Skip to content

08 · Reproducible Research Practices

"It worked when I ran it" isn't the bar — the bar is "someone else (or you, in a year) can rerun this and get the same numbers." This module covers the concrete practices that make an analysis reproducible: project structure, dependency pinning, seeding, and data versioning.

A reproducible project layout

churn-analysis/
├── data/
│   ├── raw/              # never edited by hand, never overwritten
│   └── processed/        # generated by scripts, safe to delete & regenerate
├── notebooks/
│   └── 01-exploration.ipynb
├── src/
│   ├── clean.py
│   ├── features.py
│   └── model.py
├── outputs/
│   ├── figures/
│   └── models/
├── requirements.txt
├── README.md
└── run_pipeline.py

The separation between raw (immutable, treated as read-only) and processed (fully regenerable from raw + code) is the core discipline: if processed/ and outputs/ can be deleted and rebuilt by running run_pipeline.py, the project is reproducible by construction.

Pin the environment, not just the packages

# requirements.txt — exact versions, not ranges
pandas==2.2.2
numpy==1.26.4
scikit-learn==1.4.2
statsmodels==0.14.2

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
pip freeze > requirements-lock.txt   # exact resolved versions, including transitive deps

pandas>=2.0 looks reasonable but silently lets a future install pull a version with different default behavior (pandas has changed defaults across major versions before). Pin exact versions for anything you expect someone to rerun later, and keep a full pip freeze lock file alongside the hand-written requirements for full transitive-dependency reproducibility.

Seed every source of randomness

import random
import numpy as np

SEED = 42

def set_all_seeds(seed: int = SEED) -> None:
    random.seed(seed)
    np.random.seed(seed)
    # If using scikit-learn estimators with random_state, pass seed explicitly there too
    # If using TensorFlow/PyTorch, seed those libraries' generators as well

set_all_seeds()

A script that calls set_all_seeds() once at the top, and passes random_state=SEED to every scikit-learn estimator and train/test split, removes "I can't reproduce your number" as a possible failure mode. Note that seeding doesn't guarantee bit-identical results across different library versions or hardware (some GPU operations are non-deterministic) — which is exactly why environment pinning matters too.

Version your data, not just your code

import hashlib

def file_hash(path: str) -> str:
    with open(path, "rb") as f:
        return hashlib.sha256(f.read()).hexdigest()[:12]

print(file_hash("data/raw/transactions.csv"))

a1f9c3e07b2d

Recording a hash of each raw input file (in the README, a manifest file, or a tool like DVC) answers the question "was this analysis run against the same data I have now?" — git tracks code changes well but is a poor fit for large data files; a hash-based manifest (or DVC/git-lfs for the files themselves) closes that gap cheaply.

A single entry point that reruns everything

# run_pipeline.py
from src.clean import clean_raw_data
from src.features import build_features
from src.model import train_and_evaluate

def main():
    set_all_seeds()
    raw = clean_raw_data("data/raw/transactions.csv")
    features = build_features(raw)
    metrics = train_and_evaluate(features)
    print("Final metrics:", metrics)

if __name__ == "__main__":
    main()

python run_pipeline.py

If a colleague can clone the repo, run pip install -r requirements.txt, then python run_pipeline.py, and get the same metrics you reported — the analysis is reproducible. If reproducing it requires "also run cells 3, 7, then 5 in this notebook in that order, and don't forget to manually download this other file" — it isn't, no matter how correct the numbers are.

Document assumptions and known caveats

## Known limitations (README.md excerpt)
- Data covers Jan–Jun 2024 only; seasonal effects beyond H1 are extrapolated.
- Customers with < 3 transactions were excluded (see src/clean.py:excludes_low_activity).
- Churn label defined as "no purchase in 60 days" — a stricter/looser
  window would change absolute churn rates but not the relative comparison.

A short, explicit "known limitations" section costs a few minutes to write and prevents a reader from unknowingly over-trusting a result outside the range it's valid for — arguably as important to reproducibility as the code itself, since it tells a re-runner what not to conclude even from a bit-identical rerun.

Cheat sheet

Practice What it guards against
raw/ vs processed/ separation Accidentally overwriting original data
Pinned requirements.txt + lock file "Works on my machine" dependency drift
Seed every random operation Non-reproducible numbers
Hash or version raw data files Unnoticed silent data changes
One pipeline entry point Manual, undocumented multi-step reruns
Documented limitations Results misapplied outside their valid scope

How It Actually Works

Random seeding works because "randomness" in software is almost always pseudorandom: a deterministic algorithm (a linear congruential generator, or NumPy's PCG64) generates a long sequence of numbers that only looks statistically random, entirely determined by an internal state. Calling np.random.seed(42) initializes that internal state to a fixed, known starting point, so every subsequent "random" draw follows the exact same sequence every time the program runs — reproducibility here isn't suppressing randomness, it's making the specific pseudo-random sequence identical across runs. This is also precisely why seeding doesn't survive a library version change: if the underlying generator algorithm itself changes between versions (or NumPy switches its default bit generator, as it did between legacy RandomState and the newer Generator API), the same seed produces a different sequence — the seed pins a position in an algorithm's output stream, not an absolute set of values.

Hashing raw data files (sha256) works because a cryptographic hash function maps an input of any size to a fixed-size fingerprint such that changing even a single byte of the input changes the output completely and unpredictably (the avalanche effect) — there's no way to alter the file while keeping the same hash by chance. This makes a hash an extremely cheap way to answer "is this literally the same file" without diffing the whole file's contents or trusting a filename/timestamp (which say nothing about content) — it's the same integrity-checking mechanism used to verify downloaded software hasn't been corrupted or tampered with, applied here to catch a silently-replaced or accidentally-edited raw dataset.

Environment pinning matters mechanically because pip's dependency resolver, given a range like pandas>=2.0, will install whatever the newest version satisfying that range happens to be at install time — which is a moving target. Two people running pip install -r requirements.txt a year apart can silently get different pandas versions with different default arguments (pandas has changed things like default observed= behavior in groupby across versions), producing different numeric output from identical code. pip freeze captures the fully resolved dependency graph (including transitive dependencies your requirements.txt never names directly) at a point in time, which is the only way to guarantee the exact same code runs against the exact same library versions later.

The raw/processed separation is a specific application of idempotency: because processed/ is produced purely as a pure function of raw/ + code (no manual edits ever touch it), deleting and regenerating it from run_pipeline.py is guaranteed to be safe — there is no hidden state living only in the processed files that the code doesn't know how to recreate. The moment someone hand-edits a value inside processed/, that guarantee is broken and the project silently stops being reproducible from its own inputs.

Exercise

Take a notebook you've written for an earlier module in this path. Refactor it into the src/ + run_pipeline.py structure shown above: move any function longer than ~10 lines into a .py module, add a requirements.txt with pinned versions, and confirm python run_pipeline.py reproduces the notebook's final numbers exactly.