atelier/examples/good/ai-ml-reproducibility.md

# Good Example: AI/ML Reproducible Training Run

> A training run that follows Atelier's AI/ML principles. Each aspect
> cites the principle it satisfies. Scope per D-023: this is
> engineering discipline (reproducibility, versioning, lineage,
> serving), **not** algorithm or model design — no architecture
> choice, hyperparameter tuning, or model-family comparison appears
> here.

## The Run

A training run `2026-08-05T09:12:00Z#run-42` produces model
`registry/payments-fraud@sha256:b5e1...aa0`. Every input that shaped
the model is pinned, named, and recoverable; the eval was declared
before training; the model is an addressed artifact in a registry;
the rollback path names the prior model and the prior dataset.

### The Reproducibility Contract

```yaml
# lineage/run-42.yaml — the lineage root, committed alongside the code
run_id:        2026-08-05T09:12:00Z#run-42
dataset:       s3://ml-data/train@sha256:7f3a...e21
splits:        dvc.yaml@commit a1b2c4d
code:          git@a1b2c4d
config:        configs/train.yaml@commit a1b2c4d
environment:   ghcr.io/org/train-img@sha256:9c2d...f88
eval_spec:     configs/eval.yaml@commit a1b2c4d
model_digest:  registry/payments-fraud@sha256:b5e1...aa0
status:        passed  # eval gate passed -> eligible for promotion
```

- Lose any line and the run is anecdote, not evidence. The record is
  the lineage root: a prediction cites the `model_digest`, which
  cites this `run_id`, which cites everything above.

### Data is Versioned (DVC, content-hashed)

```ini
# dvc.yaml — the split config is versioned in git, the data in the
# content-addressed object store. Both are pinned by commit + hash.
stages:
  prepare:
    cmd: python src/prepare.py --input data/raw --out data/splits
    deps:
      - data/raw
      - src/prepare.py
    outs:
      - data/splits/train.parquet
      - data/splits/val.parquet
      - data/splits/test.parquet
    # The dataset hash (sha256:7f3a...e21) is recorded in the lineage
    # contract above. "s3://ml-data/latest" would be a P2 violation.
```

```bash
# The dataset is pinned by content hash, not by a mutable path.
$ dvc get s3://ml-data/train --rev sha256:7f3a...e21
# The split is a deterministic function of (dataset version, split
# config, random seed). Two runs on the same pinned inputs produce
# the same splits.
```

### Code and Config are Versioned (git)

```yaml
# configs/train.yaml@commit a1b2c4d — versioned with the code
# (No algorithm/hyperparameter content is illustrated here — this is
#  the engineering discipline of pinning the config, not the model
#  design inside it. Per D-023, algorithm choice is out of scope.)
seed: 42
splits:
  train: data/splits/train.parquet
  val:   data/splits/val.parquet
  test:  data/splits/test.parquet   # held out, never touched by training
```

### Environment is Pinned (container digest)

```dockerfile
# The training environment is an image addressed by digest, not :latest.
# ghcr.io/org/train-img@sha256:9c2d...f88
FROM python:3.11-slim
# dependencies pinned in requirements.txt with hashes
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
```

```text
# requirements.txt — pinned + hash-pinned (pip-compile / pip-audit)
dvc==3.50.2 \
    --hash=sha256:1c8a...e7
mlflow==2.16.0 \
    --hash=sha256:9b2f...a1
# No unpinned ranges. A rerun pulls the exact same wheels.
```

### Evaluation is Defined Before Training (P4)

```yaml
# configs/eval.yaml@commit a1b2c4d — committed BEFORE training runs.
# The metrics, splits, and pass/fail thresholds are a-priori; they
# are the contract the model must satisfy to leave the experiment.
metrics:
  - name: precision_at_threshold
    threshold: ">= 0.92"
  - name: recall_at_threshold
    threshold: ">= 0.85"
  - name: false_positive_rate
    threshold: "<= 0.03"
split: data/splits/test.parquet   # held out, never in training
gate: all_metrics_pass            # AND of all thresholds; no cherry-pick
# The eval schema equals the serving input contract (serving.md P8):
# feature names, types, ranges match the production boundary exactly.
```

- Metrics chosen after seeing scores would be a P4 violation: the eval
  would be rationalizing, not measuring. See
  `domains/ai-ml/model-evaluation.md`.

### The Model is a Versioned Artifact (MLflow registry)

```bash
# After the eval gate passes, the model is registered as an immutable
# artifact addressed by digest, then promoted by stage.
$ mlflow models register \
    --name payments-fraud \
    --model-uri runs:/run-42/model \
    --description "run-42, dataset sha256:7f3a...e21, eval passed"
# registry/payments-fraud@sha256:b5e1...aa0
# Stages: None -> Staging -> Production. Promotion is a registry
# operation, not a file copy. Never "latest".
```

### The Pipeline Composes (P9)

```text
# The training flow is a pipeline with explicit stages and contracts,
# not a notebook. Each stage has named inputs and named outputs.
prepare(dataset@hash) -> split(dvc.yaml) -> train(config, env@digest)
  -> eval(eval.yaml, test@hash) -> [gate: pass] -> register(model@digest)
                                  |
                                  +-> [gate: fail] -> abort, no promote
# A notebook in this path would be a P9 violation: implicit state,
# human-dependent order, unreproducible.
```

## What Makes It Good

### Reproducibility is First Class (AI/ML P1, C1, C5)
- data + code + config + environment are all pinned. A second
  engineer on a second laptop checks out commit `a1b2c4d`, pulls the
  dataset by hash, pulls the image by digest, and reproduces the run
  bit-for-bit. The run is reviewable because it is recreatable.
- See `domains/ai-ml/first-principles.md` P1 and
  `domains/devops/first-principles.md` P1 Reproducibility.

### Data is Versioned, Not Just Code (AI/ML P2, C5, C7)
- The dataset is `s3://ml-data/train@sha256:7f3a...e21`, not
  `s3://ml-data/latest`. A model trained on "the data" is a model
  trained on an unknown input — a C1 violation. DVC pins the data the
  way git pins the code.
- See `domains/ai-ml/data-versioning.md` (dataset hashing, the DVC /
  Delta Lake / LakeFS comparison) and `domains/data/migrations.md`.

### Lineage is Traceable End-to-End (AI/ML P3, C7, C1)
- prediction → model → run-42 → dataset → source. Every edge is
  named; no orphan model. A serving regression traces back to the
  exact dataset and code that built the model, which is how drift is
  diagnosed (data drift vs concept drift vs prediction drift).
- See `domains/ai-ml/data-versioning.md` (lineage record) and
  `domains/observability/logging.md`.

### Evaluation Defined Before Training (AI/ML P4, C1, C2)
- `eval.yaml` was committed before `train` ran. The gate is
  `all_metrics_pass`; a failing metric aborts promotion. Cherry-
  picking a metric post-hoc is a correctness violation — the eval
  would no longer measure the model.
- See `domains/ai-ml/model-evaluation.md` (eval-as-a-gate) and
  `domains/testing/first-principles.md` (tests as specification).

### Models are Versioned Artifacts (AI/ML P5, C5, C6)
- The model is `registry/payments-fraud@sha256:b5e1...aa0`, promoted
  Staging → Production. A serving endpoint that pulled `latest` would
  be serving an unknown model with no rollback. The registry is to
  models what a container registry is to images.
- See `domains/ai-ml/serving.md` (the model is an addressed artifact)
  and `domains/devops/first-principles.md` P7 Immutability.

### Rollback Includes the Model (AI/ML P10, C5)
- If production regresses, the rollback restores the prior model
  digest `registry/payments-fraud@sha256:a1c4...f09` AND the prior
  serving code. A rollback that redeploys old code but keeps the new
  model has not rolled back — the model was the thing that regressed.
- See `domains/ai-ml/serving.md` (Rollback Includes the Model) and
  `domains/devops/first-principles.md` P4 Rollback First.

## What This Example Does NOT Do (And Why That's Good)

- Does **not** reference the dataset by a mutable path —
  `s3://ml-data/latest` would be a P2 violation.
- Does **not** choose metrics after seeing scores — that is a P4
  violation (rationalizing, not measuring).
- Does **not** pull `latest` from the model registry — that is a P5
  violation (unknown model, no rollback).
- Does **not** contain algorithm/architecture/hyperparameter content
  — per D-023, those are research choices, not engineering
  principles, and have no derivation in the core C-rules.
- Does **not** run from a notebook — a notebook in the pipeline path
  is a P9 violation (implicit state, unreproducible).

## Cross-Domain Links

- `domains/ai-ml/data-versioning.md` — the DVC pinning, the lineage
  record, the tool comparison (DVC / Delta Lake / LakeFS).
- `domains/ai-ml/serving.md` — the model is promoted as an addressed
  artifact; the serving boundary validates inputs against the same
  schema as the eval.
- `domains/ai-ml/model-evaluation.md` — the eval-as-a-gate that this
  run must pass before promotion.
- `domains/devops/first-principles.md` P1 Reproducibility — the
  non-negotiable this run inherits.
- `domains/data/migrations.md` — data versioning parallels schema
  migration discipline.
- `domains/observability/logging.md` — the lineage record is a
  structured, append-only log of provenance.