atelier/examples/good/ai-ml-reproducibility.md

Good Example: AI/ML Reproducible Training Run

A training run that follows Atelier's AI/ML principles. Each aspect cites the principle it satisfies. Scope per D-023: this is engineering discipline (reproducibility, versioning, lineage, serving), not algorithm or model design — no architecture choice, hyperparameter tuning, or model-family comparison appears here.

The Run

A training run 2026-08-05T09:12:00Z#run-42 produces model registry/payments-fraud@sha256:b5e1...aa0. Every input that shaped the model is pinned, named, and recoverable; the eval was declared before training; the model is an addressed artifact in a registry; the rollback path names the prior model and the prior dataset.

The Reproducibility Contract

# lineage/run-42.yaml — the lineage root, committed alongside the code
run_id:        2026-08-05T09:12:00Z#run-42
dataset:       s3://ml-data/train@sha256:7f3a...e21
splits:        dvc.yaml@commit a1b2c4d
code:          git@a1b2c4d
config:        configs/train.yaml@commit a1b2c4d
environment:   ghcr.io/org/train-img@sha256:9c2d...f88
eval_spec:     configs/eval.yaml@commit a1b2c4d
model_digest:  registry/payments-fraud@sha256:b5e1...aa0
status:        passed  # eval gate passed -> eligible for promotion

• Lose any line and the run is anecdote, not evidence. The record is the lineage root: a prediction cites the model_digest, which cites this run_id, which cites everything above.

Data is Versioned (DVC, content-hashed)

# dvc.yaml — the split config is versioned in git, the data in the
# content-addressed object store. Both are pinned by commit + hash.
stages:
  prepare:
    cmd: python src/prepare.py --input data/raw --out data/splits
    deps:
      - data/raw
      - src/prepare.py
    outs:
      - data/splits/train.parquet
      - data/splits/val.parquet
      - data/splits/test.parquet
    # The dataset hash (sha256:7f3a...e21) is recorded in the lineage
    # contract above. "s3://ml-data/latest" would be a P2 violation.

# The dataset is pinned by content hash, not by a mutable path.
$ dvc get s3://ml-data/train --rev sha256:7f3a...e21
# The split is a deterministic function of (dataset version, split
# config, random seed). Two runs on the same pinned inputs produce
# the same splits.

Code and Config are Versioned (git)

# configs/train.yaml@commit a1b2c4d — versioned with the code
# (No algorithm/hyperparameter content is illustrated here — this is
#  the engineering discipline of pinning the config, not the model
#  design inside it. Per D-023, algorithm choice is out of scope.)
seed: 42
splits:
  train: data/splits/train.parquet
  val:   data/splits/val.parquet
  test:  data/splits/test.parquet   # held out, never touched by training

Environment is Pinned (container digest)

# The training environment is an image addressed by digest, not :latest.
# ghcr.io/org/train-img@sha256:9c2d...f88
FROM python:3.11-slim
# dependencies pinned in requirements.txt with hashes
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# requirements.txt — pinned + hash-pinned (pip-compile / pip-audit)
dvc==3.50.2 \
    --hash=sha256:1c8a...e7
mlflow==2.16.0 \
    --hash=sha256:9b2f...a1
# No unpinned ranges. A rerun pulls the exact same wheels.

Evaluation is Defined Before Training (P4)

# configs/eval.yaml@commit a1b2c4d — committed BEFORE training runs.
# The metrics, splits, and pass/fail thresholds are a-priori; they
# are the contract the model must satisfy to leave the experiment.
metrics:
  - name: precision_at_threshold
    threshold: ">= 0.92"
  - name: recall_at_threshold
    threshold: ">= 0.85"
  - name: false_positive_rate
    threshold: "<= 0.03"
split: data/splits/test.parquet   # held out, never in training
gate: all_metrics_pass            # AND of all thresholds; no cherry-pick
# The eval schema equals the serving input contract (serving.md P8):
# feature names, types, ranges match the production boundary exactly.

• Metrics chosen after seeing scores would be a P4 violation: the eval would be rationalizing, not measuring. See domains/ai-ml/model-evaluation.md.

The Model is a Versioned Artifact (MLflow registry)

# After the eval gate passes, the model is registered as an immutable
# artifact addressed by digest, then promoted by stage.
$ mlflow models register \
    --name payments-fraud \
    --model-uri runs:/run-42/model \
    --description "run-42, dataset sha256:7f3a...e21, eval passed"
# registry/payments-fraud@sha256:b5e1...aa0
# Stages: None -> Staging -> Production. Promotion is a registry
# operation, not a file copy. Never "latest".

The Pipeline Composes (P9)

# The training flow is a pipeline with explicit stages and contracts,
# not a notebook. Each stage has named inputs and named outputs.
prepare(dataset@hash) -> split(dvc.yaml) -> train(config, env@digest)
  -> eval(eval.yaml, test@hash) -> [gate: pass] -> register(model@digest)
                                  |
                                  +-> [gate: fail] -> abort, no promote
# A notebook in this path would be a P9 violation: implicit state,
# human-dependent order, unreproducible.

What Makes It Good

Reproducibility is First Class (AI/ML P1, C1, C5)

• data + code + config + environment are all pinned. A second engineer on a second laptop checks out commit a1b2c4d, pulls the dataset by hash, pulls the image by digest, and reproduces the run bit-for-bit. The run is reviewable because it is recreatable.
• See domains/ai-ml/first-principles.md P1 and domains/devops/first-principles.md P1 Reproducibility.

Data is Versioned, Not Just Code (AI/ML P2, C5, C7)

• The dataset is s3://ml-data/train@sha256:7f3a...e21, not s3://ml-data/latest. A model trained on "the data" is a model trained on an unknown input — a C1 violation. DVC pins the data the way git pins the code.
• See domains/ai-ml/data-versioning.md (dataset hashing, the DVC / Delta Lake / LakeFS comparison) and domains/data/migrations.md.

Lineage is Traceable End-to-End (AI/ML P3, C7, C1)

• prediction → model → run-42 → dataset → source. Every edge is named; no orphan model. A serving regression traces back to the exact dataset and code that built the model, which is how drift is diagnosed (data drift vs concept drift vs prediction drift).
• See domains/ai-ml/data-versioning.md (lineage record) and domains/observability/logging.md.

Evaluation Defined Before Training (AI/ML P4, C1, C2)

• eval.yaml was committed before train ran. The gate is all_metrics_pass; a failing metric aborts promotion. Cherry- picking a metric post-hoc is a correctness violation — the eval would no longer measure the model.
• See domains/ai-ml/model-evaluation.md (eval-as-a-gate) and domains/testing/first-principles.md (tests as specification).

Models are Versioned Artifacts (AI/ML P5, C5, C6)

• The model is registry/payments-fraud@sha256:b5e1...aa0, promoted Staging → Production. A serving endpoint that pulled latest would be serving an unknown model with no rollback. The registry is to models what a container registry is to images.
• See domains/ai-ml/serving.md (the model is an addressed artifact) and domains/devops/first-principles.md P7 Immutability.

Rollback Includes the Model (AI/ML P10, C5)

• If production regresses, the rollback restores the prior model digest registry/payments-fraud@sha256:a1c4...f09 AND the prior serving code. A rollback that redeploys old code but keeps the new model has not rolled back — the model was the thing that regressed.
• See domains/ai-ml/serving.md (Rollback Includes the Model) and domains/devops/first-principles.md P4 Rollback First.

What This Example Does NOT Do (And Why That's Good)

• Does not reference the dataset by a mutable path — s3://ml-data/latest would be a P2 violation.
• Does not choose metrics after seeing scores — that is a P4 violation (rationalizing, not measuring).
• Does not pull latest from the model registry — that is a P5 violation (unknown model, no rollback).
• Does not contain algorithm/architecture/hyperparameter content — per D-023, those are research choices, not engineering principles, and have no derivation in the core C-rules.
• Does not run from a notebook — a notebook in the pipeline path is a P9 violation (implicit state, unreproducible).

Cross-Domain Links

• domains/ai-ml/data-versioning.md — the DVC pinning, the lineage record, the tool comparison (DVC / Delta Lake / LakeFS).
• domains/ai-ml/serving.md — the model is promoted as an addressed artifact; the serving boundary validates inputs against the same schema as the eval.
• domains/ai-ml/model-evaluation.md — the eval-as-a-gate that this run must pass before promotion.
• domains/devops/first-principles.md P1 Reproducibility — the non-negotiable this run inherits.
• domains/data/migrations.md — data versioning parallels schema migration discipline.
• domains/observability/logging.md — the lineage record is a structured, append-only log of provenance.