atelier/domains/ai-ml/data-versioning.md

Data Versioning — Derived Rules

Derives from domains/ai-ml/first-principles.md. Covers P2 (Data is Versioned, Not Just Code) and P3 (Lineage is Traceable End-to-End). Referenced by serving.md and monitoring-drift.md. Scope per D-023: engineering discipline of versioning data, not dataset content design.

Why Data Versioning (P2 Data is Versioned, Not Just Code)

• git versions code well and data badly. Datasets do not fit in git, and a dataset is not recovered from a commit hash.
• A model trained on "the data" is a model trained on an unknown input — a C1 (Correctness) violation. The dataset is a build input; it is named, hashed, and recoverable the way any build input is.
• Data versioning is the ML analogue of domains/data/migrations.md: the schema and contents of the data evolve, every evolution is a versioned migration, and every model points at a specific version.

Dataset Hashing and Lineage (P3 Lineage Traceable End-to-End)

• Every dataset version has a content hash (not a filename or a timestamp). The hash is the identity. A model's lineage record names the dataset hash it was trained on; a serving prediction names the model digest it came from.
• Lineage is a graph: prediction → model → training run → dataset → source(s). Any edge missing is an orphan (domains/observability/logging.md for the structured-log angle on lineage events).
• The lineage record is append-only. Editing it to "fix" a broken trace is the same class of violation as editing an audit log.

Train/Val/Test Split Versioning (P2, P4 Eval Defined Before Training)

• Splits are versioned with the dataset, not derived ad-hoc per run. A split is a deterministic function of (dataset version, split config, random seed). Two runs on the same pinned inputs produce the same splits.
• The eval split is held out and never touched by training. A "held out" set that leaked into training is a P4 (Evaluation Defined Before Training) violation, not just a P2 violation — the eval gate is measuring the training set, not the model.
• Cross domains/data/schema-design.md for the eval input contract: the schema of the eval set is part of the versioned artifact.

Tool Comparison (IDEATE-22, D-040)

Tool	Versioning Model	Lineage	Best For	Notes
DVC	Git-like pointers to content-addressed object store; .dvc files in git track data versions	Pipeline DAG in dvc.yaml; reproducibility via dvc repro	Teams already on git; file/directory datasets; ML pipelines	Treats data like code; shares git's history model. Object store is pluggable (S3, GCS, Azure, SSH)
Delta Lake	Table format with transaction log (ACID) + time travel via versioned commits; schema enforcement	Time travel queries; lineage via table history + catalog	Large tabular data; lakehouse; streaming + batch on the same table	Not a pipeline tool — pairs with Spark/Trino/Flink. Brings DB guarantees to object storage
LakeFS	Git-like operations (branch, commit, merge) over object storage itself	Branch model gives isolated, reproducible data branches	Data engineering teams; branch-per-experiment; CI over data	Not a table format — versions objects. Composes with Delta/Iceberg on top

• Pick one primary versioning model per platform. Mixing DVC's pointer model with Delta's transaction-log model fragments operational knowledge (C4 Locality).
• All three satisfy P2; the choice is which fits the data shape and the team's existing tooling. None is advocated over the others.

Reproducibility Contract (P1 Reproducibility is the First Class)

A reproducible training run records, in one versioned place:

run_id:        2026-08-05T09:12:00Z#run-42
dataset:       s3://ml-data/train@sha256:7f3a...e21
splits:        dvc.yaml@commit a1b2c4d
code:          git@a1b2c4d
config:        configs/train.yaml@commit a1b2c4d
environment:   ghcr.io/org/train-img@sha256:9c2d...f88
eval_spec:     configs/eval.yaml@commit a1b2c4d
model_digest:  registry/model@sha256:b5e1...aa0

• Lose any line and the run is anecdote, not evidence.
• The record is the lineage root: a prediction cites the model_digest, which cites the run_id, which cites everything above. This is how P3 (Lineage Traceable End-to-End) is satisfied in practice.

What Violates Data Versioning Discipline

Violation	Principle
Dataset referenced by s3://bucket/latest/	P2 Data is Versioned, Not Just Code
Splits regenerated with an unpinned seed per run	P2, P4 Evaluation Defined Before Training
A production model with no dataset hash in its lineage	P3 Lineage Traceable End-to-End
Editing a lineage record to "clean up" a broken trace	P3 Lineage Traceable End-to-End
Eval split reachable from the training data path	P4 Evaluation Defined Before Training
Two platforms versioning the same data with different models	C4 Locality