Jon Chery
9ebc9c8868
---ci--- project: atelier phase: 6 milestone: v0.3 status: complete requirements: covered: [ATELIER-60..91] partial: [] ---/ci---
90 lines
4.9 KiB
Markdown
90 lines
4.9 KiB
Markdown
# Data Versioning — Derived Rules
|
|
|
|
> Derives from `domains/ai-ml/first-principles.md`. Covers P2 (Data is
|
|
> Versioned, Not Just Code) and P3 (Lineage is Traceable End-to-End).
|
|
> Referenced by `serving.md` and `monitoring-drift.md`. Scope per
|
|
> D-023: engineering discipline of versioning data, not dataset
|
|
> content design.
|
|
|
|
## Why Data Versioning (P2 Data is Versioned, Not Just Code)
|
|
|
|
- `git` versions code well and data badly. Datasets do not fit in
|
|
git, and a dataset is not recovered from a commit hash.
|
|
- A model trained on "the data" is a model trained on an unknown
|
|
input — a C1 (Correctness) violation. The dataset is a build
|
|
input; it is named, hashed, and recoverable the way any build
|
|
input is.
|
|
- Data versioning is the ML analogue of `domains/data/migrations.md`:
|
|
the schema and contents of the data evolve, every evolution is a
|
|
versioned migration, and every model points at a specific version.
|
|
|
|
## Dataset Hashing and Lineage (P3 Lineage Traceable End-to-End)
|
|
|
|
- Every dataset version has a content hash (not a filename or a
|
|
timestamp). The hash is the identity. A model's lineage record
|
|
names the dataset hash it was trained on; a serving prediction
|
|
names the model digest it came from.
|
|
- Lineage is a graph: prediction → model → training run → dataset →
|
|
source(s). Any edge missing is an orphan (`domains/observability/logging.md`
|
|
for the structured-log angle on lineage events).
|
|
- The lineage record is append-only. Editing it to "fix" a broken
|
|
trace is the same class of violation as editing an audit log.
|
|
|
|
## Train/Val/Test Split Versioning (P2, P4 Eval Defined Before Training)
|
|
|
|
- Splits are versioned with the dataset, not derived ad-hoc per run.
|
|
A split is a deterministic function of (dataset version, split
|
|
config, random seed). Two runs on the same pinned inputs produce
|
|
the same splits.
|
|
- The eval split is held out and never touched by training. A "held
|
|
out" set that leaked into training is a P4 (Evaluation Defined
|
|
Before Training) violation, not just a P2 violation — the eval
|
|
gate is measuring the training set, not the model.
|
|
- Cross `domains/data/schema-design.md` for the eval input contract:
|
|
the schema of the eval set is part of the versioned artifact.
|
|
|
|
## Tool Comparison (IDEATE-22, D-040)
|
|
|
|
| Tool | Versioning Model | Lineage | Best For | Notes |
|
|
|------|------------------|---------|----------|-------|
|
|
| DVC | Git-like pointers to content-addressed object store; `.dvc` files in git track data versions | Pipeline DAG in `dvc.yaml`; reproducibility via `dvc repro` | Teams already on git; file/directory datasets; ML pipelines | Treats data like code; shares git's history model. Object store is pluggable (S3, GCS, Azure, SSH) |
|
|
| Delta Lake | Table format with transaction log (ACID) + time travel via versioned commits; schema enforcement | Time travel queries; lineage via table history + catalog | Large tabular data; lakehouse; streaming + batch on the same table | Not a pipeline tool — pairs with Spark/Trino/Flink. Brings DB guarantees to object storage |
|
|
| LakeFS | Git-like operations (branch, commit, merge) over object storage itself | Branch model gives isolated, reproducible data branches | Data engineering teams; branch-per-experiment; CI over data | Not a table format — versions objects. Composes with Delta/Iceberg on top |
|
|
|
|
- Pick one primary versioning model per platform. Mixing DVC's
|
|
pointer model with Delta's transaction-log model fragments
|
|
operational knowledge (C4 Locality).
|
|
- All three satisfy P2; the choice is which fits the data shape and
|
|
the team's existing tooling. None is advocated over the others.
|
|
|
|
## Reproducibility Contract (P1 Reproducibility is the First Class)
|
|
|
|
A reproducible training run records, in one versioned place:
|
|
|
|
```
|
|
run_id: 2026-08-05T09:12:00Z#run-42
|
|
dataset: s3://ml-data/train@sha256:7f3a...e21
|
|
splits: dvc.yaml@commit a1b2c4d
|
|
code: git@a1b2c4d
|
|
config: configs/train.yaml@commit a1b2c4d
|
|
environment: ghcr.io/org/train-img@sha256:9c2d...f88
|
|
eval_spec: configs/eval.yaml@commit a1b2c4d
|
|
model_digest: registry/model@sha256:b5e1...aa0
|
|
```
|
|
|
|
- Lose any line and the run is anecdote, not evidence.
|
|
- The record is the lineage root: a prediction cites the
|
|
`model_digest`, which cites the `run_id`, which cites everything
|
|
above. This is how P3 (Lineage Traceable End-to-End) is satisfied
|
|
in practice.
|
|
|
|
## What Violates Data Versioning Discipline
|
|
|
|
| Violation | Principle |
|
|
|-----------|-----------|
|
|
| Dataset referenced by `s3://bucket/latest/` | P2 Data is Versioned, Not Just Code |
|
|
| Splits regenerated with an unpinned seed per run | P2, P4 Evaluation Defined Before Training |
|
|
| A production model with no dataset hash in its lineage | P3 Lineage Traceable End-to-End |
|
|
| Editing a lineage record to "clean up" a broken trace | P3 Lineage Traceable End-to-End |
|
|
| Eval split reachable from the training data path | P4 Evaluation Defined Before Training |
|
|
| Two platforms versioning the same data with different models | C4 Locality | |