docs(P2): complete ai-ml domain phase

---ci--- project: atelier phase: 2 milestone: v0.3 status: complete requirements: covered: [ATELIER-65, ATELIER-66, ATELIER-67, ATELIER-68, ATELIER-69] partial: [] ---/ci---
docs(ship): P1 complete — checkpoint + roadmap status
2026-08-05 03:24:13 +00:00 · 2026-08-05 03:21:04 +00:00
7 changed files with 520 additions and 6 deletions
@@ -1,11 +1,11 @@
 {
-  "phase": 0,
+  "phase": 1,
  "stage": "complete",
  "milestone": "v0.3",
-  "phase_role": "pre_execution",
+  "phase_role": "execution",
  "project": "atelier",
  "attempts": 0,
-  "updated_at": "2026-08-05T03:17:00Z",
+  "updated_at": "2026-08-05T03:25:00Z",
-  "phase_tag": "v0.2.0",
+  "phase_tag": "v0.2.1",
-  "release_id": 468
+  "release_id": 469
 }
@@ -85,7 +85,7 @@ NFR milestone: no separate minor tag. The final patch (v0.1.5) IS the v0.2 deliv
 | Phase | Name | Type | Status | Key Deliverables |
 |-------|------|------|--------|------------------|
 | 0 | Pre-Execution | docs | complete | Spec, clarify, research, ideate, plan, PERSONAS.md (extends platform-engineer, adds ml-engineer) — shipped v0.2.0 |
-| 1 | GitOps + Operators Domain | docs | pending | domains/gitops-operators/{first-principles, argocd, flux, operators, progressive-delivery}.md |
+| 1 | GitOps + Operators Domain | docs | complete | domains/gitops-operators/{first-principles, argocd, flux, operators, progressive-delivery}.md — shipped v0.2.1 |
 | 2 | AI/ML Domain | docs | pending | domains/ai-ml/{first-principles, data-versioning, model-evaluation, serving, monitoring-drift}.md |
 | 3 | i18n + Compliance Domains | docs | pending | domains/i18n/{first-principles, locale-resources, formatting, rtl-bidi, testing-i18n}.md, domains/compliance/{first-principles, audit-logs, data-retention, policy-as-code, evidence}.md |
 | 4 | Matrix + Review Integration | docs | pending | matrix/principles-matrix.md (+40 mappings), matrix/domain-coverage.md (incl. C-rule coverage table update), review/{agent-checklist, peer-review-checklist, anti-patterns}.md, MANIFEST.md (+ examples/ listing per ATELIER-91) |
@@ -0,0 +1,90 @@
 # Data Versioning — Derived Rules
 > Derives from `domains/ai-ml/first-principles.md`. Covers P2 (Data is
 > Versioned, Not Just Code) and P3 (Lineage is Traceable End-to-End).
 > Referenced by `serving.md` and `monitoring-drift.md`. Scope per
 > D-023: engineering discipline of versioning data, not dataset
 > content design.
 ## Why Data Versioning (P2 Data is Versioned, Not Just Code)
 - `git` versions code well and data badly. Datasets do not fit in
  git, and a dataset is not recovered from a commit hash.
 - A model trained on "the data" is a model trained on an unknown
  input — a C1 (Correctness) violation. The dataset is a build
  input; it is named, hashed, and recoverable the way any build
  input is.
 - Data versioning is the ML analogue of `domains/data/migrations.md`:
  the schema and contents of the data evolve, every evolution is a
  versioned migration, and every model points at a specific version.
 ## Dataset Hashing and Lineage (P3 Lineage Traceable End-to-End)
 - Every dataset version has a content hash (not a filename or a
  timestamp). The hash is the identity. A model's lineage record
  names the dataset hash it was trained on; a serving prediction
  names the model digest it came from.
 - Lineage is a graph: prediction → model → training run → dataset →
  source(s). Any edge missing is an orphan (`domains/observability/logging.md`
  for the structured-log angle on lineage events).
 - The lineage record is append-only. Editing it to "fix" a broken
  trace is the same class of violation as editing an audit log.
 ## Train/Val/Test Split Versioning (P2, P4 Eval Defined Before Training)
 - Splits are versioned with the dataset, not derived ad-hoc per run.
  A split is a deterministic function of (dataset version, split
  config, random seed). Two runs on the same pinned inputs produce
  the same splits.
 - The eval split is held out and never touched by training. A "held
  out" set that leaked into training is a P4 (Evaluation Defined
  Before Training) violation, not just a P2 violation — the eval
  gate is measuring the training set, not the model.
 - Cross `domains/data/schema-design.md` for the eval input contract:
  the schema of the eval set is part of the versioned artifact.
 ## Tool Comparison (IDEATE-22, D-040)
 | Tool | Versioning Model | Lineage | Best For | Notes |
 |------|------------------|---------|----------|-------|
 | DVC | Git-like pointers to content-addressed object store; `.dvc` files in git track data versions | Pipeline DAG in `dvc.yaml`; reproducibility via `dvc repro` | Teams already on git; file/directory datasets; ML pipelines | Treats data like code; shares git's history model. Object store is pluggable (S3, GCS, Azure, SSH) |
 | Delta Lake | Table format with transaction log (ACID) + time travel via versioned commits; schema enforcement | Time travel queries; lineage via table history + catalog | Large tabular data; lakehouse; streaming + batch on the same table | Not a pipeline tool — pairs with Spark/Trino/Flink. Brings DB guarantees to object storage |
 | LakeFS | Git-like operations (branch, commit, merge) over object storage itself | Branch model gives isolated, reproducible data branches | Data engineering teams; branch-per-experiment; CI over data | Not a table format — versions objects. Composes with Delta/Iceberg on top |
 - Pick one primary versioning model per platform. Mixing DVC's
  pointer model with Delta's transaction-log model fragments
  operational knowledge (C4 Locality).
 - All three satisfy P2; the choice is which fits the data shape and
  the team's existing tooling. None is advocated over the others.
 ## Reproducibility Contract (P1 Reproducibility is the First Class)
 A reproducible training run records, in one versioned place:
 ```
 run_id:        2026-08-05T09:12:00Z#run-42
 dataset:       s3://ml-data/train@sha256:7f3a...e21
 splits:        dvc.yaml@commit a1b2c4d
 code:          git@a1b2c4d
 config:        configs/train.yaml@commit a1b2c4d
 environment:   ghcr.io/org/train-img@sha256:9c2d...f88
 eval_spec:     configs/eval.yaml@commit a1b2c4d
 model_digest:  registry/model@sha256:b5e1...aa0
 ```
 - Lose any line and the run is anecdote, not evidence.
 - The record is the lineage root: a prediction cites the
  `model_digest`, which cites the `run_id`, which cites everything
  above. This is how P3 (Lineage Traceable End-to-End) is satisfied
  in practice.
 ## What Violates Data Versioning Discipline
 | Violation | Principle |
 |-----------|-----------|
 | Dataset referenced by `s3://bucket/latest/` | P2 Data is Versioned, Not Just Code |
 | Splits regenerated with an unpinned seed per run | P2, P4 Evaluation Defined Before Training |
 | A production model with no dataset hash in its lineage | P3 Lineage Traceable End-to-End |
 | Editing a lineage record to "clean up" a broken trace | P3 Lineage Traceable End-to-End |
 | Eval split reachable from the training data path | P4 Evaluation Defined Before Training |
 | Two platforms versioning the same data with different models | C4 Locality |
@@ -0,0 +1,154 @@
 # AI / ML — First Principles
 > Scope per D-023: this domain covers ML **engineering discipline** —
 > data versioning, evaluation methodology, serving patterns, and drift
 > detection. It does **not** cover algorithm design, model architecture
 > selection, hyperparameter tuning, or model-family comparison. Those
 > are research choices, not engineering principles, and they have no
 > derivation in the core C-rules.
 ## 1. The Principles
 ### P1. Reproducibility is the First Class
 Every training run is reproducible from pinned data + code + config +
 environment. An unreproducible run is an unreviewable run: you cannot
 decide whether a result is correct if you cannot recreate it.
 Reproducibility is the ML analogue of `domains/devops/P1
 Reproducibility` and inherits its non-negotiable status. Lose any one
 of data, code, config, or environment pinning, and the run is
 anecdote, not evidence.
 ### P2. Data is Versioned, Not Just Code
 Datasets, features, and train/val/test splits are first-class
 versioned artifacts with content hashes and lineage. `git` alone is
 insufficient — datasets do not fit in git, and a dataset is not a
 commit hash. A model trained on "the data" is a model trained on an
 unknown input, which is a correctness violation. Version data the way
 you version code: pinned, named, and recoverable.
 ### P3. Lineage is Traceable End-to-End
 Any deployed prediction traces back through model → training run →
 dataset → source. No orphan models. A model in production with no
 lineage is a correctness defect: you cannot reason about its failure
 modes, you cannot roll it back to a known-good dataset, and you cannot
 tell whether drift is in the model or in the data that built it.
 Lineage is the audit trail of ML (`domains/observability/logging.md`).
 ### P4. Evaluation is Defined Before Training
 Metrics, splits, and acceptance thresholds are declared a priori, in
 code, before the model is trained. Cherry-picking metrics post-hoc is
 a correctness violation: the evaluation is no longer measuring the
 model, it is rationalizing it. The eval spec is a contract — it is
 reviewable, it is versioned, and it is the gate the model must pass
 before it leaves the experiment. This is the ML angle on C2 Clarity:
 the intent of the model is obvious to its reader because the eval
 declared it first.
 ### P5. Models are Versioned Artifacts
 A model is a pinned, immutable, registry-tracked artifact with a
 unique identifier. Never "the latest." A serving endpoint that pulls
 "latest" is serving an unknown model — its behavior is undefined, its
 rollback is impossible, and its lineage is broken. The model registry
 is to models what a container registry is to images
 (`domains/devops/P7 Immutability`): immutable, addressed by digest,
 promoted by stage.
 ### P6. Serving is Observable
 Inference latency, throughput, input distributions, and prediction
 confidence are first-class signals. Silent serving is a bug. A model
 in production that emits no metrics is a model you cannot operate: you
 cannot see latency regressions, you cannot see input drift, you cannot
 see a failing downstream consumer. Observability is designed in, not
 bolted on (`domains/observability/metrics.md`).
 ### P7. Drift is Expected and Detected
 Data drift, concept drift, and prediction drift are monitored as a
 matter of course. A drift signal is an incident, not a curiosity. ML
 systems decay without code changes — the world changes under the
 model — so "no code changed" is not a defense against a serving
 regression. Detecting drift is the ML-specific form of C7
 Observability: you cannot fix a model you cannot see degrading.
 ### P8. Inference Inputs are Validated
 The model's input contract — schema, value ranges, types, and
 categorical domains — is enforced at the serving boundary.
 Out-of-contract inputs are rejected, not silently scored. Scoring an
 out-of-contract input is a correctness violation: the model's output
 is undefined for inputs outside its training distribution, and
 returning a number for it is lying to the caller. This is the ML angle
 on `domains/security/input-validation.md` and inherits C1's
 non-tradeable status.
 ### P9. Pipelines Compose, Notebooks Don't
 Training and serving flows are composable pipelines with explicit
 steps, named inputs, named outputs, and contracts between stages.
 Notebooks are for exploration, not production. A notebook in the
 serving path is a correctness defect: its state is implicit, its
 order is human-dependent, and its reproducibility is whatever the last
 operator remembered. Compose pipelines; keep notebooks in the lab.
 ### P10. Rollback Includes the Model
 A serving rollback restores the prior model artifact, not just the
 prior code. Promotion is reversible at the model layer. A rollback
 that redeploys old code but keeps the new model has not rolled back —
 the model was the thing that regressed. The rollback path must name
 the prior model digest, the prior dataset version, and the prior eval
 that cleared it. This is the ML angle on `domains/devops/P4 Rollback
 First` and `domains/kubernetes/P10 Roll Forward, Roll Back`.
 ## 2. Core Principle Trace
 Each AI/ML P-rule derives from one or more core C-rules (C1–C8). The
 matrix extension lands in P4 of the v0.3 plan; the traces below are
 authoritative.
 | P-rule | Core | Why |
 |--------|------|-----|
 | P1 Reproducibility is the First Class | C1, C5 | Correctness of results; reversibility of runs |
 | P2 Data is Versioned, Not Just Code | C5, C7 | Reversibility of datasets; observability of data lineage |
 | P3 Lineage is Traceable End-to-End | C7, C1 | Observability of provenance; correctness of attribution |
 | P4 Evaluation is Defined Before Training | C1, C2 | Correctness of the eval gate; clarity of a-priori intent |
 | P5 Models are Versioned Artifacts | C5, C6 | Reversibility of model identity; composability of registry stages |
 | P6 Serving is Observable | C7 | Observability of inference |
 | P7 Drift is Expected and Detected | C7, C1 | Observability of degradation; correctness of detection |
 | P8 Inference Inputs are Validated | C1 | Correctness of the serving boundary (security subset) |
 | P9 Pipelines Compose, Notebooks Don't | C6, C2 | Composability of stages; clarity of explicit contracts |
 | P10 Rollback Includes the Model | C5 | Reversibility at the model layer |
 ## 3. What Violates These Principles
 | Violation | Principle Breached |
 |-----------|-------------------|
 | A training run that cannot be replayed from pinned inputs | P1 Reproducibility is the First Class |
 | A dataset referenced by a mutable path, not a hash | P2 Data is Versioned, Not Just Code |
 | A production model with no record of its training data | P3 Lineage is Traceable End-to-End |
 | Metrics chosen after seeing the results | P4 Evaluation is Defined Before Training |
 | A serving endpoint that pulls `latest` from the registry | P5 Models are Versioned Artifacts |
 | A model in production with no latency or throughput metrics | P6 Serving is Observable |
 | A serving regression dismissed as "no code changed" | P7 Drift is Expected and Detected |
 | An input with an out-of-range feature scored silently | P8 Inference Inputs are Validated |
 | A notebook in the serving or training pipeline path | P9 Pipelines Compose, Notebooks Don't |
 | A rollback that restores code but keeps the regressed model | P10 Rollback Includes the Model |
 ## 4. Relationship to Other Domains
 AI/ML is the engineering-discipline layer for model-bearing systems.
 It borrows the reproducibility, immutability, rollback, and
 observability disciplines of `domains/devops/` and applies them to
 the data → model → serving lifecycle. Cross-links are one-directional
 (per D-026 extended):
 - `domains/devops/P1 Reproducibility` ← P1
 - `domains/devops/P4 Rollback First` ← P10
 - `domains/devops/P5 Progressive Delivery` ← P10 (model canary)
 - `domains/devops/P7 Immutability` ← P5 (model images)
 - `domains/data/migrations.md` ← P2 (data versioning ↔ migration discipline)
 - `domains/data/schema-design.md` ← P8 (inference input contract)
 - `domains/observability/metrics.md` ← P6, P7
 - `domains/observability/logging.md` ← P3 (lineage)
 - `domains/security/input-validation.md` ← P8
 - `domains/security/secrets.md` ← P8 (serving credentials)
 - `domains/performance/backend.md` ← P6 (serving latency)
 - `domains/kubernetes/workloads.md` ← P9 (serving on k8s)
 - `domains/testing/first-principles.md` ← P4 (eval as a gate)
 - `domains/gitops-operators/first-principles.md` ← P10 (model rollback in a GitOps loop)
@@ -0,0 +1,94 @@
 # Model Evaluation — Derived Rules
 > Derives from `domains/ai-ml/first-principles.md`. Covers P4
 > (Evaluation is Defined Before Training) and the eval-as-a-gate
 > discipline. Referenced by `serving.md` (promotion gate) and
 > `monitoring-drift.md` (online eval). Scope per D-023: evaluation
 > methodology, not metric math or model-family benchmarks.
 ## Evaluation is a Gate, Not a Report (P4 Evaluation Defined Before Training)
 - The eval spec — metrics, splits, thresholds, and pass/fail
  criteria — is declared in code **before** the model is trained.
  It is versioned with the data and the code; it is reviewable; it
  is the contract the model must satisfy to leave the experiment.
 - Cherry-picking metrics after seeing results is a correctness
  violation: the eval is no longer measuring the model, it is
  rationalizing it. The a-priori spec is what makes the eval
  trustworthy.
 - This is the ML angle on `domains/testing/first-principles.md` P1
  (Tests as Specification): the eval declares the model's contract,
  the model does not declare its own success.
 ## The Eval Input Contract (P8 Inference Inputs are Validated, cross `domains/data/schema-design.md`)
 - The eval set has a schema: feature names, types, ranges, and
  categorical domains. That schema is the same schema the serving
  boundary enforces (`serving.md`, `domains/security/input-validation.md`).
 - An eval set whose schema drifted from the serving schema is
  measuring a different model than the one in production. Schema
  parity is part of the versioned eval artifact.
 - Cross `domains/data/schema-design.md`: the eval input contract is
  a schema-design problem, versioned and reviewed like any schema.
 ## Holdout Integrity (P4, P2 Data is Versioned)
 - The held-out eval set is never touched by training, feature
  selection, or threshold tuning. A "held out" set that influenced
  any training decision is not held out — it is a third training
  set, and the eval is measuring memorization.
 - Splits are versioned with the dataset (`data-versioning.md`).
  Recreating splits ad-hoc per run breaks comparability across runs.
 - Reusing a held-out set across many model iterations leaks it
  incrementally. Rotate or re-split on a cadence; record the
  rotation in lineage.
 ## Offline vs Online Evaluation (P6 Serving is Observable)
 - **Offline eval** runs before promotion: held-out data, pinned
  model, declared metrics, pass/fail gate. It answers "should this
  model ship?"
 - **Online eval** runs after promotion, on live traffic: shadow
  scoring, A/B, canary metrics. It answers "is this model behaving
  in production?" It is the bridge to `monitoring-drift.md`.
 - A model that passed offline and regressed online is not a
  contradiction — it is a signal that the offline distribution
  differs from the live one (a P7 drift signal). Both eval layers
  are required; neither substitutes for the other.
 ## Bias and Fairness Checks (Engineering Angle, P4)
 - Bias/fairness checks are part of the a-priori eval spec, not an
  afterthought. They are metrics with thresholds, declared before
  training, gated the same as any metric.
 - This doc covers the **engineering** discipline: the checks are
  versioned, gated, and recorded in lineage. The choice of which
  fairness metrics and what thresholds are policy decisions, not
  engineering principles, and are out of scope here (D-023).
 ## Eval-as-a-Gate in the Pipeline (P9 Pipelines Compose)
 - The eval is a pipeline stage with a contract: input = model
  digest + eval dataset version; output = pass/fail + metric
  report. It composes with the training stage and the promotion
  stage.
 - A promotion that bypasses the eval stage is a P4 violation,
  regardless of who approved it. The gate is in the pipeline, not
  in a human sign-off sheet.
 ```
 train -> eval(gate) -> register(promote) -> serve
                 |
                 +-- fail -> abort, no promote
 ```
 ## What Violates Evaluation Discipline
 | Violation | Principle |
 |-----------|-----------|
 | Metrics chosen after seeing the scores | P4 Evaluation Defined Before Training |
 | Held-out set used in feature selection or threshold tuning | P4, P2 |
 | Eval schema differs from serving schema | P8 Inference Inputs are Validated |
 | Promotion by human approval, bypassing the eval stage | P4, P9 Pipelines Compose |
 | A "passing" model with no online eval in production | P6 Serving is Observable |
 | Fairness checks added after a model shipped | P4 Evaluation Defined Before Training |
@@ -0,0 +1,88 @@
 # Monitoring & Drift — Derived Rules
 > Derives from `domains/ai-ml/first-principles.md`. Covers P7 (Drift
 > is Expected and Detected) and the online half of P6 (Serving is
 > Observable). Referenced by `serving.md` (online eval) and
 > `model-evaluation.md` (online layer). Scope per D-023: drift
 > detection methodology, not model retraining architecture.
 ## Drift is Expected and Detected (P7 Drift is Expected and Detected)
 - ML systems decay without code changes. The world changes under
  the model: user behavior shifts, input pipelines change,
  upstream schemas evolve. "No code changed" is not a defense
  against a serving regression.
 - A drift signal is an incident, not a curiosity. It triggers an
  alert, an investigation, and a decision (retrain, roll back, or
  accept with a recorded justification). Silent drift is the same
  class of bug as silent serving (P6).
 - Cross `domains/observability/metrics.md` for the alerting
  primitives and `domains/observability/logging.md` for the
  structured events a drift signal emits.
 ## The Three Drift Types (IDEATE-30, D-048)
 | Drift Type | What Changes | Detection Signal | Source of Truth |
 |------------|--------------|------------------|-----------------|
 | **Data drift** (input drift) | The distribution of inputs at serving time diverges from the distribution the model was trained on | Statistical distance between the live input distribution and the pinned training-set distribution (e.g., PSI, KL, KS test). Alert on threshold breach | Training dataset hash (`data-versioning.md`) + live input metrics |
 | **Concept drift** | The relationship between inputs and the target changes — the same input now maps to a different correct output | Ground-truth lag: compare delayed labels against predictions on the same inputs. Rising error rate against a stable input distribution signals concept, not data, drift | Delayed-label feedback stream + prediction log |
 | **Prediction drift** (output drift) | The distribution of the model's predictions shifts, with no change to inputs | Statistical distance between the live prediction distribution and a pinned baseline prediction distribution. Independent of inputs — catches model-internal regressions and upstream silent changes | Prediction log + baseline prediction snapshot |
 - The three signals are distinct and non-substitutable. Data drift
  catches the input changing; concept drift catches the world
  changing; prediction drift catches the model's behavior changing.
  A monitoring setup with only one is blind to two classes of
  regression.
 - Evidently AI and Great Expectations are the canonical tooling:
  Evidently for drift/statistical reports, Great Expectations for
  data-quality/contract checks at the pipeline boundary. Both
  produce the metrics that feed `domains/observability/metrics.md`.
 ## Detection Signals in Practice
 - **Data drift** compares live inputs to the **pinned training
  distribution** — not to "yesterday's inputs." Without a pinned
  baseline, drift is measured against a moving target and is
  meaningless. Cross `data-versioning.md` for how the baseline is
  pinned.
 - **Concept drift** requires ground truth, which is often delayed
  (days/weeks). The detection signal is the gap between
  prediction-time confidence and delayed-label error. A rising
  error against stable inputs is the signature.
 - **Prediction drift** needs no ground truth and no input
  comparison — it watches the model's own output distribution. It
  is the cheapest signal and the first to fire; it is also the
  least specific (any of the three drifts can move predictions).
 ## Alerting and Retraining Triggers (P7, P10 Rollback Includes the Model)
 - A drift alert is an incident. It does not auto-trigger retraining
  unsupervised — auto-retraining on drift can lock in a bad
  distribution. The alert triggers a human decision: investigate,
  retrain, roll back, or accept.
 - Retraining is a new training run (`first-principles.md` P1): it
  produces a new model digest, passes the eval gate
  (`model-evaluation.md`), and is promoted through the registry
  (`serving.md`). The prior model stays rollbackable (P10).
 - Cross `domains/observability/metrics.md` for the alert-rule
  pattern: threshold + window + severity, routed to the same
  on-call path as any production incident.
 ## Online Evaluation Bridge (P6 Serving is Observable)
 - Online eval (`model-evaluation.md`) is the live counterpart to
  drift monitoring: shadow scores and A/B canaries measure a
  candidate model against the incumbent, while drift monitoring
  measures the incumbent against its own baseline. Both feed the
  same metrics pipeline.
 ## What Violates Monitoring Discipline
 | Violation | Principle |
 |-----------|-----------|
 | Only one drift type monitored | P7 Drift is Expected and Detected |
 | Drift baseline is "yesterday's inputs," not pinned training data | P7, P2 Data is Versioned |
 | Drift alert that auto-retrains without a human gate | P7, P1 Reproducibility |
 | A serving regression dismissed as "no code changed" | P7 Drift is Expected and Detected |
 | Concept-drift check with no delayed-label feedback path | P7 Drift is Expected and Detected |
 | Prediction-distribution change with no alert | P6 Serving is Observable, P7 |
@@ -0,0 +1,88 @@
 # Serving — Derived Rules
 > Derives from `domains/ai-ml/first-principles.md`. Covers P5 (Models
 > are Versioned Artifacts), P6 (Serving is Observable), P8 (Inference
 > Inputs are Validated), and P10 (Rollback Includes the Model).
 > Referenced by `monitoring-drift.md` (online signals) and
 > `model-evaluation.md` (promotion gate). Scope per D-023: serving
 > patterns, not model architectures.
 ## The Model is an Addressed Artifact (P5 Models are Versioned Artifacts)
 - A serving endpoint pulls a model by digest, never by `latest`. A
  model pulled by `latest` is an unknown model — its behavior is
  undefined and its rollback is impossible.
 - The model registry is to models what a container registry is to
  images (`domains/devops/P7 Immutability`): immutable, addressed by
  digest, promoted by stage (staging → prod). Promotion is a
  registry operation, not a file copy.
 - A serving rollout names the model digest in its manifest. The
  digest is part of the deploy's lineage (`data-versioning.md`).
 ## Inference Inputs are Validated (P8 Inference Inputs are Validated)
 - The model's input contract — schema, types, ranges, categorical
  domains — is enforced at the serving boundary, before the model
  sees the input. Out-of-contract inputs are rejected with a
  defined error, not silently scored.
 - Scoring an out-of-contract input is a C1 (Correctness) violation:
  the model's output is undefined outside its training
  distribution, and returning a number for it is lying to the
  caller.
 - This is the ML angle on `domains/security/input-validation.md`:
  the validation lives at the boundary, the model is downstream of
  it, and the contract is versioned with the model.
 ## Serving is Observable (P6 Serving is Observable)
 - Every inference path emits: request latency, throughput, input
  distribution summaries, prediction confidence, and error counts.
  Silent serving is a bug.
 - Cross `domains/observability/metrics.md` for the metrics
  primitives (histograms, counters, gauges) and
  `domains/observability/tracing.md` for the request-level trace
  that ties an input to a prediction.
 - Latency SLAs are enforced via `domains/performance/backend.md`
  disciplines: budget the inference path, measure the tail (p99),
  alert on budget breach.
 ## Serving Patterns (P9 Pipelines Compose)
 | Pattern | When | Notes |
 |---------|------|-------|
 | Inference as a service | Default; model behind an HTTP/gRPC endpoint | KServe, Seldon Core, BentoML. Scales with traffic; model is a deployable, addressable artifact |
 | Batch inference | Offline scoring of large datasets | No latency SLA; throughput-bound. Same model digest, same input contract |
 | Embedded / in-process | Latency-critical, single-tenant | Model linked into the app. Trades observability for latency — only when the SLA demands it |
 - Canarying a model is a serving pattern, not a deployment pattern:
  shift a fraction of traffic to the new model digest, measure
  online eval (`model-evaluation.md`), abort to the prior digest on
  regression. This is `domains/devops/P5 Progressive Delivery`
  applied at the model layer.
 - Rollback restores the prior model digest (P10 Rollback Includes
  the Model). A rollback that redeploys old code but keeps the new
  model has not rolled back. Cross `domains/gitops-operators/first-principles.md`
  for the GitOps reconciliation loop that drives model rollouts.
 ## Tool Landscape (KServe / Seldon Core / BentoML)
 | Tool | Model Packaging | Deployment Surface | Notes |
 |------|-----------------|--------------------|-------|
 | KServe | InferenceService CRD; runtime predictors (v2, HuggingFace, PMML, custom) | Kubernetes-native; CRD-driven | Cross `domains/kubernetes/workloads.md`. Brings the k8s reconciliation model to serving |
 | Seldon Core | SeldonDeployment CRD; graph of predictors | Kubernetes-native; CRD-driven | Emphasizes inference graphs (fan-out, ensemble) as CRD structure |
 | BentoML | Bento (model + runtime + deps packaged); Yatai registry | Kubernetes or bare container | Focuses on packaging + registry; the Bento is the versioned artifact (P5) |
 - All three satisfy P5/P6/P8 when wired correctly; the choice is
  packaging model and deployment surface, not correctness.
 - None is advocated over the others.
 ## What Violates Serving Discipline
 | Violation | Principle |
 |-----------|-----------|
 | Endpoint pulls `latest` from the registry | P5 Models are Versioned Artifacts |
 | Out-of-range input scored silently | P8 Inference Inputs are Validated |
 | Serving path emits no latency or throughput metrics | P6 Serving is Observable |
 | Rollback redeploys code but keeps the regressed model | P10 Rollback Includes the Model |
 | A notebook in the serving path | P9 Pipelines Compose, Notebooks Don't |
 | Canary with no abort-to-prior-digest path | P10, `domains/devops/P5 Progressive Delivery` |