diff --git a/domains/ai-ml/data-versioning.md b/domains/ai-ml/data-versioning.md
new file mode 100644
index 0000000..5c4cfd5
--- /dev/null
+++ b/domains/ai-ml/data-versioning.md
@@ -0,0 +1,90 @@
+# Data Versioning — Derived Rules
+
+> Derives from `domains/ai-ml/first-principles.md`. Covers P2 (Data is
+> Versioned, Not Just Code) and P3 (Lineage is Traceable End-to-End).
+> Referenced by `serving.md` and `monitoring-drift.md`. Scope per
+> D-023: engineering discipline of versioning data, not dataset
+> content design.
+
+## Why Data Versioning (P2 Data is Versioned, Not Just Code)
+
+- `git` versions code well and data badly. Datasets do not fit in
+  git, and a dataset is not recovered from a commit hash.
+- A model trained on "the data" is a model trained on an unknown
+  input — a C1 (Correctness) violation. The dataset is a build
+  input; it is named, hashed, and recoverable the way any build
+  input is.
+- Data versioning is the ML analogue of `domains/data/migrations.md`:
+  the schema and contents of the data evolve, every evolution is a
+  versioned migration, and every model points at a specific version.
+
+## Dataset Hashing and Lineage (P3 Lineage Traceable End-to-End)
+
+- Every dataset version has a content hash (not a filename or a
+  timestamp). The hash is the identity. A model's lineage record
+  names the dataset hash it was trained on; a serving prediction
+  names the model digest it came from.
+- Lineage is a graph: prediction → model → training run → dataset →
+  source(s). Any edge missing is an orphan (`domains/observability/logging.md`
+  for the structured-log angle on lineage events).
+- The lineage record is append-only. Editing it to "fix" a broken
+  trace is the same class of violation as editing an audit log.
+
+## Train/Val/Test Split Versioning (P2, P4 Eval Defined Before Training)
+
+- Splits are versioned with the dataset, not derived ad-hoc per run.
+  A split is a deterministic function of (dataset version, split
+  config, random seed). Two runs on the same pinned inputs produce
+  the same splits.
+- The eval split is held out and never touched by training. A "held
+  out" set that leaked into training is a P4 (Evaluation Defined
+  Before Training) violation, not just a P2 violation — the eval
+  gate is measuring the training set, not the model.
+- Cross `domains/data/schema-design.md` for the eval input contract:
+  the schema of the eval set is part of the versioned artifact.
+
+## Tool Comparison (IDEATE-22, D-040)
+
+| Tool | Versioning Model | Lineage | Best For | Notes |
+|------|------------------|---------|----------|-------|
+| DVC | Git-like pointers to content-addressed object store; `.dvc` files in git track data versions | Pipeline DAG in `dvc.yaml`; reproducibility via `dvc repro` | Teams already on git; file/directory datasets; ML pipelines | Treats data like code; shares git's history model. Object store is pluggable (S3, GCS, Azure, SSH) |
+| Delta Lake | Table format with transaction log (ACID) + time travel via versioned commits; schema enforcement | Time travel queries; lineage via table history + catalog | Large tabular data; lakehouse; streaming + batch on the same table | Not a pipeline tool — pairs with Spark/Trino/Flink. Brings DB guarantees to object storage |
+| LakeFS | Git-like operations (branch, commit, merge) over object storage itself | Branch model gives isolated, reproducible data branches | Data engineering teams; branch-per-experiment; CI over data | Not a table format — versions objects. Composes with Delta/Iceberg on top |
+
+- Pick one primary versioning model per platform. Mixing DVC's
+  pointer model with Delta's transaction-log model fragments
+  operational knowledge (C4 Locality).
+- All three satisfy P2; the choice is which fits the data shape and
+  the team's existing tooling. None is advocated over the others.
+
+## Reproducibility Contract (P1 Reproducibility is the First Class)
+
+A reproducible training run records, in one versioned place:
+
+```
+run_id:        2026-08-05T09:12:00Z#run-42
+dataset:       s3://ml-data/train@sha256:7f3a...e21
+splits:        dvc.yaml@commit a1b2c4d
+code:          git@a1b2c4d
+config:        configs/train.yaml@commit a1b2c4d
+environment:   ghcr.io/org/train-img@sha256:9c2d...f88
+eval_spec:     configs/eval.yaml@commit a1b2c4d
+model_digest:  registry/model@sha256:b5e1...aa0
+```
+
+- Lose any line and the run is anecdote, not evidence.
+- The record is the lineage root: a prediction cites the
+  `model_digest`, which cites the `run_id`, which cites everything
+  above. This is how P3 (Lineage Traceable End-to-End) is satisfied
+  in practice.
+
+## What Violates Data Versioning Discipline
+
+| Violation | Principle |
+|-----------|-----------|
+| Dataset referenced by `s3://bucket/latest/` | P2 Data is Versioned, Not Just Code |
+| Splits regenerated with an unpinned seed per run | P2, P4 Evaluation Defined Before Training |
+| A production model with no dataset hash in its lineage | P3 Lineage Traceable End-to-End |
+| Editing a lineage record to "clean up" a broken trace | P3 Lineage Traceable End-to-End |
+| Eval split reachable from the training data path | P4 Evaluation Defined Before Training |
+| Two platforms versioning the same data with different models | C4 Locality |
\ No newline at end of file
diff --git a/domains/ai-ml/first-principles.md b/domains/ai-ml/first-principles.md
new file mode 100644
index 0000000..3396d29
--- /dev/null
+++ b/domains/ai-ml/first-principles.md
@@ -0,0 +1,154 @@
+# AI / ML — First Principles
+
+> Scope per D-023: this domain covers ML **engineering discipline** —
+> data versioning, evaluation methodology, serving patterns, and drift
+> detection. It does **not** cover algorithm design, model architecture
+> selection, hyperparameter tuning, or model-family comparison. Those
+> are research choices, not engineering principles, and they have no
+> derivation in the core C-rules.
+
+## 1. The Principles
+
+### P1. Reproducibility is the First Class
+Every training run is reproducible from pinned data + code + config +
+environment. An unreproducible run is an unreviewable run: you cannot
+decide whether a result is correct if you cannot recreate it.
+Reproducibility is the ML analogue of `domains/devops/P1
+Reproducibility` and inherits its non-negotiable status. Lose any one
+of data, code, config, or environment pinning, and the run is
+anecdote, not evidence.
+
+### P2. Data is Versioned, Not Just Code
+Datasets, features, and train/val/test splits are first-class
+versioned artifacts with content hashes and lineage. `git` alone is
+insufficient — datasets do not fit in git, and a dataset is not a
+commit hash. A model trained on "the data" is a model trained on an
+unknown input, which is a correctness violation. Version data the way
+you version code: pinned, named, and recoverable.
+
+### P3. Lineage is Traceable End-to-End
+Any deployed prediction traces back through model → training run →
+dataset → source. No orphan models. A model in production with no
+lineage is a correctness defect: you cannot reason about its failure
+modes, you cannot roll it back to a known-good dataset, and you cannot
+tell whether drift is in the model or in the data that built it.
+Lineage is the audit trail of ML (`domains/observability/logging.md`).
+
+### P4. Evaluation is Defined Before Training
+Metrics, splits, and acceptance thresholds are declared a priori, in
+code, before the model is trained. Cherry-picking metrics post-hoc is
+a correctness violation: the evaluation is no longer measuring the
+model, it is rationalizing it. The eval spec is a contract — it is
+reviewable, it is versioned, and it is the gate the model must pass
+before it leaves the experiment. This is the ML angle on C2 Clarity:
+the intent of the model is obvious to its reader because the eval
+declared it first.
+
+### P5. Models are Versioned Artifacts
+A model is a pinned, immutable, registry-tracked artifact with a
+unique identifier. Never "the latest." A serving endpoint that pulls
+"latest" is serving an unknown model — its behavior is undefined, its
+rollback is impossible, and its lineage is broken. The model registry
+is to models what a container registry is to images
+(`domains/devops/P7 Immutability`): immutable, addressed by digest,
+promoted by stage.
+
+### P6. Serving is Observable
+Inference latency, throughput, input distributions, and prediction
+confidence are first-class signals. Silent serving is a bug. A model
+in production that emits no metrics is a model you cannot operate: you
+cannot see latency regressions, you cannot see input drift, you cannot
+see a failing downstream consumer. Observability is designed in, not
+bolted on (`domains/observability/metrics.md`).
+
+### P7. Drift is Expected and Detected
+Data drift, concept drift, and prediction drift are monitored as a
+matter of course. A drift signal is an incident, not a curiosity. ML
+systems decay without code changes — the world changes under the
+model — so "no code changed" is not a defense against a serving
+regression. Detecting drift is the ML-specific form of C7
+Observability: you cannot fix a model you cannot see degrading.
+
+### P8. Inference Inputs are Validated
+The model's input contract — schema, value ranges, types, and
+categorical domains — is enforced at the serving boundary.
+Out-of-contract inputs are rejected, not silently scored. Scoring an
+out-of-contract input is a correctness violation: the model's output
+is undefined for inputs outside its training distribution, and
+returning a number for it is lying to the caller. This is the ML angle
+on `domains/security/input-validation.md` and inherits C1's
+non-tradeable status.
+
+### P9. Pipelines Compose, Notebooks Don't
+Training and serving flows are composable pipelines with explicit
+steps, named inputs, named outputs, and contracts between stages.
+Notebooks are for exploration, not production. A notebook in the
+serving path is a correctness defect: its state is implicit, its
+order is human-dependent, and its reproducibility is whatever the last
+operator remembered. Compose pipelines; keep notebooks in the lab.
+
+### P10. Rollback Includes the Model
+A serving rollback restores the prior model artifact, not just the
+prior code. Promotion is reversible at the model layer. A rollback
+that redeploys old code but keeps the new model has not rolled back —
+the model was the thing that regressed. The rollback path must name
+the prior model digest, the prior dataset version, and the prior eval
+that cleared it. This is the ML angle on `domains/devops/P4 Rollback
+First` and `domains/kubernetes/P10 Roll Forward, Roll Back`.
+
+## 2. Core Principle Trace
+
+Each AI/ML P-rule derives from one or more core C-rules (C1–C8). The
+matrix extension lands in P4 of the v0.3 plan; the traces below are
+authoritative.
+
+| P-rule | Core | Why |
+|--------|------|-----|
+| P1 Reproducibility is the First Class | C1, C5 | Correctness of results; reversibility of runs |
+| P2 Data is Versioned, Not Just Code | C5, C7 | Reversibility of datasets; observability of data lineage |
+| P3 Lineage is Traceable End-to-End | C7, C1 | Observability of provenance; correctness of attribution |
+| P4 Evaluation is Defined Before Training | C1, C2 | Correctness of the eval gate; clarity of a-priori intent |
+| P5 Models are Versioned Artifacts | C5, C6 | Reversibility of model identity; composability of registry stages |
+| P6 Serving is Observable | C7 | Observability of inference |
+| P7 Drift is Expected and Detected | C7, C1 | Observability of degradation; correctness of detection |
+| P8 Inference Inputs are Validated | C1 | Correctness of the serving boundary (security subset) |
+| P9 Pipelines Compose, Notebooks Don't | C6, C2 | Composability of stages; clarity of explicit contracts |
+| P10 Rollback Includes the Model | C5 | Reversibility at the model layer |
+
+## 3. What Violates These Principles
+
+| Violation | Principle Breached |
+|-----------|-------------------|
+| A training run that cannot be replayed from pinned inputs | P1 Reproducibility is the First Class |
+| A dataset referenced by a mutable path, not a hash | P2 Data is Versioned, Not Just Code |
+| A production model with no record of its training data | P3 Lineage is Traceable End-to-End |
+| Metrics chosen after seeing the results | P4 Evaluation is Defined Before Training |
+| A serving endpoint that pulls `latest` from the registry | P5 Models are Versioned Artifacts |
+| A model in production with no latency or throughput metrics | P6 Serving is Observable |
+| A serving regression dismissed as "no code changed" | P7 Drift is Expected and Detected |
+| An input with an out-of-range feature scored silently | P8 Inference Inputs are Validated |
+| A notebook in the serving or training pipeline path | P9 Pipelines Compose, Notebooks Don't |
+| A rollback that restores code but keeps the regressed model | P10 Rollback Includes the Model |
+
+## 4. Relationship to Other Domains
+
+AI/ML is the engineering-discipline layer for model-bearing systems.
+It borrows the reproducibility, immutability, rollback, and
+observability disciplines of `domains/devops/` and applies them to
+the data → model → serving lifecycle. Cross-links are one-directional
+(per D-026 extended):
+
+- `domains/devops/P1 Reproducibility` ← P1
+- `domains/devops/P4 Rollback First` ← P10
+- `domains/devops/P5 Progressive Delivery` ← P10 (model canary)
+- `domains/devops/P7 Immutability` ← P5 (model images)
+- `domains/data/migrations.md` ← P2 (data versioning ↔ migration discipline)
+- `domains/data/schema-design.md` ← P8 (inference input contract)
+- `domains/observability/metrics.md` ← P6, P7
+- `domains/observability/logging.md` ← P3 (lineage)
+- `domains/security/input-validation.md` ← P8
+- `domains/security/secrets.md` ← P8 (serving credentials)
+- `domains/performance/backend.md` ← P6 (serving latency)
+- `domains/kubernetes/workloads.md` ← P9 (serving on k8s)
+- `domains/testing/first-principles.md` ← P4 (eval as a gate)
+- `domains/gitops-operators/first-principles.md` ← P10 (model rollback in a GitOps loop)
\ No newline at end of file
diff --git a/domains/ai-ml/model-evaluation.md b/domains/ai-ml/model-evaluation.md
new file mode 100644
index 0000000..adfdbe4
--- /dev/null
+++ b/domains/ai-ml/model-evaluation.md
@@ -0,0 +1,94 @@
+# Model Evaluation — Derived Rules
+
+> Derives from `domains/ai-ml/first-principles.md`. Covers P4
+> (Evaluation is Defined Before Training) and the eval-as-a-gate
+> discipline. Referenced by `serving.md` (promotion gate) and
+> `monitoring-drift.md` (online eval). Scope per D-023: evaluation
+> methodology, not metric math or model-family benchmarks.
+
+## Evaluation is a Gate, Not a Report (P4 Evaluation Defined Before Training)
+
+- The eval spec — metrics, splits, thresholds, and pass/fail
+  criteria — is declared in code **before** the model is trained.
+  It is versioned with the data and the code; it is reviewable; it
+  is the contract the model must satisfy to leave the experiment.
+- Cherry-picking metrics after seeing results is a correctness
+  violation: the eval is no longer measuring the model, it is
+  rationalizing it. The a-priori spec is what makes the eval
+  trustworthy.
+- This is the ML angle on `domains/testing/first-principles.md` P1
+  (Tests as Specification): the eval declares the model's contract,
+  the model does not declare its own success.
+
+## The Eval Input Contract (P8 Inference Inputs are Validated, cross `domains/data/schema-design.md`)
+
+- The eval set has a schema: feature names, types, ranges, and
+  categorical domains. That schema is the same schema the serving
+  boundary enforces (`serving.md`, `domains/security/input-validation.md`).
+- An eval set whose schema drifted from the serving schema is
+  measuring a different model than the one in production. Schema
+  parity is part of the versioned eval artifact.
+- Cross `domains/data/schema-design.md`: the eval input contract is
+  a schema-design problem, versioned and reviewed like any schema.
+
+## Holdout Integrity (P4, P2 Data is Versioned)
+
+- The held-out eval set is never touched by training, feature
+  selection, or threshold tuning. A "held out" set that influenced
+  any training decision is not held out — it is a third training
+  set, and the eval is measuring memorization.
+- Splits are versioned with the dataset (`data-versioning.md`).
+  Recreating splits ad-hoc per run breaks comparability across runs.
+- Reusing a held-out set across many model iterations leaks it
+  incrementally. Rotate or re-split on a cadence; record the
+  rotation in lineage.
+
+## Offline vs Online Evaluation (P6 Serving is Observable)
+
+- **Offline eval** runs before promotion: held-out data, pinned
+  model, declared metrics, pass/fail gate. It answers "should this
+  model ship?"
+- **Online eval** runs after promotion, on live traffic: shadow
+  scoring, A/B, canary metrics. It answers "is this model behaving
+  in production?" It is the bridge to `monitoring-drift.md`.
+- A model that passed offline and regressed online is not a
+  contradiction — it is a signal that the offline distribution
+  differs from the live one (a P7 drift signal). Both eval layers
+  are required; neither substitutes for the other.
+
+## Bias and Fairness Checks (Engineering Angle, P4)
+
+- Bias/fairness checks are part of the a-priori eval spec, not an
+  afterthought. They are metrics with thresholds, declared before
+  training, gated the same as any metric.
+- This doc covers the **engineering** discipline: the checks are
+  versioned, gated, and recorded in lineage. The choice of which
+  fairness metrics and what thresholds are policy decisions, not
+  engineering principles, and are out of scope here (D-023).
+
+## Eval-as-a-Gate in the Pipeline (P9 Pipelines Compose)
+
+- The eval is a pipeline stage with a contract: input = model
+  digest + eval dataset version; output = pass/fail + metric
+  report. It composes with the training stage and the promotion
+  stage.
+- A promotion that bypasses the eval stage is a P4 violation,
+  regardless of who approved it. The gate is in the pipeline, not
+  in a human sign-off sheet.
+
+```
+train -> eval(gate) -> register(promote) -> serve
+                 |
+                 +-- fail -> abort, no promote
+```
+
+## What Violates Evaluation Discipline
+
+| Violation | Principle |
+|-----------|-----------|
+| Metrics chosen after seeing the scores | P4 Evaluation Defined Before Training |
+| Held-out set used in feature selection or threshold tuning | P4, P2 |
+| Eval schema differs from serving schema | P8 Inference Inputs are Validated |
+| Promotion by human approval, bypassing the eval stage | P4, P9 Pipelines Compose |
+| A "passing" model with no online eval in production | P6 Serving is Observable |
+| Fairness checks added after a model shipped | P4 Evaluation Defined Before Training |
\ No newline at end of file
diff --git a/domains/ai-ml/monitoring-drift.md b/domains/ai-ml/monitoring-drift.md
new file mode 100644
index 0000000..0148071
--- /dev/null
+++ b/domains/ai-ml/monitoring-drift.md
@@ -0,0 +1,88 @@
+# Monitoring & Drift — Derived Rules
+
+> Derives from `domains/ai-ml/first-principles.md`. Covers P7 (Drift
+> is Expected and Detected) and the online half of P6 (Serving is
+> Observable). Referenced by `serving.md` (online eval) and
+> `model-evaluation.md` (online layer). Scope per D-023: drift
+> detection methodology, not model retraining architecture.
+
+## Drift is Expected and Detected (P7 Drift is Expected and Detected)
+
+- ML systems decay without code changes. The world changes under
+  the model: user behavior shifts, input pipelines change,
+  upstream schemas evolve. "No code changed" is not a defense
+  against a serving regression.
+- A drift signal is an incident, not a curiosity. It triggers an
+  alert, an investigation, and a decision (retrain, roll back, or
+  accept with a recorded justification). Silent drift is the same
+  class of bug as silent serving (P6).
+- Cross `domains/observability/metrics.md` for the alerting
+  primitives and `domains/observability/logging.md` for the
+  structured events a drift signal emits.
+
+## The Three Drift Types (IDEATE-30, D-048)
+
+| Drift Type | What Changes | Detection Signal | Source of Truth |
+|------------|--------------|------------------|-----------------|
+| **Data drift** (input drift) | The distribution of inputs at serving time diverges from the distribution the model was trained on | Statistical distance between the live input distribution and the pinned training-set distribution (e.g., PSI, KL, KS test). Alert on threshold breach | Training dataset hash (`data-versioning.md`) + live input metrics |
+| **Concept drift** | The relationship between inputs and the target changes — the same input now maps to a different correct output | Ground-truth lag: compare delayed labels against predictions on the same inputs. Rising error rate against a stable input distribution signals concept, not data, drift | Delayed-label feedback stream + prediction log |
+| **Prediction drift** (output drift) | The distribution of the model's predictions shifts, with no change to inputs | Statistical distance between the live prediction distribution and a pinned baseline prediction distribution. Independent of inputs — catches model-internal regressions and upstream silent changes | Prediction log + baseline prediction snapshot |
+
+- The three signals are distinct and non-substitutable. Data drift
+  catches the input changing; concept drift catches the world
+  changing; prediction drift catches the model's behavior changing.
+  A monitoring setup with only one is blind to two classes of
+  regression.
+- Evidently AI and Great Expectations are the canonical tooling:
+  Evidently for drift/statistical reports, Great Expectations for
+  data-quality/contract checks at the pipeline boundary. Both
+  produce the metrics that feed `domains/observability/metrics.md`.
+
+## Detection Signals in Practice
+
+- **Data drift** compares live inputs to the **pinned training
+  distribution** — not to "yesterday's inputs." Without a pinned
+  baseline, drift is measured against a moving target and is
+  meaningless. Cross `data-versioning.md` for how the baseline is
+  pinned.
+- **Concept drift** requires ground truth, which is often delayed
+  (days/weeks). The detection signal is the gap between
+  prediction-time confidence and delayed-label error. A rising
+  error against stable inputs is the signature.
+- **Prediction drift** needs no ground truth and no input
+  comparison — it watches the model's own output distribution. It
+  is the cheapest signal and the first to fire; it is also the
+  least specific (any of the three drifts can move predictions).
+
+## Alerting and Retraining Triggers (P7, P10 Rollback Includes the Model)
+
+- A drift alert is an incident. It does not auto-trigger retraining
+  unsupervised — auto-retraining on drift can lock in a bad
+  distribution. The alert triggers a human decision: investigate,
+  retrain, roll back, or accept.
+- Retraining is a new training run (`first-principles.md` P1): it
+  produces a new model digest, passes the eval gate
+  (`model-evaluation.md`), and is promoted through the registry
+  (`serving.md`). The prior model stays rollbackable (P10).
+- Cross `domains/observability/metrics.md` for the alert-rule
+  pattern: threshold + window + severity, routed to the same
+  on-call path as any production incident.
+
+## Online Evaluation Bridge (P6 Serving is Observable)
+
+- Online eval (`model-evaluation.md`) is the live counterpart to
+  drift monitoring: shadow scores and A/B canaries measure a
+  candidate model against the incumbent, while drift monitoring
+  measures the incumbent against its own baseline. Both feed the
+  same metrics pipeline.
+
+## What Violates Monitoring Discipline
+
+| Violation | Principle |
+|-----------|-----------|
+| Only one drift type monitored | P7 Drift is Expected and Detected |
+| Drift baseline is "yesterday's inputs," not pinned training data | P7, P2 Data is Versioned |
+| Drift alert that auto-retrains without a human gate | P7, P1 Reproducibility |
+| A serving regression dismissed as "no code changed" | P7 Drift is Expected and Detected |
+| Concept-drift check with no delayed-label feedback path | P7 Drift is Expected and Detected |
+| Prediction-distribution change with no alert | P6 Serving is Observable, P7 |
\ No newline at end of file
diff --git a/domains/ai-ml/serving.md b/domains/ai-ml/serving.md
new file mode 100644
index 0000000..70096af
--- /dev/null
+++ b/domains/ai-ml/serving.md
@@ -0,0 +1,88 @@
+# Serving — Derived Rules
+
+> Derives from `domains/ai-ml/first-principles.md`. Covers P5 (Models
+> are Versioned Artifacts), P6 (Serving is Observable), P8 (Inference
+> Inputs are Validated), and P10 (Rollback Includes the Model).
+> Referenced by `monitoring-drift.md` (online signals) and
+> `model-evaluation.md` (promotion gate). Scope per D-023: serving
+> patterns, not model architectures.
+
+## The Model is an Addressed Artifact (P5 Models are Versioned Artifacts)
+
+- A serving endpoint pulls a model by digest, never by `latest`. A
+  model pulled by `latest` is an unknown model — its behavior is
+  undefined and its rollback is impossible.
+- The model registry is to models what a container registry is to
+  images (`domains/devops/P7 Immutability`): immutable, addressed by
+  digest, promoted by stage (staging → prod). Promotion is a
+  registry operation, not a file copy.
+- A serving rollout names the model digest in its manifest. The
+  digest is part of the deploy's lineage (`data-versioning.md`).
+
+## Inference Inputs are Validated (P8 Inference Inputs are Validated)
+
+- The model's input contract — schema, types, ranges, categorical
+  domains — is enforced at the serving boundary, before the model
+  sees the input. Out-of-contract inputs are rejected with a
+  defined error, not silently scored.
+- Scoring an out-of-contract input is a C1 (Correctness) violation:
+  the model's output is undefined outside its training
+  distribution, and returning a number for it is lying to the
+  caller.
+- This is the ML angle on `domains/security/input-validation.md`:
+  the validation lives at the boundary, the model is downstream of
+  it, and the contract is versioned with the model.
+
+## Serving is Observable (P6 Serving is Observable)
+
+- Every inference path emits: request latency, throughput, input
+  distribution summaries, prediction confidence, and error counts.
+  Silent serving is a bug.
+- Cross `domains/observability/metrics.md` for the metrics
+  primitives (histograms, counters, gauges) and
+  `domains/observability/tracing.md` for the request-level trace
+  that ties an input to a prediction.
+- Latency SLAs are enforced via `domains/performance/backend.md`
+  disciplines: budget the inference path, measure the tail (p99),
+  alert on budget breach.
+
+## Serving Patterns (P9 Pipelines Compose)
+
+| Pattern | When | Notes |
+|---------|------|-------|
+| Inference as a service | Default; model behind an HTTP/gRPC endpoint | KServe, Seldon Core, BentoML. Scales with traffic; model is a deployable, addressable artifact |
+| Batch inference | Offline scoring of large datasets | No latency SLA; throughput-bound. Same model digest, same input contract |
+| Embedded / in-process | Latency-critical, single-tenant | Model linked into the app. Trades observability for latency — only when the SLA demands it |
+
+- Canarying a model is a serving pattern, not a deployment pattern:
+  shift a fraction of traffic to the new model digest, measure
+  online eval (`model-evaluation.md`), abort to the prior digest on
+  regression. This is `domains/devops/P5 Progressive Delivery`
+  applied at the model layer.
+- Rollback restores the prior model digest (P10 Rollback Includes
+  the Model). A rollback that redeploys old code but keeps the new
+  model has not rolled back. Cross `domains/gitops-operators/first-principles.md`
+  for the GitOps reconciliation loop that drives model rollouts.
+
+## Tool Landscape (KServe / Seldon Core / BentoML)
+
+| Tool | Model Packaging | Deployment Surface | Notes |
+|------|-----------------|--------------------|-------|
+| KServe | InferenceService CRD; runtime predictors (v2, HuggingFace, PMML, custom) | Kubernetes-native; CRD-driven | Cross `domains/kubernetes/workloads.md`. Brings the k8s reconciliation model to serving |
+| Seldon Core | SeldonDeployment CRD; graph of predictors | Kubernetes-native; CRD-driven | Emphasizes inference graphs (fan-out, ensemble) as CRD structure |
+| BentoML | Bento (model + runtime + deps packaged); Yatai registry | Kubernetes or bare container | Focuses on packaging + registry; the Bento is the versioned artifact (P5) |
+
+- All three satisfy P5/P6/P8 when wired correctly; the choice is
+  packaging model and deployment surface, not correctness.
+- None is advocated over the others.
+
+## What Violates Serving Discipline
+
+| Violation | Principle |
+|-----------|-----------|
+| Endpoint pulls `latest` from the registry | P5 Models are Versioned Artifacts |
+| Out-of-range input scored silently | P8 Inference Inputs are Validated |
+| Serving path emits no latency or throughput metrics | P6 Serving is Observable |
+| Rollback redeploys code but keeps the regressed model | P10 Rollback Includes the Model |
+| A notebook in the serving path | P9 Pipelines Compose, Notebooks Don't |
+| Canary with no abort-to-prior-digest path | P10, `domains/devops/P5 Progressive Delivery` |
\ No newline at end of file