coreci/atelier
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 28 Wiki Activity

docs(P2): complete ai-ml domain phase

Browse Source 
---ci---
project: atelier
phase: 2
milestone: v0.3
status: complete
requirements:
  covered: [ATELIER-65, ATELIER-66, ATELIER-67, ATELIER-68, ATELIER-69]
  partial: []
---/ci---
This commit is contained in:
Jon Chery
2026-08-05 03:24:13 +00:00
parent ab1289a9d9
commit ba553c45af
5 changed files with 514 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
domains/ai-ml/data-versioning.md
+90
View File
@@ -0,0 +1,90 @@
# Data Versioning — Derived Rules
> Derives from `domains/ai-ml/first-principles.md`. Covers P2 (Data is
> Versioned, Not Just Code) and P3 (Lineage is Traceable End-to-End).
> Referenced by `serving.md` and `monitoring-drift.md`. Scope per
> D-023: engineering discipline of versioning data, not dataset
> content design.
## Why Data Versioning (P2 Data is Versioned, Not Just Code)
- `git` versions code well and data badly. Datasets do not fit in
git, and a dataset is not recovered from a commit hash.
- A model trained on "the data" is a model trained on an unknown
input — a C1 (Correctness) violation. The dataset is a build
input; it is named, hashed, and recoverable the way any build
input is.
- Data versioning is the ML analogue of `domains/data/migrations.md`:
the schema and contents of the data evolve, every evolution is a
versioned migration, and every model points at a specific version.
## Dataset Hashing and Lineage (P3 Lineage Traceable End-to-End)
- Every dataset version has a content hash (not a filename or a
timestamp). The hash is the identity. A model's lineage record
names the dataset hash it was trained on; a serving prediction
names the model digest it came from.
- Lineage is a graph: prediction → model → training run → dataset →
source(s). Any edge missing is an orphan (`domains/observability/logging.md`
for the structured-log angle on lineage events).
- The lineage record is append-only. Editing it to "fix" a broken
trace is the same class of violation as editing an audit log.
## Train/Val/Test Split Versioning (P2, P4 Eval Defined Before Training)
- Splits are versioned with the dataset, not derived ad-hoc per run.
A split is a deterministic function of (dataset version, split
config, random seed). Two runs on the same pinned inputs produce
the same splits.
- The eval split is held out and never touched by training. A "held
out" set that leaked into training is a P4 (Evaluation Defined
Before Training) violation, not just a P2 violation — the eval
gate is measuring the training set, not the model.
- Cross `domains/data/schema-design.md` for the eval input contract:
the schema of the eval set is part of the versioned artifact.
## Tool Comparison (IDEATE-22, D-040)
| Tool | Versioning Model | Lineage | Best For | Notes |
|------|------------------|---------|----------|-------|
| DVC | Git-like pointers to content-addressed object store; `.dvc` files in git track data versions | Pipeline DAG in `dvc.yaml`; reproducibility via `dvc repro` | Teams already on git; file/directory datasets; ML pipelines | Treats data like code; shares git's history model. Object store is pluggable (S3, GCS, Azure, SSH) |
| Delta Lake | Table format with transaction log (ACID) + time travel via versioned commits; schema enforcement | Time travel queries; lineage via table history + catalog | Large tabular data; lakehouse; streaming + batch on the same table | Not a pipeline tool — pairs with Spark/Trino/Flink. Brings DB guarantees to object storage |
| LakeFS | Git-like operations (branch, commit, merge) over object storage itself | Branch model gives isolated, reproducible data branches | Data engineering teams; branch-per-experiment; CI over data | Not a table format — versions objects. Composes with Delta/Iceberg on top |
- Pick one primary versioning model per platform. Mixing DVC's
pointer model with Delta's transaction-log model fragments
operational knowledge (C4 Locality).
- All three satisfy P2; the choice is which fits the data shape and
the team's existing tooling. None is advocated over the others.
## Reproducibility Contract (P1 Reproducibility is the First Class)
A reproducible training run records, in one versioned place:
```
run_id: 2026-08-05T09:12:00Z#run-42
dataset: s3://ml-data/train@sha256:7f3a...e21
splits: dvc.yaml@commit a1b2c4d
code: git@a1b2c4d
config: configs/train.yaml@commit a1b2c4d
environment: ghcr.io/org/train-img@sha256:9c2d...f88
eval_spec: configs/eval.yaml@commit a1b2c4d
model_digest: registry/model@sha256:b5e1...aa0
```
- Lose any line and the run is anecdote, not evidence.
- The record is the lineage root: a prediction cites the
`model_digest`, which cites the `run_id`, which cites everything
above. This is how P3 (Lineage Traceable End-to-End) is satisfied
in practice.
## What Violates Data Versioning Discipline
| Violation | Principle |
|-----------|-----------|
| Dataset referenced by `s3://bucket/latest/` | P2 Data is Versioned, Not Just Code |
| Splits regenerated with an unpinned seed per run | P2, P4 Evaluation Defined Before Training |
| A production model with no dataset hash in its lineage | P3 Lineage Traceable End-to-End |
| Editing a lineage record to "clean up" a broken trace | P3 Lineage Traceable End-to-End |
| Eval split reachable from the training data path | P4 Evaluation Defined Before Training |
| Two platforms versioning the same data with different models | C4 Locality |
domains/ai-ml/first-principles.md
+154
View File
@@ -0,0 +1,154 @@
# AI / ML — First Principles
> Scope per D-023: this domain covers ML **engineering discipline** —
> data versioning, evaluation methodology, serving patterns, and drift
> detection. It does **not** cover algorithm design, model architecture
> selection, hyperparameter tuning, or model-family comparison. Those
> are research choices, not engineering principles, and they have no
> derivation in the core C-rules.
## 1. The Principles
### P1. Reproducibility is the First Class
Every training run is reproducible from pinned data + code + config +
environment. An unreproducible run is an unreviewable run: you cannot
decide whether a result is correct if you cannot recreate it.
Reproducibility is the ML analogue of `domains/devops/P1
Reproducibility` and inherits its non-negotiable status. Lose any one
of data, code, config, or environment pinning, and the run is
anecdote, not evidence.
### P2. Data is Versioned, Not Just Code
Datasets, features, and train/val/test splits are first-class
versioned artifacts with content hashes and lineage. `git` alone is
insufficient — datasets do not fit in git, and a dataset is not a
commit hash. A model trained on "the data" is a model trained on an
unknown input, which is a correctness violation. Version data the way
you version code: pinned, named, and recoverable.
### P3. Lineage is Traceable End-to-End
Any deployed prediction traces back through model → training run →
dataset → source. No orphan models. A model in production with no
lineage is a correctness defect: you cannot reason about its failure
modes, you cannot roll it back to a known-good dataset, and you cannot
tell whether drift is in the model or in the data that built it.
Lineage is the audit trail of ML (`domains/observability/logging.md`).
### P4. Evaluation is Defined Before Training
Metrics, splits, and acceptance thresholds are declared a priori, in
code, before the model is trained. Cherry-picking metrics post-hoc is
a correctness violation: the evaluation is no longer measuring the
model, it is rationalizing it. The eval spec is a contract — it is
reviewable, it is versioned, and it is the gate the model must pass
before it leaves the experiment. This is the ML angle on C2 Clarity:
the intent of the model is obvious to its reader because the eval
declared it first.
### P5. Models are Versioned Artifacts
A model is a pinned, immutable, registry-tracked artifact with a
unique identifier. Never "the latest." A serving endpoint that pulls
"latest" is serving an unknown model — its behavior is undefined, its
rollback is impossible, and its lineage is broken. The model registry
is to models what a container registry is to images
(`domains/devops/P7 Immutability`): immutable, addressed by digest,
promoted by stage.
### P6. Serving is Observable
Inference latency, throughput, input distributions, and prediction
confidence are first-class signals. Silent serving is a bug. A model
in production that emits no metrics is a model you cannot operate: you
cannot see latency regressions, you cannot see input drift, you cannot
see a failing downstream consumer. Observability is designed in, not
bolted on (`domains/observability/metrics.md`).
### P7. Drift is Expected and Detected
Data drift, concept drift, and prediction drift are monitored as a
matter of course. A drift signal is an incident, not a curiosity. ML
systems decay without code changes — the world changes under the
model — so "no code changed" is not a defense against a serving
regression. Detecting drift is the ML-specific form of C7
Observability: you cannot fix a model you cannot see degrading.
### P8. Inference Inputs are Validated
The model's input contract — schema, value ranges, types, and
categorical domains — is enforced at the serving boundary.
Out-of-contract inputs are rejected, not silently scored. Scoring an
out-of-contract input is a correctness violation: the model's output
is undefined for inputs outside its training distribution, and
returning a number for it is lying to the caller. This is the ML angle
on `domains/security/input-validation.md` and inherits C1's
non-tradeable status.
### P9. Pipelines Compose, Notebooks Don't
Training and serving flows are composable pipelines with explicit
steps, named inputs, named outputs, and contracts between stages.
Notebooks are for exploration, not production. A notebook in the
serving path is a correctness defect: its state is implicit, its
order is human-dependent, and its reproducibility is whatever the last
operator remembered. Compose pipelines; keep notebooks in the lab.
### P10. Rollback Includes the Model
A serving rollback restores the prior model artifact, not just the
prior code. Promotion is reversible at the model layer. A rollback
that redeploys old code but keeps the new model has not rolled back —
the model was the thing that regressed. The rollback path must name
the prior model digest, the prior dataset version, and the prior eval
that cleared it. This is the ML angle on `domains/devops/P4 Rollback
First` and `domains/kubernetes/P10 Roll Forward, Roll Back`.
## 2. Core Principle Trace
Each AI/ML P-rule derives from one or more core C-rules (C1–C8). The
matrix extension lands in P4 of the v0.3 plan; the traces below are
authoritative.
| P-rule | Core | Why |
|--------|------|-----|
| P1 Reproducibility is the First Class | C1, C5 | Correctness of results; reversibility of runs |
| P2 Data is Versioned, Not Just Code | C5, C7 | Reversibility of datasets; observability of data lineage |
| P3 Lineage is Traceable End-to-End | C7, C1 | Observability of provenance; correctness of attribution |
| P4 Evaluation is Defined Before Training | C1, C2 | Correctness of the eval gate; clarity of a-priori intent |
| P5 Models are Versioned Artifacts | C5, C6 | Reversibility of model identity; composability of registry stages |
| P6 Serving is Observable | C7 | Observability of inference |
| P7 Drift is Expected and Detected | C7, C1 | Observability of degradation; correctness of detection |
| P8 Inference Inputs are Validated | C1 | Correctness of the serving boundary (security subset) |
| P9 Pipelines Compose, Notebooks Don't | C6, C2 | Composability of stages; clarity of explicit contracts |
| P10 Rollback Includes the Model | C5 | Reversibility at the model layer |
## 3. What Violates These Principles
| Violation | Principle Breached |
|-----------|-------------------|
| A training run that cannot be replayed from pinned inputs | P1 Reproducibility is the First Class |
| A dataset referenced by a mutable path, not a hash | P2 Data is Versioned, Not Just Code |
| A production model with no record of its training data | P3 Lineage is Traceable End-to-End |
| Metrics chosen after seeing the results | P4 Evaluation is Defined Before Training |
| A serving endpoint that pulls `latest` from the registry | P5 Models are Versioned Artifacts |
| A model in production with no latency or throughput metrics | P6 Serving is Observable |
| A serving regression dismissed as "no code changed" | P7 Drift is Expected and Detected |
| An input with an out-of-range feature scored silently | P8 Inference Inputs are Validated |
| A notebook in the serving or training pipeline path | P9 Pipelines Compose, Notebooks Don't |
| A rollback that restores code but keeps the regressed model | P10 Rollback Includes the Model |
## 4. Relationship to Other Domains
AI/ML is the engineering-discipline layer for model-bearing systems.
It borrows the reproducibility, immutability, rollback, and
observability disciplines of `domains/devops/` and applies them to
the data → model → serving lifecycle. Cross-links are one-directional
(per D-026 extended):
- `domains/devops/P1 Reproducibility` ← P1
- `domains/devops/P4 Rollback First` ← P10
- `domains/devops/P5 Progressive Delivery` ← P10 (model canary)
- `domains/devops/P7 Immutability` ← P5 (model images)
- `domains/data/migrations.md` ← P2 (data versioning ↔ migration discipline)
- `domains/data/schema-design.md` ← P8 (inference input contract)
- `domains/observability/metrics.md` ← P6, P7
- `domains/observability/logging.md` ← P3 (lineage)
- `domains/security/input-validation.md` ← P8
- `domains/security/secrets.md` ← P8 (serving credentials)
- `domains/performance/backend.md` ← P6 (serving latency)
- `domains/kubernetes/workloads.md` ← P9 (serving on k8s)
- `domains/testing/first-principles.md` ← P4 (eval as a gate)
- `domains/gitops-operators/first-principles.md` ← P10 (model rollback in a GitOps loop)
domains/ai-ml/model-evaluation.md
+94
View File
@@ -0,0 +1,94 @@
# Model Evaluation — Derived Rules
> Derives from `domains/ai-ml/first-principles.md`. Covers P4
> (Evaluation is Defined Before Training) and the eval-as-a-gate
> discipline. Referenced by `serving.md` (promotion gate) and
> `monitoring-drift.md` (online eval). Scope per D-023: evaluation
> methodology, not metric math or model-family benchmarks.
## Evaluation is a Gate, Not a Report (P4 Evaluation Defined Before Training)
- The eval spec — metrics, splits, thresholds, and pass/fail
criteria — is declared in code **before** the model is trained.
It is versioned with the data and the code; it is reviewable; it
is the contract the model must satisfy to leave the experiment.
- Cherry-picking metrics after seeing results is a correctness
violation: the eval is no longer measuring the model, it is
rationalizing it. The a-priori spec is what makes the eval
trustworthy.
- This is the ML angle on `domains/testing/first-principles.md` P1
(Tests as Specification): the eval declares the model's contract,
the model does not declare its own success.
## The Eval Input Contract (P8 Inference Inputs are Validated, cross `domains/data/schema-design.md`)
- The eval set has a schema: feature names, types, ranges, and
categorical domains. That schema is the same schema the serving
boundary enforces (`serving.md`, `domains/security/input-validation.md`).
- An eval set whose schema drifted from the serving schema is
measuring a different model than the one in production. Schema
parity is part of the versioned eval artifact.
- Cross `domains/data/schema-design.md`: the eval input contract is
a schema-design problem, versioned and reviewed like any schema.
## Holdout Integrity (P4, P2 Data is Versioned)
- The held-out eval set is never touched by training, feature
selection, or threshold tuning. A "held out" set that influenced
any training decision is not held out — it is a third training
set, and the eval is measuring memorization.
- Splits are versioned with the dataset (`data-versioning.md`).
Recreating splits ad-hoc per run breaks comparability across runs.
- Reusing a held-out set across many model iterations leaks it
incrementally. Rotate or re-split on a cadence; record the
rotation in lineage.
## Offline vs Online Evaluation (P6 Serving is Observable)
- **Offline eval** runs before promotion: held-out data, pinned
model, declared metrics, pass/fail gate. It answers "should this
model ship?"
- **Online eval** runs after promotion, on live traffic: shadow
scoring, A/B, canary metrics. It answers "is this model behaving
in production?" It is the bridge to `monitoring-drift.md`.
- A model that passed offline and regressed online is not a
contradiction — it is a signal that the offline distribution
differs from the live one (a P7 drift signal). Both eval layers
are required; neither substitutes for the other.
## Bias and Fairness Checks (Engineering Angle, P4)
- Bias/fairness checks are part of the a-priori eval spec, not an
afterthought. They are metrics with thresholds, declared before
training, gated the same as any metric.
- This doc covers the **engineering** discipline: the checks are
versioned, gated, and recorded in lineage. The choice of which
fairness metrics and what thresholds are policy decisions, not
engineering principles, and are out of scope here (D-023).
## Eval-as-a-Gate in the Pipeline (P9 Pipelines Compose)
- The eval is a pipeline stage with a contract: input = model
digest + eval dataset version; output = pass/fail + metric
report. It composes with the training stage and the promotion
stage.
- A promotion that bypasses the eval stage is a P4 violation,
regardless of who approved it. The gate is in the pipeline, not
in a human sign-off sheet.
```
train -> eval(gate) -> register(promote) -> serve
|
+-- fail -> abort, no promote
```
## What Violates Evaluation Discipline
| Violation | Principle |
|-----------|-----------|
| Metrics chosen after seeing the scores | P4 Evaluation Defined Before Training |
| Held-out set used in feature selection or threshold tuning | P4, P2 |
| Eval schema differs from serving schema | P8 Inference Inputs are Validated |
| Promotion by human approval, bypassing the eval stage | P4, P9 Pipelines Compose |
| A "passing" model with no online eval in production | P6 Serving is Observable |
| Fairness checks added after a model shipped | P4 Evaluation Defined Before Training |
domains/ai-ml/monitoring-drift.md
+88
View File
@@ -0,0 +1,88 @@
# Monitoring & Drift — Derived Rules
> Derives from `domains/ai-ml/first-principles.md`. Covers P7 (Drift
> is Expected and Detected) and the online half of P6 (Serving is
> Observable). Referenced by `serving.md` (online eval) and
> `model-evaluation.md` (online layer). Scope per D-023: drift
> detection methodology, not model retraining architecture.
## Drift is Expected and Detected (P7 Drift is Expected and Detected)
- ML systems decay without code changes. The world changes under
the model: user behavior shifts, input pipelines change,
upstream schemas evolve. "No code changed" is not a defense
against a serving regression.
- A drift signal is an incident, not a curiosity. It triggers an
alert, an investigation, and a decision (retrain, roll back, or
accept with a recorded justification). Silent drift is the same
class of bug as silent serving (P6).
- Cross `domains/observability/metrics.md` for the alerting
primitives and `domains/observability/logging.md` for the
structured events a drift signal emits.
## The Three Drift Types (IDEATE-30, D-048)
| Drift Type | What Changes | Detection Signal | Source of Truth |
|------------|--------------|------------------|-----------------|
| **Data drift** (input drift) | The distribution of inputs at serving time diverges from the distribution the model was trained on | Statistical distance between the live input distribution and the pinned training-set distribution (e.g., PSI, KL, KS test). Alert on threshold breach | Training dataset hash (`data-versioning.md`) + live input metrics |
| **Concept drift** | The relationship between inputs and the target changes — the same input now maps to a different correct output | Ground-truth lag: compare delayed labels against predictions on the same inputs. Rising error rate against a stable input distribution signals concept, not data, drift | Delayed-label feedback stream + prediction log |
| **Prediction drift** (output drift) | The distribution of the model's predictions shifts, with no change to inputs | Statistical distance between the live prediction distribution and a pinned baseline prediction distribution. Independent of inputs — catches model-internal regressions and upstream silent changes | Prediction log + baseline prediction snapshot |
- The three signals are distinct and non-substitutable. Data drift
catches the input changing; concept drift catches the world
changing; prediction drift catches the model's behavior changing.
A monitoring setup with only one is blind to two classes of
regression.
- Evidently AI and Great Expectations are the canonical tooling:
Evidently for drift/statistical reports, Great Expectations for
data-quality/contract checks at the pipeline boundary. Both
produce the metrics that feed `domains/observability/metrics.md`.
## Detection Signals in Practice
- **Data drift** compares live inputs to the **pinned training
distribution** — not to "yesterday's inputs." Without a pinned
baseline, drift is measured against a moving target and is
meaningless. Cross `data-versioning.md` for how the baseline is
pinned.
- **Concept drift** requires ground truth, which is often delayed
(days/weeks). The detection signal is the gap between
prediction-time confidence and delayed-label error. A rising
error against stable inputs is the signature.
- **Prediction drift** needs no ground truth and no input
comparison — it watches the model's own output distribution. It
is the cheapest signal and the first to fire; it is also the
least specific (any of the three drifts can move predictions).
## Alerting and Retraining Triggers (P7, P10 Rollback Includes the Model)
- A drift alert is an incident. It does not auto-trigger retraining
unsupervised — auto-retraining on drift can lock in a bad
distribution. The alert triggers a human decision: investigate,
retrain, roll back, or accept.
- Retraining is a new training run (`first-principles.md` P1): it
produces a new model digest, passes the eval gate
(`model-evaluation.md`), and is promoted through the registry
(`serving.md`). The prior model stays rollbackable (P10).
- Cross `domains/observability/metrics.md` for the alert-rule
pattern: threshold + window + severity, routed to the same
on-call path as any production incident.
## Online Evaluation Bridge (P6 Serving is Observable)
- Online eval (`model-evaluation.md`) is the live counterpart to
drift monitoring: shadow scores and A/B canaries measure a
candidate model against the incumbent, while drift monitoring
measures the incumbent against its own baseline. Both feed the
same metrics pipeline.
## What Violates Monitoring Discipline
| Violation | Principle |
|-----------|-----------|
| Only one drift type monitored | P7 Drift is Expected and Detected |
| Drift baseline is "yesterday's inputs," not pinned training data | P7, P2 Data is Versioned |
| Drift alert that auto-retrains without a human gate | P7, P1 Reproducibility |
| A serving regression dismissed as "no code changed" | P7 Drift is Expected and Detected |
| Concept-drift check with no delayed-label feedback path | P7 Drift is Expected and Detected |
| Prediction-distribution change with no alert | P6 Serving is Observable, P7 |
domains/ai-ml/serving.md
+88
View File
@@ -0,0 +1,88 @@
# Serving — Derived Rules
> Derives from `domains/ai-ml/first-principles.md`. Covers P5 (Models
> are Versioned Artifacts), P6 (Serving is Observable), P8 (Inference
> Inputs are Validated), and P10 (Rollback Includes the Model).
> Referenced by `monitoring-drift.md` (online signals) and
> `model-evaluation.md` (promotion gate). Scope per D-023: serving
> patterns, not model architectures.
## The Model is an Addressed Artifact (P5 Models are Versioned Artifacts)
- A serving endpoint pulls a model by digest, never by `latest`. A
model pulled by `latest` is an unknown model — its behavior is
undefined and its rollback is impossible.
- The model registry is to models what a container registry is to
images (`domains/devops/P7 Immutability`): immutable, addressed by
digest, promoted by stage (staging → prod). Promotion is a
registry operation, not a file copy.
- A serving rollout names the model digest in its manifest. The
digest is part of the deploy's lineage (`data-versioning.md`).
## Inference Inputs are Validated (P8 Inference Inputs are Validated)
- The model's input contract — schema, types, ranges, categorical
domains — is enforced at the serving boundary, before the model
sees the input. Out-of-contract inputs are rejected with a
defined error, not silently scored.
- Scoring an out-of-contract input is a C1 (Correctness) violation:
the model's output is undefined outside its training
distribution, and returning a number for it is lying to the
caller.
- This is the ML angle on `domains/security/input-validation.md`:
the validation lives at the boundary, the model is downstream of
it, and the contract is versioned with the model.
## Serving is Observable (P6 Serving is Observable)
- Every inference path emits: request latency, throughput, input
distribution summaries, prediction confidence, and error counts.
Silent serving is a bug.
- Cross `domains/observability/metrics.md` for the metrics
primitives (histograms, counters, gauges) and
`domains/observability/tracing.md` for the request-level trace
that ties an input to a prediction.
- Latency SLAs are enforced via `domains/performance/backend.md`
disciplines: budget the inference path, measure the tail (p99),
alert on budget breach.
## Serving Patterns (P9 Pipelines Compose)
| Pattern | When | Notes |
|---------|------|-------|
| Inference as a service | Default; model behind an HTTP/gRPC endpoint | KServe, Seldon Core, BentoML. Scales with traffic; model is a deployable, addressable artifact |
| Batch inference | Offline scoring of large datasets | No latency SLA; throughput-bound. Same model digest, same input contract |
| Embedded / in-process | Latency-critical, single-tenant | Model linked into the app. Trades observability for latency — only when the SLA demands it |
- Canarying a model is a serving pattern, not a deployment pattern:
shift a fraction of traffic to the new model digest, measure
online eval (`model-evaluation.md`), abort to the prior digest on
regression. This is `domains/devops/P5 Progressive Delivery`
applied at the model layer.
- Rollback restores the prior model digest (P10 Rollback Includes
the Model). A rollback that redeploys old code but keeps the new
model has not rolled back. Cross `domains/gitops-operators/first-principles.md`
for the GitOps reconciliation loop that drives model rollouts.
## Tool Landscape (KServe / Seldon Core / BentoML)
| Tool | Model Packaging | Deployment Surface | Notes |
|------|-----------------|--------------------|-------|
| KServe | InferenceService CRD; runtime predictors (v2, HuggingFace, PMML, custom) | Kubernetes-native; CRD-driven | Cross `domains/kubernetes/workloads.md`. Brings the k8s reconciliation model to serving |
| Seldon Core | SeldonDeployment CRD; graph of predictors | Kubernetes-native; CRD-driven | Emphasizes inference graphs (fan-out, ensemble) as CRD structure |
| BentoML | Bento (model + runtime + deps packaged); Yatai registry | Kubernetes or bare container | Focuses on packaging + registry; the Bento is the versioned artifact (P5) |
- All three satisfy P5/P6/P8 when wired correctly; the choice is
packaging model and deployment surface, not correctness.
- None is advocated over the others.
## What Violates Serving Discipline
| Violation | Principle |
|-----------|-----------|
| Endpoint pulls `latest` from the registry | P5 Models are Versioned Artifacts |
| Out-of-range input scored silently | P8 Inference Inputs are Validated |
| Serving path emits no latency or throughput metrics | P6 Serving is Observable |
| Rollback redeploys code but keeps the regressed model | P10 Rollback Includes the Model |
| A notebook in the serving path | P9 Pipelines Compose, Notebooks Don't |
| Canary with no abort-to-prior-digest path | P10, `domains/devops/P5 Progressive Delivery` |