atelier/domains/ai-ml/serving.md

Serving — Derived Rules

Derives from domains/ai-ml/first-principles.md. Covers P5 (Models are Versioned Artifacts), P6 (Serving is Observable), P8 (Inference Inputs are Validated), and P10 (Rollback Includes the Model). Referenced by monitoring-drift.md (online signals) and model-evaluation.md (promotion gate). Scope per D-023: serving patterns, not model architectures.

The Model is an Addressed Artifact (P5 Models are Versioned Artifacts)

• A serving endpoint pulls a model by digest, never by latest. A model pulled by latest is an unknown model — its behavior is undefined and its rollback is impossible.
• The model registry is to models what a container registry is to images (domains/devops/P7 Immutability): immutable, addressed by digest, promoted by stage (staging → prod). Promotion is a registry operation, not a file copy.
• A serving rollout names the model digest in its manifest. The digest is part of the deploy's lineage (data-versioning.md).

Inference Inputs are Validated (P8 Inference Inputs are Validated)

• The model's input contract — schema, types, ranges, categorical domains — is enforced at the serving boundary, before the model sees the input. Out-of-contract inputs are rejected with a defined error, not silently scored.
• Scoring an out-of-contract input is a C1 (Correctness) violation: the model's output is undefined outside its training distribution, and returning a number for it is lying to the caller.
• This is the ML angle on domains/security/input-validation.md: the validation lives at the boundary, the model is downstream of it, and the contract is versioned with the model.

Serving is Observable (P6 Serving is Observable)

• Every inference path emits: request latency, throughput, input distribution summaries, prediction confidence, and error counts. Silent serving is a bug.
• Cross domains/observability/metrics.md for the metrics primitives (histograms, counters, gauges) and domains/observability/tracing.md for the request-level trace that ties an input to a prediction.
• Latency SLAs are enforced via domains/performance/backend.md disciplines: budget the inference path, measure the tail (p99), alert on budget breach.

Serving Patterns (P9 Pipelines Compose)

Pattern	When	Notes
Inference as a service	Default; model behind an HTTP/gRPC endpoint	KServe, Seldon Core, BentoML. Scales with traffic; model is a deployable, addressable artifact
Batch inference	Offline scoring of large datasets	No latency SLA; throughput-bound. Same model digest, same input contract
Embedded / in-process	Latency-critical, single-tenant	Model linked into the app. Trades observability for latency — only when the SLA demands it

• Canarying a model is a serving pattern, not a deployment pattern: shift a fraction of traffic to the new model digest, measure online eval (model-evaluation.md), abort to the prior digest on regression. This is domains/devops/P5 Progressive Delivery applied at the model layer.
• Rollback restores the prior model digest (P10 Rollback Includes the Model). A rollback that redeploys old code but keeps the new model has not rolled back. Cross domains/gitops-operators/first-principles.md for the GitOps reconciliation loop that drives model rollouts.

Tool Landscape (KServe / Seldon Core / BentoML)

Tool	Model Packaging	Deployment Surface	Notes
KServe	InferenceService CRD; runtime predictors (v2, HuggingFace, PMML, custom)	Kubernetes-native; CRD-driven	Cross domains/kubernetes/workloads.md. Brings the k8s reconciliation model to serving
Seldon Core	SeldonDeployment CRD; graph of predictors	Kubernetes-native; CRD-driven	Emphasizes inference graphs (fan-out, ensemble) as CRD structure
BentoML	Bento (model + runtime + deps packaged); Yatai registry	Kubernetes or bare container	Focuses on packaging + registry; the Bento is the versioned artifact (P5)

• All three satisfy P5/P6/P8 when wired correctly; the choice is packaging model and deployment surface, not correctness.
• None is advocated over the others.

What Violates Serving Discipline

Violation	Principle
Endpoint pulls latest from the registry	P5 Models are Versioned Artifacts
Out-of-range input scored silently	P8 Inference Inputs are Validated
Serving path emits no latency or throughput metrics	P6 Serving is Observable
Rollback redeploys code but keeps the regressed model	P10 Rollback Includes the Model
A notebook in the serving path	P9 Pipelines Compose, Notebooks Don't
Canary with no abort-to-prior-digest path	P10, domains/devops/P5 Progressive Delivery