Jon Chery
9ebc9c8868
---ci--- project: atelier phase: 6 milestone: v0.3 status: complete requirements: covered: [ATELIER-60..91] partial: [] ---/ci---
88 lines
4.8 KiB
Markdown
88 lines
4.8 KiB
Markdown
# Serving — Derived Rules
|
|
|
|
> Derives from `domains/ai-ml/first-principles.md`. Covers P5 (Models
|
|
> are Versioned Artifacts), P6 (Serving is Observable), P8 (Inference
|
|
> Inputs are Validated), and P10 (Rollback Includes the Model).
|
|
> Referenced by `monitoring-drift.md` (online signals) and
|
|
> `model-evaluation.md` (promotion gate). Scope per D-023: serving
|
|
> patterns, not model architectures.
|
|
|
|
## The Model is an Addressed Artifact (P5 Models are Versioned Artifacts)
|
|
|
|
- A serving endpoint pulls a model by digest, never by `latest`. A
|
|
model pulled by `latest` is an unknown model — its behavior is
|
|
undefined and its rollback is impossible.
|
|
- The model registry is to models what a container registry is to
|
|
images (`domains/devops/P7 Immutability`): immutable, addressed by
|
|
digest, promoted by stage (staging → prod). Promotion is a
|
|
registry operation, not a file copy.
|
|
- A serving rollout names the model digest in its manifest. The
|
|
digest is part of the deploy's lineage (`data-versioning.md`).
|
|
|
|
## Inference Inputs are Validated (P8 Inference Inputs are Validated)
|
|
|
|
- The model's input contract — schema, types, ranges, categorical
|
|
domains — is enforced at the serving boundary, before the model
|
|
sees the input. Out-of-contract inputs are rejected with a
|
|
defined error, not silently scored.
|
|
- Scoring an out-of-contract input is a C1 (Correctness) violation:
|
|
the model's output is undefined outside its training
|
|
distribution, and returning a number for it is lying to the
|
|
caller.
|
|
- This is the ML angle on `domains/security/input-validation.md`:
|
|
the validation lives at the boundary, the model is downstream of
|
|
it, and the contract is versioned with the model.
|
|
|
|
## Serving is Observable (P6 Serving is Observable)
|
|
|
|
- Every inference path emits: request latency, throughput, input
|
|
distribution summaries, prediction confidence, and error counts.
|
|
Silent serving is a bug.
|
|
- Cross `domains/observability/metrics.md` for the metrics
|
|
primitives (histograms, counters, gauges) and
|
|
`domains/observability/tracing.md` for the request-level trace
|
|
that ties an input to a prediction.
|
|
- Latency SLAs are enforced via `domains/performance/backend.md`
|
|
disciplines: budget the inference path, measure the tail (p99),
|
|
alert on budget breach.
|
|
|
|
## Serving Patterns (P9 Pipelines Compose)
|
|
|
|
| Pattern | When | Notes |
|
|
|---------|------|-------|
|
|
| Inference as a service | Default; model behind an HTTP/gRPC endpoint | KServe, Seldon Core, BentoML. Scales with traffic; model is a deployable, addressable artifact |
|
|
| Batch inference | Offline scoring of large datasets | No latency SLA; throughput-bound. Same model digest, same input contract |
|
|
| Embedded / in-process | Latency-critical, single-tenant | Model linked into the app. Trades observability for latency — only when the SLA demands it |
|
|
|
|
- Canarying a model is a serving pattern, not a deployment pattern:
|
|
shift a fraction of traffic to the new model digest, measure
|
|
online eval (`model-evaluation.md`), abort to the prior digest on
|
|
regression. This is `domains/devops/P5 Progressive Delivery`
|
|
applied at the model layer.
|
|
- Rollback restores the prior model digest (P10 Rollback Includes
|
|
the Model). A rollback that redeploys old code but keeps the new
|
|
model has not rolled back. Cross `domains/gitops-operators/first-principles.md`
|
|
for the GitOps reconciliation loop that drives model rollouts.
|
|
|
|
## Tool Landscape (KServe / Seldon Core / BentoML)
|
|
|
|
| Tool | Model Packaging | Deployment Surface | Notes |
|
|
|------|-----------------|--------------------|-------|
|
|
| KServe | InferenceService CRD; runtime predictors (v2, HuggingFace, PMML, custom) | Kubernetes-native; CRD-driven | Cross `domains/kubernetes/workloads.md`. Brings the k8s reconciliation model to serving |
|
|
| Seldon Core | SeldonDeployment CRD; graph of predictors | Kubernetes-native; CRD-driven | Emphasizes inference graphs (fan-out, ensemble) as CRD structure |
|
|
| BentoML | Bento (model + runtime + deps packaged); Yatai registry | Kubernetes or bare container | Focuses on packaging + registry; the Bento is the versioned artifact (P5) |
|
|
|
|
- All three satisfy P5/P6/P8 when wired correctly; the choice is
|
|
packaging model and deployment surface, not correctness.
|
|
- None is advocated over the others.
|
|
|
|
## What Violates Serving Discipline
|
|
|
|
| Violation | Principle |
|
|
|-----------|-----------|
|
|
| Endpoint pulls `latest` from the registry | P5 Models are Versioned Artifacts |
|
|
| Out-of-range input scored silently | P8 Inference Inputs are Validated |
|
|
| Serving path emits no latency or throughput metrics | P6 Serving is Observable |
|
|
| Rollback redeploys code but keeps the regressed model | P10 Rollback Includes the Model |
|
|
| A notebook in the serving path | P9 Pipelines Compose, Notebooks Don't |
|
|
| Canary with no abort-to-prior-digest path | P10, `domains/devops/P5 Progressive Delivery` | |