atelier/domains/ai-ml/monitoring-drift.md

Monitoring & Drift — Derived Rules

Derives from domains/ai-ml/first-principles.md. Covers P7 (Drift is Expected and Detected) and the online half of P6 (Serving is Observable). Referenced by serving.md (online eval) and model-evaluation.md (online layer). Scope per D-023: drift detection methodology, not model retraining architecture.

Drift is Expected and Detected (P7 Drift is Expected and Detected)

• ML systems decay without code changes. The world changes under the model: user behavior shifts, input pipelines change, upstream schemas evolve. "No code changed" is not a defense against a serving regression.
• A drift signal is an incident, not a curiosity. It triggers an alert, an investigation, and a decision (retrain, roll back, or accept with a recorded justification). Silent drift is the same class of bug as silent serving (P6).
• Cross domains/observability/metrics.md for the alerting primitives and domains/observability/logging.md for the structured events a drift signal emits.

The Three Drift Types (IDEATE-30, D-048)

Drift Type	What Changes	Detection Signal	Source of Truth
Data drift (input drift)	The distribution of inputs at serving time diverges from the distribution the model was trained on	Statistical distance between the live input distribution and the pinned training-set distribution (e.g., PSI, KL, KS test). Alert on threshold breach	Training dataset hash (data-versioning.md) + live input metrics
Concept drift	The relationship between inputs and the target changes — the same input now maps to a different correct output	Ground-truth lag: compare delayed labels against predictions on the same inputs. Rising error rate against a stable input distribution signals concept, not data, drift	Delayed-label feedback stream + prediction log
Prediction drift (output drift)	The distribution of the model's predictions shifts, with no change to inputs	Statistical distance between the live prediction distribution and a pinned baseline prediction distribution. Independent of inputs — catches model-internal regressions and upstream silent changes	Prediction log + baseline prediction snapshot

• The three signals are distinct and non-substitutable. Data drift catches the input changing; concept drift catches the world changing; prediction drift catches the model's behavior changing. A monitoring setup with only one is blind to two classes of regression.
• Evidently AI and Great Expectations are the canonical tooling: Evidently for drift/statistical reports, Great Expectations for data-quality/contract checks at the pipeline boundary. Both produce the metrics that feed domains/observability/metrics.md.

Detection Signals in Practice

• Data drift compares live inputs to the pinned training distribution — not to "yesterday's inputs." Without a pinned baseline, drift is measured against a moving target and is meaningless. Cross data-versioning.md for how the baseline is pinned.
• Concept drift requires ground truth, which is often delayed (days/weeks). The detection signal is the gap between prediction-time confidence and delayed-label error. A rising error against stable inputs is the signature.
• Prediction drift needs no ground truth and no input comparison — it watches the model's own output distribution. It is the cheapest signal and the first to fire; it is also the least specific (any of the three drifts can move predictions).

Alerting and Retraining Triggers (P7, P10 Rollback Includes the Model)

• A drift alert is an incident. It does not auto-trigger retraining unsupervised — auto-retraining on drift can lock in a bad distribution. The alert triggers a human decision: investigate, retrain, roll back, or accept.
• Retraining is a new training run (first-principles.md P1): it produces a new model digest, passes the eval gate (model-evaluation.md), and is promoted through the registry (serving.md). The prior model stays rollbackable (P10).
• Cross domains/observability/metrics.md for the alert-rule pattern: threshold + window + severity, routed to the same on-call path as any production incident.

Online Evaluation Bridge (P6 Serving is Observable)

• Online eval (model-evaluation.md) is the live counterpart to drift monitoring: shadow scores and A/B canaries measure a candidate model against the incumbent, while drift monitoring measures the incumbent against its own baseline. Both feed the same metrics pipeline.

What Violates Monitoring Discipline

Violation	Principle
Only one drift type monitored	P7 Drift is Expected and Detected
Drift baseline is "yesterday's inputs," not pinned training data	P7, P2 Data is Versioned
Drift alert that auto-retrains without a human gate	P7, P1 Reproducibility
A serving regression dismissed as "no code changed"	P7 Drift is Expected and Detected
Concept-drift check with no delayed-label feedback path	P7 Drift is Expected and Detected
Prediction-distribution change with no alert	P6 Serving is Observable, P7