docs(P2): complete ai-ml domain phase

---ci---
project: atelier
phase: 2
milestone: v0.3
status: complete
requirements:
  covered: [ATELIER-65, ATELIER-66, ATELIER-67, ATELIER-68, ATELIER-69]
  partial: []
---/ci---

.ciagent/CHECKPOINT.json

@@ -1,9 +1,11 @@
 {
-  "phase": 0,
+  "phase": 1,
-  "stage": "clarify",
+  "stage": "complete",
   "milestone": "v0.3",
-  "phase_role": "pre_execution",
+  "phase_role": "execution",
   "project": "atelier",
   "attempts": 0,
-  "updated_at": "2026-08-05T03:08:00Z"
+  "updated_at": "2026-08-05T03:25:00Z",
+  "phase_tag": "v0.2.1",
+  "release_id": 469
 }

.ciagent/atelier/ROADMAP.md

@@ -84,8 +84,8 @@ NFR milestone: no separate minor tag. The final patch (v0.1.5) IS the v0.2 deliv
 
 | Phase | Name | Type | Status | Key Deliverables |
 |-------|------|------|--------|------------------|
-| 0 | Pre-Execution | docs | active | Spec, clarify, research, ideate, plan, PERSONAS.md (extends platform-engineer, adds ml-engineer) |
+| 0 | Pre-Execution | docs | complete | Spec, clarify, research, ideate, plan, PERSONAS.md (extends platform-engineer, adds ml-engineer) — shipped v0.2.0 |
-| 1 | GitOps + Operators Domain | docs | pending | domains/gitops-operators/{first-principles, argocd, flux, operators, progressive-delivery}.md |
+| 1 | GitOps + Operators Domain | docs | complete | domains/gitops-operators/{first-principles, argocd, flux, operators, progressive-delivery}.md — shipped v0.2.1 |
 | 2 | AI/ML Domain | docs | pending | domains/ai-ml/{first-principles, data-versioning, model-evaluation, serving, monitoring-drift}.md |
 | 3 | i18n + Compliance Domains | docs | pending | domains/i18n/{first-principles, locale-resources, formatting, rtl-bidi, testing-i18n}.md, domains/compliance/{first-principles, audit-logs, data-retention, policy-as-code, evidence}.md |
 | 4 | Matrix + Review Integration | docs | pending | matrix/principles-matrix.md (+40 mappings), matrix/domain-coverage.md (incl. C-rule coverage table update), review/{agent-checklist, peer-review-checklist, anti-patterns}.md, MANIFEST.md (+ examples/ listing per ATELIER-91) |

domains/ai-ml/data-versioning.md

@@ -0,0 +1,90 @@
+# Data Versioning — Derived Rules
+
+> Derives from `domains/ai-ml/first-principles.md`. Covers P2 (Data is
+> Versioned, Not Just Code) and P3 (Lineage is Traceable End-to-End).
+> Referenced by `serving.md` and `monitoring-drift.md`. Scope per
+> D-023: engineering discipline of versioning data, not dataset
+> content design.
+
+## Why Data Versioning (P2 Data is Versioned, Not Just Code)
+
+- `git` versions code well and data badly. Datasets do not fit in
+  git, and a dataset is not recovered from a commit hash.
+- A model trained on "the data" is a model trained on an unknown
+  input — a C1 (Correctness) violation. The dataset is a build
+  input; it is named, hashed, and recoverable the way any build
+  input is.
+- Data versioning is the ML analogue of `domains/data/migrations.md`:
+  the schema and contents of the data evolve, every evolution is a
+  versioned migration, and every model points at a specific version.
+
+## Dataset Hashing and Lineage (P3 Lineage Traceable End-to-End)
+
+- Every dataset version has a content hash (not a filename or a
+  timestamp). The hash is the identity. A model's lineage record
+  names the dataset hash it was trained on; a serving prediction
+  names the model digest it came from.
+- Lineage is a graph: prediction → model → training run → dataset →
+  source(s). Any edge missing is an orphan (`domains/observability/logging.md`
+  for the structured-log angle on lineage events).
+- The lineage record is append-only. Editing it to "fix" a broken
+  trace is the same class of violation as editing an audit log.
+
+## Train/Val/Test Split Versioning (P2, P4 Eval Defined Before Training)
+
+- Splits are versioned with the dataset, not derived ad-hoc per run.
+  A split is a deterministic function of (dataset version, split
+  config, random seed). Two runs on the same pinned inputs produce
+  the same splits.
+- The eval split is held out and never touched by training. A "held
+  out" set that leaked into training is a P4 (Evaluation Defined
+  Before Training) violation, not just a P2 violation — the eval
+  gate is measuring the training set, not the model.
+- Cross `domains/data/schema-design.md` for the eval input contract:
+  the schema of the eval set is part of the versioned artifact.
+
+## Tool Comparison (IDEATE-22, D-040)
+
+| Tool | Versioning Model | Lineage | Best For | Notes |
+|------|------------------|---------|----------|-------|
+| DVC | Git-like pointers to content-addressed object store; `.dvc` files in git track data versions | Pipeline DAG in `dvc.yaml`; reproducibility via `dvc repro` | Teams already on git; file/directory datasets; ML pipelines | Treats data like code; shares git's history model. Object store is pluggable (S3, GCS, Azure, SSH) |
+| Delta Lake | Table format with transaction log (ACID) + time travel via versioned commits; schema enforcement | Time travel queries; lineage via table history + catalog | Large tabular data; lakehouse; streaming + batch on the same table | Not a pipeline tool — pairs with Spark/Trino/Flink. Brings DB guarantees to object storage |
+| LakeFS | Git-like operations (branch, commit, merge) over object storage itself | Branch model gives isolated, reproducible data branches | Data engineering teams; branch-per-experiment; CI over data | Not a table format — versions objects. Composes with Delta/Iceberg on top |
+
+- Pick one primary versioning model per platform. Mixing DVC's
+  pointer model with Delta's transaction-log model fragments
+  operational knowledge (C4 Locality).
+- All three satisfy P2; the choice is which fits the data shape and
+  the team's existing tooling. None is advocated over the others.
+
+## Reproducibility Contract (P1 Reproducibility is the First Class)
+
+A reproducible training run records, in one versioned place:
+
+```
+run_id:        2026-08-05T09:12:00Z#run-42
+dataset:       s3://ml-data/train@sha256:7f3a...e21
+splits:        dvc.yaml@commit a1b2c4d
+code:          git@a1b2c4d
+config:        configs/train.yaml@commit a1b2c4d
+environment:   ghcr.io/org/train-img@sha256:9c2d...f88
+eval_spec:     configs/eval.yaml@commit a1b2c4d
+model_digest:  registry/model@sha256:b5e1...aa0
+```
+
+- Lose any line and the run is anecdote, not evidence.
+- The record is the lineage root: a prediction cites the
+  `model_digest`, which cites the `run_id`, which cites everything
+  above. This is how P3 (Lineage Traceable End-to-End) is satisfied
+  in practice.
+
+## What Violates Data Versioning Discipline
+
+| Violation | Principle |
+|-----------|-----------|
+| Dataset referenced by `s3://bucket/latest/` | P2 Data is Versioned, Not Just Code |
+| Splits regenerated with an unpinned seed per run | P2, P4 Evaluation Defined Before Training |
+| A production model with no dataset hash in its lineage | P3 Lineage Traceable End-to-End |
+| Editing a lineage record to "clean up" a broken trace | P3 Lineage Traceable End-to-End |
+| Eval split reachable from the training data path | P4 Evaluation Defined Before Training |
+| Two platforms versioning the same data with different models | C4 Locality |

domains/ai-ml/first-principles.md

@@ -0,0 +1,154 @@
+# AI / ML — First Principles
+
+> Scope per D-023: this domain covers ML **engineering discipline** —
+> data versioning, evaluation methodology, serving patterns, and drift
+> detection. It does **not** cover algorithm design, model architecture
+> selection, hyperparameter tuning, or model-family comparison. Those
+> are research choices, not engineering principles, and they have no
+> derivation in the core C-rules.
+
+## 1. The Principles
+
+### P1. Reproducibility is the First Class
+Every training run is reproducible from pinned data + code + config +
+environment. An unreproducible run is an unreviewable run: you cannot
+decide whether a result is correct if you cannot recreate it.
+Reproducibility is the ML analogue of `domains/devops/P1
+Reproducibility` and inherits its non-negotiable status. Lose any one
+of data, code, config, or environment pinning, and the run is
+anecdote, not evidence.
+
+### P2. Data is Versioned, Not Just Code
+Datasets, features, and train/val/test splits are first-class
+versioned artifacts with content hashes and lineage. `git` alone is
+insufficient — datasets do not fit in git, and a dataset is not a
+commit hash. A model trained on "the data" is a model trained on an
+unknown input, which is a correctness violation. Version data the way
+you version code: pinned, named, and recoverable.
+
+### P3. Lineage is Traceable End-to-End
+Any deployed prediction traces back through model → training run →
+dataset → source. No orphan models. A model in production with no
+lineage is a correctness defect: you cannot reason about its failure
+modes, you cannot roll it back to a known-good dataset, and you cannot
+tell whether drift is in the model or in the data that built it.
+Lineage is the audit trail of ML (`domains/observability/logging.md`).
+
+### P4. Evaluation is Defined Before Training
+Metrics, splits, and acceptance thresholds are declared a priori, in
+code, before the model is trained. Cherry-picking metrics post-hoc is
+a correctness violation: the evaluation is no longer measuring the
+model, it is rationalizing it. The eval spec is a contract — it is
+reviewable, it is versioned, and it is the gate the model must pass
+before it leaves the experiment. This is the ML angle on C2 Clarity:
+the intent of the model is obvious to its reader because the eval
+declared it first.
+
+### P5. Models are Versioned Artifacts
+A model is a pinned, immutable, registry-tracked artifact with a
+unique identifier. Never "the latest." A serving endpoint that pulls
+"latest" is serving an unknown model — its behavior is undefined, its
+rollback is impossible, and its lineage is broken. The model registry
+is to models what a container registry is to images
+(`domains/devops/P7 Immutability`): immutable, addressed by digest,
+promoted by stage.
+
+### P6. Serving is Observable
+Inference latency, throughput, input distributions, and prediction
+confidence are first-class signals. Silent serving is a bug. A model
+in production that emits no metrics is a model you cannot operate: you
+cannot see latency regressions, you cannot see input drift, you cannot
+see a failing downstream consumer. Observability is designed in, not
+bolted on (`domains/observability/metrics.md`).
+
+### P7. Drift is Expected and Detected
+Data drift, concept drift, and prediction drift are monitored as a
+matter of course. A drift signal is an incident, not a curiosity. ML
+systems decay without code changes — the world changes under the
+model — so "no code changed" is not a defense against a serving
+regression. Detecting drift is the ML-specific form of C7
+Observability: you cannot fix a model you cannot see degrading.
+
+### P8. Inference Inputs are Validated
+The model's input contract — schema, value ranges, types, and
+categorical domains — is enforced at the serving boundary.
+Out-of-contract inputs are rejected, not silently scored. Scoring an
+out-of-contract input is a correctness violation: the model's output
+is undefined for inputs outside its training distribution, and
+returning a number for it is lying to the caller. This is the ML angle
+on `domains/security/input-validation.md` and inherits C1's
+non-tradeable status.
+
+### P9. Pipelines Compose, Notebooks Don't
+Training and serving flows are composable pipelines with explicit
+steps, named inputs, named outputs, and contracts between stages.
+Notebooks are for exploration, not production. A notebook in the
+serving path is a correctness defect: its state is implicit, its
+order is human-dependent, and its reproducibility is whatever the last
+operator remembered. Compose pipelines; keep notebooks in the lab.
+
+### P10. Rollback Includes the Model
+A serving rollback restores the prior model artifact, not just the
+prior code. Promotion is reversible at the model layer. A rollback
+that redeploys old code but keeps the new model has not rolled back —
+the model was the thing that regressed. The rollback path must name
+the prior model digest, the prior dataset version, and the prior eval
+that cleared it. This is the ML angle on `domains/devops/P4 Rollback
+First` and `domains/kubernetes/P10 Roll Forward, Roll Back`.
+
+## 2. Core Principle Trace
+
+Each AI/ML P-rule derives from one or more core C-rules (C1–C8). The
+matrix extension lands in P4 of the v0.3 plan; the traces below are
+authoritative.
+
+| P-rule | Core | Why |
+|--------|------|-----|
+| P1 Reproducibility is the First Class | C1, C5 | Correctness of results; reversibility of runs |
+| P2 Data is Versioned, Not Just Code | C5, C7 | Reversibility of datasets; observability of data lineage |
+| P3 Lineage is Traceable End-to-End | C7, C1 | Observability of provenance; correctness of attribution |
+| P4 Evaluation is Defined Before Training | C1, C2 | Correctness of the eval gate; clarity of a-priori intent |
+| P5 Models are Versioned Artifacts | C5, C6 | Reversibility of model identity; composability of registry stages |
+| P6 Serving is Observable | C7 | Observability of inference |
+| P7 Drift is Expected and Detected | C7, C1 | Observability of degradation; correctness of detection |
+| P8 Inference Inputs are Validated | C1 | Correctness of the serving boundary (security subset) |
+| P9 Pipelines Compose, Notebooks Don't | C6, C2 | Composability of stages; clarity of explicit contracts |
+| P10 Rollback Includes the Model | C5 | Reversibility at the model layer |
+
+## 3. What Violates These Principles
+
+| Violation | Principle Breached |
+|-----------|-------------------|
+| A training run that cannot be replayed from pinned inputs | P1 Reproducibility is the First Class |
+| A dataset referenced by a mutable path, not a hash | P2 Data is Versioned, Not Just Code |
+| A production model with no record of its training data | P3 Lineage is Traceable End-to-End |
+| Metrics chosen after seeing the results | P4 Evaluation is Defined Before Training |
+| A serving endpoint that pulls `latest` from the registry | P5 Models are Versioned Artifacts |
+| A model in production with no latency or throughput metrics | P6 Serving is Observable |
+| A serving regression dismissed as "no code changed" | P7 Drift is Expected and Detected |
+| An input with an out-of-range feature scored silently | P8 Inference Inputs are Validated |
+| A notebook in the serving or training pipeline path | P9 Pipelines Compose, Notebooks Don't |
+| A rollback that restores code but keeps the regressed model | P10 Rollback Includes the Model |
+
+## 4. Relationship to Other Domains
+
+AI/ML is the engineering-discipline layer for model-bearing systems.
+It borrows the reproducibility, immutability, rollback, and
+observability disciplines of `domains/devops/` and applies them to
+the data → model → serving lifecycle. Cross-links are one-directional
+(per D-026 extended):
+
+- `domains/devops/P1 Reproducibility` ← P1
+- `domains/devops/P4 Rollback First` ← P10
+- `domains/devops/P5 Progressive Delivery` ← P10 (model canary)
+- `domains/devops/P7 Immutability` ← P5 (model images)
+- `domains/data/migrations.md` ← P2 (data versioning ↔ migration discipline)
+- `domains/data/schema-design.md` ← P8 (inference input contract)
+- `domains/observability/metrics.md` ← P6, P7
+- `domains/observability/logging.md` ← P3 (lineage)
+- `domains/security/input-validation.md` ← P8
+- `domains/security/secrets.md` ← P8 (serving credentials)
+- `domains/performance/backend.md` ← P6 (serving latency)
+- `domains/kubernetes/workloads.md` ← P9 (serving on k8s)
+- `domains/testing/first-principles.md` ← P4 (eval as a gate)
+- `domains/gitops-operators/first-principles.md` ← P10 (model rollback in a GitOps loop)

domains/ai-ml/model-evaluation.md

@@ -0,0 +1,94 @@
+# Model Evaluation — Derived Rules
+
+> Derives from `domains/ai-ml/first-principles.md`. Covers P4
+> (Evaluation is Defined Before Training) and the eval-as-a-gate
+> discipline. Referenced by `serving.md` (promotion gate) and
+> `monitoring-drift.md` (online eval). Scope per D-023: evaluation
+> methodology, not metric math or model-family benchmarks.
+
+## Evaluation is a Gate, Not a Report (P4 Evaluation Defined Before Training)
+
+- The eval spec — metrics, splits, thresholds, and pass/fail
+  criteria — is declared in code **before** the model is trained.
+  It is versioned with the data and the code; it is reviewable; it
+  is the contract the model must satisfy to leave the experiment.
+- Cherry-picking metrics after seeing results is a correctness
+  violation: the eval is no longer measuring the model, it is
+  rationalizing it. The a-priori spec is what makes the eval
+  trustworthy.
+- This is the ML angle on `domains/testing/first-principles.md` P1
+  (Tests as Specification): the eval declares the model's contract,
+  the model does not declare its own success.
+
+## The Eval Input Contract (P8 Inference Inputs are Validated, cross `domains/data/schema-design.md`)
+
+- The eval set has a schema: feature names, types, ranges, and
+  categorical domains. That schema is the same schema the serving
+  boundary enforces (`serving.md`, `domains/security/input-validation.md`).
+- An eval set whose schema drifted from the serving schema is
+  measuring a different model than the one in production. Schema
+  parity is part of the versioned eval artifact.
+- Cross `domains/data/schema-design.md`: the eval input contract is
+  a schema-design problem, versioned and reviewed like any schema.
+
+## Holdout Integrity (P4, P2 Data is Versioned)
+
+- The held-out eval set is never touched by training, feature
+  selection, or threshold tuning. A "held out" set that influenced
+  any training decision is not held out — it is a third training
+  set, and the eval is measuring memorization.
+- Splits are versioned with the dataset (`data-versioning.md`).
+  Recreating splits ad-hoc per run breaks comparability across runs.
+- Reusing a held-out set across many model iterations leaks it
+  incrementally. Rotate or re-split on a cadence; record the
+  rotation in lineage.
+
+## Offline vs Online Evaluation (P6 Serving is Observable)
+
+- **Offline eval** runs before promotion: held-out data, pinned
+  model, declared metrics, pass/fail gate. It answers "should this
+  model ship?"
+- **Online eval** runs after promotion, on live traffic: shadow
+  scoring, A/B, canary metrics. It answers "is this model behaving
+  in production?" It is the bridge to `monitoring-drift.md`.
+- A model that passed offline and regressed online is not a
+  contradiction — it is a signal that the offline distribution
+  differs from the live one (a P7 drift signal). Both eval layers
+  are required; neither substitutes for the other.
+
+## Bias and Fairness Checks (Engineering Angle, P4)
+
+- Bias/fairness checks are part of the a-priori eval spec, not an
+  afterthought. They are metrics with thresholds, declared before
+  training, gated the same as any metric.
+- This doc covers the **engineering** discipline: the checks are
+  versioned, gated, and recorded in lineage. The choice of which
+  fairness metrics and what thresholds are policy decisions, not
+  engineering principles, and are out of scope here (D-023).
+
+## Eval-as-a-Gate in the Pipeline (P9 Pipelines Compose)
+
+- The eval is a pipeline stage with a contract: input = model
+  digest + eval dataset version; output = pass/fail + metric
+  report. It composes with the training stage and the promotion
+  stage.
+- A promotion that bypasses the eval stage is a P4 violation,
+  regardless of who approved it. The gate is in the pipeline, not
+  in a human sign-off sheet.
+
+```
+train -> eval(gate) -> register(promote) -> serve
+                 |
+                 +-- fail -> abort, no promote
+```
+
+## What Violates Evaluation Discipline
+
+| Violation | Principle |
+|-----------|-----------|
+| Metrics chosen after seeing the scores | P4 Evaluation Defined Before Training |
+| Held-out set used in feature selection or threshold tuning | P4, P2 |
+| Eval schema differs from serving schema | P8 Inference Inputs are Validated |
+| Promotion by human approval, bypassing the eval stage | P4, P9 Pipelines Compose |
+| A "passing" model with no online eval in production | P6 Serving is Observable |
+| Fairness checks added after a model shipped | P4 Evaluation Defined Before Training |

domains/ai-ml/monitoring-drift.md

@@ -0,0 +1,88 @@
+# Monitoring & Drift — Derived Rules
+
+> Derives from `domains/ai-ml/first-principles.md`. Covers P7 (Drift
+> is Expected and Detected) and the online half of P6 (Serving is
+> Observable). Referenced by `serving.md` (online eval) and
+> `model-evaluation.md` (online layer). Scope per D-023: drift
+> detection methodology, not model retraining architecture.
+
+## Drift is Expected and Detected (P7 Drift is Expected and Detected)
+
+- ML systems decay without code changes. The world changes under
+  the model: user behavior shifts, input pipelines change,
+  upstream schemas evolve. "No code changed" is not a defense
+  against a serving regression.
+- A drift signal is an incident, not a curiosity. It triggers an
+  alert, an investigation, and a decision (retrain, roll back, or
+  accept with a recorded justification). Silent drift is the same
+  class of bug as silent serving (P6).
+- Cross `domains/observability/metrics.md` for the alerting
+  primitives and `domains/observability/logging.md` for the
+  structured events a drift signal emits.
+
+## The Three Drift Types (IDEATE-30, D-048)
+
+| Drift Type | What Changes | Detection Signal | Source of Truth |
+|------------|--------------|------------------|-----------------|
+| **Data drift** (input drift) | The distribution of inputs at serving time diverges from the distribution the model was trained on | Statistical distance between the live input distribution and the pinned training-set distribution (e.g., PSI, KL, KS test). Alert on threshold breach | Training dataset hash (`data-versioning.md`) + live input metrics |
+| **Concept drift** | The relationship between inputs and the target changes — the same input now maps to a different correct output | Ground-truth lag: compare delayed labels against predictions on the same inputs. Rising error rate against a stable input distribution signals concept, not data, drift | Delayed-label feedback stream + prediction log |
+| **Prediction drift** (output drift) | The distribution of the model's predictions shifts, with no change to inputs | Statistical distance between the live prediction distribution and a pinned baseline prediction distribution. Independent of inputs — catches model-internal regressions and upstream silent changes | Prediction log + baseline prediction snapshot |
+
+- The three signals are distinct and non-substitutable. Data drift
+  catches the input changing; concept drift catches the world
+  changing; prediction drift catches the model's behavior changing.
+  A monitoring setup with only one is blind to two classes of
+  regression.
+- Evidently AI and Great Expectations are the canonical tooling:
+  Evidently for drift/statistical reports, Great Expectations for
+  data-quality/contract checks at the pipeline boundary. Both
+  produce the metrics that feed `domains/observability/metrics.md`.
+
+## Detection Signals in Practice
+
+- **Data drift** compares live inputs to the **pinned training
+  distribution** — not to "yesterday's inputs." Without a pinned
+  baseline, drift is measured against a moving target and is
+  meaningless. Cross `data-versioning.md` for how the baseline is
+  pinned.
+- **Concept drift** requires ground truth, which is often delayed
+  (days/weeks). The detection signal is the gap between
+  prediction-time confidence and delayed-label error. A rising
+  error against stable inputs is the signature.
+- **Prediction drift** needs no ground truth and no input
+  comparison — it watches the model's own output distribution. It
+  is the cheapest signal and the first to fire; it is also the
+  least specific (any of the three drifts can move predictions).
+
+## Alerting and Retraining Triggers (P7, P10 Rollback Includes the Model)
+
+- A drift alert is an incident. It does not auto-trigger retraining
+  unsupervised — auto-retraining on drift can lock in a bad
+  distribution. The alert triggers a human decision: investigate,
+  retrain, roll back, or accept.
+- Retraining is a new training run (`first-principles.md` P1): it
+  produces a new model digest, passes the eval gate
+  (`model-evaluation.md`), and is promoted through the registry
+  (`serving.md`). The prior model stays rollbackable (P10).
+- Cross `domains/observability/metrics.md` for the alert-rule
+  pattern: threshold + window + severity, routed to the same
+  on-call path as any production incident.
+
+## Online Evaluation Bridge (P6 Serving is Observable)
+
+- Online eval (`model-evaluation.md`) is the live counterpart to
+  drift monitoring: shadow scores and A/B canaries measure a
+  candidate model against the incumbent, while drift monitoring
+  measures the incumbent against its own baseline. Both feed the
+  same metrics pipeline.
+
+## What Violates Monitoring Discipline
+
+| Violation | Principle |
+|-----------|-----------|
+| Only one drift type monitored | P7 Drift is Expected and Detected |
+| Drift baseline is "yesterday's inputs," not pinned training data | P7, P2 Data is Versioned |
+| Drift alert that auto-retrains without a human gate | P7, P1 Reproducibility |
+| A serving regression dismissed as "no code changed" | P7 Drift is Expected and Detected |
+| Concept-drift check with no delayed-label feedback path | P7 Drift is Expected and Detected |
+| Prediction-distribution change with no alert | P6 Serving is Observable, P7 |

domains/ai-ml/serving.md

@@ -0,0 +1,88 @@
+# Serving — Derived Rules
+
+> Derives from `domains/ai-ml/first-principles.md`. Covers P5 (Models
+> are Versioned Artifacts), P6 (Serving is Observable), P8 (Inference
+> Inputs are Validated), and P10 (Rollback Includes the Model).
+> Referenced by `monitoring-drift.md` (online signals) and
+> `model-evaluation.md` (promotion gate). Scope per D-023: serving
+> patterns, not model architectures.
+
+## The Model is an Addressed Artifact (P5 Models are Versioned Artifacts)
+
+- A serving endpoint pulls a model by digest, never by `latest`. A
+  model pulled by `latest` is an unknown model — its behavior is
+  undefined and its rollback is impossible.
+- The model registry is to models what a container registry is to
+  images (`domains/devops/P7 Immutability`): immutable, addressed by
+  digest, promoted by stage (staging → prod). Promotion is a
+  registry operation, not a file copy.
+- A serving rollout names the model digest in its manifest. The
+  digest is part of the deploy's lineage (`data-versioning.md`).
+
+## Inference Inputs are Validated (P8 Inference Inputs are Validated)
+
+- The model's input contract — schema, types, ranges, categorical
+  domains — is enforced at the serving boundary, before the model
+  sees the input. Out-of-contract inputs are rejected with a
+  defined error, not silently scored.
+- Scoring an out-of-contract input is a C1 (Correctness) violation:
+  the model's output is undefined outside its training
+  distribution, and returning a number for it is lying to the
+  caller.
+- This is the ML angle on `domains/security/input-validation.md`:
+  the validation lives at the boundary, the model is downstream of
+  it, and the contract is versioned with the model.
+
+## Serving is Observable (P6 Serving is Observable)
+
+- Every inference path emits: request latency, throughput, input
+  distribution summaries, prediction confidence, and error counts.
+  Silent serving is a bug.
+- Cross `domains/observability/metrics.md` for the metrics
+  primitives (histograms, counters, gauges) and
+  `domains/observability/tracing.md` for the request-level trace
+  that ties an input to a prediction.
+- Latency SLAs are enforced via `domains/performance/backend.md`
+  disciplines: budget the inference path, measure the tail (p99),
+  alert on budget breach.
+
+## Serving Patterns (P9 Pipelines Compose)
+
+| Pattern | When | Notes |
+|---------|------|-------|
+| Inference as a service | Default; model behind an HTTP/gRPC endpoint | KServe, Seldon Core, BentoML. Scales with traffic; model is a deployable, addressable artifact |
+| Batch inference | Offline scoring of large datasets | No latency SLA; throughput-bound. Same model digest, same input contract |
+| Embedded / in-process | Latency-critical, single-tenant | Model linked into the app. Trades observability for latency — only when the SLA demands it |
+
+- Canarying a model is a serving pattern, not a deployment pattern:
+  shift a fraction of traffic to the new model digest, measure
+  online eval (`model-evaluation.md`), abort to the prior digest on
+  regression. This is `domains/devops/P5 Progressive Delivery`
+  applied at the model layer.
+- Rollback restores the prior model digest (P10 Rollback Includes
+  the Model). A rollback that redeploys old code but keeps the new
+  model has not rolled back. Cross `domains/gitops-operators/first-principles.md`
+  for the GitOps reconciliation loop that drives model rollouts.
+
+## Tool Landscape (KServe / Seldon Core / BentoML)
+
+| Tool | Model Packaging | Deployment Surface | Notes |
+|------|-----------------|--------------------|-------|
+| KServe | InferenceService CRD; runtime predictors (v2, HuggingFace, PMML, custom) | Kubernetes-native; CRD-driven | Cross `domains/kubernetes/workloads.md`. Brings the k8s reconciliation model to serving |
+| Seldon Core | SeldonDeployment CRD; graph of predictors | Kubernetes-native; CRD-driven | Emphasizes inference graphs (fan-out, ensemble) as CRD structure |
+| BentoML | Bento (model + runtime + deps packaged); Yatai registry | Kubernetes or bare container | Focuses on packaging + registry; the Bento is the versioned artifact (P5) |
+
+- All three satisfy P5/P6/P8 when wired correctly; the choice is
+  packaging model and deployment surface, not correctness.
+- None is advocated over the others.
+
+## What Violates Serving Discipline
+
+| Violation | Principle |
+|-----------|-----------|
+| Endpoint pulls `latest` from the registry | P5 Models are Versioned Artifacts |
+| Out-of-range input scored silently | P8 Inference Inputs are Validated |
+| Serving path emits no latency or throughput metrics | P6 Serving is Observable |
+| Rollback redeploys code but keeps the regressed model | P10 Rollback Includes the Model |
+| A notebook in the serving path | P9 Pipelines Compose, Notebooks Don't |
+| Canary with no abort-to-prior-digest path | P10, `domains/devops/P5 Progressive Delivery` |

domains/gitops-operators/argocd.md

@@ -0,0 +1,176 @@
+# ArgoCD — Derived Rules
+
+> Derives from `domains/gitops-operators/first-principles.md`.
+> Applies P1–P10 to ArgoCD specifically. For the ArgoCD-vs-Flux
+> decision, see the decision matrix at the end of this doc and in
+> `flux.md`.
+
+## What ArgoCD Is (P1 Git is the Source of Truth, P3 Pull, Don't Push)
+
+- ArgoCD is a pull-based GitOps controller for Kubernetes. It runs
+  inside the target cluster, pulls desired state from git, and
+  reconciles the cluster to match. CI never holds `kubectl` rights
+  against the cluster (P3).
+- An Application is a declarative binding of "this git path" to
+  "this cluster destination." The Application CRD is the unit of
+  reconciliation. The cluster is a derivative of git, never the
+  authority (P1).
+- ArgoCD supports Helm charts, Kustomize overlays, ksonnet, and raw
+  manifests as source formats — see `domains/kubernetes/helm.md`
+  and `domains/kubernetes/kustomize.md`.
+
+## Application CRD (P2 Declarative Over Imperative, P4 Continuous Reconciliation)
+
+- An Application declares `source` (repo, path, revision, chart),
+  `destination` (server, namespace), and `syncPolicy`. The
+  reconciler loops continuously; drift is corrected automatically,
+  not on-demand (P4).
+
+```yaml
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: payments-api
+  namespace: argocd
+spec:
+  source:
+    repoURL: https://git.example.com/platform/payments
+    targetRevision: 1.2.3
+    path: manifests/prod
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: payments
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=false
+```
+
+- `automated.prune: true` deletes resources removed from git.
+  `selfHeal: true` corrects hand-edited drift back to git (P8).
+  Disable both for workloads that need manual approval gates.
+
+## App-of-Apps (P6 Operators Encode Domain Knowledge, C6 Composability)
+
+- The App-of-Apps pattern: one root Application points at a git
+  directory of child Application manifests. The root app reconciles
+  the children; the children reconcile the workloads. This is the
+  ArgoCD expression of composition — a fleet of apps as a tree of
+  Applications.
+- Use App-of-Apps for cluster bootstrapping (one repo, many
+  clusters, many apps). Do not use it as a substitute for a package
+  manager; if you are templating hundreds of near-identical
+  Applications, use a generator (ApplicationSet) instead.
+
+## Sync Waves and Hooks (P4 Continuous Reconciliation, P7 Reversibility)
+
+- Sync waves order resources within a sync: `PreSync` → `Sync` →
+  `PostSync`. Use waves to run a job before a Deployment, or a
+  migration before the app that depends on it.
+- Sync hooks (`PreSync`, `Sync`, `PostSync`, `SyncFail`) are
+  Resources annotated to execute at a wave boundary. A `SyncFail`
+  hook runs on sync failure — the abort path (P7).
+- Wave ordering is a correctness mechanism, not a performance one.
+  Mis-ordered waves (e.g., app starts before its migration job)
+  are a correctness bug.
+
+## Health and Status (P9 Failure is Observable and Surfaced)
+
+- ArgoCD assesses every resource's health (`Healthy`, `Progressing`,
+  `Degraded`, `Missing`, `Suspended`) and surfaces the aggregate as
+  Application status. Sync status (`Synced`, `OutOfSync`) reports
+  drift against git.
+- Health checks are pluggable via Lua scripts for custom CRDs. An
+  Operator-managed CRD without a health check reads as `Progressing`
+  forever — write one (see `operators.md`).
+- Out-of-sync or degraded status must emit a notification (Slack,
+  PagerDuty, webhook). Silent drift is the bug (P9). Wire status to
+  `domains/observability/metrics.md`.
+
+## Diff and Drift (P8 Reconcile, Don't Mutate by Hand, P4)
+
+- `argocd app diff` shows the diff between git and live cluster.
+  A non-empty diff on a synced app is hand-edit drift — the
+  recovery is `selfHeal`, not a manual `kubectl apply` (P8).
+- Drift detection runs continuously (P4). The gap between "git
+  changed" and "cluster matches git" is observable, not assumed.
+
+## RBAC and SSO (P10 Least Privilege Reconciliation)
+
+- ArgoCD's own RBAC governs who can view, sync, and admin
+  Applications. Bind to SSO (OIDC, SAML) for human identity; bind
+  the controller's service account to a Role scoped to the
+  namespaces it reconciles.
+- The controller's credentials must not be `cluster-admin` (P10).
+  Use namespace-scoped Roles via `ApplicationSet` namespaces or
+  cluster-wide AppProject restrictions. See
+  `domains/kubernetes/rbac.md` and `domains/security/authorization.md`.
+- AppProjects bound the blast radius of what an Application can
+  deploy (allowed repos, destinations, roles). One AppProject per
+  team or environment; the default project is for nothing in
+  production.
+
+## Multi-Cluster (P4 Locality, P10)
+
+- ArgoCD registers external clusters by secret. The controller
+  pulls from git and pushes to the registered cluster's API server.
+  The "pull, don't push" boundary (P3) is between the target
+  cluster's reconciler and CI — the controller-to-apiserver hop is
+  internal to the platform.
+- Scope each registered cluster's credentials to the namespaces
+  ArgoCD manages there. Do not register a cluster with cluster-admin
+  and call it done (P10).
+
+## Sync Windows (P5 Reversibility, P7)
+
+- Sync windows restrict when automated sync runs (e.g., no syncs
+  during business hours, or syncs only in a maintenance window).
+  They are a reversibility mechanism: a bad commit lands in git,
+  but the sync window holds it until review.
+- Sync windows do not replace health monitoring (P9). A degraded
+  app inside a window is still an incident.
+
+## Secrets (P10, cross-link security/secrets)
+
+- Do not store raw Secrets in the GitOps repo. Use a sealed-secret
+  controller (Bitnami Sealed Secrets, SOPS, External Secrets
+  Operator) so the git store holds encrypted material only. See
+  `domains/security/secrets.md` for the general secret-hygiene
+  principles.
+
+## ArgoCD vs Flux — Decision Matrix (IDEATE-21, D-039)
+
+| Axis | ArgoCD | Flux |
+|------|--------|------|
+| Architecture | Monolithic controller + Application CRD | Composable GitOps Toolkit controllers (source, kustomize, helm, notification) |
+| Reconciliation unit | Application (one CRD per app) | Kustomization / HelmRelease (one per deploy unit) |
+| UI | Web UI + CLI (full dashboard, tree view, diff viewer) | CLI-first; UI via Weave GitOps or FluxUI (add-on) |
+| Sync model | Periodic poll or webhook; sync waves + hooks | Poll + webhook; runs continuously, no explicit sync waves |
+| Multi-cluster | One ArgoCD manages many clusters (hub-and-spoke) | One Flux per cluster (per-cluster autonomy) |
+| Templating in repo | Helm, Kustomize, ksonnet, raw manifests, Jsonnet | Helm, Kustomize, raw manifests |
+| RBAC | Built-in RBAC + SSO + AppProjects | Kubernetes RBAC (no built-in RBAC layer) |
+| Progressive delivery | Argo Rollouts (sister project, tight integration) | Flagger (sister project, tight integration) |
+| Best for | Teams wanting a UI, multi-cluster from one pane, App-of-Apps bootstrapping | Teams wanting composable controllers, per-cluster autonomy, minimal footprint |
+| Watch out for | Monolithic controller scaling, UI as ops crutch, AppProject sprawl | No native UI, steeper learning curve, manual multi-cluster orchestration |
+
+- Use ArgoCD when you want a UI, central multi-cluster management,
+  and sync-wave ordering. Use Flux when you want composable
+  controllers, per-cluster autonomy, and a minimal footprint.
+- Both are CNCF graduated and both implement the OpenGitOps
+  principles. The choice is architectural fit, not correctness. See
+  `flux.md` for the Flux-side perspective.
+
+## What Violates ArgoCD Discipline
+
+| Violation | Principle |
+|-----------|-----------|
+| CI pipeline with `kubectl` rights pushing to the cluster | P3 Pull, Don't Push |
+| `argocd app set` used as the steady state instead of git | P1 Git is the Source of Truth |
+| `selfHeal: false` on a prod app with no manual gate | P8 Reconcile, Don't Mutate by Hand |
+| Controller ServiceAccount bound to `cluster-admin` | P10 Least Privilege Reconciliation |
+| Sync failure with no notification wired | P9 Failure is Observable and Surfaced |
+| AppProject with no destination restrictions in prod | P10 Least Privilege Reconciliation |
+| Raw Secret in the GitOps repo | P10, `domains/security/secrets.md` |
+| Manual `kubectl edit` on an ArgoCD-managed resource | P8 Reconcile, Don't Mutate by Hand |

domains/gitops-operators/first-principles.md

@@ -0,0 +1,131 @@
+# GitOps + Operators — First Principles
+
+## 1. The Principles
+
+### P1. Git is the Source of Truth
+Desired state lives in a versioned, immutable git store. The
+cluster is a derivative of git, never the authority. If a state
+exists only in the cluster and not in git, it is drift, not truth.
+The commit history is the audit trail and the rollback path.
+
+### P2. Declarative Over Imperative
+Express the desired cluster state, not the commands to reach it.
+A manifest says what should exist; the reconciler makes it so.
+Imperative `kubectl` is for inspection and incident response, not
+for the steady state. This is the GitOps expression of
+`domains/kubernetes/P1 Declarative Desired State` and
+`domains/infrastructure-as-code/P1 Declarative Intent`.
+
+### P3. Pull, Don't Push
+Agents running inside the target pull desired state from git; the
+target never accepts outside push credentials. No CI pipeline holds
+`kubectl` rights against the production cluster. The cluster reaches
+out to git, not the other way around. This is the security primitive
+of GitOps: the blast radius of a compromised CI is bounded by what CI
+can push, and a pull model gives CI nothing to push.
+
+### P4. Continuous Reconciliation
+The reconciliation loop is the primitive. Drift is detected and
+corrected automatically, not on-demand. A manual `apply` is an
+exception, not the workflow. The loop runs continuously; the gap
+between "git changed" and "cluster matches git" is measured in
+seconds, not tickets.
+
+### P5. State is Immutable and Versioned
+Every change to desired state is a commit. History is the audit
+trail and the rollback path. A revert is a rollback; a force-push is
+history deletion. The git store is treated like
+`domains/infrastructure-as-code/P3 State is Truth` — lose it or
+tamper with it, and you lose the ability to reason about the system.
+
+### P6. Operators Encode Domain Knowledge
+Operational expertise lives as CRDs plus controllers, not as
+runbooks that humans must remember. An operator is a control loop
+that encodes how to reconcile a specific domain (a database, a
+message queue, a certificate). The operator is the deepest
+expression of `domains/kubernetes/P1 Declarative Desired State` —
+the domain knowledge is the desired state.
+
+### P7. Progressive Delivery is Reversible by Construction
+Canary and blue-green are staged, metric-gated, and one-command
+abortable. Promotion without a rollback path is a violation. A
+rollout that cannot be aborted is a deploy, not a progressive
+delivery. This is the GitOps extension of
+`domains/devops/P5 Progressive Delivery` and
+`domains/kubernetes/P10 Roll Forward, Roll Back`.
+
+### P8. Reconcile, Don't Mutate by Hand
+Manual `kubectl apply` or `kubectl edit` on a GitOps-managed
+resource is an incident. The reconciler will overwrite the hand
+edit on the next loop; the hand edit was never truth. Drift back to
+git is the recovery, not the failure. This is the GitOps angle on
+`domains/infrastructure-as-code/P9 Drift is Recoverable`.
+
+### P9. Failure is Observable and Surfaced
+Sync failures, health degradation, and rollout-stall events emit
+status and notifications. Silent drift is the bug. A GitOps
+controller that fails to sync without surfacing the failure has
+violated the contract — you cannot fix what you cannot see
+(`domains/observability/metrics.md`).
+
+### P10. Least Privilege Reconciliation
+The controller's credentials are scoped to the namespaces and
+resources it reconciles. No `cluster-admin` GitOps robots. One
+credential set per boundary; the reconciler sees only what it
+reconciles. This is the GitOps angle on
+`domains/kubernetes/P7 RBAC by Intent, Not Identity` and
+`domains/security/authorization.md`.
+
+## 2. Core Principle Trace
+
+Each GitOps + Operators P-rule derives from one or more core
+C-rules (C1–C8). The matrix extension lands in P4 of the v0.3
+plan; the traces below are authoritative.
+
+| P-rule | Core | Why |
+|--------|------|-----|
+| P1 Git is the Source of Truth | C1, C5 | Correctness of state; reversibility via history |
+| P2 Declarative Over Imperative | C2, C3 | Clarity of intent; simplicity of mental model |
+| P3 Pull, Don't Push | C1, C4 | Correctness via security; locality of credentials |
+| P4 Continuous Reconciliation | C7, C1 | Observability of drift; correctness of convergence |
+| P5 State is Immutable and Versioned | C5 | Reversibility via version history |
+| P6 Operators Encode Domain Knowledge | C6, C2 | Composability of expertise; clarity of operational intent |
+| P7 Progressive Delivery is Reversible | C5, C1 | Reversibility of promotion; correctness of abort |
+| P8 Reconcile, Don't Mutate by Hand | C1, C7 | Correctness of single source; observability of drift |
+| P9 Failure is Observable and Surfaced | C7 | Observability of reconciliation |
+| P10 Least Privilege Reconciliation | C1, C8 | Correctness via security; economy of trust |
+
+## 3. What Violates These Principles
+
+| Violation | Principle Breached |
+|-----------|-------------------|
+| CI pipeline pushes manifests to the cluster | P3 Pull, Don't Push |
+| A resource exists in the cluster but not in git | P1 Git is the Source of Truth |
+| `kubectl edit` on a GitOps-managed resource | P8 Reconcile, Don't Mutate by Hand |
+| Reconciler with `cluster-admin` ClusterRoleBinding | P10 Least Privilege Reconciliation |
+| Sync failure with no status or notification | P9 Failure is Observable and Surfaced |
+| Canary with no abort/rollback path | P7 Progressive Delivery is Reversible |
+| Operator runbook that exists only in a wiki | P6 Operators Encode Domain Knowledge |
+| Reconciler that applies on a cron, not continuously | P4 Continuous Reconciliation |
+| Force-push rewrites GitOps repo history | P5 State is Immutable and Versioned |
+| Imperative deploy script as the steady state | P2 Declarative Over Imperative |
+
+## 4. Relationship to Other Domains
+
+GitOps + Operators is the deployment-automation layer above
+`domains/kubernetes/` and `domains/infrastructure-as-code/`. It
+borrows their declarative-reconciliation model and adds the
+git-as-source-of-truth and pull-based credential boundaries. Cross
+links are one-directional (per D-026 extended):
+
+- `domains/kubernetes/P1 Declarative Desired State` ← P2
+- `domains/kubernetes/P10 Roll Forward, Roll Back` ← P7
+- `domains/infrastructure-as-code/P1 Declarative Intent` ← P2
+- `domains/infrastructure-as-code/P3 State is Truth` ← P1, P5
+- `domains/infrastructure-as-code/P9 Drift is Recoverable` ← P4, P8
+- `domains/devops/P4 Rollback First` ← P5, P7
+- `domains/devops/P5 Progressive Delivery` ← P7
+- `domains/devops/P6 Configuration as Code` ← P1, P2
+- `domains/security/secrets.md` ← P3, P10 (reconciliation credentials)
+- `domains/security/supply-chain.md` ← P5 (signed, immutable provenance)
+- `domains/observability/metrics.md` ← P4, P9 (reconciliation + rollout metrics)

domains/gitops-operators/flux.md

@@ -0,0 +1,159 @@
+# Flux — Derived Rules
+
+> Derives from `domains/gitops-operators/first-principles.md`.
+> Applies P1–P10 to Flux specifically. For the ArgoCD-vs-Flux
+> decision, see the decision matrix at the end of this doc and in
+> `argocd.md`.
+
+## What Flux Is (P1 Git is the Source of Truth, P3 Pull, Don't Push)
+
+- Flux is a set of composable controllers — the GitOps Toolkit —
+  that run inside the target cluster, pull desired state from git
+  or OCI registries, and reconcile the cluster to match. CI never
+  holds `kubectl` rights against the cluster (P3).
+- The composable-controller architecture is a C6 (Composability)
+  exemplar: each controller does one thing (source, kustomize, helm,
+  notification) and the controllers compose into a full GitOps
+  system.
+- Flux supports Helm releases, Kustomize overlays, and raw
+  manifests — see `domains/kubernetes/helm.md` and
+  `domains/kubernetes/kustomize.md`.
+
+## GitOps Toolkit Controllers (P6 Composability, P4 Continuous Reconciliation)
+
+- **source-controller** — pulls git, Helm, OCI, and bucket sources;
+  emits artifacts (tarballs) with a digest. The source is the
+  pinned input to reconciliation (P5 versioning by digest).
+- **kustomize-controller** — reconciles Kustomization CRDs against
+  the artifacts from source-controller. Runs continuously (P4).
+- **helm-controller** — reconciles HelmRelease CRDs against Helm
+  charts from source-controller.
+- **notification-controller** — emits events and notifications for
+  sync, health, and source-readiness events (P9).
+- **image-automation-controller** (optional) — updates git with new
+  image tags when a policy matches, closing the "latest image"
+  loop declaratively.
+
+## Kustomization CRD (P2 Declarative Over Imperative, P4)
+
+- A Kustomization binds "this source" to "this target namespace"
+  with a reconciliation interval. The reconciler loops
+  continuously; drift is corrected automatically (P4).
+
+```yaml
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: payments-api
+  namespace: flux-system
+spec:
+  sourceRef:
+    kind: GitRepository
+    name: platform
+    namespace: flux-system
+  path: ./manifests/prod
+  targetNamespace: payments
+  interval: 1m
+  prune: true
+  wait: true
+  healthChecks:
+    - apiVersion: apps/v1
+      kind: Deployment
+      name: payments-api
+      namespace: payments
+```
+
+- `prune: true` deletes resources removed from git. `wait: true`
+  waits for health checks before declaring the Kustomization ready.
+  Disable prune for workloads that need manual removal gates.
+
+## HelmRelease CRD (P6 Composability, cross-link helm.md)
+
+- A HelmRelease binds a Helm chart (from a HelmRepository or OCI
+  source) to target values and a target namespace. helm-controller
+  renders and applies it. See `domains/kubernetes/helm.md` for the
+  chart model.
+- Pin the chart version in the HelmRepository or the HelmRelease.
+  Never float `latest` — unversioned charts drift (P5).
+
+## OCI Sources (P5 State is Immutable and Versioned)
+
+- source-controller can pull from OCI registries (Helm charts as
+  OCI artifacts, or generic OCI repositories). The digest is the
+  version — immutable by construction (P5).
+- OCI sources close the supply-chain loop: the manifest is signed
+  and immutable in the registry, and Flux pulls it by digest. Cross-
+  link `domains/security/supply-chain.md` for signed-provenance
+  principles.
+
+## Reconciliation and Drift (P4 Continuous Reconciliation, P8)
+
+- Flux reconciles on `interval` (default 1m) and on webhook event.
+  Drift between git and cluster is detected each interval and
+  corrected (with `prune` + `selfHeal` semantics).
+- Hand-edited drift on a Flux-managed resource is overwritten on the
+  next loop — the hand edit was never truth (P8). The recovery is
+  to fix git, not to `kubectl apply`.
+
+## Notifications and Events (P9 Failure is Observable and Surfaced)
+
+- notification-controller emits events for source readiness, sync
+  success/failure, and health transitions. Wire them to Slack,
+  PagerDuty, or a webhook. Silent drift is the bug (P9).
+- Events flow to `domains/observability/metrics.md` via the
+  notification controller's provider model — sync and health as
+  first-class signals.
+
+## RBAC and Multi-Cluster (P10 Least Privilege Reconciliation, P4)
+
+- Flux's controllers run with a ServiceAccount in `flux-system`.
+  Scope that account to the namespaces Flux reconciles. Do not bind
+  it to `cluster-admin` (P10). See `domains/kubernetes/rbac.md` and
+  `domains/security/authorization.md`.
+- Flux is per-cluster by design (one Flux install per cluster). For
+  multi-cluster, use one repo with per-cluster paths, or a fleet
+  tool that bootstraps Flux per cluster. Per-cluster autonomy is a
+  feature, not a limitation — it bounds the blast radius of a
+  compromised controller (P4 locality, P10).
+
+## Secrets (P10, cross-link security/secrets)
+
+- Do not store raw Secrets in the GitOps repo. Use the
+  SOPS-compatible decryption in kustomize-controller, or External
+  Secrets Operator, so the git store holds encrypted material only.
+  See `domains/security/secrets.md`.
+
+## ArgoCD vs Flux — Decision Matrix (IDEATE-21, D-039)
+
+| Axis | ArgoCD | Flux |
+|------|--------|------|
+| Architecture | Monolithic controller + Application CRD | Composable GitOps Toolkit controllers (source, kustomize, helm, notification) |
+| Reconciliation unit | Application (one CRD per app) | Kustomization / HelmRelease (one per deploy unit) |
+| UI | Web UI + CLI (full dashboard, tree view, diff viewer) | CLI-first; UI via Weave GitOps or FluxUI (add-on) |
+| Sync model | Periodic poll or webhook; sync waves + hooks | Poll + webhook; runs continuously, no explicit sync waves |
+| Multi-cluster | One ArgoCD manages many clusters (hub-and-spoke) | One Flux per cluster (per-cluster autonomy) |
+| Templating in repo | Helm, Kustomize, ksonnet, raw manifests, Jsonnet | Helm, Kustomize, raw manifests |
+| RBAC | Built-in RBAC + SSO + AppProjects | Kubernetes RBAC (no built-in RBAC layer) |
+| Progressive delivery | Argo Rollouts (sister project, tight integration) | Flagger (sister project, tight integration) |
+| Best for | Teams wanting a UI, multi-cluster from one pane, App-of-Apps bootstrapping | Teams wanting composable controllers, per-cluster autonomy, minimal footprint |
+| Watch out for | Monolithic controller scaling, UI as ops crutch, AppProject sprawl | No native UI, steeper learning curve, manual multi-cluster orchestration |
+
+- Use Flux when you want composable controllers, per-cluster
+  autonomy, and a minimal footprint. Use ArgoCD when you want a UI,
+  central multi-cluster management, and sync-wave ordering.
+- Both are CNCF graduated and both implement the OpenGitOps
+  principles. The choice is architectural fit, not correctness. See
+  `argocd.md` for the ArgoCD-side perspective.
+
+## What Violates Flux Discipline
+
+| Violation | Principle |
+|-----------|-----------|
+| CI pipeline with `kubectl` rights pushing to the cluster | P3 Pull, Don't Push |
+| HelmRelease with no pinned chart version | P5 State is Immutable and Versioned |
+| Flux ServiceAccount bound to `cluster-admin` | P10 Least Privilege Reconciliation |
+| Kustomization with no `healthChecks` on a prod app | P9 Failure is Observable and Surfaced |
+| No notification provider wired for sync failures | P9 Failure is Observable and Surfaced |
+| Raw Secret in the GitOps repo | P10, `domains/security/secrets.md` |
+| Manual `kubectl edit` on a Flux-managed resource | P8 Reconcile, Don't Mutate by Hand |
+| `interval: 24h` on a prod Kustomization (drift window too wide) | P4 Continuous Reconciliation |

domains/gitops-operators/operators.md

@@ -0,0 +1,140 @@
+# Operators — Derived Rules
+
+> Derives from `domains/gitops-operators/first-principles.md`.
+> Applies P6 (Operators Encode Domain Knowledge) primarily, with
+> P1, P4, P8, P9, P10. Cross-links `domains/kubernetes/workloads.md`
+> and `domains/kubernetes/rbac.md` for the underlying controller
+> model, and `domains/infrastructure-as-code/modules.md` for the
+> module-vs-operator boundary.
+
+## What an Operator Is (P6 Operators Encode Domain Knowledge)
+
+- An Operator is a Kubernetes controller that encodes human
+  operational knowledge as CRDs plus a control loop. The operator
+  reconciles a domain-specific resource (a database, a message
+  queue, a certificate, a ML model) to a desired state.
+- The operator is the deepest expression of
+  `domains/kubernetes/P1 Declarative Desired State`: the domain
+  knowledge itself is the desired state. A runbook that lives only
+  in a wiki is operational knowledge that has not been encoded —
+  the operator is the encoding (P6).
+- An operator runs inside the cluster, observes its CRDs, and acts.
+  It is a pull-based reconciler by construction — see
+  `domains/gitops-operators/first-principles.md` P3.
+
+## CRDs and Controllers (P2 Declarative Over Imperative, P4 Continuous Reconciliation)
+
+- A CustomResourceDefinition (CRD) defines the schema of the
+  domain resource. The controller watches instances of that CRD
+  and reconciles current → desired (P4).
+- The CRD is the public contract of the operator. Version it
+  (`v1alpha1` → `v1beta1` → `v1`) and preserve backward
+  compatibility — see `domains/api/versioning.md` for the general
+  API-evolution principles. A CRD is an API surface, not an
+  internal type.
+
+```yaml
+apiVersion: postgres.example.com/v1
+kind: PostgresCluster
+metadata:
+  name: payments-db
+  namespace: payments
+spec:
+  replicas: 3
+  version: "16"
+  storage:
+    size: 100Gi
+    storageClass: fast-ssd
+  backup:
+    schedule: "0 2 * * *"
+    retention: 7d
+```
+
+- The controller reconciles this spec: creates StatefulSets, PVCs,
+  Services, backup CronJobs. The user declares intent; the operator
+  makes it so (P2, P6).
+
+## The Control Loop (P4 Continuous Reconciliation, P8)
+
+- The loop watches CRD instances, compares current vs desired, and
+  acts to converge. Drift (a hand-deleted pod, a failed backup) is
+  detected and corrected each loop (P4).
+- An operator-managed resource should not be hand-edited (P8). The
+  operator owns the subordinate resources (StatefulSets, PVCs); a
+  manual `kubectl edit` on a subordinate is drift the operator will
+  overwrite.
+
+## Operator SDK and OLM (P6 Composability, C6)
+
+- The Operator SDK scaffolds a controller from a CRD (Go, Ansible,
+  Helm). Use it to avoid re-implementing the controller boilerplate.
+- Operator Lifecycle Manager (OLM) installs, updates, and manages
+  operators as first-class cluster components. OLM is the package
+  manager for operators — the operator analogue of
+  `domains/kubernetes/helm.md` for workloads.
+- An operator published via OLM is a versioned, catalog-tracked
+  artifact. Pin the operator version; do not float `latest` (P5
+  applies to operators as much as to manifests).
+
+## When to Write an Operator vs a Helm Chart (P6, C6 Composability)
+
+| Axis | Helm chart | Operator |
+|------|-----------|----------|
+| Day-2 operations | None — chart installs, you operate | Encoded — operator reconciles lifecycle (backup, resize, failover, upgrade) |
+| State | Static manifests | Live control loop watching CRDs |
+| Day-1 install | Strong fit — package and install | Overkill if install is all you need |
+| Day-2 reconcile | None — drift is manual | Continuous — drift corrected each loop |
+| Domain knowledge | Lives in runbooks + on-call | Lives in the controller code |
+| Best for | Off-the-shelf apps, stateless services, one-shot deploys | Stateful apps, complex lifecycles, day-2 automation (backup, scale, failover, version upgrades) |
+| Watch out for | Templating complexity, no day-2 reconcile | Controller complexity, multi-team maintenance burden, scope creep |
+
+- Write an operator when the day-2 operations (backup, failover,
+  resize, version upgrade) are non-trivial and repeated. Write a
+  Helm chart when install is all you need and day-2 is run by a
+  human or a separate tool.
+- Do not write an operator to wrap a Helm chart and call it day-2
+  automation — that is a Helm chart with extra steps. See
+  `domains/infrastructure-as-code/modules.md` for the
+  module-vs-copy boundary (the operator-vs-chart boundary is its
+  analogue).
+
+## Scope and Responsibility Boundaries (P10 Least Privilege, C6)
+
+- An operator owns one domain. An operator that manages databases
+  and message queues and certificates is doing three jobs — split
+  it. Scope creep is the most common operator failure mode (P6
+  violation: the encoded knowledge is no longer coherent).
+- The operator's ServiceAccount must be scoped to the resources it
+  manages (P10). A database operator that needs `cluster-admin` to
+  create a StatefulSet has the wrong RBAC — see
+  `domains/kubernetes/rbac.md` and `domains/security/authorization.md`.
+- One operator per CRD family; one ServiceAccount per operator; one
+  namespace per operator (or a shared `operators` namespace with
+  strict RoleBindings). Default namespace is for nothing in
+  production.
+
+## Failure and Observability (P9 Failure is Observable and Surfaced)
+
+- An operator must surface its reconcile status on the CRD
+  (`status.conditions`, `status.observedGeneration`). A CRD with no
+  status is an operator that fails silently (P9).
+- Wire operator events to notifications and metrics. A failed
+  backup, a stuck failover, a version-upgrade stall must emit a
+  signal — see `domains/observability/metrics.md`.
+- An operator that reconciles but does not report health is a
+  black box. The GitOps controller (ArgoCD/Flux) will read it as
+  `Progressing` forever — write the health check (see `argocd.md`
+  "Health and Status").
+
+## What Violates Operator Discipline
+
+| Violation | Principle |
+|-----------|-----------|
+| Operator that manages databases + queues + certs | P6 Operators Encode Domain Knowledge (scope creep) |
+| Operator ServiceAccount bound to `cluster-admin` | P10 Least Privilege Reconciliation |
+| CRD with no `status.conditions` | P9 Failure is Observable and Surfaced |
+| Operator with no health check wired to GitOps | P9, `argocd.md` Health and Status |
+| Unversioned CRD (`v1` shipped without alpha/beta) | P5, `domains/api/versioning.md` |
+| Manual `kubectl edit` on an operator-managed subordinate | P8 Reconcile, Don't Mutate by Hand |
+| Operator that wraps a Helm chart and adds no day-2 logic | P6 (no knowledge encoded) |
+| Operator runbook that exists only in a wiki | P6 Operators Encode Domain Knowledge |

domains/gitops-operators/progressive-delivery.md

@@ -0,0 +1,177 @@
+# Progressive Delivery — Derived Rules
+
+> Derives from `domains/gitops-operators/first-principles.md`.
+> Applies P7 (Progressive Delivery is Reversible by Construction)
+> primarily, with P4, P9. Cross-links `domains/devops/first-principles.md`
+> P4 Rollback First and P5 Progressive Delivery, and
+> `domains/observability/metrics.md` for the analysis signals.
+
+## What Progressive Delivery Is (P7 Reversible by Construction)
+
+- Progressive delivery shifts traffic in stages (canary, blue-green)
+  gated by analysis (metrics, counters, error rates). Each stage is
+  metric-checked; a failed gate aborts the rollout and reverts to
+  the prior stable version. Promotion without a rollback path is a
+  violation (P7).
+- Progressive delivery is the GitOps extension of
+  `domains/devops/P5 Progressive Delivery` and
+  `domains/kubernetes/P10 Roll Forward, Roll Back`. The k8s rolling
+  update is the floor; progressive delivery adds metric-gated
+  promotion and one-command abort.
+- Two sister projects dominate: **Argo Rollouts** (Argo ecosystem)
+  and **Flagger** (Flux ecosystem). Both implement the same pattern
+  — a Rollout CRD replaces a Deployment, an analysis drives the
+  gates, an abort reverts traffic.
+
+## The Rollout CRD (P2 Declarative Over Imperative, P7)
+
+- A Rollout (Argo Rollouts) or Canary/Flag (Flagger) is a CRD that
+  replaces the Deployment as the reconciled resource. It declares
+  the strategy (canary, blue-green), the traffic split, and the
+  analysis gates. The controller reconciles traffic and pods to
+  match.
+
+```yaml
+apiVersion: argoproj.io/v1alpha1
+kind: Rollout
+metadata:
+  name: payments-api
+  namespace: payments
+spec:
+  replicas: 10
+  selector:
+    matchLabels:
+      app: payments-api
+  template:
+    metadata:
+      labels:
+        app: payments-api
+    spec:
+      containers:
+        - name: api
+          image: registry.example.com/payments-api:1.2.3
+  strategy:
+    canary:
+      trafficRouting:
+        istio:
+          virtualService:
+            name: payments-vs
+            routes: [primary]
+      steps:
+        - setWeight: 5
+        - pause: { duration: 2m }
+        - analysis:
+            templates:
+              - templateName: success-rate
+        - setWeight: 25
+        - pause: { duration: 5m }
+        - analysis:
+            templates:
+              - templateName: success-rate
+        - setWeight: 50
+        - pause: { duration: 5m }
+        - setWeight: 100
+```
+
+- Each `setWeight` shifts traffic; each `pause` holds for
+  observation; each `analysis` runs a metric gate. A failed
+  analysis aborts the rollout and reverts traffic to the stable
+  ReplicaSet (P7).
+
+## Canary vs Blue-Green (P7, C3 Simplicity)
+
+| Strategy | Mechanism | Cost | Best for |
+|----------|-----------|------|----------|
+| Canary | Shift a small % of traffic to the new version; increase on gate success | Low (few new pods) | Most production rollouts; metric-gated, gradual |
+| Blue-Green | Run two full environments; switch traffic all-at-once | High (2× capacity) | Schema-breaking changes, instant rollback, low-frequency deploys |
+
+- Canary is the default — it is reversible by construction (P7)
+  and economical (C8). Blue-green is for changes that cannot be
+  partial (a breaking schema migration, a full cutover).
+- A canary with no analysis gate is a slow blue-green — it is not
+  progressive delivery. The gate is what makes it progressive (P7).
+
+## Analysis Templates (P9 Failure is Observable and Surfaced, P7)
+
+- An AnalysisTemplate declares the metric query, the success
+  threshold, and the count of samples. The rollout controller runs
+  the analysis at each gate; a failed analysis aborts the rollout.
+
+```yaml
+apiVersion: argoproj.io/v1alpha1
+kind: AnalysisTemplate
+metadata:
+  name: success-rate
+  namespace: payments
+spec:
+  metrics:
+    - name: success-rate
+      interval: 1m
+      successCondition: result[0] >= 0.99
+      failureLimit: 2
+      provider:
+        prometheus:
+          address: http://prometheus.observability:9090
+          query: |
+            sum(rate(http_requests_total{job="payments-api",code!~"5.."}[2m]))
+            /
+            sum(rate(http_requests_total{job="payments-api"}[2m]))
+```
+
+- `successCondition` is the gate; `failureLimit` is the tolerance
+  for transient blips. A single failed sample aborts immediately if
+  `failureLimit: 0`; tolerate noise with `failureLimit: 2`.
+- The metric is the abort signal — see `domains/observability/metrics.md`
+  for the SLI/SLO discipline that makes the gate meaningful. A gate
+  on an undefined SLO is a gate on noise.
+
+## Argo Rollouts vs Flagger (P6 Composability, P7)
+
+| Axis | Argo Rollouts | Flagger |
+|------|---------------|---------|
+| Ecosystem | Argo (ArgoCD sister project) | Flux (Flux sister project) |
+| CRD | `Rollout` (replaces `Deployment`) | `Canary` / `Flag` (wraps a `Deployment`) |
+| Traffic providers | Istio, NGINX, ALB, SMI, Traefik, Ambassador | Istio, NGINX, Linkerd, SMI, App Mesh, Gloo, Contour |
+| Analysis sources | Prometheus, Datadog, Wavefront, NewRelic, CloudWatch, Graphite, Kayenta | Prometheus, Datadog, CloudWatch, Stackdriver, Elasticsearch, Graphite |
+| Integration | Tight with ArgoCD (UI shows rollout) | Tight with Flux (events via notification-controller) |
+| Learning curve | Rollout CRD replaces Deployment (migration cost) | Wraps existing Deployment (lower migration cost) |
+| Best for | ArgoCD shops wanting rollout in the Argo UI | Flux shops wanting progressive delivery with minimal migration |
+
+- Both implement the same pattern. The choice follows your GitOps
+  controller — Argo Rollouts with ArgoCD, Flagger with Flux. Mixing
+  is possible but not idiomatic.
+
+## Abort and Rollback (P7 Reversible by Construction, P5)
+
+- An abort reverts traffic to the stable ReplicaSet immediately. A
+  rollout without a tested abort is a prototype (P7).
+- The abort must be one-command (or one-gate-failure). A
+  progressive delivery that requires manual rollback steps has
+  lost the "reversible by construction" property — it is a deploy
+  with extra steps.
+- Test the abort path in staging. An abort that has never been
+  exercised will fail when you need it most — see
+  `domains/devops/first-principles.md` P4 Rollback First.
+
+## Observability (P9 Failure is Observable and Surfaced)
+
+- Progressive delivery is only as good as its metrics. A rollout
+  gated on a metric that is not tracked is ungated — the gate is
+  theater (P9).
+- Wire rollout status (phase, weight, analysis result) to
+  notifications and dashboards. A stalled rollout with no signal is
+  silent drift (P9). See `domains/observability/metrics.md`.
+- Cross-link `domains/kubernetes/workloads.md` for the underlying
+  Deployment/ReplicaSet model that progressive delivery replaces.
+
+## What Violates Progressive Delivery Discipline
+
+| Violation | Principle |
+|-----------|-----------|
+| Canary with no analysis gate | P7 Progressive Delivery is Reversible by Construction |
+| Rollout with no tested abort path | P7, `domains/devops/P4 Rollback First` |
+| Analysis gate on an undefined SLO | P9 Failure is Observable and Surfaced |
+| Blue-green with no 2× capacity budget | C8 Economy (blue-green is a cost decision) |
+| Rollout stalled with no notification | P9 Failure is Observable and Surfaced |
+| Manual `kubectl` traffic shift on a Rollout-managed service | P8 Reconcile, Don't Mutate by Hand |
+| `failureLimit: 0` on a noisy metric (constant false aborts) | P4 Continuous Reconciliation (gate noise tolerance) |