Compare commits
4 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Jon Chery
|
32edb19c96 | ||
|
Jon Chery
|
ab1289a9d9 | ||
|
Jon Chery
|
47674969a1 | ||
|
Jon Chery
|
ce36db0579 |
@@ -1,9 +1,11 @@
|
||||
{
|
||||
"phase": 0,
|
||||
"stage": "clarify",
|
||||
"phase": 1,
|
||||
"stage": "complete",
|
||||
"milestone": "v0.3",
|
||||
"phase_role": "pre_execution",
|
||||
"phase_role": "execution",
|
||||
"project": "atelier",
|
||||
"attempts": 0,
|
||||
"updated_at": "2026-08-05T03:08:00Z"
|
||||
"updated_at": "2026-08-05T03:25:00Z",
|
||||
"phase_tag": "v0.2.1",
|
||||
"release_id": 469
|
||||
}
|
||||
@@ -84,8 +84,8 @@ NFR milestone: no separate minor tag. The final patch (v0.1.5) IS the v0.2 deliv
|
||||
|
||||
| Phase | Name | Type | Status | Key Deliverables |
|
||||
|-------|------|------|--------|------------------|
|
||||
| 0 | Pre-Execution | docs | active | Spec, clarify, research, ideate, plan, PERSONAS.md (extends platform-engineer, adds ml-engineer) |
|
||||
| 1 | GitOps + Operators Domain | docs | pending | domains/gitops-operators/{first-principles, argocd, flux, operators, progressive-delivery}.md |
|
||||
| 0 | Pre-Execution | docs | complete | Spec, clarify, research, ideate, plan, PERSONAS.md (extends platform-engineer, adds ml-engineer) — shipped v0.2.0 |
|
||||
| 1 | GitOps + Operators Domain | docs | complete | domains/gitops-operators/{first-principles, argocd, flux, operators, progressive-delivery}.md — shipped v0.2.1 |
|
||||
| 2 | AI/ML Domain | docs | pending | domains/ai-ml/{first-principles, data-versioning, model-evaluation, serving, monitoring-drift}.md |
|
||||
| 3 | i18n + Compliance Domains | docs | pending | domains/i18n/{first-principles, locale-resources, formatting, rtl-bidi, testing-i18n}.md, domains/compliance/{first-principles, audit-logs, data-retention, policy-as-code, evidence}.md |
|
||||
| 4 | Matrix + Review Integration | docs | pending | matrix/principles-matrix.md (+40 mappings), matrix/domain-coverage.md (incl. C-rule coverage table update), review/{agent-checklist, peer-review-checklist, anti-patterns}.md, MANIFEST.md (+ examples/ listing per ATELIER-91) |
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
# Data Versioning — Derived Rules
|
||||
|
||||
> Derives from `domains/ai-ml/first-principles.md`. Covers P2 (Data is
|
||||
> Versioned, Not Just Code) and P3 (Lineage is Traceable End-to-End).
|
||||
> Referenced by `serving.md` and `monitoring-drift.md`. Scope per
|
||||
> D-023: engineering discipline of versioning data, not dataset
|
||||
> content design.
|
||||
|
||||
## Why Data Versioning (P2 Data is Versioned, Not Just Code)
|
||||
|
||||
- `git` versions code well and data badly. Datasets do not fit in
|
||||
git, and a dataset is not recovered from a commit hash.
|
||||
- A model trained on "the data" is a model trained on an unknown
|
||||
input — a C1 (Correctness) violation. The dataset is a build
|
||||
input; it is named, hashed, and recoverable the way any build
|
||||
input is.
|
||||
- Data versioning is the ML analogue of `domains/data/migrations.md`:
|
||||
the schema and contents of the data evolve, every evolution is a
|
||||
versioned migration, and every model points at a specific version.
|
||||
|
||||
## Dataset Hashing and Lineage (P3 Lineage Traceable End-to-End)
|
||||
|
||||
- Every dataset version has a content hash (not a filename or a
|
||||
timestamp). The hash is the identity. A model's lineage record
|
||||
names the dataset hash it was trained on; a serving prediction
|
||||
names the model digest it came from.
|
||||
- Lineage is a graph: prediction → model → training run → dataset →
|
||||
source(s). Any edge missing is an orphan (`domains/observability/logging.md`
|
||||
for the structured-log angle on lineage events).
|
||||
- The lineage record is append-only. Editing it to "fix" a broken
|
||||
trace is the same class of violation as editing an audit log.
|
||||
|
||||
## Train/Val/Test Split Versioning (P2, P4 Eval Defined Before Training)
|
||||
|
||||
- Splits are versioned with the dataset, not derived ad-hoc per run.
|
||||
A split is a deterministic function of (dataset version, split
|
||||
config, random seed). Two runs on the same pinned inputs produce
|
||||
the same splits.
|
||||
- The eval split is held out and never touched by training. A "held
|
||||
out" set that leaked into training is a P4 (Evaluation Defined
|
||||
Before Training) violation, not just a P2 violation — the eval
|
||||
gate is measuring the training set, not the model.
|
||||
- Cross `domains/data/schema-design.md` for the eval input contract:
|
||||
the schema of the eval set is part of the versioned artifact.
|
||||
|
||||
## Tool Comparison (IDEATE-22, D-040)
|
||||
|
||||
| Tool | Versioning Model | Lineage | Best For | Notes |
|
||||
|------|------------------|---------|----------|-------|
|
||||
| DVC | Git-like pointers to content-addressed object store; `.dvc` files in git track data versions | Pipeline DAG in `dvc.yaml`; reproducibility via `dvc repro` | Teams already on git; file/directory datasets; ML pipelines | Treats data like code; shares git's history model. Object store is pluggable (S3, GCS, Azure, SSH) |
|
||||
| Delta Lake | Table format with transaction log (ACID) + time travel via versioned commits; schema enforcement | Time travel queries; lineage via table history + catalog | Large tabular data; lakehouse; streaming + batch on the same table | Not a pipeline tool — pairs with Spark/Trino/Flink. Brings DB guarantees to object storage |
|
||||
| LakeFS | Git-like operations (branch, commit, merge) over object storage itself | Branch model gives isolated, reproducible data branches | Data engineering teams; branch-per-experiment; CI over data | Not a table format — versions objects. Composes with Delta/Iceberg on top |
|
||||
|
||||
- Pick one primary versioning model per platform. Mixing DVC's
|
||||
pointer model with Delta's transaction-log model fragments
|
||||
operational knowledge (C4 Locality).
|
||||
- All three satisfy P2; the choice is which fits the data shape and
|
||||
the team's existing tooling. None is advocated over the others.
|
||||
|
||||
## Reproducibility Contract (P1 Reproducibility is the First Class)
|
||||
|
||||
A reproducible training run records, in one versioned place:
|
||||
|
||||
```
|
||||
run_id: 2026-08-05T09:12:00Z#run-42
|
||||
dataset: s3://ml-data/train@sha256:7f3a...e21
|
||||
splits: dvc.yaml@commit a1b2c4d
|
||||
code: git@a1b2c4d
|
||||
config: configs/train.yaml@commit a1b2c4d
|
||||
environment: ghcr.io/org/train-img@sha256:9c2d...f88
|
||||
eval_spec: configs/eval.yaml@commit a1b2c4d
|
||||
model_digest: registry/model@sha256:b5e1...aa0
|
||||
```
|
||||
|
||||
- Lose any line and the run is anecdote, not evidence.
|
||||
- The record is the lineage root: a prediction cites the
|
||||
`model_digest`, which cites the `run_id`, which cites everything
|
||||
above. This is how P3 (Lineage Traceable End-to-End) is satisfied
|
||||
in practice.
|
||||
|
||||
## What Violates Data Versioning Discipline
|
||||
|
||||
| Violation | Principle |
|
||||
|-----------|-----------|
|
||||
| Dataset referenced by `s3://bucket/latest/` | P2 Data is Versioned, Not Just Code |
|
||||
| Splits regenerated with an unpinned seed per run | P2, P4 Evaluation Defined Before Training |
|
||||
| A production model with no dataset hash in its lineage | P3 Lineage Traceable End-to-End |
|
||||
| Editing a lineage record to "clean up" a broken trace | P3 Lineage Traceable End-to-End |
|
||||
| Eval split reachable from the training data path | P4 Evaluation Defined Before Training |
|
||||
| Two platforms versioning the same data with different models | C4 Locality |
|
||||
@@ -0,0 +1,154 @@
|
||||
# AI / ML — First Principles
|
||||
|
||||
> Scope per D-023: this domain covers ML **engineering discipline** —
|
||||
> data versioning, evaluation methodology, serving patterns, and drift
|
||||
> detection. It does **not** cover algorithm design, model architecture
|
||||
> selection, hyperparameter tuning, or model-family comparison. Those
|
||||
> are research choices, not engineering principles, and they have no
|
||||
> derivation in the core C-rules.
|
||||
|
||||
## 1. The Principles
|
||||
|
||||
### P1. Reproducibility is the First Class
|
||||
Every training run is reproducible from pinned data + code + config +
|
||||
environment. An unreproducible run is an unreviewable run: you cannot
|
||||
decide whether a result is correct if you cannot recreate it.
|
||||
Reproducibility is the ML analogue of `domains/devops/P1
|
||||
Reproducibility` and inherits its non-negotiable status. Lose any one
|
||||
of data, code, config, or environment pinning, and the run is
|
||||
anecdote, not evidence.
|
||||
|
||||
### P2. Data is Versioned, Not Just Code
|
||||
Datasets, features, and train/val/test splits are first-class
|
||||
versioned artifacts with content hashes and lineage. `git` alone is
|
||||
insufficient — datasets do not fit in git, and a dataset is not a
|
||||
commit hash. A model trained on "the data" is a model trained on an
|
||||
unknown input, which is a correctness violation. Version data the way
|
||||
you version code: pinned, named, and recoverable.
|
||||
|
||||
### P3. Lineage is Traceable End-to-End
|
||||
Any deployed prediction traces back through model → training run →
|
||||
dataset → source. No orphan models. A model in production with no
|
||||
lineage is a correctness defect: you cannot reason about its failure
|
||||
modes, you cannot roll it back to a known-good dataset, and you cannot
|
||||
tell whether drift is in the model or in the data that built it.
|
||||
Lineage is the audit trail of ML (`domains/observability/logging.md`).
|
||||
|
||||
### P4. Evaluation is Defined Before Training
|
||||
Metrics, splits, and acceptance thresholds are declared a priori, in
|
||||
code, before the model is trained. Cherry-picking metrics post-hoc is
|
||||
a correctness violation: the evaluation is no longer measuring the
|
||||
model, it is rationalizing it. The eval spec is a contract — it is
|
||||
reviewable, it is versioned, and it is the gate the model must pass
|
||||
before it leaves the experiment. This is the ML angle on C2 Clarity:
|
||||
the intent of the model is obvious to its reader because the eval
|
||||
declared it first.
|
||||
|
||||
### P5. Models are Versioned Artifacts
|
||||
A model is a pinned, immutable, registry-tracked artifact with a
|
||||
unique identifier. Never "the latest." A serving endpoint that pulls
|
||||
"latest" is serving an unknown model — its behavior is undefined, its
|
||||
rollback is impossible, and its lineage is broken. The model registry
|
||||
is to models what a container registry is to images
|
||||
(`domains/devops/P7 Immutability`): immutable, addressed by digest,
|
||||
promoted by stage.
|
||||
|
||||
### P6. Serving is Observable
|
||||
Inference latency, throughput, input distributions, and prediction
|
||||
confidence are first-class signals. Silent serving is a bug. A model
|
||||
in production that emits no metrics is a model you cannot operate: you
|
||||
cannot see latency regressions, you cannot see input drift, you cannot
|
||||
see a failing downstream consumer. Observability is designed in, not
|
||||
bolted on (`domains/observability/metrics.md`).
|
||||
|
||||
### P7. Drift is Expected and Detected
|
||||
Data drift, concept drift, and prediction drift are monitored as a
|
||||
matter of course. A drift signal is an incident, not a curiosity. ML
|
||||
systems decay without code changes — the world changes under the
|
||||
model — so "no code changed" is not a defense against a serving
|
||||
regression. Detecting drift is the ML-specific form of C7
|
||||
Observability: you cannot fix a model you cannot see degrading.
|
||||
|
||||
### P8. Inference Inputs are Validated
|
||||
The model's input contract — schema, value ranges, types, and
|
||||
categorical domains — is enforced at the serving boundary.
|
||||
Out-of-contract inputs are rejected, not silently scored. Scoring an
|
||||
out-of-contract input is a correctness violation: the model's output
|
||||
is undefined for inputs outside its training distribution, and
|
||||
returning a number for it is lying to the caller. This is the ML angle
|
||||
on `domains/security/input-validation.md` and inherits C1's
|
||||
non-tradeable status.
|
||||
|
||||
### P9. Pipelines Compose, Notebooks Don't
|
||||
Training and serving flows are composable pipelines with explicit
|
||||
steps, named inputs, named outputs, and contracts between stages.
|
||||
Notebooks are for exploration, not production. A notebook in the
|
||||
serving path is a correctness defect: its state is implicit, its
|
||||
order is human-dependent, and its reproducibility is whatever the last
|
||||
operator remembered. Compose pipelines; keep notebooks in the lab.
|
||||
|
||||
### P10. Rollback Includes the Model
|
||||
A serving rollback restores the prior model artifact, not just the
|
||||
prior code. Promotion is reversible at the model layer. A rollback
|
||||
that redeploys old code but keeps the new model has not rolled back —
|
||||
the model was the thing that regressed. The rollback path must name
|
||||
the prior model digest, the prior dataset version, and the prior eval
|
||||
that cleared it. This is the ML angle on `domains/devops/P4 Rollback
|
||||
First` and `domains/kubernetes/P10 Roll Forward, Roll Back`.
|
||||
|
||||
## 2. Core Principle Trace
|
||||
|
||||
Each AI/ML P-rule derives from one or more core C-rules (C1–C8). The
|
||||
matrix extension lands in P4 of the v0.3 plan; the traces below are
|
||||
authoritative.
|
||||
|
||||
| P-rule | Core | Why |
|
||||
|--------|------|-----|
|
||||
| P1 Reproducibility is the First Class | C1, C5 | Correctness of results; reversibility of runs |
|
||||
| P2 Data is Versioned, Not Just Code | C5, C7 | Reversibility of datasets; observability of data lineage |
|
||||
| P3 Lineage is Traceable End-to-End | C7, C1 | Observability of provenance; correctness of attribution |
|
||||
| P4 Evaluation is Defined Before Training | C1, C2 | Correctness of the eval gate; clarity of a-priori intent |
|
||||
| P5 Models are Versioned Artifacts | C5, C6 | Reversibility of model identity; composability of registry stages |
|
||||
| P6 Serving is Observable | C7 | Observability of inference |
|
||||
| P7 Drift is Expected and Detected | C7, C1 | Observability of degradation; correctness of detection |
|
||||
| P8 Inference Inputs are Validated | C1 | Correctness of the serving boundary (security subset) |
|
||||
| P9 Pipelines Compose, Notebooks Don't | C6, C2 | Composability of stages; clarity of explicit contracts |
|
||||
| P10 Rollback Includes the Model | C5 | Reversibility at the model layer |
|
||||
|
||||
## 3. What Violates These Principles
|
||||
|
||||
| Violation | Principle Breached |
|
||||
|-----------|-------------------|
|
||||
| A training run that cannot be replayed from pinned inputs | P1 Reproducibility is the First Class |
|
||||
| A dataset referenced by a mutable path, not a hash | P2 Data is Versioned, Not Just Code |
|
||||
| A production model with no record of its training data | P3 Lineage is Traceable End-to-End |
|
||||
| Metrics chosen after seeing the results | P4 Evaluation is Defined Before Training |
|
||||
| A serving endpoint that pulls `latest` from the registry | P5 Models are Versioned Artifacts |
|
||||
| A model in production with no latency or throughput metrics | P6 Serving is Observable |
|
||||
| A serving regression dismissed as "no code changed" | P7 Drift is Expected and Detected |
|
||||
| An input with an out-of-range feature scored silently | P8 Inference Inputs are Validated |
|
||||
| A notebook in the serving or training pipeline path | P9 Pipelines Compose, Notebooks Don't |
|
||||
| A rollback that restores code but keeps the regressed model | P10 Rollback Includes the Model |
|
||||
|
||||
## 4. Relationship to Other Domains
|
||||
|
||||
AI/ML is the engineering-discipline layer for model-bearing systems.
|
||||
It borrows the reproducibility, immutability, rollback, and
|
||||
observability disciplines of `domains/devops/` and applies them to
|
||||
the data → model → serving lifecycle. Cross-links are one-directional
|
||||
(per D-026 extended):
|
||||
|
||||
- `domains/devops/P1 Reproducibility` ← P1
|
||||
- `domains/devops/P4 Rollback First` ← P10
|
||||
- `domains/devops/P5 Progressive Delivery` ← P10 (model canary)
|
||||
- `domains/devops/P7 Immutability` ← P5 (model images)
|
||||
- `domains/data/migrations.md` ← P2 (data versioning ↔ migration discipline)
|
||||
- `domains/data/schema-design.md` ← P8 (inference input contract)
|
||||
- `domains/observability/metrics.md` ← P6, P7
|
||||
- `domains/observability/logging.md` ← P3 (lineage)
|
||||
- `domains/security/input-validation.md` ← P8
|
||||
- `domains/security/secrets.md` ← P8 (serving credentials)
|
||||
- `domains/performance/backend.md` ← P6 (serving latency)
|
||||
- `domains/kubernetes/workloads.md` ← P9 (serving on k8s)
|
||||
- `domains/testing/first-principles.md` ← P4 (eval as a gate)
|
||||
- `domains/gitops-operators/first-principles.md` ← P10 (model rollback in a GitOps loop)
|
||||
@@ -0,0 +1,94 @@
|
||||
# Model Evaluation — Derived Rules
|
||||
|
||||
> Derives from `domains/ai-ml/first-principles.md`. Covers P4
|
||||
> (Evaluation is Defined Before Training) and the eval-as-a-gate
|
||||
> discipline. Referenced by `serving.md` (promotion gate) and
|
||||
> `monitoring-drift.md` (online eval). Scope per D-023: evaluation
|
||||
> methodology, not metric math or model-family benchmarks.
|
||||
|
||||
## Evaluation is a Gate, Not a Report (P4 Evaluation Defined Before Training)
|
||||
|
||||
- The eval spec — metrics, splits, thresholds, and pass/fail
|
||||
criteria — is declared in code **before** the model is trained.
|
||||
It is versioned with the data and the code; it is reviewable; it
|
||||
is the contract the model must satisfy to leave the experiment.
|
||||
- Cherry-picking metrics after seeing results is a correctness
|
||||
violation: the eval is no longer measuring the model, it is
|
||||
rationalizing it. The a-priori spec is what makes the eval
|
||||
trustworthy.
|
||||
- This is the ML angle on `domains/testing/first-principles.md` P1
|
||||
(Tests as Specification): the eval declares the model's contract,
|
||||
the model does not declare its own success.
|
||||
|
||||
## The Eval Input Contract (P8 Inference Inputs are Validated, cross `domains/data/schema-design.md`)
|
||||
|
||||
- The eval set has a schema: feature names, types, ranges, and
|
||||
categorical domains. That schema is the same schema the serving
|
||||
boundary enforces (`serving.md`, `domains/security/input-validation.md`).
|
||||
- An eval set whose schema drifted from the serving schema is
|
||||
measuring a different model than the one in production. Schema
|
||||
parity is part of the versioned eval artifact.
|
||||
- Cross `domains/data/schema-design.md`: the eval input contract is
|
||||
a schema-design problem, versioned and reviewed like any schema.
|
||||
|
||||
## Holdout Integrity (P4, P2 Data is Versioned)
|
||||
|
||||
- The held-out eval set is never touched by training, feature
|
||||
selection, or threshold tuning. A "held out" set that influenced
|
||||
any training decision is not held out — it is a third training
|
||||
set, and the eval is measuring memorization.
|
||||
- Splits are versioned with the dataset (`data-versioning.md`).
|
||||
Recreating splits ad-hoc per run breaks comparability across runs.
|
||||
- Reusing a held-out set across many model iterations leaks it
|
||||
incrementally. Rotate or re-split on a cadence; record the
|
||||
rotation in lineage.
|
||||
|
||||
## Offline vs Online Evaluation (P6 Serving is Observable)
|
||||
|
||||
- **Offline eval** runs before promotion: held-out data, pinned
|
||||
model, declared metrics, pass/fail gate. It answers "should this
|
||||
model ship?"
|
||||
- **Online eval** runs after promotion, on live traffic: shadow
|
||||
scoring, A/B, canary metrics. It answers "is this model behaving
|
||||
in production?" It is the bridge to `monitoring-drift.md`.
|
||||
- A model that passed offline and regressed online is not a
|
||||
contradiction — it is a signal that the offline distribution
|
||||
differs from the live one (a P7 drift signal). Both eval layers
|
||||
are required; neither substitutes for the other.
|
||||
|
||||
## Bias and Fairness Checks (Engineering Angle, P4)
|
||||
|
||||
- Bias/fairness checks are part of the a-priori eval spec, not an
|
||||
afterthought. They are metrics with thresholds, declared before
|
||||
training, gated the same as any metric.
|
||||
- This doc covers the **engineering** discipline: the checks are
|
||||
versioned, gated, and recorded in lineage. The choice of which
|
||||
fairness metrics and what thresholds are policy decisions, not
|
||||
engineering principles, and are out of scope here (D-023).
|
||||
|
||||
## Eval-as-a-Gate in the Pipeline (P9 Pipelines Compose)
|
||||
|
||||
- The eval is a pipeline stage with a contract: input = model
|
||||
digest + eval dataset version; output = pass/fail + metric
|
||||
report. It composes with the training stage and the promotion
|
||||
stage.
|
||||
- A promotion that bypasses the eval stage is a P4 violation,
|
||||
regardless of who approved it. The gate is in the pipeline, not
|
||||
in a human sign-off sheet.
|
||||
|
||||
```
|
||||
train -> eval(gate) -> register(promote) -> serve
|
||||
|
|
||||
+-- fail -> abort, no promote
|
||||
```
|
||||
|
||||
## What Violates Evaluation Discipline
|
||||
|
||||
| Violation | Principle |
|
||||
|-----------|-----------|
|
||||
| Metrics chosen after seeing the scores | P4 Evaluation Defined Before Training |
|
||||
| Held-out set used in feature selection or threshold tuning | P4, P2 |
|
||||
| Eval schema differs from serving schema | P8 Inference Inputs are Validated |
|
||||
| Promotion by human approval, bypassing the eval stage | P4, P9 Pipelines Compose |
|
||||
| A "passing" model with no online eval in production | P6 Serving is Observable |
|
||||
| Fairness checks added after a model shipped | P4 Evaluation Defined Before Training |
|
||||
@@ -0,0 +1,88 @@
|
||||
# Monitoring & Drift — Derived Rules
|
||||
|
||||
> Derives from `domains/ai-ml/first-principles.md`. Covers P7 (Drift
|
||||
> is Expected and Detected) and the online half of P6 (Serving is
|
||||
> Observable). Referenced by `serving.md` (online eval) and
|
||||
> `model-evaluation.md` (online layer). Scope per D-023: drift
|
||||
> detection methodology, not model retraining architecture.
|
||||
|
||||
## Drift is Expected and Detected (P7 Drift is Expected and Detected)
|
||||
|
||||
- ML systems decay without code changes. The world changes under
|
||||
the model: user behavior shifts, input pipelines change,
|
||||
upstream schemas evolve. "No code changed" is not a defense
|
||||
against a serving regression.
|
||||
- A drift signal is an incident, not a curiosity. It triggers an
|
||||
alert, an investigation, and a decision (retrain, roll back, or
|
||||
accept with a recorded justification). Silent drift is the same
|
||||
class of bug as silent serving (P6).
|
||||
- Cross `domains/observability/metrics.md` for the alerting
|
||||
primitives and `domains/observability/logging.md` for the
|
||||
structured events a drift signal emits.
|
||||
|
||||
## The Three Drift Types (IDEATE-30, D-048)
|
||||
|
||||
| Drift Type | What Changes | Detection Signal | Source of Truth |
|
||||
|------------|--------------|------------------|-----------------|
|
||||
| **Data drift** (input drift) | The distribution of inputs at serving time diverges from the distribution the model was trained on | Statistical distance between the live input distribution and the pinned training-set distribution (e.g., PSI, KL, KS test). Alert on threshold breach | Training dataset hash (`data-versioning.md`) + live input metrics |
|
||||
| **Concept drift** | The relationship between inputs and the target changes — the same input now maps to a different correct output | Ground-truth lag: compare delayed labels against predictions on the same inputs. Rising error rate against a stable input distribution signals concept, not data, drift | Delayed-label feedback stream + prediction log |
|
||||
| **Prediction drift** (output drift) | The distribution of the model's predictions shifts, with no change to inputs | Statistical distance between the live prediction distribution and a pinned baseline prediction distribution. Independent of inputs — catches model-internal regressions and upstream silent changes | Prediction log + baseline prediction snapshot |
|
||||
|
||||
- The three signals are distinct and non-substitutable. Data drift
|
||||
catches the input changing; concept drift catches the world
|
||||
changing; prediction drift catches the model's behavior changing.
|
||||
A monitoring setup with only one is blind to two classes of
|
||||
regression.
|
||||
- Evidently AI and Great Expectations are the canonical tooling:
|
||||
Evidently for drift/statistical reports, Great Expectations for
|
||||
data-quality/contract checks at the pipeline boundary. Both
|
||||
produce the metrics that feed `domains/observability/metrics.md`.
|
||||
|
||||
## Detection Signals in Practice
|
||||
|
||||
- **Data drift** compares live inputs to the **pinned training
|
||||
distribution** — not to "yesterday's inputs." Without a pinned
|
||||
baseline, drift is measured against a moving target and is
|
||||
meaningless. Cross `data-versioning.md` for how the baseline is
|
||||
pinned.
|
||||
- **Concept drift** requires ground truth, which is often delayed
|
||||
(days/weeks). The detection signal is the gap between
|
||||
prediction-time confidence and delayed-label error. A rising
|
||||
error against stable inputs is the signature.
|
||||
- **Prediction drift** needs no ground truth and no input
|
||||
comparison — it watches the model's own output distribution. It
|
||||
is the cheapest signal and the first to fire; it is also the
|
||||
least specific (any of the three drifts can move predictions).
|
||||
|
||||
## Alerting and Retraining Triggers (P7, P10 Rollback Includes the Model)
|
||||
|
||||
- A drift alert is an incident. It does not auto-trigger retraining
|
||||
unsupervised — auto-retraining on drift can lock in a bad
|
||||
distribution. The alert triggers a human decision: investigate,
|
||||
retrain, roll back, or accept.
|
||||
- Retraining is a new training run (`first-principles.md` P1): it
|
||||
produces a new model digest, passes the eval gate
|
||||
(`model-evaluation.md`), and is promoted through the registry
|
||||
(`serving.md`). The prior model stays rollbackable (P10).
|
||||
- Cross `domains/observability/metrics.md` for the alert-rule
|
||||
pattern: threshold + window + severity, routed to the same
|
||||
on-call path as any production incident.
|
||||
|
||||
## Online Evaluation Bridge (P6 Serving is Observable)
|
||||
|
||||
- Online eval (`model-evaluation.md`) is the live counterpart to
|
||||
drift monitoring: shadow scores and A/B canaries measure a
|
||||
candidate model against the incumbent, while drift monitoring
|
||||
measures the incumbent against its own baseline. Both feed the
|
||||
same metrics pipeline.
|
||||
|
||||
## What Violates Monitoring Discipline
|
||||
|
||||
| Violation | Principle |
|
||||
|-----------|-----------|
|
||||
| Only one drift type monitored | P7 Drift is Expected and Detected |
|
||||
| Drift baseline is "yesterday's inputs," not pinned training data | P7, P2 Data is Versioned |
|
||||
| Drift alert that auto-retrains without a human gate | P7, P1 Reproducibility |
|
||||
| A serving regression dismissed as "no code changed" | P7 Drift is Expected and Detected |
|
||||
| Concept-drift check with no delayed-label feedback path | P7 Drift is Expected and Detected |
|
||||
| Prediction-distribution change with no alert | P6 Serving is Observable, P7 |
|
||||
@@ -0,0 +1,88 @@
|
||||
# Serving — Derived Rules
|
||||
|
||||
> Derives from `domains/ai-ml/first-principles.md`. Covers P5 (Models
|
||||
> are Versioned Artifacts), P6 (Serving is Observable), P8 (Inference
|
||||
> Inputs are Validated), and P10 (Rollback Includes the Model).
|
||||
> Referenced by `monitoring-drift.md` (online signals) and
|
||||
> `model-evaluation.md` (promotion gate). Scope per D-023: serving
|
||||
> patterns, not model architectures.
|
||||
|
||||
## The Model is an Addressed Artifact (P5 Models are Versioned Artifacts)
|
||||
|
||||
- A serving endpoint pulls a model by digest, never by `latest`. A
|
||||
model pulled by `latest` is an unknown model — its behavior is
|
||||
undefined and its rollback is impossible.
|
||||
- The model registry is to models what a container registry is to
|
||||
images (`domains/devops/P7 Immutability`): immutable, addressed by
|
||||
digest, promoted by stage (staging → prod). Promotion is a
|
||||
registry operation, not a file copy.
|
||||
- A serving rollout names the model digest in its manifest. The
|
||||
digest is part of the deploy's lineage (`data-versioning.md`).
|
||||
|
||||
## Inference Inputs are Validated (P8 Inference Inputs are Validated)
|
||||
|
||||
- The model's input contract — schema, types, ranges, categorical
|
||||
domains — is enforced at the serving boundary, before the model
|
||||
sees the input. Out-of-contract inputs are rejected with a
|
||||
defined error, not silently scored.
|
||||
- Scoring an out-of-contract input is a C1 (Correctness) violation:
|
||||
the model's output is undefined outside its training
|
||||
distribution, and returning a number for it is lying to the
|
||||
caller.
|
||||
- This is the ML angle on `domains/security/input-validation.md`:
|
||||
the validation lives at the boundary, the model is downstream of
|
||||
it, and the contract is versioned with the model.
|
||||
|
||||
## Serving is Observable (P6 Serving is Observable)
|
||||
|
||||
- Every inference path emits: request latency, throughput, input
|
||||
distribution summaries, prediction confidence, and error counts.
|
||||
Silent serving is a bug.
|
||||
- Cross `domains/observability/metrics.md` for the metrics
|
||||
primitives (histograms, counters, gauges) and
|
||||
`domains/observability/tracing.md` for the request-level trace
|
||||
that ties an input to a prediction.
|
||||
- Latency SLAs are enforced via `domains/performance/backend.md`
|
||||
disciplines: budget the inference path, measure the tail (p99),
|
||||
alert on budget breach.
|
||||
|
||||
## Serving Patterns (P9 Pipelines Compose)
|
||||
|
||||
| Pattern | When | Notes |
|
||||
|---------|------|-------|
|
||||
| Inference as a service | Default; model behind an HTTP/gRPC endpoint | KServe, Seldon Core, BentoML. Scales with traffic; model is a deployable, addressable artifact |
|
||||
| Batch inference | Offline scoring of large datasets | No latency SLA; throughput-bound. Same model digest, same input contract |
|
||||
| Embedded / in-process | Latency-critical, single-tenant | Model linked into the app. Trades observability for latency — only when the SLA demands it |
|
||||
|
||||
- Canarying a model is a serving pattern, not a deployment pattern:
|
||||
shift a fraction of traffic to the new model digest, measure
|
||||
online eval (`model-evaluation.md`), abort to the prior digest on
|
||||
regression. This is `domains/devops/P5 Progressive Delivery`
|
||||
applied at the model layer.
|
||||
- Rollback restores the prior model digest (P10 Rollback Includes
|
||||
the Model). A rollback that redeploys old code but keeps the new
|
||||
model has not rolled back. Cross `domains/gitops-operators/first-principles.md`
|
||||
for the GitOps reconciliation loop that drives model rollouts.
|
||||
|
||||
## Tool Landscape (KServe / Seldon Core / BentoML)
|
||||
|
||||
| Tool | Model Packaging | Deployment Surface | Notes |
|
||||
|------|-----------------|--------------------|-------|
|
||||
| KServe | InferenceService CRD; runtime predictors (v2, HuggingFace, PMML, custom) | Kubernetes-native; CRD-driven | Cross `domains/kubernetes/workloads.md`. Brings the k8s reconciliation model to serving |
|
||||
| Seldon Core | SeldonDeployment CRD; graph of predictors | Kubernetes-native; CRD-driven | Emphasizes inference graphs (fan-out, ensemble) as CRD structure |
|
||||
| BentoML | Bento (model + runtime + deps packaged); Yatai registry | Kubernetes or bare container | Focuses on packaging + registry; the Bento is the versioned artifact (P5) |
|
||||
|
||||
- All three satisfy P5/P6/P8 when wired correctly; the choice is
|
||||
packaging model and deployment surface, not correctness.
|
||||
- None is advocated over the others.
|
||||
|
||||
## What Violates Serving Discipline
|
||||
|
||||
| Violation | Principle |
|
||||
|-----------|-----------|
|
||||
| Endpoint pulls `latest` from the registry | P5 Models are Versioned Artifacts |
|
||||
| Out-of-range input scored silently | P8 Inference Inputs are Validated |
|
||||
| Serving path emits no latency or throughput metrics | P6 Serving is Observable |
|
||||
| Rollback redeploys code but keeps the regressed model | P10 Rollback Includes the Model |
|
||||
| A notebook in the serving path | P9 Pipelines Compose, Notebooks Don't |
|
||||
| Canary with no abort-to-prior-digest path | P10, `domains/devops/P5 Progressive Delivery` |
|
||||
@@ -0,0 +1,176 @@
|
||||
# ArgoCD — Derived Rules
|
||||
|
||||
> Derives from `domains/gitops-operators/first-principles.md`.
|
||||
> Applies P1–P10 to ArgoCD specifically. For the ArgoCD-vs-Flux
|
||||
> decision, see the decision matrix at the end of this doc and in
|
||||
> `flux.md`.
|
||||
|
||||
## What ArgoCD Is (P1 Git is the Source of Truth, P3 Pull, Don't Push)
|
||||
|
||||
- ArgoCD is a pull-based GitOps controller for Kubernetes. It runs
|
||||
inside the target cluster, pulls desired state from git, and
|
||||
reconciles the cluster to match. CI never holds `kubectl` rights
|
||||
against the cluster (P3).
|
||||
- An Application is a declarative binding of "this git path" to
|
||||
"this cluster destination." The Application CRD is the unit of
|
||||
reconciliation. The cluster is a derivative of git, never the
|
||||
authority (P1).
|
||||
- ArgoCD supports Helm charts, Kustomize overlays, ksonnet, and raw
|
||||
manifests as source formats — see `domains/kubernetes/helm.md`
|
||||
and `domains/kubernetes/kustomize.md`.
|
||||
|
||||
## Application CRD (P2 Declarative Over Imperative, P4 Continuous Reconciliation)
|
||||
|
||||
- An Application declares `source` (repo, path, revision, chart),
|
||||
`destination` (server, namespace), and `syncPolicy`. The
|
||||
reconciler loops continuously; drift is corrected automatically,
|
||||
not on-demand (P4).
|
||||
|
||||
```yaml
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: payments-api
|
||||
namespace: argocd
|
||||
spec:
|
||||
source:
|
||||
repoURL: https://git.example.com/platform/payments
|
||||
targetRevision: 1.2.3
|
||||
path: manifests/prod
|
||||
destination:
|
||||
server: https://kubernetes.default.svc
|
||||
namespace: payments
|
||||
syncPolicy:
|
||||
automated:
|
||||
prune: true
|
||||
selfHeal: true
|
||||
syncOptions:
|
||||
- CreateNamespace=false
|
||||
```
|
||||
|
||||
- `automated.prune: true` deletes resources removed from git.
|
||||
`selfHeal: true` corrects hand-edited drift back to git (P8).
|
||||
Disable both for workloads that need manual approval gates.
|
||||
|
||||
## App-of-Apps (P6 Operators Encode Domain Knowledge, C6 Composability)
|
||||
|
||||
- The App-of-Apps pattern: one root Application points at a git
|
||||
directory of child Application manifests. The root app reconciles
|
||||
the children; the children reconcile the workloads. This is the
|
||||
ArgoCD expression of composition — a fleet of apps as a tree of
|
||||
Applications.
|
||||
- Use App-of-Apps for cluster bootstrapping (one repo, many
|
||||
clusters, many apps). Do not use it as a substitute for a package
|
||||
manager; if you are templating hundreds of near-identical
|
||||
Applications, use a generator (ApplicationSet) instead.
|
||||
|
||||
## Sync Waves and Hooks (P4 Continuous Reconciliation, P7 Reversibility)
|
||||
|
||||
- Sync waves order resources within a sync: `PreSync` → `Sync` →
|
||||
`PostSync`. Use waves to run a job before a Deployment, or a
|
||||
migration before the app that depends on it.
|
||||
- Sync hooks (`PreSync`, `Sync`, `PostSync`, `SyncFail`) are
|
||||
Resources annotated to execute at a wave boundary. A `SyncFail`
|
||||
hook runs on sync failure — the abort path (P7).
|
||||
- Wave ordering is a correctness mechanism, not a performance one.
|
||||
Mis-ordered waves (e.g., app starts before its migration job)
|
||||
are a correctness bug.
|
||||
|
||||
## Health and Status (P9 Failure is Observable and Surfaced)
|
||||
|
||||
- ArgoCD assesses every resource's health (`Healthy`, `Progressing`,
|
||||
`Degraded`, `Missing`, `Suspended`) and surfaces the aggregate as
|
||||
Application status. Sync status (`Synced`, `OutOfSync`) reports
|
||||
drift against git.
|
||||
- Health checks are pluggable via Lua scripts for custom CRDs. An
|
||||
Operator-managed CRD without a health check reads as `Progressing`
|
||||
forever — write one (see `operators.md`).
|
||||
- Out-of-sync or degraded status must emit a notification (Slack,
|
||||
PagerDuty, webhook). Silent drift is the bug (P9). Wire status to
|
||||
`domains/observability/metrics.md`.
|
||||
|
||||
## Diff and Drift (P8 Reconcile, Don't Mutate by Hand, P4)
|
||||
|
||||
- `argocd app diff` shows the diff between git and live cluster.
|
||||
A non-empty diff on a synced app is hand-edit drift — the
|
||||
recovery is `selfHeal`, not a manual `kubectl apply` (P8).
|
||||
- Drift detection runs continuously (P4). The gap between "git
|
||||
changed" and "cluster matches git" is observable, not assumed.
|
||||
|
||||
## RBAC and SSO (P10 Least Privilege Reconciliation)
|
||||
|
||||
- ArgoCD's own RBAC governs who can view, sync, and admin
|
||||
Applications. Bind to SSO (OIDC, SAML) for human identity; bind
|
||||
the controller's service account to a Role scoped to the
|
||||
namespaces it reconciles.
|
||||
- The controller's credentials must not be `cluster-admin` (P10).
|
||||
Use namespace-scoped Roles via `ApplicationSet` namespaces or
|
||||
cluster-wide AppProject restrictions. See
|
||||
`domains/kubernetes/rbac.md` and `domains/security/authorization.md`.
|
||||
- AppProjects bound the blast radius of what an Application can
|
||||
deploy (allowed repos, destinations, roles). One AppProject per
|
||||
team or environment; the default project is for nothing in
|
||||
production.
|
||||
|
||||
## Multi-Cluster (P4 Locality, P10)
|
||||
|
||||
- ArgoCD registers external clusters by secret. The controller
|
||||
pulls from git and pushes to the registered cluster's API server.
|
||||
The "pull, don't push" boundary (P3) is between the target
|
||||
cluster's reconciler and CI — the controller-to-apiserver hop is
|
||||
internal to the platform.
|
||||
- Scope each registered cluster's credentials to the namespaces
|
||||
ArgoCD manages there. Do not register a cluster with cluster-admin
|
||||
and call it done (P10).
|
||||
|
||||
## Sync Windows (P5 Reversibility, P7)
|
||||
|
||||
- Sync windows restrict when automated sync runs (e.g., no syncs
|
||||
during business hours, or syncs only in a maintenance window).
|
||||
They are a reversibility mechanism: a bad commit lands in git,
|
||||
but the sync window holds it until review.
|
||||
- Sync windows do not replace health monitoring (P9). A degraded
|
||||
app inside a window is still an incident.
|
||||
|
||||
## Secrets (P10, cross-link security/secrets)
|
||||
|
||||
- Do not store raw Secrets in the GitOps repo. Use a sealed-secret
|
||||
controller (Bitnami Sealed Secrets, SOPS, External Secrets
|
||||
Operator) so the git store holds encrypted material only. See
|
||||
`domains/security/secrets.md` for the general secret-hygiene
|
||||
principles.
|
||||
|
||||
## ArgoCD vs Flux — Decision Matrix (IDEATE-21, D-039)
|
||||
|
||||
| Axis | ArgoCD | Flux |
|
||||
|------|--------|------|
|
||||
| Architecture | Monolithic controller + Application CRD | Composable GitOps Toolkit controllers (source, kustomize, helm, notification) |
|
||||
| Reconciliation unit | Application (one CRD per app) | Kustomization / HelmRelease (one per deploy unit) |
|
||||
| UI | Web UI + CLI (full dashboard, tree view, diff viewer) | CLI-first; UI via Weave GitOps or FluxUI (add-on) |
|
||||
| Sync model | Periodic poll or webhook; sync waves + hooks | Poll + webhook; runs continuously, no explicit sync waves |
|
||||
| Multi-cluster | One ArgoCD manages many clusters (hub-and-spoke) | One Flux per cluster (per-cluster autonomy) |
|
||||
| Templating in repo | Helm, Kustomize, ksonnet, raw manifests, Jsonnet | Helm, Kustomize, raw manifests |
|
||||
| RBAC | Built-in RBAC + SSO + AppProjects | Kubernetes RBAC (no built-in RBAC layer) |
|
||||
| Progressive delivery | Argo Rollouts (sister project, tight integration) | Flagger (sister project, tight integration) |
|
||||
| Best for | Teams wanting a UI, multi-cluster from one pane, App-of-Apps bootstrapping | Teams wanting composable controllers, per-cluster autonomy, minimal footprint |
|
||||
| Watch out for | Monolithic controller scaling, UI as ops crutch, AppProject sprawl | No native UI, steeper learning curve, manual multi-cluster orchestration |
|
||||
|
||||
- Use ArgoCD when you want a UI, central multi-cluster management,
|
||||
and sync-wave ordering. Use Flux when you want composable
|
||||
controllers, per-cluster autonomy, and a minimal footprint.
|
||||
- Both are CNCF graduated and both implement the OpenGitOps
|
||||
principles. The choice is architectural fit, not correctness. See
|
||||
`flux.md` for the Flux-side perspective.
|
||||
|
||||
## What Violates ArgoCD Discipline
|
||||
|
||||
| Violation | Principle |
|
||||
|-----------|-----------|
|
||||
| CI pipeline with `kubectl` rights pushing to the cluster | P3 Pull, Don't Push |
|
||||
| `argocd app set` used as the steady state instead of git | P1 Git is the Source of Truth |
|
||||
| `selfHeal: false` on a prod app with no manual gate | P8 Reconcile, Don't Mutate by Hand |
|
||||
| Controller ServiceAccount bound to `cluster-admin` | P10 Least Privilege Reconciliation |
|
||||
| Sync failure with no notification wired | P9 Failure is Observable and Surfaced |
|
||||
| AppProject with no destination restrictions in prod | P10 Least Privilege Reconciliation |
|
||||
| Raw Secret in the GitOps repo | P10, `domains/security/secrets.md` |
|
||||
| Manual `kubectl edit` on an ArgoCD-managed resource | P8 Reconcile, Don't Mutate by Hand |
|
||||
@@ -0,0 +1,131 @@
|
||||
# GitOps + Operators — First Principles
|
||||
|
||||
## 1. The Principles
|
||||
|
||||
### P1. Git is the Source of Truth
|
||||
Desired state lives in a versioned, immutable git store. The
|
||||
cluster is a derivative of git, never the authority. If a state
|
||||
exists only in the cluster and not in git, it is drift, not truth.
|
||||
The commit history is the audit trail and the rollback path.
|
||||
|
||||
### P2. Declarative Over Imperative
|
||||
Express the desired cluster state, not the commands to reach it.
|
||||
A manifest says what should exist; the reconciler makes it so.
|
||||
Imperative `kubectl` is for inspection and incident response, not
|
||||
for the steady state. This is the GitOps expression of
|
||||
`domains/kubernetes/P1 Declarative Desired State` and
|
||||
`domains/infrastructure-as-code/P1 Declarative Intent`.
|
||||
|
||||
### P3. Pull, Don't Push
|
||||
Agents running inside the target pull desired state from git; the
|
||||
target never accepts outside push credentials. No CI pipeline holds
|
||||
`kubectl` rights against the production cluster. The cluster reaches
|
||||
out to git, not the other way around. This is the security primitive
|
||||
of GitOps: the blast radius of a compromised CI is bounded by what CI
|
||||
can push, and a pull model gives CI nothing to push.
|
||||
|
||||
### P4. Continuous Reconciliation
|
||||
The reconciliation loop is the primitive. Drift is detected and
|
||||
corrected automatically, not on-demand. A manual `apply` is an
|
||||
exception, not the workflow. The loop runs continuously; the gap
|
||||
between "git changed" and "cluster matches git" is measured in
|
||||
seconds, not tickets.
|
||||
|
||||
### P5. State is Immutable and Versioned
|
||||
Every change to desired state is a commit. History is the audit
|
||||
trail and the rollback path. A revert is a rollback; a force-push is
|
||||
history deletion. The git store is treated like
|
||||
`domains/infrastructure-as-code/P3 State is Truth` — lose it or
|
||||
tamper with it, and you lose the ability to reason about the system.
|
||||
|
||||
### P6. Operators Encode Domain Knowledge
|
||||
Operational expertise lives as CRDs plus controllers, not as
|
||||
runbooks that humans must remember. An operator is a control loop
|
||||
that encodes how to reconcile a specific domain (a database, a
|
||||
message queue, a certificate). The operator is the deepest
|
||||
expression of `domains/kubernetes/P1 Declarative Desired State` —
|
||||
the domain knowledge is the desired state.
|
||||
|
||||
### P7. Progressive Delivery is Reversible by Construction
|
||||
Canary and blue-green are staged, metric-gated, and one-command
|
||||
abortable. Promotion without a rollback path is a violation. A
|
||||
rollout that cannot be aborted is a deploy, not a progressive
|
||||
delivery. This is the GitOps extension of
|
||||
`domains/devops/P5 Progressive Delivery` and
|
||||
`domains/kubernetes/P10 Roll Forward, Roll Back`.
|
||||
|
||||
### P8. Reconcile, Don't Mutate by Hand
|
||||
Manual `kubectl apply` or `kubectl edit` on a GitOps-managed
|
||||
resource is an incident. The reconciler will overwrite the hand
|
||||
edit on the next loop; the hand edit was never truth. Drift back to
|
||||
git is the recovery, not the failure. This is the GitOps angle on
|
||||
`domains/infrastructure-as-code/P9 Drift is Recoverable`.
|
||||
|
||||
### P9. Failure is Observable and Surfaced
|
||||
Sync failures, health degradation, and rollout-stall events emit
|
||||
status and notifications. Silent drift is the bug. A GitOps
|
||||
controller that fails to sync without surfacing the failure has
|
||||
violated the contract — you cannot fix what you cannot see
|
||||
(`domains/observability/metrics.md`).
|
||||
|
||||
### P10. Least Privilege Reconciliation
|
||||
The controller's credentials are scoped to the namespaces and
|
||||
resources it reconciles. No `cluster-admin` GitOps robots. One
|
||||
credential set per boundary; the reconciler sees only what it
|
||||
reconciles. This is the GitOps angle on
|
||||
`domains/kubernetes/P7 RBAC by Intent, Not Identity` and
|
||||
`domains/security/authorization.md`.
|
||||
|
||||
## 2. Core Principle Trace
|
||||
|
||||
Each GitOps + Operators P-rule derives from one or more core
|
||||
C-rules (C1–C8). The matrix extension lands in P4 of the v0.3
|
||||
plan; the traces below are authoritative.
|
||||
|
||||
| P-rule | Core | Why |
|
||||
|--------|------|-----|
|
||||
| P1 Git is the Source of Truth | C1, C5 | Correctness of state; reversibility via history |
|
||||
| P2 Declarative Over Imperative | C2, C3 | Clarity of intent; simplicity of mental model |
|
||||
| P3 Pull, Don't Push | C1, C4 | Correctness via security; locality of credentials |
|
||||
| P4 Continuous Reconciliation | C7, C1 | Observability of drift; correctness of convergence |
|
||||
| P5 State is Immutable and Versioned | C5 | Reversibility via version history |
|
||||
| P6 Operators Encode Domain Knowledge | C6, C2 | Composability of expertise; clarity of operational intent |
|
||||
| P7 Progressive Delivery is Reversible | C5, C1 | Reversibility of promotion; correctness of abort |
|
||||
| P8 Reconcile, Don't Mutate by Hand | C1, C7 | Correctness of single source; observability of drift |
|
||||
| P9 Failure is Observable and Surfaced | C7 | Observability of reconciliation |
|
||||
| P10 Least Privilege Reconciliation | C1, C8 | Correctness via security; economy of trust |
|
||||
|
||||
## 3. What Violates These Principles
|
||||
|
||||
| Violation | Principle Breached |
|
||||
|-----------|-------------------|
|
||||
| CI pipeline pushes manifests to the cluster | P3 Pull, Don't Push |
|
||||
| A resource exists in the cluster but not in git | P1 Git is the Source of Truth |
|
||||
| `kubectl edit` on a GitOps-managed resource | P8 Reconcile, Don't Mutate by Hand |
|
||||
| Reconciler with `cluster-admin` ClusterRoleBinding | P10 Least Privilege Reconciliation |
|
||||
| Sync failure with no status or notification | P9 Failure is Observable and Surfaced |
|
||||
| Canary with no abort/rollback path | P7 Progressive Delivery is Reversible |
|
||||
| Operator runbook that exists only in a wiki | P6 Operators Encode Domain Knowledge |
|
||||
| Reconciler that applies on a cron, not continuously | P4 Continuous Reconciliation |
|
||||
| Force-push rewrites GitOps repo history | P5 State is Immutable and Versioned |
|
||||
| Imperative deploy script as the steady state | P2 Declarative Over Imperative |
|
||||
|
||||
## 4. Relationship to Other Domains
|
||||
|
||||
GitOps + Operators is the deployment-automation layer above
|
||||
`domains/kubernetes/` and `domains/infrastructure-as-code/`. It
|
||||
borrows their declarative-reconciliation model and adds the
|
||||
git-as-source-of-truth and pull-based credential boundaries. Cross
|
||||
links are one-directional (per D-026 extended):
|
||||
|
||||
- `domains/kubernetes/P1 Declarative Desired State` ← P2
|
||||
- `domains/kubernetes/P10 Roll Forward, Roll Back` ← P7
|
||||
- `domains/infrastructure-as-code/P1 Declarative Intent` ← P2
|
||||
- `domains/infrastructure-as-code/P3 State is Truth` ← P1, P5
|
||||
- `domains/infrastructure-as-code/P9 Drift is Recoverable` ← P4, P8
|
||||
- `domains/devops/P4 Rollback First` ← P5, P7
|
||||
- `domains/devops/P5 Progressive Delivery` ← P7
|
||||
- `domains/devops/P6 Configuration as Code` ← P1, P2
|
||||
- `domains/security/secrets.md` ← P3, P10 (reconciliation credentials)
|
||||
- `domains/security/supply-chain.md` ← P5 (signed, immutable provenance)
|
||||
- `domains/observability/metrics.md` ← P4, P9 (reconciliation + rollout metrics)
|
||||
@@ -0,0 +1,159 @@
|
||||
# Flux — Derived Rules
|
||||
|
||||
> Derives from `domains/gitops-operators/first-principles.md`.
|
||||
> Applies P1–P10 to Flux specifically. For the ArgoCD-vs-Flux
|
||||
> decision, see the decision matrix at the end of this doc and in
|
||||
> `argocd.md`.
|
||||
|
||||
## What Flux Is (P1 Git is the Source of Truth, P3 Pull, Don't Push)
|
||||
|
||||
- Flux is a set of composable controllers — the GitOps Toolkit —
|
||||
that run inside the target cluster, pull desired state from git
|
||||
or OCI registries, and reconcile the cluster to match. CI never
|
||||
holds `kubectl` rights against the cluster (P3).
|
||||
- The composable-controller architecture is a C6 (Composability)
|
||||
exemplar: each controller does one thing (source, kustomize, helm,
|
||||
notification) and the controllers compose into a full GitOps
|
||||
system.
|
||||
- Flux supports Helm releases, Kustomize overlays, and raw
|
||||
manifests — see `domains/kubernetes/helm.md` and
|
||||
`domains/kubernetes/kustomize.md`.
|
||||
|
||||
## GitOps Toolkit Controllers (P6 Composability, P4 Continuous Reconciliation)
|
||||
|
||||
- **source-controller** — pulls git, Helm, OCI, and bucket sources;
|
||||
emits artifacts (tarballs) with a digest. The source is the
|
||||
pinned input to reconciliation (P5 versioning by digest).
|
||||
- **kustomize-controller** — reconciles Kustomization CRDs against
|
||||
the artifacts from source-controller. Runs continuously (P4).
|
||||
- **helm-controller** — reconciles HelmRelease CRDs against Helm
|
||||
charts from source-controller.
|
||||
- **notification-controller** — emits events and notifications for
|
||||
sync, health, and source-readiness events (P9).
|
||||
- **image-automation-controller** (optional) — updates git with new
|
||||
image tags when a policy matches, closing the "latest image"
|
||||
loop declaratively.
|
||||
|
||||
## Kustomization CRD (P2 Declarative Over Imperative, P4)
|
||||
|
||||
- A Kustomization binds "this source" to "this target namespace"
|
||||
with a reconciliation interval. The reconciler loops
|
||||
continuously; drift is corrected automatically (P4).
|
||||
|
||||
```yaml
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: payments-api
|
||||
namespace: flux-system
|
||||
spec:
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: platform
|
||||
namespace: flux-system
|
||||
path: ./manifests/prod
|
||||
targetNamespace: payments
|
||||
interval: 1m
|
||||
prune: true
|
||||
wait: true
|
||||
healthChecks:
|
||||
- apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: payments-api
|
||||
namespace: payments
|
||||
```
|
||||
|
||||
- `prune: true` deletes resources removed from git. `wait: true`
|
||||
waits for health checks before declaring the Kustomization ready.
|
||||
Disable prune for workloads that need manual removal gates.
|
||||
|
||||
## HelmRelease CRD (P6 Composability, cross-link helm.md)
|
||||
|
||||
- A HelmRelease binds a Helm chart (from a HelmRepository or OCI
|
||||
source) to target values and a target namespace. helm-controller
|
||||
renders and applies it. See `domains/kubernetes/helm.md` for the
|
||||
chart model.
|
||||
- Pin the chart version in the HelmRepository or the HelmRelease.
|
||||
Never float `latest` — unversioned charts drift (P5).
|
||||
|
||||
## OCI Sources (P5 State is Immutable and Versioned)
|
||||
|
||||
- source-controller can pull from OCI registries (Helm charts as
|
||||
OCI artifacts, or generic OCI repositories). The digest is the
|
||||
version — immutable by construction (P5).
|
||||
- OCI sources close the supply-chain loop: the manifest is signed
|
||||
and immutable in the registry, and Flux pulls it by digest. Cross-
|
||||
link `domains/security/supply-chain.md` for signed-provenance
|
||||
principles.
|
||||
|
||||
## Reconciliation and Drift (P4 Continuous Reconciliation, P8)
|
||||
|
||||
- Flux reconciles on `interval` (default 1m) and on webhook event.
|
||||
Drift between git and cluster is detected each interval and
|
||||
corrected (with `prune` + `selfHeal` semantics).
|
||||
- Hand-edited drift on a Flux-managed resource is overwritten on the
|
||||
next loop — the hand edit was never truth (P8). The recovery is
|
||||
to fix git, not to `kubectl apply`.
|
||||
|
||||
## Notifications and Events (P9 Failure is Observable and Surfaced)
|
||||
|
||||
- notification-controller emits events for source readiness, sync
|
||||
success/failure, and health transitions. Wire them to Slack,
|
||||
PagerDuty, or a webhook. Silent drift is the bug (P9).
|
||||
- Events flow to `domains/observability/metrics.md` via the
|
||||
notification controller's provider model — sync and health as
|
||||
first-class signals.
|
||||
|
||||
## RBAC and Multi-Cluster (P10 Least Privilege Reconciliation, P4)
|
||||
|
||||
- Flux's controllers run with a ServiceAccount in `flux-system`.
|
||||
Scope that account to the namespaces Flux reconciles. Do not bind
|
||||
it to `cluster-admin` (P10). See `domains/kubernetes/rbac.md` and
|
||||
`domains/security/authorization.md`.
|
||||
- Flux is per-cluster by design (one Flux install per cluster). For
|
||||
multi-cluster, use one repo with per-cluster paths, or a fleet
|
||||
tool that bootstraps Flux per cluster. Per-cluster autonomy is a
|
||||
feature, not a limitation — it bounds the blast radius of a
|
||||
compromised controller (P4 locality, P10).
|
||||
|
||||
## Secrets (P10, cross-link security/secrets)
|
||||
|
||||
- Do not store raw Secrets in the GitOps repo. Use the
|
||||
SOPS-compatible decryption in kustomize-controller, or External
|
||||
Secrets Operator, so the git store holds encrypted material only.
|
||||
See `domains/security/secrets.md`.
|
||||
|
||||
## ArgoCD vs Flux — Decision Matrix (IDEATE-21, D-039)
|
||||
|
||||
| Axis | ArgoCD | Flux |
|
||||
|------|--------|------|
|
||||
| Architecture | Monolithic controller + Application CRD | Composable GitOps Toolkit controllers (source, kustomize, helm, notification) |
|
||||
| Reconciliation unit | Application (one CRD per app) | Kustomization / HelmRelease (one per deploy unit) |
|
||||
| UI | Web UI + CLI (full dashboard, tree view, diff viewer) | CLI-first; UI via Weave GitOps or FluxUI (add-on) |
|
||||
| Sync model | Periodic poll or webhook; sync waves + hooks | Poll + webhook; runs continuously, no explicit sync waves |
|
||||
| Multi-cluster | One ArgoCD manages many clusters (hub-and-spoke) | One Flux per cluster (per-cluster autonomy) |
|
||||
| Templating in repo | Helm, Kustomize, ksonnet, raw manifests, Jsonnet | Helm, Kustomize, raw manifests |
|
||||
| RBAC | Built-in RBAC + SSO + AppProjects | Kubernetes RBAC (no built-in RBAC layer) |
|
||||
| Progressive delivery | Argo Rollouts (sister project, tight integration) | Flagger (sister project, tight integration) |
|
||||
| Best for | Teams wanting a UI, multi-cluster from one pane, App-of-Apps bootstrapping | Teams wanting composable controllers, per-cluster autonomy, minimal footprint |
|
||||
| Watch out for | Monolithic controller scaling, UI as ops crutch, AppProject sprawl | No native UI, steeper learning curve, manual multi-cluster orchestration |
|
||||
|
||||
- Use Flux when you want composable controllers, per-cluster
|
||||
autonomy, and a minimal footprint. Use ArgoCD when you want a UI,
|
||||
central multi-cluster management, and sync-wave ordering.
|
||||
- Both are CNCF graduated and both implement the OpenGitOps
|
||||
principles. The choice is architectural fit, not correctness. See
|
||||
`argocd.md` for the ArgoCD-side perspective.
|
||||
|
||||
## What Violates Flux Discipline
|
||||
|
||||
| Violation | Principle |
|
||||
|-----------|-----------|
|
||||
| CI pipeline with `kubectl` rights pushing to the cluster | P3 Pull, Don't Push |
|
||||
| HelmRelease with no pinned chart version | P5 State is Immutable and Versioned |
|
||||
| Flux ServiceAccount bound to `cluster-admin` | P10 Least Privilege Reconciliation |
|
||||
| Kustomization with no `healthChecks` on a prod app | P9 Failure is Observable and Surfaced |
|
||||
| No notification provider wired for sync failures | P9 Failure is Observable and Surfaced |
|
||||
| Raw Secret in the GitOps repo | P10, `domains/security/secrets.md` |
|
||||
| Manual `kubectl edit` on a Flux-managed resource | P8 Reconcile, Don't Mutate by Hand |
|
||||
| `interval: 24h` on a prod Kustomization (drift window too wide) | P4 Continuous Reconciliation |
|
||||
@@ -0,0 +1,140 @@
|
||||
# Operators — Derived Rules
|
||||
|
||||
> Derives from `domains/gitops-operators/first-principles.md`.
|
||||
> Applies P6 (Operators Encode Domain Knowledge) primarily, with
|
||||
> P1, P4, P8, P9, P10. Cross-links `domains/kubernetes/workloads.md`
|
||||
> and `domains/kubernetes/rbac.md` for the underlying controller
|
||||
> model, and `domains/infrastructure-as-code/modules.md` for the
|
||||
> module-vs-operator boundary.
|
||||
|
||||
## What an Operator Is (P6 Operators Encode Domain Knowledge)
|
||||
|
||||
- An Operator is a Kubernetes controller that encodes human
|
||||
operational knowledge as CRDs plus a control loop. The operator
|
||||
reconciles a domain-specific resource (a database, a message
|
||||
queue, a certificate, a ML model) to a desired state.
|
||||
- The operator is the deepest expression of
|
||||
`domains/kubernetes/P1 Declarative Desired State`: the domain
|
||||
knowledge itself is the desired state. A runbook that lives only
|
||||
in a wiki is operational knowledge that has not been encoded —
|
||||
the operator is the encoding (P6).
|
||||
- An operator runs inside the cluster, observes its CRDs, and acts.
|
||||
It is a pull-based reconciler by construction — see
|
||||
`domains/gitops-operators/first-principles.md` P3.
|
||||
|
||||
## CRDs and Controllers (P2 Declarative Over Imperative, P4 Continuous Reconciliation)
|
||||
|
||||
- A CustomResourceDefinition (CRD) defines the schema of the
|
||||
domain resource. The controller watches instances of that CRD
|
||||
and reconciles current → desired (P4).
|
||||
- The CRD is the public contract of the operator. Version it
|
||||
(`v1alpha1` → `v1beta1` → `v1`) and preserve backward
|
||||
compatibility — see `domains/api/versioning.md` for the general
|
||||
API-evolution principles. A CRD is an API surface, not an
|
||||
internal type.
|
||||
|
||||
```yaml
|
||||
apiVersion: postgres.example.com/v1
|
||||
kind: PostgresCluster
|
||||
metadata:
|
||||
name: payments-db
|
||||
namespace: payments
|
||||
spec:
|
||||
replicas: 3
|
||||
version: "16"
|
||||
storage:
|
||||
size: 100Gi
|
||||
storageClass: fast-ssd
|
||||
backup:
|
||||
schedule: "0 2 * * *"
|
||||
retention: 7d
|
||||
```
|
||||
|
||||
- The controller reconciles this spec: creates StatefulSets, PVCs,
|
||||
Services, backup CronJobs. The user declares intent; the operator
|
||||
makes it so (P2, P6).
|
||||
|
||||
## The Control Loop (P4 Continuous Reconciliation, P8)
|
||||
|
||||
- The loop watches CRD instances, compares current vs desired, and
|
||||
acts to converge. Drift (a hand-deleted pod, a failed backup) is
|
||||
detected and corrected each loop (P4).
|
||||
- An operator-managed resource should not be hand-edited (P8). The
|
||||
operator owns the subordinate resources (StatefulSets, PVCs); a
|
||||
manual `kubectl edit` on a subordinate is drift the operator will
|
||||
overwrite.
|
||||
|
||||
## Operator SDK and OLM (P6 Composability, C6)
|
||||
|
||||
- The Operator SDK scaffolds a controller from a CRD (Go, Ansible,
|
||||
Helm). Use it to avoid re-implementing the controller boilerplate.
|
||||
- Operator Lifecycle Manager (OLM) installs, updates, and manages
|
||||
operators as first-class cluster components. OLM is the package
|
||||
manager for operators — the operator analogue of
|
||||
`domains/kubernetes/helm.md` for workloads.
|
||||
- An operator published via OLM is a versioned, catalog-tracked
|
||||
artifact. Pin the operator version; do not float `latest` (P5
|
||||
applies to operators as much as to manifests).
|
||||
|
||||
## When to Write an Operator vs a Helm Chart (P6, C6 Composability)
|
||||
|
||||
| Axis | Helm chart | Operator |
|
||||
|------|-----------|----------|
|
||||
| Day-2 operations | None — chart installs, you operate | Encoded — operator reconciles lifecycle (backup, resize, failover, upgrade) |
|
||||
| State | Static manifests | Live control loop watching CRDs |
|
||||
| Day-1 install | Strong fit — package and install | Overkill if install is all you need |
|
||||
| Day-2 reconcile | None — drift is manual | Continuous — drift corrected each loop |
|
||||
| Domain knowledge | Lives in runbooks + on-call | Lives in the controller code |
|
||||
| Best for | Off-the-shelf apps, stateless services, one-shot deploys | Stateful apps, complex lifecycles, day-2 automation (backup, scale, failover, version upgrades) |
|
||||
| Watch out for | Templating complexity, no day-2 reconcile | Controller complexity, multi-team maintenance burden, scope creep |
|
||||
|
||||
- Write an operator when the day-2 operations (backup, failover,
|
||||
resize, version upgrade) are non-trivial and repeated. Write a
|
||||
Helm chart when install is all you need and day-2 is run by a
|
||||
human or a separate tool.
|
||||
- Do not write an operator to wrap a Helm chart and call it day-2
|
||||
automation — that is a Helm chart with extra steps. See
|
||||
`domains/infrastructure-as-code/modules.md` for the
|
||||
module-vs-copy boundary (the operator-vs-chart boundary is its
|
||||
analogue).
|
||||
|
||||
## Scope and Responsibility Boundaries (P10 Least Privilege, C6)
|
||||
|
||||
- An operator owns one domain. An operator that manages databases
|
||||
and message queues and certificates is doing three jobs — split
|
||||
it. Scope creep is the most common operator failure mode (P6
|
||||
violation: the encoded knowledge is no longer coherent).
|
||||
- The operator's ServiceAccount must be scoped to the resources it
|
||||
manages (P10). A database operator that needs `cluster-admin` to
|
||||
create a StatefulSet has the wrong RBAC — see
|
||||
`domains/kubernetes/rbac.md` and `domains/security/authorization.md`.
|
||||
- One operator per CRD family; one ServiceAccount per operator; one
|
||||
namespace per operator (or a shared `operators` namespace with
|
||||
strict RoleBindings). Default namespace is for nothing in
|
||||
production.
|
||||
|
||||
## Failure and Observability (P9 Failure is Observable and Surfaced)
|
||||
|
||||
- An operator must surface its reconcile status on the CRD
|
||||
(`status.conditions`, `status.observedGeneration`). A CRD with no
|
||||
status is an operator that fails silently (P9).
|
||||
- Wire operator events to notifications and metrics. A failed
|
||||
backup, a stuck failover, a version-upgrade stall must emit a
|
||||
signal — see `domains/observability/metrics.md`.
|
||||
- An operator that reconciles but does not report health is a
|
||||
black box. The GitOps controller (ArgoCD/Flux) will read it as
|
||||
`Progressing` forever — write the health check (see `argocd.md`
|
||||
"Health and Status").
|
||||
|
||||
## What Violates Operator Discipline
|
||||
|
||||
| Violation | Principle |
|
||||
|-----------|-----------|
|
||||
| Operator that manages databases + queues + certs | P6 Operators Encode Domain Knowledge (scope creep) |
|
||||
| Operator ServiceAccount bound to `cluster-admin` | P10 Least Privilege Reconciliation |
|
||||
| CRD with no `status.conditions` | P9 Failure is Observable and Surfaced |
|
||||
| Operator with no health check wired to GitOps | P9, `argocd.md` Health and Status |
|
||||
| Unversioned CRD (`v1` shipped without alpha/beta) | P5, `domains/api/versioning.md` |
|
||||
| Manual `kubectl edit` on an operator-managed subordinate | P8 Reconcile, Don't Mutate by Hand |
|
||||
| Operator that wraps a Helm chart and adds no day-2 logic | P6 (no knowledge encoded) |
|
||||
| Operator runbook that exists only in a wiki | P6 Operators Encode Domain Knowledge |
|
||||
@@ -0,0 +1,177 @@
|
||||
# Progressive Delivery — Derived Rules
|
||||
|
||||
> Derives from `domains/gitops-operators/first-principles.md`.
|
||||
> Applies P7 (Progressive Delivery is Reversible by Construction)
|
||||
> primarily, with P4, P9. Cross-links `domains/devops/first-principles.md`
|
||||
> P4 Rollback First and P5 Progressive Delivery, and
|
||||
> `domains/observability/metrics.md` for the analysis signals.
|
||||
|
||||
## What Progressive Delivery Is (P7 Reversible by Construction)
|
||||
|
||||
- Progressive delivery shifts traffic in stages (canary, blue-green)
|
||||
gated by analysis (metrics, counters, error rates). Each stage is
|
||||
metric-checked; a failed gate aborts the rollout and reverts to
|
||||
the prior stable version. Promotion without a rollback path is a
|
||||
violation (P7).
|
||||
- Progressive delivery is the GitOps extension of
|
||||
`domains/devops/P5 Progressive Delivery` and
|
||||
`domains/kubernetes/P10 Roll Forward, Roll Back`. The k8s rolling
|
||||
update is the floor; progressive delivery adds metric-gated
|
||||
promotion and one-command abort.
|
||||
- Two sister projects dominate: **Argo Rollouts** (Argo ecosystem)
|
||||
and **Flagger** (Flux ecosystem). Both implement the same pattern
|
||||
— a Rollout CRD replaces a Deployment, an analysis drives the
|
||||
gates, an abort reverts traffic.
|
||||
|
||||
## The Rollout CRD (P2 Declarative Over Imperative, P7)
|
||||
|
||||
- A Rollout (Argo Rollouts) or Canary/Flag (Flagger) is a CRD that
|
||||
replaces the Deployment as the reconciled resource. It declares
|
||||
the strategy (canary, blue-green), the traffic split, and the
|
||||
analysis gates. The controller reconciles traffic and pods to
|
||||
match.
|
||||
|
||||
```yaml
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Rollout
|
||||
metadata:
|
||||
name: payments-api
|
||||
namespace: payments
|
||||
spec:
|
||||
replicas: 10
|
||||
selector:
|
||||
matchLabels:
|
||||
app: payments-api
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: payments-api
|
||||
spec:
|
||||
containers:
|
||||
- name: api
|
||||
image: registry.example.com/payments-api:1.2.3
|
||||
strategy:
|
||||
canary:
|
||||
trafficRouting:
|
||||
istio:
|
||||
virtualService:
|
||||
name: payments-vs
|
||||
routes: [primary]
|
||||
steps:
|
||||
- setWeight: 5
|
||||
- pause: { duration: 2m }
|
||||
- analysis:
|
||||
templates:
|
||||
- templateName: success-rate
|
||||
- setWeight: 25
|
||||
- pause: { duration: 5m }
|
||||
- analysis:
|
||||
templates:
|
||||
- templateName: success-rate
|
||||
- setWeight: 50
|
||||
- pause: { duration: 5m }
|
||||
- setWeight: 100
|
||||
```
|
||||
|
||||
- Each `setWeight` shifts traffic; each `pause` holds for
|
||||
observation; each `analysis` runs a metric gate. A failed
|
||||
analysis aborts the rollout and reverts traffic to the stable
|
||||
ReplicaSet (P7).
|
||||
|
||||
## Canary vs Blue-Green (P7, C3 Simplicity)
|
||||
|
||||
| Strategy | Mechanism | Cost | Best for |
|
||||
|----------|-----------|------|----------|
|
||||
| Canary | Shift a small % of traffic to the new version; increase on gate success | Low (few new pods) | Most production rollouts; metric-gated, gradual |
|
||||
| Blue-Green | Run two full environments; switch traffic all-at-once | High (2× capacity) | Schema-breaking changes, instant rollback, low-frequency deploys |
|
||||
|
||||
- Canary is the default — it is reversible by construction (P7)
|
||||
and economical (C8). Blue-green is for changes that cannot be
|
||||
partial (a breaking schema migration, a full cutover).
|
||||
- A canary with no analysis gate is a slow blue-green — it is not
|
||||
progressive delivery. The gate is what makes it progressive (P7).
|
||||
|
||||
## Analysis Templates (P9 Failure is Observable and Surfaced, P7)
|
||||
|
||||
- An AnalysisTemplate declares the metric query, the success
|
||||
threshold, and the count of samples. The rollout controller runs
|
||||
the analysis at each gate; a failed analysis aborts the rollout.
|
||||
|
||||
```yaml
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: AnalysisTemplate
|
||||
metadata:
|
||||
name: success-rate
|
||||
namespace: payments
|
||||
spec:
|
||||
metrics:
|
||||
- name: success-rate
|
||||
interval: 1m
|
||||
successCondition: result[0] >= 0.99
|
||||
failureLimit: 2
|
||||
provider:
|
||||
prometheus:
|
||||
address: http://prometheus.observability:9090
|
||||
query: |
|
||||
sum(rate(http_requests_total{job="payments-api",code!~"5.."}[2m]))
|
||||
/
|
||||
sum(rate(http_requests_total{job="payments-api"}[2m]))
|
||||
```
|
||||
|
||||
- `successCondition` is the gate; `failureLimit` is the tolerance
|
||||
for transient blips. A single failed sample aborts immediately if
|
||||
`failureLimit: 0`; tolerate noise with `failureLimit: 2`.
|
||||
- The metric is the abort signal — see `domains/observability/metrics.md`
|
||||
for the SLI/SLO discipline that makes the gate meaningful. A gate
|
||||
on an undefined SLO is a gate on noise.
|
||||
|
||||
## Argo Rollouts vs Flagger (P6 Composability, P7)
|
||||
|
||||
| Axis | Argo Rollouts | Flagger |
|
||||
|------|---------------|---------|
|
||||
| Ecosystem | Argo (ArgoCD sister project) | Flux (Flux sister project) |
|
||||
| CRD | `Rollout` (replaces `Deployment`) | `Canary` / `Flag` (wraps a `Deployment`) |
|
||||
| Traffic providers | Istio, NGINX, ALB, SMI, Traefik, Ambassador | Istio, NGINX, Linkerd, SMI, App Mesh, Gloo, Contour |
|
||||
| Analysis sources | Prometheus, Datadog, Wavefront, NewRelic, CloudWatch, Graphite, Kayenta | Prometheus, Datadog, CloudWatch, Stackdriver, Elasticsearch, Graphite |
|
||||
| Integration | Tight with ArgoCD (UI shows rollout) | Tight with Flux (events via notification-controller) |
|
||||
| Learning curve | Rollout CRD replaces Deployment (migration cost) | Wraps existing Deployment (lower migration cost) |
|
||||
| Best for | ArgoCD shops wanting rollout in the Argo UI | Flux shops wanting progressive delivery with minimal migration |
|
||||
|
||||
- Both implement the same pattern. The choice follows your GitOps
|
||||
controller — Argo Rollouts with ArgoCD, Flagger with Flux. Mixing
|
||||
is possible but not idiomatic.
|
||||
|
||||
## Abort and Rollback (P7 Reversible by Construction, P5)
|
||||
|
||||
- An abort reverts traffic to the stable ReplicaSet immediately. A
|
||||
rollout without a tested abort is a prototype (P7).
|
||||
- The abort must be one-command (or one-gate-failure). A
|
||||
progressive delivery that requires manual rollback steps has
|
||||
lost the "reversible by construction" property — it is a deploy
|
||||
with extra steps.
|
||||
- Test the abort path in staging. An abort that has never been
|
||||
exercised will fail when you need it most — see
|
||||
`domains/devops/first-principles.md` P4 Rollback First.
|
||||
|
||||
## Observability (P9 Failure is Observable and Surfaced)
|
||||
|
||||
- Progressive delivery is only as good as its metrics. A rollout
|
||||
gated on a metric that is not tracked is ungated — the gate is
|
||||
theater (P9).
|
||||
- Wire rollout status (phase, weight, analysis result) to
|
||||
notifications and dashboards. A stalled rollout with no signal is
|
||||
silent drift (P9). See `domains/observability/metrics.md`.
|
||||
- Cross-link `domains/kubernetes/workloads.md` for the underlying
|
||||
Deployment/ReplicaSet model that progressive delivery replaces.
|
||||
|
||||
## What Violates Progressive Delivery Discipline
|
||||
|
||||
| Violation | Principle |
|
||||
|-----------|-----------|
|
||||
| Canary with no analysis gate | P7 Progressive Delivery is Reversible by Construction |
|
||||
| Rollout with no tested abort path | P7, `domains/devops/P4 Rollback First` |
|
||||
| Analysis gate on an undefined SLO | P9 Failure is Observable and Surfaced |
|
||||
| Blue-green with no 2× capacity budget | C8 Economy (blue-green is a cost decision) |
|
||||
| Rollout stalled with no notification | P9 Failure is Observable and Surfaced |
|
||||
| Manual `kubectl` traffic shift on a Rollout-managed service | P8 Reconcile, Don't Mutate by Hand |
|
||||
| `failureLimit: 0` on a noisy metric (constant false aborts) | P4 Continuous Reconciliation (gate noise tolerance) |
|
||||
Reference in New Issue
Block a user