coreci/atelier
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 28 Wiki Activity

docs(P5): complete examples + cross-links phase

Browse Source 
---ci---
project: atelier
phase: 5
milestone: v0.3
status: complete
requirements:
  covered: [ATELIER-86, ATELIER-87, ATELIER-88]
  partial: []
---/ci---
This commit is contained in:
Jon Chery
2026-08-05 03:40:37 +00:00
parent 648fcb5d85
commit 4c8700e5a4
5 changed files with 839 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
examples/good/ai-ml-reproducibility.md
+226
View File
@@ -0,0 +1,226 @@
# Good Example: AI/ML Reproducible Training Run
> A training run that follows Atelier's AI/ML principles. Each aspect
> cites the principle it satisfies. Scope per D-023: this is
> engineering discipline (reproducibility, versioning, lineage,
> serving), **not** algorithm or model design — no architecture
> choice, hyperparameter tuning, or model-family comparison appears
> here.
## The Run
A training run `2026-08-05T09:12:00Z#run-42` produces model
`registry/payments-fraud@sha256:b5e1...aa0`. Every input that shaped
the model is pinned, named, and recoverable; the eval was declared
before training; the model is an addressed artifact in a registry;
the rollback path names the prior model and the prior dataset.
### The Reproducibility Contract
```yaml
# lineage/run-42.yaml — the lineage root, committed alongside the code
run_id: 2026-08-05T09:12:00Z#run-42
dataset: s3://ml-data/train@sha256:7f3a...e21
splits: dvc.yaml@commit a1b2c4d
code: git@a1b2c4d
config: configs/train.yaml@commit a1b2c4d
environment: ghcr.io/org/train-img@sha256:9c2d...f88
eval_spec: configs/eval.yaml@commit a1b2c4d
model_digest: registry/payments-fraud@sha256:b5e1...aa0
status: passed # eval gate passed -> eligible for promotion
```
- Lose any line and the run is anecdote, not evidence. The record is
the lineage root: a prediction cites the `model_digest`, which
cites this `run_id`, which cites everything above.
### Data is Versioned (DVC, content-hashed)
```ini
# dvc.yaml — the split config is versioned in git, the data in the
# content-addressed object store. Both are pinned by commit + hash.
stages:
prepare:
cmd: python src/prepare.py --input data/raw --out data/splits
deps:
- data/raw
- src/prepare.py
outs:
- data/splits/train.parquet
- data/splits/val.parquet
- data/splits/test.parquet
# The dataset hash (sha256:7f3a...e21) is recorded in the lineage
# contract above. "s3://ml-data/latest" would be a P2 violation.
```
```bash
# The dataset is pinned by content hash, not by a mutable path.
$ dvc get s3://ml-data/train --rev sha256:7f3a...e21
# The split is a deterministic function of (dataset version, split
# config, random seed). Two runs on the same pinned inputs produce
# the same splits.
```
### Code and Config are Versioned (git)
```yaml
# configs/train.yaml@commit a1b2c4d — versioned with the code
# (No algorithm/hyperparameter content is illustrated here — this is
# the engineering discipline of pinning the config, not the model
# design inside it. Per D-023, algorithm choice is out of scope.)
seed: 42
splits:
train: data/splits/train.parquet
val: data/splits/val.parquet
test: data/splits/test.parquet # held out, never touched by training
```
### Environment is Pinned (container digest)
```dockerfile
# The training environment is an image addressed by digest, not :latest.
# ghcr.io/org/train-img@sha256:9c2d...f88
FROM python:3.11-slim
# dependencies pinned in requirements.txt with hashes
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
```
```text
# requirements.txt — pinned + hash-pinned (pip-compile / pip-audit)
dvc==3.50.2 \
--hash=sha256:1c8a...e7
mlflow==2.16.0 \
--hash=sha256:9b2f...a1
# No unpinned ranges. A rerun pulls the exact same wheels.
```
### Evaluation is Defined Before Training (P4)
```yaml
# configs/eval.yaml@commit a1b2c4d — committed BEFORE training runs.
# The metrics, splits, and pass/fail thresholds are a-priori; they
# are the contract the model must satisfy to leave the experiment.
metrics:
- name: precision_at_threshold
threshold: ">= 0.92"
- name: recall_at_threshold
threshold: ">= 0.85"
- name: false_positive_rate
threshold: "<= 0.03"
split: data/splits/test.parquet # held out, never in training
gate: all_metrics_pass # AND of all thresholds; no cherry-pick
# The eval schema equals the serving input contract (serving.md P8):
# feature names, types, ranges match the production boundary exactly.
```
- Metrics chosen after seeing scores would be a P4 violation: the eval
would be rationalizing, not measuring. See
`domains/ai-ml/model-evaluation.md`.
### The Model is a Versioned Artifact (MLflow registry)
```bash
# After the eval gate passes, the model is registered as an immutable
# artifact addressed by digest, then promoted by stage.
$ mlflow models register \
--name payments-fraud \
--model-uri runs:/run-42/model \
--description "run-42, dataset sha256:7f3a...e21, eval passed"
# registry/payments-fraud@sha256:b5e1...aa0
# Stages: None -> Staging -> Production. Promotion is a registry
# operation, not a file copy. Never "latest".
```
### The Pipeline Composes (P9)
```text
# The training flow is a pipeline with explicit stages and contracts,
# not a notebook. Each stage has named inputs and named outputs.
prepare(dataset@hash) -> split(dvc.yaml) -> train(config, env@digest)
-> eval(eval.yaml, test@hash) -> [gate: pass] -> register(model@digest)
|
+-> [gate: fail] -> abort, no promote
# A notebook in this path would be a P9 violation: implicit state,
# human-dependent order, unreproducible.
```
## What Makes It Good
### Reproducibility is First Class (AI/ML P1, C1, C5)
- data + code + config + environment are all pinned. A second
engineer on a second laptop checks out commit `a1b2c4d`, pulls the
dataset by hash, pulls the image by digest, and reproduces the run
bit-for-bit. The run is reviewable because it is recreatable.
- See `domains/ai-ml/first-principles.md` P1 and
`domains/devops/first-principles.md` P1 Reproducibility.
### Data is Versioned, Not Just Code (AI/ML P2, C5, C7)
- The dataset is `s3://ml-data/train@sha256:7f3a...e21`, not
`s3://ml-data/latest`. A model trained on "the data" is a model
trained on an unknown input — a C1 violation. DVC pins the data the
way git pins the code.
- See `domains/ai-ml/data-versioning.md` (dataset hashing, the DVC /
Delta Lake / LakeFS comparison) and `domains/data/migrations.md`.
### Lineage is Traceable End-to-End (AI/ML P3, C7, C1)
- prediction → model → run-42 → dataset → source. Every edge is
named; no orphan model. A serving regression traces back to the
exact dataset and code that built the model, which is how drift is
diagnosed (data drift vs concept drift vs prediction drift).
- See `domains/ai-ml/data-versioning.md` (lineage record) and
`domains/observability/logging.md`.
### Evaluation Defined Before Training (AI/ML P4, C1, C2)
- `eval.yaml` was committed before `train` ran. The gate is
`all_metrics_pass`; a failing metric aborts promotion. Cherry-
picking a metric post-hoc is a correctness violation — the eval
would no longer measure the model.
- See `domains/ai-ml/model-evaluation.md` (eval-as-a-gate) and
`domains/testing/first-principles.md` (tests as specification).
### Models are Versioned Artifacts (AI/ML P5, C5, C6)
- The model is `registry/payments-fraud@sha256:b5e1...aa0`, promoted
Staging → Production. A serving endpoint that pulled `latest` would
be serving an unknown model with no rollback. The registry is to
models what a container registry is to images.
- See `domains/ai-ml/serving.md` (the model is an addressed artifact)
and `domains/devops/first-principles.md` P7 Immutability.
### Rollback Includes the Model (AI/ML P10, C5)
- If production regresses, the rollback restores the prior model
digest `registry/payments-fraud@sha256:a1c4...f09` AND the prior
serving code. A rollback that redeploys old code but keeps the new
model has not rolled back — the model was the thing that regressed.
- See `domains/ai-ml/serving.md` (Rollback Includes the Model) and
`domains/devops/first-principles.md` P4 Rollback First.
## What This Example Does NOT Do (And Why That's Good)
- Does **not** reference the dataset by a mutable path —
`s3://ml-data/latest` would be a P2 violation.
- Does **not** choose metrics after seeing scores — that is a P4
violation (rationalizing, not measuring).
- Does **not** pull `latest` from the model registry — that is a P5
violation (unknown model, no rollback).
- Does **not** contain algorithm/architecture/hyperparameter content
— per D-023, those are research choices, not engineering
principles, and have no derivation in the core C-rules.
- Does **not** run from a notebook — a notebook in the pipeline path
is a P9 violation (implicit state, unreproducible).
## Cross-Domain Links
- `domains/ai-ml/data-versioning.md` — the DVC pinning, the lineage
record, the tool comparison (DVC / Delta Lake / LakeFS).
- `domains/ai-ml/serving.md` — the model is promoted as an addressed
artifact; the serving boundary validates inputs against the same
schema as the eval.
- `domains/ai-ml/model-evaluation.md` — the eval-as-a-gate that this
run must pass before promotion.
- `domains/devops/first-principles.md` P1 Reproducibility — the
non-negotiable this run inherits.
- `domains/data/migrations.md` — data versioning parallels schema
migration discipline.
- `domains/observability/logging.md` — the lineage record is a
structured, append-only log of provenance.
examples/good/gitops-pr.md
+197
View File
@@ -0,0 +1,197 @@
# Good Example: GitOps Pull Request
> A pull request that changes ArgoCD Application manifests following
> Atelier's GitOps + Operators principles. Each aspect cites the
> principle it satisfies.
## The PR
A PR titled `promote payments-api 1.2.3 to prod` opened against the
GitOps repo `platform/gitops`. It changes the `targetRevision` of the
payments-api Application from `1.2.2` to `1.2.3`, adds a sync-wave
annotation to a new migration Job, and tightens the AppProject's
destination allow-list. CI runs plan/diff; nothing pushes to the
cluster.
### The Commit
```yaml
# manifests/prod/payments-api.yaml — the only file changed
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: payments-api
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
source:
repoURL: https://git.example.com/platform/payments
targetRevision: 1.2.3 # was 1.2.2 — pinned, not latest
path: manifests/prod
destination:
server: https://kubernetes.default.svc
namespace: payments
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=false
- PrunePropagationPolicy=foreground
```
```yaml
# manifests/prod/payments-db-migration.yaml — new file, wave-ordered
apiVersion: batch/v1
kind: Job
metadata:
name: payments-db-migrate-1.2.3
namespace: payments
annotations:
argocd.argoproj.io/sync-wave: "-1" # PreSync: run before the app
spec:
backoffLimit: 0
ttlSecondsAfterFinished: 86400
template:
spec:
restartPolicy: OnFailure
serviceAccountName: payments-migrator
containers:
- name: migrate
image: registry.example.com/payments/migrate:1.2.3
args: ["up", "--to", "1.2.3"]
```
```yaml
# projects/payments.yaml — AppProject tightened in the same PR
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: payments
namespace: argocd
spec:
sourceRepos:
- https://git.example.com/platform/payments
destinations:
- server: https://kubernetes.default.svc
namespace: payments # prod-app can only deploy to payments ns
clusterResourceWhitelist:
- group: ""
kind: Namespace # allowed to create its own namespace
roles:
- name: payments-team
policies:
- p, proj:payments:payments-team, applications, sync, payments/*, allow
```
### The CI Pipeline (runs on the PR, before merge)
```text
# .github/workflows/gitops-plan.yml (illustrative steps)
- name: validate manifests
run: argocd app manifests manifests/prod/ | kubeconform -strict
- name: diff against live cluster (read-only, no apply)
run: argocd app diff payments-api --server $ARGOCD_SERVER --auth-token $READ_ONLY_TOKEN
# CI holds a READ-ONLY ArgoCD token. It never holds kubectl rights.
# A non-empty diff is the PR's proposed change, rendered for review.
- name: opa gate (admission policy pre-check)
run: opa eval -i manifests/prod/ -d policies/ "data.k8s.admission.deny"
# Policy violations fail the PR before merge, not after deploy.
```
## What Makes It Good
### Git is the Source of Truth (GitOps P1, C1 Correctness)
- The promotion is a commit. The cluster's desired state is a
derivative of this repo; the repo is the authority. If the change is
wrong, `git revert` is the rollback — the recovery path is the
history.
- See `domains/gitops-operators/first-principles.md` P1 and
`domains/gitops-operators/argocd.md` (Application CRD).
### Pull, Don't Push (GitOps P3, C4 Locality)
- CI holds a **read-only** ArgoCD token for `app diff`. It holds no
`kubectl` rights against the production cluster. The cluster's
ArgoCD controller pulls the merged commit; nothing pushes to the
cluster. A compromised CI token can read, not deploy.
- See `domains/gitops-operators/argocd.md` (RBAC and SSO) and
`domains/gitops-operators/flux.md` for the same pull boundary from
the Flux side.
### State is Immutable and Versioned (GitOps P5, C5 Reversibility)
- `targetRevision: 1.2.3` — the Application pins a specific chart
revision, not `latest`. The commit that changed it is a permanent
record; `git revert` restores `1.2.2` and ArgoCD's `selfHeal`
converges the cluster back. No force-push; history is the audit
trail.
- See `domains/gitops-operators/first-principles.md` P5 and
`domains/infrastructure-as-code/state.md` (State is Truth).
### Sync Waves Order Correctness (GitOps P4, C1)
- The migration Job carries `argocd.argoproj.io/sync-wave: "-1"` so
it runs in `PreSync` before the payments-api Deployment that
depends on the new schema. Wave ordering is a correctness
mechanism, not performance — the app starting before its migration
is a correctness bug.
- See `domains/gitops-operators/argocd.md` (Sync Waves and Hooks).
### Reconcile, Don't Mutate by Hand (GitOps P8)
- `selfHeal: true` + `prune: true` means a hand-edited drift on a
managed resource is overwritten on the next loop. The fix for drift
is a new commit, not `kubectl edit`. The PR author does not SSH into
the cluster to "fix" anything.
- See `domains/gitops-operators/argocd.md` (Diff and Drift) and
`domains/gitops-operators/first-principles.md` P8.
### Least Privilege Reconciliation (GitOps P10, C8 Economy)
- The AppProject `payments` restricts the Application to the
`payments` namespace and the `payments` repo. The controller's
ServiceAccount (not shown) is bound to a namespace-scoped Role, not
`cluster-admin`. The PR *tightens* the allow-list — least privilege
is a direction, not a one-time setting.
- See `domains/gitops-operators/argocd.md` (RBAC and SSO) and
`domains/kubernetes/rbac.md`.
### Policy is a Gate (Compliance P5, cross-link)
- The `opa eval` step runs the admission policy against the proposed
manifests before merge. A violation fails the PR; the non-compliant
state is never realized. Detection is not enforcement; this is
enforcement.
- See `domains/compliance/policy-as-code.md` and
`domains/devops/ci-cd.md`.
### Failure is Observable (GitOps P9)
- A sync failure or health degradation on `payments-api` emits
ArgoCD status (`Degraded` / `OutOfSync`) and a notification. Silent
drift is the bug; this PR does not disable notifications.
- See `domains/gitops-operators/argocd.md` (Health and Status) and
`domains/observability/metrics.md`.
## What This PR Does NOT Do (And Why That's Good)
- Does **not** run `kubectl apply` from CI — that is the push pattern,
a P3 violation (see `examples/bad/` for the anti-pattern).
- Does **not** use `argocd app set` as the steady state — the change
is in git, not in an imperative command's history.
- Does **not** store raw Secrets in the GitOps repo — secrets arrive
via Sealed Secrets / SOPS / External Secrets, encrypted in git.
- Does **not** float `targetRevision: latest` — the Application pins
a version; "latest" is an unknown model of the system.
## Cross-Domain Links
- `domains/gitops-operators/argocd.md` — the Application CRD, sync
waves, RBAC/AppProjects, and the pull model.
- `domains/gitops-operators/flux.md` — the same PR pattern from the
Flux side (Kustomization CRD, per-cluster autonomy).
- `domains/kubernetes/workloads.md` — the Deployment/Job the
Application reconciles.
- `domains/kubernetes/rbac.md` — the ServiceAccount + Role the
controller and the migration Job run as.
- `domains/compliance/policy-as-code.md` — the OPA gate is a
compliance-as-a-gate enforcement point.
- `domains/devops/P4 Rollback First` — `git revert` is the rollback;
`selfHeal` is the convergence.