diff --git a/domains/gitops-operators/argocd.md b/domains/gitops-operators/argocd.md new file mode 100644 index 0000000..a5afd5b --- /dev/null +++ b/domains/gitops-operators/argocd.md @@ -0,0 +1,176 @@ +# ArgoCD — Derived Rules + +> Derives from `domains/gitops-operators/first-principles.md`. +> Applies P1–P10 to ArgoCD specifically. For the ArgoCD-vs-Flux +> decision, see the decision matrix at the end of this doc and in +> `flux.md`. + +## What ArgoCD Is (P1 Git is the Source of Truth, P3 Pull, Don't Push) + +- ArgoCD is a pull-based GitOps controller for Kubernetes. It runs + inside the target cluster, pulls desired state from git, and + reconciles the cluster to match. CI never holds `kubectl` rights + against the cluster (P3). +- An Application is a declarative binding of "this git path" to + "this cluster destination." The Application CRD is the unit of + reconciliation. The cluster is a derivative of git, never the + authority (P1). +- ArgoCD supports Helm charts, Kustomize overlays, ksonnet, and raw + manifests as source formats — see `domains/kubernetes/helm.md` + and `domains/kubernetes/kustomize.md`. + +## Application CRD (P2 Declarative Over Imperative, P4 Continuous Reconciliation) + +- An Application declares `source` (repo, path, revision, chart), + `destination` (server, namespace), and `syncPolicy`. The + reconciler loops continuously; drift is corrected automatically, + not on-demand (P4). + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: payments-api + namespace: argocd +spec: + source: + repoURL: https://git.example.com/platform/payments + targetRevision: 1.2.3 + path: manifests/prod + destination: + server: https://kubernetes.default.svc + namespace: payments + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=false +``` + +- `automated.prune: true` deletes resources removed from git. + `selfHeal: true` corrects hand-edited drift back to git (P8). + Disable both for workloads that need manual approval gates. + +## App-of-Apps (P6 Operators Encode Domain Knowledge, C6 Composability) + +- The App-of-Apps pattern: one root Application points at a git + directory of child Application manifests. The root app reconciles + the children; the children reconcile the workloads. This is the + ArgoCD expression of composition — a fleet of apps as a tree of + Applications. +- Use App-of-Apps for cluster bootstrapping (one repo, many + clusters, many apps). Do not use it as a substitute for a package + manager; if you are templating hundreds of near-identical + Applications, use a generator (ApplicationSet) instead. + +## Sync Waves and Hooks (P4 Continuous Reconciliation, P7 Reversibility) + +- Sync waves order resources within a sync: `PreSync` → `Sync` → + `PostSync`. Use waves to run a job before a Deployment, or a + migration before the app that depends on it. +- Sync hooks (`PreSync`, `Sync`, `PostSync`, `SyncFail`) are + Resources annotated to execute at a wave boundary. A `SyncFail` + hook runs on sync failure — the abort path (P7). +- Wave ordering is a correctness mechanism, not a performance one. + Mis-ordered waves (e.g., app starts before its migration job) + are a correctness bug. + +## Health and Status (P9 Failure is Observable and Surfaced) + +- ArgoCD assesses every resource's health (`Healthy`, `Progressing`, + `Degraded`, `Missing`, `Suspended`) and surfaces the aggregate as + Application status. Sync status (`Synced`, `OutOfSync`) reports + drift against git. +- Health checks are pluggable via Lua scripts for custom CRDs. An + Operator-managed CRD without a health check reads as `Progressing` + forever — write one (see `operators.md`). +- Out-of-sync or degraded status must emit a notification (Slack, + PagerDuty, webhook). Silent drift is the bug (P9). Wire status to + `domains/observability/metrics.md`. + +## Diff and Drift (P8 Reconcile, Don't Mutate by Hand, P4) + +- `argocd app diff` shows the diff between git and live cluster. + A non-empty diff on a synced app is hand-edit drift — the + recovery is `selfHeal`, not a manual `kubectl apply` (P8). +- Drift detection runs continuously (P4). The gap between "git + changed" and "cluster matches git" is observable, not assumed. + +## RBAC and SSO (P10 Least Privilege Reconciliation) + +- ArgoCD's own RBAC governs who can view, sync, and admin + Applications. Bind to SSO (OIDC, SAML) for human identity; bind + the controller's service account to a Role scoped to the + namespaces it reconciles. +- The controller's credentials must not be `cluster-admin` (P10). + Use namespace-scoped Roles via `ApplicationSet` namespaces or + cluster-wide AppProject restrictions. See + `domains/kubernetes/rbac.md` and `domains/security/authorization.md`. +- AppProjects bound the blast radius of what an Application can + deploy (allowed repos, destinations, roles). One AppProject per + team or environment; the default project is for nothing in + production. + +## Multi-Cluster (P4 Locality, P10) + +- ArgoCD registers external clusters by secret. The controller + pulls from git and pushes to the registered cluster's API server. + The "pull, don't push" boundary (P3) is between the target + cluster's reconciler and CI — the controller-to-apiserver hop is + internal to the platform. +- Scope each registered cluster's credentials to the namespaces + ArgoCD manages there. Do not register a cluster with cluster-admin + and call it done (P10). + +## Sync Windows (P5 Reversibility, P7) + +- Sync windows restrict when automated sync runs (e.g., no syncs + during business hours, or syncs only in a maintenance window). + They are a reversibility mechanism: a bad commit lands in git, + but the sync window holds it until review. +- Sync windows do not replace health monitoring (P9). A degraded + app inside a window is still an incident. + +## Secrets (P10, cross-link security/secrets) + +- Do not store raw Secrets in the GitOps repo. Use a sealed-secret + controller (Bitnami Sealed Secrets, SOPS, External Secrets + Operator) so the git store holds encrypted material only. See + `domains/security/secrets.md` for the general secret-hygiene + principles. + +## ArgoCD vs Flux — Decision Matrix (IDEATE-21, D-039) + +| Axis | ArgoCD | Flux | +|------|--------|------| +| Architecture | Monolithic controller + Application CRD | Composable GitOps Toolkit controllers (source, kustomize, helm, notification) | +| Reconciliation unit | Application (one CRD per app) | Kustomization / HelmRelease (one per deploy unit) | +| UI | Web UI + CLI (full dashboard, tree view, diff viewer) | CLI-first; UI via Weave GitOps or FluxUI (add-on) | +| Sync model | Periodic poll or webhook; sync waves + hooks | Poll + webhook; runs continuously, no explicit sync waves | +| Multi-cluster | One ArgoCD manages many clusters (hub-and-spoke) | One Flux per cluster (per-cluster autonomy) | +| Templating in repo | Helm, Kustomize, ksonnet, raw manifests, Jsonnet | Helm, Kustomize, raw manifests | +| RBAC | Built-in RBAC + SSO + AppProjects | Kubernetes RBAC (no built-in RBAC layer) | +| Progressive delivery | Argo Rollouts (sister project, tight integration) | Flagger (sister project, tight integration) | +| Best for | Teams wanting a UI, multi-cluster from one pane, App-of-Apps bootstrapping | Teams wanting composable controllers, per-cluster autonomy, minimal footprint | +| Watch out for | Monolithic controller scaling, UI as ops crutch, AppProject sprawl | No native UI, steeper learning curve, manual multi-cluster orchestration | + +- Use ArgoCD when you want a UI, central multi-cluster management, + and sync-wave ordering. Use Flux when you want composable + controllers, per-cluster autonomy, and a minimal footprint. +- Both are CNCF graduated and both implement the OpenGitOps + principles. The choice is architectural fit, not correctness. See + `flux.md` for the Flux-side perspective. + +## What Violates ArgoCD Discipline + +| Violation | Principle | +|-----------|-----------| +| CI pipeline with `kubectl` rights pushing to the cluster | P3 Pull, Don't Push | +| `argocd app set` used as the steady state instead of git | P1 Git is the Source of Truth | +| `selfHeal: false` on a prod app with no manual gate | P8 Reconcile, Don't Mutate by Hand | +| Controller ServiceAccount bound to `cluster-admin` | P10 Least Privilege Reconciliation | +| Sync failure with no notification wired | P9 Failure is Observable and Surfaced | +| AppProject with no destination restrictions in prod | P10 Least Privilege Reconciliation | +| Raw Secret in the GitOps repo | P10, `domains/security/secrets.md` | +| Manual `kubectl edit` on an ArgoCD-managed resource | P8 Reconcile, Don't Mutate by Hand | \ No newline at end of file diff --git a/domains/gitops-operators/first-principles.md b/domains/gitops-operators/first-principles.md new file mode 100644 index 0000000..9dce538 --- /dev/null +++ b/domains/gitops-operators/first-principles.md @@ -0,0 +1,131 @@ +# GitOps + Operators — First Principles + +## 1. The Principles + +### P1. Git is the Source of Truth +Desired state lives in a versioned, immutable git store. The +cluster is a derivative of git, never the authority. If a state +exists only in the cluster and not in git, it is drift, not truth. +The commit history is the audit trail and the rollback path. + +### P2. Declarative Over Imperative +Express the desired cluster state, not the commands to reach it. +A manifest says what should exist; the reconciler makes it so. +Imperative `kubectl` is for inspection and incident response, not +for the steady state. This is the GitOps expression of +`domains/kubernetes/P1 Declarative Desired State` and +`domains/infrastructure-as-code/P1 Declarative Intent`. + +### P3. Pull, Don't Push +Agents running inside the target pull desired state from git; the +target never accepts outside push credentials. No CI pipeline holds +`kubectl` rights against the production cluster. The cluster reaches +out to git, not the other way around. This is the security primitive +of GitOps: the blast radius of a compromised CI is bounded by what CI +can push, and a pull model gives CI nothing to push. + +### P4. Continuous Reconciliation +The reconciliation loop is the primitive. Drift is detected and +corrected automatically, not on-demand. A manual `apply` is an +exception, not the workflow. The loop runs continuously; the gap +between "git changed" and "cluster matches git" is measured in +seconds, not tickets. + +### P5. State is Immutable and Versioned +Every change to desired state is a commit. History is the audit +trail and the rollback path. A revert is a rollback; a force-push is +history deletion. The git store is treated like +`domains/infrastructure-as-code/P3 State is Truth` — lose it or +tamper with it, and you lose the ability to reason about the system. + +### P6. Operators Encode Domain Knowledge +Operational expertise lives as CRDs plus controllers, not as +runbooks that humans must remember. An operator is a control loop +that encodes how to reconcile a specific domain (a database, a +message queue, a certificate). The operator is the deepest +expression of `domains/kubernetes/P1 Declarative Desired State` — +the domain knowledge is the desired state. + +### P7. Progressive Delivery is Reversible by Construction +Canary and blue-green are staged, metric-gated, and one-command +abortable. Promotion without a rollback path is a violation. A +rollout that cannot be aborted is a deploy, not a progressive +delivery. This is the GitOps extension of +`domains/devops/P5 Progressive Delivery` and +`domains/kubernetes/P10 Roll Forward, Roll Back`. + +### P8. Reconcile, Don't Mutate by Hand +Manual `kubectl apply` or `kubectl edit` on a GitOps-managed +resource is an incident. The reconciler will overwrite the hand +edit on the next loop; the hand edit was never truth. Drift back to +git is the recovery, not the failure. This is the GitOps angle on +`domains/infrastructure-as-code/P9 Drift is Recoverable`. + +### P9. Failure is Observable and Surfaced +Sync failures, health degradation, and rollout-stall events emit +status and notifications. Silent drift is the bug. A GitOps +controller that fails to sync without surfacing the failure has +violated the contract — you cannot fix what you cannot see +(`domains/observability/metrics.md`). + +### P10. Least Privilege Reconciliation +The controller's credentials are scoped to the namespaces and +resources it reconciles. No `cluster-admin` GitOps robots. One +credential set per boundary; the reconciler sees only what it +reconciles. This is the GitOps angle on +`domains/kubernetes/P7 RBAC by Intent, Not Identity` and +`domains/security/authorization.md`. + +## 2. Core Principle Trace + +Each GitOps + Operators P-rule derives from one or more core +C-rules (C1–C8). The matrix extension lands in P4 of the v0.3 +plan; the traces below are authoritative. + +| P-rule | Core | Why | +|--------|------|-----| +| P1 Git is the Source of Truth | C1, C5 | Correctness of state; reversibility via history | +| P2 Declarative Over Imperative | C2, C3 | Clarity of intent; simplicity of mental model | +| P3 Pull, Don't Push | C1, C4 | Correctness via security; locality of credentials | +| P4 Continuous Reconciliation | C7, C1 | Observability of drift; correctness of convergence | +| P5 State is Immutable and Versioned | C5 | Reversibility via version history | +| P6 Operators Encode Domain Knowledge | C6, C2 | Composability of expertise; clarity of operational intent | +| P7 Progressive Delivery is Reversible | C5, C1 | Reversibility of promotion; correctness of abort | +| P8 Reconcile, Don't Mutate by Hand | C1, C7 | Correctness of single source; observability of drift | +| P9 Failure is Observable and Surfaced | C7 | Observability of reconciliation | +| P10 Least Privilege Reconciliation | C1, C8 | Correctness via security; economy of trust | + +## 3. What Violates These Principles + +| Violation | Principle Breached | +|-----------|-------------------| +| CI pipeline pushes manifests to the cluster | P3 Pull, Don't Push | +| A resource exists in the cluster but not in git | P1 Git is the Source of Truth | +| `kubectl edit` on a GitOps-managed resource | P8 Reconcile, Don't Mutate by Hand | +| Reconciler with `cluster-admin` ClusterRoleBinding | P10 Least Privilege Reconciliation | +| Sync failure with no status or notification | P9 Failure is Observable and Surfaced | +| Canary with no abort/rollback path | P7 Progressive Delivery is Reversible | +| Operator runbook that exists only in a wiki | P6 Operators Encode Domain Knowledge | +| Reconciler that applies on a cron, not continuously | P4 Continuous Reconciliation | +| Force-push rewrites GitOps repo history | P5 State is Immutable and Versioned | +| Imperative deploy script as the steady state | P2 Declarative Over Imperative | + +## 4. Relationship to Other Domains + +GitOps + Operators is the deployment-automation layer above +`domains/kubernetes/` and `domains/infrastructure-as-code/`. It +borrows their declarative-reconciliation model and adds the +git-as-source-of-truth and pull-based credential boundaries. Cross +links are one-directional (per D-026 extended): + +- `domains/kubernetes/P1 Declarative Desired State` ← P2 +- `domains/kubernetes/P10 Roll Forward, Roll Back` ← P7 +- `domains/infrastructure-as-code/P1 Declarative Intent` ← P2 +- `domains/infrastructure-as-code/P3 State is Truth` ← P1, P5 +- `domains/infrastructure-as-code/P9 Drift is Recoverable` ← P4, P8 +- `domains/devops/P4 Rollback First` ← P5, P7 +- `domains/devops/P5 Progressive Delivery` ← P7 +- `domains/devops/P6 Configuration as Code` ← P1, P2 +- `domains/security/secrets.md` ← P3, P10 (reconciliation credentials) +- `domains/security/supply-chain.md` ← P5 (signed, immutable provenance) +- `domains/observability/metrics.md` ← P4, P9 (reconciliation + rollout metrics) \ No newline at end of file diff --git a/domains/gitops-operators/flux.md b/domains/gitops-operators/flux.md new file mode 100644 index 0000000..3d9c916 --- /dev/null +++ b/domains/gitops-operators/flux.md @@ -0,0 +1,159 @@ +# Flux — Derived Rules + +> Derives from `domains/gitops-operators/first-principles.md`. +> Applies P1–P10 to Flux specifically. For the ArgoCD-vs-Flux +> decision, see the decision matrix at the end of this doc and in +> `argocd.md`. + +## What Flux Is (P1 Git is the Source of Truth, P3 Pull, Don't Push) + +- Flux is a set of composable controllers — the GitOps Toolkit — + that run inside the target cluster, pull desired state from git + or OCI registries, and reconcile the cluster to match. CI never + holds `kubectl` rights against the cluster (P3). +- The composable-controller architecture is a C6 (Composability) + exemplar: each controller does one thing (source, kustomize, helm, + notification) and the controllers compose into a full GitOps + system. +- Flux supports Helm releases, Kustomize overlays, and raw + manifests — see `domains/kubernetes/helm.md` and + `domains/kubernetes/kustomize.md`. + +## GitOps Toolkit Controllers (P6 Composability, P4 Continuous Reconciliation) + +- **source-controller** — pulls git, Helm, OCI, and bucket sources; + emits artifacts (tarballs) with a digest. The source is the + pinned input to reconciliation (P5 versioning by digest). +- **kustomize-controller** — reconciles Kustomization CRDs against + the artifacts from source-controller. Runs continuously (P4). +- **helm-controller** — reconciles HelmRelease CRDs against Helm + charts from source-controller. +- **notification-controller** — emits events and notifications for + sync, health, and source-readiness events (P9). +- **image-automation-controller** (optional) — updates git with new + image tags when a policy matches, closing the "latest image" + loop declaratively. + +## Kustomization CRD (P2 Declarative Over Imperative, P4) + +- A Kustomization binds "this source" to "this target namespace" + with a reconciliation interval. The reconciler loops + continuously; drift is corrected automatically (P4). + +```yaml +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: payments-api + namespace: flux-system +spec: + sourceRef: + kind: GitRepository + name: platform + namespace: flux-system + path: ./manifests/prod + targetNamespace: payments + interval: 1m + prune: true + wait: true + healthChecks: + - apiVersion: apps/v1 + kind: Deployment + name: payments-api + namespace: payments +``` + +- `prune: true` deletes resources removed from git. `wait: true` + waits for health checks before declaring the Kustomization ready. + Disable prune for workloads that need manual removal gates. + +## HelmRelease CRD (P6 Composability, cross-link helm.md) + +- A HelmRelease binds a Helm chart (from a HelmRepository or OCI + source) to target values and a target namespace. helm-controller + renders and applies it. See `domains/kubernetes/helm.md` for the + chart model. +- Pin the chart version in the HelmRepository or the HelmRelease. + Never float `latest` — unversioned charts drift (P5). + +## OCI Sources (P5 State is Immutable and Versioned) + +- source-controller can pull from OCI registries (Helm charts as + OCI artifacts, or generic OCI repositories). The digest is the + version — immutable by construction (P5). +- OCI sources close the supply-chain loop: the manifest is signed + and immutable in the registry, and Flux pulls it by digest. Cross- + link `domains/security/supply-chain.md` for signed-provenance + principles. + +## Reconciliation and Drift (P4 Continuous Reconciliation, P8) + +- Flux reconciles on `interval` (default 1m) and on webhook event. + Drift between git and cluster is detected each interval and + corrected (with `prune` + `selfHeal` semantics). +- Hand-edited drift on a Flux-managed resource is overwritten on the + next loop — the hand edit was never truth (P8). The recovery is + to fix git, not to `kubectl apply`. + +## Notifications and Events (P9 Failure is Observable and Surfaced) + +- notification-controller emits events for source readiness, sync + success/failure, and health transitions. Wire them to Slack, + PagerDuty, or a webhook. Silent drift is the bug (P9). +- Events flow to `domains/observability/metrics.md` via the + notification controller's provider model — sync and health as + first-class signals. + +## RBAC and Multi-Cluster (P10 Least Privilege Reconciliation, P4) + +- Flux's controllers run with a ServiceAccount in `flux-system`. + Scope that account to the namespaces Flux reconciles. Do not bind + it to `cluster-admin` (P10). See `domains/kubernetes/rbac.md` and + `domains/security/authorization.md`. +- Flux is per-cluster by design (one Flux install per cluster). For + multi-cluster, use one repo with per-cluster paths, or a fleet + tool that bootstraps Flux per cluster. Per-cluster autonomy is a + feature, not a limitation — it bounds the blast radius of a + compromised controller (P4 locality, P10). + +## Secrets (P10, cross-link security/secrets) + +- Do not store raw Secrets in the GitOps repo. Use the + SOPS-compatible decryption in kustomize-controller, or External + Secrets Operator, so the git store holds encrypted material only. + See `domains/security/secrets.md`. + +## ArgoCD vs Flux — Decision Matrix (IDEATE-21, D-039) + +| Axis | ArgoCD | Flux | +|------|--------|------| +| Architecture | Monolithic controller + Application CRD | Composable GitOps Toolkit controllers (source, kustomize, helm, notification) | +| Reconciliation unit | Application (one CRD per app) | Kustomization / HelmRelease (one per deploy unit) | +| UI | Web UI + CLI (full dashboard, tree view, diff viewer) | CLI-first; UI via Weave GitOps or FluxUI (add-on) | +| Sync model | Periodic poll or webhook; sync waves + hooks | Poll + webhook; runs continuously, no explicit sync waves | +| Multi-cluster | One ArgoCD manages many clusters (hub-and-spoke) | One Flux per cluster (per-cluster autonomy) | +| Templating in repo | Helm, Kustomize, ksonnet, raw manifests, Jsonnet | Helm, Kustomize, raw manifests | +| RBAC | Built-in RBAC + SSO + AppProjects | Kubernetes RBAC (no built-in RBAC layer) | +| Progressive delivery | Argo Rollouts (sister project, tight integration) | Flagger (sister project, tight integration) | +| Best for | Teams wanting a UI, multi-cluster from one pane, App-of-Apps bootstrapping | Teams wanting composable controllers, per-cluster autonomy, minimal footprint | +| Watch out for | Monolithic controller scaling, UI as ops crutch, AppProject sprawl | No native UI, steeper learning curve, manual multi-cluster orchestration | + +- Use Flux when you want composable controllers, per-cluster + autonomy, and a minimal footprint. Use ArgoCD when you want a UI, + central multi-cluster management, and sync-wave ordering. +- Both are CNCF graduated and both implement the OpenGitOps + principles. The choice is architectural fit, not correctness. See + `argocd.md` for the ArgoCD-side perspective. + +## What Violates Flux Discipline + +| Violation | Principle | +|-----------|-----------| +| CI pipeline with `kubectl` rights pushing to the cluster | P3 Pull, Don't Push | +| HelmRelease with no pinned chart version | P5 State is Immutable and Versioned | +| Flux ServiceAccount bound to `cluster-admin` | P10 Least Privilege Reconciliation | +| Kustomization with no `healthChecks` on a prod app | P9 Failure is Observable and Surfaced | +| No notification provider wired for sync failures | P9 Failure is Observable and Surfaced | +| Raw Secret in the GitOps repo | P10, `domains/security/secrets.md` | +| Manual `kubectl edit` on a Flux-managed resource | P8 Reconcile, Don't Mutate by Hand | +| `interval: 24h` on a prod Kustomization (drift window too wide) | P4 Continuous Reconciliation | \ No newline at end of file diff --git a/domains/gitops-operators/operators.md b/domains/gitops-operators/operators.md new file mode 100644 index 0000000..31d3107 --- /dev/null +++ b/domains/gitops-operators/operators.md @@ -0,0 +1,140 @@ +# Operators — Derived Rules + +> Derives from `domains/gitops-operators/first-principles.md`. +> Applies P6 (Operators Encode Domain Knowledge) primarily, with +> P1, P4, P8, P9, P10. Cross-links `domains/kubernetes/workloads.md` +> and `domains/kubernetes/rbac.md` for the underlying controller +> model, and `domains/infrastructure-as-code/modules.md` for the +> module-vs-operator boundary. + +## What an Operator Is (P6 Operators Encode Domain Knowledge) + +- An Operator is a Kubernetes controller that encodes human + operational knowledge as CRDs plus a control loop. The operator + reconciles a domain-specific resource (a database, a message + queue, a certificate, a ML model) to a desired state. +- The operator is the deepest expression of + `domains/kubernetes/P1 Declarative Desired State`: the domain + knowledge itself is the desired state. A runbook that lives only + in a wiki is operational knowledge that has not been encoded — + the operator is the encoding (P6). +- An operator runs inside the cluster, observes its CRDs, and acts. + It is a pull-based reconciler by construction — see + `domains/gitops-operators/first-principles.md` P3. + +## CRDs and Controllers (P2 Declarative Over Imperative, P4 Continuous Reconciliation) + +- A CustomResourceDefinition (CRD) defines the schema of the + domain resource. The controller watches instances of that CRD + and reconciles current → desired (P4). +- The CRD is the public contract of the operator. Version it + (`v1alpha1` → `v1beta1` → `v1`) and preserve backward + compatibility — see `domains/api/versioning.md` for the general + API-evolution principles. A CRD is an API surface, not an + internal type. + +```yaml +apiVersion: postgres.example.com/v1 +kind: PostgresCluster +metadata: + name: payments-db + namespace: payments +spec: + replicas: 3 + version: "16" + storage: + size: 100Gi + storageClass: fast-ssd + backup: + schedule: "0 2 * * *" + retention: 7d +``` + +- The controller reconciles this spec: creates StatefulSets, PVCs, + Services, backup CronJobs. The user declares intent; the operator + makes it so (P2, P6). + +## The Control Loop (P4 Continuous Reconciliation, P8) + +- The loop watches CRD instances, compares current vs desired, and + acts to converge. Drift (a hand-deleted pod, a failed backup) is + detected and corrected each loop (P4). +- An operator-managed resource should not be hand-edited (P8). The + operator owns the subordinate resources (StatefulSets, PVCs); a + manual `kubectl edit` on a subordinate is drift the operator will + overwrite. + +## Operator SDK and OLM (P6 Composability, C6) + +- The Operator SDK scaffolds a controller from a CRD (Go, Ansible, + Helm). Use it to avoid re-implementing the controller boilerplate. +- Operator Lifecycle Manager (OLM) installs, updates, and manages + operators as first-class cluster components. OLM is the package + manager for operators — the operator analogue of + `domains/kubernetes/helm.md` for workloads. +- An operator published via OLM is a versioned, catalog-tracked + artifact. Pin the operator version; do not float `latest` (P5 + applies to operators as much as to manifests). + +## When to Write an Operator vs a Helm Chart (P6, C6 Composability) + +| Axis | Helm chart | Operator | +|------|-----------|----------| +| Day-2 operations | None — chart installs, you operate | Encoded — operator reconciles lifecycle (backup, resize, failover, upgrade) | +| State | Static manifests | Live control loop watching CRDs | +| Day-1 install | Strong fit — package and install | Overkill if install is all you need | +| Day-2 reconcile | None — drift is manual | Continuous — drift corrected each loop | +| Domain knowledge | Lives in runbooks + on-call | Lives in the controller code | +| Best for | Off-the-shelf apps, stateless services, one-shot deploys | Stateful apps, complex lifecycles, day-2 automation (backup, scale, failover, version upgrades) | +| Watch out for | Templating complexity, no day-2 reconcile | Controller complexity, multi-team maintenance burden, scope creep | + +- Write an operator when the day-2 operations (backup, failover, + resize, version upgrade) are non-trivial and repeated. Write a + Helm chart when install is all you need and day-2 is run by a + human or a separate tool. +- Do not write an operator to wrap a Helm chart and call it day-2 + automation — that is a Helm chart with extra steps. See + `domains/infrastructure-as-code/modules.md` for the + module-vs-copy boundary (the operator-vs-chart boundary is its + analogue). + +## Scope and Responsibility Boundaries (P10 Least Privilege, C6) + +- An operator owns one domain. An operator that manages databases + and message queues and certificates is doing three jobs — split + it. Scope creep is the most common operator failure mode (P6 + violation: the encoded knowledge is no longer coherent). +- The operator's ServiceAccount must be scoped to the resources it + manages (P10). A database operator that needs `cluster-admin` to + create a StatefulSet has the wrong RBAC — see + `domains/kubernetes/rbac.md` and `domains/security/authorization.md`. +- One operator per CRD family; one ServiceAccount per operator; one + namespace per operator (or a shared `operators` namespace with + strict RoleBindings). Default namespace is for nothing in + production. + +## Failure and Observability (P9 Failure is Observable and Surfaced) + +- An operator must surface its reconcile status on the CRD + (`status.conditions`, `status.observedGeneration`). A CRD with no + status is an operator that fails silently (P9). +- Wire operator events to notifications and metrics. A failed + backup, a stuck failover, a version-upgrade stall must emit a + signal — see `domains/observability/metrics.md`. +- An operator that reconciles but does not report health is a + black box. The GitOps controller (ArgoCD/Flux) will read it as + `Progressing` forever — write the health check (see `argocd.md` + "Health and Status"). + +## What Violates Operator Discipline + +| Violation | Principle | +|-----------|-----------| +| Operator that manages databases + queues + certs | P6 Operators Encode Domain Knowledge (scope creep) | +| Operator ServiceAccount bound to `cluster-admin` | P10 Least Privilege Reconciliation | +| CRD with no `status.conditions` | P9 Failure is Observable and Surfaced | +| Operator with no health check wired to GitOps | P9, `argocd.md` Health and Status | +| Unversioned CRD (`v1` shipped without alpha/beta) | P5, `domains/api/versioning.md` | +| Manual `kubectl edit` on an operator-managed subordinate | P8 Reconcile, Don't Mutate by Hand | +| Operator that wraps a Helm chart and adds no day-2 logic | P6 (no knowledge encoded) | +| Operator runbook that exists only in a wiki | P6 Operators Encode Domain Knowledge | \ No newline at end of file diff --git a/domains/gitops-operators/progressive-delivery.md b/domains/gitops-operators/progressive-delivery.md new file mode 100644 index 0000000..516ddfe --- /dev/null +++ b/domains/gitops-operators/progressive-delivery.md @@ -0,0 +1,177 @@ +# Progressive Delivery — Derived Rules + +> Derives from `domains/gitops-operators/first-principles.md`. +> Applies P7 (Progressive Delivery is Reversible by Construction) +> primarily, with P4, P9. Cross-links `domains/devops/first-principles.md` +> P4 Rollback First and P5 Progressive Delivery, and +> `domains/observability/metrics.md` for the analysis signals. + +## What Progressive Delivery Is (P7 Reversible by Construction) + +- Progressive delivery shifts traffic in stages (canary, blue-green) + gated by analysis (metrics, counters, error rates). Each stage is + metric-checked; a failed gate aborts the rollout and reverts to + the prior stable version. Promotion without a rollback path is a + violation (P7). +- Progressive delivery is the GitOps extension of + `domains/devops/P5 Progressive Delivery` and + `domains/kubernetes/P10 Roll Forward, Roll Back`. The k8s rolling + update is the floor; progressive delivery adds metric-gated + promotion and one-command abort. +- Two sister projects dominate: **Argo Rollouts** (Argo ecosystem) + and **Flagger** (Flux ecosystem). Both implement the same pattern + — a Rollout CRD replaces a Deployment, an analysis drives the + gates, an abort reverts traffic. + +## The Rollout CRD (P2 Declarative Over Imperative, P7) + +- A Rollout (Argo Rollouts) or Canary/Flag (Flagger) is a CRD that + replaces the Deployment as the reconciled resource. It declares + the strategy (canary, blue-green), the traffic split, and the + analysis gates. The controller reconciles traffic and pods to + match. + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Rollout +metadata: + name: payments-api + namespace: payments +spec: + replicas: 10 + selector: + matchLabels: + app: payments-api + template: + metadata: + labels: + app: payments-api + spec: + containers: + - name: api + image: registry.example.com/payments-api:1.2.3 + strategy: + canary: + trafficRouting: + istio: + virtualService: + name: payments-vs + routes: [primary] + steps: + - setWeight: 5 + - pause: { duration: 2m } + - analysis: + templates: + - templateName: success-rate + - setWeight: 25 + - pause: { duration: 5m } + - analysis: + templates: + - templateName: success-rate + - setWeight: 50 + - pause: { duration: 5m } + - setWeight: 100 +``` + +- Each `setWeight` shifts traffic; each `pause` holds for + observation; each `analysis` runs a metric gate. A failed + analysis aborts the rollout and reverts traffic to the stable + ReplicaSet (P7). + +## Canary vs Blue-Green (P7, C3 Simplicity) + +| Strategy | Mechanism | Cost | Best for | +|----------|-----------|------|----------| +| Canary | Shift a small % of traffic to the new version; increase on gate success | Low (few new pods) | Most production rollouts; metric-gated, gradual | +| Blue-Green | Run two full environments; switch traffic all-at-once | High (2× capacity) | Schema-breaking changes, instant rollback, low-frequency deploys | + +- Canary is the default — it is reversible by construction (P7) + and economical (C8). Blue-green is for changes that cannot be + partial (a breaking schema migration, a full cutover). +- A canary with no analysis gate is a slow blue-green — it is not + progressive delivery. The gate is what makes it progressive (P7). + +## Analysis Templates (P9 Failure is Observable and Surfaced, P7) + +- An AnalysisTemplate declares the metric query, the success + threshold, and the count of samples. The rollout controller runs + the analysis at each gate; a failed analysis aborts the rollout. + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: AnalysisTemplate +metadata: + name: success-rate + namespace: payments +spec: + metrics: + - name: success-rate + interval: 1m + successCondition: result[0] >= 0.99 + failureLimit: 2 + provider: + prometheus: + address: http://prometheus.observability:9090 + query: | + sum(rate(http_requests_total{job="payments-api",code!~"5.."}[2m])) + / + sum(rate(http_requests_total{job="payments-api"}[2m])) +``` + +- `successCondition` is the gate; `failureLimit` is the tolerance + for transient blips. A single failed sample aborts immediately if + `failureLimit: 0`; tolerate noise with `failureLimit: 2`. +- The metric is the abort signal — see `domains/observability/metrics.md` + for the SLI/SLO discipline that makes the gate meaningful. A gate + on an undefined SLO is a gate on noise. + +## Argo Rollouts vs Flagger (P6 Composability, P7) + +| Axis | Argo Rollouts | Flagger | +|------|---------------|---------| +| Ecosystem | Argo (ArgoCD sister project) | Flux (Flux sister project) | +| CRD | `Rollout` (replaces `Deployment`) | `Canary` / `Flag` (wraps a `Deployment`) | +| Traffic providers | Istio, NGINX, ALB, SMI, Traefik, Ambassador | Istio, NGINX, Linkerd, SMI, App Mesh, Gloo, Contour | +| Analysis sources | Prometheus, Datadog, Wavefront, NewRelic, CloudWatch, Graphite, Kayenta | Prometheus, Datadog, CloudWatch, Stackdriver, Elasticsearch, Graphite | +| Integration | Tight with ArgoCD (UI shows rollout) | Tight with Flux (events via notification-controller) | +| Learning curve | Rollout CRD replaces Deployment (migration cost) | Wraps existing Deployment (lower migration cost) | +| Best for | ArgoCD shops wanting rollout in the Argo UI | Flux shops wanting progressive delivery with minimal migration | + +- Both implement the same pattern. The choice follows your GitOps + controller — Argo Rollouts with ArgoCD, Flagger with Flux. Mixing + is possible but not idiomatic. + +## Abort and Rollback (P7 Reversible by Construction, P5) + +- An abort reverts traffic to the stable ReplicaSet immediately. A + rollout without a tested abort is a prototype (P7). +- The abort must be one-command (or one-gate-failure). A + progressive delivery that requires manual rollback steps has + lost the "reversible by construction" property — it is a deploy + with extra steps. +- Test the abort path in staging. An abort that has never been + exercised will fail when you need it most — see + `domains/devops/first-principles.md` P4 Rollback First. + +## Observability (P9 Failure is Observable and Surfaced) + +- Progressive delivery is only as good as its metrics. A rollout + gated on a metric that is not tracked is ungated — the gate is + theater (P9). +- Wire rollout status (phase, weight, analysis result) to + notifications and dashboards. A stalled rollout with no signal is + silent drift (P9). See `domains/observability/metrics.md`. +- Cross-link `domains/kubernetes/workloads.md` for the underlying + Deployment/ReplicaSet model that progressive delivery replaces. + +## What Violates Progressive Delivery Discipline + +| Violation | Principle | +|-----------|-----------| +| Canary with no analysis gate | P7 Progressive Delivery is Reversible by Construction | +| Rollout with no tested abort path | P7, `domains/devops/P4 Rollback First` | +| Analysis gate on an undefined SLO | P9 Failure is Observable and Surfaced | +| Blue-green with no 2× capacity budget | C8 Economy (blue-green is a cost decision) | +| Rollout stalled with no notification | P9 Failure is Observable and Surfaced | +| Manual `kubectl` traffic shift on a Rollout-managed service | P8 Reconcile, Don't Mutate by Hand | +| `failureLimit: 0` on a noisy metric (constant false aborts) | P4 Continuous Reconciliation (gate noise tolerance) | \ No newline at end of file