atelier/domains/gitops-operators/progressive-delivery.md

Progressive Delivery — Derived Rules

Derives from domains/gitops-operators/first-principles.md. Applies P7 (Progressive Delivery is Reversible by Construction) primarily, with P4, P9. Cross-links domains/devops/first-principles.md P4 Rollback First and P5 Progressive Delivery, and domains/observability/metrics.md for the analysis signals.

What Progressive Delivery Is (P7 Reversible by Construction)

• Progressive delivery shifts traffic in stages (canary, blue-green) gated by analysis (metrics, counters, error rates). Each stage is metric-checked; a failed gate aborts the rollout and reverts to the prior stable version. Promotion without a rollback path is a violation (P7).
• Progressive delivery is the GitOps extension of domains/devops/P5 Progressive Delivery and domains/kubernetes/P10 Roll Forward, Roll Back. The k8s rolling update is the floor; progressive delivery adds metric-gated promotion and one-command abort.
• Two sister projects dominate: Argo Rollouts (Argo ecosystem) and Flagger (Flux ecosystem). Both implement the same pattern — a Rollout CRD replaces a Deployment, an analysis drives the gates, an abort reverts traffic.

The Rollout CRD (P2 Declarative Over Imperative, P7)

• A Rollout (Argo Rollouts) or Canary/Flag (Flagger) is a CRD that replaces the Deployment as the reconciled resource. It declares the strategy (canary, blue-green), the traffic split, and the analysis gates. The controller reconciles traffic and pods to match.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: payments-api
  namespace: payments
spec:
  replicas: 10
  selector:
    matchLabels:
      app: payments-api
  template:
    metadata:
      labels:
        app: payments-api
    spec:
      containers:
        - name: api
          image: registry.example.com/payments-api:1.2.3
  strategy:
    canary:
      trafficRouting:
        istio:
          virtualService:
            name: payments-vs
            routes: [primary]
      steps:
        - setWeight: 5
        - pause: { duration: 2m }
        - analysis:
            templates:
              - templateName: success-rate
        - setWeight: 25
        - pause: { duration: 5m }
        - analysis:
            templates:
              - templateName: success-rate
        - setWeight: 50
        - pause: { duration: 5m }
        - setWeight: 100

• Each setWeight shifts traffic; each pause holds for observation; each analysis runs a metric gate. A failed analysis aborts the rollout and reverts traffic to the stable ReplicaSet (P7).

Canary vs Blue-Green (P7, C3 Simplicity)

Strategy	Mechanism	Cost	Best for
Canary	Shift a small % of traffic to the new version; increase on gate success	Low (few new pods)	Most production rollouts; metric-gated, gradual
Blue-Green	Run two full environments; switch traffic all-at-once	High (2× capacity)	Schema-breaking changes, instant rollback, low-frequency deploys

• Canary is the default — it is reversible by construction (P7) and economical (C8). Blue-green is for changes that cannot be partial (a breaking schema migration, a full cutover).
• A canary with no analysis gate is a slow blue-green — it is not progressive delivery. The gate is what makes it progressive (P7).

Analysis Templates (P9 Failure is Observable and Surfaced, P7)

• An AnalysisTemplate declares the metric query, the success threshold, and the count of samples. The rollout controller runs the analysis at each gate; a failed analysis aborts the rollout.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
  namespace: payments
spec:
  metrics:
    - name: success-rate
      interval: 1m
      successCondition: result[0] >= 0.99
      failureLimit: 2
      provider:
        prometheus:
          address: http://prometheus.observability:9090
          query: |
            sum(rate(http_requests_total{job="payments-api",code!~"5.."}[2m]))
            /
            sum(rate(http_requests_total{job="payments-api"}[2m]))

• successCondition is the gate; failureLimit is the tolerance for transient blips. A single failed sample aborts immediately if failureLimit: 0; tolerate noise with failureLimit: 2.
• The metric is the abort signal — see domains/observability/metrics.md for the SLI/SLO discipline that makes the gate meaningful. A gate on an undefined SLO is a gate on noise.

Argo Rollouts vs Flagger (P6 Composability, P7)

Axis	Argo Rollouts	Flagger
Ecosystem	Argo (ArgoCD sister project)	Flux (Flux sister project)
CRD	Rollout (replaces Deployment)	Canary / Flag (wraps a Deployment)
Traffic providers	Istio, NGINX, ALB, SMI, Traefik, Ambassador	Istio, NGINX, Linkerd, SMI, App Mesh, Gloo, Contour
Analysis sources	Prometheus, Datadog, Wavefront, NewRelic, CloudWatch, Graphite, Kayenta	Prometheus, Datadog, CloudWatch, Stackdriver, Elasticsearch, Graphite
Integration	Tight with ArgoCD (UI shows rollout)	Tight with Flux (events via notification-controller)
Learning curve	Rollout CRD replaces Deployment (migration cost)	Wraps existing Deployment (lower migration cost)
Best for	ArgoCD shops wanting rollout in the Argo UI	Flux shops wanting progressive delivery with minimal migration

• Both implement the same pattern. The choice follows your GitOps controller — Argo Rollouts with ArgoCD, Flagger with Flux. Mixing is possible but not idiomatic.

Abort and Rollback (P7 Reversible by Construction, P5)

• An abort reverts traffic to the stable ReplicaSet immediately. A rollout without a tested abort is a prototype (P7).
• The abort must be one-command (or one-gate-failure). A progressive delivery that requires manual rollback steps has lost the "reversible by construction" property — it is a deploy with extra steps.
• Test the abort path in staging. An abort that has never been exercised will fail when you need it most — see domains/devops/first-principles.md P4 Rollback First.

Observability (P9 Failure is Observable and Surfaced)

• Progressive delivery is only as good as its metrics. A rollout gated on a metric that is not tracked is ungated — the gate is theater (P9).
• Wire rollout status (phase, weight, analysis result) to notifications and dashboards. A stalled rollout with no signal is silent drift (P9). See domains/observability/metrics.md.
• Cross-link domains/kubernetes/workloads.md for the underlying Deployment/ReplicaSet model that progressive delivery replaces.

What Violates Progressive Delivery Discipline

Violation	Principle
Canary with no analysis gate	P7 Progressive Delivery is Reversible by Construction
Rollout with no tested abort path	P7, domains/devops/P4 Rollback First
Analysis gate on an undefined SLO	P9 Failure is Observable and Surfaced
Blue-green with no 2× capacity budget	C8 Economy (blue-green is a cost decision)
Rollout stalled with no notification	P9 Failure is Observable and Surfaced
Manual kubectl traffic shift on a Rollout-managed service	P8 Reconcile, Don't Mutate by Hand
failureLimit: 0 on a noisy metric (constant false aborts)	P4 Continuous Reconciliation (gate noise tolerance)