atelier/domains/gitops-operators/operators.md

Operators — Derived Rules

Derives from domains/gitops-operators/first-principles.md. Applies P6 (Operators Encode Domain Knowledge) primarily, with P1, P4, P8, P9, P10. Cross-links domains/kubernetes/workloads.md and domains/kubernetes/rbac.md for the underlying controller model, and domains/infrastructure-as-code/modules.md for the module-vs-operator boundary.

What an Operator Is (P6 Operators Encode Domain Knowledge)

• An Operator is a Kubernetes controller that encodes human operational knowledge as CRDs plus a control loop. The operator reconciles a domain-specific resource (a database, a message queue, a certificate, a ML model) to a desired state.
• The operator is the deepest expression of domains/kubernetes/P1 Declarative Desired State: the domain knowledge itself is the desired state. A runbook that lives only in a wiki is operational knowledge that has not been encoded — the operator is the encoding (P6).
• An operator runs inside the cluster, observes its CRDs, and acts. It is a pull-based reconciler by construction — see domains/gitops-operators/first-principles.md P3.

CRDs and Controllers (P2 Declarative Over Imperative, P4 Continuous Reconciliation)

• A CustomResourceDefinition (CRD) defines the schema of the domain resource. The controller watches instances of that CRD and reconciles current → desired (P4).
• The CRD is the public contract of the operator. Version it (v1alpha1 → v1beta1 → v1) and preserve backward compatibility — see domains/api/versioning.md for the general API-evolution principles. A CRD is an API surface, not an internal type.

apiVersion: postgres.example.com/v1
kind: PostgresCluster
metadata:
  name: payments-db
  namespace: payments
spec:
  replicas: 3
  version: "16"
  storage:
    size: 100Gi
    storageClass: fast-ssd
  backup:
    schedule: "0 2 * * *"
    retention: 7d

• The controller reconciles this spec: creates StatefulSets, PVCs, Services, backup CronJobs. The user declares intent; the operator makes it so (P2, P6).

The Control Loop (P4 Continuous Reconciliation, P8)

• The loop watches CRD instances, compares current vs desired, and acts to converge. Drift (a hand-deleted pod, a failed backup) is detected and corrected each loop (P4).
• An operator-managed resource should not be hand-edited (P8). The operator owns the subordinate resources (StatefulSets, PVCs); a manual kubectl edit on a subordinate is drift the operator will overwrite.

Operator SDK and OLM (P6 Composability, C6)

• The Operator SDK scaffolds a controller from a CRD (Go, Ansible, Helm). Use it to avoid re-implementing the controller boilerplate.
• Operator Lifecycle Manager (OLM) installs, updates, and manages operators as first-class cluster components. OLM is the package manager for operators — the operator analogue of domains/kubernetes/helm.md for workloads.
• An operator published via OLM is a versioned, catalog-tracked artifact. Pin the operator version; do not float latest (P5 applies to operators as much as to manifests).

When to Write an Operator vs a Helm Chart (P6, C6 Composability)

Axis	Helm chart	Operator
Day-2 operations	None — chart installs, you operate	Encoded — operator reconciles lifecycle (backup, resize, failover, upgrade)
State	Static manifests	Live control loop watching CRDs
Day-1 install	Strong fit — package and install	Overkill if install is all you need
Day-2 reconcile	None — drift is manual	Continuous — drift corrected each loop
Domain knowledge	Lives in runbooks + on-call	Lives in the controller code
Best for	Off-the-shelf apps, stateless services, one-shot deploys	Stateful apps, complex lifecycles, day-2 automation (backup, scale, failover, version upgrades)
Watch out for	Templating complexity, no day-2 reconcile	Controller complexity, multi-team maintenance burden, scope creep

• Write an operator when the day-2 operations (backup, failover, resize, version upgrade) are non-trivial and repeated. Write a Helm chart when install is all you need and day-2 is run by a human or a separate tool.
• Do not write an operator to wrap a Helm chart and call it day-2 automation — that is a Helm chart with extra steps. See domains/infrastructure-as-code/modules.md for the module-vs-copy boundary (the operator-vs-chart boundary is its analogue).

Scope and Responsibility Boundaries (P10 Least Privilege, C6)

• An operator owns one domain. An operator that manages databases and message queues and certificates is doing three jobs — split it. Scope creep is the most common operator failure mode (P6 violation: the encoded knowledge is no longer coherent).
• The operator's ServiceAccount must be scoped to the resources it manages (P10). A database operator that needs cluster-admin to create a StatefulSet has the wrong RBAC — see domains/kubernetes/rbac.md and domains/security/authorization.md.
• One operator per CRD family; one ServiceAccount per operator; one namespace per operator (or a shared operators namespace with strict RoleBindings). Default namespace is for nothing in production.

Failure and Observability (P9 Failure is Observable and Surfaced)

• An operator must surface its reconcile status on the CRD (status.conditions, status.observedGeneration). A CRD with no status is an operator that fails silently (P9).
• Wire operator events to notifications and metrics. A failed backup, a stuck failover, a version-upgrade stall must emit a signal — see domains/observability/metrics.md.
• An operator that reconciles but does not report health is a black box. The GitOps controller (ArgoCD/Flux) will read it as Progressing forever — write the health check (see argocd.md "Health and Status").

What Violates Operator Discipline

Violation	Principle
Operator that manages databases + queues + certs	P6 Operators Encode Domain Knowledge (scope creep)
Operator ServiceAccount bound to cluster-admin	P10 Least Privilege Reconciliation
CRD with no status.conditions	P9 Failure is Observable and Surfaced
Operator with no health check wired to GitOps	P9, argocd.md Health and Status
Unversioned CRD (v1 shipped without alpha/beta)	P5, domains/api/versioning.md
Manual kubectl edit on an operator-managed subordinate	P8 Reconcile, Don't Mutate by Hand
Operator that wraps a Helm chart and adds no day-2 logic	P6 (no knowledge encoded)
Operator runbook that exists only in a wiki	P6 Operators Encode Domain Knowledge