orca/.ciagent/RESEARCH_v0.11.md

Research: v0.11 Production Hardening

Source material

Five research documents were ingested 2026-08-07 as directional input (not verbatim) for v0.11 Phase 0. The current ciagent files (R-001…R-016, D-001…D-206) are authoritative and take precedence; where research conflicted, the ciagent files won. The research drove the adoption of R-017…R-020 and D-215…D-237 (see PROJECT.md, PRD_v0.11.md).

Doc	Theme	Adopted as
1	Ingress hybrid (nft DNAT → Traefik on 127.0.0.1:8443)	R-017, D-215..D-226, REQ-099..REQ-102
2	Platform-engineer playbook (8 differentiators, TCO, honest trade-offs)	README positioning (Q5=A), CLI surface gap analysis
3	Strategic positioning ("be Proxmox-for-bare-metal, not K8s-without-K8s")	README framing (Q5=A: Nomad-inspired, honest trade-offs table from doc 3, not Proxmox-first lead)
4	Drift detection cadence (R-018/R-019/R-020, tiered cadence, hard gate)	R-018, R-019, R-020, D-227..D-237, REQ-103..REQ-113
5	Drift detection concrete impl (systemd Path units, orca-drift-notify.sh, orca-remediate.sh)	D-227..D-237 detail, REQ-103..REQ-113

Thread A — Ingress hardening (doc 1)

What changes vs v0.9/v0.10

Traefik static config gains entryPoints.websecure.address: 127.0.0.1:8443 (default) instead of :443. A new nftables emitter renders /etc/nftables.d/orca.nft with DNAT rules. Certs, mTLS, dynamic config, and the workload SPIFFE validation path are completely unchanged. Only the address line shifts + one new emitter + orca doctor nft + orca nft ... CLI.

Defense in depth

Two layers: kernel (nftables: SYN flood, rate limit, GeoIP, conntrack) and application (Traefik: mTLS, SNI, ACL, dynamic routing, health checks). Neither can replace the other; they catch different attack classes.

Codebase reality (verified 2026-08-07)

• internal/emitter/traefik.go + traefik_atomic.go exist (v0.9 P02). The static-config emitter is where the address: line change lands.
• internal/emitter/systemd.go exists. New .path/.service unit types extend this emitter pattern (shared with drift detection, doc 5).
• internal/emitter/nft.go does not exist — greenfield, ~200 LoC.
• scripts/ has orca-verify-render.sh but not orca-aggregate.sh, orca-pull.sh, orca-apply-render.sh, orca-remediate.sh — all are v0.11 P09/P10 scope.

Thread B — Drift detection + transactional plane (docs 4 + 5)

Architecture

systemd Path units (R-001-clean; systemd is OS, not Orca) watch critical paths via inotify. On change, a oneshot service computes sha256 and writes an event JSON to /etc/orca/state/drift-events/. The lead's aggregator timer (10s, C-11) rsyncs these events, validates against the applied txn manifest, and triggers remediation for auto-remediable paths.

Tiered cadence

Tier	Detection	Auto-remediate	Latency
Critical	Path unit + 5s polling backstop	yes (config files only)	~10s
Standard	30s polling	optional (systemd units: require approval)	30s
Default	60s polling	no	60s

R-020 hard gate

Applier refuses new txns if pre-flight consistency check fails. Override: --force flag + per-namespace scoping (Q4=A) — a drifted peer in ns-A does not block ns-B.

Codebase reality (verified 2026-08-07)

• internal/store/node_repo.go:80 and internal/store/job_task_repo.go:82 already use iter.Seq[T]. Doc 5's iter.Seq2[Event, error] is the natural extension per D-017 (settled, shipped v0.3).
• internal/paths/paths.go:86 has TxnDir() — the txn staging dir the drift detector hooks into.
• internal/emit/contract.go has the Go↔bash render-contract anti-drift (C-16). The runtime drift detector (doc 5) is net-new.
• internal/drift/ package does not exist — greenfield, ~500 LoC.
• No orca system user creation in code — net-new operational requirement (REQ-111).
• No NFS detection at peer setup — net-new (REQ-112, D-233).
• doctor.go has an OS-drift check (one-shot, on-demand) but not a 60s runtime drift-polling loop. Doc 5's design is net-new scope.

Alignment with existing gates

• C-09 (orca-pull.sh failure contract) — R-020 refines "deterministic state" into an explicit refusal contract.
• C-11 (lead-side watchdog meta-timer) — doc 5's aggregator extension is the input C-11 monitors.
• REQ-075 (lead applier execution model) — doc 5's orca-remediate.sh is literally the same code path as a normal txn-apply, triggered by drift instead of a new submission.

Thread C — Positioning/messaging (docs 2 + 3)

Consistent with locked vision

The vision is "A minimalist, offline-first, CLI-first orchestration engine inspired by HashiCorp Nomad" — explicitly Nomad-inspired, not K8s. Doc 3's recommendation ("be Proxmox-for-bare-metal, not K8s-without-the-complexity") is consistent with the locked vision.

Where doc 3 diverges (resolved per Q5=A)

Doc 3 recommends "leading with Proxmox positioning." But R-003 says "Proxmox can never be lead." Leading the project identity with a node type that can't be the lead is subtly contradictory. Q5=A decision: README uses the Nomad-inspired, OS-as-cluster framing (locked vision), mentions Proxmox as one node type, and incorporates doc 3's "honest trade-offs" table but not its Proxmox-first lead-positioning advice.

CLI surface gap analysis (doc 2)

Doc 2's playbook cites ~10 CLI commands. Verified against the live codebase (internal/cli/*.go):

Exist today: orca init, orca node {join,leave,list,key-reset, capacity}, orca job {run,list,stop,logs}, orca ns {list,create, delete,inspect,validate}, orca cert {ca-init,gen,show,renew, fingerprint}, orca doctor {cert,network,db,os,proxmox}, orca audit list, orca status, orca version, orca daemon (deprecated).

Not in v0.11 ROADMAP, added per Q2=C: orca cluster rotate-lead (REQ-114, P14b), orca upgrade (REQ-115, P14a), orca job migrate (REQ-116, P05), orca logs --all-nodes --since (REQ-117, P06), orca doctor mTLS (REQ-118, P15.5).

Already in v0.11 ROADMAP: orca node drain (P05), orca job lint (P11), orca job verify (P12), orca restore (P07), orca backup (P04).

Unverified performance claims in doc 3

Doc 3's "10s applier timer = 10,000x slower than K8s informers" and "60s drift polling" are forward-looking design constraints, not current-state limitations — no applier timer or drift-polling loop exists in the codebase. These are answered by R-018/R-019/R-020 (doc 4 + doc 5): the drift detector is a backstop, not the primary detector, and critical paths get ~10s latency via systemd Path units.

Persona assessment

v0.11 touches these territories:

Territory	Persona	Phases
internal/cli/**, internal/drift/**, internal/nft/**	backend-engineer	P10, P15.5, P05, P06, P14a, P14b
internal/emitter/**, internal/sshpush/**	backend-engineer + lead-developer	P09, P10, P15.5
internal/store/**, migrations	data-engineer	P14a (data migration)
scripts/orca-*.sh	backend-engineer (bash tooling)	P09, P10
docs/**, README.md, examples/**	lead-developer + docs-engineer (phase-specific)	P15, P08
Threat model, security review, mTLS, secrets	security-engineer	P03, P15.5
nftables, Traefik binding, cluster mesh	network-engineer	P15.5, P09
Test coverage, integration harness	devops-engineer (phase-specific)	P08

No frontend-engineer work (no UI). The data-engineer persona is reactivated for P14a (v0.8→v1.0 data migration). A docs-engineer custom persona is created for P15 (README) and P08 (integration test docs). See PERSONAS.md for the updated roster.