diff --git a/.ciagent/CHECKPOINT.json b/.ciagent/CHECKPOINT.json index 69e0d94..4591afe 100644 --- a/.ciagent/CHECKPOINT.json +++ b/.ciagent/CHECKPOINT.json @@ -1,14 +1,13 @@ { "phase": 0, - "stage": "clarify", + "stage": "research", "milestone": "v0.11", "milestone_slug": "production-hardening", "phase_role": "pre_execution", "attempts": 0, - "updated_at": "2026-08-07T00:01:00Z", + "updated_at": "2026-08-07T00:02:00Z", "milestone_complete": false, "previous_milestone": "v0.10", "research_docs_ingested": 5, - "locked_decisions": {"Q1": "A", "Q2": "C", "Q3": "A", "Q4": "A", "Q5": "A"}, - "clarifications": {"C1": "hybrid-default-fresh-migrate-existing", "C2": "thin-wrapper", "C3": "drain-reschedule", "C4": "cooldown-on-success-only", "C5": "chain-plus-live-probe"} + "locked_decisions": {"Q1": "A", "Q2": "C", "Q3": "A", "Q4": "A", "Q5": "A"} } diff --git a/.ciagent/PERSONAS.md b/.ciagent/PERSONAS.md index d7d402a..ba6d61b 100644 --- a/.ciagent/PERSONAS.md +++ b/.ciagent/PERSONAS.md @@ -210,4 +210,48 @@ reason: | | `security-engineer` reactivated | Owns SSH keygen, TOFU host-key, sudoers, PVE role — first-class security surface. | | `devops-engineer` deactivated | v0.6 has no packaging/distribution surface. | | `network-engineer` remains deactivated | No transport/mTLS surface. | -| `frontend-engineer` remains deactivated | No web UI. | \ No newline at end of file +| `frontend-engineer` remains deactivated | No web UI. | +## v0.11 Update (Production Hardening) + +The v0.9 persona roster carries forward to v0.11 with these additions: + +### Roster changes + +- **lead-developer**: RETAINED — owns `orca cluster rotate-lead` (P14b), + `orca upgrade` (P14a), README framing (P15, Q5=A Nomad-inspired), + milestone coordination. +- **backend-engineer**: RETAINED — owns `internal/drift/` (P10, ~500 LoC + greenfield), `internal/emitter/nft.go` (P15.5, ~200 LoC greenfield), + `orca drift` CLI tree (P10), `orca nft` CLI (P15.5), `orca job migrate` + (P05), `orca logs --all-nodes` (P06), `orca doctor mTLS`/`orca doctor nft` + (P15.5), `scripts/orca-drift-notify.sh` + `orca-remediate.sh` (P10). + Frameworks: cobra, `iter.Seq2` (D-017 extension), `signal.NotifyContext` + (D-023), golang.org/x/crypto/ssh (existing). +- **data-engineer**: REACTIVATED for P14a — owns v0.8→v1.0 data migration + (REQ-066), schema migration for `orca upgrade` binding cutover. Was + deactivated in v0.10 (docs-only milestone); reactivated for the + migration phase. +- **security-engineer**: RETAINED — owns threat model (P15.5, C-19), + secrets subsystem (P03), `orca doctor mTLS` (P15.5), ingress-hybrid + trust-boundary review (R-017), drift-detection threat model (R-020 + deadlock, secret exclusion D-234). +- **network-engineer**: RETAINED — owns nftables emitter (P15.5, R-017), + Traefik binding cutover (P14a/P15.5), cross-node cluster mesh (D-219, + unchanged private IP), drift-detection network paths (NFS detection + D-233, SSH fanout for aggregator). +- **devops-engineer**: RETAINED — owns `scripts/orca-aggregate.sh` + extension (P09, D-237), `scripts/orca-drift-notify.sh` (P10), + `scripts/orca-remediate.sh` (P10), systemd Path unit emitter (P10), + drift-detection integration tests (P08: auto-remediation, NFS fallback, + cooldown, secret exclusion), `orca` system user setup (P10, REQ-111). +- **docs-engineer**: PHASE-SPECIFIC (P15) — owns README refresh (Q5=A + Nomad-inspired framing, honest-trade-offs table from research doc 3). + Created for P15; removed after phase completes. +- **cli-engineer**: remains DEACTIVATED — CLI surface growth is owned by + lead-developer + backend-engineer. +- **frontend-engineer**: remains DEACTIVATED — no web UI. + +### Phase-specific personas + +- `docs-engineer`: active for P15 only (README refresh). Removed after + phase completes. diff --git a/.ciagent/RESEARCH_v0.11.md b/.ciagent/RESEARCH_v0.11.md new file mode 100644 index 0000000..ea0154f --- /dev/null +++ b/.ciagent/RESEARCH_v0.11.md @@ -0,0 +1,164 @@ +# Research: v0.11 Production Hardening + +## Source material + +Five research documents were ingested 2026-08-07 as directional input +(not verbatim) for v0.11 Phase 0. The current ciagent files +(R-001…R-016, D-001…D-206) are authoritative and take precedence; where +research conflicted, the ciagent files won. The research drove the +adoption of R-017…R-020 and D-215…D-237 (see PROJECT.md, PRD_v0.11.md). + +| Doc | Theme | Adopted as | +|-----|-------|------------| +| 1 | Ingress hybrid (nft DNAT → Traefik on 127.0.0.1:8443) | R-017, D-215..D-226, REQ-099..REQ-102 | +| 2 | Platform-engineer playbook (8 differentiators, TCO, honest trade-offs) | README positioning (Q5=A), CLI surface gap analysis | +| 3 | Strategic positioning ("be Proxmox-for-bare-metal, not K8s-without-K8s") | README framing (Q5=A: Nomad-inspired, honest trade-offs table from doc 3, not Proxmox-first lead) | +| 4 | Drift detection cadence (R-018/R-019/R-020, tiered cadence, hard gate) | R-018, R-019, R-020, D-227..D-237, REQ-103..REQ-113 | +| 5 | Drift detection concrete impl (systemd Path units, orca-drift-notify.sh, orca-remediate.sh) | D-227..D-237 detail, REQ-103..REQ-113 | + +## Thread A — Ingress hardening (doc 1) + +### What changes vs v0.9/v0.10 + +Traefik static config gains `entryPoints.websecure.address: 127.0.0.1:8443` +(default) instead of `:443`. A new nftables emitter renders +`/etc/nftables.d/orca.nft` with DNAT rules. Certs, mTLS, dynamic config, +and the workload SPIFFE validation path are **completely unchanged**. +Only the `address` line shifts + one new emitter + `orca doctor nft` + +`orca nft ...` CLI. + +### Defense in depth + +Two layers: kernel (nftables: SYN flood, rate limit, GeoIP, conntrack) +and application (Traefik: mTLS, SNI, ACL, dynamic routing, health +checks). Neither can replace the other; they catch different attack +classes. + +### Codebase reality (verified 2026-08-07) + +- `internal/emitter/traefik.go` + `traefik_atomic.go` exist (v0.9 P02). + The static-config emitter is where the `address:` line change lands. +- `internal/emitter/systemd.go` exists. New `.path`/`.service` unit + types extend this emitter pattern (shared with drift detection, doc 5). +- `internal/emitter/nft.go` does **not** exist — greenfield, ~200 LoC. +- `scripts/` has `orca-verify-render.sh` but **not** `orca-aggregate.sh`, + `orca-pull.sh`, `orca-apply-render.sh`, `orca-remediate.sh` — all are + v0.11 P09/P10 scope. + +## Thread B — Drift detection + transactional plane (docs 4 + 5) + +### Architecture + +systemd Path units (R-001-clean; systemd is OS, not Orca) watch critical +paths via inotify. On change, a oneshot service computes sha256 and +writes an event JSON to `/etc/orca/state/drift-events/`. The lead's +aggregator timer (10s, C-11) rsyncs these events, validates against the +applied txn manifest, and triggers remediation for auto-remediable +paths. + +### Tiered cadence + +| Tier | Detection | Auto-remediate | Latency | +|------|-----------|----------------|--------| +| Critical | Path unit + 5s polling backstop | yes (config files only) | ~10s | +| Standard | 30s polling | optional (systemd units: require approval) | 30s | +| Default | 60s polling | no | 60s | + +### R-020 hard gate + +Applier refuses new txns if pre-flight consistency check fails. Override: +`--force` flag + per-namespace scoping (Q4=A) — a drifted peer in ns-A +does not block ns-B. + +### Codebase reality (verified 2026-08-07) + +- `internal/store/node_repo.go:80` and `internal/store/job_task_repo.go:82` + already use `iter.Seq[T]`. Doc 5's `iter.Seq2[Event, error]` is the + natural extension per D-017 (settled, shipped v0.3). +- `internal/paths/paths.go:86` has `TxnDir()` — the txn staging dir the + drift detector hooks into. +- `internal/emit/contract.go` has the Go↔bash render-contract anti-drift + (C-16). The *runtime* drift detector (doc 5) is net-new. +- `internal/drift/` package does **not** exist — greenfield, ~500 LoC. +- No `orca` system user creation in code — net-new operational + requirement (REQ-111). +- No NFS detection at peer setup — net-new (REQ-112, D-233). +- `doctor.go` has an OS-drift *check* (one-shot, on-demand) but **not** + a 60s runtime drift-polling loop. Doc 5's design is net-new scope. + +### Alignment with existing gates + +- **C-09** (`orca-pull.sh` failure contract) — R-020 refines + "deterministic state" into an explicit refusal contract. +- **C-11** (lead-side watchdog meta-timer) — doc 5's aggregator + extension is the input C-11 monitors. +- **REQ-075** (lead applier execution model) — doc 5's `orca-remediate.sh` + is literally the same code path as a normal txn-apply, triggered by + drift instead of a new submission. + +## Thread C — Positioning/messaging (docs 2 + 3) + +### Consistent with locked vision + +The vision is *"A minimalist, offline-first, CLI-first orchestration +engine inspired by HashiCorp Nomad"* — explicitly Nomad-inspired, not +K8s. Doc 3's recommendation ("be Proxmox-for-bare-metal, not +K8s-without-the-complexity") is consistent with the locked vision. + +### Where doc 3 diverges (resolved per Q5=A) + +Doc 3 recommends "leading with Proxmox positioning." But R-003 says +"Proxmox can never be lead." Leading the *project identity* with a node +type that can't be the lead is subtly contradictory. **Q5=A decision**: +README uses the Nomad-inspired, OS-as-cluster framing (locked vision), +mentions Proxmox as one node type, and incorporates doc 3's "honest +trade-offs" table but not its Proxmox-first lead-positioning advice. + +### CLI surface gap analysis (doc 2) + +Doc 2's playbook cites ~10 CLI commands. Verified against the live +codebase (`internal/cli/*.go`): + +**Exist today**: `orca init`, `orca node {join,leave,list,key-reset, +capacity}`, `orca job {run,list,stop,logs}`, `orca ns {list,create, +delete,inspect,validate}`, `orca cert {ca-init,gen,show,renew, +fingerprint}`, `orca doctor {cert,network,db,os,proxmox}`, `orca audit +list`, `orca status`, `orca version`, `orca daemon` (deprecated). + +**Not in v0.11 ROADMAP, added per Q2=C**: `orca cluster rotate-lead` +(REQ-114, P14b), `orca upgrade` (REQ-115, P14a), `orca job migrate` +(REQ-116, P05), `orca logs --all-nodes --since` (REQ-117, P06), +`orca doctor mTLS` (REQ-118, P15.5). + +**Already in v0.11 ROADMAP**: `orca node drain` (P05), `orca job lint` +(P11), `orca job verify` (P12), `orca restore` (P07), `orca backup` +(P04). + +### Unverified performance claims in doc 3 + +Doc 3's "10s applier timer = 10,000x slower than K8s informers" and +"60s drift polling" are **forward-looking design constraints**, not +current-state limitations — no applier timer or drift-polling loop +exists in the codebase. These are answered by R-018/R-019/R-020 +(doc 4 + doc 5): the drift detector is a backstop, not the primary +detector, and critical paths get ~10s latency via systemd Path units. + +## Persona assessment + +v0.11 touches these territories: + +| Territory | Persona | Phases | +|-----------|---------|--------| +| `internal/cli/**`, `internal/drift/**`, `internal/nft/**` | backend-engineer | P10, P15.5, P05, P06, P14a, P14b | +| `internal/emitter/**`, `internal/sshpush/**` | backend-engineer + lead-developer | P09, P10, P15.5 | +| `internal/store/**`, migrations | data-engineer | P14a (data migration) | +| `scripts/orca-*.sh` | backend-engineer (bash tooling) | P09, P10 | +| `docs/**`, `README.md`, `examples/**` | lead-developer + docs-engineer (phase-specific) | P15, P08 | +| Threat model, security review, mTLS, secrets | security-engineer | P03, P15.5 | +| nftables, Traefik binding, cluster mesh | network-engineer | P15.5, P09 | +| Test coverage, integration harness | devops-engineer (phase-specific) | P08 | + +No frontend-engineer work (no UI). The data-engineer persona is +reactivated for P14a (v0.8→v1.0 data migration). A docs-engineer custom +persona is created for P15 (README) and P08 (integration test docs). +See `PERSONAS.md` for the updated roster.