docs(P00): create phase plans — 14 phases, 15 REQs, vertical slices

Phase decomposition with wave ordering and persona assignments: - P01: Toolchain vulns (security-engineer, 1 task) - P02: Injection hardening (backend-engineer, 12 tasks) - P03: Scheduler wiring + jobspec parser (lead-developer, 13 tasks) - P04: ACL enforcement + WebAuthn reg auth (backend-engineer, 11 tasks) - P05: Seal/audit CLI + chain race + key zeroing (security-engineer, 11 tasks) - P06: auth init-idp real + auth register (security-engineer, 7 tasks) - P07: Concurrency safety (data-engineer+backend-engineer, 14 tasks) - P08: Transport and SSH safety (backend-engineer, 12 tasks) - P09: Migration and operational safety (data-engineer, 10 tasks) - P10: Observability and metrics (backend-engineer, 4 tasks) - P11: Doc drift round 2 (lead-developer, 14 tasks) - P12: type linux + UAT plan + signoff (lead-developer+uat-engineer, 8 tasks) - P13: Final review + ship + audit (7 tasks) Each phase independently shippable. Vertical slice integrity preserved. ---ci--- project: orca phase: 0 milestone: v0.13 status: plan ---/ci---
2026-08-07 18:44:59 +00:00
parent a2a651e628
commit 82bfab1da3
1 changed files with 426 additions and 0 deletions
@@ -0,0 +1,426 @@
+# PLAN v0.13: Production Hardening Round 2 + UAT Plan
+
+**Status**: complete (2026-08-07). 14 phases (P0 + P01..P12 + P13
+final). Each phase ships a patch tag on the v0.12.x line. This plan
+references requirement IDs from REQUIREMENTS.md and follows the
+vertical-slice integrity rule (each phase is independently shippable).
+
+## Phase 0: Pre-execution (this phase)
+
+**Status**: complete. SPECIFY → CLARIFY → RESEARCH → IDEATE → PLAN →
+GRILL → SHIP. Ships as `v0.12.0`.
+
+## Phase 1: Toolchain & dependency vulns (REQ-149)
+
+**Tag**: `v0.12.1` | **Type**: fix | **Persona**: security-engineer
+
+### Wave 1 (single task)
+- **T1**: Bump `go.mod` from `go 1.25.0` to `go 1.25.12` (or latest
+  1.25.x). Run `go mod tidy`. Run `govulncheck -show verbose ./...` and
+  triage the 6 imported third-party vulns. Bump any dep with a
+  reachable trace (webauthn, cobra, modernc/sqlite, go-jose, coreos/
+  go-oidc, x/crypto, oauth2). Verify `make build && make test && make
+  lint` all pass.
+
+### Must-haves
+- [ ] `go.mod` declares `go 1.25.12`+
+- [ ] `govulncheck ./...` reports zero stdlib vulns with call traces
+- [ ] `make build && make test && make lint` pass
+
+## Phase 2: Input validation & injection hardening (REQ-150)
+
+**Tag**: `v0.12.2` | **Type**: fix | **Persona**: backend-engineer
+
+### Wave 1 (11 sub-fixes, all in `internal/`)
+- **T1**: `orca logs --job` — validate against `^[A-Za-z0-9_-]+$`;
+  replace `fmt.Sprintf("journalctl -u %q", ...)` with `shellQuote`
+  (critical: backtick RCE via SSH fanout)
+- **T2**: pprof `isLoopback(":6060")` — treat empty host as non-
+  loopback/bind-all; reject unless explicit public-allow flag wired;
+  remove phantom `--pprof-allow-public` references; make loopback-only
+  a hard invariant
+- **T3**: backup restore tar-slip — replace `HasPrefix(name, "..")`
+  with `filepath.Rel(target, dest)` containment check
+- **T4**: `orca txn rollback` — validate txn ID against `^T-[0-9a-f]{16}$`
+- **T5**: `orca nft diff --against` — validate txn ID before
+  `filepath.Join`
+- **T6**: `drain stopAlloc` — validate `allocID` against
+  `^[A-Za-z0-9_-]+$` before `systemctl stop`
+- **T7**: `cluster_compat` — `shellQuote(first)` for peer dir name
+- **T8**: `runtime/podman.go` — use `shellQuote(image)` not `%q`
+- **T9**: nft `TrustedProbes` — validate each entry with
+  `net.ParseIP`/`net.ParseCIDR`; fix ipv4/ipv6 mismatch
+- **T10**: sudoers — validate `--proxmox-user`/`--proxmox-role` against
+  `^[a-z_][a-z0-9_-]{0,31}$`; write to fixed `/etc/sudoers.d/orca`;
+  `shellQuote` all pveum/useradd; `validateSudoers` check actual file
+- **T11**: `nft country block add` — validate `^[A-Z]{2}$`
+
+### Wave 2 (tests)
+- **T12**: Add injection/traversal regression tests for each sub-fix;
+  extend `tests/security_integration_test.go` with negative tests
+
+### Must-haves
+- [ ] All 11 injection/traversal vectors fixed with validation
+- [ ] Regression tests for each vector
+- [ ] `tests/security_integration_test.go` passes
+
+## Phase 3: Scheduler/deployment wiring + jobspec parser (REQ-151, REQ-152)
+
+**Tag**: `v0.12.3` | **Type**: feat | **Persona**: lead-developer
+
+### Wave 1 (jobspec parser fixes — REQ-152)
+- **T1**: Add `case "schedule":` and `case "timeout":` to top-level
+  switch in `internal/jobspec/markdown.go`
+- **T2**: Fix DaemonSet — parser must not default `Count` to 1 for
+  DaemonSet (validator rejects `Count != 0`)
+- **T3**: `restart:` policy → systemd `Restart=`/`StartLimitBurst` in
+  `internal/emitter/systemd.go`
+- **T4**: Add `job lint` warnings for advisory-only fields (cron,
+  health, update, affinity) — honest "not enforced in this version"
+
+### Wave 2 (scheduler wiring — REQ-151)
+- **T5**: Wire `internal/scheduler.Schedule()` into `orca job run` —
+  replace local `exec.CommandContext` path with: evaluate constraints/
+  capacity/affinity → render systemd units → SSH-push to target
+- **T6**: `--target` overrides scheduler selection (manual pinning)
+- **T7**: Without `--target`, scheduler bin-packs across `ready` nodes
+- **T8**: Local fallback when no remote nodes registered (single-node
+  dev mode — preserves backward compatibility)
+- **T9**: `systemd-analyze verify` on rendered unit before deploy
+
+### Wave 3 (tests)
+- **T10**: Scheduler constraint/capacity/affinity enforcement tests
+- **T11**: DaemonSet spec passes lint and runs
+- **T12**: `timeout:` on Jobs enforced (kill after duration)
+- **T13**: Local fallback test (no remote nodes)
+
+### Must-haves
+- [ ] `orca job run --target <node>` deploys via SSH-push to remote
+- [ ] Scheduler evaluates constraints/capacity/affinity
+- [ ] DaemonSet works (schedule parsed, Count correct)
+- [ ] `timeout:` enforced on Jobs
+- [ ] `restart:` translated to systemd unit
+- [ ] Local fallback when no remote nodes
+- [ ] `systemd-analyze verify` before deploy
+
+## Phase 4: ACL enforcement + WebAuthn registration auth (REQ-153)
+
+**Tag**: `v0.12.4` | **Type**: fix | **Persona**: backend-engineer
+
+### Wave 1 (ACL wiring)
+- **T1**: Wire `acl.Check` into `dispatch_handler.go` — extract OIDC
+  sub/SPIFFE SVID from mTLS peer cert, check against ACL
+- **T2**: Wire `acl.Check` into `jobs_handler.go`, `nodes_handler.go`,
+  `tasks_handler.go`, `health_handler.go`
+- **T3**: Wire `acl.Check` into `internal/sshpush/` — validate
+  `ORCA_OIDC_TOKEN` bearer against JWKS
+- **T4**: Wire `acl.Check` into `internal/txn/txn.go` apply path
+- **T5**: Thread OIDC sub/SVID into audit `actor` field
+- **T6**: Fix `acl.json` mode 0644→0600
+- **T7**: Add flock on `acl.json` for concurrent grant/revoke
+- **T8**: Bootstrap ACL: grant `cluster-admin` to init cert's SVID
+
+### Wave 2 (WebAuthn registration auth)
+- **T9**: Fix WebAuthn unauthenticated registration — require existing
+  session or admin bootstrap token; no overwriting existing creds
+  without re-auth
+
+### Wave 3 (tests)
+- **T10**: Extend `tests/security_integration_test.go` with deny-by-
+  default enforcement test per handler
+- **T11**: WebAuthn registration auth test (unauthenticated rejected)
+
+### Must-haves
+- [ ] `acl.Check` called in all 5 daemon handlers + sshpush + txn
+- [ ] `acl.json` mode 0600
+- [ ] Audit actor = OIDC sub/SVID
+- [ ] WebAuthn registration requires auth
+- [ ] Bootstrap ACL grants cluster-admin to init SVID
+- [ ] Deny-by-default enforcement tests pass
+
+## Phase 5: Seal/audit CLI + chain race + key zeroing (REQ-154)
+
+**Tag**: `v0.12.5` | **Type**: feat+fix | **Persona**: security-engineer
+
+### Wave 1 (CLI commands)
+- **T1**: Implement `orca cluster seal`/`unseal` (wraps `internal/seal/`;
+  OIDC token exchange; Shamir 3-of-5 shards; sealed blob 0600)
+- **T2**: Implement `orca doctor audit` (wraps `AuditRepo.VerifyChain`)
+- **T3**: Implement `orca doctor modes` (wraps `EnforceFileModes`)
+
+### Wave 2 (fixes)
+- **T4**: Fix audit hash-chain race — `Append` uses `BEGIN IMMEDIATE`
+  transaction
+- **T5**: Fix `secrets rotate-master` to actually re-seal to OIDC
+- **T6**: Zero master key / namespace keys / SVID private keys after
+  use (defense-in-depth)
+
+### Wave 3 (tests)
+- **T7**: Seal→unseal→secrets get round-trip test
+- **T8**: `doctor audit` tamper-detection test
+- **T9**: `doctor modes` 0644-rejection test
+- **T10**: Audit chain concurrent-write integrity test
+- **T11**: Key zeroing verification test
+
+### Must-haves
+- [ ] `orca cluster seal`/`unseal` work (round-trip)
+- [ ] `orca doctor audit` verifies chain
+- [ ] `orca doctor modes` checks file modes
+- [ ] Audit chain survives concurrent appends
+- [ ] `secrets rotate-master` re-seals to OIDC
+- [ ] Keys zeroed after use
+
+## Phase 6: auth init-idp real + auth register (REQ-155)
+
+**Tag**: `v0.12.6` | **Type**: feat | **Persona**: security-engineer
+
+### Wave 1
+- **T1**: Implement `orca auth init-idp` — render Dex systemd unit +
+  config template + Traefik dynamic route from `internal/webauthn/`
+  connector; RP ID = cluster Traefik domain; HTTPS via step-ca cert;
+  atomic deploy with rollback
+- **T2**: Implement `orca auth register` (browser flow to WebAuthn
+  registration endpoint)
+- **T3**: `loadOIDCConfig` config-file loading (`oidc.issuer` in config)
+- **T4**: `orca doctor oidc` health check
+
+### Wave 2 (tests)
+- **T5**: Hermetic Dex+Traefik config render test
+- **T6**: `doctor oidc` health check test
+- **T7**: Virtual-authenticator WebAuthn flow test (C-38)
+
+### Must-haves
+- [ ] `auth init-idp` deploys Dex+Traefik+systemd
+- [ ] `auth register` opens browser flow
+- [ ] `oidc.issuer` loadable from config file
+- [ ] `doctor oidc` health check works
+
+## Phase 7: Concurrency safety (REQ-156)
+
+**Tag**: `v0.12.7` | **Type**: fix | **Persona**: data-engineer + backend-engineer
+
+### Wave 1 (SQLite)
+- **T1**: Add `busy_timeout(5000)` + `SetMaxOpenConns(1)` to all 4 DSNs
+  (store, cache, recovery, webauthn)
+
+### Wave 2 (flocks + locks)
+- **T2**: Secrets file flock (concurrent `secrets set` on same ns)
+- **T3**: Upgrade lock file (refuse concurrent `orca upgrade`)
+- **T4**: Backup lock file
+- **T5**: Cache invalidation by write commands (node join/leave, ns
+  create/delete, job run/stop)
+- **T6**: `Executor.Run` mutex scope fix (hold only for DB inserts)
+- **T7**: `ns create` atomic dir+ns.md write
+- **T8**: `writeCurrentLead` atomic write
+- **T9**: Consolidate 3 divergent `writeAtomic` impls onto
+  `security.WriteAtomic`
+- **T10**: WebAuthn session stores guarded with `sync.Mutex`
+
+### Wave 3 (tests)
+- **T11**: Concurrent secrets set test (no data loss)
+- **T12**: Concurrent upgrade rejection test
+- **T13**: Cache invalidation read-after-write test
+- **T14**: SQLite concurrent writer test (no "database is locked")
+
+### Must-haves
+- [ ] All SQLite DSNs have busy_timeout
+- [ ] Concurrent secrets set preserves all writes
+- [ ] Concurrent upgrade rejected
+- [ ] Cache invalidated by writes (read-after-write consistency)
+- [ ] WebAuthn session stores thread-safe
+
+## Phase 8: Transport & SSH safety (REQ-157)
+
+**Tag**: `v0.12.8` | **Type**: fix | **Persona**: backend-engineer
+
+### Wave 1
+- **T1**: Replace substring matching in `transport.IsTransient` AND
+  `sshpush.isTransient` with typed sentinels (`errors.Is`)
+- **T2**: `rotateSSHKeys` 2-phase atomic swap
+- **T3**: `known_hosts` flock field read by `dial()`
+- **T4**: IPv6 `net.JoinHostPort` in proxmox SSH dial + drain
+  `splitHostPort`
+- **T5**: Explicit timeouts for peer-setup, drift remediate/ack, txn
+  rollback, job restart
+- **T6**: `verifyCutover` use `security.ClientTLSConfig` with orca CA
+- **T7**: OIDC callback server `ReadHeaderTimeout: 5s`
+- **T8**: Root SIGINT/SIGTERM handler for non-watch commands
+
+### Wave 2 (tests)
+- **T9**: Typed-error classification test
+- **T10**: rotate-lead 2-phase with partial-peer failure test
+- **T11**: IPv6 SSH dial test
+- **T12**: Signal handling clean-exit test
+
+### Must-haves
+- [ ] No substring matching in transport retry logic
+- [ ] rotateSSHKeys atomic 2-phase
+- [ ] IPv6 addresses work in SSH dial
+- [ ] All SSH commands have explicit timeouts
+- [ ] SIGINT/SIGTERM triggers clean exit
+
+## Phase 9: Migration & operational safety (REQ-158)
+
+**Tag**: `v0.12.9` | **Type**: fix | **Persona**: data-engineer
+
+### Wave 1
+- **T1**: Migration transaction + torn-write fix
+- **T2**: `job stop` real `systemctl stop` via SSH
+- **T3**: DB retention/compaction for jobs/tasks/audit_log
+- **T4**: `orca logs --lines` cap + `--since` upper bound
+- **T5**: Cache DB mode 0600
+- **T6**: `upgrade.go` cutover backup-file + atomic-rename
+
+### Wave 2 (tests)
+- **T7**: Migration transaction-rollback test
+- **T8**: `job stop` actually-stops test
+- **T9**: DB retention compaction test
+- **T10**: Logs `--lines` cap test
+
+### Must-haves
+- [ ] Migration is transactional + recoverable from torn write
+- [ ] `job stop` sends `systemctl stop` via SSH
+- [ ] DB retention prevents unbounded growth
+- [ ] Logs output is bounded
+
+## Phase 10: Observability & metrics (REQ-159)
+
+**Tag**: `v0.12.10` | **Type**: feat | **Persona**: backend-engineer
+
+### Wave 1
+- **T1**: Add metrics: `orca_jobs_by_state`, `orca_drift_events_total`,
+  `orca_ssh_errors_total`, `orca_txn_apply_total`,
+  `orca_txn_rollback_total`, `orca_acl_denials_total`,
+  `orca_audit_chain_head`
+- **T2**: New `docs/metrics.md` with Prometheus scrape config
+- **T3**: Security headers middleware on daemon
+
+### Wave 2 (tests)
+- **T4**: Metric exposition format + counter increment tests
+
+### Must-haves
+- [ ] 7 new metrics exposed at /metrics
+- [ ] `docs/metrics.md` exists
+- [ ] Security headers set on daemon responses
+
+## Phase 11: Doc drift round 2 (REQ-160)
+
+**Tag**: `v0.12.11` | **Type**: docs | **Persona**: lead-developer
+
+### Wave 1 (README + CHANGELOG)
+- **T1**: README — update status banner, latest tag, subcommand table
+  (add auth/nft/peer-setup/secrets rotate-master), correct "mTLS by
+  default" claim, add missing docs to table
+- **T2**: CHANGELOG regen
+
+### Wave 2 (docs/*)
+- **T3**: `docs/cli.md` — complete rewrite covering all ~40 subcommands
+- **T4**: `docs/webauthn.md` — add `auth register`
+- **T5**: `docs/namespace.md` — add inherit/set-constraint
+- **T6**: `docs/install.md`+`docker.md` — update version refs
+- **T7**: `docs/security-runbook.md` — match P05 reality
+- **T8**: `docs/security-scanning.md` — gosec.json
+
+### Wave 3 (code-level doc fixes)
+- **T9**: Fix `verify-reqs` bold-format regex (bypasses v0.12)
+- **T10**: Fix ROADMAP/REQUIREMENTS v0.12 status hygiene
+- **T11**: `internal/proxmox/bootstrap.go` comments (password→key auth)
+- **T12**: Deprecate `orca status` stub
+- **T13**: Help text fixes (`job run` HCL→markdown, `job stop`
+  daemon→SSH-push)
+- **T14**: `make verify-docs` target (cli.md ↔ `orca --help`)
+
+### Must-haves
+- [ ] README accurate (status, tag, subcommands, claims)
+- [ ] `docs/cli.md` covers all subcommands
+- [ ] `verify-reqs` works for v0.12 and v0.13
+- [ ] `make verify-docs` passes
+
+## Phase 12: --type linux + UAT plan + signoff (REQ-161, REQ-162, REQ-163)
+
+**Tag**: `v0.12.12` | **Type**: feat | **Persona**: lead-developer + uat-engineer
+
+### Wave 1 (--type linux — REQ-161)
+- **T1**: Implement `internal/linux/bootstrap.go` (mirrors Proxmox
+  pattern without PVE role/sudoers)
+- **T2**: Wire `orca node join --type linux --host <ip> --ssh-user root
+  --ssh-key <path>`
+
+### Wave 2 (UAT plan — REQ-162)
+- **T3**: Write `docs/uat.md` — 3-host topology, step-by-step, claim
+  matrix (~35 claims), signoff procedure
+
+### Wave 3 (UAT signoff — REQ-163)
+- **T4**: Write `scripts/uat-signoff.sh` — ~35 named assertions,
+  idempotent, read-only, exit 0 iff all pass
+- **T5**: Write `scripts/uat-smoke.sh` — pure-CLI subset for CI validate
+
+### Wave 4 (tests)
+- **T6**: `--type linux` bootstrap round-trip test (mock SSH)
+- **T7**: `uat-signoff.sh` syntax + assertion-count test
+- **T8**: `uat-smoke.sh` in `.coreci.yml` validate
+
+### Must-haves
+- [ ] `orca node join --type linux` works (SSH bootstrap)
+- [ ] `docs/uat.md` covers 3-host topology + all claims
+- [ ] `scripts/uat-signoff.sh` has ~35 assertions, idempotent
+- [ ] `scripts/uat-smoke.sh` runs in CI
+
+## Phase 13: Final review + ship + audit
+
+**Tag**: `v0.12.13` = v0.13 milestone release | **Type**: chore
+
+### Wave 1
+- **T1**: `ciagent-review` — multi-persona code review across P01..P12
+- **T2**: `ciagent-audit` — reconstruction test, branch hygiene, commit
+  discipline; fix any remaining verify-reqs discrepancies
+- **T3**: Update REQUIREMENTS.md — mark all v0.13 REQs as complete
+- **T4**: Update ROADMAP.md — mark v0.13 as **COMPLETE**
+- **T5**: Merge `phase/13` → `milestone/v0.13` → `main`
+- **T6**: Tag `v0.12.13` (milestone release)
+- **T7**: Create release with full milestone summary
+
+### Must-haves
+- [ ] All v0.13 REQs marked complete in REQUIREMENTS.md
+- [ ] ROADMAP.md marks v0.13 COMPLETE (with bold)
+- [ ] `verify-reqs` passes for v0.12 and v0.13
+- [ ] Milestone merged to main
+- [ ] `v0.12.13` tag created
+- [ ] v1.0.0 NOT cut (deferred for UAT signoff)
+
+## Wave ordering summary
+
+| Phase | Waves | Tasks | Depends on |
+|-------|-------|-------|------------|
+| P01 | 1 | 1 | P0 |
+| P02 | 2 | 12 | P0 |
+| P03 | 3 | 13 | P0 |
+| P04 | 3 | 11 | P0 (P03 for scheduler context) |
+| P05 | 3 | 11 | P0 |
+| P06 | 2 | 7 | P05 (seal) |
+| P07 | 3 | 14 | P0 |
+| P08 | 2 | 12 | P0 |
+| P09 | 2 | 10 | P0 |
+| P10 | 2 | 4 | P04 (acl denials metric), P05 (audit chain head) |
+| P11 | 3 | 14 | P01..P10 (docs reflect reality) |
+| P12 | 4 | 8 | P03 (scheduler for UAT), P04 (ACL for UAT) |
+| P13 | 1 | 7 | P01..P12 |
+
+## Vertical slice integrity
+
+Each phase is independently shippable:
+- P01 (toolchain) — bumps go version, no API change
+- P02 (injection) — validates inputs, no API change
+- P03 (scheduler) — changes `job run` behavior (local→remote), local
+  fallback preserves backward compat
+- P04 (ACL) — adds enforcement, bootstrap ACL prevents lockout
+- P05 (seal) — adds new CLI commands, no breaking change
+- P06 (init-idp) — replaces stub, no breaking change
+- P07 (concurrency) — adds locks/timeouts, no API change
+- P08 (transport) — replaces substring with typed errors, no API change
+- P09 (migration) — fixes migration safety + job stop, job stop is
+  behavioral change (soft→hard stop) — documented
+- P10 (metrics) — adds metrics, no API change
+- P11 (docs) — docs only, no code behavior change
+- P12 (UAT) — adds new command + docs + scripts, no breaking change
+- P13 (final) — review + ship, no new features