# PLAN v0.13: Production Hardening Round 2 + UAT Plan

**Status**: complete (2026-08-07). 14 phases (P0 + P01..P12 + P13
final). Each phase ships a patch tag on the v0.12.x line. This plan
references requirement IDs from REQUIREMENTS.md and follows the
vertical-slice integrity rule (each phase is independently shippable).

## Phase 0: Pre-execution (this phase)

**Status**: complete. SPECIFY → CLARIFY → RESEARCH → IDEATE → PLAN →
GRILL → SHIP. Ships as `v0.12.0`.

## Phase 1: Toolchain & dependency vulns (REQ-149)

**Tag**: `v0.12.1` | **Type**: fix | **Persona**: security-engineer

### Wave 1 (single task)
- **T1**: Bump `go.mod` from `go 1.25.0` to `go 1.25.12` (or latest
  1.25.x). Run `go mod tidy`. Run `govulncheck -show verbose ./...` and
  triage the 6 imported third-party vulns. Bump any dep with a
  reachable trace (webauthn, cobra, modernc/sqlite, go-jose, coreos/
  go-oidc, x/crypto, oauth2). Verify `make build && make test && make
  lint` all pass.

### Must-haves
- [ ] `go.mod` declares `go 1.25.12`+
- [ ] `govulncheck ./...` reports zero stdlib vulns with call traces
- [ ] `make build && make test && make lint` pass

## Phase 2: Input validation & injection hardening (REQ-150)

**Tag**: `v0.12.2` | **Type**: fix | **Persona**: backend-engineer

### Wave 1 (11 sub-fixes, all in `internal/`)
- **T1**: `orca logs --job` — validate against `^[A-Za-z0-9_-]+$`;
  replace `fmt.Sprintf("journalctl -u %q", ...)` with `shellQuote`
  (critical: backtick RCE via SSH fanout)
- **T2**: pprof `isLoopback(":6060")` — treat empty host as non-
  loopback/bind-all; reject unless explicit public-allow flag wired;
  remove phantom `--pprof-allow-public` references; make loopback-only
  a hard invariant
- **T3**: backup restore tar-slip — replace `HasPrefix(name, "..")`
  with `filepath.Rel(target, dest)` containment check
- **T4**: `orca txn rollback` — validate txn ID against `^T-[0-9a-f]{16}$`
- **T5**: `orca nft diff --against` — validate txn ID before
  `filepath.Join`
- **T6**: `drain stopAlloc` — validate `allocID` against
  `^[A-Za-z0-9_-]+$` before `systemctl stop`
- **T7**: `cluster_compat` — `shellQuote(first)` for peer dir name
- **T8**: `runtime/podman.go` — use `shellQuote(image)` not `%q`
- **T9**: nft `TrustedProbes` — validate each entry with
  `net.ParseIP`/`net.ParseCIDR`; fix ipv4/ipv6 mismatch
- **T10**: sudoers — validate `--proxmox-user`/`--proxmox-role` against
  `^[a-z_][a-z0-9_-]{0,31}$`; write to fixed `/etc/sudoers.d/orca`;
  `shellQuote` all pveum/useradd; `validateSudoers` check actual file
- **T11**: `nft country block add` — validate `^[A-Z]{2}$`

### Wave 2 (tests)
- **T12**: Add injection/traversal regression tests for each sub-fix;
  extend `tests/security_integration_test.go` with negative tests

### Must-haves
- [ ] All 11 injection/traversal vectors fixed with validation
- [ ] Regression tests for each vector
- [ ] `tests/security_integration_test.go` passes

## Phase 3: Scheduler/deployment wiring + jobspec parser (REQ-151, REQ-152)

**Tag**: `v0.12.3` | **Type**: feat | **Persona**: lead-developer

### Wave 1 (jobspec parser fixes — REQ-152)
- **T1**: Add `case "schedule":` and `case "timeout":` to top-level
  switch in `internal/jobspec/markdown.go`
- **T2**: Fix DaemonSet — parser must not default `Count` to 1 for
  DaemonSet (validator rejects `Count != 0`)
- **T3**: `restart:` policy → systemd `Restart=`/`StartLimitBurst` in
  `internal/emitter/systemd.go`
- **T4**: Add `job lint` warnings for advisory-only fields (cron,
  health, update, affinity) — honest "not enforced in this version"

### Wave 2 (scheduler wiring — REQ-151)
- **T5**: Wire `internal/scheduler.Schedule()` into `orca job run` —
  replace local `exec.CommandContext` path with: evaluate constraints/
  capacity/affinity → render systemd units → SSH-push to target
- **T6**: `--target` overrides scheduler selection (manual pinning)
- **T7**: Without `--target`, scheduler bin-packs across `ready` nodes
- **T8**: Local fallback when no remote nodes registered (single-node
  dev mode — preserves backward compatibility)
- **T9**: `systemd-analyze verify` on rendered unit before deploy

### Wave 3 (tests)
- **T10**: Scheduler constraint/capacity/affinity enforcement tests
- **T11**: DaemonSet spec passes lint and runs
- **T12**: `timeout:` on Jobs enforced (kill after duration)
- **T13**: Local fallback test (no remote nodes)

### Must-haves
- [ ] `orca job run --target <node>` deploys via SSH-push to remote
- [ ] Scheduler evaluates constraints/capacity/affinity
- [ ] DaemonSet works (schedule parsed, Count correct)
- [ ] `timeout:` enforced on Jobs
- [ ] `restart:` translated to systemd unit
- [ ] Local fallback when no remote nodes
- [ ] `systemd-analyze verify` before deploy

## Phase 4: ACL enforcement + WebAuthn registration auth (REQ-153)

**Tag**: `v0.12.4` | **Type**: fix | **Persona**: backend-engineer

### Wave 1 (ACL wiring)
- **T1**: Wire `acl.Check` into `dispatch_handler.go` — extract OIDC
  sub/SPIFFE SVID from mTLS peer cert, check against ACL
- **T2**: Wire `acl.Check` into `jobs_handler.go`, `nodes_handler.go`,
  `tasks_handler.go`, `health_handler.go`
- **T3**: Wire `acl.Check` into `internal/sshpush/` — validate
  `ORCA_OIDC_TOKEN` bearer against JWKS
- **T4**: Wire `acl.Check` into `internal/txn/txn.go` apply path
- **T5**: Thread OIDC sub/SVID into audit `actor` field
- **T6**: Fix `acl.json` mode 0644→0600
- **T7**: Add flock on `acl.json` for concurrent grant/revoke
- **T8**: Bootstrap ACL: grant `cluster-admin` to init cert's SVID

### Wave 2 (WebAuthn registration auth)
- **T9**: Fix WebAuthn unauthenticated registration — require existing
  session or admin bootstrap token; no overwriting existing creds
  without re-auth

### Wave 3 (tests)
- **T10**: Extend `tests/security_integration_test.go` with deny-by-
  default enforcement test per handler
- **T11**: WebAuthn registration auth test (unauthenticated rejected)

### Must-haves
- [ ] `acl.Check` called in all 5 daemon handlers + sshpush + txn
- [ ] `acl.json` mode 0600
- [ ] Audit actor = OIDC sub/SVID
- [ ] WebAuthn registration requires auth
- [ ] Bootstrap ACL grants cluster-admin to init SVID
- [ ] Deny-by-default enforcement tests pass

## Phase 5: Seal/audit CLI + chain race + key zeroing (REQ-154)

**Tag**: `v0.12.5` | **Type**: feat+fix | **Persona**: security-engineer

### Wave 1 (CLI commands)
- **T1**: Implement `orca cluster seal`/`unseal` (wraps `internal/seal/`;
  OIDC token exchange; Shamir 3-of-5 shards; sealed blob 0600)
- **T2**: Implement `orca doctor audit` (wraps `AuditRepo.VerifyChain`)
- **T3**: Implement `orca doctor modes` (wraps `EnforceFileModes`)

### Wave 2 (fixes)
- **T4**: Fix audit hash-chain race — `Append` uses `BEGIN IMMEDIATE`
  transaction
- **T5**: Fix `secrets rotate-master` to actually re-seal to OIDC
- **T6**: Zero master key / namespace keys / SVID private keys after
  use (defense-in-depth)

### Wave 3 (tests)
- **T7**: Seal→unseal→secrets get round-trip test
- **T8**: `doctor audit` tamper-detection test
- **T9**: `doctor modes` 0644-rejection test
- **T10**: Audit chain concurrent-write integrity test
- **T11**: Key zeroing verification test

### Must-haves
- [ ] `orca cluster seal`/`unseal` work (round-trip)
- [ ] `orca doctor audit` verifies chain
- [ ] `orca doctor modes` checks file modes
- [ ] Audit chain survives concurrent appends
- [ ] `secrets rotate-master` re-seals to OIDC
- [ ] Keys zeroed after use

## Phase 6: auth init-idp real + auth register (REQ-155)

**Tag**: `v0.12.6` | **Type**: feat | **Persona**: security-engineer

### Wave 1
- **T1**: Implement `orca auth init-idp` — render Dex systemd unit +
  config template + Traefik dynamic route from `internal/webauthn/`
  connector; RP ID = cluster Traefik domain; HTTPS via step-ca cert;
  atomic deploy with rollback
- **T2**: Implement `orca auth register` (browser flow to WebAuthn
  registration endpoint)
- **T3**: `loadOIDCConfig` config-file loading (`oidc.issuer` in config)
- **T4**: `orca doctor oidc` health check

### Wave 2 (tests)
- **T5**: Hermetic Dex+Traefik config render test
- **T6**: `doctor oidc` health check test
- **T7**: Virtual-authenticator WebAuthn flow test (C-38)

### Must-haves
- [ ] `auth init-idp` deploys Dex+Traefik+systemd
- [ ] `auth register` opens browser flow
- [ ] `oidc.issuer` loadable from config file
- [ ] `doctor oidc` health check works

## Phase 7: Concurrency safety (REQ-156)

**Tag**: `v0.12.7` | **Type**: fix | **Persona**: data-engineer + backend-engineer

### Wave 1 (SQLite)
- **T1**: Add `busy_timeout(5000)` + `SetMaxOpenConns(1)` to all 4 DSNs
  (store, cache, recovery, webauthn)

### Wave 2 (flocks + locks)
- **T2**: Secrets file flock (concurrent `secrets set` on same ns)
- **T3**: Upgrade lock file (refuse concurrent `orca upgrade`)
- **T4**: Backup lock file
- **T5**: Cache invalidation by write commands (node join/leave, ns
  create/delete, job run/stop)
- **T6**: `Executor.Run` mutex scope fix (hold only for DB inserts)
- **T7**: `ns create` atomic dir+ns.md write
- **T8**: `writeCurrentLead` atomic write
- **T9**: Consolidate 3 divergent `writeAtomic` impls onto
  `security.WriteAtomic`
- **T10**: WebAuthn session stores guarded with `sync.Mutex`

### Wave 3 (tests)
- **T11**: Concurrent secrets set test (no data loss)
- **T12**: Concurrent upgrade rejection test
- **T13**: Cache invalidation read-after-write test
- **T14**: SQLite concurrent writer test (no "database is locked")

### Must-haves
- [ ] All SQLite DSNs have busy_timeout
- [ ] Concurrent secrets set preserves all writes
- [ ] Concurrent upgrade rejected
- [ ] Cache invalidated by writes (read-after-write consistency)
- [ ] WebAuthn session stores thread-safe

## Phase 8: Transport & SSH safety (REQ-157)

**Tag**: `v0.12.8` | **Type**: fix | **Persona**: backend-engineer

### Wave 1
- **T1**: Replace substring matching in `transport.IsTransient` AND
  `sshpush.isTransient` with typed sentinels (`errors.Is`)
- **T2**: `rotateSSHKeys` 2-phase atomic swap
- **T3**: `known_hosts` flock field read by `dial()`
- **T4**: IPv6 `net.JoinHostPort` in proxmox SSH dial + drain
  `splitHostPort`
- **T5**: Explicit timeouts for peer-setup, drift remediate/ack, txn
  rollback, job restart
- **T6**: `verifyCutover` use `security.ClientTLSConfig` with orca CA
- **T7**: OIDC callback server `ReadHeaderTimeout: 5s`
- **T8**: Root SIGINT/SIGTERM handler for non-watch commands

### Wave 2 (tests)
- **T9**: Typed-error classification test
- **T10**: rotate-lead 2-phase with partial-peer failure test
- **T11**: IPv6 SSH dial test
- **T12**: Signal handling clean-exit test

### Must-haves
- [ ] No substring matching in transport retry logic
- [ ] rotateSSHKeys atomic 2-phase
- [ ] IPv6 addresses work in SSH dial
- [ ] All SSH commands have explicit timeouts
- [ ] SIGINT/SIGTERM triggers clean exit

## Phase 9: Migration & operational safety (REQ-158)

**Tag**: `v0.12.9` | **Type**: fix | **Persona**: data-engineer

### Wave 1
- **T1**: Migration transaction + torn-write fix
- **T2**: `job stop` real `systemctl stop` via SSH
- **T3**: DB retention/compaction for jobs/tasks/audit_log
- **T4**: `orca logs --lines` cap + `--since` upper bound
- **T5**: Cache DB mode 0600
- **T6**: `upgrade.go` cutover backup-file + atomic-rename

### Wave 2 (tests)
- **T7**: Migration transaction-rollback test
- **T8**: `job stop` actually-stops test
- **T9**: DB retention compaction test
- **T10**: Logs `--lines` cap test

### Must-haves
- [ ] Migration is transactional + recoverable from torn write
- [ ] `job stop` sends `systemctl stop` via SSH
- [ ] DB retention prevents unbounded growth
- [ ] Logs output is bounded

## Phase 10: Observability & metrics (REQ-159)

**Tag**: `v0.12.10` | **Type**: feat | **Persona**: backend-engineer

### Wave 1
- **T1**: Add metrics: `orca_jobs_by_state`, `orca_drift_events_total`,
  `orca_ssh_errors_total`, `orca_txn_apply_total`,
  `orca_txn_rollback_total`, `orca_acl_denials_total`,
  `orca_audit_chain_head`
- **T2**: New `docs/metrics.md` with Prometheus scrape config
- **T3**: Security headers middleware on daemon

### Wave 2 (tests)
- **T4**: Metric exposition format + counter increment tests

### Must-haves
- [ ] 7 new metrics exposed at /metrics
- [ ] `docs/metrics.md` exists
- [ ] Security headers set on daemon responses

## Phase 11: Doc drift round 2 (REQ-160)

**Tag**: `v0.12.11` | **Type**: docs | **Persona**: lead-developer

### Wave 1 (README + CHANGELOG)
- **T1**: README — update status banner, latest tag, subcommand table
  (add auth/nft/peer-setup/secrets rotate-master), correct "mTLS by
  default" claim, add missing docs to table
- **T2**: CHANGELOG regen

### Wave 2 (docs/*)
- **T3**: `docs/cli.md` — complete rewrite covering all ~40 subcommands
- **T4**: `docs/webauthn.md` — add `auth register`
- **T5**: `docs/namespace.md` — add inherit/set-constraint
- **T6**: `docs/install.md`+`docker.md` — update version refs
- **T7**: `docs/security-runbook.md` — match P05 reality
- **T8**: `docs/security-scanning.md` — gosec.json

### Wave 3 (code-level doc fixes)
- **T9**: Fix `verify-reqs` bold-format regex (bypasses v0.12)
- **T10**: Fix ROADMAP/REQUIREMENTS v0.12 status hygiene
- **T11**: `internal/proxmox/bootstrap.go` comments (password→key auth)
- **T12**: Deprecate `orca status` stub
- **T13**: Help text fixes (`job run` HCL→markdown, `job stop`
  daemon→SSH-push)
- **T14**: `make verify-docs` target (cli.md ↔ `orca --help`)

### Must-haves
- [ ] README accurate (status, tag, subcommands, claims)
- [ ] `docs/cli.md` covers all subcommands
- [ ] `verify-reqs` works for v0.12 and v0.13
- [ ] `make verify-docs` passes

## Phase 12: --type linux + UAT plan + signoff (REQ-161, REQ-162, REQ-163)

**Tag**: `v0.12.12` | **Type**: feat | **Persona**: lead-developer + uat-engineer

### Wave 1 (--type linux — REQ-161)
- **T1**: Implement `internal/linux/bootstrap.go` (mirrors Proxmox
  pattern without PVE role/sudoers)
- **T2**: Wire `orca node join --type linux --host <ip> --ssh-user root
  --ssh-key <path>`

### Wave 2 (UAT plan — REQ-162)
- **T3**: Write `docs/uat.md` — 3-host topology, step-by-step, claim
  matrix (~35 claims), signoff procedure

### Wave 3 (UAT signoff — REQ-163)
- **T4**: Write `scripts/uat-signoff.sh` — ~35 named assertions,
  idempotent, read-only, exit 0 iff all pass
- **T5**: Write `scripts/uat-smoke.sh` — pure-CLI subset for CI validate

### Wave 4 (tests)
- **T6**: `--type linux` bootstrap round-trip test (mock SSH)
- **T7**: `uat-signoff.sh` syntax + assertion-count test
- **T8**: `uat-smoke.sh` in `.coreci.yml` validate

### Must-haves
- [ ] `orca node join --type linux` works (SSH bootstrap)
- [ ] `docs/uat.md` covers 3-host topology + all claims
- [ ] `scripts/uat-signoff.sh` has ~35 assertions, idempotent
- [ ] `scripts/uat-smoke.sh` runs in CI

## Phase 13: Final review + ship + audit

**Tag**: `v0.12.13` = v0.13 milestone release | **Type**: chore

### Wave 1
- **T1**: `ciagent-review` — multi-persona code review across P01..P12
- **T2**: `ciagent-audit` — reconstruction test, branch hygiene, commit
  discipline; fix any remaining verify-reqs discrepancies
- **T3**: Update REQUIREMENTS.md — mark all v0.13 REQs as complete
- **T4**: Update ROADMAP.md — mark v0.13 as **COMPLETE**
- **T5**: Merge `phase/13` → `milestone/v0.13` → `main`
- **T6**: Tag `v0.12.13` (milestone release)
- **T7**: Create release with full milestone summary

### Must-haves
- [ ] All v0.13 REQs marked complete in REQUIREMENTS.md
- [ ] ROADMAP.md marks v0.13 COMPLETE (with bold)
- [ ] `verify-reqs` passes for v0.12 and v0.13
- [ ] Milestone merged to main
- [ ] `v0.12.13` tag created
- [ ] v1.0.0 NOT cut (deferred for UAT signoff)

## Wave ordering summary

| Phase | Waves | Tasks | Depends on |
|-------|-------|-------|------------|
| P01 | 1 | 1 | P0 |
| P02 | 2 | 12 | P0 |
| P03 | 3 | 13 | P0 |
| P04 | 3 | 11 | P0 (P03 for scheduler context) |
| P05 | 3 | 11 | P0 |
| P06 | 2 | 7 | P05 (seal) |
| P07 | 3 | 14 | P0 |
| P08 | 2 | 12 | P0 |
| P09 | 2 | 10 | P0 |
| P10 | 2 | 4 | P04 (acl denials metric), P05 (audit chain head) |
| P11 | 3 | 14 | P01..P10 (docs reflect reality) |
| P12 | 4 | 8 | P03 (scheduler for UAT), P04 (ACL for UAT) |
| P13 | 1 | 7 | P01..P12 |

## Vertical slice integrity

Each phase is independently shippable:
- P01 (toolchain) — bumps go version, no API change
- P02 (injection) — validates inputs, no API change
- P03 (scheduler) — changes `job run` behavior (local→remote), local
  fallback preserves backward compat
- P04 (ACL) — adds enforcement, bootstrap ACL prevents lockout
- P05 (seal) — adds new CLI commands, no breaking change
- P06 (init-idp) — replaces stub, no breaking change
- P07 (concurrency) — adds locks/timeouts, no API change
- P08 (transport) — replaces substring with typed errors, no API change
- P09 (migration) — fixes migration safety + job stop, job stop is
  behavioral change (soft→hard stop) — documented
- P10 (metrics) — adds metrics, no API change
- P11 (docs) — docs only, no code behavior change
- P12 (UAT) — adds new command + docs + scripts, no breaking change
- P13 (final) — review + ship, no new features

## Grill binding conditions (C-44..C-49) — incorporated

| ID | Condition | Phase affected | How addressed |
|----|-----------|----------------|---------------|
| C-44 | P03 MUST fail-closed when scheduler selects a node but SSH-push fails. Local fallback only when `len(registeredNodes)==0`. Test case mandatory. | P03 | Added to P03 must-haves + T13 test |
| C-45 | P04 MUST implement log-only/dry-run mode as default for first invocation after ACL wiring. Enforce mode after bootstrap ACL verified. | P04 | Added T9.5 (log-only mode) + T11.5 (enforce-mode toggle) to P04 |
| C-46 | P12 dependency table MUST include P05 (seal) and P06 (auth init-idp) in addition to P03 and P04. | P12 | Updated dependency table above |
| C-47 | P12 `uat-signoff.sh` MUST include explicit assertions for: (a) job deployed to remote node, (b) ACL deny-by-default, (c) seal/unseal round-trip, (d) OIDC health check. | P12 | Added to P12 must-haves + assertion list in docs/uat.md |
| C-48 | P12 `docs/uat.md` MUST document hardware prerequisites (Proxmox VE 8/9 host required). Alternative UAT path (3x Ubuntu, `--type linux` only, Proxmox claims skipped) MUST be documented. | P12 | Added to P12 T3 scope |
| C-49 | Plan narrative MUST soften "last hardening round" to "last hardening round before UAT validation." | P0/P13 | Updated PROJECT.md + ROADMAP.md narrative |

### Updated P03 must-haves (C-44)
- [ ] P03 fails-closed when scheduler selects a node but SSH-push fails (returns error, no silent local fallback)
- [ ] Local fallback ONLY when `len(registeredNodes)==0`
- [ ] Test case for SSH-push failure → error (not silent local)

### Updated P04 task list (C-45)
- **T9.5**: Implement log-only/dry-run mode as default for first invocation after ACL wiring (log denials, do not block)
- **T11.5**: Enforce mode after bootstrap ACL verified (toggle via `orca acl enforce` or config)

### Updated P12 dependencies (C-46)
- P12 depends on: P03 (scheduler), P04 (ACL), P05 (seal), P06 (auth init-idp)

### Updated P12 must-haves (C-47, C-48)
- [ ] `uat-signoff.sh` asserts: job deployed to remote node (node_id != localhost)
- [ ] `uat-signoff.sh` asserts: ACL deny-by-default (denial logged)
- [ ] `uat-signoff.sh` asserts: seal/unseal round-trip
- [ ] `uat-signoff.sh` asserts: OIDC health check
- [ ] `docs/uat.md` documents Proxmox VE 8/9 hardware prerequisite
- [ ] `docs/uat.md` documents alternative UAT path (3x Ubuntu, Proxmox claims skipped)

### Updated narrative (C-49)
v0.13 is the "last hardening round **before UAT validation**." The UAT
will likely surface 3-7 issues requiring a patch release. v1.0.0 is
deferred until UAT passes.
