From 3f5e5de7299becfdbe4cee9087c425af37f88ebb Mon Sep 17 00:00:00 2001 From: Jon Chery Date: Fri, 7 Aug 2026 18:44:05 +0000 Subject: [PATCH] =?UTF-8?q?docs(P00):=20research=20findings=20=E2=80=94=20?= =?UTF-8?q?threat=20model=20round=203=20(~60=20gaps,=20F26-F101)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three deep codebase sweeps (security, reliability, feature/doc): - Critical: job run runs locally (scheduler dead code), jobspec parser drops schedule/timeout, verify-reqs bypassed, logs --job RCE, pprof bypass, tar-slip, WebAuthn unauthenticated registration - High: 8 injection vectors, Go 1.25.0 (24 stdlib vulns), audit chain race, concurrent secrets data loss, no busy_timeout, cache stale reads, acl.Check zero calls, mTLS claim false, docs missing 25 subcommands - Medium: key zeroing, cache DB mode, writeAtomic consolidation, WebAuthn session mutex, IPv6, SSH timeouts, DB retention, logs unbounded R-022 (scheduler wiring) and R-023 (zero-trust enforcement) adopted as load-bearing architectural changes. ARCHITECTURE.md updated with deltas. PERSONAS.md updated (security-engineer added, uat-engineer phase-specific). ---ci--- project: orca phase: 0 milestone: v0.13 status: research ---/ci--- --- .ciagent/ARCHITECTURE.md | 86 ++++++++++++ .ciagent/CHECKPOINT.json | 2 +- .ciagent/PERSONAS.md | 277 +++++-------------------------------- .ciagent/RESEARCH_v0.13.md | 163 ++++++++++++++++++++++ 4 files changed, 285 insertions(+), 243 deletions(-) create mode 100644 .ciagent/RESEARCH_v0.13.md diff --git a/.ciagent/ARCHITECTURE.md b/.ciagent/ARCHITECTURE.md index 15b2961..344b722 100644 --- a/.ciagent/ARCHITECTURE.md +++ b/.ciagent/ARCHITECTURE.md @@ -786,3 +786,89 @@ The v0.12 milestone is gated by binding conditions C-29..C-38 (see GRILL_v0.12.md). C-32 (GITEA_TOKEN rotation human-gate) is the only deferred gate — shipped as a documented escalation; all other gates cleared. The load-bearing rule is R-021 (no Orca password/token paths). +--- + +## v0.13 Architecture Deltas — Production Hardening Round 2 + +### R-022: Scheduler/Deployment Wiring + +`orca job run` now deploys to remote nodes via the pipeline: +``` +scheduler.Schedule(spec, nodes) → emitter.Render(unit) → sshpush.Deploy(target, unit) +``` +- The local `exec.CommandContext` path in `internal/engine/executor.go` + is removed for the dispatch path. Local execution is the fallback + when no remote nodes are registered (single-node dev mode). +- `internal/scheduler.Schedule()` evaluates CEL constraints, capacity + fit, and affinity scoring against registered nodes. +- `internal/emitter/systemd.go` renders the unit; `systemd-analyze + verify` validates before deploy. +- `internal/sshpush` pushes the unit + env file to the target node. +- `--target ` overrides scheduler selection (manual pinning). +- Without `--target`, the scheduler bin-packs across all `ready` nodes. + +### R-023: Zero-Trust Enforcement Wiring + +`acl.Check` is invoked on every request path: +- **Daemon handlers** (`dispatch`/`jobs`/`nodes`/`tasks`/`health`): + extract OIDC `sub`/SPIFFE SVID from mTLS peer cert → `acl.Check(acl, + identity, namespace, verb)` → deny-by-default. +- **SSH-push applier** (`internal/sshpush/`): validate `ORCA_OIDC_TOKEN` + bearer against JWKS before applying any txn. +- **Txn apply** (`internal/txn/`): same bearer validation. +- Audit `actor` field carries the OIDC `sub` or SPIFFE SVID (not + "cli"/"daemon"). +- `acl.json` mode is 0600 (not 0644). +- WebAuthn registration (`/orca/webauthn/register`) requires an + existing authenticated session or admin bootstrap token. + +### New Components + +- `internal/linux/bootstrap.go` — Ubuntu/Debian SSH-join (mirrors + `internal/proxmox/bootstrap.go` without PVE role/sudoers). Deploys + orca pubkey, creates `orca` system user, creates drift-events dir. + Key-auth only (R-021). Invoked via `orca node join --type linux`. +- `internal/cli/cluster_seal.go` — `orca cluster seal`/`unseal` CLI + (wraps `internal/seal/` library; OIDC token exchange → unwrap master + key → zeroed on shutdown; Shamir 3-of-5 shards at seal time). +- `internal/cli/doctor_audit.go` — `orca doctor audit` (wraps + `AuditRepo.VerifyChain`). +- `internal/cli/doctor_modes.go` — `orca doctor modes` (wraps + `EnforceFileModes` across ORCA_HOME). + +### New Artifacts + +- `docs/uat.md` — UAT plan (3-host topology, step-by-step, claim matrix) +- `scripts/uat-signoff.sh` — v1.0 gate signoff script (~35 assertions, + idempotent, read-only) +- `scripts/uat-smoke.sh` — CI-tested pure-CLI subset of signoff +- `docs/metrics.md` — expanded Prometheus metric set reference + +### jobspec Parser Fixes + +- `schedule:` and `timeout:` now parsed at top level (previously + silently dropped by the markdown parser's default case). +- DaemonSet: parser no longer defaults `Count` to 1 (validator rejects + `Count != 0` for DaemonSet). +- `restart:` policy translated to systemd `Restart=`/`StartLimitBurst` + in the emitter. +- `job lint` emits honest "not enforced in this version" warnings for + advisory-only fields (cron, health, update, affinity). + +### Concurrency Safety + +- All SQLite DSNs set `busy_timeout(5000)` + `SetMaxOpenConns(1)`. +- Secrets file flock prevents concurrent-write data loss. +- Upgrade/backup lock files prevent concurrent cutover/clobber. +- Cache invalidated by write commands (read-after-write consistency). +- Audit `Append` uses `BEGIN IMMEDIATE` transaction (chain race fixed). +- WebAuthn session stores guarded with `sync.Mutex`. + +### Transport Safety + +- Typed sentinels replace substring matching in both `transport` and + `sshpush` packages. +- `rotateSSHKeys` 2-phase atomic swap (stage → swap → verify → cleanup). +- IPv6 `net.JoinHostPort` in all SSH dial paths. +- Explicit timeouts on all SSH commands. +- Root SIGINT/SIGTERM handler for clean exit on non-watch commands. diff --git a/.ciagent/CHECKPOINT.json b/.ciagent/CHECKPOINT.json index 33129b0..7bb7097 100644 --- a/.ciagent/CHECKPOINT.json +++ b/.ciagent/CHECKPOINT.json @@ -1,6 +1,6 @@ { "phase": 0, - "stage": "clarify", + "stage": "research", "milestone": "v0.13", "milestone_slug": "production-hardening-2", "phase_role": "pre_execution", diff --git a/.ciagent/PERSONAS.md b/.ciagent/PERSONAS.md index ba6d61b..e440956 100644 --- a/.ciagent/PERSONAS.md +++ b/.ciagent/PERSONAS.md @@ -4,254 +4,47 @@ active: - backend-engineer - data-engineer - security-engineer - - network-engineer - - devops-engineer deactivated: - cli-engineer - frontend-engineer -phase_specific: [] -reason: | - Orca v0.9 is the first DIRECTION-CHANGE milestone in the project's - history. It supersedes the shipped v0.1–v0.8 architecture per the adopted - PRD (.ciagent/PRD_v0.9.md). The re-architecture deprecates the daemon/ - transport/internal-CA/HCL/single-namespace stack and builds a CLI-only/ - SSH-push/step-ca/Markdown-frontmatter/multi-namespace stack plus 8 - net-new subsystems. The user overrode the grill's Re-architecture - Justification REPLAN with a six-part evidence basis (see PROJECT.md - Supersession Table). The ci-griller's 19 binding conditions (C-01..C-19) - and 10 phase challenges (PC-01..PC-10) are adopted as execution gates - (see GRILL_v0.9.md). - - Roster changes vs v0.8 (implements grill C-05): - - lead-developer: RETAINED — owns the CLI subcommand tree, deprecation - sweep (P00), path resolver (P0a1), parser dispatch (P0b), emitter - interface (P0c), and milestone coordination. - - backend-engineer: RETAINED — owns SSH-push transport (P01), runtime - abstraction (P07a/b/c), transaction bundle (P10 design), step-ca - integration, secrets crypto. Frameworks updated: golang.org/x/crypto/ssh - (existing), golang.org/x/crypto/ssh/knownhosts (existing); pending - deps: bytecodealliance/wasmtime-go (C-01 gate), smallstep/cli (I-B-004). - - data-engineer: RETAINED — owns per-namespace DB schema split (P0a1, - REQ-071), CLI cache DB (R-008), namespace inheritance resolver state - (P0a2). Frameworks: modernc/sqlite. - - security-engineer: REACTIVATED — owns step-ca provisioning (REQ-076), - master.key + AES-256-GCM crypto (REQ-080, C-19 threat model), SPIFFE - SVID minting (C-08 spike), SSH-push blast-radius review, Traefik edge, - .env.secrets threat model. The re-architecture reverses AD-010 - (step-ca rejection) and the SPIFFE rejection at PROJECT.md:94; both - reversals are justified in the Supersession Table. - - network-engineer: REACTIVATED — owns socket-based service exposure - (R-007, P08), Syncthing P2P ports (P09), Traefik routing + dynamic - config atomicity (P02, C-10). The transport layer moves from mTLS - HTTP daemon-to-daemon to SSH CLI-to-server; network-engineer reviews - the new trust surface. - - devops-engineer: REACTIVATED — owns bash scripts (scripts/orca-*.sh, - C-15..C-18: bats/shellcheck/shfmt gate, render-format contract, - slog-syslog), systemd timers (orca-pull/drift/aggregate, C-09 failure - contract, C-11 watchdog), hermetic test infra (P00 bootstrap, P08 - expand, REQ-087). - - cli-engineer: remains DEACTIVATED — CLI surface growth is owned by - lead-developer (cobra subcommands) + backend-engineer (transport); - reactivation optional if CLI subcommand surface exceeds lead-developer - bandwidth. - - frontend-engineer: remains DEACTIVATED — no web UI (unchanged from - v0.1 onward; R-014 makes Markdown canonical, not a web UI). ---- - -# Personas: Orca - -## v0.9 persona assessment (supersedes v0.8) - -The v0.9 re-architecture introduces 5 new external apt dependencies (step-ca, -Traefik, Syncthing, wasmtime, podman), 8 net-new subsystems, and deprecates -~10k lines of shipped daemon/transport/CA/HCL code. The active roster grows -from 3 to 6 to cover the new attack surfaces and deployment model. Territory -enforcement remains in `warn` mode per config.json. - -### lead-developer -- **Domain**: coordination -- **Frameworks**: `cobra`, `net/http/httptest`, `testing` -- **Constraints**: `boundary-enforcement`, `offline-first`, `no-redundant-implementations`, `coverage-floor-70` -- **Territory**: `cmd/**`, `internal/cli/**`, `cmd/verify-reqs/**`, `Makefile`, `.coreci.yml`, `.ciagent/**` -- **Active**: true -- **Reason**: Owns P01 coverage for `cmd/orca` (smoke test of `main()`/`cli.Execute()`), `internal/cli` coverage for the non-node, non-daemon subcommands (`cert *`, `doctor *`, `audit list`, `status`, `version`), and the P03 `cmd/verify-reqs/main.go` Go program + `make verify-reqs` Makefile target + `.coreci.yml` validate-pipeline hook. Added `coverage-floor-70` constraint (D-047 tiered floor: 70% for the 6 under-50% packages, 50% for the 3 zero-test packages). Added `testing` + `net/http/httptest` to frameworks (test-only phase). - -### backend-engineer -- **Domain**: backend -- **Frameworks**: `cobra`, `net/http`, `net/http/httptest`, `golang.org/x/crypto/ssh`, `golang.org/x/crypto/ssh/knownhosts`, `testing` -- **Constraints**: `API-first`, `error-handling`, `minimal-dependencies`, `security-first`, `tofu-host-key-pinning`, `pinned-host-key-fail-closed`, `atomic-file-rewrite`, `coverage-floor-70` -- **Territory**: `internal/transport/**`, `internal/engine/**`, `internal/proxmox/**`, `internal/cli/node.go`, `internal/daemon/**` (tests only) -- **Active**: true -- **Reason**: Owns P01 coverage for `internal/transport` (httptest.NewTLSServer for mTLS + stubDispatcher for DispatchClient) and `internal/engine` (LocalExecutor stubs + PeerRegistry in-memory tests). Owns P02 SSH trust hardening: `--host-key-fingerprint` pinned callback in `internal/proxmox/bootstrap.go` (D-045 OpenSSH SHA256:base64 format, AD-027/AD-028), the TOFU capture-fix (knownhosts.New returns KeyError{Want:[]} on first connect — must capture-and-persist via knownhosts.Line, AD-029 atomic rewrite), the `sessionRunner` seam refactor (P01 enabler for proxmox coverage), and `internal/cli/node.go` `--host-key-fingerprint` flag + `key-reset` subcommand (D-046 local known_hosts only). Frameworks updated: `connectrpc` REMOVED (not in go.mod per AD-014 — config.json still lists it but it's a stale entry), `golang.org/x/crypto/ssh` + `knownhosts` ADDED (direct dep since v0.6 D-030). Added `pinned-host-key-fail-closed` + `atomic-file-rewrite` + `coverage-floor-70` constraints. - -### data-engineer -- **Domain**: data -- **Frameworks**: `modernc/sqlite`, `iter`, `hashicorp/hcl/v2`, `testing` -- **Constraints**: `schema-first`, `migration-safe`, `local-storage-only`, `no-goroutine-leak`, `nullable-column-handling`, `coverage-floor-70` -- **Territory**: `internal/store/**`, `internal/audit/**`, `internal/certpaths/**`, `internal/jobspec/**`, `internal/model/**`, `internal/store/migrations/**` -- **Active**: true -- **Reason**: Owns P01 coverage for `internal/store` (including the missing `cert_repo_test.go` — a v0.7 P01 leftover; Insert/Get/List/ListByNode/LatestForKind/PruneOlderThan/Delete + N=3 rotation history per REQ-025), `internal/audit` (sqlite-backed audit_log row asserts via `engine.Audit` + `store.AuditRepo`, slog capture via test handler), `internal/certpaths` (path-join asserts with temp dir + ORCA_HOME/ORCA_DB env), and `internal/jobspec` (golden-file HCL fixtures in a new `testdata/` dir + error-path table for Parse/Validate/ParseFile). Frameworks updated: `iter` + `hashicorp/hcl/v2` added (matches actual go.mod — jobspec uses hclsimple; store Watch uses iter.Seq). Added `coverage-floor-70` constraint. - -### cli-engineer -- **Active**: false (v0.8) -- **Reason**: Deactivated — merged into lead-developer. The cli coverage work is test-only; `--host-key-fingerprint` and `key-reset` are a 1-flag and 1-subcommand addition to the existing `internal/cli/node.go`, not a new CLI subsystem. - -### security-engineer -- **Active**: false (v0.8) -- **Reason**: Deactivated — v0.8 refines the existing proxmox SSH trust surface (pinned host-key callback, key-reset known_hosts rewrite) but does NOT add new security architecture (no new CA, no new X.509, no new crypto). The trust work is backend-engineer territory (SSH dialer + known_hosts file manipulation). The `internal/security/sshkey.go` is unchanged in v0.8. Was active in v0.6 (SSH keygen + sudoers), deactivated in v0.7, remains deactivated in v0.8. - -### devops-engineer -- **Active**: false (v0.8) -- **Reason**: Deactivated — `verify-reqs` is a Go program (`cmd/verify-reqs/main.go`), not a CI/packaging change. The `.coreci.yml` edit is a 3-line validate-pipeline hook (lead-developer territory). No install.sh, Dockerfile, or release-pipeline surface in v0.8. - -### network-engineer -- **Active**: false (v0.8) -- **Reason**: Deactivated — no transport/mTLS surface change. `internal/transport` coverage is test-only on the existing mTLS layer (httptest.NewTLSServer, no new TLS config). The SSH trust work is point-to-point bootstrap, not the mTLS mesh network-engineer owns. - -### frontend-engineer -- **Active**: false (v0.8) -- **Reason**: No web UI in Orca (unchanged from v0.1 onward). - -## Territory Enforcement - -- **Mode**: `warn` (per `config.json`) -- **Behavior**: Out-of-territory file changes log a warning but do not block. -- **Key overlaps in v0.8** (lead-developer adjudicates): - - `internal/cli/node.go` — backend-engineer (`--host-key-fingerprint` flag + `key-reset` subcommand + proxmox pass-through) vs lead-developer (cli coverage tests). Boundary: backend owns the command implementation; lead owns the test files (`node_test.go`). - - `internal/proxmox/bootstrap.go` — backend-engineer (pinned callback, TOFU fix, sessionRunner seam) vs data-engineer (no overlap — proxmox has no store/audit code). Clean boundary. - - `cmd/verify-reqs/main.go` — lead-developer (Go program + Makefile + .coreci.yml) vs data-engineer (no overlap — verify-reqs parses markdown, not DB). Clean boundary. - - `internal/store/cert_repo_test.go` — data-engineer (test file) vs backend-engineer (no overlap — cert_repo is data territory). Clean boundary. - -## v0.8 vs v0.7 Persona Diff - -| Change | Rationale | -|--------|-----------| -| `lead-developer` retained | Owns cmd/orca smoke test, internal/cli coverage (non-node subcommands), cmd/verify-reqs Go program. | -| `backend-engineer` retained | Owns internal/transport + internal/engine tests + SSH trust-surface in proxmox + cli/node. Frameworks corrected: connectrpc removed (not in go.mod), x/crypto/ssh added. | -| `data-engineer` retained | Owns internal/store (cert_repo gap) + internal/audit + internal/certpaths + internal/jobspec tests. Frameworks corrected: iter + hcl/v2 added. | -| `security-engineer` remains deactivated | v0.8 refines existing SSH trust surface, no new security architecture. | -| `cli-engineer` remains deactivated | Merged into lead-developer (test-only + 1 flag + 1 subcommand). | -| `devops-engineer` remains deactivated | verify-reqs is a Go program, not CI/packaging. | -| `network-engineer` remains deactivated | No transport/mTLS surface change (test-only). | -| `frontend-engineer` remains deactivated | No web UI. | - ---- - -## v0.10 Docs & Install Milestone — Persona Configuration - -```yaml ---- -active: - - lead-developer - - backend-engineer - - docs-engineer -deactivated: - - data-engineer - - security-engineer - network-engineer - devops-engineer - - cli-engineer - - frontend-engineer phase_specific: - - docs-engineer + - uat-engineer (P12 only) reason: | - v0.10 is a documentation + install-hardening milestone. It touches two - territories: scripts/ (release.sh, install.sh — bash, backend-engineer) - and docs/ + examples/ + README.md (markdown, lead-developer + - docs-engineer). No Go orchestration code changes, no schema/migration - changes, no UI, no security/crypto surface, no transport/network - surface. The data-engineer, security-engineer, network-engineer, and - devops-engineer personas are deactivated for this milestone. ---- -``` + Orca v0.13 is a production-hardening milestone. The active roster is + trimmed to the four personas that own the hardening work: + - lead-developer: coordinates phase decomposition, owns scheduler + wiring (R-022) and jobspec parser fixes (P03) + - backend-engineer: owns ACL enforcement wiring (R-023), injection + hardening (P02), transport/SSH safety (P08), concurrency (P07) + - data-engineer: owns SQLite busy_timeout, audit chain race fix, + migration safety, DB retention (P05, P07, P09) + - security-engineer: owns toolchain vulns (P01), seal/audit CLI + (P05), auth init-idp (P06), key zeroing, WebAuthn reg auth (P04) + + network-engineer and devops-engineer are deactivated — their territory + (nft ruleset, collector scripts) is covered by backend-engineer in + this milestone. cli-engineer and frontend-engineer remain deactivated + (no CLI framework or UI work). + + uat-engineer is phase-specific for P12 (UAT plan + signoff script). + + Territory enforcement is warn mode (config.json + personas.territory_enforcement=warn). Cross-territory fixes (e.g. a + fix that touches both daemon handlers and SQLite) are allowed with a + warning. -### lead-developer (v0.10) -- **Active**: true -- **Territory**: `docs/**/*.md`, `examples/**`, `README.md`, - `.ciagent/**/*.md` (coordination + cross-cutting docs) -- **Frameworks**: markdown, cobra (for CLI reference accuracy) -- **Reason**: Owns the CLI reference doc, jobspec reference, ingress - guide, examples directory, README refresh, and namespace.md update. - Coordinates factual accuracy against the live codebase. + Framework alignment (from go.mod): + - lead-developer: cobra + - backend-engineer: cobra, connectrpc + - data-engineer: modernc/sqlite + - security-engineer: go-webauthn, go-jose, x/crypto + - uat-engineer: bash, bats -### backend-engineer (v0.10) -- **Active**: true -- **Territory**: `scripts/release.sh`, `scripts/install.sh`, - `scripts/tests/*.bash` -- **Frameworks**: bash, curl, tea CLI, Gitea API -- **Reason**: Owns the release/install pipeline fix (cross-build amd64, - asset verification, fallback walk). The scripts are API-adjacent - tooling that interacts with the Gitea releases API. - -### docs-engineer (v0.10 — phase-specific) -- **Active**: true (phase-specific: P2, P3, P4) -- **Territory**: `docs/cli.md`, `docs/jobspec.md`, `docs/ingress.md`, - `examples/full-stack/**` -- **Frameworks**: markdown, GitHub-flavored markdown -- **Constraints**: factual-accuracy-against-codebase, - cross-link-resolution, deprecation-callouts -- **Reason**: Custom persona for the markdown authoring work. Ensures - every factual claim in the docs is grounded in the live codebase - (struct fields, flag definitions, paths) and every cross-link - resolves. Removed after P4. - -### Deactivated personas (v0.10) -- **data-engineer**: no schema/migration work this milestone. -- **security-engineer**: no crypto/threat-model work this milestone. -- **network-engineer**: no transport/socket work this milestone. -- **devops-engineer**: no packaging/distribution work beyond the - release.sh fix (owned by backend-engineer). -- **cli-engineer**: no new CLI commands this milestone. -- **frontend-engineer**: no web UI (unchanged from v0.1). - -| Change | Rationale | -|--------|-----------| -| `data-engineer` reactivated | Owns migration 0006 + NodeRepo schema extension (kind/os columns). | -| `security-engineer` reactivated | Owns SSH keygen, TOFU host-key, sudoers, PVE role — first-class security surface. | -| `devops-engineer` deactivated | v0.6 has no packaging/distribution surface. | -| `network-engineer` remains deactivated | No transport/mTLS surface. | -| `frontend-engineer` remains deactivated | No web UI. | -## v0.11 Update (Production Hardening) - -The v0.9 persona roster carries forward to v0.11 with these additions: - -### Roster changes - -- **lead-developer**: RETAINED — owns `orca cluster rotate-lead` (P14b), - `orca upgrade` (P14a), README framing (P15, Q5=A Nomad-inspired), - milestone coordination. -- **backend-engineer**: RETAINED — owns `internal/drift/` (P10, ~500 LoC - greenfield), `internal/emitter/nft.go` (P15.5, ~200 LoC greenfield), - `orca drift` CLI tree (P10), `orca nft` CLI (P15.5), `orca job migrate` - (P05), `orca logs --all-nodes` (P06), `orca doctor mTLS`/`orca doctor nft` - (P15.5), `scripts/orca-drift-notify.sh` + `orca-remediate.sh` (P10). - Frameworks: cobra, `iter.Seq2` (D-017 extension), `signal.NotifyContext` - (D-023), golang.org/x/crypto/ssh (existing). -- **data-engineer**: REACTIVATED for P14a — owns v0.8→v1.0 data migration - (REQ-066), schema migration for `orca upgrade` binding cutover. Was - deactivated in v0.10 (docs-only milestone); reactivated for the - migration phase. -- **security-engineer**: RETAINED — owns threat model (P15.5, C-19), - secrets subsystem (P03), `orca doctor mTLS` (P15.5), ingress-hybrid - trust-boundary review (R-017), drift-detection threat model (R-020 - deadlock, secret exclusion D-234). -- **network-engineer**: RETAINED — owns nftables emitter (P15.5, R-017), - Traefik binding cutover (P14a/P15.5), cross-node cluster mesh (D-219, - unchanged private IP), drift-detection network paths (NFS detection - D-233, SSH fanout for aggregator). -- **devops-engineer**: RETAINED — owns `scripts/orca-aggregate.sh` - extension (P09, D-237), `scripts/orca-drift-notify.sh` (P10), - `scripts/orca-remediate.sh` (P10), systemd Path unit emitter (P10), - drift-detection integration tests (P08: auto-remediation, NFS fallback, - cooldown, secret exclusion), `orca` system user setup (P10, REQ-111). -- **docs-engineer**: PHASE-SPECIFIC (P15) — owns README refresh (Q5=A - Nomad-inspired framing, honest-trade-offs table from research doc 3). - Created for P15; removed after phase completes. -- **cli-engineer**: remains DEACTIVATED — CLI surface growth is owned by - lead-developer + backend-engineer. -- **frontend-engineer**: remains DEACTIVATED — no web UI. - -### Phase-specific personas - -- `docs-engineer`: active for P15 only (README refresh). Removed after - phase completes. + Constraint alignment: + - All personas: offline-first, no-redundant-implementations + - backend-engineer: API-first, error-handling, security-first + - data-engineer: schema-first, migration-safe, local-storage-only + - security-engineer: deny-by-default, zero-trust, no-passwords (R-021) + - uat-engineer: idempotent, read-only, claim-coverage diff --git a/.ciagent/RESEARCH_v0.13.md b/.ciagent/RESEARCH_v0.13.md new file mode 100644 index 0000000..e54ec69 --- /dev/null +++ b/.ciagent/RESEARCH_v0.13.md @@ -0,0 +1,163 @@ +# RESEARCH v0.13: Production Hardening Round 2 — Threat Model & Gap Analysis + +**Status**: complete (2026-08-07). Three deep codebase sweeps (security, +reliability, feature/doc claims) performed via parallel sub-agents. +~60 gaps surfaced beyond v0.12. Findings drive the 15 new requirements +(REQ-149..REQ-163) and 14-phase plan. + +## Methodology + +Three parallel `explore` agents investigated the codebase: +1. **Security sweep** — input validation, injection, SSH, crypto, TLS, + race conditions, SQL, secrets, backup, pprof, rate limiting, memory, + dependencies, toolchain vulns. +2. **Reliability sweep** — idempotency, concurrency, SQLite, partial + failure, SSH fanout, timeouts, systemd, journald, cache, watch + streams, scheduler, capacity, namespace isolation, DB growth, time, + signals, temp files, flock. +3. **Feature/doc sweep** — README claims, docs/*, examples/*, Makefile, + .coreci.yml, CHANGELOG, REQUIREMENTS/ROADMAP consistency, help text, + deprecation warnings, WASM claim. + +Each agent produced a structured report with file:line evidence. This +document synthesizes the findings into the v0.13 plan. + +## Threat Model Round 3 — Findings + +### Critical (must fix in v0.13) + +| ID | Finding | file:line | REQ | +|----|---------|-----------|-----| +| F26 | `orca job run` runs locally via `exec.CommandContext` — scheduler/emitter/SSH-push are dead code; documented deployment model non-functional | `internal/cli/job.go:352-372`, `internal/engine/executor.go:150-180` | REQ-151 | +| F27 | jobspec `schedule:` and `timeout:` silently dropped by markdown parser — DaemonSet fundamentally broken | `internal/jobspec/markdown.go:480-573` | REQ-152 | +| F28 | `verify-reqs` gate bypassed for v0.12 (bold-format regex mismatch) | `cmd/verify-reqs/main.go:29` | REQ-160 | +| F29 | Command injection in `orca logs --job` via `%q`+backtick (RCE via SSH fanout) | `internal/cli/logs.go:283,289` | REQ-150 | +| F30 | pprof loopback bypass via `:6060` (empty host = bind-all) | `internal/daemon/pprof.go:21-29` | REQ-150 | +| F31 | Tar-slip in backup restore (`a/../../etc/passwd` bypasses `HasPrefix(name,"..")`) | `internal/backup/backup.go:302-304` | REQ-150 | +| F32 | Unauthenticated WebAuthn registration (account takeover) | `internal/webauthn/connector.go:85,120` | REQ-153 | +| F33 | ROADMAP marks v0.12 COMPLETE but seal/unseal/init-idp/auth-register don't exist | `.ciagent/ROADMAP.md:403` | REQ-154,155 | + +### High (must fix in v0.13) + +| ID | Finding | file:line | REQ | +|----|---------|-----------|-----| +| F34 | nft ruleset injection via unvalidated `TrustedProbes` IPs | `internal/emitter/nft.go:101-107` | REQ-150 | +| F35 | sudoers/shell injection via `--proxmox-user`/`--proxmox-role` | `internal/proxmox/bootstrap.go:445-452` | REQ-150 | +| F36 | `validateSudoers` checks wrong filename when `ProxmoxUser != "orca"` | `internal/proxmox/bootstrap.go:474` | REQ-150 | +| F37 | `orca txn rollback` shell injection via unvalidated txn ID | `internal/cli/txn.go:240-241` | REQ-150 | +| F38 | `orca nft diff --against` path traversal | `internal/cli/nft.go:225` | REQ-150 | +| F39 | `drain stopAlloc` stored injection from compromised peer | `internal/cli/drain.go:132` | REQ-150 | +| F40 | `cluster_compat` stored injection from peer | `internal/cli/cluster_compat.go:399` | REQ-150 | +| F41 | podman `image` `%q` backtick injection | `internal/runtime/podman.go:67` | REQ-150 | +| F42 | Go toolchain 1.25.0 — 24 stdlib vulns (tar, tls, x509, http, pem...) | `go.mod:3` | REQ-149 | +| F43 | No SQLite `busy_timeout` — "database is locked" under concurrency | `internal/store/store.go:21` | REQ-156 | +| F44 | Audit hash-chain race — concurrent appends corrupt tamper-evidence | `internal/store/audit_repo.go:908-919` | REQ-154 | +| F45 | Concurrent `secrets set` silently loses data (no flock) | `internal/cli/secrets.go:135-148` | REQ-156 | +| F46 | Concurrent `orca upgrade` races on Traefik cutover + binary install | `internal/cli/upgrade.go:111` | REQ-156 | +| F47 | Cache never invalidated by writes — stale reads after join/create/run | `internal/cli/cache.go:763-770` | REQ-156 | +| F48 | `acl.Check` called zero times — v0.12 zero-trust not wired | `internal/daemon/`, `internal/sshpush/` | REQ-153 | +| F49 | `acl.json` mode 0644 (should be 0600 per REQ-145) | `internal/cli/acl.go:152` | REQ-153 | +| F50 | README "mTLS by default" is false — SSH-push is canonical, mTLS deprecated | `README.md`, `internal/cli/node.go:93-98` | REQ-160 | +| F51 | `docs/cli.md` missing ~25 subcommands; CHANGELOG stale at v0.1 | `docs/cli.md:4`, `CHANGELOG.md:9-32` | REQ-160 | +| F52 | `docs/security-runbook.md` documents seal/unseal/doctor audit that don't exist | `docs/security-runbook.md:5-11,23` | REQ-160 | +| F53 | `docs/webauthn.md` documents `orca auth register` that doesn't exist | `docs/webauthn.md:13` | REQ-155,160 | +| F54 | `auth init-idp` is a stub — v0.12 R-021 load-bearing change has no working IdP | `internal/cli/auth.go:151-155` | REQ-155 | +| F55 | `secrets rotate-master` writes raw key, doesn't re-seal to OIDC | `internal/cli/secrets.go:358` | REQ-154 | +| F56 | `orca cluster seal`/`unseal` documented but not implemented | `docs/security-runbook.md:3-9` | REQ-154 | +| F57 | `orca doctor audit` documented but not implemented | `docs/security-runbook.md:18` | REQ-154 | +| F58 | `orca doctor modes` not implemented (REQ-130) | `internal/security/ca.go:236` | REQ-154 | +| F59 | Audit actor field is "cli"/"daemon" not OIDC sub/SVID | `internal/cli/drain.go`, `internal/daemon/server.go` | REQ-153 | +| F60 | `Executor.Run` holds mutex for whole job duration | `internal/engine/executor.go:101-103` | REQ-156 | +| F61 | `splitHostPort` in drain.go breaks IPv6 addresses | `internal/cli/drain.go:68-74` | REQ-157 | +| F62 | `transport.IsTransient` + `sshpush.isTransient` both use substring matching | `internal/transport/retry.go:44`, `internal/sshpush/transport.go:395-414` | REQ-157 | +| F63 | `rotateSSHKeys` partial-result window (old key overwritten before all peers updated) | `internal/cli/rotate_lead.go:132` | REQ-157 | +| F64 | `known_hosts` flock field stored but not read by `dial()` | `internal/sshpush/transport.go:60-63` | REQ-157 | +| F65 | `verifyCutover` uses default http.Client against orca CA (will fail TLS verification) | `internal/cli/upgrade.go:313-314` | REQ-157 | +| F66 | v0.8→v0.11 migration torn-write window (crash after rename, before schema fixup) | `internal/migration/migrate.go:135-140` | REQ-158 | +| F67 | `job stop` is soft-stop only (doesn't signal process) | `internal/cli/job.go:266` | REQ-158 | +| F68 | `upgrade.go` cutover uses direct `sed -i` (no backup file) | `internal/cli/upgrade.go:performCutover` | REQ-158 | +| F69 | `nft country block add` validates length but not content; uses `%q` | `internal/cli/nft.go:136,259` | REQ-150 | +| F70 | `--type linux` reserved but unimplemented | `internal/model/node.go:29` | REQ-161 | +| F71 | No UAT/E2E test doc exists | repo-wide | REQ-162,163 | + +### Medium (fix in v0.13) + +| ID | Finding | file:line | REQ | +|----|---------|-----------|-----| +| F72 | Master/SVID keys never zeroed from memory after use | throughout `internal/secrets/`, `internal/seal/` | REQ-154 | +| F73 | Cache DB mode 0644 (not 0600) | `internal/cache/cache.go:61-64` | REQ-158 | +| F74 | `writeAtomic0600`/collector: predictable tmp, no cleanup, leaks | `internal/identity/oidc.go:134`, `internal/cli/collector.go:179` | REQ-156 | +| F75 | `cli/acl.go writeAtomicFile` no fsync (durability gap) | `internal/cli/acl.go:161-181` | REQ-156 | +| F76 | WebAuthn session stores unsynchronized global maps (data race) | `internal/webauthn/connector.go:67,171` | REQ-156 | +| F77 | `loadOIDCConfig` TODO for config-file loading | `internal/cli/auth.go:168` | REQ-155 | +| F78 | No retention/compaction for jobs/tasks/audit_log tables | `internal/store/` | REQ-158 | +| F79 | `orca logs` no `--lines` cap, `--since` unbounded (OOM risk) | `internal/cli/logs.go:173-185` | REQ-158 | +| F80 | `ns create` non-atomic (partial dir creation on mid-failure) | `internal/cli/ns.go:906-918` | REQ-156 | +| F81 | `writeCurrentLead` non-atomic `os.WriteFile` | `internal/cli/rotate_lead.go:315-322` | REQ-156 | +| F82 | `secrets set` doesn't validate namespace exists (creates phantom ns) | `internal/cli/secrets.go:130` | REQ-156 | +| F83 | `backup` has no lock; concurrent backups may clobber | `internal/cli/backup.go:42-68` | REQ-156 | +| F84 | Root command has no SIGINT/SIGTERM handler for non-watch commands | `cmd/orca/main.go:17-22` | REQ-157 | +| F85 | SSH commands without explicit timeouts (peer-setup, drift, txn rollback, job restart) | various | REQ-157 | +| F86 | Rendered systemd units never validated (`systemd-analyze verify`) before deploy | `internal/emitter/systemd.go:80-98` | REQ-151 | +| F87 | OIDC callback HTTP server has no timeouts (slowloris) | `internal/identity/oidc.go:244` | REQ-157 | +| F88 | No security headers on daemon TLS surface | `internal/daemon/health.go:93` | REQ-159 | +| F89 | `orca status` returns hardcoded v0.1 stub, not deprecated | `internal/cli/status.go:22` | REQ-160 | +| F90 | `job run` help text says "HCL spec file" but HCL is deprecated | `internal/cli/job.go:47-48` | REQ-160 | +| F91 | README subcommand table omits `auth`, `nft`, `peer-setup` | `README.md` | REQ-160 | +| F92 | `docs/namespace.md` omits `inherit`/`set-constraint` | `docs/namespace.md:114-134` | REQ-160 | +| F93 | README "latest tag: v0.10.19" is stale (actual: v0.11.29) | `README.md:30,39` | REQ-160 | +| F94 | `docs/install.md`+`docker.md` reference stale v0.4.x and deprecated daemon | `docs/install.md:42,62`, `docs/docker.md:21,43` | REQ-160 | +| F95 | IPv6 host not bracketed in proxmox SSH dial | `internal/proxmox/bootstrap.go:140` | REQ-157 | + +### Low (fix in v0.13 where cheap, document otherwise) + +| ID | Finding | file:line | REQ | +|----|---------|-----------|-----| +| F96 | `--pprof-allow-public` documented but never implemented | `internal/daemon/pprof.go:37,42,43` | REQ-150 | +| F97 | `nft country block add` weak code validation | `internal/cli/nft.go:136` | REQ-150 | +| F98 | `cert show`/`fingerprint` don't emit deprecation warnings | `internal/cli/cert.go` | REQ-160 | +| F99 | `docs/namespace.md` references `orca doctor --legacy-paths` that doesn't exist | `docs/namespace.md:165` | REQ-160 | +| F100 | `release.sh` only builds linux-amd64; install.sh advertises arm64 | `scripts/release.sh:94-102` | accepted (D-193) | +| F101 | `docs/cli.md` version example shows "v0.9.1" but default is "0.1.0-dev" | `docs/cli.md:253` | REQ-160 | + +## CLEAN categories (verified, no new findings) + +- **SQL injection in `internal/store/`** — all queries use `?` placeholders +- **TLS version/cipher policy** — TLS 1.3 only, AEAD cipher allowlist +- **SSH key generation** — Ed25519, `crypto/rand`, PKCS8, 0600 +- **TOFU host-key pinning** — fail-closed on mismatch, constant-time comparison +- **Self-signed cert generation** — RSA 3072, 128-bit serial, correct KeyUsage +- **Nonce reuse in secrets** — fresh 12-byte nonce per line from `crypto/rand` +- **Gitleaks / secrets in git history** — only test fixtures +- **Secrets logged in errors** — only keys/namespaces logged, never values +- **CSRF on HTTP surfaces** — daemon is GET-only, no state-changing GETs +- **Watch streams (iter.Seq)** — pull-based, defer cleanup, no goroutine leak +- **DNS resolution** — bounded by `net.Dialer{Timeout: 15s}` +- **Multi-namespace DB isolation** — per-ns file layout + +## Accepted residual risks (documented, not fixed) + +1. OIDC tokens plaintext at rest (0600) — sealing on every CLI invocation conflicts with "no orca binary on servers" model +2. HSTS on daemon — mTLS-only API, no browser-facing surface +3. DNS resolution timeout — bounded by `net.Dialer{Timeout: 15s}` +4. Temp file cleanup on SIGKILL — orphaned temp files, operator-visible +5. Flock timeout on NFS — stuck holder is rare; `tryFlockEx` exists +6. "WASM-first" pillar aspirational — document as "WASM runtime available, process is default" +7. arm64/armv7 release — D-193 deferred; install.sh detection is forward-looking +8. OIDC callback slowloris — loopback, short-lived, single CLI invocation +9. `--pprof-allow-public` flag — remove references, make loopback-only a hard invariant + +## Architecture updates (for ARCHITECTURE.md) + +- **R-022**: `orca job run` deploys via scheduler → emitter → SSH-push (local exec path removed) +- **R-023**: Zero-trust enforcement wired (`acl.Check` on every request path) +- New component: `internal/linux/bootstrap.go` (Ubuntu/Debian SSH-join, mirrors Proxmox pattern) +- New artifact: `docs/uat.md` + `scripts/uat-signoff.sh` (v1.0 gate) +- New artifact: `docs/metrics.md` (expanded Prometheus metric set) + +## Conclusion + +Three deep sweeps found ~60 gaps. v0.13 closes all critical/high/medium +(REQ-149..REQ-163, 14 phases). 9 low-severity residual risks are +documented and accepted. This is the last hardening round. v1.0.0 is +gated on the UAT signoff script delivered by P12.