orca/.ciagent/PROJECT.md

Project: Orca

What This Is

A minimalist, offline-first, CLI-first orchestration engine inspired by HashiCorp Nomad, prioritizing stability, security, and simplicity over feature richness. Single-binary distribution, no container runtime, no cloud dependencies, no K8s-level complexity.

Vision

A minimalist, offline-first, CLI-first orchestration engine inspired by HashiCorp Nomad, prioritizing stability, security, and simplicity over feature richness.

Objective

Build a lightweight system to manage and execute workloads across a set of nodes, keeping complexity far below that of Kubernetes.

Requirements

• CLI First: Primary interaction through a CLI tool.
• Offline First: Functional without constant internet connectivity.
• AI First: Designed to be easily discoverable and manageable by AI agents.
• Stability & Security: Prioritize security fixes and bug fixes over new features.
• Language: Written in Go 1.25+.
• Simplicity: Minimalist implementation, avoiding the "K8s complexity trap".

Constraints

• No web UI as a primary requirement.
• Must not implement K8s-level complexity.
• Feature development must move slowly to ensure stability.
• Only CI system allowed: CoreCI (git.cloudinit.dev/coreci/coreci).
• Gitea remote: git.cloudinit.dev/coreci/orca.

Clarified Decisions (D-series, full autonomy)

ID	Question	Decision	Rationale	Confidence
D-001	Single binary or multi-binary distribution?	Single binary	Simpler distribution; subcommands baked into one orca binary. Aligns with simplicity pillar.	0.95
D-002	Local state store technology?	modernc/sqlite (pure Go, CGO-free)	Cross-compile friendly, no CGO dependency, single file on disk, mature.	0.92
D-003	Inter-node communication?	Embedded HTTP (net/http) over loopback, mTLS for cross-node	No external RPC framework needed for v0.1. HTTP suffices.	0.85
D-004	Scheduling algorithm for v0.1?	Single-node only (no scheduling)	Multi-node scheduling is out of scope for v0.1. Tasks run on the node they're submitted to.	0.90
D-005	CLI output format?	Human-readable by default, --json flag for machine consumption	Serves both humans and AI agents.	0.95
D-006	Job/task definition format?	HCL or YAML in .hcl/.yaml files	Familiar to Nomad/HashiCorp users; simpler than JSON for humans.	0.88
D-186	Bash scripts coverage gate: count toward Go gate or exempt?	Exempt from Go coverage gate; compensating control: bats tests (C-15) + shellcheck + shfmt in CI; every script must have >=1 happy-path and >=1 failure-path bats test	Bash is a different language surface from Go; the 70%/50% Go coverage gate (D-042/D-047) is Go-specific. Forcing bash into the Go gate would require a coverage tool that does not exist for bash. The compensating control (bats + shellcheck + shfmt) provides equivalent discipline.	0.82
D-007	Authentication?	mTLS for v0.1, token-based deferred	mTLS is the most secure default. Tokens can be added later if needed.	0.80
D-008	Container runtime?	Direct process execution (no container runtime) for v0.1	Avoids the Docker/container dependency. Pure process management.	0.85
D-009	Configuration file location?	~/.orca/config.hcl and /etc/orca/orca.hcl	Standard XDG-style paths.	0.90
D-010	Logging format?	Structured JSON via log/slog	Native Go 1.21+ slog, no external dependency.	0.95
D-011	v0.2 mTLS cert authority model?	Internal CA with CSR join	One node bootstraps a local CA; peers generate CSRs and submit them to the CA for signing. CA cert is the trust anchor. More secure than self-signed per-node (single trust root) without the operational complexity of an external PKI.	0.92
D-012	v0.2 CA bootstrap & cert distribution?	Operator-mediated, fingerprint-verified	Bootstrap node writes ~/.orca/ca.crt and ~/.orca/ca.key (mode 0600). Operator copies ca.crt to peers; peers verify by SHA-256 fingerprint at orca node join --ca-fingerprint <sha256>. No automated secret distribution.	0.85
D-013	v0.2 cert validity & rotation?	Server certs 90 days, CA cert 10 years, rotate 30 days before expiry	Server certs are short-lived (compromise window small); CA is long-lived (manual rotation is expensive). orca cert renew reissues server certs automatically.	0.90
D-014	v0.2 mTLS handshake timing?	Eager — at orca node join time	Fail fast on bad certs, misconfigurations, or CA mismatches. Lazy handshake would let stale configs run until first request, complicating debugging.	0.88
D-015	v0.2 minimum TLS version & cipher suites?	TLS 1.3 only; AEAD cipher allowlist	MinVersion=tls.VersionTLS13, CipherSuites limited to TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_AES_128_GCM_SHA256. No TLS 1.2 fallback.	0.92
D-016	v0.2 security-scanning placement?	validate pipeline of .coreci.yml, gates merges to main	gosec baseline JSON checked into repo; new findings fail the build. govulncheck ./... exit-on-known. Pre-commit hook with gitleaks is opt-in (developer machine).	0.85
D-017	v0.2 iter.Seq API surface?	orca job list --watch and orca node list --watch	Pull-based iter.Seq[Job] / iter.Seq[Node]; cancellation via context.Context; signal.NotifyContext on ctrl-c. Backpressure is implicit (consumer-driven).	0.90
D-018	v0.2 multi-node scheduling algorithm?	Bin-packing by available CPU/memory, FIFO within a node	Simple, deterministic, matches D-004 minimalism. Cross-node dispatch via ConnectRPC orca.v1.Dispatch service. Retry on transient failures with exponential backoff.	0.85

Out of Scope

• Full-blown Kubernetes-compatible API.
• Complex cloud-provider integrations.
• GUI-based management consoles.
• Container runtime integration.
• Service mesh / sidecar injection.
• Auto-scaling / horizontal pod autoscaler.
• External PKI / Let's Encrypt / cert transparency logs.
• gRPC framework dependency (ConnectRPC in config.json frameworks but not in go.mod; v0.2 uses stdlib net/http with h2c for orca.v1.Dispatch — see ARCHITECTURE.md AD-014).

v0.2 Scope Summary

v0.2 is a focused 4-phase milestone that turns Orca from a single-node process executor into a small cluster engine with strong transport security and richer I/O. The 4 phases are:

• P01 — mTLS handshake + internal CA with CSR join. Internal CA, CSR join, eager handshake at node join, TLS 1.3 + AEAD allowlist. See ARCHITECTURE.md Flow 1 + Flow 2.
• P02 — Multi-node scheduling & job dispatch. Best-fit bin-packing by CPU/memory, FIFO within a node, orca.v1.Dispatch over mTLS. See ARCHITECTURE.md Flow 3.
• P03 — gosec + govulncheck + gitleaks in CI. gosec baseline JSON in repo, govulncheck ./... in validate pipeline, gitleaks in pre-commit (opt-in).
• P04 — iter.Seq streaming for --watch flags. Go 1.25+ range-over-func semantics, context.Context cancellation, signal.NotifyContext on ctrl-c. See ARCHITECTURE.md Flow 4.

The vision ("minimalist, offline-first, CLI-first orchestration engine") is unchanged. v0.2 is a hardening + small-cluster extension, not a direction change.

Key Decisions

The 18 D-series decisions (D-001..D-018) are recorded in the "Clarified Decisions" table above. The 10 v0.1 decisions (D-001..D-010) are stable and unchanged in v0.2. The 8 v0.2 decisions (D-011..D-018) were auto-resolved under full autonomy and are summarized here:

• D-011: Internal CA with CSR join (vs. self-signed per-node or SPIFFE). Single trust root, no external PKI, CSR workflow.
• D-012: Operator-mediated CA cert distribution with fingerprint verify (no automated secret distribution — matches offline-first principle).
• D-013: 90d server certs, 10y CA cert, 30d pre-expiry rotation.
• D-014: Eager mTLS handshake at orca node join time (fail fast).
• D-015: TLS 1.3 only, AEAD cipher allowlist (no TLS 1.2 fallback).
• D-016: gosec+govulncheck in validate pipeline of .coreci.yml (gates merges to main). gitleaks in pre-commit (opt-in).
• D-017: iter.Seq for orca job list --watch and orca node list --watch (pull-based, ctx cancellation, ctrl-c via signal.NotifyContext).
• D-018: Bin-packing by CPU/memory with FIFO within node; JSON-over-HTTP orca.v1.Dispatch for cross-node (no ConnectRPC dep).

v0.3 Clarified Decisions (D-series, full autonomy)

v0.3 is a lean 2-execution-phase milestone completing the streaming and doctor work deferred from v0.2. The 6 v0.3 decisions (D-019..D-024) were auto-resolved under full autonomy:

ID	Question	Decision	Rationale	Confidence
D-019	Watch refresh mechanism?	Poll-based, 1s ticker	Simpler than event channel; no daemon coupling for CLI; matches offline-first.	0.90
D-020	Watch output format (REQ-030)?	Table by default; --watch --json streams one-line JSON per event	Consistent with D-005 --json convention; serves humans + AI agents.	0.92
D-021	Doctor network check scope?	Probe configured peer addresses via mTLS /healthz handshake; PASS/WARN/FAIL per peer	Reuses existing transport client; read-only.	0.85
D-022	Doctor db check scope?	PRAGMA integrity_check + migration version query	Already specced in ARCHITECTURE.md §5; minimal surface.	0.95
D-023	iter.Seq cancellation?	signal.NotifyContext on SIGINT/SIGTERM	Per D-017 + ARCHITECTURE Flow 4.	0.95
D-024	--watch applies to job list only, or node list too?	Both orca job list --watch and orca node list --watch	Per ARCHITECTURE.md CLI layer + D-017.	0.92

v0.3 Scope Summary

v0.3 is a focused 2-execution-phase milestone completing the work deferred from v0.2 that was NOT already shipped in P08-P10. A codebase audit during re-init SPECIFY confirmed that REQ-014, REQ-027, REQ-028, REQ-029, REQ-031, REQ-037, REQ-039, REQ-040 all shipped in P08-P10 despite stale REQUIREMENTS.md marking them Pending. The remaining work:

• P01 — iter.Seq streaming for --watch flags. Go 1.25+ range-over-func semantics, pull-based iter.Seq[Job] / iter.Seq[Node], context.Context cancellation, signal.NotifyContext on ctrl-c. Applies to both orca job list --watch and orca node list --watch. Covers REQ-022, REQ-030.
• P02 — orca doctor network + db full implementation. Replaces the P01 stubs (NetworkStub, DBStub) with real checks: peer reachability via mTLS /healthz probe; SQLite PRAGMA integrity_check + migration version. Covers REQ-032 (completion).

The vision ("minimalist, offline-first, CLI-first orchestration engine") is unchanged. v0.3 is a completion milestone, not a direction change.

v0.5 Scope Summary — Distribution

v0.5 is a 3-execution-phase milestone that makes Orca installable, distributable, and containerized. The engine functionality from v0.1–v0.3 is unchanged; this milestone is purely about delivery surface:

• P01 — Namespace unification. A single ORCA_HOME environment variable becomes the namespace root for all on-disk state (db, certs, init, daemon). A --system flag on the root command selects the system-level namespace root /root/.orca. Backward compatible: empty ORCA_HOME → ~/.orca. Covers REQ-041, REQ-042.
• P02 — install.sh + in-place update. A 1-liner installer pulls the release binary from the public Gitea release URL, installs at user level by default (~/.local/bin/orca) or system level (/usr/local/bin/orca) with --system. Re-running updates the binary in place while preserving config/db/certs in the namespace dir. Idempotent. Covers REQ-043, REQ-044. Also updates README quickstart (REQ-016 completion).
• P03 — Docker release. A multi-stage Dockerfile builds a distroless image; scripts/release.sh and .coreci.yml publish the image to the Gitea container registry per release. Covers REQ-046.
• P04 — Final review + ship + audit. Milestone release.

The vision ("minimalist, offline-first, CLI-first orchestration engine") is unchanged. v0.5 is a distribution milestone, not a direction change.

v0.5 Clarified Decisions (D-series, full autonomy)

The 5 v0.5 decisions (D-025..D-029) were auto-resolved under full autonomy during the CLARIFY stage:

ID	Question	Decision	Rationale	Confidence
D-025	System-level namespace path layout?	/root/.orca (mirror of user-level ~/.orca)	Consistent shape with user-level; just a different root. Matches the user's "starts at /root" wording. Single dir keeps it simple.	0.90
D-026	Namespace override mechanism at runtime?	Unify on ORCA_HOME as single namespace root for all components (db, certs, init, daemon). Add --system flag that sets root to /root/.orca.	ORCA_HOME already exists for certs; extend to all components. Backward compatible (empty → ~/.orca). One knob, not many.	0.92
D-027	Docker registry target?	Gitea built-in container registry (git.cloudinit.dev/coreci/orca)	Keeps everything in one forge; uses Gitea's native registry. Consistent with REQ-045 (public repo → public image pulls).	0.88
D-028	How to make releases publicly accessible (REQ-045)?	Flip repo visibility to public via tea repos edit coreci/orca --private=false during P0 ship	Simplest path to anonymous downloads; enables both install.sh pulls and docker pulls. Pre-existing .env leak already suppressed via gitleaks baseline + rotate-forward (commit 00127ce).	0.85
D-029	install.sh default version?	Latest release (query Gitea releases API), optional --version vX.Y.Z to pin	Matches typical 1-liner installer UX; users get newest by default, can pin for reproducibility.	0.92

v0.5 Operational prerequisite (P0 ship)

The Gitea repo coreci/orca is currently private (returns 404 unauthenticated). P0 ship flips visibility to public via tea repos edit coreci/orca --private=false so that install.sh can pull release binaries unauthenticated (REQ-045). This is an operational step performed during the P0 ship, verified by an unauth curl against the releases API.

v0.6 Scope Summary — Node Bootstrap & Proxmox

v0.6 is a 3-execution-phase milestone that turns orca init from a bare mkdir into a full single-node cluster bootstrap, and adds Proxmox 8 & 9 as a first-class remote node type joined over SSH with least-privilege role delegation. The engine functionality from v0.1–v0.5 is unchanged; this milestone is about bootstrap ergonomics and heterogeneous node support:

• P01 — orca init full bootstrap. A single orca init call now: (a) creates the namespace dir (~/.orca or /root/.orca with --system); (b) runs all DB migrations including the new 0006 (nodes.kind, nodes.os — backward-compatible nullable columns); (c) bootstraps the internal CA via security.CAInit if ca.crt is absent; (d) generates the server cert via security.GenerateCSR + ca.SignCSR if server.crt is absent; (e) auto-detects the local OS via /etc/os-release ID= field (ubuntu/debian/alpine); (f) registers a localhost node with kind=localhost, os=<detected>, addr=localhost:8443 if no localhost node exists yet. After orca init, orca doctor MUST pass with zero FAILs. Idempotent: re-running orca init is a no-op (or refresh) for already-provisioned artifacts. Covers REQ-047, REQ-048, REQ-049.
• P02 — Proxmox SSH join. orca node join --type proxmox --host <addr> --user root --password <pw> (password via flag or $ORCA_PROXMOX_PASSWORD, never persisted) bootstraps a remote Proxmox 8/9 host via golang.org/x/crypto/ssh (new direct dep). Steps: (1) SSH password-auth; (2) generate or load orca's SSH keypair (~/.orca/orca_ssh_key / .pub, 0600/0644); (3) deploy pubkey to remote ~orca/.ssh/authorized_keys; (4) create orca user (config-overridable name via --proxmox-user, default orca); (5) create PVE custom role OrcaOperator (config-overridable via --proxmox-role) with privileges VM.Audit, Datastore.AllocateSpace, SDN.Use; (6) assign role to orca user on /; (7) drop /etc/sudoers.d/orca allowlist (pct, qm, pvesh, apt-get, dpkg — no shell-escape commands); (8) record node row kind=proxmox, os=pve, audit log. Idempotent re-run. Covers REQ-050, REQ-051.
• P03 — doctor os + doctor proxmox. Extends orca doctor with two new checks: doctor os re-runs /etc/os-release detection and verifies it matches the stored localhost node row's os field (drift = WARN); doctor proxmox iterates kind=proxmox nodes and SSH-probes each with pveversion / pvecmd status (3s timeout per peer per D-038 pattern), reporting PASS/WARN/FAIL per node. All bootstrap + join actions emit structured audit-log entries. Covers REQ-052.
• P04 — Final review + ship + audit. Milestone release.

The vision ("minimalist, offline-first, CLI-first orchestration engine") is unchanged. v0.6 is a bootstrap-ergonomics + heterogeneous- nodes milestone, not a direction change.

v0.6 Clarified Decisions (D-series, full autonomy)

The 8 v0.6 decisions (D-030..D-037) were resolved during the CLARIFY stage — D-030..D-034 confirmed by the operator in plan mode, D-035..D-037 auto-resolved at full autonomy within the clarify_budget:

ID	Question	Decision	Rationale	Confidence
D-030	SSH library for Proxmox join?	golang.org/x/crypto/ssh	Stdlib-adjacent, well-maintained, single new direct dep. Matches orca's minimal-deps ethos. Shell-out to /usr/bin/ssh would require openssh-client on the orca host and complicate password-auth + idempotent pubkey deploy.	0.92 (operator-confirmed)
D-031	Proxmox join password handling?	Flag/env only, never persisted	--password flag or $ORCA_PROXMOX_PASSWORD is used once to deploy the orca pubkey + create the orca user; the password is never written to SQLite. Subsequent orca→Proxmox access uses the deployed SSH key.	0.95 (operator-confirmed)
D-032	Localhost OS auto-detect signal?	/etc/os-release ID= field	Parse ID= from /etc/os-release; map ubuntu/debian/alpine → node os. Falls back to linux (unknown) if none match. Simplest reliable signal across the three target distros.	0.93 (operator-confirmed)
D-033	Least-privilege Proxmox role granularity?	Custom PVE role OrcaOperator with VM.Audit, Datastore.AllocateSpace, SDN.Use + /etc/sudoers.d/orca allowlist (pct, qm, pvesh, apt-get, dpkg)	Config-overridable role + user names. Sufficient for "manage the host, VMs/CTs, storage, packages" without granting root shell. Built-in PVEAuditor is too read-only; full Administrator is too broad.	0.88 (operator-confirmed)
D-034	Node kind/os schema?	Add nodes.kind + nodes.os columns via migration 0006	Schema-first, queryable, doctor can branch on kind. Nullable with localhost/"" defaults for existing rows (backward-compatible). data-engineer owns the migration.	0.94 (operator-confirmed)
D-035	SSH host-key verification on first Proxmox connect?	TOFU: pin on first connect, refuse on mismatch thereafter	First connect uses ssh.InsecureIgnoreHostKey to capture the host key; it is then persisted to ~/.orca/known_hosts (or the nodes metadata) and all subsequent connects require a match. Balances first-run ergonomics against MITM risk on subsequent runs. Switching to pre-pinned keys is a future enhancement.	0.82 (auto)
D-036	orca init idempotency semantics for already-provisioned artifacts?	Skip-and-refresh, never overwrite	If ca.crt exists → load it (no regen). If server.crt exists → keep it (no reissue). If a localhost node row exists → update last_seen + re-detect os, never insert a duplicate. If DB migrations are ahead → no-op. If ~/.orca exists → MkdirAll is a no-op. Idempotent re-run is a hard requirement (REQ-047).	0.95 (auto)
D-037	orca SSH keypair location + algorithm?	~/.orca/orca_ssh_key (0600) + ~/.orca/orca_ssh_key.pub (0644), Ed25519	Ed25519 keys are smaller, faster, and more secure than RSA for SSH auth. Stored in the orca namespace dir alongside ca.crt/server.crt so ORCA_HOME relocation works. File modes mirror the cert file-mode discipline (REQ-033 spirit). Generated lazily on first orca node join --type proxmox, not at orca init (localhost doesn't need SSH).	0.90 (auto)

v0.6 clarification notes

• D-035 TOFU caveat: TOFU (trust-on-first-use) is the standard SSH UX and matches the operator-mediated model from D-012 (CA cert distribution). The operator is expected to verify the host key fingerprint out-of-band on first connect if the network is untrusted. A future milestone may add --host-key-fingerprint pin flag to orca node join --type proxmox for pre-pinned deployments.
• D-036 idempotency: re-running orca init on a node that already has a localhost row updates last_seen and re-detects os (in case the host OS was upgraded) but does NOT change the node ID or joined_at. This makes orca init safe to put in a systemd ExecStartPre or a config-management runbook.
• D-037 Ed25519: golang.org/x/crypto/ssh + golang.org/x/crypto/ed25519 are in the same module; no additional direct dep beyond D-030.

v0.7 Clarified Decisions (D-series, full autonomy)

The 5 v0.7 decisions (D-038..D-042) were auto-resolved at full autonomy within the clarify_budget (10):

ID	Question	Decision	Rationale	Confidence
D-038	Config file format — HCL or YAML?	HCL	D-009 already specced config.hcl. HCL is already a direct dep (hashicorp/hcl/v2 for jobspec). Adding YAML would introduce a second parser dep — violates minimal-deps. Use the existing hclparse pkg from jobspec.	0.93
D-039	Config precedence order (flag vs env vs file vs default)?	flag > env > file > default	Standard layered config: the most explicit (flag) wins, then the runtime (env), then the persisted (file), then the built-in default. Matches cobra/viper convention without the viper dep.	0.92
D-040	pprof security — bind to localhost only, or operator-chosen addr?	Operator-chosen --pprof <addr> (default disabled)	Default disabled keeps the minimalist posture. Operator picks the addr — localhost for dev, unix socket for prod. Separate mux so it never touches the mTLS daemon listener. No auth (pprof is operator-only, addr is the gate).	0.85
D-041	cert command registration — where in root command order?	After cert is unreachable today, append after node in rootCmd.AddCommand order	Alphabetical-ish with the existing cluster (audit, daemon, doctor, init, job, node, cert, status, version). No behavior change to existing commands.	0.88
D-042	Coverage target — 50% floor or higher?	50% floor per package, 70% target for new packages	50% is achievable for the concurrent packages (engine, transport) without heroic mock effort; 70% is the floor for new code in P02/P04. Avoids a "raise coverage everywhere" rathole.	0.85

v0.7 Scope Summary — Hardening & Completion

v0.7 is a 4-execution-phase NFR milestone that closes out gaps surfaced by the v0.7 IDEATE stage: an unreachable command tree, a missing config file layer, low test coverage in core packages, and the long-deferred pprof endpoint. The engine functionality from v0.1–v0.6 is unchanged; this milestone is purely about correctness, coverage, and operability:

• P01 — Register orca cert command tree + cert_repo tests. The internal/cli/cert.go command (cert ca-init, cert gen, cert show, cert renew, cert fingerprint) is fully implemented but never wired into rootCmd. This phase adds the missing rootCmd.AddCommand(newCertCmd(...)) and adds the missing internal/store/cert_repo_test.go. Covers REQ-053.
• P02 — HCL config file parsing (config.hcl). D-009 specified ~/.orca/config.hcl and /etc/orca/orca.hcl as config locations, but no HCL config-file parser exists — the CLI relies entirely on flags and env vars. This phase adds a minimal internal/config package that loads config.hcl (keys: db_path, listen_addr, ca_path, server_cert_path, server_key_path, node_capacity), merges with env/flag overrides (flag > env > file > default), and surfaces it via --config flag on the root command. Covers REQ-054.
• P03 — Test coverage uplift. Adds tests for the lowest-coverage packages: internal/engine (executor, dispatcher, peer — currently 8.3%), internal/transport (mtls, dispatch, handshake_log — currently 26.3%), internal/proxmox (bootstrap SSH path — currently 5.1%), and internal/audit (no tests). Target: every package ≥ 50% coverage. Covers REQ-055.
• P04 — --pprof opt-in on orca daemon. Adds the long-deferred I-308 pprof endpoint behind an opt-in --pprof <addr> flag (default disabled). net/http/pprof mounted on a separate mux so it never touches the mTLS daemon listener. Covers REQ-056.
• P05 — Final review + ship + audit. Milestone release.

The vision ("minimalist, offline-first, CLI-first orchestration engine") is unchanged. v0.7 is a hardening milestone, not a direction change. Milestone type: NFR (all phases are fix/test/chore); the final phase's progressive patch IS the deliverable per run.md versioning logic. Tags run on the v0.6.x patch line: v0.6.0 (P0) … v0.6.5 (P05 = milestone release).

v0.8 Scope Summary — Coverage & Trust Hardening

v0.8 is a 3-execution-phase NFR milestone that continues the hardening theme opened by v0.7. v0.7 P03 (REQ-055) lifted four packages to ≥ 50%, but a coverage re-baseline after v0.7 ship shows the floor was insufficient: internal/engine regressed to 8.3%, internal/proxmox to 5.1%, and four more packages sit between 26% and 48%. Three packages (internal/audit, internal/certpaths, cmd/orca) still have no test files at all. v0.8 also closes the two "future enhancement" hooks explicitly deferred in v0.6 — SSH host-key pre-pinning (D-035 caveat) and orca node key-reset (RESEARCH_v0.6 §80) — and adds a requirements-hygiene gate so the stale-REQ-status drift seen in REQUIREMENTS.md after v0.7 ship cannot recur:

• P01 — Coverage uplift round 2. Raise six under-50% packages to ≥ 70% and add first tests for the three zero-test packages. Covers REQ-057.
• P02 — SSH trust hardening. --host-key-fingerprint pre-pin flag on orca node join --type proxmox + orca node key-reset <node> command. Covers REQ-058, REQ-059.
• P03 — Requirements-hygiene gate. make verify-reqs target + verify-stage assertion that ROADMAP Complete ↔ REQUIREMENTS Complete. Covers REQ-060.
• P04 — Final review + ship + audit. Milestone release.

The vision is unchanged. v0.8 is a hardening milestone, not a direction change. Milestone type: NFR (all phases are test/feat-chore on the trust surface — see CLARIFY D-043 for the feat vs chore classification of P02); the final phase's progressive patch IS the deliverable per run.md versioning logic. Tags run on the v0.7.x patch line: v0.7.0 (P0) … v0.7.4 (P04 = milestone release).

v0.8 Clarified Decisions (D-series, full autonomy)

The 5 v0.8 decisions (D-043..D-047) were auto-resolved at full autonomy within the clarify_budget (10):

ID	Question	Decision	Rationale	Confidence
D-043	Is P02 (SSH trust hardening) a feat phase or a chore phase? It adds a new flag + a new subcommand.	chore (trust-surface hardening), not feat	Both --host-key-fingerprint and orca node key-reset refine the existing orca node join --type proxmox flow and the existing TOFU known_hosts store (D-035). No new orchestration capability, no new node kind, no new API. They close a security gap explicitly deferred in v0.6, not open new surface area. Per run.md versioning logic this keeps v0.8 NFR (all phases fix/test/chore/perf/refactor).	0.84
D-044	Where does --host-key-fingerprint live — on orca node join or only on --type proxmox?	On orca node join (root of the join subcommand), validated when --type proxmox	The flag is generic (any future SSH-joined node kind will use it); gating it to --type proxmox only would require re-adding it later. Validation (flag requires --type proxmox today) happens in RunE, not in the flag declaration, so the flag is declared once on node join and the type check emits a clear error for non-proxmox types until other SSH-joined kinds exist.	0.86
D-045	--host-key-fingerprint format — raw hex, sha256:-prefixed, or OpenSSH SHA256:base64?	OpenSSH SHA256:base64 (the format ssh-keyscan -E sha256 -D - emits and operators expect)	Matches the fingerprint format operators already see from ssh-keyscan and orca node join's own Result.HostKeyFingerprint output. Accept only SHA256:-prefixed base64; reject raw hex with a clear error. Internally decode base64 → compare against ssh.PublicKey Marshal + sha256.	0.88
D-046	Does orca node key-reset <node> also revoke the orca pubkey on the remote host, or only clear the local known_hosts entry?	Local known_hosts entry only	Revoking the remote authorized_keys entry would orphan a working node (next dispatch would fail auth). key-reset is the local "forget this host's key" operation (mirrors ssh-keygen -R host); re-establishing trust is a separate orca node join re-run. Audit-log the reset with actor, node, event=node.key_reset.	0.90
D-047	Coverage target for P01 — 70% floor or higher?	70% floor for the 6 under-50% packages; 50% floor for the 3 zero-test packages (internal/audit, internal/certpaths, cmd/orca) as a first-toe-hold	70% across the board for the already-tested packages matches D-042's "70% target for new packages" and is achievable without heroic mock effort. For the zero-test packages, going 0→50% is the realistic single-phase step (0→70% risks a coverage rathole on cmd/orca which is glue code); a future milestone can lift them to 70%.	0.82

---

v0.9/v0.10 — Re-architecture Scope Summary (Supersedes v0.1–v0.8 architecture)

v0.9 is the first DIRECTION-CHANGE milestone in the project's history. It supersedes the shipped v0.1–v0.8 architecture per the adopted PRD (.ciagent/PRD_v0.9.md). The re-architecture deprecates the daemon/ transport/internal-CA/HCL/single-namespace stack and builds a CLI-only/ SSH-push/step-ca/Markdown-frontmatter/multi-namespace stack plus 8 net-new subsystems.

Override Justification (Re-architecture Justification axis)

The ci-griller returned REPLAN (0.70) on the Re-architecture Justification axis, noting the PRD reverses 6 documented decisions without new evidence and that the incremental-additive path was not evaluated. The user reviewed the fork and overrode the direction with a six-part evidence basis. The override is recorded verbatim below; each part addresses a reversal that the grill flagged as unjustified.

1. The v0.8 daemon model is operationally failing in the target environment — R-001 ("no orca binary on any server") is a response to measured pain, not preference.
2. step-ca is externally mandated (D-101) — the operator environment requires an external CA; AD-010's "too heavyweight" rationale is no longer operative.
3. Multi-tenancy is a hard product requirement (R-002) — real multi-tenant use cases cannot be served by the single-namespace layout; the "no multi-tenancy" anti-pattern is obsolete.
4. WASM is a hard workload requirement (D-088) — workloads are WASM, not processes; os/exec is insufficient; the "no container runtime" anti-pattern is reversed.
5. SSH-push is the only viable deployment target for the operator's bare-Linux/Proxmox environment — installing/maintaining an orca daemon on every peer is operationally infeasible.
6. Simplicity/vision correction — the v0.1-v0.8 daemon model was a wrong turn against the original CLI-first vision; the re-architecture corrects the vision.

Supersession Table (AD-series reversals, recorded per grill PC-09)

Old decision	Was	Superseded by	Evidence basis
AD-010 (ARCHITECTURE.md:463)	step-ca/cfssl/vault-pki "too heavyweight"	D-101 (step-ca)	Override ground 2 (external mandate)
SPIFFE rejection (PROJECT.md:94)	internal CA chosen over SPIFFE	D-068 (SPIFFE SVIDs)	Override ground 3 (multi-tenancy requires per-workload identity)
No-container-runtime (ARCHITECTURE.md:477)	explicit anti-pattern	D-088 (5 runtimes; wasmtime primary)	Override ground 4 (WASM is the workload profile)
No-multi-tenancy (ARCHITECTURE.md:478)	explicit anti-pattern	D-158 / R-002 (many namespaces under ORCA_HOME)	Override ground 3 (hard multi-tenant product req)
AD-007 (HCL canonical)	HCL for jobspec	R-013 / R-014 (Markdown canonical; HCL legacy)	PRD §8 (Markdown + body preservation is the operator-facing format)
Daemon-on-every-node	orca daemon on all peers	R-001 (no orca binary on any server)	Override grounds 1 + 5 (daemon failing; SSH-push only viable target)

The 19 binding conditions (C-01..C-19) and 10 phase challenges (PC-01..PC-10) from GRILL_v0.9.md are adopted as execution gates. The 30 net-new requirements (REQ-061..REQ-090) from IDEATION_v0.9.md are recorded in REQUIREMENTS.md. The reordered phase plan is in ROADMAP.md.

v0.9 Clarified Decisions (D-series, full autonomy — Phase 0 pre-execution)

ID	Question	Decision	Rationale	Confidence
D-101	Cluster CA: internal Go CA (AD-010) or step-ca (external)?	step-ca (apt-installed)	Externally mandated per override ground 2; AD-010's "too heavyweight" rationale reversed. CLI wraps step CLI via SSH (no Go step-ca client library — keep zero-new-dep posture if possible, or add github.com/smallstep/cli as a dep). Gated by C-07 (CA migration spec).	0.74
D-068	Workload identity: internal X.509 CA or SPIFFE SVIDs?	SPIFFE SVIDs minted at submit time via step-ca	Multi-tenancy (override ground 3) requires per-workload identity model; SPIFFE is the standard. SPIFFE ID spiffe://orca/ns/<ns>/job/<name>/alloc/<id> as SAN. Gated by C-08 (mint spike in v0.10-P01.5; fallback to mTLS identity if spike fails).	0.72
D-088	Runtime: direct os/exec only (D-008) or multi-runtime?	5 runtimes: wasm (wasmtime primary), podman, process, pve-vm, pve-ct	WASM is the primary workload (override ground 4). processRuntime wraps existing executor.go; others are net-new. Split P07a/b/c per grill PC-10. P07b gated by C-01 (wasmtime/CGO eval).	0.82
D-158	Namespace model: single flat root or multi-namespace?	Multi-namespace under ORCA_HOME (R-002)	Hard multi-tenant product requirement (override ground 3). _defaults/ implicit root; cluster/ for cluster-wide; per-namespace db/, .env, .env.secrets, jobs/, alloc/, ns.md. No namespace column in SQLite.	0.84
D-179	Jobspec format: HCL canonical (AD-007) or Markdown?	Markdown with YAML frontmatter canonical (R-013); HCL legacy	PRD §8 — Markdown + body preservation is the operator-facing format. HCL adapter (REQ-064) preserves orca job run old-spec.hcl during migration.	0.85
D-185	Re-architecture justification: incremental additive or full re-architecture?	Full re-architecture (overridden by user)	Six-part evidence basis above; the grill's REPLAN mechanics (PC-01..PC-10, C-01..C-19) adopted as gates. The incremental-additive path was evaluated and rejected on grounds 1 + 5 (daemon failing; SSH-push only viable).	0.88
D-187	wasmtime Go binding (bytecodealliance/wasmtime-go) is CGO-based — does adopting it revoke D-002 (modernc/sqlite CGO-free cross-compile story)?	Use the wasmtime CLI (apt-installed on peer) via SSH exec; do NOT import wasmtime-go.	The Go binding links libwasmtime via cgo and would revoke D-002's CGO-free cross-compile story. The CLI-via-SSH approach (same pattern as podman/qm/pct) avoids CGO entirely. internal/runtime/wasm.go imports only stdlib + sshpush. CGO_ENABLED=0 go build ./... succeeds. C-01 grill gate SATISFIED; D-002 NOT revoked. Full evaluation in internal/runtime/C01_WASMTIME_CGO_EVAL.md.	0.90

---

v0.10 Docs & Install Milestone — Scope Summary

v0.10 is a focused milestone that closes the documentation gap left by the v0.9 re-architecture and fixes the release/install pipeline bug that caused install.sh to resolve to v0.4.5 instead of the latest release. The v0.9 re-architecture shipped a complete CLI surface (markdown jobspec, orca ns, orca node capacity, CLI-side scheduler, emitters, Traefik ingress) but no operator-facing reference documentation. This milestone ships that documentation plus a worked full-stack example with ingress configured, and hardens the release pipeline so every Gitea release carries a Linux binary asset.

Root cause of the v0.4.5 install

The v0.8.x releases (v0.8.0 through v0.8.15) shipped with zero binary assets attached to their Gitea releases. scripts/install.sh resolves "latest" by hitting /releases/latest (returns v0.8.15), then looks for orca-v0.8.15-linux-amd64.tar.gz in that release's assets. Since the asset is missing, install.sh errors out — there is no fallback walk to older releases that DO carry a binary. The user's v0.4.5 install came from an earlier run or a pinned --version. The fix is forward: harden scripts/release.sh to cross-build the amd64 tarball and verify the asset attached post-create; harden scripts/install.sh to walk backward through releases if the latest lacks the asset.

v0.10 Phases

• Phase 0 (pre-execution): specify → clarify → research → ideate → plan → grill. Tag v0.9.0.
• Phase P1 — release/install fix (REQ-097, REQ-098): cross-build amd64 tarball in release.sh, post-create asset verification, install.sh fallback walk. Tag v0.9.1.
• Phase P2 — CLI + jobspec + ingress docs (REQ-091, REQ-092, REQ-093): docs/cli.md, docs/jobspec.md, docs/ingress.md. Tag v0.9.2.
• Phase P3 — full-stack examples (REQ-094): examples/full-stack/ with 5 valid jobspecs + rendered artifacts + walkthrough README. Tag v0.9.3.
• Phase P4 — README + namespace.md refresh (REQ-095, REQ-096): README subcommand table + install example + docs/examples sections; docs/namespace.md v0.9 layout. Tag v0.9.4.
• Phase P5 — final review + ship + audit (milestone release). Tag v0.9.5 = v0.10.0 milestone release.

Milestone type: feature (P1 ships fix phases; P2/P3/P4 ship docs phases; at least one non-docs phase makes this a feature milestone per the versioning logic). Tags run on the v0.9.x patch line. The milestone branch label is milestone/v0.10-docs-cli-examples.

The vision ("minimalist, offline-first, CLI-first orchestration engine") is unchanged. v0.10 is a documentation + install-hardening milestone, not a direction change. It builds on the v0.9 re-architecture foundation without modifying any Go orchestration code.

v0.10 Clarified Decisions (D-series, full autonomy — Phase 0 pre-execution)

ID	Question	Decision	Rationale	Confidence
D-188	Should the CLI docs be a single docs/cli.md reference or a per-command docs/cli/ subdirectory?	Single docs/cli.md reference	Mirrors the existing flat docs/ pattern (install.md, docker.md, namespace.md, security-scanning.md). One file is more discoverable for a CLI tool and avoids navigation overhead. A per-command subdirectory diverges from the established layout.	0.92
D-189	Should the examples live in examples/full-stack/ or in testdata/?	examples/full-stack/ as a new top-level directory	testdata/ holds legacy HCL fixtures (hello.hcl, fail.hcl) used by Go tests; mixing operator-facing examples with test fixtures conflates audiences. A new examples/ directory is the conventional location for worked examples and is what an operator expects to find.	0.93
D-190	How deep should the ingress/Traefik documentation go?	Dedicated docs/ingress.md plus a worked example in examples/full-stack/	Ingress is the user's explicit ask ("full stack with ingress configured") and the Traefik/service-block model (R-007 socket vs TCP, atomic reload, drain, TLS) is non-trivial. A dedicated doc is the clearest answer; a section buried in docs/cli.md would be less discoverable.	0.90
D-191	Should the docs frame the v0.9 canonical path or document both v0.8 and v0.9 equally?	Document the v0.9 canonical path; flag deprecated surface with callout boxes	The v0.8 daemon/mTLS/HCL path is deprecated and scheduled for removal in v0.10-P14. Documenting it as primary misleads new operators; documenting both equally doubles the surface and risks documenting soon-removed code. Callout boxes with "deprecated in v0.9, removed in v0.10" point operators to the canonical path.	0.91
D-192	Should the existing v0.8.15 release be backfilled with a binary asset, or only fix the pipeline forward?	Fix forward only; no backfill	Backfilling a past release is an ops task, not a docs milestone deliverable. The next tagged phase (this milestone's P1 ship at v0.9.1) will be the first correctly-asseted release; install.sh's new fallback walk handles the gap until then.	0.88
D-193	Should release.sh build only linux-amd64 or also linux-arm64?	Cross-build linux-amd64 explicitly (host-arch-independent); arm64 deferred to a follow-up	The install.sh user base is amd64 today (the .coreci.yml release step hardcodes --asset orca-${VERSION}-linux-amd64.tar.gz). Building amd64 regardless of host arch (via GOOS=linux GOARCH=amd64 go build) guarantees the asset the install script expects. arm64 support is a separate enhancement.	0.85
D-194	Should install.sh add a --check dry-run mode?	Yes, lightweight	A dry-run mode (--check) that prints the version + asset URL + install path without writing is cheap to add and useful for debugging the "which release will I get?" question that the v0.4.5 incident surfaced.	0.80

v0.11 Clarified Decisions (D-series, full autonomy — Phase 0 pre-execution)

The following 23 decisions (D-215…D-237) extend the locked D-series (ends at D-206). They derive from 5 research documents ingested 2026-08-07 covering ingress hardening, drift detection, platform-engineer positioning, strategic framing, and the systemd Path unit implementation. Operator decisions Q1=A, Q2=C, Q3=A, Q4=A, Q5=A are adopted.

Ingress hybrid (D-215…D-226, from research doc 1)

ID	Question	Decision	Rationale	Confidence
D-215	Public-binding default?	Hybrid: nft DNAT → Traefik on 127.0.0.1:8443	Defense-in-depth (kernel + app layer); mature pattern (kube-proxy, Linkerd2-proxy, F5/HAProxy+nginx). Smaller Traefik attack surface. R-017.	0.93
D-216	Opt-out?	orca cluster config --public-binding=traefik-on-public-ip for the simple case	Operators who want simplicity get it with a one-line config change.	0.94
D-217	nftables emitter?	Yes; renders /etc/nftables.d/orca.nft; idempotent nft -f apply	Same emitter pattern as Traefik/systemd emitters (R-001-clean).	0.93
D-218	nftables tool vs iptables?	nft (modern) over legacy iptables	Atomic rule-set swap; modern kernel API.	0.96
D-219	Cross-node cluster mesh?	Stays bound on private IP 192.168.x.x:8443; unchanged	Avoids adding iptables rules for cross-node mesh; keeps mesh logic unchanged.	0.94
D-220	Traefik address in static config?	127.0.0.1:8443 in default, :443 in opt-out	Single line change; certs/mTLS/dynamic config unchanged.	0.97
D-221	orca doctor nft?	Yes; checks table, expected rules, file hash; drift detection via hash comparison	Parity with orca doctor traefik; integrates with R-018 critical_paths.	0.95
D-222	Rate-limit meter?	ora_rl set as part of the default rule set; configurable via orca nft rate limit set	Kernel-level line-rate rate limiting; defense against SYN floods.	0.91
D-223	GeoIP blocking?	Operator-opt-in via orca nft country block add; cli + ipset extension	Not a default; operators opt in.	0.88
D-224	nftables not iptables in .coreci.yml pipelines?	Yes; integration tests use nft exclusively	Matches D-218.	0.94
D-225	Per-workload ingress: native coexists with hybrid default?	Yes; service { ingress: native } opts into pure iptables + stunnel sidecars	Workload-level opt-in; doesn't affect cluster default.	0.93
D-226	nftables rule hash baseline?	cluster/state/baseline.nft.hash per peer; drift detection per §17	Integrates with R-018 drift detection.	0.90

Drift detection (D-227…D-237, from research doc 5)

ID	Question	Decision	Rationale	Confidence
D-227	Drift detection architecture?	systemd Path units for critical paths + 60s polling backstop + auto-remediation	R-001-clean (systemd is OS, not Orca); ~10s event-driven latency on critical paths. R-018/R-019.	0.94
D-228	Path unit event payload?	Oneshot service; receives path via %f; computes sha256; writes event JSON to /etc/orca/state/drift-events/	Stateless, self-contained, idempotent.	0.93
D-229	Lead-side pickup?	Aggregator timer reads each peer's drift-events/, validates against applied txn hashes, triggers remediation	Reuses existing 10s aggregator cadence (C-11); single SSH pull per tick.	0.94
D-230	Critical path polling cadence?	5s backstop; systemd Path unit provides ~10s event-driven latency	Closes the gap to K8s-comparable drift detection on critical paths.	0.92
D-231	Auto-remediation policy?	Per-path config; critical paths default to auto; systemd units default to require-approval	Config files are safe to re-push; service units may need careful ordering (don't restart serving workloads).	0.93
D-232	Remediation rate limit?	5-minute cooldown per path; applies only on SUCCESSFUL remediation; transient failures retry on next aggregator tick	Prevents loops from buggy external actors; avoids a 30s network blip blocking re-remediation for 5 min (refined per CLARIFY C4).	0.92
D-233	NFS path detection?	orca node setup detects NFS mounts; falls back to polling for affected paths	systemd Path units use inotify which doesn't work across NFS.	0.90
D-234	Secrets path exclusion?	/etc/orca/credentials/* excluded from drift detection	Re-remediating secrets might clobber intentional out-of-band rotation.	0.95
D-235	EnvironmentFile drift?	Triggers orca job restart <name> instead of file-level remediation	Workload already running won't pick up env changes without a restart.	0.89
D-236	orca drift watch semantics?	iter.Seq2[Event, error] per D-017; signal.NotifyContext per D-023; default 2s poll	Consistent with existing --watch pattern (D-017/D-023).	0.95
D-237	Aggregator timer changes?	Existing 10s cadence; extended to also pull drift-events/ and remediate	Reuses C-11 aggregator; no new timer.	0.95

P01.5 — SPIFFE SVID minting spike (gate C-08, D-068)

C-08 SPIFFE mint spike: PASS. The step CLI (smallstep step-ca) accepts a spiffe:// URI in --san and emits a cert whose URI SAN (x509 subjectAltName URI entry) carries the SPIFFE URI. The fallback to mTLS identity (per D-068 / C-08) is NOT needed; D-068 stands.

• SPIFFE URI format (locked): spiffe://orca.local/ns/<namespace>/sa/<service-account>/<alloc-id> — trust domain orca.local; ns/<ns> scopes the workload to an Orca namespace (R-002); sa/<sa> is the service-account; <alloc-id> makes the SVID unique per allocation.
• step CLI command (locked): step ca certificate <spiffe-id> <cert> <key> --san <spiffe-id> --not-after 24h --provisioner orca-admin --password-file /dev/stdin --force
• Cert parsing (locked): pem.Decode → x509.ParseCertificate → iterate cert.URIs and match the expected SPIFFE URI (parsed as *url.URL, compared by canonical string). Missing URI SAN → ErrSpiffeURIMissing (cert rejected before reaching the workload).
• Implementation: internal/identity/spiffe.go — SpiffeURI, MintSVID, VerifySVID, SpiffeIDFromCert, SubjectFromSpiffe.
• Tests: internal/identity/spiffe_test.go — mock transport (execer) returns a self-signed cert minted in-process via crypto/x509.CreateCertificate with URIs: []*url.URL{spiffeURI}, exercising the exact production parsing path. 15 tests, all pass.
• Spike result record: internal/identity/SPIFFE_SPIKE_RESULT.md.

v0.12 Scope Summary — Security Hardening (Zero-Trust Identity)

v0.12 is a 27-execution-phase feature milestone dedicated to comprehensive security hardening across the entire attack surface, including the operating system itself. The threat-model review (v0.11 closeout + Phase 0 RESEARCH) surfaced 25 distinct findings (F1..F25) spanning injection, traversal, ACL, audit, crypto, OS scripts, emitters, sudoers, system users, file modes, daemon auth, backup, SQLite, install.sh, and migration. v0.12 closes all of them and adopts a zero-trust identity model as the load-bearing architectural change.

Load-bearing rule adopted in Phase 0

R-021: Orca never issues, stores, or accepts human-identity credentials. Human identity is exclusively external (OIDC). Machine identity is exclusively mTLS/SPIFFE. No passwords, no Orca-issued tokens, no CA-key passphrases.

Zero-trust identity model

Two identity layers, zero overlap:

• Human operators → OIDC (external IdP, BYO) OR the bundled Dex with a WebAuthn (passkeys) connector as the default password-free authenticator. orca auth login / orca auth register open the default browser to the Dex WebAuthn endpoint via OIDC authorization-code + PKCE + local loopback redirect. After the WebAuthn ceremony (biometric/security key), Dex redirects back with an auth code; CLI exchanges for a short-lived ID token (1h) + refresh. Headless/CI fallback: device-code flow.
• Machine-to-machine → mTLS + SPIFFE SVIDs (unchanged from v0.11).

The "no Orca credentials" invariant holds: passkeys are public-key credentials (the private key never leaves the authenticator); the WebAuthn credential DB stores only public keys + credential IDs + sign counts. No passwords, no Orca-issued tokens, no CA-key passphrases anywhere in the system.

Master key sealing

The secrets master key (32 random bytes) is sealed to OIDC — wrapped by a key derived from an OIDC token exchange at unseal time. orca cluster unseal (operator authenticates via OIDC → token exchange → unwrap master key into memory → zeroed on shutdown). The raw master key never touches disk. Shamir 3-of-5 recovery: at seal time, 5 shards are printed and the operator stores them offline. If the IdP is permanently lost AND a quorum of shards is unavailable, the cluster is unrecoverable by design (documented residual risk; no backdoor).

New requirements (REQ-119..REQ-148)

30 net-new requirements derived from the threat-model findings and the zero-trust identity model. See REQUIREMENTS.md and ROADMAP.md for the full mapping. Highlights:

• REQ-119..121: command injection, path traversal, txn path allowlist
• REQ-144: OIDC client + bundled Dex (BYO-IdP override)
• REQ-145: ACL rewrite (remove KindToken, add KindOidc, enforce)
• REQ-146: remove all password/token paths (breaking)
• REQ-147: master key seal-to-OIDC + Shamir recovery
• REQ-148: WebAuthn connector for Dex (passkeys, browser auth+register)
• REQ-122..143: integrity, crypto, OS scripts, emitters, sudoers, system users, SQLite, migration, dual-write closure, transport, drift auth, integration tests, docs, final review

v0.12 Clarified Decisions (D-series, full autonomy)

The 10 v0.12 decisions (D-238..D-247) were resolved during CLARIFY under full autonomy (autonomy.level=full, workflow.no_hitl=true):

ID	Question	Decision	Rationale	Confidence
D-238	Milestone version?	v0.12 (minor, not v1.0)	v1.0.0 stays deferred for post-UAT per v0.11 PRD; v0.12 is a minor feature milestone. Tags on v0.11.x patch line.	0.95
D-239	OIDC provider model?	Bundled Dex by default + BYO external IdP override	Zero-trust out of the box without external setup; oidc.issuer repoint switches to BYO.	0.90
D-240	Bundled Dex upstream authenticator (password-free)?	WebAuthn (passkeys) connector	Public-key credentials; private key never leaves authenticator; reinforces "no passwords" invariant (R-021).	0.88
D-241	Master key sealing model?	Seal to OIDC + Shamir 3-of-5 recovery	No password anywhere; quorum recovery if IdP lost; no backdoor.	0.85
D-242	CLI browser flow?	OIDC auth-code + PKCE + local loopback redirect	Standard OIDC browser flow; secure for public clients; headless fallback via device-code.	0.92
D-243	WebAuthn RP ID / secure context?	Traefik-served cluster domain (step-ca cert, R-017)	WebAuthn requires HTTPS; Traefik already provides it; RP ID configurable via orca auth init-idp.	0.90
D-244	Passkey storage?	SQLite at ClusterDir()/webauthn-credentials.db (0600); public keys only	Public keys are not secrets; 0600 file mode for integrity; no passphrase wrapping needed.	0.92
D-245	Headless/CI auth fallback?	Device-code flow	No browser in CI; device-code is the standard OIDC headless path.	0.90
D-246	Token storage at rest?	~/.orca/credentials.json (0600); short-lived (1h) + refresh	Standard OIDC token storage; 0600; refresh handles rotation; no long-lived Orca-issued tokens.	0.92
D-247	Breaking-change handling for password/token removal?	orca upgrade refuses v0.11 clusters using --password/bare-tokens without --accept-identity-migration	No silent breakage; explicit migration gate; documented cutover.	0.90

v0.12 is a HARDENING + IDENTITY milestone, not a direction change

The vision ("minimalist, offline-first, CLI-first orchestration engine inspired by HashiCorp Nomad") is unchanged. v0.12 closes the security-surface gaps surfaced by the v0.11 threat model and adopts a zero-trust identity model. The offline-first principle (R-003) is preserved: the bundled Dex can run on the lead (offline), and the mTLS-only path remains for the single-operator fully-offline case (no human authn needed — the operator holds the pre-staged SSH key + mTLS cert; no password, no token).

v0.13: Production Hardening Round 2 + UAT Plan (IN PROGRESS)

v0.12 (Security Hardening) is COMPLETE. v0.13 is the final hardening round before UAT validation. The UAT will likely surface 3-7 issues requiring a patch release. v1.0.0 is deferred until UAT passes.

v0.13 is the final hardening round before the v1.0.0 production-ready tag. Three deep codebase sweeps (security, reliability, feature/doc claims) surfaced ~60 gaps beyond v0.12. The most critical:

1. orca job run runs locally via exec.CommandContext — the scheduler/emitter/SSH-push pipeline is dead code. The documented deployment model (deploy to Proxmox/Ubuntu worker) is non-functional. R-022 fixes this.
2. jobspec schedule:/timeout: silently dropped by the markdown parser — DaemonSet is fundamentally broken (parser defaults Count=1, validator rejects Count!=0, schedule never parsed).
3. acl.Check called zero times — v0.12's headline zero-trust feature is library-complete but not wired into any request path. R-023 fixes this.
4. Command injection vectors — orca logs --job backtick RCE via %q (bash executes command substitution in double quotes), tar-slip in backup restore, sudoers injection via --proxmox-user/--role, txn rollback shell injection, and 7 more.
5. Go toolchain 1.25.0 — 24 stdlib vulns with call traces in orca (archive/tar, crypto/tls, crypto/x509, net/http, encoding/pem...).
6. Concurrency hazards — audit hash-chain race (concurrent appends corrupt tamper-evidence), concurrent secrets set silently loses data (no flock), no SQLite busy_timeout (database is locked), concurrent orca upgrade races on Traefik cutover.
7. Cache never invalidated by writes — stale reads for 10–60s after node join/ns create/job run.
8. Massive doc drift — README "mTLS by default" is false (SSH-push is canonical), docs/cli.md missing ~25 subcommands, CHANGELOG stale at v0.1, verify-reqs gate bypassed for v0.12.

v0.13 closes all critical/high/medium findings (15 new requirements, 14 phases) and delivers the UAT plan + signoff script that gates the v1.0.0 cut.

v0.14 Milestone: Ingress Bootstrap Completeness

Scope: ensure that linux & proxmox types are properly bootstrapped with traefik during cluster init or node join. All cluster endpoints are provisioned as sockets (R-007); routing between jobs and services depends on traefik being present and properly configured. v0.13 shipped traefik binary + systemd unit + empty dynamic dir but never wrote the static config nor applied nft rules — orca-traefik.service fails on a fresh orca init and orca doctor nft FAILs. v0.14 replaces the binary+systemd model with a podman container running a custom orca-traefik image, and completes the nft SNAT+DNAT ingress stack on every node type.

New load-bearing rule: R-024 — Traefik runs exclusively as a podman container, deployed from the orca-traefik image published per release. Every orca-managed ingress surface bootstraps: nft DNAT (:443→127.0.0.1:8443, :80→127.0.0.1:8080) + SNAT/MASQUERADE postrouting + podman run -d --restart=always --network host -v /etc/traefik/dynamic:/etc/traefik/dynamic:Z -v /etc/orca/step-ca-root.crt:/etc/orca/step-ca-root.crt:ro git.cloudinit.dev/coreci/orca-traefik:<tag>. No node joins without a functional podman-traefik ingress data plane.

Three topologies (per operator constraints):

1. Linux: host → nft → podman run orca-traefik (host network)
2. Proxmox Native: host → nft → LXC (nesting=1) → podman run orca-traefik
3. Proxmox Floating-IP: LXC (owns floating IP) → nft (inside LXC) → podman run orca-traefik

Milestone type: feature (multiple feat phases). Tags on v0.13.x patch line: v0.13.0 (P0) ... v0.13.8 (P8 final = v0.14 milestone release). 9 phases, 9 net-new requirements (REQ-171..REQ-179).

v0.14 Decisions (D-series, full autonomy)

ID	Question	Decision	Rationale	Confidence
D-255	Traefik deployment model?	Podman container from custom orca-traefik image	Operator constraint: traefik always deployed as a container. Replaces v0.13 binary+systemd. Image bakes static config.	0.95
D-256	Container network mode?	--network host	Binds 127.0.0.1:8080/8443 directly on host/LXC loopback; nft DNAT targets that. No port publishing complexity.	0.92
D-257	TLS cert resolver in image?	No certResolver; tls: {} for v0.14, real mTLS deferred to v0.15	Grill G-003 (confidence 0.55 < 0.60) auto-resolved to defer. Traefik v3.3 certificatesResolvers only supports acme/tailscale, not CA-file. Drop certResolver: orca (broken). Emit tls: {} in dynamic config. Real mTLS via dynamic tls.certificates + tls.options.default.clientAuth.caFiles lands in v0.15 when step-ca mints server certs.	0.90
D-258	Dynamic config volume?	Mount /etc/traefik/dynamic from host	Zero changes to existing deployRemote WriteFile path (job_dispatch.go:243). File provider watches it.	0.95
D-259	Floating-IP mode: register PVE host too?	Yes — PVE host as proxmox + ingress LXC as linux	PVE host needed in registry for pct/qm workload dispatch. Both register.	0.92
D-260	--ingress-mode persistence?	Store IngressMode on model.Node	doctor ingress needs to know which check path to run. Schema migration.	0.90
D-261	MAC generation when --mac omitted?	Generate random 02:XX:... in interactive mode; require --mac in --json mode	Interactive: generate + confirm. Non-interactive: explicit required (no silent generation).	0.88
D-262	Proxmox native nft DNAT target?	LXC bridge IP (not 127.0.0.1)	LXC has its own network namespace; 127.0.0.1 on PVE host ≠ LXC loopback. NftClusterConfig.DNATTarget field (default 127.0.0.1:8443; native mode = <lxc-ip>:8443).	0.90
D-263	LXC podman requirements?	--features nesting=1,keyctl=1 + apt-get install podman	Ubuntu 24.04 LXC template has no podman preinstalled. Nesting+keyctl required for podman in unprivileged LXC.	0.88

v0.13 Decisions (D-series, full autonomy)

ID	Question	Decision	Rationale	Confidence
D-248	Milestone version?	v0.13 (minor, not v1.0)	v1.0.0 stays deferred for UAT signoff; v0.13 is a minor feature milestone. Tags on v0.12.x patch line.	0.95
D-249	UAT validation mechanism?	Operator-driven docs/uat.md + scripts/uat-signoff.sh assertions	Operator builds real cluster (3 hosts), runs signoff script, pastes output. Exit 0 iff all ~35 assertions pass.	0.92
D-250	Hardening phase scope?	All 8 themes, 14 phases	"No limit on phases" per operator; comprehensive to avoid a round 3.	0.90
D-251	Ubuntu worker onboarding?	Implement --type linux SSH-join	NodeKindLinux is reserved but unimplemented; UAT plan needs first-class worker onboarding. Proxmox stays --type proxmox.	0.88
D-252	job stop semantics?	Real systemctl stop via SSH	Honest semantics matching job restart pattern; UAT assumes stop actually stops.	0.90
D-253	UAT cluster topology?	3 hosts: lead Ubuntu + pve01 Proxmox + worker01 Ubuntu	Minimal topology covering both node types + migrate-between-hosts.	0.92
D-254	UAT signoff script re-runnable?	Idempotent — read + non-mutating assertions only	Operator can iterate; no destructive ops.	0.95

v0.13 is the LAST hardening round

Three deep sweeps (security, reliability, feature/doc) were performed to ensure no gap is missed. 9 low-severity residual risks are documented and accepted (OIDC tokens plaintext at rest, HSTS on daemon, DNS timeout, temp file cleanup on SIGKILL, flock timeout on NFS, WASM-first aspirational, arm64 release, OIDC callback slowloris, pprof-allow-public flag). v0.13 closes everything else. The v1.0.0 tag is cut only after the UAT signoff script passes.