orca/.ciagent/PROJECT.md

Project: Orca

What This Is

A minimalist, offline-first, CLI-first orchestration engine inspired by HashiCorp Nomad, prioritizing stability, security, and simplicity over feature richness. Single-binary distribution, no container runtime, no cloud dependencies, no K8s-level complexity.

Vision

A minimalist, offline-first, CLI-first orchestration engine inspired by HashiCorp Nomad, prioritizing stability, security, and simplicity over feature richness.

Objective

Build a lightweight system to manage and execute workloads across a set of nodes, keeping complexity far below that of Kubernetes.

Requirements

• CLI First: Primary interaction through a CLI tool.
• Offline First: Functional without constant internet connectivity.
• AI First: Designed to be easily discoverable and manageable by AI agents.
• Stability & Security: Prioritize security fixes and bug fixes over new features.
• Language: Written in Go 1.25+.
• Simplicity: Minimalist implementation, avoiding the "K8s complexity trap".

Constraints

• No web UI as a primary requirement.
• Must not implement K8s-level complexity.
• Feature development must move slowly to ensure stability.
• Only CI system allowed: CoreCI (git.cloudinit.dev/coreci/coreci).
• Gitea remote: git.cloudinit.dev/coreci/orca.

Clarified Decisions (D-series, full autonomy)

ID	Question	Decision	Rationale	Confidence
D-001	Single binary or multi-binary distribution?	Single binary	Simpler distribution; subcommands baked into one orca binary. Aligns with simplicity pillar.	0.95
D-002	Local state store technology?	modernc/sqlite (pure Go, CGO-free)	Cross-compile friendly, no CGO dependency, single file on disk, mature.	0.92
D-003	Inter-node communication?	Embedded HTTP (net/http) over loopback, mTLS for cross-node	No external RPC framework needed for v0.1. HTTP suffices.	0.85
D-004	Scheduling algorithm for v0.1?	Single-node only (no scheduling)	Multi-node scheduling is out of scope for v0.1. Tasks run on the node they're submitted to.	0.90
D-005	CLI output format?	Human-readable by default, --json flag for machine consumption	Serves both humans and AI agents.	0.95
D-006	Job/task definition format?	HCL or YAML in .hcl/.yaml files	Familiar to Nomad/HashiCorp users; simpler than JSON for humans.	0.88
D-186	Bash scripts coverage gate: count toward Go gate or exempt?	Exempt from Go coverage gate; compensating control: bats tests (C-15) + shellcheck + shfmt in CI; every script must have >=1 happy-path and >=1 failure-path bats test	Bash is a different language surface from Go; the 70%/50% Go coverage gate (D-042/D-047) is Go-specific. Forcing bash into the Go gate would require a coverage tool that does not exist for bash. The compensating control (bats + shellcheck + shfmt) provides equivalent discipline.	0.82
D-007	Authentication?	mTLS for v0.1, token-based deferred	mTLS is the most secure default. Tokens can be added later if needed.	0.80
D-008	Container runtime?	Direct process execution (no container runtime) for v0.1	Avoids the Docker/container dependency. Pure process management.	0.85
D-009	Configuration file location?	~/.orca/config.hcl and /etc/orca/orca.hcl	Standard XDG-style paths.	0.90
D-010	Logging format?	Structured JSON via log/slog	Native Go 1.21+ slog, no external dependency.	0.95
D-011	v0.2 mTLS cert authority model?	Internal CA with CSR join	One node bootstraps a local CA; peers generate CSRs and submit them to the CA for signing. CA cert is the trust anchor. More secure than self-signed per-node (single trust root) without the operational complexity of an external PKI.	0.92
D-012	v0.2 CA bootstrap & cert distribution?	Operator-mediated, fingerprint-verified	Bootstrap node writes ~/.orca/ca.crt and ~/.orca/ca.key (mode 0600). Operator copies ca.crt to peers; peers verify by SHA-256 fingerprint at orca node join --ca-fingerprint <sha256>. No automated secret distribution.	0.85
D-013	v0.2 cert validity & rotation?	Server certs 90 days, CA cert 10 years, rotate 30 days before expiry	Server certs are short-lived (compromise window small); CA is long-lived (manual rotation is expensive). orca cert renew reissues server certs automatically.	0.90
D-014	v0.2 mTLS handshake timing?	Eager — at orca node join time	Fail fast on bad certs, misconfigurations, or CA mismatches. Lazy handshake would let stale configs run until first request, complicating debugging.	0.88
D-015	v0.2 minimum TLS version & cipher suites?	TLS 1.3 only; AEAD cipher allowlist	MinVersion=tls.VersionTLS13, CipherSuites limited to TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_AES_128_GCM_SHA256. No TLS 1.2 fallback.	0.92
D-016	v0.2 security-scanning placement?	validate pipeline of .coreci.yml, gates merges to main	gosec baseline JSON checked into repo; new findings fail the build. govulncheck ./... exit-on-known. Pre-commit hook with gitleaks is opt-in (developer machine).	0.85
D-017	v0.2 iter.Seq API surface?	orca job list --watch and orca node list --watch	Pull-based iter.Seq[Job] / iter.Seq[Node]; cancellation via context.Context; signal.NotifyContext on ctrl-c. Backpressure is implicit (consumer-driven).	0.90
D-018	v0.2 multi-node scheduling algorithm?	Bin-packing by available CPU/memory, FIFO within a node	Simple, deterministic, matches D-004 minimalism. Cross-node dispatch via ConnectRPC orca.v1.Dispatch service. Retry on transient failures with exponential backoff.	0.85

Out of Scope

• Full-blown Kubernetes-compatible API.
• Complex cloud-provider integrations.
• GUI-based management consoles.
• Container runtime integration.
• Service mesh / sidecar injection.
• Auto-scaling / horizontal pod autoscaler.
• External PKI / Let's Encrypt / cert transparency logs.
• gRPC framework dependency (ConnectRPC in config.json frameworks but not in go.mod; v0.2 uses stdlib net/http with h2c for orca.v1.Dispatch — see ARCHITECTURE.md AD-014).

v0.2 Scope Summary

v0.2 is a focused 4-phase milestone that turns Orca from a single-node process executor into a small cluster engine with strong transport security and richer I/O. The 4 phases are:

• P01 — mTLS handshake + internal CA with CSR join. Internal CA, CSR join, eager handshake at node join, TLS 1.3 + AEAD allowlist. See ARCHITECTURE.md Flow 1 + Flow 2.
• P02 — Multi-node scheduling & job dispatch. Best-fit bin-packing by CPU/memory, FIFO within a node, orca.v1.Dispatch over mTLS. See ARCHITECTURE.md Flow 3.
• P03 — gosec + govulncheck + gitleaks in CI. gosec baseline JSON in repo, govulncheck ./... in validate pipeline, gitleaks in pre-commit (opt-in).
• P04 — iter.Seq streaming for --watch flags. Go 1.25+ range-over-func semantics, context.Context cancellation, signal.NotifyContext on ctrl-c. See ARCHITECTURE.md Flow 4.

The vision ("minimalist, offline-first, CLI-first orchestration engine") is unchanged. v0.2 is a hardening + small-cluster extension, not a direction change.

Key Decisions

The 18 D-series decisions (D-001..D-018) are recorded in the "Clarified Decisions" table above. The 10 v0.1 decisions (D-001..D-010) are stable and unchanged in v0.2. The 8 v0.2 decisions (D-011..D-018) were auto-resolved under full autonomy and are summarized here:

• D-011: Internal CA with CSR join (vs. self-signed per-node or SPIFFE). Single trust root, no external PKI, CSR workflow.
• D-012: Operator-mediated CA cert distribution with fingerprint verify (no automated secret distribution — matches offline-first principle).
• D-013: 90d server certs, 10y CA cert, 30d pre-expiry rotation.
• D-014: Eager mTLS handshake at orca node join time (fail fast).
• D-015: TLS 1.3 only, AEAD cipher allowlist (no TLS 1.2 fallback).
• D-016: gosec+govulncheck in validate pipeline of .coreci.yml (gates merges to main). gitleaks in pre-commit (opt-in).
• D-017: iter.Seq for orca job list --watch and orca node list --watch (pull-based, ctx cancellation, ctrl-c via signal.NotifyContext).
• D-018: Bin-packing by CPU/memory with FIFO within node; JSON-over-HTTP orca.v1.Dispatch for cross-node (no ConnectRPC dep).

v0.3 Clarified Decisions (D-series, full autonomy)

v0.3 is a lean 2-execution-phase milestone completing the streaming and doctor work deferred from v0.2. The 6 v0.3 decisions (D-019..D-024) were auto-resolved under full autonomy:

ID	Question	Decision	Rationale	Confidence
D-019	Watch refresh mechanism?	Poll-based, 1s ticker	Simpler than event channel; no daemon coupling for CLI; matches offline-first.	0.90
D-020	Watch output format (REQ-030)?	Table by default; --watch --json streams one-line JSON per event	Consistent with D-005 --json convention; serves humans + AI agents.	0.92
D-021	Doctor network check scope?	Probe configured peer addresses via mTLS /healthz handshake; PASS/WARN/FAIL per peer	Reuses existing transport client; read-only.	0.85
D-022	Doctor db check scope?	PRAGMA integrity_check + migration version query	Already specced in ARCHITECTURE.md §5; minimal surface.	0.95
D-023	iter.Seq cancellation?	signal.NotifyContext on SIGINT/SIGTERM	Per D-017 + ARCHITECTURE Flow 4.	0.95
D-024	--watch applies to job list only, or node list too?	Both orca job list --watch and orca node list --watch	Per ARCHITECTURE.md CLI layer + D-017.	0.92

v0.3 Scope Summary

v0.3 is a focused 2-execution-phase milestone completing the work deferred from v0.2 that was NOT already shipped in P08-P10. A codebase audit during re-init SPECIFY confirmed that REQ-014, REQ-027, REQ-028, REQ-029, REQ-031, REQ-037, REQ-039, REQ-040 all shipped in P08-P10 despite stale REQUIREMENTS.md marking them Pending. The remaining work:

• P01 — iter.Seq streaming for --watch flags. Go 1.25+ range-over-func semantics, pull-based iter.Seq[Job] / iter.Seq[Node], context.Context cancellation, signal.NotifyContext on ctrl-c. Applies to both orca job list --watch and orca node list --watch. Covers REQ-022, REQ-030.
• P02 — orca doctor network + db full implementation. Replaces the P01 stubs (NetworkStub, DBStub) with real checks: peer reachability via mTLS /healthz probe; SQLite PRAGMA integrity_check + migration version. Covers REQ-032 (completion).

The vision ("minimalist, offline-first, CLI-first orchestration engine") is unchanged. v0.3 is a completion milestone, not a direction change.

v0.5 Scope Summary — Distribution

v0.5 is a 3-execution-phase milestone that makes Orca installable, distributable, and containerized. The engine functionality from v0.1–v0.3 is unchanged; this milestone is purely about delivery surface:

• P01 — Namespace unification. A single ORCA_HOME environment variable becomes the namespace root for all on-disk state (db, certs, init, daemon). A --system flag on the root command selects the system-level namespace root /root/.orca. Backward compatible: empty ORCA_HOME → ~/.orca. Covers REQ-041, REQ-042.
• P02 — install.sh + in-place update. A 1-liner installer pulls the release binary from the public Gitea release URL, installs at user level by default (~/.local/bin/orca) or system level (/usr/local/bin/orca) with --system. Re-running updates the binary in place while preserving config/db/certs in the namespace dir. Idempotent. Covers REQ-043, REQ-044. Also updates README quickstart (REQ-016 completion).
• P03 — Docker release. A multi-stage Dockerfile builds a distroless image; scripts/release.sh and .coreci.yml publish the image to the Gitea container registry per release. Covers REQ-046.
• P04 — Final review + ship + audit. Milestone release.

The vision ("minimalist, offline-first, CLI-first orchestration engine") is unchanged. v0.5 is a distribution milestone, not a direction change.

v0.5 Clarified Decisions (D-series, full autonomy)

The 5 v0.5 decisions (D-025..D-029) were auto-resolved under full autonomy during the CLARIFY stage:

ID	Question	Decision	Rationale	Confidence
D-025	System-level namespace path layout?	/root/.orca (mirror of user-level ~/.orca)	Consistent shape with user-level; just a different root. Matches the user's "starts at /root" wording. Single dir keeps it simple.	0.90
D-026	Namespace override mechanism at runtime?	Unify on ORCA_HOME as single namespace root for all components (db, certs, init, daemon). Add --system flag that sets root to /root/.orca.	ORCA_HOME already exists for certs; extend to all components. Backward compatible (empty → ~/.orca). One knob, not many.	0.92
D-027	Docker registry target?	Gitea built-in container registry (git.cloudinit.dev/coreci/orca)	Keeps everything in one forge; uses Gitea's native registry. Consistent with REQ-045 (public repo → public image pulls).	0.88
D-028	How to make releases publicly accessible (REQ-045)?	Flip repo visibility to public via tea repos edit coreci/orca --private=false during P0 ship	Simplest path to anonymous downloads; enables both install.sh pulls and docker pulls. Pre-existing .env leak already suppressed via gitleaks baseline + rotate-forward (commit 00127ce).	0.85
D-029	install.sh default version?	Latest release (query Gitea releases API), optional --version vX.Y.Z to pin	Matches typical 1-liner installer UX; users get newest by default, can pin for reproducibility.	0.92

v0.5 Operational prerequisite (P0 ship)

The Gitea repo coreci/orca is currently private (returns 404 unauthenticated). P0 ship flips visibility to public via tea repos edit coreci/orca --private=false so that install.sh can pull release binaries unauthenticated (REQ-045). This is an operational step performed during the P0 ship, verified by an unauth curl against the releases API.

v0.6 Scope Summary — Node Bootstrap & Proxmox

v0.6 is a 3-execution-phase milestone that turns orca init from a bare mkdir into a full single-node cluster bootstrap, and adds Proxmox 8 & 9 as a first-class remote node type joined over SSH with least-privilege role delegation. The engine functionality from v0.1–v0.5 is unchanged; this milestone is about bootstrap ergonomics and heterogeneous node support:

• P01 — orca init full bootstrap. A single orca init call now: (a) creates the namespace dir (~/.orca or /root/.orca with --system); (b) runs all DB migrations including the new 0006 (nodes.kind, nodes.os — backward-compatible nullable columns); (c) bootstraps the internal CA via security.CAInit if ca.crt is absent; (d) generates the server cert via security.GenerateCSR + ca.SignCSR if server.crt is absent; (e) auto-detects the local OS via /etc/os-release ID= field (ubuntu/debian/alpine); (f) registers a localhost node with kind=localhost, os=<detected>, addr=localhost:8443 if no localhost node exists yet. After orca init, orca doctor MUST pass with zero FAILs. Idempotent: re-running orca init is a no-op (or refresh) for already-provisioned artifacts. Covers REQ-047, REQ-048, REQ-049.
• P02 — Proxmox SSH join. orca node join --type proxmox --host <addr> --user root --password <pw> (password via flag or $ORCA_PROXMOX_PASSWORD, never persisted) bootstraps a remote Proxmox 8/9 host via golang.org/x/crypto/ssh (new direct dep). Steps: (1) SSH password-auth; (2) generate or load orca's SSH keypair (~/.orca/orca_ssh_key / .pub, 0600/0644); (3) deploy pubkey to remote ~orca/.ssh/authorized_keys; (4) create orca user (config-overridable name via --proxmox-user, default orca); (5) create PVE custom role OrcaOperator (config-overridable via --proxmox-role) with privileges VM.Audit, Datastore.AllocateSpace, SDN.Use; (6) assign role to orca user on /; (7) drop /etc/sudoers.d/orca allowlist (pct, qm, pvesh, apt-get, dpkg — no shell-escape commands); (8) record node row kind=proxmox, os=pve, audit log. Idempotent re-run. Covers REQ-050, REQ-051.
• P03 — doctor os + doctor proxmox. Extends orca doctor with two new checks: doctor os re-runs /etc/os-release detection and verifies it matches the stored localhost node row's os field (drift = WARN); doctor proxmox iterates kind=proxmox nodes and SSH-probes each with pveversion / pvecmd status (3s timeout per peer per D-038 pattern), reporting PASS/WARN/FAIL per node. All bootstrap + join actions emit structured audit-log entries. Covers REQ-052.
• P04 — Final review + ship + audit. Milestone release.

The vision ("minimalist, offline-first, CLI-first orchestration engine") is unchanged. v0.6 is a bootstrap-ergonomics + heterogeneous- nodes milestone, not a direction change.

v0.6 Clarified Decisions (D-series, full autonomy)

The 8 v0.6 decisions (D-030..D-037) were resolved during the CLARIFY stage — D-030..D-034 confirmed by the operator in plan mode, D-035..D-037 auto-resolved at full autonomy within the clarify_budget:

ID	Question	Decision	Rationale	Confidence
D-030	SSH library for Proxmox join?	golang.org/x/crypto/ssh	Stdlib-adjacent, well-maintained, single new direct dep. Matches orca's minimal-deps ethos. Shell-out to /usr/bin/ssh would require openssh-client on the orca host and complicate password-auth + idempotent pubkey deploy.	0.92 (operator-confirmed)
D-031	Proxmox join password handling?	Flag/env only, never persisted	--password flag or $ORCA_PROXMOX_PASSWORD is used once to deploy the orca pubkey + create the orca user; the password is never written to SQLite. Subsequent orca→Proxmox access uses the deployed SSH key.	0.95 (operator-confirmed)
D-032	Localhost OS auto-detect signal?	/etc/os-release ID= field	Parse ID= from /etc/os-release; map ubuntu/debian/alpine → node os. Falls back to linux (unknown) if none match. Simplest reliable signal across the three target distros.	0.93 (operator-confirmed)
D-033	Least-privilege Proxmox role granularity?	Custom PVE role OrcaOperator with VM.Audit, Datastore.AllocateSpace, SDN.Use + /etc/sudoers.d/orca allowlist (pct, qm, pvesh, apt-get, dpkg)	Config-overridable role + user names. Sufficient for "manage the host, VMs/CTs, storage, packages" without granting root shell. Built-in PVEAuditor is too read-only; full Administrator is too broad.	0.88 (operator-confirmed)
D-034	Node kind/os schema?	Add nodes.kind + nodes.os columns via migration 0006	Schema-first, queryable, doctor can branch on kind. Nullable with localhost/"" defaults for existing rows (backward-compatible). data-engineer owns the migration.	0.94 (operator-confirmed)
D-035	SSH host-key verification on first Proxmox connect?	TOFU: pin on first connect, refuse on mismatch thereafter	First connect uses ssh.InsecureIgnoreHostKey to capture the host key; it is then persisted to ~/.orca/known_hosts (or the nodes metadata) and all subsequent connects require a match. Balances first-run ergonomics against MITM risk on subsequent runs. Switching to pre-pinned keys is a future enhancement.	0.82 (auto)
D-036	orca init idempotency semantics for already-provisioned artifacts?	Skip-and-refresh, never overwrite	If ca.crt exists → load it (no regen). If server.crt exists → keep it (no reissue). If a localhost node row exists → update last_seen + re-detect os, never insert a duplicate. If DB migrations are ahead → no-op. If ~/.orca exists → MkdirAll is a no-op. Idempotent re-run is a hard requirement (REQ-047).	0.95 (auto)
D-037	orca SSH keypair location + algorithm?	~/.orca/orca_ssh_key (0600) + ~/.orca/orca_ssh_key.pub (0644), Ed25519	Ed25519 keys are smaller, faster, and more secure than RSA for SSH auth. Stored in the orca namespace dir alongside ca.crt/server.crt so ORCA_HOME relocation works. File modes mirror the cert file-mode discipline (REQ-033 spirit). Generated lazily on first orca node join --type proxmox, not at orca init (localhost doesn't need SSH).	0.90 (auto)

v0.6 clarification notes

• D-035 TOFU caveat: TOFU (trust-on-first-use) is the standard SSH UX and matches the operator-mediated model from D-012 (CA cert distribution). The operator is expected to verify the host key fingerprint out-of-band on first connect if the network is untrusted. A future milestone may add --host-key-fingerprint pin flag to orca node join --type proxmox for pre-pinned deployments.
• D-036 idempotency: re-running orca init on a node that already has a localhost row updates last_seen and re-detects os (in case the host OS was upgraded) but does NOT change the node ID or joined_at. This makes orca init safe to put in a systemd ExecStartPre or a config-management runbook.
• D-037 Ed25519: golang.org/x/crypto/ssh + golang.org/x/crypto/ed25519 are in the same module; no additional direct dep beyond D-030.

v0.7 Clarified Decisions (D-series, full autonomy)

The 5 v0.7 decisions (D-038..D-042) were auto-resolved at full autonomy within the clarify_budget (10):

ID	Question	Decision	Rationale	Confidence
D-038	Config file format — HCL or YAML?	HCL	D-009 already specced config.hcl. HCL is already a direct dep (hashicorp/hcl/v2 for jobspec). Adding YAML would introduce a second parser dep — violates minimal-deps. Use the existing hclparse pkg from jobspec.	0.93
D-039	Config precedence order (flag vs env vs file vs default)?	flag > env > file > default	Standard layered config: the most explicit (flag) wins, then the runtime (env), then the persisted (file), then the built-in default. Matches cobra/viper convention without the viper dep.	0.92
D-040	pprof security — bind to localhost only, or operator-chosen addr?	Operator-chosen --pprof <addr> (default disabled)	Default disabled keeps the minimalist posture. Operator picks the addr — localhost for dev, unix socket for prod. Separate mux so it never touches the mTLS daemon listener. No auth (pprof is operator-only, addr is the gate).	0.85
D-041	cert command registration — where in root command order?	After cert is unreachable today, append after node in rootCmd.AddCommand order	Alphabetical-ish with the existing cluster (audit, daemon, doctor, init, job, node, cert, status, version). No behavior change to existing commands.	0.88
D-042	Coverage target — 50% floor or higher?	50% floor per package, 70% target for new packages	50% is achievable for the concurrent packages (engine, transport) without heroic mock effort; 70% is the floor for new code in P02/P04. Avoids a "raise coverage everywhere" rathole.	0.85

v0.7 Scope Summary — Hardening & Completion

v0.7 is a 4-execution-phase NFR milestone that closes out gaps surfaced by the v0.7 IDEATE stage: an unreachable command tree, a missing config file layer, low test coverage in core packages, and the long-deferred pprof endpoint. The engine functionality from v0.1–v0.6 is unchanged; this milestone is purely about correctness, coverage, and operability:

• P01 — Register orca cert command tree + cert_repo tests. The internal/cli/cert.go command (cert ca-init, cert gen, cert show, cert renew, cert fingerprint) is fully implemented but never wired into rootCmd. This phase adds the missing rootCmd.AddCommand(newCertCmd(...)) and adds the missing internal/store/cert_repo_test.go. Covers REQ-053.
• P02 — HCL config file parsing (config.hcl). D-009 specified ~/.orca/config.hcl and /etc/orca/orca.hcl as config locations, but no HCL config-file parser exists — the CLI relies entirely on flags and env vars. This phase adds a minimal internal/config package that loads config.hcl (keys: db_path, listen_addr, ca_path, server_cert_path, server_key_path, node_capacity), merges with env/flag overrides (flag > env > file > default), and surfaces it via --config flag on the root command. Covers REQ-054.
• P03 — Test coverage uplift. Adds tests for the lowest-coverage packages: internal/engine (executor, dispatcher, peer — currently 8.3%), internal/transport (mtls, dispatch, handshake_log — currently 26.3%), internal/proxmox (bootstrap SSH path — currently 5.1%), and internal/audit (no tests). Target: every package ≥ 50% coverage. Covers REQ-055.
• P04 — --pprof opt-in on orca daemon. Adds the long-deferred I-308 pprof endpoint behind an opt-in --pprof <addr> flag (default disabled). net/http/pprof mounted on a separate mux so it never touches the mTLS daemon listener. Covers REQ-056.
• P05 — Final review + ship + audit. Milestone release.

The vision ("minimalist, offline-first, CLI-first orchestration engine") is unchanged. v0.7 is a hardening milestone, not a direction change. Milestone type: NFR (all phases are fix/test/chore); the final phase's progressive patch IS the deliverable per run.md versioning logic. Tags run on the v0.6.x patch line: v0.6.0 (P0) … v0.6.5 (P05 = milestone release).

v0.8 Scope Summary — Coverage & Trust Hardening

v0.8 is a 3-execution-phase NFR milestone that continues the hardening theme opened by v0.7. v0.7 P03 (REQ-055) lifted four packages to ≥ 50%, but a coverage re-baseline after v0.7 ship shows the floor was insufficient: internal/engine regressed to 8.3%, internal/proxmox to 5.1%, and four more packages sit between 26% and 48%. Three packages (internal/audit, internal/certpaths, cmd/orca) still have no test files at all. v0.8 also closes the two "future enhancement" hooks explicitly deferred in v0.6 — SSH host-key pre-pinning (D-035 caveat) and orca node key-reset (RESEARCH_v0.6 §80) — and adds a requirements-hygiene gate so the stale-REQ-status drift seen in REQUIREMENTS.md after v0.7 ship cannot recur:

• P01 — Coverage uplift round 2. Raise six under-50% packages to ≥ 70% and add first tests for the three zero-test packages. Covers REQ-057.
• P02 — SSH trust hardening. --host-key-fingerprint pre-pin flag on orca node join --type proxmox + orca node key-reset <node> command. Covers REQ-058, REQ-059.
• P03 — Requirements-hygiene gate. make verify-reqs target + verify-stage assertion that ROADMAP Complete ↔ REQUIREMENTS Complete. Covers REQ-060.
• P04 — Final review + ship + audit. Milestone release.

The vision is unchanged. v0.8 is a hardening milestone, not a direction change. Milestone type: NFR (all phases are test/feat-chore on the trust surface — see CLARIFY D-043 for the feat vs chore classification of P02); the final phase's progressive patch IS the deliverable per run.md versioning logic. Tags run on the v0.7.x patch line: v0.7.0 (P0) … v0.7.4 (P04 = milestone release).

v0.8 Clarified Decisions (D-series, full autonomy)

The 5 v0.8 decisions (D-043..D-047) were auto-resolved at full autonomy within the clarify_budget (10):

ID	Question	Decision	Rationale	Confidence
D-043	Is P02 (SSH trust hardening) a feat phase or a chore phase? It adds a new flag + a new subcommand.	chore (trust-surface hardening), not feat	Both --host-key-fingerprint and orca node key-reset refine the existing orca node join --type proxmox flow and the existing TOFU known_hosts store (D-035). No new orchestration capability, no new node kind, no new API. They close a security gap explicitly deferred in v0.6, not open new surface area. Per run.md versioning logic this keeps v0.8 NFR (all phases fix/test/chore/perf/refactor).	0.84
D-044	Where does --host-key-fingerprint live — on orca node join or only on --type proxmox?	On orca node join (root of the join subcommand), validated when --type proxmox	The flag is generic (any future SSH-joined node kind will use it); gating it to --type proxmox only would require re-adding it later. Validation (flag requires --type proxmox today) happens in RunE, not in the flag declaration, so the flag is declared once on node join and the type check emits a clear error for non-proxmox types until other SSH-joined kinds exist.	0.86
D-045	--host-key-fingerprint format — raw hex, sha256:-prefixed, or OpenSSH SHA256:base64?	OpenSSH SHA256:base64 (the format ssh-keyscan -E sha256 -D - emits and operators expect)	Matches the fingerprint format operators already see from ssh-keyscan and orca node join's own Result.HostKeyFingerprint output. Accept only SHA256:-prefixed base64; reject raw hex with a clear error. Internally decode base64 → compare against ssh.PublicKey Marshal + sha256.	0.88
D-046	Does orca node key-reset <node> also revoke the orca pubkey on the remote host, or only clear the local known_hosts entry?	Local known_hosts entry only	Revoking the remote authorized_keys entry would orphan a working node (next dispatch would fail auth). key-reset is the local "forget this host's key" operation (mirrors ssh-keygen -R host); re-establishing trust is a separate orca node join re-run. Audit-log the reset with actor, node, event=node.key_reset.	0.90
D-047	Coverage target for P01 — 70% floor or higher?	70% floor for the 6 under-50% packages; 50% floor for the 3 zero-test packages (internal/audit, internal/certpaths, cmd/orca) as a first-toe-hold	70% across the board for the already-tested packages matches D-042's "70% target for new packages" and is achievable without heroic mock effort. For the zero-test packages, going 0→50% is the realistic single-phase step (0→70% risks a coverage rathole on cmd/orca which is glue code); a future milestone can lift them to 70%.	0.82

---

v0.9/v0.10 — Re-architecture Scope Summary (Supersedes v0.1–v0.8 architecture)

v0.9 is the first DIRECTION-CHANGE milestone in the project's history. It supersedes the shipped v0.1–v0.8 architecture per the adopted PRD (.ciagent/PRD_v0.9.md). The re-architecture deprecates the daemon/ transport/internal-CA/HCL/single-namespace stack and builds a CLI-only/ SSH-push/step-ca/Markdown-frontmatter/multi-namespace stack plus 8 net-new subsystems.

Override Justification (Re-architecture Justification axis)

The ci-griller returned REPLAN (0.70) on the Re-architecture Justification axis, noting the PRD reverses 6 documented decisions without new evidence and that the incremental-additive path was not evaluated. The user reviewed the fork and overrode the direction with a six-part evidence basis. The override is recorded verbatim below; each part addresses a reversal that the grill flagged as unjustified.

1. The v0.8 daemon model is operationally failing in the target environment — R-001 ("no orca binary on any server") is a response to measured pain, not preference.
2. step-ca is externally mandated (D-101) — the operator environment requires an external CA; AD-010's "too heavyweight" rationale is no longer operative.
3. Multi-tenancy is a hard product requirement (R-002) — real multi-tenant use cases cannot be served by the single-namespace layout; the "no multi-tenancy" anti-pattern is obsolete.
4. WASM is a hard workload requirement (D-088) — workloads are WASM, not processes; os/exec is insufficient; the "no container runtime" anti-pattern is reversed.
5. SSH-push is the only viable deployment target for the operator's bare-Linux/Proxmox environment — installing/maintaining an orca daemon on every peer is operationally infeasible.
6. Simplicity/vision correction — the v0.1-v0.8 daemon model was a wrong turn against the original CLI-first vision; the re-architecture corrects the vision.

Supersession Table (AD-series reversals, recorded per grill PC-09)

Old decision	Was	Superseded by	Evidence basis
AD-010 (ARCHITECTURE.md:463)	step-ca/cfssl/vault-pki "too heavyweight"	D-101 (step-ca)	Override ground 2 (external mandate)
SPIFFE rejection (PROJECT.md:94)	internal CA chosen over SPIFFE	D-068 (SPIFFE SVIDs)	Override ground 3 (multi-tenancy requires per-workload identity)
No-container-runtime (ARCHITECTURE.md:477)	explicit anti-pattern	D-088 (5 runtimes; wasmtime primary)	Override ground 4 (WASM is the workload profile)
No-multi-tenancy (ARCHITECTURE.md:478)	explicit anti-pattern	D-158 / R-002 (many namespaces under ORCA_HOME)	Override ground 3 (hard multi-tenant product req)
AD-007 (HCL canonical)	HCL for jobspec	R-013 / R-014 (Markdown canonical; HCL legacy)	PRD §8 (Markdown + body preservation is the operator-facing format)
Daemon-on-every-node	orca daemon on all peers	R-001 (no orca binary on any server)	Override grounds 1 + 5 (daemon failing; SSH-push only viable target)

The 19 binding conditions (C-01..C-19) and 10 phase challenges (PC-01..PC-10) from GRILL_v0.9.md are adopted as execution gates. The 30 net-new requirements (REQ-061..REQ-090) from IDEATION_v0.9.md are recorded in REQUIREMENTS.md. The reordered phase plan is in ROADMAP.md.

v0.9 Clarified Decisions (D-series, full autonomy — Phase 0 pre-execution)

ID	Question	Decision	Rationale	Confidence
D-101	Cluster CA: internal Go CA (AD-010) or step-ca (external)?	step-ca (apt-installed)	Externally mandated per override ground 2; AD-010's "too heavyweight" rationale reversed. CLI wraps step CLI via SSH (no Go step-ca client library — keep zero-new-dep posture if possible, or add github.com/smallstep/cli as a dep). Gated by C-07 (CA migration spec).	0.74
D-068	Workload identity: internal X.509 CA or SPIFFE SVIDs?	SPIFFE SVIDs minted at submit time via step-ca	Multi-tenancy (override ground 3) requires per-workload identity model; SPIFFE is the standard. SPIFFE ID spiffe://orca/ns/<ns>/job/<name>/alloc/<id> as SAN. Gated by C-08 (mint spike in v0.10-P01.5; fallback to mTLS identity if spike fails).	0.72
D-088	Runtime: direct os/exec only (D-008) or multi-runtime?	5 runtimes: wasm (wasmtime primary), podman, process, pve-vm, pve-ct	WASM is the primary workload (override ground 4). processRuntime wraps existing executor.go; others are net-new. Split P07a/b/c per grill PC-10. P07b gated by C-01 (wasmtime/CGO eval).	0.82
D-158	Namespace model: single flat root or multi-namespace?	Multi-namespace under ORCA_HOME (R-002)	Hard multi-tenant product requirement (override ground 3). _defaults/ implicit root; cluster/ for cluster-wide; per-namespace db/, .env, .env.secrets, jobs/, alloc/, ns.md. No namespace column in SQLite.	0.84
D-179	Jobspec format: HCL canonical (AD-007) or Markdown?	Markdown with YAML frontmatter canonical (R-013); HCL legacy	PRD §8 — Markdown + body preservation is the operator-facing format. HCL adapter (REQ-064) preserves orca job run old-spec.hcl during migration.	0.85
D-185	Re-architecture justification: incremental additive or full re-architecture?	Full re-architecture (overridden by user)	Six-part evidence basis above; the grill's REPLAN mechanics (PC-01..PC-10, C-01..C-19) adopted as gates. The incremental-additive path was evaluated and rejected on grounds 1 + 5 (daemon failing; SSH-push only viable).	0.88
D-187	wasmtime Go binding (bytecodealliance/wasmtime-go) is CGO-based — does adopting it revoke D-002 (modernc/sqlite CGO-free cross-compile story)?	Use the wasmtime CLI (apt-installed on peer) via SSH exec; do NOT import wasmtime-go.	The Go binding links libwasmtime via cgo and would revoke D-002's CGO-free cross-compile story. The CLI-via-SSH approach (same pattern as podman/qm/pct) avoids CGO entirely. internal/runtime/wasm.go imports only stdlib + sshpush. CGO_ENABLED=0 go build ./... succeeds. C-01 grill gate SATISFIED; D-002 NOT revoked. Full evaluation in internal/runtime/C01_WASMTIME_CGO_EVAL.md.	0.90

---

v0.10 Docs & Install Milestone — Scope Summary

v0.10 is a focused milestone that closes the documentation gap left by the v0.9 re-architecture and fixes the release/install pipeline bug that caused install.sh to resolve to v0.4.5 instead of the latest release. The v0.9 re-architecture shipped a complete CLI surface (markdown jobspec, orca ns, orca node capacity, CLI-side scheduler, emitters, Traefik ingress) but no operator-facing reference documentation. This milestone ships that documentation plus a worked full-stack example with ingress configured, and hardens the release pipeline so every Gitea release carries a Linux binary asset.

Root cause of the v0.4.5 install

The v0.8.x releases (v0.8.0 through v0.8.15) shipped with zero binary assets attached to their Gitea releases. scripts/install.sh resolves "latest" by hitting /releases/latest (returns v0.8.15), then looks for orca-v0.8.15-linux-amd64.tar.gz in that release's assets. Since the asset is missing, install.sh errors out — there is no fallback walk to older releases that DO carry a binary. The user's v0.4.5 install came from an earlier run or a pinned --version. The fix is forward: harden scripts/release.sh to cross-build the amd64 tarball and verify the asset attached post-create; harden scripts/install.sh to walk backward through releases if the latest lacks the asset.

v0.10 Phases

• Phase 0 (pre-execution): specify → clarify → research → ideate → plan → grill. Tag v0.9.0.
• Phase P1 — release/install fix (REQ-097, REQ-098): cross-build amd64 tarball in release.sh, post-create asset verification, install.sh fallback walk. Tag v0.9.1.
• Phase P2 — CLI + jobspec + ingress docs (REQ-091, REQ-092, REQ-093): docs/cli.md, docs/jobspec.md, docs/ingress.md. Tag v0.9.2.
• Phase P3 — full-stack examples (REQ-094): examples/full-stack/ with 5 valid jobspecs + rendered artifacts + walkthrough README. Tag v0.9.3.
• Phase P4 — README + namespace.md refresh (REQ-095, REQ-096): README subcommand table + install example + docs/examples sections; docs/namespace.md v0.9 layout. Tag v0.9.4.
• Phase P5 — final review + ship + audit (milestone release). Tag v0.9.5 = v0.10.0 milestone release.

Milestone type: feature (P1 ships fix phases; P2/P3/P4 ship docs phases; at least one non-docs phase makes this a feature milestone per the versioning logic). Tags run on the v0.9.x patch line. The milestone branch label is milestone/v0.10-docs-cli-examples.

The vision ("minimalist, offline-first, CLI-first orchestration engine") is unchanged. v0.10 is a documentation + install-hardening milestone, not a direction change. It builds on the v0.9 re-architecture foundation without modifying any Go orchestration code.

v0.10 Clarified Decisions (D-series, full autonomy — Phase 0 pre-execution)

ID	Question	Decision	Rationale	Confidence
D-188	Should the CLI docs be a single docs/cli.md reference or a per-command docs/cli/ subdirectory?	Single docs/cli.md reference	Mirrors the existing flat docs/ pattern (install.md, docker.md, namespace.md, security-scanning.md). One file is more discoverable for a CLI tool and avoids navigation overhead. A per-command subdirectory diverges from the established layout.	0.92
D-189	Should the examples live in examples/full-stack/ or in testdata/?	examples/full-stack/ as a new top-level directory	testdata/ holds legacy HCL fixtures (hello.hcl, fail.hcl) used by Go tests; mixing operator-facing examples with test fixtures conflates audiences. A new examples/ directory is the conventional location for worked examples and is what an operator expects to find.	0.93
D-190	How deep should the ingress/Traefik documentation go?	Dedicated docs/ingress.md plus a worked example in examples/full-stack/	Ingress is the user's explicit ask ("full stack with ingress configured") and the Traefik/service-block model (R-007 socket vs TCP, atomic reload, drain, TLS) is non-trivial. A dedicated doc is the clearest answer; a section buried in docs/cli.md would be less discoverable.	0.90
D-191	Should the docs frame the v0.9 canonical path or document both v0.8 and v0.9 equally?	Document the v0.9 canonical path; flag deprecated surface with callout boxes	The v0.8 daemon/mTLS/HCL path is deprecated and scheduled for removal in v0.10-P14. Documenting it as primary misleads new operators; documenting both equally doubles the surface and risks documenting soon-removed code. Callout boxes with "deprecated in v0.9, removed in v0.10" point operators to the canonical path.	0.91
D-192	Should the existing v0.8.15 release be backfilled with a binary asset, or only fix the pipeline forward?	Fix forward only; no backfill	Backfilling a past release is an ops task, not a docs milestone deliverable. The next tagged phase (this milestone's P1 ship at v0.9.1) will be the first correctly-asseted release; install.sh's new fallback walk handles the gap until then.	0.88
D-193	Should release.sh build only linux-amd64 or also linux-arm64?	Cross-build linux-amd64 explicitly (host-arch-independent); arm64 deferred to a follow-up	The install.sh user base is amd64 today (the .coreci.yml release step hardcodes --asset orca-${VERSION}-linux-amd64.tar.gz). Building amd64 regardless of host arch (via GOOS=linux GOARCH=amd64 go build) guarantees the asset the install script expects. arm64 support is a separate enhancement.	0.85
D-194	Should install.sh add a --check dry-run mode?	Yes, lightweight	A dry-run mode (--check) that prints the version + asset URL + install path without writing is cheap to add and useful for debugging the "which release will I get?" question that the v0.4.5 incident surfaced.	0.80