# Project: Orca

## What This Is

A minimalist, offline-first, CLI-first orchestration engine inspired by HashiCorp Nomad, prioritizing stability, security, and simplicity over feature richness. Single-binary distribution, no container runtime, no cloud dependencies, no K8s-level complexity.

## Vision

A minimalist, offline-first, CLI-first orchestration engine inspired by HashiCorp Nomad, prioritizing stability, security, and simplicity over feature richness.

## Objective
Build a lightweight system to manage and execute workloads across a set of nodes, keeping complexity far below that of Kubernetes.

## Requirements
- **CLI First**: Primary interaction through a CLI tool.
- **Offline First**: Functional without constant internet connectivity.
- **AI First**: Designed to be easily discoverable and manageable by AI agents.
- **Stability & Security**: Prioritize security fixes and bug fixes over new features.
- **Language**: Written in Go 1.25+.
- **Simplicity**: Minimalist implementation, avoiding the "K8s complexity trap".

## Constraints
- No web UI as a primary requirement.
- Must not implement K8s-level complexity.
- Feature development must move slowly to ensure stability.
- Only CI system allowed: CoreCI (git.cloudinit.dev/coreci/coreci).
- Gitea remote: git.cloudinit.dev/coreci/orca.

## Clarified Decisions (D-series, full autonomy)

| ID | Question | Decision | Rationale | Confidence |
|----|----------|----------|-----------|------------|
| D-001 | Single binary or multi-binary distribution? | **Single binary** | Simpler distribution; subcommands baked into one `orca` binary. Aligns with simplicity pillar. | 0.95 |
| D-002 | Local state store technology? | **modernc/sqlite (pure Go, CGO-free)** | Cross-compile friendly, no CGO dependency, single file on disk, mature. | 0.92 |
| D-003 | Inter-node communication? | **Embedded HTTP (net/http) over loopback, mTLS for cross-node** | No external RPC framework needed for v0.1. HTTP suffices. | 0.85 |
| D-004 | Scheduling algorithm for v0.1? | **Single-node only (no scheduling)** | Multi-node scheduling is out of scope for v0.1. Tasks run on the node they're submitted to. | 0.90 |
| D-005 | CLI output format? | **Human-readable by default, `--json` flag for machine consumption** | Serves both humans and AI agents. | 0.95 |
| D-006 | Job/task definition format? | **HCL or YAML in `.hcl`/`.yaml` files** | Familiar to Nomad/HashiCorp users; simpler than JSON for humans. | 0.88 |
| D-186 | Bash scripts coverage gate: count toward Go gate or exempt? | **Exempt from Go coverage gate; compensating control: bats tests (C-15) + shellcheck + shfmt in CI; every script must have >=1 happy-path and >=1 failure-path bats test** | Bash is a different language surface from Go; the 70%/50% Go coverage gate (D-042/D-047) is Go-specific. Forcing bash into the Go gate would require a coverage tool that does not exist for bash. The compensating control (bats + shellcheck + shfmt) provides equivalent discipline. | 0.82 |
| D-007 | Authentication? | **mTLS for v0.1, token-based deferred** | mTLS is the most secure default. Tokens can be added later if needed. | 0.80 |
| D-008 | Container runtime? | **Direct process execution (no container runtime) for v0.1** | Avoids the Docker/container dependency. Pure process management. | 0.85 |
| D-009 | Configuration file location? | **`~/.orca/config.hcl` and `/etc/orca/orca.hcl`** | Standard XDG-style paths. | 0.90 |
| D-010 | Logging format? | **Structured JSON via `log/slog`** | Native Go 1.21+ slog, no external dependency. | 0.95 |
| D-011 | v0.2 mTLS cert authority model? | **Internal CA with CSR join** | One node bootstraps a local CA; peers generate CSRs and submit them to the CA for signing. CA cert is the trust anchor. More secure than self-signed per-node (single trust root) without the operational complexity of an external PKI. | 0.92 |
| D-012 | v0.2 CA bootstrap & cert distribution? | **Operator-mediated, fingerprint-verified** | Bootstrap node writes `~/.orca/ca.crt` and `~/.orca/ca.key` (mode 0600). Operator copies `ca.crt` to peers; peers verify by SHA-256 fingerprint at `orca node join --ca-fingerprint <sha256>`. No automated secret distribution. | 0.85 |
| D-013 | v0.2 cert validity & rotation? | **Server certs 90 days, CA cert 10 years, rotate 30 days before expiry** | Server certs are short-lived (compromise window small); CA is long-lived (manual rotation is expensive). `orca cert renew` reissues server certs automatically. | 0.90 |
| D-014 | v0.2 mTLS handshake timing? | **Eager — at `orca node join` time** | Fail fast on bad certs, misconfigurations, or CA mismatches. Lazy handshake would let stale configs run until first request, complicating debugging. | 0.88 |
| D-015 | v0.2 minimum TLS version & cipher suites? | **TLS 1.3 only; AEAD cipher allowlist** | MinVersion=tls.VersionTLS13, CipherSuites limited to TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_AES_128_GCM_SHA256. No TLS 1.2 fallback. | 0.92 |
| D-016 | v0.2 security-scanning placement? | **`validate` pipeline of `.coreci.yml`, gates merges to main** | `gosec` baseline JSON checked into repo; new findings fail the build. `govulncheck ./...` exit-on-known. Pre-commit hook with `gitleaks` is opt-in (developer machine). | 0.85 |
| D-017 | v0.2 `iter.Seq` API surface? | **`orca job list --watch` and `orca node list --watch`** | Pull-based `iter.Seq[Job]` / `iter.Seq[Node]`; cancellation via `context.Context`; `signal.NotifyContext` on ctrl-c. Backpressure is implicit (consumer-driven). | 0.90 |
| D-018 | v0.2 multi-node scheduling algorithm? | **Bin-packing by available CPU/memory, FIFO within a node** | Simple, deterministic, matches D-004 minimalism. Cross-node dispatch via ConnectRPC `orca.v1.Dispatch` service. Retry on transient failures with exponential backoff. | 0.85 |

## Out of Scope
- Full-blown Kubernetes-compatible API.
- Complex cloud-provider integrations.
- GUI-based management consoles.
- Container runtime integration.
- Service mesh / sidecar injection.
- Auto-scaling / horizontal pod autoscaler.
- External PKI / Let's Encrypt / cert transparency logs.
- gRPC framework dependency (ConnectRPC in `config.json` frameworks but
  not in `go.mod`; v0.2 uses stdlib `net/http` with h2c for
  `orca.v1.Dispatch` — see ARCHITECTURE.md AD-014).

## v0.2 Scope Summary

v0.2 is a focused 4-phase milestone that turns Orca from a single-node
process executor into a small cluster engine with strong transport
security and richer I/O. The 4 phases are:

- **P01 — mTLS handshake + internal CA with CSR join.** Internal CA, CSR
  join, eager handshake at `node join`, TLS 1.3 + AEAD allowlist.
  See ARCHITECTURE.md Flow 1 + Flow 2.
- **P02 — Multi-node scheduling & job dispatch.** Best-fit bin-packing by
  CPU/memory, FIFO within a node, `orca.v1.Dispatch` over mTLS. See
  ARCHITECTURE.md Flow 3.
- **P03 — `gosec` + `govulncheck` + `gitleaks` in CI.** `gosec` baseline
  JSON in repo, `govulncheck ./...` in `validate` pipeline, `gitleaks`
  in pre-commit (opt-in).
- **P04 — `iter.Seq` streaming for `--watch` flags.** Go 1.25+ range-over-func
  semantics, `context.Context` cancellation, `signal.NotifyContext` on
  ctrl-c. See ARCHITECTURE.md Flow 4.

The vision ("minimalist, offline-first, CLI-first orchestration engine")
is unchanged. v0.2 is a hardening + small-cluster extension, not a
direction change.

## Key Decisions

The 18 D-series decisions (D-001..D-018) are recorded in the "Clarified
Decisions" table above. The 10 v0.1 decisions (D-001..D-010) are stable
and unchanged in v0.2. The 8 v0.2 decisions (D-011..D-018) were
auto-resolved under full autonomy and are summarized here:

- **D-011: Internal CA with CSR join** (vs. self-signed per-node or SPIFFE).
  Single trust root, no external PKI, CSR workflow.
- **D-012: Operator-mediated CA cert distribution with fingerprint verify**
  (no automated secret distribution — matches offline-first principle).
- **D-013: 90d server certs, 10y CA cert, 30d pre-expiry rotation.**
- **D-014: Eager mTLS handshake at `orca node join` time** (fail fast).
- **D-015: TLS 1.3 only, AEAD cipher allowlist** (no TLS 1.2 fallback).
- **D-016: `gosec`+`govulncheck` in `validate` pipeline of `.coreci.yml`**
  (gates merges to main). `gitleaks` in pre-commit (opt-in).
- **D-017: `iter.Seq` for `orca job list --watch` and `orca node list --watch`**
  (pull-based, ctx cancellation, ctrl-c via `signal.NotifyContext`).
- **D-018: Bin-packing by CPU/memory with FIFO within node; JSON-over-HTTP
  orca.v1.Dispatch for cross-node** (no ConnectRPC dep).

## v0.3 Clarified Decisions (D-series, full autonomy)

v0.3 is a lean 2-execution-phase milestone completing the streaming and
doctor work deferred from v0.2. The 6 v0.3 decisions (D-019..D-024)
were auto-resolved under full autonomy:

| ID | Question | Decision | Rationale | Confidence |
|----|----------|----------|-----------|------------|
| D-019 | Watch refresh mechanism? | **Poll-based, 1s ticker** | Simpler than event channel; no daemon coupling for CLI; matches offline-first. | 0.90 |
| D-020 | Watch output format (REQ-030)? | **Table by default; `--watch --json` streams one-line JSON per event** | Consistent with D-005 `--json` convention; serves humans + AI agents. | 0.92 |
| D-021 | Doctor network check scope? | **Probe configured peer addresses via mTLS `/healthz` handshake; PASS/WARN/FAIL per peer** | Reuses existing transport client; read-only. | 0.85 |
| D-022 | Doctor db check scope? | **`PRAGMA integrity_check` + migration version query** | Already specced in ARCHITECTURE.md §5; minimal surface. | 0.95 |
| D-023 | iter.Seq cancellation? | **`signal.NotifyContext` on SIGINT/SIGTERM** | Per D-017 + ARCHITECTURE Flow 4. | 0.95 |
| D-024 | `--watch` applies to job list only, or node list too? | **Both `orca job list --watch` and `orca node list --watch`** | Per ARCHITECTURE.md CLI layer + D-017. | 0.92 |

## v0.3 Scope Summary

v0.3 is a focused 2-execution-phase milestone completing the work
deferred from v0.2 that was NOT already shipped in P08-P10. A codebase
audit during re-init SPECIFY confirmed that REQ-014, REQ-027, REQ-028,
REQ-029, REQ-031, REQ-037, REQ-039, REQ-040 all shipped in P08-P10
despite stale REQUIREMENTS.md marking them Pending. The remaining work:

- **P01 — `iter.Seq` streaming for `--watch` flags.** Go 1.25+
  range-over-func semantics, pull-based `iter.Seq[Job]` /
  `iter.Seq[Node]`, `context.Context` cancellation,
  `signal.NotifyContext` on ctrl-c. Applies to both `orca job list
  --watch` and `orca node list --watch`. Covers REQ-022, REQ-030.
- **P02 — `orca doctor` network + db full implementation.** Replaces
  the P01 stubs (`NetworkStub`, `DBStub`) with real checks: peer
  reachability via mTLS `/healthz` probe; SQLite `PRAGMA
  integrity_check` + migration version. Covers REQ-032 (completion).

The vision ("minimalist, offline-first, CLI-first orchestration
engine") is unchanged. v0.3 is a completion milestone, not a direction
change.

## v0.5 Scope Summary — Distribution

v0.5 is a 3-execution-phase milestone that makes Orca installable,
distributable, and containerized. The engine functionality from
v0.1–v0.3 is unchanged; this milestone is purely about **delivery
surface**:

- **P01 — Namespace unification.** A single `ORCA_HOME` environment
  variable becomes the namespace root for *all* on-disk state (db,
  certs, init, daemon). A `--system` flag on the root command selects
  the system-level namespace root `/root/.orca`. Backward compatible:
  empty `ORCA_HOME` → `~/.orca`. Covers REQ-041, REQ-042.
- **P02 — `install.sh` + in-place update.** A 1-liner installer pulls
  the release binary from the public Gitea release URL, installs at
  user level by default (`~/.local/bin/orca`) or system level
  (`/usr/local/bin/orca`) with `--system`. Re-running updates the
  binary in place while preserving config/db/certs in the namespace
  dir. Idempotent. Covers REQ-043, REQ-044. Also updates README
  quickstart (REQ-016 completion).
- **P03 — Docker release.** A multi-stage `Dockerfile` builds a
  distroless image; `scripts/release.sh` and `.coreci.yml` publish the
  image to the Gitea container registry per release. Covers REQ-046.
- **P04 — Final review + ship + audit.** Milestone release.

The vision ("minimalist, offline-first, CLI-first orchestration
engine") is unchanged. v0.5 is a distribution milestone, not a
direction change.

## v0.5 Clarified Decisions (D-series, full autonomy)

The 5 v0.5 decisions (D-025..D-029) were auto-resolved under full
autonomy during the CLARIFY stage:

| ID | Question | Decision | Rationale | Confidence |
|----|----------|----------|-----------|------------|
| D-025 | System-level namespace path layout? | **`/root/.orca`** (mirror of user-level `~/.orca`) | Consistent shape with user-level; just a different root. Matches the user's "starts at /root" wording. Single dir keeps it simple. | 0.90 |
| D-026 | Namespace override mechanism at runtime? | **Unify on `ORCA_HOME`** as single namespace root for all components (db, certs, init, daemon). Add `--system` flag that sets root to `/root/.orca`. | `ORCA_HOME` already exists for certs; extend to all components. Backward compatible (empty → `~/.orca`). One knob, not many. | 0.92 |
| D-027 | Docker registry target? | **Gitea built-in container registry** (`git.cloudinit.dev/coreci/orca`) | Keeps everything in one forge; uses Gitea's native registry. Consistent with REQ-045 (public repo → public image pulls). | 0.88 |
| D-028 | How to make releases publicly accessible (REQ-045)? | **Flip repo visibility to public** via `tea repos edit coreci/orca --private=false` during P0 ship | Simplest path to anonymous downloads; enables both install.sh pulls and docker pulls. Pre-existing `.env` leak already suppressed via gitleaks baseline + rotate-forward (commit 00127ce). | 0.85 |
| D-029 | install.sh default version? | **Latest release** (query Gitea releases API), optional `--version vX.Y.Z` to pin | Matches typical 1-liner installer UX; users get newest by default, can pin for reproducibility. | 0.92 |

### v0.5 Operational prerequisite (P0 ship)

The Gitea repo `coreci/orca` is currently **private** (returns 404
unauthenticated). P0 ship flips visibility to public via `tea repos
edit coreci/orca --private=false` so that `install.sh` can pull
release binaries unauthenticated (REQ-045). This is an operational
step performed during the P0 ship, verified by an unauth `curl`
against the releases API.

## v0.6 Scope Summary — Node Bootstrap & Proxmox

v0.6 is a 3-execution-phase milestone that turns `orca init` from a
bare `mkdir` into a full single-node cluster bootstrap, and adds
Proxmox 8 & 9 as a first-class remote node type joined over SSH with
least-privilege role delegation. The engine functionality from
v0.1–v0.5 is unchanged; this milestone is about **bootstrap
ergonomics** and **heterogeneous node support**:

- **P01 — `orca init` full bootstrap.** A single `orca init` call now:
  (a) creates the namespace dir (`~/.orca` or `/root/.orca` with
  `--system`); (b) runs all DB migrations including the new 0006
  (`nodes.kind`, `nodes.os` — backward-compatible nullable columns);
  (c) bootstraps the internal CA via `security.CAInit` if `ca.crt` is
  absent; (d) generates the server cert via `security.GenerateCSR` +
  `ca.SignCSR` if `server.crt` is absent; (e) auto-detects the local
  OS via `/etc/os-release` `ID=` field (ubuntu/debian/alpine); (f)
  registers a `localhost` node with `kind=localhost`, `os=<detected>`,
  `addr=localhost:8443` if no localhost node exists yet. After
  `orca init`, `orca doctor` MUST pass with zero FAILs. Idempotent:
  re-running `orca init` is a no-op (or refresh) for already-provisioned
  artifacts. Covers REQ-047, REQ-048, REQ-049.
- **P02 — Proxmox SSH join.** `orca node join --type proxmox --host
  <addr> --user root --password <pw>` (password via flag or
  `$ORCA_PROXMOX_PASSWORD`, **never persisted**) bootstraps a remote
  Proxmox 8/9 host via `golang.org/x/crypto/ssh` (new direct dep).
  Steps: (1) SSH password-auth; (2) generate or load orca's SSH
  keypair (`~/.orca/orca_ssh_key` / `.pub`, 0600/0644); (3) deploy
  pubkey to remote `~orca/.ssh/authorized_keys`; (4) create `orca`
  user (config-overridable name via `--proxmox-user`, default `orca`);
  (5) create PVE custom role `OrcaOperator` (config-overridable via
  `--proxmox-role`) with privileges `VM.Audit`,
  `Datastore.AllocateSpace`, `SDN.Use`; (6) assign role to `orca`
  user on `/`; (7) drop `/etc/sudoers.d/orca` allowlist (`pct`, `qm`,
  `pvesh`, `apt-get`, `dpkg` — no shell-escape commands); (8) record
  node row `kind=proxmox`, `os=pve`, audit log. Idempotent re-run.
  Covers REQ-050, REQ-051.
- **P03 — `doctor os` + `doctor proxmox`.** Extends `orca doctor`
  with two new checks: `doctor os` re-runs `/etc/os-release` detection
  and verifies it matches the stored localhost node row's `os` field
  (drift = WARN); `doctor proxmox` iterates `kind=proxmox` nodes and
  SSH-probes each with `pveversion` / `pvecmd status` (3s timeout per
  peer per D-038 pattern), reporting PASS/WARN/FAIL per node. All
  bootstrap + join actions emit structured audit-log entries. Covers
  REQ-052.
- **P04 — Final review + ship + audit.** Milestone release.

The vision ("minimalist, offline-first, CLI-first orchestration
engine") is unchanged. v0.6 is a bootstrap-ergonomics + heterogeneous-
nodes milestone, not a direction change.

## v0.6 Clarified Decisions (D-series, full autonomy)

The 8 v0.6 decisions (D-030..D-037) were resolved during the CLARIFY
stage — D-030..D-034 confirmed by the operator in plan mode, D-035..D-037
auto-resolved at full autonomy within the `clarify_budget`:

| ID | Question | Decision | Rationale | Confidence |
|----|----------|----------|-----------|------------|
| D-030 | SSH library for Proxmox join? | **`golang.org/x/crypto/ssh`** | Stdlib-adjacent, well-maintained, single new direct dep. Matches orca's minimal-deps ethos. Shell-out to `/usr/bin/ssh` would require openssh-client on the orca host and complicate password-auth + idempotent pubkey deploy. | 0.92 (operator-confirmed) |
| D-031 | Proxmox join password handling? | **Flag/env only, never persisted** | `--password` flag or `$ORCA_PROXMOX_PASSWORD` is used once to deploy the orca pubkey + create the `orca` user; the password is never written to SQLite. Subsequent orca→Proxmox access uses the deployed SSH key. | 0.95 (operator-confirmed) |
| D-032 | Localhost OS auto-detect signal? | **`/etc/os-release` `ID=` field** | Parse `ID=` from `/etc/os-release`; map `ubuntu`/`debian`/`alpine` → node `os`. Falls back to `linux` (unknown) if none match. Simplest reliable signal across the three target distros. | 0.93 (operator-confirmed) |
| D-033 | Least-privilege Proxmox role granularity? | **Custom PVE role `OrcaOperator`** with `VM.Audit`, `Datastore.AllocateSpace`, `SDN.Use` + `/etc/sudoers.d/orca` allowlist (`pct`, `qm`, `pvesh`, `apt-get`, `dpkg`) | Config-overridable role + user names. Sufficient for "manage the host, VMs/CTs, storage, packages" without granting root shell. Built-in `PVEAuditor` is too read-only; full `Administrator` is too broad. | 0.88 (operator-confirmed) |
| D-034 | Node kind/os schema? | **Add `nodes.kind` + `nodes.os` columns via migration 0006** | Schema-first, queryable, doctor can branch on kind. Nullable with `localhost`/`""` defaults for existing rows (backward-compatible). data-engineer owns the migration. | 0.94 (operator-confirmed) |
| D-035 | SSH host-key verification on first Proxmox connect? | **TOFU: pin on first connect, refuse on mismatch thereafter** | First connect uses `ssh.InsecureIgnoreHostKey` to capture the host key; it is then persisted to `~/.orca/known_hosts` (or the nodes metadata) and all subsequent connects require a match. Balances first-run ergonomics against MITM risk on subsequent runs. Switching to pre-pinned keys is a future enhancement. | 0.82 (auto) |
| D-036 | `orca init` idempotency semantics for already-provisioned artifacts? | **Skip-and-refresh, never overwrite** | If `ca.crt` exists → load it (no regen). If `server.crt` exists → keep it (no reissue). If a localhost node row exists → update `last_seen` + re-detect `os`, never insert a duplicate. If DB migrations are ahead → no-op. If `~/.orca` exists → MkdirAll is a no-op. Idempotent re-run is a hard requirement (REQ-047). | 0.95 (auto) |
| D-037 | orca SSH keypair location + algorithm? | **`~/.orca/orca_ssh_key` (0600) + `~/.orca/orca_ssh_key.pub` (0644), Ed25519** | Ed25519 keys are smaller, faster, and more secure than RSA for SSH auth. Stored in the orca namespace dir alongside ca.crt/server.crt so `ORCA_HOME` relocation works. File modes mirror the cert file-mode discipline (REQ-033 spirit). Generated lazily on first `orca node join --type proxmox`, not at `orca init` (localhost doesn't need SSH). | 0.90 (auto) |

### v0.6 clarification notes

- **D-035 TOFU caveat**: TOFU (trust-on-first-use) is the standard SSH
  UX and matches the operator-mediated model from D-012 (CA cert
  distribution). The operator is expected to verify the host key
  fingerprint out-of-band on first connect if the network is
  untrusted. A future milestone may add `--host-key-fingerprint` pin
  flag to `orca node join --type proxmox` for pre-pinned deployments.
- **D-036 idempotency**: re-running `orca init` on a node that already
  has a localhost row updates `last_seen` and re-detects `os` (in case
  the host OS was upgraded) but does NOT change the node `ID` or
  `joined_at`. This makes `orca init` safe to put in a systemd
  ExecStartPre or a config-management runbook.
- **D-037 Ed25519**: `golang.org/x/crypto/ssh` + `golang.org/x/crypto/ed25519`
  are in the same module; no additional direct dep beyond D-030.

## v0.7 Clarified Decisions (D-series, full autonomy)

The 5 v0.7 decisions (D-038..D-042) were auto-resolved at full autonomy
within the `clarify_budget` (10):

| ID | Question | Decision | Rationale | Confidence |
|----|----------|----------|-----------|------------|
| D-038 | Config file format — HCL or YAML? | **HCL** | D-009 already specced `config.hcl`. HCL is already a direct dep (hashicorp/hcl/v2 for jobspec). Adding YAML would introduce a second parser dep — violates minimal-deps. Use the existing `hclparse` pkg from jobspec. | 0.93 |
| D-039 | Config precedence order (flag vs env vs file vs default)? | **flag > env > file > default** | Standard layered config: the most explicit (flag) wins, then the runtime (env), then the persisted (file), then the built-in default. Matches cobra/viper convention without the viper dep. | 0.92 |
| D-040 | pprof security — bind to localhost only, or operator-chosen addr? | **Operator-chosen `--pprof <addr>` (default disabled)** | Default disabled keeps the minimalist posture. Operator picks the addr — localhost for dev, unix socket for prod. Separate mux so it never touches the mTLS daemon listener. No auth (pprof is operator-only, addr is the gate). | 0.85 |
| D-041 | cert command registration — where in root command order? | **After `cert` is unreachable today, append after `node` in rootCmd.AddCommand order** | Alphabetical-ish with the existing cluster (audit, daemon, doctor, init, job, node, cert, status, version). No behavior change to existing commands. | 0.88 |
| D-042 | Coverage target — 50% floor or higher? | **50% floor per package, 70% target for new packages** | 50% is achievable for the concurrent packages (engine, transport) without heroic mock effort; 70% is the floor for new code in P02/P04. Avoids a "raise coverage everywhere" rathole. | 0.85 |

## v0.7 Scope Summary — Hardening & Completion

v0.7 is a 4-execution-phase **NFR milestone** that closes out gaps
surfaced by the v0.7 IDEATE stage: an unreachable command tree, a
missing config file layer, low test coverage in core packages, and the
long-deferred pprof endpoint. The engine functionality from v0.1–v0.6
is unchanged; this milestone is purely about **correctness, coverage,
and operability**:

- **P01 — Register `orca cert` command tree + cert_repo tests.** The
  `internal/cli/cert.go` command (`cert ca-init`, `cert gen`, `cert
  show`, `cert renew`, `cert fingerprint`) is fully implemented but
  never wired into `rootCmd`. This phase adds the missing
  `rootCmd.AddCommand(newCertCmd(...))` and adds the missing
  `internal/store/cert_repo_test.go`. Covers REQ-053.
- **P02 — HCL config file parsing (`config.hcl`).** D-009 specified
  `~/.orca/config.hcl` and `/etc/orca/orca.hcl` as config locations,
  but no HCL config-file parser exists — the CLI relies entirely on
  flags and env vars. This phase adds a minimal `internal/config`
  package that loads `config.hcl` (keys: `db_path`, `listen_addr`,
  `ca_path`, `server_cert_path`, `server_key_path`, `node_capacity`),
  merges with env/flag overrides (flag > env > file > default), and
  surfaces it via `--config` flag on the root command. Covers
  REQ-054.
- **P03 — Test coverage uplift.** Adds tests for the lowest-coverage
  packages: `internal/engine` (executor, dispatcher, peer — currently
  8.3%), `internal/transport` (mtls, dispatch, handshake_log —
  currently 26.3%), `internal/proxmox` (bootstrap SSH path —
  currently 5.1%), and `internal/audit` (no tests). Target: every
  package ≥ 50% coverage. Covers REQ-055.
- **P04 — `--pprof` opt-in on `orca daemon`.** Adds the long-deferred
  I-308 pprof endpoint behind an opt-in `--pprof <addr>` flag (default
  disabled). `net/http/pprof` mounted on a separate mux so it never
  touches the mTLS daemon listener. Covers REQ-056.
- **P05 — Final review + ship + audit.** Milestone release.

The vision ("minimalist, offline-first, CLI-first orchestration
engine") is unchanged. v0.7 is a hardening milestone, not a direction
change. Milestone type: NFR (all phases are fix/test/chore); the final
phase's progressive patch IS the deliverable per `run.md` versioning
logic. Tags run on the v0.6.x patch line: `v0.6.0` (P0) … `v0.6.5` (P05
= milestone release).

## v0.8 Scope Summary — Coverage & Trust Hardening

v0.8 is a 3-execution-phase **NFR milestone** that continues the
hardening theme opened by v0.7. v0.7 P03 (REQ-055) lifted four
packages to ≥ 50%, but a coverage re-baseline after v0.7 ship shows
the floor was insufficient: `internal/engine` regressed to 8.3%,
`internal/proxmox` to 5.1%, and four more packages sit between 26% and
48%. Three packages (`internal/audit`, `internal/certpaths`,
`cmd/orca`) still have **no test files at all**. v0.8 also closes the
two "future enhancement" hooks explicitly deferred in v0.6 — SSH
host-key pre-pinning (D-035 caveat) and `orca node key-reset`
(RESEARCH_v0.6 §80) — and adds a requirements-hygiene gate so the
stale-REQ-status drift seen in REQUIREMENTS.md after v0.7 ship cannot
recur:

- **P01 — Coverage uplift round 2.** Raise six under-50% packages to
  ≥ 70% and add first tests for the three zero-test packages. Covers
  REQ-057.
- **P02 — SSH trust hardening.** `--host-key-fingerprint` pre-pin flag
  on `orca node join --type proxmox` + `orca node key-reset <node>`
  command. Covers REQ-058, REQ-059.
- **P03 — Requirements-hygiene gate.** `make verify-reqs` target +
  verify-stage assertion that ROADMAP `Complete` ↔ REQUIREMENTS
  `Complete`. Covers REQ-060.
- **P04 — Final review + ship + audit.** Milestone release.

The vision is unchanged. v0.8 is a hardening milestone, not a
direction change. Milestone type: NFR (all phases are test/feat-chore
on the trust surface — see CLARIFY D-043 for the `feat` vs `chore`
classification of P02); the final phase's progressive patch IS the
deliverable per `run.md` versioning logic. Tags run on the **v0.7.x**
patch line: `v0.7.0` (P0) … `v0.7.4` (P04 = milestone release).

## v0.8 Clarified Decisions (D-series, full autonomy)

The 5 v0.8 decisions (D-043..D-047) were auto-resolved at full autonomy
within the `clarify_budget` (10):

| ID | Question | Decision | Rationale | Confidence |
|----|----------|----------|-----------|------------|
| D-043 | Is P02 (SSH trust hardening) a `feat` phase or a `chore` phase? It adds a new flag + a new subcommand. | **`chore` (trust-surface hardening), not `feat`** | Both `--host-key-fingerprint` and `orca node key-reset` refine the *existing* `orca node join --type proxmox` flow and the existing TOFU `known_hosts` store (D-035). No new orchestration capability, no new node kind, no new API. They close a security gap explicitly deferred in v0.6, not open new surface area. Per `run.md` versioning logic this keeps v0.8 NFR (all phases fix/test/chore/perf/refactor). | 0.84 |
| D-044 | Where does `--host-key-fingerprint` live — on `orca node join` or only on `--type proxmox`? | **On `orca node join` (root of the join subcommand), validated when `--type proxmox`** | The flag is generic (any future SSH-joined node kind will use it); gating it to `--type proxmox` only would require re-adding it later. Validation (`flag requires --type proxmox today`) happens in `RunE`, not in the flag declaration, so the flag is declared once on `node join` and the type check emits a clear error for non-proxmox types until other SSH-joined kinds exist. | 0.86 |
| D-045 | `--host-key-fingerprint` format — raw hex, `sha256:`-prefixed, or OpenSSH `SHA256:base64`? | **OpenSSH `SHA256:base64` (the format `ssh-keyscan -E sha256 -D -` emits and operators expect)** | Matches the fingerprint format operators already see from `ssh-keyscan` and `orca node join`'s own `Result.HostKeyFingerprint` output. Accept only `SHA256:`-prefixed base64; reject raw hex with a clear error. Internally decode base64 → compare against `ssh.PublicKey` Marshal + sha256. | 0.88 |
| D-046 | Does `orca node key-reset <node>` also revoke the orca pubkey on the remote host, or only clear the local `known_hosts` entry? | **Local `known_hosts` entry only** | Revoking the remote authorized_keys entry would orphan a working node (next dispatch would fail auth). `key-reset` is the local "forget this host's key" operation (mirrors `ssh-keygen -R host`); re-establishing trust is a separate `orca node join` re-run. Audit-log the reset with `actor`, `node`, `event=node.key_reset`. | 0.90 |
| D-047 | Coverage target for P01 — 70% floor or higher? | **70% floor for the 6 under-50% packages; 50% floor for the 3 zero-test packages (`internal/audit`, `internal/certpaths`, `cmd/orca`) as a first-toe-hold** | 70% across the board for the already-tested packages matches D-042's "70% target for new packages" and is achievable without heroic mock effort. For the zero-test packages, going 0→50% is the realistic single-phase step (0→70% risks a coverage rathole on `cmd/orca` which is glue code); a future milestone can lift them to 70%. | 0.82 |

---

# v0.9/v0.10 — Re-architecture Scope Summary (Supersedes v0.1–v0.8 architecture)

v0.9 is the first DIRECTION-CHANGE milestone in the project's history.
It supersedes the shipped v0.1–v0.8 architecture per the adopted PRD
(`.ciagent/PRD_v0.9.md`). The re-architecture deprecates the daemon/
transport/internal-CA/HCL/single-namespace stack and builds a CLI-only/
SSH-push/step-ca/Markdown-frontmatter/multi-namespace stack plus 8
net-new subsystems.

## Override Justification (Re-architecture Justification axis)

The ci-griller returned REPLAN (0.70) on the Re-architecture Justification
axis, noting the PRD reverses 6 documented decisions without new evidence
and that the incremental-additive path was not evaluated. The user reviewed
the fork and overrode the *direction* with a six-part evidence basis. The
override is recorded verbatim below; each part addresses a reversal that
the grill flagged as unjustified.

1. **The v0.8 daemon model is operationally failing** in the target
   environment — R-001 ("no orca binary on any server") is a response to
   measured pain, not preference.
2. **step-ca is externally mandated** (D-101) — the operator environment
   requires an external CA; AD-010's "too heavyweight" rationale is no
   longer operative.
3. **Multi-tenancy is a hard product requirement** (R-002) — real
   multi-tenant use cases cannot be served by the single-namespace layout;
   the "no multi-tenancy" anti-pattern is obsolete.
4. **WASM is a hard workload requirement** (D-088) — workloads are WASM, not
   processes; `os/exec` is insufficient; the "no container runtime"
   anti-pattern is reversed.
5. **SSH-push is the only viable deployment target** for the operator's
   bare-Linux/Proxmox environment — installing/maintaining an orca daemon
   on every peer is operationally infeasible.
6. **Simplicity/vision correction** — the v0.1-v0.8 daemon model was a
   wrong turn against the original CLI-first vision; the re-architecture
   corrects the vision.

## Supersession Table (AD-series reversals, recorded per grill PC-09)

| Old decision | Was | Superseded by | Evidence basis |
|---|---|---|---|
| AD-010 (ARCHITECTURE.md:463) | step-ca/cfssl/vault-pki "too heavyweight" | **D-101** (step-ca) | Override ground 2 (external mandate) |
| SPIFFE rejection (PROJECT.md:94) | internal CA chosen over SPIFFE | **D-068** (SPIFFE SVIDs) | Override ground 3 (multi-tenancy requires per-workload identity) |
| No-container-runtime (ARCHITECTURE.md:477) | explicit anti-pattern | **D-088** (5 runtimes; wasmtime primary) | Override ground 4 (WASM is the workload profile) |
| No-multi-tenancy (ARCHITECTURE.md:478) | explicit anti-pattern | **D-158 / R-002** (many namespaces under ORCA_HOME) | Override ground 3 (hard multi-tenant product req) |
| AD-007 (HCL canonical) | HCL for jobspec | **R-013 / R-014** (Markdown canonical; HCL legacy) | PRD §8 (Markdown + body preservation is the operator-facing format) |
| Daemon-on-every-node | `orca daemon` on all peers | **R-001** (no orca binary on any server) | Override grounds 1 + 5 (daemon failing; SSH-push only viable target) |

The 19 binding conditions (C-01..C-19) and 10 phase challenges
(PC-01..PC-10) from `GRILL_v0.9.md` are adopted as execution gates.
The 30 net-new requirements (REQ-061..REQ-090) from `IDEATION_v0.9.md`
are recorded in `REQUIREMENTS.md`. The reordered phase plan is in
`ROADMAP.md`.

## v0.9 Clarified Decisions (D-series, full autonomy — Phase 0 pre-execution)

| ID | Question | Decision | Rationale | Confidence |
|----|----------|----------|-----------|------------|
| D-101 | Cluster CA: internal Go CA (AD-010) or step-ca (external)? | **step-ca (apt-installed)** | Externally mandated per override ground 2; AD-010's "too heavyweight" rationale reversed. CLI wraps `step` CLI via SSH (no Go step-ca client library — keep zero-new-dep posture if possible, or add `github.com/smallstep/cli` as a dep). **Gated by C-07** (CA migration spec). | 0.74 |
| D-068 | Workload identity: internal X.509 CA or SPIFFE SVIDs? | **SPIFFE SVIDs minted at submit time via step-ca** | Multi-tenancy (override ground 3) requires per-workload identity model; SPIFFE is the standard. SPIFFE ID `spiffe://orca/ns/<ns>/job/<name>/alloc/<id>` as SAN. **Gated by C-08** (mint spike in v0.10-P01.5; fallback to mTLS identity if spike fails). | 0.72 |
| D-088 | Runtime: direct os/exec only (D-008) or multi-runtime? | **5 runtimes: wasm (wasmtime primary), podman, process, pve-vm, pve-ct** | WASM is the primary workload (override ground 4). `processRuntime` wraps existing `executor.go`; others are net-new. Split P07a/b/c per grill PC-10. **P07b gated by C-01** (wasmtime/CGO eval). | 0.82 |
| D-158 | Namespace model: single flat root or multi-namespace? | **Multi-namespace under ORCA_HOME (R-002)** | Hard multi-tenant product requirement (override ground 3). `_defaults/` implicit root; `cluster/` for cluster-wide; per-namespace `db/`, `.env`, `.env.secrets`, `jobs/`, `alloc/`, `ns.md`. No namespace column in SQLite. | 0.84 |
| D-179 | Jobspec format: HCL canonical (AD-007) or Markdown? | **Markdown with YAML frontmatter canonical (R-013); HCL legacy** | PRD §8 — Markdown + body preservation is the operator-facing format. HCL adapter (REQ-064) preserves `orca job run old-spec.hcl` during migration. | 0.85 |
| D-185 | Re-architecture justification: incremental additive or full re-architecture? | **Full re-architecture (overridden by user)** | Six-part evidence basis above; the grill's REPLAN mechanics (PC-01..PC-10, C-01..C-19) adopted as gates. The incremental-additive path was evaluated and rejected on grounds 1 + 5 (daemon failing; SSH-push only viable). | 0.88 |
| D-187 | wasmtime Go binding (bytecodealliance/wasmtime-go) is CGO-based — does adopting it revoke D-002 (modernc/sqlite CGO-free cross-compile story)? | **Use the wasmtime CLI (apt-installed on peer) via SSH exec; do NOT import wasmtime-go.** | The Go binding links libwasmtime via cgo and would revoke D-002's CGO-free cross-compile story. The CLI-via-SSH approach (same pattern as podman/qm/pct) avoids CGO entirely. `internal/runtime/wasm.go` imports only stdlib + sshpush. `CGO_ENABLED=0 go build ./...` succeeds. C-01 grill gate SATISFIED; D-002 NOT revoked. Full evaluation in `internal/runtime/C01_WASMTIME_CGO_EVAL.md`. | 0.90 |

---

# v0.10 Docs & Install Milestone — Scope Summary

v0.10 is a focused milestone that closes the documentation gap left by
the v0.9 re-architecture and fixes the release/install pipeline bug that
caused `install.sh` to resolve to v0.4.5 instead of the latest release.
The v0.9 re-architecture shipped a complete CLI surface (markdown
jobspec, `orca ns`, `orca node capacity`, CLI-side scheduler, emitters,
Traefik ingress) but no operator-facing reference documentation. This
milestone ships that documentation plus a worked full-stack example
with ingress configured, and hardens the release pipeline so every
Gitea release carries a Linux binary asset.

## Root cause of the v0.4.5 install

The v0.8.x releases (v0.8.0 through v0.8.15) shipped with **zero binary
assets attached** to their Gitea releases. `scripts/install.sh` resolves
"latest" by hitting `/releases/latest` (returns v0.8.15), then looks for
`orca-v0.8.15-linux-amd64.tar.gz` in that release's assets. Since the
asset is missing, install.sh errors out — there is no fallback walk to
older releases that DO carry a binary. The user's v0.4.5 install came
from an earlier run or a pinned `--version`. The fix is forward: harden
`scripts/release.sh` to cross-build the amd64 tarball and verify the
asset attached post-create; harden `scripts/install.sh` to walk
backward through releases if the latest lacks the asset.

## v0.10 Phases

- **Phase 0 (pre-execution)**: specify → clarify → research → ideate → plan → grill. Tag `v0.9.0`.
- **Phase P1 — release/install fix** (REQ-097, REQ-098): cross-build amd64 tarball in release.sh, post-create asset verification, install.sh fallback walk. Tag `v0.9.1`.
- **Phase P2 — CLI + jobspec + ingress docs** (REQ-091, REQ-092, REQ-093): `docs/cli.md`, `docs/jobspec.md`, `docs/ingress.md`. Tag `v0.9.2`.
- **Phase P3 — full-stack examples** (REQ-094): `examples/full-stack/` with 5 valid jobspecs + rendered artifacts + walkthrough README. Tag `v0.9.3`.
- **Phase P4 — README + namespace.md refresh** (REQ-095, REQ-096): README subcommand table + install example + docs/examples sections; `docs/namespace.md` v0.9 layout. Tag `v0.9.4`.
- **Phase P5 — final review + ship + audit** (milestone release). Tag `v0.9.5` = v0.10.0 milestone release.

**Milestone type**: feature (P1 ships `fix` phases; P2/P3/P4 ship `docs`
phases; at least one non-docs phase makes this a feature milestone per
the versioning logic). Tags run on the v0.9.x patch line. The milestone
branch label is `milestone/v0.10-docs-cli-examples`.

The vision ("minimalist, offline-first, CLI-first orchestration
engine") is unchanged. v0.10 is a documentation + install-hardening
milestone, not a direction change. It builds on the v0.9
re-architecture foundation without modifying any Go orchestration code.

## v0.10 Clarified Decisions (D-series, full autonomy — Phase 0 pre-execution)

| ID | Question | Decision | Rationale | Confidence |
|----|----------|----------|-----------|------------|
| D-188 | Should the CLI docs be a single `docs/cli.md` reference or a per-command `docs/cli/` subdirectory? | **Single `docs/cli.md` reference** | Mirrors the existing flat `docs/` pattern (install.md, docker.md, namespace.md, security-scanning.md). One file is more discoverable for a CLI tool and avoids navigation overhead. A per-command subdirectory diverges from the established layout. | 0.92 |
| D-189 | Should the examples live in `examples/full-stack/` or in `testdata/`? | **`examples/full-stack/` as a new top-level directory** | `testdata/` holds legacy HCL fixtures (`hello.hcl`, `fail.hcl`) used by Go tests; mixing operator-facing examples with test fixtures conflates audiences. A new `examples/` directory is the conventional location for worked examples and is what an operator expects to find. | 0.93 |
| D-190 | How deep should the ingress/Traefik documentation go? | **Dedicated `docs/ingress.md` plus a worked example in `examples/full-stack/`** | Ingress is the user's explicit ask ("full stack with ingress configured") and the Traefik/service-block model (R-007 socket vs TCP, atomic reload, drain, TLS) is non-trivial. A dedicated doc is the clearest answer; a section buried in `docs/cli.md` would be less discoverable. | 0.90 |
| D-191 | Should the docs frame the v0.9 canonical path or document both v0.8 and v0.9 equally? | **Document the v0.9 canonical path; flag deprecated surface with callout boxes** | The v0.8 daemon/mTLS/HCL path is deprecated and scheduled for removal in v0.10-P14. Documenting it as primary misleads new operators; documenting both equally doubles the surface and risks documenting soon-removed code. Callout boxes with "deprecated in v0.9, removed in v0.10" point operators to the canonical path. | 0.91 |
| D-192 | Should the existing v0.8.15 release be backfilled with a binary asset, or only fix the pipeline forward? | **Fix forward only; no backfill** | Backfilling a past release is an ops task, not a docs milestone deliverable. The next tagged phase (this milestone's P1 ship at v0.9.1) will be the first correctly-asseted release; install.sh's new fallback walk handles the gap until then. | 0.88 |
| D-193 | Should `release.sh` build only `linux-amd64` or also `linux-arm64`? | **Cross-build `linux-amd64` explicitly (host-arch-independent); arm64 deferred to a follow-up** | The install.sh user base is amd64 today (the `.coreci.yml` release step hardcodes `--asset orca-${VERSION}-linux-amd64.tar.gz`). Building amd64 regardless of host arch (via `GOOS=linux GOARCH=amd64 go build`) guarantees the asset the install script expects. arm64 support is a separate enhancement. | 0.85 |
| D-194 | Should `install.sh` add a `--check` dry-run mode? | **Yes, lightweight** | A dry-run mode (`--check`) that prints the version + asset URL + install path without writing is cheap to add and useful for debugging the "which release will I get?" question that the v0.4.5 incident surfaced. | 0.80 |

## v0.11 Clarified Decisions (D-series, full autonomy — Phase 0 pre-execution)

The following 23 decisions (D-215…D-237) extend the locked D-series
(ends at D-206). They derive from 5 research documents ingested
2026-08-07 covering ingress hardening, drift detection, platform-engineer
positioning, strategic framing, and the systemd Path unit implementation.
Operator decisions Q1=A, Q2=C, Q3=A, Q4=A, Q5=A are adopted.

### Ingress hybrid (D-215…D-226, from research doc 1)

| ID | Question | Decision | Rationale | Confidence |
|----|----------|----------|-----------|------------|
| D-215 | Public-binding default? | **Hybrid: nft DNAT → Traefik on `127.0.0.1:8443`** | Defense-in-depth (kernel + app layer); mature pattern (kube-proxy, Linkerd2-proxy, F5/HAProxy+nginx). Smaller Traefik attack surface. R-017. | 0.93 |
| D-216 | Opt-out? | **`orca cluster config --public-binding=traefik-on-public-ip` for the simple case** | Operators who want simplicity get it with a one-line config change. | 0.94 |
| D-217 | nftables emitter? | **Yes; renders `/etc/nftables.d/orca.nft`; idempotent `nft -f` apply** | Same emitter pattern as Traefik/systemd emitters (R-001-clean). | 0.93 |
| D-218 | nftables tool vs iptables? | **`nft` (modern) over legacy `iptables`** | Atomic rule-set swap; modern kernel API. | 0.96 |
| D-219 | Cross-node cluster mesh? | **Stays bound on private IP `192.168.x.x:8443`; unchanged** | Avoids adding iptables rules for cross-node mesh; keeps mesh logic unchanged. | 0.94 |
| D-220 | Traefik `address` in static config? | **`127.0.0.1:8443` in default, `:443` in opt-out** | Single line change; certs/mTLS/dynamic config unchanged. | 0.97 |
| D-221 | `orca doctor nft`? | **Yes; checks table, expected rules, file hash; drift detection via hash comparison** | Parity with `orca doctor traefik`; integrates with R-018 critical_paths. | 0.95 |
| D-222 | Rate-limit meter? | **`ora_rl` set as part of the default rule set; configurable via `orca nft rate limit set`** | Kernel-level line-rate rate limiting; defense against SYN floods. | 0.91 |
| D-223 | GeoIP blocking? | **Operator-opt-in via `orca nft country block add`**; cli + ipset extension | Not a default; operators opt in. | 0.88 |
| D-224 | `nftables` not `iptables` in `.coreci.yml` pipelines? | **Yes; integration tests use `nft` exclusively** | Matches D-218. | 0.94 |
| D-225 | Per-workload `ingress: native` coexists with hybrid default? | **Yes; `service { ingress: native }` opts into pure iptables + stunnel sidecars** | Workload-level opt-in; doesn't affect cluster default. | 0.93 |
| D-226 | `nftables` rule hash baseline? | **`cluster/state/baseline.nft.hash` per peer; drift detection per §17** | Integrates with R-018 drift detection. | 0.90 |

### Drift detection (D-227…D-237, from research doc 5)

| ID | Question | Decision | Rationale | Confidence |
|----|----------|----------|-----------|------------|
| D-227 | Drift detection architecture? | **systemd Path units for critical paths + 60s polling backstop + auto-remediation** | R-001-clean (systemd is OS, not Orca); ~10s event-driven latency on critical paths. R-018/R-019. | 0.94 |
| D-228 | Path unit event payload? | **Oneshot service; receives path via `%f`; computes sha256; writes event JSON to `/etc/orca/state/drift-events/`** | Stateless, self-contained, idempotent. | 0.93 |
| D-229 | Lead-side pickup? | **Aggregator timer reads each peer's drift-events/, validates against applied txn hashes, triggers remediation** | Reuses existing 10s aggregator cadence (C-11); single SSH pull per tick. | 0.94 |
| D-230 | Critical path polling cadence? | **5s backstop; systemd Path unit provides ~10s event-driven latency** | Closes the gap to K8s-comparable drift detection on critical paths. | 0.92 |
| D-231 | Auto-remediation policy? | **Per-path config; critical paths default to auto; systemd units default to require-approval** | Config files are safe to re-push; service units may need careful ordering (don't restart serving workloads). | 0.93 |
| D-232 | Remediation rate limit? | **5-minute cooldown per path; applies only on SUCCESSFUL remediation; transient failures retry on next aggregator tick** | Prevents loops from buggy external actors; avoids a 30s network blip blocking re-remediation for 5 min (refined per CLARIFY C4). | 0.92 |
| D-233 | NFS path detection? | **`orca node setup` detects NFS mounts; falls back to polling for affected paths** | systemd Path units use inotify which doesn't work across NFS. | 0.90 |
| D-234 | Secrets path exclusion? | **`/etc/orca/credentials/*` excluded from drift detection** | Re-remediating secrets might clobber intentional out-of-band rotation. | 0.95 |
| D-235 | EnvironmentFile drift? | **Triggers `orca job restart <name>` instead of file-level remediation** | Workload already running won't pick up env changes without a restart. | 0.89 |
| D-236 | `orca drift watch` semantics? | **`iter.Seq2[Event, error]` per D-017; `signal.NotifyContext` per D-023; default 2s poll** | Consistent with existing `--watch` pattern (D-017/D-023). | 0.95 |
| D-237 | Aggregator timer changes? | **Existing 10s cadence; extended to also pull drift-events/ and remediate** | Reuses C-11 aggregator; no new timer. | 0.95 |

### P01.5 — SPIFFE SVID minting spike (gate C-08, D-068)

**C-08 SPIFFE mint spike: PASS.** The `step` CLI (smallstep step-ca)
accepts a `spiffe://` URI in `--san` and emits a cert whose URI SAN
(x509 subjectAltName URI entry) carries the SPIFFE URI. The fallback to
mTLS identity (per D-068 / C-08) is NOT needed; D-068 stands.

- **SPIFFE URI format (locked):**
  `spiffe://orca.local/ns/<namespace>/sa/<service-account>/<alloc-id>`
  — trust domain `orca.local`; `ns/<ns>` scopes the workload to an
  Orca namespace (R-002); `sa/<sa>` is the service-account; `<alloc-id>`
  makes the SVID unique per allocation.
- **step CLI command (locked):**
  `step ca certificate <spiffe-id> <cert> <key> --san <spiffe-id> --not-after 24h --provisioner orca-admin --password-file /dev/stdin --force`
- **Cert parsing (locked):** `pem.Decode` → `x509.ParseCertificate` →
  iterate `cert.URIs` and match the expected SPIFFE URI (parsed as
  `*url.URL`, compared by canonical string). Missing URI SAN →
  `ErrSpiffeURIMissing` (cert rejected before reaching the workload).
- **Implementation:** `internal/identity/spiffe.go` — `SpiffeURI`,
  `MintSVID`, `VerifySVID`, `SpiffeIDFromCert`, `SubjectFromSpiffe`.
- **Tests:** `internal/identity/spiffe_test.go` — mock transport
  (`execer`) returns a self-signed cert minted in-process via
  `crypto/x509.CreateCertificate` with `URIs: []*url.URL{spiffeURI}`,
  exercising the exact production parsing path. 15 tests, all pass.
- **Spike result record:** `internal/identity/SPIFFE_SPIKE_RESULT.md`.

## v0.12 Scope Summary — Security Hardening (Zero-Trust Identity)

v0.12 is a 27-execution-phase feature milestone dedicated to
comprehensive security hardening across the entire attack surface,
**including the operating system itself**. The threat-model review
(v0.11 closeout + Phase 0 RESEARCH) surfaced 25 distinct findings
(F1..F25) spanning injection, traversal, ACL, audit, crypto, OS
scripts, emitters, sudoers, system users, file modes, daemon auth,
backup, SQLite, install.sh, and migration. v0.12 closes all of them
and adopts a **zero-trust identity model** as the load-bearing
architectural change.

### Load-bearing rule adopted in Phase 0

**R-021**: *Orca never issues, stores, or accepts human-identity
credentials. Human identity is exclusively external (OIDC). Machine
identity is exclusively mTLS/SPIFFE. No passwords, no Orca-issued
tokens, no CA-key passphrases.*

### Zero-trust identity model

Two identity layers, zero overlap:

- **Human operators** → OIDC (external IdP, BYO) OR the **bundled Dex**
  with a **WebAuthn (passkeys) connector** as the default
  password-free authenticator. `orca auth login` / `orca auth register`
  open the default browser to the Dex WebAuthn endpoint via OIDC
  authorization-code + PKCE + local loopback redirect. After the
  WebAuthn ceremony (biometric/security key), Dex redirects back with
  an auth code; CLI exchanges for a short-lived ID token (1h) +
  refresh. Headless/CI fallback: device-code flow.
- **Machine-to-machine** → mTLS + SPIFFE SVIDs (unchanged from v0.11).

The "no Orca credentials" invariant holds: passkeys are public-key
credentials (the private key never leaves the authenticator); the
WebAuthn credential DB stores only public keys + credential IDs +
sign counts. No passwords, no Orca-issued tokens, no CA-key
passphrases anywhere in the system.

### Master key sealing

The secrets master key (32 random bytes) is **sealed to OIDC** —
wrapped by a key derived from an OIDC token exchange at unseal time.
`orca cluster unseal` (operator authenticates via OIDC → token
exchange → unwrap master key into memory → zeroed on shutdown). The
raw master key never touches disk. **Shamir 3-of-5 recovery**: at seal
time, 5 shards are printed and the operator stores them offline. If
the IdP is permanently lost AND a quorum of shards is unavailable, the
cluster is unrecoverable by design (documented residual risk; no
backdoor).

### New requirements (REQ-119..REQ-148)

30 net-new requirements derived from the threat-model findings and the
zero-trust identity model. See REQUIREMENTS.md and ROADMAP.md for the
full mapping. Highlights:

- REQ-119..121: command injection, path traversal, txn path allowlist
- REQ-144: OIDC client + bundled Dex (BYO-IdP override)
- REQ-145: ACL rewrite (remove KindToken, add KindOidc, enforce)
- REQ-146: remove all password/token paths (breaking)
- REQ-147: master key seal-to-OIDC + Shamir recovery
- REQ-148: WebAuthn connector for Dex (passkeys, browser auth+register)
- REQ-122..143: integrity, crypto, OS scripts, emitters, sudoers,
  system users, SQLite, migration, dual-write closure, transport,
  drift auth, integration tests, docs, final review

### v0.12 Clarified Decisions (D-series, full autonomy)

The 10 v0.12 decisions (D-238..D-247) were resolved during CLARIFY
under full autonomy (autonomy.level=full, workflow.no_hitl=true):

| ID | Question | Decision | Rationale | Confidence |
|----|----------|----------|-----------|------------|
| D-238 | Milestone version? | **v0.12 (minor, not v1.0)** | v1.0.0 stays deferred for post-UAT per v0.11 PRD; v0.12 is a minor feature milestone. Tags on v0.11.x patch line. | 0.95 |
| D-239 | OIDC provider model? | **Bundled Dex by default + BYO external IdP override** | Zero-trust out of the box without external setup; `oidc.issuer` repoint switches to BYO. | 0.90 |
| D-240 | Bundled Dex upstream authenticator (password-free)? | **WebAuthn (passkeys) connector** | Public-key credentials; private key never leaves authenticator; reinforces "no passwords" invariant (R-021). | 0.88 |
| D-241 | Master key sealing model? | **Seal to OIDC + Shamir 3-of-5 recovery** | No password anywhere; quorum recovery if IdP lost; no backdoor. | 0.85 |
| D-242 | CLI browser flow? | **OIDC auth-code + PKCE + local loopback redirect** | Standard OIDC browser flow; secure for public clients; headless fallback via device-code. | 0.92 |
| D-243 | WebAuthn RP ID / secure context? | **Traefik-served cluster domain (step-ca cert, R-017)** | WebAuthn requires HTTPS; Traefik already provides it; RP ID configurable via `orca auth init-idp`. | 0.90 |
| D-244 | Passkey storage? | **SQLite at ClusterDir()/webauthn-credentials.db (0600); public keys only** | Public keys are not secrets; 0600 file mode for integrity; no passphrase wrapping needed. | 0.92 |
| D-245 | Headless/CI auth fallback? | **Device-code flow** | No browser in CI; device-code is the standard OIDC headless path. | 0.90 |
| D-246 | Token storage at rest? | **~/.orca/credentials.json (0600); short-lived (1h) + refresh** | Standard OIDC token storage; 0600; refresh handles rotation; no long-lived Orca-issued tokens. | 0.92 |
| D-247 | Breaking-change handling for password/token removal? | **`orca upgrade` refuses v0.11 clusters using --password/bare-tokens without --accept-identity-migration** | No silent breakage; explicit migration gate; documented cutover. | 0.90 |

### v0.12 is a HARDENING + IDENTITY milestone, not a direction change

The vision ("minimalist, offline-first, CLI-first orchestration
engine inspired by HashiCorp Nomad") is unchanged. v0.12 closes the
security-surface gaps surfaced by the v0.11 threat model and adopts a
zero-trust identity model. The offline-first principle (R-003) is
preserved: the bundled Dex can run on the lead (offline), and the
mTLS-only path remains for the single-operator fully-offline case (no
human authn needed — the operator holds the pre-staged SSH key + mTLS
cert; no password, no token).

### v0.13: Production Hardening Round 2 + UAT Plan (IN PROGRESS)

v0.12 (Security Hardening) is COMPLETE. v0.13 is the **final hardening round before UAT validation**. The UAT will likely surface 3-7 issues requiring a patch release. v1.0.0 is deferred until UAT passes.

v0.13 is the **final hardening
round** before the v1.0.0 production-ready tag. Three deep codebase
sweeps (security, reliability, feature/doc claims) surfaced ~60 gaps
beyond v0.12. The most critical:

1. **`orca job run` runs locally** via `exec.CommandContext` — the
   scheduler/emitter/SSH-push pipeline is dead code. The documented
   deployment model (deploy to Proxmox/Ubuntu worker) is non-functional.
   **R-022** fixes this.
2. **jobspec `schedule:`/`timeout:` silently dropped** by the markdown
   parser — DaemonSet is fundamentally broken (parser defaults Count=1,
   validator rejects Count!=0, schedule never parsed).
3. **`acl.Check` called zero times** — v0.12's headline zero-trust
   feature is library-complete but not wired into any request path.
   **R-023** fixes this.
4. **Command injection vectors** — `orca logs --job` backtick RCE via
   `%q` (bash executes command substitution in double quotes), tar-slip
   in backup restore, sudoers injection via `--proxmox-user`/`--role`,
   `txn rollback` shell injection, and 7 more.
5. **Go toolchain 1.25.0** — 24 stdlib vulns with call traces in orca
   (archive/tar, crypto/tls, crypto/x509, net/http, encoding/pem...).
6. **Concurrency hazards** — audit hash-chain race (concurrent appends
   corrupt tamper-evidence), concurrent `secrets set` silently loses
   data (no flock), no SQLite `busy_timeout` (database is locked),
   concurrent `orca upgrade` races on Traefik cutover.
7. **Cache never invalidated by writes** — stale reads for 10–60s
   after `node join`/`ns create`/`job run`.
8. **Massive doc drift** — README "mTLS by default" is false (SSH-push
   is canonical), `docs/cli.md` missing ~25 subcommands, CHANGELOG
   stale at v0.1, `verify-reqs` gate bypassed for v0.12.

v0.13 closes all critical/high/medium findings (15 new requirements,
14 phases) and delivers the **UAT plan + signoff script** that gates
the v1.0.0 cut.

### v0.14 Milestone: Ingress Bootstrap Completeness

**Scope**: ensure that linux & proxmox types are properly bootstrapped
with traefik during cluster init or node join. All cluster endpoints are
provisioned as sockets (R-007); routing between jobs and services
depends on traefik being present and properly configured. v0.13 shipped
traefik binary + systemd unit + empty dynamic dir but never wrote the
static config nor applied nft rules — `orca-traefik.service` fails on a
fresh `orca init` and `orca doctor nft` FAILs. v0.14 replaces the
binary+systemd model with a **podman container** running a custom
`orca-traefik` image, and completes the nft SNAT+DNAT ingress stack on
every node type.

**New load-bearing rule**: **R-024** — Traefik runs exclusively as a
podman container, deployed from the `orca-traefik` image published per
release. Every orca-managed ingress surface bootstraps: nft DNAT
(`:443→127.0.0.1:8443`, `:80→127.0.0.1:8080`) + SNAT/MASQUERADE
postrouting + `podman run -d --restart=always --network host -v
/etc/traefik/dynamic:/etc/traefik/dynamic:Z -v
/etc/orca/step-ca-root.crt:/etc/orca/step-ca-root.crt:ro
git.cloudinit.dev/coreci/orca-traefik:<tag>`. No node joins without a
functional podman-traefik ingress data plane.

**Three topologies** (per operator constraints):
1. **Linux**: host → nft → `podman run orca-traefik` (host network)
2. **Proxmox Native**: host → nft → LXC (nesting=1) → `podman run orca-traefik`
3. **Proxmox Floating-IP**: LXC (owns floating IP) → nft (inside LXC) → `podman run orca-traefik`

**Milestone type**: feature (multiple `feat` phases). Tags on v0.13.x
patch line: `v0.13.0` (P0) ... `v0.13.8` (P8 final = v0.14 milestone
release). 9 phases, 9 net-new requirements (REQ-171..REQ-179).

### v0.14 Decisions (D-series, full autonomy)

| ID | Question | Decision | Rationale | Confidence |
|----|----------|----------|-----------|------------|
| D-255 | Traefik deployment model? | **Podman container from custom `orca-traefik` image** | Operator constraint: traefik always deployed as a container. Replaces v0.13 binary+systemd. Image bakes static config. | 0.95 |
| D-256 | Container network mode? | **`--network host`** | Binds 127.0.0.1:8080/8443 directly on host/LXC loopback; nft DNAT targets that. No port publishing complexity. | 0.92 |
| D-257 | TLS cert resolver in image? | **No certResolver; `tls: {}` for v0.14, real mTLS deferred to v0.15** | Grill G-003 (confidence 0.55 < 0.60) auto-resolved to defer. Traefik v3.3 `certificatesResolvers` only supports acme/tailscale, not CA-file. Drop `certResolver: orca` (broken). Emit `tls: {}` in dynamic config. Real mTLS via dynamic `tls.certificates` + `tls.options.default.clientAuth.caFiles` lands in v0.15 when step-ca mints server certs. | 0.90 |
| D-258 | Dynamic config volume? | **Mount `/etc/traefik/dynamic` from host** | Zero changes to existing `deployRemote` WriteFile path (`job_dispatch.go:243`). File provider watches it. | 0.95 |
| D-259 | Floating-IP mode: register PVE host too? | **Yes — PVE host as `proxmox` + ingress LXC as `linux`** | PVE host needed in registry for `pct`/`qm` workload dispatch. Both register. | 0.92 |
| D-260 | `--ingress-mode` persistence? | **Store `IngressMode` on `model.Node`** | `doctor ingress` needs to know which check path to run. Schema migration. | 0.90 |
| D-261 | MAC generation when `--mac` omitted? | **Generate random `02:XX:...` in interactive mode; require `--mac` in `--json` mode** | Interactive: generate + confirm. Non-interactive: explicit required (no silent generation). | 0.88 |
| D-262 | Proxmox native nft DNAT target? | **LXC bridge IP (not 127.0.0.1)** | LXC has its own network namespace; 127.0.0.1 on PVE host ≠ LXC loopback. `NftClusterConfig.DNATTarget` field (default `127.0.0.1:8443`; native mode = `<lxc-ip>:8443`). | 0.90 |
| D-263 | LXC podman requirements? | **`--features nesting=1,keyctl=1` + `apt-get install podman`** | Ubuntu 24.04 LXC template has no podman preinstalled. Nesting+keyctl required for podman in unprivileged LXC. | 0.88 |

### v0.13 Decisions (D-series, full autonomy)

| ID | Question | Decision | Rationale | Confidence |
|----|----------|----------|-----------|------------|
| D-248 | Milestone version? | **v0.13 (minor, not v1.0)** | v1.0.0 stays deferred for UAT signoff; v0.13 is a minor feature milestone. Tags on v0.12.x patch line. | 0.95 |
| D-249 | UAT validation mechanism? | **Operator-driven `docs/uat.md` + `scripts/uat-signoff.sh` assertions** | Operator builds real cluster (3 hosts), runs signoff script, pastes output. Exit 0 iff all ~35 assertions pass. | 0.92 |
| D-250 | Hardening phase scope? | **All 8 themes, 14 phases** | "No limit on phases" per operator; comprehensive to avoid a round 3. | 0.90 |
| D-251 | Ubuntu worker onboarding? | **Implement `--type linux` SSH-join** | `NodeKindLinux` is reserved but unimplemented; UAT plan needs first-class worker onboarding. Proxmox stays `--type proxmox`. | 0.88 |
| D-252 | `job stop` semantics? | **Real `systemctl stop` via SSH** | Honest semantics matching `job restart` pattern; UAT assumes stop actually stops. | 0.90 |
| D-253 | UAT cluster topology? | **3 hosts: lead Ubuntu + pve01 Proxmox + worker01 Ubuntu** | Minimal topology covering both node types + migrate-between-hosts. | 0.92 |
| D-254 | UAT signoff script re-runnable? | **Idempotent — read + non-mutating assertions only** | Operator can iterate; no destructive ops. | 0.95 |

### v0.13 is the LAST hardening round

Three deep sweeps (security, reliability, feature/doc) were performed
to ensure no gap is missed. 9 low-severity residual risks are
documented and accepted (OIDC tokens plaintext at rest, HSTS on
daemon, DNS timeout, temp file cleanup on SIGKILL, flock timeout on
NFS, WASM-first aspirational, arm64 release, OIDC callback slowloris,
pprof-allow-public flag). v0.13 closes everything else. The v1.0.0
tag is cut only after the UAT signoff script passes.
