orca/.ciagent/PROJECT.md

# Project: Orca

## What This Is

A minimalist, offline-first, CLI-first orchestration engine inspired by HashiCorp Nomad, prioritizing stability, security, and simplicity over feature richness. Single-binary distribution, no container runtime, no cloud dependencies, no K8s-level complexity.

## Vision

A minimalist, offline-first, CLI-first orchestration engine inspired by HashiCorp Nomad, prioritizing stability, security, and simplicity over feature richness.

## Objective
Build a lightweight system to manage and execute workloads across a set of nodes, keeping complexity far below that of Kubernetes.

## Requirements
- **CLI First**: Primary interaction through a CLI tool.
- **Offline First**: Functional without constant internet connectivity.
- **AI First**: Designed to be easily discoverable and manageable by AI agents.
- **Stability & Security**: Prioritize security fixes and bug fixes over new features.
- **Language**: Written in Go 1.25+.
- **Simplicity**: Minimalist implementation, avoiding the "K8s complexity trap".

## Constraints
- No web UI as a primary requirement.
- Must not implement K8s-level complexity.
- Feature development must move slowly to ensure stability.
- Only CI system allowed: CoreCI (git.cloudinit.dev/coreci/coreci).
- Gitea remote: git.cloudinit.dev/coreci/orca.

## Clarified Decisions (D-series, full autonomy)

| ID | Question | Decision | Rationale | Confidence |
|----|----------|----------|-----------|------------|
| D-001 | Single binary or multi-binary distribution? | **Single binary** | Simpler distribution; subcommands baked into one `orca` binary. Aligns with simplicity pillar. | 0.95 |
| D-002 | Local state store technology? | **modernc/sqlite (pure Go, CGO-free)** | Cross-compile friendly, no CGO dependency, single file on disk, mature. | 0.92 |
| D-003 | Inter-node communication? | **Embedded HTTP (net/http) over loopback, mTLS for cross-node** | No external RPC framework needed for v0.1. HTTP suffices. | 0.85 |
| D-004 | Scheduling algorithm for v0.1? | **Single-node only (no scheduling)** | Multi-node scheduling is out of scope for v0.1. Tasks run on the node they're submitted to. | 0.90 |
| D-005 | CLI output format? | **Human-readable by default, `--json` flag for machine consumption** | Serves both humans and AI agents. | 0.95 |
| D-006 | Job/task definition format? | **HCL or YAML in `.hcl`/`.yaml` files** | Familiar to Nomad/HashiCorp users; simpler than JSON for humans. | 0.88 |
| D-186 | Bash scripts coverage gate: count toward Go gate or exempt? | **Exempt from Go coverage gate; compensating control: bats tests (C-15) + shellcheck + shfmt in CI; every script must have >=1 happy-path and >=1 failure-path bats test** | Bash is a different language surface from Go; the 70%/50% Go coverage gate (D-042/D-047) is Go-specific. Forcing bash into the Go gate would require a coverage tool that does not exist for bash. The compensating control (bats + shellcheck + shfmt) provides equivalent discipline. | 0.82 |
| D-007 | Authentication? | **mTLS for v0.1, token-based deferred** | mTLS is the most secure default. Tokens can be added later if needed. | 0.80 |
| D-008 | Container runtime? | **Direct process execution (no container runtime) for v0.1** | Avoids the Docker/container dependency. Pure process management. | 0.85 |
| D-009 | Configuration file location? | **`~/.orca/config.hcl` and `/etc/orca/orca.hcl`** | Standard XDG-style paths. | 0.90 |
| D-010 | Logging format? | **Structured JSON via `log/slog`** | Native Go 1.21+ slog, no external dependency. | 0.95 |
| D-011 | v0.2 mTLS cert authority model? | **Internal CA with CSR join** | One node bootstraps a local CA; peers generate CSRs and submit them to the CA for signing. CA cert is the trust anchor. More secure than self-signed per-node (single trust root) without the operational complexity of an external PKI. | 0.92 |
| D-012 | v0.2 CA bootstrap & cert distribution? | **Operator-mediated, fingerprint-verified** | Bootstrap node writes `~/.orca/ca.crt` and `~/.orca/ca.key` (mode 0600). Operator copies `ca.crt` to peers; peers verify by SHA-256 fingerprint at `orca node join --ca-fingerprint <sha256>`. No automated secret distribution. | 0.85 |
| D-013 | v0.2 cert validity & rotation? | **Server certs 90 days, CA cert 10 years, rotate 30 days before expiry** | Server certs are short-lived (compromise window small); CA is long-lived (manual rotation is expensive). `orca cert renew` reissues server certs automatically. | 0.90 |
| D-014 | v0.2 mTLS handshake timing? | **Eager — at `orca node join` time** | Fail fast on bad certs, misconfigurations, or CA mismatches. Lazy handshake would let stale configs run until first request, complicating debugging. | 0.88 |
| D-015 | v0.2 minimum TLS version & cipher suites? | **TLS 1.3 only; AEAD cipher allowlist** | MinVersion=tls.VersionTLS13, CipherSuites limited to TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_AES_128_GCM_SHA256. No TLS 1.2 fallback. | 0.92 |
| D-016 | v0.2 security-scanning placement? | **`validate` pipeline of `.coreci.yml`, gates merges to main** | `gosec` baseline JSON checked into repo; new findings fail the build. `govulncheck ./...` exit-on-known. Pre-commit hook with `gitleaks` is opt-in (developer machine). | 0.85 |
| D-017 | v0.2 `iter.Seq` API surface? | **`orca job list --watch` and `orca node list --watch`** | Pull-based `iter.Seq[Job]` / `iter.Seq[Node]`; cancellation via `context.Context`; `signal.NotifyContext` on ctrl-c. Backpressure is implicit (consumer-driven). | 0.90 |
| D-018 | v0.2 multi-node scheduling algorithm? | **Bin-packing by available CPU/memory, FIFO within a node** | Simple, deterministic, matches D-004 minimalism. Cross-node dispatch via ConnectRPC `orca.v1.Dispatch` service. Retry on transient failures with exponential backoff. | 0.85 |

## Out of Scope
- Full-blown Kubernetes-compatible API.
- Complex cloud-provider integrations.
- GUI-based management consoles.
- Container runtime integration.
- Service mesh / sidecar injection.
- Auto-scaling / horizontal pod autoscaler.
- External PKI / Let's Encrypt / cert transparency logs.
- gRPC framework dependency (ConnectRPC in `config.json` frameworks but
  not in `go.mod`; v0.2 uses stdlib `net/http` with h2c for
  `orca.v1.Dispatch` — see ARCHITECTURE.md AD-014).

## v0.2 Scope Summary

v0.2 is a focused 4-phase milestone that turns Orca from a single-node
process executor into a small cluster engine with strong transport
security and richer I/O. The 4 phases are:

- **P01 — mTLS handshake + internal CA with CSR join.** Internal CA, CSR
  join, eager handshake at `node join`, TLS 1.3 + AEAD allowlist.
  See ARCHITECTURE.md Flow 1 + Flow 2.
- **P02 — Multi-node scheduling & job dispatch.** Best-fit bin-packing by
  CPU/memory, FIFO within a node, `orca.v1.Dispatch` over mTLS. See
  ARCHITECTURE.md Flow 3.
- **P03 — `gosec` + `govulncheck` + `gitleaks` in CI.** `gosec` baseline
  JSON in repo, `govulncheck ./...` in `validate` pipeline, `gitleaks`
  in pre-commit (opt-in).
- **P04 — `iter.Seq` streaming for `--watch` flags.** Go 1.25+ range-over-func
  semantics, `context.Context` cancellation, `signal.NotifyContext` on
  ctrl-c. See ARCHITECTURE.md Flow 4.

The vision ("minimalist, offline-first, CLI-first orchestration engine")
is unchanged. v0.2 is a hardening + small-cluster extension, not a
direction change.

## Key Decisions

The 18 D-series decisions (D-001..D-018) are recorded in the "Clarified
Decisions" table above. The 10 v0.1 decisions (D-001..D-010) are stable
and unchanged in v0.2. The 8 v0.2 decisions (D-011..D-018) were
auto-resolved under full autonomy and are summarized here:

- **D-011: Internal CA with CSR join** (vs. self-signed per-node or SPIFFE).
  Single trust root, no external PKI, CSR workflow.
- **D-012: Operator-mediated CA cert distribution with fingerprint verify**
  (no automated secret distribution — matches offline-first principle).
- **D-013: 90d server certs, 10y CA cert, 30d pre-expiry rotation.**
- **D-014: Eager mTLS handshake at `orca node join` time** (fail fast).
- **D-015: TLS 1.3 only, AEAD cipher allowlist** (no TLS 1.2 fallback).
- **D-016: `gosec`+`govulncheck` in `validate` pipeline of `.coreci.yml`**
  (gates merges to main). `gitleaks` in pre-commit (opt-in).
- **D-017: `iter.Seq` for `orca job list --watch` and `orca node list --watch`**
  (pull-based, ctx cancellation, ctrl-c via `signal.NotifyContext`).
- **D-018: Bin-packing by CPU/memory with FIFO within node; JSON-over-HTTP
  orca.v1.Dispatch for cross-node** (no ConnectRPC dep).

## v0.3 Clarified Decisions (D-series, full autonomy)

v0.3 is a lean 2-execution-phase milestone completing the streaming and
doctor work deferred from v0.2. The 6 v0.3 decisions (D-019..D-024)
were auto-resolved under full autonomy:

| ID | Question | Decision | Rationale | Confidence |
|----|----------|----------|-----------|------------|
| D-019 | Watch refresh mechanism? | **Poll-based, 1s ticker** | Simpler than event channel; no daemon coupling for CLI; matches offline-first. | 0.90 |
| D-020 | Watch output format (REQ-030)? | **Table by default; `--watch --json` streams one-line JSON per event** | Consistent with D-005 `--json` convention; serves humans + AI agents. | 0.92 |
| D-021 | Doctor network check scope? | **Probe configured peer addresses via mTLS `/healthz` handshake; PASS/WARN/FAIL per peer** | Reuses existing transport client; read-only. | 0.85 |
| D-022 | Doctor db check scope? | **`PRAGMA integrity_check` + migration version query** | Already specced in ARCHITECTURE.md §5; minimal surface. | 0.95 |
| D-023 | iter.Seq cancellation? | **`signal.NotifyContext` on SIGINT/SIGTERM** | Per D-017 + ARCHITECTURE Flow 4. | 0.95 |
| D-024 | `--watch` applies to job list only, or node list too? | **Both `orca job list --watch` and `orca node list --watch`** | Per ARCHITECTURE.md CLI layer + D-017. | 0.92 |

## v0.3 Scope Summary

v0.3 is a focused 2-execution-phase milestone completing the work
deferred from v0.2 that was NOT already shipped in P08-P10. A codebase
audit during re-init SPECIFY confirmed that REQ-014, REQ-027, REQ-028,
REQ-029, REQ-031, REQ-037, REQ-039, REQ-040 all shipped in P08-P10
despite stale REQUIREMENTS.md marking them Pending. The remaining work:

- **P01 — `iter.Seq` streaming for `--watch` flags.** Go 1.25+
  range-over-func semantics, pull-based `iter.Seq[Job]` /
  `iter.Seq[Node]`, `context.Context` cancellation,
  `signal.NotifyContext` on ctrl-c. Applies to both `orca job list
  --watch` and `orca node list --watch`. Covers REQ-022, REQ-030.
- **P02 — `orca doctor` network + db full implementation.** Replaces
  the P01 stubs (`NetworkStub`, `DBStub`) with real checks: peer
  reachability via mTLS `/healthz` probe; SQLite `PRAGMA
  integrity_check` + migration version. Covers REQ-032 (completion).

The vision ("minimalist, offline-first, CLI-first orchestration
engine") is unchanged. v0.3 is a completion milestone, not a direction
change.

## v0.5 Scope Summary — Distribution

v0.5 is a 3-execution-phase milestone that makes Orca installable,
distributable, and containerized. The engine functionality from
v0.1–v0.3 is unchanged; this milestone is purely about **delivery
surface**:

- **P01 — Namespace unification.** A single `ORCA_HOME` environment
  variable becomes the namespace root for *all* on-disk state (db,
  certs, init, daemon). A `--system` flag on the root command selects
  the system-level namespace root `/root/.orca`. Backward compatible:
  empty `ORCA_HOME` → `~/.orca`. Covers REQ-041, REQ-042.
- **P02 — `install.sh` + in-place update.** A 1-liner installer pulls
  the release binary from the public Gitea release URL, installs at
  user level by default (`~/.local/bin/orca`) or system level
  (`/usr/local/bin/orca`) with `--system`. Re-running updates the
  binary in place while preserving config/db/certs in the namespace
  dir. Idempotent. Covers REQ-043, REQ-044. Also updates README
  quickstart (REQ-016 completion).
- **P03 — Docker release.** A multi-stage `Dockerfile` builds a
  distroless image; `scripts/release.sh` and `.coreci.yml` publish the
  image to the Gitea container registry per release. Covers REQ-046.
- **P04 — Final review + ship + audit.** Milestone release.

The vision ("minimalist, offline-first, CLI-first orchestration
engine") is unchanged. v0.5 is a distribution milestone, not a
direction change.

## v0.5 Clarified Decisions (D-series, full autonomy)

The 5 v0.5 decisions (D-025..D-029) were auto-resolved under full
autonomy during the CLARIFY stage:

| ID | Question | Decision | Rationale | Confidence |
|----|----------|----------|-----------|------------|
| D-025 | System-level namespace path layout? | **`/root/.orca`** (mirror of user-level `~/.orca`) | Consistent shape with user-level; just a different root. Matches the user's "starts at /root" wording. Single dir keeps it simple. | 0.90 |
| D-026 | Namespace override mechanism at runtime? | **Unify on `ORCA_HOME`** as single namespace root for all components (db, certs, init, daemon). Add `--system` flag that sets root to `/root/.orca`. | `ORCA_HOME` already exists for certs; extend to all components. Backward compatible (empty → `~/.orca`). One knob, not many. | 0.92 |
| D-027 | Docker registry target? | **Gitea built-in container registry** (`git.cloudinit.dev/coreci/orca`) | Keeps everything in one forge; uses Gitea's native registry. Consistent with REQ-045 (public repo → public image pulls). | 0.88 |
| D-028 | How to make releases publicly accessible (REQ-045)? | **Flip repo visibility to public** via `tea repos edit coreci/orca --private=false` during P0 ship | Simplest path to anonymous downloads; enables both install.sh pulls and docker pulls. Pre-existing `.env` leak already suppressed via gitleaks baseline + rotate-forward (commit 00127ce). | 0.85 |
| D-029 | install.sh default version? | **Latest release** (query Gitea releases API), optional `--version vX.Y.Z` to pin | Matches typical 1-liner installer UX; users get newest by default, can pin for reproducibility. | 0.92 |

### v0.5 Operational prerequisite (P0 ship)

The Gitea repo `coreci/orca` is currently **private** (returns 404
unauthenticated). P0 ship flips visibility to public via `tea repos
edit coreci/orca --private=false` so that `install.sh` can pull
release binaries unauthenticated (REQ-045). This is an operational
step performed during the P0 ship, verified by an unauth `curl`
against the releases API.

## v0.6 Scope Summary — Node Bootstrap & Proxmox

v0.6 is a 3-execution-phase milestone that turns `orca init` from a
bare `mkdir` into a full single-node cluster bootstrap, and adds
Proxmox 8 & 9 as a first-class remote node type joined over SSH with
least-privilege role delegation. The engine functionality from
v0.1–v0.5 is unchanged; this milestone is about **bootstrap
ergonomics** and **heterogeneous node support**:

- **P01 — `orca init` full bootstrap.** A single `orca init` call now:
  (a) creates the namespace dir (`~/.orca` or `/root/.orca` with
  `--system`); (b) runs all DB migrations including the new 0006
  (`nodes.kind`, `nodes.os` — backward-compatible nullable columns);
  (c) bootstraps the internal CA via `security.CAInit` if `ca.crt` is
  absent; (d) generates the server cert via `security.GenerateCSR` +
  `ca.SignCSR` if `server.crt` is absent; (e) auto-detects the local
  OS via `/etc/os-release` `ID=` field (ubuntu/debian/alpine); (f)
  registers a `localhost` node with `kind=localhost`, `os=<detected>`,
  `addr=localhost:8443` if no localhost node exists yet. After
  `orca init`, `orca doctor` MUST pass with zero FAILs. Idempotent:
  re-running `orca init` is a no-op (or refresh) for already-provisioned
  artifacts. Covers REQ-047, REQ-048, REQ-049.
- **P02 — Proxmox SSH join.** `orca node join --type proxmox --host
  <addr> --user root --password <pw>` (password via flag or
  `$ORCA_PROXMOX_PASSWORD`, **never persisted**) bootstraps a remote
  Proxmox 8/9 host via `golang.org/x/crypto/ssh` (new direct dep).
  Steps: (1) SSH password-auth; (2) generate or load orca's SSH
  keypair (`~/.orca/orca_ssh_key` / `.pub`, 0600/0644); (3) deploy
  pubkey to remote `~orca/.ssh/authorized_keys`; (4) create `orca`
  user (config-overridable name via `--proxmox-user`, default `orca`);
  (5) create PVE custom role `OrcaOperator` (config-overridable via
  `--proxmox-role`) with privileges `VM.Audit`,
  `Datastore.AllocateSpace`, `SDN.Use`; (6) assign role to `orca`
  user on `/`; (7) drop `/etc/sudoers.d/orca` allowlist (`pct`, `qm`,
  `pvesh`, `apt-get`, `dpkg` — no shell-escape commands); (8) record
  node row `kind=proxmox`, `os=pve`, audit log. Idempotent re-run.
  Covers REQ-050, REQ-051.
- **P03 — `doctor os` + `doctor proxmox`.** Extends `orca doctor`
  with two new checks: `doctor os` re-runs `/etc/os-release` detection
  and verifies it matches the stored localhost node row's `os` field
  (drift = WARN); `doctor proxmox` iterates `kind=proxmox` nodes and
  SSH-probes each with `pveversion` / `pvecmd status` (3s timeout per
  peer per D-038 pattern), reporting PASS/WARN/FAIL per node. All
  bootstrap + join actions emit structured audit-log entries. Covers
  REQ-052.
- **P04 — Final review + ship + audit.** Milestone release.

The vision ("minimalist, offline-first, CLI-first orchestration
engine") is unchanged. v0.6 is a bootstrap-ergonomics + heterogeneous-
nodes milestone, not a direction change.

## v0.6 Clarified Decisions (D-series, full autonomy)

The 8 v0.6 decisions (D-030..D-037) were resolved during the CLARIFY
stage — D-030..D-034 confirmed by the operator in plan mode, D-035..D-037
auto-resolved at full autonomy within the `clarify_budget`:

| ID | Question | Decision | Rationale | Confidence |
|----|----------|----------|-----------|------------|
| D-030 | SSH library for Proxmox join? | **`golang.org/x/crypto/ssh`** | Stdlib-adjacent, well-maintained, single new direct dep. Matches orca's minimal-deps ethos. Shell-out to `/usr/bin/ssh` would require openssh-client on the orca host and complicate password-auth + idempotent pubkey deploy. | 0.92 (operator-confirmed) |
| D-031 | Proxmox join password handling? | **Flag/env only, never persisted** | `--password` flag or `$ORCA_PROXMOX_PASSWORD` is used once to deploy the orca pubkey + create the `orca` user; the password is never written to SQLite. Subsequent orca→Proxmox access uses the deployed SSH key. | 0.95 (operator-confirmed) |
| D-032 | Localhost OS auto-detect signal? | **`/etc/os-release` `ID=` field** | Parse `ID=` from `/etc/os-release`; map `ubuntu`/`debian`/`alpine` → node `os`. Falls back to `linux` (unknown) if none match. Simplest reliable signal across the three target distros. | 0.93 (operator-confirmed) |
| D-033 | Least-privilege Proxmox role granularity? | **Custom PVE role `OrcaOperator`** with `VM.Audit`, `Datastore.AllocateSpace`, `SDN.Use` + `/etc/sudoers.d/orca` allowlist (`pct`, `qm`, `pvesh`, `apt-get`, `dpkg`) | Config-overridable role + user names. Sufficient for "manage the host, VMs/CTs, storage, packages" without granting root shell. Built-in `PVEAuditor` is too read-only; full `Administrator` is too broad. | 0.88 (operator-confirmed) |
| D-034 | Node kind/os schema? | **Add `nodes.kind` + `nodes.os` columns via migration 0006** | Schema-first, queryable, doctor can branch on kind. Nullable with `localhost`/`""` defaults for existing rows (backward-compatible). data-engineer owns the migration. | 0.94 (operator-confirmed) |
| D-035 | SSH host-key verification on first Proxmox connect? | **TOFU: pin on first connect, refuse on mismatch thereafter** | First connect uses `ssh.InsecureIgnoreHostKey` to capture the host key; it is then persisted to `~/.orca/known_hosts` (or the nodes metadata) and all subsequent connects require a match. Balances first-run ergonomics against MITM risk on subsequent runs. Switching to pre-pinned keys is a future enhancement. | 0.82 (auto) |
| D-036 | `orca init` idempotency semantics for already-provisioned artifacts? | **Skip-and-refresh, never overwrite** | If `ca.crt` exists → load it (no regen). If `server.crt` exists → keep it (no reissue). If a localhost node row exists → update `last_seen` + re-detect `os`, never insert a duplicate. If DB migrations are ahead → no-op. If `~/.orca` exists → MkdirAll is a no-op. Idempotent re-run is a hard requirement (REQ-047). | 0.95 (auto) |
| D-037 | orca SSH keypair location + algorithm? | **`~/.orca/orca_ssh_key` (0600) + `~/.orca/orca_ssh_key.pub` (0644), Ed25519** | Ed25519 keys are smaller, faster, and more secure than RSA for SSH auth. Stored in the orca namespace dir alongside ca.crt/server.crt so `ORCA_HOME` relocation works. File modes mirror the cert file-mode discipline (REQ-033 spirit). Generated lazily on first `orca node join --type proxmox`, not at `orca init` (localhost doesn't need SSH). | 0.90 (auto) |

### v0.6 clarification notes

- **D-035 TOFU caveat**: TOFU (trust-on-first-use) is the standard SSH
  UX and matches the operator-mediated model from D-012 (CA cert
  distribution). The operator is expected to verify the host key
  fingerprint out-of-band on first connect if the network is
  untrusted. A future milestone may add `--host-key-fingerprint` pin
  flag to `orca node join --type proxmox` for pre-pinned deployments.
- **D-036 idempotency**: re-running `orca init` on a node that already
  has a localhost row updates `last_seen` and re-detects `os` (in case
  the host OS was upgraded) but does NOT change the node `ID` or
  `joined_at`. This makes `orca init` safe to put in a systemd
  ExecStartPre or a config-management runbook.
- **D-037 Ed25519**: `golang.org/x/crypto/ssh` + `golang.org/x/crypto/ed25519`
  are in the same module; no additional direct dep beyond D-030.

## v0.7 Clarified Decisions (D-series, full autonomy)

The 5 v0.7 decisions (D-038..D-042) were auto-resolved at full autonomy
within the `clarify_budget` (10):

| ID | Question | Decision | Rationale | Confidence |
|----|----------|----------|-----------|------------|
| D-038 | Config file format — HCL or YAML? | **HCL** | D-009 already specced `config.hcl`. HCL is already a direct dep (hashicorp/hcl/v2 for jobspec). Adding YAML would introduce a second parser dep — violates minimal-deps. Use the existing `hclparse` pkg from jobspec. | 0.93 |
| D-039 | Config precedence order (flag vs env vs file vs default)? | **flag > env > file > default** | Standard layered config: the most explicit (flag) wins, then the runtime (env), then the persisted (file), then the built-in default. Matches cobra/viper convention without the viper dep. | 0.92 |
| D-040 | pprof security — bind to localhost only, or operator-chosen addr? | **Operator-chosen `--pprof <addr>` (default disabled)** | Default disabled keeps the minimalist posture. Operator picks the addr — localhost for dev, unix socket for prod. Separate mux so it never touches the mTLS daemon listener. No auth (pprof is operator-only, addr is the gate). | 0.85 |
| D-041 | cert command registration — where in root command order? | **After `cert` is unreachable today, append after `node` in rootCmd.AddCommand order** | Alphabetical-ish with the existing cluster (audit, daemon, doctor, init, job, node, cert, status, version). No behavior change to existing commands. | 0.88 |
| D-042 | Coverage target — 50% floor or higher? | **50% floor per package, 70% target for new packages** | 50% is achievable for the concurrent packages (engine, transport) without heroic mock effort; 70% is the floor for new code in P02/P04. Avoids a "raise coverage everywhere" rathole. | 0.85 |

## v0.7 Scope Summary — Hardening & Completion

v0.7 is a 4-execution-phase **NFR milestone** that closes out gaps
surfaced by the v0.7 IDEATE stage: an unreachable command tree, a
missing config file layer, low test coverage in core packages, and the
long-deferred pprof endpoint. The engine functionality from v0.1–v0.6
is unchanged; this milestone is purely about **correctness, coverage,
and operability**:

- **P01 — Register `orca cert` command tree + cert_repo tests.** The
  `internal/cli/cert.go` command (`cert ca-init`, `cert gen`, `cert
  show`, `cert renew`, `cert fingerprint`) is fully implemented but
  never wired into `rootCmd`. This phase adds the missing
  `rootCmd.AddCommand(newCertCmd(...))` and adds the missing
  `internal/store/cert_repo_test.go`. Covers REQ-053.
- **P02 — HCL config file parsing (`config.hcl`).** D-009 specified
  `~/.orca/config.hcl` and `/etc/orca/orca.hcl` as config locations,
  but no HCL config-file parser exists — the CLI relies entirely on
  flags and env vars. This phase adds a minimal `internal/config`
  package that loads `config.hcl` (keys: `db_path`, `listen_addr`,
  `ca_path`, `server_cert_path`, `server_key_path`, `node_capacity`),
  merges with env/flag overrides (flag > env > file > default), and
  surfaces it via `--config` flag on the root command. Covers
  REQ-054.
- **P03 — Test coverage uplift.** Adds tests for the lowest-coverage
  packages: `internal/engine` (executor, dispatcher, peer — currently
  8.3%), `internal/transport` (mtls, dispatch, handshake_log —
  currently 26.3%), `internal/proxmox` (bootstrap SSH path —
  currently 5.1%), and `internal/audit` (no tests). Target: every
  package ≥ 50% coverage. Covers REQ-055.
- **P04 — `--pprof` opt-in on `orca daemon`.** Adds the long-deferred
  I-308 pprof endpoint behind an opt-in `--pprof <addr>` flag (default
  disabled). `net/http/pprof` mounted on a separate mux so it never
  touches the mTLS daemon listener. Covers REQ-056.
- **P05 — Final review + ship + audit.** Milestone release.

The vision ("minimalist, offline-first, CLI-first orchestration
engine") is unchanged. v0.7 is a hardening milestone, not a direction
change. Milestone type: NFR (all phases are fix/test/chore); the final
phase's progressive patch IS the deliverable per `run.md` versioning
logic. Tags run on the v0.6.x patch line: `v0.6.0` (P0) … `v0.6.5` (P05
= milestone release).

## v0.8 Scope Summary — Coverage & Trust Hardening

v0.8 is a 3-execution-phase **NFR milestone** that continues the
hardening theme opened by v0.7. v0.7 P03 (REQ-055) lifted four
packages to ≥ 50%, but a coverage re-baseline after v0.7 ship shows
the floor was insufficient: `internal/engine` regressed to 8.3%,
`internal/proxmox` to 5.1%, and four more packages sit between 26% and
48%. Three packages (`internal/audit`, `internal/certpaths`,
`cmd/orca`) still have **no test files at all**. v0.8 also closes the
two "future enhancement" hooks explicitly deferred in v0.6 — SSH
host-key pre-pinning (D-035 caveat) and `orca node key-reset`
(RESEARCH_v0.6 §80) — and adds a requirements-hygiene gate so the
stale-REQ-status drift seen in REQUIREMENTS.md after v0.7 ship cannot
recur:

- **P01 — Coverage uplift round 2.** Raise six under-50% packages to
  ≥ 70% and add first tests for the three zero-test packages. Covers
  REQ-057.
- **P02 — SSH trust hardening.** `--host-key-fingerprint` pre-pin flag
  on `orca node join --type proxmox` + `orca node key-reset <node>`
  command. Covers REQ-058, REQ-059.
- **P03 — Requirements-hygiene gate.** `make verify-reqs` target +
  verify-stage assertion that ROADMAP `Complete` ↔ REQUIREMENTS
  `Complete`. Covers REQ-060.
- **P04 — Final review + ship + audit.** Milestone release.

The vision is unchanged. v0.8 is a hardening milestone, not a
direction change. Milestone type: NFR (all phases are test/feat-chore
on the trust surface — see CLARIFY D-043 for the `feat` vs `chore`
classification of P02); the final phase's progressive patch IS the
deliverable per `run.md` versioning logic. Tags run on the **v0.7.x**
patch line: `v0.7.0` (P0) … `v0.7.4` (P04 = milestone release).

## v0.8 Clarified Decisions (D-series, full autonomy)

The 5 v0.8 decisions (D-043..D-047) were auto-resolved at full autonomy
within the `clarify_budget` (10):

| ID | Question | Decision | Rationale | Confidence |
|----|----------|----------|-----------|------------|
| D-043 | Is P02 (SSH trust hardening) a `feat` phase or a `chore` phase? It adds a new flag + a new subcommand. | **`chore` (trust-surface hardening), not `feat`** | Both `--host-key-fingerprint` and `orca node key-reset` refine the *existing* `orca node join --type proxmox` flow and the existing TOFU `known_hosts` store (D-035). No new orchestration capability, no new node kind, no new API. They close a security gap explicitly deferred in v0.6, not open new surface area. Per `run.md` versioning logic this keeps v0.8 NFR (all phases fix/test/chore/perf/refactor). | 0.84 |
| D-044 | Where does `--host-key-fingerprint` live — on `orca node join` or only on `--type proxmox`? | **On `orca node join` (root of the join subcommand), validated when `--type proxmox`** | The flag is generic (any future SSH-joined node kind will use it); gating it to `--type proxmox` only would require re-adding it later. Validation (`flag requires --type proxmox today`) happens in `RunE`, not in the flag declaration, so the flag is declared once on `node join` and the type check emits a clear error for non-proxmox types until other SSH-joined kinds exist. | 0.86 |
| D-045 | `--host-key-fingerprint` format — raw hex, `sha256:`-prefixed, or OpenSSH `SHA256:base64`? | **OpenSSH `SHA256:base64` (the format `ssh-keyscan -E sha256 -D -` emits and operators expect)** | Matches the fingerprint format operators already see from `ssh-keyscan` and `orca node join`'s own `Result.HostKeyFingerprint` output. Accept only `SHA256:`-prefixed base64; reject raw hex with a clear error. Internally decode base64 → compare against `ssh.PublicKey` Marshal + sha256. | 0.88 |
| D-046 | Does `orca node key-reset <node>` also revoke the orca pubkey on the remote host, or only clear the local `known_hosts` entry? | **Local `known_hosts` entry only** | Revoking the remote authorized_keys entry would orphan a working node (next dispatch would fail auth). `key-reset` is the local "forget this host's key" operation (mirrors `ssh-keygen -R host`); re-establishing trust is a separate `orca node join` re-run. Audit-log the reset with `actor`, `node`, `event=node.key_reset`. | 0.90 |
| D-047 | Coverage target for P01 — 70% floor or higher? | **70% floor for the 6 under-50% packages; 50% floor for the 3 zero-test packages (`internal/audit`, `internal/certpaths`, `cmd/orca`) as a first-toe-hold** | 70% across the board for the already-tested packages matches D-042's "70% target for new packages" and is achievable without heroic mock effort. For the zero-test packages, going 0→50% is the realistic single-phase step (0→70% risks a coverage rathole on `cmd/orca` which is glue code); a future milestone can lift them to 70%. | 0.82 |

---

# v0.9/v0.10 — Re-architecture Scope Summary (Supersedes v0.1–v0.8 architecture)

v0.9 is the first DIRECTION-CHANGE milestone in the project's history.
It supersedes the shipped v0.1–v0.8 architecture per the adopted PRD
(`.ciagent/PRD_v0.9.md`). The re-architecture deprecates the daemon/
transport/internal-CA/HCL/single-namespace stack and builds a CLI-only/
SSH-push/step-ca/Markdown-frontmatter/multi-namespace stack plus 8
net-new subsystems.

## Override Justification (Re-architecture Justification axis)

The ci-griller returned REPLAN (0.70) on the Re-architecture Justification
axis, noting the PRD reverses 6 documented decisions without new evidence
and that the incremental-additive path was not evaluated. The user reviewed
the fork and overrode the *direction* with a six-part evidence basis. The
override is recorded verbatim below; each part addresses a reversal that
the grill flagged as unjustified.

1. **The v0.8 daemon model is operationally failing** in the target
   environment — R-001 ("no orca binary on any server") is a response to
   measured pain, not preference.
2. **step-ca is externally mandated** (D-101) — the operator environment
   requires an external CA; AD-010's "too heavyweight" rationale is no
   longer operative.
3. **Multi-tenancy is a hard product requirement** (R-002) — real
   multi-tenant use cases cannot be served by the single-namespace layout;
   the "no multi-tenancy" anti-pattern is obsolete.
4. **WASM is a hard workload requirement** (D-088) — workloads are WASM, not
   processes; `os/exec` is insufficient; the "no container runtime"
   anti-pattern is reversed.
5. **SSH-push is the only viable deployment target** for the operator's
   bare-Linux/Proxmox environment — installing/maintaining an orca daemon
   on every peer is operationally infeasible.
6. **Simplicity/vision correction** — the v0.1-v0.8 daemon model was a
   wrong turn against the original CLI-first vision; the re-architecture
   corrects the vision.

## Supersession Table (AD-series reversals, recorded per grill PC-09)

| Old decision | Was | Superseded by | Evidence basis |
|---|---|---|---|
| AD-010 (ARCHITECTURE.md:463) | step-ca/cfssl/vault-pki "too heavyweight" | **D-101** (step-ca) | Override ground 2 (external mandate) |
| SPIFFE rejection (PROJECT.md:94) | internal CA chosen over SPIFFE | **D-068** (SPIFFE SVIDs) | Override ground 3 (multi-tenancy requires per-workload identity) |
| No-container-runtime (ARCHITECTURE.md:477) | explicit anti-pattern | **D-088** (5 runtimes; wasmtime primary) | Override ground 4 (WASM is the workload profile) |
| No-multi-tenancy (ARCHITECTURE.md:478) | explicit anti-pattern | **D-158 / R-002** (many namespaces under ORCA_HOME) | Override ground 3 (hard multi-tenant product req) |
| AD-007 (HCL canonical) | HCL for jobspec | **R-013 / R-014** (Markdown canonical; HCL legacy) | PRD §8 (Markdown + body preservation is the operator-facing format) |
| Daemon-on-every-node | `orca daemon` on all peers | **R-001** (no orca binary on any server) | Override grounds 1 + 5 (daemon failing; SSH-push only viable target) |

The 19 binding conditions (C-01..C-19) and 10 phase challenges
(PC-01..PC-10) from `GRILL_v0.9.md` are adopted as execution gates.
The 30 net-new requirements (REQ-061..REQ-090) from `IDEATION_v0.9.md`
are recorded in `REQUIREMENTS.md`. The reordered phase plan is in
`ROADMAP.md`.

## v0.9 Clarified Decisions (D-series, full autonomy — Phase 0 pre-execution)

| ID | Question | Decision | Rationale | Confidence |
|----|----------|----------|-----------|------------|
| D-101 | Cluster CA: internal Go CA (AD-010) or step-ca (external)? | **step-ca (apt-installed)** | Externally mandated per override ground 2; AD-010's "too heavyweight" rationale reversed. CLI wraps `step` CLI via SSH (no Go step-ca client library — keep zero-new-dep posture if possible, or add `github.com/smallstep/cli` as a dep). **Gated by C-07** (CA migration spec). | 0.74 |
| D-068 | Workload identity: internal X.509 CA or SPIFFE SVIDs? | **SPIFFE SVIDs minted at submit time via step-ca** | Multi-tenancy (override ground 3) requires per-workload identity model; SPIFFE is the standard. SPIFFE ID `spiffe://orca/ns/<ns>/job/<name>/alloc/<id>` as SAN. **Gated by C-08** (mint spike in v0.10-P01.5; fallback to mTLS identity if spike fails). | 0.72 |
| D-088 | Runtime: direct os/exec only (D-008) or multi-runtime? | **5 runtimes: wasm (wasmtime primary), podman, process, pve-vm, pve-ct** | WASM is the primary workload (override ground 4). `processRuntime` wraps existing `executor.go`; others are net-new. Split P07a/b/c per grill PC-10. **P07b gated by C-01** (wasmtime/CGO eval). | 0.82 |
| D-158 | Namespace model: single flat root or multi-namespace? | **Multi-namespace under ORCA_HOME (R-002)** | Hard multi-tenant product requirement (override ground 3). `_defaults/` implicit root; `cluster/` for cluster-wide; per-namespace `db/`, `.env`, `.env.secrets`, `jobs/`, `alloc/`, `ns.md`. No namespace column in SQLite. | 0.84 |
| D-179 | Jobspec format: HCL canonical (AD-007) or Markdown? | **Markdown with YAML frontmatter canonical (R-013); HCL legacy** | PRD §8 — Markdown + body preservation is the operator-facing format. HCL adapter (REQ-064) preserves `orca job run old-spec.hcl` during migration. | 0.85 |
| D-185 | Re-architecture justification: incremental additive or full re-architecture? | **Full re-architecture (overridden by user)** | Six-part evidence basis above; the grill's REPLAN mechanics (PC-01..PC-10, C-01..C-19) adopted as gates. The incremental-additive path was evaluated and rejected on grounds 1 + 5 (daemon failing; SSH-push only viable). | 0.88 |
| D-187 | wasmtime Go binding (bytecodealliance/wasmtime-go) is CGO-based — does adopting it revoke D-002 (modernc/sqlite CGO-free cross-compile story)? | **Use the wasmtime CLI (apt-installed on peer) via SSH exec; do NOT import wasmtime-go.** | The Go binding links libwasmtime via cgo and would revoke D-002's CGO-free cross-compile story. The CLI-via-SSH approach (same pattern as podman/qm/pct) avoids CGO entirely. `internal/runtime/wasm.go` imports only stdlib + sshpush. `CGO_ENABLED=0 go build ./...` succeeds. C-01 grill gate SATISFIED; D-002 NOT revoked. Full evaluation in `internal/runtime/C01_WASMTIME_CGO_EVAL.md`. | 0.90 |

---

# v0.10 Docs & Install Milestone — Scope Summary

v0.10 is a focused milestone that closes the documentation gap left by
the v0.9 re-architecture and fixes the release/install pipeline bug that
caused `install.sh` to resolve to v0.4.5 instead of the latest release.
The v0.9 re-architecture shipped a complete CLI surface (markdown
jobspec, `orca ns`, `orca node capacity`, CLI-side scheduler, emitters,
Traefik ingress) but no operator-facing reference documentation. This
milestone ships that documentation plus a worked full-stack example
with ingress configured, and hardens the release pipeline so every
Gitea release carries a Linux binary asset.

## Root cause of the v0.4.5 install

The v0.8.x releases (v0.8.0 through v0.8.15) shipped with **zero binary
assets attached** to their Gitea releases. `scripts/install.sh` resolves
"latest" by hitting `/releases/latest` (returns v0.8.15), then looks for
`orca-v0.8.15-linux-amd64.tar.gz` in that release's assets. Since the
asset is missing, install.sh errors out — there is no fallback walk to
older releases that DO carry a binary. The user's v0.4.5 install came
from an earlier run or a pinned `--version`. The fix is forward: harden
`scripts/release.sh` to cross-build the amd64 tarball and verify the
asset attached post-create; harden `scripts/install.sh` to walk
backward through releases if the latest lacks the asset.

## v0.10 Phases

- **Phase 0 (pre-execution)**: specify → clarify → research → ideate → plan → grill. Tag `v0.9.0`.
- **Phase P1 — release/install fix** (REQ-097, REQ-098): cross-build amd64 tarball in release.sh, post-create asset verification, install.sh fallback walk. Tag `v0.9.1`.
- **Phase P2 — CLI + jobspec + ingress docs** (REQ-091, REQ-092, REQ-093): `docs/cli.md`, `docs/jobspec.md`, `docs/ingress.md`. Tag `v0.9.2`.
- **Phase P3 — full-stack examples** (REQ-094): `examples/full-stack/` with 5 valid jobspecs + rendered artifacts + walkthrough README. Tag `v0.9.3`.
- **Phase P4 — README + namespace.md refresh** (REQ-095, REQ-096): README subcommand table + install example + docs/examples sections; `docs/namespace.md` v0.9 layout. Tag `v0.9.4`.
- **Phase P5 — final review + ship + audit** (milestone release). Tag `v0.9.5` = v0.10.0 milestone release.

**Milestone type**: feature (P1 ships `fix` phases; P2/P3/P4 ship `docs`
phases; at least one non-docs phase makes this a feature milestone per
the versioning logic). Tags run on the v0.9.x patch line. The milestone
branch label is `milestone/v0.10-docs-cli-examples`.

The vision ("minimalist, offline-first, CLI-first orchestration
engine") is unchanged. v0.10 is a documentation + install-hardening
milestone, not a direction change. It builds on the v0.9
re-architecture foundation without modifying any Go orchestration code.

## v0.10 Clarified Decisions (D-series, full autonomy — Phase 0 pre-execution)

| ID | Question | Decision | Rationale | Confidence |
|----|----------|----------|-----------|------------|
| D-188 | Should the CLI docs be a single `docs/cli.md` reference or a per-command `docs/cli/` subdirectory? | **Single `docs/cli.md` reference** | Mirrors the existing flat `docs/` pattern (install.md, docker.md, namespace.md, security-scanning.md). One file is more discoverable for a CLI tool and avoids navigation overhead. A per-command subdirectory diverges from the established layout. | 0.92 |
| D-189 | Should the examples live in `examples/full-stack/` or in `testdata/`? | **`examples/full-stack/` as a new top-level directory** | `testdata/` holds legacy HCL fixtures (`hello.hcl`, `fail.hcl`) used by Go tests; mixing operator-facing examples with test fixtures conflates audiences. A new `examples/` directory is the conventional location for worked examples and is what an operator expects to find. | 0.93 |
| D-190 | How deep should the ingress/Traefik documentation go? | **Dedicated `docs/ingress.md` plus a worked example in `examples/full-stack/`** | Ingress is the user's explicit ask ("full stack with ingress configured") and the Traefik/service-block model (R-007 socket vs TCP, atomic reload, drain, TLS) is non-trivial. A dedicated doc is the clearest answer; a section buried in `docs/cli.md` would be less discoverable. | 0.90 |
| D-191 | Should the docs frame the v0.9 canonical path or document both v0.8 and v0.9 equally? | **Document the v0.9 canonical path; flag deprecated surface with callout boxes** | The v0.8 daemon/mTLS/HCL path is deprecated and scheduled for removal in v0.10-P14. Documenting it as primary misleads new operators; documenting both equally doubles the surface and risks documenting soon-removed code. Callout boxes with "deprecated in v0.9, removed in v0.10" point operators to the canonical path. | 0.91 |
| D-192 | Should the existing v0.8.15 release be backfilled with a binary asset, or only fix the pipeline forward? | **Fix forward only; no backfill** | Backfilling a past release is an ops task, not a docs milestone deliverable. The next tagged phase (this milestone's P1 ship at v0.9.1) will be the first correctly-asseted release; install.sh's new fallback walk handles the gap until then. | 0.88 |
| D-193 | Should `release.sh` build only `linux-amd64` or also `linux-arm64`? | **Cross-build `linux-amd64` explicitly (host-arch-independent); arm64 deferred to a follow-up** | The install.sh user base is amd64 today (the `.coreci.yml` release step hardcodes `--asset orca-${VERSION}-linux-amd64.tar.gz`). Building amd64 regardless of host arch (via `GOOS=linux GOARCH=amd64 go build`) guarantees the asset the install script expects. arm64 support is a separate enhancement. | 0.85 |
| D-194 | Should `install.sh` add a `--check` dry-run mode? | **Yes, lightweight** | A dry-run mode (`--check`) that prints the version + asset URL + install path without writing is cheap to add and useful for debugging the "which release will I get?" question that the v0.4.5 incident surfaced. | 0.80 |