orca/.ciagent/PROJECT.md

Project: Orca

What This Is

A minimalist, offline-first, CLI-first orchestration engine inspired by HashiCorp Nomad, prioritizing stability, security, and simplicity over feature richness. Single-binary distribution, no container runtime, no cloud dependencies, no K8s-level complexity.

Vision

A minimalist, offline-first, CLI-first orchestration engine inspired by HashiCorp Nomad, prioritizing stability, security, and simplicity over feature richness.

Objective

Build a lightweight system to manage and execute workloads across a set of nodes, keeping complexity far below that of Kubernetes.

Requirements

• CLI First: Primary interaction through a CLI tool.
• Offline First: Functional without constant internet connectivity.
• AI First: Designed to be easily discoverable and manageable by AI agents.
• Stability & Security: Prioritize security fixes and bug fixes over new features.
• Language: Written in Go 1.25+.
• Simplicity: Minimalist implementation, avoiding the "K8s complexity trap".

Constraints

• No web UI as a primary requirement.
• Must not implement K8s-level complexity.
• Feature development must move slowly to ensure stability.
• Only CI system allowed: CoreCI (git.cloudinit.dev/coreci/coreci).
• Gitea remote: git.cloudinit.dev/coreci/orca.

Clarified Decisions (D-series, full autonomy)

ID	Question	Decision	Rationale	Confidence
D-001	Single binary or multi-binary distribution?	Single binary	Simpler distribution; subcommands baked into one orca binary. Aligns with simplicity pillar.	0.95
D-002	Local state store technology?	modernc/sqlite (pure Go, CGO-free)	Cross-compile friendly, no CGO dependency, single file on disk, mature.	0.92
D-003	Inter-node communication?	Embedded HTTP (net/http) over loopback, mTLS for cross-node	No external RPC framework needed for v0.1. HTTP suffices.	0.85
D-004	Scheduling algorithm for v0.1?	Single-node only (no scheduling)	Multi-node scheduling is out of scope for v0.1. Tasks run on the node they're submitted to.	0.90
D-005	CLI output format?	Human-readable by default, --json flag for machine consumption	Serves both humans and AI agents.	0.95
D-006	Job/task definition format?	HCL or YAML in .hcl/.yaml files	Familiar to Nomad/HashiCorp users; simpler than JSON for humans.	0.88
D-007	Authentication?	mTLS for v0.1, token-based deferred	mTLS is the most secure default. Tokens can be added later if needed.	0.80
D-008	Container runtime?	Direct process execution (no container runtime) for v0.1	Avoids the Docker/container dependency. Pure process management.	0.85
D-009	Configuration file location?	~/.orca/config.hcl and /etc/orca/orca.hcl	Standard XDG-style paths.	0.90
D-010	Logging format?	Structured JSON via log/slog	Native Go 1.21+ slog, no external dependency.	0.95
D-011	v0.2 mTLS cert authority model?	Internal CA with CSR join	One node bootstraps a local CA; peers generate CSRs and submit them to the CA for signing. CA cert is the trust anchor. More secure than self-signed per-node (single trust root) without the operational complexity of an external PKI.	0.92
D-012	v0.2 CA bootstrap & cert distribution?	Operator-mediated, fingerprint-verified	Bootstrap node writes ~/.orca/ca.crt and ~/.orca/ca.key (mode 0600). Operator copies ca.crt to peers; peers verify by SHA-256 fingerprint at orca node join --ca-fingerprint <sha256>. No automated secret distribution.	0.85
D-013	v0.2 cert validity & rotation?	Server certs 90 days, CA cert 10 years, rotate 30 days before expiry	Server certs are short-lived (compromise window small); CA is long-lived (manual rotation is expensive). orca cert renew reissues server certs automatically.	0.90
D-014	v0.2 mTLS handshake timing?	Eager — at orca node join time	Fail fast on bad certs, misconfigurations, or CA mismatches. Lazy handshake would let stale configs run until first request, complicating debugging.	0.88
D-015	v0.2 minimum TLS version & cipher suites?	TLS 1.3 only; AEAD cipher allowlist	MinVersion=tls.VersionTLS13, CipherSuites limited to TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_AES_128_GCM_SHA256. No TLS 1.2 fallback.	0.92
D-016	v0.2 security-scanning placement?	validate pipeline of .coreci.yml, gates merges to main	gosec baseline JSON checked into repo; new findings fail the build. govulncheck ./... exit-on-known. Pre-commit hook with gitleaks is opt-in (developer machine).	0.85
D-017	v0.2 iter.Seq API surface?	orca job list --watch and orca node list --watch	Pull-based iter.Seq[Job] / iter.Seq[Node]; cancellation via context.Context; signal.NotifyContext on ctrl-c. Backpressure is implicit (consumer-driven).	0.90
D-018	v0.2 multi-node scheduling algorithm?	Bin-packing by available CPU/memory, FIFO within a node	Simple, deterministic, matches D-004 minimalism. Cross-node dispatch via ConnectRPC orca.v1.Dispatch service. Retry on transient failures with exponential backoff.	0.85

Out of Scope

• Full-blown Kubernetes-compatible API.
• Complex cloud-provider integrations.
• GUI-based management consoles.
• Container runtime integration.
• Service mesh / sidecar injection.
• Auto-scaling / horizontal pod autoscaler.
• External PKI / Let's Encrypt / cert transparency logs.
• gRPC framework dependency (ConnectRPC in config.json frameworks but not in go.mod; v0.2 uses stdlib net/http with h2c for orca.v1.Dispatch — see ARCHITECTURE.md AD-014).

v0.2 Scope Summary

v0.2 is a focused 4-phase milestone that turns Orca from a single-node process executor into a small cluster engine with strong transport security and richer I/O. The 4 phases are:

• P01 — mTLS handshake + internal CA with CSR join. Internal CA, CSR join, eager handshake at node join, TLS 1.3 + AEAD allowlist. See ARCHITECTURE.md Flow 1 + Flow 2.
• P02 — Multi-node scheduling & job dispatch. Best-fit bin-packing by CPU/memory, FIFO within a node, orca.v1.Dispatch over mTLS. See ARCHITECTURE.md Flow 3.
• P03 — gosec + govulncheck + gitleaks in CI. gosec baseline JSON in repo, govulncheck ./... in validate pipeline, gitleaks in pre-commit (opt-in).
• P04 — iter.Seq streaming for --watch flags. Go 1.25+ range-over-func semantics, context.Context cancellation, signal.NotifyContext on ctrl-c. See ARCHITECTURE.md Flow 4.

The vision ("minimalist, offline-first, CLI-first orchestration engine") is unchanged. v0.2 is a hardening + small-cluster extension, not a direction change.

Key Decisions

The 18 D-series decisions (D-001..D-018) are recorded in the "Clarified Decisions" table above. The 10 v0.1 decisions (D-001..D-010) are stable and unchanged in v0.2. The 8 v0.2 decisions (D-011..D-018) were auto-resolved under full autonomy and are summarized here:

• D-011: Internal CA with CSR join (vs. self-signed per-node or SPIFFE). Single trust root, no external PKI, CSR workflow.
• D-012: Operator-mediated CA cert distribution with fingerprint verify (no automated secret distribution — matches offline-first principle).
• D-013: 90d server certs, 10y CA cert, 30d pre-expiry rotation.
• D-014: Eager mTLS handshake at orca node join time (fail fast).
• D-015: TLS 1.3 only, AEAD cipher allowlist (no TLS 1.2 fallback).
• D-016: gosec+govulncheck in validate pipeline of .coreci.yml (gates merges to main). gitleaks in pre-commit (opt-in).
• D-017: iter.Seq for orca job list --watch and orca node list --watch (pull-based, ctx cancellation, ctrl-c via signal.NotifyContext).
• D-018: Bin-packing by CPU/memory with FIFO within node; JSON-over-HTTP orca.v1.Dispatch for cross-node (no ConnectRPC dep).

v0.3 Clarified Decisions (D-series, full autonomy)

v0.3 is a lean 2-execution-phase milestone completing the streaming and doctor work deferred from v0.2. The 6 v0.3 decisions (D-019..D-024) were auto-resolved under full autonomy:

ID	Question	Decision	Rationale	Confidence
D-019	Watch refresh mechanism?	Poll-based, 1s ticker	Simpler than event channel; no daemon coupling for CLI; matches offline-first.	0.90
D-020	Watch output format (REQ-030)?	Table by default; --watch --json streams one-line JSON per event	Consistent with D-005 --json convention; serves humans + AI agents.	0.92
D-021	Doctor network check scope?	Probe configured peer addresses via mTLS /healthz handshake; PASS/WARN/FAIL per peer	Reuses existing transport client; read-only.	0.85
D-022	Doctor db check scope?	PRAGMA integrity_check + migration version query	Already specced in ARCHITECTURE.md §5; minimal surface.	0.95
D-023	iter.Seq cancellation?	signal.NotifyContext on SIGINT/SIGTERM	Per D-017 + ARCHITECTURE Flow 4.	0.95
D-024	--watch applies to job list only, or node list too?	Both orca job list --watch and orca node list --watch	Per ARCHITECTURE.md CLI layer + D-017.	0.92

v0.3 Scope Summary

v0.3 is a focused 2-execution-phase milestone completing the work deferred from v0.2 that was NOT already shipped in P08-P10. A codebase audit during re-init SPECIFY confirmed that REQ-014, REQ-027, REQ-028, REQ-029, REQ-031, REQ-037, REQ-039, REQ-040 all shipped in P08-P10 despite stale REQUIREMENTS.md marking them Pending. The remaining work:

• P01 — iter.Seq streaming for --watch flags. Go 1.25+ range-over-func semantics, pull-based iter.Seq[Job] / iter.Seq[Node], context.Context cancellation, signal.NotifyContext on ctrl-c. Applies to both orca job list --watch and orca node list --watch. Covers REQ-022, REQ-030.
• P02 — orca doctor network + db full implementation. Replaces the P01 stubs (NetworkStub, DBStub) with real checks: peer reachability via mTLS /healthz probe; SQLite PRAGMA integrity_check + migration version. Covers REQ-032 (completion).

The vision ("minimalist, offline-first, CLI-first orchestration engine") is unchanged. v0.3 is a completion milestone, not a direction change.

v0.5 Scope Summary — Distribution

v0.5 is a 3-execution-phase milestone that makes Orca installable, distributable, and containerized. The engine functionality from v0.1–v0.3 is unchanged; this milestone is purely about delivery surface:

• P01 — Namespace unification. A single ORCA_HOME environment variable becomes the namespace root for all on-disk state (db, certs, init, daemon). A --system flag on the root command selects the system-level namespace root /root/.orca. Backward compatible: empty ORCA_HOME → ~/.orca. Covers REQ-041, REQ-042.
• P02 — install.sh + in-place update. A 1-liner installer pulls the release binary from the public Gitea release URL, installs at user level by default (~/.local/bin/orca) or system level (/usr/local/bin/orca) with --system. Re-running updates the binary in place while preserving config/db/certs in the namespace dir. Idempotent. Covers REQ-043, REQ-044. Also updates README quickstart (REQ-016 completion).
• P03 — Docker release. A multi-stage Dockerfile builds a distroless image; scripts/release.sh and .coreci.yml publish the image to the Gitea container registry per release. Covers REQ-046.
• P04 — Final review + ship + audit. Milestone release.

The vision ("minimalist, offline-first, CLI-first orchestration engine") is unchanged. v0.5 is a distribution milestone, not a direction change.

v0.5 Clarified Decisions (D-series, full autonomy)

The 5 v0.5 decisions (D-025..D-029) were auto-resolved under full autonomy during the CLARIFY stage:

ID	Question	Decision	Rationale	Confidence
D-025	System-level namespace path layout?	/root/.orca (mirror of user-level ~/.orca)	Consistent shape with user-level; just a different root. Matches the user's "starts at /root" wording. Single dir keeps it simple.	0.90
D-026	Namespace override mechanism at runtime?	Unify on ORCA_HOME as single namespace root for all components (db, certs, init, daemon). Add --system flag that sets root to /root/.orca.	ORCA_HOME already exists for certs; extend to all components. Backward compatible (empty → ~/.orca). One knob, not many.	0.92
D-027	Docker registry target?	Gitea built-in container registry (git.cloudinit.dev/coreci/orca)	Keeps everything in one forge; uses Gitea's native registry. Consistent with REQ-045 (public repo → public image pulls).	0.88
D-028	How to make releases publicly accessible (REQ-045)?	Flip repo visibility to public via tea repos edit coreci/orca --private=false during P0 ship	Simplest path to anonymous downloads; enables both install.sh pulls and docker pulls. Pre-existing .env leak already suppressed via gitleaks baseline + rotate-forward (commit 00127ce).	0.85
D-029	install.sh default version?	Latest release (query Gitea releases API), optional --version vX.Y.Z to pin	Matches typical 1-liner installer UX; users get newest by default, can pin for reproducibility.	0.92

v0.5 Operational prerequisite (P0 ship)

The Gitea repo coreci/orca is currently private (returns 404 unauthenticated). P0 ship flips visibility to public via tea repos edit coreci/orca --private=false so that install.sh can pull release binaries unauthenticated (REQ-045). This is an operational step performed during the P0 ship, verified by an unauth curl against the releases API.

v0.6 Scope Summary — Node Bootstrap & Proxmox

v0.6 is a 3-execution-phase milestone that turns orca init from a bare mkdir into a full single-node cluster bootstrap, and adds Proxmox 8 & 9 as a first-class remote node type joined over SSH with least-privilege role delegation. The engine functionality from v0.1–v0.5 is unchanged; this milestone is about bootstrap ergonomics and heterogeneous node support:

• P01 — orca init full bootstrap. A single orca init call now: (a) creates the namespace dir (~/.orca or /root/.orca with --system); (b) runs all DB migrations including the new 0006 (nodes.kind, nodes.os — backward-compatible nullable columns); (c) bootstraps the internal CA via security.CAInit if ca.crt is absent; (d) generates the server cert via security.GenerateCSR + ca.SignCSR if server.crt is absent; (e) auto-detects the local OS via /etc/os-release ID= field (ubuntu/debian/alpine); (f) registers a localhost node with kind=localhost, os=<detected>, addr=localhost:8443 if no localhost node exists yet. After orca init, orca doctor MUST pass with zero FAILs. Idempotent: re-running orca init is a no-op (or refresh) for already-provisioned artifacts. Covers REQ-047, REQ-048, REQ-049.
• P02 — Proxmox SSH join. orca node join --type proxmox --host <addr> --user root --password <pw> (password via flag or $ORCA_PROXMOX_PASSWORD, never persisted) bootstraps a remote Proxmox 8/9 host via golang.org/x/crypto/ssh (new direct dep). Steps: (1) SSH password-auth; (2) generate or load orca's SSH keypair (~/.orca/orca_ssh_key / .pub, 0600/0644); (3) deploy pubkey to remote ~orca/.ssh/authorized_keys; (4) create orca user (config-overridable name via --proxmox-user, default orca); (5) create PVE custom role OrcaOperator (config-overridable via --proxmox-role) with privileges VM.Audit, Datastore.AllocateSpace, SDN.Use; (6) assign role to orca user on /; (7) drop /etc/sudoers.d/orca allowlist (pct, qm, pvesh, apt-get, dpkg — no shell-escape commands); (8) record node row kind=proxmox, os=pve, audit log. Idempotent re-run. Covers REQ-050, REQ-051.
• P03 — doctor os + doctor proxmox. Extends orca doctor with two new checks: doctor os re-runs /etc/os-release detection and verifies it matches the stored localhost node row's os field (drift = WARN); doctor proxmox iterates kind=proxmox nodes and SSH-probes each with pveversion / pvecmd status (3s timeout per peer per D-038 pattern), reporting PASS/WARN/FAIL per node. All bootstrap + join actions emit structured audit-log entries. Covers REQ-052.
• P04 — Final review + ship + audit. Milestone release.

The vision ("minimalist, offline-first, CLI-first orchestration engine") is unchanged. v0.6 is a bootstrap-ergonomics + heterogeneous- nodes milestone, not a direction change.

v0.6 Clarified Decisions (D-series, full autonomy)

The 8 v0.6 decisions (D-030..D-037) were resolved during the CLARIFY stage — D-030..D-034 confirmed by the operator in plan mode, D-035..D-037 auto-resolved at full autonomy within the clarify_budget:

ID	Question	Decision	Rationale	Confidence
D-030	SSH library for Proxmox join?	golang.org/x/crypto/ssh	Stdlib-adjacent, well-maintained, single new direct dep. Matches orca's minimal-deps ethos. Shell-out to /usr/bin/ssh would require openssh-client on the orca host and complicate password-auth + idempotent pubkey deploy.	0.92 (operator-confirmed)
D-031	Proxmox join password handling?	Flag/env only, never persisted	--password flag or $ORCA_PROXMOX_PASSWORD is used once to deploy the orca pubkey + create the orca user; the password is never written to SQLite. Subsequent orca→Proxmox access uses the deployed SSH key.	0.95 (operator-confirmed)
D-032	Localhost OS auto-detect signal?	/etc/os-release ID= field	Parse ID= from /etc/os-release; map ubuntu/debian/alpine → node os. Falls back to linux (unknown) if none match. Simplest reliable signal across the three target distros.	0.93 (operator-confirmed)
D-033	Least-privilege Proxmox role granularity?	Custom PVE role OrcaOperator with VM.Audit, Datastore.AllocateSpace, SDN.Use + /etc/sudoers.d/orca allowlist (pct, qm, pvesh, apt-get, dpkg)	Config-overridable role + user names. Sufficient for "manage the host, VMs/CTs, storage, packages" without granting root shell. Built-in PVEAuditor is too read-only; full Administrator is too broad.	0.88 (operator-confirmed)
D-034	Node kind/os schema?	Add nodes.kind + nodes.os columns via migration 0006	Schema-first, queryable, doctor can branch on kind. Nullable with localhost/"" defaults for existing rows (backward-compatible). data-engineer owns the migration.	0.94 (operator-confirmed)
D-035	SSH host-key verification on first Proxmox connect?	TOFU: pin on first connect, refuse on mismatch thereafter	First connect uses ssh.InsecureIgnoreHostKey to capture the host key; it is then persisted to ~/.orca/known_hosts (or the nodes metadata) and all subsequent connects require a match. Balances first-run ergonomics against MITM risk on subsequent runs. Switching to pre-pinned keys is a future enhancement.	0.82 (auto)
D-036	orca init idempotency semantics for already-provisioned artifacts?	Skip-and-refresh, never overwrite	If ca.crt exists → load it (no regen). If server.crt exists → keep it (no reissue). If a localhost node row exists → update last_seen + re-detect os, never insert a duplicate. If DB migrations are ahead → no-op. If ~/.orca exists → MkdirAll is a no-op. Idempotent re-run is a hard requirement (REQ-047).	0.95 (auto)
D-037	orca SSH keypair location + algorithm?	~/.orca/orca_ssh_key (0600) + ~/.orca/orca_ssh_key.pub (0644), Ed25519	Ed25519 keys are smaller, faster, and more secure than RSA for SSH auth. Stored in the orca namespace dir alongside ca.crt/server.crt so ORCA_HOME relocation works. File modes mirror the cert file-mode discipline (REQ-033 spirit). Generated lazily on first orca node join --type proxmox, not at orca init (localhost doesn't need SSH).	0.90 (auto)

v0.6 clarification notes

• D-035 TOFU caveat: TOFU (trust-on-first-use) is the standard SSH UX and matches the operator-mediated model from D-012 (CA cert distribution). The operator is expected to verify the host key fingerprint out-of-band on first connect if the network is untrusted. A future milestone may add --host-key-fingerprint pin flag to orca node join --type proxmox for pre-pinned deployments.
• D-036 idempotency: re-running orca init on a node that already has a localhost row updates last_seen and re-detects os (in case the host OS was upgraded) but does NOT change the node ID or joined_at. This makes orca init safe to put in a systemd ExecStartPre or a config-management runbook.
• D-037 Ed25519: golang.org/x/crypto/ssh + golang.org/x/crypto/ed25519 are in the same module; no additional direct dep beyond D-030.

v0.7 Clarified Decisions (D-series, full autonomy)

The 5 v0.7 decisions (D-038..D-042) were auto-resolved at full autonomy within the clarify_budget (10):

ID	Question	Decision	Rationale	Confidence
D-038	Config file format — HCL or YAML?	HCL	D-009 already specced config.hcl. HCL is already a direct dep (hashicorp/hcl/v2 for jobspec). Adding YAML would introduce a second parser dep — violates minimal-deps. Use the existing hclparse pkg from jobspec.	0.93
D-039	Config precedence order (flag vs env vs file vs default)?	flag > env > file > default	Standard layered config: the most explicit (flag) wins, then the runtime (env), then the persisted (file), then the built-in default. Matches cobra/viper convention without the viper dep.	0.92
D-040	pprof security — bind to localhost only, or operator-chosen addr?	Operator-chosen --pprof <addr> (default disabled)	Default disabled keeps the minimalist posture. Operator picks the addr — localhost for dev, unix socket for prod. Separate mux so it never touches the mTLS daemon listener. No auth (pprof is operator-only, addr is the gate).	0.85
D-041	cert command registration — where in root command order?	After cert is unreachable today, append after node in rootCmd.AddCommand order	Alphabetical-ish with the existing cluster (audit, daemon, doctor, init, job, node, cert, status, version). No behavior change to existing commands.	0.88
D-042	Coverage target — 50% floor or higher?	50% floor per package, 70% target for new packages	50% is achievable for the concurrent packages (engine, transport) without heroic mock effort; 70% is the floor for new code in P02/P04. Avoids a "raise coverage everywhere" rathole.	0.85

v0.7 Scope Summary — Hardening & Completion

v0.7 is a 4-execution-phase NFR milestone that closes out gaps surfaced by the v0.7 IDEATE stage: an unreachable command tree, a missing config file layer, low test coverage in core packages, and the long-deferred pprof endpoint. The engine functionality from v0.1–v0.6 is unchanged; this milestone is purely about correctness, coverage, and operability:

• P01 — Register orca cert command tree + cert_repo tests. The internal/cli/cert.go command (cert ca-init, cert gen, cert show, cert renew, cert fingerprint) is fully implemented but never wired into rootCmd. This phase adds the missing rootCmd.AddCommand(newCertCmd(...)) and adds the missing internal/store/cert_repo_test.go. Covers REQ-053.
• P02 — HCL config file parsing (config.hcl). D-009 specified ~/.orca/config.hcl and /etc/orca/orca.hcl as config locations, but no HCL config-file parser exists — the CLI relies entirely on flags and env vars. This phase adds a minimal internal/config package that loads config.hcl (keys: db_path, listen_addr, ca_path, server_cert_path, server_key_path, node_capacity), merges with env/flag overrides (flag > env > file > default), and surfaces it via --config flag on the root command. Covers REQ-054.
• P03 — Test coverage uplift. Adds tests for the lowest-coverage packages: internal/engine (executor, dispatcher, peer — currently 8.3%), internal/transport (mtls, dispatch, handshake_log — currently 26.3%), internal/proxmox (bootstrap SSH path — currently 5.1%), and internal/audit (no tests). Target: every package ≥ 50% coverage. Covers REQ-055.
• P04 — --pprof opt-in on orca daemon. Adds the long-deferred I-308 pprof endpoint behind an opt-in --pprof <addr> flag (default disabled). net/http/pprof mounted on a separate mux so it never touches the mTLS daemon listener. Covers REQ-056.
• P05 — Final review + ship + audit. Milestone release.

The vision ("minimalist, offline-first, CLI-first orchestration engine") is unchanged. v0.7 is a hardening milestone, not a direction change. Milestone type: NFR (all phases are fix/test/chore); the final phase's progressive patch IS the deliverable per run.md versioning logic. Tags run on the v0.5.x patch line: v0.5.5 (P0) … v0.5.9 (P05 = milestone release).