Jon Chery
fc94326b0e
P00 — Re-architecture Foundation (deprecation/migration/test-infra/persona/docs). Deprecation sweep (REQ-068, REQ-072, REQ-089): - Add // Deprecated: doc comments to internal/daemon (R-001), internal/transport (REQ-073), internal/security/ca.go+csr.go (D-101/REQ-076), internal/engine/ dispatcher.go+peer.go (CLI-side scheduler), internal/cli/daemon.go. - orca daemon emits slog.Warn deprecation banner on every run (ungated); fires R-001 + v0.10-P05 drain-and-stop + v0.10-P14 deletion. - orca cert and orca node join (mTLS path) emit deprecation warnings; proxmox SSH path (the v0.9 replacement) does not warn. - Add --no-deprecation-warnings global flag on root command (PersistentPreRunE) for orca upgrade migrations. - 12 new daemon/cert/node deprecation tests in internal/cli/daemon_test.go (cli coverage 81.9%, warnDeprecated 100%). - Add DEPRECATED banners to v0.8 sections of ARCHITECTURE.md (verified the v0.9 supersession section + Supersession Table from prior turn are present). Bash tooling gate (grill C-06, C-15, C-16, C-17, C-18): - scripts/tests/test_helper.bash + example_test.bash — bats framework + helpers. - scripts/lib/orca-log.sh — slog-compatible JSON logging to syslog (C-17). - scripts/orca-verify-render.sh — render-contract validator skeleton (C-16). - scripts/tests/orca-log_test.bash + orca-verify-render_test.bash — 20 bats tests total (happy + failure paths per C-15). - .shellcheckrc — project shellcheck config. - Makefile: test-bash + lint-bash targets (graceful skip if tools missing); wired into test + lint targets. - internal/emit/contract.go + contract_test.go — versioned JSON render contract (orca.emit/v1) between Go emitters and bash appliers (C-16). - .ciagent/BASH_CAPABILITY_MAP_v0.9.md — maps shipped internal/transport capabilities to bash-side equivalents or accepted drops (C-18). - D-186 recorded in PROJECT.md: bash exempt from Go coverage gate; compensating control is bats + shellcheck + shfmt (C-06). verify-reqs: 90 requirements consistent. Build/test/lint/fmt all green. 20 bats tests pass. Go tests pass. No v0.8 code deleted — only marked deprecated (deletion deferred to v0.10-P14 per REQ-090 dual-write window). ---ci--- project: orca phase: P00 milestone: v0.9 status: execute ---/ci---
731 lines
41 KiB
Markdown
731 lines
41 KiB
Markdown
# Architecture: Orca
|
||
|
||
## System Overview
|
||
|
||
Orca is a single-binary, offline-first orchestration engine. The system consists
|
||
of three logical layers (CLI, Daemon, Engine) compiled into one `orca` binary
|
||
and selected via subcommands. v0.2 adds a **cross-node transport layer** (mTLS)
|
||
and a **dispatcher** for multi-node job execution.
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||
│ orca (single binary, v0.2) │
|
||
├─────────────────────────────────────────────────────────────────────────────┤
|
||
│ CLI Layer (Cobra) │
|
||
│ ├── orca version │
|
||
│ ├── orca init # local node bootstrap │
|
||
│ ├── orca cert {init,join,renew,show} # NEW (P01) │
|
||
│ ├── orca status │
|
||
│ ├── orca node {join,leave,list} # join = mTLS handshake (P01) │
|
||
│ │ └── orca node list --watch # NEW iter.Seq (P04) │
|
||
│ ├── orca job {run,list,stop,logs} │
|
||
│ │ └── orca job list --watch # NEW iter.Seq (P04) │
|
||
│ ├── orca doctor # NEW (P01) — diagnostics │
|
||
│ │ ├── orca doctor cert │
|
||
│ │ ├── orca doctor network │
|
||
│ │ └── orca doctor db │
|
||
│ └── orca daemon │
|
||
├─────────────────────────────────────────────────────────────────────────────┤
|
||
│ Daemon Layer (net/http over h2c, mTLS in P01) │
|
||
│ ├── /healthz (liveness) │
|
||
│ ├── /readyz (readiness) │
|
||
│ ├── /v1/jobs/* (job control API) │
|
||
│ ├── /v1/nodes/* (node registry API) │
|
||
│ ├── /v1/tasks/* (task lifecycle API) │
|
||
│ ├── /orca.v1.Dispatch/... # NEW (P02) — cross-node dispatch │
|
||
│ └── /orca.v1.Register/... # NEW (P02) — peer join ack │
|
||
├─────────────────────────────────────────────────────────────────────────────┤
|
||
│ Transport Layer (NEW — internal/transport) │
|
||
│ ├── mTLS client (dialer pool per peer) │
|
||
│ ├── mTLS server config (TLS 1.3 only, AEAD allowlist) │
|
||
│ ├── Retry+backoff (exponential, jittered, capped) │
|
||
│ └── Graceful disconnect (ctx-aware Conn.Close) │
|
||
├─────────────────────────────────────────────────────────────────────────────┤
|
||
│ Core Engine │
|
||
│ ├── Node Registry (in-memory + SQLite persistence) │
|
||
│ ├── Task Executor (os/exec with WaitDelay, Go 1.25+) │
|
||
│ ├── Job Scheduler (single-node FIFO; bin-pack P02) │
|
||
│ ├── Dispatcher # NEW internal/engine/dispatcher.go │
|
||
│ │ ├── Local decision (does this job fit on this node?) │
|
||
│ │ ├── Remote dispatch (POST to peer via transport) │
|
||
│ │ └── Streaming callback (iter.Seq[DispatchResult] for CLI) │
|
||
│ ├── Security Manager # NEW internal/security (P01) │
|
||
│ │ ├── CA lifecycle (init, fingerprint, sign CSR) │
|
||
│ │ ├── Server cert lifecycle (issue, renew, rotate) │
|
||
│ │ ├── mTLS config builder │
|
||
│ │ └── Cert store (filesystem + SQLite metadata) │
|
||
│ └── Audit Logger (log/slog JSON handler) │
|
||
├─────────────────────────────────────────────────────────────────────────────┤
|
||
│ State Store (modernc/sqlite, CGO-free) │
|
||
│ ~/.orca/orca.db │
|
||
│ ├── nodes, jobs, tasks, audit_log (v0.1) │
|
||
│ └── certs # NEW (P01) — CA + server certs │
|
||
└─────────────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
## Component Details
|
||
|
||
### 1. CLI Layer (`cmd/orca`, `internal/cli`)
|
||
|
||
- **Framework**: Cobra (industry standard, familiar to operators)
|
||
- **v0.1 subcommands**: `version`, `init`, `status`, `node`, `job`, `daemon`
|
||
- **v0.2 additions (P01)**: `orca cert {init,join,renew,show}`
|
||
- **v0.2 additions (P04)**: `--watch` flag on `orca job list` and `orca node list`
|
||
- **v0.2 additions (P01)**: `orca doctor` subcommand (see §5 below)
|
||
- **Output**: Human-readable by default; `--json` flag for machine consumption
|
||
- **Discovery**: All subcommands self-document via Cobra's auto-generated help
|
||
- **Watch semantics (P04)**: `--watch` consumes `iter.Seq[Job|Node]`, exits on
|
||
ctrl-c (via `signal.NotifyContext`), refreshes on internal change events.
|
||
|
||
### 2. Daemon Layer (`internal/daemon`)
|
||
|
||
> **⚠️ DEPRECATED in v0.9**: This section describes the v0.8 architecture, superseded by the v0.9 re-architecture. See the "v0.9 Architecture" section at the bottom of this file and `.ciagent/PRD_v0.9.md`.
|
||
|
||
- **Server**: `net/http` with `http.ServeMux` (no external router)
|
||
- **TLS (P01)**: `crypto/tls` with `MinVersion=tls.VersionTLS13` and
|
||
AEAD cipher allowlist
|
||
(`TLS_AES_256_GCM_SHA384`, `TLS_CHACHA20_POLY1305_SHA256`,
|
||
`TLS_AES_128_GCM_SHA256`)
|
||
- **Ports**: Configurable (default `:8443` for API+mTLS, `:8080` for health)
|
||
- **Graceful Shutdown**: `signal.NotifyContext` with SIGINT/SIGTERM
|
||
- **v0.2 endpoints (P02)**:
|
||
- `POST /orca.v1.Dispatch/Submit` — receive cross-node job submission
|
||
- `POST /orca.v1.Dispatch/Status` — query dispatched job status
|
||
- `POST /orca.v1.Register/Hello` — peer join ack (used during `orca node join`)
|
||
|
||
### 3. Transport Layer (`internal/transport`, NEW in P01/P02)
|
||
|
||
> **⚠️ DEPRECATED in v0.9**: This section describes the v0.8 architecture, superseded by the v0.9 re-architecture. See the "v0.9 Architecture" section at the bottom of this file and `.ciagent/PRD_v0.9.md`.
|
||
|
||
- **Client**: `http.Client` with `http.Transport.TLSClientConfig` populated
|
||
from `internal/security.NewClientTLSConfig`
|
||
- **Server**: `http.Server.TLSConfig` populated from
|
||
`internal/security.NewServerTLSConfig`
|
||
- **Retry policy**: exponential backoff with jitter (start 100ms, x2, cap 5s,
|
||
max 5 attempts); only idempotent verbs (`GET`, `HEAD`, `OPTIONS`) are
|
||
retried automatically; `POST` retries require an explicit
|
||
`X-Orca-Idempotency-Key` header
|
||
- **Conn lifecycle**: `context.Context`-aware dials and reads; graceful
|
||
`Close` on `ctx.Done()`
|
||
- **Peer dial pool**: small `sync.Map` of `peerID → *http.Client` to reuse
|
||
TLS handshakes (TCP keep-alive) within a session
|
||
|
||
### 4. Core Engine (`internal/engine`)
|
||
|
||
> **⚠️ DEPRECATED in v0.9**: This section describes the v0.8 architecture, superseded by the v0.9 re-architecture. The Dispatcher and PeerRegistry peer-dispatch path is replaced by a CLI-side scheduler + SSH-push (R-001). See the "v0.9 Architecture" section at the bottom of this file and `.ciagent/PRD_v0.9.md`.
|
||
|
||
- **Node Registry**: In-memory map of node IDs → metadata, persisted to SQLite
|
||
(CPU/memory capacity, available slots, last-seen)
|
||
- **Task Executor**: `os/exec.CommandContext` with `WaitDelay` (Go 1.25+) for
|
||
clean process termination
|
||
- **Job Scheduler (v0.2 P02)**:
|
||
- **Algorithm**: best-fit bin-packing by `available_cpu` and `available_memory`
|
||
- **Within-node ordering**: FIFO queue
|
||
- **Cross-node**: if local node is full, `Dispatcher.Submit(peer, job)` is
|
||
invoked; peers are tried in round-robin order
|
||
- **Fallback**: if all peers reject, return `ErrNoFit` and requeue
|
||
- **Dispatcher (NEW, P02)**:
|
||
- `Submit(peerID, spec) (jobID, error)` — blocking call with retry
|
||
- `Watch(peerID) iter.Seq[DispatchEvent]` — pull-style event stream for the
|
||
CLI's `--watch` flag
|
||
- Stateless: every call uses the latest mTLS client config and peer address
|
||
- **Security Manager (NEW, P01)**:
|
||
- `InitCA(commonName) (*CA, error)` — generates a self-signed CA, writes
|
||
`ca.crt` (0644) and `ca.key` (0600) to `~/.orca/`
|
||
- `Fingerprint(certPath) (sha256hex, error)` — used by `orca cert join`
|
||
- `SignServerCert(csr, validity) (*cert, error)` — signs a CSR with the CA
|
||
- `IssueServerCert(nodeName, dnsNames, ips) (*cert, *key, error)` — generates
|
||
a keypair + CSR + signs it, returns PEM bytes for `orca cert join --server`
|
||
- `ServerTLSConfig() (*tls.Config, error)` — loads `server.crt`/`server.key`
|
||
and the CA pool from disk
|
||
- `ClientTLSConfig(caPath) (*tls.Config, error)` — returns a client config
|
||
pinned to the supplied CA
|
||
- **Rotation policy**: server certs valid 90d; CA cert valid 10y. On
|
||
`orca cert renew`, `IssueServerCert` is called and the daemon
|
||
gracefully reloads the in-process `tls.Config` via `GetCertificate`
|
||
hot-swap (no restart required)
|
||
- **Audit Logger**: `slog.NewJSONHandler(os.Stderr, ...)` with structured fields
|
||
|
||
### 5. Doctor (`internal/doctor`, NEW in P01)
|
||
|
||
- **Purpose**: operator-facing diagnostics; runs read-only checks against
|
||
the local state and reports PASS/WARN/FAIL.
|
||
- **Subcommands**:
|
||
- `orca doctor` — runs all checks
|
||
- `orca doctor cert` — cert/CA health (file modes, expiry windows, SAN
|
||
presence, fingerprint pinning match — see REQ-026, REQ-033,
|
||
REQ-034, REQ-036)
|
||
- `orca doctor network` — peer reachability over mTLS (per-peer handshake
|
||
sanity, last-seen delta)
|
||
- `orca doctor db` — SQLite integrity check (`PRAGMA integrity_check`)
|
||
+ migration version
|
||
- **Output**: human-readable by default; `--json` for machine consumption
|
||
- **No state changes**: doctor is strictly read-only. It can be run
|
||
while the daemon is down (where possible) or while it's up.
|
||
- **Initial implementation in P01** (cert checks only); `network` and
|
||
`db` checks land in subsequent phases as their state becomes
|
||
available.
|
||
|
||
### 6. State Store (`internal/store`)
|
||
|
||
- **Driver**: `modernc.org/sqlite` (pure Go, CGO-free)
|
||
- **Location**: `~/.orca/orca.db` (user-mode) or `/var/lib/orca/orca.db` (system-mode)
|
||
- **Schema (v0.1)**: `nodes`, `jobs`, `tasks`, `audit_log` tables
|
||
- **Schema (v0.2 P01)**: NEW `certs` table
|
||
```sql
|
||
CREATE TABLE certs (
|
||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||
kind TEXT NOT NULL, -- 'ca' | 'server'
|
||
node_id TEXT, -- NULL for CA
|
||
serial_hex TEXT NOT NULL, -- x509.SerialNumber.Hex()
|
||
subject_cn TEXT NOT NULL,
|
||
issuer_cn TEXT NOT NULL,
|
||
not_before INTEGER NOT NULL, -- unix seconds
|
||
not_after INTEGER NOT NULL, -- unix seconds
|
||
fingerprint TEXT NOT NULL, -- sha256 of DER, hex
|
||
source_path TEXT NOT NULL, -- on-disk PEM path
|
||
created_at INTEGER NOT NULL
|
||
);
|
||
CREATE INDEX idx_certs_node_kind ON certs(node_id, kind);
|
||
CREATE INDEX idx_certs_not_after ON certs(not_after);
|
||
```
|
||
- **Migrations**: Embedded SQL files, applied on startup
|
||
- **Migration 0004** is added in P01 with the schema above
|
||
|
||
## Data Model
|
||
|
||
### Node (v0.1, extended in v0.2 P02)
|
||
```go
|
||
type Node struct {
|
||
ID string
|
||
Name string
|
||
Address string
|
||
State string
|
||
JoinedAt time.Time
|
||
LastSeen time.Time
|
||
Metadata map[string]string
|
||
// v0.2 P02 — capacity for bin-packing
|
||
Capacity NodeCapacity
|
||
}
|
||
|
||
type NodeCapacity struct {
|
||
CPUMillicores int // total, e.g. 4000 = 4 cores
|
||
MemoryBytes int64 // total RAM
|
||
CPUUsed int // currently allocated
|
||
MemoryUsed int64 // currently allocated
|
||
}
|
||
|
||
func (c NodeCapacity) AvailableCPU() int { return c.CPUMillicores - c.CPUUsed }
|
||
func (c NodeCapacity) AvailableMemory() int64 { return c.MemoryBytes - c.MemoryUsed }
|
||
```
|
||
|
||
### Job (v0.1)
|
||
```go
|
||
type Job struct {
|
||
ID string
|
||
Name string
|
||
Spec string
|
||
Status string
|
||
CreatedAt time.Time
|
||
StartedAt *time.Time
|
||
EndedAt *time.Time
|
||
}
|
||
```
|
||
|
||
### Task (v0.1)
|
||
```go
|
||
type Task struct {
|
||
ID string
|
||
JobID string
|
||
Command string
|
||
Args []string
|
||
Env []string
|
||
PID int
|
||
ExitCode int
|
||
Status string
|
||
CreatedAt time.Time
|
||
StartedAt *time.Time
|
||
EndedAt *time.Time
|
||
}
|
||
```
|
||
|
||
### Certificate (NEW, v0.2 P01)
|
||
```go
|
||
type Cert struct {
|
||
Kind CertKind // CertCA | CertServer
|
||
NodeID string // empty for CA
|
||
SerialHex string
|
||
SubjectCN string
|
||
IssuerCN string
|
||
NotBefore time.Time
|
||
NotAfter time.Time
|
||
Fingerprint string // sha256 of DER (hex)
|
||
SourcePath string // PEM path on disk
|
||
CreatedAt time.Time
|
||
}
|
||
```
|
||
|
||
## v0.2 Component Graph (ASCII)
|
||
|
||
```
|
||
┌─────────────────┐
|
||
│ Operator Host │
|
||
│ (orca CLI) │
|
||
└────────┬────────┘
|
||
│ 1. cert join --ca-fingerprint <sha>
|
||
▼
|
||
┌──────────────────────────────────────────────────────────────────────────────┐
|
||
│ NODE A (Bootstrap / CA holder) │
|
||
│ │
|
||
│ ┌──────────────┐ CSR ┌────────────────────┐ PEM sign ┌────────┐ │
|
||
│ │ orca cert │──────────▶│ internal/security │─────────────▶│ CA │ │
|
||
│ │ {init,join} │ │ .SignServerCert() │ │ key │ │
|
||
│ └──────────────┘ └────────────────────┘ │ 0600 │ │
|
||
│ ┌──└────────┘ │
|
||
│ ┌─────────────────┐ │ │
|
||
│ │ internal/daemon │ ◀──tls.Config── internal/security │ │
|
||
│ │ (http.Server) │ .ServerTLSConfig() │ │
|
||
│ │ │ │ │
|
||
│ │ /v1/jobs/* │ │ │
|
||
│ │ /v1/nodes/* │ │ │
|
||
│ │ /orca.v1.* │ │ │
|
||
│ └────────┬────────┘ │ │
|
||
│ │ Submit(job) (bin-pack) │ │
|
||
│ ▼ │ │
|
||
│ ┌─────────────────┐ │ │
|
||
│ │ internal/engine │ │ │
|
||
│ │ .scheduler │──── if local fits → executor │ │
|
||
│ │ .dispatcher │──── else → Submit(peer, job) ───────────┼──┐ │
|
||
│ └─────────────────┘ │ │ │
|
||
│ │ │ mTLS │
|
||
└──────────────────────────────────────────────────────────────┼──┼───────────┘
|
||
│ │
|
||
ORCA NODE NETWORK │ │
|
||
│ │
|
||
┌──────────────────────────────────────────────────────────────┼──┼───────────┐
|
||
│ NODE B (Peer) │ │ │
|
||
│ │ │ │
|
||
│ ┌─────────────────┐ ◀── TLS 1.3 handshake ─────────────────┘ │ │
|
||
│ │ internal/daemon │ │ │
|
||
│ │ (http.Server) │ POST /orca.v1.Dispatch/Submit │ │
|
||
│ │ │──── 200 + jobID │ │
|
||
│ └────────┬────────┘ │ │
|
||
│ │ │ │
|
||
│ ▼ │ │
|
||
│ ┌─────────────────┐ │ │
|
||
│ │ internal/engine │ │ │
|
||
│ │ .scheduler (FIFO) │ │
|
||
│ │ .executor │ │
|
||
│ └─────────────────┘ │ │
|
||
└──────────────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
## v0.2 Flows
|
||
|
||
### Flow 1: Cert Issuance (CA-init → CSR → sign → install) — P01
|
||
|
||
```
|
||
Operator (Node A) Operator (Node B)
|
||
───────────────── ─────────────────
|
||
orca cert init
|
||
↳ InitCA("orca-ca")
|
||
↳ write ca.crt (0644), ca.key (0600)
|
||
↳ record in certs table (kind='ca')
|
||
orca cert join --ca-fingerprint <sha>
|
||
↳ operator copies ca.crt → Node B
|
||
↳ verifies fingerprint matches local
|
||
--ca-fingerprint arg
|
||
↳ IssueServerCert("node-b", SANs)
|
||
↳ generate 2048-bit RSA key
|
||
↳ build CSR with SANs
|
||
↳ read ca.crt + ca.key
|
||
↳ sign CSR (90d validity)
|
||
↳ write server.crt (0644),
|
||
server.key (0600)
|
||
↳ record in certs table
|
||
(kind='server', node_id='node-b')
|
||
```
|
||
|
||
### Flow 2: mTLS Handshake at `node join` — P01
|
||
|
||
```
|
||
Node B (joiner) Node A (CA holder)
|
||
──────────────── ─────────────────
|
||
orca node join --name node-b
|
||
--ca-fingerprint <sha>
|
||
--peer node-a:8443
|
||
↳ load ca.crt → verify sha256 == --ca-fingerprint
|
||
↳ load server.crt + server.key
|
||
↳ tls.Config{MinVersion: TLS1.3, ...}
|
||
↳ ClientHello (SNI=node-a)
|
||
◀── ServerHello (TLS 1.3)
|
||
◀── Certificate (Node A's cert)
|
||
↳ verify Node A's cert chains to ca.crt ◀── CertificateRequest
|
||
↳ send Certificate (Node B's cert) ◀── Finished
|
||
↳ Finished
|
||
↳ GET /healthz (over mTLS) — sanity check
|
||
↳ POST /v1/nodes (over mTLS) — register
|
||
↳ insert into nodes table
|
||
↳ audit log
|
||
↳ 200 OK
|
||
↳ record node_a in peers table
|
||
↳ audit log
|
||
```
|
||
|
||
### Flow 3: Job Dispatch (CLI → dispatcher → peer) — P02
|
||
|
||
```
|
||
User (Node A CLI) Node A (scheduler) Node B (peer)
|
||
────────────────── ─────────────────── ─────────────
|
||
orca job run spec.hcl
|
||
↳ parse HCL
|
||
↳ POST /v1/jobs (mTLS, local)
|
||
↳ scheduler.Submit(spec)
|
||
↳ bin-pack: spec.cpu + spec.mem
|
||
↳ local node A has 2000mc + 4GiB free → fit!
|
||
↳ executor.Run(spec)
|
||
↳ 202 Accepted + jobID
|
||
↳ returns jobID
|
||
```
|
||
|
||
```
|
||
User (Node A CLI) Node A (scheduler) Node B (peer)
|
||
────────────────── ─────────────────── ─────────────
|
||
↳ scheduler.Submit(spec)
|
||
↳ bin-pack: spec.cpu + spec.mem
|
||
↳ local node A has 0 free → NO FIT
|
||
↳ dispatcher.Submit(peer="node-b", spec)
|
||
↳ load transport.Client("node-b")
|
||
↳ POST /orca.v1.Dispatch/Submit
|
||
↳ mTLS handshake
|
||
↳ authenticate cert
|
||
↳ scheduler.Submit(spec)
|
||
↳ executor.Run(spec)
|
||
↳ 200 OK + jobID
|
||
↳ 200 OK + jobID
|
||
↳ return jobID to local caller
|
||
↳ returns jobID
|
||
```
|
||
|
||
### Flow 4: iter.Seq Streaming (`--watch`) — P04
|
||
|
||
```
|
||
User orca job list --watch
|
||
──── ─────────────────────
|
||
ctx, cancel := signal.NotifyContext(ctx, os.Interrupt)
|
||
defer cancel()
|
||
seq := store.Jobs().Watch(ctx)
|
||
for job := range seq {
|
||
print(job) // human or --json
|
||
}
|
||
// ctrl-c → ctx.Done() → seq stops yielding
|
||
```
|
||
|
||
Internally `store.Jobs().Watch(ctx) iter.Seq[Job]` polls the
|
||
`jobs` table on a 1s ticker (or subscribes to an in-process
|
||
notifier channel) and yields the current snapshot of each job
|
||
until `ctx.Done()`. The store repo implements `iter.Seq[Job]`
|
||
as a function that takes a `yield func(Job) bool` callback.
|
||
|
||
## Security Architecture
|
||
|
||
> **⚠️ DEPRECATED in v0.9**: This section describes the v0.8 internal-CA architecture, superseded by the v0.9 re-architecture (step-ca, D-101/REQ-076). See the "v0.9 Architecture" section at the bottom of this file and `.ciagent/PRD_v0.9.md`.
|
||
|
||
### Authentication
|
||
- **v0.1**: mTLS for all API endpoints (self-signed CA)
|
||
- **v0.2 P01**: Internal CA with CSR join (see Flow 1 + 2)
|
||
- **v0.2+**: Token-based auth deferred to v0.3+
|
||
|
||
### Cert Rotation
|
||
- Server certs: 90-day validity, rotate at 60 days (30d before expiry)
|
||
- CA cert: 10-year validity, manual rotation
|
||
- Hot-swap: `tls.Config.GetCertificate` callback re-reads the
|
||
`server.crt`/`server.key` files on each handshake so `orca cert renew`
|
||
takes effect without a daemon restart.
|
||
|
||
### Audit Logging
|
||
- All state-changing operations emit structured log records
|
||
- Fields: `timestamp`, `actor`, `action`, `resource`, `result`, `error`
|
||
- Stored in SQLite `audit_log` table and stderr (JSON)
|
||
- **v0.2 P01 additions**: `cert.issued`, `cert.renewed`, `cert.joined`,
|
||
`node.handshake_ok`, `node.handshake_failed`
|
||
|
||
### Input Validation
|
||
- All CLI inputs validated via Cobra's `Args`/`ValidArgs` functions
|
||
- All API inputs validated at handler boundary
|
||
- HCL/YAML specs parsed with strict schemas
|
||
|
||
## Key Architectural Decisions (v0.1 + v0.2)
|
||
|
||
> **⚠️ DEPRECATED in v0.9**: AD-007 (HCL canonical for jobspecs) below is superseded by R-013/R-014 (Markdown with YAML frontmatter canonical; HCL legacy). See the "v0.9 Architecture" section at the bottom of this file and `.ciagent/PRD_v0.9.md`.
|
||
|
||
| ID | Decision | Rationale |
|
||
|----|----------|-----------|
|
||
| AD-001 | Single binary with subcommands | Simpler distribution, aligns with simplicity pillar |
|
||
| AD-002 | modernc/sqlite (CGO-free) | Cross-compile friendly, no CGO toolchain needed |
|
||
| AD-003 | net/http (no framework) | Stdlib suffices; avoids external router dependency |
|
||
| AD-004 | os/exec with WaitDelay (Go 1.25+) | Clean process termination, native to Go |
|
||
| AD-005 | Cobra for CLI | Industry standard, familiar to operators |
|
||
| AD-006 | slog for logging | Native to Go 1.21+, no external dependency |
|
||
| AD-007 | HCL for job specs | Familiar to Nomad/HashiCorp users |
|
||
| AD-008 | Single-node scheduling (v0.1) | Multi-node scheduling deferred to v0.2+ |
|
||
| AD-009 | Internal CA, no external PKI (v0.2) | Self-contained, no operational PKI requirement |
|
||
| AD-010 | Roll-our-own CA in `crypto/x509` (v0.2) | step-ca/cfssl/vault-pki too heavyweight for Orca's footprint |
|
||
| AD-011 | Operator-mediated CA cert distribution (v0.2) | No secret distribution over the wire; matches offline-first |
|
||
| AD-012 | TLS 1.3 only, AEAD allowlist (v0.2) | Modern crypto only; no downgrade risk |
|
||
| AD-013 | Eager mTLS at `node join` (v0.2) | Fail fast; don't defer handshake to first request |
|
||
| AD-014 | `orca.v1.Dispatch` via stdlib h2c (v0.2) | ConnectRPC not in go.mod; stdlib suffices for a single-RPC service |
|
||
| AD-015 | Best-fit bin-packing (v0.2) | Simple, deterministic, optimal for small fleets |
|
||
| AD-016 | iter.Seq for streaming lists (v0.2) | Go 1.25+ native, context-aware, pull semantics |
|
||
|
||
## Anti-Patterns (Explicitly Avoided)
|
||
|
||
- No controller/agent split (single binary)
|
||
- No CRDs / custom resource definitions
|
||
- No web UI (CLI-only)
|
||
- No service mesh
|
||
- No container runtime integration
|
||
- No multi-tenancy
|
||
- No cloud provider integrations
|
||
- No auto-scaling
|
||
- No admission controllers
|
||
- No complex scheduling algorithms (best-fit only)
|
||
- No gRPC framework dependency (stdlib net/http with h2c, Go 1.25+ native)
|
||
- No external PKI / no cert transparency logs (offline-first)
|
||
|
||
## Dependency Map (minimal — v0.2 adds zero direct deps)
|
||
|
||
```
|
||
github.com/spf13/cobra # CLI framework
|
||
github.com/hashicorp/hcl/v2 # HCL parser
|
||
modernc.org/sqlite # SQLite (pure Go)
|
||
github.com/google/uuid # UUID generation
|
||
```
|
||
|
||
Total: ~4 direct dependencies. No web framework, no ORM, no RPC framework,
|
||
no PKI library. mTLS via `crypto/tls` and `crypto/x509` (stdlib).
|
||
|
||
> **ConnectRPC note**: `.ciagent/config.json` lists `connectrpc` in
|
||
> `frameworks`, but the actual `go.mod` does not depend on
|
||
> `connectrpc.com/connect`. v0.2 falls back to plain `net/http` with
|
||
> HTTP/2 cleartext (h2c) for `orca.v1.Dispatch`. The protocol is a
|
||
> simple JSON-over-HTTP POST: client sends
|
||
> `{"spec": "..."}` to `/orca.v1.Dispatch/Submit`; server replies
|
||
> `{"job_id": "..."}`. This keeps the zero-new-dep promise and the
|
||
> codebase coherent with the rest of the daemon's `http.ServeMux`.
|
||
|
||
## Deployment Model (v0.2)
|
||
|
||
```
|
||
Operator Machine Node A (CA holder) Node B (Peer)
|
||
──────────────── ───────────────── ─────────────
|
||
orca CLI orca daemon orca daemon
|
||
│ │ ▲ │ ▲
|
||
│ mTLS handshake │ │ mTLS │ │
|
||
│ at `node join` ────────────┼──┘ │ │
|
||
│ │ │ │
|
||
│ submit job ───POST────────▶│ POST (cross-node) ──────────▶│
|
||
│ │ mTLS only │ │
|
||
│ │ │ │
|
||
│ ◀───────jobID──────────────│ ◀─────jobID (200 OK)─────────│
|
||
│ │ │
|
||
│ orca job list --watch │ │
|
||
│ (iter.Seq stream) ◀────────│── polls local + stream events│
|
||
```
|
||
|
||
For v0.2, one node must be the CA holder (`orca cert init` was run
|
||
on it). The CA holder's `ca.crt` is copied to each peer manually by
|
||
the operator; peers do not auto-fetch it.
|
||
|
||
## v0.6 Architecture Addendum — Node Bootstrap & Proxmox
|
||
|
||
### `orca init` Full Bootstrap (REQ-047, REQ-048, REQ-049)
|
||
|
||
`orca init` transforms from a bare `mkdir` into a full single-node
|
||
cluster bootstrap. The sequence (idempotent per D-036):
|
||
|
||
```
|
||
orca init
|
||
1. MkdirAll(certpaths.Dir(), 0o755) # namespace dir
|
||
2. store.Open(certpaths.DBPath()) # runs migrations 0001..0006
|
||
3. security.CAInit(dir, "orca-internal-ca") # idempotent fast-path
|
||
4. if !exists(server.crt):
|
||
GenerateCSR("localhost", ["localhost","127.0.0.1"])
|
||
ca.SignCSR(csr) → WriteCert + WriteKey # server cert (skip if present)
|
||
5. os := detectOS() # /etc/os-release ID=
|
||
6. node := Node{kind:"localhost", os:os, name:"localhost", addr:"localhost:8443"}
|
||
if GetByName("localhost") exists:
|
||
UpdateLastSeenAndOS(id, os) # refresh, keep id/joined_at
|
||
else:
|
||
NodeRepo.Insert(node) # first-run insert
|
||
7. print summary (CA fp, server cert fp, os, node id)
|
||
```
|
||
|
||
After `orca init`, `orca doctor` MUST pass with zero FAILs.
|
||
|
||
### Node Schema Extension (REQ-049)
|
||
|
||
Migration 0006 adds two nullable columns to `nodes`:
|
||
|
||
```sql
|
||
ALTER TABLE nodes ADD COLUMN kind TEXT; -- localhost | linux | proxmox
|
||
ALTER TABLE nodes ADD COLUMN os TEXT; -- ubuntu | debian | alpine | pve | linux
|
||
```
|
||
|
||
Existing rows get SQL NULL → mapped to `""` in Go (`sql.NullString`).
|
||
`Node` struct gains `Kind string` + `OS string` fields (JSON tags
|
||
`kind,omitempty` / `os,omitempty`). `NodeRepo` extends all
|
||
INSERT/SELECT/scanNode calls; adds `GetByName(ctx, name)` and
|
||
`UpdateLastSeenAndOS(ctx, id, os)` helpers.
|
||
|
||
### Proxmox SSH Bootstrap (REQ-050, REQ-051)
|
||
|
||
```
|
||
orca node join --type proxmox --host <addr> --user root --password <pw>
|
||
│ password from --password or $ORCA_PROXMOX_PASSWORD (never persisted, D-031)
|
||
▼
|
||
internal/proxmox.BootstrapProxmox(ctx, opts)
|
||
1. GenerateOrLoadSSHKey(certpaths.Dir()) # Ed25519, ~/.orca/orca_ssh_key{,.pub}
|
||
2. SSH dial (password auth, knownhosts.New TOFU) # capture host key on first connect
|
||
3. Deploy pubkey → ~orca/.ssh/authorized_keys # via session heredoc (no SFTP dep)
|
||
4. useradd -m orca # create Linux system user (config-overridable name)
|
||
5. pveum role add OrcaOperator --privs "VM.Audit Datastore.AllocateSpace SDN.Use"
|
||
(idempotent: probe pveum role list first)
|
||
6. pveum user add orca@pam -comment "Orca automation user"
|
||
(idempotent: probe pveum user list first)
|
||
7. pveum acl modify / -user orca@pam -role OrcaOperator
|
||
(idempotent: modify creates or updates)
|
||
8. Write /etc/sudoers.d/orca (mode 0440):
|
||
orca ALL=(root) NOPASSWD: NOEXEC: /usr/bin/pct, /usr/bin/qm
|
||
orca ALL=(root) NOPASSWD: /usr/bin/apt-get, /usr/bin/dpkg
|
||
9. visudo -cf /etc/sudoers.d/orca # validate; abort on error
|
||
10. NodeRepo.Insert(Node{kind:"proxmox", os:"pve", name:host, addr:host})
|
||
11. Audit log: proxmox.bootstrap_ok (host, user, role, fp)
|
||
```
|
||
|
||
**`pvesh` excluded from sudoers** — `pvesh` can trigger the API
|
||
`/nodes/{node}/execute` endpoint which spawns shell commands
|
||
server-side, bypassing sudo's `NOEXEC` tag. API access is via the
|
||
`OrcaOperator` PVE role + `orca@pam` user (PVE RBAC), not sudo'd `pvesh`.
|
||
|
||
### Doctor Extensions (REQ-052)
|
||
|
||
- **`doctor os`**: re-runs `detectOS()` from `/etc/os-release`, compares
|
||
to the stored localhost node's `os` field. Drift = WARN (OS upgraded
|
||
since init? re-run `orca init` to refresh). Match = PASS.
|
||
- **`doctor proxmox`**: iterates `kind=proxmox` nodes, SSH-probes each
|
||
with `pveversion` (3s timeout per peer, clones `doctor.Network()`
|
||
pattern). PASS = reachable + pveversion exits 0. WARN = zero proxmox
|
||
nodes (single-node cluster is legitimate). FAIL = any node
|
||
unreachable or pveversion fails.
|
||
|
||
### SSH Key Handling (D-037)
|
||
|
||
- **Location**: `~/.orca/orca_ssh_key` (0600) + `~/.orca/orca_ssh_key.pub` (0644)
|
||
- **Algorithm**: Ed25519 (smaller, faster, more secure than RSA for SSH)
|
||
- **Generation**: lazy — on first `orca node join --type proxmox`, NOT at `orca init` (localhost doesn't need SSH)
|
||
- **Format**: PKCS8 PEM (consistent with `ca.key`/`server.key`; `ssh.ParsePrivateKey` accepts it)
|
||
- **TOFU host keys**: `~/.orca/known_hosts` (OpenSSH format via `knownhosts.New`)
|
||
|
||
### Dependency Map (v0.6 addition)
|
||
|
||
```
|
||
golang.org/x/crypto v0.54.0 # SSH (ssh + ssh/knownhosts + ed25519)
|
||
└─ golang.org/x/sys v0.47.0 # indirect (bumped from v0.42.0)
|
||
└─ golang.org/x/term v0.45.0 # indirect (pulled by ssh for PTY)
|
||
```
|
||
|
||
Total direct deps: 5 (was 4). One new direct dep (`x/crypto`). Matches
|
||
D-030 minimal-deps rationale. No SFTP module (file upload via session
|
||
heredoc).
|
||
|
||
### v0.6 Architectural Decisions (AD-017..AD-021)
|
||
|
||
| ID | Decision | Rationale |
|
||
|----|----------|-----------|
|
||
| AD-017 | `orca init` = full bootstrap (CA + cert + db + localhost node) | Single command produces a working cluster; `orca doctor` passes post-init. Idempotent (D-036). |
|
||
| AD-018 | Proxmox join via SSH (golang.org/x/crypto/ssh), not PVE REST API | SSH is the universal Proxmox management entry point; REST API would require API token bootstrap (chicken-and-egg). One new direct dep (D-030). |
|
||
| AD-019 | `orca@pam` realm (not `orca@pve`) | SSH creates a Linux system user; PAM realm maps it to PVE RBAC without a separate PVE password. `@pve` requires interactive password prompt over non-PTY SSH (hangs). |
|
||
| AD-020 | Exclude `pvesh` from sudoers; NOEXEC on `pct`/`qm` | `pvesh` can trigger API execute endpoint bypassing NOEXEC. `pct`/`qm` are Perl scripts via dynamically-linked perl → NOEXEC effective. `apt-get`/`dpkg` need exec for maintainer scripts → no NOEXEC. |
|
||
| AD-021 | TOFU host-key via `knownhosts.New` | Avoids deprecated `ssh.InsecureIgnoreHostKey`. Capture-on-first-connect, verify-on-subsequent. Fail closed on mismatch (operator runs key-reset). |
|
||
|
||
---
|
||
|
||
# v0.9 Architecture (Supersedes v0.8)
|
||
|
||
> **⚠️ v0.9 DIRECTION CHANGE**: This section supersedes the v0.1–v0.8
|
||
> architecture described above. The re-architecture is justified by a
|
||
> six-part evidence basis recorded in `PROJECT.md` (Supersession Table).
|
||
> The v0.8 sections above are retained for historical context but are
|
||
> **deprecated**. The 16 load-bearing rules (R-001…R-016) in
|
||
> `PRD_v0.9.md` are now the canonical invariants.
|
||
|
||
## Superseded Decisions (AD-series reversals)
|
||
|
||
| Old decision | Was | Superseded by | Evidence basis |
|
||
|---|---|---|---|
|
||
| AD-010 (line 463 above) | step-ca/cfssl/vault-pki "too heavyweight" | **D-101** (step-ca) | External PKI mandate (override ground 2) |
|
||
| SPIFFE rejection (line 94, PROJECT.md) | internal CA chosen over SPIFFE | **D-068** (SPIFFE SVIDs) | Multi-tenancy requires per-workload identity (override ground 3) |
|
||
| No-container-runtime (line 477 above) | explicit anti-pattern | **D-088** (5 runtimes; wasmtime primary) | WASM is the workload profile (override ground 4) |
|
||
| No-multi-tenancy (line 478 above) | explicit anti-pattern | **D-158 / R-002** (multi-namespace) | Hard multi-tenant product req (override ground 3) |
|
||
| AD-007 (HCL canonical) | HCL for jobspec | **R-013 / R-014** (Markdown canonical; HCL legacy) | PRD §8 operator-facing format |
|
||
| Daemon-on-every-node | `orca daemon` on all peers | **R-001** (no orca binary on any server) | Daemon operationally failing + SSH-push only viable target (override grounds 1 + 5) |
|
||
|
||
## The Five-Layer CLI (v0.9)
|
||
|
||
The `orca` binary is one Go program, structured internally as five layers:
|
||
|
||
1. **CLI subcommand tree** (cobra) — `internal/cli/`
|
||
2. **Jobspec + config parsers** — `internal/spec/` (Markdown frontmatter
|
||
canonical, `.md`/`.yaml`/`.hcl` dispatcher per R-013/R-014)
|
||
3. **Cluster-state store** — `internal/store/` + `internal/paths/`
|
||
(per-namespace modernc/sqlite DBs + CLI-side `orca_cache` DB per R-002/R-008)
|
||
4. **Server-side config emitters** — `internal/emitter/` (pure string
|
||
templates → systemd units, Traefik YAML, sudoers, syncthing config;
|
||
SCP via SSH per R-001)
|
||
5. **Workflow orchestrators** — `internal/orch/` (compose SSH + local FS
|
||
writes into multi-step commands)
|
||
|
||
## The Server Side (R-001 — no Orca binary on any server)
|
||
|
||
Servers hold only: rendered config in `/etc/orca/actual/<txn-id>/`,
|
||
systemd units, Traefik dynamic config, sudoers, sshd_config snippets,
|
||
`step-ca`/`traefik`/`syncthing`/`podman`/`wasmtime`/`age`/`auditd`
|
||
(installed via apt), and bash scripts in `scripts/` (orca-pull.sh,
|
||
orca-drift.sh, orca-collect.sh, orca-aggregate.sh, orca-apply-render.sh,
|
||
orca-verify-render.sh, orca-rollback-render.sh, orca-cleanup-credentials.sh).
|
||
Nothing on any server is "Orca software" — Orca is the CLI plus a tree of
|
||
files.
|
||
|
||
## Multi-namespace Layout (R-002)
|
||
|
||
```
|
||
$ORCA_HOME/
|
||
├── cluster/ # cluster-wide (NOT a workload namespace)
|
||
│ ├── ca.crt, ca.key # step-ca root (R-006, D-101)
|
||
│ ├── master.key # AES-256-GCM root (R-011, mode 0600)
|
||
│ ├── config.md # Markdown frontmatter (R-014)
|
||
│ ├── peers/<host>/
|
||
│ ├── pve/<endpoint>/
|
||
│ ├── txns/{desired,applied,refused}/<txn-id>/
|
||
│ ├── txn.sqlite
|
||
│ └── state/
|
||
├── _defaults/ # implicit root namespace (always exists)
|
||
│ ├── ns.md
|
||
│ ├── .env, .env.secrets
|
||
│ ├── db/orca.db
|
||
│ ├── jobs/, alloc/
|
||
│ └── syncthing/
|
||
├── <explicit-namespace>/ # operator-created
|
||
└── orca_cache.db # CLI-side cache (R-008)
|
||
```
|
||
|
||
## Execution gates (from GRILL_v0.9.md)
|
||
|
||
The 19 binding conditions (C-01..C-19) and 10 phase challenges
|
||
(PC-01..PC-10) gate specific phases. See `GRILL_v0.9.md` for the full
|
||
list. Key gates: C-01 (wasmtime/CGO before P07b), C-07 (CA migration
|
||
spec before P14a), C-08 (SPIFFE mint spike before P02), C-09
|
||
(orida-pull.sh failure contract before P10), C-19 (threat model before
|
||
P15.5).
|