orca/.ciagent/ARCHITECTURE.md

Architecture: Orca

System Overview

Orca is a single-binary, offline-first orchestration engine. The system consists of three logical layers (CLI, Daemon, Engine) compiled into one orca binary and selected via subcommands. v0.2 adds a cross-node transport layer (mTLS) and a dispatcher for multi-node job execution.

┌─────────────────────────────────────────────────────────────────────────────┐
│                       orca (single binary, v0.2)                             │
├─────────────────────────────────────────────────────────────────────────────┤
│  CLI Layer (Cobra)                                                          │
│  ├── orca version                                                           │
│  ├── orca init                          # local node bootstrap               │
│  ├── orca cert {init,join,renew,show}   # NEW (P01)                         │
│  ├── orca status                                                           │
│  ├── orca node {join,leave,list}        # join = mTLS handshake (P01)      │
│  │   └── orca node list --watch         # NEW iter.Seq (P04)                │
│  ├── orca job {run,list,stop,logs}                                         │
│  │   └── orca job list --watch          # NEW iter.Seq (P04)                │
│  ├── orca doctor                       # NEW (P01) — diagnostics            │
│  │   ├── orca doctor cert                                                   │
│  │   ├── orca doctor network                                                │
│  │   └── orca doctor db                                                     │
│  └── orca daemon                                                           │
├─────────────────────────────────────────────────────────────────────────────┤
│  Daemon Layer (net/http over h2c, mTLS in P01)                              │
│  ├── /healthz                          (liveness)                           │
│  ├── /readyz                           (readiness)                          │
│  ├── /v1/jobs/*                        (job control API)                    │
│  ├── /v1/nodes/*                       (node registry API)                  │
│  ├── /v1/tasks/*                       (task lifecycle API)                 │
│  ├── /orca.v1.Dispatch/...             # NEW (P02) — cross-node dispatch    │
│  └── /orca.v1.Register/...             # NEW (P02) — peer join ack           │
├─────────────────────────────────────────────────────────────────────────────┤
│  Transport Layer (NEW — internal/transport)                                  │
│  ├── mTLS client                       (dialer pool per peer)               │
│  ├── mTLS server config                (TLS 1.3 only, AEAD allowlist)       │
│  ├── Retry+backoff                     (exponential, jittered, capped)       │
│  └── Graceful disconnect               (ctx-aware Conn.Close)               │
├─────────────────────────────────────────────────────────────────────────────┤
│  Core Engine                                                                │
│  ├── Node Registry                     (in-memory + SQLite persistence)     │
│  ├── Task Executor                     (os/exec with WaitDelay, Go 1.25+)   │
│  ├── Job Scheduler                     (single-node FIFO; bin-pack P02)     │
│  ├── Dispatcher                        # NEW internal/engine/dispatcher.go │
│  │   ├── Local decision                (does this job fit on this node?)    │
│  │   ├── Remote dispatch               (POST to peer via transport)         │
│  │   └── Streaming callback            (iter.Seq[DispatchResult] for CLI)   │
│  ├── Security Manager                  # NEW internal/security (P01)        │
│  │   ├── CA lifecycle                  (init, fingerprint, sign CSR)        │
│  │   ├── Server cert lifecycle         (issue, renew, rotate)                │
│  │   ├── mTLS config builder                                              │
│  │   └── Cert store                    (filesystem + SQLite metadata)       │
│  └── Audit Logger                      (log/slog JSON handler)              │
├─────────────────────────────────────────────────────────────────────────────┤
│  State Store (modernc/sqlite, CGO-free)                                     │
│  ~/.orca/orca.db                                                            │
│  ├── nodes, jobs, tasks, audit_log    (v0.1)                                │
│  └── certs                             # NEW (P01) — CA + server certs      │
└─────────────────────────────────────────────────────────────────────────────┘

Component Details

1. CLI Layer (cmd/orca, internal/cli)

• Framework: Cobra (industry standard, familiar to operators)
• v0.1 subcommands: version, init, status, node, job, daemon
• v0.2 additions (P01): orca cert {init,join,renew,show}
• v0.2 additions (P04): --watch flag on orca job list and orca node list
• v0.2 additions (P01): orca doctor subcommand (see §5 below)
• Output: Human-readable by default; --json flag for machine consumption
• Discovery: All subcommands self-document via Cobra's auto-generated help
• Watch semantics (P04): --watch consumes iter.Seq[Job|Node], exits on ctrl-c (via signal.NotifyContext), refreshes on internal change events.

2. Daemon Layer (internal/daemon)

⚠️ DEPRECATED in v0.9: This section describes the v0.8 architecture, superseded by the v0.9 re-architecture. See the "v0.9 Architecture" section at the bottom of this file and .ciagent/PRD_v0.9.md.

• Server: net/http with http.ServeMux (no external router)
• TLS (P01): crypto/tls with MinVersion=tls.VersionTLS13 and AEAD cipher allowlist (TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_AES_128_GCM_SHA256)
• Ports: Configurable (default :8443 for API+mTLS, :8080 for health)
• Graceful Shutdown: signal.NotifyContext with SIGINT/SIGTERM
• v0.2 endpoints (P02):
  • POST /orca.v1.Dispatch/Submit — receive cross-node job submission
  • POST /orca.v1.Dispatch/Status — query dispatched job status
  • POST /orca.v1.Register/Hello — peer join ack (used during orca node join)

3. Transport Layer (internal/transport, NEW in P01/P02)

⚠️ DEPRECATED in v0.9: This section describes the v0.8 architecture, superseded by the v0.9 re-architecture. See the "v0.9 Architecture" section at the bottom of this file and .ciagent/PRD_v0.9.md.

• Client: http.Client with http.Transport.TLSClientConfig populated from internal/security.NewClientTLSConfig
• Server: http.Server.TLSConfig populated from internal/security.NewServerTLSConfig
• Retry policy: exponential backoff with jitter (start 100ms, x2, cap 5s, max 5 attempts); only idempotent verbs (GET, HEAD, OPTIONS) are retried automatically; POST retries require an explicit X-Orca-Idempotency-Key header
• Conn lifecycle: context.Context-aware dials and reads; graceful Close on ctx.Done()
• Peer dial pool: small sync.Map of peerID → *http.Client to reuse TLS handshakes (TCP keep-alive) within a session

4. Core Engine (internal/engine)

⚠️ DEPRECATED in v0.9: This section describes the v0.8 architecture, superseded by the v0.9 re-architecture. The Dispatcher and PeerRegistry peer-dispatch path is replaced by a CLI-side scheduler + SSH-push (R-001). See the "v0.9 Architecture" section at the bottom of this file and .ciagent/PRD_v0.9.md.

• Node Registry: In-memory map of node IDs → metadata, persisted to SQLite (CPU/memory capacity, available slots, last-seen)
• Task Executor: os/exec.CommandContext with WaitDelay (Go 1.25+) for clean process termination
• Job Scheduler (v0.2 P02):
  • Algorithm: best-fit bin-packing by available_cpu and available_memory
  • Within-node ordering: FIFO queue
  • Cross-node: if local node is full, Dispatcher.Submit(peer, job) is invoked; peers are tried in round-robin order
  • Fallback: if all peers reject, return ErrNoFit and requeue
• Dispatcher (NEW, P02):
  • Submit(peerID, spec) (jobID, error) — blocking call with retry
  • Watch(peerID) iter.Seq[DispatchEvent] — pull-style event stream for the CLI's --watch flag
  • Stateless: every call uses the latest mTLS client config and peer address
• Security Manager (NEW, P01):
  • InitCA(commonName) (*CA, error) — generates a self-signed CA, writes ca.crt (0644) and ca.key (0600) to ~/.orca/
  • Fingerprint(certPath) (sha256hex, error) — used by orca cert join
  • SignServerCert(csr, validity) (*cert, error) — signs a CSR with the CA
  • IssueServerCert(nodeName, dnsNames, ips) (*cert, *key, error) — generates a keypair + CSR + signs it, returns PEM bytes for orca cert join --server
  • ServerTLSConfig() (*tls.Config, error) — loads server.crt/server.key and the CA pool from disk
  • ClientTLSConfig(caPath) (*tls.Config, error) — returns a client config pinned to the supplied CA
  • Rotation policy: server certs valid 90d; CA cert valid 10y. On orca cert renew, IssueServerCert is called and the daemon gracefully reloads the in-process tls.Config via GetCertificate hot-swap (no restart required)
• Audit Logger: slog.NewJSONHandler(os.Stderr, ...) with structured fields

5. Doctor (internal/doctor, NEW in P01)

• Purpose: operator-facing diagnostics; runs read-only checks against the local state and reports PASS/WARN/FAIL.
• Subcommands:
  • orca doctor — runs all checks
  • orca doctor cert — cert/CA health (file modes, expiry windows, SAN presence, fingerprint pinning match — see REQ-026, REQ-033, REQ-034, REQ-036)
  • orca doctor network — peer reachability over mTLS (per-peer handshake sanity, last-seen delta)
  • orca doctor db — SQLite integrity check (PRAGMA integrity_check)
    • migration version
• Output: human-readable by default; --json for machine consumption
• No state changes: doctor is strictly read-only. It can be run while the daemon is down (where possible) or while it's up.
• Initial implementation in P01 (cert checks only); network and db checks land in subsequent phases as their state becomes available.

6. State Store (internal/store)

• Driver: modernc.org/sqlite (pure Go, CGO-free)
• Location: ~/.orca/orca.db (user-mode) or /var/lib/orca/orca.db (system-mode)
• Schema (v0.1): nodes, jobs, tasks, audit_log tables
• Schema (v0.2 P01): NEW certs table

  CREATE TABLE certs (
      id            INTEGER PRIMARY KEY AUTOINCREMENT,
      kind          TEXT    NOT NULL,           -- 'ca' | 'server'
      node_id       TEXT,                       -- NULL for CA
      serial_hex    TEXT    NOT NULL,           -- x509.SerialNumber.Hex()
      subject_cn    TEXT    NOT NULL,
      issuer_cn     TEXT    NOT NULL,
      not_before    INTEGER NOT NULL,           -- unix seconds
      not_after     INTEGER NOT NULL,           -- unix seconds
      fingerprint   TEXT    NOT NULL,           -- sha256 of DER, hex
      source_path   TEXT    NOT NULL,           -- on-disk PEM path
      created_at    INTEGER NOT NULL
  );
  CREATE INDEX idx_certs_node_kind ON certs(node_id, kind);
  CREATE INDEX idx_certs_not_after ON certs(not_after);
  

• Migrations: Embedded SQL files, applied on startup
• Migration 0004 is added in P01 with the schema above

Data Model

Node (v0.1, extended in v0.2 P02)

type Node struct {
    ID        string
    Name      string
    Address   string
    State     string
    JoinedAt  time.Time
    LastSeen  time.Time
    Metadata  map[string]string
    // v0.2 P02 — capacity for bin-packing
    Capacity  NodeCapacity
}

type NodeCapacity struct {
    CPUMillicores int   // total, e.g. 4000 = 4 cores
    MemoryBytes   int64 // total RAM
    CPUUsed       int   // currently allocated
    MemoryUsed    int64 // currently allocated
}

func (c NodeCapacity) AvailableCPU() int       { return c.CPUMillicores - c.CPUUsed }
func (c NodeCapacity) AvailableMemory() int64  { return c.MemoryBytes - c.MemoryUsed }

Job (v0.1)

type Job struct {
    ID        string
    Name      string
    Spec      string
    Status    string
    CreatedAt time.Time
    StartedAt *time.Time
    EndedAt   *time.Time
}

Task (v0.1)

type Task struct {
    ID        string
    JobID     string
    Command   string
    Args      []string
    Env       []string
    PID       int
    ExitCode  int
    Status    string
    CreatedAt time.Time
    StartedAt *time.Time
    EndedAt   *time.Time
}

Certificate (NEW, v0.2 P01)

type Cert struct {
    Kind        CertKind   // CertCA | CertServer
    NodeID      string     // empty for CA
    SerialHex   string
    SubjectCN   string
    IssuerCN    string
    NotBefore   time.Time
    NotAfter    time.Time
    Fingerprint string     // sha256 of DER (hex)
    SourcePath  string     // PEM path on disk
    CreatedAt   time.Time
}

v0.2 Component Graph (ASCII)

                                  ┌─────────────────┐
                                  │  Operator Host  │
                                  │  (orca CLI)     │
                                  └────────┬────────┘
                                           │ 1. cert join --ca-fingerprint <sha>
                                           ▼
┌──────────────────────────────────────────────────────────────────────────────┐
│                              NODE A (Bootstrap / CA holder)                  │
│                                                                              │
│  ┌──────────────┐    CSR     ┌────────────────────┐   PEM sign   ┌────────┐  │
│  │ orca cert    │──────────▶│ internal/security   │─────────────▶│  CA    │  │
│  │ {init,join}  │            │ .SignServerCert()  │              │  key   │  │
│  └──────────────┘            └────────────────────┘              │ 0600   │  │
│                                                              ┌──└────────┘  │
│  ┌─────────────────┐                                         │              │
│  │ internal/daemon │ ◀──tls.Config── internal/security        │              │
│  │ (http.Server)   │              .ServerTLSConfig()          │              │
│  │                 │                                         │              │
│  │ /v1/jobs/*      │                                         │              │
│  │ /v1/nodes/*     │                                         │              │
│  │ /orca.v1.*      │                                         │              │
│  └────────┬────────┘                                         │              │
│           │  Submit(job)  (bin-pack)                         │              │
│           ▼                                                  │              │
│  ┌─────────────────┐                                         │              │
│  │ internal/engine │                                         │              │
│  │ .scheduler      │──── if local fits → executor            │              │
│  │ .dispatcher     │──── else → Submit(peer, job) ───────────┼──┐           │
│  └─────────────────┘                                         │  │           │
│                                                              │  │ mTLS      │
└──────────────────────────────────────────────────────────────┼──┼───────────┘
                                                               │  │
                                            ORCA NODE NETWORK │  │
                                                               │  │
┌──────────────────────────────────────────────────────────────┼──┼───────────┐
│                              NODE B (Peer)                   │  │           │
│                                                              │  │           │
│  ┌─────────────────┐  ◀── TLS 1.3 handshake ─────────────────┘  │           │
│  │ internal/daemon │                                              │           │
│  │ (http.Server)   │  POST /orca.v1.Dispatch/Submit              │           │
│  │                 │──── 200 + jobID                              │           │
│  └────────┬────────┘                                              │           │
│           │                                                       │           │
│           ▼                                                       │           │
│  ┌─────────────────┐                                              │           │
│  │ internal/engine │                                              │           │
│  │ .scheduler (FIFO)                                              │           │
│  │ .executor                                                     │           │
│  └─────────────────┘                                              │           │
└──────────────────────────────────────────────────────────────────────────────┘

v0.2 Flows

Flow 1: Cert Issuance (CA-init → CSR → sign → install) — P01

Operator (Node A)                       Operator (Node B)
─────────────────                       ─────────────────
orca cert init
  ↳ InitCA("orca-ca")
  ↳ write ca.crt (0644), ca.key (0600)
  ↳ record in certs table (kind='ca')
                                         orca cert join --ca-fingerprint <sha>
                                           ↳ operator copies ca.crt → Node B
                                           ↳ verifies fingerprint matches local
                                              --ca-fingerprint arg
                                           ↳ IssueServerCert("node-b", SANs)
                                              ↳ generate 2048-bit RSA key
                                              ↳ build CSR with SANs
                                              ↳ read ca.crt + ca.key
                                              ↳ sign CSR (90d validity)
                                           ↳ write server.crt (0644),
                                              server.key (0600)
                                           ↳ record in certs table
                                              (kind='server', node_id='node-b')

Flow 2: mTLS Handshake at node join — P01

Node B (joiner)                                  Node A (CA holder)
────────────────                                  ─────────────────
orca node join --name node-b
  --ca-fingerprint <sha>
  --peer node-a:8443
  ↳ load ca.crt → verify sha256 == --ca-fingerprint
  ↳ load server.crt + server.key
  ↳ tls.Config{MinVersion: TLS1.3, ...}
  ↳ ClientHello (SNI=node-a)
                                                ◀── ServerHello (TLS 1.3)
                                                ◀── Certificate (Node A's cert)
  ↳ verify Node A's cert chains to ca.crt       ◀── CertificateRequest
  ↳ send Certificate (Node B's cert)            ◀── Finished
  ↳ Finished
  ↳ GET /healthz (over mTLS) — sanity check
  ↳ POST /v1/nodes (over mTLS) — register
                                                 ↳ insert into nodes table
                                                 ↳ audit log
  ↳ 200 OK
  ↳ record node_a in peers table
  ↳ audit log

Flow 3: Job Dispatch (CLI → dispatcher → peer) — P02

User (Node A CLI)             Node A (scheduler)            Node B (peer)
──────────────────             ───────────────────            ─────────────
orca job run spec.hcl
  ↳ parse HCL
  ↳ POST /v1/jobs (mTLS, local)
                                ↳ scheduler.Submit(spec)
                                ↳ bin-pack: spec.cpu + spec.mem
                                ↳ local node A has 2000mc + 4GiB free → fit!
                                ↳ executor.Run(spec)
                                ↳ 202 Accepted + jobID
  ↳ returns jobID

User (Node A CLI)             Node A (scheduler)            Node B (peer)
──────────────────             ───────────────────            ─────────────
                                ↳ scheduler.Submit(spec)
                                ↳ bin-pack: spec.cpu + spec.mem
                                ↳ local node A has 0 free    → NO FIT
                                ↳ dispatcher.Submit(peer="node-b", spec)
                                  ↳ load transport.Client("node-b")
                                  ↳ POST /orca.v1.Dispatch/Submit
                                                                ↳ mTLS handshake
                                                                ↳ authenticate cert
                                                                ↳ scheduler.Submit(spec)
                                                                ↳ executor.Run(spec)
                                                                ↳ 200 OK + jobID
                                  ↳ 200 OK + jobID
                                ↳ return jobID to local caller
  ↳ returns jobID

Flow 4: iter.Seq Streaming (--watch) — P04

User                              orca job list --watch
────                              ─────────────────────
                                  ctx, cancel := signal.NotifyContext(ctx, os.Interrupt)
                                  defer cancel()
                                  seq := store.Jobs().Watch(ctx)
                                  for job := range seq {
                                    print(job)  // human or --json
                                  }
                                  // ctrl-c → ctx.Done() → seq stops yielding

Internally store.Jobs().Watch(ctx) iter.Seq[Job] polls the jobs table on a 1s ticker (or subscribes to an in-process notifier channel) and yields the current snapshot of each job until ctx.Done(). The store repo implements iter.Seq[Job] as a function that takes a yield func(Job) bool callback.

Security Architecture

⚠️ DEPRECATED in v0.9: This section describes the v0.8 internal-CA architecture, superseded by the v0.9 re-architecture (step-ca, D-101/REQ-076). See the "v0.9 Architecture" section at the bottom of this file and .ciagent/PRD_v0.9.md.

Authentication

• v0.1: mTLS for all API endpoints (self-signed CA)
• v0.2 P01: Internal CA with CSR join (see Flow 1 + 2)
• v0.2+: Token-based auth deferred to v0.3+

Cert Rotation

• Server certs: 90-day validity, rotate at 60 days (30d before expiry)
• CA cert: 10-year validity, manual rotation
• Hot-swap: tls.Config.GetCertificate callback re-reads the server.crt/server.key files on each handshake so orca cert renew takes effect without a daemon restart.

Audit Logging

• All state-changing operations emit structured log records
• Fields: timestamp, actor, action, resource, result, error
• Stored in SQLite audit_log table and stderr (JSON)
• v0.2 P01 additions: cert.issued, cert.renewed, cert.joined, node.handshake_ok, node.handshake_failed

Input Validation

• All CLI inputs validated via Cobra's Args/ValidArgs functions
• All API inputs validated at handler boundary
• HCL/YAML specs parsed with strict schemas

Key Architectural Decisions (v0.1 + v0.2)

⚠️ DEPRECATED in v0.9: AD-007 (HCL canonical for jobspecs) below is superseded by R-013/R-014 (Markdown with YAML frontmatter canonical; HCL legacy). See the "v0.9 Architecture" section at the bottom of this file and .ciagent/PRD_v0.9.md.

ID	Decision	Rationale
AD-001	Single binary with subcommands	Simpler distribution, aligns with simplicity pillar
AD-002	modernc/sqlite (CGO-free)	Cross-compile friendly, no CGO toolchain needed
AD-003	net/http (no framework)	Stdlib suffices; avoids external router dependency
AD-004	os/exec with WaitDelay (Go 1.25+)	Clean process termination, native to Go
AD-005	Cobra for CLI	Industry standard, familiar to operators
AD-006	slog for logging	Native to Go 1.21+, no external dependency
AD-007	HCL for job specs	Familiar to Nomad/HashiCorp users
AD-008	Single-node scheduling (v0.1)	Multi-node scheduling deferred to v0.2+
AD-009	Internal CA, no external PKI (v0.2)	Self-contained, no operational PKI requirement
AD-010	Roll-our-own CA in crypto/x509 (v0.2)	step-ca/cfssl/vault-pki too heavyweight for Orca's footprint
AD-011	Operator-mediated CA cert distribution (v0.2)	No secret distribution over the wire; matches offline-first
AD-012	TLS 1.3 only, AEAD allowlist (v0.2)	Modern crypto only; no downgrade risk
AD-013	Eager mTLS at node join (v0.2)	Fail fast; don't defer handshake to first request
AD-014	orca.v1.Dispatch via stdlib h2c (v0.2)	ConnectRPC not in go.mod; stdlib suffices for a single-RPC service
AD-015	Best-fit bin-packing (v0.2)	Simple, deterministic, optimal for small fleets
AD-016	iter.Seq for streaming lists (v0.2)	Go 1.25+ native, context-aware, pull semantics

Anti-Patterns (Explicitly Avoided)

• No controller/agent split (single binary)
• No CRDs / custom resource definitions
• No web UI (CLI-only)
• No service mesh
• No container runtime integration
• No multi-tenancy
• No cloud provider integrations
• No auto-scaling
• No admission controllers
• No complex scheduling algorithms (best-fit only)
• No gRPC framework dependency (stdlib net/http with h2c, Go 1.25+ native)
• No external PKI / no cert transparency logs (offline-first)

Dependency Map (minimal — v0.2 adds zero direct deps)

github.com/spf13/cobra           # CLI framework
github.com/hashicorp/hcl/v2      # HCL parser
modernc.org/sqlite               # SQLite (pure Go)
github.com/google/uuid           # UUID generation

Total: ~4 direct dependencies. No web framework, no ORM, no RPC framework, no PKI library. mTLS via crypto/tls and crypto/x509 (stdlib).

ConnectRPC note: .ciagent/config.json lists connectrpc in frameworks, but the actual go.mod does not depend on connectrpc.com/connect. v0.2 falls back to plain net/http with HTTP/2 cleartext (h2c) for orca.v1.Dispatch. The protocol is a simple JSON-over-HTTP POST: client sends {"spec": "..."} to /orca.v1.Dispatch/Submit; server replies {"job_id": "..."}. This keeps the zero-new-dep promise and the codebase coherent with the rest of the daemon's http.ServeMux.

Deployment Model (v0.2)

Operator Machine             Node A (CA holder)              Node B (Peer)
────────────────             ─────────────────                ─────────────
orca CLI                     orca daemon                      orca daemon
   │                            │  ▲                            │  ▲
   │ mTLS handshake             │  │ mTLS                       │  │
   │ at `node join` ────────────┼──┘                            │  │
   │                            │                               │  │
   │ submit job ───POST────────▶│  POST (cross-node) ──────────▶│
   │                            │  mTLS only                    │  │
   │                            │                               │  │
   │ ◀───────jobID──────────────│  ◀─────jobID (200 OK)─────────│
   │                            │                               │
   │ orca job list --watch      │                               │
   │ (iter.Seq stream) ◀────────│── polls local + stream events│

For v0.2, one node must be the CA holder (orca cert init was run on it). The CA holder's ca.crt is copied to each peer manually by the operator; peers do not auto-fetch it.

v0.6 Architecture Addendum — Node Bootstrap & Proxmox

orca init Full Bootstrap (REQ-047, REQ-048, REQ-049)

orca init transforms from a bare mkdir into a full single-node cluster bootstrap. The sequence (idempotent per D-036):

orca init
  1. MkdirAll(certpaths.Dir(), 0o755)              # namespace dir
  2. store.Open(certpaths.DBPath())                # runs migrations 0001..0006
  3. security.CAInit(dir, "orca-internal-ca")      # idempotent fast-path
  4. if !exists(server.crt):
       GenerateCSR("localhost", ["localhost","127.0.0.1"])
       ca.SignCSR(csr) → WriteCert + WriteKey      # server cert (skip if present)
  5. os := detectOS()                              # /etc/os-release ID=
  6. node := Node{kind:"localhost", os:os, name:"localhost", addr:"localhost:8443"}
     if GetByName("localhost") exists:
       UpdateLastSeenAndOS(id, os)                 # refresh, keep id/joined_at
     else:
       NodeRepo.Insert(node)                       # first-run insert
  7. print summary (CA fp, server cert fp, os, node id)

After orca init, orca doctor MUST pass with zero FAILs.

Node Schema Extension (REQ-049)

Migration 0006 adds two nullable columns to nodes:

ALTER TABLE nodes ADD COLUMN kind TEXT;  -- localhost | linux | proxmox
ALTER TABLE nodes ADD COLUMN os TEXT;    -- ubuntu | debian | alpine | pve | linux

Existing rows get SQL NULL → mapped to "" in Go (sql.NullString). Node struct gains Kind string + OS string fields (JSON tags kind,omitempty / os,omitempty). NodeRepo extends all INSERT/SELECT/scanNode calls; adds GetByName(ctx, name) and UpdateLastSeenAndOS(ctx, id, os) helpers.

Proxmox SSH Bootstrap (REQ-050, REQ-051)

orca node join --type proxmox --host <addr> --user root --password <pw>
  │ password from --password or $ORCA_PROXMOX_PASSWORD (never persisted, D-031)
  ▼
internal/proxmox.BootstrapProxmox(ctx, opts)
  1. GenerateOrLoadSSHKey(certpaths.Dir())         # Ed25519, ~/.orca/orca_ssh_key{,.pub}
  2. SSH dial (password auth, knownhosts.New TOFU) # capture host key on first connect
  3. Deploy pubkey → ~orca/.ssh/authorized_keys    # via session heredoc (no SFTP dep)
  4. useradd -m orca                                # create Linux system user (config-overridable name)
  5. pveum role add OrcaOperator --privs "VM.Audit Datastore.AllocateSpace SDN.Use"
     (idempotent: probe pveum role list first)
  6. pveum user add orca@pam -comment "Orca automation user"
     (idempotent: probe pveum user list first)
  7. pveum acl modify / -user orca@pam -role OrcaOperator
     (idempotent: modify creates or updates)
  8. Write /etc/sudoers.d/orca (mode 0440):
       orca ALL=(root) NOPASSWD: NOEXEC: /usr/bin/pct, /usr/bin/qm
       orca ALL=(root) NOPASSWD: /usr/bin/apt-get, /usr/bin/dpkg
  9. visudo -cf /etc/sudoers.d/orca                 # validate; abort on error
  10. NodeRepo.Insert(Node{kind:"proxmox", os:"pve", name:host, addr:host})
  11. Audit log: proxmox.bootstrap_ok (host, user, role, fp)

pvesh excluded from sudoers — pvesh can trigger the API /nodes/{node}/execute endpoint which spawns shell commands server-side, bypassing sudo's NOEXEC tag. API access is via the OrcaOperator PVE role + orca@pam user (PVE RBAC), not sudo'd pvesh.

Doctor Extensions (REQ-052)

• doctor os: re-runs detectOS() from /etc/os-release, compares to the stored localhost node's os field. Drift = WARN (OS upgraded since init? re-run orca init to refresh). Match = PASS.
• doctor proxmox: iterates kind=proxmox nodes, SSH-probes each with pveversion (3s timeout per peer, clones doctor.Network() pattern). PASS = reachable + pveversion exits 0. WARN = zero proxmox nodes (single-node cluster is legitimate). FAIL = any node unreachable or pveversion fails.

SSH Key Handling (D-037)

• Location: ~/.orca/orca_ssh_key (0600) + ~/.orca/orca_ssh_key.pub (0644)
• Algorithm: Ed25519 (smaller, faster, more secure than RSA for SSH)
• Generation: lazy — on first orca node join --type proxmox, NOT at orca init (localhost doesn't need SSH)
• Format: PKCS8 PEM (consistent with ca.key/server.key; ssh.ParsePrivateKey accepts it)
• TOFU host keys: ~/.orca/known_hosts (OpenSSH format via knownhosts.New)

Dependency Map (v0.6 addition)

golang.org/x/crypto v0.54.0        # SSH (ssh + ssh/knownhosts + ed25519)
  └─ golang.org/x/sys v0.47.0      # indirect (bumped from v0.42.0)
  └─ golang.org/x/term v0.45.0     # indirect (pulled by ssh for PTY)

Total direct deps: 5 (was 4). One new direct dep (x/crypto). Matches D-030 minimal-deps rationale. No SFTP module (file upload via session heredoc).

v0.6 Architectural Decisions (AD-017..AD-021)

ID	Decision	Rationale
AD-017	orca init = full bootstrap (CA + cert + db + localhost node)	Single command produces a working cluster; orca doctor passes post-init. Idempotent (D-036).
AD-018	Proxmox join via SSH (golang.org/x/crypto/ssh), not PVE REST API	SSH is the universal Proxmox management entry point; REST API would require API token bootstrap (chicken-and-egg). One new direct dep (D-030).
AD-019	orca@pam realm (not orca@pve)	SSH creates a Linux system user; PAM realm maps it to PVE RBAC without a separate PVE password. @pve requires interactive password prompt over non-PTY SSH (hangs).
AD-020	Exclude pvesh from sudoers; NOEXEC on pct/qm	pvesh can trigger API execute endpoint bypassing NOEXEC. pct/qm are Perl scripts via dynamically-linked perl → NOEXEC effective. apt-get/dpkg need exec for maintainer scripts → no NOEXEC.
AD-021	TOFU host-key via knownhosts.New	Avoids deprecated ssh.InsecureIgnoreHostKey. Capture-on-first-connect, verify-on-subsequent. Fail closed on mismatch (operator runs key-reset).

---

v0.9 Architecture (Supersedes v0.8)

⚠️ v0.9 DIRECTION CHANGE: This section supersedes the v0.1–v0.8 architecture described above. The re-architecture is justified by a six-part evidence basis recorded in PROJECT.md (Supersession Table). The v0.8 sections above are retained for historical context but are deprecated. The 16 load-bearing rules (R-001…R-016) in PRD_v0.9.md are now the canonical invariants.

Superseded Decisions (AD-series reversals)

Old decision	Was	Superseded by	Evidence basis
AD-010 (line 463 above)	step-ca/cfssl/vault-pki "too heavyweight"	D-101 (step-ca)	External PKI mandate (override ground 2)
SPIFFE rejection (line 94, PROJECT.md)	internal CA chosen over SPIFFE	D-068 (SPIFFE SVIDs)	Multi-tenancy requires per-workload identity (override ground 3)
No-container-runtime (line 477 above)	explicit anti-pattern	D-088 (5 runtimes; wasmtime primary)	WASM is the workload profile (override ground 4)
No-multi-tenancy (line 478 above)	explicit anti-pattern	D-158 / R-002 (multi-namespace)	Hard multi-tenant product req (override ground 3)
AD-007 (HCL canonical)	HCL for jobspec	R-013 / R-014 (Markdown canonical; HCL legacy)	PRD §8 operator-facing format
Daemon-on-every-node	orca daemon on all peers	R-001 (no orca binary on any server)	Daemon operationally failing + SSH-push only viable target (override grounds 1 + 5)

The Five-Layer CLI (v0.9)

The orca binary is one Go program, structured internally as five layers:

1. CLI subcommand tree (cobra) — internal/cli/
2. Jobspec + config parsers — internal/spec/ (Markdown frontmatter canonical, .md/.yaml/.hcl dispatcher per R-013/R-014)
3. Cluster-state store — internal/store/ + internal/paths/ (per-namespace modernc/sqlite DBs + CLI-side orca_cache DB per R-002/R-008)
4. Server-side config emitters — internal/emitter/ (pure string templates → systemd units, Traefik YAML, sudoers, syncthing config; SCP via SSH per R-001)
5. Workflow orchestrators — internal/sshpush/ (compose SSH + local FS writes into multi-step commands)

The Server Side (R-001 — no Orca binary on any server)

Servers hold only: rendered config in /etc/orca/actual/<txn-id>/, systemd units, Traefik dynamic config, sudoers, sshd_config snippets, step-ca/traefik/syncthing/podman/wasmtime/age/auditd (installed via apt), and bash scripts in scripts/ (orca-pull.sh, orca-drift.sh, orca-collect.sh, orca-aggregate.sh, orca-apply-render.sh, orca-verify-render.sh, orca-rollback-render.sh, orca-cleanup-credentials.sh). Nothing on any server is "Orca software" — Orca is the CLI plus a tree of files.

Multi-namespace Layout (R-002)

$ORCA_HOME/
├── cluster/                  # cluster-wide (NOT a workload namespace)
│   ├── ca.crt, ca.key        # step-ca root (R-006, D-101)
│   ├── master.key            # AES-256-GCM root (R-011, mode 0600)
│   ├── config.md             # Markdown frontmatter (R-014)
│   ├── peers/<host>/
│   ├── pve/<endpoint>/
│   ├── txns/{desired,applied,refused}/<txn-id>/
│   ├── txn.sqlite
│   └── state/
├── _defaults/                # implicit root namespace (always exists)
│   ├── ns.md
│   ├── .env, .env.secrets
│   ├── db/orca.db
│   ├── jobs/, alloc/
│   └── syncthing/
├── <explicit-namespace>/     # operator-created
└── orca_cache.db            # CLI-side cache (R-008)

Execution gates (from GRILL_v0.9.md)

The 19 binding conditions (C-01..C-19) and 10 phase challenges (PC-01..PC-10) gate specific phases. See GRILL_v0.9.md for the full list. Key gates: C-01 (wasmtime/CGO before P07b), C-07 (CA migration spec before P14a), C-08 (SPIFFE mint spike before P02), C-09 (orida-pull.sh failure contract before P10), C-19 (threat model before P15.5).

v0.9–v0.12 Component Addendum (post-rearchitecture packages)

The v0.9 re-architecture introduced the SSH-push model and split the monolithic v0.8 transport layer into focused packages. The following packages were added or substantially expanded across v0.9–v0.12 and are part of the canonical component graph:

Workload & runtime layer

• internal/runtime/ — runtime abstraction (process/podman/wasm/pve-vm/pve-ct), 5 backends (REQ-078, C-01)
• internal/scheduler/ — CLI-side scheduler, CEL constraints, affinity (REQ-083)
• internal/jobspec/ — job specification parsing & validation
• internal/spec/ — update stanza + lifecycle hooks
• internal/engine/ — dispatcher, executor, peer, registry, audit, scheduler

State & persistence layer

• internal/model/ — core data model (Node, Job, Task, Certificate, Alloc)
• internal/store/ — cluster-state store, per-namespace modernc/sqlite
• internal/paths/ — path resolution for the multi-namespace layout (R-002)
• internal/certpaths/ — certificate path helpers (known_hosts, CA material)
• internal/cache/ — CLI-side orca_cache SQLite (R-008)
• internal/migration/ — v0.8→v1.0 data migration (REQ-066, C-07)
• internal/txn/ — transactional plane, apply-path allowlist (REQ-075, REQ-079)
• internal/ns/ — namespace subcommands, inheritance, constraints (REQ-068)

Transport & bootstrap layer

• internal/sshpush/ — v0.9 SSH-push transport, fanout, idempotency (R-001, C-18)
• internal/cluster/ — lead rules, rotate-lead, mixed-version tolerance
• internal/proxmox/ — Proxmox API + host-key TOFU (D-035)
• internal/stepca/ — step-ca integration (REQ-076)
• internal/storage/ — Syncthing storage replication + conflict resolution (REQ-081)
• internal/backup/ — backup/restore, signed tarball (HMAC-SHA256)
• internal/secrets/ — per-namespace AES-256-GCM + HKDF-SHA256 (REQ-080)
• internal/emit/ — emit contract (systemd units, Traefik YAML, sudoers, syncthing)
• internal/emitter/ — server-side config emitters (renders internal/emit contract)
• internal/osdetect/ — OS detection for renderer dispatch (R-013/R-014)

Drift detection layer

• internal/drift/ — drift detection collector + aggregator (REQ-103..113; R-018/R-019/R-020)

Security & identity layer (v0.12 — Zero-Trust Identity)

• internal/identity/ — OIDC client + auth CLI (REQ-144)
• internal/seal/ — master key seal-to-OIDC + Shamir 3-of-5 (REQ-147, D-241, C-35)
• internal/webauthn/ — WebAuthn connector for Dex (REQ-148, D-240, C-38)
• internal/acl/ — ACL rewrite to OIDC claims, deny-by-default (REQ-122, REQ-145)
• internal/audit/ — audit log tamper-evidence (REQ-125, F2)
• internal/security/ — SVID chain validation, daemon auth, file-mode enforcement (REQ-123, REQ-124, REQ-126)
• internal/config/ — cluster config parsing, frontmatter dispatch (R-014)

Deprecated / dual-write (removed in v1.x)

• internal/transport/ — v0.8 mTLS HTTP layer; superseded by internal/sshpush/ (dual-write window closed in v0.12 P07; full deletion deferred to v1.x per P23_DUAL_WRITE_DECISION.md)

Execution gates (v0.12)

The v0.12 milestone is gated by binding conditions C-29..C-38 (see GRILL_v0.12.md). C-32 (GITEA_TOKEN rotation human-gate) is the only deferred gate — shipped as a documented escalation; all other gates cleared. The load-bearing rule is R-021 (no Orca password/token paths).

v0.13 Architecture Deltas — Production Hardening Round 2

R-022: Scheduler/Deployment Wiring

orca job run now deploys to remote nodes via the pipeline:

scheduler.Schedule(spec, nodes) → emitter.Render(unit) → sshpush.Deploy(target, unit)

• The local exec.CommandContext path in internal/engine/executor.go is removed for the dispatch path. Local execution is the fallback when no remote nodes are registered (single-node dev mode).
• internal/scheduler.Schedule() evaluates CEL constraints, capacity fit, and affinity scoring against registered nodes.
• internal/emitter/systemd.go renders the unit; systemd-analyze verify validates before deploy.
• internal/sshpush pushes the unit + env file to the target node.
• --target <node> overrides scheduler selection (manual pinning).
• Without --target, the scheduler bin-packs across all ready nodes.

R-023: Zero-Trust Enforcement Wiring

acl.Check is invoked on every request path:

• Daemon handlers (dispatch/jobs/nodes/tasks/health): extract OIDC sub/SPIFFE SVID from mTLS peer cert → acl.Check(acl, identity, namespace, verb) → deny-by-default.
• SSH-push applier (internal/sshpush/): validate ORCA_OIDC_TOKEN bearer against JWKS before applying any txn.
• Txn apply (internal/txn/): same bearer validation.
• Audit actor field carries the OIDC sub or SPIFFE SVID (not "cli"/"daemon").
• acl.json mode is 0600 (not 0644).
• WebAuthn registration (/orca/webauthn/register) requires an existing authenticated session or admin bootstrap token.

New Components

• internal/linux/bootstrap.go — Ubuntu/Debian SSH-join (mirrors internal/proxmox/bootstrap.go without PVE role/sudoers). Deploys orca pubkey, creates orca system user, creates drift-events dir. Key-auth only (R-021). Invoked via orca node join --type linux.
• internal/cli/cluster_seal.go — orca cluster seal/unseal CLI (wraps internal/seal/ library; OIDC token exchange → unwrap master key → zeroed on shutdown; Shamir 3-of-5 shards at seal time).
• internal/cli/doctor_audit.go — orca doctor audit (wraps AuditRepo.VerifyChain).
• internal/cli/doctor_modes.go — orca doctor modes (wraps EnforceFileModes across ORCA_HOME).

New Artifacts

• docs/uat.md — UAT plan (3-host topology, step-by-step, claim matrix)
• scripts/uat-signoff.sh — v1.0 gate signoff script (~35 assertions, idempotent, read-only)
• scripts/uat-smoke.sh — CI-tested pure-CLI subset of signoff
• docs/metrics.md — expanded Prometheus metric set reference

jobspec Parser Fixes

• schedule: and timeout: now parsed at top level (previously silently dropped by the markdown parser's default case).
• DaemonSet: parser no longer defaults Count to 1 (validator rejects Count != 0 for DaemonSet).
• restart: policy translated to systemd Restart=/StartLimitBurst in the emitter.
• job lint emits honest "not enforced in this version" warnings for advisory-only fields (cron, health, update, affinity).

Concurrency Safety

• All SQLite DSNs set busy_timeout(5000) + SetMaxOpenConns(1).
• Secrets file flock prevents concurrent-write data loss.
• Upgrade/backup lock files prevent concurrent cutover/clobber.
• Cache invalidated by write commands (read-after-write consistency).
• Audit Append uses BEGIN IMMEDIATE transaction (chain race fixed).
• WebAuthn session stores guarded with sync.Mutex.

Transport Safety

• Typed sentinels replace substring matching in both transport and sshpush packages.
• rotateSSHKeys 2-phase atomic swap (stage → swap → verify → cleanup).
• IPv6 net.JoinHostPort in all SSH dial paths.
• Explicit timeouts on all SSH commands.
• Root SIGINT/SIGTERM handler for clean exit on non-watch commands.

v0.14 Deltas — Ingress Bootstrap Completeness (R-024)

R-024: Traefik as Podman Container

Traefik runs exclusively as a podman container, deployed from the custom orca-traefik image (published per release via Dockerfile.traefik

• scripts/release.sh + .coreci.yml container-publish-traefik).

The v0.13 binary+systemd install (internal/traefik/install.go) is replaced by an idempotent podman container reconciler (EnsureTraefikContainerLocal/Remote). The container runs with --network host, --restart=unless-stopped, and volume mounts for traefik.yml (static config), dynamic (dynamic config), and step-ca-root.crt (future mTLS). No SELinux :Z flag.

Three Ingress Topologies

1. Linux (orca init / orca node join --type linux): host → nft DNAT → podman traefik (host network). internal/ingress/bootstrap.go → BootstrapLocalIngress / BootstrapRemoteIngress.

2. Proxmox Native (--ingress-mode native, default): PVE host → nft DNAT (target = LXC bridge IP) → LXC (--features nesting=1,keyctl=1,fuse=1) → podman traefik. internal/proxmox/bootstrap.go → provisionNativeIngressLXC.

3. Proxmox Floating-IP (--ingress-mode floating-ip): LXC owns the floating IP (net0 bridge=vmbr0,hwaddr=<mac>, ip=<floating-ip>/<prefix>,gw=<gateway>) → nft inside LXC → podman traefik. The ingress LXC is registered as a linux node (name=ingress) so orca job run pushes traefik dynamic config. internal/proxmox/ingress_lxc.go → ProvisionIngressLXC.

nft Emitter Changes

internal/emitter/nft.go:

• DNATTarget field (C-51: validated via net.ParseIP). Default 127.0.0.1; proxmox native uses LXC bridge IP.
• EnableSNAT field + postrouting masquerade chain: ip saddr 127.0.0.0/8 oifname != "lo" masquerade (research Topic 1).
• Input/forward chain priority shifted from filter (=0) to -10 (research Topic 2: pve-firewall coexistence — avoids same-priority undefined evaluation order).

TLS Model

v0.14 drops certResolver: orca from the dynamic config (traefik v3.3 only supports acme/tailscale resolvers, not CA-file-based). The dynamic config emits tls: {} (traefik default cert). Real mTLS via tls.certificates + tls.options.default.clientAuth.caFiles is deferred to v0.15 (grill G-003, confidence 0.55 < 0.60).

Migration 0009

ALTER TABLE nodes ADD COLUMN ingress_mode TEXT NOT NULL DEFAULT ''; Values: "" (legacy), "native", "floating-ip". IngressMode field on model.Node.

New CLI

• orca doctor ingress — verifies podman container running, nft DNAT+SNAT, dynamic dir, step-ca root CA.
• --ingress-mode flag on orca node join --type proxmox.
• --floating-ip, --gateway, --mac, --net-prefix flags for floating-IP mode.