orca/.ciagent/ARCHITECTURE.md

Architecture: Orca

System Overview

Orca is a single-binary, offline-first orchestration engine. The system consists of three logical layers (CLI, Daemon, Engine) compiled into one orca binary and selected via subcommands. v0.2 adds a cross-node transport layer (mTLS) and a dispatcher for multi-node job execution.

┌─────────────────────────────────────────────────────────────────────────────┐
│                       orca (single binary, v0.2)                             │
├─────────────────────────────────────────────────────────────────────────────┤
│  CLI Layer (Cobra)                                                          │
│  ├── orca version                                                           │
│  ├── orca init                          # local node bootstrap               │
│  ├── orca cert {init,join,renew,show}   # NEW (P01)                         │
│  ├── orca status                                                           │
│  ├── orca node {join,leave,list}        # join = mTLS handshake (P01)      │
│  │   └── orca node list --watch         # NEW iter.Seq (P04)                │
│  ├── orca job {run,list,stop,logs}                                         │
│  │   └── orca job list --watch          # NEW iter.Seq (P04)                │
│  ├── orca doctor                       # NEW (P01) — diagnostics            │
│  │   ├── orca doctor cert                                                   │
│  │   ├── orca doctor network                                                │
│  │   └── orca doctor db                                                     │
│  └── orca daemon                                                           │
├─────────────────────────────────────────────────────────────────────────────┤
│  Daemon Layer (net/http over h2c, mTLS in P01)                              │
│  ├── /healthz                          (liveness)                           │
│  ├── /readyz                           (readiness)                          │
│  ├── /v1/jobs/*                        (job control API)                    │
│  ├── /v1/nodes/*                       (node registry API)                  │
│  ├── /v1/tasks/*                       (task lifecycle API)                 │
│  ├── /orca.v1.Dispatch/...             # NEW (P02) — cross-node dispatch    │
│  └── /orca.v1.Register/...             # NEW (P02) — peer join ack           │
├─────────────────────────────────────────────────────────────────────────────┤
│  Transport Layer (NEW — internal/transport)                                  │
│  ├── mTLS client                       (dialer pool per peer)               │
│  ├── mTLS server config                (TLS 1.3 only, AEAD allowlist)       │
│  ├── Retry+backoff                     (exponential, jittered, capped)       │
│  └── Graceful disconnect               (ctx-aware Conn.Close)               │
├─────────────────────────────────────────────────────────────────────────────┤
│  Core Engine                                                                │
│  ├── Node Registry                     (in-memory + SQLite persistence)     │
│  ├── Task Executor                     (os/exec with WaitDelay, Go 1.25+)   │
│  ├── Job Scheduler                     (single-node FIFO; bin-pack P02)     │
│  ├── Dispatcher                        # NEW internal/engine/dispatcher.go │
│  │   ├── Local decision                (does this job fit on this node?)    │
│  │   ├── Remote dispatch               (POST to peer via transport)         │
│  │   └── Streaming callback            (iter.Seq[DispatchResult] for CLI)   │
│  ├── Security Manager                  # NEW internal/security (P01)        │
│  │   ├── CA lifecycle                  (init, fingerprint, sign CSR)        │
│  │   ├── Server cert lifecycle         (issue, renew, rotate)                │
│  │   ├── mTLS config builder                                              │
│  │   └── Cert store                    (filesystem + SQLite metadata)       │
│  └── Audit Logger                      (log/slog JSON handler)              │
├─────────────────────────────────────────────────────────────────────────────┤
│  State Store (modernc/sqlite, CGO-free)                                     │
│  ~/.orca/orca.db                                                            │
│  ├── nodes, jobs, tasks, audit_log    (v0.1)                                │
│  └── certs                             # NEW (P01) — CA + server certs      │
└─────────────────────────────────────────────────────────────────────────────┘

Component Details

1. CLI Layer (cmd/orca, internal/cli)

• Framework: Cobra (industry standard, familiar to operators)
• v0.1 subcommands: version, init, status, node, job, daemon
• v0.2 additions (P01): orca cert {init,join,renew,show}
• v0.2 additions (P04): --watch flag on orca job list and orca node list
• v0.2 additions (P01): orca doctor subcommand (see §5 below)
• Output: Human-readable by default; --json flag for machine consumption
• Discovery: All subcommands self-document via Cobra's auto-generated help
• Watch semantics (P04): --watch consumes iter.Seq[Job|Node], exits on ctrl-c (via signal.NotifyContext), refreshes on internal change events.

2. Daemon Layer (internal/daemon)

• Server: net/http with http.ServeMux (no external router)
• TLS (P01): crypto/tls with MinVersion=tls.VersionTLS13 and AEAD cipher allowlist (TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_AES_128_GCM_SHA256)
• Ports: Configurable (default :8443 for API+mTLS, :8080 for health)
• Graceful Shutdown: signal.NotifyContext with SIGINT/SIGTERM
• v0.2 endpoints (P02):
  • POST /orca.v1.Dispatch/Submit — receive cross-node job submission
  • POST /orca.v1.Dispatch/Status — query dispatched job status
  • POST /orca.v1.Register/Hello — peer join ack (used during orca node join)

3. Transport Layer (internal/transport, NEW in P01/P02)

• Client: http.Client with http.Transport.TLSClientConfig populated from internal/security.NewClientTLSConfig
• Server: http.Server.TLSConfig populated from internal/security.NewServerTLSConfig
• Retry policy: exponential backoff with jitter (start 100ms, x2, cap 5s, max 5 attempts); only idempotent verbs (GET, HEAD, OPTIONS) are retried automatically; POST retries require an explicit X-Orca-Idempotency-Key header
• Conn lifecycle: context.Context-aware dials and reads; graceful Close on ctx.Done()
• Peer dial pool: small sync.Map of peerID → *http.Client to reuse TLS handshakes (TCP keep-alive) within a session

4. Core Engine (internal/engine)

• Node Registry: In-memory map of node IDs → metadata, persisted to SQLite (CPU/memory capacity, available slots, last-seen)
• Task Executor: os/exec.CommandContext with WaitDelay (Go 1.25+) for clean process termination
• Job Scheduler (v0.2 P02):
  • Algorithm: best-fit bin-packing by available_cpu and available_memory
  • Within-node ordering: FIFO queue
  • Cross-node: if local node is full, Dispatcher.Submit(peer, job) is invoked; peers are tried in round-robin order
  • Fallback: if all peers reject, return ErrNoFit and requeue
• Dispatcher (NEW, P02):
  • Submit(peerID, spec) (jobID, error) — blocking call with retry
  • Watch(peerID) iter.Seq[DispatchEvent] — pull-style event stream for the CLI's --watch flag
  • Stateless: every call uses the latest mTLS client config and peer address
• Security Manager (NEW, P01):
  • InitCA(commonName) (*CA, error) — generates a self-signed CA, writes ca.crt (0644) and ca.key (0600) to ~/.orca/
  • Fingerprint(certPath) (sha256hex, error) — used by orca cert join
  • SignServerCert(csr, validity) (*cert, error) — signs a CSR with the CA
  • IssueServerCert(nodeName, dnsNames, ips) (*cert, *key, error) — generates a keypair + CSR + signs it, returns PEM bytes for orca cert join --server
  • ServerTLSConfig() (*tls.Config, error) — loads server.crt/server.key and the CA pool from disk
  • ClientTLSConfig(caPath) (*tls.Config, error) — returns a client config pinned to the supplied CA
  • Rotation policy: server certs valid 90d; CA cert valid 10y. On orca cert renew, IssueServerCert is called and the daemon gracefully reloads the in-process tls.Config via GetCertificate hot-swap (no restart required)
• Audit Logger: slog.NewJSONHandler(os.Stderr, ...) with structured fields

5. Doctor (internal/doctor, NEW in P01)

• Purpose: operator-facing diagnostics; runs read-only checks against the local state and reports PASS/WARN/FAIL.
• Subcommands:
  • orca doctor — runs all checks
  • orca doctor cert — cert/CA health (file modes, expiry windows, SAN presence, fingerprint pinning match — see REQ-026, REQ-033, REQ-034, REQ-036)
  • orca doctor network — peer reachability over mTLS (per-peer handshake sanity, last-seen delta)
  • orca doctor db — SQLite integrity check (PRAGMA integrity_check)
    • migration version
• Output: human-readable by default; --json for machine consumption
• No state changes: doctor is strictly read-only. It can be run while the daemon is down (where possible) or while it's up.
• Initial implementation in P01 (cert checks only); network and db checks land in subsequent phases as their state becomes available.

6. State Store (internal/store)

• Driver: modernc.org/sqlite (pure Go, CGO-free)
• Location: ~/.orca/orca.db (user-mode) or /var/lib/orca/orca.db (system-mode)
• Schema (v0.1): nodes, jobs, tasks, audit_log tables
• Schema (v0.2 P01): NEW certs table

  CREATE TABLE certs (
      id            INTEGER PRIMARY KEY AUTOINCREMENT,
      kind          TEXT    NOT NULL,           -- 'ca' | 'server'
      node_id       TEXT,                       -- NULL for CA
      serial_hex    TEXT    NOT NULL,           -- x509.SerialNumber.Hex()
      subject_cn    TEXT    NOT NULL,
      issuer_cn     TEXT    NOT NULL,
      not_before    INTEGER NOT NULL,           -- unix seconds
      not_after     INTEGER NOT NULL,           -- unix seconds
      fingerprint   TEXT    NOT NULL,           -- sha256 of DER, hex
      source_path   TEXT    NOT NULL,           -- on-disk PEM path
      created_at    INTEGER NOT NULL
  );
  CREATE INDEX idx_certs_node_kind ON certs(node_id, kind);
  CREATE INDEX idx_certs_not_after ON certs(not_after);
  

• Migrations: Embedded SQL files, applied on startup
• Migration 0004 is added in P01 with the schema above

Data Model

Node (v0.1, extended in v0.2 P02)

type Node struct {
    ID        string
    Name      string
    Address   string
    State     string
    JoinedAt  time.Time
    LastSeen  time.Time
    Metadata  map[string]string
    // v0.2 P02 — capacity for bin-packing
    Capacity  NodeCapacity
}

type NodeCapacity struct {
    CPUMillicores int   // total, e.g. 4000 = 4 cores
    MemoryBytes   int64 // total RAM
    CPUUsed       int   // currently allocated
    MemoryUsed    int64 // currently allocated
}

func (c NodeCapacity) AvailableCPU() int       { return c.CPUMillicores - c.CPUUsed }
func (c NodeCapacity) AvailableMemory() int64  { return c.MemoryBytes - c.MemoryUsed }

Job (v0.1)

type Job struct {
    ID        string
    Name      string
    Spec      string
    Status    string
    CreatedAt time.Time
    StartedAt *time.Time
    EndedAt   *time.Time
}

Task (v0.1)

type Task struct {
    ID        string
    JobID     string
    Command   string
    Args      []string
    Env       []string
    PID       int
    ExitCode  int
    Status    string
    CreatedAt time.Time
    StartedAt *time.Time
    EndedAt   *time.Time
}

Certificate (NEW, v0.2 P01)

type Cert struct {
    Kind        CertKind   // CertCA | CertServer
    NodeID      string     // empty for CA
    SerialHex   string
    SubjectCN   string
    IssuerCN    string
    NotBefore   time.Time
    NotAfter    time.Time
    Fingerprint string     // sha256 of DER (hex)
    SourcePath  string     // PEM path on disk
    CreatedAt   time.Time
}

v0.2 Component Graph (ASCII)

                                  ┌─────────────────┐
                                  │  Operator Host  │
                                  │  (orca CLI)     │
                                  └────────┬────────┘
                                           │ 1. cert join --ca-fingerprint <sha>
                                           ▼
┌──────────────────────────────────────────────────────────────────────────────┐
│                              NODE A (Bootstrap / CA holder)                  │
│                                                                              │
│  ┌──────────────┐    CSR     ┌────────────────────┐   PEM sign   ┌────────┐  │
│  │ orca cert    │──────────▶│ internal/security   │─────────────▶│  CA    │  │
│  │ {init,join}  │            │ .SignServerCert()  │              │  key   │  │
│  └──────────────┘            └────────────────────┘              │ 0600   │  │
│                                                              ┌──└────────┘  │
│  ┌─────────────────┐                                         │              │
│  │ internal/daemon │ ◀──tls.Config── internal/security        │              │
│  │ (http.Server)   │              .ServerTLSConfig()          │              │
│  │                 │                                         │              │
│  │ /v1/jobs/*      │                                         │              │
│  │ /v1/nodes/*     │                                         │              │
│  │ /orca.v1.*      │                                         │              │
│  └────────┬────────┘                                         │              │
│           │  Submit(job)  (bin-pack)                         │              │
│           ▼                                                  │              │
│  ┌─────────────────┐                                         │              │
│  │ internal/engine │                                         │              │
│  │ .scheduler      │──── if local fits → executor            │              │
│  │ .dispatcher     │──── else → Submit(peer, job) ───────────┼──┐           │
│  └─────────────────┘                                         │  │           │
│                                                              │  │ mTLS      │
└──────────────────────────────────────────────────────────────┼──┼───────────┘
                                                               │  │
                                            ORCA NODE NETWORK │  │
                                                               │  │
┌──────────────────────────────────────────────────────────────┼──┼───────────┐
│                              NODE B (Peer)                   │  │           │
│                                                              │  │           │
│  ┌─────────────────┐  ◀── TLS 1.3 handshake ─────────────────┘  │           │
│  │ internal/daemon │                                              │           │
│  │ (http.Server)   │  POST /orca.v1.Dispatch/Submit              │           │
│  │                 │──── 200 + jobID                              │           │
│  └────────┬────────┘                                              │           │
│           │                                                       │           │
│           ▼                                                       │           │
│  ┌─────────────────┐                                              │           │
│  │ internal/engine │                                              │           │
│  │ .scheduler (FIFO)                                              │           │
│  │ .executor                                                     │           │
│  └─────────────────┘                                              │           │
└──────────────────────────────────────────────────────────────────────────────┘

v0.2 Flows

Flow 1: Cert Issuance (CA-init → CSR → sign → install) — P01

Operator (Node A)                       Operator (Node B)
─────────────────                       ─────────────────
orca cert init
  ↳ InitCA("orca-ca")
  ↳ write ca.crt (0644), ca.key (0600)
  ↳ record in certs table (kind='ca')
                                         orca cert join --ca-fingerprint <sha>
                                           ↳ operator copies ca.crt → Node B
                                           ↳ verifies fingerprint matches local
                                              --ca-fingerprint arg
                                           ↳ IssueServerCert("node-b", SANs)
                                              ↳ generate 2048-bit RSA key
                                              ↳ build CSR with SANs
                                              ↳ read ca.crt + ca.key
                                              ↳ sign CSR (90d validity)
                                           ↳ write server.crt (0644),
                                              server.key (0600)
                                           ↳ record in certs table
                                              (kind='server', node_id='node-b')

Flow 2: mTLS Handshake at node join — P01

Node B (joiner)                                  Node A (CA holder)
────────────────                                  ─────────────────
orca node join --name node-b
  --ca-fingerprint <sha>
  --peer node-a:8443
  ↳ load ca.crt → verify sha256 == --ca-fingerprint
  ↳ load server.crt + server.key
  ↳ tls.Config{MinVersion: TLS1.3, ...}
  ↳ ClientHello (SNI=node-a)
                                                ◀── ServerHello (TLS 1.3)
                                                ◀── Certificate (Node A's cert)
  ↳ verify Node A's cert chains to ca.crt       ◀── CertificateRequest
  ↳ send Certificate (Node B's cert)            ◀── Finished
  ↳ Finished
  ↳ GET /healthz (over mTLS) — sanity check
  ↳ POST /v1/nodes (over mTLS) — register
                                                 ↳ insert into nodes table
                                                 ↳ audit log
  ↳ 200 OK
  ↳ record node_a in peers table
  ↳ audit log

Flow 3: Job Dispatch (CLI → dispatcher → peer) — P02

User (Node A CLI)             Node A (scheduler)            Node B (peer)
──────────────────             ───────────────────            ─────────────
orca job run spec.hcl
  ↳ parse HCL
  ↳ POST /v1/jobs (mTLS, local)
                                ↳ scheduler.Submit(spec)
                                ↳ bin-pack: spec.cpu + spec.mem
                                ↳ local node A has 2000mc + 4GiB free → fit!
                                ↳ executor.Run(spec)
                                ↳ 202 Accepted + jobID
  ↳ returns jobID

User (Node A CLI)             Node A (scheduler)            Node B (peer)
──────────────────             ───────────────────            ─────────────
                                ↳ scheduler.Submit(spec)
                                ↳ bin-pack: spec.cpu + spec.mem
                                ↳ local node A has 0 free    → NO FIT
                                ↳ dispatcher.Submit(peer="node-b", spec)
                                  ↳ load transport.Client("node-b")
                                  ↳ POST /orca.v1.Dispatch/Submit
                                                                ↳ mTLS handshake
                                                                ↳ authenticate cert
                                                                ↳ scheduler.Submit(spec)
                                                                ↳ executor.Run(spec)
                                                                ↳ 200 OK + jobID
                                  ↳ 200 OK + jobID
                                ↳ return jobID to local caller
  ↳ returns jobID

Flow 4: iter.Seq Streaming (--watch) — P04

User                              orca job list --watch
────                              ─────────────────────
                                  ctx, cancel := signal.NotifyContext(ctx, os.Interrupt)
                                  defer cancel()
                                  seq := store.Jobs().Watch(ctx)
                                  for job := range seq {
                                    print(job)  // human or --json
                                  }
                                  // ctrl-c → ctx.Done() → seq stops yielding

Internally store.Jobs().Watch(ctx) iter.Seq[Job] polls the jobs table on a 1s ticker (or subscribes to an in-process notifier channel) and yields the current snapshot of each job until ctx.Done(). The store repo implements iter.Seq[Job] as a function that takes a yield func(Job) bool callback.

Security Architecture

Authentication

• v0.1: mTLS for all API endpoints (self-signed CA)
• v0.2 P01: Internal CA with CSR join (see Flow 1 + 2)
• v0.2+: Token-based auth deferred to v0.3+

Cert Rotation

• Server certs: 90-day validity, rotate at 60 days (30d before expiry)
• CA cert: 10-year validity, manual rotation
• Hot-swap: tls.Config.GetCertificate callback re-reads the server.crt/server.key files on each handshake so orca cert renew takes effect without a daemon restart.

Audit Logging

• All state-changing operations emit structured log records
• Fields: timestamp, actor, action, resource, result, error
• Stored in SQLite audit_log table and stderr (JSON)
• v0.2 P01 additions: cert.issued, cert.renewed, cert.joined, node.handshake_ok, node.handshake_failed

Input Validation

• All CLI inputs validated via Cobra's Args/ValidArgs functions
• All API inputs validated at handler boundary
• HCL/YAML specs parsed with strict schemas

Key Architectural Decisions (v0.1 + v0.2)

ID	Decision	Rationale
AD-001	Single binary with subcommands	Simpler distribution, aligns with simplicity pillar
AD-002	modernc/sqlite (CGO-free)	Cross-compile friendly, no CGO toolchain needed
AD-003	net/http (no framework)	Stdlib suffices; avoids external router dependency
AD-004	os/exec with WaitDelay (Go 1.25+)	Clean process termination, native to Go
AD-005	Cobra for CLI	Industry standard, familiar to operators
AD-006	slog for logging	Native to Go 1.21+, no external dependency
AD-007	HCL for job specs	Familiar to Nomad/HashiCorp users
AD-008	Single-node scheduling (v0.1)	Multi-node scheduling deferred to v0.2+
AD-009	Internal CA, no external PKI (v0.2)	Self-contained, no operational PKI requirement
AD-010	Roll-our-own CA in crypto/x509 (v0.2)	step-ca/cfssl/vault-pki too heavyweight for Orca's footprint
AD-011	Operator-mediated CA cert distribution (v0.2)	No secret distribution over the wire; matches offline-first
AD-012	TLS 1.3 only, AEAD allowlist (v0.2)	Modern crypto only; no downgrade risk
AD-013	Eager mTLS at node join (v0.2)	Fail fast; don't defer handshake to first request
AD-014	orca.v1.Dispatch via stdlib h2c (v0.2)	ConnectRPC not in go.mod; stdlib suffices for a single-RPC service
AD-015	Best-fit bin-packing (v0.2)	Simple, deterministic, optimal for small fleets
AD-016	iter.Seq for streaming lists (v0.2)	Go 1.25+ native, context-aware, pull semantics

Anti-Patterns (Explicitly Avoided)

• No controller/agent split (single binary)
• No CRDs / custom resource definitions
• No web UI (CLI-only)
• No service mesh
• No container runtime integration
• No multi-tenancy
• No cloud provider integrations
• No auto-scaling
• No admission controllers
• No complex scheduling algorithms (best-fit only)
• No gRPC framework dependency (stdlib net/http with h2c, Go 1.25+ native)
• No external PKI / no cert transparency logs (offline-first)

Dependency Map (minimal — v0.2 adds zero direct deps)

github.com/spf13/cobra           # CLI framework
github.com/hashicorp/hcl/v2      # HCL parser
modernc.org/sqlite               # SQLite (pure Go)
github.com/google/uuid           # UUID generation

Total: ~4 direct dependencies. No web framework, no ORM, no RPC framework, no PKI library. mTLS via crypto/tls and crypto/x509 (stdlib).

ConnectRPC note: .ciagent/config.json lists connectrpc in frameworks, but the actual go.mod does not depend on connectrpc.com/connect. v0.2 falls back to plain net/http with HTTP/2 cleartext (h2c) for orca.v1.Dispatch. The protocol is a simple JSON-over-HTTP POST: client sends {"spec": "..."} to /orca.v1.Dispatch/Submit; server replies {"job_id": "..."}. This keeps the zero-new-dep promise and the codebase coherent with the rest of the daemon's http.ServeMux.

Deployment Model (v0.2)

Operator Machine             Node A (CA holder)              Node B (Peer)
────────────────             ─────────────────                ─────────────
orca CLI                     orca daemon                      orca daemon
   │                            │  ▲                            │  ▲
   │ mTLS handshake             │  │ mTLS                       │  │
   │ at `node join` ────────────┼──┘                            │  │
   │                            │                               │  │
   │ submit job ───POST────────▶│  POST (cross-node) ──────────▶│
   │                            │  mTLS only                    │  │
   │                            │                               │  │
   │ ◀───────jobID──────────────│  ◀─────jobID (200 OK)─────────│
   │                            │                               │
   │ orca job list --watch      │                               │
   │ (iter.Seq stream) ◀────────│── polls local + stream events│

For v0.2, one node must be the CA holder (orca cert init was run on it). The CA holder's ca.crt is copied to each peer manually by the operator; peers do not auto-fetch it.