# Architecture: Orca

## System Overview

Orca is a single-binary, offline-first orchestration engine. The system consists
of three logical layers (CLI, Daemon, Engine) compiled into one `orca` binary
and selected via subcommands. v0.2 adds a **cross-node transport layer** (mTLS)
and a **dispatcher** for multi-node job execution.

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                       orca (single binary, v0.2)                             │
├─────────────────────────────────────────────────────────────────────────────┤
│  CLI Layer (Cobra)                                                          │
│  ├── orca version                                                           │
│  ├── orca init                          # local node bootstrap               │
│  ├── orca cert {init,join,renew,show}   # NEW (P01)                         │
│  ├── orca status                                                           │
│  ├── orca node {join,leave,list}        # join = mTLS handshake (P01)      │
│  │   └── orca node list --watch         # NEW iter.Seq (P04)                │
│  ├── orca job {run,list,stop,logs}                                         │
│  │   └── orca job list --watch          # NEW iter.Seq (P04)                │
│  ├── orca doctor                       # NEW (P01) — diagnostics            │
│  │   ├── orca doctor cert                                                   │
│  │   ├── orca doctor network                                                │
│  │   └── orca doctor db                                                     │
│  └── orca daemon                                                           │
├─────────────────────────────────────────────────────────────────────────────┤
│  Daemon Layer (net/http over h2c, mTLS in P01)                              │
│  ├── /healthz                          (liveness)                           │
│  ├── /readyz                           (readiness)                          │
│  ├── /v1/jobs/*                        (job control API)                    │
│  ├── /v1/nodes/*                       (node registry API)                  │
│  ├── /v1/tasks/*                       (task lifecycle API)                 │
│  ├── /orca.v1.Dispatch/...             # NEW (P02) — cross-node dispatch    │
│  └── /orca.v1.Register/...             # NEW (P02) — peer join ack           │
├─────────────────────────────────────────────────────────────────────────────┤
│  Transport Layer (NEW — internal/transport)                                  │
│  ├── mTLS client                       (dialer pool per peer)               │
│  ├── mTLS server config                (TLS 1.3 only, AEAD allowlist)       │
│  ├── Retry+backoff                     (exponential, jittered, capped)       │
│  └── Graceful disconnect               (ctx-aware Conn.Close)               │
├─────────────────────────────────────────────────────────────────────────────┤
│  Core Engine                                                                │
│  ├── Node Registry                     (in-memory + SQLite persistence)     │
│  ├── Task Executor                     (os/exec with WaitDelay, Go 1.25+)   │
│  ├── Job Scheduler                     (single-node FIFO; bin-pack P02)     │
│  ├── Dispatcher                        # NEW internal/engine/dispatcher.go │
│  │   ├── Local decision                (does this job fit on this node?)    │
│  │   ├── Remote dispatch               (POST to peer via transport)         │
│  │   └── Streaming callback            (iter.Seq[DispatchResult] for CLI)   │
│  ├── Security Manager                  # NEW internal/security (P01)        │
│  │   ├── CA lifecycle                  (init, fingerprint, sign CSR)        │
│  │   ├── Server cert lifecycle         (issue, renew, rotate)                │
│  │   ├── mTLS config builder                                              │
│  │   └── Cert store                    (filesystem + SQLite metadata)       │
│  └── Audit Logger                      (log/slog JSON handler)              │
├─────────────────────────────────────────────────────────────────────────────┤
│  State Store (modernc/sqlite, CGO-free)                                     │
│  ~/.orca/orca.db                                                            │
│  ├── nodes, jobs, tasks, audit_log    (v0.1)                                │
│  └── certs                             # NEW (P01) — CA + server certs      │
└─────────────────────────────────────────────────────────────────────────────┘
```

## Component Details

### 1. CLI Layer (`cmd/orca`, `internal/cli`)

- **Framework**: Cobra (industry standard, familiar to operators)
- **v0.1 subcommands**: `version`, `init`, `status`, `node`, `job`, `daemon`
- **v0.2 additions (P01)**: `orca cert {init,join,renew,show}`
- **v0.2 additions (P04)**: `--watch` flag on `orca job list` and `orca node list`
- **v0.2 additions (P01)**: `orca doctor` subcommand (see §5 below)
- **Output**: Human-readable by default; `--json` flag for machine consumption
- **Discovery**: All subcommands self-document via Cobra's auto-generated help
- **Watch semantics (P04)**: `--watch` consumes `iter.Seq[Job|Node]`, exits on
  ctrl-c (via `signal.NotifyContext`), refreshes on internal change events.

### 2. Daemon Layer (`internal/daemon`)

> **⚠️ DEPRECATED in v0.9**: This section describes the v0.8 architecture, superseded by the v0.9 re-architecture. See the "v0.9 Architecture" section at the bottom of this file and `.ciagent/PRD_v0.9.md`.

- **Server**: `net/http` with `http.ServeMux` (no external router)
- **TLS (P01)**: `crypto/tls` with `MinVersion=tls.VersionTLS13` and
  AEAD cipher allowlist
  (`TLS_AES_256_GCM_SHA384`, `TLS_CHACHA20_POLY1305_SHA256`,
  `TLS_AES_128_GCM_SHA256`)
- **Ports**: Configurable (default `:8443` for API+mTLS, `:8080` for health)
- **Graceful Shutdown**: `signal.NotifyContext` with SIGINT/SIGTERM
- **v0.2 endpoints (P02)**:
  - `POST /orca.v1.Dispatch/Submit` — receive cross-node job submission
  - `POST /orca.v1.Dispatch/Status` — query dispatched job status
  - `POST /orca.v1.Register/Hello` — peer join ack (used during `orca node join`)

### 3. Transport Layer (`internal/transport`, NEW in P01/P02)

> **⚠️ DEPRECATED in v0.9**: This section describes the v0.8 architecture, superseded by the v0.9 re-architecture. See the "v0.9 Architecture" section at the bottom of this file and `.ciagent/PRD_v0.9.md`.

- **Client**: `http.Client` with `http.Transport.TLSClientConfig` populated
  from `internal/security.NewClientTLSConfig`
- **Server**: `http.Server.TLSConfig` populated from
  `internal/security.NewServerTLSConfig`
- **Retry policy**: exponential backoff with jitter (start 100ms, x2, cap 5s,
  max 5 attempts); only idempotent verbs (`GET`, `HEAD`, `OPTIONS`) are
  retried automatically; `POST` retries require an explicit
  `X-Orca-Idempotency-Key` header
- **Conn lifecycle**: `context.Context`-aware dials and reads; graceful
  `Close` on `ctx.Done()`
- **Peer dial pool**: small `sync.Map` of `peerID → *http.Client` to reuse
  TLS handshakes (TCP keep-alive) within a session

### 4. Core Engine (`internal/engine`)

> **⚠️ DEPRECATED in v0.9**: This section describes the v0.8 architecture, superseded by the v0.9 re-architecture. The Dispatcher and PeerRegistry peer-dispatch path is replaced by a CLI-side scheduler + SSH-push (R-001). See the "v0.9 Architecture" section at the bottom of this file and `.ciagent/PRD_v0.9.md`.

- **Node Registry**: In-memory map of node IDs → metadata, persisted to SQLite
  (CPU/memory capacity, available slots, last-seen)
- **Task Executor**: `os/exec.CommandContext` with `WaitDelay` (Go 1.25+) for
  clean process termination
- **Job Scheduler (v0.2 P02)**:
  - **Algorithm**: best-fit bin-packing by `available_cpu` and `available_memory`
  - **Within-node ordering**: FIFO queue
  - **Cross-node**: if local node is full, `Dispatcher.Submit(peer, job)` is
    invoked; peers are tried in round-robin order
  - **Fallback**: if all peers reject, return `ErrNoFit` and requeue
- **Dispatcher (NEW, P02)**:
  - `Submit(peerID, spec) (jobID, error)` — blocking call with retry
  - `Watch(peerID) iter.Seq[DispatchEvent]` — pull-style event stream for the
    CLI's `--watch` flag
  - Stateless: every call uses the latest mTLS client config and peer address
- **Security Manager (NEW, P01)**:
  - `InitCA(commonName) (*CA, error)` — generates a self-signed CA, writes
    `ca.crt` (0644) and `ca.key` (0600) to `~/.orca/`
  - `Fingerprint(certPath) (sha256hex, error)` — used by `orca cert join`
  - `SignServerCert(csr, validity) (*cert, error)` — signs a CSR with the CA
  - `IssueServerCert(nodeName, dnsNames, ips) (*cert, *key, error)` — generates
    a keypair + CSR + signs it, returns PEM bytes for `orca cert join --server`
  - `ServerTLSConfig() (*tls.Config, error)` — loads `server.crt`/`server.key`
    and the CA pool from disk
  - `ClientTLSConfig(caPath) (*tls.Config, error)` — returns a client config
    pinned to the supplied CA
  - **Rotation policy**: server certs valid 90d; CA cert valid 10y. On
    `orca cert renew`, `IssueServerCert` is called and the daemon
    gracefully reloads the in-process `tls.Config` via `GetCertificate`
    hot-swap (no restart required)
- **Audit Logger**: `slog.NewJSONHandler(os.Stderr, ...)` with structured fields

### 5. Doctor (`internal/doctor`, NEW in P01)

- **Purpose**: operator-facing diagnostics; runs read-only checks against
  the local state and reports PASS/WARN/FAIL.
- **Subcommands**:
  - `orca doctor` — runs all checks
  - `orca doctor cert` — cert/CA health (file modes, expiry windows, SAN
    presence, fingerprint pinning match — see REQ-026, REQ-033,
    REQ-034, REQ-036)
  - `orca doctor network` — peer reachability over mTLS (per-peer handshake
    sanity, last-seen delta)
  - `orca doctor db` — SQLite integrity check (`PRAGMA integrity_check`)
    + migration version
- **Output**: human-readable by default; `--json` for machine consumption
- **No state changes**: doctor is strictly read-only. It can be run
  while the daemon is down (where possible) or while it's up.
- **Initial implementation in P01** (cert checks only); `network` and
  `db` checks land in subsequent phases as their state becomes
  available.

### 6. State Store (`internal/store`)

- **Driver**: `modernc.org/sqlite` (pure Go, CGO-free)
- **Location**: `~/.orca/orca.db` (user-mode) or `/var/lib/orca/orca.db` (system-mode)
- **Schema (v0.1)**: `nodes`, `jobs`, `tasks`, `audit_log` tables
- **Schema (v0.2 P01)**: NEW `certs` table
  ```sql
  CREATE TABLE certs (
      id            INTEGER PRIMARY KEY AUTOINCREMENT,
      kind          TEXT    NOT NULL,           -- 'ca' | 'server'
      node_id       TEXT,                       -- NULL for CA
      serial_hex    TEXT    NOT NULL,           -- x509.SerialNumber.Hex()
      subject_cn    TEXT    NOT NULL,
      issuer_cn     TEXT    NOT NULL,
      not_before    INTEGER NOT NULL,           -- unix seconds
      not_after     INTEGER NOT NULL,           -- unix seconds
      fingerprint   TEXT    NOT NULL,           -- sha256 of DER, hex
      source_path   TEXT    NOT NULL,           -- on-disk PEM path
      created_at    INTEGER NOT NULL
  );
  CREATE INDEX idx_certs_node_kind ON certs(node_id, kind);
  CREATE INDEX idx_certs_not_after ON certs(not_after);
  ```
- **Migrations**: Embedded SQL files, applied on startup
- **Migration 0004** is added in P01 with the schema above

## Data Model

### Node (v0.1, extended in v0.2 P02)
```go
type Node struct {
    ID        string
    Name      string
    Address   string
    State     string
    JoinedAt  time.Time
    LastSeen  time.Time
    Metadata  map[string]string
    // v0.2 P02 — capacity for bin-packing
    Capacity  NodeCapacity
}

type NodeCapacity struct {
    CPUMillicores int   // total, e.g. 4000 = 4 cores
    MemoryBytes   int64 // total RAM
    CPUUsed       int   // currently allocated
    MemoryUsed    int64 // currently allocated
}

func (c NodeCapacity) AvailableCPU() int       { return c.CPUMillicores - c.CPUUsed }
func (c NodeCapacity) AvailableMemory() int64  { return c.MemoryBytes - c.MemoryUsed }
```

### Job (v0.1)
```go
type Job struct {
    ID        string
    Name      string
    Spec      string
    Status    string
    CreatedAt time.Time
    StartedAt *time.Time
    EndedAt   *time.Time
}
```

### Task (v0.1)
```go
type Task struct {
    ID        string
    JobID     string
    Command   string
    Args      []string
    Env       []string
    PID       int
    ExitCode  int
    Status    string
    CreatedAt time.Time
    StartedAt *time.Time
    EndedAt   *time.Time
}
```

### Certificate (NEW, v0.2 P01)
```go
type Cert struct {
    Kind        CertKind   // CertCA | CertServer
    NodeID      string     // empty for CA
    SerialHex   string
    SubjectCN   string
    IssuerCN    string
    NotBefore   time.Time
    NotAfter    time.Time
    Fingerprint string     // sha256 of DER (hex)
    SourcePath  string     // PEM path on disk
    CreatedAt   time.Time
}
```

## v0.2 Component Graph (ASCII)

```
                                  ┌─────────────────┐
                                  │  Operator Host  │
                                  │  (orca CLI)     │
                                  └────────┬────────┘
                                           │ 1. cert join --ca-fingerprint <sha>
                                           ▼
┌──────────────────────────────────────────────────────────────────────────────┐
│                              NODE A (Bootstrap / CA holder)                  │
│                                                                              │
│  ┌──────────────┐    CSR     ┌────────────────────┐   PEM sign   ┌────────┐  │
│  │ orca cert    │──────────▶│ internal/security   │─────────────▶│  CA    │  │
│  │ {init,join}  │            │ .SignServerCert()  │              │  key   │  │
│  └──────────────┘            └────────────────────┘              │ 0600   │  │
│                                                              ┌──└────────┘  │
│  ┌─────────────────┐                                         │              │
│  │ internal/daemon │ ◀──tls.Config── internal/security        │              │
│  │ (http.Server)   │              .ServerTLSConfig()          │              │
│  │                 │                                         │              │
│  │ /v1/jobs/*      │                                         │              │
│  │ /v1/nodes/*     │                                         │              │
│  │ /orca.v1.*      │                                         │              │
│  └────────┬────────┘                                         │              │
│           │  Submit(job)  (bin-pack)                         │              │
│           ▼                                                  │              │
│  ┌─────────────────┐                                         │              │
│  │ internal/engine │                                         │              │
│  │ .scheduler      │──── if local fits → executor            │              │
│  │ .dispatcher     │──── else → Submit(peer, job) ───────────┼──┐           │
│  └─────────────────┘                                         │  │           │
│                                                              │  │ mTLS      │
└──────────────────────────────────────────────────────────────┼──┼───────────┘
                                                               │  │
                                            ORCA NODE NETWORK │  │
                                                               │  │
┌──────────────────────────────────────────────────────────────┼──┼───────────┐
│                              NODE B (Peer)                   │  │           │
│                                                              │  │           │
│  ┌─────────────────┐  ◀── TLS 1.3 handshake ─────────────────┘  │           │
│  │ internal/daemon │                                              │           │
│  │ (http.Server)   │  POST /orca.v1.Dispatch/Submit              │           │
│  │                 │──── 200 + jobID                              │           │
│  └────────┬────────┘                                              │           │
│           │                                                       │           │
│           ▼                                                       │           │
│  ┌─────────────────┐                                              │           │
│  │ internal/engine │                                              │           │
│  │ .scheduler (FIFO)                                              │           │
│  │ .executor                                                     │           │
│  └─────────────────┘                                              │           │
└──────────────────────────────────────────────────────────────────────────────┘
```

## v0.2 Flows

### Flow 1: Cert Issuance (CA-init → CSR → sign → install) — P01

```
Operator (Node A)                       Operator (Node B)
─────────────────                       ─────────────────
orca cert init
  ↳ InitCA("orca-ca")
  ↳ write ca.crt (0644), ca.key (0600)
  ↳ record in certs table (kind='ca')
                                         orca cert join --ca-fingerprint <sha>
                                           ↳ operator copies ca.crt → Node B
                                           ↳ verifies fingerprint matches local
                                              --ca-fingerprint arg
                                           ↳ IssueServerCert("node-b", SANs)
                                              ↳ generate 2048-bit RSA key
                                              ↳ build CSR with SANs
                                              ↳ read ca.crt + ca.key
                                              ↳ sign CSR (90d validity)
                                           ↳ write server.crt (0644),
                                              server.key (0600)
                                           ↳ record in certs table
                                              (kind='server', node_id='node-b')
```

### Flow 2: mTLS Handshake at `node join` — P01

```
Node B (joiner)                                  Node A (CA holder)
────────────────                                  ─────────────────
orca node join --name node-b
  --ca-fingerprint <sha>
  --peer node-a:8443
  ↳ load ca.crt → verify sha256 == --ca-fingerprint
  ↳ load server.crt + server.key
  ↳ tls.Config{MinVersion: TLS1.3, ...}
  ↳ ClientHello (SNI=node-a)
                                                ◀── ServerHello (TLS 1.3)
                                                ◀── Certificate (Node A's cert)
  ↳ verify Node A's cert chains to ca.crt       ◀── CertificateRequest
  ↳ send Certificate (Node B's cert)            ◀── Finished
  ↳ Finished
  ↳ GET /healthz (over mTLS) — sanity check
  ↳ POST /v1/nodes (over mTLS) — register
                                                 ↳ insert into nodes table
                                                 ↳ audit log
  ↳ 200 OK
  ↳ record node_a in peers table
  ↳ audit log
```

### Flow 3: Job Dispatch (CLI → dispatcher → peer) — P02

```
User (Node A CLI)             Node A (scheduler)            Node B (peer)
──────────────────             ───────────────────            ─────────────
orca job run spec.hcl
  ↳ parse HCL
  ↳ POST /v1/jobs (mTLS, local)
                                ↳ scheduler.Submit(spec)
                                ↳ bin-pack: spec.cpu + spec.mem
                                ↳ local node A has 2000mc + 4GiB free → fit!
                                ↳ executor.Run(spec)
                                ↳ 202 Accepted + jobID
  ↳ returns jobID
```

```
User (Node A CLI)             Node A (scheduler)            Node B (peer)
──────────────────             ───────────────────            ─────────────
                                ↳ scheduler.Submit(spec)
                                ↳ bin-pack: spec.cpu + spec.mem
                                ↳ local node A has 0 free    → NO FIT
                                ↳ dispatcher.Submit(peer="node-b", spec)
                                  ↳ load transport.Client("node-b")
                                  ↳ POST /orca.v1.Dispatch/Submit
                                                                ↳ mTLS handshake
                                                                ↳ authenticate cert
                                                                ↳ scheduler.Submit(spec)
                                                                ↳ executor.Run(spec)
                                                                ↳ 200 OK + jobID
                                  ↳ 200 OK + jobID
                                ↳ return jobID to local caller
  ↳ returns jobID
```

### Flow 4: iter.Seq Streaming (`--watch`) — P04

```
User                              orca job list --watch
────                              ─────────────────────
                                  ctx, cancel := signal.NotifyContext(ctx, os.Interrupt)
                                  defer cancel()
                                  seq := store.Jobs().Watch(ctx)
                                  for job := range seq {
                                    print(job)  // human or --json
                                  }
                                  // ctrl-c → ctx.Done() → seq stops yielding
```

Internally `store.Jobs().Watch(ctx) iter.Seq[Job]` polls the
`jobs` table on a 1s ticker (or subscribes to an in-process
notifier channel) and yields the current snapshot of each job
until `ctx.Done()`. The store repo implements `iter.Seq[Job]`
as a function that takes a `yield func(Job) bool` callback.

## Security Architecture

> **⚠️ DEPRECATED in v0.9**: This section describes the v0.8 internal-CA architecture, superseded by the v0.9 re-architecture (step-ca, D-101/REQ-076). See the "v0.9 Architecture" section at the bottom of this file and `.ciagent/PRD_v0.9.md`.

### Authentication
- **v0.1**: mTLS for all API endpoints (self-signed CA)
- **v0.2 P01**: Internal CA with CSR join (see Flow 1 + 2)
- **v0.2+**: Token-based auth deferred to v0.3+

### Cert Rotation
- Server certs: 90-day validity, rotate at 60 days (30d before expiry)
- CA cert: 10-year validity, manual rotation
- Hot-swap: `tls.Config.GetCertificate` callback re-reads the
  `server.crt`/`server.key` files on each handshake so `orca cert renew`
  takes effect without a daemon restart.

### Audit Logging
- All state-changing operations emit structured log records
- Fields: `timestamp`, `actor`, `action`, `resource`, `result`, `error`
- Stored in SQLite `audit_log` table and stderr (JSON)
- **v0.2 P01 additions**: `cert.issued`, `cert.renewed`, `cert.joined`,
  `node.handshake_ok`, `node.handshake_failed`

### Input Validation
- All CLI inputs validated via Cobra's `Args`/`ValidArgs` functions
- All API inputs validated at handler boundary
- HCL/YAML specs parsed with strict schemas

## Key Architectural Decisions (v0.1 + v0.2)

> **⚠️ DEPRECATED in v0.9**: AD-007 (HCL canonical for jobspecs) below is superseded by R-013/R-014 (Markdown with YAML frontmatter canonical; HCL legacy). See the "v0.9 Architecture" section at the bottom of this file and `.ciagent/PRD_v0.9.md`.

| ID | Decision | Rationale |
|----|----------|-----------|
| AD-001 | Single binary with subcommands | Simpler distribution, aligns with simplicity pillar |
| AD-002 | modernc/sqlite (CGO-free) | Cross-compile friendly, no CGO toolchain needed |
| AD-003 | net/http (no framework) | Stdlib suffices; avoids external router dependency |
| AD-004 | os/exec with WaitDelay (Go 1.25+) | Clean process termination, native to Go |
| AD-005 | Cobra for CLI | Industry standard, familiar to operators |
| AD-006 | slog for logging | Native to Go 1.21+, no external dependency |
| AD-007 | HCL for job specs | Familiar to Nomad/HashiCorp users |
| AD-008 | Single-node scheduling (v0.1) | Multi-node scheduling deferred to v0.2+ |
| AD-009 | Internal CA, no external PKI (v0.2) | Self-contained, no operational PKI requirement |
| AD-010 | Roll-our-own CA in `crypto/x509` (v0.2) | step-ca/cfssl/vault-pki too heavyweight for Orca's footprint |
| AD-011 | Operator-mediated CA cert distribution (v0.2) | No secret distribution over the wire; matches offline-first |
| AD-012 | TLS 1.3 only, AEAD allowlist (v0.2) | Modern crypto only; no downgrade risk |
| AD-013 | Eager mTLS at `node join` (v0.2) | Fail fast; don't defer handshake to first request |
| AD-014 | `orca.v1.Dispatch` via stdlib h2c (v0.2) | ConnectRPC not in go.mod; stdlib suffices for a single-RPC service |
| AD-015 | Best-fit bin-packing (v0.2) | Simple, deterministic, optimal for small fleets |
| AD-016 | iter.Seq for streaming lists (v0.2) | Go 1.25+ native, context-aware, pull semantics |

## Anti-Patterns (Explicitly Avoided)

- No controller/agent split (single binary)
- No CRDs / custom resource definitions
- No web UI (CLI-only)
- No service mesh
- No container runtime integration
- No multi-tenancy
- No cloud provider integrations
- No auto-scaling
- No admission controllers
- No complex scheduling algorithms (best-fit only)
- No gRPC framework dependency (stdlib net/http with h2c, Go 1.25+ native)
- No external PKI / no cert transparency logs (offline-first)

## Dependency Map (minimal — v0.2 adds zero direct deps)

```
github.com/spf13/cobra           # CLI framework
github.com/hashicorp/hcl/v2      # HCL parser
modernc.org/sqlite               # SQLite (pure Go)
github.com/google/uuid           # UUID generation
```

Total: ~4 direct dependencies. No web framework, no ORM, no RPC framework,
no PKI library. mTLS via `crypto/tls` and `crypto/x509` (stdlib).

> **ConnectRPC note**: `.ciagent/config.json` lists `connectrpc` in
> `frameworks`, but the actual `go.mod` does not depend on
> `connectrpc.com/connect`. v0.2 falls back to plain `net/http` with
> HTTP/2 cleartext (h2c) for `orca.v1.Dispatch`. The protocol is a
> simple JSON-over-HTTP POST: client sends
> `{"spec": "..."}` to `/orca.v1.Dispatch/Submit`; server replies
> `{"job_id": "..."}`. This keeps the zero-new-dep promise and the
> codebase coherent with the rest of the daemon's `http.ServeMux`.

## Deployment Model (v0.2)

```
Operator Machine             Node A (CA holder)              Node B (Peer)
────────────────             ─────────────────                ─────────────
orca CLI                     orca daemon                      orca daemon
   │                            │  ▲                            │  ▲
   │ mTLS handshake             │  │ mTLS                       │  │
   │ at `node join` ────────────┼──┘                            │  │
   │                            │                               │  │
   │ submit job ───POST────────▶│  POST (cross-node) ──────────▶│
   │                            │  mTLS only                    │  │
   │                            │                               │  │
   │ ◀───────jobID──────────────│  ◀─────jobID (200 OK)─────────│
   │                            │                               │
   │ orca job list --watch      │                               │
   │ (iter.Seq stream) ◀────────│── polls local + stream events│
```

For v0.2, one node must be the CA holder (`orca cert init` was run
on it). The CA holder's `ca.crt` is copied to each peer manually by
the operator; peers do not auto-fetch it.

## v0.6 Architecture Addendum — Node Bootstrap & Proxmox

### `orca init` Full Bootstrap (REQ-047, REQ-048, REQ-049)

`orca init` transforms from a bare `mkdir` into a full single-node
cluster bootstrap. The sequence (idempotent per D-036):

```
orca init
  1. MkdirAll(certpaths.Dir(), 0o755)              # namespace dir
  2. store.Open(certpaths.DBPath())                # runs migrations 0001..0006
  3. security.CAInit(dir, "orca-internal-ca")      # idempotent fast-path
  4. if !exists(server.crt):
       GenerateCSR("localhost", ["localhost","127.0.0.1"])
       ca.SignCSR(csr) → WriteCert + WriteKey      # server cert (skip if present)
  5. os := detectOS()                              # /etc/os-release ID=
  6. node := Node{kind:"localhost", os:os, name:"localhost", addr:"localhost:8443"}
     if GetByName("localhost") exists:
       UpdateLastSeenAndOS(id, os)                 # refresh, keep id/joined_at
     else:
       NodeRepo.Insert(node)                       # first-run insert
  7. print summary (CA fp, server cert fp, os, node id)
```

After `orca init`, `orca doctor` MUST pass with zero FAILs.

### Node Schema Extension (REQ-049)

Migration 0006 adds two nullable columns to `nodes`:

```sql
ALTER TABLE nodes ADD COLUMN kind TEXT;  -- localhost | linux | proxmox
ALTER TABLE nodes ADD COLUMN os TEXT;    -- ubuntu | debian | alpine | pve | linux
```

Existing rows get SQL NULL → mapped to `""` in Go (`sql.NullString`).
`Node` struct gains `Kind string` + `OS string` fields (JSON tags
`kind,omitempty` / `os,omitempty`). `NodeRepo` extends all
INSERT/SELECT/scanNode calls; adds `GetByName(ctx, name)` and
`UpdateLastSeenAndOS(ctx, id, os)` helpers.

### Proxmox SSH Bootstrap (REQ-050, REQ-051)

```
orca node join --type proxmox --host <addr> --user root --password <pw>
  │ password from --password or $ORCA_PROXMOX_PASSWORD (never persisted, D-031)
  ▼
internal/proxmox.BootstrapProxmox(ctx, opts)
  1. GenerateOrLoadSSHKey(certpaths.Dir())         # Ed25519, ~/.orca/orca_ssh_key{,.pub}
  2. SSH dial (password auth, knownhosts.New TOFU) # capture host key on first connect
  3. Deploy pubkey → ~orca/.ssh/authorized_keys    # via session heredoc (no SFTP dep)
  4. useradd -m orca                                # create Linux system user (config-overridable name)
  5. pveum role add OrcaOperator --privs "VM.Audit Datastore.AllocateSpace SDN.Use"
     (idempotent: probe pveum role list first)
  6. pveum user add orca@pam -comment "Orca automation user"
     (idempotent: probe pveum user list first)
  7. pveum acl modify / -user orca@pam -role OrcaOperator
     (idempotent: modify creates or updates)
  8. Write /etc/sudoers.d/orca (mode 0440):
       orca ALL=(root) NOPASSWD: NOEXEC: /usr/bin/pct, /usr/bin/qm
       orca ALL=(root) NOPASSWD: /usr/bin/apt-get, /usr/bin/dpkg
  9. visudo -cf /etc/sudoers.d/orca                 # validate; abort on error
  10. NodeRepo.Insert(Node{kind:"proxmox", os:"pve", name:host, addr:host})
  11. Audit log: proxmox.bootstrap_ok (host, user, role, fp)
```

**`pvesh` excluded from sudoers** — `pvesh` can trigger the API
`/nodes/{node}/execute` endpoint which spawns shell commands
server-side, bypassing sudo's `NOEXEC` tag. API access is via the
`OrcaOperator` PVE role + `orca@pam` user (PVE RBAC), not sudo'd `pvesh`.

### Doctor Extensions (REQ-052)

- **`doctor os`**: re-runs `detectOS()` from `/etc/os-release`, compares
  to the stored localhost node's `os` field. Drift = WARN (OS upgraded
  since init? re-run `orca init` to refresh). Match = PASS.
- **`doctor proxmox`**: iterates `kind=proxmox` nodes, SSH-probes each
  with `pveversion` (3s timeout per peer, clones `doctor.Network()`
  pattern). PASS = reachable + pveversion exits 0. WARN = zero proxmox
  nodes (single-node cluster is legitimate). FAIL = any node
  unreachable or pveversion fails.

### SSH Key Handling (D-037)

- **Location**: `~/.orca/orca_ssh_key` (0600) + `~/.orca/orca_ssh_key.pub` (0644)
- **Algorithm**: Ed25519 (smaller, faster, more secure than RSA for SSH)
- **Generation**: lazy — on first `orca node join --type proxmox`, NOT at `orca init` (localhost doesn't need SSH)
- **Format**: PKCS8 PEM (consistent with `ca.key`/`server.key`; `ssh.ParsePrivateKey` accepts it)
- **TOFU host keys**: `~/.orca/known_hosts` (OpenSSH format via `knownhosts.New`)

### Dependency Map (v0.6 addition)

```
golang.org/x/crypto v0.54.0        # SSH (ssh + ssh/knownhosts + ed25519)
  └─ golang.org/x/sys v0.47.0      # indirect (bumped from v0.42.0)
  └─ golang.org/x/term v0.45.0     # indirect (pulled by ssh for PTY)
```

Total direct deps: 5 (was 4). One new direct dep (`x/crypto`). Matches
D-030 minimal-deps rationale. No SFTP module (file upload via session
heredoc).

### v0.6 Architectural Decisions (AD-017..AD-021)

| ID | Decision | Rationale |
|----|----------|-----------|
| AD-017 | `orca init` = full bootstrap (CA + cert + db + localhost node) | Single command produces a working cluster; `orca doctor` passes post-init. Idempotent (D-036). |
| AD-018 | Proxmox join via SSH (golang.org/x/crypto/ssh), not PVE REST API | SSH is the universal Proxmox management entry point; REST API would require API token bootstrap (chicken-and-egg). One new direct dep (D-030). |
| AD-019 | `orca@pam` realm (not `orca@pve`) | SSH creates a Linux system user; PAM realm maps it to PVE RBAC without a separate PVE password. `@pve` requires interactive password prompt over non-PTY SSH (hangs). |
| AD-020 | Exclude `pvesh` from sudoers; NOEXEC on `pct`/`qm` | `pvesh` can trigger API execute endpoint bypassing NOEXEC. `pct`/`qm` are Perl scripts via dynamically-linked perl → NOEXEC effective. `apt-get`/`dpkg` need exec for maintainer scripts → no NOEXEC. |
| AD-021 | TOFU host-key via `knownhosts.New` | Avoids deprecated `ssh.InsecureIgnoreHostKey`. Capture-on-first-connect, verify-on-subsequent. Fail closed on mismatch (operator runs key-reset). |

---

# v0.9 Architecture (Supersedes v0.8)

> **⚠️ v0.9 DIRECTION CHANGE**: This section supersedes the v0.1–v0.8
> architecture described above. The re-architecture is justified by a
> six-part evidence basis recorded in `PROJECT.md` (Supersession Table).
> The v0.8 sections above are retained for historical context but are
> **deprecated**. The 16 load-bearing rules (R-001…R-016) in
> `PRD_v0.9.md` are now the canonical invariants.

## Superseded Decisions (AD-series reversals)

| Old decision | Was | Superseded by | Evidence basis |
|---|---|---|---|
| AD-010 (line 463 above) | step-ca/cfssl/vault-pki "too heavyweight" | **D-101** (step-ca) | External PKI mandate (override ground 2) |
| SPIFFE rejection (line 94, PROJECT.md) | internal CA chosen over SPIFFE | **D-068** (SPIFFE SVIDs) | Multi-tenancy requires per-workload identity (override ground 3) |
| No-container-runtime (line 477 above) | explicit anti-pattern | **D-088** (5 runtimes; wasmtime primary) | WASM is the workload profile (override ground 4) |
| No-multi-tenancy (line 478 above) | explicit anti-pattern | **D-158 / R-002** (multi-namespace) | Hard multi-tenant product req (override ground 3) |
| AD-007 (HCL canonical) | HCL for jobspec | **R-013 / R-014** (Markdown canonical; HCL legacy) | PRD §8 operator-facing format |
| Daemon-on-every-node | `orca daemon` on all peers | **R-001** (no orca binary on any server) | Daemon operationally failing + SSH-push only viable target (override grounds 1 + 5) |

## The Five-Layer CLI (v0.9)

The `orca` binary is one Go program, structured internally as five layers:

1. **CLI subcommand tree** (cobra) — `internal/cli/`
2. **Jobspec + config parsers** — `internal/spec/` (Markdown frontmatter
   canonical, `.md`/`.yaml`/`.hcl` dispatcher per R-013/R-014)
3. **Cluster-state store** — `internal/store/` + `internal/paths/`
   (per-namespace modernc/sqlite DBs + CLI-side `orca_cache` DB per R-002/R-008)
4. **Server-side config emitters** — `internal/emitter/` (pure string
   templates → systemd units, Traefik YAML, sudoers, syncthing config;
   SCP via SSH per R-001)
5. **Workflow orchestrators** — `internal/sshpush/` (compose SSH + local FS
   writes into multi-step commands)

## The Server Side (R-001 — no Orca binary on any server)

Servers hold only: rendered config in `/etc/orca/actual/<txn-id>/`,
systemd units, Traefik dynamic config, sudoers, sshd_config snippets,
`step-ca`/`traefik`/`syncthing`/`podman`/`wasmtime`/`age`/`auditd`
(installed via apt), and bash scripts in `scripts/` (orca-pull.sh,
orca-drift.sh, orca-collect.sh, orca-aggregate.sh, orca-apply-render.sh,
orca-verify-render.sh, orca-rollback-render.sh, orca-cleanup-credentials.sh).
Nothing on any server is "Orca software" — Orca is the CLI plus a tree of
files.

## Multi-namespace Layout (R-002)

```
$ORCA_HOME/
├── cluster/                  # cluster-wide (NOT a workload namespace)
│   ├── ca.crt, ca.key        # step-ca root (R-006, D-101)
│   ├── master.key            # AES-256-GCM root (R-011, mode 0600)
│   ├── config.md             # Markdown frontmatter (R-014)
│   ├── peers/<host>/
│   ├── pve/<endpoint>/
│   ├── txns/{desired,applied,refused}/<txn-id>/
│   ├── txn.sqlite
│   └── state/
├── _defaults/                # implicit root namespace (always exists)
│   ├── ns.md
│   ├── .env, .env.secrets
│   ├── db/orca.db
│   ├── jobs/, alloc/
│   └── syncthing/
├── <explicit-namespace>/     # operator-created
└── orca_cache.db            # CLI-side cache (R-008)
```

## Execution gates (from GRILL_v0.9.md)

The 19 binding conditions (C-01..C-19) and 10 phase challenges
(PC-01..PC-10) gate specific phases. See `GRILL_v0.9.md` for the full
list. Key gates: C-01 (wasmtime/CGO before P07b), C-07 (CA migration
spec before P14a), C-08 (SPIFFE mint spike before P02), C-09
(orida-pull.sh failure contract before P10), C-19 (threat model before
P15.5).

## v0.9–v0.12 Component Addendum (post-rearchitecture packages)

The v0.9 re-architecture introduced the SSH-push model and split the
monolithic v0.8 transport layer into focused packages. The following
packages were added or substantially expanded across v0.9–v0.12 and are
part of the canonical component graph:

### Workload & runtime layer
- `internal/runtime/` — runtime abstraction (process/podman/wasm/pve-vm/pve-ct), 5 backends (REQ-078, C-01)
- `internal/scheduler/` — CLI-side scheduler, CEL constraints, affinity (REQ-083)
- `internal/jobspec/` — job specification parsing & validation
- `internal/spec/` — update stanza + lifecycle hooks
- `internal/engine/` — dispatcher, executor, peer, registry, audit, scheduler

### State & persistence layer
- `internal/model/` — core data model (Node, Job, Task, Certificate, Alloc)
- `internal/store/` — cluster-state store, per-namespace modernc/sqlite
- `internal/paths/` — path resolution for the multi-namespace layout (R-002)
- `internal/certpaths/` — certificate path helpers (known_hosts, CA material)
- `internal/cache/` — CLI-side orca_cache SQLite (R-008)
- `internal/migration/` — v0.8→v1.0 data migration (REQ-066, C-07)
- `internal/txn/` — transactional plane, apply-path allowlist (REQ-075, REQ-079)
- `internal/ns/` — namespace subcommands, inheritance, constraints (REQ-068)

### Transport & bootstrap layer
- `internal/sshpush/` — v0.9 SSH-push transport, fanout, idempotency (R-001, C-18)
- `internal/cluster/` — lead rules, rotate-lead, mixed-version tolerance
- `internal/proxmox/` — Proxmox API + host-key TOFU (D-035)
- `internal/stepca/` — step-ca integration (REQ-076)
- `internal/storage/` — Syncthing storage replication + conflict resolution (REQ-081)
- `internal/backup/` — backup/restore, signed tarball (HMAC-SHA256)
- `internal/secrets/` — per-namespace AES-256-GCM + HKDF-SHA256 (REQ-080)
- `internal/emit/` — emit contract (systemd units, Traefik YAML, sudoers, syncthing)
- `internal/emitter/` — server-side config emitters (renders `internal/emit` contract)
- `internal/osdetect/` — OS detection for renderer dispatch (R-013/R-014)

### Drift detection layer
- `internal/drift/` — drift detection collector + aggregator (REQ-103..113; R-018/R-019/R-020)

### Security & identity layer (v0.12 — Zero-Trust Identity)
- `internal/identity/` — OIDC client + auth CLI (REQ-144)
- `internal/seal/` — master key seal-to-OIDC + Shamir 3-of-5 (REQ-147, D-241, C-35)
- `internal/webauthn/` — WebAuthn connector for Dex (REQ-148, D-240, C-38)
- `internal/acl/` — ACL rewrite to OIDC claims, deny-by-default (REQ-122, REQ-145)
- `internal/audit/` — audit log tamper-evidence (REQ-125, F2)
- `internal/security/` — SVID chain validation, daemon auth, file-mode enforcement (REQ-123, REQ-124, REQ-126)
- `internal/config/` — cluster config parsing, frontmatter dispatch (R-014)

### Deprecated / dual-write (removed in v1.x)
- `internal/transport/` — v0.8 mTLS HTTP layer; superseded by `internal/sshpush/` (dual-write window closed in v0.12 P07; full deletion deferred to v1.x per P23_DUAL_WRITE_DECISION.md)

## Execution gates (v0.12)

The v0.12 milestone is gated by binding conditions C-29..C-38 (see
GRILL_v0.12.md). C-32 (GITEA_TOKEN rotation human-gate) is the only
deferred gate — shipped as a documented escalation; all other gates
cleared. The load-bearing rule is R-021 (no Orca password/token paths).
---

## v0.13 Architecture Deltas — Production Hardening Round 2

### R-022: Scheduler/Deployment Wiring

`orca job run` now deploys to remote nodes via the pipeline:
```
scheduler.Schedule(spec, nodes) → emitter.Render(unit) → sshpush.Deploy(target, unit)
```
- The local `exec.CommandContext` path in `internal/engine/executor.go`
  is removed for the dispatch path. Local execution is the fallback
  when no remote nodes are registered (single-node dev mode).
- `internal/scheduler.Schedule()` evaluates CEL constraints, capacity
  fit, and affinity scoring against registered nodes.
- `internal/emitter/systemd.go` renders the unit; `systemd-analyze
  verify` validates before deploy.
- `internal/sshpush` pushes the unit + env file to the target node.
- `--target <node>` overrides scheduler selection (manual pinning).
- Without `--target`, the scheduler bin-packs across all `ready` nodes.

### R-023: Zero-Trust Enforcement Wiring

`acl.Check` is invoked on every request path:
- **Daemon handlers** (`dispatch`/`jobs`/`nodes`/`tasks`/`health`):
  extract OIDC `sub`/SPIFFE SVID from mTLS peer cert → `acl.Check(acl,
  identity, namespace, verb)` → deny-by-default.
- **SSH-push applier** (`internal/sshpush/`): validate `ORCA_OIDC_TOKEN`
  bearer against JWKS before applying any txn.
- **Txn apply** (`internal/txn/`): same bearer validation.
- Audit `actor` field carries the OIDC `sub` or SPIFFE SVID (not
  "cli"/"daemon").
- `acl.json` mode is 0600 (not 0644).
- WebAuthn registration (`/orca/webauthn/register`) requires an
  existing authenticated session or admin bootstrap token.

### New Components

- `internal/linux/bootstrap.go` — Ubuntu/Debian SSH-join (mirrors
  `internal/proxmox/bootstrap.go` without PVE role/sudoers). Deploys
  orca pubkey, creates `orca` system user, creates drift-events dir.
  Key-auth only (R-021). Invoked via `orca node join --type linux`.
- `internal/cli/cluster_seal.go` — `orca cluster seal`/`unseal` CLI
  (wraps `internal/seal/` library; OIDC token exchange → unwrap master
  key → zeroed on shutdown; Shamir 3-of-5 shards at seal time).
- `internal/cli/doctor_audit.go` — `orca doctor audit` (wraps
  `AuditRepo.VerifyChain`).
- `internal/cli/doctor_modes.go` — `orca doctor modes` (wraps
  `EnforceFileModes` across ORCA_HOME).

### New Artifacts

- `docs/uat.md` — UAT plan (3-host topology, step-by-step, claim matrix)
- `scripts/uat-signoff.sh` — v1.0 gate signoff script (~35 assertions,
  idempotent, read-only)
- `scripts/uat-smoke.sh` — CI-tested pure-CLI subset of signoff
- `docs/metrics.md` — expanded Prometheus metric set reference

### jobspec Parser Fixes

- `schedule:` and `timeout:` now parsed at top level (previously
  silently dropped by the markdown parser's default case).
- DaemonSet: parser no longer defaults `Count` to 1 (validator rejects
  `Count != 0` for DaemonSet).
- `restart:` policy translated to systemd `Restart=`/`StartLimitBurst`
  in the emitter.
- `job lint` emits honest "not enforced in this version" warnings for
  advisory-only fields (cron, health, update, affinity).

### Concurrency Safety

- All SQLite DSNs set `busy_timeout(5000)` + `SetMaxOpenConns(1)`.
- Secrets file flock prevents concurrent-write data loss.
- Upgrade/backup lock files prevent concurrent cutover/clobber.
- Cache invalidated by write commands (read-after-write consistency).
- Audit `Append` uses `BEGIN IMMEDIATE` transaction (chain race fixed).
- WebAuthn session stores guarded with `sync.Mutex`.

### Transport Safety

- Typed sentinels replace substring matching in both `transport` and
  `sshpush` packages.
- `rotateSSHKeys` 2-phase atomic swap (stage → swap → verify → cleanup).
- IPv6 `net.JoinHostPort` in all SSH dial paths.
- Explicit timeouts on all SSH commands.
- Root SIGINT/SIGTERM handler for clean exit on non-watch commands.

## v0.14 Deltas — Ingress Bootstrap Completeness (R-024)

### R-024: Traefik as Podman Container

Traefik runs exclusively as a podman container, deployed from the
custom `orca-traefik` image (published per release via `Dockerfile.traefik`
+ `scripts/release.sh` + `.coreci.yml container-publish-traefik`).

The v0.13 binary+systemd install (`internal/traefik/install.go`) is
replaced by an idempotent podman container reconciler
(`EnsureTraefikContainerLocal`/`Remote`). The container runs with
`--network host`, `--restart=unless-stopped`, and volume mounts for
`traefik.yml` (static config), `dynamic` (dynamic config), and
`step-ca-root.crt` (future mTLS). No SELinux `:Z` flag.

### Three Ingress Topologies

1. **Linux** (`orca init` / `orca node join --type linux`):
   host → nft DNAT → podman traefik (host network).
   `internal/ingress/bootstrap.go` → `BootstrapLocalIngress` /
   `BootstrapRemoteIngress`.

2. **Proxmox Native** (`--ingress-mode native`, default):
   PVE host → nft DNAT (target = LXC bridge IP) → LXC
   (`--features nesting=1,keyctl=1,fuse=1`) → podman traefik.
   `internal/proxmox/bootstrap.go` → `provisionNativeIngressLXC`.

3. **Proxmox Floating-IP** (`--ingress-mode floating-ip`):
   LXC owns the floating IP (`net0 bridge=vmbr0,hwaddr=<mac>,
   ip=<floating-ip>/<prefix>,gw=<gateway>`) → nft inside LXC →
   podman traefik. The ingress LXC is registered as a `linux` node
   (name=`ingress`) so `orca job run` pushes traefik dynamic config.
   `internal/proxmox/ingress_lxc.go` → `ProvisionIngressLXC`.

### nft Emitter Changes

`internal/emitter/nft.go`:
- `DNATTarget` field (C-51: validated via `net.ParseIP`). Default
  `127.0.0.1`; proxmox native uses LXC bridge IP.
- `EnableSNAT` field + postrouting masquerade chain: `ip saddr
  127.0.0.0/8 oifname != "lo" masquerade` (research Topic 1).
- Input/forward chain priority shifted from `filter` (=0) to `-10`
  (research Topic 2: pve-firewall coexistence — avoids same-priority
  undefined evaluation order).

### TLS Model

v0.14 drops `certResolver: orca` from the dynamic config (traefik v3.3
only supports `acme`/`tailscale` resolvers, not CA-file-based). The
dynamic config emits `tls: {}` (traefik default cert). Real mTLS via
`tls.certificates` + `tls.options.default.clientAuth.caFiles` is
deferred to v0.15 (grill G-003, confidence 0.55 < 0.60).

### Migration 0009

`ALTER TABLE nodes ADD COLUMN ingress_mode TEXT NOT NULL DEFAULT '';`
Values: `""` (legacy), `"native"`, `"floating-ip"`. `IngressMode` field
on `model.Node`.

### New CLI

- `orca doctor ingress` — verifies podman container running, nft
  DNAT+SNAT, dynamic dir, step-ca root CA.
- `--ingress-mode` flag on `orca node join --type proxmox`.
- `--floating-ip`, `--gateway`, `--mac`, `--net-prefix` flags for
  floating-IP mode.
