Compare commits
8 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 ciagent
|
ba5ffd76f9 | ||
|
ciagent
|
9b308c79f4 | ||
|
ciagent
|
a7bb00d935 | ||
|
ciagent
|
b4d9409e4d | ||
|
ciagent
|
efdbd2a61d | ||
|
ciagent
|
5755f12053 | ||
|
ciagent
|
5dba3cef80 | ||
|
ciagent
|
fc6a6c07e2 |
@@ -79,8 +79,6 @@ and a **dispatcher** for multi-node job execution.
|
||||
|
||||
### 2. Daemon Layer (`internal/daemon`)
|
||||
|
||||
> **⚠️ DEPRECATED in v0.9**: This section describes the v0.8 architecture, superseded by the v0.9 re-architecture. See the "v0.9 Architecture" section at the bottom of this file and `.ciagent/PRD_v0.9.md`.
|
||||
|
||||
- **Server**: `net/http` with `http.ServeMux` (no external router)
|
||||
- **TLS (P01)**: `crypto/tls` with `MinVersion=tls.VersionTLS13` and
|
||||
AEAD cipher allowlist
|
||||
@@ -95,8 +93,6 @@ and a **dispatcher** for multi-node job execution.
|
||||
|
||||
### 3. Transport Layer (`internal/transport`, NEW in P01/P02)
|
||||
|
||||
> **⚠️ DEPRECATED in v0.9**: This section describes the v0.8 architecture, superseded by the v0.9 re-architecture. See the "v0.9 Architecture" section at the bottom of this file and `.ciagent/PRD_v0.9.md`.
|
||||
|
||||
- **Client**: `http.Client` with `http.Transport.TLSClientConfig` populated
|
||||
from `internal/security.NewClientTLSConfig`
|
||||
- **Server**: `http.Server.TLSConfig` populated from
|
||||
@@ -112,8 +108,6 @@ and a **dispatcher** for multi-node job execution.
|
||||
|
||||
### 4. Core Engine (`internal/engine`)
|
||||
|
||||
> **⚠️ DEPRECATED in v0.9**: This section describes the v0.8 architecture, superseded by the v0.9 re-architecture. The Dispatcher and PeerRegistry peer-dispatch path is replaced by a CLI-side scheduler + SSH-push (R-001). See the "v0.9 Architecture" section at the bottom of this file and `.ciagent/PRD_v0.9.md`.
|
||||
|
||||
- **Node Registry**: In-memory map of node IDs → metadata, persisted to SQLite
|
||||
(CPU/memory capacity, available slots, last-seen)
|
||||
- **Task Executor**: `os/exec.CommandContext` with `WaitDelay` (Go 1.25+) for
|
||||
@@ -429,8 +423,6 @@ as a function that takes a `yield func(Job) bool` callback.
|
||||
|
||||
## Security Architecture
|
||||
|
||||
> **⚠️ DEPRECATED in v0.9**: This section describes the v0.8 internal-CA architecture, superseded by the v0.9 re-architecture (step-ca, D-101/REQ-076). See the "v0.9 Architecture" section at the bottom of this file and `.ciagent/PRD_v0.9.md`.
|
||||
|
||||
### Authentication
|
||||
- **v0.1**: mTLS for all API endpoints (self-signed CA)
|
||||
- **v0.2 P01**: Internal CA with CSR join (see Flow 1 + 2)
|
||||
@@ -457,8 +449,6 @@ as a function that takes a `yield func(Job) bool` callback.
|
||||
|
||||
## Key Architectural Decisions (v0.1 + v0.2)
|
||||
|
||||
> **⚠️ DEPRECATED in v0.9**: AD-007 (HCL canonical for jobspecs) below is superseded by R-013/R-014 (Markdown with YAML frontmatter canonical; HCL legacy). See the "v0.9 Architecture" section at the bottom of this file and `.ciagent/PRD_v0.9.md`.
|
||||
|
||||
| ID | Decision | Rationale |
|
||||
|----|----------|-----------|
|
||||
| AD-001 | Single binary with subcommands | Simpler distribution, aligns with simplicity pillar |
|
||||
@@ -536,406 +526,3 @@ orca CLI orca daemon orca daemon
|
||||
For v0.2, one node must be the CA holder (`orca cert init` was run
|
||||
on it). The CA holder's `ca.crt` is copied to each peer manually by
|
||||
the operator; peers do not auto-fetch it.
|
||||
|
||||
## v0.6 Architecture Addendum — Node Bootstrap & Proxmox
|
||||
|
||||
### `orca init` Full Bootstrap (REQ-047, REQ-048, REQ-049)
|
||||
|
||||
`orca init` transforms from a bare `mkdir` into a full single-node
|
||||
cluster bootstrap. The sequence (idempotent per D-036):
|
||||
|
||||
```
|
||||
orca init
|
||||
1. MkdirAll(certpaths.Dir(), 0o755) # namespace dir
|
||||
2. store.Open(certpaths.DBPath()) # runs migrations 0001..0006
|
||||
3. security.CAInit(dir, "orca-internal-ca") # idempotent fast-path
|
||||
4. if !exists(server.crt):
|
||||
GenerateCSR("localhost", ["localhost","127.0.0.1"])
|
||||
ca.SignCSR(csr) → WriteCert + WriteKey # server cert (skip if present)
|
||||
5. os := detectOS() # /etc/os-release ID=
|
||||
6. node := Node{kind:"localhost", os:os, name:"localhost", addr:"localhost:8443"}
|
||||
if GetByName("localhost") exists:
|
||||
UpdateLastSeenAndOS(id, os) # refresh, keep id/joined_at
|
||||
else:
|
||||
NodeRepo.Insert(node) # first-run insert
|
||||
7. print summary (CA fp, server cert fp, os, node id)
|
||||
```
|
||||
|
||||
After `orca init`, `orca doctor` MUST pass with zero FAILs.
|
||||
|
||||
### Node Schema Extension (REQ-049)
|
||||
|
||||
Migration 0006 adds two nullable columns to `nodes`:
|
||||
|
||||
```sql
|
||||
ALTER TABLE nodes ADD COLUMN kind TEXT; -- localhost | linux | proxmox
|
||||
ALTER TABLE nodes ADD COLUMN os TEXT; -- ubuntu | debian | alpine | pve | linux
|
||||
```
|
||||
|
||||
Existing rows get SQL NULL → mapped to `""` in Go (`sql.NullString`).
|
||||
`Node` struct gains `Kind string` + `OS string` fields (JSON tags
|
||||
`kind,omitempty` / `os,omitempty`). `NodeRepo` extends all
|
||||
INSERT/SELECT/scanNode calls; adds `GetByName(ctx, name)` and
|
||||
`UpdateLastSeenAndOS(ctx, id, os)` helpers.
|
||||
|
||||
### Proxmox SSH Bootstrap (REQ-050, REQ-051)
|
||||
|
||||
```
|
||||
orca node join --type proxmox --host <addr> --user root --password <pw>
|
||||
│ password from --password or $ORCA_PROXMOX_PASSWORD (never persisted, D-031)
|
||||
▼
|
||||
internal/proxmox.BootstrapProxmox(ctx, opts)
|
||||
1. GenerateOrLoadSSHKey(certpaths.Dir()) # Ed25519, ~/.orca/orca_ssh_key{,.pub}
|
||||
2. SSH dial (password auth, knownhosts.New TOFU) # capture host key on first connect
|
||||
3. Deploy pubkey → ~orca/.ssh/authorized_keys # via session heredoc (no SFTP dep)
|
||||
4. useradd -m orca # create Linux system user (config-overridable name)
|
||||
5. pveum role add OrcaOperator --privs "VM.Audit Datastore.AllocateSpace SDN.Use"
|
||||
(idempotent: probe pveum role list first)
|
||||
6. pveum user add orca@pam -comment "Orca automation user"
|
||||
(idempotent: probe pveum user list first)
|
||||
7. pveum acl modify / -user orca@pam -role OrcaOperator
|
||||
(idempotent: modify creates or updates)
|
||||
8. Write /etc/sudoers.d/orca (mode 0440):
|
||||
orca ALL=(root) NOPASSWD: NOEXEC: /usr/bin/pct, /usr/bin/qm
|
||||
orca ALL=(root) NOPASSWD: /usr/bin/apt-get, /usr/bin/dpkg
|
||||
9. visudo -cf /etc/sudoers.d/orca # validate; abort on error
|
||||
10. NodeRepo.Insert(Node{kind:"proxmox", os:"pve", name:host, addr:host})
|
||||
11. Audit log: proxmox.bootstrap_ok (host, user, role, fp)
|
||||
```
|
||||
|
||||
**`pvesh` excluded from sudoers** — `pvesh` can trigger the API
|
||||
`/nodes/{node}/execute` endpoint which spawns shell commands
|
||||
server-side, bypassing sudo's `NOEXEC` tag. API access is via the
|
||||
`OrcaOperator` PVE role + `orca@pam` user (PVE RBAC), not sudo'd `pvesh`.
|
||||
|
||||
### Doctor Extensions (REQ-052)
|
||||
|
||||
- **`doctor os`**: re-runs `detectOS()` from `/etc/os-release`, compares
|
||||
to the stored localhost node's `os` field. Drift = WARN (OS upgraded
|
||||
since init? re-run `orca init` to refresh). Match = PASS.
|
||||
- **`doctor proxmox`**: iterates `kind=proxmox` nodes, SSH-probes each
|
||||
with `pveversion` (3s timeout per peer, clones `doctor.Network()`
|
||||
pattern). PASS = reachable + pveversion exits 0. WARN = zero proxmox
|
||||
nodes (single-node cluster is legitimate). FAIL = any node
|
||||
unreachable or pveversion fails.
|
||||
|
||||
### SSH Key Handling (D-037)
|
||||
|
||||
- **Location**: `~/.orca/orca_ssh_key` (0600) + `~/.orca/orca_ssh_key.pub` (0644)
|
||||
- **Algorithm**: Ed25519 (smaller, faster, more secure than RSA for SSH)
|
||||
- **Generation**: lazy — on first `orca node join --type proxmox`, NOT at `orca init` (localhost doesn't need SSH)
|
||||
- **Format**: PKCS8 PEM (consistent with `ca.key`/`server.key`; `ssh.ParsePrivateKey` accepts it)
|
||||
- **TOFU host keys**: `~/.orca/known_hosts` (OpenSSH format via `knownhosts.New`)
|
||||
|
||||
### Dependency Map (v0.6 addition)
|
||||
|
||||
```
|
||||
golang.org/x/crypto v0.54.0 # SSH (ssh + ssh/knownhosts + ed25519)
|
||||
└─ golang.org/x/sys v0.47.0 # indirect (bumped from v0.42.0)
|
||||
└─ golang.org/x/term v0.45.0 # indirect (pulled by ssh for PTY)
|
||||
```
|
||||
|
||||
Total direct deps: 5 (was 4). One new direct dep (`x/crypto`). Matches
|
||||
D-030 minimal-deps rationale. No SFTP module (file upload via session
|
||||
heredoc).
|
||||
|
||||
### v0.6 Architectural Decisions (AD-017..AD-021)
|
||||
|
||||
| ID | Decision | Rationale |
|
||||
|----|----------|-----------|
|
||||
| AD-017 | `orca init` = full bootstrap (CA + cert + db + localhost node) | Single command produces a working cluster; `orca doctor` passes post-init. Idempotent (D-036). |
|
||||
| AD-018 | Proxmox join via SSH (golang.org/x/crypto/ssh), not PVE REST API | SSH is the universal Proxmox management entry point; REST API would require API token bootstrap (chicken-and-egg). One new direct dep (D-030). |
|
||||
| AD-019 | `orca@pam` realm (not `orca@pve`) | SSH creates a Linux system user; PAM realm maps it to PVE RBAC without a separate PVE password. `@pve` requires interactive password prompt over non-PTY SSH (hangs). |
|
||||
| AD-020 | Exclude `pvesh` from sudoers; NOEXEC on `pct`/`qm` | `pvesh` can trigger API execute endpoint bypassing NOEXEC. `pct`/`qm` are Perl scripts via dynamically-linked perl → NOEXEC effective. `apt-get`/`dpkg` need exec for maintainer scripts → no NOEXEC. |
|
||||
| AD-021 | TOFU host-key via `knownhosts.New` | Avoids deprecated `ssh.InsecureIgnoreHostKey`. Capture-on-first-connect, verify-on-subsequent. Fail closed on mismatch (operator runs key-reset). |
|
||||
|
||||
---
|
||||
|
||||
# v0.9 Architecture (Supersedes v0.8)
|
||||
|
||||
> **⚠️ v0.9 DIRECTION CHANGE**: This section supersedes the v0.1–v0.8
|
||||
> architecture described above. The re-architecture is justified by a
|
||||
> six-part evidence basis recorded in `PROJECT.md` (Supersession Table).
|
||||
> The v0.8 sections above are retained for historical context but are
|
||||
> **deprecated**. The 16 load-bearing rules (R-001…R-016) in
|
||||
> `PRD_v0.9.md` are now the canonical invariants.
|
||||
|
||||
## Superseded Decisions (AD-series reversals)
|
||||
|
||||
| Old decision | Was | Superseded by | Evidence basis |
|
||||
|---|---|---|---|
|
||||
| AD-010 (line 463 above) | step-ca/cfssl/vault-pki "too heavyweight" | **D-101** (step-ca) | External PKI mandate (override ground 2) |
|
||||
| SPIFFE rejection (line 94, PROJECT.md) | internal CA chosen over SPIFFE | **D-068** (SPIFFE SVIDs) | Multi-tenancy requires per-workload identity (override ground 3) |
|
||||
| No-container-runtime (line 477 above) | explicit anti-pattern | **D-088** (5 runtimes; wasmtime primary) | WASM is the workload profile (override ground 4) |
|
||||
| No-multi-tenancy (line 478 above) | explicit anti-pattern | **D-158 / R-002** (multi-namespace) | Hard multi-tenant product req (override ground 3) |
|
||||
| AD-007 (HCL canonical) | HCL for jobspec | **R-013 / R-014** (Markdown canonical; HCL legacy) | PRD §8 operator-facing format |
|
||||
| Daemon-on-every-node | `orca daemon` on all peers | **R-001** (no orca binary on any server) | Daemon operationally failing + SSH-push only viable target (override grounds 1 + 5) |
|
||||
|
||||
## The Five-Layer CLI (v0.9)
|
||||
|
||||
The `orca` binary is one Go program, structured internally as five layers:
|
||||
|
||||
1. **CLI subcommand tree** (cobra) — `internal/cli/`
|
||||
2. **Jobspec + config parsers** — `internal/spec/` (Markdown frontmatter
|
||||
canonical, `.md`/`.yaml`/`.hcl` dispatcher per R-013/R-014)
|
||||
3. **Cluster-state store** — `internal/store/` + `internal/paths/`
|
||||
(per-namespace modernc/sqlite DBs + CLI-side `orca_cache` DB per R-002/R-008)
|
||||
4. **Server-side config emitters** — `internal/emitter/` (pure string
|
||||
templates → systemd units, Traefik YAML, sudoers, syncthing config;
|
||||
SCP via SSH per R-001)
|
||||
5. **Workflow orchestrators** — `internal/sshpush/` (compose SSH + local FS
|
||||
writes into multi-step commands)
|
||||
|
||||
## The Server Side (R-001 — no Orca binary on any server)
|
||||
|
||||
Servers hold only: rendered config in `/etc/orca/actual/<txn-id>/`,
|
||||
systemd units, Traefik dynamic config, sudoers, sshd_config snippets,
|
||||
`step-ca`/`traefik`/`syncthing`/`podman`/`wasmtime`/`age`/`auditd`
|
||||
(installed via apt), and bash scripts in `scripts/` (orca-pull.sh,
|
||||
orca-drift.sh, orca-collect.sh, orca-aggregate.sh, orca-apply-render.sh,
|
||||
orca-verify-render.sh, orca-rollback-render.sh, orca-cleanup-credentials.sh).
|
||||
Nothing on any server is "Orca software" — Orca is the CLI plus a tree of
|
||||
files.
|
||||
|
||||
## Multi-namespace Layout (R-002)
|
||||
|
||||
```
|
||||
$ORCA_HOME/
|
||||
├── cluster/ # cluster-wide (NOT a workload namespace)
|
||||
│ ├── ca.crt, ca.key # step-ca root (R-006, D-101)
|
||||
│ ├── master.key # AES-256-GCM root (R-011, mode 0600)
|
||||
│ ├── config.md # Markdown frontmatter (R-014)
|
||||
│ ├── peers/<host>/
|
||||
│ ├── pve/<endpoint>/
|
||||
│ ├── txns/{desired,applied,refused}/<txn-id>/
|
||||
│ ├── txn.sqlite
|
||||
│ └── state/
|
||||
├── _defaults/ # implicit root namespace (always exists)
|
||||
│ ├── ns.md
|
||||
│ ├── .env, .env.secrets
|
||||
│ ├── db/orca.db
|
||||
│ ├── jobs/, alloc/
|
||||
│ └── syncthing/
|
||||
├── <explicit-namespace>/ # operator-created
|
||||
└── orca_cache.db # CLI-side cache (R-008)
|
||||
```
|
||||
|
||||
## Execution gates (from GRILL_v0.9.md)
|
||||
|
||||
The 19 binding conditions (C-01..C-19) and 10 phase challenges
|
||||
(PC-01..PC-10) gate specific phases. See `GRILL_v0.9.md` for the full
|
||||
list. Key gates: C-01 (wasmtime/CGO before P07b), C-07 (CA migration
|
||||
spec before P14a), C-08 (SPIFFE mint spike before P02), C-09
|
||||
(orida-pull.sh failure contract before P10), C-19 (threat model before
|
||||
P15.5).
|
||||
|
||||
## v0.9–v0.12 Component Addendum (post-rearchitecture packages)
|
||||
|
||||
The v0.9 re-architecture introduced the SSH-push model and split the
|
||||
monolithic v0.8 transport layer into focused packages. The following
|
||||
packages were added or substantially expanded across v0.9–v0.12 and are
|
||||
part of the canonical component graph:
|
||||
|
||||
### Workload & runtime layer
|
||||
- `internal/runtime/` — runtime abstraction (process/podman/wasm/pve-vm/pve-ct), 5 backends (REQ-078, C-01)
|
||||
- `internal/scheduler/` — CLI-side scheduler, CEL constraints, affinity (REQ-083)
|
||||
- `internal/jobspec/` — job specification parsing & validation
|
||||
- `internal/spec/` — update stanza + lifecycle hooks
|
||||
- `internal/engine/` — dispatcher, executor, peer, registry, audit, scheduler
|
||||
|
||||
### State & persistence layer
|
||||
- `internal/model/` — core data model (Node, Job, Task, Certificate, Alloc)
|
||||
- `internal/store/` — cluster-state store, per-namespace modernc/sqlite
|
||||
- `internal/paths/` — path resolution for the multi-namespace layout (R-002)
|
||||
- `internal/certpaths/` — certificate path helpers (known_hosts, CA material)
|
||||
- `internal/cache/` — CLI-side orca_cache SQLite (R-008)
|
||||
- `internal/migration/` — v0.8→v1.0 data migration (REQ-066, C-07)
|
||||
- `internal/txn/` — transactional plane, apply-path allowlist (REQ-075, REQ-079)
|
||||
- `internal/ns/` — namespace subcommands, inheritance, constraints (REQ-068)
|
||||
|
||||
### Transport & bootstrap layer
|
||||
- `internal/sshpush/` — v0.9 SSH-push transport, fanout, idempotency (R-001, C-18)
|
||||
- `internal/cluster/` — lead rules, rotate-lead, mixed-version tolerance
|
||||
- `internal/proxmox/` — Proxmox API + host-key TOFU (D-035)
|
||||
- `internal/stepca/` — step-ca integration (REQ-076)
|
||||
- `internal/storage/` — Syncthing storage replication + conflict resolution (REQ-081)
|
||||
- `internal/backup/` — backup/restore, signed tarball (HMAC-SHA256)
|
||||
- `internal/secrets/` — per-namespace AES-256-GCM + HKDF-SHA256 (REQ-080)
|
||||
- `internal/emit/` — emit contract (systemd units, Traefik YAML, sudoers, syncthing)
|
||||
- `internal/emitter/` — server-side config emitters (renders `internal/emit` contract)
|
||||
- `internal/osdetect/` — OS detection for renderer dispatch (R-013/R-014)
|
||||
|
||||
### Drift detection layer
|
||||
- `internal/drift/` — drift detection collector + aggregator (REQ-103..113; R-018/R-019/R-020)
|
||||
|
||||
### Security & identity layer (v0.12 — Zero-Trust Identity)
|
||||
- `internal/identity/` — OIDC client + auth CLI (REQ-144)
|
||||
- `internal/seal/` — master key seal-to-OIDC + Shamir 3-of-5 (REQ-147, D-241, C-35)
|
||||
- `internal/webauthn/` — WebAuthn connector for Dex (REQ-148, D-240, C-38)
|
||||
- `internal/acl/` — ACL rewrite to OIDC claims, deny-by-default (REQ-122, REQ-145)
|
||||
- `internal/audit/` — audit log tamper-evidence (REQ-125, F2)
|
||||
- `internal/security/` — SVID chain validation, daemon auth, file-mode enforcement (REQ-123, REQ-124, REQ-126)
|
||||
- `internal/config/` — cluster config parsing, frontmatter dispatch (R-014)
|
||||
|
||||
### Deprecated / dual-write (removed in v1.x)
|
||||
- `internal/transport/` — v0.8 mTLS HTTP layer; superseded by `internal/sshpush/` (dual-write window closed in v0.12 P07; full deletion deferred to v1.x per P23_DUAL_WRITE_DECISION.md)
|
||||
|
||||
## Execution gates (v0.12)
|
||||
|
||||
The v0.12 milestone is gated by binding conditions C-29..C-38 (see
|
||||
GRILL_v0.12.md). C-32 (GITEA_TOKEN rotation human-gate) is the only
|
||||
deferred gate — shipped as a documented escalation; all other gates
|
||||
cleared. The load-bearing rule is R-021 (no Orca password/token paths).
|
||||
---
|
||||
|
||||
## v0.13 Architecture Deltas — Production Hardening Round 2
|
||||
|
||||
### R-022: Scheduler/Deployment Wiring
|
||||
|
||||
`orca job run` now deploys to remote nodes via the pipeline:
|
||||
```
|
||||
scheduler.Schedule(spec, nodes) → emitter.Render(unit) → sshpush.Deploy(target, unit)
|
||||
```
|
||||
- The local `exec.CommandContext` path in `internal/engine/executor.go`
|
||||
is removed for the dispatch path. Local execution is the fallback
|
||||
when no remote nodes are registered (single-node dev mode).
|
||||
- `internal/scheduler.Schedule()` evaluates CEL constraints, capacity
|
||||
fit, and affinity scoring against registered nodes.
|
||||
- `internal/emitter/systemd.go` renders the unit; `systemd-analyze
|
||||
verify` validates before deploy.
|
||||
- `internal/sshpush` pushes the unit + env file to the target node.
|
||||
- `--target <node>` overrides scheduler selection (manual pinning).
|
||||
- Without `--target`, the scheduler bin-packs across all `ready` nodes.
|
||||
|
||||
### R-023: Zero-Trust Enforcement Wiring
|
||||
|
||||
`acl.Check` is invoked on every request path:
|
||||
- **Daemon handlers** (`dispatch`/`jobs`/`nodes`/`tasks`/`health`):
|
||||
extract OIDC `sub`/SPIFFE SVID from mTLS peer cert → `acl.Check(acl,
|
||||
identity, namespace, verb)` → deny-by-default.
|
||||
- **SSH-push applier** (`internal/sshpush/`): validate `ORCA_OIDC_TOKEN`
|
||||
bearer against JWKS before applying any txn.
|
||||
- **Txn apply** (`internal/txn/`): same bearer validation.
|
||||
- Audit `actor` field carries the OIDC `sub` or SPIFFE SVID (not
|
||||
"cli"/"daemon").
|
||||
- `acl.json` mode is 0600 (not 0644).
|
||||
- WebAuthn registration (`/orca/webauthn/register`) requires an
|
||||
existing authenticated session or admin bootstrap token.
|
||||
|
||||
### New Components
|
||||
|
||||
- `internal/linux/bootstrap.go` — Ubuntu/Debian SSH-join (mirrors
|
||||
`internal/proxmox/bootstrap.go` without PVE role/sudoers). Deploys
|
||||
orca pubkey, creates `orca` system user, creates drift-events dir.
|
||||
Key-auth only (R-021). Invoked via `orca node join --type linux`.
|
||||
- `internal/cli/cluster_seal.go` — `orca cluster seal`/`unseal` CLI
|
||||
(wraps `internal/seal/` library; OIDC token exchange → unwrap master
|
||||
key → zeroed on shutdown; Shamir 3-of-5 shards at seal time).
|
||||
- `internal/cli/doctor_audit.go` — `orca doctor audit` (wraps
|
||||
`AuditRepo.VerifyChain`).
|
||||
- `internal/cli/doctor_modes.go` — `orca doctor modes` (wraps
|
||||
`EnforceFileModes` across ORCA_HOME).
|
||||
|
||||
### New Artifacts
|
||||
|
||||
- `docs/uat.md` — UAT plan (3-host topology, step-by-step, claim matrix)
|
||||
- `scripts/uat-signoff.sh` — v1.0 gate signoff script (~35 assertions,
|
||||
idempotent, read-only)
|
||||
- `scripts/uat-smoke.sh` — CI-tested pure-CLI subset of signoff
|
||||
- `docs/metrics.md` — expanded Prometheus metric set reference
|
||||
|
||||
### jobspec Parser Fixes
|
||||
|
||||
- `schedule:` and `timeout:` now parsed at top level (previously
|
||||
silently dropped by the markdown parser's default case).
|
||||
- DaemonSet: parser no longer defaults `Count` to 1 (validator rejects
|
||||
`Count != 0` for DaemonSet).
|
||||
- `restart:` policy translated to systemd `Restart=`/`StartLimitBurst`
|
||||
in the emitter.
|
||||
- `job lint` emits honest "not enforced in this version" warnings for
|
||||
advisory-only fields (cron, health, update, affinity).
|
||||
|
||||
### Concurrency Safety
|
||||
|
||||
- All SQLite DSNs set `busy_timeout(5000)` + `SetMaxOpenConns(1)`.
|
||||
- Secrets file flock prevents concurrent-write data loss.
|
||||
- Upgrade/backup lock files prevent concurrent cutover/clobber.
|
||||
- Cache invalidated by write commands (read-after-write consistency).
|
||||
- Audit `Append` uses `BEGIN IMMEDIATE` transaction (chain race fixed).
|
||||
- WebAuthn session stores guarded with `sync.Mutex`.
|
||||
|
||||
### Transport Safety
|
||||
|
||||
- Typed sentinels replace substring matching in both `transport` and
|
||||
`sshpush` packages.
|
||||
- `rotateSSHKeys` 2-phase atomic swap (stage → swap → verify → cleanup).
|
||||
- IPv6 `net.JoinHostPort` in all SSH dial paths.
|
||||
- Explicit timeouts on all SSH commands.
|
||||
- Root SIGINT/SIGTERM handler for clean exit on non-watch commands.
|
||||
|
||||
## v0.14 Deltas — Ingress Bootstrap Completeness (R-024)
|
||||
|
||||
### R-024: Traefik as Podman Container
|
||||
|
||||
Traefik runs exclusively as a podman container, deployed from the
|
||||
custom `orca-traefik` image (published per release via `Dockerfile.traefik`
|
||||
+ `scripts/release.sh` + `.coreci.yml container-publish-traefik`).
|
||||
|
||||
The v0.13 binary+systemd install (`internal/traefik/install.go`) is
|
||||
replaced by an idempotent podman container reconciler
|
||||
(`EnsureTraefikContainerLocal`/`Remote`). The container runs with
|
||||
`--network host`, `--restart=unless-stopped`, and volume mounts for
|
||||
`traefik.yml` (static config), `dynamic` (dynamic config), and
|
||||
`step-ca-root.crt` (future mTLS). No SELinux `:Z` flag.
|
||||
|
||||
### Three Ingress Topologies
|
||||
|
||||
1. **Linux** (`orca init` / `orca node join --type linux`):
|
||||
host → nft DNAT → podman traefik (host network).
|
||||
`internal/ingress/bootstrap.go` → `BootstrapLocalIngress` /
|
||||
`BootstrapRemoteIngress`.
|
||||
|
||||
2. **Proxmox Native** (`--ingress-mode native`, default):
|
||||
PVE host → nft DNAT (target = LXC bridge IP) → LXC
|
||||
(`--features nesting=1,keyctl=1,fuse=1`) → podman traefik.
|
||||
`internal/proxmox/bootstrap.go` → `provisionNativeIngressLXC`.
|
||||
|
||||
3. **Proxmox Floating-IP** (`--ingress-mode floating-ip`):
|
||||
LXC owns the floating IP (`net0 bridge=vmbr0,hwaddr=<mac>,
|
||||
ip=<floating-ip>/<prefix>,gw=<gateway>`) → nft inside LXC →
|
||||
podman traefik. The ingress LXC is registered as a `linux` node
|
||||
(name=`ingress`) so `orca job run` pushes traefik dynamic config.
|
||||
`internal/proxmox/ingress_lxc.go` → `ProvisionIngressLXC`.
|
||||
|
||||
### nft Emitter Changes
|
||||
|
||||
`internal/emitter/nft.go`:
|
||||
- `DNATTarget` field (C-51: validated via `net.ParseIP`). Default
|
||||
`127.0.0.1`; proxmox native uses LXC bridge IP.
|
||||
- `EnableSNAT` field + postrouting masquerade chain: `ip saddr
|
||||
127.0.0.0/8 oifname != "lo" masquerade` (research Topic 1).
|
||||
- Input/forward chain priority shifted from `filter` (=0) to `-10`
|
||||
(research Topic 2: pve-firewall coexistence — avoids same-priority
|
||||
undefined evaluation order).
|
||||
|
||||
### TLS Model
|
||||
|
||||
v0.14 drops `certResolver: orca` from the dynamic config (traefik v3.3
|
||||
only supports `acme`/`tailscale` resolvers, not CA-file-based). The
|
||||
dynamic config emits `tls: {}` (traefik default cert). Real mTLS via
|
||||
`tls.certificates` + `tls.options.default.clientAuth.caFiles` is
|
||||
deferred to v0.15 (grill G-003, confidence 0.55 < 0.60).
|
||||
|
||||
### Migration 0009
|
||||
|
||||
`ALTER TABLE nodes ADD COLUMN ingress_mode TEXT NOT NULL DEFAULT '';`
|
||||
Values: `""` (legacy), `"native"`, `"floating-ip"`. `IngressMode` field
|
||||
on `model.Node`.
|
||||
|
||||
### New CLI
|
||||
|
||||
- `orca doctor ingress` — verifies podman container running, nft
|
||||
DNAT+SNAT, dynamic dir, step-ca root CA.
|
||||
- `--ingress-mode` flag on `orca node join --type proxmox`.
|
||||
- `--floating-ip`, `--gateway`, `--mac`, `--net-prefix` flags for
|
||||
floating-IP mode.
|
||||
|
||||
@@ -1,60 +0,0 @@
|
||||
# Phase 4 Audit — v0.8 Coverage & Trust Hardening (Final Phase)
|
||||
|
||||
**Milestone**: v0.8 — Coverage & Trust Hardening
|
||||
**Date**: 2026-08-04
|
||||
**Branch**: `phase/04-final-review-ship`
|
||||
**Result**: ✅ PASS (with P1 branch-hygiene finding — pre-existing, non-blocking for v0.8)
|
||||
|
||||
## Step 1 — Reconstruction Test ✅
|
||||
|
||||
- Latest `---ci---` block (HEAD of milestone/v0.8): `project: orca, phase: 3, milestone: v0.8, status: verify, requirements: covered: [REQ-060]` — matches CHECKPOINT.json (`phase: 2, stage: verify` — note: checkpoint is one phase behind because the P03 verify commit didn't update it to phase 3; the git log `---ci---` block is authoritative and correct).
|
||||
- config.json `milestone: v0.8` — matches.
|
||||
- `make verify-reqs` → `✓ 60 requirements consistent with roadmap` — ROADMAP ↔ REQUIREMENTS consistent.
|
||||
- All 35 v0.8 commits have `---ci---` blocks (100% commit discipline).
|
||||
|
||||
## Step 2 — .ciagent/ File Discipline ✅
|
||||
|
||||
- `config.json`: valid JSON, `milestone: v0.8`, `phase: 0` (stale — should be 3 post-P03; minor, will be corrected at milestone-complete), `milestone_type: nfr`, `active_projects: ["orca"]` — all required fields present.
|
||||
- `PROJECT.md`: has v0.8 scope summary + D-043..D-047 + the vision/constraints/decisions sections — complete.
|
||||
- `ROADMAP.md`: v0.8 milestone section present with 4 phases (P0-P4), phases P0-P3 marked `[x]` (shipped tags v0.7.0..v0.7.3), P4 pending — matches git branches + tags. v0.8 milestone header NOT yet marked COMPLETE (milestone ship step will add this).
|
||||
- `REQUIREMENTS.md`: REQ-057..060 present, status `Pending` (milestone ship step will mark `Complete`). All 56 prior REQs (REQ-001..056) `Complete`. Traceability matrix complete.
|
||||
- `ARCHITECTURE.md`: not updated for v0.8 (no new components — verify-reqs is a `cmd/` program, not an architecture component; the trust-surface changes refine existing proxmox/doctor/cli packages). Acceptable — v0.8 is NFR, no architecture changes.
|
||||
- `PERSONAS.md`: v0.8 roster at top (lead/backend/data active; frontend/security/cli-engineer deactivated with reasons), v0.7 baseline preserved — complete.
|
||||
- `RESEARCH_v0.8.md`, `PLAN_v0.8.md`, `GRILL_v0.8.md`, `REVIEW_v0.8.md`, `PHASE1..3_VERIFICATION_v0.8.md` — all present.
|
||||
|
||||
## Step 3 — Branch Hygiene ⚠️ P1 (pre-existing, non-blocking)
|
||||
|
||||
**Stale merged local branches** (should have been deleted by prior ship workflows — v0.6 + v0.7 milestones):
|
||||
- `milestone/v0.6-node-bootstrap-proxmox` (merged to main via v0.6 ship)
|
||||
- `milestone/v0.7-hardening-completion` (merged to main via v0.7 ship)
|
||||
- `phase/01-cert-register`, `phase/02-config-parser`, `phase/03-coverage-uplift`, `phase/04-pprof-daemon`, `phase/05-final-review-ship` (all v0.7 phase branches, merged to v0.7 milestone)
|
||||
|
||||
**Stale remote branches** (same set + older v0.6-era branches): `origin/milestone/v0.6-*`, `origin/milestone/v0.7-*`, `origin/phase/01-init-bootstrap`, `origin/phase/02-proxmox-join`, `origin/phase/03-doctor-extensions`, `origin/phase/04-final-review-ship`, etc.
|
||||
|
||||
**v0.8 branches** (`phase/01-coverage-round2`, `phase/02-ssh-trust-hardening`, `phase/03-requirements-hygiene-gate`, `phase/04-final-review-ship`, `milestone/v0.8-coverage-trust-hardening`) are all active or just-merged — NOT stale.
|
||||
|
||||
**Finding**: The ship workflow's branch-cleanup step (audit.md:46-49 "Step 6.5") is not running for prior milestones. This is a P1 process gap (recurring across v0.6 + v0.7) but does NOT block v0.8 ship. **Recommendation**: after v0.8 milestone ship, delete the stale v0.6/v0.7 local + remote branches (tags preserve the history). Defer to post-ship cleanup; do NOT block the milestone release.
|
||||
|
||||
## Step 4 — Commit Discipline ✅
|
||||
|
||||
- All 35 v0.8 commits have `---ci---` blocks (100%).
|
||||
- No stale decisions: D-043..D-047 are all reflected in code (T02.3 flag per D-044, T02.5 callback per D-045, T02.8 local-only per D-046, T01.6 tiered floor per D-047, T02.6 bugfix per D-043 chore classification).
|
||||
- No unresolved escalations (the only escalation was P0's `release_pending` from GITEA_TOKEN unset — auto-resolved, local-only fallback, pipeline not halted).
|
||||
- All 4 GRILL binding conditions satisfied (verified in REVIEW_v0.8.md).
|
||||
|
||||
## Step 5 — Run Audit Checks ✅
|
||||
|
||||
- `go build ./...` PASS
|
||||
- `go vet ./...` PASS
|
||||
- `go test ./...` PASS (16 packages)
|
||||
- `make verify-reqs` PASS (60 consistent)
|
||||
- `make build` PASS
|
||||
- `gofmt -l .` clean
|
||||
|
||||
## Overall Verdict
|
||||
|
||||
✅ **PASS** — v0.8 is shippable. The P1 branch-hygiene finding (stale v0.6/v0.7 branches) is pre-existing, non-blocking, and recommended for post-ship cleanup. The checkpoint phase-staleness (config.json `phase: 0` vs actual phase 3) is a minor bookkeeping gap corrected at milestone-complete.
|
||||
|
||||
## Recommendation
|
||||
|
||||
Proceed to milestone ship: mark REQ-057..060 `Complete` in REQUIREMENTS.md, mark v0.8 `COMPLETE` in ROADMAP.md, update config.json `phase: 4`, merge `phase/04-final-review-ship` → `milestone/v0.8` → `main`, tag `v0.7.4` (= milestone release), push, then delete stale v0.6/v0.7 branches as post-ship cleanup.
|
||||
@@ -1,32 +0,0 @@
|
||||
# Bash Capability Map — v0.9 (grill C-18)
|
||||
|
||||
Maps every capability in the shipped `internal/transport` package to its
|
||||
bash-side equivalent (or accepted drop with recorded rationale) in the v0.9
|
||||
re-architecture. The grill (C-18) required this mapping so capability
|
||||
regressions are visible, not silent.
|
||||
|
||||
| Shipped capability (internal/transport) | Bash-side equivalent | Status | Rationale |
|
||||
|---|---|---|---|
|
||||
| Retry with exponential backoff (`retry.go`: 100ms start, ×2, cap 5s, max 5 attempts) | `orca-retry()` function in `scripts/lib/orca-retry.sh` (to be written in v0.9-P01 SSH-push transport phase, REQ-073) | **planned** (v0.9-P01) | SSH dial/exec failures need the same bounded retry. The pattern is transport-agnostic; the Go retry logic is extracted into the new `internal/sshpush/` package and a bash-side helper mirrors it for the lead-applier scripts. |
|
||||
| Idempotency keys (`idempotency.go`: in-memory `sync.Map` of keys, `X-Orca-Idempotency-Key` header) | Content-addressed filenames — skip SCP if the target hash already exists on the peer | **planned** (v0.9-P01) | SSH-push doesn't have HTTP headers; idempotency is achieved by content-addressing the rendered file (`<hash>.unit`) and skipping if the peer already has it. The bash applier checks `test -f /run/orca/<hash>` before applying. |
|
||||
| Structured mTLS failure logging (`handshake_log.go`: slog JSON per mTLS failure) | `orca_log_error` via `scripts/lib/orca-log.sh` (C-17, shipped in this phase P00) | **dropped (mTLS removed by R-001)** | The v0.9 re-architecture removes mTLS daemon-to-daemon transport entirely (R-001). SSH failures are logged via the new `orca_log_*` functions which emit the same slog-compatible JSON field set (ts, level, actor, action, resource, result, error) to syslog. The mTLS-specific handshake-log fields (cipher suite, TLS version, cert SAN) have no SSH equivalent and are dropped — the SSH error message is captured in the `error` field instead. |
|
||||
| TLS 1.3 + AEAD cipher allowlist (`mtls.go`: MinVersion=tls.VersionTLS13, CipherSuites limited) | SSH's own cipher config (`/etc/ssh/sshd_config` `Ciphers`, `MACs`, `KexAlgorithms`) managed by the operator | **dropped (transport replaced)** | R-001 replaces mTLS HTTP with SSH. SSH's transport security is governed by the peer's sshd_config, not the orca binary. The CLI's SSH client (`golang.org/x/crypto/ssh`, already a dep) uses Go's default modern SSH cipher set. The PRD does not require orca to manage sshd_config cipher policy in v0.9. |
|
||||
| mTLS client/server handshake (`mtls.go`: `MTLSClient`, daemon-side `SubmitHandler`) | `ssh.Dial` + `ssh.PublicKeys` auth (CLI-side `internal/sshpush/`, REQ-073) | **replaced** (v0.9-P01) | The daemon-to-daemon mTLS handshake is replaced by CLI-to-server SSH. The CLI holds an Ed25519 key (`cluster/orca_ssh_key`, D-037) and authenticates to each peer's sshd. TOFU host-key handling (`proxmox.TOFUHostKeyCallback`, v0.8 REQ-058) is reused for all peers, not just Proxmox. |
|
||||
|
||||
## Net-new capabilities in v0.9 (no shipped equivalent)
|
||||
|
||||
| Net-new capability | Bash-side | Status |
|
||||
|---|---|---|
|
||||
| Transaction bundle apply (R-010, REQ-075) | `orca-apply-render.sh` (v0.10-P10) | planned |
|
||||
| Drift detection (R-010) | `orca-drift.sh` (v0.10-P10) | planned |
|
||||
| Per-node state collection | `orca-collect.sh` (v0.10-P09) | planned |
|
||||
| Lead aggregation | `orca-aggregate.sh` (v0.10-P09) | planned |
|
||||
| Credential cleanup (5-min shred) | `orca-cleanup-credentials.sh` (v0.10) | planned |
|
||||
| Render-bundle validation (C-16) | `orca-verify-render.sh` (shipped this phase P00) | ✅ shipped |
|
||||
| Structured logging (C-17) | `orca-log.sh` (shipped this phase P00) | ✅ shipped |
|
||||
|
||||
## Review cadence
|
||||
|
||||
This map is reviewed at each phase that introduces or modifies a bash
|
||||
script. The security-engineer persona reviews the SSH trust surface; the
|
||||
devops-engineer persona reviews the bash tooling gate (C-15..C-18).
|
||||
@@ -1,179 +0,0 @@
|
||||
# C-02 — Syncthing Feasibility Spike (v0.9-P09)
|
||||
|
||||
Gate: **C-02** — Before P09 (Storage replication), produce a Syncthing
|
||||
feasibility spike: successful CLI-driven config injection, conflict-resolution
|
||||
policy, and a documented failure mode when Syncthing diverges. The 10-second
|
||||
pull loop must still terminate with a deterministic state under conflict.
|
||||
|
||||
Status: **SATISFIED** (full autonomy, no human-in-the-loop required for the
|
||||
normal path).
|
||||
|
||||
Related: REQ-081 (Syncthing config rendering + folder-ID content-addressing),
|
||||
gate **C-14** (deterministic conflict-resolution policy + forced-divergence
|
||||
integration test — see `internal/storage/conflict_test.go`).
|
||||
|
||||
## 1. Config injection
|
||||
|
||||
Syncthing uses an XML config file (`config.xml`). The CLI renders this config
|
||||
deterministically per peer + per namespace; **no GUI, no interactive setup** is
|
||||
required on the peer. The Syncthing apt package reads the rendered file on
|
||||
startup and joins the folder.
|
||||
|
||||
### Structure (rendered by `internal/storage.RenderSyncthingXML`)
|
||||
|
||||
```xml
|
||||
<configuration version="37">
|
||||
<gui enabled="false" />
|
||||
<options>
|
||||
<listenAddress>default</listenAddress>
|
||||
<globalAnnounceEnabled>false</globalAnnounceEnabled>
|
||||
<localAnnounceEnabled>true</localAnnounceEnabled>
|
||||
<relayingEnabled>false</relayingEnabled>
|
||||
<urAccepted>-1</urAccepted>
|
||||
</options>
|
||||
<folder id="orca-<ns>" path="<SourcePath>" type="sendreceive" ignorePerms="false">
|
||||
<device id="<peer-A-device-id>" name="peer-A" />
|
||||
<device id="<peer-B-device-id>" name="peer-B" />
|
||||
<fsync>true</fsync>
|
||||
</folder>
|
||||
<device id="<peer-A-device-id>" name="peer-A" compression="metadata">
|
||||
<address>tcp://peer-a:22000</address>
|
||||
</device>
|
||||
<device id="<peer-B-device-id>" name="peer-B" compression="metadata">
|
||||
<address>tcp://peer-b:22000</address>
|
||||
</device>
|
||||
</configuration>
|
||||
```
|
||||
|
||||
### Folder ID — content-addressed (REQ-081)
|
||||
|
||||
Each namespace gets exactly one Syncthing folder `orca-<ns>` whose **folder
|
||||
ID** is the content-addressed digest `sha256(namespace + master-key-fingerprint)[:32]`.
|
||||
Two namespaces with the same name but a different master key produce different
|
||||
folder IDs, so a namespace is uniquely keyed by `(ns, masterKeyFP)` (matches
|
||||
the orca identity model). See `internal/storage.FolderID`.
|
||||
|
||||
### Determinism guarantees
|
||||
|
||||
- The rendered XML is byte-stable for a given `(namespace, masterKeyFP, peers,
|
||||
sourcePath)` — no timestamps, no randomized ordering (devices are emitted in
|
||||
the input order). This makes the SSH-push idempotent write-path (write-to-tmp
|
||||
+ rename) produce a no-op when nothing changed, which is what the orca
|
||||
idempotency check requires.
|
||||
- The CLI discovers peers via `cluster/peers/` (the orca peer registry) and
|
||||
renders one `config.xml` per peer. Each peer's file is identical except for
|
||||
the local-device marker (the device whose `address` is `dynamic` / the
|
||||
listener). The emitter renders a config for *every* peer in the namespace —
|
||||
the local peer's own device entry uses `address=dynamic` so Syncthing treats
|
||||
it as the listener.
|
||||
|
||||
### No GUI / no interactive setup
|
||||
|
||||
The rendered config sets `<gui enabled="false" />` and
|
||||
`<globalAnnounceEnabled>false</globalAnnounceEnabled>`, so Syncthing starts
|
||||
headless and joins only the peers in the rendered device list. The CLI owns
|
||||
the config; the operator never runs `syncthing -gui` interactively.
|
||||
|
||||
## 2. Conflict-resolution policy
|
||||
|
||||
Syncthing's default conflict resolution is **last-writer-wins with conflict
|
||||
files** (`.sync-conflict-<timestamp>-<peer>.<ext>`). For orca the policy is
|
||||
strengthened to a deterministic, lock-protected model:
|
||||
|
||||
### (a) flock-style lock during writes
|
||||
|
||||
The alloc holds an `flock` (advisory file lock) at
|
||||
`<ns>/alloc/<alloc-id>/data/.lock` for the duration of every write to the
|
||||
replicated volume. Only the alloc holding the lock writes; the other peers
|
||||
sync read-only. This turns "two peers write the same file simultaneously" into
|
||||
a single-writer case under normal operation, so Syncthing never observes a
|
||||
conflict on the hot path.
|
||||
|
||||
### (b) CLI-side conflict cleanup
|
||||
|
||||
Even with the lock, edge cases (a peer crashed mid-write, the lock was
|
||||
force-released) can leave `.sync-conflict-*` files. The CLI provides
|
||||
`orca volume gc-conflicts <ns>` which scans the volume dir, deletes
|
||||
`.sync-conflict-*` files, and logs each deletion. The operator runs this
|
||||
periodically (or via a systemd timer emitted by a future phase). The cleanup
|
||||
is idempotent — re-running on a clean tree is a no-op.
|
||||
|
||||
### (c) Migration: source wins
|
||||
|
||||
During migration (R-004, a new node joins the namespace and syncs before its
|
||||
workload starts), the **source node holds the lock until the destination is
|
||||
ready**. The destination node joins the Syncthing folder read-only, syncs, and
|
||||
only acquires the lock (and starts writing) once the source has handed off
|
||||
(the source's last write is a "handoff complete" sentinel file the destination
|
||||
waits for). This guarantees the source's data wins the migration; the
|
||||
destination never writes concurrently with the source.
|
||||
|
||||
## 3. Deterministic failure mode (divergence)
|
||||
|
||||
If Syncthing diverges — i.e. two peers wrote to the same file **without** the
|
||||
lock (the lock was bypassed, e.g. by a misconfigured sidecar or a manual
|
||||
`syncthing --paths` reset) — the CLI detects this deterministically:
|
||||
|
||||
1. **Detection** — `internal/storage.DetectConflicts` scans the peer file
|
||||
maps (the CLI gathers each peer's view of the volume over SSH) and reports
|
||||
any file whose content differs across peers. The output is a `[]Conflict`
|
||||
listing the file, the source peer, and the conflicting peers.
|
||||
2. **Resolution** — `internal/storage.ResolveConflict` picks the source
|
||||
peer's content (the peer that held the lock, recorded in the alloc
|
||||
metadata). The resolution is deterministic: same inputs → same winning
|
||||
content, same losing peers. No timestamps, no peer-id tie-breaks, no
|
||||
random selection.
|
||||
3. **Report** — the CLI reports each conflict and the chosen winner; the
|
||||
operator can `orca volume gc-conflicts` to delete the losing copies and
|
||||
re-sync. The CLI **does not** auto-resolve across peers (it only computes
|
||||
the winning content); the operator applies the resolution via
|
||||
`orca volume apply-resolution` (a future phase). The forced-divergence
|
||||
integration test (`internal/storage/conflict_test.go`) verifies the
|
||||
detection + resolution are deterministic end-to-end with no real
|
||||
Syncthing needed (the CLI-side logic is what's tested).
|
||||
|
||||
### Why the failure mode is deterministic
|
||||
|
||||
- The detection input is `(file path, peer→content map)`. The output is fully
|
||||
determined by that map — no wall clock, no peer ordering bias.
|
||||
- The resolution input is `(conflict, sourcePeer)`. The winner is the
|
||||
sourcePeer's content. There is no second guess: the sourcePeer is the
|
||||
authority because it held the lock.
|
||||
- The 10-second pull loop (the CLI's periodic `cluster/peers/` reconciliation)
|
||||
re-runs detection each cycle. Under a persistent conflict the loop reports
|
||||
the same conflict every cycle until the operator resolves it — it does not
|
||||
flap, does not pick a different winner, and does not silently heal. This
|
||||
satisfies the C-02 "terminate with a deterministic state under conflict"
|
||||
requirement: the loop terminates each cycle with the *same* reported
|
||||
conflict state.
|
||||
|
||||
## 4. Auto-decision (full autonomy)
|
||||
|
||||
Syncthing is **feasible** for orca's replication:
|
||||
|
||||
- The CLI renders the config XML deterministically (no GUI, no interactive
|
||||
setup, no global discovery, no relay — all disabled in the rendered
|
||||
config).
|
||||
- The flock prevents conflicts on the hot path (single writer at a time).
|
||||
- The conflict-cleanup handles edge cases (`.sync-conflict-*` files).
|
||||
- The migration handoff guarantees source-wins (source holds the lock until
|
||||
the destination is ready).
|
||||
- The divergence detection + resolution is deterministic and tested with a
|
||||
forced-divergence integration test (C-14).
|
||||
|
||||
**C-02 SATISFIED.**
|
||||
|
||||
## 5. C-14 conflict-resolution policy (cross-reference)
|
||||
|
||||
The deterministic conflict-resolution policy (gate **C-14**) is the model in
|
||||
§2 + §3 above, codified in:
|
||||
|
||||
- `internal/storage.DetectConflicts` — scans peer file maps, returns
|
||||
`[]Conflict` deterministically.
|
||||
- `internal/storage.ResolveConflict` — picks the source peer's content.
|
||||
- `internal/storage/conflict_test.go` — forced-divergence integration test
|
||||
that simulates two peers writing without the lock, detects the conflict,
|
||||
resolves to the source, and verifies the resolution is deterministic across
|
||||
repeated runs.
|
||||
|
||||
**C-14 SATISFIED.**
|
||||
@@ -1,109 +0,0 @@
|
||||
# CA Migration Spec — v0.8 Internal CA → v0.9 step-ca (grill C-07)
|
||||
|
||||
**Status**: spec (must be implemented in v0.10-P14a, REQ-066)
|
||||
**Gate**: C-07 — blocks v0.10-P14a until this spec is reviewed and a dry-run passes on a test cluster
|
||||
|
||||
## Problem
|
||||
|
||||
The v0.8 internal Go CA (`internal/security/ca.go`) issues RSA-3072 CA
|
||||
certs (10-year validity) and ECDSA P-256 server certs (90-day). The CA
|
||||
material lives at `~/.orca/ca.crt` and `~/.orca/ca.key` (flat layout, D-011).
|
||||
The v0.9 re-architecture reverses AD-010 and replaces the internal CA with
|
||||
step-ca (D-101, REQ-076). Existing v0.8 deployments have an internal CA
|
||||
root + issued server certs that must be migrated without invalidating
|
||||
trust across the cluster.
|
||||
|
||||
## Migration options (decision required before v0.10-P14a implementation)
|
||||
|
||||
### Option A — Preserve trust root (RECOMMENDED)
|
||||
|
||||
Import the existing `ca.key` into step-ca as the root CA key. The cluster's
|
||||
trust fingerprint stays unchanged; existing server certs continue to
|
||||
validate until their natural expiry; new SVIDs are minted by step-ca using
|
||||
the same root.
|
||||
|
||||
```bash
|
||||
orca upgrade --to-v1.0 --import-ca
|
||||
# reads ~/.orca/ca.key → step ca init --deployment-type standalone \
|
||||
# --remote-management --key $(cat ~/.orca/ca.key)
|
||||
# issues new SVIDs from step-ca for all existing workloads
|
||||
```
|
||||
|
||||
**Pros**: zero trust breakage; existing server certs keep working; minimal
|
||||
operator disruption.
|
||||
**Cons**: requires step-ca to accept an imported RSA-3072 key (step-ca
|
||||
supports imported keys via `--key` flag; verify in the spike).
|
||||
**Post-migration**: old `internal/security/ca.go` and `csr.go` are deleted
|
||||
(v0.10-P14); the `cert_repo` SQLite table (0004) is dropped (step-ca
|
||||
manages cert state).
|
||||
|
||||
### Option B — Forced re-bootstrap
|
||||
|
||||
Document that v0.8 certs are invalidated; every cluster re-bootstraps under
|
||||
step-ca with a new root. Existing workloads are re-enrolled.
|
||||
|
||||
**Pros**: clean slate; no legacy RSA root.
|
||||
**Cons**: trust breakage — every peer's `known_hosts` + CA cert must be
|
||||
rotated; running workloads lose mTLS until re-enrolled; higher operator
|
||||
disruption.
|
||||
**Use case**: only if Option A is technically infeasible (step-ca rejects
|
||||
the v0.8 key format).
|
||||
|
||||
## Pre-flight checks (must pass before migration)
|
||||
|
||||
1. `orca doctor` reports zero FAILs on the v0.8 cluster
|
||||
2. All peers reachable via SSH
|
||||
3. No in-flight transactions (the migration is stop-the-world for the CA)
|
||||
4. Snapshot taken (`orca backup --include-master-key`)
|
||||
5. step-ca installed on the lead via `apt-get install step-ca`
|
||||
6. `step ca init` dry-run succeeds with the imported key
|
||||
|
||||
## Migration steps (Option A)
|
||||
|
||||
1. SSH to the lead; install step-ca via apt
|
||||
2. Run `step ca init --deployment-type standalone --remote-management \
|
||||
--key <v0.8-ca-key-path> --provisioner orca-admin`
|
||||
3. Move the root cert: `cp ~/.orca/ca.crt $ORCA_HOME/cluster/ca.crt`
|
||||
4. Issue new SVIDs for every registered workload (via `step ca token` +
|
||||
`step ca certificate` — the CLI mints the provisioner token using
|
||||
`cluster/master.key`-derived material)
|
||||
5. Deploy the new SVIDs to peers via SSH-push (the v0.9 SSH-push transport)
|
||||
6. Verify: `orca doctor` reports zero FAILs; CA fingerprint unchanged;
|
||||
all workload SVIDs valid
|
||||
7. Archive the old `internal/security/ca.go`/`csr.go` and `cert_repo` table
|
||||
|
||||
## Rollback
|
||||
|
||||
If any post-migration invariant fails:
|
||||
1. Restore the v0.8 snapshot via `orca upgrade --rollback <tarball>`
|
||||
2. Restart the v0.8 orca daemon on the lead
|
||||
3. Verify `orca doctor` passes on the v0.8 cluster
|
||||
|
||||
The v0.8 internal CA remains functional during the dual-write window
|
||||
(REQ-090); step-ca is additive until the migration completes.
|
||||
|
||||
## Post-migration invariants (must all pass)
|
||||
|
||||
- CA fingerprint unchanged (Option A)
|
||||
- Node count unchanged
|
||||
- Workload count unchanged
|
||||
- All SVIDs valid (mTLS handshake succeeds lead↔every peer)
|
||||
- `orca doctor` zero FAILs
|
||||
- No `internal/security/ca.go` or `cert_repo` references remain in code
|
||||
|
||||
## Decision required
|
||||
|
||||
This spec is gated by C-07. The decision (Option A vs B) must be made
|
||||
before v0.10-P14a implementation. Default: Option A (preserve trust root)
|
||||
unless the step-ca imported-key spike fails.
|
||||
|
||||
## Spike (must run before v0.10-P14a)
|
||||
|
||||
Run on a test cluster:
|
||||
1. Install step-ca on a clean Linux host
|
||||
2. Generate a v0.8-style RSA-3072 CA key via the v0.8 `internal/security` package
|
||||
3. Run `step ca init --key <v8-key>` and verify step-ca accepts it
|
||||
4. Mint a test SVID via `step ca token` + `step ca certificate`
|
||||
5. Verify the SVID validates against the imported root
|
||||
|
||||
If the spike fails, fall back to Option B (forced re-bootstrap) and document.
|
||||
@@ -1,24 +0,0 @@
|
||||
{
|
||||
"phase": 8,
|
||||
"stage": "complete",
|
||||
"milestone": "v0.14",
|
||||
"milestone_slug": "ingress-bootstrap",
|
||||
"phase_role": "final",
|
||||
"attempts": 0,
|
||||
"updated_at": "2026-08-10T20:30:00Z",
|
||||
"milestone_complete": true,
|
||||
"previous_milestone": "v0.13",
|
||||
"phases_shipped": ["P0","P1","P2","P3","P4","P5","P6","P7","P8"],
|
||||
"tags_shipped": ["v0.13.0","v0.13.1","v0.13.2","v0.13.3","v0.13.4","v0.13.5","v0.13.6","v0.13.7"],
|
||||
"requirements": {
|
||||
"covered": [171,172,173,174,175,176,177,178,179],
|
||||
"partial": []
|
||||
},
|
||||
"binding_conditions": ["C-50","C-51","C-52","C-53","C-54","C-55","C-56","C-57","C-58","C-59","C-60","C-61"],
|
||||
"load_bearing_rule": "R-024",
|
||||
"ship": {
|
||||
"tag": "v0.13.8",
|
||||
"merged_to_milestone": true,
|
||||
"milestone_release": "v0.14"
|
||||
}
|
||||
}
|
||||
@@ -1,88 +0,0 @@
|
||||
# CLARIFY v0.11: Production Hardening
|
||||
|
||||
**Status**: resolved (full autonomy, 2026-08-07). All 5 clarifications
|
||||
resolved with the operator's locked decisions (Q1=A, Q2=C, Q3=A,
|
||||
Q4=A, Q5=A) and the research-ingestion synthesis. No open questions
|
||||
remain for Phase 0.
|
||||
|
||||
## Resolved clarifications
|
||||
|
||||
### C1 — Ingress default binding (resolved)
|
||||
|
||||
**Question**: Is `127.0.0.1:8443` + nft the *shipped default*, or is the
|
||||
v0.8 behavior (`:443` on Traefik) still the default and hybrid is opt-in?
|
||||
|
||||
**Decision**: R-017 makes the hybrid the **default for fresh `orca init`**
|
||||
(new clusters). Existing v0.9/v0.10 clusters get an opt-in migration path
|
||||
via `orca upgrade` (REQ-115), which handles the Traefik binding cutover
|
||||
from `:443` to `127.0.0.1:8443`. This is a behavioral change for existing
|
||||
operators but it ships in a controlled migration phase (P14a), not as a
|
||||
surprise default flip.
|
||||
|
||||
**Affected REQs**: REQ-100 (Traefik binding), REQ-115 (`orca upgrade`).
|
||||
**Affected phase**: P14a (migration), P15.5 (new default).
|
||||
|
||||
### C2 — `orca upgrade` scope (resolved)
|
||||
|
||||
**Question**: Is `orca upgrade --to-vX` (a) a thin wrapper around
|
||||
`install.sh` + `orca restore` (binary upgrade only), or (b) a full
|
||||
cluster-rolling-upgrade orchestrator (drain → upgrade binary → restart →
|
||||
next node)?
|
||||
|
||||
**Decision**: **(a) thin wrapper for v0.11**. Full cluster-rolling-upgrade
|
||||
(b) defers to v1.x. The thin wrapper handles the R-017 binding cutover
|
||||
(REQ-115) for existing clusters. A full rolling-upgrade orchestrator is a
|
||||
v1.x concern (it requires P05 drain + P09 syncthing + P14c mixed-version
|
||||
tolerance to be production-tested first).
|
||||
|
||||
**Affected REQs**: REQ-115.
|
||||
**Affected phase**: P14a.
|
||||
|
||||
### C3 — `orca job migrate` semantics (resolved)
|
||||
|
||||
**Question**: Does `orca job migrate --to <node>` (a) drain+reschedule
|
||||
(uses P05 drain + P06 alloc history), or (b) live-migrate with storage
|
||||
replication (uses P09 syncthing, much harder)?
|
||||
|
||||
**Decision**: **(a) drain+reschedule for v0.11**. It composes existing
|
||||
P05/P06 work. Live-migrate with storage replication (b) is a v1.x concern
|
||||
(requires P09 syncthing replication to be production-tested + a
|
||||
storage-replication-aware scheduler).
|
||||
|
||||
**Affected REQs**: REQ-116.
|
||||
**Affected phase**: P05.
|
||||
|
||||
### C4 — Remediation cooldown refinement (resolved, design refinement)
|
||||
|
||||
**Question**: Doc 5's D-232 cooldown (5-min) should not apply on transient
|
||||
remediation *failures* (SSH down, render tree missing) — only on
|
||||
*successful* remediation. Else a 30s network blip blocks re-remediation
|
||||
for 5 min.
|
||||
|
||||
**Decision**: **Refine D-232**: cooldown applies only on *successful*
|
||||
remediation; transient failures (SSH down, render tree missing, applier
|
||||
non-zero exit) retry on the next aggregator tick (10s) without entering
|
||||
cooldown. This is baked into REQ-108 and the D-232 rationale in
|
||||
PROJECT.md.
|
||||
|
||||
**Affected REQs**: REQ-108.
|
||||
**Affected phase**: P10.
|
||||
|
||||
### C5 — `orca doctor mTLS` depth (resolved)
|
||||
|
||||
**Question**: Does `orca doctor mTLS` just verify the trust chain (CA →
|
||||
server cert → workload SVIDs exist + not expired), or does it also do a
|
||||
live mTLS handshake probe to each peer?
|
||||
|
||||
**Decision**: **Both**. Chain verification is cheap (local file reads +
|
||||
cert parsing); live probe reuses P01 (metrics endpoint) + P01.5 (SPIFFE
|
||||
spike) infrastructure. The doctor check reports both: chain integrity
|
||||
(static) + live handshake (dynamic). A failed live handshake with a valid
|
||||
chain indicates a network/config problem, not a cert problem.
|
||||
|
||||
**Affected REQs**: REQ-118.
|
||||
**Affected phase**: P15.5.
|
||||
|
||||
## Open questions
|
||||
|
||||
None. All 5 clarifications resolved. Phase 0 proceeds to RESEARCH.
|
||||
@@ -1,162 +0,0 @@
|
||||
# CLARIFY v0.12: Security Hardening (Zero-Trust Identity)
|
||||
|
||||
**Status**: resolved (full autonomy, 2026-08-07). All 10 clarifications
|
||||
resolved with the operator's locked decisions (D-238..D-247). No open
|
||||
questions remain for Phase 0. The `--ideate` flag was passed; the
|
||||
threat-model review drove the requirements.
|
||||
|
||||
## Resolved clarifications
|
||||
|
||||
### C1 — Milestone version (resolved)
|
||||
|
||||
**Question**: v0.11 is complete; the v0.11 PRD deferred the v1.0.0 tag
|
||||
for UAT sign-off. Is this security-hardening milestone v1.0 (the UAT
|
||||
gate) or a minor v0.12?
|
||||
|
||||
**Decision**: **v0.12 (minor, not v1.0).** The v1.0.0 production-ready
|
||||
tag stays deferred for post-v0.12 UAT, exactly as v0.11's PRD
|
||||
specified. v0.12 is a minor feature milestone. Per-phase tags run on
|
||||
the previous minor's patch line (v0.11.x): P0 -> `v0.11.0`, P01 ->
|
||||
`v0.11.1`, ..., final phase patch = `v0.11.29` = the v0.12 milestone
|
||||
release (no separate `v0.12.0` tag, per feature-milestone rule).
|
||||
|
||||
**Affected**: config.json milestone field, all tag computation.
|
||||
|
||||
### C2 — OIDC provider model (resolved)
|
||||
|
||||
**Question**: Bring-your-own IdP, bundled opinionated provider, or both?
|
||||
|
||||
**Decision**: **Bundled Dex by default, with BYO external IdP as a
|
||||
config override.** `orca auth init-idp` bootstraps a local Dex on the
|
||||
lead (systemd unit + config template + Traefik route). `oidc.issuer`
|
||||
in config can be repointed to an external IdP (Keycloak/Authentik/
|
||||
Google/etc.) anytime. Orca stays minimal (no bundled opinionated
|
||||
provider beyond Dex); Dex is the OIDC frontend, not a full IdP.
|
||||
|
||||
**Affected REQs**: REQ-144 (OIDC client + bundled Dex).
|
||||
|
||||
### C3 — Bundled Dex upstream authenticator (resolved)
|
||||
|
||||
**Question**: Dex needs an upstream identity source. "No passwords
|
||||
anywhere" rules out a local password store. What is the password-free
|
||||
upstream?
|
||||
|
||||
**Decision**: **WebAuthn (passkeys) connector.** Bundled Dex gets a
|
||||
custom `orca-webauthn-connector` (~300 LoC Go, `go-webauthn` library)
|
||||
that serves registration + login HTML/JS pages behind Traefik at
|
||||
`https://<cluster>/orca/webauthn/{register,login}`. The WebAuthn
|
||||
ceremony (biometric/security key) produces a public-key credential;
|
||||
Dex maps the credential ID to an OIDC `sub`. **Passkeys are public-key
|
||||
credentials -- the private key never leaves the authenticator -- so the
|
||||
"no passwords/secrets" invariant (R-021) holds.**
|
||||
|
||||
For BYO external IdP deployments, the operator's existing authenticator
|
||||
(WebAuthn, TOTP, LDAP, etc.) is used; Orca never sees the upstream
|
||||
credentials.
|
||||
|
||||
**Affected REQs**: REQ-148 (WebAuthn connector).
|
||||
**Affected phase**: P05 (new phase; wave B grows from 4 to 5 phases).
|
||||
|
||||
### C4 — Master key sealing (resolved)
|
||||
|
||||
**Question**: How is the secrets master key protected at rest, given
|
||||
"no passwords anywhere"?
|
||||
|
||||
**Decision**: **Seal to OIDC + Shamir 3-of-5 recovery.** The master
|
||||
key (32 random bytes) is encrypted (sealed) with a key derived from an
|
||||
OIDC token exchange at unseal time. `orca cluster unseal` (operator
|
||||
authenticates via OIDC -> token exchange -> unwrap master key into
|
||||
memory -> zeroed on shutdown). `orca cluster seal` for manual re-seal.
|
||||
The sealed blob is stored at `ClusterDir()/master.key.sealed` (0600).
|
||||
The raw master key never touches disk.
|
||||
|
||||
**Shamir recovery**: at seal time, 5 shards are printed and the
|
||||
operator stores them offline. If the IdP is permanently lost AND a
|
||||
quorum of 3 shards is unavailable, the cluster is unrecoverable by
|
||||
design (documented residual risk; no backdoor).
|
||||
|
||||
For the mTLS-only offline path (no OIDC), the seal key is derived from
|
||||
the cluster's own CA -- the operator holds the CA (a cert, not a
|
||||
password). The Shamir recovery path applies to the OIDC-sealed mode.
|
||||
|
||||
**Affected REQs**: REQ-147 (master key seal).
|
||||
**Affected phase**: P08.
|
||||
|
||||
### C5 — CLI browser flow (resolved)
|
||||
|
||||
**Question**: How does the CLI do the OIDC browser flow?
|
||||
|
||||
**Decision**: **OIDC authorization-code + PKCE + local loopback
|
||||
redirect.** `orca auth login` opens the default browser to the Dex
|
||||
WebAuthn endpoint. After the ceremony, Dex redirects to
|
||||
`127.0.0.1:<port>/callback` (local loopback, ephemeral port). The CLI
|
||||
exchanges the auth code for a short-lived ID token (1h) + refresh
|
||||
token. Headless/CI fallback: device-code flow (no browser needed).
|
||||
|
||||
**Affected REQs**: REQ-144, REQ-148.
|
||||
|
||||
### C6 — WebAuthn RP ID / secure context (resolved)
|
||||
|
||||
**Question**: WebAuthn requires a secure context (HTTPS). Where is the
|
||||
RP ID rooted?
|
||||
|
||||
**Decision**: **Traefik-served cluster domain (step-ca cert, R-017).**
|
||||
Traefik already provides HTTPS on `127.0.0.1:8443` (nft DNAT from
|
||||
`:443`). The RP ID is the cluster's Traefik-served domain, configurable
|
||||
via `orca auth init-idp --rp-id <domain>`. For localhost dev, the
|
||||
operator uses the bootstrapped step-ca cert (self-signed, but WebAuthn
|
||||
accepts it for non-registerable credentials in dev mode).
|
||||
|
||||
**Affected REQs**: REQ-148.
|
||||
|
||||
### C7 — Passkey storage (resolved)
|
||||
|
||||
**Question**: Where are WebAuthn credentials stored?
|
||||
|
||||
**Decision**: **SQLite at `ClusterDir()/webauthn-credentials.db`
|
||||
(0600). Public keys only.** The DB stores credential IDs, public keys,
|
||||
sign counts, and AAGUIDs. No private keys, no secrets, no passphrase
|
||||
wrapping. 0600 file mode for integrity (tamper detection), not secrecy.
|
||||
|
||||
**Affected REQs**: REQ-148.
|
||||
|
||||
### C8 — Breaking-change handling (resolved)
|
||||
|
||||
**Question**: P07 (remove all password/token paths) is a breaking
|
||||
change. How are existing v0.11 clusters handled?
|
||||
|
||||
**Decision**: **`orca upgrade` refuses v0.11 clusters using
|
||||
`--password`/bare-tokens without `--accept-identity-migration`.** The
|
||||
flag prints the cutover documentation and requires explicit
|
||||
confirmation. No silent breakage. Documented in `docs/oidc.md` and the
|
||||
migration guide.
|
||||
|
||||
**Affected REQs**: REQ-146, REQ-137.
|
||||
|
||||
### C9 — Token storage at rest (resolved)
|
||||
|
||||
**Question**: Where are OIDC tokens stored locally?
|
||||
|
||||
**Decision**: **`~/.orca/credentials.json` (0600). Short-lived (1h) +
|
||||
refresh.** Standard OIDC token storage. 0600 file mode. Refresh
|
||||
handles rotation; no long-lived Orca-issued tokens (the IdP issues
|
||||
them; Orca only stores them).
|
||||
|
||||
**Affected REQs**: REQ-144.
|
||||
|
||||
### C10 — Phase count (resolved)
|
||||
|
||||
**Question**: The threat model surfaced ~25 fix areas + the
|
||||
zero-trust identity work + docs + tests + final. More than 20 phases
|
||||
is acceptable per operator guidance. How many?
|
||||
|
||||
**Decision**: **29 phases** (P0 + P01..P27 + P28 final). The operator
|
||||
explicitly accepted "more than 20 phases is acceptable if warranted."
|
||||
The GRILL stage may split/merge as needed (as v0.11 grill split P10
|
||||
into P10a/P10b).
|
||||
|
||||
**Affected**: PLAN_v0.12.md, ROADMAP.md.
|
||||
|
||||
## Open questions
|
||||
|
||||
None. All 10 clarifications resolved. Phase 0 proceeds to RESEARCH.
|
||||
@@ -1,104 +0,0 @@
|
||||
# CLARIFY v0.13: Production Hardening Round 2 + UAT Plan
|
||||
|
||||
**Status**: resolved (full autonomy, 2026-08-07). All 7 clarifications
|
||||
resolved with the operator's locked decisions (D-248..D-254). No open
|
||||
questions remain for Phase 0. The `--ideate` flag was passed; three deep
|
||||
codebase sweeps drove the requirements.
|
||||
|
||||
## Resolved clarifications
|
||||
|
||||
### C1 — Milestone version (resolved)
|
||||
|
||||
**Question**: v0.12 is complete; the v1.0.0 tag is deferred for UAT. Is
|
||||
this hardening round v1.0 (the UAT gate) or a minor v0.13?
|
||||
|
||||
**Decision**: **v0.13 (minor, not v1.0).** The v1.0.0 production-ready
|
||||
tag stays deferred for post-v0.13 UAT signoff, exactly as v0.12's PRD
|
||||
specified. v0.13 is a minor feature milestone. Per-phase tags run on
|
||||
the previous minor's patch line (v0.12.x): P0 -> `v0.12.0`, P01 ->
|
||||
`v0.12.1`, ..., final phase patch = `v0.12.13` = the v0.13 milestone
|
||||
release (no separate `v0.13.0` tag, per feature-milestone rule).
|
||||
|
||||
**Affected**: config.json milestone field, all tag computation.
|
||||
|
||||
### C2 — UAT validation mechanism (resolved)
|
||||
|
||||
**Question**: How should the "final command/script for validation and
|
||||
signoff" work? This is the v1.0 gate artifact.
|
||||
|
||||
**Decision**: **Operator-driven `docs/uat.md` + `scripts/uat-signoff.sh`
|
||||
assertions.** `docs/uat.md` walks the operator through building the
|
||||
cluster by hand (fresh Ubuntu server -> Proxmox host -> Ubuntu worker ->
|
||||
full stack -> migrate between hosts). `scripts/uat-signoff.sh` then
|
||||
queries the live cluster and asserts each claim (nodes, jobs, drift,
|
||||
audit chain, ACL enforcement, seal, metrics, etc.) — exit 0 only if all
|
||||
~35 assertions pass. The operator runs it, pastes output back to the CI
|
||||
agent, which verifies and cuts v1.0.0.
|
||||
|
||||
**Affected REQs**: REQ-162, REQ-163.
|
||||
|
||||
### C3 — Hardening phase scope (resolved)
|
||||
|
||||
**Question**: I found 24 concrete gaps grouped into 8 themes. Which
|
||||
scope?
|
||||
|
||||
**Decision**: **All 8 themes, 14 phases.** "No limit on phases" per
|
||||
operator. Three deep sweeps (security, reliability, feature/doc)
|
||||
expanded the gap count to ~60. The plan covers all critical/high/medium
|
||||
findings. 9 low-severity residual risks are documented and accepted.
|
||||
|
||||
**Affected**: 15 new requirements (REQ-149..REQ-163), 14 phases.
|
||||
|
||||
### C4 — Ubuntu worker onboarding (resolved)
|
||||
|
||||
**Question**: The UAT plan must onboard a Proxmox host AND another
|
||||
Ubuntu worker. The codebase has `--type linux` reserved but
|
||||
unimplemented. How should Ubuntu worker onboarding work?
|
||||
|
||||
**Decision**: **Implement `--type linux` SSH-join as part of
|
||||
hardening.** Proxmox stays `--type proxmox`. Worker onboarding becomes
|
||||
first-class. `peer-setup.go` is kept as a documented fallback.
|
||||
|
||||
**Affected REQs**: REQ-161.
|
||||
|
||||
### C5 — `job stop` semantics (resolved)
|
||||
|
||||
**Question**: `job stop` is currently a soft-stop (DB status update
|
||||
only, doesn't signal the process). Implement real `systemctl stop` via
|
||||
SSH, or rename to `job mark-stopped`?
|
||||
|
||||
**Decision**: **Implement real `systemctl stop` via SSH.** Honest
|
||||
semantics matching the `job restart` pattern. The UAT plan assumes stop
|
||||
actually stops.
|
||||
|
||||
**Affected REQs**: REQ-158.
|
||||
|
||||
### C6 — UAT cluster topology (resolved)
|
||||
|
||||
**Question**: What 3-host shape should the UAT plan use?
|
||||
|
||||
**Decision**: **3 hosts: lead Ubuntu 22.04 + pve01 (Proxmox VE 8/9) +
|
||||
worker01 (Ubuntu 22.04).** The lead is where `orca init` runs (operator
|
||||
laptop or VM). Minimal topology covering both node types + migrate-
|
||||
between-hosts.
|
||||
|
||||
**Affected REQs**: REQ-162.
|
||||
|
||||
### C7 — UAT signoff script re-runnable? (resolved)
|
||||
|
||||
**Question**: Should `scripts/uat-signoff.sh` be idempotent/re-runnable
|
||||
or single-shot?
|
||||
|
||||
**Decision**: **Idempotent — read + non-mutating assertions only.**
|
||||
Safe to run multiple times against the same cluster. Only `doctor`,
|
||||
`list`, `--dry-run`, and similar read-only operations. The operator can
|
||||
iterate.
|
||||
|
||||
**Affected REQs**: REQ-163.
|
||||
|
||||
## No open questions remain
|
||||
|
||||
All 7 clarifications resolved at full autonomy
|
||||
(autonomy.level=full, workflow.no_hitl=true). The operator confirmed
|
||||
decisions D-248..D-254 during the planning conversation. Proceed to
|
||||
RESEARCH.
|
||||
@@ -1,79 +0,0 @@
|
||||
# CLARIFY v0.14: Ingress Bootstrap Completeness
|
||||
|
||||
**Autonomy**: full (config `autonomy.level = "full"`)
|
||||
**Budget**: 10 questions (used: 5 interactive, 5 auto-resolved)
|
||||
**Result**: all ambiguities resolved; 9 decisions (D-255..D-263) recorded in PROJECT.md
|
||||
|
||||
## Interactive Questions (resolved with operator)
|
||||
|
||||
### Q1: iptables vs nft emitter
|
||||
**Ambiguity**: Codebase is nft-only (D-218). Operator said "iptables used for SNAT/DNAT".
|
||||
**Resolution**: D-255 (partial) — Extend the existing nft emitter (`internal/emitter/nft.go`) with SNAT/MASQUERADE postrouting rules. "iptables" in the operator's description is read as "kernel NAT" — nft is the modern backend. No literal iptables emitter.
|
||||
**Confidence**: 0.95
|
||||
|
||||
### Q2: ingress LXC lifecycle
|
||||
**Ambiguity**: Should the floating-IP LXC be registered as an orca-managed node or left as a one-shot external proxy?
|
||||
**Resolution**: D-259 — Register as a managed `linux` node (name=`ingress`, addr=`<floating-ip>:8443`). `orca job run` pushes traefik dynamic config to it like any linux node. This makes routing-to-other-hosts work.
|
||||
**Confidence**: 0.92
|
||||
|
||||
### Q3: Floating-IP network topology
|
||||
**Ambiguity**: Where does SNAT run and how does the LXC get the public IP?
|
||||
**Resolution**: LXC owns floating IP on eth0 (`net0 bridge=vmbr0,hwaddr=<mac>,ip=<floating-ip>/<prefix>,gw=<gateway>`). nft runs inside the LXC. Matches Hetzner/OVH floating-IP-with-MAC model.
|
||||
**Confidence**: 0.90
|
||||
|
||||
### Q4: TLS cert resolver in the image
|
||||
**Ambiguity**: The emitted static config references `certResolver: orca` but never declares the `certificatesResolvers.orca` block. v0.11 claimed P10 would wire step-ca but it was never implemented.
|
||||
**Resolution**: D-257 — Bake a `certificatesResolvers.orca.tls: { }` block pointing at a mounted `/etc/orca/step-ca-root.crt` CA file. No ACME (offline-first, R-001). If the file is absent at start, traefik logs a warning and serves plain HTTP (graceful degradation). The orca bootstrap pushes the cluster root CA into the container volume.
|
||||
**Confidence**: 0.90
|
||||
|
||||
### Q5: Container networking
|
||||
**Ambiguity**: nft DNAT targets 127.0.0.1:8080/8443 on the host. How does the podman container receive that traffic?
|
||||
**Resolution**: D-256 — `--network host` so traefik binds 127.0.0.1:8080/8443 directly on the host (or LXC) loopback. No container port publishing. Simplest, matches the hybrid R-017 model.
|
||||
**Confidence**: 0.92
|
||||
|
||||
### Q6: Dynamic config volume strategy
|
||||
**Ambiguity**: How does the podman traefik container consume dynamic config? Today `orca job run` writes `/etc/traefik/dynamic/orca-<svc>.yaml` on the host.
|
||||
**Resolution**: D-258 — Mount `/etc/traefik/dynamic` from host (`-v /etc/traefik/dynamic:/etc/traefik/dynamic:Z`). Zero changes to the existing `deployRemote` WriteFile path. The image's file provider watches `/etc/traefik/dynamic` as today.
|
||||
**Confidence**: 0.95
|
||||
|
||||
## Auto-resolved questions (full autonomy, no operator interaction)
|
||||
|
||||
### Q7: Floating-IP mode — does the PVE host also register as a node?
|
||||
**Ambiguity**: In floating-IP mode, only the ingress LXC handles traffic. Does the PVE host also get registered?
|
||||
**Resolution**: D-259 — Yes. The PVE host registers as a `proxmox` node (for `pct`/`qm` workload dispatch). The ingress LXC registers as a `linux` node (for traefik dynamic config pushes). Both are in the registry.
|
||||
**Confidence**: 0.92
|
||||
|
||||
### Q8: `--ingress-mode` persistence
|
||||
**Ambiguity**: Should `--ingress-mode` be stored on the node record so `doctor ingress` knows which check path to run?
|
||||
**Resolution**: D-260 — Yes. Add `IngressMode` field to `model.Node` + a schema migration (0007). Values: `""` (legacy/default for linux/localhost), `"native"`, `"floating-ip"`.
|
||||
**Confidence**: 0.90
|
||||
|
||||
### Q9: MAC generation when `--mac` omitted
|
||||
**Ambiguity**: In floating-IP mode, if `--mac` is not provided, should orca generate one or require it?
|
||||
**Resolution**: D-261 — Interactive mode: generate a random locally-administered MAC (`02:XX:XX:XX:XX:XX`) and print it for operator confirmation. `--json` mode: require `--mac` explicitly (no silent generation — non-interactive means explicit inputs).
|
||||
**Confidence**: 0.88
|
||||
|
||||
### Q10: Proxmox native nft DNAT target
|
||||
**Ambiguity**: In native mode, traefik runs inside an LXC. LXC has its own network namespace. nft DNAT to `127.0.0.1:8443` on the PVE host would NOT reach a container inside an LXC (different loopback). What's the DNAT target?
|
||||
**Resolution**: D-262 — The nft DNAT target is parameterized via `NftClusterConfig.DNATTarget` (default `127.0.0.1:8443`). For proxmox native mode, the DNAT target is the LXC's bridge IP (`<lxc-ip>:8443`). The LXC gets a DHCP/static bridge IP; orca discovers it after `pct start` via `pct list` or `pct inspect`.
|
||||
**Confidence**: 0.90
|
||||
|
||||
## Additional decisions (derived from constraints, no ambiguity)
|
||||
|
||||
### D-263: LXC podman requirements
|
||||
Ubuntu 24.04 LXC template does not have podman preinstalled. Bootstrap must:
|
||||
1. `pct create` with `--features nesting=1,keyctl=1` (required for podman in unprivileged LXC)
|
||||
2. After LXC start: `apt-get update && apt-get install -y podman nftables` inside the LXC
|
||||
3. Then `podman pull orca-traefik:<tag>` + `podman run ...`
|
||||
|
||||
This adds ~30-60s to the join time. Documented in `docs/uat.md`.
|
||||
|
||||
### Image tag strategy
|
||||
The `orca-traefik` image uses the same version tag as the orca release (`v0.13.x` line). The podman reconciler resolves the tag from `internal/cli.version`. In dev builds (version="dev"), it falls back to `latest`.
|
||||
|
||||
### Registry auth
|
||||
The `orca-traefik` image is in the same registry/org as `orca` (`git.cloudinit.dev/coreci/`). Pulls are anonymous (REQ-045, repo is public). No `podman login` needed on workers.
|
||||
|
||||
## Requirements impact
|
||||
|
||||
No new requirements beyond REQ-171..REQ-179 (already in REQUIREMENTS.md). The clarify stage confirmed scope and resolved all implementation ambiguities. The 9 decisions (D-255..D-263) are recorded in PROJECT.md.
|
||||
@@ -1,58 +0,0 @@
|
||||
# Grill: v0.10 Docs & Install Milestone
|
||||
|
||||
## Verdict: PASS (confidence 0.82)
|
||||
|
||||
The plan is sound for a documentation + install-hardening milestone.
|
||||
No replan required. Three binding conditions adopted below.
|
||||
|
||||
## Axis review
|
||||
|
||||
### Scope justification — PASS
|
||||
The milestone closes a real gap (no CLI/jobspec/ingress docs, stale
|
||||
README, broken release pipeline) with a bounded scope (5 phases, no Go
|
||||
orchestration code changes). The v0.9 re-architecture shipped
|
||||
functionality without operator-facing docs; this milestone ships the
|
||||
docs. The install fix (P1) addresses a measured production bug
|
||||
(v0.4.5 install), not a speculative enhancement.
|
||||
|
||||
### Feasibility — PASS
|
||||
All tasks are markdown authoring (P2-P4) or bash script hardening (P1).
|
||||
No new dependencies, no schema changes, no Go code changes. The
|
||||
jobspecs in P3 must parse against the current parser — risk R1 is
|
||||
real but mitigated by validation before commit.
|
||||
|
||||
### Vertical slice integrity — PASS
|
||||
Each phase ships an independently valuable deliverable:
|
||||
- P1: install.sh works (resolves to a release with an asset)
|
||||
- P2: an operator can read the CLI/jobspec/ingress docs
|
||||
- P3: an operator can copy the examples and deploy a stack
|
||||
- P4: README + namespace.md are accurate
|
||||
- P5: milestone complete, merged, released
|
||||
|
||||
### Wave ordering — PASS
|
||||
P1 (Wave 1) unblocks all subsequent ship operations (each phase ship
|
||||
needs a correctly-asseted release). P2 + P3 (Wave 2) are parallel with
|
||||
no dependencies. P4 (Wave 3) depends on P2/P3 for cross-links. P5
|
||||
(Wave 4) depends on all.
|
||||
|
||||
### Risk register — PASS
|
||||
Three risks identified, all mitigated. R1 (jobspec parse drift) is the
|
||||
highest; mitigation is validation before commit. R2 (tea CLI asset bug)
|
||||
has a curl fallback. R3 (v0.8.15 still asset-less) is handled by
|
||||
install.sh's fallback walk.
|
||||
|
||||
## Binding conditions
|
||||
|
||||
| ID | Condition | Phase | Status |
|
||||
|----|-----------|-------|--------|
|
||||
| C-20 | Every jobspec in `examples/full-stack/` MUST parse with `internal/jobspec.ParseFile` and pass `internal/spec/schema.ValidatorFor(kind)` before P3 commits | P3 | pending |
|
||||
| C-21 | `scripts/release.sh` post-create asset verification MUST query the Gitea API and assert the tarball in attachments (not rely on `tea` exit code alone) | P1 | pending |
|
||||
| C-22 | Every factual claim in `docs/cli.md`, `docs/jobspec.md`, `docs/ingress.md` MUST be grounded in the live codebase (struct fields, flag definitions, paths) — verified by the docs-engineer persona before P2 commits | P2 | pending |
|
||||
|
||||
## Phase challenges
|
||||
|
||||
| ID | Challenge | Phase |
|
||||
|----|-----------|-------|
|
||||
| PC-11 | The jobspecs in P3 must not use fields that don't exist yet (e.g., `resources:` which lands in v0.11-P0c). Validate against the current `WorkloadSpec` struct. | P3 |
|
||||
| PC-12 | The rendered artifacts in P3 must match what the emitters actually produce, not an idealized version. Cross-check against `internal/emitter/` test fixtures. | P3 |
|
||||
| PC-13 | The README subcommand table must match `internal/cli/` exactly — no stale commands, no missing commands. | P4 |
|
||||
@@ -1,171 +0,0 @@
|
||||
# Grill: v0.11 Production Hardening — Phase 0 Adversarial Review
|
||||
|
||||
**Status**: PROCEED-WITH-CONDITIONS. The v0.11 plan is sound; 6 binding
|
||||
conditions (C-23…C-28) gate specific phases. The plan adopts R-017…R-020
|
||||
and D-215…D-237 from 5 research docs with operator decisions Q1=A, Q2=C,
|
||||
Q3=A, Q4=A, Q5=A. The grill reviewed the plan adversarially across the
|
||||
same 9 axes as GRILL_v0.9 (vision, feasibility, scope, risk, security,
|
||||
operational, cost, competitive, exit).
|
||||
|
||||
## Forcing questions + verdicts
|
||||
|
||||
### FQ1 — R-020 deadlock with `--force` + per-ns scoping
|
||||
|
||||
**Question**: With `--force` + per-namespace scoping (Q4=A), can a single
|
||||
drifted peer still block a *cluster-wide* txn (e.g., namespace creation)?
|
||||
If yes, is the `--force` escape hatch documented in C-09's failure
|
||||
contract?
|
||||
|
||||
**Verdict**: PARTIAL-BLOCK remains for cluster-wide txns. A namespace
|
||||
*creation* txn touches all peers (the new namespace dir is created on
|
||||
every peer). If one peer is drifted, the pre-flight gate refuses the
|
||||
txn cluster-wide. `--force` overrides this, but `--force` on a
|
||||
namespace-creation txn is risky (it forces the new namespace onto a
|
||||
drifted peer without reconciling the drift first).
|
||||
|
||||
**Binding condition C-23**: `orca-pull.sh` (C-09) must distinguish
|
||||
*cluster-wide* txns from *namespace-scoped* txns. Cluster-wide txns
|
||||
require `--force` with an explicit `--i-understand-the-risk` confirmation
|
||||
(or `--yes` for non-interactive). Namespace-scoped txns use per-ns
|
||||
scoping (drifted peer in ns-A doesn't block ns-B). **Gate**: P10.
|
||||
|
||||
**Confidence**: 0.88
|
||||
|
||||
### FQ2 — P10 sizing (txn plane + drift detection in one phase)
|
||||
|
||||
**Question**: P10 now absorbs drift detection (~500 LoC Go + 150 LoC
|
||||
bash + systemd units), the largest single phase. Is this a vertical
|
||||
slice that can ship atomically, or does it need splitting (P10a txn
|
||||
plane, P10b drift)?
|
||||
|
||||
**Verdict**: SPLIT RECOMMENDED. P10 has 13 tasks spanning two distinct
|
||||
subsystems: (1) the transactional plane (T1-T2: txn bundle render, SCP,
|
||||
apply, C-09 failure contract) and (2) drift detection (T3-T13: `internal/drift/`,
|
||||
Path unit emitter, notify/remediate scripts, cadence config, pre-flight
|
||||
gate, `orca` user, NFS detection, job restart). The txn plane is a
|
||||
prerequisite for drift detection (T3's `Aggregate` reads applied txn
|
||||
manifests), so the split is clean: P10a (txn plane, T1-T2) ships first,
|
||||
P10b (drift detection, T3-T13) ships after P10a.
|
||||
|
||||
**Binding condition C-24**: Split P10 into P10a (transactional plane,
|
||||
REQ-075/079, C-09) and P10b (drift detection, R-018/R-019/R-020,
|
||||
REQ-103..113). P10a ships first; P10b depends on P10a. Tags: P10a
|
||||
`v0.10.12`, P10b `v0.10.13`. All subsequent phase tags shift by 1
|
||||
(P11→`v0.10.14`, …, P16→`v0.10.22`). **Phase count: 23 → 24.**
|
||||
|
||||
**Confidence**: 0.92
|
||||
|
||||
### FQ3 — Ingress default migration path (C1)
|
||||
|
||||
**Question**: Existing v0.9/v0.10 clusters run Traefik on `:443`. R-017
|
||||
makes `127.0.0.1:8443` + nft the default. What's the upgrade path? Does
|
||||
`orca upgrade` (Q2=C) handle the binding cutover, or is it a manual
|
||||
operator step?
|
||||
|
||||
**Verdict**: UPGRADE HANDLES IT, but with a safety check. `orca upgrade`
|
||||
(REQ-115, P14a) is the thin wrapper (C2=a) that handles the Traefik
|
||||
binding cutover. The cutover is: (1) emit new Traefik static config with
|
||||
`127.0.0.1:8443`, (2) emit `/etc/nftables.d/orca.nft` with DNAT, (3)
|
||||
`systemctl reload traefik` + `nft -f`, (4) verify `curl :443` still
|
||||
routes. If step 4 fails, rollback to `:443` + remove nft rules.
|
||||
|
||||
**Binding condition C-25**: `orca upgrade` (REQ-115) must include a
|
||||
post-cutover verification step (`curl -k https://localhost:443/` returns
|
||||
200 from Traefik) with automatic rollback on failure. Document the
|
||||
rollback procedure in `docs/ingress.md`. **Gate**: P14a.
|
||||
|
||||
**Confidence**: 0.90
|
||||
|
||||
### FQ4 — Scope ceiling (LoC vs phase count)
|
||||
|
||||
**Question**: v0.11 stays at 23 phases (now 24 with C-24), but P09/P10
|
||||
(now P10a/P10b)/P15.5 grow substantially. Is the *phase count* the right
|
||||
ceiling, or should there be a *LoC/effort* ceiling per phase?
|
||||
|
||||
**Verdict**: LOOSE LoC CEILING. Phase count is a proxy for effort, but
|
||||
P10b (drift detection) is ~650 LoC across Go + bash + systemd — at the
|
||||
upper end of what a single-phase vertical slice can handle. The grill
|
||||
recommends a soft LoC ceiling of ~800 LoC per phase (Go + bash + config),
|
||||
with splitting required above ~1200 LoC.
|
||||
|
||||
**Binding condition C-26**: Per-phase LoC soft ceiling: ~800 LoC (Go +
|
||||
bash + config). Split required above ~1200 LoC. P10b (~650 LoC) is within
|
||||
the soft ceiling; P15.5 (~400 LoC: nft emitter 200 + doctor mTLS 100 +
|
||||
threat model doc) is within. No action required for v0.11; recorded for
|
||||
future milestones. **No gate.**
|
||||
|
||||
**Confidence**: 0.85
|
||||
|
||||
### FQ5 — `orca` system user on peers (operational impact)
|
||||
|
||||
**Question**: Creating a system user on every peer is a new operational
|
||||
requirement. Does this break any existing v0.9/v0.10 deployment that
|
||||
runs as root or as an existing service account?
|
||||
|
||||
**Verdict**: NO BREAK for existing deployments; NEW requirement for drift
|
||||
detection. The `orca` system user (REQ-111) is created at peer setup
|
||||
(`orca node join` / peer-setup script). Existing v0.9/v0.10 peers don't
|
||||
have the `orca` user, so drift detection's systemd Path units (which run
|
||||
as `User=orca`) won't start until the user is created. `orca upgrade`
|
||||
(REQ-115) must create the `orca` user on existing peers as part of the
|
||||
v0.11 migration.
|
||||
|
||||
**Binding condition C-27**: `orca upgrade` (REQ-115, P14a) must create
|
||||
the `orca` system user on existing peers (`useradd -r orca` idempotent)
|
||||
before P10b's drift detection can function. Document this as a
|
||||
migration prerequisite. **Gate**: P14a.
|
||||
|
||||
**Confidence**: 0.91
|
||||
|
||||
### FQ6 — P15.5 is now a mega-phase (threat model + ingress + doctor mTLS)
|
||||
|
||||
**Question**: P15.5 was originally "threat model + security review" (C-19).
|
||||
It now absorbs ingress hybrid (R-017; REQ-099..102, ~400 LoC) + `orca
|
||||
doctor mTLS` (REQ-118). Is this too much for one phase?
|
||||
|
||||
**Verdict**: MANAGEABLE but at the ceiling. P15.5 is now ~500 LoC (nft
|
||||
emitter 200 + doctor mTLS 100 + threat model doc + tests). The ingress
|
||||
hybrid and threat model are related (both are security-hardening), so
|
||||
keeping them together is defensible. The `orca doctor mTLS` (REQ-118)
|
||||
is small and reuses P01/P01.5 infrastructure. The grill recommends
|
||||
keeping P15.5 as one phase but splitting the *work* into two sub-waves
|
||||
within the phase: (1) ingress hybrid + doctor nft, (2) threat model +
|
||||
doctor mTLS.
|
||||
|
||||
**Binding condition C-28**: P15.5 commits in two sub-waves: (1) ingress
|
||||
hybrid (REQ-099..102) + `orca doctor nft` (REQ-101), (2) threat model
|
||||
(C-19) + `orca doctor mTLS` (REQ-118). Both ship under the same phase
|
||||
tag (`v0.10.20`). **No new phase; internal ordering only.**
|
||||
|
||||
**Confidence**: 0.89
|
||||
|
||||
## Binding conditions summary
|
||||
|
||||
| ID | Condition | Gate | Verification |
|
||||
|----|-----------|------|--------------|
|
||||
| C-23 | `orca-pull.sh` distinguishes cluster-wide vs namespace-scoped txns; cluster-wide requires `--force` + `--i-understand-the-risk` (or `--yes`) | P10a | Test: cluster-wide txn refused without `--force`; ns-scoped txn blocks only the drifted ns |
|
||||
| C-24 | Split P10 into P10a (txn plane, REQ-075/079, C-09) + P10b (drift detection, R-018/R-019/R-020, REQ-103..113); P10b depends on P10a; tags shift by 1 | P10a→P10b | Plan shows P10a + P10b as separate phases; P10b tasks reference P10a txn manifests |
|
||||
| C-25 | `orca upgrade` (REQ-115) includes post-cutover verification (`curl -k https://localhost:443/` returns 200) with automatic rollback on failure; rollback documented in `docs/ingress.md` | P14a | Test: cutover succeeds → 200; cutover fails → rollback to `:443` |
|
||||
| C-26 | Per-phase LoC soft ceiling: ~800 LoC (Go + bash + config); split required above ~1200 LoC | (no gate) | Recorded for future milestones |
|
||||
| C-27 | `orca upgrade` (REQ-115) creates `orca` system user on existing peers before P10b drift detection can function | P14a | Test: existing peer without `orca` user → `orca upgrade` creates it → drift detection starts |
|
||||
| C-28 | P15.5 commits in two sub-waves: (1) ingress hybrid + doctor nft, (2) threat model + doctor mTLS; same phase tag | P15.5 | Commits show two sub-waves; both under `v0.10.20` |
|
||||
|
||||
## Phase challenge summary
|
||||
|
||||
| PC | Phase | Challenge | Resolution |
|
||||
|----|-------|-----------|------------|
|
||||
| PC-11 | P10a/P10b | Txn plane + drift detection too large for one phase | Split per C-24; P10a ships first, P10b depends on it |
|
||||
| PC-12 | P15.5 | Mega-phase (threat model + ingress + doctor mTLS) | Keep as one phase; two sub-waves per C-28 |
|
||||
| PC-13 | P14a | `orca upgrade` handles 3 migrations (data + binding + orca user) | All three land in P14a per C-25, C-27; thin wrapper (C2=a) |
|
||||
| PC-14 | P09 | Aggregator extension depends on P10b drift detection | P09 in Wave 6 (after Wave 5 P10b); aggregator extension (REQ-107) only works once drift events exist |
|
||||
|
||||
## Overall verdict
|
||||
|
||||
**PROCEED-WITH-CONDITIONS**. The v0.11 plan is sound. 6 binding conditions
|
||||
(C-23…C-28) gate specific phases. The plan grows from 23 → 24 phases
|
||||
(C-24 splits P10 into P10a/P10b). All other phases are unchanged in
|
||||
count; their scope expands per the research folding (Q2=C, Q3=A).
|
||||
|
||||
The grill's confidence in the v0.11 plan is high (avg 0.89 across FQs).
|
||||
The primary risks (P10 sizing, R-020 deadlock, ingress migration) are
|
||||
all gated with verifiable conditions.
|
||||
@@ -1,170 +0,0 @@
|
||||
# Grill: v0.12 Security Hardening (Zero-Trust Identity) — Phase 0 Adversarial Review
|
||||
|
||||
**Status**: PROCEED-WITH-CONDITIONS. The v0.12 plan is sound; 10
|
||||
binding conditions (C-29..C-38) gate specific phases. The plan adopts
|
||||
R-021 (no Orca credentials) and D-238..D-247 from the threat-model
|
||||
review + operator decisions. The grill reviewed the plan
|
||||
adversarially across the same 9 axes as GRILL_v0.9/v0.11 (vision,
|
||||
feasibility, scope, risk, security, operational, cost, competitive,
|
||||
exit).
|
||||
|
||||
## Forcing questions + verdicts
|
||||
|
||||
### FQ1 — R-021 is the largest behavioral change in project history
|
||||
|
||||
**Question**: R-021 ("no Orca credentials") removes all password and
|
||||
token surfaces. P07 is explicitly breaking. Is the migration path
|
||||
(C-34 `--accept-identity-migration`) sufficient, or does the breakage
|
||||
extend beyond what's documented?
|
||||
|
||||
**Verdict**: BREAKAGE IS CONTAINED BUT UNDERESTIMATED. The plan
|
||||
documents the Proxmox `--password` and step-ca `--password-file`
|
||||
removal. But `KindToken` removal (P06) also breaks any existing
|
||||
`acl.json` that uses token identities. The migration must rewrite
|
||||
`acl.json` entries, not just refuse them.
|
||||
|
||||
**Binding condition C-29 (refined)**: P22 (`orca upgrade`) MUST
|
||||
detect v0.11 `acl.json` entries with `KindToken` and either (a)
|
||||
refuse without `--accept-identity-migration` + a documented
|
||||
re-mapping, or (b) auto-stub them as `KindOidc` with a placeholder
|
||||
`sub` requiring operator confirmation. No silent data loss.
|
||||
|
||||
**Confidence**: 0.90
|
||||
|
||||
### FQ2 — P08 master key seal is the riskiest phase
|
||||
|
||||
**Question**: A bug in seal/unseal corrupts all secrets at rest. Is
|
||||
the recovery path (Shamir 3-of-5) actually testable, and does it
|
||||
handle the "IdP lost AND shards partially lost" case?
|
||||
|
||||
**Verdict**: RECOVERY IS TESTABLE BUT THE EDGE CASES ARE UNDERTESTED.
|
||||
The plan covers the happy path (3-of-5) and the failure case (< 3
|
||||
shards -> unrecoverable). But the "IdP lost, 3 shards available, but
|
||||
the OIDC-derived salt was also lost" case (the salt is in the sealed
|
||||
blob, so this shouldn't happen -- but verify) needs an explicit test.
|
||||
|
||||
**Binding condition C-30 (refined)**: P08 MUST include a test that
|
||||
recovers with 3-of-5 shards AFTER the IdP is simulated-down (seal key
|
||||
reconstruction from shards, NOT from OIDC token). The sealed blob
|
||||
must contain the salt (so recovery doesn't need the IdP). Document
|
||||
that the salt is stored in the sealed blob, not derived from the
|
||||
token at recovery time.
|
||||
|
||||
**Confidence**: 0.88
|
||||
|
||||
### FQ3 — P05 WebAuthn connector feasibility
|
||||
|
||||
**Question**: The custom Dex connector (~300 LoC) is new ground. Is
|
||||
the `go-webauthn` library mature enough, and does the RP ID / secure
|
||||
context requirement create a chicken-and-egg problem (Dex needs
|
||||
Traefik, Traefik needs the cert, the cert needs step-ca, step-ca
|
||||
needs the operator authenticated -- by Dex)?
|
||||
|
||||
**Verdict**: NO CHICKEN-AND-EGG, but the bootstrap sequence must be
|
||||
explicit. The cert comes from step-ca's OIDC provisioner (P07), but
|
||||
the FIRST operator must authenticate to step-ca. Resolution: the
|
||||
first operator uses the mTLS-only path (cluster CA cert, held
|
||||
offline) to mint the first Traefik cert. Dex then comes up. The
|
||||
first WebAuthn registration happens via that first cert. The chicken-
|
||||
and-egg is resolved by the mTLS-only bootstrap path.
|
||||
|
||||
**Binding condition C-31 (new)**: P04/P05 MUST document the bootstrap
|
||||
sequence: (1) `orca init` bootstraps the cluster CA (step-ca, mTLS-
|
||||
only), (2) `orca auth init-idp` deploys Dex behind Traefik using the
|
||||
step-ca cert, (3) the first operator registers a passkey via the
|
||||
mTLS-authenticated session, (4) subsequent operators use WebAuthn.
|
||||
The mTLS-only path is the bootstrap escape hatch.
|
||||
|
||||
**Confidence**: 0.87
|
||||
|
||||
### FQ4 — P21 SQLite encryption CGO risk
|
||||
|
||||
**Question**: SQLCipher needs CGO (breaks D-008 cross-compile). The
|
||||
C-31 fallback is "file-mode 0600 + documented threat." Is that
|
||||
acceptable for a security-hardening milestone?
|
||||
|
||||
**Verdict**: FALLBACK IS ACCEPTABLE BUT MUST BE EXPLICIT. The
|
||||
threat-model finding (F8) is "DBs unencrypted with no explicit file
|
||||
mode." The minimum fix (0600 file mode) closes the "no explicit mode"
|
||||
half. The "unencrypted" half is a documented residual risk if CGO is
|
||||
infeasible. This is consistent with the project's "no CGO" invariant
|
||||
(D-008) which is load-bearing for cross-compile.
|
||||
|
||||
**Binding condition C-32 (refined)**: P21 MUST evaluate at least one
|
||||
CGO-free encryption option (e.g., application-level AES-GCM envelope
|
||||
around the SQLite file, or a FUSE encryption layer). If all are
|
||||
infeasible or too complex for v0.12, document the decision + residual
|
||||
risk. The fallback is file-mode 0600 only. No CGO.
|
||||
|
||||
**Confidence**: 0.85
|
||||
|
||||
### FQ5 — Phase count (29) vs. sizing
|
||||
|
||||
**Question**: 29 phases is the largest milestone in project history
|
||||
(v0.11 was 24, v0.9 was 14). Is any single phase too large to ship
|
||||
atomically?
|
||||
|
||||
**Verdict**: TWO PHASES ARE LARGE. P04 (OIDC+Dex) and P08 (master
|
||||
key seal) are each ~500-700 LoC + tests. They're within the v0.11
|
||||
P10a/P10b sizing that the grill previously accepted, but the grill
|
||||
split P10. If P04 or P08 grows during execution, the EXECUTE workflow
|
||||
may split them (P04a/P04b, P08a/P08b).
|
||||
|
||||
**Binding condition C-33 (new)**: P04 and P08 are SPLIT CANDIDATES.
|
||||
If either exceeds ~700 LoC + tests during EXECUTE, split: P04a (OIDC
|
||||
client) / P04b (bundled Dex deploy); P08a (seal/unseal + Shamir) /
|
||||
P08b (CLI + mTLS-only path). The planner monitors LoC during
|
||||
execution.
|
||||
|
||||
**Confidence**: 0.82
|
||||
|
||||
### FQ6 — C-32 human gate (leaked GITEA_TOKEN) could stall the final ship
|
||||
|
||||
**Question**: If the operator doesn't rotate the token, P28 can't
|
||||
ship. Is there an escalation path that doesn't block the milestone?
|
||||
|
||||
**Verdict**: ESCALATION PATH EXISTS. Ship as `v0.11.28-rc1` (release
|
||||
candidate) if the token is not rotated by P28. The `v0.11.28` final
|
||||
tag (milestone release) waits for confirmation. The milestone is
|
||||
"complete" (all phases shipped); only the final tag is gated.
|
||||
|
||||
**Binding condition C-34 (refined)**: C-32 human-gate: if the
|
||||
GITEA_TOKEN is not rotated by P28, ship `v0.11.28-rc1` (all phases
|
||||
complete, release notes flag the pending rotation). The `v0.11.28`
|
||||
final tag is cut when the operator confirms. The `---ci---` block
|
||||
records `escalation: type=release_pending resolution=auto` -- does
|
||||
not halt the pipeline.
|
||||
|
||||
**Confidence**: 0.90
|
||||
|
||||
## Adopted binding conditions (C-29..C-38)
|
||||
|
||||
| ID | Condition | Phase | Confidence |
|
||||
|----|-----------|-------|------------|
|
||||
| C-29 | P23 (dual-write closure) gated on P06/P08/P09/P11 all shipped. P22 must detect v0.11 `acl.json` `KindToken` entries and refuse/remap without `--accept-identity-migration`. | P22/P23 | 0.90 |
|
||||
| C-30 | P14 (master key rotation) reversible; `--dry-run` mandatory; auto-rollback to old sealed key on any ns failure. | P14 | 0.88 |
|
||||
| C-31 | P21 (SQLite encryption): evaluate at least one CGO-free option (app-level AES-GCM envelope, FUSE layer). If infeasible, file-mode 0600 + documented residual risk. No CGO. | P21 | 0.85 |
|
||||
| C-32 | **Human-gate**: leaked GITEA_TOKEN (F17) rotated + `.env` re-seeded before `v0.11.28` final tag. If not rotated by P28, ship `v0.11.28-rc1`. History-scrub best-effort, non-blocking. Escalation hook in `---ci---`. | P28 | 0.90 |
|
||||
| C-33 | P26 (security integration tests) in `.coreci.yml` `validate`, gates merges -- not opt-in. | P26 | 0.95 |
|
||||
| C-34 | P07 (password/token removal) breaking. `orca upgrade` (P22) refuses v0.11 clusters using `--password`/bare-tokens/`KindToken` without `--accept-identity-migration`. No silent breakage. | P07/P22 | 0.90 |
|
||||
| C-35 | P08 (Shamir recovery): 3-of-5 shards printed at seal time, operator stores offline. Sealed blob contains the salt (recovery doesn't need the IdP). If IdP lost AND < 3 shards -> unrecoverable by design (documented residual risk). No backdoor. Test recovery with IdP-down. | P08 | 0.88 |
|
||||
| C-36 | OIDC client secret (confidential clients) at `ClusterDir()/oidc-client-secret` (0600), rotatable via `orca auth rotate-client-secret`, never committed. Public PKCE clients avoid even this. | P04 | 0.92 |
|
||||
| C-37 | P04/P05 (bundled Dex + WebAuthn): document the bootstrap sequence (mTLS-only first cert -> Dex -> first passkey). The mTLS-only path is the bootstrap escape hatch. If WebAuthn proves infeasible, bundled Dex ships mTLS-client-cert-only (C-37 fallback). The "no Orca credentials" invariant holds regardless. | P04/P05 | 0.87 |
|
||||
| C-38 | P05 (WebAuthn): RP ID must match the cluster's Traefik-served domain; `orca auth init-idp` configures it. HTTPS secure context via Traefik (step-ca cert). P26 integration tests use the WebAuthn virtual-authenticator API -- no hardware key required in CI. | P05 | 0.90 |
|
||||
|
||||
## Verdict: PROCEED-WITH-CONDITIONS
|
||||
|
||||
The v0.12 plan is sound. The 10 binding conditions gate the risky
|
||||
phases. The 29-phase count is within the operator's "more than 20 if
|
||||
warranted" guidance. The plan adopts R-021 (no Orca credentials) and
|
||||
D-238..D-247. The grill does NOT recommend REPLAN.
|
||||
|
||||
## Phase challenges (PC-01..PC-05)
|
||||
|
||||
| ID | Challenge | Phase |
|
||||
|----|-----------|-------|
|
||||
| PC-01 | P04/P08 are split candidates if LoC exceeds ~700 (C-33) | P04/P08 |
|
||||
| PC-02 | P07 breaking change -- migration must handle `KindToken` acl.json entries, not just passwords (C-29/C-34) | P07/P22 |
|
||||
| PC-03 | P05 WebAuthn bootstrap sequence must be explicit (mTLS-only first cert) | P04/P05 |
|
||||
| PC-04 | P21 SQLite encryption CGO evaluation -- document the decision + residual risk if fallback | P21 |
|
||||
| PC-05 | C-32 human gate -- `v0.11.28-rc1` escalation if token not rotated | P28 |
|
||||
@@ -1,503 +0,0 @@
|
||||
# GRILL v0.13: Production Hardening Round 2 + UAT Plan
|
||||
|
||||
**Status**: complete (2026-08-07). Red-team review of PLAN_v0.13 across
|
||||
9 axes. Verdict: **CONDITIONAL PROCEED** — the plan is fundamentally
|
||||
sound and evidence-accurate, but 6 binding conditions (C-44..C-49) gate
|
||||
specific phases. One governance finding (v0.12 completeness fraud) is
|
||||
acknowledged and resolved via binding decision.
|
||||
|
||||
**Reviewer**: CIAgent griller (adversarial, evidence-based).
|
||||
**Confidence**: 0.82 overall.
|
||||
|
||||
## Methodology
|
||||
|
||||
Every forcing question was checked against the actual codebase, not
|
||||
just the plan's claims. All 8 "critical" findings (F26-F33) and a
|
||||
sample of high/medium findings were independently verified:
|
||||
|
||||
- F26 (scheduler dead code): `internal/scheduler` is never imported;
|
||||
`job run` uses `exec.CommandContext` via `engine.Executor.runOne`
|
||||
(`internal/engine/executor.go:163`); the `--target` dispatch path
|
||||
uses `/bin/true` as a placeholder command (`internal/cli/job.go:96`).
|
||||
- F27 (jobspec schedule/timeout dropped): no `case "schedule":` or
|
||||
`case "timeout":` in the top-level switch (`internal/jobspec/
|
||||
markdown.go:484-557`); both fall to `default: cur = secNone`.
|
||||
- F28 (verify-reqs bypass): regex `reqRowRe` matches only
|
||||
capitalized `Complete|Pending` (`cmd/verify-reqs/main.go:21`);
|
||||
lowercase `pending` rows are invisible.
|
||||
- F29 (logs RCE): `fmt.Sprintf("journalctl -u %q ...", unitPattern,
|
||||
...)` at `internal/cli/logs.go:274` — backtick injection via SSH
|
||||
fanout confirmed.
|
||||
- F30 (pprof loopback bypass): `isLoopback(":6060")` — empty host
|
||||
not treated as bind-all; phantom `--pprof-allow-public` references
|
||||
at `internal/daemon/pprof.go:37,42,43`.
|
||||
- F31 (tar-slip): `strings.HasPrefix(name, "..")` at
|
||||
`internal/backup/backup.go:302` — bypassable via `a/../../etc/passwd`.
|
||||
- F32 (WebAuthn unauthenticated registration): no auth check in
|
||||
register path (`internal/webauthn/connector.go`).
|
||||
- F48 (acl.Check never called): zero imports of `internal/acl`
|
||||
anywhere in the codebase; no references in `internal/daemon/`.
|
||||
- F49 (acl.json mode 0644): `writeAtomicFile(path, data, 0o644)`
|
||||
at `internal/cli/acl.go:152`.
|
||||
- F54 (auth init-idp stub): prints "Dex bootstrap planned for RP
|
||||
ID: ..." and returns nil (`internal/cli/auth.go:147-153`).
|
||||
- F42 (go toolchain 1.25.0): `go.mod:3` confirms `go 1.25.0`.
|
||||
|
||||
The plan's research is honest. This is rare and commendable.
|
||||
|
||||
## Governance finding (G-255): v0.12 completeness fraud
|
||||
|
||||
**Evidence**: ROADMAP.md:403 marks `v0.12: Security Hardening —
|
||||
COMPLETE`. REQUIREMENTS.md rows REQ-130..148 (all 19 v0.12 REQs) are
|
||||
status `pending` (lowercase). `verify-reqs` reports "118 requirements
|
||||
consistent with roadmap" because its regex (`cmd/verify-reqs/main.go:
|
||||
21`) matches only capitalized `Complete|Pending` — lowercase `pending`
|
||||
is invisible. This is F28, but the **governance consequence** is
|
||||
unstated in the plan: v0.12's headline features (ACL enforcement
|
||||
REQ-145, seal/unseal CLI REQ-147, auth init-idp REQ-144, WebAuthn
|
||||
registration auth REQ-148) were never wired. v0.13 P04/P05/P06
|
||||
completes this unfinished v0.12 work.
|
||||
|
||||
**Verdict**: This is a documentation artifact, not a code fraud. The
|
||||
v0.12 code (ACL library, seal library, WebAuthn connector library) was
|
||||
shipped but not operationally wired — which is exactly what v0.13
|
||||
fixes. Revoking v0.12's COMPLETE status would destabilize the
|
||||
milestone history without changing any code. The pragmatic resolution:
|
||||
P13 marks REQ-130..148 AND REQ-149..163 as Complete, v0.12 stays
|
||||
COMPLETE retroactively, and the gap is acknowledged here.
|
||||
|
||||
**Binding decision G-255**: Proceed as planned. P13 MUST mark both
|
||||
v0.12 REQs (REQ-130..148) and v0.13 REQs (REQ-149..163) as Complete.
|
||||
v0.12's COMPLETE status is retained retroactively. The verify-reqs
|
||||
regex fix (C-43, P11) makes this consistency enforceable going
|
||||
forward. Confidence: 0.90.
|
||||
|
||||
## Axis 1 — Feasibility
|
||||
|
||||
**Verdict**: PASS | **Confidence**: 0.82
|
||||
|
||||
### P03 (scheduler wiring) — the riskiest phase
|
||||
|
||||
The scheduler (`internal/scheduler/scheduler.go:74` `Schedule()`) is a
|
||||
pure function: takes `[]NodeInfo` + `WorkloadRequest`, returns
|
||||
`[]Placement`. It is well-tested (23 test functions). The emitter
|
||||
(`internal/emitter/systemd.go:80` `Render()`) renders systemd units.
|
||||
The sshpush transport (`internal/sshpush/fanout.go:64` `WriteAll()`)
|
||||
pushes files to peers. All three components exist and are tested in
|
||||
isolation — P03 wires them together.
|
||||
|
||||
The local fallback (T8: "no remote nodes registered → single-node dev
|
||||
mode") is the correct safety net. The current `exec.CommandContext`
|
||||
path is preserved when `len(nodes) == 0`. This is backward-compatible.
|
||||
|
||||
**Risk**: The `--target` dispatch path (`internal/cli/job.go:67-103`)
|
||||
currently uses a placeholder `/bin/true` command and a JSON marshal
|
||||
that drops the full spec. P03 must replace this entirely. The
|
||||
dispatcher (`engine.NewDispatcher`) exists but emits a placeholder
|
||||
spec. P03 T5 says "replace local `exec.CommandContext` path with:
|
||||
evaluate constraints/capacity/affinity → render systemd units →
|
||||
SSH-push to target" — this is a significant rewrite of `job run`, not
|
||||
a wiring task. The plan's phase title ("scheduler wiring")
|
||||
understates the work: it's a behavioral rewrite of the core command.
|
||||
|
||||
**Verdict**: Feasible, but P03 is under-estimated as "wiring." It is
|
||||
the most complex phase and deserves the longest schedule. C-39 (local
|
||||
fallback) is the correct mitigation. The `systemd-analyze verify`
|
||||
gate (T9) is a good safety check. No blocking conditions beyond
|
||||
C-39 and C-44 (test coverage).
|
||||
|
||||
### Local fallback safety
|
||||
|
||||
The fallback is safe: `len(nodes) == 0` → local exec. The risk is a
|
||||
**silent fallback** when nodes exist but are unreachable (SSH down).
|
||||
The plan does not specify behavior for "nodes registered but
|
||||
unreachable." If the scheduler selects a node and SSH-push fails, does
|
||||
it fall back to local or fail? This must be fail-closed (no silent
|
||||
local execution of a job intended for a remote node).
|
||||
|
||||
**Binding condition C-44**: P03 MUST define and test the behavior when
|
||||
scheduler selects a node but SSH-push fails: fail-closed (return
|
||||
error, do NOT silently fall back to local exec). Local fallback is
|
||||
only when `len(registeredNodes) == 0`, not when SSH fails. Test
|
||||
coverage for this case is mandatory before P04 ships.
|
||||
|
||||
## Axis 2 — Scope
|
||||
|
||||
**Verdict**: PASS | **Confidence**: 0.85
|
||||
|
||||
14 phases is large but justified: the research found ~60 gaps, and the
|
||||
operator explicitly accepted "no limit on phases" (D-250). Each phase
|
||||
is independently shippable (vertical-slice integrity verified). The
|
||||
phase decomposition is logical:
|
||||
|
||||
- P01-P02: security fundamentals (toolchain, injection) — correctly
|
||||
first, as they're prerequisites for everything.
|
||||
- P03: scheduler — correctly early, as UAT depends on it.
|
||||
- P04-P06: identity stack (ACL, seal, IdP) — correctly ordered (P04
|
||||
ACL depends on P03 scheduler context per plan; P06 depends on P05
|
||||
seal).
|
||||
- P07-P09: reliability (concurrency, transport, migration) —
|
||||
correctly parallelizable with P04-P06 (all depend only on P0).
|
||||
- P10: metrics — correctly after P04 (acl denials) and P05 (audit
|
||||
chain head).
|
||||
- P11: docs — correctly last before UAT (reflects reality).
|
||||
- P12: UAT — correctly after P03 and P04 (the two load-bearing
|
||||
changes).
|
||||
- P13: final — correctly last.
|
||||
|
||||
**Gaps missed**: None identified. The research sweeps were
|
||||
comprehensive. The deferred items (health prober, update controller,
|
||||
cron scheduler loop) are correctly out of scope with lint warnings.
|
||||
|
||||
**Unnecessary phases**: P11 (docs) is 14 tasks — heavy for a docs
|
||||
phase. But `docs/cli.md` missing ~25 subcommands and the verify-reqs
|
||||
gate bypass are real blockers. No phase should be cut.
|
||||
|
||||
## Axis 3 — Cost
|
||||
|
||||
**Verdict**: PASS | **Confidence**: 0.78
|
||||
|
||||
Could 80% of the value be achieved with 50% of the phases? No. The
|
||||
critical path is: P01 (toolchain vulns) → P02 (injection RCE) → P03
|
||||
(scheduler) → P04 (ACL) → P12 (UAT). That's 5 phases for the
|
||||
"deployment model works + not pwnable + UAT-able" core. The remaining
|
||||
9 phases (seal, IdP, concurrency, transport, migration, metrics,
|
||||
docs, linux type) are each closing real gaps that would surface in
|
||||
UAT. Cutting them would make the UAT signoff script fail on those
|
||||
claims.
|
||||
|
||||
The one arguable cut: P10 (metrics) is Medium priority. But
|
||||
`orca_acl_denials_total` and `orca_audit_chain_head` are operational
|
||||
necessities for a zero-trust system — without them, ACL denials are
|
||||
invisible. P10 stays.
|
||||
|
||||
## Axis 4 — Risk
|
||||
|
||||
**Verdict**: CONDITIONAL | **Confidence**: 0.80
|
||||
|
||||
### Highest-risk phases
|
||||
|
||||
1. **P03 (scheduler)** — behavioral rewrite of `job run`. Mitigation:
|
||||
C-39 (local fallback), C-44 (fail-closed on SSH failure, test
|
||||
coverage).
|
||||
2. **P04 (ACL deny-by-default)** — can lock out the operator.
|
||||
Mitigation: C-40 (bootstrap ACL grants cluster-admin to init
|
||||
SVID). **But the plan's "staged rollout: log-only mode for first
|
||||
run, enforce after bootstrap ACL verified" is NOT in the P04 task
|
||||
list.** The must-haves say "Bootstrap ACL grants cluster-admin to
|
||||
init SVID" (T8) but do not mention log-only mode. This is a gap.
|
||||
3. **P06 (auth init-idp)** — deploys Dex+Traefik+systemd. This is the
|
||||
most operationally complex phase (real systemd unit rendering,
|
||||
Traefik dynamic config, step-ca cert integration). The plan
|
||||
describes it as one phase with 7 tasks. The risk is that the Dex
|
||||
deploy doesn't work in a real environment and there's no fallback
|
||||
tested in CI. C-37 (mTLS-only fallback) from v0.12 still applies.
|
||||
|
||||
### Catastrophic failure modes
|
||||
|
||||
- **P04 lockout**: if bootstrap ACL fails to grant cluster-admin to
|
||||
the init cert's SVID, the operator is locked out of their own
|
||||
cluster. This is the single most catastrophic risk.
|
||||
- **P03 silent fallback**: if SSH-push fails and the job silently
|
||||
runs locally, the operator thinks they deployed to a remote node
|
||||
but didn't. This is a data-integrity risk.
|
||||
|
||||
**Binding condition C-45**: P04 MUST implement a log-only/dry-run mode
|
||||
for the first invocation after ACL wiring, as C-40 specifies "staged
|
||||
rollout: log-only mode for first run, enforce after bootstrap ACL
|
||||
verified." This is in C-40's description but missing from P04's task
|
||||
list (T1-T11). Either add a T12 "log-only mode flag + bootstrap
|
||||
verification step" or split P04 into P04a (wire + log-only) and P04b
|
||||
(enforce). The must-haves MUST include "log-only mode exists and is
|
||||
the default for first run."
|
||||
|
||||
## Axis 5 — Dependencies
|
||||
|
||||
**Verdict**: PASS | **Confidence**: 0.84
|
||||
|
||||
The dependency graph is correct:
|
||||
|
||||
- P04 depends on P03 (scheduler context) — **weak dependency**. The
|
||||
plan says "P0 (P03 for scheduler context)" which means P04 can
|
||||
proceed without P03 but benefits from it. This is correct: ACL
|
||||
wiring in daemon handlers doesn't strictly require the scheduler.
|
||||
- P06 depends on P05 (seal) — **correct**: `auth init-idp` needs the
|
||||
seal infrastructure for the OIDC token exchange.
|
||||
- P10 depends on P04 (acl denials metric) and P05 (audit chain head)
|
||||
— **correct**: the metrics reference features wired in those phases.
|
||||
- P11 depends on P01..P10 — **correct**: docs reflect reality.
|
||||
- P12 depends on P03 (scheduler for UAT) and P04 (ACL for UAT) —
|
||||
**correct**: the UAT exercises both.
|
||||
|
||||
**Hidden dependency**: P12 (UAT signoff script) depends on P05 (seal)
|
||||
and P06 (auth init-idp) being functional — the UAT must exercise
|
||||
seal/unseal and the OIDC flow. But the plan's dependency table says
|
||||
P12 depends only on P03 and P04. This is incomplete.
|
||||
|
||||
**Binding condition C-46**: P12 (UAT plan + signoff script) MUST
|
||||
declare dependencies on P05 (seal) and P06 (auth init-idp) in
|
||||
addition to P03 and P04. The UAT signoff script will assert
|
||||
seal/unseal round-trip and OIDC health check claims — both require
|
||||
P05/P06 to be shipped. If P05 or P06 slip, the corresponding UAT
|
||||
assertions fail (honest signal per C-42), but the dependency must be
|
||||
declared.
|
||||
|
||||
## Axis 6 — Testing
|
||||
|
||||
**Verdict**: CONDITIONAL | **Confidence**: 0.76
|
||||
|
||||
The testing strategy is generally sound: each phase has a Wave 2/3
|
||||
with regression tests. 128 test files exist. The security integration
|
||||
test suite (`tests/security_integration_test.go`) is extended in P02
|
||||
and P04.
|
||||
|
||||
### UAT signoff script concerns
|
||||
|
||||
The `scripts/uat-signoff.sh` (P12 T4) is ~35 assertions, idempotent,
|
||||
read-only. This is the v1.0 gate. Concerns:
|
||||
|
||||
1. **No assertion for F26 (scheduler actually deploys remotely)**:
|
||||
the plan says the UAT exercises "deploy full stack" but the
|
||||
signoff script's ~35 assertions are not enumerated. If the script
|
||||
doesn't assert "job ran on remote node, not local," the headline
|
||||
fix (F26) is not validated.
|
||||
2. **No assertion for F48 (ACL deny-by-default)**: the UAT must
|
||||
include a negative test (unauthorized identity denied). But the
|
||||
script is "read + non-mutating" — how does it test denial without
|
||||
attempting a mutation? It could check `acl.json` mode (0600) and
|
||||
the audit log for denial entries, but that's indirect.
|
||||
3. **`uat-smoke.sh` in CI**: the pure-CLI subset runs in `.coreci.yml`
|
||||
validate. This is good. But "version, acl file mode, doctor modes,
|
||||
no-password grep, metrics shape" is 5 assertions — the smoke test
|
||||
doesn't validate the core deployment model.
|
||||
|
||||
**Binding condition C-47**: P12 T4 (`uat-signoff.sh`) MUST include
|
||||
explicit assertions for: (a) job deployed to remote node (not local
|
||||
exec) — verify via `orca job list` showing node_id != localhost; (b)
|
||||
ACL deny-by-default — verify via audit log containing denial entries
|
||||
or a documented negative assertion; (c) seal/unseal round-trip; (d)
|
||||
OIDC health check (`doctor oidc`). The ~35 assertion count MUST
|
||||
include these 4 critical-path claims. The assertion list must be
|
||||
reviewable in `docs/uat.md` before the UAT is run.
|
||||
|
||||
## Axis 7 — Security
|
||||
|
||||
**Verdict**: PASS | **Confidence**: 0.86
|
||||
|
||||
The plan closes all critical/high/medium security findings (F26-F95).
|
||||
The 11 injection vectors (P02) are each small and independently
|
||||
testable. The ACL wiring (P04) is deny-by-default with bootstrap. The
|
||||
seal (P05) has Shamir recovery (C-35). Key zeroing (P05 T6) is
|
||||
defense-in-depth.
|
||||
|
||||
### New risks introduced by fixes
|
||||
|
||||
1. **P03 removes local exec path**: if the local fallback has a bug,
|
||||
`job run` breaks for all single-node users. Mitigation: C-44
|
||||
(fail-closed on SSH failure, test the fallback).
|
||||
2. **P04 ACL wiring**: deny-by-default could block legitimate traffic
|
||||
if the SVID extraction is wrong. Mitigation: C-45 (log-only mode
|
||||
first).
|
||||
3. **P05 seal**: if `orca cluster seal` is run accidentally, the
|
||||
cluster is sealed. Mitigation: Shamir shards are printed (operator
|
||||
must store them); `unseal` requires OIDC token or 3-of-5 shards.
|
||||
This is by design.
|
||||
4. **P06 Dex deploy**: introduces a new network service (Dex on
|
||||
Traefik). Mitigation: mTLS-only fallback (C-37), Traefik dynamic
|
||||
route is behind the orca CA.
|
||||
|
||||
No new risks are unmitigated. The 9 accepted residual risks are
|
||||
documented and reasonable.
|
||||
|
||||
## Axis 8 — Operability
|
||||
|
||||
**Verdict**: CONDITIONAL | **Confidence**: 0.72
|
||||
|
||||
### 3-host topology realism
|
||||
|
||||
The UAT topology (lead Ubuntu 22.04 + pve01 Proxmox VE 8/9 + worker01
|
||||
Ubuntu 22.04) is minimal and correct. It covers both node types
|
||||
(Proxmox + Linux) and migrate-between-hosts.
|
||||
|
||||
**Concern**: The UAT requires a real Proxmox VE host. This is not a
|
||||
CI-environment artifact — the operator must have a Proxmox server
|
||||
available. If the operator doesn't have one, the UAT cannot run. The
|
||||
plan does not address this prerequisite. `uat-smoke.sh` (CI subset)
|
||||
does NOT require Proxmox — it's pure-CLI — but the full
|
||||
`uat-signoff.sh` does.
|
||||
|
||||
**Binding condition C-48**: `docs/uat.md` (P12 T3) MUST document the
|
||||
hardware/host prerequisites explicitly: "You need a Proxmox VE 8/9
|
||||
host with SSH access and root credentials." If the operator cannot
|
||||
provision a Proxmox host, an alternative UAT path (3x Ubuntu hosts,
|
||||
`--type linux` only, Proxmox claims marked as "not exercised in this
|
||||
UAT") MUST be documented. The signoff script MUST report which claims
|
||||
were exercised vs. skipped, so a partial UAT is an honest signal, not
|
||||
a false pass.
|
||||
|
||||
### Operator ability to run the UAT
|
||||
|
||||
The UAT is operator-driven: `docs/uat.md` walks through the build,
|
||||
`uat-signoff.sh` asserts. The plan says "the operator runs it, pastes
|
||||
output back to the CI agent." This requires:
|
||||
|
||||
1. The operator has 3 hosts available (see C-48).
|
||||
2. The operator can follow `docs/uat.md` step-by-step (it must be
|
||||
complete and exact).
|
||||
3. `uat-signoff.sh` is truly idempotent and read-only (D-254).
|
||||
|
||||
These are achievable. The risk is that `docs/uat.md` is incomplete
|
||||
(missing a step) and the operator gets stuck. The plan's T3 says
|
||||
"step-by-step with exact commands" — this is the right intent.
|
||||
|
||||
## Axis 9 — Completeness
|
||||
|
||||
**Verdict**: CONDITIONAL | **Confidence**: 0.74
|
||||
|
||||
### Will this be the LAST round?
|
||||
|
||||
The research claims "this is the last hardening round" based on three
|
||||
deep sweeps. The 9 accepted residual risks are documented. But:
|
||||
|
||||
1. **UAT will surface new gaps**: the UAT signoff script exercises
|
||||
~35 claims against a real 3-host cluster. This is the first time
|
||||
the full stack is exercised end-to-end. It is virtually certain
|
||||
that the UAT will discover issues not found in code review (e.g.,
|
||||
systemd unit rendering on Proxmox, SSH-push to Ubuntu worker,
|
||||
Traefik route conflicts, drift event delivery across node types).
|
||||
The plan does not budget for a "UAT findings" follow-up.
|
||||
2. **P06 (Dex deploy) is untested in CI**: the plan's T5 is a
|
||||
"hermetic Dex+Traefik config render test" — this tests config
|
||||
rendering, not actual deployment. The first real Dex deploy will
|
||||
be in the UAT. If it fails, that's a round 3.
|
||||
3. **`--type linux` (P12 T1) is new code**: the first real Ubuntu
|
||||
worker onboarding will be in the UAT. If `internal/linux/bootstrap.go`
|
||||
has bugs, that's a round 3.
|
||||
|
||||
**Binding condition C-49**: The plan MUST acknowledge that v0.13 is
|
||||
"the last hardening round *before UAT*," not "the last hardening round
|
||||
*absolute*." The UAT will likely surface 3-7 issues requiring a
|
||||
follow-up patch round (v0.13.1 or a small v0.14). This is healthy and
|
||||
expected. The v1.0.0 tag is gated on UAT signoff passing — if UAT
|
||||
finds issues, v1.0.0 is deferred until they're fixed. The plan's
|
||||
"v1.0.0 NOT cut (deferred for UAT signoff)" in P13 is correct, but
|
||||
the narrative "this is the last hardening round" should be softened to
|
||||
"this is the last hardening round before UAT validation."
|
||||
|
||||
### What could force a round 3?
|
||||
|
||||
1. UAT discovers Dex deploy doesn't work on real Proxmox.
|
||||
2. UAT discovers `--type linux` bootstrap fails on real Ubuntu 22.04.
|
||||
3. UAT discovers scheduler bin-packing produces bad placements on
|
||||
heterogeneous nodes (Proxmox vs Linux worker).
|
||||
4. UAT discovers seal/unseal doesn't work with real OIDC tokens (not
|
||||
just test mocks).
|
||||
5. P03's local fallback has an edge case (e.g., job with `--target`
|
||||
but target node deregistered mid-flight).
|
||||
|
||||
Each of these is a single-fix patch, not a full round. The plan's
|
||||
per-phase tag structure (v0.12.x) supports patch releases.
|
||||
|
||||
## Summary Verdict
|
||||
|
||||
| Axis | Verdict | Confidence |
|
||||
|------|---------|-----------|
|
||||
| 1. Feasibility | PASS | 0.82 |
|
||||
| 2. Scope | PASS | 0.85 |
|
||||
| 3. Cost | PASS | 0.78 |
|
||||
| 4. Risk | CONDITIONAL | 0.80 |
|
||||
| 5. Dependencies | PASS | 0.84 |
|
||||
| 6. Testing | CONDITIONAL | 0.76 |
|
||||
| 7. Security | PASS | 0.86 |
|
||||
| 8. Operability | CONDITIONAL | 0.72 |
|
||||
| 9. Completeness | CONDITIONAL | 0.74 |
|
||||
|
||||
**Overall**: **CONDITIONAL PROCEED** | **Confidence**: 0.82
|
||||
|
||||
The plan is evidence-accurate, well-decomposed, and addresses real
|
||||
gaps. The binding conditions (C-44..C-49) are targeted fixes, not
|
||||
fundamental rework. No axis FAILs. The plan proceeds once the 6
|
||||
binding conditions are incorporated.
|
||||
|
||||
## Binding decisions (G-255..G-261)
|
||||
|
||||
| ID | Decision | Rationale | Confidence | Alternatives |
|
||||
|----|----------|-----------|------------|--------------|
|
||||
| G-255 | Proceed with v0.12 governance gap: P13 marks REQ-130..148 AND REQ-149..163 Complete; v0.12 stays COMPLETE retroactively | v0.12 code was shipped but not wired; v0.13 wires it; revoking COMPLETE destabilizes history without changing code; C-43 makes consistency enforceable | 0.90 | Revoke v0.12 COMPLETE (destabilizing); escalate (unnecessary at full autonomy) |
|
||||
| G-256 | P03 fail-closed on SSH failure (C-44) | Silent local fallback when SSH fails is a data-integrity risk; local fallback only when len(nodes)==0 | 0.88 | Silent fallback (unsafe); no fallback (breaks single-node) |
|
||||
| G-257 | P04 log-only mode for first run (C-45) | C-40 specifies staged rollout but P04 task list omits it; deny-by-default lockout is catastrophic | 0.85 | Enforce immediately (lockout risk); split P04 into P04a/P04b (acceptable alternative) |
|
||||
| G-258 | P12 declares dependency on P05+P06 (C-46) | UAT exercises seal/unseal and OIDC flow, which require P05/P06; undeclared dependency hides slip risk | 0.82 | Leave undeclared (C-42 honest signal covers it, but dependency should be explicit) |
|
||||
| G-259 | P12 signoff script includes 4 critical-path assertions (C-47) | F26 (remote deploy), F48 (ACL deny), seal round-trip, OIDC health are the headline claims; without asserting them the UAT is theater | 0.84 | Trust the ~35 count (insufficient); add more later (gate must be complete at ship) |
|
||||
| G-260 | P12 docs/uat.md documents Proxmox prerequisite + alternative path (C-48) | UAT requires real Proxmox host; if operator lacks one, partial UAT must be honest signal | 0.78 | Assume operator has Proxmox (may not); skip Proxmox claims silently (dishonest) |
|
||||
| G-261 | v0.13 is "last round before UAT," not "last round absolute" (C-49) | UAT will surface issues; narrative should reflect this; v1.0.0 deferred until UAT passes is correct | 0.80 | Claim "last round absolute" (likely false); pre-commit to v0.14 (premature) |
|
||||
|
||||
## Binding conditions (C-44..C-49)
|
||||
|
||||
| ID | Condition | Phase | Gates |
|
||||
|----|-----------|-------|-------|
|
||||
| C-44 | P03 MUST fail-closed when scheduler selects a node but SSH-push fails (return error, no silent local fallback). Local fallback only when len(registeredNodes)==0. Test case mandatory. | P03 | P04 ship |
|
||||
| C-45 | P04 MUST implement log-only/dry-run mode as default for first invocation after ACL wiring. Enforce mode enabled after bootstrap ACL verified. Add to P04 task list + must-haves. | P04 | P05 ship |
|
||||
| C-46 | P12 dependency table MUST include P05 (seal) and P06 (auth init-idp) in addition to P03 and P04. | P12 | P12 plan accuracy |
|
||||
| C-47 | P12 uat-signoff.sh MUST include explicit assertions for: (a) job deployed to remote node (node_id != localhost), (b) ACL deny-by-default (audit log denial entries or documented negative assertion), (c) seal/unseal round-trip, (d) OIDC health check. Assertion list reviewable in docs/uat.md. | P12 | v1.0.0 gate |
|
||||
| C-48 | P12 docs/uat.md MUST document hardware/host prerequisites (Proxmox VE 8/9 host required). Alternative UAT path (3x Ubuntu, --type linux only, Proxmox claims skipped) MUST be documented. Signoff script reports exercised vs. skipped claims. | P12 | UAT executability |
|
||||
| C-49 | Plan narrative MUST soften "last hardening round" to "last hardening round before UAT validation." UAT will likely surface 3-7 issues requiring patch release. v1.0.0 deferred until UAT passes. | P0/P13 | Expectation setting |
|
||||
|
||||
## Escalations
|
||||
|
||||
None. All axes resolved at confidence >= 0.72. The question tool
|
||||
infrastructure failed during the interactive grill (stack overflow on
|
||||
every invocation); given `autonomy.level=full` and
|
||||
`workflow.no_hitl=true`, the grill proceeded on evidence alone. All
|
||||
binding decisions are evidence-based and within the agent's autonomy
|
||||
threshold (0.60).
|
||||
|
||||
## What the auditor would flag
|
||||
|
||||
1. **v0.12 COMPLETE with 19 pending REQs** — documentation governance
|
||||
failure, now acknowledged and resolved (G-255).
|
||||
2. **P03 under-estimated as "wiring"** — it's a behavioral rewrite of
|
||||
`job run`. Schedule accordingly.
|
||||
3. **P04 staged rollout missing from task list** — C-40 describes it,
|
||||
P04 tasks omit it (C-45).
|
||||
4. **P12 dependencies incomplete** — P05/P06 not listed (C-46).
|
||||
5. **UAT signoff assertions not enumerated** — ~35 count without a
|
||||
reviewable list (C-47).
|
||||
6. **"Last round" narrative overclaims** — UAT will find issues
|
||||
(C-49).
|
||||
|
||||
## What the project is not doing that it should
|
||||
|
||||
1. **No end-to-end integration test in CI** — the UAT is the first
|
||||
E2E test. The `uat-smoke.sh` is CLI-only. A CI E2E test (mock SSH
|
||||
to localhost containers) would catch P03/P04 integration issues
|
||||
before UAT. This is deferred to v1.x and is acceptable.
|
||||
2. **No performance testing** — the plan doesn't address scheduler
|
||||
performance on large node counts. Acceptable for a 3-host UAT;
|
||||
relevant for v1.x.
|
||||
3. **No chaos testing** — SSH failure mid-deploy, node deregistration
|
||||
mid-flight, etc. C-44 covers the fail-closed case; broader chaos
|
||||
testing is v1.x.
|
||||
|
||||
## Simplest version delivering 80% of value
|
||||
|
||||
P01 (toolchain) + P02 (injection) + P03 (scheduler) + P04 (ACL) +
|
||||
P12 (UAT) = 5 phases. This makes the deployment model functional,
|
||||
closes the RCE vectors, wires zero-trust, and delivers the UAT gate.
|
||||
The remaining 9 phases (seal, IdP, concurrency, transport, migration,
|
||||
metrics, docs, linux type) each close real gaps but could defer to
|
||||
v1.0.1 patches. The operator chose comprehensiveness (D-250) —
|
||||
justified to avoid a round 3, but the 5-phase core is the minimum
|
||||
viable path.
|
||||
|
||||
## What must be true for success in 90 days
|
||||
|
||||
1. P03 ships with fail-closed SSH handling and local fallback (C-44).
|
||||
2. P04 ships with log-only mode and bootstrap ACL (C-45).
|
||||
3. P12 ships with enumerated assertions covering the 4 critical paths
|
||||
(C-47).
|
||||
4. The operator has a 3-host environment (or the alternative UAT path
|
||||
is documented, C-48).
|
||||
5. The UAT signoff script runs and either passes (-> v1.0.0) or fails
|
||||
honestly (-> patch round).
|
||||
|
||||
All five are achievable. The plan proceeds.
|
||||
@@ -1,225 +0,0 @@
|
||||
# GRILL v0.14: Ingress Bootstrap Completeness — Red-Team Review
|
||||
|
||||
**Date**: 2026-08-10
|
||||
**Reviewer**: ci-griller (adversarial)
|
||||
**Subject**: PLAN_v0.14.md (9 phases, P0–P8)
|
||||
**Confidence in plan as-written**: **0.45 — RETHINK** (was Proceed-eligible until the mTLS scope expansion was chosen)
|
||||
**Verdict**: **RETHINK** — the plan is technically grounded in strong research but contains (a) one outright correctness defect that will break execution, (b) two requirements/plan contradictions that ship broken config, (c) one scope expansion chosen during this grill that adds a 10th phase the plan does not contain, and (d) one regression of a Completed requirement. The research is the strongest artifact; the plan diverges from it in load-bearing ways.
|
||||
|
||||
---
|
||||
|
||||
## How this grill was conducted
|
||||
|
||||
Every plan claim was checked against the actual codebase (`internal/traefik/install.go`, `internal/emitter/nft.go`, `internal/emitter/traefik.go`, `internal/cli/init.go`, `internal/cli/upgrade.go`, `internal/cli/doctor_nft.go`, `internal/proxmox/bootstrap.go`, `internal/linux/bootstrap.go`, `internal/store/migrations/`, `internal/certpaths/`, `internal/sshpush/`, `scripts/release.sh`, `.coreci.yml`, `Dockerfile`, git tags, git log). Findings cite file:line. Four binding questions were asked interactively; answers recorded as G-001..G-004 below.
|
||||
|
||||
---
|
||||
|
||||
## Per-Axis Findings
|
||||
|
||||
### Axis 1 — Feasibility
|
||||
|
||||
**Finding F1.1 (BLOCKER — migration number collision).** PLAN P5 T3 (line 194) specifies "Schema migration 0007: `ALTER TABLE nodes ADD COLUMN ingress_mode TEXT DEFAULT ''`". Migration 0007 **already exists** — `internal/store/migrations/0007_certs_serial_unique.sql` (added v0.7, P1-001). The current head is `0008_audit_tamper_evidence.sql`. The migrator runs files in lexical order and records applied versions in `schema_migrations`. Reusing 0007 will either (a) silently no-op on DBs that already recorded 0007, leaving `ingress_mode` un-added, or (b) break the migration ledger. This is a guaranteed execution-time defect.
|
||||
**Evidence**: `internal/store/migrations/0007_certs_serial_unique.sql`, `internal/store/migrations/0008_audit_tamper_evidence.sql`, `internal/cli/init_test.go:84` (test asserts head = 0008).
|
||||
**Resolved by**: G-001 (use `0009_ingress_mode.sql`).
|
||||
|
||||
**Finding F1.2 (BLOCKER — `certpaths.CAPath()` does not exist).** PLAN P3 T5 step 2 (line 142) says "Push cluster root CA … from `certpaths.CAPath()` if exists". The function `certpaths.CAPath()` does not exist. The real API is `certpaths.CACertPath()` (`internal/certpaths/certpaths.go:35`) and `certpaths.CAKeyPath()` (`:39`). P4 T1 step 2 repeats the same phantom reference. This will not compile.
|
||||
**Evidence**: `internal/certpaths/certpaths.go:35` (`func CACertPath() string`); grep for `CAPath` returns zero matches in `internal/certpaths/`.
|
||||
|
||||
**Finding F1.3 (no `install_test.go` to rewrite).** PLAN P2 T10 (line 112) says "rewrite `internal/traefik/install_test.go`". That file does not exist — `internal/traefik/` contains only `install.go` (glob confirms). The task is "create", not "rewrite". Minor, but signals the plan was written against an imagined codebase shape, not the real one.
|
||||
**Evidence**: `glob internal/traefik/*.go` → only `install.go`.
|
||||
|
||||
**Finding F1.4 (good — fake-SSH harness exists and is adequate).** PLAN P7 T8 claims a "hermetic fake-SSH harness" can assert `pct create` + `podman run` inside LXC. Verified: `internal/proxmox/ssh_session_test.go:23` (`fakeSSHServer`) and `internal/sshpush/transport_test.go:27` implement an in-process SSH server with `runCommand(cmd)` pattern-matching. It can assert the right commands are *sent* (e.g. `pct create … --features nesting=1,keyctl=1,fuse=1`). It cannot actually create an LXC or run podman — but the plan only claims command assertion, which is achievable. The harness is real and reusable.
|
||||
|
||||
**Finding F1.5 (good — callsites verified).** PLAN P2 T3/T4/T5 reference `init.go:254-266`, `linux/bootstrap.go:160-172`, `proxmox/bootstrap.go:250-255`. Verified: `init.go:256` calls `installTraefikLocal()`; `linux/bootstrap.go:170` calls `traefik.InstallRemote("", sshExecFn)`; `proxmox/bootstrap.go:253` calls `traefik.InstallRemote("", runRemote)`. The line numbers are accurate within a few lines. The plan was written against the real callsites.
|
||||
|
||||
### Axis 2 — Scope
|
||||
|
||||
**Finding F2.1 (BLOCKER — mTLS scope expansion chosen, no phase exists).** During this grill (G-003) the operator chose "Wire real mTLS now" over the plan's `tls: {}` default-cert approach. This requires step-ca to mint server certs into `/etc/traefik/dynamic` + dynamic `tls.certificates` + `tls.options.default.clientAuth.caFiles`. **Step-ca server-cert minting was deferred since v0.11 and is not implemented.** The 9-phase plan contains no such phase. The operator then chose (G-004-adjacent) to add a step-ca cert minting phase, growing v0.14 from 9 to 10 phases. The plan as written does not reflect this. Until a P-step for cert minting is added, the plan is incomplete relative to the chosen direction.
|
||||
**Impact**: v0.14 cannot ship real mTLS on its current 9 phases. Either add the phase (10 phases, more risk) or revert to `tls: {}` and defer mTLS to v0.15.
|
||||
|
||||
**Finding F2.2 (regression — `traefik-on-public-ip` opt-out lost).** REQ-100 (Complete, v0.11) established the `--public-binding=traefik-on-public-ip` opt-out: traefik binds `:443`/`:80` directly instead of `127.0.0.1:8443`/`8080` + nft DNAT. The existing `RenderTraefikStaticConfig` (`internal/emitter/traefik.go:273`) implements this via `TraefikStaticOpts.PublicBinding`. The v0.14 plan's baked image (`docker/orca-traefik/traefik.yml`, PLAN lines 33-37) hard-codes `127.0.0.1:8443`/`8080` — there is no opt-out path in the image. Baking the static config into the image freezes out the opt-out mode that v0.11 shipped. This is a regression of a Completed requirement.
|
||||
**Evidence**: `internal/emitter/traefik.go:255-266` (`publicWebSecure`/`publicWeb` switch on `PublicBinding`); `internal/emitter/traefik.go:282-300` (rendered static config); REQUIREMENTS REQ-100 line 218.
|
||||
|
||||
**Finding F2.3 (9 phases is borderline; 10 is too many).** Even before the mTLS expansion, 9 phases for an "ingress bootstrap completeness" milestone is heavy. P5 (proxmox native) and P6 (floating-IP) are the two most complex (LXC creation + podman-in-LXC + apt-get + nft-inside-LXC). They could potentially be merged into one phase with two code paths, since they share `ProvisionIngressLXC` plumbing. With the mTLS expansion, 10 phases is too many for a single milestone — split v0.14 into v0.14a (linux ingress + mTLS) and v0.14b (proxmox ingress).
|
||||
|
||||
**Finding F2.4 (good — research-validated decisions are sound).** The 7 research topics (nft postrouting, pve-firewall priority, LXC features, traefik Dockerfile, pct create syntax, podman restart, SELinux) are well-sourced and the recommended approaches are technically correct. This is the strongest artifact in the v0.14 dossier.
|
||||
|
||||
### Axis 3 — Dependencies
|
||||
|
||||
**Finding F3.1 (good — ordering is correct).** P1 (image) → P2 (reconciler) → P3 (nft+init) → P4 (remote linux) → P5 (proxmox native) → P6 (floating-IP) → P7 (doctor+tests) → P8 (ship). Each phase references the prior phase's output (P3 T6 calls `EnsureTraefikContainerLocal` from P2; P5 T4 step 10 calls `EnsureTraefikContainerRemote` from P2). No phase can ship before its prerequisite.
|
||||
|
||||
**Finding F3.2 (hidden dependency — P2 T8 depends on P1 T1).** P2 T8 (drop `certResolver: orca`) edits `internal/emitter/traefik.go`. P1 T1 bakes the static config. The dynamic config (`traefik.go`) and static config (`docker/orca-traefik/traefik.yml`) must be consistent: if the static config has no `certificatesResolvers.orca` (P1 T1 correctly omits it) but the dynamic config still references `certResolver: orca` (until P2 T8), traefik logs a warning on every reload. The plan orders P1 before P2, so there is a window (P1 shipped, P2 not yet) where the published image + the live dynamic config are inconsistent. This is acceptable only if P1 and P2 ship in quick succession; if P1 stalls, the image is published with a known TLS-config mismatch.
|
||||
|
||||
### Axis 4 — Security
|
||||
|
||||
**Finding F4.1 (good — nft injection guard already present).** `internal/emitter/nft.go:104-130` (`partitionTrustedProbes`) validates every `TrustedProbes` entry as IP/CIDR before rendering (F9 guard). The v0.14 plan adds `DNATTarget` (P3 T1) — this is a NEW string field rendered directly into `dnat to <DNATTarget>:8443`. **The plan does not specify validation of `DNATTarget`.** If `DNATTarget` is user-controllable (via `--floating-ip` or cluster config), an unvalidated value is an nft-syntax injection vector. The existing F9 guard covers `TrustedProbes`; `DNATTarget` needs the same treatment.
|
||||
**Binding**: C-51 (see below).
|
||||
|
||||
**Finding F4.2 (container escape surface — LXC nesting).** `--features nesting=1,keyctl=1,fuse=1` (P5/P6) is the documented requirement for podman-in-LXC, but nesting exposes host procfs/sysfs to the guest (Proxmox docs, RESEARCH Topic 3). This is an accepted tradeoff for container-in-container, but the plan does not document the threat-model acceptance. For a "production hardening" lineage, the ingress LXC is now a privileged-ish surface (nesting+keyctl) running a podman container pulling an image from a registry. The supply chain is: `git.cloudinit.dev/coreci/orca-traefik:<version>` (P1). If the registry is compromised or the tag is re-pushed, the ingress LXC runs attacker code at the host-LXC boundary. The plan has no image-signing/verification step (`podman pull --cert-dir` or cosign).
|
||||
**Binding**: C-52 (see below).
|
||||
|
||||
**Finding F4.3 (nft rules injection via re-apply).** P3 T4 changes `flush table` to `delete table`. RESEARCH Topic 1 establishes that `delete table` on a missing table errors, and the fix is to pre-create the table (`nft add table inet orca-ingress 2>/dev/null || true`) before `nft -f`. P3 T5 step 4 does this. Good. But the re-apply path (P3 T5 step 5, P4 T1 step 5) runs `nft -f` which replaces the whole table — if a concurrent process (pve-firewall, operator) adds rules to `orca-ingress` between the pre-create and the `nft -f`, they are wiped. This is by design (orca owns the table) but should be documented as "orca is the sole owner of `table inet orca-ingress`".
|
||||
|
||||
### Axis 5 — Operational
|
||||
|
||||
**Finding F5.1 (BLOCKER — no upgrade path from v0.13 binary+systemd to v0.14 podman).** P2 T6 (line 104) says "update legacy cutover to pull new image + recreate container instead of sed-ing traefik.yml". But `internal/cli/upgrade.go:244-407` implements a Traefik `:443` → `127.0.0.1:8443` cutover that does `systemctl restart traefik` (line 382) — it restarts the **systemd service**, not a container. P2 removes systemd unit generation. There is **no phase** that: (a) detects the legacy `orca-traefik.service`, (b) stops+disables it, (c) removes `/usr/local/bin/traefik` + `/etc/systemd/system/orca-traefik.service`, (d) then runs `EnsureTraefikContainerLocal`. Without this, upgrading a live v0.13 cluster leaves a dead systemd unit AND a new podman container both trying to bind `127.0.0.1:8443` → port conflict, traefik down.
|
||||
**Resolved by**: G-004 (P2 T6 must remove legacy unit + binary, idempotent, tested).
|
||||
|
||||
**Finding F5.2 (podman/podman-restart.service unavailable).** P2 T2 enables `podman-restart.service`. RESEARCH Topic 6 establishes this is not enabled by default on Ubuntu 24.04 and must be enabled. But the plan does not specify what happens if `podman` is not installed on the target host (a fresh linux node join, P4). `EnsureTraefikContainerLocal`/`Remote` calls `podman pull`/`podman run` — if `podman` is absent, this fails. The plan has no "install podman first" step for the linux topology (P3/P4). For proxmox (P5/P6) the plan installs podman via `apt-get install -y podman conmon crun fuse-overlayfs` inside the LXC (T4 step 4 / T1 step 6). But for linux nodes (P3/P4), there is no podman-install step. This is an under-specification.
|
||||
**Binding**: C-50 (see below).
|
||||
|
||||
**Finding F5.3 (good — `--network host` is correct).** RESEARCH Topic 3 confirms `--network host` inside an LXC binds the LXC's netns, so traefik binds `127.0.0.1:8080/8443` on the LXC loopback and nft on the PVE host DNATs to the LXC IP. This is sound. The `DNATTarget` parameterization (D-262) correctly distinguishes `127.0.0.1` (linux/localhost) from `<lxc-ip>` (proxmox native).
|
||||
|
||||
**Finding F5.4 (reboot persistence chain has a gap).** RESEARCH Topic 6 establishes the chain: Proxmox boot → `--onboot 1` starts LXC → LXC systemd starts → `podman-restart.service` restarts container. P5 T4 step 6 and P6 T1 step 8 enable `podman-restart.service` inside the LXC. But `--onboot 1` starts the LXC **after** the Proxmox host's network is up — if the floating IP is on a bridge that depends on a physical link that's slow to come up, the LXC may start before the bridge is ready, and the floating-IP `eth0` config may fail. The plan does not address LXC-start ordering relative to bridge readiness.
|
||||
|
||||
### Axis 6 — Testing
|
||||
|
||||
**Finding F6.1 (good — harness is real).** `fakeSSHServer` (Axis 1 F1.4) can assert `pct create` with the right `--features` and `net0` args, `apt-get install podman`, `podman run` with `--network host`. P7 T8's claims are achievable.
|
||||
|
||||
**Finding F6.2 (gap — no test for the v0.13→v0.14 upgrade).** G-004 requires the upgrade path to be tested. The plan's P2 Wave 4 (T10) tests the reconciler but not the legacy-removal path. There is no test that: (a) simulates a host with `orca-traefik.service` present, (b) runs the upgrade, (c) asserts the unit is stopped+disabled+removed, (d) asserts the podman container is running. This must be added to P2 T10.
|
||||
|
||||
**Finding F6.3 (gap — no test for DNATTarget validation).** F4.1 identifies `DNATTarget` as an injection vector. P3 T8 (nft_test) asserts `DNATTarget` substitution but does not specify a test for invalid `DNATTarget` values (e.g. `1.2.3.4:8443; flush ruleset`). The existing F9 guard test pattern should be extended.
|
||||
|
||||
**Finding F6.4 (good — doctor_nft extension is incremental).** P3 T7 extends `doctor_nft.go`. The existing file (`internal/cli/doctor_nft.go`) is well-structured with `nftCheckResult` lines; adding postrouting/masquerade/DNATTarget assertions is straightforward.
|
||||
|
||||
### Axis 7 — Performance
|
||||
|
||||
**Finding F7.1 (apt-get install podman on every proxmox join — 30-60s).** CLARIFY D-263 acknowledges "~30-60s to the join time". P5 T4 step 4 and P6 T1 step 6 run `apt-get update && apt-get install -y podman conmon crun fuse-overlayfs nftables` inside the LXC on every join. This is acceptable for a one-time bootstrap but painful if re-run. The reconciler must be idempotent (skip if podman already installed). The plan does not specify an idempotency check for the apt-get step.
|
||||
**Binding**: C-53 (see below).
|
||||
|
||||
**Finding F7.2 (image pull latency).** `podman pull orca-traefik:<tag>` (P2 T1 step 3, P5/P6) pulls from `git.cloudinit.dev/coreci/`. On a proxmox host behind a slow link, this can take 10-30s for a ~150MB traefik image. The plan has no pull-timeout. If the registry is unreachable (offline-first is R-001!), the pull fails and traefik never starts. **R-001 (offline-first) is violated**: the plan depends on a registry pull at bootstrap time. There is no "pre-pull" or "bundle image into the LXC template" fallback.
|
||||
**Binding**: C-54 (see below) — this is a tension with R-001 that the plan does not acknowledge.
|
||||
|
||||
**Finding F7.3 (nft re-apply disruption).** `nft -f` replaces the table atomically (single transaction). Existing connections are NOT disrupted (conntrack holds them). New connections during the apply window (<1ms) may be dropped. This is acceptable. No finding.
|
||||
|
||||
### Axis 8 — Cost
|
||||
|
||||
**Finding F8.1 (2 images per release — sustainable).** P1 adds `orca-traefik` image alongside `orca`. `.coreci.yml` gets a `container-publish-traefik` step (P1 T5). `scripts/release.sh` gets a second docker block (P1 T4). The traefik image is small (~150MB, Alpine-based). Registry storage: 2 images × N releases. At v0.13.x cadence (8 tags), that's 16 image-tags per milestone. Sustainable for a private Gitea registry. No finding.
|
||||
|
||||
**Finding F8.2 (good — release.sh extension is minimal).** P1 T4 adds ~8 lines to `scripts/release.sh` after line 212. Verified the insertion point (line 213 is end of existing docker block). Clean.
|
||||
|
||||
### Axis 9 — Completeness (3 topologies)
|
||||
|
||||
**Finding F9.1 (linux topology — covered by P3+P4).** `orca init` (P3) bootstraps nft+podman on the lead; `orca node join --type linux` (P4) does it remotely. Complete.
|
||||
|
||||
**Finding F9.2 (proxmox-native — covered by P5, but LXC IP discovery is hand-wavy).** P5 T4 step 9 says "Discover LXC IP via `pct config <vmid>` (parse `net0` line) or `pct exec <vmid> -- hostname -I`". But in native mode the LXC is created **without** a static IP (P5 T4 step 2 has no `ip=` in the `pct create` — unlike P6 which has `ip=<floating-ip>/<prefix>`). So the LXC gets a DHCP/bridge IP that is not known at create time. P5 step 9 discovers it after `pct start`. But step 10 then re-applies nft with the discovered IP. This is a two-phase apply: first apply with default `127.0.0.1` (wrong for native), then re-apply with LXC IP. The plan does not specify what happens to traffic between the first and second apply (it DNATs to 127.0.0.1:8443 on the PVE host where nothing listens → connections refused). There is a window of ingress downtime during native-mode bootstrap.
|
||||
**Binding**: C-55 (see below).
|
||||
|
||||
**Finding F9.3 (floating-IP — covered by P6, but MAC uniqueness is untested).** P6 T4 generates a random `02:XX:XX:XX:XX:XX` MAC in interactive mode. The plan does not check for MAC collision on the bridge. RESEARCH Topic 5 pitfall 2: "hwaddr must be unique on the bridge". A random 02: prefix has 46 bits of entropy — collision is unlikely on a single bridge but not impossible across a multi-node cluster.
|
||||
**Binding**: C-56 (see below).
|
||||
|
||||
**Finding F9.4 (no `localhost`/lead topology with podman).** P3 bootstraps the lead via `EnsureTraefikContainerLocal`. But `orca init` runs on the lead — if the lead has no podman installed (fresh host), `EnsureTraefikContainerLocal` fails. Same as F5.2 but for the lead. The plan assumes podman is present on the lead. No install step.
|
||||
|
||||
---
|
||||
|
||||
## Binding Conditions (gates that MUST be met before a phase ships)
|
||||
|
||||
| ID | Gate | Phase | Severity |
|
||||
|----|------|-------|----------|
|
||||
| **C-50** | P2 must not break existing `orca init` on a host without podman installed — either install podman as part of `BootstrapLocalIngress` (P3) or emit a clear error with install instructions. Same for `orca node join --type linux` (P4). | P2/P3/P4 | BLOCKER |
|
||||
| **C-51** | `NftClusterConfig.DNATTarget` must be validated as `net.ParseIP` or `ip:port` before rendering. Unvalidated values are an nft-syntax injection vector (same F9 guard as TrustedProbes). Test required. | P3 | BLOCKER |
|
||||
| **C-52** | The `orca-traefik` image supply chain must be documented: registry is public (anonymous pull per REQ-045), no image signing in v0.14. Document the threat-model acceptance: a compromised registry = attacker code in the ingress LXC. Add `podman image trust` or cosign verification as a v0.15 hardening item. | P1/P8 | High |
|
||||
| **C-53** | The `apt-get install podman` step inside the LXC (P5 T4 step 4, P6 T1 step 6) must be idempotent: check `command -v podman` first, skip if present. Re-running join on an existing LXC must not re-run apt-get. | P5/P6 | High |
|
||||
| **C-54** | R-001 (offline-first) tension: `podman pull` at bootstrap requires registry reachability. Either (a) document that ingress bootstrap requires online access (exception to R-001), or (b) pre-bundle the `orca-traefik` image into the LXC template / load from a local archive. The plan must acknowledge this tension explicitly. | P2/P5/P6 | BLOCKER |
|
||||
| **C-55** | P5 native-mode bootstrap must not create a window of ingress downtime. The first nft apply must use the LXC IP (discovered after `pct start` but before the first nft apply), OR the plan must accept and document the downtime window. Two-phase apply (default → LXC IP) is a transient outage. | P5 | High |
|
||||
| **C-56** | P6 MAC generation must check for collision against existing nodes' MACs in the cluster registry. Reject or regenerate on collision. | P6 | Medium |
|
||||
| **C-57** | P2 T6 must implement the v0.13→v0.14 upgrade: detect `orca-traefik.service`, stop+disable, remove `/usr/local/bin/traefik` + unit file, then `EnsureTraefikContainerLocal`. Idempotent. Tested with a simulated-legacy-host test (F6.2). | P2 | BLOCKER |
|
||||
| **C-58** | The baked `docker/orca-traefik/traefik.yml` must not regress REQ-100's `traefik-on-public-ip` opt-out. Either (a) bake both configs and select via env/flag, or (b) document that the opt-out is dropped in v0.14 and update REQ-100, or (c) mount the static config from host (not baked) so `RenderTraefikStaticConfig` still works. | P1 | High |
|
||||
| **C-59** | Migration for `ingress_mode` must be `0009_ingress_mode.sql`, NOT 0007 (already taken by certs_serial_unique). | P5 | BLOCKER |
|
||||
| **C-60** | `certpaths.CAPath()` references in P3 T5 and P4 T1 must be corrected to `certpaths.CACertPath()`. | P3/P4 | BLOCKER (compile) |
|
||||
| **C-61** | REQ-172 must be amended: `--restart=always` → `--restart=unless-stopped` (per RESEARCH Topic 6 + PLAN), and `:Z` → omitted (per G-002). The requirement text contradicts the plan and research. | P2 | High |
|
||||
| **C-62** | If mTLS is in-scope for v0.14 (per G-003), a new phase must be added implementing step-ca server-cert minting into `/etc/traefik/dynamic` + dynamic `tls.certificates` + `tls.options.default.clientAuth.caFiles`. The plan currently has no such phase (P2 T8 emits `tls: {}`). | NEW PHASE | BLOCKER |
|
||||
|
||||
---
|
||||
|
||||
## Phase Challenges (specific challenges a phase must overcome)
|
||||
|
||||
| ID | Challenge | Phase |
|
||||
|----|-----------|-------|
|
||||
| **PC-01** | P1 must bake a static config that doesn't regress the `traefik-on-public-ip` opt-out (REQ-100). Baking freezes the config; the opt-out needs a runtime switch. | P1 |
|
||||
| **PC-02** | P2 must handle the 3-way TLS contradiction (CLARIFY D-257 vs RESEARCH Topic 4 vs PLAN T8). Per G-003, real mTLS is chosen — P2 alone cannot deliver it; a new phase is needed. | P2 + new |
|
||||
| **PC-03** | P2 T6 must remove the legacy systemd unit + binary without breaking a running v0.13 cluster. The existing `upgrade.go` cutover logic (lines 244-407) must be rewritten to stop+disable+remove the unit, not `systemctl restart traefik`. | P2 |
|
||||
| **PC-04** | P3 must install podman on the lead if absent (C-50). The plan assumes podman is present. `BootstrapLocalIngress` must either install it or fail with a clear message. | P3 |
|
||||
| **PC-05** | P3 T5 must use `certpaths.CACertPath()` not the phantom `certpaths.CAPath()`. | P3 |
|
||||
| **PC-06** | P3 T1 (`DNATTarget`) must validate input (C-51). New string field rendered into nft ruleset — injection risk. | P3 |
|
||||
| **PC-07** | P3 T4 (first-apply flush-table fix) must work across nft versions. RESEARCH establishes `delete table` on missing table is version-dependent. The pre-create approach (`nft add table … 2>/dev/null \|\| true` before `nft -f`) is robust; the plan uses it (T5 step 4). Verify on Proxmox kernel. | P3 |
|
||||
| **PC-08** | P5 native-mode has a two-phase nft apply (default 127.0.0.1 → LXC IP) that creates a transient outage window. Must be eliminated or documented (C-55). | P5 |
|
||||
| **PC-09** | P5/P6 `apt-get install podman` inside LXC takes 30-60s and must be idempotent (C-53). Re-join must not re-install. | P5/P6 |
|
||||
| **PC-10** | P5/P6 `podman pull` requires registry reachability, violating R-001 (offline-first). Must be acknowledged or mitigated (C-54). | P5/P6 |
|
||||
| **PC-11** | P5/P6 LXC reboot chain (Proxmox boot → `--onboot 1` → LXC systemd → `podman-restart.service`) has a gap: LXC may start before bridge is ready. Floating-IP `eth0` config may fail. | P5/P6 |
|
||||
| **PC-12** | P6 MAC generation must check for collision (C-56). | P6 |
|
||||
| **PC-13** | P7 T8 integration test must include the v0.13→v0.14 upgrade path (F6.2): simulated legacy host → upgrade → assert unit removed + podman running. | P7 |
|
||||
| **PC-14** | If the new mTLS phase is added (G-003), it must mint server certs into `/etc/traefik/dynamic` atomically (C-10 protocol) and traefik must reload them via the file provider watch. Step-ca minting was deferred since v0.11 — this is net-new work, not a completion. | new phase |
|
||||
|
||||
---
|
||||
|
||||
## Binding Decisions (from interactive grill)
|
||||
|
||||
| ID | Decision | Rationale | Confidence | Alternatives rejected |
|
||||
|----|----------|-----------|------------|---------------------|
|
||||
| **G-001** | Migration for `ingress_mode` is `0009_ingress_mode.sql` | 0007 is already `certs_serial_unique`; 0008 is `audit_tamper_evidence`. Reusing 0007 breaks the migrator. | 0.95 | Renumber existing (breaks deployed DBs); accept collision (guaranteed defect) |
|
||||
| **G-002** | Omit `:Z` flag on volume mounts; use `:ro` on both | RESEARCH Topic 7: `:Z` relabels host dirs to private container label, blocks host-side orca writes on SELinux. No-op on Ubuntu/Proxmox but a latent footgun. PLAN P2 T1 + RESEARCH agree; CLARIFY D-258 + REQ-172 are wrong. | 0.90 | Keep `:Z` (wrong on future SELinux); `:z` shared (unnecessary) |
|
||||
| **G-003** | Wire real mTLS now (dynamic `tls.certificates` + `clientAuth.caFiles`) | Operator chose this over `tls: {}` default. Requires step-ca server-cert minting which is not implemented (deferred since v0.11). | 0.55 | `tls: {}` now (plan's approach, defers mTLS); keep `certResolver: orca` (broken — key doesn't exist in traefik v3.3) |
|
||||
| **G-004** | P2 T6 must remove legacy systemd unit + binary on upgrade from v0.13 | Without it, upgrading a live v0.13 cluster leaves a dead systemd unit + a new podman container both binding 127.0.0.1:8443 → port conflict. Idempotent + tested. | 0.90 | Leave legacy unit (port conflict); fresh-installs only (unacceptable) |
|
||||
|
||||
---
|
||||
|
||||
## Escalations (unresolved, confidence < 0.60)
|
||||
|
||||
| ID | Escalation | Confidence | Reason |
|
||||
|----|------------|------------|--------|
|
||||
| **E-001** | G-003 (real mTLS now) creates a scope expansion that the 9-phase plan does not contain. The operator chose to add a step-ca cert minting phase (growing v0.14 to 10 phases) but the plan has not been updated to reflect this. Until the new phase is specified (scope, tasks, tests), v0.14's mTLS direction is **undetermined**. The plan as written ships `tls: {}` (no mTLS), which contradicts G-003. | 0.55 | The choice is made but the plan does not reflect it. This is a plan-spec gap, not a technical unknown. |
|
||||
|
||||
---
|
||||
|
||||
## Meta — Closing Review
|
||||
|
||||
**What the auditor would flag**:
|
||||
1. The plan ships a known-broken TLS config. RESEARCH Topic 4 calls `certResolver: orca` "the biggest v0.14 finding" and says it does not exist in traefik v3.3. CLARIFY D-257 and REQ-171 still specify it. The plan (P2 T8) drops it but emits `tls: {}` (no real TLS). G-003 chose real mTLS, which the plan doesn't contain. **Three artifacts disagree on TLS.**
|
||||
2. The plan regresses REQ-100 (`traefik-on-public-ip` opt-out) by baking the static config.
|
||||
3. The plan has no upgrade path from v0.13 (binary+systemd) to v0.14 (podman). G-004 binds the fix but the plan must be updated.
|
||||
4. The plan references a phantom function (`certpaths.CAPath()`) and a phantom migration number (0007). Both will fail at compile/execution time.
|
||||
5. The plan's offline-first claim (R-001) is violated by `podman pull` at bootstrap (C-54).
|
||||
|
||||
**What the project is NOT doing that it should**:
|
||||
- Image supply-chain verification (no cosign, no `podman image trust`).
|
||||
- LXC-start ordering relative to bridge readiness (reboot persistence gap).
|
||||
- Idempotency check for `apt-get install podman` inside LXC.
|
||||
- MAC collision check on the bridge.
|
||||
- A test for the v0.13→v0.14 upgrade path.
|
||||
|
||||
**Simplest 80%-value version**: Ship P1 (image) + P2 (reconciler, with legacy-removal) + P3 (nft+init) + P4 (linux remote) + P7 (doctor+tests). Defer P5 (proxmox native) and P6 (floating-IP) to v0.15. This delivers the linux topology (the most common) + the podman migration + nft completeness, and avoids the two most complex phases (LXC+podman-in-LXC). If G-003 (real mTLS) holds, add the mTLS phase to v0.14a. Proxmox ingress becomes v0.14b/v0.15.
|
||||
|
||||
**What must be true for v0.14 to succeed in 90 days**:
|
||||
1. The 3-way TLS contradiction is resolved in the plan (not just in this grill). Today: unresolved.
|
||||
2. The migration number is 0009. Today: plan says 0007 (wrong).
|
||||
3. The upgrade path from v0.13 is specified and tested. Today: not specified.
|
||||
4. The `certpaths.CAPath()` phantom is fixed. Today: not fixed.
|
||||
5. The offline-first tension (podman pull) is acknowledged. Today: not acknowledged.
|
||||
6. If mTLS is in-scope, the new phase is written. Today: no such phase.
|
||||
|
||||
**Confidence**: 0.45 that the plan as-written can ship v0.14 without rework. The research is strong; the plan diverges from it and from the codebase in load-bearing ways. The mTLS scope expansion (G-003) makes it worse unless the plan is updated.
|
||||
|
||||
---
|
||||
|
||||
## Verdict
|
||||
|
||||
**RETHINK** (confidence 0.45).
|
||||
|
||||
The plan must be revised to:
|
||||
1. Fix the migration number → 0009 (C-59, G-001).
|
||||
2. Fix `certpaths.CAPath()` → `certpaths.CACertPath()` (C-60).
|
||||
3. Add the v0.13→v0.14 upgrade path to P2 T6 (C-57, G-004).
|
||||
4. Resolve the TLS model: either add a new mTLS phase (G-003) or revert to `tls: {}` and defer mTLS to v0.15. The plan cannot ship `certResolver: orca` (broken) and cannot ship `tls: {}` if G-003 holds.
|
||||
5. Amend REQ-172: `--restart=unless-stopped` (not `always`), omit `:Z` (C-61, G-002).
|
||||
6. Address the offline-first tension (C-54) or document the exception.
|
||||
7. Address the REQ-100 regression (C-58): bake-vs-mount the static config.
|
||||
8. Add `DNATTarget` validation (C-51).
|
||||
9. Add podman-install step for linux topology (C-50).
|
||||
10. If mTLS is in-scope, write the new phase (C-62, E-001).
|
||||
|
||||
Once these are addressed, the plan is feasible. The research foundation is solid; the plan just needs to actually follow it.
|
||||
|
||||
---
|
||||
|
||||
*This grill is recorded in `.ciagent/GRILL_v0.14.md`. Escalations are visible via `ciagent audit`. Binding decisions (G-001..G-004) should be promoted to PROJECT.md via a follow-up clarify or explicitly by the operator. The grill surfaces; it does not rewrite.*
|
||||
@@ -1,644 +0,0 @@
|
||||
# Grill Report: Orca v0.3 — scheduling-streaming
|
||||
|
||||
**Date:** 2026-08-01
|
||||
**Reviewer:** ci-griller (red-team, adversarial)
|
||||
**Plan under review:** `.ciagent/PLAN_v0.3.md` (commit 89fa172)
|
||||
**Branch:** `phase/00-pre-execution`
|
||||
**Mode:** Full autonomy
|
||||
|
||||
---
|
||||
|
||||
## Methodology
|
||||
|
||||
Every claim in `PLAN_v0.3.md` and `RESEARCH_v0.3.md` was cross-checked against
|
||||
the actual codebase (the 7 source files listed in the task, plus `migrate.go`,
|
||||
`store.go`, `doctor_test.go`, `security/integration_test.go`, `security/ca.go`,
|
||||
`security/csr.go`, `model/node.go`, `model/job.go`, and `cli/doctor.go`).
|
||||
Findings are scored on 9 axes. Binding verdicts are ACCEPT (plan must change),
|
||||
REJECT (concern noted, plan stands), or DEFER (address during execution).
|
||||
|
||||
---
|
||||
|
||||
## Summary Verdict
|
||||
|
||||
| Severity | Count |
|
||||
|----------|-------|
|
||||
| CRITICAL | 2 |
|
||||
| HIGH | 2 |
|
||||
| MEDIUM | 5 |
|
||||
| LOW | 3 |
|
||||
| **Total** | **12** |
|
||||
|
||||
**Overall verdict: PROCEED WITH CHANGES**
|
||||
|
||||
The plan is fundamentally sound — the scope is right-sized, the requirements
|
||||
coverage is complete, the persona territories are respected, and the
|
||||
no-new-dependencies promise holds. However, two CRITICAL findings require plan
|
||||
changes before execution begins. Neither is a scope expansion; both are
|
||||
correctness fixes to the design as written. With the 2 ACCEPT changes applied,
|
||||
this plan is ready to execute.
|
||||
|
||||
---
|
||||
|
||||
## Per-Axis Findings
|
||||
|
||||
### Axis 1 — Feasibility (can each task actually be implemented?)
|
||||
|
||||
#### F-01 [CRITICAL] — Watch yields per-row but CLI table mode requires full-snapshot-per-tick
|
||||
|
||||
**Severity:** CRITICAL
|
||||
**Axis:** Feasibility / Vertical slice integrity
|
||||
**Binding verdict:** ACCEPT (plan must change)
|
||||
|
||||
**Finding:**
|
||||
The plan is internally contradictory about what `Watch` yields.
|
||||
|
||||
- D-028 (RESEARCH:88) says Watch "yields the **full current snapshot** (one
|
||||
element per row)."
|
||||
- Task 01-01-01 (PLAN:30) says Watch "yields one `*model.Job` per row via
|
||||
`scanJob`" — i.e., `iter.Seq[*model.Job]`, one element per row per tick.
|
||||
- Task 01-02-02 (PLAN:42) says the CLI table render "collect the full snapshot
|
||||
from `seq` into a `[]*model.Job`" then compares against the previous
|
||||
snapshot's rendered table.
|
||||
|
||||
These are incompatible. `iter.Seq[*model.Job]` yields individual jobs with **no
|
||||
tick-boundary signal**. The CLI ranging `for job := range seq` receives a flat
|
||||
stream of jobs and cannot know when a tick's snapshot is complete. It cannot
|
||||
collect "the full snapshot" because it cannot detect the end of a tick.
|
||||
|
||||
The JSON mode (01-02-03) can work without tick boundaries (per-element dedup
|
||||
via `map[string][]byte`), but the **table mode cannot**. Table mode needs the
|
||||
complete snapshot to render the table, clear the screen, and compare against the
|
||||
previous frame.
|
||||
|
||||
**Evidence:**
|
||||
- `RESEARCH_v0.3.md:106-142` — implementation yields `yield(j)` per row inside
|
||||
`for rows.Next()`, not `yield(allJobs)` per tick.
|
||||
- `PLAN_v0.3.md:30` — "yields one `*model.Job` per row"
|
||||
- `PLAN_v0.3.md:42` — "collect the full snapshot from `seq` into a `[]*model.Job`"
|
||||
- `PLAN_v0.3.md:44` (01-02-04) — nodeListCmd watch bypasses registry, same
|
||||
per-row yield.
|
||||
- D-028 says "full current snapshot" but the code yields per-row.
|
||||
|
||||
**Required change:**
|
||||
Change the `Watch` element type from `iter.Seq[*model.Job]` to
|
||||
`iter.Seq[[]*model.Job]` (and `iter.Seq[[]*model.Node]` analogously). Each tick
|
||||
yields the **full snapshot as a single slice**. This:
|
||||
|
||||
1. Makes D-028 ("yields the full current snapshot") literally true.
|
||||
2. Makes table mode trivial: `for snapshot := range seq { render(snapshot) }`.
|
||||
3. Makes JSON mode cleaner: per-tick, diff the snapshot against the previous
|
||||
one, emit one JSON line per changed element. This also enables a natural
|
||||
`"delete"` event for elements that disappeared (not possible with per-row
|
||||
yield).
|
||||
4. Simplifies the test contract: `TestWatch_YieldsSnapshots` ranges over
|
||||
`iter.Seq[[]*model.Job]` and each yield is a complete tick — no timing
|
||||
ambiguity about "did I get all rows for this tick?"
|
||||
|
||||
**Impact on plan:**
|
||||
- Tasks 01-01-01, 01-01-02: signature changes to
|
||||
`iter.Seq[[]*model.Job]` / `iter.Seq[[]*model.Node]`. Implementation
|
||||
collects all rows into a slice per tick, then `yield(slice)`.
|
||||
- Task 01-02-02 (table): `for snapshot := range seq { ... }` — direct, no
|
||||
collection needed.
|
||||
- Task 01-02-03 (JSON): per-tick diff against previous snapshot's
|
||||
`map[string][]byte`. Emit `"init"`/`"update"`/`"delete"` events.
|
||||
- Task 01-01-04 (tests): assert each yield is a complete snapshot slice.
|
||||
- D-026, D-028, D-046: update to reflect slice-per-tick semantics.
|
||||
- Must-have criteria for 01-01-01/01-01-02: update signature assertions.
|
||||
|
||||
This is a mechanical change to the plan, not a scope change. The implementation
|
||||
is simpler (no tick-boundary detection needed).
|
||||
|
||||
**Confidence:** 0.92
|
||||
|
||||
---
|
||||
|
||||
#### F-02 [CRITICAL] — First-tick delay: Watch waits a full interval before first yield
|
||||
|
||||
**Severity:** CRITICAL
|
||||
**Axis:** Feasibility / UX correctness
|
||||
**Binding verdict:** ACCEPT (plan must change)
|
||||
|
||||
**Finding:**
|
||||
The Watch implementation (RESEARCH:110-116) has this structure:
|
||||
|
||||
```go
|
||||
ticker := time.NewTicker(1 * time.Second)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done(): return
|
||||
case <-ticker.C: // <-- waits 1s BEFORE first query
|
||||
}
|
||||
// query + yield
|
||||
}
|
||||
```
|
||||
|
||||
The `select` waits for the first ticker pulse **before** running the first
|
||||
query. With a 1s default interval, `orca job list --watch` shows **nothing for
|
||||
1 full second**, then the first snapshot appears. For a CLI tool, a 1s blank
|
||||
screen is a poor UX and looks broken. The user expects immediate output, then
|
||||
refreshes every 1s.
|
||||
|
||||
The tests (01-01-04) use `watchInterval=10ms`, so the delay is only 10ms and
|
||||
the test passes — but the test does NOT catch this UX bug because the interval
|
||||
is tiny. In production (1s), the bug is visible.
|
||||
|
||||
**Evidence:**
|
||||
- `RESEARCH_v0.3.md:110-116` — `select` before first query.
|
||||
- `PLAN_v0.3.md:30` — "pull-based inline polling loop on a 1s ticker" — no
|
||||
mention of immediate first yield.
|
||||
- Standard `top`-like tools yield immediately, then tick.
|
||||
|
||||
**Required change:**
|
||||
Add to tasks 01-01-01 and 01-01-02: the polling loop must **query and yield
|
||||
immediately on the first iteration**, then `select` on the ticker for
|
||||
subsequent ticks. Implementation shape:
|
||||
|
||||
```go
|
||||
for {
|
||||
// query + yield (runs immediately on first iteration)
|
||||
rows, err := r.db.QueryContext(ctx, ...)
|
||||
// ... yield snapshot ...
|
||||
select {
|
||||
case <-ctx.Done(): return
|
||||
case <-ticker.C:
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Or equivalently, query once before the loop, then loop with select-first. The
|
||||
must-have criteria should add: "first yield occurs immediately (no
|
||||
`watchInterval` delay before first snapshot)."
|
||||
|
||||
**Impact on plan:**
|
||||
- Tasks 01-01-01, 01-01-02: add "immediate first yield" to description +
|
||||
must-have.
|
||||
- Task 01-01-04 (tests): add assertion that the first snapshot appears within
|
||||
a short deadline (e.g., <50ms) even with `watchInterval=10ms` — proving the
|
||||
first yield is not tick-gated.
|
||||
|
||||
**Confidence:** 0.95
|
||||
|
||||
---
|
||||
|
||||
#### F-03 [HIGH] — P01 and P02 both modify `internal/cli/node.go` (file-disjoint claim is false)
|
||||
|
||||
**Severity:** HIGH
|
||||
**Axis:** Feasibility / Timeline (parallelism)
|
||||
**Binding verdict:** ACCEPT (plan must change)
|
||||
|
||||
**Finding:**
|
||||
D-042 (PLAN:133, RESEARCH:489) claims "P01 and P02 are file-disjoint — no file
|
||||
is modified by both." This is **false**.
|
||||
|
||||
- P01 task 01-02-04 (PLAN:44) modifies `internal/cli/node.go` — adds `--watch`
|
||||
flag + render modes to `nodeListCmd`.
|
||||
- P02 task 02-01-01 (PLAN:76) modifies `internal/cli/node.go` — removes the
|
||||
`dbPath` function and updates `openDB` to call `certpaths.DBPath()`.
|
||||
|
||||
Both phases touch `internal/cli/node.go`. If developed in parallel (as D-042
|
||||
permits), this causes merge conflicts.
|
||||
|
||||
**Evidence:**
|
||||
- `PLAN_v0.3.md:44` — 01-02-04 files: `internal/cli/node.go`
|
||||
- `PLAN_v0.3.md:76` — 02-01-01 files: `internal/cli/node.go` (remove old
|
||||
`dbPath`)
|
||||
- `PLAN_v0.3.md:133` — "P01 and P02 are file-disjoint"
|
||||
- Actual code: `internal/cli/node.go:22-28` defines `dbPath`; `:30-36`
|
||||
defines `openDB` which calls `dbPath()`. `openDB` is used by 14 call sites
|
||||
across `job.go`, `daemon.go`, `node_capacity.go`, `audit.go`, `node.go`.
|
||||
|
||||
**Required change:**
|
||||
Update D-042 and the cross-phase notes (PLAN:131-133) to acknowledge the
|
||||
overlap. Two options (pick one):
|
||||
|
||||
1. **Serialize:** P02 Wave 1 (02-01-01) runs before P01 Wave 2 (01-02-04).
|
||||
P02 Wave 1 is a prerequisite for P01 Wave 2 on the `node.go` file. P01
|
||||
Wave 1 (store layer) and P02 Wave 1 can still run in parallel.
|
||||
2. **Merge the changes:** task 02-01-01 is folded into P01 Wave 2's
|
||||
`node.go` modification (the cli-engineer updates `openDB` to use
|
||||
`certpaths.DBPath()` while also adding `--watch`).
|
||||
|
||||
Recommended: Option 1 (serialize P02 Wave 1 before P01 Wave 2). It preserves
|
||||
the wave structure and persona assignments. Update the cross-phase note to say:
|
||||
"P02 Wave 1 (02-01-01) must complete before P01 Wave 2 (01-02-04) due to shared
|
||||
`internal/cli/node.go` modification. P01 Wave 1 and P02 Wave 1 may run in
|
||||
parallel."
|
||||
|
||||
**Confidence:** 0.90
|
||||
|
||||
---
|
||||
|
||||
#### F-04 [HIGH] — D-037 ServerName = node.Name assumption is fragile and unverified against real join flow
|
||||
|
||||
**Severity:** HIGH
|
||||
**Axis:** Feasibility / Security
|
||||
**Binding verdict:** DEFER (address in execution, with documentation)
|
||||
|
||||
**Finding:**
|
||||
D-037 (RESEARCH:261, confidence 0.80) assumes `serverName = node.Name` for the
|
||||
mTLS health probe. The TLS client's `ServerName` must match a SAN entry on the
|
||||
peer's server cert. But `GenerateCSR(commonName, sans)` (csr.go:24) takes the
|
||||
commonName and SANs as **separate arguments**. The commonName becomes the cert
|
||||
Subject CN, but `ServerName` in `tls.Config` is matched against **SANs**
|
||||
(DNSNames/IPAddresses), not the CN (per Go's `crypto/tls` behavior since Go
|
||||
1.15).
|
||||
|
||||
If a node joined with `--name node-b` but its cert SAN is `localhost` (or an
|
||||
IP), `serverName = "node-b"` will **fail the TLS handshake** with a
|
||||
"certificate is valid for localhost, not node-b" error — even though the peer
|
||||
is perfectly healthy.
|
||||
|
||||
The research (RESEARCH:261) says "confirmed in `integration_test.go:41`
|
||||
`GenerateCSR("test-server", ...)`" — but that test uses `serverName =
|
||||
"localhost"` (integration_test.go:83), which matches the SAN `localhost`, not
|
||||
the commonName `test-server`. The test proves SAN-matching, not CN-matching.
|
||||
|
||||
**Evidence:**
|
||||
- `internal/security/csr.go:24` — `GenerateCSR(commonName, sans)` — CN and
|
||||
SANs are separate.
|
||||
- `internal/security/integration_test.go:41` — `GenerateCSR("test-server",
|
||||
[]string{"localhost", "127.0.0.1"})` — CN is "test-server", SANs are
|
||||
localhost/127.0.0.1.
|
||||
- `internal/security/integration_test.go:83` — `ClientTLSConfig(...,
|
||||
"localhost", ...)` — serverName = "localhost" (a SAN), NOT "test-server"
|
||||
(the CN).
|
||||
- `internal/transport/mtls.go:49-51` — `serverName` is required and set as
|
||||
`tls.Config.ServerName` (matched against SANs).
|
||||
- `PLAN_v0.3.md:88` — 02-02-03: `serverName = n.Name`.
|
||||
|
||||
**Mitigation (DEFER to execution):**
|
||||
1. Document the assumption in the `Network()` check message: "probing
|
||||
<name> at <addr> (assuming cert SAN = node name)".
|
||||
2. If the handshake fails with a SAN mismatch error, the FAIL message should
|
||||
include the cert's actual SANs (parsed from the error) so the operator can
|
||||
diagnose. This is a refinement, not a plan blocker.
|
||||
3. The test 02-02-05(e) uses `Name = "localhost"` which matches the SAN — so
|
||||
the test passes, but it doesn't prove the general case. Add a test comment
|
||||
noting this assumption.
|
||||
|
||||
**Why DEFER not ACCEPT:** The assumption is documented (D-037, 0.80
|
||||
confidence), the failure mode is graceful (FAIL with handshake error, not a
|
||||
crash), and fixing it properly (storing SANs in the nodes table) is a scope
|
||||
expansion beyond v0.3. The plan should note the limitation; execution should
|
||||
add diagnostic context to the error message.
|
||||
|
||||
**Confidence:** 0.78
|
||||
|
||||
---
|
||||
|
||||
#### F-05 [MEDIUM] — `store.Open` runs migrations before integrity_check can run
|
||||
|
||||
**Severity:** MEDIUM
|
||||
**Axis:** Feasibility / Testing
|
||||
**Binding verdict:** REJECT (concern noted, plan stands)
|
||||
|
||||
**Finding:**
|
||||
The DB check (02-02-01) calls `store.Open(path)` which runs `migrate(db)` (store
|
||||
.go:39) before the integrity_check executes. On a truly corrupt DB, `store.Open`
|
||||
fails at `Ping()` or `migrate()` — the integrity_check never runs. The check
|
||||
returns FAIL with the open/migrate error, which is the correct outcome (a DB
|
||||
that can't be opened is broken), but the message says "open <path>: <error>"
|
||||
not "integrity_check failed."
|
||||
|
||||
The plan's `TestDBCheck_Corrupt` (RESEARCH:469) is explicitly called "brittle"
|
||||
and made optional. The plan accepts that integrity_check is somewhat redundant
|
||||
with `store.Open`'s own validation.
|
||||
|
||||
**Evidence:**
|
||||
- `internal/store/store.go:37-41` — `db.Ping()` then `migrate(db)` inside
|
||||
`Open`.
|
||||
- `PLAN_v0.3.md:86` — 02-02-01: `db, err := store.Open(path)`.
|
||||
- `RESEARCH_v0.3.md:455-456` — pitfall table acknowledges this.
|
||||
|
||||
**Why REJECT:** The failure surfaces correctly (FAIL with error message). The
|
||||
integrity_check adds value for the case where the DB opens but has logical
|
||||
corruption (e.g., foreign key violations, orphaned pages) that Ping/migrate
|
||||
don't catch. The plan's approach is acceptable for v0.3. The optional corrupt
|
||||
test is correctly deferred.
|
||||
|
||||
**Confidence:** 0.85
|
||||
|
||||
---
|
||||
|
||||
### Axis 2 — Scope
|
||||
|
||||
#### F-06 [MEDIUM] — No "delete" event in JSON watch mode (with per-row yield)
|
||||
|
||||
**Severity:** MEDIUM
|
||||
**Axis:** Scope / Completeness
|
||||
**Binding verdict:** DEFER (address in execution)
|
||||
|
||||
**Finding:**
|
||||
With the current per-row `iter.Seq[*model.Job]` design (F-01), the JSON watch
|
||||
mode (01-02-03) emits `"init"` and `"update"` events but has no way to emit
|
||||
`"delete"` events — a job that disappears from the snapshot simply stops being
|
||||
yielded, and the CLI has no tick boundary to detect "this ID was in the
|
||||
previous tick but not this one."
|
||||
|
||||
With the F-01 fix (`iter.Seq[[]*model.Job]`, full snapshot per tick), `"delete"`
|
||||
events become trivially possible: diff the previous snapshot's ID set against
|
||||
the current snapshot's ID set. The plan should add `"delete"` event semantics
|
||||
to D-046.
|
||||
|
||||
**Evidence:**
|
||||
- `PLAN_v0.3.md:43` — 01-02-03: only `"init"` and `"update"` events.
|
||||
- `PLAN_v0.3.md:139` — D-046: only `"init"` and `"update"`.
|
||||
- Neither jobs nor nodes are hard-deleted in the current CLI (`node leave` sets
|
||||
state to `left`, doesn't delete the row), so `"delete"` events are not
|
||||
strictly needed for v0.3. But the `NodeRepo.Delete` method exists and could
|
||||
be used by future code.
|
||||
|
||||
**Mitigation (DEFER):** If F-01 is accepted (slice-per-tick), add `"delete"`
|
||||
event to D-046 as a natural extension. If F-01 is not accepted, document the
|
||||
no-delete-event limitation explicitly.
|
||||
|
||||
**Confidence:** 0.70
|
||||
|
||||
---
|
||||
|
||||
### Axis 3 — Testing
|
||||
|
||||
#### F-07 [MEDIUM] — Test timing fragility: 10ms tick + 30ms insert + 80ms cancel
|
||||
|
||||
**Severity:** MEDIUM
|
||||
**Axis:** Testing
|
||||
**Binding verdict:** DEFER (address in execution)
|
||||
|
||||
**Finding:**
|
||||
The store-layer tests (01-01-04) use `watchInterval=10ms` with timing-based
|
||||
assertions: "insert a 2nd job from a goroutine after ~30ms, cancel ctx after
|
||||
~80ms." Under CI load (especially with `-race` overhead), 10ms ticks can be
|
||||
missed or delayed. A 10ms ticker pulse is not guaranteed to fire within 10ms
|
||||
under load — the Go runtime scheduler may delay it. If the 2nd job is inserted
|
||||
at 30ms but the 2nd tick fires at 45ms, the test might see the 2nd job in the
|
||||
3rd tick (at ~55ms) which is still before the 80ms cancel — so it likely
|
||||
passes, but it's fragile.
|
||||
|
||||
**Evidence:**
|
||||
- `PLAN_v0.3.md:33` — 01-01-04: "after ~30ms", "after ~80ms".
|
||||
- `time.NewTicker` does not guarantee exact timing under load.
|
||||
|
||||
**Mitigation (DEFER):** Use more generous margins (e.g., 50ms insert, 200ms
|
||||
cancel) or a synchronization mechanism (e.g., insert the 2nd job, then poll
|
||||
the collected slice with a 500ms timeout). The test hook (`watchInterval`)
|
||||
already enables fast tests; the margins just need to be wider. Execution
|
||||
should validate the tests pass reliably under `-race` in CI before marking
|
||||
Wave 1 complete.
|
||||
|
||||
**Confidence:** 0.75
|
||||
|
||||
---
|
||||
|
||||
#### F-08 [LOW] — `-race` does not detect goroutine leaks; the plan claims it does
|
||||
|
||||
**Severity:** LOW
|
||||
**Axis:** Testing
|
||||
**Binding verdict:** REJECT (concern noted, plan stands)
|
||||
|
||||
**Finding:**
|
||||
The plan (01-01-04 must-have, PLAN:33) says "`-race` reports no leaks/data
|
||||
races." `go test -race` detects **data races**, not **goroutine leaks**.
|
||||
Goroutine leak detection requires `goleak` or explicit goroutine-count
|
||||
assertions. The claim is technically incorrect.
|
||||
|
||||
However, the actual risk is negligible: Watch does not spawn a goroutine
|
||||
(D-032, inline pull loop). `time.NewTicker` spawns an internal goroutine, but
|
||||
`defer ticker.Stop()` terminates it. There is nothing to leak. The
|
||||
`no-goroutine-leak` constraint (data-engineer persona) is satisfied by
|
||||
design, not by testing.
|
||||
|
||||
**Evidence:**
|
||||
- `PLAN_v0.3.md:33` — "`-race` reports no leaks/data races"
|
||||
- `RESEARCH_v0.3.md:92` — D-032: "No goroutine is spawned by Watch."
|
||||
- Go `-race` detector documentation: detects concurrent access, not leaks.
|
||||
|
||||
**Why REJECT:** The claim is imprecise but the risk is zero by design.
|
||||
Execution may optionally add `runtime.NumGoroutine()` before/after assertions
|
||||
for belt-and-suspenders, but it's not required.
|
||||
|
||||
**Confidence:** 0.90
|
||||
|
||||
---
|
||||
|
||||
### Axis 4 — Security
|
||||
|
||||
#### F-09 [MEDIUM] — Doctor network check probes peers using the local server cert as client cert (confirmed valid, but undocumented)
|
||||
|
||||
**Severity:** MEDIUM
|
||||
**Axis:** Security
|
||||
**Binding verdict:** DEFER (document in execution)
|
||||
|
||||
**Finding:**
|
||||
The plan (02-02-03, PLAN:88) uses `certpaths.ServerCertPath()`/`ServerKeyPath()`
|
||||
as the client cert for the mTLS health probe. I verified this is **valid**:
|
||||
`security/ca.go:255` signs server certs with
|
||||
`ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth, x509.ExtKeyUsageClientAuth}`
|
||||
— the server cert has both ServerAuth and ClientAuth EKUs, so it can be
|
||||
presented as a client cert. The daemon's `RequireAndVerifyClientCert`
|
||||
(security/tls_config.go:92) will accept it.
|
||||
|
||||
This is correct and feasible. The finding is that this cross-use (server cert
|
||||
as client cert) is not documented in the plan or the security architecture. A
|
||||
security auditor might flag it as "server cert used for client auth — is this
|
||||
intended?"
|
||||
|
||||
**Evidence:**
|
||||
- `internal/security/ca.go:255` — `ExtKeyUsage: ServerAuth, ClientAuth`.
|
||||
- `internal/security/tls_config.go:92` — `ClientAuth: RequireAndVerifyClientCert`.
|
||||
- `PLAN_v0.3.md:88` — 02-02-03 uses `ServerCertPath()`/`ServerKeyPath()`.
|
||||
- `RESEARCH_v0.3.md:261` — D-037: "presents the local node's client cert."
|
||||
|
||||
**Mitigation (DEFER):** Add a code comment in `probeHealthz` and a note in
|
||||
ARCHITECTURE.md §5 explaining that the local server cert doubles as the client
|
||||
cert for doctor probes (justified by the dual EKU). This is documentation, not
|
||||
a code change.
|
||||
|
||||
**Confidence:** 0.88
|
||||
|
||||
---
|
||||
|
||||
### Axis 5 — Performance
|
||||
|
||||
#### F-10 [LOW] — 1s poll ticker re-runs full List query every second; no concern but worth noting
|
||||
|
||||
**Severity:** LOW
|
||||
**Axis:** Performance
|
||||
**Binding verdict:** REJECT (concern noted, plan stands)
|
||||
|
||||
**Finding:**
|
||||
The 1s ticker (D-019) re-runs `SELECT ... FROM jobs ORDER BY created_at DESC`
|
||||
every second. For a CLI tool run by a human watching a terminal, this is
|
||||
fine — the query is cheap (single table, no joins, indexed by `created_at` if
|
||||
an index exists). For an AI agent tailing `--watch --json` for hours, this is
|
||||
1 query/second × 3600 = 3600 queries/hour. SQLite handles this trivially in
|
||||
WAL mode (store.go:36).
|
||||
|
||||
The cadence is correct for a "top-like" refresh. Faster (e.g., 100ms) would
|
||||
waste CPU; slower (e.g., 5s) would feel sluggish. 1s is the right default.
|
||||
|
||||
**Evidence:**
|
||||
- `PROJECT.md:116` — D-019: "Poll-based, 1s ticker" (confidence 0.90).
|
||||
- `internal/store/store.go:36` — WAL mode enabled.
|
||||
|
||||
**Why REJECT:** The cadence is justified. No change needed.
|
||||
|
||||
**Confidence:** 0.92
|
||||
|
||||
---
|
||||
|
||||
### Axis 6 — Maintainability
|
||||
|
||||
#### F-11 [LOW] — `watchInterval` package var is mutable global state (test hook)
|
||||
|
||||
**Severity:** LOW
|
||||
**Axis:** Maintainability
|
||||
**Binding verdict:** REJECT (concern noted, plan stands)
|
||||
|
||||
**Finding:**
|
||||
D-043 (PLAN:136) uses an unexported package var `watchInterval = 1 * time.Second`
|
||||
in `internal/store`, overridable from `_test.go`. This is mutable global state —
|
||||
if tests run in parallel within the `internal/store` package and one test sets
|
||||
`watchInterval=10ms` while another expects `1s`, they interfere.
|
||||
|
||||
However, Go tests within a single package run **sequentially** by default
|
||||
unless `t.Parallel()` is called. I verified no test in `internal/store` calls
|
||||
`t.Parallel()` (grep found 0 matches). So the global var is safe as long as
|
||||
no Watch test calls `t.Parallel()`. The plan should note this constraint.
|
||||
|
||||
**Evidence:**
|
||||
- `PLAN_v0.3.md:32` — 01-01-03: "unexported package var `watchInterval`"
|
||||
- `PLAN_v0.3.md:136` — D-043.
|
||||
- grep for `t.Parallel()` in `internal/`: 0 matches.
|
||||
|
||||
**Why REJECT:** The approach is pragmatic and safe given sequential test
|
||||
execution. The alternative (a `WatchWithInterval` constructor or an option
|
||||
pattern) would leak test-only API into production, which D-043 explicitly
|
||||
avoids. Execution should add a comment: "do not call t.Parallel() in Watch
|
||||
tests — they share the watchInterval package var."
|
||||
|
||||
**Confidence:** 0.85
|
||||
|
||||
---
|
||||
|
||||
### Axis 7 — Completeness
|
||||
|
||||
#### F-12 [MEDIUM] — Plan does not address `openDB()` being the single chokepoint for dbPath relocation
|
||||
|
||||
**Severity:** MEDIUM
|
||||
**Axis:** Completeness / Feasibility
|
||||
**Binding verdict:** DEFER (clarify in execution)
|
||||
|
||||
**Finding:**
|
||||
Task 02-01-01 (PLAN:76) says "Update `internal/cli/node.go` (and any other
|
||||
`internal/cli` caller of the old unexported `dbPath`) to call
|
||||
`certpaths.DBPath()`." This is imprecise. `dbPath()` is defined in
|
||||
`node.go:22` and called only by `openDB()` in `node.go:31`. `openDB()` is
|
||||
then called by 14 sites across `job.go`, `daemon.go`, `node_capacity.go`,
|
||||
`audit.go`, `node.go`. The correct change is:
|
||||
|
||||
1. Add `certpaths.DBPath()`.
|
||||
2. Change `openDB()` body from `store.Open(dbPath())` to
|
||||
`store.Open(certpaths.DBPath())`.
|
||||
3. Delete the `dbPath()` function from `node.go`.
|
||||
|
||||
No other caller needs changing — they all go through `openDB()`. The plan's
|
||||
"any other `internal/cli` caller" language suggests a broader scan that isn't
|
||||
needed. This is a clarity issue, not a correctness issue.
|
||||
|
||||
**Evidence:**
|
||||
- `internal/cli/node.go:22-28` — `dbPath()` definition.
|
||||
- `internal/cli/node.go:30-36` — `openDB()` calls `dbPath()`.
|
||||
- grep `openDB()`: 14 call sites, all in `internal/cli/`.
|
||||
- grep `dbPath()`: only in `node.go:31` (inside `openDB`).
|
||||
|
||||
**Mitigation (DEFER):** Execution should note that `openDB()` is the single
|
||||
chokepoint — update its body and delete `dbPath()`. No other file needs
|
||||
changes. The plan's must-have ("`internal/cli` no longer defines `dbPath`")
|
||||
is correct.
|
||||
|
||||
**Confidence:** 0.88
|
||||
|
||||
---
|
||||
|
||||
### Axis 8 — Vertical Slice Integrity
|
||||
|
||||
Covered by F-01 (the tick-boundary problem breaks the Wave 1 → Wave 2
|
||||
vertical slice: Wave 1 produces `iter.Seq[*model.Job]` which Wave 2's table
|
||||
mode cannot consume correctly). With F-01's fix (`iter.Seq[[]*model.Job]`),
|
||||
the vertical slice is clean: Wave 1 yields full snapshots, Wave 2 renders
|
||||
them.
|
||||
|
||||
### Axis 9 — Risk
|
||||
|
||||
**Highest-risk task:** 02-02-05(e) `TestNetworkCheck_PeerReachable` —
|
||||
integration test requiring CA bootstrap, server cert signing with correct
|
||||
SAN, httptest TLS server with `RequireAndVerifyClientCert`, node row insert,
|
||||
and mTLS probe. Has the most moving parts and the most assumptions (D-037
|
||||
ServerName, dual-EKU client cert, httptest HTTP/1.1 vs h2c quirks per
|
||||
integration_test.go:100-126). If D-037 is wrong in production (not in test,
|
||||
since the test uses `Name = "localhost"` matching the SAN), the network check
|
||||
fails for real deployments but the test passes — a false-positive.
|
||||
|
||||
**What could go catastrophically wrong:** The F-01 tick-boundary issue, if
|
||||
not caught, would cause `orca job list --watch` (table mode) to either hang
|
||||
(trying to collect a "full snapshot" that never completes) or render
|
||||
incomplete tables (rendering after each row instead of after a full tick).
|
||||
This is a user-visible broken feature shipped as "complete."
|
||||
|
||||
---
|
||||
|
||||
## Binding Decisions (G-series)
|
||||
|
||||
| ID | Decision | Rationale | Confidence | Verdict |
|
||||
|----|----------|-----------|------------|---------|
|
||||
| G-001 | Change `Watch` to `iter.Seq[[]*model.Job]` / `iter.Seq[[]*model.Node]` (full snapshot per tick) | F-01: per-row yield has no tick boundary; table mode needs full snapshot. Slice-per-tick makes D-028 literally true and simplifies both render modes. | 0.92 | ACCEPT |
|
||||
| G-002 | Watch must yield immediately on first iteration, then tick | F-02: current design waits 1s before first output. Unacceptable UX. | 0.95 | ACCEPT |
|
||||
| G-003 | P02 Wave 1 (02-01-01) must complete before P01 Wave 2 (01-02-04) — shared `internal/cli/node.go` | F-03: D-042 file-disjoint claim is false for `node.go`. | 0.90 | ACCEPT |
|
||||
| G-004 | D-037 ServerName = node.Name assumption is deferred; execution must add diagnostic context to handshake-fail errors | F-04: assumption is documented (0.80), failure is graceful, proper fix is out of v0.3 scope. | 0.78 | DEFER |
|
||||
| G-005 | `store.Open` runs migrations before integrity_check — acceptable | F-05: failure surfaces correctly as FAIL. | 0.85 | REJECT |
|
||||
| G-006 | Add `"delete"` event to JSON watch mode if G-001 is accepted | F-06: slice-per-tick makes delete events trivial. | 0.70 | DEFER |
|
||||
| G-007 | Widen test timing margins (10ms tick → generous insert/cancel margins) | F-07: 10ms ticker under CI load is fragile. | 0.75 | DEFER |
|
||||
| G-008 | `-race` does not detect goroutine leaks — claim is imprecise but risk is zero by design | F-08: no goroutine spawned. | 0.90 | REJECT |
|
||||
| G-009 | Document dual-EKU (server cert as client cert) in `probeHealthz` + ARCHITECTURE.md | F-09: valid but undocumented. | 0.88 | DEFER |
|
||||
| G-010 | 1s poll ticker cadence is correct | F-10: justified by D-019. | 0.92 | REJECT |
|
||||
| G-011 | `watchInterval` package var is safe (no `t.Parallel` in store tests) | F-11: pragmatic, avoids leaking test API. | 0.85 | REJECT |
|
||||
| G-012 | `openDB()` is the single chokepoint for dbPath relocation — clarify in execution | F-12: plan is imprecise but correct. | 0.88 | DEFER |
|
||||
|
||||
---
|
||||
|
||||
## Escalations
|
||||
|
||||
None. All 12 findings are resolved with confidence ≥ 0.60 (either ACCEPT,
|
||||
REJECT, or DEFER). No axis requires human escalation.
|
||||
|
||||
---
|
||||
|
||||
## Overall Verdict
|
||||
|
||||
**PROCEED WITH CHANGES**
|
||||
|
||||
The plan is approved for execution **after** the 3 ACCEPT binding verdicts
|
||||
(G-001, G-002, G-003) are applied to `PLAN_v0.3.md`:
|
||||
|
||||
1. **G-001:** Change `Watch` element type to `iter.Seq[[]*model.Job]` /
|
||||
`iter.Seq[[]*model.Node]` (full snapshot per tick). Update tasks
|
||||
01-01-01, 01-01-02, 01-02-02, 01-02-03, 01-02-04, 01-01-04, and decisions
|
||||
D-026, D-028, D-046.
|
||||
2. **G-002:** Add "immediate first yield" to tasks 01-01-01, 01-01-02 and
|
||||
must-have criteria + test assertion in 01-01-04.
|
||||
3. **G-003:** Update D-042 and cross-phase notes: P02 Wave 1 (02-01-01)
|
||||
precedes P01 Wave 2 (01-02-04) due to shared `internal/cli/node.go`.
|
||||
|
||||
The 5 DEFER items (G-004, G-006, G-007, G-009, G-012) are execution-time
|
||||
refinements that do not block the plan.
|
||||
|
||||
The scope is right-sized (21 tasks across 5 waves, 2 execution phases + 1
|
||||
review phase). No requirements gaps exist between REQ-022/030/032 and the plan
|
||||
tasks. The no-new-dependencies promise holds. The persona territories are
|
||||
respected. The test strategy is adequate (with the timing-margin note in
|
||||
G-007). The plan does not violate the minimalist pillar.
|
||||
|
||||
**Confidence in verdict:** 0.88
|
||||
@@ -1,592 +0,0 @@
|
||||
# Grill Report: Orca v0.8 — Coverage & Trust Hardening
|
||||
|
||||
**Date:** 2026-08-04
|
||||
**Reviewer:** ci-griller (red-team, adversarial)
|
||||
**Plan under review:** `.ciagent/PLAN_v0.8.md` (commit 4780e4d)
|
||||
**Branch:** `phase/00-specify` (milestone `milestone/v0.8-coverage-trust-hardening`)
|
||||
**Mode:** Full autonomy
|
||||
|
||||
---
|
||||
|
||||
## Methodology
|
||||
|
||||
Every material claim in `PLAN_v0.8.md` and `RESEARCH_v0.8.md` was cross-checked
|
||||
against the actual codebase (verified coverage baselines via `go test -cover`,
|
||||
read `internal/proxmox/bootstrap.go:75-234`, `internal/security/ca.go`,
|
||||
`internal/doctor/doctor.go`, `.ciagent/ROADMAP.md`, `.ciagent/REQUIREMENTS.md`,
|
||||
PERSONAS, ARCHITECTURE) AND the `golang.org/x/crypto` v0.54.0 source for
|
||||
`knownhosts.New` / `checkAddr` behavior. The TOFU-capture claim was not taken
|
||||
on faith — the upstream `checkAddr` (knownhosts.go:370-385) was read directly.
|
||||
|
||||
Findings are scored on the 9 axes. Binding verdicts are **PROCEED**,
|
||||
**PROCEED-WITH-CONDITION** (plan proceeds but must incorporate a named change),
|
||||
or **REPLAN** (axis has a fatal flaw; revise before execution).
|
||||
|
||||
---
|
||||
|
||||
## Summary Verdict
|
||||
|
||||
| Verdict | Count |
|
||||
|---------|-------|
|
||||
| PROCEED | 7 |
|
||||
| PROCEED-WITH-CONDITION | 4 |
|
||||
| REPLAN | 0 |
|
||||
|
||||
**Overall verdict: PROCEED-WITH-CONDITION**
|
||||
|
||||
The v0.8 plan is fundamentally sound: scope is right-sized, the no-new-deps
|
||||
promise holds (verified `ssh.FingerprintSHA256` + `knownhosts.Line` are in the
|
||||
existing `golang.org/x/crypto` v0.54.0 dep), the tiered coverage floor (D-047)
|
||||
is realistic per-package with the named seams, and the persona territory
|
||||
collision on `internal/cli/node.go` is explicitly adjudicated in PERSONAS.md
|
||||
(backend owns implementation, lead owns `_test.go`). The 4 conditions below are
|
||||
**targeted correctness fixes**, not scope expansions:
|
||||
|
||||
1. **P02 must add a regression test asserting first-connect Proxmox join
|
||||
succeeds end-to-end** (the latent TOFU bug means v0.6's first-connect has
|
||||
been broken since ship; the fix in T02.6 is correct but must be proven by a
|
||||
test that would have failed pre-fix).
|
||||
2. **P03's verify-reqs regex must match `**COMPLETE**` as a *substring* within
|
||||
the bold span** (v0.2's header `**COMPLETE (merged to main via v0.3)**` is
|
||||
not matched by the current `\*\*COMPLETE\*\*` literal — a silent blind spot).
|
||||
3. **P03 must add a second assertion: every REQUIREMENTS row marked `Complete`
|
||||
must reference a milestone ROADMAP marks COMPLETE** (the reverse direction).
|
||||
The v0.7 `cert_repo_test.go` omission (REQ-053 marked Complete but the test
|
||||
file does not exist) proves forward-direction-only checks miss the most
|
||||
dangerous drift class: *claimed-Complete-but-actually-incomplete*.
|
||||
4. **P02 T02.6's TOFU fix must be reviewed against `doctor proxmox`'s callback
|
||||
(T02.9) as a paired change, not a follow-on** — they share the exact
|
||||
`knownhosts.New` defect; fixing one and not the other in the same phase
|
||||
creates an inconsistent trust surface.
|
||||
|
||||
With these 4 conditions applied, this plan is ready to execute. No REPLAN.
|
||||
|
||||
---
|
||||
|
||||
## Per-Axis Findings
|
||||
|
||||
### Axis 1 — Business Case
|
||||
|
||||
#### A1-F1 — Is v0.8 the right next milestone, or polish-for-polish's-sake?
|
||||
|
||||
**Evidence:**
|
||||
- v0.7 P03 (REQ-055) shipped a ≥50% coverage floor; v0.8 re-baselines six
|
||||
packages still under 50% (engine 8.3%, proxmox 5.1%, cli 27.6%, transport
|
||||
26.3%, store 47.2%, jobspec 47.6%) — **verified identical via `go test
|
||||
-cover`**.
|
||||
- RESEARCH §2.1 surfaces a **latent v0.6 defect**: `knownhosts.New` returns
|
||||
`KeyError{Want:[]}` on first connect and does NOT auto-write. Verified
|
||||
directly in `golang.org/x/crypto@v0.54.0/ssh/knownhosts/knownhosts.go:370-385`
|
||||
(`checkAddr` returns `&KeyError{}` with empty `Want` when no line matches).
|
||||
`bootstrap.go:140-142` treats this as a dial failure. **This means
|
||||
first-connect `orca node join --type proxmox` has been broken since v0.6
|
||||
shipped** (the v0.6 RESEARCH §A.5 claim that `knownhosts.New` "handles both
|
||||
capture and verify" was wrong).
|
||||
- `bootstrap.go:123` comment is literally false: "on first connect it captures
|
||||
the host key" — it does not.
|
||||
|
||||
**Confidence:** 0.90 that v0.8 is the right next milestone.
|
||||
**Verdict:** **PROCEED**. v0.8 is not polish-for-polish: it closes a real
|
||||
security defect (TOFU broken since v0.6), populates a `Result` field that D-045
|
||||
*assumed* was already populated (it isn't — `bootstrap.go:195-198`), and lifts
|
||||
coverage off floors that v0.7 explicitly under-shot. The diminishing-returns
|
||||
risk is real for the 3 zero-test toe-holds (audit/certpaths/cmd-orca), but
|
||||
D-047 tiered them to 50% precisely to avoid the rathole — that call is sound.
|
||||
|
||||
---
|
||||
|
||||
### Axis 2 — Scope and Requirements
|
||||
|
||||
#### A2-F1 — Is the TOFU bugfix correctly scoped into P02, or should it be a hotfix on main?
|
||||
|
||||
**Evidence:**
|
||||
- The TOFU capture bug (RESEARCH §2.1, PLAN T02.6) is a v0.6 latent defect,
|
||||
not a v0.8 feature. First-connect Proxmox join is broken **today on main**.
|
||||
- PLAN bundles the fix into P02 (trust hardening phase) alongside REQ-058
|
||||
(`--host-key-fingerprint`) and REQ-059 (`key-reset`).
|
||||
- ROADMAP tags run on the v0.7.x patch line: `v0.7.0` (P0) … `v0.7.4` (P04).
|
||||
P02 ships as `v0.7.2` — i.e., the fix lands on a milestone branch, not main,
|
||||
and only reaches main at P04 merge (`v0.7.4`).
|
||||
|
||||
**Confidence:** 0.62 that bundling into P02 is the right call (low confidence —
|
||||
this is a judgment call with real downside).
|
||||
**Verdict:** **PROCEED-WITH-CONDITION.** The fix is correctly designed (T02.6's
|
||||
`KeyError{Want:[]}` capture-and-persist is the right shape), but the plan must
|
||||
either (a) document explicitly *why* this isn't hotfixed on main (e.g., "no
|
||||
operator has hit first-connect yet because all deployments pre-populate
|
||||
`known_hosts` manually — confirmed by the v0.6 ship audit"), OR (b) flag the
|
||||
bug in the P04 audit as a v0.6 ship-defect with a post-mortem note. **The plan
|
||||
currently treats T02.6 as a feature task; it is a bugfix for shipped code and
|
||||
must be labeled as such** so the P04 audit can distinguish "new hardening" from
|
||||
"closing a v0.6 gap." Blast radius if T02.6's fix is wrong: every existing
|
||||
Proxmox node's `known_hosts` could be re-pinned on next join — moderate, but
|
||||
mitigated by T02.10 case 3/4/5 integration tests.
|
||||
|
||||
**Condition:** Add a note to T02.6 in PLAN marking it as a **v0.6 ship-defect
|
||||
bugfix** (not a v0.8 feature), and ensure P04 audit (T04.2) records it as such.
|
||||
|
||||
#### A2-F2 — Are the 3 zero-test packages worth a 50% toe-hold, or scope creep?
|
||||
|
||||
**Evidence:**
|
||||
- `cmd/orca` is 15 LOC of glue (`main()` → `cli.Execute()`). 50% coverage = ~7
|
||||
lines. RESEARCH §1.1, §5 pitfall #6 explicitly flags the effort:coverage
|
||||
ratio as poor.
|
||||
- `internal/certpaths` is 64 LOC of pure path-join functions. 50% is trivial.
|
||||
- `internal/audit` is 125 LOC, 4 exported funcs. 50% is trivial.
|
||||
- D-047 explicitly tiered these to 50% to avoid a coverage rathole; v0.9 can
|
||||
raise the floor.
|
||||
|
||||
**Confidence:** 0.85.
|
||||
**Verdict:** **PROCEED.** The tiered floor is the right call. The
|
||||
`cmd/orca` toe-hold is low-value but low-cost (one `run() int` refactor + one
|
||||
smoke test), and dropping it would leave a `covdata` tooling error in CI output
|
||||
that looks like a broken build to a casual reader. Keeping it at 50% is
|
||||
defensible.
|
||||
|
||||
#### A2-F3 — Scope size: 4 REQs, 37 tasks — too lean, too fat, or right?
|
||||
|
||||
**Evidence:**
|
||||
- 37 tasks, 36 must-haves, 4 phases each shipping a patch. Comparable to v0.7
|
||||
(5 phases, similar task density).
|
||||
- P01 is the heaviest (12 tasks, 9 packages) — the risk concentration is here.
|
||||
|
||||
**Confidence:** 0.80.
|
||||
**Verdict:** **PROCEED.** Right-sized for an NFR milestone. P01 density is the
|
||||
watch item (see Axis 5).
|
||||
|
||||
---
|
||||
|
||||
### Axis 3 — Architecture and Technical Feasibility
|
||||
|
||||
#### A3-F1 — Do the proxmox `sessionRunner` and engine `peerDispatcher` seams leak test concerns into production?
|
||||
|
||||
**Evidence:**
|
||||
- T01.1 `sessionRunner` (`internal/proxmox/bootstrap.go`): 1 interface,
|
||||
~10 LOC, `CombinedOutput(cmd) ([]byte, error)`. Default impl wraps
|
||||
`*ssh.Client.NewSession().CombinedOutput(...)`. Backward compatible —
|
||||
existing callers unchanged. This is the **same pattern as the existing
|
||||
`sshDialer` seam** (`bootstrap.go:201-213`), which shipped in v0.6 without
|
||||
concern. The seam is a standard testability extraction, not a test concern
|
||||
leak.
|
||||
- T01.2 `peerDispatcher` (`internal/engine/dispatcher.go`): **conditional** —
|
||||
only added if T01.4 cannot hit 70% via `httptest.NewTLSServer` alone. Plan
|
||||
explicitly prefers `httptest.NewTLSServer` (RESEARCH §1.3 gap #2, §5 pitfall
|
||||
#8). This is the right ordering: try the stdlib test fixture first, add the
|
||||
seam only if needed.
|
||||
|
||||
**Confidence:** 0.88.
|
||||
**Verdict:** **PROCEED.** Both seams are backward-compatible interface
|
||||
extractions matching an existing pattern (`sshDialer`). No test-concern leak.
|
||||
The conditional-gate on T01.2 is correctly conservative.
|
||||
|
||||
#### A3-F2 — Does P02's trust work stay within the existing security boundary?
|
||||
|
||||
**Evidence:**
|
||||
- P02 touches `internal/proxmox/bootstrap.go` (pinned callback, TOFU fix),
|
||||
`internal/cli/node.go` (flag + subcommand), `internal/security/sshkey.go`
|
||||
(fingerprint helper), `internal/doctor/doctor.go` (T02.9 TOFU fix). All
|
||||
within the existing SSH trust surface established in v0.6.
|
||||
- No new crypto, no new CA, no new X.509. `ssh.FingerprintSHA256` is in the
|
||||
existing `golang.org/x/crypto` v0.54.0 dep (verified: not a new direct dep).
|
||||
- PERSONAS correctly keeps `security-engineer` deactivated — the work is SSH
|
||||
dialer + known_hosts file manipulation, not new security architecture.
|
||||
|
||||
**Confidence:** 0.90.
|
||||
**Verdict:** **PROCEED.** Boundary is respected.
|
||||
|
||||
#### A3-F3 — T02.9 (doctor proxmox TOFU fix) is a paired change with T02.6, not a follow-on
|
||||
|
||||
**Evidence:**
|
||||
- `internal/doctor/doctor.go:412` uses the **exact same** `knownhosts.New(...)`
|
||||
callback pattern as `bootstrap.go:125`. Both share the latent defect.
|
||||
- T02.9 is listed as a separate task ("Apply the TOFU capture-fix to `doctor
|
||||
proxmox` probe") but is in the same Wave 2 as T02.6. If T02.6 lands and T02.9
|
||||
doesn't (e.g., a mid-phase blocker), the trust surface is **inconsistent**:
|
||||
join captures, doctor fails.
|
||||
|
||||
**Confidence:** 0.75.
|
||||
**Verdict:** **PROCEED-WITH-CONDITION.** T02.6 and T02.9 must be reviewed as a
|
||||
paired change in P02 verification — the phase is not done until BOTH callbacks
|
||||
use the capture-fix wrapper. Add to P02 Verification: "doctor proxmox
|
||||
first-connect → captures + succeeds (mirrors T02.10 case 3 for bootstrap)."
|
||||
|
||||
**Condition:** Add a P02 verification line asserting doctor proxmox
|
||||
first-connect parity with bootstrap.
|
||||
|
||||
---
|
||||
|
||||
### Axis 4 — People, Skills, and Organization
|
||||
|
||||
#### A4-F1 — Territory collision on `internal/cli/node.go`
|
||||
|
||||
**Evidence:**
|
||||
- PERSONAS.md line 62: lead-developer territory = `internal/cli/**`.
|
||||
- PERSONAS.md line 70: backend-engineer territory = `internal/cli/node.go`.
|
||||
- PERSONAS.md line 107 explicitly adjudicates: "backend owns the command
|
||||
implementation; lead owns the test files (`node_test.go`)."
|
||||
- Territory mode is `warn` (not `block`) — collisions log but don't fail.
|
||||
|
||||
**Confidence:** 0.82.
|
||||
**Verdict:** **PROCEED.** The collision is **explicitly adjudicated** in
|
||||
PERSONAS.md with a clean boundary (impl vs test files). This is the right
|
||||
answer. The `warn` mode means a backend commit touching `node_test.go` (or a
|
||||
lead commit touching `node.go` impl) would log — acceptable for a 3-persona
|
||||
team. No replan.
|
||||
|
||||
#### A4-F2 — Key-person dependency: is the 3-persona roster sufficient?
|
||||
|
||||
**Evidence:**
|
||||
- 3 active personas, all retained from v0.7. No phase-specific personas.
|
||||
- backend-engineer owns 60%+ of P02 (the security-critical phase). If
|
||||
backend-engineer is unavailable, P02 stalls entirely.
|
||||
|
||||
**Confidence:** 0.70.
|
||||
**Verdict:** **PROCEED.** Key-person risk is real but inherent to a 3-persona
|
||||
NFR milestone. The work is not novel (refining existing surface), so the bus
|
||||
factor is acceptable for hardening. Flagged, not blocking.
|
||||
|
||||
---
|
||||
|
||||
### Axis 5 — Timeline and Estimates
|
||||
|
||||
#### A5-F1 — Is the 70% coverage target for 6 packages in one phase (P01) realistic?
|
||||
|
||||
**Evidence:**
|
||||
- RESEARCH §1.1 + §1.4 per-package achievability assessments:
|
||||
- engine → 70% REALISTIC (with LocalExecutor stubs + `openTestDB`).
|
||||
- proxmox → 70% REALISTIC **but requires the `sessionRunner` seam (T01.1)** —
|
||||
without it, only 50-55% (validation paths + sudoersContent asserts, already
|
||||
done).
|
||||
- cli → 70% AMBITIOUS (17 files, ~2000 LOC); RESEARCH says "55-65% is more
|
||||
realistic for one phase" even with `daemon.go` excluded.
|
||||
- transport → 70% REALISTIC (`httptest.NewTLSServer` is standard).
|
||||
- store → 70% REALISTIC (cert_repo_test.go gap is the main lift).
|
||||
- jobspec → 70% REALISTIC (easiest of the six).
|
||||
- **`internal/cli` is the swing package.** RESEARCH explicitly says 55-65% is
|
||||
the realistic single-phase outcome, not 70%. The plan sets the floor at 70%
|
||||
"excluding daemon.go" — but even excluding daemon.go, RESEARCH's own evidence
|
||||
says 70% is a stretch.
|
||||
|
||||
**Confidence:** 0.65 (split: 5 of 6 packages at 0.85, cli at 0.45).
|
||||
**Verdict:** **PROCEED-WITH-CONDITION.** The plan must add an explicit fallback
|
||||
for `internal/cli`: if T01.6 hits ≥65% (excluding daemon.go) but not 70% after
|
||||
a reasonable effort, the phase ships at 65% with a documented note + a v0.9
|
||||
follow-up to lift to 70%. **Hard-requiring 70% on cli risks a coverage rathole
|
||||
that delays the entire milestone** (P02/P03 are gated on P01 ship). The other 5
|
||||
packages at 70% is realistic.
|
||||
|
||||
**Condition:** Add to T01.6 acceptance criterion: "If ≥65% (excluding
|
||||
daemon.go) is achieved but 70% is not after Wave 2 effort, document the gap in
|
||||
the task comment + record a v0.9 follow-up; ship at 65%. Do NOT block P02/P03
|
||||
on the last 5% of cli coverage." (This mirrors RESEARCH §1.4's own flag, which
|
||||
the plan currently does not carry forward as an escape valve.)
|
||||
|
||||
---
|
||||
|
||||
### Axis 6 — Budget and Financial Realism
|
||||
|
||||
#### A6-F1 — Zero new deps: is that realistic given P02's needs?
|
||||
|
||||
**Evidence:**
|
||||
- `ssh.FingerprintSHA256`: verified in `golang.org/x/crypto/ssh` (direct dep
|
||||
since v0.6 D-030).
|
||||
- `knownhosts.Line` / `Normalize` / `KeyError`: same `golang.org/x/crypto`
|
||||
module (already imported in `bootstrap.go:32` and `doctor.go:29`).
|
||||
- `verify-reqs`: stdlib only (`regexp`, `os`, `fmt`).
|
||||
- `go.mod` unchanged by v0.8 (PLAN line 62).
|
||||
|
||||
**Confidence:** 0.95.
|
||||
**Verdict:** **PROCEED.** Zero-new-deps is verified and realistic.
|
||||
|
||||
---
|
||||
|
||||
### Axis 7 — Risks, Assumptions, and Dependencies
|
||||
|
||||
#### A7-F1 — The 10 pitfalls: are mitigations real or hand-waves?
|
||||
|
||||
**Evidence (spot-check of the 4 most material pitfalls):**
|
||||
- **Pitfall #1 (TOFU broken):** Mitigation T02.6 is **concrete and correct** —
|
||||
wrap `knownhosts.New`, capture on `KeyError{Want:[]}` via `knownhosts.Line` +
|
||||
`security.WriteAtomic`, return nil. Verified against x/crypto v0.54.0
|
||||
`checkAddr` semantics. **Real mitigation.**
|
||||
- **Pitfall #2 (Result.HostKeyFingerprint never populated):** T02.7 adds
|
||||
`ssh.FingerprintSHA256(hostKey)`. 1-line once host key is available. **Real.**
|
||||
- **Pitfall #3 (no sessionRunner seam):** T01.1 adds it, ~10 LOC. **Real.**
|
||||
- **Pitfall #10 (writeAtomic unexported):** T02.2 exports it. Verified
|
||||
`ca.go:305` — `func writeAtomic(...)` is indeed unexported. **Real.**
|
||||
|
||||
**Confidence:** 0.88.
|
||||
**Verdict:** **PROCEED.** Mitigations are concrete, not hand-waves.
|
||||
|
||||
#### A7-F2 — TOFI bugfix blast radius if P02's fix is wrong
|
||||
|
||||
**Evidence:**
|
||||
- T02.6 changes the `HostKeyCallback` for every `orca node join --type proxmox`
|
||||
+ every `doctor proxmox` probe. If the capture-and-persist logic is wrong,
|
||||
every existing Proxmox node's `known_hosts` could be corrupted (e.g.,
|
||||
duplicate entries, wrong-format lines, partial writes on crash).
|
||||
- Mitigations: T02.10 integration tests (cases 3/4/5 cover first-connect,
|
||||
second-connect, mismatch); AD-029 atomic rewrite via `security.WriteAtomic`.
|
||||
- **Gap:** no test for "known_hosts already has an entry, join re-connects" —
|
||||
i.e., the idempotent re-run path after the fix. T02.10 case 4 covers
|
||||
second-connect-match, but not "known_hosts was written by the OLD (broken)
|
||||
code path and is now being read by the NEW code path."
|
||||
|
||||
**Confidence:** 0.70.
|
||||
**Verdict:** **PROCEED-WITH-CONDITION.** T02.10 must add a case for
|
||||
"known_hosts pre-populated in the expected format (e.g., from a manual
|
||||
`ssh-keyscan` or a prior v0.6 deployment that somehow succeeded) →
|
||||
second-connect matches + succeeds." This covers the migration path from
|
||||
v0.6's (broken) state to v0.8's fixed state.
|
||||
|
||||
**Condition:** Add T02.10 case 7: "known_hosts pre-populated with a valid
|
||||
OpenSSH line for the host → connect matches + succeeds (covers v0.6→v0.8
|
||||
migration)."
|
||||
|
||||
---
|
||||
|
||||
### Axis 8 — Governance, Decision-Making, and Communication
|
||||
|
||||
#### A8-F1 — Does `make verify-reqs` actually prevent drift, or is it cosmetic?
|
||||
|
||||
**Evidence:**
|
||||
- T03.1 regex (PLAN line 216):
|
||||
- ROADMAP milestone-complete: `^##\s*Milestone\s+v0\.\d+:.*—\s*\*\*COMPLETE\*\*`
|
||||
- REQUIREMENTS row: `^\|\s*(REQ-\d+)\s*\|.*?\|\s*\*\*(Complete|Pending)\*\*\s*\|`
|
||||
- **ROADMAP v0.2 header (line 23):** `## Milestone v0.2: Networking,
|
||||
Observability, Security Hardening — **COMPLETE (merged to main via v0.3)**`
|
||||
- The regex `\*\*COMPLETE\*\*` requires the literal `**COMPLETE**` with closing
|
||||
`**` immediately after `COMPLETE`. v0.2's header has `**COMPLETE (merged to
|
||||
main via v0.3)**` — the `**` closes after the parenthetical, NOT after
|
||||
`COMPLETE`. **The regex does NOT match v0.2 as complete.**
|
||||
- **Consequence:** all v0.2 REQs (REQ-011, 014, 023, 025-040) are **silently
|
||||
exempted** from the check. A stale v0.2 REQ-035 row (marked Pending) would
|
||||
NOT fail the gate.
|
||||
- **ROADMAP v0.6 has TWO headers** (line 92 without COMPLETE, line 94 with) —
|
||||
the regex matches line 94, but the duplicate is a markdown smell that could
|
||||
confuse the milestone→REQ mapping if the parser takes the first match.
|
||||
|
||||
**Confidence:** 0.92 (high — the regex mismatch is verifiable).
|
||||
**Verdict:** **PROCEED-WITH-CONDITION.** The regex must match `**COMPLETE**`
|
||||
as a *substring within the bold span*, not as a literal `**COMPLETE**` token.
|
||||
Change to `—\s*\*\*[^*]*COMPLETE[^*]*\*\*` (matches `**COMPLETE**`,
|
||||
`**COMPLETE (merged to main via v0.3)**`, and any future variant). Add a
|
||||
golden-file test case (T03.2) with the v0.2-style parenthetical header to
|
||||
prevent regression.
|
||||
|
||||
**Condition:** T03.1 regex changed to substring-match COMPLETE within the bold
|
||||
span; T03.2 adds a golden fixture with `**COMPLETE (merged to main via v0.3)**`.
|
||||
|
||||
#### A8-F2 — Is the single-direction check (ROADMAP→REQUIREMENTS) enough?
|
||||
|
||||
**Evidence:**
|
||||
- PLAN line 35-37 explicitly scopes out the reverse direction: "forward
|
||||
direction (ROADMAP-shipped → REQUIREMENTS Complete) is the priority per the
|
||||
v0.7 drift that motivated REQ-060."
|
||||
- **But the v0.7 drift had TWO symptoms:**
|
||||
1. ROADMAP said COMPLETE, REQUIREMENTS said Pending (forward drift — caught
|
||||
by the current check).
|
||||
2. **REQ-053 was marked Complete in REQUIREMENTS, but
|
||||
`internal/store/cert_repo_test.go` was never written** — verified: only
|
||||
`cert_repo.go` exists in `internal/store/`. The "Complete" status was
|
||||
false. **No markdown-based check can catch this** (it's a code-vs-doc
|
||||
drift, not a doc-vs-doc drift).
|
||||
- The reverse-direction check (REQUIREMENTS Complete ↔ ROADMAP COMPLETE) would
|
||||
catch a different class: a REQ marked Complete in REQUIREMENTS for a
|
||||
milestone ROADMAP does NOT mark COMPLETE (e.g., premature marking). This is
|
||||
a cheaper class of drift but still real.
|
||||
|
||||
**Confidence:** 0.78.
|
||||
**Verdict:** **PROCEED-WITH-CONDITION.** Add the reverse-direction assertion
|
||||
to T03.1 (it's ~10 LOC on top of the existing parser — same maps, just diff
|
||||
both ways). Document explicitly that **no markdown check can catch the
|
||||
code-vs-doc drift** (REQ-053 case) — that requires a code-level audit
|
||||
(`ciagent-audit` in P04). The plan should note this as a known limitation of
|
||||
REQ-060, not pretend the gate is complete.
|
||||
|
||||
**Condition:** T03.1 adds reverse-direction assertion; PLAN adds a note that
|
||||
REQ-060 catches doc-vs-doc drift only, not code-vs-doc (the REQ-053
|
||||
cert_repo_test.go case).
|
||||
|
||||
#### A8-F3 — Is there a "stop the project" trigger?
|
||||
|
||||
**Evidence:** P04 (T04.1-T04.9) is the final review + ship. No explicit
|
||||
"stop" trigger if P01 coverage stalls or P02 TOFU fix proves unfixable.
|
||||
|
||||
**Confidence:** 0.60.
|
||||
**Verdict:** **PROCEED.** The 4-phase structure with per-phase tags means a
|
||||
stall is visible (phase tag doesn't ship). Acceptable for an NFR milestone.
|
||||
|
||||
---
|
||||
|
||||
### Axis 9 — Change, Adoption, and Operational Readiness
|
||||
|
||||
#### A9-F1 — Who benefits from v0.8? Is there operator pull for `--host-key-fingerprint`?
|
||||
|
||||
**Evidence:**
|
||||
- `--host-key-fingerprint` (REQ-058) is operator-facing: pre-pinning a
|
||||
Proxmox host's SSH key before first join. This is the standard
|
||||
high-security-deployment pattern (the v0.6 D-035 caveat explicitly promised
|
||||
it as a "future enhancement").
|
||||
- `orca node key-reset` (REQ-059) is operator-facing: the `ssh-keygen -R`
|
||||
equivalent for orca's known_hosts.
|
||||
- The TOFU bugfix (T02.6) benefits **every operator who has tried
|
||||
first-connect Proxmox join since v0.6** — i.e., it fixes a feature that was
|
||||
advertised as working but wasn't.
|
||||
- Coverage uplift (REQ-057) is developer-facing (no operator pull).
|
||||
- verify-reqs (REQ-060) is internal-governance (no operator pull).
|
||||
|
||||
**Confidence:** 0.82.
|
||||
**Verdict:** **PROCEED.** The trust features have real operator pull
|
||||
(pre-pinning is a documented security best practice; the v0.6 caveat promised
|
||||
it). The coverage + hygiene work is internal-debt paydown — justified by the
|
||||
v0.7 under-shot, not by operator demand. The mix is appropriate for an NFR
|
||||
milestone.
|
||||
|
||||
#### A9-F2 — Rollback plan if P02's trust changes go wrong
|
||||
|
||||
**Evidence:**
|
||||
- P02 changes `HostKeyCallback` for all Proxmox joins + doctor probes. If the
|
||||
capture-fix corrupts `known_hosts`, the rollback is: revert the phase commit
|
||||
+ manually restore `known_hosts` from backup.
|
||||
- No data migration in P02 (known_hosts is a flat file; atomic rewrite via
|
||||
`WriteAtomic` preserves crash safety).
|
||||
- `key-reset` (T02.8) is local-only (D-046) — no remote side effects to
|
||||
reverse.
|
||||
|
||||
**Confidence:** 0.80.
|
||||
**Verdict:** **PROCEED.** Rollback is straightforward (revert + file restore).
|
||||
The atomic-rewrite requirement (AD-029) is the right mitigation.
|
||||
|
||||
---
|
||||
|
||||
## Binding Verdicts Table
|
||||
|
||||
| # | Axis | Finding | Verdict | Condition | Confidence |
|
||||
|---|------|---------|---------|-----------|------------|
|
||||
| A2-F1 | Scope | TOFU bugfix is a v0.6 ship-defect bundled into P02 as a feature task | PROCEED-WITH-CONDITION | Label T02.6 as a v0.6 bugfix in PLAN; P04 audit records it as a ship-defect closure | 0.62 |
|
||||
| A2-F2 | Scope | 3 zero-test packages at 50% toe-hold | PROCEED | — | 0.85 |
|
||||
| A2-F3 | Scope | 37 tasks / 4 phases size | PROCEED | — | 0.80 |
|
||||
| A1-F1 | Business | v0.8 is the right next milestone (not polish) | PROCEED | — | 0.90 |
|
||||
| A3-F1 | Architecture | sessionRunner + peerDispatcher seams do not leak test concerns | PROCEED | — | 0.88 |
|
||||
| A3-F2 | Architecture | P02 stays within existing security boundary | PROCEED | — | 0.90 |
|
||||
| A3-F3 | Architecture | T02.6 + T02.9 are paired changes (bootstrap + doctor share the defect) | PROCEED-WITH-CONDITION | Add P02 verification line for doctor proxmox first-connect parity with bootstrap | 0.75 |
|
||||
| A4-F1 | People | internal/cli/node.go territory collision adjudicated | PROCEED | — | 0.82 |
|
||||
| A4-F2 | People | Key-person risk on backend-engineer in P02 | PROCEED | — | 0.70 |
|
||||
| A5-F1 | Timeline | 70% cli coverage in one phase is a stretch (RESEARCH says 55-65%) | PROCEED-WITH-CONDITION | Add escape valve: ship cli at 65% if 70% not reached after Wave 2; do not block P02/P03 | 0.65 |
|
||||
| A6-F1 | Budget | Zero new deps verified | PROCEED | — | 0.95 |
|
||||
| A7-F1 | Risks | 10 pitfalls mitigations are concrete | PROCEED | — | 0.88 |
|
||||
| A7-F2 | Risks | TOFU fix blast radius — no migration-path test | PROCEED-WITH-CONDITION | Add T02.10 case 7: known_hosts pre-populated → second-connect matches (v0.6→v0.8 migration) | 0.70 |
|
||||
| A8-F1 | Governance | verify-reqs regex does not match v0.2's `**COMPLETE (merged...)**` header | PROCEED-WITH-CONDITION | Change regex to substring-match COMPLETE within bold span; add golden fixture | 0.92 |
|
||||
| A8-F2 | Governance | Single-direction check misses reverse drift + code-vs-doc drift (REQ-053 case) | PROCEED-WITH-CONDITION | Add reverse-direction assertion; document that code-vs-doc drift is out of scope for REQ-060 | 0.78 |
|
||||
| A8-F3 | Governance | No explicit "stop" trigger | PROCEED | — | 0.60 |
|
||||
| A9-F1 | Adoption | Operator pull exists for trust features; coverage/hygiene is internal debt | PROCEED | — | 0.82 |
|
||||
| A9-F2 | Adoption | Rollback plan is straightforward (revert + file restore) | PROCEED | — | 0.80 |
|
||||
|
||||
---
|
||||
|
||||
## Required Plan Changes (4 conditions)
|
||||
|
||||
1. **T02.6 labeling (A2-F1):** Add a note to T02.6 in `PLAN_v0.8.md` marking
|
||||
it as a **v0.6 ship-defect bugfix** (first-connect Proxmox join has been
|
||||
broken since v0.6 shipped due to `knownhosts.New` returning
|
||||
`KeyError{Want:[]}` with no capture-and-persist). P04 audit (T04.2) must
|
||||
record it as a ship-defect closure, not a v0.8 feature.
|
||||
|
||||
2. **P02 verification parity for doctor (A3-F3):** Add to Phase 2 Verification:
|
||||
"`doctor proxmox` first-connect on a node with empty known_hosts → captures
|
||||
the key + writes known_hosts + probe succeeds (mirrors T02.10 case 3 for
|
||||
bootstrap). T02.6 and T02.9 are a paired change; the phase is not complete
|
||||
until both callbacks use the capture-fix wrapper."
|
||||
|
||||
3. **T01.6 cli coverage escape valve (A5-F1):** Add to T01.6 acceptance
|
||||
criterion: "If ≥65% (excluding `daemon.go`) is achieved but 70% is not after
|
||||
Wave 2 effort, document the gap in a test-file comment + record a v0.9
|
||||
follow-up; ship P01 at 65% for cli. Do NOT block P02/P03 on the last 5% of
|
||||
cli coverage." (Carries forward RESEARCH §1.4's own flag as an explicit
|
||||
escape valve.)
|
||||
|
||||
4. **verify-reqs regex + reverse direction (A8-F1 + A8-F2):**
|
||||
- Change T03.1 ROADMAP-complete regex from
|
||||
`^##\s*Milestone\s+v0\.\d+:.*—\s*\*\*COMPLETE\*\*` to
|
||||
`^##\s*Milestone\s+v0\.\d+:.*—\s*\*\*[^*]*COMPLETE[^*]*\*\*` (substring
|
||||
match within the bold span — handles `**COMPLETE**`,
|
||||
`**COMPLETE (merged to main via v0.3)**`, and future variants).
|
||||
- Add T03.2 golden fixture: a ROADMAP with
|
||||
`**COMPLETE (merged to main via v0.3)**` → assert the milestone is
|
||||
detected as complete.
|
||||
- Add reverse-direction assertion to T03.1: every REQUIREMENTS row marked
|
||||
`**Complete**` must reference a milestone ROADMAP marks COMPLETE (catches
|
||||
premature-Complete drift).
|
||||
- Add a PLAN note: "REQ-060 catches doc-vs-doc drift only. Code-vs-doc
|
||||
drift (e.g., REQ-053 marked Complete but `cert_repo_test.go` missing —
|
||||
verified missing in v0.7 ship) is NOT caught by this gate; it requires
|
||||
the P04 `ciagent-audit` code-level review."
|
||||
|
||||
Additionally (lower-priority, from A7-F2):
|
||||
|
||||
5. **T02.10 case 7 (A7-F2):** Add integration test case: "known_hosts
|
||||
pre-populated with a valid OpenSSH line for the host (simulating a v0.6
|
||||
deployment or manual `ssh-keyscan`) → connect matches + succeeds. Covers
|
||||
the v0.6→v0.8 migration path."
|
||||
|
||||
---
|
||||
|
||||
## Escalations
|
||||
|
||||
None. All 9 axes resolved at confidence ≥ 0.60. No axis requires escalation to
|
||||
the operator; the 4 conditions are within the plan-author's authority to apply
|
||||
before P01 execution begins.
|
||||
|
||||
---
|
||||
|
||||
## What the Plan Is NOT Doing (and should it?)
|
||||
|
||||
- **Not lifting the 3 zero-test packages to 70%.** Correct per D-047 — deferred
|
||||
to v0.9. Not a gap.
|
||||
- **Not adding a `peerDispatcher` seam unless needed.** Correct — conditional
|
||||
on T01.4's 70% via `httptest.NewTLSServer`. Not a gap.
|
||||
- **Not pre-populating `known_hosts` from a remote keyscan API.** Correct —
|
||||
TOFU + manual `--host-key-fingerprint` cover the v0.8 surface. Not a gap.
|
||||
- **Not catching code-vs-doc drift in verify-reqs.** **Known limitation** —
|
||||
REQ-060 is a markdown-vs-markdown check. The REQ-053
|
||||
`cert_repo_test.go`-missing case proves this class of drift is real. P04
|
||||
`ciagent-audit` is the backstop. Documented in condition #4.
|
||||
|
||||
---
|
||||
|
||||
## Simplest 80%-of-the-value version
|
||||
|
||||
If forced to cut v0.8 to its smallest valuable form: **keep P02 (trust
|
||||
hardening + TOFU bugfix) and P03 (verify-reqs); drop P01's coverage uplift for
|
||||
the 3 zero-test packages + cli.** The TOFU bugfix alone (T02.6 + T02.9) fixes a
|
||||
shipped security defect — that's the highest-value work. The verify-reqs gate
|
||||
prevents the v0.7 drift from recurring. The coverage uplift on the 6
|
||||
under-50% packages is valuable but not urgent; the 3 zero-test toe-holds are
|
||||
the lowest-value work in the milestone. **The plan as written does not over-
|
||||
scope** — it includes all of the above because the marginal cost is low — but
|
||||
if P01 slips, the 3 toe-holds + cli are the first cuts to make.
|
||||
|
||||
---
|
||||
|
||||
## What Would Have to Be True for v0.8 to Succeed in the Next 90 Days
|
||||
|
||||
1. The `sessionRunner` seam (T01.1) unlocks proxmox 70% — **plausible** (same
|
||||
pattern as the existing `sshDialer` seam).
|
||||
2. `httptest.NewTLSServer` suffices for transport 70% without a new seam —
|
||||
**plausible** (standard Go testing fixture).
|
||||
3. The TOFU capture-fix (T02.6) is correct — **plausible** (verified against
|
||||
x/crypto v0.54.0 semantics; integration tests T02.10 cover the cases).
|
||||
4. `verify-reqs` regex matches all ROADMAP milestone header variants — **NOT
|
||||
true today** (v0.2 header mismatch — condition #4 fixes this).
|
||||
5. cli hits 70% in one phase — **NOT confirmed** (RESEARCH says 55-65%;
|
||||
condition #3 adds the escape valve).
|
||||
|
||||
(4) and (5) are the two conditions that move the plan from "optimistic" to
|
||||
"sound." Both are addressed by the 4 required changes.
|
||||
|
||||
---
|
||||
|
||||
**End of grill report.** Apply the 4 conditions to `PLAN_v0.8.md` before P01
|
||||
execution. No REPLAN; no escalations. Overall verdict: **PROCEED-WITH-
|
||||
CONDITION** (confidence 0.78).
|
||||
@@ -1,485 +0,0 @@
|
||||
# Grill v0.9 — Adversarial Review of Re-Architecture
|
||||
|
||||
**Reviewer**: ci-griller (adversarial red-team)
|
||||
**Date**: 2026-08-05
|
||||
**Subject**: PRD that SUPERSEDES shipped v0.8 architecture; user committed to full re-architecture
|
||||
**Default stance**: infeasible / over-scoped / too costly until evidence forces otherwise
|
||||
|
||||
## Resolution note (recorded after grill completion)
|
||||
|
||||
The grill returned an overall **REPLAN** verdict (0.74) on three axes
|
||||
(Scope, Migration, Re-architecture Justification). The user reviewed the fork
|
||||
and **overrode the Re-architecture Justification axis' *direction*** with a
|
||||
recorded six-part evidence basis (see PROJECT.md Supersession Table):
|
||||
|
||||
1. The v0.8 daemon model is operationally failing in the target environment.
|
||||
2. step-ca is externally mandated.
|
||||
3. Multi-tenancy is a hard product requirement.
|
||||
4. WASM is a hard workload requirement.
|
||||
5. SSH-push is the only viable deployment target for the operator's environment.
|
||||
6. Simplicity/vision correction — the v0.1-v0.8 daemon model was a wrong turn.
|
||||
|
||||
Per the override, the three REPLAN axes' **direction** is settled (the
|
||||
re-architecture proceeds). Their **mechanics** remain as binding work items:
|
||||
- **Scope** mechanics → reorder phases (PC-01..PC-10), add deprecation sweep
|
||||
phase, split heavy phases.
|
||||
- **Migration** mechanics → split P14 into P14a/P14b/P14c, design migration
|
||||
ordering in v0.9-P00.
|
||||
- **Security** mechanics → threat model in v0.10-P15.5 (C-19).
|
||||
|
||||
The 19 binding conditions (C-01..C-19) and 10 phase challenges (PC-01..PC-10)
|
||||
are adopted in full as execution gates.
|
||||
|
||||
---
|
||||
|
||||
## Axis 1 — Feasibility
|
||||
|
||||
**Forcing questions**: Can five external apt packages (step-ca, Traefik,
|
||||
Syncthing, wasmtime, podman) truly be orchestrated from a single stateless CLI
|
||||
over SSH with no Orca-side code on the server, while still satisfying the
|
||||
"single binary, minimal deps" constraint? Is the SSH-push-to-bare-servers
|
||||
model sound at the latency/reliability required for a 10-second pull loop?
|
||||
wasmtime's canonical Go binding (`bytecodealliance/wasmtime-go`) is CGO — does
|
||||
wasmtime integration break the cross-compile story (D-002 modernc/sqlite was
|
||||
chosen for exactly CGO-freedom)?
|
||||
|
||||
**Evidence**: Constraint conflict between PROJECT.md:5 ("no container runtime")
|
||||
and PRD R-001 (podman as one of 5 runtimes). D-002 selected modernc/sqlite for
|
||||
"Cross-compile friendly, no CGO dependency." No evidence in the PRD that a
|
||||
CGO-free wasmtime binding exists. The PRD itself was not checked in (now
|
||||
resolved: `.ciagent/PRD_v0.9.md`).
|
||||
|
||||
**Verdict**: PROCEED-WITH-CONDITION
|
||||
**Confidence**: 0.62
|
||||
|
||||
**Binding conditions**:
|
||||
- **C-01**: Before P07b (wasmtime), produce a written evaluation of wasmtime Go
|
||||
bindings including CGO impact on the cross-compile target matrix. If
|
||||
wasmtime-go requires CGO, either (a) drop wasmtime as *primary* runtime and
|
||||
promote podman/process, or (b) explicitly revoke D-002's CGO-free rationale
|
||||
with a documented scope-consequence note. No silent reversal.
|
||||
- **C-02**: Before P09 (Storage replication), produce a Syncthing feasibility
|
||||
spike: successful CLI-driven config injection, conflict-resolution policy,
|
||||
and a documented failure mode when Syncthing diverges. The 10-second pull
|
||||
loop must still terminate with a deterministic state under conflict.
|
||||
- **C-03**: The PRD must be checked into `.ciagent/` before any v0.9 phase
|
||||
begins execution. ✅ Resolved — committed as `.ciagent/PRD_v0.9.md`.
|
||||
|
||||
**Rationale**: SSH-push is individually feasible — Ansible, Salt prove the
|
||||
pattern. The aggregate is the risk: five daemons, all configured over SSH,
|
||||
with bash as the reconciliation language. The wasmtime/CGO conflict could
|
||||
silently break the build story; must be spiked before commitment.
|
||||
|
||||
## Axis 2 — Scope
|
||||
|
||||
**Forcing questions**: Is the 27-phase plan realistically scoped when it
|
||||
simultaneously deprecates 7 shipped subsystems and adds 8 net-new subsystems?
|
||||
The deprecation of ~10k lines of shipped daemon/transport/CA code is not listed
|
||||
as a phase. v0.9 P0a..P10 ship 10 phases of workload features before the
|
||||
transactional control plane (R-010 deferred to v0.10 P10) — is that intentional
|
||||
or a sequencing error? Hidden requirements (step-ca self-upgrade, master.key
|
||||
rotation, Syncthing version drift)?
|
||||
|
||||
**Evidence**: v0.9 phase ordering ships P0a..P10 workloads, then P10 lead rules
|
||||
+ migration *last*. The transactional plane (R-010) is deferred to v0.10 P10 —
|
||||
two milestones away. v0.8 was a 4-phase NFR milestone; v0.6 was 4-phase feature.
|
||||
The PRD's v0.9 (11) + v1.0 (16) = 27 phases is 3-4× prior milestone size with
|
||||
no evidence the throughput model was re-validated. No phase is labeled
|
||||
"deprecate daemon/transport/internal-CA."
|
||||
|
||||
**Verdict**: REPLAN (mechanics — direction settled by override)
|
||||
**Confidence**: 0.78
|
||||
|
||||
**Mechanics adopted**:
|
||||
- **PC-01**: Move the transactional plane primitives forward. The
|
||||
transactional primitives (desired-state, lead-applier, drift, rollback) are
|
||||
the substrate every workload phase depends on. Design spike in v0.9-P00;
|
||||
full implementation in v0.10-P10 per PRD ordering (workloads first is accepted
|
||||
given the dual-write window mitigation in I-C-006).
|
||||
- **PC-02**: Add `v0.9-P00 — Deprecation sweep` as an explicit phase. Must land
|
||||
before any new feature phase so coverage gates don't measure dead packages.
|
||||
- **PC-03**: Split migration: `v0.9-P00b — Migration design + dry-run` (early,
|
||||
parallel to deprecation) and `v0.10-P14 — Production migration` (final).
|
||||
Migration design must inform every earlier phase, not be informed by them.
|
||||
|
||||
**Rationale**: 27 phases framed as "two milestones" while simultaneously
|
||||
deleting 10k lines and adding 8 subsystems is a multi-quarter effort. The
|
||||
deprecation work is a real phase that was not on the plan. With the override
|
||||
and the v0.9-P00 additions, the plan is now structurally sound.
|
||||
|
||||
## Axis 3 — Cost / Effort
|
||||
|
||||
**Forcing questions**: Realistic phase count if each phase is held to the same
|
||||
4-layer verification bar (REQ-060) and 70% coverage floor (D-042/D-047)?
|
||||
Personas active: 3 of 8; 5 dormant map directly to the 5 new apt dependencies.
|
||||
Deprecation cost — deleting 10k lines, rewriting tests, removing coverage-gate
|
||||
packages? The bash scripts (8 in §26.D) are a net-new language surface; bash
|
||||
testing frameworks not in current dep map — what's the cost?
|
||||
|
||||
**Evidence**: Active roster has 3 of 8 active; the 5 dormant personas map
|
||||
directly to the 5 new apt dependencies. v0.8 took 4 phases for a pure
|
||||
test/coverage milestone; v0.10 includes 11 distinct subsystems in one
|
||||
"milestone." No bash test infrastructure exists today.
|
||||
|
||||
**Verdict**: PROCEED-WITH-CONDITION
|
||||
**Confidence**: 0.70
|
||||
|
||||
**Binding conditions**:
|
||||
- **C-04**: Produce a per-phase sizing estimate using v0.6/v0.7/v0.8 actuals
|
||||
as the analogous baseline. If realistic phase count exceeds 35, the
|
||||
milestone must be split into v0.9 + v0.10 (three milestones), not two.
|
||||
- **C-05**: Reactivate or explicitly assign coverage for the dormant personas'
|
||||
domains (security, network, devops); no "dormant" = "unowned."
|
||||
- **C-06**: Decide and document whether bash scripts count toward the coverage
|
||||
gate. If exempt, the exemption is recorded as a binding decision with a
|
||||
compensating control (bats/shellcheck/shfmt in CI). If not exempt, the effort
|
||||
estimate must include bash test authoring.
|
||||
|
||||
**Rationale**: The work is physically doable, but the framing as "two
|
||||
milestones" is a cost fiction. The realistic shape is three milestones minimum,
|
||||
with the deprecation work as its own phase and bash testing either added to
|
||||
the gate or explicitly exempted with a documented compensating control.
|
||||
|
||||
## Axis 4 — Technical Risk
|
||||
|
||||
**Forcing questions**: CA migration — PRD reverses AD-010 and replaces the
|
||||
shipped internal Go CA. What is the migration path for existing `ca.crt`/
|
||||
`ca.key`/`server.crt`/`server.key` on every running cluster? SPIFFE SVID
|
||||
minting at submit time (D-068) reverses the PROJECT.md:94 SPIFFE rejection —
|
||||
has anyone prototyped the mint-at-submit path? Lead-applier as bash + systemd
|
||||
with no Orca code on the server — when `orca-pull.sh` fails mid-render, what
|
||||
is the recovery? Traefik dynamic config atomicity — mid-write, Traefik may
|
||||
re-read a half-written file. Syncthing replication correctness on a 10-second
|
||||
pull loop means the lead may render against stale state.
|
||||
|
||||
**Evidence**: AD-010 (ARCHITECTURE.md:463) is an explicit documented decision
|
||||
*against* step-ca. The PRD reversal has no recorded re-evidence of what changed
|
||||
(now resolved by the override justification). SPIFFE rejection at PROJECT.md:94
|
||||
is the same pattern. No mention in the PRD of a tmpfile+rename protocol for
|
||||
Traefik config, no Syncthing conflict-resolution policy, no `orca-pull.sh`
|
||||
failure semantics. The shipped `internal/transport/mtls.go` had
|
||||
retry+backoff+idempotency (REQ-037). The bash replacement has no equivalent
|
||||
specified.
|
||||
|
||||
**Verdict**: PROCEED-WITH-CONDITION
|
||||
**Confidence**: 0.72
|
||||
|
||||
**Binding conditions**:
|
||||
- **C-07**: Before P0a, write a CA migration spec: either (a) preserve existing
|
||||
`ca.crt` trust root and import into step-ca, or (b) document forced
|
||||
re-bootstrap as an accepted breaking change with per-cluster upgrade
|
||||
procedure. Cannot be deferred.
|
||||
- **C-08**: Before the first SPIFFE-touching phase (v0.10 P02 ACL), produce a
|
||||
working spike of step-ca JWT-SVID or X.509-SVID minting from the orca CLI
|
||||
(v0.10-P01.5). If the spike fails, SPIFFE is deferred and ACL falls back to
|
||||
mTLS identity (which the shipped model already had).
|
||||
- **C-09**: Define and test the `orca-pull.sh` failure contract: idempotent
|
||||
re-run, bounded retry, deterministic state on partial failure, syslog
|
||||
emission on every failure with a structured tag the CLI can scrape.
|
||||
- **C-10**: Define the Traefik config atomicity protocol (tmpfile + fsync +
|
||||
rename) and verify Traefik's behavior on malformed config (does it
|
||||
hold-last-good or fail?). Documented, tested.
|
||||
|
||||
**Rationale**: Each of the five technical unknowns is independently survivable
|
||||
with a spike; the risk is that all five land in the same milestone without
|
||||
any of them being spiked first. The CA-migration and SPIFFE items reverse
|
||||
documented rejections and so carry the highest re-evidence burden (now met by
|
||||
the override). The bash-control-plane risk is the one most likely to produce a
|
||||
"works in demo, fails in week 3 of production" failure mode.
|
||||
|
||||
## Axis 5 — Migration Risk
|
||||
|
||||
**Forcing questions**: §24 covers *data* migration (cert paths,
|
||||
config.hcl→config.md, db relocation). It does *not* cover *daemon cutover*: how
|
||||
do you stop `orca daemon` on every peer without losing the in-flight
|
||||
allocations those daemons are supervising? What happens to running
|
||||
allocations during `orca upgrade --to-v1.0`? The old model has the daemon as
|
||||
process parent; the new model has systemd units emitted by the CLI — there is
|
||||
no process-parent continuity. The transition period where some peers are v0.8
|
||||
(daemon) and some are v1.0 (no daemon) — what is the failure mode? In-flight
|
||||
jobs during upgrade — wait for drain, force-kill, or queue-and-replay?
|
||||
|
||||
**Evidence**: §24 covers cert paths, config.hcl→config.md, db relocation —
|
||||
three file-layout migrations. It omits four operational migrations: daemon
|
||||
cutover, running-allocation adoption, mixed-version cluster, in-flight jobs.
|
||||
The shipped executor (`internal/engine/executor.go:163`) uses
|
||||
`os/exec.CommandContext` — the daemon is the process parent. systemd units
|
||||
emit by the CLI would be a *different* parent (systemd). Process reparenting
|
||||
is not portable across the orca model. "Atomic, auto-rollback" is asserted for
|
||||
§24 but no trigger, no unit, no boundary is defined.
|
||||
|
||||
**Verdict**: REPLAN (mechanics — direction settled by override)
|
||||
**Confidence**: 0.82
|
||||
|
||||
**Mechanics adopted**:
|
||||
- **PC-04**: Split P14 into `v0.10-P14a — Data migration` (current scope),
|
||||
`v0.10-P14b — Daemon cutover + running-allocation adoption`,
|
||||
`v0.10-P14c — Mixed-version cluster tolerance + no-orca-on-server enforcement`.
|
||||
Three sub-phases, each with its own integration test.
|
||||
|
||||
**Rationale**: The migration plan as described covers the easy third (file
|
||||
layout) and omits the hard two-thirds (running processes and mixed-version
|
||||
clusters). A re-architecture that has no answer for "what happens to running
|
||||
workloads during the upgrade" is not shippable. With the P14 split, the plan
|
||||
is now complete.
|
||||
|
||||
## Axis 6 — Operational Risk
|
||||
|
||||
**Forcing questions**: When `orca-pull.sh` fails on the lead, what happens to
|
||||
workloads? When step-ca is down, can new workloads start? When Syncthing
|
||||
conflicts, what is the conflict-resolution policy? The lead's systemd timers
|
||||
drift when the lead is under load — how is timer starvation detected? No Orca
|
||||
binary on the server means no `orca doctor` on the server — the shipped doctor
|
||||
(REQ-032, REQ-052) ran locally on each node; the new model requires every
|
||||
diagnostic to be SSH-pushed from the CLI.
|
||||
|
||||
**Evidence**: The shipped `orca doctor` runs locally (ARCHITECTURE.md §5,
|
||||
REQ-032). The PRD's R-001 ("no orca binary on any server") implicitly deletes
|
||||
server-side doctor. The shipped model had `orca daemon` on every node
|
||||
providing `/healthz` — a local liveness signal. The new model has no
|
||||
server-side health producer. step-ca as a single point of failure is
|
||||
documented in step-ca's own operations guide (out-of-band knowledge).
|
||||
|
||||
**Verdict**: PROCEED-WITH-CONDITION
|
||||
**Confidence**: 0.68
|
||||
|
||||
**Binding conditions**:
|
||||
- **C-11**: Define the lead-side watchdog: a meta-timer that fires when
|
||||
`orca-pull.sh` has not successfully run in N seconds, emitting a structured
|
||||
alert. Document the alert path (syslog? CLI-pull?).
|
||||
- **C-12**: Document step-ca's HA story. If step-ca is single-node, that
|
||||
decision is recorded as an accepted SPOF with the mitigation being
|
||||
"workloads continue to run; only new submits are blocked." If step-ca is
|
||||
multi-node, the RAFT/sync story is part of the orca plan and must be sized.
|
||||
- **C-13**: Replace server-side doctor with a CLI-driven equivalent that
|
||||
SSH-probes every node and reconstructs the health view the daemon used to
|
||||
provide locally. This is a new requirement, not a feature; added as
|
||||
I-C-002 / v0.10-P14c.
|
||||
- **C-14**: Syncthing conflict-resolution policy must be deterministic,
|
||||
documented, and tested with a forced-divergence integration test.
|
||||
|
||||
**Rationale**: The operational model replaces a distributed system (daemons
|
||||
with health endpoints) with a centralized polling system (CLI over SSH) and a
|
||||
bash control plane on the lead. The mitigations are knowable but unspecified.
|
||||
|
||||
## Axis 7 — Security
|
||||
|
||||
**Forcing questions**: The master.key (AES-256-GCM for `.env.secrets`) is mode
|
||||
0600 on the CLI host with no passphrase — stolen key = all secrets in
|
||||
plaintext. The shipped model distributed keys with operator mediation (D-012).
|
||||
SSH is now the primary transport to every server — does the orca SSH key have
|
||||
a passphrase, or is it also bare 0600? The sudoers allowlist on peers grants
|
||||
the `orca` user privileged command access — does it grow to include
|
||||
`systemctl restart traefik`, `step ca ...`, `podman ...`? Five new attack
|
||||
surfaces: step-ca, Traefik, Syncthing, wasmtime, podman. SPIFFE SVIDs minted
|
||||
at submit time means the CLI holds the minting authority — if the CLI host is
|
||||
compromised, it mints valid SVIDs for the whole cluster.
|
||||
|
||||
**Evidence**: Shipped security posture: mTLS daemon-to-daemon, internal CA on
|
||||
a node, operator-mediated CA cert distribution (D-012 "no secret distribution
|
||||
over the wire, matches offline-first"). The shipped model was deliberately
|
||||
designed to avoid secret transport. New posture: CLI holds master.key (no
|
||||
passphrase), CLI mints SVIDs, SSH from CLI to every server with a (presumably)
|
||||
un-passphrased Ed25519 key, 5 daemons on every server each with their own
|
||||
attack surface. ARCHITECTURE.md:464 "AD-011 Operator-mediated CA cert
|
||||
distribution: No secret distribution over the wire." The new model puts a
|
||||
master.key on the CLI and uses SSH to push to every server — secret-over-the-wire
|
||||
is now the default.
|
||||
|
||||
**Verdict**: REPLAN (mechanics — direction settled by override)
|
||||
**Confidence**: 0.74
|
||||
|
||||
**Mechanics adopted**:
|
||||
- **C-19**: Write a threat model for the new posture before any
|
||||
security-touching phase (v0.10-P15.5). Defend master.key + CLI mint authority
|
||||
or revise. The shipped model deliberately avoided putting a single stealable
|
||||
file on a single host that decrypts all secrets and mints all identities.
|
||||
The threat model must document why the new posture is acceptable or specify
|
||||
mitigations (OS keyring, hardware secret, split keys).
|
||||
|
||||
**Rationale**: The re-architecture reverses the offline-first, no-secret-transport
|
||||
principle (AD-011) and centralizes minting authority + secret encryption on
|
||||
the CLI host with no passphrase. A threat model must be written and the
|
||||
master.key + CLI-mint-authority design defended or revised before any
|
||||
security-touching phase begins.
|
||||
|
||||
## Axis 8 — Maintainability
|
||||
|
||||
**Forcing questions**: The PRD moves logic from Go (type-safe, tested, in the
|
||||
orca binary, gated by REQ-057 coverage) to bash (untyped, hard to test, 8
|
||||
scripts in `scripts/`). How will the 8 bash scripts be tested under the
|
||||
project's coverage gate? Drift between Go-side emitters and bash-side appliers
|
||||
— when the Go side changes a render format, the bash side must change in
|
||||
lockstep; there is no compiler to catch this. The shipped `internal/transport`
|
||||
had retry, backoff, idempotency keys, structured mTLS failure logs. The bash
|
||||
replacement has none specified. Bash has no native structured logging (the
|
||||
project standard is slog JSON, REQ-008). The 8 scripts are a new language
|
||||
surface in a Go-only project.
|
||||
|
||||
**Evidence**: PROJECT.md:5 vision: "minimalist, offline-first, CLI-first
|
||||
orchestration engine prioritizing stability, security, and simplicity over
|
||||
feature richness." An 8-script bash control plane is not minimal by any prior
|
||||
definition used in this project. The shipped code has structured slog JSON
|
||||
logging (REQ-008), audit log (REQ-006), error wrapping (REQ-018), context
|
||||
propagation (REQ-017). Bash has none of these natively. No bash test
|
||||
framework in current dep map; no `bats`/`shunit2` reference. The 70%/50%
|
||||
coverage gate (D-042/D-047) is Go-specific.
|
||||
|
||||
**Verdict**: PROCEED-WITH-CONDITION
|
||||
**Confidence**: 0.66
|
||||
|
||||
**Binding conditions**:
|
||||
- **C-15**: Adopt a bash testing framework (bats or shunit2) and a static-analysis
|
||||
gate (`shellcheck`, `shfmt -d`) in CoreCI before any bash script ships. Bash
|
||||
scripts must have at least one integration test covering the happy path and
|
||||
one covering the failure path.
|
||||
- **C-16**: Define a **render-format contract** between Go emitters and bash
|
||||
appliers. Minimum: a versioned JSON schema for every rendered artifact,
|
||||
validated on both sides. The bash side rejects unparseable input with a
|
||||
structured error, never silently.
|
||||
- **C-17**: Bash scripts must emit slog-compatible JSON to syslog with the same
|
||||
field set (timestamp, actor, action, resource, result, error) as the Go
|
||||
audit log (REQ-006). No unstructured text in audit.
|
||||
- **C-18**: Every capability present in shipped `internal/transport` (retry,
|
||||
backoff, idempotency, structured mTLS failure logs) must have a documented
|
||||
bash-side equivalent or be explicitly accepted as dropped with a recorded
|
||||
rationale. Capability regressions must be visible, not silent.
|
||||
|
||||
**Rationale**: Bash is not inherently unmaintainable, but bash *in a Go-only,
|
||||
coverage-gated, structured-logging project* is a language-without-rails. Without
|
||||
the four conditions above, the bash control plane becomes the part of the
|
||||
codebase that everyone is afraid to touch by v0.10 P05. The drift between Go
|
||||
emitters and bash appliers is the single most likely source of "works on the
|
||||
CLI's machine, fails on the lead" bugs.
|
||||
|
||||
## Axis 9 — Re-Architecture Justification
|
||||
|
||||
**Forcing questions**: The PRD reverses 6 documented decisions (AD-010
|
||||
step-ca, SPIFFE rejection, no-container-runtime, no-multi-tenancy,
|
||||
HCL-canonical, daemon-on-every-node). For each reversal, what *new evidence*
|
||||
since the original decision justifies the reversal? The shipped v0.8 model is
|
||||
*working* — 8 milestones, REQ-001..060 Complete, 4-layer verification passing,
|
||||
coverage gates met. What is the *specific failure* of the shipped model that
|
||||
an incremental extension could not fix? What would be *lost* by incrementally
|
||||
extending the shipped model: add workload kinds, add secrets, add a
|
||||
transactional layer *on top of the daemon*? Is this re-architecture driven by a
|
||||
*real operational pain* or by an *architectural preference*?
|
||||
|
||||
**Evidence**: ROADMAP.md and PROJECT.md: every milestone from v0.1 to v0.8
|
||||
explicitly says "the vision is unchanged; this milestone is not a direction
|
||||
change." v0.9/v0.10 is the *first* milestone in the project's history that
|
||||
reverses the vision's anti-patterns. AD-010's rationale: "step-ca/cfssl/
|
||||
vault-pki too heavyweight for Orca's footprint." Nothing in the original PRD
|
||||
suggested Orca's footprint changed. The shipped model's `internal/transport`
|
||||
provides retry, backoff, idempotency, structured mTLS failure logs. The PRD
|
||||
replaces this with bash + systemd + SSH. No evidence the shipped transport
|
||||
was a source of operational pain.
|
||||
|
||||
**Verdict**: REPLAN (direction overridden by user with recorded justification)
|
||||
**Confidence**: 0.70
|
||||
|
||||
**Override**: The user provided a six-part evidence basis that addresses the
|
||||
reversal of each documented decision (see PROJECT.md Supersession Table):
|
||||
operational failure of the daemon model, external step-ca mandate, hard
|
||||
multi-tenancy requirement, hard WASM requirement, SSH-push as the only viable
|
||||
deployment target, and vision correction. The override is recorded; the
|
||||
direction holds.
|
||||
|
||||
**Residual mechanics**: The incremental-additive alternative was evaluated
|
||||
(the grill's Open Q1, Q2, Q10) and rejected on the grounds that the daemon
|
||||
model is operationally failing (ground 1) and SSH-push is the only viable
|
||||
deployment target (ground 5) — both of which foreclose the additive path.
|
||||
|
||||
**Rationale**: The default assumption — that a re-architecture of working
|
||||
shipped code is a mistake unless the case is overwhelming — is now met by the
|
||||
six-part justification. The re-architecture proceeds.
|
||||
|
||||
---
|
||||
|
||||
# Overall Verdict
|
||||
|
||||
**Verdict**: PROCEED-WITH-CONDITION (direction settled by override; mechanics gated by C-01..C-19)
|
||||
**Confidence**: 0.74
|
||||
|
||||
**Summary**: The re-architecture is technically feasible in pieces but
|
||||
structurally large as a single two-milestone jump. The override justification
|
||||
closes the Re-architecture Justification axis with a six-part evidence basis.
|
||||
The remaining mechanics: reorder phases (PC-01..PC-10), split heavy phases,
|
||||
add the v0.9-P00 deprecation/migration-ordering pre-phase, split P14 into
|
||||
three sub-phases, write the threat model in P15.5, and gate the 19 binding
|
||||
conditions (C-01..C-19) as execution gates. If the C-04 sizing estimate exceeds
|
||||
35 phases, the milestone splits into v0.9 + v0.10 + v1.0.
|
||||
|
||||
# Binding Conditions (aggregated — execution gates)
|
||||
|
||||
| ID | Condition | Blocks phase | Testable how |
|
||||
|----|-----------|--------------|--------------|
|
||||
| C-01 | Evaluate wasmtime Go binding CGO impact; if CGO-required, drop wasmtime as primary or revoke D-002 | v0.9-P07b | Build matrix spike on linux/amd64+arm64; revocation decision recorded |
|
||||
| C-02 | Syncthing feasibility spike: config injection, conflict policy, deterministic failure mode | v0.9-P09 | Spike report + forced-divergence integration test |
|
||||
| C-03 | Check PRD into `.ciagent/PRD_v0.9.md` before any v0.9 phase begins | (gate) | ✅ Resolved — file committed |
|
||||
| C-04 | Per-phase sizing estimate vs v0.6/v0.7/v0.8 actuals; if >35, split into v0.9+v0.10 | v0.9 start | ✅ RESOLVED — operator decision: keep 2 milestones (v0.9+v0.10), keep all phases (40 total), v1.0 UAT-gated after v0.10 |
|
||||
| C-05 | Reactivate or assign dormant persona domains (security, network, devops) | v0.9-P00 | PERSONAS.md updated with named owners |
|
||||
| C-06 | Decide bash coverage-gate status; if exempt, record compensating control | v0.9-P00 | Decision recorded in PROJECT.md D-series; CI pipeline shows the gate |
|
||||
| C-07 | CA migration spec: preserve existing trust root or document forced re-bootstrap | v0.10-P14a | Spec doc + migration dry-run on test cluster |
|
||||
| C-08 | SPIFFE SVID minting spike; if fails, fall back to mTLS identity | v0.10-P02 (spike in P01.5) | Working SVID mint from orca CLI in sandbox |
|
||||
| C-09 | `orca-pull.sh` failure contract: idempotent re-run, bounded retry, deterministic state, structured syslog | v0.10-P10 | Failure-path integration test + syslog structured-tag verification |
|
||||
| C-10 | Traefik config atomicity protocol (tmpfile+fsync+rename) + malformed-config behavior verified | v0.9-P02 | Atomic-rename test + Traefik malconfig-hold-last-good assertion |
|
||||
| C-11 | Lead-side watchdog meta-timer for `orca-pull.sh` starvation, with structured alert path | v0.10-P09 | Watchdog fires on injected pull failure; alert received |
|
||||
| C-12 | Document step-ca HA story; if single-node, record as accepted SPOF with mitigation | v0.10-P09 | Decision doc; if HA, RAFT/sync story in orca plan |
|
||||
| C-13 | Replace server-side doctor with CLI-SSH-driven equivalent | v0.10-P14c | New REQ-086 in REQUIREMENTS.md; integration test SSH-probes N nodes |
|
||||
| C-14 | Syncthing conflict-resolution policy deterministic + forced-divergence integration test | v0.10-P09 | Test induces divergence; resolves to single deterministic state |
|
||||
| C-15 | Bash testing framework (bats/shunit2) + shellcheck + shfmt in CoreCI before any bash ships | v0.9-P00 | CI pipeline green with the gate on a sample script |
|
||||
| C-16 | Versioned JSON-schema render-format contract between Go emitters and bash appliers | v0.9-P00 | Schema file in repo; both sides validate; mismatch fails CI |
|
||||
| C-17 | Bash scripts emit slog-compatible JSON to syslog with audit-log field set (REQ-006) | v0.9-P00 | Syslog capture test verifies field-presence + JSON parse |
|
||||
| C-18 | Document bash-side equivalents (or accepted drops) for shipped transport capabilities | v0.9-P00 | Capability-mapping doc in `.ciagent/` |
|
||||
| C-19 | Write a threat model for the new posture; defend master.key + CLI mint authority or revise | v0.10-P15.5 | Threat-model doc reviewed and committed; design revised if regression found |
|
||||
|
||||
# Phase Plan Challenges
|
||||
|
||||
| # | Phase | Problem | Fix |
|
||||
|---|-------|---------|-----|
|
||||
| PC-01 | v0.9 P0a–P10 | Ship 10 phases of workload features before the transactional control plane | Design spike in v0.9-P00; full impl in v0.10-P10 per PRD ordering (workloads first accepted with dual-write mitigation) |
|
||||
| PC-02 | (missing) | Deprecation of ~10k lines of daemon/transport/CA code is not a phase | Add `v0.9-P00 — Deprecation sweep` as explicit phase before any new feature phase |
|
||||
| PC-03 | v0.9 P10 | Migration is the last phase of v0.9 but is highest-risk | Split: migration design in v0.9-P00 (early), implementation in v0.10-P14 (final) |
|
||||
| PC-04 | v0.10 P14 | Covers data migration only; omits running-allocation cutover, mixed-version cluster, rollback trigger | Split into P14a (data), P14b (daemon cutover), P14c (mixed-version tolerance) |
|
||||
| PC-05 | v0.10 P02 | SPIFFE is a documented reversal with no spike; lands before spike possible | Insert `v0.10-P01.5 — SPIFFE mint spike` as hard gate before P02 |
|
||||
| PC-06 | v0.10 P10 | Transactional plane depends on lead-applier bash scripts (C-09) not gated | Reorder to v0.9-P00 design + add C-09 gate |
|
||||
| PC-07 | v0.10 P15/P16 | README before security threat model | Add `v0.10-P15.5 — Threat model + security review` before final review |
|
||||
| PC-08 | (missing) | No phase replaces server-side `orca doctor` | Add as I-C-002 / v0.10-P14c (CLI-SSH-driven doctor) |
|
||||
| PC-09 | v0.9 P09 | Syncthing lands before feasibility spike (C-02) | Spike must precede P09; if P09 is the spike, rename + gate on spike success |
|
||||
| PC-10 | v0.9 P07 | Five runtimes in one phase, including wasmtime (CGO risk) and pve-vm/pve-ct | Split: P07a (process+podman), P07b (wasmtime, C-01 gated), P07c (pve-vm+ct) |
|
||||
|
||||
# Open Questions (feed back to IDEATE/PLAN; resolved where noted)
|
||||
|
||||
1. **What measured operational failure of the shipped v0.8 daemon model is the re-architecture responding to?** — ✅ Resolved by override ground 1.
|
||||
2. **Can the v0.9 scope be delivered as additive extensions?** — ✅ Resolved: rejected per override grounds 1 + 5.
|
||||
3. **What is the wasmtime/CGO resolution?** — Closes via C-01 spike in v0.9-P07b.
|
||||
4. **What is the master.key threat model?** — Closes via C-19 in v0.10-P15.5.
|
||||
5. **What is the rollback unit of work for §24, and what triggers it?** — Must be answered in v0.9-P00 txn-design spike (I-B-007).
|
||||
6. **Is step-ca single-node acceptable as a cluster SPOF?** — Closes via C-12 in v0.10-P09.
|
||||
7. **Can the bash control plane be reduced?** — Closes in v0.9-P00 (fold 3+ scripts into Go-side SSH invocations where possible).
|
||||
8. **What is the realistic phase count?** — Closes via C-04 sizing before v0.9 starts; if >35, the plan becomes three milestones.
|
||||
9. **Does the PRD's reversal of 6 documented decisions require a formal AD-series supersession?** — ✅ Resolved: supersession table recorded in PROJECT.md + ARCHITECTURE.md.
|
||||
10. **What is the smallest possible version of this re-architecture that delivers 80% of the value?** — ✅ Resolved: the override rejected the incremental-additive path; the full re-architecture proceeds per the six-part justification.
|
||||
|
||||
# Binding Decisions (this grill session, G-001..G-009)
|
||||
|
||||
| ID | Decision | Confidence |
|
||||
|----|----------|-----------|
|
||||
| G-001 | Feasibility: PROCEED-WITH-CONDITION (C-01..C-03) | 0.62 |
|
||||
| G-002 | Scope: REPLAN mechanics (PC-01..PC-03) — direction settled by override | 0.78 |
|
||||
| G-003 | Cost: PROCEED-WITH-CONDITION (C-04..C-06) | 0.70 |
|
||||
| G-004 | Tech Risk: PROCEED-WITH-CONDITION (C-07..C-10) | 0.72 |
|
||||
| G-005 | Migration: REPLAN mechanics (PC-04) — direction settled by override | 0.82 |
|
||||
| G-006 | Op Risk: PROCEED-WITH-CONDITION (C-11..C-14) | 0.68 |
|
||||
| G-007 | Security: REPLAN mechanics (C-19) — direction settled by override | 0.74 |
|
||||
| G-008 | Maintainability: PROCEED-WITH-CONDITION (C-15..C-18) | 0.66 |
|
||||
| G-009 | Re-architecture Justification: direction overridden by user with six-part evidence basis; mechanics closed | 0.70 |
|
||||
|
||||
# Escalations (auto-resolved under full autonomy)
|
||||
|
||||
| E-ID | Item | Auto-decision | Mitigation |
|
||||
|------|------|---------------|-----------|
|
||||
| E-01 | Whether the re-architecture is justified vs incremental | OVERRIDDEN by user — direction holds | Six-part evidence basis recorded in PROJECT.md Supersession Table |
|
||||
| E-02 | Whether master.key passphrase-less posture is acceptable | REPLAN mechanics — threat model first | C-19 in v0.10-P15.5; if threat model shows regression vs shipped, revise design |
|
||||
| E-03 | Whether 27 phases fit in 2 milestones | Auto-split if sizing exceeds 35 | ✅ RESOLVED — operator: keep 2 milestones (v0.9+v0.10), keep all phases, v1.0 UAT-gated |
|
||||
@@ -1,39 +0,0 @@
|
||||
# Ideation: v0.10 Docs & Install Milestone
|
||||
|
||||
## Tier 1 — Mechanical (codebase-grounded, no new deps)
|
||||
|
||||
| ID | Idea | Source | Accepted | REQ |
|
||||
|----|------|--------|----------|-----|
|
||||
| I-M-091 | `docs/cli.md` comprehensive CLI reference | README subcommand table is stale (missing cert/daemon/doctor/audit/ns/node-capacity/node-key-reset); no `docs/` CLI reference exists | ✅ | REQ-091 |
|
||||
| I-M-092 | `docs/jobspec.md` markdown frontmatter schema reference | Operators must read `internal/jobspec/markdown.go` source to author jobspecs; no reference doc exists | ✅ | REQ-092 |
|
||||
| I-M-093 | `docs/ingress.md` Traefik ingress reference | The service→Traefik mapping (R-007, atomic reload, drain, TLS) is undocumented; the user explicitly asked for "ingress configured" | ✅ | REQ-093 |
|
||||
| I-M-094 | `examples/full-stack/` with 5 valid jobspecs + rendered artifacts + walkthrough | No examples directory exists; `testdata/` holds legacy HCL test fixtures, not operator examples | ✅ | REQ-094 |
|
||||
| I-M-095 | README.md refresh (status, subcommand table, install example, dev targets, docs/examples sections) | README says "v0.1: Foundation"; subcommand table missing 5 commands; install example pins v0.4.2 | ✅ | REQ-095 |
|
||||
| I-M-096 | `docs/namespace.md` v0.9 multi-namespace layout update | Documents the v0.8 flat layout, not the v0.9 `cluster/`+`_defaults/`+per-ns layout | ✅ | REQ-096 |
|
||||
|
||||
## Tier 2 — Backend-enriched (API/behavior-grounded)
|
||||
|
||||
| ID | Idea | Source | Accepted | REQ |
|
||||
|----|------|--------|----------|-----|
|
||||
| I-B-097 | `scripts/release.sh` cross-build amd64 + post-create asset verification | v0.8.x releases shipped with zero binary assets; install.sh resolves to v0.8.15 then errors on missing tarball; root cause of v0.4.5 install | ✅ | REQ-097 |
|
||||
| I-B-098 | `scripts/install.sh` asset fallback walk + `--check` dry-run | install.sh has no fallback when the latest release lacks the expected tarball; a broken release blocks all installs | ✅ | REQ-098 |
|
||||
|
||||
## Tier 3 — Cross-project (deferred — single-project mode)
|
||||
|
||||
No cross-project ideas. Orca is single-project mode.
|
||||
|
||||
## Rejected ideas
|
||||
|
||||
- **Backfill the existing v0.8.15 release with a binary asset** —
|
||||
rejected per D-192. Backfilling a past release is an ops task, not a
|
||||
docs milestone deliverable. The next tagged phase (P1 ship at v0.9.1)
|
||||
will be the first correctly-asseted release; install.sh's fallback
|
||||
walk handles the gap.
|
||||
- **Document both v0.8 and v0.9 paths equally** — rejected per D-191.
|
||||
The v0.8 path is deprecated and scheduled for removal; documenting it
|
||||
as primary misleads new operators.
|
||||
- **arm64 tarball in release.sh** — rejected for this milestone per
|
||||
D-193. The install user base is amd64 today; arm64 is a separate
|
||||
enhancement.
|
||||
- **Per-command `docs/cli/*.md` subdirectory** — rejected per D-188.
|
||||
Single-file `docs/cli.md` matches the existing flat `docs/` layout.
|
||||
@@ -1,183 +0,0 @@
|
||||
# Ideation v0.12: Security Hardening (Zero-Trust Identity)
|
||||
|
||||
**Status**: 30 ideas accepted (0 skipped, 0 modified). All from Tier 1
|
||||
(mechanical analysis of the threat-model review) and Tier 2
|
||||
(backend-enriched prioritization). The `--ideate` flag was passed;
|
||||
ideation ran between RESEARCH and PLAN per run.md Step 3.
|
||||
|
||||
## Tier 1 — Mechanical analysis
|
||||
|
||||
### 2.1 Git-native pattern mining
|
||||
|
||||
The v0.11 milestone shipped 24 phases with a threat model in P15.5
|
||||
(gate C-19). The threat model identified residual risks but did not
|
||||
close them -- it documented them for v1.x. The v0.12 ideation ingests
|
||||
that threat model as the primary signal source.
|
||||
|
||||
**Repeated lessons** (from v0.8..v0.11 `---ci---` blocks):
|
||||
- "Deprecated but still load-bearing" appears 6 times across v0.8..v0.11
|
||||
(legacy CA, mTLS transport, daemon, certpaths, step-ca password
|
||||
provisioner, `hmacSHA256` dead code). The dual-write window is the
|
||||
single largest attack-surface expander. -> **F16 / REQ-138**.
|
||||
- "TOFU by default, pre-pin optional" appears 4 times (v0.6 SSH join,
|
||||
v0.8 host-key-fingerprint, v0.11 drift scripts). TOFU is a
|
||||
first-connect MITM risk. -> **F15 / REQ-139** (known_hosts tightening).
|
||||
- "File modes checked at write, not at read" appears 3 times (v0.2
|
||||
cert modes, v0.5 namespace dirs, v0.11 master key). -> **F13 /
|
||||
REQ-130**.
|
||||
|
||||
**Low-confidence decisions** (confidence < 0.85 in `---ci---` blocks):
|
||||
- D-007 (mTLS for v0.1, tokens deferred) -- 0.80. v0.12 closes the
|
||||
token gap via OIDC (no Orca-issued tokens; the IdP issues them).
|
||||
- D-028 (repo visibility flip for public releases) -- 0.85. v0.12
|
||||
adds install.sh checksum verification (F14) as defense-in-depth.
|
||||
|
||||
**Escalation types**:
|
||||
- `release_pending` (v0.8..v0.11 ship fallbacks) -- not security-relevant.
|
||||
- `human_validation` (v0.11 C-19 threat model) -- v0.12 is the
|
||||
comprehensive closure of those documented risks.
|
||||
|
||||
**Compound solutions** (generalized patterns):
|
||||
- The "shellQuote + regression test" pattern from v0.8 SSH trust
|
||||
hardening (REQ-058) generalizes to all SSH-exec interpolation sites
|
||||
(podman, wasm, aggregate.sh). -> **F3 / REQ-119**.
|
||||
- The "atomic temp + chmod + fsync + rename" pattern from
|
||||
`WriteAtomic` (ca.go) generalizes to migration `copyFile` and
|
||||
backup restore. -> **F19 / REQ-137**.
|
||||
|
||||
**Partial requirements**: none (v0.11 shipped all REQs complete).
|
||||
|
||||
### 2.2 Coverage gap analysis
|
||||
|
||||
All v0.11 REQs are Complete. The v0.12 requirements are net-new from
|
||||
the threat model -- no pending/in_progress REQs to close.
|
||||
|
||||
### 2.3 Verification layer inversion
|
||||
|
||||
- **Structural**: `internal/security/ca.go` (legacy CA) documented as
|
||||
deprecated but still compiled and load-bearing. -> F16.
|
||||
- **Behavioral**: `internal/runtime/podman.go`, `wasm.go` have no
|
||||
command-injection regression tests. -> F3.
|
||||
- **Security**: No STRIDE analysis for the OIDC/WebAuthn data flow
|
||||
(new in v0.12). -> addressed by REQ-142 (docs).
|
||||
- **Quality**: `classifyDialErr` substring matching is a known code
|
||||
smell flagged in v0.9 research. -> F25.
|
||||
|
||||
### 2.4 Architectural drift detection
|
||||
|
||||
- `internal/acl/` exists but is not wired into any enforcement point
|
||||
(documented as "future" since v0.9). -> F1.
|
||||
- `internal/identity/spiffe.go` `VerifySVID` skips chain validation
|
||||
(documented as "trust is implicit via SSH channel" in v0.11 P01.5
|
||||
spike result). -> F9.
|
||||
- `internal/emitter/nft.go` ships SYN-flood + rate-limit but no
|
||||
conntrack/default-deny (the v0.11 emitter met the REQ but not
|
||||
defense-in-depth best practice). -> F21.
|
||||
|
||||
### 2.5 Spec-driven improvement
|
||||
|
||||
- R-021 ("no Orca credentials") is the new spec invariant. Every
|
||||
existing password/token surface is a spec violation under R-021.
|
||||
-> F1, F12, F17, REQ-144..148.
|
||||
- The v0.11 PRD's deferred-v1.x list included "master.key
|
||||
passphrase-less 0600 (consider OS keyring in v1.x)." v0.12 closes
|
||||
this via seal-to-OIDC (no passphrase, no OS keyring dependency --
|
||||
OIDC is the unwrap mechanism).
|
||||
|
||||
## Tier 2 — Backend-enriched analysis
|
||||
|
||||
### 2.6 Prioritization
|
||||
|
||||
Ranked by (1) severity, (2) OS-surface exposure (per user instruction
|
||||
"includes the operating system itself"), (3) ease of addressing:
|
||||
|
||||
1. **F3 command injection** (Critical, OS-touching, shellQuote is a
|
||||
well-understood fix) -> P01.
|
||||
2. **F4 path traversal** (Critical, OS-touching, validateNamespaceName
|
||||
is trivial) -> P02.
|
||||
3. **F5 txn arbitrary paths** (Critical, OS-touching, prefix allowlist)
|
||||
-> P03.
|
||||
4. **F1 ACL unenforced** (Critical, foundational for OIDC authz) ->
|
||||
P06 (after P04/P05 identity).
|
||||
5. **F6 daemon no auth** (High, OS-touching) -> P09.
|
||||
6. **F2 audit not tamper-evident** (Critical, integrity) -> P10.
|
||||
7. **F9 SVID no chain** (High, identity) -> P11.
|
||||
8. **F7 backup symlink** (High, OS-touching) -> P12.
|
||||
9. **F10 step-ca /tmp** (High, OS-touching) -> P13.
|
||||
10. **F12 master key rotation** (High, crypto) -> P14.
|
||||
11. **F11 aggregate.sh JSON injection** (High, OS-touching) -> P16.
|
||||
12. **F14 install.sh no checksum** (High, OS-touching) -> P17.
|
||||
13. **F21 nftables** (Medium, OS-touching) -> P18.
|
||||
14. **F22 sudoers** (Medium, OS-touching) -> P19.
|
||||
15. **F23 system user** (Medium, OS-touching) -> P20.
|
||||
16. **F8 SQLite** (High, OS-touching) -> P21.
|
||||
17. **F19 migration** (Medium, OS-touching) -> P22.
|
||||
18. **F16 dual-write closure** (Medium, surface reduction) -> P23.
|
||||
19. **F15/F25 transport** (Medium/Low) -> P24.
|
||||
20. **F18 drift auth** (Medium) -> P25.
|
||||
21. **Integration tests** (gate) -> P26.
|
||||
22. **Docs** (gate) -> P27.
|
||||
23. **Final review** (gate) -> P28.
|
||||
|
||||
The zero-trust identity work (P04 OIDC+Dex, P05 WebAuthn, P07 password
|
||||
removal, P08 master key seal) is wave B because it's the architectural
|
||||
foundation -- P06 (ACL) and P09 (daemon auth) depend on it.
|
||||
|
||||
### 2.7 Novel improvement suggestions
|
||||
|
||||
- **WebAuthn as the bundled password-free authenticator** (operator
|
||||
decision D-240). This is beyond pattern matching -- it's the
|
||||
strongest available authentication primitive and directly satisfies
|
||||
R-021. The `go-webauthn` library is mature; the custom Dex connector
|
||||
is ~300 LoC.
|
||||
- **Shamir 3-of-5 master key recovery** (operator decision D-241).
|
||||
Standard threshold cryptography; no backdoor; documented residual
|
||||
risk.
|
||||
- **Bundled Dex** (operator decision D-239). Zero-trust out of the
|
||||
box without external setup; BYO override preserves flexibility.
|
||||
|
||||
### 2.8 Chaos engineering ideation
|
||||
|
||||
- **What if the OIDC provider is unavailable?** -> `orca cluster
|
||||
unseal` fails; cluster runs on in-memory master key until shutdown
|
||||
(no new secrets operations). Shamir recovery if permanent. Doc'd.
|
||||
- **What if a peer's drift event is forged?** -> REQ-140 (per-peer
|
||||
HMAC).
|
||||
- **What if the master key is compromised?** -> REQ-129 (rotation,
|
||||
re-seal to OIDC). All historical secrets still compromised (no
|
||||
forward secrecy) -- documented residual risk.
|
||||
- **What if install.sh is MITM'd?** -> REQ-132 (checksum+GPG).
|
||||
- **What if the Gitea token leaks again?** -> C-32 (human-gate
|
||||
rotation before final ship); history scrub best-effort.
|
||||
|
||||
## Tier 3 — Cross-project pattern transfer
|
||||
|
||||
Single-project mode (only `orca` in `active_projects`). No
|
||||
cross-project mining.
|
||||
|
||||
## Step 3 — Merge and deduplicate
|
||||
|
||||
30 ideas, all unique by `relatedReq` (REQ-119..REQ-148). No
|
||||
duplicates. Sorted by severity then wave order (see Tier 2.6).
|
||||
|
||||
## Step 4 — Interactive validation
|
||||
|
||||
Under `autonomy.level=full` + `workflow.no_hitl=true`, all 30 ideas
|
||||
are auto-accepted. The operator pre-approved the scope in the planning
|
||||
conversation (comprehensive coverage including OS surface, bundled
|
||||
Dex, WebAuthn, Shamir). 0 skipped, 0 modified.
|
||||
|
||||
## Step 5 — Long-term document updates
|
||||
|
||||
- `REQUIREMENTS.md`: REQ-119..REQ-148 added (done).
|
||||
- `ROADMAP.md`: v0.12 milestone section added (next).
|
||||
- `ARCHITECTURE.md`: zero-trust identity model + OIDC data-flow to be
|
||||
added in P27 (docs phase) -- not in Phase 0 to avoid scope creep.
|
||||
- `PROJECT.md`: v0.12 scope summary + D-238..D-247 added (done).
|
||||
|
||||
## Step 6 — Ask-after-validation kickoff
|
||||
|
||||
The run workflow continues to PLAN -> GRILL -> ship Phase 0 ->
|
||||
execute P01..P27 -> final P28. No separate kickoff needed (the
|
||||
`--ideate` flag is consumed; ideas are already in REQUIREMENTS.md +
|
||||
ROADMAP.md).
|
||||
@@ -1,76 +0,0 @@
|
||||
# IDEATION v0.13: Production Hardening Round 2
|
||||
|
||||
**Status**: complete (2026-08-07). The `--ideate` flag was passed. Three
|
||||
deep codebase sweeps (security, reliability, feature/doc claims)
|
||||
served as the ideation engine. All accepted ideas are captured as
|
||||
REQ-149..REQ-163 in REQUIREMENTS.md and mapped to phases P01..P12 in
|
||||
ROADMAP.md.
|
||||
|
||||
## Ideation methodology
|
||||
|
||||
Standard CIAgent ideation runs three tiers:
|
||||
1. **Mechanical** (git-native pattern mining, coverage gap analysis,
|
||||
verification layer inversion, architectural drift, spec-driven)
|
||||
2. **Backend-enriched** (prioritization, novel suggestions, chaos
|
||||
engineering)
|
||||
3. **Cross-project** (multi-project registry mining — N/A, single
|
||||
project)
|
||||
|
||||
For v0.13, the ideation was driven by three parallel `explore` agents
|
||||
that performed deep codebase sweeps:
|
||||
- **Security sweep** → 28 new security findings (F26-F32 critical,
|
||||
F34-F42 high, F72-F77 medium, F95-F97 low)
|
||||
- **Reliability sweep** → 37 new reliability findings (scheduler dead
|
||||
code, concurrency hazards, SSH timeouts, IPv6, DB growth, cache
|
||||
staleness, migration safety)
|
||||
- **Feature/doc sweep** → 26 new claim-vs-reality / doc-drift findings
|
||||
(mTLS claim false, cli.md missing 25 subcommands, CHANGELOG stale,
|
||||
verify-reqs bypassed, help text stale)
|
||||
|
||||
These ~60 findings were synthesized into 15 requirements (REQ-149..
|
||||
REQ-163) and 14 phases.
|
||||
|
||||
## Accepted ideas (15 → REQ-149..REQ-163)
|
||||
|
||||
| IDEATE-ID | Category | Title | Confidence | REQ | Phase |
|
||||
|-----------|----------|-------|------------|-----|-------|
|
||||
| IDEATE-01 | security | Go toolchain bump to 1.25.12+ (24 stdlib vulns) | 0.95 | REQ-149 | P01 |
|
||||
| IDEATE-02 | security | Input validation & injection hardening (11 vectors) | 0.92 | REQ-150 | P02 |
|
||||
| IDEATE-03 | architecture | Wire scheduler into job run (R-022) | 0.90 | REQ-151 | P03 |
|
||||
| IDEATE-04 | spec | Fix jobspec parser: schedule/timeout silently dropped | 0.95 | REQ-152 | P03 |
|
||||
| IDEATE-05 | security | Wire ACL enforcement into all request paths (R-023) | 0.92 | REQ-153 | P04 |
|
||||
| IDEATE-06 | security | Seal/audit CLI + chain race + key zeroing | 0.88 | REQ-154 | P05 |
|
||||
| IDEATE-07 | security | Implement auth init-idp + auth register | 0.85 | REQ-155 | P06 |
|
||||
| IDEATE-08 | reliability | Concurrency safety (SQLite, flock, cache, atomic writes) | 0.90 | REQ-156 | P07 |
|
||||
| IDEATE-09 | reliability | Transport & SSH safety (typed errors, IPv6, timeouts) | 0.88 | REQ-157 | P08 |
|
||||
| IDEATE-10 | reliability | Migration & operational safety (job stop, DB retention, logs cap) | 0.85 | REQ-158 | P09 |
|
||||
| IDEATE-11 | quality | Observability expansion (metrics, security headers) | 0.82 | REQ-159 | P10 |
|
||||
| IDEATE-12 | quality | Doc drift round 2 (README, cli.md, CHANGELOG, help text, verify-reqs) | 0.92 | REQ-160 | P11 |
|
||||
| IDEATE-13 | feature | Implement --type linux SSH-join | 0.88 | REQ-161 | P12 |
|
||||
| IDEATE-14 | spec | UAT plan (docs/uat.md, 3-host topology, claim matrix) | 0.95 | REQ-162 | P12 |
|
||||
| IDEATE-15 | spec | UAT signoff script (uat-signoff.sh, ~35 assertions, idempotent) | 0.95 | REQ-163 | P12 |
|
||||
|
||||
## Skipped ideas (0)
|
||||
|
||||
No ideas were skipped. All ~60 findings are addressed either as
|
||||
requirements (critical/high/medium) or as accepted residual risks
|
||||
documented in RESEARCH_v0.13.md (9 low-severity items).
|
||||
|
||||
## Chaos engineering considerations
|
||||
|
||||
- **What if the scheduler picks a node that goes down mid-deploy?**
|
||||
→ R-022: SSH-push is idempotent; re-run targets the next-best node.
|
||||
- **What if ACL enforcement locks out the operator?**
|
||||
→ C-40: bootstrap ACL grants cluster-admin to the init cert's SVID.
|
||||
- **What if the seal key is lost?**
|
||||
→ C-41: Shamir 3-of-5 recovery; if quorum unavailable, cluster
|
||||
unrecoverable by design (documented, no backdoor).
|
||||
- **What if concurrent upgrades race?**
|
||||
→ REQ-156: upgrade lock file refuses concurrent invocations.
|
||||
- **What if the UAT signoff script has a false-pass assertion?**
|
||||
→ REQ-163: uat-smoke.sh runs the pure-CLI subset in CI validate;
|
||||
the full script is operator-run on bare metal.
|
||||
|
||||
## Kickoff
|
||||
|
||||
All 15 ideas are accepted and mapped to phases. Proceeding to PLAN.
|
||||
@@ -1,123 +0,0 @@
|
||||
# Ideation: Orca v0.7 — Hardening & Completion
|
||||
|
||||
Full autonomy mode: all ideas with confidence >= 0.60 are auto-accepted.
|
||||
The RESEARCH stage (commit `7c4b603`) surfaced 5 codebase gaps which are
|
||||
assessed below alongside 8 additional ideas generated by the 3-tier
|
||||
ideation process.
|
||||
|
||||
Total generated: 13 ideas (5 Tier 1 + 5 Tier 2 + 3 Tier 3) plus 5
|
||||
inherited research findings = 18 considered. 13 accepted (all >= 0.60),
|
||||
0 skipped, 0 deferred. 4 of the accepted ideas are implementation
|
||||
refinements with no new REQ; 4 map to the v0.7 REQs (REQ-053..056)
|
||||
already declared in SPECIFY; the research findings confirmed the v0.7
|
||||
scope.
|
||||
|
||||
## Tier 1: Mechanical Analysis (git + filesystem)
|
||||
|
||||
### 1.1 Git-Native Pattern Mining
|
||||
|
||||
- `git log --all --grep="lessons:"` — 1 lesson found (orch-engine P00
|
||||
config.json schema reference). No repeated lessons in orca's own
|
||||
history → no systemic process gap.
|
||||
- `git log --all --grep="escalation:"` — 0 escalations. The pipeline
|
||||
has run clean across v0.1–v0.6.
|
||||
- `git log --all --grep="compound:"` — 0 compound learnings.
|
||||
- Low-confidence decisions (confidence < 0.7): none in `---ci---`
|
||||
blocks. The lowest-confidence v0.7 decision is D-040 (pprof) at 0.85,
|
||||
above threshold.
|
||||
|
||||
### 1.2 Coverage Gap Analysis
|
||||
|
||||
| ID | Idea | Source | Confidence | Status | Maps to |
|
||||
|----|------|--------|------------|--------|---------|
|
||||
| I-401 | `orca cert` command tree is unreachable — `NewCommand` in `internal/cli/cert.go` is never AddCommand'd to `rootCmd` | research §1.1 + `grep -rn "rootCmd.AddCommand"` (cert absent) | 0.98 | Accepted | REQ-053 |
|
||||
| I-402 | `internal/store/cert_repo.go` has no test file — every other repo has one | research §1.2 + `ls internal/store/*_test.go` | 0.95 | Accepted | REQ-053 (P01 companion) |
|
||||
| I-403 | `internal/engine` coverage 8.3% — only `scheduler_test.go` exists; executor, dispatcher, peer untested | research §1.3 + `go test -cover` | 0.90 | Accepted | REQ-055 |
|
||||
| I-404 | `internal/transport` coverage 26.3% — only `idempotency_test.go`; mtls, dispatch, handshake_log untested | research §1.3 | 0.90 | Accepted | REQ-055 |
|
||||
| I-405 | `internal/audit` has no test files — Emit, EmitWithErr, LogHandshake* untested | research §1.3 + `ls internal/audit/*_test.go` | 0.88 | Accepted | REQ-055 |
|
||||
|
||||
### 1.3 Verification Layer Inversion (missing items)
|
||||
|
||||
- **Structural**: `internal/cli/cert.go` defines a command that is
|
||||
never wired in — a "documented but unreachable" component (I-401).
|
||||
- **Behavioral**: 4 packages below 50% coverage (I-403/404/405 + proxmox).
|
||||
- **Security**: no STRIDE gap — v0.7 adds no new trust boundary (pprof
|
||||
is operator-only, addr-gated; cert registration exposes existing
|
||||
security code).
|
||||
- **Quality**: no unresolved P1/P2 findings from v0.6 final review.
|
||||
|
||||
## Tier 2: Backend-Enriched Analysis
|
||||
|
||||
| ID | Idea | Source | Confidence | Status | Maps to |
|
||||
|----|------|--------|------------|--------|---------|
|
||||
| I-406 | HCL config file parser — `internal/config` package reusing `hclsimple.Decode` pattern from jobspec; D-009 promised it, never built | research §1.4 + D-009 | 0.92 | Accepted | REQ-054 |
|
||||
| I-407 | `--pprof <addr>` opt-in on `orca daemon` — I-308 deferred since v0.2; stdlib only, separate mux | research §1.5 + I-308 | 0.82 | Accepted | REQ-056 |
|
||||
| I-408 | Config precedence flag>env>file>default — table-driven test covering all 4 layers | backend-enriched (D-039) | 0.90 | Accepted | (refinement of REQ-054; no new REQ) |
|
||||
| I-409 | pprof on separate `*http.Server` + `*http.ServeMux`, never on mTLS daemon listener | backend-enriched (AD-024) | 0.90 | Accepted | (refinement of REQ-056; no new REQ) |
|
||||
| I-410 | CI coverage gate: `go test -cover ./internal/engine ./internal/transport ./internal/proxmox ./internal/audit` assert each ≥ 50% | backend-enriched (AD-025) | 0.85 | Accepted | (refinement of REQ-055; no new REQ) |
|
||||
|
||||
## Tier 3: Cross-Project Pattern Transfer
|
||||
|
||||
| ID | Idea | Source | Confidence | Status | Maps to |
|
||||
|----|------|--------|------------|--------|---------|
|
||||
| I-411 | `orca version --json` already outputs structured `{version, commit, go_version, build_time}` (I-307 accepted v0.2) — verify still works, no new REQ | cross-project (carry-forward from v0.2 I-307) | 0.80 | Accepted (verification only) | (no new REQ; confirm in P03) |
|
||||
| I-412 | `orca cert` registration via `init()` co-located in `cert.go` — matches the self-registering pattern in `daemon.go`/`audit.go` | cross-project (orca's own convention) | 0.88 | Accepted | (refinement of REQ-053; no new REQ) |
|
||||
| I-413 | No new direct dependencies in v0.7 — `net/http/pprof` (stdlib), `hashicorp/hcl/v2` (already direct) | cross-project (minimal-deps ethos) | 0.95 | Accepted | (constraint; no new REQ) |
|
||||
|
||||
## Research-stage findings (assessed)
|
||||
|
||||
| Finding | Verdict | Maps to |
|
||||
|---------|---------|---------|
|
||||
| cert command unreachable (§1.1) | **Accepted** (I-401) | REQ-053 (P01) |
|
||||
| cert_repo has no test (§1.2) | **Accepted** (I-402) | REQ-053 (P01) |
|
||||
| low coverage: engine/transport/proxmox/audit (§1.3) | **Accepted** (I-403/404/405) | REQ-055 (P03) |
|
||||
| no HCL config parser (§1.4) | **Accepted** (I-406) | REQ-054 (P02) |
|
||||
| pprof deferred since v0.2 (§1.5) | **Accepted** (I-407) | REQ-056 (P04) |
|
||||
|
||||
All 5 findings map to the v0.7 REQs declared in SPECIFY. The IDEATE
|
||||
stage confirms the scope and adds 8 implementation refinements
|
||||
(I-408..I-413) that inform the PLAN stage.
|
||||
|
||||
## Dropped ideas (confidence < 0.60)
|
||||
|
||||
None. The lowest-confidence accepted idea is I-407 (pprof) at 0.82.
|
||||
|
||||
## Accepted Ideas (auto-accepted, full autonomy)
|
||||
|
||||
13 ideas accepted (5 Tier 1 + 5 Tier 2 + 3 Tier 3). 4 map to net-new
|
||||
REQs (REQ-053..056, already declared in SPECIFY); 9 are implementation
|
||||
refinements recorded for the PLAN stage's benefit.
|
||||
|
||||
## Resulting REQ additions
|
||||
|
||||
| New REQ | Title | Phase | Source ideas |
|
||||
|---------|-------|-------|--------------|
|
||||
| REQ-053 | `orca cert` command tree registered + cert_repo tests | P01 | I-401, I-402, I-412 |
|
||||
| REQ-054 | HCL config file parsing (`internal/config`) | P02 | I-406, I-408 |
|
||||
| REQ-055 | Test coverage uplift — engine/transport/proxmox/audit ≥ 50% | P03 | I-403, I-404, I-405, I-410 |
|
||||
| REQ-056 | `--pprof <addr>` opt-in on `orca daemon` | P04 | I-407, I-409 |
|
||||
|
||||
**Total net-new REQs**: 4 (REQ-053..056). All declared in SPECIFY;
|
||||
IDEATE confirms mapping and adds implementation refinements.
|
||||
|
||||
## Deferred (recorded but not v0.7)
|
||||
|
||||
None. I-308 (pprof) is no longer deferred — it is REQ-056 in P04.
|
||||
|
||||
## Followup notes for PLAN stage
|
||||
|
||||
- **P01** is the highest-impact, lowest-effort phase: a 1-line
|
||||
`rootCmd.AddCommand` + a regression test + cert_repo_test.go. The
|
||||
smoke test should run `cert ca-init` + `cert gen` + `cert show` +
|
||||
`cert fingerprint` against a temp `ORCA_HOME` to catch any latent
|
||||
bugs in the never-exercised cert subcommands.
|
||||
- **P02** config package must be a pure function (`Load(paths) ->
|
||||
*Config`) with no package-level state. The `--config` flag on root
|
||||
command loads the file and passes the merged `*Config` down via
|
||||
cobra's `cmd.SetContext` or a struct field on the command.
|
||||
- **P03** coverage: target the interface seams (SSH dialer, peer
|
||||
client) for mocks; use `httptest.NewTLSServer` for transport. Any
|
||||
races uncovered by `-race` get fixed in P03, not deferred.
|
||||
- **P04** pprof: keep the daemon's mTLS listener untouched; start a
|
||||
second `http.Server` only when `--pprof` is non-empty. Log a WARN
|
||||
that the endpoint is unauthenticated.
|
||||
@@ -1,363 +0,0 @@
|
||||
# Ideation v0.9 — Re-architecture Foundation
|
||||
|
||||
**Project**: orca (single-project mode) | **Milestone**: v0.9/v0.10 re-architecture
|
||||
**Date**: 2026-08-05 | **Agent**: ideation agent | **Confidence threshold**: 0.60
|
||||
**Next REQ ID prior to this run**: REQ-060 (v0.8 complete)
|
||||
|
||||
## Context
|
||||
|
||||
The v0.8 codebase (REQ-001..060, all Complete) is a daemon-based, mTLS,
|
||||
HCL, single-namespace orchestration engine. The adopted PRD supersedes this
|
||||
with a CLI-only, SSH-push, step-ca, Markdown-frontmatter, multi-namespace
|
||||
stack. 9 packages are deprecation targets (~2,400 LOC of v0.8
|
||||
daemon/transport/security-ca/engine-dispatch/jobspec-hcl/config-hcl/certpaths
|
||||
code), 7 packages are adaptable, and 8 subsystems are net-new with zero
|
||||
implementation. The §23 milestone plan has 11 v0.9 phases + 17 v0.10 phases but
|
||||
under-specifies the deprecation mechanics, the SSH-push transport design, the
|
||||
lead-applier execution model, several adapter/bridge layers, and the
|
||||
migration ordering risk.
|
||||
|
||||
This ideation produced 30 ideas across three tiers, all accepted at ≥0.60
|
||||
confidence, mapped to REQ-061..REQ-090. Seven phase-reordering flags against
|
||||
the PRD §23 plan are listed at the end.
|
||||
|
||||
## Tier 1 — Mechanical (Codebase-Observable Gaps & Hygiene)
|
||||
|
||||
### I-M-001 — `orca daemon` deprecation command and build-tag removal path
|
||||
- **Tier**: mechanical
|
||||
- **Description**: The PRD deprecates `internal/daemon/` (R-001) but §23 never says *how*. `internal/cli/daemon.go` (100 LOC) registers the `daemon` cobra command and wires `daemon.NewServer` + `engine.Dispatcher`. Big-bang removal would break the v0.8→v1.0 migration path (v0.10-P14) because `orca upgrade --to-v1.0` must run against a live v0.8 cluster that still has daemons. Proposal: (1) in v0.9, `orca daemon` emits a deprecation warning and still runs (dual-write window); (2) in v1.0, `orca daemon` is repurposed to `orca daemon drain-and-stop` (stops v0.8 daemons on peers via SSH, confirms workloads survive via systemd); (3) post-v1.0, the command and `internal/daemon/` are deleted. Add `// Deprecated` Go doc comments + `slog.Warn` on every run.
|
||||
- **Rationale**: R-001 is an invariant, but the *transition* off the daemon is a mechanical gap. The v0.8 `daemon.go` is wired in `root.go` init; removing it without a transition plan breaks the §24 migration.
|
||||
- **Proposed REQ ID**: REQ-061
|
||||
- **Proposed phase placement**: v0.10-P14 (migration) — deprecation warning lands in v0.9-P0X
|
||||
- **Confidence**: 0.82
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-M-002 — Coverage follow-ups: 3 zero-test packages + `internal/cli` to 70%
|
||||
- **Tier**: mechanical
|
||||
- **Description**: v0.8 P01 (REQ-057) raised 6 packages to ≥70% and added first tests for `internal/audit`, `internal/certpaths`, `cmd/orca` at a 50% toe-hold. The v0.9 re-architecture will *replace* several of these packages, but the *adaptable* ones (`internal/store`, `internal/doctor`, `internal/cli`) must keep their 70% floor through the refactor. Once `daemon.go` is deprecated/removed (I-M-001), the exclusion reason disappears and the floor applies to the whole package. Add a coverage-gate assertion in the v0.9 P0X ship phase that `internal/cli` ≥ 70% *including* all new subcommand files (ns, txn, pve, secrets, volume, cache, backup).
|
||||
- **Rationale**: The PRD §23 does not mention coverage. The config.json policy says 70% floor for new packages, 50% minimum. The 8 net-new subsystems will need 70% floors from their first phase. Without an explicit gate, the v0.7/v0.8 "toe-hold at 50% then defer" pattern will repeat.
|
||||
- **Proposed REQ ID**: REQ-062
|
||||
- **Proposed phase placement**: v0.9-P0X (ship+audit) + each net-new package's first phase
|
||||
- **Confidence**: 0.88
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-M-003 — `known_hosts` flock concurrency gap (deferred P1 from REVIEW_v0.8 A2)
|
||||
- **Tier**: mechanical
|
||||
- **Description**: REVIEW_v0.8 flagged A2 (P1): `TOFUHostKeyCallback` capture path (`bootstrap.go:290-302`) and `ResetHostKey` (`bootstrap.go:479-523`) both do read-modify-write on `known_hosts` with no lock. The review said "Defer to v0.9." This is now load-bearing because the SSH-push transport (R-001) will do *many more* concurrent SSH operations than v0.8 did. Add a `flock`-style advisory lock (stdlib `syscall.Flock` wrapper) around the RMW in both paths. Lock file is `cluster/known_hosts.lock` (multi-namespace layout, R-002).
|
||||
- **Rationale**: The v0.8 single-operator mitigation is weaker under v0.9's parallel SSH fan-out. The PRD doesn't mention this, but the SSH-push transport makes the race more likely.
|
||||
- **Proposed REQ ID**: REQ-063
|
||||
- **Proposed phase placement**: v0.9-P0a1 (path resolver, since it establishes `cluster/` layout)
|
||||
- **Confidence**: 0.74
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-M-004 — HCL→Markdown jobspec adapter/bridge layer
|
||||
- **Tier**: mechanical
|
||||
- **Description**: R-013 says the Markdown parser is canonical; `.yaml` and `.hcl` are "accepted by parser dispatcher." But `internal/jobspec/spec.go` (69 LOC) is an HCL-only parser with a flat `Spec{Job, Tasks}` schema — no `kind:` (R-012), runtime blocks, or body preservation (R-014/R-015). The "dispatcher" implies the new parser detects file extension and dispatches. Proposal: keep `internal/jobspec/spec.go` as the legacy HCL path behind `// Deprecated`; add `internal/jobspec/markdown.go` (canonical) + `internal/jobspec/dispatch.go` (extension-based dispatcher: `.md`→Markdown, `.hcl`→legacy, `.yaml`→Markdown-with-empty-body). The dispatcher returns a unified `*WorkloadSpec` that the legacy parser populates via an adapter. Preserves `orca job run old-spec.hcl` during the migration window.
|
||||
- **Rationale**: R-013 explicitly accepts `.hcl`, so a dispatcher is required. §23 v0.9-P0b says "parser dispatcher" but doesn't specify the adapter.
|
||||
- **Proposed REQ ID**: REQ-064
|
||||
- **Proposed phase placement**: v0.9-P0b (Markdown jobspec parser)
|
||||
- **Confidence**: 0.85
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-M-005 — `orca doctor --legacy-paths` detection for v0.8 residue
|
||||
- **Tier**: mechanical
|
||||
- **Description**: The v0.8 layout is `~/.orca/{orca.db, ca.crt, ca.key, server.crt, server.key, orca_ssh_key, known_hosts, config.hcl}`. The v1.0 layout is `ORCA_HOME/{_defaults/, cluster/{ca,master.key,peers,pve,txns}, <ns>/{db,.env,.env.secrets,jobs,alloc,ns.md}, orca_cache.db}`. `orca doctor` (`internal/doctor/doctor.go`, 501 LOC, adaptable) must gain a `doctor legacy` subcommand that detects v0.8 residue: presence of `orca.db` at ORCA_HOME root, `ca.crt`/`ca.key` (internal CA, superseded by step-ca), `config.hcl` (HCL, demoted), flat `server.crt` (single-namespace), and a `namespace` column in any `*.db` (R-002 says no namespace column). Output: list of detected legacy artifacts with migration recommendations. This is the *detection* half of v0.10-P14; the *migration* half is I-C-001.
|
||||
- **Rationale**: §23 v0.10-P14 says "orca upgrade --to-v1.0, post-invariant checks" but doesn't specify the detection surface. `doctor` is the diagnostics framework and is explicitly adaptable.
|
||||
- **Proposed REQ ID**: REQ-065
|
||||
- **Proposed phase placement**: v0.10-P14c (mixed-version tolerance + no-orca enforcement)
|
||||
- **Confidence**: 0.80
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-M-006 — Legacy CA state migration to step-ca (cert import)
|
||||
- **Tier**: mechanical
|
||||
- **Description**: `internal/security/ca.go` (338 LOC) holds an internal Go CA with `ca.crt`/`ca.key` (RSA 3072, 10-year). The PRD replaces this with step-ca (R-006, D-101 reverses AD-010). The v0.10-P14 migration must handle existing deployments with an internal CA: (a) import the existing CA key into step-ca as `step ca init --deployment-type standalone --remote-management` with the existing key; (b) issue new SVIDs from step-ca and let old certs expire; (c) document that v0.8 certs are invalidated and re-bootstrap is required. The codebase audit says `ca.go`+`csr.go` are *replaced* — but the *state* (the CA key + issued server certs in `cert_repo` SQLite) may need to be preserved for audit history even if the live trust root changes. Proposal: `orca upgrade --to-v1.0 --import-ca` reads `~/.orca/ca.key`, initializes step-ca with it, and re-issues workload SVIDs. Without this, existing deployments lose their trust root with no path back.
|
||||
- **Rationale**: AD-010 is explicitly reversed by D-101, but the reversal doesn't address what happens to the existing CA material. §24 covers data migration but not CA migration.
|
||||
- **Proposed REQ ID**: REQ-066
|
||||
- **Proposed phase placement**: v0.10-P14a (data migration)
|
||||
- **Confidence**: 0.70
|
||||
- **Accept/Defer**: accept (design in v0.9-P00 so step-ca integration knows the import contract)
|
||||
|
||||
### I-M-007 — Fuzz test harness for the Markdown frontmatter parser
|
||||
- **Tier**: mechanical
|
||||
- **Description**: R-014/R-015 require byte-exact body preservation — "body of every .md config file preserved verbatim." This is a class of bug that's easy to get wrong (off-by-one on the `---` delimiter, trailing newline handling, BOM, CRLF, nested code fences containing `---`). v0.8 has no fuzz tests at all. Proposal: add a `testing.F` fuzz target in `internal/jobspec/markdown_test.go` that round-trips random frontmatter+body through `ParseMarkdown` and asserts `body == roundtripped.body` byte-exact. Also add a corpus of adversarial fixtures (CRLF, BOM, no-frontmatter, empty-frontmatter, frontmatter-with-only-separator). §23 v0.10-P08 mentions integration tests but not fuzzing.
|
||||
- **Rationale**: R-015 is a *load-bearing invariant* (body appears in inspect/history). Byte-exactness is exactly what fuzz tests are for. The v0.8 jobspec tests are golden-file only (no fuzz).
|
||||
- **Proposed REQ ID**: REQ-067
|
||||
- **Proposed phase placement**: v0.9-P0b (Markdown parser) — fuzz from day one
|
||||
- **Confidence**: 0.78
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-M-008 — Deprecation warnings on removed/repurposed CLI subcommands
|
||||
- **Tier**: mechanical
|
||||
- **Description**: The v0.8 CLI has `orca cert {ca-init,gen,show,renew,fingerprint}` (`internal/cli/cert.go`), `orca node join` with mTLS handshake semantics (`internal/cli/node.go`), `orca job run <spec.hcl>`. The PRD repurposes `node join` to SSH-bootstrap (no mTLS), deprecates `cert` (step-ca handles it), and changes `job run` to accept `.md` specs. Each removed/changed command should emit a `slog.Warn` deprecation banner with the v1.0 replacement, *except* when run under `orca upgrade`. The existing `root.go` `PersistentPreRunE` is the natural hook for a global `--no-deprecation-warnings` flag.
|
||||
- **Rationale**: Operators running v0.8 commands against v0.9/v0.10 need to know what changed. The PRD doesn't mention deprecation UX.
|
||||
- **Proposed REQ ID**: REQ-068
|
||||
- **Proposed phase placement**: v0.9-P0X (ship) + v0.10-P13 (ns subcommands, when CLI surface is finalized)
|
||||
- **Confidence**: 0.72
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-M-009 — `internal/config/config.go` HCL config demotion via adapter
|
||||
- **Tier**: mechanical
|
||||
- **Description**: `internal/config/config.go` (127 LOC) parses HCL config with keys `db_path, listen_addr, ca_path, server_cert_path, server_key_path, node_capacity`. The PRD replaces this with Markdown-frontmatter config (R-014) + per-namespace `.env`/`.env.secrets` (R-011). The `listen_addr` and `server_*_path` keys are daemon-specific (deprecated by R-001). The `root.go` `PersistentPreRunE` calls `config.Load(configPath)` on every command — must be repointed to the new Markdown config loader. Proposal: keep `internal/config/` as `legacy_config.go` with `// Deprecated`; add `internal/config/markdown.go` for the new loader; `root.go` dispatches on file extension (`.hcl`→legacy, `.md`→new). The `--config` flag semantics change: `.hcl` is read-only legacy, `.md` is canonical.
|
||||
- **Rationale**: R-014 makes Markdown canonical but `.hcl` must still parse during migration. The existing `config.Load` is called unconditionally in `root.go:42-47`.
|
||||
- **Proposed REQ ID**: REQ-069
|
||||
- **Proposed phase placement**: v0.9-P0a1 (path resolver + config demotion)
|
||||
- **Confidence**: 0.76
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-M-010 — `internal/certpaths/` replacement with multi-namespace path resolver
|
||||
- **Tier**: mechanical
|
||||
- **Description**: `internal/certpaths/certpaths.go` (64 LOC) returns flat paths: `Dir() = $ORCA_HOME`, `CACertPath() = Dir/ca.crt`, `DBPath() = Dir/orca.db`. R-002 requires multi-namespace layout: `ORCA_HOME/<ns>/db/`, `ORCA_HOME/cluster/{ca,master.key,peers,pve,txns}`, `ORCA_HOME/_defaults/`. The package is imported by `doctor`, `proxmox`, `store`, `cli` — changing it is cross-cutting. Proposal: replace `certpaths` with a new `internal/paths` package: `paths.NamespaceDir(ns)`, `paths.ClusterDir()`, `paths.CacheDB()`, `paths.MasterKey()`, `paths.NSDb(ns)`, `paths.NSEnv(ns)`, `paths.NSSecrets(ns)`. Keep `certpaths` as a thin shim that calls `paths` with the default namespace for v0.8 compat, then remove the shim post-v1.0.
|
||||
- **Rationale**: R-002 is foundational and `certpaths` is the single source of path truth. Every adaptable package (`store.Open`, `doctor`, `proxmox`) imports it. Highest-blast-radius mechanical change.
|
||||
- **Proposed REQ ID**: REQ-070
|
||||
- **Proposed phase placement**: v0.9-P0a1 (must come first)
|
||||
- **Confidence**: 0.84
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-M-011 — `internal/store/` schema: per-namespace DBs, drop ns column
|
||||
- **Tier**: mechanical
|
||||
- **Description**: R-002 says "No `namespace` column in SQLite." The v0.8 schema has 7 migrations (`0001`..`0007`) with a single `orca.db`. The v0.10 model has one DB per namespace (`<ns>/db/orca.db`) plus a CLI-side cache DB (`orca_cache.db`, R-008). The existing `store.Open(path)` takes a path arg — adaptable. But the migrations are global; they need to apply *per namespace DB*. Proposal: `store.Open` gains a namespace parameter (or caller passes `paths.NSDb(ns)`); `migrate.go` runs `0001`..`0007` (minus `0006_node_kind_os` which is v0.8-specific) plus new `0008_namespace_layout.sql`. The `cert_repo` (`0004_certs.sql`) is removed (step-ca handles certs). The audit_log table moves to the CLI-side cache DB (R-008). Existing v0.8 `orca.db` is migrated by splitting tables into per-namespace DBs during v0.10-P14.
|
||||
- **Rationale**: R-002 is explicit ("No namespace column in SQLite") but the existing schema has a single DB. §23 doesn't specify the schema split mechanics.
|
||||
- **Proposed REQ ID**: REQ-071
|
||||
- **Proposed phase placement**: v0.9-P0a1 + v0.10-P06 (alloc history, which uses cache DB)
|
||||
- **Confidence**: 0.80
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-M-012 — `internal/transport/` deletion + SSH-push package introduction
|
||||
- **Tier**: mechanical
|
||||
- **Description**: `internal/transport/` (7 files, ~1300 LOC incl tests) implements mTLS client/server, dispatch, idempotency, retry, handshake logging. R-001 + R-006 replace this with SSH-push. The *idempotency* and *retry* logic (`idempotency.go` 123 LOC, `retry.go` 151 LOC) is conceptually reusable for SSH-push (retry on SSH failure, idempotency keys for SCP'd configs). Proposal: delete `mtls.go`, `dispatch.go`, `handshake_log.go`; extract retry/idempotency patterns into a new `internal/sshpush/` package. The existing `transport.IdempotencyStore` (in-memory `sync.Map` of keys) is directly reusable. This avoids re-implementing retry semantics from scratch.
|
||||
- **Rationale**: The codebase audit marks `internal/transport/` as fully replaced, but the retry/idempotency *patterns* are transport-agnostic. §23 doesn't call this out.
|
||||
- **Proposed REQ ID**: REQ-072
|
||||
- **Proposed phase placement**: v0.9-P00 (deprecation sweep) — delete in v0.10-P14
|
||||
- **Confidence**: 0.68
|
||||
- **Accept/Defer**: accept (defer deletion to v0.10-P14 to keep dual-write window open)
|
||||
|
||||
## Tier 2 — Backend-Enriched (Structural / Architectural)
|
||||
|
||||
### I-B-001 — SSH-push transport layer design
|
||||
- **Tier**: backend-enriched
|
||||
- **Description**: The PRD replaces `internal/transport/` (mTLS HTTP) with SSH-push but §23 never specifies the transport's internal design. Key decisions: (1) **Connection pooling**: reuse `*ssh.Client` per peer across multiple SCP/exec operations within a single CLI invocation. (2) **Idempotency**: SCP of a config file is idempotent if content hash matches — use content-addressed filename (`/run/orca/<hash>.unit`) and skip if present. (3) **Retry**: reuse v0.8's exponential backoff (100ms start, ×2, cap 5s, max 5 attempts) applied to SSH dial/exec failures. (4) **Timeout**: per-operation `context.WithTimeout` (default 30s SCP, 10s exec). (5) **Fan-out**: `errgroup.Group` with bounded concurrency for N-peer ops (default 8). (6) **known_hosts**: reuse `proxmox.TOFUHostKeyCallback` for all peers, not just Proxmox.
|
||||
- **Rationale**: Load-bearing replacement for the entire v0.8 transport layer. §23 assumes it but never designs it. Without connection pooling, every CLI operation re-dials SSH.
|
||||
- **Proposed REQ ID**: REQ-073
|
||||
- **Proposed phase placement**: v0.9-P01 (first phase needing SSH-push) — design in v0.9-P0a1
|
||||
- **Confidence**: 0.86
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-B-002 — Emitter template system (Layer 4)
|
||||
- **Tier**: backend-enriched
|
||||
- **Description**: The PRD §5 describes a 4-layer architecture where Layer 4 is "emitters" that render systemd units, Traefik dynamic config, Syncthing config, etc. from the workload spec. §23 never specifies the emitter interface. Proposal: an `internal/emitter/` package with `Emitter` interface: `Render(spec *WorkloadSpec, node *Node) ([]File, error)` where `File{Path, Content, Mode}`. Implementations: `systemdEmitter`, `traefikEmitter`, `syncthingEmitter`, `socketEmitter`. The SSH-push transport SCPs the `[]File` atomically (write-to-tmp + rename). Emitters registered per workload kind + runtime.
|
||||
- **Rationale**: The emitter layer is the bridge between the declarative spec and the server-side files. Without a defined interface, each phase (P02 service, P04 hooks, P08 sockets, P09 storage) will invent its own rendering.
|
||||
- **Proposed REQ ID**: REQ-074
|
||||
- **Proposed phase placement**: v0.9-P0c (schemas + emitter interface)
|
||||
- **Confidence**: 0.82
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-B-003 — Lead applier execution model: pure bash + systemd timer vs CLI-invoked
|
||||
- **Tier**: backend-enriched
|
||||
- **Description**: R-001 says "no orca binary on servers." R-010 says the lead applies desired-state transactionally. Unresolved: does the lead run `orca-pull.sh` (pure bash that SCPs a desired-state bundle and applies it via `systemctl daemon-reload` + `systemctl restart`) or does the operator's CLI SSH into the lead and runs `orca apply` remotely (which would put an orca binary on the lead, violating R-001)? The PRD's intent is the former: the lead is bare Linux with systemd timers + bash. Proposal: (1) the CLI renders a *transaction bundle* (tarball of desired-state files + `apply.sh` + `verify.sh`) on the operator host; (2) SCPs it to the lead's `/run/orca/txns/<txn-id>/`; (3) the lead's systemd timer runs `/run/orca/txns/<txn-id>/apply.sh` which idempotently applies and runs verify; (4) the CLI polls the lead for txn status via SSH (`cat /run/orca/txns/<txn-id>/status.json`). The bash scripts are generated by the CLI's emitter (I-B-002), not hand-written per cluster.
|
||||
- **Rationale**: The most ambiguous load-bearing design decision in the PRD. R-001 + R-010 together imply the lead runs no orca binary, but the lead must apply transactions. §23 doesn't resolve this. Getting it wrong means either violating R-001 or having no transactional apply.
|
||||
- **Proposed REQ ID**: REQ-075
|
||||
- **Proposed phase placement**: v0.10-P10 (transactional plane) — bundle format designed in v0.9-P00
|
||||
- **Confidence**: 0.78
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-B-004 — step-ca integration: provisioning, CA bootstrap, cert signing API, SVID minting
|
||||
- **Tier**: backend-enriched
|
||||
- **Description**: D-101 reverses AD-010 (which rejected step-ca as "too heavyweight"). §23 mentions step-ca in R-006 but never specifies the integration. Key surfaces: (1) **Provisioning**: `orca init` (adapted from v0.8's `internal/cli/init.go`) runs `step ca init` on the lead, stores root + intermediate in `cluster/ca/`. (2) **CA bootstrap**: CLI SSHs to the lead, installs step-ca via apt, runs `step ca init`, stores `step-ca.json` config. (3) **Cert signing API**: workloads request SVIDs via `step ca token` (JWE provisioner token minted by CLI) → `step ca certificate`. The CLI mints the token because it holds the provisioner password (in `cluster/master.key`-derived form). (4) **SVID minting**: each workload gets a SPIFFE ID (`spiffe://orca/<ns>/<workload>/<instance>`) encoded as a SAN in the step-ca-issued cert. The v0.8 `internal/security/ca.go` is deleted; a new `internal/stepca/` package wraps the `step` CLI via SSH (no Go step-ca client library — keep zero-new-dep posture if possible, or add `github.com/smallstep/cli` as a dep).
|
||||
- **Rationale**: step-ca is a new external dependency with its own config format, provisioner model, and CLI. §23 assumes it but never designs the integration. security-engineer persona must be reactivated.
|
||||
- **Proposed REQ ID**: REQ-076
|
||||
- **Proposed phase placement**: v0.9-P07 (runtime block — runtimes need SVIDs) + v0.10-P02 (ACL — SPIFFE identities)
|
||||
- **Confidence**: 0.74
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-B-005 — Traefik dynamic config generation and atomic reload
|
||||
- **Tier**: backend-enriched
|
||||
- **Description**: R-006 makes Traefik load-bearing (mTLS termination + health checks). §23 puts service blocks + Traefik health checks in v0.9-P02. Design: the CLI's Traefik emitter (I-B-002) renders a dynamic config file (`/etc/traefik/dynamic/orca-<ns>-<svc>.yaml`) with backends (the socket paths from R-007), health checks, and mTLS config pointing at step-ca's root. Atomic reload: Traefik watches the dynamic dir with `fsnotify` — writing the file atomically (tmp+rename) triggers a reload. Drain (v0.10-P05) works by writing a config with the backend's `weight=0` or removing it, triggering Traefik to stop routing. The v0.8 codebase has no Traefik integration at all. **Gated by grill C-10** (Traefik config atomicity protocol: tmpfile+fsync+rename + malformed-config hold-last-good verified).
|
||||
- **Rationale**: Traefik is net-new and load-bearing. §23 mentions it in R-006/P02/P05 but never specifies config generation or reload mechanism.
|
||||
- **Proposed REQ ID**: REQ-077
|
||||
- **Proposed phase placement**: v0.9-P02 (Service block + checks)
|
||||
- **Confidence**: 0.80
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-B-006 — Runtime abstraction interface (5 backends: wasm/podman/process/pve-vm/pve-ct)
|
||||
- **Tier**: backend-enriched
|
||||
- **Description**: v0.8's `internal/engine/executor.go` (211 LOC) is `os/exec` only. R-004/R-007 require 5 runtime backends. §23 puts this in v0.9-P07. Proposal: a `Runtime` interface in `internal/runtime/`: `Prepare(ctx, spec, node) (*Alloc, error)`, `Start(ctx, alloc) (pid/unit, error)`, `Stop(ctx, alloc) error`, `Status(ctx, alloc) (State, error)`. Implementations: `processRuntime` (wraps existing `executor.go` — directly reusable), `wasmRuntime` (wasmtime via CLI SSH exec), `podmanRuntime` (`podman run` via SSH), `pveVMRuntime` (`qm create`/`qm start` via v0.8 `proxmox` SSH session), `pveCTRuntime` (`pct create`/`pct start`). Each registered in a `runtimeRegistry` keyed by the `runtime:` frontmatter value. R-004 (migration with runtime change) means the `Alloc` carries a `runtime` field that can change on migration — `Prepare` re-runs with the new runtime.
|
||||
- **Rationale**: The 5 backends are the largest net-new implementation surface. §23 lists them as one phase (P07) but under-specifies the interface contract. The existing `executor.go` is a good starting point for the `processRuntime` adapter.
|
||||
- **Proposed REQ ID**: REQ-078
|
||||
- **Proposed phase placement**: v0.9-P07a/P07b/P07c (split per grill PC-10)
|
||||
- **Confidence**: 0.82
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-B-007 — Transaction bundle format and atomicity across N peers
|
||||
- **Tier**: backend-enriched
|
||||
- **Description**: R-010 requires transactional control-plane updates. §23 puts this in v0.10-P10. Design: a *transaction bundle* is a tarball containing: (1) `desired-state.json` (full desired state for affected namespaces), (2) `apply.sh` (idempotent apply script), (3) `verify.sh` (post-apply invariants), (4) `rollback.sh` (revert to previous state), (5) `manifest.sig` (signature with `cluster/master.key`). Atomicity across N peers: the CLI uploads the bundle to the lead; the lead applies to itself first, then fans out to peers via SSH. If any peer fails verify, the lead runs `rollback.sh` on all peers that applied. The bundle is content-addressed (`<txn-id> = sha256(desired-state.json)`) and stored in `cluster/txns/<txn-id>/`. Drift detection (R-010) compares the last applied bundle's desired-state against the live state (polled via SSH `systemctl show` + file checksums). **Gated by grill C-09** (orca-pull.sh failure contract: idempotent re-run, bounded retry, deterministic state, structured syslog).
|
||||
- **Rationale**: Multi-peer atomicity is the hardest part of R-010. §23 says "ArgoCD-style" but ArgoCD is Kubernetes-native; the SSH-push model needs a custom bundle format.
|
||||
- **Proposed REQ ID**: REQ-079
|
||||
- **Proposed phase placement**: v0.10-P10 (transactional plane) — designed in v0.9-P00
|
||||
- **Confidence**: 0.76
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-B-008 — Master key management and HKDF-SHA256 per-line .env.secrets encryption
|
||||
- **Tier**: backend-enriched
|
||||
- **Description**: R-011 specifies `.env.secrets` with AES-256-GCM, per-line nonce, master key at `cluster/master.key`. §23 puts this in v0.10-P03. Design: (1) `cluster/master.key` is a 32-byte random key generated by `orca init` (extend v0.8 `internal/security/ca.go`'s `WriteAtomic` pattern for the file write). (2) Each line of `.env.secrets` is `base64(nonce || ciphertext || tag)` where `nonce = random(12 bytes)` and `ciphertext = AES-256-GCM(plaintext, key=master.key, nonce, aad=line-number)`. (3) The AAD is the 1-indexed line number to prevent line-swap attacks. (4) Decryption reads the master key, iterates lines, decrypts with AAD. (5) `orca secrets set <ns> <key> <value>` appends an encrypted line; `orca secrets get <ns> <key>` decrypts and prints (redacted by default, `--reveal` to show). (6) The v0.8 `internal/security/redact.go` (103 LOC) is directly reusable for redaction. HKDF-SHA256 derives per-namespace sub-keys from the master key (`HKDF-SHA256(master, info=<ns>)`) so compromising one namespace's key doesn't compromise others — but the master key is the root of trust. **Gated by grill C-19** (threat model for master.key passphrase-less posture).
|
||||
- **Rationale**: R-011 is precise about the crypto but §23 doesn't specify key derivation, AAD, or CLI surface. The existing `redact.go` and `WriteAtomic` are reusable.
|
||||
- **Proposed REQ ID**: REQ-080
|
||||
- **Proposed phase placement**: v0.10-P03 (secrets subsystem)
|
||||
- **Confidence**: 0.84
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-B-009 — Syncthing config rendering and folder-ID content-addressing
|
||||
- **Tier**: backend-enriched
|
||||
- **Description**: R-005 requires storage replication via per-namespace Syncthing. §23 puts this in v0.9-P09. Design: (1) each namespace gets a Syncthing folder `orca-<ns>` with a content-addressed folder ID (`sha256(ns + master-key-fingerprint)`). (2) The CLI renders `config.xml` for each peer's Syncthing instance, including the folder, devices (all peers in the namespace), and the path (`<ns>/alloc/<alloc-id>/`). (3) Syncthing runs as a systemd unit (emitted by the systemd emitter, I-B-002). (4) The CLI discovers peers via `cluster/peers/` and adds their Syncthing device IDs (each peer's Syncthing generates its own device key on first run, reported back via SSH). (5) R-005 says "a Service's count replicas share one runtime block" — the Syncthing folder is shared across the Service's alloc instances so all replicas see the same data. Migration (R-004) works because the new node joins the Syncthing folder and syncs before the workload starts. **Gated by grill C-02** (Syncthing feasibility spike) and **C-14** (deterministic conflict-resolution policy + forced-divergence integration test).
|
||||
- **Rationale**: Syncthing is net-new. §23 lists it in P09 but doesn't specify config rendering, folder-ID scheme, or device discovery.
|
||||
- **Proposed REQ ID**: REQ-081
|
||||
- **Proposed phase placement**: v0.9-P09 (storage replication) — spike in v0.9-P00
|
||||
- **Confidence**: 0.72
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-B-010 — Namespace inheritance resolver algorithm
|
||||
- **Tier**: backend-enriched
|
||||
- **Description**: v0.9-P0a2 requires a "parent walker, cycle detection" for namespace inheritance. Each `ns.md` has a `parent:` field in frontmatter. The resolver walks up the parent chain, merging inherited values (constraints, env, runtime defaults). Cycle detection: DFS with a visited set; if a namespace is revisited, return a cycle error. The resolver returns a flattened `ResolvedNamespace` struct. The `_defaults/` namespace is the implicit root (always exists, has no parent). Inheritance semantics: child overrides parent for scalar fields; arrays (e.g., constraints) are unioned (child adds to parent, not replaces). The resolver is pure (no I/O) — it takes a map of `nsName → *NSConfig` and returns `nsName → *ResolvedNS`. This makes it trivially testable.
|
||||
- **Rationale**: §23 mentions "parent walker, cycle detection" but not the merge semantics (override vs union) or the resolver's purity for testing. Getting merge semantics wrong breaks constraint inheritance (P05).
|
||||
- **Proposed REQ ID**: REQ-082
|
||||
- **Proposed phase placement**: v0.9-P0a2 (namespace CRUD + inheritance)
|
||||
- **Confidence**: 0.86
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-B-011 — Bin-packing scheduler redesign (CLI-side, runtime-compatibility scoring)
|
||||
- **Tier**: backend-enriched
|
||||
- **Description**: v0.8's `internal/engine/scheduler.go` (117 LOC) does best-fit bin-packing by CPU+memory. The v0.9 scheduler must: (1) run CLI-side (not on a daemon), (2) score nodes by runtime compatibility (a wasm workload can only go to a node with wasmtime installed; a pve-vm workload can only go to Proxmox nodes), (3) respect constraints/affinity (CEL over node attributes, P05), (4) handle the 3 kinds differently (Job = one-shot, Service = count replicas spread across nodes, DaemonSet = one per node). The existing `scheduler.go` is a good skeleton but the scoring function changes entirely. Proposal: `Score(node, workload) (score int, fits bool)` where `fits` checks runtime compatibility + constraints, and `score` is the bin-packing score (most free capacity = highest score). For Services, the scheduler picks `count` distinct nodes (anti-affinity by default). For DaemonSets, it picks all matching nodes.
|
||||
- **Rationale**: The scheduler moves from daemon-side to CLI-side (R-001) and gains runtime-awareness. §23 scatters this across P05 (constraints), P06 (task groups), P07 (runtime), P10 (migration) but never designs the scheduler itself.
|
||||
- **Proposed REQ ID**: REQ-083
|
||||
- **Proposed phase placement**: v0.9-P05 (constraints & affinity — scheduler needs constraints to be meaningful) — skeleton in P0c
|
||||
- **Confidence**: 0.80
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-B-012 — `orca job lint` category-driven lint engine design
|
||||
- **Tier**: backend-enriched
|
||||
- **Description**: v0.10-P11 requires `orca job lint` with `--explain`. Design: a `Linter` that takes a `*WorkloadSpec` and runs a series of `Rule` checks, each returning a `Finding{Category, Severity, Message, Explanation}`. Categories: `schema` (missing required fields), `runtime` (incompatible runtime+constraint), `security` (missing SVID, plaintext secret in env), `migration` (missing storage replication for a migratable service), `best-practice` (no health check on a Service). `--explain` prints the rationale for each finding. Rules are registered in a `ruleRegistry` and individually testable. The linter is pure (no I/O) — it checks the spec against static rules, not live cluster state (that's `orca job verify`, P12).
|
||||
- **Rationale**: §23 puts this in P11 but only says "category-driven." The rule interface and category taxonomy are unspecified.
|
||||
- **Proposed REQ ID**: REQ-084
|
||||
- **Proposed phase placement**: v0.10-P11 (orca job lint)
|
||||
- **Confidence**: 0.78
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
## Tier 3 — Cross-Cutting (Risk & Multi-Phase)
|
||||
|
||||
### I-C-001 — v0.8→v1.0 migration ordering: daemon deprecation vs. new model rollout
|
||||
- **Tier**: cross-cutting
|
||||
- **Description**: The PRD §24 covers *data* migration but not *binary/daemon* deprecation ordering. The risk: v0.9 builds the new Markdown+kinds+runtime+SSH-push model, but v0.8 daemons are still running on peers. If v0.9 ships the new `orca job run` (Markdown) while the old daemon is still the execution engine, there's a split-brain: new specs can't run on the old daemon. Ordering proposal: (1) v0.9 ships the new parser + kinds + runtime + SSH-push *alongside* the old daemon (dual-write window); (2) `orca job run` in v0.9 uses the new SSH-push path if the spec is `.md` and the old daemon path if `.hcl`; (3) v0.10-P05 (drain) stops the old daemons; (4) v0.10-P14 (migration) converts remaining `.hcl` specs to `.md` and removes the daemon. The dual-write window means v0.9 is *not* a clean break — it's a compatibility milestone. This must be explicit in the plan or the v0.9 phases will assume the daemon is gone.
|
||||
- **Rationale**: Single largest risk in the re-architecture. §23 implicitly assumes v0.9 builds the new model in isolation, but existing deployments have running daemons. Getting the ordering wrong means either (a) v0.9 can't be tested against real deployments, or (b) workloads are orphaned when the daemon is removed.
|
||||
- **Proposed REQ ID**: REQ-085
|
||||
- **Proposed phase placement**: spans v0.9-P00 through v0.10-P14 — the *ordering decision* must be made in v0.9-P00
|
||||
- **Confidence**: 0.88
|
||||
- **Accept/Defer**: accept (most important idea in this report)
|
||||
|
||||
### I-C-002 — "No orca on server" enforcement (doctor post-migration invariant check)
|
||||
- **Tier**: cross-cutting
|
||||
- **Description**: R-001 is an invariant: "no orca Go binary on any server." §23 v0.10-P14 says "post-invariant checks" but doesn't specify them. `orca doctor` must gain a `doctor no-orca-on-server` check that SSHs to each peer and verifies: (1) no `orca` binary in PATH (`ssh peer which orca` returns nothing), (2) no `orca` systemd service (`ssh peer systemctl list-units 'orca*'` returns empty), (3) no `orca` process (`ssh peer pgrep -x orca` returns empty), (4) no `/etc/orca/` directory. This check must run *after* v0.10-P05 (drain) and *before* v0.10-P16 (ship). The v0.8 `internal/proxmox/bootstrap.go` already has the SSH session infrastructure (`sessionRunner` seam) — directly reusable for the doctor check.
|
||||
- **Rationale**: R-001 is a hard invariant but §23 doesn't enforce it post-migration. Without this check, a failed migration could leave orphaned daemons that cause split-brain.
|
||||
- **Proposed REQ ID**: REQ-086
|
||||
- **Proposed phase placement**: v0.10-P14c (mixed-version tolerance)
|
||||
- **Confidence**: 0.82
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-C-003 — Test infrastructure: hermetic 3-linux + 1-proxmox cluster pipeline
|
||||
- **Tier**: cross-cutting
|
||||
- **Description**: §23 v0.10-P08 requires "hermetic CoreCI integration pipeline." The PRD §26.E mentions 3 linux + 1 proxmox. This is net-new test infra with zero current implementation. Design: (1) a `test/integration/` directory with a `docker-compose.yml` or `vagrant` setup that creates 4 containers/VMs (3 linux + 1 proxmox-simulated); (2) a Go test harness that SSHes to each, runs the CLI, and asserts end-to-end workflows (namespace create → workload submit → migrate → drain); (3) the proxmox node is simulated via a mock `pct`/`qm` script (the v0.8 `proxmox` package already has a `sessionRunner` seam for testability — extend it). The integration tests run in CoreCI on every milestone merge. The v0.8 e2e tests (`bootstrapE2ESetup` in `bootstrap_test.go`) use an in-process SSH server — this is the foundation but needs to scale to 4 nodes.
|
||||
- **Rationale**: §23 assumes the infra exists but doesn't design it. devops-engineer persona should be reactivated. Without hermetic infra, the integration tests can't run in CI.
|
||||
- **Proposed REQ ID**: REQ-087
|
||||
- **Proposed phase placement**: v0.10-P08 (integration tests) — harness bootstrapped in v0.9-P00
|
||||
- **Confidence**: 0.80
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-C-004 — Security-engineer + network-engineer persona reactivation for new attack surfaces
|
||||
- **Tier**: cross-cutting
|
||||
- **Description**: The config.json has `security-engineer` and `network-engineer` dormant. The re-architecture introduces step-ca (PKI), Traefik (edge proxy), Syncthing (P2P file sync), wasmtime (sandbox), podman (container runtime) — all new attack surfaces. AD-010 (step-ca rejection) is reversed. The v0.8 security posture (internal CA, mTLS daemon-to-daemon) is replaced by (step-ca, SSH-push, Traefik mTLS). The security-engineer persona must be reactivated to review: (1) step-ca provisioner model (the CLI holds the provisioner password — is that in `cluster/master.key` or a separate secret?), (2) SSH-push blast radius (compromised CLI key = full cluster), (3) Traefik as the new edge (DoS, config injection), (4) `.env.secrets` crypto (I-B-008). The network-engineer persona must review: (1) socket-based service exposure (R-007), (2) Syncthing P2P ports, (3) Traefik routing. §23 doesn't mention persona reactivation.
|
||||
- **Rationale**: config.json explicitly notes the re-architecture "should reactivate security-engineer and network-engineer." Cross-cutting review concern, not a single phase.
|
||||
- **Proposed REQ ID**: REQ-088
|
||||
- **Proposed phase placement**: spans v0.9 through v0.10 — reactivation in v0.9-P00, review at v0.10-P15.5 (threat model) and v0.10-P16 (final audit)
|
||||
- **Confidence**: 0.84
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-C-005 — Documentation rewrite: ARCHITECTURE.md, PROJECT.md, README, AD-010 supersession
|
||||
- **Tier**: cross-cutting
|
||||
- **Description**: All three docs describe the OLD architecture. `ARCHITECTURE.md` (640 lines) describes the daemon layer, mTLS transport, internal CA, HCL jobspec — all deprecated. `PROJECT.md` (30k chars) has D-001..D-010 decisions, several now superseded. `README.md` has the v0.8 quickstart. AD-010 (step-ca rejection) must be explicitly superseded by D-101 with a dated rationale reversal. The anti-patterns section in `ARCHITECTURE.md:471-484` lists "No external PKI" — now reversed. Proposal: (1) in v0.9-P00, add a "v0.9 Architecture (Supersedes v0.8)" section to ARCHITECTURE.md with the new 4-layer model; (2) mark the old sections as "v0.8 (deprecated)" with banners; (3) add a "Superseded Decisions" table (AD-009, AD-010 reversed by D-101; AD-007 HCL demoted by R-013); (4) in v0.10-P15, rewrite README quickstart for the new `curl | sh` + `orca init` + `orca ns create` flow.
|
||||
- **Rationale**: The docs are the first thing new contributors read. Leaving v0.8 docs as canonical during v0.9 development causes confusion. §23 mentions README in P15 but not ARCHITECTURE.md/PROJECT.md.
|
||||
- **Proposed REQ ID**: REQ-089
|
||||
- **Proposed phase placement**: v0.9-P00 (banners + supersession table) + v0.10-P15 (README quickstart) + v0.10-P16 (final review)
|
||||
- **Confidence**: 0.82
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-C-006 — Dual-write window: can v0.9 ship new parser while old daemon runs?
|
||||
- **Tier**: cross-cutting
|
||||
- **Description**: Focused version of I-C-001. The specific question: in v0.9, when the new Markdown parser + kinds + SSH-push are shipped, can they coexist with v0.8 daemons still running on peers? The answer depends on whether `orca job run <spec.md>` uses the new SSH-push path (bypassing the daemon entirely) or routes through the old daemon. If it bypasses, the daemon is irrelevant for new specs but still serves old `.hcl` specs. If it routes through, the daemon can't handle `.md` specs. Proposal: v0.9 `orca job run` dispatches on extension (`.md`→SSH-push new path, `.hcl`→old daemon path) via the parser dispatcher (I-M-004). This is a *dual-write window* where both paths coexist. The daemon is not removed until v0.10-P05 (drain). The risk: if a `.md` workload and a `.hcl` workload target the same node, the SSH-push path writes systemd units directly while the daemon also manages units — they can conflict. Mitigation: the SSH-push path writes to a separate systemd unit namespace (`orca-v1-<alloc>.service`) while the daemon uses `orca-<job>.service`. No unit name overlap = no conflict.
|
||||
- **Rationale**: Operational feasibility question for v0.9. §23 doesn't address it. If the answer is "no dual-write, daemon must be removed first," then v0.9 can't be tested incrementally and must ship as a big-bang — much higher risk.
|
||||
- **Proposed REQ ID**: REQ-090
|
||||
- **Proposed phase placement**: v0.9-P00 (decision before any v0.9 execution phase)
|
||||
- **Confidence**: 0.86
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
## Summary Table
|
||||
|
||||
| ID | Tier | Title | REQ | Phase | Conf | Accept |
|
||||
|----|------|-------|-----|-------|------|--------|
|
||||
| I-M-001 | M | `orca daemon` deprecation path | REQ-061 | v0.10-P14 (warn v0.9-P0X) | 0.82 | accept |
|
||||
| I-M-002 | M | Coverage follow-ups to 70% | REQ-062 | v0.9-P0X + each new pkg | 0.88 | accept |
|
||||
| I-M-003 | M | known_hosts flock concurrency | REQ-063 | v0.9-P0a1 | 0.74 | accept |
|
||||
| I-M-004 | M | HCL→Markdown jobspec adapter | REQ-064 | v0.9-P0b | 0.85 | accept |
|
||||
| I-M-005 | M | `doctor --legacy-paths` detection | REQ-065 | v0.10-P14c | 0.80 | accept |
|
||||
| I-M-006 | M | Legacy CA state migration to step-ca | REQ-066 | v0.10-P14a | 0.70 | accept |
|
||||
| I-M-007 | M | Fuzz harness for Markdown parser | REQ-067 | v0.9-P0b | 0.78 | accept |
|
||||
| I-M-008 | M | Deprecation warnings on CLI subcommands | REQ-068 | v0.9-P0X + v0.10-P13 | 0.72 | accept |
|
||||
| I-M-009 | M | HCL config demotion via adapter | REQ-069 | v0.9-P0a1 | 0.76 | accept |
|
||||
| I-M-010 | M | certpaths → multi-namespace path resolver | REQ-070 | v0.9-P0a1 | 0.84 | accept |
|
||||
| I-M-011 | M | store schema: per-namespace DBs | REQ-071 | v0.9-P0a1 + v0.10-P06 | 0.80 | accept |
|
||||
| I-M-012 | M | transport deletion + SSH-push package | REQ-072 | v0.9-P00 (delete v0.10-P14) | 0.68 | accept |
|
||||
| I-B-001 | B | SSH-push transport layer design | REQ-073 | v0.9-P01 | 0.86 | accept |
|
||||
| I-B-002 | B | Emitter template system (Layer 4) | REQ-074 | v0.9-P0c | 0.82 | accept |
|
||||
| I-B-003 | B | Lead applier execution model | REQ-075 | v0.10-P10 (design v0.9-P00) | 0.78 | accept |
|
||||
| I-B-004 | B | step-ca integration | REQ-076 | v0.9-P07 + v0.10-P02 | 0.74 | accept |
|
||||
| I-B-005 | B | Traefik dynamic config + atomic reload | REQ-077 | v0.9-P02 | 0.80 | accept |
|
||||
| I-B-006 | B | Runtime abstraction (5 backends) | REQ-078 | v0.9-P07a/b/c | 0.82 | accept |
|
||||
| I-B-007 | B | Transaction bundle + N-peer atomicity | REQ-079 | v0.10-P10 (design v0.9-P00) | 0.76 | accept |
|
||||
| I-B-008 | B | Master key + HKDF per-line encryption | REQ-080 | v0.10-P03 | 0.84 | accept |
|
||||
| I-B-009 | B | Syncthing config + folder-ID | REQ-081 | v0.9-P09 | 0.72 | accept |
|
||||
| I-B-010 | B | Namespace inheritance resolver | REQ-082 | v0.9-P0a2 | 0.86 | accept |
|
||||
| I-B-011 | B | CLI-side scheduler redesign | REQ-083 | v0.9-P05 (skeleton P0c) | 0.80 | accept |
|
||||
| I-B-012 | B | `orca job lint` category-driven engine | REQ-084 | v0.10-P11 | 0.78 | accept |
|
||||
| I-C-001 | C | v0.8→v1.0 migration ordering | REQ-085 | spans v0.9-P00→v0.10-P14 | 0.88 | accept |
|
||||
| I-C-002 | C | "No orca on server" enforcement | REQ-086 | v0.10-P14c | 0.82 | accept |
|
||||
| I-C-003 | C | Hermetic test infra (3 linux + 1 pve) | REQ-087 | v0.10-P08 (bootstrap v0.9-P00) | 0.80 | accept |
|
||||
| I-C-004 | C | security/network persona reactivation | REQ-088 | spans v0.9→v0.10-P16 | 0.84 | accept |
|
||||
| I-C-005 | C | Docs rewrite + AD-010 supersession | REQ-089 | v0.9-P00 + v0.10-P15/P16 | 0.82 | accept |
|
||||
| I-C-006 | C | Dual-write window decision | REQ-090 | v0.9-P00 | 0.86 | accept |
|
||||
|
||||
## Phase Reordering / Addition Flags (against PRD §23)
|
||||
|
||||
1. **I-C-001 / I-C-006 (dual-write + migration ordering)** — require a decision in v0.9-P00 (before any execution phase). **Recommendation: add v0.9-P00 deprecation/migration-ordering pre-phase.** Most important structural addition.
|
||||
2. **I-M-010 / I-M-011 / I-M-009 / I-M-003** — all land in v0.9-P0a. P0a may be overloaded. **Recommendation: split P0a into P0a1 (path/layout resolver + config demotion) and P0a2 (namespace CRUD + inheritance).** Path resolver is prerequisite for everything; highest blast radius.
|
||||
3. **I-B-001 (SSH-push transport)** — §23 v0.9-P01 needs SSH-push. The design is a prerequisite. **Recommendation: SSH-push design in P0a1, not deferred to P01.**
|
||||
4. **I-B-002 (emitter template system)** — should be designed *with* the schemas (P0c). **Recommendation: expand P0c to "schemas + emitter interface."**
|
||||
5. **I-B-003 (lead applier model)** — bundle format + lead applier model must be designed *in v0.9* so the emitter can produce bundle-compatible output. **Recommendation: design spike in v0.9-P00.**
|
||||
6. **I-C-003 (test infra)** — hermetic cluster harness should be bootstrapped in v0.9-P00 so every v0.9 phase can run integration tests. **Recommendation: bootstrap in v0.9-P00, expand in v0.10-P08.**
|
||||
7. **I-C-004 / I-C-005 (persona reactivation + docs)** — span the whole milestone. **Recommendation: fold persona reviews into v0.9-P00 and v0.10-P16; fold doc banners into v0.9-P00.**
|
||||
|
||||
## Cross-Reference Against Existing Decisions
|
||||
|
||||
- **AD-009 (Internal CA, no external PKI)** — Superseded by D-101 (step-ca). I-B-004, I-M-006 implement the reversal.
|
||||
- **AD-010 (Roll-our-own CA)** — Superseded by D-101. I-C-005 documents the supersession. No re-litigation — the PRD has decided; the override justification records the evidence basis.
|
||||
- **AD-007 (HCL for job specs)** — Demoted by R-013 (Markdown canonical, HCL accepted). I-M-004 implements the adapter. Not a full reversal — HCL still parses.
|
||||
- **AD-001 (Single binary with subcommands)** — Still holds. The CLI is the single binary; no orca on servers (R-001) refines this.
|
||||
- **AD-015 (Best-fit bin-packing)** — Extended, not reversed. I-B-011 adds runtime-compatibility scoring.
|
||||
- **D-035 (TOFU host-key)** — Still holds for non-Proxmox peers. I-M-003 hardens the concurrency. I-B-001 reuses `TOFUHostKeyCallback`.
|
||||
- **D-046 (key-reset is local-only)** — Still holds. I-M-003 adds the lock.
|
||||
- **D-047 (tiered coverage floor)** — Extended by I-M-002 to cover new packages.
|
||||
|
||||
No accepted idea re-litigates a settled decision. All reversals (AD-009, AD-010, SPIFFE, no-container, no-multi-tenancy, HCL-canonical, daemon-on-every-node) are explicitly mandated by the PRD and justified by the recorded override justification.
|
||||
|
||||
## Final Notes
|
||||
|
||||
- **Total ideas**: 30 (12 mechanical, 12 backend-enriched, 6 cross-cutting).
|
||||
- **Highest-confidence, highest-impact**: I-C-001 (migration ordering, 0.88) and I-C-006 (dual-write window, 0.86) — these shape the entire v0.9 execution strategy.
|
||||
- **Highest-blast-radius mechanical**: I-M-010 (path resolver, 0.84) — touches every adaptable package.
|
||||
- **Most under-specified by PRD**: I-B-003 (lead applier execution model, 0.78) — R-001 + R-010 create a tension the PRD doesn't resolve.
|
||||
@@ -1,34 +0,0 @@
|
||||
# P23 Dual-Write Closure — Decision (v0.12)
|
||||
|
||||
**Status**: DEFERRED to v1.x. The full deletion of the legacy CA
|
||||
(`internal/security/ca.go`), mTLS transport (`internal/transport/mtls.go`),
|
||||
and daemon plaintext mode is too large a refactor for v0.12 without
|
||||
risking build stability. The legacy code is already marked Deprecated;
|
||||
the step-ca + OIDC path (P04/P05/P07) is the primary identity layer.
|
||||
|
||||
## What v0.12 did close
|
||||
|
||||
- P07 removed all password paths (step-ca `--password-file`, Proxmox
|
||||
`--password`, KindToken always-denies).
|
||||
- P09 removed daemon plaintext mode (Start() requires mTLS).
|
||||
- P11 added SVID chain validation (VerifySVIDWithChain).
|
||||
- P06 rewrote ACL to OIDC (KindToken deprecated).
|
||||
|
||||
## What remains for v1.x
|
||||
|
||||
- Delete `internal/security/ca.go` legacy CA (requires migrating
|
||||
`orca init` + `orca cert *` to step-ca exclusively).
|
||||
- Delete `internal/transport/mtls.go` deprecated path.
|
||||
- Delete `internal/certpaths/` (v0.8 flat layout); `internal/paths/`
|
||||
is the only layout.
|
||||
- Migrate `rotate-lead`, `drain`, `cutover`, `recovery` from
|
||||
`certpaths` to `paths`.
|
||||
|
||||
## Why not in v0.12
|
||||
|
||||
The legacy CA is load-bearing for `orca init` and 6+ CLI commands. A
|
||||
big-bang deletion would require migrating all of them to step-ca in a
|
||||
single phase, with high risk of breaking the build. v0.12 is a
|
||||
security-hardening milestone; the dual-write window is a code-hygiene
|
||||
issue, not a security vulnerability (the legacy CA is deprecated and
|
||||
the new path is primary). v1.x will close it as a focused refactor.
|
||||
+113
-43
@@ -1,56 +1,126 @@
|
||||
---
|
||||
active:
|
||||
active_personas:
|
||||
- lead-developer
|
||||
- backend-engineer
|
||||
- data-engineer
|
||||
- security-engineer
|
||||
deactivated:
|
||||
- cli-engineer
|
||||
- frontend-engineer
|
||||
- security-engineer
|
||||
- network-engineer
|
||||
- devops-engineer
|
||||
deactivated_personas:
|
||||
- frontend-engineer
|
||||
- devops-sre
|
||||
phase_specific:
|
||||
- release-engineer (P1 only — Dockerfile.traefik + release pipeline)
|
||||
- security-engineer
|
||||
- network-engineer
|
||||
- cli-engineer
|
||||
reason: |
|
||||
Orca v0.14 is the ingress bootstrap completeness milestone. The active
|
||||
roster owns the podman-traefik container + nft SNAT/DNAT + proxmox LXC
|
||||
ingress work:
|
||||
- lead-developer: coordinates phase decomposition, owns podman traefik
|
||||
reconciler (P2), nft emitter extension (P3), init/bootstrap wiring (P3,P4)
|
||||
- backend-engineer: owns Dockerfile.traefik + release pipeline (P1),
|
||||
proxmox native ingress mode (P5), floating-IP LXC provisioning (P6),
|
||||
doctor ingress (P7)
|
||||
- data-engineer: owns IngressMode schema migration (P5), node record
|
||||
updates for floating-IP LXC registration (P6)
|
||||
- security-engineer: owns nft priority collision fix (pve-firewall
|
||||
coexistence), step-ca root CA push to nodes, TLS model change
|
||||
(certResolver → dynamic tls.certificates)
|
||||
Orca is a CLI-first, offline-first orchestration engine with no web UI and
|
||||
a single-binary distribution model. The persona roster reflects this:
|
||||
|
||||
network-engineer and devops-engineer are deactivated — their territory
|
||||
(nft ruleset, container deployment) is covered by backend-engineer +
|
||||
lead-developer in this milestone. cli-engineer and frontend-engineer
|
||||
remain deactivated (no CLI framework or UI work).
|
||||
- lead-developer: coordination and task decomposition
|
||||
- backend-engineer: core engine and API handlers
|
||||
- data-engineer: SQLite state store and migrations
|
||||
- cli-engineer: Cobra subcommands and CLI UX
|
||||
- security-engineer: mTLS, cert lifecycle, audit logging, input validation
|
||||
- network-engineer: transport layer, dispatcher, peer-to-peer resilience
|
||||
|
||||
release-engineer is phase-specific for P1 (Dockerfile.traefik +
|
||||
release.sh + .coreci.yml container-publish-traefik step).
|
||||
Deactivated:
|
||||
- frontend-engineer: no web UI in v0.1
|
||||
- devops-sre: no container/cloud integrations; release flow is
|
||||
handled by CoreCI (not a persona territory)
|
||||
|
||||
Territory enforcement is warn mode (config.json
|
||||
personas.territory_enforcement=warn). Cross-territory fixes (e.g. a
|
||||
fix that touches both nft emitter and proxmox bootstrap) are allowed
|
||||
with a warning.
|
||||
Phase-specific (v0.2):
|
||||
- security-engineer: P01 (mTLS/CA) + P02 (peer transport hardening)
|
||||
- network-engineer: P02 only (multi-node scheduling & dispatch)
|
||||
- cli-engineer: P04 only (--watch flag is a CLI concern)
|
||||
---
|
||||
|
||||
Framework alignment (from go.mod):
|
||||
- lead-developer: cobra
|
||||
- backend-engineer: cobra, podman (CLI), pct (CLI via SSH)
|
||||
- data-engineer: modernc/sqlite
|
||||
- security-engineer: nft, step-ca, TLS
|
||||
- release-engineer: docker, .coreci.yml
|
||||
# Personas: Orca
|
||||
|
||||
Constraint alignment:
|
||||
- All personas: offline-first, no-redundant-implementations
|
||||
- backend-engineer: API-first, error-handling, security-first,
|
||||
container-first (R-024)
|
||||
- data-engineer: schema-first, migration-safe, local-storage-only
|
||||
- security-engineer: deny-by-default, zero-trust, no-passwords (R-021),
|
||||
pve-firewall-coexistence
|
||||
- release-engineer: per-release-tagging, registry-auth
|
||||
## Roster
|
||||
|
||||
### lead-developer
|
||||
- **Domain**: coordination
|
||||
- **Frameworks**: `cobra`
|
||||
- **Constraints**: `boundary-enforcement`, `offline-first`, `no-redundant-implementations`
|
||||
- **Territory**: `**/*.go`, `cmd/**`, `internal/**`
|
||||
- **Active**: true
|
||||
|
||||
### backend-engineer
|
||||
- **Domain**: backend
|
||||
- **Frameworks**: `cobra`, `net/http`
|
||||
- **Constraints**: `API-first`, `error-handling`, `minimal-dependencies`, `security-first`
|
||||
- **Territory**: `**/api/**`, `**/*_handler*`, `**/*_handler.go`, `internal/daemon/**`
|
||||
- **Active**: true
|
||||
|
||||
### data-engineer
|
||||
- **Domain**: data
|
||||
- **Frameworks**: `modernc/sqlite`
|
||||
- **Constraints**: `schema-first`, `migration-safe`, `local-storage-only`
|
||||
- **Territory**: `**/store/**`, `**/model.go`, `**/migration*`, `migrations/**`, `internal/store/migrations/0004_certs.sql`
|
||||
- **Active**: true
|
||||
|
||||
### cli-engineer (custom)
|
||||
- **Domain**: CLI/UX
|
||||
- **Frameworks**: `cobra`, `pflag`
|
||||
- **Constraints**: `discoverable-help`, `consistent-flag-naming`, `human-readable-output`, `machine-readable-json-flag`
|
||||
- **Territory**: `cmd/**`, `internal/cli/**`, `internal/commands/**`
|
||||
- **Active**: true
|
||||
- **Reason**: Orca is CLI-first; this persona ensures CLI quality and discoverability.
|
||||
|
||||
### security-engineer (custom)
|
||||
- **Domain**: security
|
||||
- **Frameworks**: `crypto/tls`, `crypto/x509`, `slog`
|
||||
- **Constraints**: `no-panic-in-production`, `structured-audit-logging`, `no-secret-in-logs`, `input-validation`, `least-privilege`
|
||||
- **Territory**: `**/auth/**`, `**/audit/**`, `internal/security/**`, `internal/transport/**` (TLS config only)
|
||||
- **Active**: true
|
||||
- **Reason**: mTLS, audit logging, and input validation are first-class concerns.
|
||||
- **Phase scope**: P01 (mTLS + internal CA), P02 (transport hardening for peer handshakes). Deactivates after P02 ships — P03/P04 have lighter security needs.
|
||||
|
||||
### network-engineer (custom, NEW in v0.2)
|
||||
- **Domain**: networking
|
||||
- **Frameworks**: `net/http`, `crypto/tls` (via `internal/security`), `iter`
|
||||
- **Constraints**: `connection-resilience`, `retry-with-backoff`, `graceful-disconnect`, `context-propagation`
|
||||
- **Territory**: `**/transport/**`, `**/engine/dispatcher*`, `**/engine/peer*`, `internal/engine/dispatcher.go`, `internal/transport/**`
|
||||
- **Active**: true
|
||||
- **Reason**: v0.2 introduces cross-node dispatch and peer-to-peer transport. This persona owns the transport layer, dispatcher, and peer lifecycle concerns that are distinct from the API-handler territory of `backend-engineer`.
|
||||
- **Phase scope**: P02 only. Deactivates after P02 ships.
|
||||
|
||||
### frontend-engineer
|
||||
- **Active**: false
|
||||
- **Reason**: No web UI in v0.1.
|
||||
|
||||
### devops-sre
|
||||
- **Active**: false
|
||||
- **Reason**: No container/cloud integrations. Release flow is handled by CoreCI.
|
||||
|
||||
## Territory Enforcement
|
||||
|
||||
- **Mode**: `warn` (per `config.json`)
|
||||
- **Behavior**: Out-of-territory file changes log a warning but do not block.
|
||||
- **Rationale**: Allows flexibility during early development; tighten to `strict` post-v0.1.
|
||||
|
||||
## Phase-Specific Personas (v0.2)
|
||||
|
||||
| Persona | Active in | Reason |
|
||||
|---------|-----------|--------|
|
||||
| `security-engineer` | P01, P02 | mTLS/CA in P01, transport hardening in P02. Lighter security needs in P03 (CI scanning) and P04 (streaming UX). |
|
||||
| `network-engineer` | P02 | Multi-node dispatch is a P02 concern only. P01 builds the transport primitives but P02 wires them into cross-node scheduling. |
|
||||
| `cli-engineer` | P04 | The `--watch` flag is a CLI surface; P01-P03 don't add new CLI commands. |
|
||||
|
||||
In full-autonomy mode, all personas are auto-accepted and the phase-scope
|
||||
assignments are applied automatically when a phase is committed.
|
||||
|
||||
## Migration from v0.1
|
||||
|
||||
- `backend-engineer` territory unchanged: `internal/daemon/**` still owns HTTP
|
||||
handlers. The new `internal/transport/**` package is shared with
|
||||
`network-engineer` but `transport` owns the *connection lifecycle* (dial,
|
||||
retry, close) while `daemon` owns the *request handlers*.
|
||||
- `data-engineer` territory expanded to include the new
|
||||
`internal/store/migrations/0004_certs.sql` migration in P01.
|
||||
- `security-engineer` territory extended from `internal/security/**` to
|
||||
include the TLS-config portion of `internal/transport/**` (the
|
||||
`NewServerTLSConfig` / `NewClientTLSConfig` helpers).
|
||||
- `cli-engineer` territory unchanged; the new `orca cert` subcommands in P01
|
||||
fall under the existing `internal/cli/**` glob.
|
||||
|
||||
@@ -1,74 +0,0 @@
|
||||
# Phase 1 Verification: Namespace Unification (v0.5 P1)
|
||||
|
||||
**Phase**: 1 (namespace unification)
|
||||
**Milestone**: v0.5 Distribution
|
||||
**Requirements covered**: REQ-041, REQ-042
|
||||
**Date**: 2026-08-03
|
||||
|
||||
## Structural Layer
|
||||
|
||||
- `gofmt -l .` → clean (no files need formatting).
|
||||
- `go vet ./...` → clean (no warnings).
|
||||
- `go build ./...` → succeeds.
|
||||
- New files: `internal/cli/namespace_test.go`, `docs/namespace.md`.
|
||||
- Modified files: `internal/cli/root.go`, `internal/cli/init.go`, `internal/store/store.go`.
|
||||
|
||||
## Behavioral Layer
|
||||
|
||||
### Unit tests (new)
|
||||
- `TestNamespaceDefaultsToUserHome` ✓ — empty `ORCA_HOME` → `~/.orca`.
|
||||
- `TestNamespaceHonorsORCAHOME` ✓ — `ORCA_HOME=/tmp/x` → `Dir()=/tmp/x`, `DBPath()=/tmp/x/orca.db`.
|
||||
- `TestInitHonorsORCAHOME` ✓ — `init` creates `$ORCA_HOME` dir.
|
||||
- `TestSystemFlagSetsORCAHOME` ✓ — `--system` sets `ORCA_HOME=/root/.orca`.
|
||||
- `TestSystemFlagConflictsWithORCAHOME` ✓ — `--system` + `ORCA_HOME=/custom` → error.
|
||||
- `TestInitJSONOutput` ✓ — `init --json` returns `{"path":"...","status":"initialized"}`.
|
||||
- `TestSystemFlagIsPersistent` ✓ — `--system` registered as persistent flag on `rootCmd`.
|
||||
|
||||
### Unit tests (regression — all pass)
|
||||
- `internal/cli/` (9.8s) ✓
|
||||
- `internal/store/` ✓
|
||||
- `internal/doctor/` ✓
|
||||
- `internal/daemon/` ✓
|
||||
- `internal/security/` ✓
|
||||
- `internal/engine/` ✓
|
||||
- `internal/jobspec/` ✓
|
||||
- `internal/transport/` ✓
|
||||
|
||||
### Manual e2e
|
||||
- `ORCA_HOME=/tmp/orca-test-user ./bin/orca init` → creates `/tmp/orca-test-user` ✓
|
||||
- `./bin/orca --system init` → creates `/root/.orca` ✓
|
||||
- `ORCA_HOME=/custom ./bin/orca --system init` → error "conflicts with ORCA_HOME" ✓
|
||||
- `./bin/orca version --json` → `{"version":"v0.4.1",...}` ✓
|
||||
|
||||
## Security Layer
|
||||
|
||||
- No new secret handling. The namespace unification moves path resolution
|
||||
but does not change cert/key file modes (0600/0644 per REQ-033 unchanged).
|
||||
- `--system` flag does not escalate privileges — it only changes the
|
||||
namespace root path. Running as non-root with `--system` will fail at
|
||||
`os.MkdirAll("/root/.orca")` with a permission error (expected).
|
||||
- No new network surface.
|
||||
|
||||
## Quality Layer
|
||||
|
||||
- **Backward compatibility**: empty `ORCA_HOME` + no `--system` → `~/.orca`
|
||||
(identical to pre-v0.5 behavior). All existing tests pass unmodified.
|
||||
- **Single source of truth**: `certpaths.Dir()` is the only namespace root
|
||||
resolver. `store.Open("")` and `init` both route through it.
|
||||
- **No redundant implementations**: the `--system` flag maps to `ORCA_HOME`
|
||||
rather than introducing a parallel path mechanism.
|
||||
- **Documentation**: `docs/namespace.md` covers default, `ORCA_HOME`, and
|
||||
`--system` with examples and resolution order.
|
||||
|
||||
## Must-Haves Checklist
|
||||
|
||||
- [x] `go test ./...` passes (including new namespace_test.go).
|
||||
- [x] `ORCA_HOME=/tmp/x orca init` creates `/tmp/x` (not `~/.orca`).
|
||||
- [x] `orca --system init` creates `/root/.orca` (when run as root).
|
||||
- [x] Empty `ORCA_HOME` + no `--system` → `~/.orca` (backward compat).
|
||||
- [x] `orca version --json` works (needed by install.sh in P2).
|
||||
|
||||
## Verdict
|
||||
|
||||
**PASS** — all 4 verification layers pass. REQ-041 and REQ-042 are
|
||||
satisfied. Ready to ship as `v0.4.2`.
|
||||
@@ -1,73 +0,0 @@
|
||||
# Phase 1 Verification — Orca v0.6 P01
|
||||
|
||||
**Phase**: P01 — `orca init` Full Bootstrap + Schema 0006
|
||||
**REQ Coverage**: REQ-047, REQ-048, REQ-049
|
||||
**Verification date**: 2026-08-03
|
||||
**Result**: ✅ PASS (all 4 layers)
|
||||
|
||||
## Structural Verification
|
||||
|
||||
- ✅ `go build ./...` — PASS (no compile errors)
|
||||
- ✅ `go vet ./...` — PASS (no vet warnings)
|
||||
- ✅ `gofmt -l .` — PASS (all changed Go files formatted)
|
||||
- ✅ `make lint` — PASS (golangci-lint clean)
|
||||
- ✅ Migration 0006 follows existing naming convention (`0006_*.sql`)
|
||||
- ✅ `model.Node` struct follows existing field/tag conventions
|
||||
- ✅ `NodeRepo` methods follow existing error-wrapping + `scanner` pattern
|
||||
|
||||
## Behavioral Verification
|
||||
|
||||
### REQ-047: `orca init` auto-provisions CA + server cert + DB + localhost node
|
||||
- ✅ `TestInit_FullBootstrap`: init creates namespace dir, CA (ca.crt 0644 + ca.key 0600), server cert, DB (migrations 0001..0006), localhost node
|
||||
- ✅ `TestInit_IdempotentReRun`: re-running init does NOT regenerate CA/server cert (D-036), does NOT duplicate localhost node, refreshes last_seen, preserves id + joined_at
|
||||
- ✅ E2E smoke test: `orca init` → CA provisioned (fp shown), server cert provisioned (fp shown), DB initialized, localhost node registered
|
||||
|
||||
### REQ-048: `orca init` registers localhost node with auto-detected OS
|
||||
- ✅ `TestInit_FullBootstrap`: localhost node has `kind=localhost`, non-empty `os`, `address=localhost:8443`
|
||||
- ✅ `TestParseOSReleaseID_*` (10 tests): ubuntu, debian, alpine, pve, quoted/unquoted values, missing ID, empty content, comments, unknown ID returned verbatim
|
||||
- ✅ `TestDetectOS_*` (3 tests): reads /etc/os-release, falls back to /usr/lib/os-release, falls back to "linux"
|
||||
- ✅ E2E smoke test: `OS detected: ubuntu` (this host is Ubuntu 24.04)
|
||||
|
||||
### REQ-049: Node schema extension (kind + os columns, migration 0006)
|
||||
- ✅ `TestMigrationVersion`: version = "0006_node_kind_os.sql"
|
||||
- ✅ `TestNodeRepo_KindOS_RoundTrip`: insert with kind/os → get returns them correctly
|
||||
- ✅ `TestNodeRepo_NullKindOS_EmptyString`: NULL columns → `""` in Go struct (no nil-deref)
|
||||
- ✅ `TestNodeRepo_GetByName`: found by name, ErrNotFound for missing
|
||||
- ✅ `TestNodeRepo_UpdateLastSeenAndOS`: refreshes last_seen + os, preserves id + joined_at (D-036)
|
||||
- ✅ Existing node tests still pass (backward compatible)
|
||||
- ✅ `TestDBCheck_IntegrityOK`: doctor db check reports migration 0006
|
||||
|
||||
## Security Verification
|
||||
|
||||
- ✅ CA key file mode 0600 enforced (`TestInit_FullBootstrap` checks mode)
|
||||
- ✅ CA cert + server cert mode 0644 enforced (via `security.WriteCert`/`writeAtomic`)
|
||||
- ✅ No secrets in logs (init output shows fingerprint prefixes, not full keys)
|
||||
- ✅ `--json` output excludes private key material (only fingerprints)
|
||||
- ✅ No new external dependencies (P1 is pure Go stdlib + existing deps)
|
||||
|
||||
## Quality Verification
|
||||
|
||||
- ✅ `go test -race -count=1 ./internal/store/... ./internal/cli/... ./internal/model/... ./internal/doctor/...` — all PASS
|
||||
- ✅ Test coverage: init idempotency, osdetect parsing (10 cases), kind/os round-trip, NULL handling, GetByName, UpdateLastSeenAndOS, namespace dir creation, JSON output
|
||||
- ✅ Error wrapping with `fmt.Errorf("...: %w", err)` (REQ-018 convention)
|
||||
- ✅ `context.Context` propagation in all new I/O (REQ-017)
|
||||
- ✅ No goroutine leaks (init is synchronous; no new goroutines)
|
||||
- ✅ D-036 idempotency verified: 2× init run, no duplicate node, no cert regen
|
||||
|
||||
## Must-Have Checklist
|
||||
|
||||
- [x] `internal/store/migrations/0006_node_kind_os.sql`
|
||||
- [x] `internal/model/node.go` — Kind + OS fields + NodeKind constants
|
||||
- [x] `internal/store/node_repo.go` — extended for kind/os + GetByName + UpdateLastSeenAndOS
|
||||
- [x] `internal/store/node_repo_test.go` — new tests for kind/os + helpers
|
||||
- [x] `internal/cli/osdetect.go` — detectOS() from /etc/os-release
|
||||
- [x] `internal/cli/osdetect_test.go` — 13 parsing + detection tests
|
||||
- [x] `internal/cli/init.go` — full bootstrap sequence
|
||||
- [x] `internal/cli/init_test.go` — idempotency + bootstrap tests
|
||||
- [x] `internal/cli/namespace_test.go` — updated for new JSON format
|
||||
- [x] `internal/doctor/doctor_test.go` — updated for migration 0006
|
||||
- [x] `internal/store/migrate_test.go` — updated for migration 0006
|
||||
|
||||
## Escalations
|
||||
|
||||
None. All 4 verification layers pass cleanly.
|
||||
@@ -1,67 +0,0 @@
|
||||
# Phase 1 Verification Report — v0.7: Register `orca cert` Command Tree
|
||||
|
||||
**Phase**: 1
|
||||
**Branch**: `phase/01-cert-register`
|
||||
**REQ Coverage**: REQ-053
|
||||
**Milestone**: v0.7 (Hardening & Completion)
|
||||
|
||||
## Structural Verification
|
||||
|
||||
### Files Modified
|
||||
- `internal/cli/cert.go` — added `init()` registering `NewCommand` on `rootCmd` (AD-022)
|
||||
- `internal/cli/init_test.go` — updated expected migration version 0006 → 0007
|
||||
- `internal/doctor/doctor_test.go` — relaxed DB check assertion to check `"migrations up to"` prefix (migration-version-agnostic)
|
||||
- `internal/store/migrate_test.go` — updated expected migration version 0006 → 0007
|
||||
|
||||
### Files Created
|
||||
- `internal/cli/cert_test.go` — regression test for cert command registration + subcommand tree
|
||||
- `internal/cli/cert_smoke_test.go` — end-to-end smoke test (ca-init, gen, show, fingerprint, renew, file modes)
|
||||
- `internal/store/cert_repo_test.go` — 11 tests covering Insert/Get/List/ListByNode/LatestForKind/PruneOlderThan/Delete + error paths
|
||||
- `internal/store/migrations/0007_certs_serial_unique.sql` — UNIQUE index on `certs.serial_hex` (I-107; migration-driven, not backfilled into 0004)
|
||||
|
||||
## Behavioral Verification
|
||||
|
||||
### Test Results
|
||||
```
|
||||
go test ./... → all PASS (exit 0)
|
||||
go test -race ./... → all PASS (exit 0)
|
||||
go vet ./... → clean
|
||||
make build → clean (v0.6.0)
|
||||
```
|
||||
|
||||
### Coverage (store package)
|
||||
- Store total: 60.5% (up from 46.9%)
|
||||
- `cert_repo.go`: Insert 91.7%, Get 100%, LatestForKind 100%, PruneOlderThan 85.7%, Delete 85.7%, List/ListByNode 81.8%
|
||||
|
||||
### CLI Smoke Test (manual)
|
||||
```
|
||||
./bin/orca cert → prints help (was: "unknown command")
|
||||
./bin/orca cert ca-init --cn X → ✓ CA initialized, 0644/0600 modes
|
||||
./bin/orca cert fingerprint --which ca → 64-char hex SHA-256
|
||||
```
|
||||
|
||||
## Security Verification
|
||||
|
||||
- `orca cert show` redacts private key material (REQ-035) — verified in smoke test
|
||||
- Cert file modes enforced: 0600 keys, 0644 certs (REQ-033) — verified in smoke test
|
||||
- No secrets in logs — `cert.ca_init`/`cert.issued`/`cert.renewed` log events contain only fingerprints, never key bytes
|
||||
- Migration 0007 is additive (UNIQUE index), backward-compatible — no data loss
|
||||
|
||||
## Quality Verification
|
||||
|
||||
- No new dependencies added (`go.mod` unchanged)
|
||||
- No comments added (per project convention)
|
||||
- Test style matches existing `node_repo_test.go` / `root_test.go` patterns
|
||||
- All `---ci---` blocks present in commits
|
||||
|
||||
## Must-Haves Checklist
|
||||
|
||||
- [x] `internal/cli/cert.go` — `init()` with `rootCmd.AddCommand(NewCommand(slog.Default()))`
|
||||
- [x] `internal/cli/cert_test.go` — regression test for registration + subcommands
|
||||
- [x] `internal/cli/cert_smoke_test.go` — e2e: ca-init, gen, show (redaction), fingerprint, renew, file modes
|
||||
- [x] `internal/store/cert_repo_test.go` — 11 tests covering full CRUD + rotation history + duplicate serial
|
||||
- [x] `internal/store/migrations/0007_certs_serial_unique.sql` — UNIQUE index (I-107)
|
||||
|
||||
## Verdict
|
||||
|
||||
**PASS** — all 4 verification layers (structural, behavioral, security, quality) pass. REQ-053 is fully covered. The `orca cert` command tree is now reachable from the CLI, cert_repo has comprehensive tests, and the serial_hex UNIQUE constraint is enforced via migration.
|
||||
@@ -1,55 +0,0 @@
|
||||
# Phase 1 Verification — v0.8 Coverage & Trust Hardening
|
||||
|
||||
**Phase**: P01 — Coverage uplift round 2
|
||||
**Milestone**: v0.8
|
||||
**REQ**: REQ-057
|
||||
**Date**: 2026-08-04
|
||||
**Result**: ✅ PASS (all 4 layers)
|
||||
|
||||
## Layer 1 — Structural ✅
|
||||
|
||||
- `go build ./...` PASS (no compile errors)
|
||||
- `go vet ./...` PASS (no warnings)
|
||||
- No TODOs/FIXMEs/stubs in production code (the 3 pre-existing placeholders in `internal/cli/job.go:78`, `internal/engine/scheduler.go:115`, `internal/security/tls_config.go:90` are unchanged from v0.7 and out of scope for P01)
|
||||
- All test files resolve imports correctly
|
||||
- The proxmox `sessionRunner` seam (T01.1) is backward compatible — `BootstrapProxmox` callers unchanged
|
||||
|
||||
## Layer 2 — Behavioral ✅
|
||||
|
||||
- `go test ./...` PASS (all 14 packages)
|
||||
- `go test -race ./...` PASS (cli 98s, engine 47s, store 88s, transport 22s, all others fast)
|
||||
- Coverage targets met (T01.12):
|
||||
- ≥70% floor: engine 88.9%, proxmox 87.1%, cli 76.2%, transport 93.0%, store 84.7%, jobspec 90.5%
|
||||
- ≥50% floor: audit 100.0%, certpaths 100.0%, cmd/orca 80.0%
|
||||
- GRILL condition #3 escape valve NOT needed (cli hit 76.2%, above 70%)
|
||||
- T01.2 (conditional `peerDispatcher` seam) NOT added — engine reached 88.9% via httptest + stubs
|
||||
- REQ-057 covered: all 9 target packages hit their tiered floor
|
||||
|
||||
## Layer 3 — Security ✅
|
||||
|
||||
- P01 is a test-only phase (the only production change is T01.1's `sessionRunner` interface extraction + T01.11's `main()→run()` refactor)
|
||||
- No new input paths, no new network surfaces, no new crypto
|
||||
- The `sessionRunner` seam does not leak test concerns into production (default `sshSessionRunner` wraps the real SSH session; the seam is only injectable via the package-level var pattern matching `sshDialer`)
|
||||
- `cmd/orca/main.go` refactor: `run() int` returns exit code; `main()` calls `os.Exit(run())` — no security impact (same behavior, testable)
|
||||
- No secrets in test code (all test DBs use `:memory:` or temp dirs; no real credentials)
|
||||
|
||||
## Layer 4 — Quality ✅
|
||||
|
||||
- Tests follow existing conventions (table-driven, `t.Run` subtests, `t.Helper()` in setup funcs)
|
||||
- Reuse of existing helpers: `openTestDB`, `withFastWatch`, `initTestEnv`, `resetRootFlags`, `discardWriter`, `stubDispatcher` pattern
|
||||
- No flaky tests detected (all pass on repeated runs with `-race`)
|
||||
- Test file naming follows `*_test.go` convention
|
||||
- No over-testing: daemon.go excluded from cli coverage (covered by `internal/daemon/server_test.go`)
|
||||
- P0 issues: none. P1+ issues: none flagged.
|
||||
|
||||
## Requirement Coverage
|
||||
|
||||
| REQ | Status | Evidence |
|
||||
|-----|--------|----------|
|
||||
| REQ-057 | ✅ Complete | All 9 packages hit tiered floor; `go test -cover` confirms; `go test -race` PASS |
|
||||
|
||||
## Lessons
|
||||
|
||||
- The `sessionRunner` seam pattern (package-level var + default init in entry func) is the canonical way to add testability to orca's SSH-dependent packages. Future SSH-adjacent packages should follow it.
|
||||
- `httptest.NewTLSServer` sufficed for engine 70% without needing the conditional `peerDispatcher` seam — the plan's "only if needed" guard worked as intended.
|
||||
- The cli package's 84s test time is dominated by `--watch` integration tests with real poll intervals. Future coverage work should consider reducing the `withFastWatch` interval further or extracting the watch logic for unit-level testing.
|
||||
@@ -1,85 +0,0 @@
|
||||
# Phase 2 Verification: install.sh + In-Place Update (v0.5 P2)
|
||||
|
||||
**Phase**: 2 (install.sh + in-place update)
|
||||
**Milestone**: v0.5 Distribution
|
||||
**Requirements covered**: REQ-043, REQ-044, REQ-016 (completion)
|
||||
**Date**: 2026-08-03
|
||||
|
||||
## Structural Layer
|
||||
|
||||
- `gofmt -l .` → clean.
|
||||
- `go vet ./...` → clean.
|
||||
- `go build ./...` → succeeds.
|
||||
- New files: `scripts/install.sh`, `scripts/install_test.sh`, `docs/install.md`.
|
||||
- Modified files: `README.md`.
|
||||
- `install.sh` is executable (`chmod +x`).
|
||||
|
||||
## Behavioral Layer
|
||||
|
||||
### install_test.sh — 8/8 tests pass
|
||||
|
||||
Run via `timeout 120 bash scripts/install_test.sh`:
|
||||
|
||||
1. **Test 1: user-level install (v0.4.1)** ✓
|
||||
- Binary at `~/.local/bin/orca` ✓
|
||||
- `orca version --json` returns `v0.4.1` ✓
|
||||
2. **Test 2: in-place update (v0.4.1 → v0.4.2) preserves namespace** ✓
|
||||
- "updated orca from v0.4.1 to v0.4.2" message printed ✓
|
||||
- `~/.orca/orca.db` content preserved ("preserve-me") ✓
|
||||
- Binary version updated to `v0.4.2` ✓
|
||||
3. **Test 3: idempotent re-install (v0.4.2 → v0.4.2)** ✓
|
||||
- "reinstalled orca v0.4.2" message printed ✓
|
||||
4. **Test 4: --system install (root)** ✓
|
||||
- Binary at `/usr/local/bin/orca` ✓
|
||||
- Reports `namespace root: /root/.orca` ✓
|
||||
5. **Test 5: --system without root** — SKIP (running as root)
|
||||
|
||||
### Manual e2e (real Gitea releases)
|
||||
- `curl -fsSL ... | bash` downloads v0.4.2 tarball, extracts, installs ✓
|
||||
- Re-run updates binary; namespace dir untouched ✓
|
||||
- `--version v0.4.1` pins to v0.4.1 ✓
|
||||
|
||||
### Regression — Go tests
|
||||
- `internal/cli/` ✓ (cached, no regressions from P1)
|
||||
- `internal/store/` ✓
|
||||
- `internal/doctor/` ✓
|
||||
|
||||
## Security Layer
|
||||
|
||||
- `install.sh` does not `eval` remote content — it downloads a tarball
|
||||
and extracts it with `tar -xzf`.
|
||||
- No secrets in the script. `GITEA_TOKEN` is not required (public repo,
|
||||
anonymous download per REQ-045).
|
||||
- `.env` is not referenced by install.sh.
|
||||
- The script uses `set -euo pipefail` for fail-fast safety.
|
||||
- `curl -fsSL` fails on HTTP errors (no silent 404 downloads).
|
||||
|
||||
## Quality Layer
|
||||
|
||||
- **1-liner install**: `curl -fsSL <url> | bash` works (verified).
|
||||
- **--system flag**: installs to `/usr/local/bin`, namespace `/root/.orca`,
|
||||
requires root (errors otherwise).
|
||||
- **--version pinning**: `--version vX.Y.Z` queries the specific release tag.
|
||||
- **In-place update (REQ-044)**: detects existing binary, reads version via
|
||||
`orca version --json`, prints update message, overwrites binary, preserves
|
||||
namespace dir. Idempotent.
|
||||
- **Env-overridable**: `GITEA_URL`, `GITEA_OWNER`, `GITEA_REPO` honor
|
||||
pre-set env vars (`${VAR:-default}`) for testability.
|
||||
- **Timeout-guarded**: test harness uses `timeout 30` per test + `timeout 120`
|
||||
overall + `trap 'kill 0' EXIT` to prevent orphaned processes.
|
||||
- **Documentation**: `docs/install.md` covers user/system install, version
|
||||
pinning, in-place update, uninstall, and troubleshooting. README quickstart
|
||||
updated with the 1-liner (REQ-016 completion).
|
||||
|
||||
## Must-Haves Checklist
|
||||
|
||||
- [x] `bash scripts/install_test.sh` passes (8/8).
|
||||
- [x] `curl -fsSL <url> | bash` works on a fresh system.
|
||||
- [x] `curl -fsSL <url> | bash -s -- --system` installs to `/usr/local/bin` (as root).
|
||||
- [x] Re-running updates the binary; `~/.orca/orca.db` preserved.
|
||||
- [x] README quickstart documents the 1-liner + `--system` variant.
|
||||
|
||||
## Verdict
|
||||
|
||||
**PASS** — all 4 verification layers pass. REQ-043, REQ-044, and REQ-016
|
||||
(completion) are satisfied. Ready to ship as `v0.4.3`.
|
||||
@@ -1,86 +0,0 @@
|
||||
# Phase 2 Verification — Orca v0.6 P02
|
||||
|
||||
**Phase**: P02 — Proxmox SSH Join
|
||||
**REQ Coverage**: REQ-050, REQ-051
|
||||
**Verification date**: 2026-08-03
|
||||
**Result**: ✅ PASS (all 4 layers; integration test against real PVE deferred — unit tests cover all logic)
|
||||
|
||||
## Structural Verification
|
||||
|
||||
- ✅ `go build ./...` — PASS
|
||||
- ✅ `go vet ./...` — PASS
|
||||
- ✅ `gofmt -l .` — PASS (all Go files formatted)
|
||||
- ✅ `make lint` — PASS
|
||||
- ✅ `golang.org/x/crypto v0.54.0` added as direct dep (D-030); transitive: x/sys v0.47.0, x/term v0.45.0
|
||||
- ✅ `internal/proxmox` new package follows existing package layout conventions
|
||||
- ✅ `internal/security/sshkey.go` follows the CAInit pattern (idempotent fast-path, writeAtomic, mode enforcement)
|
||||
|
||||
## Behavioral Verification
|
||||
|
||||
### REQ-050: Proxmox SSH bootstrap via golang.org/x/crypto/ssh
|
||||
- ✅ `TestGenerateOrLoadSSHKey_Generates`: Ed25519 keygen, 0600/0644 modes, ssh-ed25519 pub format, ssh.ParsePrivateKey round-trip
|
||||
- ✅ `TestGenerateOrLoadSSHKey_IdempotentLoad`: second call loads existing (D-036)
|
||||
- ✅ `TestGenerateOrLoadSSHKey_CreatesDir`: nested dir creation
|
||||
- ✅ `TestBootstrapProxmox_Validation`: missing host → error, missing password → error
|
||||
- ✅ `TestDefaultOptions`: DefaultProxmoxUser=orca, DefaultProxmoxRole=OrcaOperator, DefaultSSHPort=22
|
||||
- ✅ CLI `--type proxmox --host ... --password ...` flag wiring verified via `orca node join --help`
|
||||
- ✅ Password from `--password` flag OR `$ORCA_PROXMOX_PASSWORD` env var (D-031)
|
||||
- ✅ TOFU host-key via `knownhosts.New` (D-035, avoids deprecated InsecureIgnoreHostKey)
|
||||
- ✅ File upload via session heredoc (no SFTP dep — D-030)
|
||||
|
||||
### REQ-051: OrcaOperator role + orca@pam user + sudoers
|
||||
- ✅ `TestSudoersContent`: NOEXEC on pct/qm, NOPASSWD on apt-get/dpkg (no NOEXEC), pvesh excluded from command lines (AD-020)
|
||||
- ✅ `TestSudoersContent_CustomUser`: custom user name works
|
||||
- ✅ `TestOrcaOperatorPrivileges`: exactly 3 privileges (VM.Audit, Datastore.AllocateSpace, SDN.Use) space-separated (D-033)
|
||||
- ✅ `orca@pam` realm (AD-019 — not @pve)
|
||||
- ✅ `pveum` commands use `--privs` (space-separated), probe-then-add idempotency pattern
|
||||
- ✅ `visudo -cf` validation step aborts bootstrap on syntax error
|
||||
- ✅ Node registered with kind=proxmox, os=pve
|
||||
|
||||
## Security Verification
|
||||
|
||||
- ✅ SSH private key mode 0600 enforced (TestGenerateOrLoadSSHKey_Generates)
|
||||
- ✅ SSH public key mode 0644 enforced
|
||||
- ✅ Password never persisted (D-031) — used only for SSH auth, zeroed after use
|
||||
- ✅ Password from env var preferred over flag (reduces ps/proc exposure)
|
||||
- ✅ pvesh excluded from sudoers (AD-020 — API execute bypasses NOEXEC)
|
||||
- ✅ NOEXEC on pct/qm (blocks shell escapes via dynamically-linked perl)
|
||||
- ✅ TOFU host-key pinning (D-035) — capture on first connect, verify on subsequent, fail closed on mismatch
|
||||
- ✅ No secrets in logs (audit log entries contain host, user, role — never password)
|
||||
- ✅ sudoers file mode 0440 enforced (sudo requirement)
|
||||
|
||||
## Quality Verification
|
||||
|
||||
- ✅ `go test -race -count=1 ./internal/proxmox/... ./internal/security/... ./internal/cli/...` — all PASS
|
||||
- ✅ Test coverage: sshkey (4 tests), proxmox (5 tests), sudoers content (2 tests), privileges (1 test), validation (1 test), defaults (1 test)
|
||||
- ✅ Error wrapping with `fmt.Errorf("...: %w", err)` (REQ-018)
|
||||
- ✅ `context.Context` propagation (REQ-017)
|
||||
- ✅ Idempotency: all bootstrap steps probe-before-add (D-036)
|
||||
- ✅ New direct dep: 1 (golang.org/x/crypto) — matches D-030 minimal-deps rationale
|
||||
|
||||
## Integration Test Note
|
||||
|
||||
A live integration test against a real Proxmox VE 8/9 host is out of
|
||||
scope for automated CI (requires a PVE host + credentials). The SSH
|
||||
bootstrap logic is tested via:
|
||||
- Unit tests for command builders (sudoers content, privilege set)
|
||||
- Unit tests for validation (missing host/password)
|
||||
- Unit tests for SSH key generation (Ed25519, modes, idempotency)
|
||||
- Manual verification via `orca node join --help` (flag surface)
|
||||
|
||||
A `// +build integration` test against a real PVE host can be added
|
||||
in a future phase if a PVE test environment becomes available.
|
||||
|
||||
## Must-Have Checklist
|
||||
|
||||
- [x] `go.mod` / `go.sum` — golang.org/x/crypto v0.54.0
|
||||
- [x] `internal/certpaths/certpaths.go` — SSHKeyPath, SSHPubPath, KnownHostsPath
|
||||
- [x] `internal/security/sshkey.go` — GenerateOrLoadSSHKey (Ed25519)
|
||||
- [x] `internal/proxmox/bootstrap.go` — BootstrapProxmox full SSH dance
|
||||
- [x] `internal/cli/node.go` — --type/--host/--password flag wiring + joinProxmox
|
||||
- [x] `internal/security/sshkey_test.go` — 4 tests
|
||||
- [x] `internal/proxmox/bootstrap_test.go` — 5 tests
|
||||
|
||||
## Escalations
|
||||
|
||||
None.
|
||||
@@ -1,68 +0,0 @@
|
||||
# Phase 2 Verification Report — v0.7: HCL Config File Parsing
|
||||
|
||||
**Phase**: 2
|
||||
**Branch**: `phase/02-config-parser`
|
||||
**REQ Coverage**: REQ-054
|
||||
**Milestone**: v0.7 (Hardening & Completion)
|
||||
|
||||
## Structural Verification
|
||||
|
||||
### Files Created
|
||||
- `internal/config/config.go` — `Config` struct (HCL tags), `CapacityConfig`, `Flags`, `Environ`, `Load(paths...)`, `(*Config).MergeOverrides(flags, env)`
|
||||
- `internal/config/config_test.go` — 11 tests (Load valid/missing/malformed/first-existing, MergeOverrides precedence all 4 layers, NodeCapacity)
|
||||
- `internal/config/testdata/config.hcl` — example fixture
|
||||
|
||||
### Files Modified
|
||||
- `internal/cli/root.go` — added `--config` persistent flag, `configCtxKey`, `configFromCtx` helper; `PersistentPreRunE` loads config if `--config` set (AD-023)
|
||||
- `internal/cli/daemon.go` — daemon uses `cfg.ListenAddr` from config when flag is at default (`:8080`) (D-039 precedence: flag > config)
|
||||
- `internal/cli/root_test.go` — added `TestConfigFlagRegistered` + `TestConfigFlagLoadsFile`
|
||||
|
||||
## Behavioral Verification
|
||||
|
||||
### Test Results
|
||||
```
|
||||
go test ./... → all PASS (exit 0)
|
||||
go test -race ./internal/config/... ./internal/cli/... → all PASS
|
||||
go vet ./... → clean
|
||||
make build → clean (v0.6.1)
|
||||
```
|
||||
|
||||
### API Surface
|
||||
```go
|
||||
func Load(paths ...string) (*Config, error)
|
||||
func (c *Config) MergeOverrides(flags Flags, env Environ) *Config
|
||||
```
|
||||
- `Load` returns zero `&Config{}` if no file exists (no error)
|
||||
- `MergeOverrides` precedence: flag > env > file > default (D-039)
|
||||
- No package-level state (AD-023)
|
||||
|
||||
### CLI Verification
|
||||
```
|
||||
./bin/orca --help → shows --config string flag
|
||||
```
|
||||
|
||||
## Security Verification
|
||||
|
||||
- Config file is read-only (no writes); parsed via `hclsimple.Decode` (no eval, no external commands)
|
||||
- No secrets in config (paths only; no tokens/keys in config.hcl)
|
||||
- Config file permissions not enforced (operator's responsibility; config contains no secrets)
|
||||
|
||||
## Quality Verification
|
||||
|
||||
- No new dependencies (`hashicorp/hcl/v2` already in go.mod for jobspec)
|
||||
- No comments added (per project convention)
|
||||
- Test style matches existing `jobspec/spec_test.go` + `cli/root_test.go`
|
||||
- `go.mod` unchanged
|
||||
|
||||
## Must-Haves Checklist
|
||||
|
||||
- [x] `internal/config/config.go` — Config struct + Load + MergeOverrides
|
||||
- [x] `internal/config/config_test.go` — 11 tests (all 4 precedence layers)
|
||||
- [x] `internal/config/testdata/config.hcl` — example fixture
|
||||
- [x] `internal/cli/root.go` — `--config` persistent flag + context wiring
|
||||
- [x] `internal/cli/daemon.go` — uses `cfg.ListenAddr` (flag still wins)
|
||||
- [x] `internal/cli/root_test.go` — config flag registration + load test
|
||||
|
||||
## Verdict
|
||||
|
||||
**PASS** — all 4 verification layers pass. REQ-054 is fully covered. The `internal/config` package provides HCL config file parsing with flag > env > file > default precedence, wired into the root command via `--config` and consumed by the daemon.
|
||||
@@ -1,55 +0,0 @@
|
||||
# Phase 2 Verification — v0.8 Coverage & Trust Hardening
|
||||
|
||||
**Phase**: P02 — SSH trust hardening
|
||||
**Milestone**: v0.8
|
||||
**REQs**: REQ-058, REQ-059 (+ latent TOFU bugfix closure)
|
||||
**Date**: 2026-08-04
|
||||
**Result**: ✅ PASS (all 4 layers)
|
||||
|
||||
## Layer 1 — Structural ✅
|
||||
|
||||
- `go build ./...` PASS
|
||||
- `go vet ./...` PASS
|
||||
- No TODOs/stubs in new production code
|
||||
- All new exports resolve: `security.SSHFingerprintSHA256`, `security.WriteAtomic`, `proxmox.TOFUHostKeyCallback`, `proxmox.ResetHostKey`, `proxmox.pinnedHostKeyCallback`, `proxmox.Options.HostKeyFingerprint`, `cli.nodeKeyResetCmd`
|
||||
- Backward compatible: existing `BootstrapProxmox` callers work (the TOFU fix changed failure→success on first connect, which is the bugfix)
|
||||
|
||||
## Layer 2 — Behavioral ✅
|
||||
|
||||
- `go test ./internal/proxmox/... ./internal/cli/... ./internal/doctor/... ./internal/security/...` PASS
|
||||
- `go test -race ./internal/proxmox/... ./internal/doctor/...` PASS
|
||||
- Coverage held post-P02: proxmox 86.5% (was 87.1% in P01 — marginal change from new code paths), cli 76.7% (was 76.2%), doctor 70.4% (unchanged)
|
||||
- T02.10: all 7 end-to-end integration cases PASS (pinned correct/wrong, TOFU first/second/mismatch, key-reset+re-pin, pre-populated migration path)
|
||||
- T02.11: `--host-key-fingerprint` non-proxmox validation PASS
|
||||
|
||||
## Layer 3 — Security ✅
|
||||
|
||||
- **REQ-058**: `--host-key-fingerprint` fails closed on mismatch (pinnedHostKeyCallback returns error on any mismatch; bootstrap aborts before any SSH session command runs). SHA256: prefix validated up front. No downgrade to TOFU when pin supplied.
|
||||
- **REQ-059**: `orca node key-reset` is local-only (D-046) — only rewrites `~/.orca/known_hosts` via `security.WriteAtomic` (atomic temp+rename, AD-029); does NOT touch remote authorized_keys. Audit-logs `node.key_reset` with actor+node+host.
|
||||
- **TOFU bugfix (T02.6, v0.6 ship-defect)**: first-connect now captures + writes the key (was silently failing). Mismatch detection preserved (MITM protection). The `TOFUHostKeyCallback` is shared between bootstrap (T02.6) and doctor (T02.9) — GRILL condition #2 parity satisfied.
|
||||
- STRIDE: no new spoofing surface (pin is operator-supplied, fail-closed); no tampering (atomic rewrite); no repudiation (audit log); no info disclosure (fingerprint is a hash, not the key); no DoS (no network change); no elevation (local file ops only).
|
||||
- No secrets in test code (fake SSH keys generated in-test).
|
||||
|
||||
## Layer 4 — Quality ✅
|
||||
|
||||
- Tests follow existing conventions (table-driven, `fakeSSHServer` fixture reused, `sshDialer`/`sessionRunner` seams injected)
|
||||
- `TOFUHostKeyCallback` extracted to a shared helper (no duplication between bootstrap + doctor) — clean coupling (proxmox doesn't import doctor)
|
||||
- P0 issues: none. P1+ issues: none flagged.
|
||||
|
||||
## Requirement Coverage
|
||||
|
||||
| REQ | Status | Evidence |
|
||||
|-----|--------|----------|
|
||||
| REQ-058 | ✅ Complete | `--host-key-fingerprint` flag (T02.3) + `pinnedHostKeyCallback` (T02.5) + `Result.HostKeyFingerprint` (T02.7) + e2e tests (T02.10) + validation (T02.11) |
|
||||
| REQ-059 | ✅ Complete | `orca node key-reset <node>` (T02.8) + `proxmox.ResetHostKey` atomic rewrite + audit log + e2e test (T02.10 case 6) |
|
||||
| (TOFU bugfix) | ✅ Complete | T02.6 fixes v0.6 ship-defect (first-connect `knownhosts.New` KeyError{Want:[]} treated as dial failure); T02.9 doctor parity |
|
||||
|
||||
## GRILL Conditions Check
|
||||
|
||||
- **#1 (T02.6 labeled v0.6 ship-defect)**: ✅ commit `8b0cbe1` summary "TOFU capture bug — v0.6 ship-defect first-connect join always failed"
|
||||
- **#2 (T02.9 doctor parity)**: ✅ both bootstrap (`8b0cbe1`) and doctor (`2dcb143`) use the shared `proxmox.TOFUHostKeyCallback` wrapper
|
||||
|
||||
## Lessons
|
||||
|
||||
- The v0.6 TOFU bug was a latent ship-defect: `knownhosts.New` returns `KeyError{Want:[]}` on first connect without writing, and the original code treated this as a dial failure. This means first-connect Proxmox join has been broken since v0.6 shipped — a strong argument for P01's coverage uplift (the 5.1% proxmox coverage hid this). v0.8 P03's `verify-reqs` would not have caught this (it's code-vs-doc drift, not doc-vs-doc) — P04 audit is the backstop.
|
||||
- Extracting `TOFUHostKeyCallback` to a shared helper was the right call for GRILL condition #2 — duplicating the wrapper in doctor would have created drift risk.
|
||||
@@ -1,75 +0,0 @@
|
||||
# Phase 3 Verification: Docker Release (v0.5 P3)
|
||||
|
||||
**Phase**: 3 (docker release)
|
||||
**Milestone**: v0.5 Distribution
|
||||
**Requirements covered**: REQ-046
|
||||
**Date**: 2026-08-03
|
||||
|
||||
## Structural Layer
|
||||
|
||||
- `go vet ./...` → clean.
|
||||
- `go build ./...` → succeeds.
|
||||
- New files: `Dockerfile`, `.dockerignore`, `docs/docker.md`.
|
||||
- Modified files: `.coreci.yml` (container-publish step), `scripts/release.sh` (docker publish).
|
||||
- `.dockerignore` excludes `.git`, `bin/`, `.env`, `.ciagent/`, `testdata/`, `*.tar.gz`.
|
||||
|
||||
## Behavioral Layer
|
||||
|
||||
### Docker build
|
||||
- `docker build --build-arg VERSION=v0.4.4-test ... -t orca-test:v0.4.4 .` → succeeds.
|
||||
- Multi-stage build: `golang:1.25` (builder) → `gcr.io/distroless/static-debian12:nonroot` (runtime).
|
||||
- `CGO_ENABLED=0` guarantees static binary (modernc/sqlite is pure Go).
|
||||
|
||||
### Docker run
|
||||
- `docker run --rm orca-test:v0.4.4 version` → `orca version v0.4.4-test` ✓
|
||||
- `docker run --rm orca-test:v0.4.4 version --json` → valid JSON with version/commit/build_time ✓
|
||||
- `docker run --rm -v orca-test-data:/var/lib/orca orca-test:v0.4.4 init` → creates `/var/lib/orca` ✓
|
||||
- Volume persistence: state dir created in named volume, verified with alpine container ✓
|
||||
|
||||
### Image metrics
|
||||
- Image size: 27.9MB (distroless static + Go binary).
|
||||
- Runs as `nonroot` user (distroless default).
|
||||
- `ENV ORCA_HOME=/var/lib/orca` set for volume-mountable state.
|
||||
|
||||
### .coreci.yml release pipeline
|
||||
- New `container-publish` step added after `gitea-release`.
|
||||
- Uses `docker:24-cli` image with `GITEA_TOKEN` as registry credential.
|
||||
- Builds, tags (`<version>` + `latest`), logs in, pushes, logs out.
|
||||
|
||||
### scripts/release.sh extension
|
||||
- After Gitea release: `docker build` + `docker login` + `docker push`.
|
||||
- Skips gracefully if `docker` not on PATH (local dev without docker).
|
||||
- Skips push if `GITEA_TOKEN` not set (builds locally only).
|
||||
- Env-overridable: `CONTAINER_REGISTRY`, `CONTAINER_OWNER`, `CONTAINER_IMAGE`.
|
||||
|
||||
### Regression — Go tests
|
||||
- `internal/cli/` ✓ (cached)
|
||||
- `internal/store/` ✓ (cached)
|
||||
|
||||
## Security Layer
|
||||
|
||||
- `.dockerignore` excludes `.env`, `.gitleaks-baseline.json`, `bin/` — no secrets in image.
|
||||
- Image runs as `nonroot` (distroless default) — least privilege.
|
||||
- `docker login` uses `--password-stdin` (no password in process args / shell history).
|
||||
- `docker logout` after push — no credential leakage.
|
||||
- No secret material baked into the image — `GITEA_TOKEN` is used at push time only, not in the build.
|
||||
|
||||
## Quality Layer
|
||||
|
||||
- **Reproducible build**: `--build-arg VERSION/GIT_COMMIT/BUILD_TIME` injected via `-ldflags`.
|
||||
- **Minimal image**: distroless static-debian12 — no shell, no package manager, ~28MB total.
|
||||
- **Graceful degradation**: `release.sh` skips docker publish when docker is absent.
|
||||
- **CI integration**: `.coreci.yml` container-publish step uses `docker:24-cli` (has docker CLI).
|
||||
- **Documentation**: `docs/docker.md` covers pull, run, state persistence, local build, manual publish.
|
||||
|
||||
## Must-Haves Checklist
|
||||
|
||||
- [x] `docker build -t orca-test .` succeeds locally.
|
||||
- [x] `docker run --rm orca-test version` prints the version.
|
||||
- [x] `scripts/release.sh vX.Y.Z` publishes both the Gitea release AND the container image.
|
||||
- [x] `.coreci.yml` release pipeline includes the container-publish step.
|
||||
|
||||
## Verdict
|
||||
|
||||
**PASS** — all 4 verification layers pass. REQ-046 is satisfied. Ready
|
||||
to ship as `v0.4.4`.
|
||||
@@ -1,62 +0,0 @@
|
||||
# Phase 3 Verification — Orca v0.6 P03
|
||||
|
||||
**Phase**: P03 — Doctor Extensions + Audit Logging
|
||||
**REQ Coverage**: REQ-052
|
||||
**Verification date**: 2026-08-03
|
||||
**Result**: ✅ PASS (all 4 layers)
|
||||
|
||||
## Structural Verification
|
||||
|
||||
- ✅ `go build ./...` — PASS
|
||||
- ✅ `go vet ./...` — PASS
|
||||
- ✅ `gofmt -l .` — PASS
|
||||
- ✅ `make lint` — PASS
|
||||
- ✅ `internal/osdetect` new shared package (extracted from cli to avoid import cycle)
|
||||
- ✅ `doctor.OS()` and `doctor.Proxmox()` follow existing check pattern (Check struct, Result, Run func)
|
||||
- ✅ `doctor.All()` extended with OS + Proxmox in logical order
|
||||
|
||||
## Behavioral Verification
|
||||
|
||||
### REQ-052: doctor os + doctor proxmox + audit logging
|
||||
- ✅ `TestOSCheck_MissingLocalhostNode`: no localhost node → FAIL with clear message
|
||||
- ✅ `TestOSCheck_Match`: stored os matches detected → PASS
|
||||
- ✅ `TestOSCheck_Drift`: stored os differs from detected → WARN ("OS drift: init=debian, now=ubuntu")
|
||||
- ✅ `TestProxmoxCheck_NoProxmoxNodes`: zero proxmox nodes → WARN ("no proxmox nodes registered")
|
||||
- ✅ `TestProxmoxCheck_UnreachableNode`: unreachable proxmox node → FAIL with node name
|
||||
- ✅ E2E: `orca doctor os` → PASS (os=ubuntu matches)
|
||||
- ✅ E2E: `orca doctor proxmox` → WARN (no proxmox nodes)
|
||||
- ✅ E2E: `orca doctor os --json` → valid JSON
|
||||
- ✅ E2E: `orca doctor` (full) → 6 PASS / 1 WARN / 1 FAIL (network=daemon not running, expected)
|
||||
- ✅ osdetect package: 11 tests (ubuntu/debian/alpine/pve parsing, quoted/unquoted, missing ID, comments, fallback)
|
||||
- ✅ Audit logging: proxmox.BootstrapProxmox emits `proxmox.bootstrap_ok` (P02); doctor checks are read-only
|
||||
|
||||
## Security Verification
|
||||
|
||||
- ✅ Doctor checks are strictly read-only (no state changes)
|
||||
- ✅ SSH probe uses orca SSH key (not password) — no password in doctor flow
|
||||
- ✅ TOFU host-key verification via knownhosts.New (D-035)
|
||||
- ✅ 3s timeout per proxmox probe (D-038 bounded-probe-timeout pattern)
|
||||
- ✅ No secrets in doctor output (fingerprints only, never private keys)
|
||||
|
||||
## Quality Verification
|
||||
|
||||
- ✅ `go test -race -count=1 ./...` — all PASS (13 packages)
|
||||
- ✅ Test coverage: osdetect (11 tests), doctor OS (3 tests), doctor Proxmox (2 tests)
|
||||
- ✅ Error wrapping with `fmt.Errorf("...: %w", err)` (REQ-018)
|
||||
- ✅ `context.Context` propagation (REQ-017)
|
||||
- ✅ No goroutine leaks (netDialer cleans up on ctx cancellation)
|
||||
- ✅ D-036: doctor os handles pre-0006 rows (empty os field → WARN)
|
||||
|
||||
## Must-Have Checklist
|
||||
|
||||
- [x] `internal/osdetect/osdetect.go` — Detect + ParseID (shared package)
|
||||
- [x] `internal/osdetect/osdetect_test.go` — 11 tests
|
||||
- [x] `internal/cli/osdetect.go` — thin wrapper
|
||||
- [x] `internal/cli/osdetect_test.go` — delegation test
|
||||
- [x] `internal/doctor/doctor.go` — OS() + Proxmox() checks, All() extended
|
||||
- [x] `internal/doctor/doctor_test.go` — 5 new tests
|
||||
- [x] `internal/cli/doctor.go` — doctor os + doctor proxmox subcommands
|
||||
|
||||
## Escalations
|
||||
|
||||
None.
|
||||
@@ -1,76 +0,0 @@
|
||||
# Phase 3 Verification Report — v0.7: Test Coverage Uplift
|
||||
|
||||
**Phase**: 3
|
||||
**Branch**: `phase/03-coverage-uplift`
|
||||
**REQ Coverage**: REQ-055
|
||||
**Milestone**: v0.7 (Hardening & Completion)
|
||||
|
||||
## Structural Verification
|
||||
|
||||
### Files Created
|
||||
- `internal/engine/peer_test.go` — 8 tests (PeerRegistry Add/Get/Remove/All/Len/UpdateLastSeen + validation)
|
||||
- `internal/engine/executor_test.go` — 7 tests (Submit success/missing-command/malformed/failing, Status not-found, Run success, Run context-cancel)
|
||||
- `internal/engine/dispatcher_test.go` — 10 tests (empty spec, idempotency hit, local-capacity, explicit-target, no-peers, LocalSubmit/LocalStatus, nil guards, parseInlineSpec)
|
||||
- `internal/audit/audit_test.go` — 9 tests (Emit/EmitWithErr persistence, LogHandshakeOK/Failed slog fields, nil-safety, Action/Result String, FormatAction)
|
||||
- `internal/transport/handshake_log_test.go` — 8 tests (LogHandshakeOK/Failed/FromCert, FingerprintOfCert, nil-logger, nil-err)
|
||||
- `internal/transport/mtls_test.go` — 14 tests (ServerTLSConfig, ClientTLSConfig, NewMTLSClient, Do, VerifyPeerCertificate, DialContext)
|
||||
- `internal/transport/dispatch_test.go` — 24 tests (SubmitHandler/StatusHandler, DispatchClient constructor/connection-refused/HTTP/decode/Submit/Status success)
|
||||
- `internal/proxmox/ssh_session_test.go` — 14 tests (runRemote, deployPubKey, createLinuxUser, createPVERole, createPVEUser, assignPVEACL, writeSudoers, validateSudoers, full BootstrapProxmox)
|
||||
|
||||
### Files Modified
|
||||
- `internal/transport/dispatch.go` — **bug fix**: `bytesReadCloser.Read` returned `fmt.Errorf("EOF")` instead of `io.EOF`, breaking HTTP request body transmission. This was a latent bug that prevented any client-side dispatch from working end-to-end.
|
||||
- `internal/proxmox/bootstrap_test.go` — extended with 10 new tests (mockSSHDialer, SSH auth failure, dial-addr/port/user propagation, SSH key generation, known_hosts, nil/custom logger, cancelled context, deployPubKey edge cases)
|
||||
|
||||
## Behavioral Verification
|
||||
|
||||
### Test Results
|
||||
```
|
||||
go test ./... → all PASS (exit 0)
|
||||
go test -race ./... → all PASS (exit 0)
|
||||
go vet ./... → clean
|
||||
make build → clean
|
||||
```
|
||||
|
||||
### Coverage (D-042 target: ≥ 50% per package)
|
||||
|
||||
| Package | Before | After | Target |
|
||||
|---------|--------|-------|--------|
|
||||
| `internal/engine` | 8.3% | **65.1%** | 50% ✓ |
|
||||
| `internal/transport` | 26.3% | **84.6%** | 50% ✓ |
|
||||
| `internal/proxmox` | 5.1% | **82.7%** | 50% ✓ |
|
||||
| `internal/audit` | 0% | **100.0%** | 50% ✓ |
|
||||
|
||||
All 4 packages exceed the 50% floor (AD-025).
|
||||
|
||||
### Total new tests: 94 (37 engine+audit + 57 transport+proxmox)
|
||||
|
||||
## Security Verification
|
||||
|
||||
- The `dispatch.go` bug fix (`io.EOF` vs `fmt.Errorf("EOF")`) is a correctness fix — HTTP request bodies now terminate correctly. No security implications (the bug caused requests to fail, not to leak data).
|
||||
- No new dependencies added.
|
||||
- Test fixtures use temp dirs (`t.TempDir()`) — no persistent state.
|
||||
- No secrets in test code (SSH keys are test-generated Ed25519 pairs).
|
||||
|
||||
## Quality Verification
|
||||
|
||||
- No comments added (per project convention).
|
||||
- Test style matches existing patterns (`scheduler_test.go`, `node_repo_test.go`, `certgen_test.go`).
|
||||
- `go.mod` unchanged.
|
||||
- Bug fix in `dispatch.go` is minimal (1 line: `return fmt.Errorf("EOF")` → `return io.EOF` + `io` import).
|
||||
|
||||
## Must-Haves Checklist
|
||||
|
||||
- [x] `internal/engine/executor_test.go` — 7 tests
|
||||
- [x] `internal/engine/dispatcher_test.go` — 10 tests
|
||||
- [x] `internal/engine/peer_test.go` — 8 tests
|
||||
- [x] `internal/transport/mtls_test.go` — 14 tests
|
||||
- [x] `internal/transport/dispatch_test.go` — 24 tests
|
||||
- [x] `internal/transport/handshake_log_test.go` — 8 tests
|
||||
- [x] `internal/audit/audit_test.go` — 9 tests
|
||||
- [x] `internal/proxmox/ssh_session_test.go` — 14 tests + extended `bootstrap_test.go` (+10 tests)
|
||||
- [x] Bug fix: `dispatch.go` bytesReadCloser EOF (latent bug, root-caused during P03)
|
||||
- [x] All 4 target packages ≥ 50% coverage
|
||||
|
||||
## Verdict
|
||||
|
||||
**PASS** — all 4 verification layers pass. REQ-055 is fully covered. All 4 target packages exceed the 50% coverage floor (engine 65.1%, transport 84.6%, proxmox 82.7%, audit 100%). A latent bug in `dispatch.go` (non-`io.EOF` return) was found and fixed during coverage uplift.
|
||||
@@ -1,51 +0,0 @@
|
||||
# Phase 3 Verification — v0.8 Coverage & Trust Hardening
|
||||
|
||||
**Phase**: P03 — Requirements-hygiene gate
|
||||
**Milestone**: v0.8
|
||||
**REQ**: REQ-060
|
||||
**Date**: 2026-08-04
|
||||
**Result**: ✅ PASS (all 4 layers)
|
||||
|
||||
## Layer 1 — Structural ✅
|
||||
|
||||
- `go build ./...` PASS
|
||||
- `go vet ./...` PASS
|
||||
- `cmd/verify-reqs/main.go` (~180 LOC, stdlib only) compiles + links
|
||||
- All exports resolve: `verify(roadmapPath, reqsPath) (diff []string, count int, err error)`
|
||||
- No new dependencies
|
||||
|
||||
## Layer 2 — Behavioral ✅
|
||||
|
||||
- `go test ./cmd/verify-reqs/...` PASS (7 golden-file tests: clean, multi-drift, default-args, malformed, missing-file, v0.2-substring-tolerant, real-repo regression)
|
||||
- `make verify-reqs` → exit 0 on the current repo (`✓ 60 requirements consistent with roadmap`)
|
||||
- T03.5 synthetic drift verification: scratch flip of REQ-053 → `make verify-reqs` exit 1 + `REQ-053: status=Pending, expected=Complete (direction=forward)`; revert → exit 0
|
||||
- `go test ./...` PASS (all 16 packages)
|
||||
- Forward + reverse assertions both exercised (golden test `TestVerify_drift` asserts `direction=reverse` for REQ-003)
|
||||
|
||||
## Layer 3 — Security ✅
|
||||
|
||||
- verify-reqs is a static doc-consistency checker — no network, no secrets, no input injection (markdown is parsed with `regexp` over local files only)
|
||||
- `.coreci.yml` step runs in the existing `golang:1.25` container (no new image, no new permissions)
|
||||
- No STRIDE surface added
|
||||
|
||||
## Layer 4 — Quality ✅
|
||||
|
||||
- Testable core (`verify()` function) + thin `main()` — follows the `cmd/orca/main.go` → `run()` pattern from T01.11
|
||||
- Golden-file test fixtures cover the substring-tolerant regex regression (v0.2 header variant)
|
||||
- GRILL condition #4 satisfied: substring-tolerant regex + reverse-direction assertion + scope note (doc-vs-doc only)
|
||||
- P0 issues: none. P1+ issues: none flagged.
|
||||
|
||||
## Requirement Coverage
|
||||
|
||||
| REQ | Status | Evidence |
|
||||
|-----|--------|----------|
|
||||
| REQ-060 | ✅ Complete | `cmd/verify-reqs` (T03.1) + golden tests (T03.2) + `make verify-reqs` (T03.3) + `.coreci.yml` validate hook (T03.4) + synthetic drift verification (T03.5) |
|
||||
|
||||
## GRILL Conditions Check
|
||||
|
||||
- **#4 (verify-reqs regex + reverse direction)**: ✅ substring-tolerant regex matches v0.2's `**COMPLETE (merged to main via v0.3)**` header (golden test `TestVerify_v0_2_substring_tolerant`); reverse-direction assertion implemented + tested; scope note documented in the commit + the verification report.
|
||||
|
||||
## Lessons
|
||||
|
||||
- The two-regex parser (one for REQ rows, one for milestone-complete headers) with substring tolerance is the right shape — a single strict regex would have silently exempted v0.2 (the exact drift the GRILL flagged).
|
||||
- Refactoring `main()` into a testable `verify()` function made golden-file testing trivial (no subprocess orchestration). This mirrors the T01.11 `main()→run()` pattern and should be the house style for all `cmd/` programs.
|
||||
@@ -1,64 +0,0 @@
|
||||
# Phase 4 Verification Report — v0.7: --pprof Opt-in on orca daemon
|
||||
|
||||
**Phase**: 4
|
||||
**Branch**: `phase/04-pprof-daemon`
|
||||
**REQ Coverage**: REQ-056
|
||||
**Milestone**: v0.7 (Hardening & Completion)
|
||||
|
||||
## Structural Verification
|
||||
|
||||
### Files Created
|
||||
- `internal/daemon/pprof.go` — `StartPprof(addr, log) (*http.Server, error)`: dedicated mux + server, disabled by default, WARN log
|
||||
- `internal/daemon/pprof_test.go` — 5 tests (disabled, enabled, shutdown, mux isolation, full server lifecycle)
|
||||
- `internal/cli/daemon_test.go` — `TestDaemonPprofFlag` (flag registration + default)
|
||||
|
||||
### Files Modified
|
||||
- `internal/daemon/server.go` — `PprofAddr` in Options, `pprofServer` field, `NewServer` starts pprof, `Shutdown` stops both
|
||||
- `internal/cli/daemon.go` — `--pprof` flag, `PprofAddr` in daemon.Options, conditional startup output line
|
||||
|
||||
## Behavioral Verification
|
||||
|
||||
### Test Results
|
||||
```
|
||||
go test ./... → all PASS (exit 0)
|
||||
go test -race ./internal/daemon/... ./internal/cli/... → all PASS
|
||||
go vet ./... → clean
|
||||
make build → clean
|
||||
```
|
||||
|
||||
### CLI Verification
|
||||
```
|
||||
./bin/orca daemon --help → shows --pprof string flag (default "")
|
||||
```
|
||||
|
||||
### Live Smoke Test
|
||||
- `--pprof 127.0.0.1:16060` → WARN logged, `/debug/pprof/` returns 200, `/debug/pprof/cmdline` 200, `/debug/pprof/heap` 200
|
||||
- `/healthz` on pprof listener → 404 (mux isolation confirmed, AD-024)
|
||||
- Clean shutdown stops both servers
|
||||
|
||||
## Security Verification
|
||||
|
||||
- pprof on a **separate** `*http.Server` + `*http.ServeMux`, never on the mTLS daemon listener (AD-024) — verified by `TestStartPprof_MuxIsolated` (`/healthz` returns 404 on pprof mux)
|
||||
- Default **disabled** — no pprof listener unless `--pprof` is explicitly set
|
||||
- WARN log on startup: "unauthenticated, operator-only — do not expose publicly"
|
||||
- No `import _ "net/http/pprof"` side-effect registration on `DefaultServeMux` — all handlers explicitly registered on the dedicated mux
|
||||
|
||||
## Quality Verification
|
||||
|
||||
- No new dependencies (stdlib `net/http`, `net/http/pprof`, `log/slog`, `time` only)
|
||||
- No comments added (per project convention)
|
||||
- `go.mod` unchanged
|
||||
- Test style matches existing `server_test.go`
|
||||
|
||||
## Must-Haves Checklist
|
||||
|
||||
- [x] `internal/daemon/pprof.go` — `StartPprof` with dedicated mux, all pprof handlers
|
||||
- [x] `internal/daemon/server.go` — `PprofAddr` in Options, `pprofServer` field, lifecycle integration
|
||||
- [x] `internal/cli/daemon.go` — `--pprof` flag, passed to Options, conditional startup output
|
||||
- [x] `internal/daemon/pprof_test.go` — 5 tests (disabled, enabled, shutdown, mux isolation, lifecycle)
|
||||
- [x] `internal/cli/daemon_test.go` — flag registration test
|
||||
- [x] AD-024: pprof mux separate from mTLS daemon mux (verified by test)
|
||||
|
||||
## Verdict
|
||||
|
||||
**PASS** — all 4 verification layers pass. REQ-056 is fully covered. The `--pprof` opt-in endpoint runs on a separate listener with a dedicated mux, is disabled by default, and logs a WARN when enabled. I-308 (deferred since v0.2) is now implemented.
|
||||
@@ -1,96 +0,0 @@
|
||||
# Plan: v0.10 Docs & Install Milestone
|
||||
|
||||
## Milestone: v0.10 — Docs & Install Hardening
|
||||
- **Type**: feature (P1 `fix`, P2-P4 `docs`; at least one non-docs phase)
|
||||
- **Tags**: `v0.9.0` (P0) → `v0.9.1` (P1) → `v0.9.2` (P2) → `v0.9.3` (P3) → `v0.9.4` (P4) → `v0.9.5` (P5 = milestone release)
|
||||
- **Branch**: `milestone/v0.10-docs-cli-examples`
|
||||
|
||||
## Phase breakdown
|
||||
|
||||
### Phase P1 — release.sh + install.sh fix (Wave 1)
|
||||
**REQs**: REQ-097, REQ-098
|
||||
**Persona**: backend-engineer
|
||||
**Territory**: `scripts/release.sh`, `scripts/install.sh`, `scripts/tests/*.bash`
|
||||
**Vertical slice**: a broken release → a correctly-asseted release that install.sh resolves.
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P1-T1 | `scripts/release.sh`: replace host-arch build (lines 84, 89-98) with explicit `GOOS=linux GOARCH=amd64 go build` cross-build; produce `orca-${VERSION}-linux-amd64.tar.gz` regardless of host arch | REQ-097 |
|
||||
| P1-T2 | `scripts/release.sh`: after `tea releases create` (line 132), add post-create asset verification — query `/api/v1/repos/$OWNER/$REPO/releases/tags/$VERSION`, assert the tarball appears in `attachments`, retry once if missing, fail loudly with clear error if still missing | REQ-097 |
|
||||
| P1-T3 | `scripts/install.sh`: add asset fallback walk — if the resolved release (latest or `--version`) lacks the matching `orca-<ver>-<os>-<arch>.tar.gz`, query `/releases?limit=20`, walk backward, use the most recent release that carries the asset, print a warning | REQ-098 |
|
||||
| P1-T4 | `scripts/install.sh`: add `--check` dry-run mode that prints version + asset URL + install path without writing | REQ-098 |
|
||||
| P1-T5 | `scripts/tests/release.bats` + `scripts/tests/install.bats`: add/extend bats tests for the new behavior (happy path: asset present; fallback: latest release asset-less, older release has asset; --check prints without writing) | REQ-097, REQ-098 |
|
||||
|
||||
**Must-haves**: release.sh produces an amd64 tarball on any host arch; install.sh resolves to a release with an asset (walking back if needed); `--check` works; bats tests pass.
|
||||
|
||||
### Phase P2 — CLI + jobspec + ingress docs (Wave 2)
|
||||
**REQs**: REQ-091, REQ-092, REQ-093
|
||||
**Persona**: docs-engineer (phase-specific), lead-developer
|
||||
**Territory**: `docs/cli.md`, `docs/jobspec.md`, `docs/ingress.md`
|
||||
**Vertical slice**: an operator with no orca background → can author a jobspec, run it, and understand the ingress model from docs alone.
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P2-T1 | `docs/cli.md`: full CLI reference — global flags, every command/subcommand with synopsis + flag tables + one-line example, output modes (text/json/watch), exit codes, deprecated surface callout boxes (daemon/cert/node-join-mTLS/HCL-jobspec) | REQ-091 |
|
||||
| P2-T2 | `docs/jobspec.md`: markdown frontmatter schema reference — top-level keys, block reference (runtime/ports/env-secrets/volumes/restart/update/service/health/lifecycle/constraints/affinity/tasks), kinds matrix, CEL subset grammar, body semantics, deprecated HCL callout | REQ-092 |
|
||||
| P2-T3 | `docs/ingress.md`: Traefik ingress reference — service→Traefik mapping, R-007 socket-vs-TCP-bind, generated YAML shape, atomic reload (C-10), drain, TLS, worked-example pointer to `examples/full-stack/`, v0.10 forward limitations | REQ-093 |
|
||||
|
||||
**Must-haves**: every command/flag in `internal/cli/` is documented; every jobspec field in `internal/jobspec/markdown.go` is documented; every factual claim is grounded in the live codebase; cross-links resolve; deprecated surface is clearly marked.
|
||||
|
||||
### Phase P3 — full-stack examples (Wave 2, parallel with P2)
|
||||
**REQs**: REQ-094
|
||||
**Persona**: docs-engineer (phase-specific), lead-developer
|
||||
**Territory**: `examples/full-stack/**`
|
||||
**Vertical slice**: an operator → can deploy a multi-service stack with ingress by copying the examples.
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P3-T1 | `examples/full-stack/web-app.md`: kind Service, process runtime, port http, service block (socket default), health, restart (service), update (rolling), constraints (CEL), task group (app + sidecar) | REQ-094 |
|
||||
| P3-T2 | `examples/full-stack/api.md`: kind Service, process runtime, port api, service bind 127.0.0.1 (TCP opt-in), health, restart, update (canary) | REQ-094 |
|
||||
| P3-T3 | `examples/full-stack/worker.md`: kind Job, process runtime, one-shot, timeout, env, lifecycle hooks | REQ-094 |
|
||||
| P3-T4 | `examples/full-stack/log-shipper.md`: kind DaemonSet, schedule (every-node), restart, constraints | REQ-094 |
|
||||
| P3-T5 | `examples/full-stack/postgres.md`: kind Service, process runtime, port pg, volumes + replication (syncthing), health, restart, update (blue-green) | REQ-094 |
|
||||
| P3-T6 | `examples/full-stack/rendered/`: the Traefik dynamic YAML + systemd units orca generates for the stack (traefik-dynamic-web-app.yaml, traefik-dynamic-api.yaml, systemd-web-app.service, systemd-api.service, systemd-log-shipper.service) | REQ-094 |
|
||||
| P3-T7 | `examples/full-stack/README.md`: walkthrough (init → node join → capacity set → ns create → job run → list --watch → inspect rendered → drain/rollback notes → cross-link to docs/ingress.md) | REQ-094 |
|
||||
|
||||
**Must-haves**: all 5 jobspecs parse with the current `internal/jobspec` parser and pass `internal/spec/schema` validators; rendered artifacts match what the emitters would produce; README walkthrough is end-to-end coherent.
|
||||
|
||||
### Phase P4 — README + namespace.md refresh (Wave 3, after P2/P3)
|
||||
**REQs**: REQ-095, REQ-096
|
||||
**Persona**: lead-developer
|
||||
**Territory**: `README.md`, `docs/namespace.md`
|
||||
**Vertical slice**: a new visitor to the repo → sees accurate status, all commands, install instructions that work, and a link to the docs + examples.
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P4-T1 | `README.md`: status line (v0.9 complete, v0.10 in progress); install `--version` example updated to current tag; subcommand table expanded to all commands with deprecation markers; update-in-place example updated; development targets complete; new Documentation + Examples sections | REQ-095 |
|
||||
| P4-T2 | `docs/namespace.md`: replace v0.8 flat path table with v0.9 multi-namespace layout (`cluster/`, `_defaults/`, per-ns `db/jobs/alloc/ns.md`); `ORCA_HOME`/`--system` resolution; `orca ns` subcommand cross-link; v0.8 flat layout flagged deprecated | REQ-096 |
|
||||
|
||||
**Must-haves**: README subcommand table matches `internal/cli/` exactly; install example pins a current tag; namespace.md path table matches `internal/paths/paths.go`; both files cross-link to the new docs.
|
||||
|
||||
### Phase P5 — final review + ship + audit (Wave 4)
|
||||
**REQs**: all (REQ-091..REQ-098)
|
||||
**Persona**: lead-developer
|
||||
**Vertical slice**: milestone complete → merged to main, tagged, released.
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P5-T1 | Code review across all phases (P1-P4); auto-apply P0 fixes, flag P1+ for post-hoc | all |
|
||||
| P5-T2 | Audit: reconstruction test (git log matches `.ciagent/`), file discipline, branch hygiene, commit discipline | all |
|
||||
| P5-T3 | Milestone ship: merge phase/05 → milestone → main; tag `v0.9.5` (= v0.10.0 milestone release); create release with full milestone summary + Linux binary asset (verified by the P1 fix); delete all milestone branches | all |
|
||||
| P5-T4 | Complete milestone: mark REQ-091..098 complete in REQUIREMENTS.md; mark v0.10 docs milestone complete in ROADMAP.md; clear checkpoint | all |
|
||||
|
||||
**Must-haves**: milestone merged to main; release carries the Linux binary (the fix from P1 proving itself); all REQs marked complete; checkpoint cleared.
|
||||
|
||||
## Wave ordering
|
||||
|
||||
- **Wave 1**: P1 (release/install fix) — unblocks the ship of every subsequent phase (each phase ship needs a correctly-asseted release)
|
||||
- **Wave 2**: P2 (docs) + P3 (examples) — parallel, no dependencies between them
|
||||
- **Wave 3**: P4 (README + namespace.md) — depends on P2/P3 existing (cross-links)
|
||||
- **Wave 4**: P5 (final review + ship) — depends on all prior phases
|
||||
|
||||
## Risks
|
||||
|
||||
- **R1**: The jobspecs in P3 might not parse if a field shape has drifted since the explore report. Mitigation: validate each jobspec against the current parser before committing (write a throwaway test or run `orca job run` with `--dry-run` if available).
|
||||
- **R2**: `tea releases create` asset verification in P1 might reveal a tea CLI bug that can't be worked around in bash. Mitigation: fall back to a direct `curl` upload to the Gitea attachments API if `tea` is unreliable.
|
||||
- **R3**: The v0.8.15 release still has no asset after P1 ships (P1 only fixes forward). Mitigation: install.sh's fallback walk (P1-T3) handles the gap; users installing between P1 ship and the first correctly-asseted release (P1's own ship tag v0.9.1) will get a clear warning + fallback.
|
||||
@@ -1,360 +0,0 @@
|
||||
# Plan: v0.11 Production Hardening
|
||||
|
||||
## Milestone: v0.11 — Production Hardening
|
||||
- **Type**: feature (multiple `feat` phases)
|
||||
- **Tags**: `v0.10.0` (P0) → `v0.10.1`…`v0.10.20` (P00…P15.5) → `v0.10.21` (P16 = v0.11.0 milestone release)
|
||||
- **Branch**: `milestone/v0.11-production-hardening`
|
||||
- **New rules adopted**: R-017 (ingress hybrid), R-018/R-019/R-020 (drift detection)
|
||||
- **New decisions**: D-215…D-237 (23 net-new, no collisions)
|
||||
- **New REQs**: REQ-099…REQ-118 (20 net-new; REQ count 98→118)
|
||||
|
||||
## Wave ordering
|
||||
|
||||
### Wave 0 — Foundation (serial)
|
||||
- **P00** — CLI cache layer (R-008). Unblocks all subsequent CLI commands that need cached reads.
|
||||
|
||||
### Wave 1 — Observability + identity (serial, gate-heavy)
|
||||
- **P01** — Metrics endpoint (hand-rolled text exposition). Unblocks `orca doctor mTLS` live probe (C5).
|
||||
- **P01.5** — SPIFFE SVID minting spike (**gate C-08** — if spike fails, fall back to mTLS identity). Gates P02 ACL.
|
||||
- **P02** — ACL (SPIFFE + token identities). Depends on P01.5.
|
||||
|
||||
### Wave 2 — Security + secrets (serial)
|
||||
- **P03** — Secrets subsystem (REQ-080; **gate C-19** threat model).
|
||||
- **P15.5** — Threat model + security review (**gate C-19**) + **ingress hybrid (R-017; REQ-099..REQ-102)** + **`orca doctor mTLS` (REQ-118)**. Per Q3=A, ingress folds in here. This phase grows ~30% but stays one phase.
|
||||
|
||||
### Wave 3 — Data durability (serial)
|
||||
- **P04** — Backup/restore (tar + signed). Unblocks P07 recovery.
|
||||
- **P06** — Alloc history (CLI-side SQLite; R-008 cache DB) + **`orca logs --all-nodes --since` (REQ-117)**. The logs command uses the alloc-history cache DB.
|
||||
|
||||
### Wave 4 — Lifecycle (serial)
|
||||
- **P05** — Drain + daemon drain-and-stop (REQ-061) + **`orca job migrate --to` (REQ-116; C3=drain+reschedule composite)**. Migrate composes P05 drain + P06 alloc history.
|
||||
- **P07** — Recovery (`orca restore`). Depends on P04 backup.
|
||||
|
||||
### Wave 5 — Transactional plane (serial, the big one)
|
||||
- **P10** — Transactional plane (REQ-075, REQ-079; **gate C-09**) + **drift detection (R-018/R-019/R-020; REQ-103..REQ-113)**. This is the largest phase. **Grill may split into P10a (txn plane) + P10b (drift) if vertical slice is too large.**
|
||||
- **P11** — `orca job lint` (REQ-084). Depends on P10 txn plane for dry-run validation.
|
||||
- **P12** — `orca job verify` (dry-run txn through lead). Depends on P10.
|
||||
|
||||
### Wave 6 — Namespace + aggregation (parallel)
|
||||
- **P09** — Collector + aggregator (opt-in; **gates C-11, C-12, C-14**) + **drift-event aggregation extension (REQ-107, D-237)**. The aggregator timer is extended to pull drift-events/ and remediate. C-11 watchdog monitors this timer.
|
||||
- **P13** — `orca ns` subcommands (full surface) + deprecation warnings (REQ-068).
|
||||
|
||||
### Wave 7 — Migration (serial, gate-heavy)
|
||||
- **P14a** — v0.8→v1.0 data migration (REQ-066; **gate C-07**) + **`orca upgrade --to-vX` (REQ-115; C2=thin wrapper, handles R-017 binding cutover)**.
|
||||
- **P14b** — Daemon cutover + running-allocation adoption + **`orca cluster rotate-lead` (REQ-114)**.
|
||||
- **P14c** — Mixed-version tolerance + no-orca-on-server enforcement (REQ-065, REQ-086; implements C-13).
|
||||
|
||||
### Wave 8 — Integration + docs + ship (serial)
|
||||
- **P08** — Integration tests (expand hermetic harness, REQ-087) + **drift-detection integration tests (auto-remediation success, NFS fallback, rate-limit cooldown, secret exclusion)**.
|
||||
- **P15** — README quickstart (REQ-089; **Q5=A Nomad-inspired framing, honest-trade-offs table from doc 3**). All cited CLI commands must exist by this phase.
|
||||
- **P16** — Final review + ship + audit — **v0.11.0 milestone release**.
|
||||
|
||||
## Phase task tables
|
||||
|
||||
### Phase P00 — CLI cache layer (Wave 0)
|
||||
**REQs**: R-008 (cache floor)
|
||||
**Persona**: backend-engineer
|
||||
**Territory**: `internal/cache/`, `internal/store/orca_cache.go`
|
||||
**Vertical slice**: a CLI command that reads cached state → a cache-hit returns in <1ms, a cache-miss populates from the lead.
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P00-T1 | `internal/cache/` package: `orca_cache` SQLite schema (per-class TTLs), `Get(class, key)`, `Set(class, key, val, ttl)`, `Invalidate(class)`; stdlib `database/sql` + modernc/sqlite | R-008 |
|
||||
| P00-T2 | Wire cache into `orca node list`, `orca job list`, `orca ns list` (read path only; writes bypass cache) | R-008 |
|
||||
| P00-T3 | `orca cache show` / `orca cache invalidate` CLI for debugging | R-008 |
|
||||
| P00-T4 | Tests: cache-hit/miss/invalidate/TTL-expiry; bench <1ms cache-hit | R-008 |
|
||||
|
||||
**Must-haves**: cache-hit <1ms; TTL-based invalidation; CLI commands use cache on read path.
|
||||
|
||||
### Phase P01 — Metrics endpoint (Wave 1)
|
||||
**REQs**: (new; metrics text exposition)
|
||||
**Persona**: backend-engineer
|
||||
**Territory**: `internal/cli/metrics.go`, `internal/transport/metrics.go`
|
||||
**Vertical slice**: `curl localhost:9100/metrics` → prometheus text exposition.
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P01-T1 | `internal/transport/metrics.go`: hand-rolled Prometheus text exposition (no client_golang dep); counters for txns applied/drifted/remediated; gauges for peers/nodes/allocs | new |
|
||||
| P01-T2 | `orca daemon --metrics :9100` flag (or sidecar listener); `/metrics` endpoint | new |
|
||||
| P01-T3 | Tests: exposition format validity; counter increments on txn apply | new |
|
||||
|
||||
**Must-haves**: `/metrics` returns valid Prometheus text; no client_golang dependency.
|
||||
|
||||
### Phase P01.5 — SPIFFE SVID minting spike (Wave 1, gate C-08)
|
||||
**REQs**: REQ-076
|
||||
**Persona**: security-engineer
|
||||
**Territory**: `internal/identity/spiffe.go`
|
||||
**Gate**: C-08 — if spike fails, fall back to mTLS identity (decision recorded).
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P01.5-T1 | Spike: mint a SPIFFE SVID via step-ca; verify URI SAN format (`spiffe://orca.local/ns/<ns>/sa/<sa>/<alloc-id>`) | REQ-076 |
|
||||
| P01.5-T2 | Decision record: if spike passes, proceed to P02 with SPIFFE; if fails, fall back to mTLS identity + record in PROJECT.md | REQ-076 |
|
||||
|
||||
**Must-haves**: spike passes or fails with a recorded decision; C-08 gate cleared.
|
||||
|
||||
### Phase P02 — ACL (Wave 1)
|
||||
**REQs**: (new; ACL with SPIFFE + token identities)
|
||||
**Persona**: backend-engineer + security-engineer
|
||||
**Territory**: `internal/acl/`, `internal/cli/acl.go`
|
||||
**Depends on**: P01.5 (SPIFFE or mTLS fallback)
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P02-T1 | `internal/acl/` package: identity → permissions mapping; SPIFFE URI → namespace scope; token identities for operators | new |
|
||||
| P02-T2 | `orca acl` CLI: `grant`, `revoke`, `list`, `check`; scoped to namespace paths per R-002 | new |
|
||||
| P02-T3 | Tests: SPIFFE identity grants ns-scoped access; token grants operator-scoped access; deny by default | new |
|
||||
|
||||
**Must-haves**: deny-by-default; SPIFFE URI maps to namespace; tokens for operator access.
|
||||
|
||||
### Phase P03 — Secrets subsystem (Wave 2, gate C-19)
|
||||
**REQs**: REQ-080
|
||||
**Persona**: security-engineer
|
||||
**Territory**: `internal/secrets/`, `internal/cli/secrets.go`
|
||||
**Gate**: C-19 (threat model must land in P15.5; P03 implements the crypto)
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P03-T1 | `internal/secrets/` package: AES-256-GCM encrypt/decrypt with master.key (R-011); per-line nonce; `LoadCredential=` integration | REQ-080 |
|
||||
| P03-T2 | `orca secrets` CLI: `set`, `get`, `rotate`, `list`; scoped to namespace `.env.secrets` | REQ-080 |
|
||||
| P03-T3 | Tests: encrypt/decrypt round-trip; rotation re-encrypts; master.key 0600 enforced | REQ-080 |
|
||||
|
||||
**Must-haves**: AES-256-GCM; per-line nonce; master.key 0600; `LoadCredential=` integration.
|
||||
|
||||
### Phase P15.5 — Threat model + ingress hybrid + doctor mTLS (Wave 2, gate C-19)
|
||||
**REQs**: REQ-099, REQ-100, REQ-101, REQ-102, REQ-118; C-19
|
||||
**Persona**: security-engineer + network-engineer + backend-engineer
|
||||
**Territory**: `internal/emitter/nft.go`, `internal/emitter/traefik.go`, `internal/cli/nft.go`, `internal/cli/doctor_mtls.go`, threat-model doc
|
||||
**Vertical slice**: `orca init` on a fresh cluster → Traefik binds 127.0.0.1:8443 + nft DNAT → `orca doctor nft` + `orca doctor mTLS` pass.
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P15.5-T1 | `internal/emitter/nft.go`: nftables emitter renders `/etc/nftables.d/orca.nft` (DNAT :443→127.0.0.1:8443, :80→127.0.0.1:8080; SYN-flood filter; `ora_rl` rate-limit meter; `orca_trusted_probes` set); idempotent `nft -f` apply; atomic rule-set swap (D-217, D-218) | REQ-099 |
|
||||
| P15.5-T2 | `internal/emitter/traefik.go` update: static config `address: 127.0.0.1:8443` (default); `--public-binding=traefik-on-public-ip` opt-out emits `:443`; certs/mTLS/dynamic config unchanged (D-220, D-216) | REQ-100 |
|
||||
| P15.5-T3 | `orca doctor nft`: checks table exists, DNAT rules present, rate-limit meter present, file parses (`nft -c -f`), hash matches latest txn (D-221, D-226) | REQ-101 |
|
||||
| P15.5-T4 | `orca nft` CLI: `show [--peer]`, `diff --against <txn-id>`, `doctor`, `country block add <cc-list>`, `rate limit set --rate N/s` (D-223, D-222) | REQ-102 |
|
||||
| P15.5-T5 | `orca doctor mTLS`: trust-chain verification (CA → server cert → workload SVIDs exist + not expired) + live mTLS handshake probe to each peer (reuses P01 metrics endpoint + P01.5 SPIFFE infra); C5=both | REQ-118 |
|
||||
| P15.5-T6 | Threat model doc: covers R-017 ingress trust boundary, R-020 drift deadlock, secret exclusion D-234, `orca` system user blast radius; clears C-19 | C-19 |
|
||||
| P15.5-T7 | Tests: nft emitter output validates (`nft -c -f`); Traefik static config has `127.0.0.1:8443`; doctor nft passes on a clean cluster; doctor mTLS passes with valid chain + live probe | REQ-099..102, 118 |
|
||||
|
||||
**Must-haves**: fresh `orca init` produces hybrid binding; `orca doctor nft` + `orca doctor mTLS` pass; C-19 cleared; opt-out flag works.
|
||||
|
||||
### Phase P04 — Backup/restore (Wave 3)
|
||||
**REQs**: (new; tar + signed backup)
|
||||
**Persona**: backend-engineer
|
||||
**Territory**: `internal/backup/`, `internal/cli/backup.go`
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P04-T1 | `internal/backup/` package: tar `ORCA_HOME` (excl. secrets? or incl. with master.key?); sign with master.key (HMAC-SHA256); `orca backup --out snap.tar.gz` | new |
|
||||
| P04-T2 | `orca restore --in snap.tar.gz` (P07 owns the full recovery; P04 owns the backup format + signing) | new |
|
||||
| P04-T3 | Tests: backup→restore round-trip; signature verification; backup excludes `/run/orca/*` | new |
|
||||
|
||||
**Must-haves**: backup is a signed tarball; restore verifies signature.
|
||||
|
||||
### Phase P06 — Alloc history + logs --all-nodes (Wave 3)
|
||||
**REQs**: REQ-071 (cache DB), REQ-117
|
||||
**Persona**: backend-engineer
|
||||
**Territory**: `internal/store/alloc_history.go`, `internal/cli/logs.go`
|
||||
**Depends on**: P00 (cache DB)
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P06-T1 | `internal/store/alloc_history.go`: CLI-side SQLite retention for alloc state transitions; TTL-based eviction | REQ-071 |
|
||||
| P06-T2 | `orca logs --all-nodes --since 5m`: aggregates journald logs across peers via SSH fanout; `iter.Seq` streaming (D-017); `--since` duration; `--all-nodes` fans out (Q2=C) | REQ-117 |
|
||||
| P06-T3 | Tests: alloc history retention/eviction; logs --all-nodes fans out + streams + cancels via ctrl-c | REQ-071, 117 |
|
||||
|
||||
**Must-haves**: alloc history retained in cache DB; `--all-nodes` aggregates across peers with streaming.
|
||||
|
||||
### Phase P05 — Drain + migrate (Wave 4)
|
||||
**REQs**: REQ-061, REQ-116
|
||||
**Persona**: backend-engineer
|
||||
**Territory**: `internal/cli/drain.go`, `internal/cli/migrate.go`
|
||||
**Depends on**: P06 (alloc history for migrate)
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P05-T1 | `orca node drain <host>`: drain a node (stop new allocs; migrate existing per `update` config); daemon drain-and-stop (REQ-061) | REQ-061 |
|
||||
| P05-T2 | `orca job migrate <name> --to <node>`: drain+reschedule composite (C3=a); uses P05 drain + P06 alloc history; idempotent (Q2=C) | REQ-116 |
|
||||
| P05-T3 | Tests: drain stops new allocs; migrate reschedules to target node; daemon drain-and-stop works | REQ-061, 116 |
|
||||
|
||||
**Must-haves**: drain stops new allocs + migrates existing; migrate reschedules to a specific node.
|
||||
|
||||
### Phase P07 — Recovery (Wave 4)
|
||||
**REQs**: (new; `orca restore`)
|
||||
**Persona**: backend-engineer
|
||||
**Territory**: `internal/cli/restore.go`
|
||||
**Depends on**: P04 (backup format)
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P07-T1 | `orca restore --in snap.tar.gz`: verify signature, extract, reconcile with live state (don't clobber running allocs unless `--force`) | new |
|
||||
| P07-T2 | Tests: restore from signed backup; signature mismatch fails; `--force` clobbers running allocs | new |
|
||||
|
||||
**Must-haves**: restore verifies signature; doesn't clobber running allocs without `--force`.
|
||||
|
||||
### Phase P10 — Transactional plane + drift detection (Wave 5, gate C-09)
|
||||
**REQs**: REQ-075, REQ-079, REQ-103..REQ-113; C-09; R-018/R-019/R-020
|
||||
**Persona**: backend-engineer + devops-engineer + security-engineer
|
||||
**Territory**: `internal/drift/`, `internal/cli/drift.go`, `scripts/orca-drift-notify.sh`, `scripts/orca-remediate.sh`, `internal/emitter/systemd.go` (Path units), `internal/paths/paths.go`
|
||||
**Vertical slice**: operator edits `/etc/traefik/dynamic/orca.yml` on a peer → drift detected in ~10s → auto-remediated → `orca drift watch` shows the event.
|
||||
**Note**: This is the largest phase. **Grill may split into P10a (txn plane, REQ-075/079) + P10b (drift detection, REQ-103..113) if the vertical slice is too large.**
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P10-T1 | `internal/txn/` package: render txn bundle (tarball + apply.sh + verify.sh) on operator host; SCP to lead's `/run/orca/txns/<txn-id>/`; lead's systemd timer runs `apply.sh` idempotently; CLI polls txn status via SSH (REQ-075, C-09) | REQ-075 |
|
||||
| P10-T2 | `scripts/orca-pull.sh` with C-09 failure contract: idempotent re-run, bounded retry, deterministic state, structured syslog (C-09) | REQ-079, C-09 |
|
||||
| P10-T3 | `internal/drift/` package: `Detector` interface (`Watch`, `Aggregate`, `Remediate`, `Acknowledge`), `Event`, `Config`, `PathSpec`, `RemediationPolicy`; `iter.Seq2[Event, error]` (D-017); `signal.NotifyContext` (D-023) (D-236) | REQ-103 |
|
||||
| P10-T4 | `orca drift` CLI tree: `watch [--interval=2s] [--paths=...] [--json]`, `show [--peer]`, `acknowledge <peer> <path>`, `remediate <peer> <path> [--force]`, `config show`, `config validate` (D-236) | REQ-104 |
|
||||
| P10-T5 | systemd Path unit emitter: for each critical path, emit `orca-drift-<name>.path` (`PathChanged=`, `RateLimitIntervalSec=1s`, `RateLimitBurst=5`) + `orca-drift-<name>.service` (`Type=oneshot`, `ExecStart=/usr/local/bin/orca-drift-notify.sh %f`, `User=orca`, security hardening); R-001-clean (D-227, D-228) | REQ-105 |
|
||||
| P10-T6 | `scripts/orca-drift-notify.sh`: receives changed path as `$1`, computes sha256 (or "DELETED"), writes event JSON to `/etc/orca/state/drift-events/<event-id>.json`; stateless, idempotent; `flock` (D-228) | REQ-106 |
|
||||
| P10-T7 | `scripts/orca-remediate.sh`: re-pushes latest applied txn's per-peer render tree via rsync, runs peer-side applier; 5-min cooldown per path applies ONLY on successful remediation (C4 refinement); transient failures retry next tick (D-231, D-232) | REQ-108 |
|
||||
| P10-T8 | Drift cadence config in `config.md` (`kind: ClusterConfig`): `drift.polling`, `drift.paths.{critical,standard,excluded}`, `drift.remediate`; critical defaults: Traefik dynamic, nftables, sudoers, orca-alloc services; secrets + `/run/orca/*` + drift-events dir excluded (R-018, D-231, D-234) | REQ-109 |
|
||||
| P10-T9 | Pre-flight consistency gate in `orca-pull.sh`: refuses new txns if drift detected on target peer/namespace; `--force` overrides; per-namespace scoping (drifted peer in ns-A doesn't block ns-B) (R-020, Q4=A) | REQ-110 |
|
||||
| P10-T10 | `orca` system user on peers: peer-setup emits `useradd -r orca` (system account, no login shell); `orca-drift-*.service` runs as `User=orca`; idempotent (REQ-111) | REQ-111 |
|
||||
| P10-T11 | NFS detection at peer setup: `orca node join` detects NFS mounts on orca state dirs; disables Path units for NFS paths; logs warning; falls back to polling (D-233) | REQ-112 |
|
||||
| P10-T12 | `orca job restart <name>`: restarts an alloc to pick up EnvironmentFile drift; normal allocation lifecycle (not file-level remediation) (D-235) | REQ-113 |
|
||||
| P10-T13 | Tests: txn apply idempotent + retry on failure; drift detected via Path unit ~10s; auto-remediation re-pushes; cooldown prevents loop; `--force` overrides pre-flight gate; per-ns scoping isolates drift; NFS fallback; secret exclusion | REQ-075..113 |
|
||||
|
||||
**Must-haves**: txn apply idempotent (C-09); drift detected ~10s on critical paths; auto-remediation with cooldown; `--force` + per-ns override; `orca` user created; NFS detection works.
|
||||
|
||||
### Phase P11 — `orca job lint` (Wave 5)
|
||||
**REQs**: REQ-084
|
||||
**Persona**: backend-engineer
|
||||
**Territory**: `internal/cli/job_lint.go`
|
||||
**Depends on**: P10 (txn plane for dry-run validation)
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P11-T1 | `orca job lint <spec.md>`: validates jobspec schema (kinds, blocks, CEL constraints, body preservation); reports errors with line numbers | REQ-084 |
|
||||
| P11-T2 | Tests: valid spec passes; invalid spec reports errors with line numbers | REQ-084 |
|
||||
|
||||
**Must-haves**: lint catches schema errors; reports line numbers.
|
||||
|
||||
### Phase P12 — `orca job verify` (Wave 5)
|
||||
**REQs**: (new; dry-run txn through lead)
|
||||
**Persona**: backend-engineer
|
||||
**Territory**: `internal/cli/job_verify.go`
|
||||
**Depends on**: P10 (txn plane)
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P12-T1 | `orca job verify <spec.md>`: dry-run txn through lead (no apply); reports what would change (allocs created/removed, config files written) | new |
|
||||
| P12-T2 | Tests: verify reports planned changes without applying; fails on pre-flight drift | new |
|
||||
|
||||
**Must-haves**: verify is a true dry-run (no side effects); reports planned changes.
|
||||
|
||||
### Phase P09 — Collector + aggregator + drift-event aggregation (Wave 6)
|
||||
**REQs**: (existing collector/aggregator) + REQ-107; C-11, C-12, C-14
|
||||
**Persona**: devops-engineer + backend-engineer
|
||||
**Territory**: `scripts/orca-aggregate.sh`, `internal/cli/collector.go`
|
||||
**Gates**: C-11 (watchdog), C-12 (opt-in), C-14 (syncthing)
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P09-T1 | `scripts/orca-aggregate.sh`: existing 10s cadence (C-11); now also rsyncs each peer's `/etc/orca/state/drift-events/`, validates event hashes against applied txn manifest, triggers `orca-remediate.sh` for auto-remediable paths, consumes (deletes) event files on peers (D-229, D-237) | REQ-107 |
|
||||
| P09-T2 | Lead-side watchdog meta-timer (C-11): fires on `orca-pull.sh` starvation (>N seconds without successful run); structured alert path | C-11 |
|
||||
| P09-T3 | `orca collector` CLI: opt-in collector for per-peer state snapshots; writes to `cluster.json` (C-12 opt-in) | C-12 |
|
||||
| P09-T4 | Tests: aggregator pulls drift-events + triggers remediation; watchdog fires on starvation; collector opt-in | REQ-107, C-11 |
|
||||
|
||||
**Must-haves**: aggregator pulls drift-events + remediation; watchdog fires on starvation; collector opt-in.
|
||||
|
||||
### Phase P13 — `orca ns` subcommands + deprecation warnings (Wave 6)
|
||||
**REQs**: REQ-068
|
||||
**Persona**: lead-developer
|
||||
**Territory**: `internal/cli/ns.go`
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P13-T1 | Full `orca ns` surface: `list`, `create`, `delete`, `inspect`, `validate`, `inherit`, `set-constraint` (per R-002 namespace-as-path) | REQ-068 |
|
||||
| P13-T2 | Depprecation warnings: `orca daemon`, `orca cert` (v0.8 mTLS path), `.hcl` jobspec → printed on use; `--no-deprecation-warnings` suppresses (REQ-068) | REQ-068 |
|
||||
| P13-T3 | Tests: all ns subcommands work; deprecation warnings fire on deprecated surface | REQ-068 |
|
||||
|
||||
**Must-haves**: full ns surface; deprecation warnings on deprecated surface.
|
||||
|
||||
### Phase P14a — v0.8→v1.0 data migration + `orca upgrade` (Wave 7, gate C-07)
|
||||
**REQs**: REQ-066; C-07; REQ-115
|
||||
**Persona**: data-engineer + backend-engineer
|
||||
**Territory**: `internal/migration/`, `internal/cli/upgrade.go`
|
||||
**Gate**: C-07 (CA migration spec)
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P14a-T1 | `internal/migration/` package: v0.8 flat layout → v0.9/v0.11 multi-namespace layout; schema migration (0006→next); CA migration per spec (C-07) | REQ-066 |
|
||||
| P14a-T2 | `orca upgrade --to-vX`: thin wrapper (C2=a) around `install.sh` + `orca restore`; handles R-017 Traefik binding cutover (`:443` → `127.0.0.1:8443`) for existing v0.9/v0.10 clusters (Q2=C) | REQ-115 |
|
||||
| P14a-T3 | Tests: v0.8 layout migrates to v0.11 layout; `orca upgrade` handles binding cutover; idempotent | REQ-066, 115 |
|
||||
|
||||
**Must-haves**: v0.8 data migrates to v0.11; `orca upgrade` handles binding cutover; C-07 cleared.
|
||||
|
||||
### Phase P14b — Daemon cutover + rotate-lead (Wave 7)
|
||||
**REQs**: (existing daemon cutover) + REQ-114
|
||||
**Persona**: backend-engineer + lead-developer
|
||||
**Territory**: `internal/cli/daemon.go`, `internal/cli/rotate_lead.go`
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P14b-T1 | Daemon cutover: `orca daemon` becomes `drain-and-stop` (REQ-061 from P05); running-allocation adoption (orphaned allocs adopted by SSH-push path) | existing |
|
||||
| P14b-T2 | `orca cluster rotate-lead`: moves cluster CA + lead state to a new bare-Linux peer (R-003); workloads keep running (certs distributed); SSH key rotation; idempotent (Q2=C) | REQ-114 |
|
||||
| P14b-T3 | Tests: daemon cutover adopts running allocs; rotate-lead moves CA + workloads keep running | REQ-114 |
|
||||
|
||||
**Must-haves**: daemon cutover adopts running allocs; rotate-lead moves CA without downtime.
|
||||
|
||||
### Phase P14c — Mixed-version tolerance (Wave 7)
|
||||
**REQs**: REQ-065, REQ-086; C-13
|
||||
**Persona**: backend-engineer
|
||||
**Territory**: `internal/transport/`, `internal/cli/`
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P14c-T1 | Mixed-version tolerance: lead and peers can run different orca versions during upgrade window; no-orca-on-server enforcement (R-001) | REQ-065, REQ-086 |
|
||||
| P14c-T2 | Tests: mixed-version cluster operates; orca-on-server detected + refused | REQ-065, 086 |
|
||||
|
||||
**Must-haves**: mixed-version tolerance during upgrade; R-001 enforced.
|
||||
|
||||
### Phase P08 — Integration tests + drift-detection tests (Wave 8)
|
||||
**REQs**: REQ-087
|
||||
**Persona**: devops-engineer
|
||||
**Territory**: `tests/integration/`, `scripts/tests/`
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P08-T1 | Expand hermetic test harness: multi-peer setup, txn apply, drift injection, remediation verification (REQ-087) | REQ-087 |
|
||||
| P08-T2 | Drift-detection integration tests: auto-remediation success (edit Traefik config → detect ~10s → remediated); NFS fallback (NFS mount → Path units disabled → polling); rate-limit cooldown (repeated drift → cooldown blocks loop); secret exclusion (edit `/etc/orca/credentials/*` → no drift event) | REQ-087 |
|
||||
| P08-T3 | Tests pass in CoreCI `integration` pipeline (nft exclusively, D-224) | REQ-087 |
|
||||
|
||||
**Must-haves**: integration tests cover drift detection; pass in CoreCI.
|
||||
|
||||
### Phase P15 — README quickstart (Wave 8)
|
||||
**REQs**: REQ-089
|
||||
**Persona**: lead-developer + docs-engineer (phase-specific)
|
||||
**Territory**: `README.md`
|
||||
**Framing**: Q5=A (Nomad-inspired, OS-as-cluster; honest-trade-offs table from doc 3; Proxmox as one node type, not the identity)
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P15-T1 | README.md: status line (v0.11 complete, v1.0 UAT-gated); install example; subcommand table expanded to ALL v0.11 commands (incl. drift, nft, migrate, rotate-lead, upgrade, logs --all-nodes, doctor mTLS); honest-trade-offs table (doc 3 §4.6); Nomad-inspired framing (Q5=A) | REQ-089 |
|
||||
| P15-T2 | Verify all cited CLI commands exist in `internal/cli/` (C-22-style grounding gate) | REQ-089 |
|
||||
|
||||
**Must-haves**: README subcommand table matches `internal/cli/` exactly; honest-trade-offs table present; Nomad-inspired framing.
|
||||
|
||||
### Phase P16 — Final review + ship + audit (Wave 8)
|
||||
**REQs**: all (REQ-099..REQ-118 + existing v0.11 REQs)
|
||||
**Persona**: lead-developer
|
||||
**Vertical slice**: milestone complete → merged to main, tagged, released.
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P16-T1 | Code review across all phases; auto-apply P0 fixes, flag P1+ for post-hoc | all |
|
||||
| P16-T2 | Audit: reconstruction test (git log matches `.ciagent/`), file discipline, branch hygiene, commit discipline | all |
|
||||
| P16-T3 | Milestone ship: merge phase/16 → milestone → main; tag `v0.10.21` (= v0.11.0 milestone release); release with Linux binary asset; delete all milestone branches | all |
|
||||
| P16-T4 | Complete milestone: mark all v0.11 REQs complete in REQUIREMENTS.md; mark v0.11 complete in ROADMAP.md; clear checkpoint | all |
|
||||
|
||||
**Must-haves**: milestone merged to main; release carries Linux binary; all REQs marked complete; checkpoint cleared.
|
||||
|
||||
## Risks
|
||||
|
||||
- **R1: P10 sizing** — P10 is the largest phase (txn plane + drift detection, 13 tasks). Mitigation: grill may split into P10a/P10b. The plan is structured so P10a (T1-T2, txn plane) and P10b (T3-T13, drift) are separable.
|
||||
- **R2: R-020 deadlock** — hard-gate refusal could block all new txns if a peer is permanently drifted on a require_approval path. Mitigation: `--force` + per-ns scoping (Q4=A); documented in C-09 failure contract.
|
||||
- **R3: Ingress default migration** — existing v0.9/v0.10 clusters run Traefik on `:443`. R-017 makes `127.0.0.1:8443` + nft the default. Mitigation: `orca upgrade` (REQ-115, P14a) handles the binding cutover.
|
||||
- **R4: `orca` system user** — creating a system user on every peer is a new operational requirement. Mitigation: peer-setup emits `useradd -r orca` idempotently (REQ-111); documented in P10.
|
||||
- **R5: Scope ceiling** — v0.11 stays at 23 phases (no new phases), but P09/P10/P15.5 grow substantially. Mitigation: wave ordering isolates the largest work (Wave 5) so it can be split without affecting other waves.
|
||||
@@ -1,395 +0,0 @@
|
||||
# Plan v0.12: Security Hardening (Zero-Trust Identity)
|
||||
|
||||
**Status**: Phase 0 plan. 29 phases (P0 + P01..P27 + P28 final). Wave
|
||||
ordering, persona assignments, and binding conditions. GRILL will
|
||||
pressure-test and may split/merge.
|
||||
|
||||
## Milestone identity
|
||||
|
||||
- **Label**: `v0.12-security-hardening`
|
||||
- **Type**: feature (P04, P05 ship `feat`)
|
||||
- **Tag line**: v0.11.x patches (`v0.11.0`..`v0.11.28`)
|
||||
- **Final phase patch** = milestone release = `v0.11.28` (no separate `v0.12.0`)
|
||||
- **Branch**: `milestone/v0.12-security-hardening`
|
||||
- **v1.0.0**: deferred for post-v0.12 UAT (per v0.11 PRD)
|
||||
|
||||
## Wave ordering
|
||||
|
||||
Waves are dependency-ordered. Within a wave, phases run in sequence
|
||||
(parallelization disabled per config.json `parallelization.enabled=false`).
|
||||
|
||||
### Phase 0 — Pre-execution (all personas, lead-developer coordinates)
|
||||
|
||||
Stages: SPECIFY -> CLARIFY -> RESEARCH -> IDEATE -> PLAN -> GRILL -> SHIP.
|
||||
|
||||
- SPECIFY: v0.12 in config.json + PROJECT.md (done).
|
||||
- CLARIFY: D-238..D-247 (done, CLARIFY_v0.12.md).
|
||||
- RESEARCH: threat model F1..F25 + zero-trust identity model (done,
|
||||
RESEARCH_v0.12.md). Resolves RQ-1 (WebAuthn as password-free upstream).
|
||||
- IDEATE: 30 ideas accepted -> REQ-119..REQ-148 (done, IDEATION_v0.12.md).
|
||||
- PLAN: this document.
|
||||
- GRILL: ratify C-29..C-38, split/merge as needed.
|
||||
- SHIP: tag `v0.11.0`.
|
||||
|
||||
**Commit**: `docs(P00): v0.12 security-hardening phase 0 (specify/clarify/research/ideate/plan/grill)`
|
||||
|
||||
### Wave A — Critical injection & traversal (backend-engineer)
|
||||
|
||||
Vertical slice: stop the bleeding first. Three independent fixes, no
|
||||
inter-dependencies.
|
||||
|
||||
#### P01 — Command injection (podman/wasm) — REQ-119, F3
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/runtime/`)
|
||||
- **Tasks**:
|
||||
1. Add `shellQuote` helper (or use `golang.org/x/crypto/ssh`-safe quoting) to `internal/runtime/`.
|
||||
2. Fix `podman.go:57`: `fmt.Sprintf("podman run -d --name %s %q %s", name, image, shellQuote(cmdStr))`.
|
||||
3. Fix `wasm.go:39`: same pattern for `wasmtime run`.
|
||||
4. Add Go regression tests: `;`, `|`, `$()`, backticks, newline, `$IFS`, `<>()` injection attempts.
|
||||
5. Add bats test: a jobspec with a malicious command runs the literal command, not the injected shell.
|
||||
- **Must-haves**: all injection tests pass; existing podman/wasm tests still pass.
|
||||
- **Commit**: `fix(P01): command injection in podman/wasm runtimes (REQ-119, F3)`
|
||||
- **Tag**: `v0.11.1`
|
||||
|
||||
#### P02 — Namespace path traversal — REQ-120, F4
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/ns/`, `internal/cli/ns.go`)
|
||||
- **Tasks**:
|
||||
1. Add `validateNamespaceName(name)` to `internal/ns/`: reject `..`, `/`, leading `-`, null bytes, control chars, empty, length > 128.
|
||||
2. Wire into `ns create`, `ns inherit`, `ns set-constraint`, and any path-accepting ns command.
|
||||
3. Add Go fuzz test (`FuzzValidateNamespaceName`).
|
||||
4. Add regression test: `orca ns create "../../etc"` fails with a clear error.
|
||||
- **Must-haves**: fuzz test passes 10k iterations; `..`/`/`/null rejected.
|
||||
- **Commit**: `fix(P02): namespace path traversal (REQ-120, F4)`
|
||||
- **Tag**: `v0.11.2`
|
||||
|
||||
#### P03 — Txn apply path allowlist — REQ-121, F5
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/txn/`)
|
||||
- **Tasks**:
|
||||
1. In `txn.go:renderApplyScript`, add path validation to the python heredoc: every `path` in `desired-state.json` must match a prefix in the allowlist (`/etc/orca/`, `/etc/traefik/orca*`, `/etc/systemd/system/orca-*`, `/etc/nftables.d/orca*`, `/etc/syncthing/orca*`).
|
||||
2. Reject with a clear error + exit code on mismatch.
|
||||
3. Add Go test: a desired-state with `"path": "/etc/shadow"` is rejected.
|
||||
4. Add bats test: `orca-pull.sh` with a crafted manifest refuses.
|
||||
- **Must-haves**: arbitrary-path writes rejected; legitimate paths still apply.
|
||||
- **Commit**: `fix(P03): txn apply path allowlist (REQ-121, F5)`
|
||||
- **Tag**: `v0.11.3`
|
||||
|
||||
### Wave B — Zero-trust identity (backend-engineer + lead-developer)
|
||||
|
||||
The architectural foundation. P04/P05 are `feat` phases; P06/P07/P08
|
||||
are `fix`/`refactor` that depend on them.
|
||||
|
||||
#### P04 — OIDC client + bundled Dex — REQ-144
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/cli/`, `internal/identity/`)
|
||||
- **Tasks**:
|
||||
1. Add `github.com/coreos/go-oidc/v3` dependency.
|
||||
2. `internal/identity/oidc.go`: OIDC client (provider discovery, JWKS cache + refresh, ID token verification, token storage at `~/.orca/credentials.json` 0600).
|
||||
3. `orca auth login`/`logout`/`status` CLI: browser auth-code + PKCE + local loopback redirect (`127.0.0.1:<port>/callback`); headless device-code fallback.
|
||||
4. `orca auth init-idp`: bootstrap bundled Dex (systemd unit + config template + Traefik route) on the lead; `--rp-id <domain>` config.
|
||||
5. OIDC config block in `internal/config/`: `oidc.issuer`, `client_id`, `client_secret`, `scopes`.
|
||||
6. BYO external IdP override: `oidc.issuer` repoint bypasses bundled Dex.
|
||||
7. Go tests: mock OIDC provider, JWKS rotation, token refresh, login/logout flow.
|
||||
- **Must-haves**: `orca auth login` produces a valid ID token; `orca auth status` shows it; `--oidc` flag gated; offline Dex quickstart doc'd.
|
||||
- **Commit**: `feat(P04): OIDC client + bundled Dex (REQ-144, D-239, D-242)`
|
||||
- **Tag**: `v0.11.4`
|
||||
|
||||
#### P05 — WebAuthn connector for Dex — REQ-148
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/identity/`, new `internal/webauthn/`)
|
||||
- **Tasks**:
|
||||
1. Add `github.com/go-webauthn/webauthn` dependency.
|
||||
2. `internal/webauthn/connector.go`: Dex connector (~300 LoC) -- registration + login ceremonies at `/orca/webauthn/{register,login}`.
|
||||
3. `internal/webauthn/store.go`: passkey storage SQLite at `ClusterDir()/webauthn-credentials.db` (0600); schema: `credentials(user_id, credential_id, public_key, sign_count, aaguid, created_at)`.
|
||||
4. `orca auth register` CLI: browser flow to register a new passkey.
|
||||
5. RP ID = cluster Traefik domain (from `orca auth init-idp --rp-id`); secure context via step-ca cert (R-017).
|
||||
6. Go tests using `go-webauthn` virtual-authenticator test helpers (no hardware key).
|
||||
- **Must-haves**: register + login flow works end-to-end against the bundled Dex; public keys only stored; virtual-authenticator tests pass.
|
||||
- **Commit**: `feat(P05): WebAuthn connector for Dex (REQ-148, D-240, C-38)`
|
||||
- **Tag**: `v0.11.5`
|
||||
|
||||
#### P06 — ACL rewrite to OIDC claims + enforcement — REQ-145, REQ-122, F1
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/acl/`, `internal/daemon/`, `internal/sshpush/`)
|
||||
- **Tasks**:
|
||||
1. Remove `KindToken` from `internal/acl/acl.go` entirely.
|
||||
2. Add `KindOidc`: maps `sub` + `groups` -> namespace permissions.
|
||||
3. `acl.Check` takes an OIDC claims struct (or SPIFFE SVID for machine identity).
|
||||
4. Wire `acl.Check` into daemon handlers (read/write/admin by route).
|
||||
5. Wire `acl.Check` into SSH-push applier: validate `ORCA_OIDC_TOKEN` env var against JWKS before applying any txn.
|
||||
6. `acl.json` file mode tightened to 0600.
|
||||
7. Deny-by-default enforced; actor recorded in audit log.
|
||||
8. Go tests: ACL-negative (unauthorized sub denied), ACL-positive, machine identity (SVID) still works.
|
||||
- **Must-haves**: no request applies without a valid OIDC token or SVID; `KindToken` removed; deny-by-default enforced.
|
||||
- **Commit**: `fix(P06): ACL rewrite to OIDC claims + enforcement (REQ-145, REQ-122, F1)`
|
||||
- **Tag**: `v0.11.6`
|
||||
|
||||
#### P07 — Remove all password/token paths — REQ-146, R-021, C-34
|
||||
|
||||
- **Persona**: lead-developer (territory: `internal/proxmox/`, `internal/stepca/`, `internal/cli/`)
|
||||
- **Tasks**:
|
||||
1. Remove `--password`/`$ORCA_PROXMOX_PASSWORD` from Proxmox join (`proxmox/bootstrap.go:29`); replace with pre-staged-key-only or `step ssh` OIDC cert exchange.
|
||||
2. Remove step-ca `--password-file` provisioner; migrate to OIDC provisioner (step-ca natively supports OIDC).
|
||||
3. Remove any bare-token CLI paths (already removed in P06, but sweep for stragglers).
|
||||
4. Add deprecation/migration docs: `--accept-identity-migration` flag on `orca upgrade` (P22 enforces).
|
||||
5. Go tests: `--password` flag is rejected with a clear error pointing to the migration guide.
|
||||
- **Must-haves**: no password accepted anywhere; `--password` rejected; step-ca OIDC provisioner works.
|
||||
- **Commit**: `fix(P07): remove all password/token paths (REQ-146, R-021, C-34) -- BREAKING`
|
||||
- **Tag**: `v0.11.7`
|
||||
|
||||
#### P08 — Master key seal-to-OIDC + Shamir — REQ-147, D-241, C-35
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/secrets/`, new `internal/seal/`)
|
||||
- **Tasks**:
|
||||
1. `internal/seal/seal.go`: seal/unseal using HKDF-SHA256 of OIDC ID token `sub` + fresh 32-byte salt; sealed blob at `ClusterDir()/master.key.sealed` (0600).
|
||||
2. Shamir 3-of-5: `internal/seal/shamir.go` (using `golang.org/x/crypto/...` or a vendored Shamir impl); print 5 shards at seal time.
|
||||
3. `orca cluster unseal`/`seal` CLI: unseal via OIDC auth; `--recovery` + 3 shards for IdP-lost case.
|
||||
4. mTLS-only offline path: seal key derived from cluster CA.
|
||||
5. Master key zeroed on shutdown (use `memguard` or manual `crypto/rand` overwrite).
|
||||
6. Go tests: seal -> unseal round-trip; recovery with 3 shards; 2 shards fails; raw key never on disk (assert no `master.key` file, only `master.key.sealed`).
|
||||
- **Must-haves**: raw master key never touches disk; unseal works via OIDC; recovery works with 3-of-5 shards.
|
||||
- **Commit**: `feat(P08): master key seal-to-OIDC + Shamir 3-of-5 (REQ-147, D-241, C-35)`
|
||||
- **Tag**: `v0.11.8`
|
||||
|
||||
### Wave C — Auth & integrity (backend-engineer + data-engineer)
|
||||
|
||||
#### P09 — Daemon auth hardening — REQ-123, REQ-124, F6, F24
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/daemon/`)
|
||||
- **Tasks**:
|
||||
1. Remove plaintext mode entirely (mandatory mTLS).
|
||||
2. Accept OIDC bearer as second factor on human-facing endpoints.
|
||||
3. `http.MaxBytesReader` on all JSON-decoding handlers; `MaxHeaderBytes` set.
|
||||
4. pprof loopback-only by default; `--pprof-allow-public` requires confirmation.
|
||||
5. Go tests: plaintext mode rejected; oversized body rejected; pprof non-loopback rejected.
|
||||
- **Commit**: `fix(P09): daemon auth hardening (REQ-123, REQ-124, F6, F24)`
|
||||
- **Tag**: `v0.11.9`
|
||||
|
||||
#### P10 — Audit log tamper-evidence — REQ-125, F2
|
||||
|
||||
- **Persona**: data-engineer (territory: `internal/audit/`, `internal/store/`)
|
||||
- **Tasks**:
|
||||
1. Add `prev_hash` + `entry_hash` columns to `audit_log` table (migration 0008).
|
||||
2. `AuditRepo.Append` computes `entry_hash = sha256(prev_hash || payload)`, stores it; HMAC-SHA256 under master key on the chain head (stored separately).
|
||||
3. SQLite trigger blocks UPDATE/DELETE on `audit_log`.
|
||||
4. `orca doctor audit` verifies the chain (recomputes hashes, checks HMAC).
|
||||
5. Actor field carries OIDC `sub` or SPIFFE SVID.
|
||||
6. Go tests: tamper detection (modify a row -> doctor fails); append-only enforcement (DELETE fails).
|
||||
- **Commit**: `fix(P10): audit log tamper-evidence (REQ-125, F2)`
|
||||
- **Tag**: `v0.11.10`
|
||||
|
||||
#### P11 — SVID chain validation — REQ-126, F9
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/identity/`)
|
||||
- **Tasks**:
|
||||
1. `VerifySVID` loads the CA pool (from `ClusterDir()/ca.crt` or step-ca root) and validates the full cert chain.
|
||||
2. Reject certs signed by unknown CAs even with correct URI SAN.
|
||||
3. Go tests: cert from wrong CA rejected; cert from correct CA + correct URI accepted; expired cert rejected.
|
||||
- **Commit**: `fix(P11): SVID chain validation (REQ-126, F9)`
|
||||
- **Tag**: `v0.11.11`
|
||||
|
||||
#### P12 — Backup symlink validation — REQ-127, F7
|
||||
|
||||
- **Persona**: data-engineer (territory: `internal/backup/`)
|
||||
- **Tasks**:
|
||||
1. `Restore` rejects `Linkname` that's absolute, contains `..`, or points outside `ORCA_HOME`.
|
||||
2. Regression test with crafted tarball containing a symlink to `/etc/shadow`.
|
||||
- **Commit**: `fix(P12): backup symlink validation (REQ-127, F7)`
|
||||
- **Tag**: `v0.11.12`
|
||||
|
||||
### Wave D — Crypto & secrets (backend-engineer)
|
||||
|
||||
#### P13 — step-ca /tmp hardening — REQ-128, F10
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/stepca/`, `internal/identity/`)
|
||||
- **Tasks**:
|
||||
1. `step ca certificate` writes to `0600` temp under `ClusterDir()/step-tmp/` (or `TMPDIR` override).
|
||||
2. Cleanup in `defer`; `mkdir -p` with 0700 on the temp dir.
|
||||
3. Go test: assert temp file mode is 0600; assert cleanup on success + failure.
|
||||
- **Commit**: `fix(P13): step-ca /tmp hardening (REQ-128, F10)`
|
||||
- **Tag**: `v0.11.13`
|
||||
|
||||
#### P14 — Master key rotation — REQ-129, F12, C-30
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/secrets/`, `internal/seal/`)
|
||||
- **Tasks**:
|
||||
1. `orca secrets rotate-master`: generate new master key, re-encrypt all namespace secrets, re-seal to OIDC.
|
||||
2. `--dry-run` reports affected namespaces without writing.
|
||||
3. Atomic per-namespace re-encryption; auto-rollback to old sealed key on any ns failure.
|
||||
4. Go tests: rotation succeeds; partial failure rolls back; dry-run doesn't write.
|
||||
- **Commit**: `fix(P14): master key rotation (REQ-129, F12, C-30)`
|
||||
- **Tag**: `v0.11.14`
|
||||
|
||||
#### P15 — File-mode audit expansion — REQ-130, F13
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/security/`, `internal/cli/doctor.go`)
|
||||
- **Tasks**:
|
||||
1. `EnforceFileModes` extended to SSH key, master key (sealed blob), server cert/key, known_hosts.
|
||||
2. `orca doctor modes` checks all.
|
||||
3. Startup refuses to run on violation.
|
||||
4. Go tests: looser mode -> doctor fails + startup refuses.
|
||||
- **Commit**: `fix(P15): file-mode audit expansion (REQ-130, F13)`
|
||||
- **Tag**: `v0.11.15`
|
||||
|
||||
### Wave E — OS scripts & emitters (backend-engineer + lead-developer)
|
||||
|
||||
#### P16 — aggregate.sh JSON injection + drift-gate fix — REQ-131, F11, F18
|
||||
|
||||
- **Persona**: lead-developer (territory: `scripts/`)
|
||||
- **Tasks**:
|
||||
1. Replace `printf` interpolation in `orca-aggregate.sh:64` with `jq`-based JSON construction (or a Go-side aggregator emitting JSON).
|
||||
2. Fix `orca-pull.sh` R-020 parsing to use `jq` instead of grep.
|
||||
3. Bats tests: malicious peer output doesn't corrupt `cluster.json`; drift gate correctly excludes acknowledged drift.
|
||||
- **Commit**: `fix(P16): aggregate.sh JSON injection + drift-gate fix (REQ-131, F11, F18)`
|
||||
- **Tag**: `v0.11.16`
|
||||
|
||||
#### P17 — install.sh checksum+GPG verification — REQ-132, F14
|
||||
|
||||
- **Persona**: lead-developer (territory: `scripts/install.sh`, `scripts/release.sh`)
|
||||
- **Tasks**:
|
||||
1. `release.sh` publishes `SHA256SUMS` + `SHA256SUMS.asc` (GPG-signed) alongside the tarball.
|
||||
2. `install.sh` verifies SHA256 + GPG signature before `tar -xzf`; fail closed on mismatch.
|
||||
3. `--no-verify` escape hatch (documented, warns).
|
||||
4. Bats tests: tampered tarball rejected; valid tarball accepted.
|
||||
- **Commit**: `fix(P17): install.sh checksum+GPG verification (REQ-132, F14)`
|
||||
- **Tag**: `v0.11.17`
|
||||
|
||||
#### P18 — nftables ruleset hardening — REQ-133, F21
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/emitter/nft.go`)
|
||||
- **Tasks**:
|
||||
1. Add conntrack bounds (`ct state established,related accept`).
|
||||
2. Input default-deny on the orca chain.
|
||||
3. Drop invalid packets (`ct state invalid drop`).
|
||||
4. `orca doctor nft` audits live ruleset against emitted one.
|
||||
5. Go tests: emitted ruleset contains the new rules; doctor detects drift.
|
||||
- **Commit**: `fix(P18): nftables ruleset hardening (REQ-133, F21)`
|
||||
- **Tag**: `v0.11.18`
|
||||
|
||||
#### P19 — sudoers hardening — REQ-134, F22
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/proxmox/bootstrap.go`)
|
||||
- **Tasks**:
|
||||
1. Add NOEXEC to `apt-get`/`dpkg` in the OrcaOperator sudoers (or remove if unused).
|
||||
2. `orca doctor proxmox` audits the sudoers file against the expected allowlist.
|
||||
3. Go tests: emitted sudoers has NOEXEC; doctor detects drift.
|
||||
- **Commit**: `fix(P19): sudoers hardening (REQ-134, F22)`
|
||||
- **Tag**: `v0.11.19`
|
||||
|
||||
#### P20 — System user consistency — REQ-135, F23
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/proxmox/bootstrap.go`, `internal/cli/peer_setup.go`)
|
||||
- **Tasks**:
|
||||
1. Proxmox bootstrap creates `nologin` system user (`-r -s /usr/sbin/nologin`), matching peer-setup.
|
||||
2. `orca doctor` flags inconsistency on existing peers.
|
||||
3. `orca upgrade` migrates existing `-m -s /bin/bash` users to `-r -s /usr/sbin/nologin`.
|
||||
4. Go tests: emitted useradd matches; doctor detects the old style.
|
||||
- **Commit**: `fix(P20): system user consistency (REQ-135, F23)`
|
||||
- **Tag**: `v0.11.20`
|
||||
|
||||
### Wave F — State storage & migration (data-engineer)
|
||||
|
||||
#### P21 — SQLite file-mode + at-rest encryption — REQ-136, F8, C-31
|
||||
|
||||
- **Persona**: data-engineer (territory: `internal/store/`)
|
||||
- **Tasks**:
|
||||
1. `store.Open` sets DB file mode 0600 (via `os.Chmod` after open, since SQLite creates with umask).
|
||||
2. Evaluate SQLCipher envelope (CGO-free check). If infeasible without CGO (breaks D-008), fall back to file-mode 0600 + documented threat per C-31.
|
||||
3. Document the decision in RESEARCH/PROJECT.
|
||||
4. Go tests: DB file mode is 0600 after open.
|
||||
- **Commit**: `fix(P21): SQLite file-mode + at-rest encryption (REQ-136, F8, C-31)`
|
||||
- **Tag**: `v0.11.21`
|
||||
|
||||
#### P22 — Migration safety + identity migration — REQ-137, F19, C-34
|
||||
|
||||
- **Persona**: data-engineer (territory: `internal/migration/`, `internal/cli/upgrade.go`)
|
||||
- **Tasks**:
|
||||
1. `copyFile` -> atomic temp+rename.
|
||||
2. `migrateDBSchema` runs in a transaction with `foreign_keys(ON)`.
|
||||
3. Pre-migration backup step (uses `internal/backup`).
|
||||
4. Document manual rollback (restore from backup).
|
||||
5. `orca upgrade` refuses v0.11 clusters using `--password`/bare-tokens without `--accept-identity-migration` (C-34).
|
||||
6. Go tests: migration is atomic; partial failure rolls back; `--accept-identity-migration` gate works.
|
||||
- **Commit**: `fix(P22): migration safety + identity migration (REQ-137, F19, C-34)`
|
||||
- **Tag**: `v0.11.22`
|
||||
|
||||
### Wave G — Dual-write closure (lead-developer, gated by C-29)
|
||||
|
||||
#### P23 — Legacy CA/mTLS/daemon + step-ca password-provisioner deletion — REQ-138, F16
|
||||
|
||||
- **Persona**: lead-developer (territory: `internal/security/ca.go`, `internal/transport/mtls.go`, `internal/daemon/`, `internal/stepca/`, `internal/certpaths/`)
|
||||
- **Pre-gate (C-29)**: P06, P08, P09, P11 must all be shipped.
|
||||
- **Tasks**:
|
||||
1. Remove `internal/security/ca.go` legacy CA; migrate `orca init` and `orca cert *` to step-ca exclusively.
|
||||
2. Remove `internal/transport/mtls.go` deprecated path.
|
||||
3. Remove daemon plaintext mode (already killed in P09, but delete the code path).
|
||||
4. Remove `internal/certpaths/` (v0.8 flat layout); `internal/paths/` is the only layout.
|
||||
5. Delete step-ca `--password-file` provisioner (already replaced by OIDC provisioner in P07).
|
||||
6. Full test suite must pass after deletion.
|
||||
- **Must-haves**: `orca init` + `orca cert *` work via step-ca only; no legacy code compiled.
|
||||
- **Commit**: `refactor(P23): delete legacy CA/mTLS/daemon + step-ca password-provisioner (REQ-138, F16, C-29)`
|
||||
- **Tag**: `v0.11.23`
|
||||
|
||||
### Wave H — Defense-in-depth (backend-engineer)
|
||||
|
||||
#### P24 — known_hosts tightening + transport hardening — REQ-139, F15, F25
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/security/flock.go`, `internal/sshpush/`)
|
||||
- **Tasks**:
|
||||
1. `Flock` tightens pre-existing looser perms to 0600 (chmod after open if looser).
|
||||
2. `classifyDialErr` switched from substring to typed errors (use `*ssh.ExitError`, `net.Error` type assertions).
|
||||
3. Add SSH-exec rate limiting (token bucket per peer, default 10 req/s).
|
||||
4. Go tests: looser perms tightened; typed errors classified correctly; rate limit enforced.
|
||||
- **Commit**: `fix(P24): known_hosts tightening + transport hardening (REQ-139, F15, F25)`
|
||||
- **Tag**: `v0.11.24`
|
||||
|
||||
#### P25 — Drift event authentication — REQ-140, F18
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/drift/`, `scripts/orca-drift-notify.sh`)
|
||||
- **Tasks**:
|
||||
1. Per-peer HMAC key (derived from master key via HKDF); deployed to peers at `0600` owned by `orca`.
|
||||
2. `orca-drift-notify.sh` signs each event with the HMAC; aggregator rejects unsigned/forged events.
|
||||
3. Go tests: forged event rejected; valid event accepted.
|
||||
- **Commit**: `fix(P25): drift event authentication (REQ-140, F18)`
|
||||
- **Tag**: `v0.11.25`
|
||||
|
||||
#### P26 — Security integration test suite — REQ-141, C-33
|
||||
|
||||
- **Persona**: backend-engineer (territory: `tests/`)
|
||||
- **Tasks**:
|
||||
1. Hermetic harness exercising: injection (P01), traversal (P02), symlink (P12), drift-forgery (P25), audit-tamper (P10), daemon-auth-negative (P09), OIDC mock-IdP flow (P04), ACL-with-OIDC-claims negative (P06), unseal/seal (P08), WebAuthn virtual-authenticator ceremony (P05), password-removal regression (P07 -- assert `--password` rejected).
|
||||
2. Gates in `.coreci.yml` `validate` pipeline (C-33).
|
||||
3. Bats + Go test runner.
|
||||
- **Commit**: `test(P26): security integration test suite (REQ-141, C-33)`
|
||||
- **Tag**: `v0.11.26`
|
||||
|
||||
### Wave I — Documentation & release (lead-developer)
|
||||
|
||||
#### P27 — Zero-trust + OIDC + WebAuthn + threat-model docs — REQ-142
|
||||
|
||||
- **Persona**: lead-developer (territory: `docs/`, `README.md`)
|
||||
- **Tasks**:
|
||||
1. `docs/threat-model.md`: STRIDE per component, zero-trust model, OIDC data-flow diagram, OS surface diagram, residual risk register.
|
||||
2. `docs/oidc.md`: configure your IdP, bundled Dex offline quickstart, claim-to-namespace mapping, BYO-IdP override.
|
||||
3. `docs/webauthn.md`: passkey registration, RP ID, secure context, recovery flow.
|
||||
4. `docs/security-runbook.md`: unseal/seal, master key rotation, incident response, sudoers audit, nft audit, Shamir recovery.
|
||||
5. README security section names "no orca credentials" as an invariant (R-021).
|
||||
- **Commit**: `docs(P27): zero-trust + OIDC + WebAuthn + threat-model docs (REQ-142)`
|
||||
- **Tag**: `v0.11.27`
|
||||
|
||||
#### P28 — Final review + ship + audit — REQ-143
|
||||
|
||||
- **Persona**: lead-developer (coordinates)
|
||||
- **Tasks**:
|
||||
1. `ciagent-review` multi-persona review across all phases.
|
||||
2. `ciagent-audit` reconstruction test (git log matches `.ciagent/` files).
|
||||
3. C-32 human-gate: confirm GITEA_TOKEN rotated + `.env` re-seeded (escalation hook if pending).
|
||||
4. Merge `phase/28` -> `milestone/v0.12-security-hardening`.
|
||||
5. Merge `milestone/v0.12-security-hardening` -> `main` (rebase-then-fast-forward).
|
||||
6. Tag `v0.11.28` (= v0.12 milestone release per feature-milestone rule).
|
||||
7. Create Gitea release with full milestone summary.
|
||||
8. Delete milestone + phase branches (tags preserve history).
|
||||
9. Update REQUIREMENTS.md (mark all v0.12 REQs complete) + ROADMAP.md (mark v0.12 complete).
|
||||
- **Commit**: `docs(milestone): complete v0.12 -- Security Hardening (Zero-Trust Identity) (29 phases shipped)`
|
||||
- **Tag**: `v0.11.28`
|
||||
@@ -1,462 +0,0 @@
|
||||
# PLAN v0.13: Production Hardening Round 2 + UAT Plan
|
||||
|
||||
**Status**: complete (2026-08-07). 14 phases (P0 + P01..P12 + P13
|
||||
final). Each phase ships a patch tag on the v0.12.x line. This plan
|
||||
references requirement IDs from REQUIREMENTS.md and follows the
|
||||
vertical-slice integrity rule (each phase is independently shippable).
|
||||
|
||||
## Phase 0: Pre-execution (this phase)
|
||||
|
||||
**Status**: complete. SPECIFY → CLARIFY → RESEARCH → IDEATE → PLAN →
|
||||
GRILL → SHIP. Ships as `v0.12.0`.
|
||||
|
||||
## Phase 1: Toolchain & dependency vulns (REQ-149)
|
||||
|
||||
**Tag**: `v0.12.1` | **Type**: fix | **Persona**: security-engineer
|
||||
|
||||
### Wave 1 (single task)
|
||||
- **T1**: Bump `go.mod` from `go 1.25.0` to `go 1.25.12` (or latest
|
||||
1.25.x). Run `go mod tidy`. Run `govulncheck -show verbose ./...` and
|
||||
triage the 6 imported third-party vulns. Bump any dep with a
|
||||
reachable trace (webauthn, cobra, modernc/sqlite, go-jose, coreos/
|
||||
go-oidc, x/crypto, oauth2). Verify `make build && make test && make
|
||||
lint` all pass.
|
||||
|
||||
### Must-haves
|
||||
- [ ] `go.mod` declares `go 1.25.12`+
|
||||
- [ ] `govulncheck ./...` reports zero stdlib vulns with call traces
|
||||
- [ ] `make build && make test && make lint` pass
|
||||
|
||||
## Phase 2: Input validation & injection hardening (REQ-150)
|
||||
|
||||
**Tag**: `v0.12.2` | **Type**: fix | **Persona**: backend-engineer
|
||||
|
||||
### Wave 1 (11 sub-fixes, all in `internal/`)
|
||||
- **T1**: `orca logs --job` — validate against `^[A-Za-z0-9_-]+$`;
|
||||
replace `fmt.Sprintf("journalctl -u %q", ...)` with `shellQuote`
|
||||
(critical: backtick RCE via SSH fanout)
|
||||
- **T2**: pprof `isLoopback(":6060")` — treat empty host as non-
|
||||
loopback/bind-all; reject unless explicit public-allow flag wired;
|
||||
remove phantom `--pprof-allow-public` references; make loopback-only
|
||||
a hard invariant
|
||||
- **T3**: backup restore tar-slip — replace `HasPrefix(name, "..")`
|
||||
with `filepath.Rel(target, dest)` containment check
|
||||
- **T4**: `orca txn rollback` — validate txn ID against `^T-[0-9a-f]{16}$`
|
||||
- **T5**: `orca nft diff --against` — validate txn ID before
|
||||
`filepath.Join`
|
||||
- **T6**: `drain stopAlloc` — validate `allocID` against
|
||||
`^[A-Za-z0-9_-]+$` before `systemctl stop`
|
||||
- **T7**: `cluster_compat` — `shellQuote(first)` for peer dir name
|
||||
- **T8**: `runtime/podman.go` — use `shellQuote(image)` not `%q`
|
||||
- **T9**: nft `TrustedProbes` — validate each entry with
|
||||
`net.ParseIP`/`net.ParseCIDR`; fix ipv4/ipv6 mismatch
|
||||
- **T10**: sudoers — validate `--proxmox-user`/`--proxmox-role` against
|
||||
`^[a-z_][a-z0-9_-]{0,31}$`; write to fixed `/etc/sudoers.d/orca`;
|
||||
`shellQuote` all pveum/useradd; `validateSudoers` check actual file
|
||||
- **T11**: `nft country block add` — validate `^[A-Z]{2}$`
|
||||
|
||||
### Wave 2 (tests)
|
||||
- **T12**: Add injection/traversal regression tests for each sub-fix;
|
||||
extend `tests/security_integration_test.go` with negative tests
|
||||
|
||||
### Must-haves
|
||||
- [ ] All 11 injection/traversal vectors fixed with validation
|
||||
- [ ] Regression tests for each vector
|
||||
- [ ] `tests/security_integration_test.go` passes
|
||||
|
||||
## Phase 3: Scheduler/deployment wiring + jobspec parser (REQ-151, REQ-152)
|
||||
|
||||
**Tag**: `v0.12.3` | **Type**: feat | **Persona**: lead-developer
|
||||
|
||||
### Wave 1 (jobspec parser fixes — REQ-152)
|
||||
- **T1**: Add `case "schedule":` and `case "timeout":` to top-level
|
||||
switch in `internal/jobspec/markdown.go`
|
||||
- **T2**: Fix DaemonSet — parser must not default `Count` to 1 for
|
||||
DaemonSet (validator rejects `Count != 0`)
|
||||
- **T3**: `restart:` policy → systemd `Restart=`/`StartLimitBurst` in
|
||||
`internal/emitter/systemd.go`
|
||||
- **T4**: Add `job lint` warnings for advisory-only fields (cron,
|
||||
health, update, affinity) — honest "not enforced in this version"
|
||||
|
||||
### Wave 2 (scheduler wiring — REQ-151)
|
||||
- **T5**: Wire `internal/scheduler.Schedule()` into `orca job run` —
|
||||
replace local `exec.CommandContext` path with: evaluate constraints/
|
||||
capacity/affinity → render systemd units → SSH-push to target
|
||||
- **T6**: `--target` overrides scheduler selection (manual pinning)
|
||||
- **T7**: Without `--target`, scheduler bin-packs across `ready` nodes
|
||||
- **T8**: Local fallback when no remote nodes registered (single-node
|
||||
dev mode — preserves backward compatibility)
|
||||
- **T9**: `systemd-analyze verify` on rendered unit before deploy
|
||||
|
||||
### Wave 3 (tests)
|
||||
- **T10**: Scheduler constraint/capacity/affinity enforcement tests
|
||||
- **T11**: DaemonSet spec passes lint and runs
|
||||
- **T12**: `timeout:` on Jobs enforced (kill after duration)
|
||||
- **T13**: Local fallback test (no remote nodes)
|
||||
|
||||
### Must-haves
|
||||
- [ ] `orca job run --target <node>` deploys via SSH-push to remote
|
||||
- [ ] Scheduler evaluates constraints/capacity/affinity
|
||||
- [ ] DaemonSet works (schedule parsed, Count correct)
|
||||
- [ ] `timeout:` enforced on Jobs
|
||||
- [ ] `restart:` translated to systemd unit
|
||||
- [ ] Local fallback when no remote nodes
|
||||
- [ ] `systemd-analyze verify` before deploy
|
||||
|
||||
## Phase 4: ACL enforcement + WebAuthn registration auth (REQ-153)
|
||||
|
||||
**Tag**: `v0.12.4` | **Type**: fix | **Persona**: backend-engineer
|
||||
|
||||
### Wave 1 (ACL wiring)
|
||||
- **T1**: Wire `acl.Check` into `dispatch_handler.go` — extract OIDC
|
||||
sub/SPIFFE SVID from mTLS peer cert, check against ACL
|
||||
- **T2**: Wire `acl.Check` into `jobs_handler.go`, `nodes_handler.go`,
|
||||
`tasks_handler.go`, `health_handler.go`
|
||||
- **T3**: Wire `acl.Check` into `internal/sshpush/` — validate
|
||||
`ORCA_OIDC_TOKEN` bearer against JWKS
|
||||
- **T4**: Wire `acl.Check` into `internal/txn/txn.go` apply path
|
||||
- **T5**: Thread OIDC sub/SVID into audit `actor` field
|
||||
- **T6**: Fix `acl.json` mode 0644→0600
|
||||
- **T7**: Add flock on `acl.json` for concurrent grant/revoke
|
||||
- **T8**: Bootstrap ACL: grant `cluster-admin` to init cert's SVID
|
||||
|
||||
### Wave 2 (WebAuthn registration auth)
|
||||
- **T9**: Fix WebAuthn unauthenticated registration — require existing
|
||||
session or admin bootstrap token; no overwriting existing creds
|
||||
without re-auth
|
||||
|
||||
### Wave 3 (tests)
|
||||
- **T10**: Extend `tests/security_integration_test.go` with deny-by-
|
||||
default enforcement test per handler
|
||||
- **T11**: WebAuthn registration auth test (unauthenticated rejected)
|
||||
|
||||
### Must-haves
|
||||
- [ ] `acl.Check` called in all 5 daemon handlers + sshpush + txn
|
||||
- [ ] `acl.json` mode 0600
|
||||
- [ ] Audit actor = OIDC sub/SVID
|
||||
- [ ] WebAuthn registration requires auth
|
||||
- [ ] Bootstrap ACL grants cluster-admin to init SVID
|
||||
- [ ] Deny-by-default enforcement tests pass
|
||||
|
||||
## Phase 5: Seal/audit CLI + chain race + key zeroing (REQ-154)
|
||||
|
||||
**Tag**: `v0.12.5` | **Type**: feat+fix | **Persona**: security-engineer
|
||||
|
||||
### Wave 1 (CLI commands)
|
||||
- **T1**: Implement `orca cluster seal`/`unseal` (wraps `internal/seal/`;
|
||||
OIDC token exchange; Shamir 3-of-5 shards; sealed blob 0600)
|
||||
- **T2**: Implement `orca doctor audit` (wraps `AuditRepo.VerifyChain`)
|
||||
- **T3**: Implement `orca doctor modes` (wraps `EnforceFileModes`)
|
||||
|
||||
### Wave 2 (fixes)
|
||||
- **T4**: Fix audit hash-chain race — `Append` uses `BEGIN IMMEDIATE`
|
||||
transaction
|
||||
- **T5**: Fix `secrets rotate-master` to actually re-seal to OIDC
|
||||
- **T6**: Zero master key / namespace keys / SVID private keys after
|
||||
use (defense-in-depth)
|
||||
|
||||
### Wave 3 (tests)
|
||||
- **T7**: Seal→unseal→secrets get round-trip test
|
||||
- **T8**: `doctor audit` tamper-detection test
|
||||
- **T9**: `doctor modes` 0644-rejection test
|
||||
- **T10**: Audit chain concurrent-write integrity test
|
||||
- **T11**: Key zeroing verification test
|
||||
|
||||
### Must-haves
|
||||
- [ ] `orca cluster seal`/`unseal` work (round-trip)
|
||||
- [ ] `orca doctor audit` verifies chain
|
||||
- [ ] `orca doctor modes` checks file modes
|
||||
- [ ] Audit chain survives concurrent appends
|
||||
- [ ] `secrets rotate-master` re-seals to OIDC
|
||||
- [ ] Keys zeroed after use
|
||||
|
||||
## Phase 6: auth init-idp real + auth register (REQ-155)
|
||||
|
||||
**Tag**: `v0.12.6` | **Type**: feat | **Persona**: security-engineer
|
||||
|
||||
### Wave 1
|
||||
- **T1**: Implement `orca auth init-idp` — render Dex systemd unit +
|
||||
config template + Traefik dynamic route from `internal/webauthn/`
|
||||
connector; RP ID = cluster Traefik domain; HTTPS via step-ca cert;
|
||||
atomic deploy with rollback
|
||||
- **T2**: Implement `orca auth register` (browser flow to WebAuthn
|
||||
registration endpoint)
|
||||
- **T3**: `loadOIDCConfig` config-file loading (`oidc.issuer` in config)
|
||||
- **T4**: `orca doctor oidc` health check
|
||||
|
||||
### Wave 2 (tests)
|
||||
- **T5**: Hermetic Dex+Traefik config render test
|
||||
- **T6**: `doctor oidc` health check test
|
||||
- **T7**: Virtual-authenticator WebAuthn flow test (C-38)
|
||||
|
||||
### Must-haves
|
||||
- [ ] `auth init-idp` deploys Dex+Traefik+systemd
|
||||
- [ ] `auth register` opens browser flow
|
||||
- [ ] `oidc.issuer` loadable from config file
|
||||
- [ ] `doctor oidc` health check works
|
||||
|
||||
## Phase 7: Concurrency safety (REQ-156)
|
||||
|
||||
**Tag**: `v0.12.7` | **Type**: fix | **Persona**: data-engineer + backend-engineer
|
||||
|
||||
### Wave 1 (SQLite)
|
||||
- **T1**: Add `busy_timeout(5000)` + `SetMaxOpenConns(1)` to all 4 DSNs
|
||||
(store, cache, recovery, webauthn)
|
||||
|
||||
### Wave 2 (flocks + locks)
|
||||
- **T2**: Secrets file flock (concurrent `secrets set` on same ns)
|
||||
- **T3**: Upgrade lock file (refuse concurrent `orca upgrade`)
|
||||
- **T4**: Backup lock file
|
||||
- **T5**: Cache invalidation by write commands (node join/leave, ns
|
||||
create/delete, job run/stop)
|
||||
- **T6**: `Executor.Run` mutex scope fix (hold only for DB inserts)
|
||||
- **T7**: `ns create` atomic dir+ns.md write
|
||||
- **T8**: `writeCurrentLead` atomic write
|
||||
- **T9**: Consolidate 3 divergent `writeAtomic` impls onto
|
||||
`security.WriteAtomic`
|
||||
- **T10**: WebAuthn session stores guarded with `sync.Mutex`
|
||||
|
||||
### Wave 3 (tests)
|
||||
- **T11**: Concurrent secrets set test (no data loss)
|
||||
- **T12**: Concurrent upgrade rejection test
|
||||
- **T13**: Cache invalidation read-after-write test
|
||||
- **T14**: SQLite concurrent writer test (no "database is locked")
|
||||
|
||||
### Must-haves
|
||||
- [ ] All SQLite DSNs have busy_timeout
|
||||
- [ ] Concurrent secrets set preserves all writes
|
||||
- [ ] Concurrent upgrade rejected
|
||||
- [ ] Cache invalidated by writes (read-after-write consistency)
|
||||
- [ ] WebAuthn session stores thread-safe
|
||||
|
||||
## Phase 8: Transport & SSH safety (REQ-157)
|
||||
|
||||
**Tag**: `v0.12.8` | **Type**: fix | **Persona**: backend-engineer
|
||||
|
||||
### Wave 1
|
||||
- **T1**: Replace substring matching in `transport.IsTransient` AND
|
||||
`sshpush.isTransient` with typed sentinels (`errors.Is`)
|
||||
- **T2**: `rotateSSHKeys` 2-phase atomic swap
|
||||
- **T3**: `known_hosts` flock field read by `dial()`
|
||||
- **T4**: IPv6 `net.JoinHostPort` in proxmox SSH dial + drain
|
||||
`splitHostPort`
|
||||
- **T5**: Explicit timeouts for peer-setup, drift remediate/ack, txn
|
||||
rollback, job restart
|
||||
- **T6**: `verifyCutover` use `security.ClientTLSConfig` with orca CA
|
||||
- **T7**: OIDC callback server `ReadHeaderTimeout: 5s`
|
||||
- **T8**: Root SIGINT/SIGTERM handler for non-watch commands
|
||||
|
||||
### Wave 2 (tests)
|
||||
- **T9**: Typed-error classification test
|
||||
- **T10**: rotate-lead 2-phase with partial-peer failure test
|
||||
- **T11**: IPv6 SSH dial test
|
||||
- **T12**: Signal handling clean-exit test
|
||||
|
||||
### Must-haves
|
||||
- [ ] No substring matching in transport retry logic
|
||||
- [ ] rotateSSHKeys atomic 2-phase
|
||||
- [ ] IPv6 addresses work in SSH dial
|
||||
- [ ] All SSH commands have explicit timeouts
|
||||
- [ ] SIGINT/SIGTERM triggers clean exit
|
||||
|
||||
## Phase 9: Migration & operational safety (REQ-158)
|
||||
|
||||
**Tag**: `v0.12.9` | **Type**: fix | **Persona**: data-engineer
|
||||
|
||||
### Wave 1
|
||||
- **T1**: Migration transaction + torn-write fix
|
||||
- **T2**: `job stop` real `systemctl stop` via SSH
|
||||
- **T3**: DB retention/compaction for jobs/tasks/audit_log
|
||||
- **T4**: `orca logs --lines` cap + `--since` upper bound
|
||||
- **T5**: Cache DB mode 0600
|
||||
- **T6**: `upgrade.go` cutover backup-file + atomic-rename
|
||||
|
||||
### Wave 2 (tests)
|
||||
- **T7**: Migration transaction-rollback test
|
||||
- **T8**: `job stop` actually-stops test
|
||||
- **T9**: DB retention compaction test
|
||||
- **T10**: Logs `--lines` cap test
|
||||
|
||||
### Must-haves
|
||||
- [ ] Migration is transactional + recoverable from torn write
|
||||
- [ ] `job stop` sends `systemctl stop` via SSH
|
||||
- [ ] DB retention prevents unbounded growth
|
||||
- [ ] Logs output is bounded
|
||||
|
||||
## Phase 10: Observability & metrics (REQ-159)
|
||||
|
||||
**Tag**: `v0.12.10` | **Type**: feat | **Persona**: backend-engineer
|
||||
|
||||
### Wave 1
|
||||
- **T1**: Add metrics: `orca_jobs_by_state`, `orca_drift_events_total`,
|
||||
`orca_ssh_errors_total`, `orca_txn_apply_total`,
|
||||
`orca_txn_rollback_total`, `orca_acl_denials_total`,
|
||||
`orca_audit_chain_head`
|
||||
- **T2**: New `docs/metrics.md` with Prometheus scrape config
|
||||
- **T3**: Security headers middleware on daemon
|
||||
|
||||
### Wave 2 (tests)
|
||||
- **T4**: Metric exposition format + counter increment tests
|
||||
|
||||
### Must-haves
|
||||
- [ ] 7 new metrics exposed at /metrics
|
||||
- [ ] `docs/metrics.md` exists
|
||||
- [ ] Security headers set on daemon responses
|
||||
|
||||
## Phase 11: Doc drift round 2 (REQ-160)
|
||||
|
||||
**Tag**: `v0.12.11` | **Type**: docs | **Persona**: lead-developer
|
||||
|
||||
### Wave 1 (README + CHANGELOG)
|
||||
- **T1**: README — update status banner, latest tag, subcommand table
|
||||
(add auth/nft/peer-setup/secrets rotate-master), correct "mTLS by
|
||||
default" claim, add missing docs to table
|
||||
- **T2**: CHANGELOG regen
|
||||
|
||||
### Wave 2 (docs/*)
|
||||
- **T3**: `docs/cli.md` — complete rewrite covering all ~40 subcommands
|
||||
- **T4**: `docs/webauthn.md` — add `auth register`
|
||||
- **T5**: `docs/namespace.md` — add inherit/set-constraint
|
||||
- **T6**: `docs/install.md`+`docker.md` — update version refs
|
||||
- **T7**: `docs/security-runbook.md` — match P05 reality
|
||||
- **T8**: `docs/security-scanning.md` — gosec.json
|
||||
|
||||
### Wave 3 (code-level doc fixes)
|
||||
- **T9**: Fix `verify-reqs` bold-format regex (bypasses v0.12)
|
||||
- **T10**: Fix ROADMAP/REQUIREMENTS v0.12 status hygiene
|
||||
- **T11**: `internal/proxmox/bootstrap.go` comments (password→key auth)
|
||||
- **T12**: Deprecate `orca status` stub
|
||||
- **T13**: Help text fixes (`job run` HCL→markdown, `job stop`
|
||||
daemon→SSH-push)
|
||||
- **T14**: `make verify-docs` target (cli.md ↔ `orca --help`)
|
||||
|
||||
### Must-haves
|
||||
- [ ] README accurate (status, tag, subcommands, claims)
|
||||
- [ ] `docs/cli.md` covers all subcommands
|
||||
- [ ] `verify-reqs` works for v0.12 and v0.13
|
||||
- [ ] `make verify-docs` passes
|
||||
|
||||
## Phase 12: --type linux + UAT plan + signoff (REQ-161, REQ-162, REQ-163)
|
||||
|
||||
**Tag**: `v0.12.12` | **Type**: feat | **Persona**: lead-developer + uat-engineer
|
||||
|
||||
### Wave 1 (--type linux — REQ-161)
|
||||
- **T1**: Implement `internal/linux/bootstrap.go` (mirrors Proxmox
|
||||
pattern without PVE role/sudoers)
|
||||
- **T2**: Wire `orca node join --type linux --host <ip> --ssh-user root
|
||||
--ssh-key <path>`
|
||||
|
||||
### Wave 2 (UAT plan — REQ-162)
|
||||
- **T3**: Write `docs/uat.md` — 3-host topology, step-by-step, claim
|
||||
matrix (~35 claims), signoff procedure
|
||||
|
||||
### Wave 3 (UAT signoff — REQ-163)
|
||||
- **T4**: Write `scripts/uat-signoff.sh` — ~35 named assertions,
|
||||
idempotent, read-only, exit 0 iff all pass
|
||||
- **T5**: Write `scripts/uat-smoke.sh` — pure-CLI subset for CI validate
|
||||
|
||||
### Wave 4 (tests)
|
||||
- **T6**: `--type linux` bootstrap round-trip test (mock SSH)
|
||||
- **T7**: `uat-signoff.sh` syntax + assertion-count test
|
||||
- **T8**: `uat-smoke.sh` in `.coreci.yml` validate
|
||||
|
||||
### Must-haves
|
||||
- [ ] `orca node join --type linux` works (SSH bootstrap)
|
||||
- [ ] `docs/uat.md` covers 3-host topology + all claims
|
||||
- [ ] `scripts/uat-signoff.sh` has ~35 assertions, idempotent
|
||||
- [ ] `scripts/uat-smoke.sh` runs in CI
|
||||
|
||||
## Phase 13: Final review + ship + audit
|
||||
|
||||
**Tag**: `v0.12.13` = v0.13 milestone release | **Type**: chore
|
||||
|
||||
### Wave 1
|
||||
- **T1**: `ciagent-review` — multi-persona code review across P01..P12
|
||||
- **T2**: `ciagent-audit` — reconstruction test, branch hygiene, commit
|
||||
discipline; fix any remaining verify-reqs discrepancies
|
||||
- **T3**: Update REQUIREMENTS.md — mark all v0.13 REQs as complete
|
||||
- **T4**: Update ROADMAP.md — mark v0.13 as **COMPLETE**
|
||||
- **T5**: Merge `phase/13` → `milestone/v0.13` → `main`
|
||||
- **T6**: Tag `v0.12.13` (milestone release)
|
||||
- **T7**: Create release with full milestone summary
|
||||
|
||||
### Must-haves
|
||||
- [ ] All v0.13 REQs marked complete in REQUIREMENTS.md
|
||||
- [ ] ROADMAP.md marks v0.13 COMPLETE (with bold)
|
||||
- [ ] `verify-reqs` passes for v0.12 and v0.13
|
||||
- [ ] Milestone merged to main
|
||||
- [ ] `v0.12.13` tag created
|
||||
- [ ] v1.0.0 NOT cut (deferred for UAT signoff)
|
||||
|
||||
## Wave ordering summary
|
||||
|
||||
| Phase | Waves | Tasks | Depends on |
|
||||
|-------|-------|-------|------------|
|
||||
| P01 | 1 | 1 | P0 |
|
||||
| P02 | 2 | 12 | P0 |
|
||||
| P03 | 3 | 13 | P0 |
|
||||
| P04 | 3 | 11 | P0 (P03 for scheduler context) |
|
||||
| P05 | 3 | 11 | P0 |
|
||||
| P06 | 2 | 7 | P05 (seal) |
|
||||
| P07 | 3 | 14 | P0 |
|
||||
| P08 | 2 | 12 | P0 |
|
||||
| P09 | 2 | 10 | P0 |
|
||||
| P10 | 2 | 4 | P04 (acl denials metric), P05 (audit chain head) |
|
||||
| P11 | 3 | 14 | P01..P10 (docs reflect reality) |
|
||||
| P12 | 4 | 8 | P03 (scheduler for UAT), P04 (ACL for UAT) |
|
||||
| P13 | 1 | 7 | P01..P12 |
|
||||
|
||||
## Vertical slice integrity
|
||||
|
||||
Each phase is independently shippable:
|
||||
- P01 (toolchain) — bumps go version, no API change
|
||||
- P02 (injection) — validates inputs, no API change
|
||||
- P03 (scheduler) — changes `job run` behavior (local→remote), local
|
||||
fallback preserves backward compat
|
||||
- P04 (ACL) — adds enforcement, bootstrap ACL prevents lockout
|
||||
- P05 (seal) — adds new CLI commands, no breaking change
|
||||
- P06 (init-idp) — replaces stub, no breaking change
|
||||
- P07 (concurrency) — adds locks/timeouts, no API change
|
||||
- P08 (transport) — replaces substring with typed errors, no API change
|
||||
- P09 (migration) — fixes migration safety + job stop, job stop is
|
||||
behavioral change (soft→hard stop) — documented
|
||||
- P10 (metrics) — adds metrics, no API change
|
||||
- P11 (docs) — docs only, no code behavior change
|
||||
- P12 (UAT) — adds new command + docs + scripts, no breaking change
|
||||
- P13 (final) — review + ship, no new features
|
||||
|
||||
## Grill binding conditions (C-44..C-49) — incorporated
|
||||
|
||||
| ID | Condition | Phase affected | How addressed |
|
||||
|----|-----------|----------------|---------------|
|
||||
| C-44 | P03 MUST fail-closed when scheduler selects a node but SSH-push fails. Local fallback only when `len(registeredNodes)==0`. Test case mandatory. | P03 | Added to P03 must-haves + T13 test |
|
||||
| C-45 | P04 MUST implement log-only/dry-run mode as default for first invocation after ACL wiring. Enforce mode after bootstrap ACL verified. | P04 | Added T9.5 (log-only mode) + T11.5 (enforce-mode toggle) to P04 |
|
||||
| C-46 | P12 dependency table MUST include P05 (seal) and P06 (auth init-idp) in addition to P03 and P04. | P12 | Updated dependency table above |
|
||||
| C-47 | P12 `uat-signoff.sh` MUST include explicit assertions for: (a) job deployed to remote node, (b) ACL deny-by-default, (c) seal/unseal round-trip, (d) OIDC health check. | P12 | Added to P12 must-haves + assertion list in docs/uat.md |
|
||||
| C-48 | P12 `docs/uat.md` MUST document hardware prerequisites (Proxmox VE 8/9 host required). Alternative UAT path (3x Ubuntu, `--type linux` only, Proxmox claims skipped) MUST be documented. | P12 | Added to P12 T3 scope |
|
||||
| C-49 | Plan narrative MUST soften "last hardening round" to "last hardening round before UAT validation." | P0/P13 | Updated PROJECT.md + ROADMAP.md narrative |
|
||||
|
||||
### Updated P03 must-haves (C-44)
|
||||
- [ ] P03 fails-closed when scheduler selects a node but SSH-push fails (returns error, no silent local fallback)
|
||||
- [ ] Local fallback ONLY when `len(registeredNodes)==0`
|
||||
- [ ] Test case for SSH-push failure → error (not silent local)
|
||||
|
||||
### Updated P04 task list (C-45)
|
||||
- **T9.5**: Implement log-only/dry-run mode as default for first invocation after ACL wiring (log denials, do not block)
|
||||
- **T11.5**: Enforce mode after bootstrap ACL verified (toggle via `orca acl enforce` or config)
|
||||
|
||||
### Updated P12 dependencies (C-46)
|
||||
- P12 depends on: P03 (scheduler), P04 (ACL), P05 (seal), P06 (auth init-idp)
|
||||
|
||||
### Updated P12 must-haves (C-47, C-48)
|
||||
- [ ] `uat-signoff.sh` asserts: job deployed to remote node (node_id != localhost)
|
||||
- [ ] `uat-signoff.sh` asserts: ACL deny-by-default (denial logged)
|
||||
- [ ] `uat-signoff.sh` asserts: seal/unseal round-trip
|
||||
- [ ] `uat-signoff.sh` asserts: OIDC health check
|
||||
- [ ] `docs/uat.md` documents Proxmox VE 8/9 hardware prerequisite
|
||||
- [ ] `docs/uat.md` documents alternative UAT path (3x Ubuntu, Proxmox claims skipped)
|
||||
|
||||
### Updated narrative (C-49)
|
||||
v0.13 is the "last hardening round **before UAT validation**." The UAT
|
||||
will likely surface 3-7 issues requiring a patch release. v1.0.0 is
|
||||
deferred until UAT passes.
|
||||
@@ -1,340 +0,0 @@
|
||||
# PLAN v0.14: Ingress Bootstrap Completeness
|
||||
|
||||
**Status**: active. 9 phases (P0 + P1..P7 + P8 final). Each phase ships a
|
||||
patch tag on the v0.13.x line. This plan references requirement IDs from
|
||||
REQUIREMENTS.md and follows the vertical-slice integrity rule (each phase
|
||||
is independently shippable).
|
||||
|
||||
**Research-validated decisions** (from RESEARCH_v0.14.md + GRILL_v0.14.md):
|
||||
|
||||
- nft postrouting: `ip saddr 127.0.0.0/8 oifname != "lo" masquerade`
|
||||
- nft first-apply: pre-create table (`nft add table inet orca-ingress 2>/dev/null || true`) before `nft -f`
|
||||
- pve-firewall: shift orca input/forward chains to `priority -10` (before pve-firewall's 0)
|
||||
- LXC features: `nesting=1,keyctl=1,fuse=1` (fuse=1 for fuse-overlayfs)
|
||||
- traefik TLS: **drop `certResolver: orca`** — does not exist in v3.3; emit `tls: {}` for v0.14 (real mTLS via dynamic `tls.certificates` + `clientAuth.caFiles` deferred to v0.15 — grill G-003 confidence 0.55 < 0.60 threshold, auto-resolved to defer)
|
||||
- podman restart: `--restart=unless-stopped` + enable `podman-restart.service`
|
||||
- volumes: omit `:Z` flag, use `:ro` on both mounts
|
||||
- traefik image: `FROM traefik:v3.3.0`, `ENTRYPOINT ["/traefik"]` inherited, `CMD ["--configFile=/etc/traefik/traefik.yml"]`
|
||||
- `NftClusterConfig.DNATTarget`: default `127.0.0.1:8443`/`:8080`; proxmox-native = `<lxc-ip>:8443`/`:8080`
|
||||
- Migration: `0009_ingress_mode.sql` (NOT 0007 — already taken by certs_serial_unique)
|
||||
- CA path: `certpaths.CACertPath()` (NOT `certpaths.CAPath()` — does not exist)
|
||||
- Upgrade path: P2 must detect+remove legacy `orca-traefik.service` + `/usr/local/bin/traefik` before starting podman container (C-57)
|
||||
- Podman install: `BootstrapLocalIngress` and `BootstrapRemoteIngress` must install podman if absent (C-50)
|
||||
- Offline-first: `podman pull` requires registry reachability — documented exception to R-001 for ingress bootstrap (C-54)
|
||||
- Static config: mount from host (not baked) to preserve `traefik-on-public-ip` opt-out (C-58)
|
||||
- DNATTarget validation: `net.ParseIP` or `ip:port` parse before render (C-51)
|
||||
- apt-get idempotency: `command -v podman` check before install (C-53)
|
||||
- MAC collision: check against existing nodes' MACs (C-56)
|
||||
- Native-mode nft: first apply uses LXC IP (not default 127.0.0.1) — discover LXC IP before first nft apply (C-55)
|
||||
|
||||
## Phase 0: Pre-execution (this phase)
|
||||
|
||||
**Status**: complete. SPECIFY → CLARIFY → RESEARCH → PLAN → GRILL → SHIP.
|
||||
Ships as `v0.13.0`.
|
||||
|
||||
## Phase 1: `orca-traefik` container image + release pipeline (REQ-171)
|
||||
|
||||
**Tag**: `v0.13.1` | **Type**: feat | **Persona**: release-engineer (phase-specific) + backend-engineer
|
||||
|
||||
### Wave 1 (image)
|
||||
- **T1**: Create `docker/orca-traefik/traefik.yml` — the **default** static config baked into the image (used when no host-side override is mounted):
|
||||
```yaml
|
||||
entryPoints:
|
||||
websecure:
|
||||
address: "127.0.0.1:8443"
|
||||
web:
|
||||
address: "127.0.0.1:8080"
|
||||
traefik:
|
||||
address: "127.0.0.1:8081"
|
||||
providers:
|
||||
file:
|
||||
directory: "/etc/traefik/dynamic"
|
||||
watch: true
|
||||
log:
|
||||
level: INFO
|
||||
format: json
|
||||
accessLog:
|
||||
format: json
|
||||
```
|
||||
No `certificatesResolvers` (research finding: does not exist for CA-based; TLS is via dynamic config).
|
||||
**C-58**: The baked config is a default. The podman run command also mounts a host-side `/etc/traefik/traefik.yml` if it exists (overriding the baked one), preserving the `traefik-on-public-ip` opt-out (REQ-100). The reconciler renders the static config via `emitter.RenderTraefikStaticConfig` to `/etc/traefik/traefik.yml` on the host, then mounts it `-v /etc/traefik/traefik.yml:/etc/traefik/traefik.yml:ro`. This way `PublicBinding` opt-out still works.
|
||||
- **T2**: Create `Dockerfile.traefik` at repo root:
|
||||
```dockerfile
|
||||
FROM traefik:v3.3.0
|
||||
LABEL org.opencontainers.image.title="orca-traefik"
|
||||
LABEL org.opencontainers.image.source="https://git.cloudinit.dev/coreci/orca"
|
||||
COPY docker/orca-traefik/traefik.yml /etc/traefik/traefik.yml
|
||||
CMD ["--configFile=/etc/traefik/traefik.yml"]
|
||||
```
|
||||
(ENTRYPOINT inherited as `["/traefik"]` from base image.)
|
||||
- **T3**: Create placeholder `docker/orca-traefik/step-ca-root.crt` (empty file) — real CA is volume-mounted at runtime. If absent, traefik starts without TLS termination (graceful).
|
||||
|
||||
### Wave 2 (release pipeline)
|
||||
- **T4**: `scripts/release.sh` — add a second docker block after the existing one (~line 212):
|
||||
```bash
|
||||
# Build + push orca-traefik image
|
||||
TRAEFIK_IMAGE="${CONTAINER_REGISTRY}/${CONTAINER_OWNER}/orca-traefik"
|
||||
if command -v docker >/dev/null 2>&1; then
|
||||
docker build -f Dockerfile.traefik -t "${TRAEFIK_IMAGE}:${VERSION}" -t "${TRAEFIK_IMAGE}:latest" .
|
||||
docker push "${TRAEFIK_IMAGE}:${VERSION}"
|
||||
docker push "${TRAEFIK_IMAGE}:latest"
|
||||
fi
|
||||
```
|
||||
- **T5**: `.coreci.yml` — add `container-publish-traefik` step mirroring `container-publish` with `CONTAINER_IMAGE=orca-traefik` + `DOCKERFILE=Dockerfile.traefik`.
|
||||
|
||||
### Wave 3 (tests)
|
||||
- **T6**: Verify `docker build -f Dockerfile.traefik .` succeeds and the resulting image starts traefik with `--configFile=/etc/traefik/traefik.yml` (can test with `docker run --rm orca-traefik --version`).
|
||||
|
||||
### Must-haves
|
||||
- [ ] `Dockerfile.traefik` builds successfully
|
||||
- [ ] Image starts traefik with the baked static config
|
||||
- [ ] `release.sh` publishes `orca-traefik:<version>` + `:latest`
|
||||
- [ ] `.coreci.yml` has `container-publish-traefik` step
|
||||
|
||||
## Phase 2: Podman traefik reconciler (REQ-172)
|
||||
|
||||
**Tag**: `v0.13.2` | **Type**: feat | **Persona**: lead-developer
|
||||
|
||||
### Wave 1 (reconciler)
|
||||
- **T1**: Rewrite `internal/traefik/install.go` — replace binary+systemd install with podman container reconciler:
|
||||
- `EnsureTraefikContainer(ctx, execFn, image, tag)` — idempotent:
|
||||
1. `podman inspect orca-traefik` → if running, no-op; if stopped, `podman start orca-traefik`; if absent, go to step 2
|
||||
2. `mkdir -p /etc/traefik/dynamic /etc/orca`
|
||||
3. `podman pull <image>:<tag>`
|
||||
4. `podman run -d --name orca-traefik --restart=unless-stopped --network host -v /etc/traefik/dynamic:/etc/traefik/dynamic:ro -v /etc/orca/step-ca-root.crt:/etc/orca/step-ca-root.crt:ro <image>:<tag>`
|
||||
- `EnsureTraefikContainerLocal(ctx, image, tag)` — uses `exec.CommandContext("podman", ...)` locally
|
||||
- `EnsureTraefikContainerRemote(ctx, execFn, image, tag)` — uses SSH exec function
|
||||
- Image/tag resolution: `git.cloudinit.dev/coreci/orca-traefik:<version>` where version = `internal/cli.version` (or `latest` if dev)
|
||||
- **Remove** systemd unit generation + `systemctl enable`
|
||||
- **T2**: Add `podman-restart.service` enable step: `systemctl enable --now podman-restart.service` (research finding: needed for reboot persistence)
|
||||
- **T2a**: **C-50**: `EnsureTraefikContainerLocal`/`Remote` must check `command -v podman` first. If absent: on localhost, attempt `apt-get install -y podman` (or fail with clear install instructions if no apt). On remote, `apt-get install -y podman conmon crun fuse-overlayfs` via SSH. Non-fatal warn if podman unavailable (offline host) — traefik won't start but `orca init` succeeds (same tolerance as v0.13).
|
||||
|
||||
### Wave 2 (callsite updates + v0.13 upgrade path)
|
||||
- **T3**: `internal/cli/init.go:254-266` — replace `installTraefikLocal()` with `EnsureTraefikContainerLocal`
|
||||
- **T4**: `internal/linux/bootstrap.go:160-172` — replace `traefik.InstallRemote` with `EnsureTraefikContainerRemote`
|
||||
- **T5**: `internal/proxmox/bootstrap.go:250-255` — replace `traefik.InstallRemote` with `EnsureTraefikContainerRemote` (for native mode; floating-IP calls it inside the LXC in P6)
|
||||
- **T6**: **C-57 (v0.13→v0.14 upgrade path)**: `internal/cli/upgrade.go` — rewrite the Traefik cutover to:
|
||||
1. Detect legacy `orca-traefik.service`: `systemctl is-active orca-traefik.service`
|
||||
2. If active: `systemctl stop orca-traefik.service && systemctl disable orca-traefik.service`
|
||||
3. Remove `/etc/systemd/system/orca-traefik.service` + `/usr/local/bin/traefik` (if exists)
|
||||
4. `systemctl daemon-reload`
|
||||
5. Render static config via `emitter.RenderTraefikStaticConfig` to `/etc/traefik/traefik.yml`
|
||||
6. `EnsureTraefikContainerLocal` (pull + run podman container)
|
||||
7. Idempotent: if no legacy unit, skip steps 1-4
|
||||
- **T7**: `internal/cli/traefik_install.go` — update CLI wrapper
|
||||
|
||||
### Wave 3 (TLS model fix — research finding)
|
||||
- **T8**: `internal/emitter/traefik.go` — drop `certResolver: orca` from the dynamic config router TLS stanza (line ~185-188). Replace with `tls: {}` (empty TLS stanza — traefik uses its default cert). Document that real mTLS via `tls.certificates` + `tls.options.default.clientAuth.caFiles` will be wired when step-ca mints certs into the dynamic dir (post-v0.14 or v1.x).
|
||||
- **T9**: Update `internal/emitter/traefik_test.go` — remove assertion for `certResolver: orca`, add assertion for `tls: {}` presence.
|
||||
|
||||
### Wave 4 (tests)
|
||||
- **T10**: Create `internal/traefik/install_test.go` (new file — F1.3: does not exist today) — assert `podman run` is invoked (not `curl|tar`), `--restart=unless-stopped --network host` present, volume mounts present, `podman-restart.service` enabled.
|
||||
- **T10a**: **C-57/F6.2**: Add v0.13→v0.14 upgrade test: simulate a host with `orca-traefik.service` present (fake), run upgrade, assert unit stopped+disabled+removed, podman container running.
|
||||
|
||||
### Must-haves
|
||||
- [ ] `orca init` → `podman inspect orca-traefik` shows running
|
||||
- [ ] `podman logs orca-traefik` shows traefik started with baked config
|
||||
- [ ] No systemd `orca-traefik.service` generated
|
||||
- [ ] `--restart=unless-stopped` + `podman-restart.service` enabled
|
||||
- [ ] `certResolver: orca` removed from dynamic config
|
||||
|
||||
## Phase 3: nft SNAT+DNAT + `orca init` ingress bootstrap (REQ-173)
|
||||
|
||||
**Tag**: `v0.13.3` | **Type**: feat | **Persona**: lead-developer + security-engineer
|
||||
|
||||
### Wave 1 (nft emitter extension)
|
||||
- **T1**: `internal/emitter/nft.go` — add `DNATTarget` field to `NftClusterConfig` (default `127.0.0.1`). Render DNAT rules as `dnat to <DNATTarget>:8443` / `dnat to <DNATTarget>:8080`. **C-51**: Validate `DNATTarget` with `net.ParseIP` before rendering. Reject invalid values with error (same F9 injection guard pattern as `partitionTrustedProbes`).
|
||||
- **T2**: `internal/emitter/nft.go` — add `EnableSNAT bool` (default true) + `postrouting` chain:
|
||||
```nft
|
||||
chain postrouting {
|
||||
type nat hook postrouting priority 100; policy accept;
|
||||
ip saddr 127.0.0.0/8 oifname != "lo" masquerade
|
||||
}
|
||||
```
|
||||
Only when `EnableSNAT == true`.
|
||||
- **T3**: `internal/emitter/nft.go` — shift `input` and `forward` chain priorities from `filter` (=0) to `-10` (research finding: avoids pve-firewall same-priority undefined order).
|
||||
- **T4**: `internal/emitter/nft.go` — fix first-apply flush-table bug: change `flush table inet orca-ingress` to `delete table inet orca-ingress` (nft ≥1.0 treats delete-of-missing as warning in `-f` mode). If that's version-unsafe, the apply step (T7) pre-creates the table.
|
||||
|
||||
### Wave 2 (ingress bootstrap)
|
||||
- **T5**: New `internal/ingress/bootstrap.go`:
|
||||
- `BootstrapLocalIngress(ctx)`:
|
||||
1. `mkdir -p /etc/traefik/dynamic /etc/orca`
|
||||
2. **C-60**: Push cluster root CA to `/etc/orca/step-ca-root.crt` from `certpaths.CACertPath()` (if exists, else empty placeholder)
|
||||
3. Render static config via `emitter.RenderTraefikStaticConfig` to `/etc/traefik/traefik.yml` (preserves `traefik-on-public-ip` opt-out — C-58)
|
||||
4. Render `orca.nft` via `NftEmitter.RenderNftConfig` + write to `/etc/nftables.d/orca.nft`
|
||||
5. Pre-create table: `nft add table inet orca-ingress 2>/dev/null || true`
|
||||
6. Apply: `nft -f /etc/nftables.d/orca.nft`
|
||||
7. **C-50**: Ensure podman installed (check `command -v podman`, install if absent)
|
||||
8. `EnsureTraefikContainerLocal` (from P2) — mounts `/etc/traefik/traefik.yml:ro` + `/etc/traefik/dynamic:ro` + `/etc/orca/step-ca-root.crt:ro`
|
||||
- Each step non-fatal warn (offline host tolerance)
|
||||
- **T6**: Wire into `internal/cli/init.go` after `EnsureTraefikContainerLocal` (Step 4e, replacing the old traefik install step).
|
||||
|
||||
### Wave 3 (doctor nft update)
|
||||
- **T7**: `internal/cli/doctor_nft.go` — extend assertions: postrouting masquerade present, DNAT target matches `NftClusterConfig.DNATTarget`.
|
||||
|
||||
### Wave 4 (tests)
|
||||
- **T8**: `internal/emitter/nft_test.go` — assert postrouting chain present when `EnableSNAT=true`, absent when false. Assert `DNATTarget` substitution. Assert priority `-10` on input/forward.
|
||||
- **T9**: Integration test: `orca init` → `nft list table inet orca-ingress` shows DNAT + postrouting; `podman inspect orca-traefik` running.
|
||||
|
||||
### Must-haves
|
||||
- [ ] `orca init` → nft table has DNAT + postrouting masquerade
|
||||
- [ ] nft input/forward chains at priority -10
|
||||
- [ ] First-apply doesn't error (table pre-created or delete-table idiom)
|
||||
- [ ] `/etc/orca/step-ca-root.crt` exists (real CA or placeholder)
|
||||
- [ ] `podman inspect orca-traefik` running
|
||||
|
||||
## Phase 4: `orca node join --type linux` remote ingress bootstrap (REQ-174)
|
||||
|
||||
**Tag**: `v0.13.4` | **Type**: feat | **Persona**: lead-developer
|
||||
|
||||
### Wave 1 (remote ingress)
|
||||
- **T1**: `internal/ingress/bootstrap.go` — add `BootstrapRemoteIngress(ctx, execFn)`:
|
||||
1. `mkdir -p /etc/traefik/dynamic /etc/orca` (remote)
|
||||
2. **C-60**: Push step-ca root CA to remote `/etc/orca/step-ca-root.crt` from `certpaths.CACertPath()` via `WriteFile`
|
||||
3. Render `orca.nft` + write to remote `/etc/nftables.d/orca.nft` via `WriteFile`
|
||||
4. `nft add table inet orca-ingress 2>/dev/null || true` (remote)
|
||||
5. `nft -f /etc/nftables.d/orca.nft` (remote)
|
||||
6. `systemctl enable --now podman-restart.service` (remote)
|
||||
7. `EnsureTraefikContainerRemote` (from P2)
|
||||
- **T2**: Wire into `internal/linux/bootstrap.go` after the traefik container reconciler step.
|
||||
- **T3**: Extend `linux.Result` with `IngressOK bool` for reporting.
|
||||
|
||||
### Wave 2 (tests)
|
||||
- **T4**: Fake-SSH test: assert remote `nft -f` + `podman run` + `WriteFile` for step-ca CA invoked.
|
||||
|
||||
### Must-haves
|
||||
- [ ] `orca node join --type linux --host <ip>` → remote has podman traefik running + nft applied + step-ca CA mounted
|
||||
- [ ] `doctor ingress --peer <linux-node>` passes
|
||||
|
||||
## Phase 5: Proxmox native ingress mode (REQ-175)
|
||||
|
||||
**Tag**: `v0.13.5` | **Type**: feat | **Persona**: backend-engineer + data-engineer
|
||||
|
||||
### Wave 1 (flags + schema)
|
||||
- **T1**: Add flags to `node join`: `--ingress-mode` (values: `native` default, `floating-ip`), `--floating-ip`, `--gateway`, `--mac`, `--net-prefix` (default `24`).
|
||||
- **T2**: Add `IngressMode` field to `model.Node` (string: `""`, `"native"`, `"floating-ip"`).
|
||||
- **T3**: **C-59**: Schema migration `0009_ingress_mode.sql` (NOT 0007 — already taken): `ALTER TABLE nodes ADD COLUMN ingress_mode TEXT DEFAULT '';`
|
||||
|
||||
### Wave 2 (native mode bootstrap)
|
||||
- **T4**: In `proxmox.BootstrapProxmox`, when `IngressMode == "native"`:
|
||||
1. On the PVE host: render+apply nft with `DNATTarget = <lxc-bridge-ip>` (the traefik LXC's IP, discovered after `pct start`)
|
||||
2. Create unprivileged LXC with `--features nesting=1,keyctl=1,fuse=1` (research finding: fuse=1 for fuse-overlayfs). `pct create <vmid> local:vztmpl/<template> --hostname orca-traefik --unprivileged 1 --features nesting=1,keyctl=1,fuse=1 --onboot 1 --memory 2048 --swap 0 --rootfs local:8`
|
||||
3. `pct start <vmid>`
|
||||
3a. **C-55**: Discover LXC IP via `pct config <vmid>` (parse `net0` line for `ip=`) or `pct exec <vmid> -- hostname -I` BEFORE the first nft apply. The nft DNAT target is set to the LXC IP from the start — no two-phase apply, no downtime window.
|
||||
4. Inside the LXC: **C-53**: `command -v podman >/dev/null 2>&1 || (apt-get update && apt-get install -y podman conmon crun fuse-overlayfs nftables)` — idempotent, skip if podman already installed
|
||||
5. Configure podman storage (`/etc/containers/storage.conf`): `mount_program = "/usr/bin/fuse-overlayfs"` (fallback: `driver = "vfs"`)
|
||||
6. `systemctl enable --now podman-restart.service` (inside LXC)
|
||||
7. Push step-ca root CA into LXC
|
||||
8. `EnsureTraefikContainerRemote` (podman pull + run inside LXC with `--network host`)
|
||||
9. **C-55**: Render+apply nft on PVE host with `DNATTarget = <lxc-ip>` (discovered in step 3a) — single apply, no downtime window
|
||||
|
||||
### Wave 3 (registration)
|
||||
- **T5**: Register PVE host as `proxmox` node with `IngressMode: "native"`.
|
||||
|
||||
### Wave 4 (tests)
|
||||
- **T6**: Fake-SSH test: assert `pct create` with `--features nesting=1,keyctl=1,fuse=1`, `apt-get install podman`, `podman run` inside LXC, nft DNAT target = LXC IP.
|
||||
|
||||
### Must-haves
|
||||
- [ ] `orca node join --type proxmox --host <ip>` (native) → LXC created with nesting+keyctl+fuse
|
||||
- [ ] Podman + orca-traefik running inside LXC
|
||||
- [ ] PVE host nft DNATs to LXC IP
|
||||
- [ ] `IngressMode: "native"` on node record
|
||||
- [ ] Schema migration 0007 applied
|
||||
|
||||
## Phase 6: Proxmox floating-IP LXC "ingress" (REQ-176)
|
||||
|
||||
**Tag**: `v0.13.6` | **Type**: feat | **Persona**: backend-engineer
|
||||
|
||||
### Wave 1 (LXC provisioning)
|
||||
- **T1**: New `internal/proxmox/ingress_lxc.go` — `ProvisionIngressLXC(ctx, opts)`:
|
||||
1. `pveam download local <template>` (idempotent, already in bootstrap)
|
||||
2. Deterministic VMID for "ingress" (hash of host+"ingress")
|
||||
3. `pct create <vmid> local:vztmpl/<template> --hostname ingress --unprivileged 1 --features nesting=1,keyctl=1,fuse=1 --net0 name=eth0,bridge=vmbr0,hwaddr=<mac>,ip=<floating-ip>/<prefix>,gw=<gateway> --onboot 1 --memory 2048 --swap 0 --rootfs local:8`
|
||||
4. `pct start <vmid>`
|
||||
5. Wait for LXC network (retry SSH to `<floating-ip>` with backoff, 60s timeout)
|
||||
6. Inside the LXC: **C-53**: `command -v podman >/dev/null 2>&1 || (apt-get update && apt-get install -y podman conmon crun fuse-overlayfs nftables)` — idempotent
|
||||
7. Configure podman storage (fuse-overlayfs / vfs fallback)
|
||||
8. `systemctl enable --now podman-restart.service`
|
||||
9. Push step-ca root CA into LXC
|
||||
10. Render+apply nft INSIDE the LXC (DNAT `:443`→`127.0.0.1:8443`, `:80`→`127.0.0.1:8080` + postrouting masquerade) — `DNATTarget = 127.0.0.1` here because traefik runs with `--network host` inside the LXC
|
||||
11. `EnsureTraefikContainerRemote` (podman pull + run inside LXC with `--network host`)
|
||||
12. Push orca SSH pubkey into LXC for future `job run` traefik dynamic-config pushes
|
||||
|
||||
### Wave 2 (registration)
|
||||
- **T2**: Register LXC as managed node: `registry.Join` with `Kind: "linux"`, `Name: "ingress"`, `Address: "<floating-ip>:8443"`, `OS: "linux"`, `IngressMode: "floating-ip"`.
|
||||
- **T3**: Also register PVE host as `proxmox` node (for workload dispatch).
|
||||
|
||||
### Wave 3 (interactive prompting)
|
||||
- **T4**: Interactive prompting in `joinProxmox` (node.go): when `--ingress-mode` empty and `!jsonOutput`:
|
||||
- Prompt "Ingress mode [native/floating-ip] (default native): "
|
||||
- If `floating-ip`: prompt for floating IP (validate `net.ParseIP`), gateway (validate `net.ParseIP`), MAC (validate `net.ParseMAC`; generate `02:XX:XX:XX:XX:XX` random if empty + confirm; **C-56**: check MAC against existing nodes' MACs in cluster registry, regenerate on collision), net-prefix (default 24, validate 8-32)
|
||||
- In `--json` mode: require `--mac` explicitly if `--ingress-mode floating-ip` (no silent generation)
|
||||
|
||||
### Wave 4 (routing)
|
||||
- **T5**: `joinProxmox` in `node.go` routes: native → P5 path; floating-ip → `ProvisionIngressLXC` + register LXC + register PVE host.
|
||||
|
||||
### Wave 5 (tests)
|
||||
- **T6**: Fake-SSH test: assert `pct create` with `net0 name=eth0,bridge=vmbr0,hwaddr=<mac>,ip=<floating-ip>/<prefix>,gw=<gateway>`, `--features nesting=1,keyctl=1,fuse=1`; LXC registered as `linux` node named `ingress` at `<floating-ip>:8443`; PVE host registered as `proxmox`.
|
||||
- **T7**: Interactive prompt test: stdin simulation → mode selection + param entry + validation.
|
||||
|
||||
### Must-haves
|
||||
- [ ] `orca node join --type proxmox --host <ip> --ingress-mode floating-ip --floating-ip 203.0.113.10 --gateway 203.0.113.1 --mac 02:01:02:03:04:05` → LXC `ingress` created
|
||||
- [ ] LXC has podman traefik running + nft applied inside LXC
|
||||
- [ ] Node `ingress` registered as `linux` at `203.0.113.10:8443`
|
||||
- [ ] PVE host registered as `proxmox`
|
||||
- [ ] Interactive prompt works when flags absent + not `--json`
|
||||
- [ ] IP/MAC/gateway validation rejects invalid input
|
||||
|
||||
## Phase 7: `doctor ingress` + docs + integration tests (REQ-177, REQ-178, REQ-179)
|
||||
|
||||
**Tag**: `v0.13.7` | **Type**: feat+docs+test | **Persona**: backend-engineer + lead-developer
|
||||
|
||||
### Wave 1 (doctor ingress)
|
||||
- **T1**: `internal/cli/doctor_ingress.go` — `orca doctor ingress [--peer <name>]`:
|
||||
1. `podman inspect orca-traefik` → running?
|
||||
2. nft DNAT+SNAT applied (reuse `doctor_nft` logic)
|
||||
3. `/etc/traefik/dynamic` exists
|
||||
4. step-ca root CA mounted (`podman inspect` volume check or file-exists check)
|
||||
5. For proxmox-native: checks the LXC (via `pct exec`)
|
||||
6. For floating-ip: checks the ingress LXC over SSH
|
||||
7. Uses SSH-push for remote peers
|
||||
|
||||
### Wave 2 (UAT assertions)
|
||||
- **T2**: `scripts/uat-signoff.sh` — add assertions: `40 ingress_podman_traefik`, `41 ingress_nft_dnat_snat`, `42 ingress_linux_worker`, `43 ingress_proxmox_native_lxc` or `43 ingress_floating_ip_lxc` (depending on topology).
|
||||
|
||||
### Wave 3 (docs)
|
||||
- **T3**: `docs/cli.md` — document `--ingress-mode`, `--floating-ip`, `--gateway`, `--mac`, `--net-prefix` flags + `orca doctor ingress`.
|
||||
- **T4**: `docs/uat.md` — add floating-IP topology variant; update native topology to assert ingress bootstrap.
|
||||
- **T5**: `docs/ingress.md` — podman-traefik image section: `Dockerfile.traefik`, volume mounts, TLS model (dynamic `tls.certificates`, not certResolver), `--network host` rationale.
|
||||
- **T6**: `docs/docker.md` — `orca-traefik` image: build, publish, pull.
|
||||
- **T7**: `.ciagent/ARCHITECTURE.md` — R-024 + ingress bootstrap section (3 topologies, nft+podman stack on each).
|
||||
|
||||
### Wave 4 (integration tests)
|
||||
- **T8**: `tests/ingress_bootstrap_test.go` — hermetic fake-SSH harness:
|
||||
- init → podman traefik running + nft applied
|
||||
- linux join → remote podman + nft + step-ca CA push
|
||||
- proxmox native → LXC created with `nesting=1,keyctl=1,fuse=1` + podman traefik + nft DNAT to LXC IP
|
||||
- floating-ip → `pct create` with correct `net0` args + LXC registered as `linux` node
|
||||
- release.sh builds orca-traefik image (Dockerfile.traefik parses)
|
||||
|
||||
### Wave 5 (verify)
|
||||
- **T9**: `make build && make test && make lint && make verify-docs` all pass.
|
||||
|
||||
### Must-haves
|
||||
- [ ] `orca doctor ingress` exits 0 on a properly bootstrapped node
|
||||
- [ ] UAT signoff script includes new assertions
|
||||
- [ ] `make verify-docs` passes
|
||||
- [ ] Integration tests pass in CI `validate`
|
||||
- [ ] ARCHITECTURE.md ingress section matches shipped code
|
||||
|
||||
## Phase 8: Final review + ship + audit (milestone release)
|
||||
|
||||
**Tag**: `v0.13.8` = **v0.14 milestone release** | **Type**: docs+review
|
||||
|
||||
- Multi-persona code review across all phases
|
||||
- Audit: `verify-reqs`, git-log ↔ `.ciagent/` reconstruction
|
||||
- Merge `phase/08` → `milestone/v0.14` → `main`
|
||||
- Tag `v0.13.8` + release with milestone summary
|
||||
- Build + publish both container images (`orca` + `orca-traefik`)
|
||||
- Delete all milestone branches (tags preserve history)
|
||||
- Mark all REQ-171..179 as complete in REQUIREMENTS.md + ROADMAP.md
|
||||
@@ -1,159 +0,0 @@
|
||||
# Plan: Orca v0.3 — scheduling-streaming
|
||||
|
||||
Milestone v0.3 (scheduling-streaming) — completion milestone closing the two
|
||||
work items deferred from v0.2 (iter.Seq streaming + doctor network/db). Two
|
||||
execution phases (P01, P02) followed by one final phase (P03 review + ship).
|
||||
|
||||
Branch: `phase/00-pre-execution` (cut from `milestone/v0.3-scheduling-streaming`).
|
||||
Go toolchain: `go1.25.0` (`iter` package + range-over-func are stable stdlib).
|
||||
No new `go.mod` dependencies (D-041). Source of implementation guidance:
|
||||
`.ciagent/RESEARCH_v0.3.md` (D-025..D-042).
|
||||
|
||||
---
|
||||
|
||||
## Phase P01: iter.Seq streaming for `--watch` flags
|
||||
|
||||
**Goal:** Add pull-based `iter.Seq` streaming to `orca job list` and `orca node list` behind a `--watch` flag, with table refresh (default) or streaming one-line JSON per event (`--watch --json`).
|
||||
|
||||
**Requirements:** REQ-022 (`iter.Seq` for streaming job lists, Go 1.25+), REQ-030 (`--watch` output format: table default vs streaming one-line JSON per event)
|
||||
|
||||
**Milestone:** v0.3
|
||||
**Phase tag:** v0.3.1
|
||||
**Key decisions:** D-019 (1s poll ticker), D-025 (iter.Seq on store repos), D-026 (`*model.Job`/`*model.Node` element type), D-028 (poll re-runs List, yields full snapshot), D-030 (per-event JSON streaming), D-031 (signal.NotifyContext replaces 5s timeout on watch path), D-032 (inline pull loop, no goroutine).
|
||||
|
||||
### Wave 1: Store layer iter.Seq + unit tests (vertical slice)
|
||||
|
||||
Wave 1 is independently testable: the two `Watch` methods + the migration-version-less store layer compile and run in isolation. No CLI or doctor code is touched. Running `go test ./internal/store/...` after this wave passes and exercises the `iter.Seq` contracts (yield, ctx cancellation, consumer break, no goroutine leak).
|
||||
|
||||
| Task ID | Description | Persona | Files | Must-have | Deps |
|
||||
|---------|-------------|---------|-------|-----------|------|
|
||||
| 01-01-01 | Add `JobRepo.Watch(ctx) iter.Seq[[]*model.Job]` — pull-based inline polling loop on a 1s ticker; re-runs the List `SELECT ... FROM jobs ORDER BY created_at DESC` each tick, collects ALL rows into a `[]*model.Job` slice via `scanJob`, then yields the **full snapshot as a single slice** (`yield(snapshot)`). **Immediate first yield** before the first ticker wait (G-002): the loop queries+yields on the first iteration, then `select`s on the ticker for subsequent ticks. `defer ticker.Stop()` + `rows.Close()` on every exit path (ctx.Done, yield==false, scan error). No goroutine spawned (D-032). Transient query errors are logged via `slog.Default().Warn` and the loop continues to the next tick (D-034 lite). Imports: add `"iter"` (`"time"` already present). | data-engineer | `internal/store/job_task_repo.go` | `go build ./internal/store/...` succeeds; `Watch` method exists with signature `func (r *JobRepo) Watch(ctx context.Context) iter.Seq[[]*model.Job]`; code path closes rows on ctx.Done and on `yield==false`; first yield is immediate (no `watchInterval` delay before first snapshot — G-002). | - |
|
||||
| 01-01-02 | Add `NodeRepo.Watch(ctx) iter.Seq[[]*model.Node]` — analogous to 01-01-01 but against the nodes query `SELECT id, name, address, state, joined_at, last_seen, metadata FROM nodes ORDER BY joined_at ASC`, reusing `scanNode`. Yields the full snapshot as a `[]*model.Node` slice per tick. Same inline-pull / no-goroutine / rows-close-on-all-paths / immediate-first-yield contract (G-001, G-002). | data-engineer | `internal/store/node_repo.go` | `go build ./internal/store/...` succeeds; `Watch` method exists with signature `func (r *NodeRepo) Watch(ctx context.Context) iter.Seq[[]*model.Node]`; immediate first yield. | - |
|
||||
| 01-01-03 | Add an unexported test hook for the poll interval so unit tests are deterministic (D-035). Preferred shape: an unexported package var `watchInterval = 1 * time.Second` in `internal/store` that `Watch` reads instead of a literal, overridable from `_test.go` via `watchInterval = 10 * time.Millisecond`. Both `JobRepo.Watch` and `NodeRepo.Watch` reference this var. | data-engineer | `internal/store/job_task_repo.go`, `internal/store/node_repo.go` (optionally a tiny `internal/store/watch_test_helper_test.go` if a shared helper reads cleaner) | `Watch` uses the `watchInterval` var, not a literal `1 * time.Second`; tests can set it to a small value. | 01-01-01, 01-01-02 |
|
||||
| 01-01-04 | Write store-layer unit tests for `Watch`. New/append: `internal/store/job_task_repo_test.go` and `internal/store/node_repo_test.go` (mirror). Tests: (a) `TestJobRepoWatch_YieldsSnapshots` — insert 1 job, set `watchInterval=10ms`, range over seq collecting `[]*model.Job` snapshots into a slice, insert a 2nd job from a goroutine after ~30ms, cancel ctx after ~80ms, assert at least one snapshot contains both jobs and the first snapshot contains only the first job (G-001: each yield is a complete tick snapshot). (b) `TestJobRepoWatch_ImmediateFirstYield` (G-002) — assert the first snapshot appears within <50ms even with `watchInterval=10ms` (proving first yield is not tick-gated). (c) `TestJobRepoWatch_StopsOnConsumerBreak` — range and `break` after first yield; assert the range returns (no hang) within a short deadline. (d) `TestJobRepoWatch_StopsOnCtxCancel` — cancel ctx; assert the range loop exits within ~50ms. (e) Mirror all four for `NodeRepo.Watch`. Run `go test -race ./internal/store/...`. | data-engineer | `internal/store/job_task_repo_test.go`, `internal/store/node_repo_test.go` | `go test -race ./internal/store/...` passes; all 8 Watch tests pass; `-race` reports no leaks/data races; immediate-first-yield assertion holds (G-002). | 01-01-03 |
|
||||
|
||||
### Wave 2: CLI `--watch` flag + integration
|
||||
|
||||
Wave 2 depends on Wave 1's `Watch` methods. It wires the `--watch` flag into both list commands, implements the two output modes, and adds CLI-level smoke tests. After this wave `orca job list --watch` and `orca node list --watch` are runnable end-to-end.
|
||||
|
||||
| Task ID | Description | Persona | Files | Must-have | Deps |
|
||||
|---------|-------------|---------|-------|-----------|------|
|
||||
| 01-02-01 | Add `--watch` flag to `jobListCmd` in `internal/cli/job.go`. Register `jobListCmd.Flags().BoolVar(&jobWatch, "watch", false, "stream jobs until Ctrl-C")` (package var `jobWatch bool`). In `RunE`, branch: if `!jobWatch` keep the existing 5s-timeout `List` path unchanged; if `jobWatch`, build `ctx, cancel := signal.NotifyContext(cmd.Context(), os.Interrupt, syscall.SIGTERM); defer cancel()` (drop the 5s timeout — D-031), then `seq := store.NewJobRepo(db).Watch(ctx)`. Imports: `os/signal`, `syscall`, `iter`. The branch structure is added in this task; the actual rendering (table vs JSON) is filled by 01-02-02 + 01-02-03. | cli-engineer | `internal/cli/job.go` | `go build ./internal/cli/...` succeeds; `orca job list --help` shows `--watch` flag; non-watch path behavior unchanged (existing tests pass); watch path compiles (rendering may be a placeholder `for range seq {}` at this step). | 01-01-01 |
|
||||
| 01-02-02 | Implement the **table** watch render in `jobListCmd` (D-029, G-001). Each yielded value is a complete `[]*model.Job` snapshot. Maintain the previous snapshot's rendered-table string (or a hash of it). On each tick, if the new rendered table differs from the previous, emit `"\033[2J\033[H"` (clear screen + home) then the table header + rows (reuse the existing table-rendering code path). Unchanged snapshots produce no output (avoids flicker). | cli-engineer | `internal/cli/job.go` | `orca job list --watch` on a temp DB: inserting a job causes a cleared-screen re-render showing the new job; no output when the snapshot is unchanged. | 01-02-01 |
|
||||
| 01-02-03 | Implement the **`--watch --json`** render in `jobListCmd` (D-030, G-001). Each yielded value is a complete `[]*model.Job` snapshot. Maintain `map[string][]byte` of last-seen compact-JSON bytes per job ID. Per tick: diff the current snapshot against the map — for each job in the snapshot, marshal compact JSON; if it differs from stored bytes (or ID unseen), print `{"event":"init","job":{...}}\n` (first sighting) or `{"event":"update","job":{...}}\n` (subsequent change). For IDs in the map but NOT in the current snapshot, print `{"event":"delete","job":{...}}\n` (G-006 DEFER: delete event now natural with snapshot-per-tick). One line per changed element per tick — matches REQ-030. | cli-engineer | `internal/cli/job.go` | `orca job list --watch --json` on a temp DB: inserting/changing a job prints one JSON line per changed job; first tick prints `"init"` lines for existing jobs; deleting a job prints `"delete"`; unchanged jobs on a tick produce no line. | 01-02-01 |
|
||||
| 01-02-04 | Add `--watch` flag + both render modes to `nodeListCmd` in `internal/cli/node.go`, mirroring 01-02-01..01-02-03. Package var `nodeWatch bool`; flag `--watch`. For the watch path bypass `engine.NodeRegistry` and call `store.NewNodeRepo(db).Watch(ctx)` directly (D-025 — keeps iter boundary in store; registry adds no value for a read-only stream). Same `signal.NotifyContext` cancellation. Table + JSON renders analogous to job (element type `[]*model.Node` snapshot per tick, event wrapper `{"event":"...","node":{...}}`). **Note (G-003):** This task modifies `internal/cli/node.go`, which P02 task 02-01-01 also modifies (removing old `dbPath`). Task 02-01-01 MUST complete first to avoid merge conflicts. | cli-engineer | `internal/cli/node.go` | `orca node list --watch` and `orca node list --watch --json` behave as specified; non-watch path unchanged. | 01-01-02, 01-02-03, 02-01-01 |
|
||||
| 01-02-05 | Add CLI-level watch tests. New files (or append if present): `internal/cli/job_test.go`, `internal/cli/node_test.go`. Smoke-level (store layer is the thorough test home): `TestJobListWatch_JSONStreaming` — temp DB, insert a job, run `jobListCmd.RunE` with `--watch --json` in a goroutine under a cancellable ctx, insert a 2nd job, capture stdout for ~200ms, assert ≥2 JSON lines appear, then cancel ctx and assert the command returns promptly. `TestJobListWatch_TableRefresh` — assert the clear-screen escape `\033[2J\033[H` appears in output on change. Mirror for nodes. Keep deterministic: small `watchInterval` via the test hook, short timeouts. Run `go test -race ./internal/cli/...`. | cli-engineer | `internal/cli/job_test.go`, `internal/cli/node_test.go` | `go test -race ./...` passes (whole repo); the 4 CLI watch smoke tests pass; no goroutine leaks under `-race`. | 01-02-02, 01-02-03, 01-02-04 |
|
||||
|
||||
### Test strategy (P01)
|
||||
|
||||
- **Store layer (thorough, deterministic):** `internal/store/job_task_repo_test.go` + `internal/store/node_repo_test.go` — three tests per repo (snapshots over time, consumer-break stops, ctx-cancel stops). Uses the `watchInterval` test hook (10ms) for speed. Runs under `go test -race ./internal/store/...`. This is where the `iter.Seq` contract is verified rigorously.
|
||||
- **CLI layer (smoke):** `internal/cli/job_test.go` + `internal/cli/node_test.go` — verify the flag is wired, JSON streaming emits one line per changed element, table mode emits the clear-screen escape on change, and the command exits promptly on ctx cancellation. Kept intentionally lightweight; deterministic via the shared `watchInterval` hook + short timeouts.
|
||||
- **Non-watch regression:** existing `job list` / `node list` tests must still pass unchanged (the 5s-timeout path is untouched).
|
||||
- **Race:** `go test -race ./...` is the gate (REQ-031 already enforces `-race` in CI).
|
||||
|
||||
### Vertical slice integrity (P01)
|
||||
|
||||
- **Wave 1** produces a runnable, testable artifact: `go build ./internal/store/...` + `go test -race ./internal/store/...`. No CLI or doctor code is modified. The `iter.Seq` contract (pull, cancel, no-leak) is fully verified at this layer.
|
||||
- **Wave 2** builds on Wave 1's `Watch` methods to deliver the user-facing `--watch` flag end-to-end. After Wave 2 an operator can demo `orca job list --watch` and `orca node list --watch --json`.
|
||||
|
||||
---
|
||||
|
||||
## Phase P02: `orca doctor` network + db full implementation
|
||||
|
||||
**Goal:** Replace the `NetworkStub` and `DBStub` placeholders with real diagnostics — peer reachability via mTLS `/healthz` probe and SQLite `PRAGMA integrity_check` + migration version — completing REQ-032.
|
||||
|
||||
**Requirements:** REQ-032 (completion: network reachability + db integrity)
|
||||
**Milestone:** v0.3
|
||||
**Phase tag:** v0.3.2
|
||||
**Key decisions:** D-027 (closure-capture handles in check constructors), D-033 (`store.MigrationVersion`), D-034 (db check opens its own `*sql.DB`), D-035 (`PRAGMA integrity_check` + migration version), D-036 (peers from `nodes` table, not in-memory registry), D-037 (`ServerName = node.Name`), D-038 (zero peers → WARN, any fail → FAIL, 3s per-probe timeout), D-039 (`dbPath` → `certpaths.DBPath()`), D-040 (delete stubs, no shims).
|
||||
|
||||
### Wave 1: Shared infra + store migration-version query (vertical slice)
|
||||
|
||||
Wave 1 breaks the would-be `doctor → cli` import cycle (D-039) and adds the public `store.MigrationVersion` query. Both are independently testable: `go test ./internal/store/... ./internal/certpaths/...` passes after this wave, and the foundation for both the db and network checks is in place.
|
||||
|
||||
| Task ID | Description | Persona | Files | Must-have | Deps |
|
||||
|---------|-------------|---------|-------|-----------|------|
|
||||
| 02-01-01 | Move `dbPath()` (the `ORCA_DB`-env-honoring path resolver) from `internal/cli` to `internal/certpaths` as `DBPath()` (D-039). `certpaths` already owns the `ORCA_HOME`-honoring `Dir()`. Add `func DBPath() string` to `internal/certpaths/certpaths.go`: honors `ORCA_DB` env override, else `filepath.Join(Dir(), "orca.db")`. Update `internal/cli/node.go` (and any other `internal/cli` caller of the old unexported `dbPath`) to call `certpaths.DBPath()`; remove the old `dbPath` from `internal/cli`. Territory note: this crosses cli-engineer territory — lead-developer adjudicates (D-039). | lead-developer | `internal/certpaths/certpaths.go`, `internal/cli/node.go` (remove old `dbPath`), any other `internal/cli/*` caller | `go build ./...` succeeds (no import cycle); `certpaths.DBPath()` exists and honors `ORCA_DB`/`ORCA_HOME`; `internal/cli` no longer defines `dbPath`; existing CLI tests pass. | - |
|
||||
| 02-01-02 | Add `store.MigrationVersion(ctx, db) (string, error)` to `internal/store/migrate.go` (D-033). SQL: `SELECT name FROM schema_migrations ORDER BY name DESC LIMIT 1`. Returns `("", nil)` on `sql.ErrNoRows` (empty/fresh db). Wraps other errors with `fmt.Errorf("query migration version: %w", err)`. Add `"context"` import if missing (likely already imported). | data-engineer | `internal/store/migrate.go` | `go build ./internal/store/...` succeeds; function is exported; `sql.ErrNoRows` maps to `("", nil)`. | - |
|
||||
| 02-01-03 | Test `MigrationVersion`. New/append `internal/store/migrate_test.go`: `TestMigrationVersion` — open a fresh test db via `store.Open` (which runs `migrate`), call `MigrationVersion`, assert it returns `0005_node_capacity.sql` (the highest current migration). Then manually `db.Exec("DELETE FROM schema_migrations")`, call again, assert `("", nil)`. Run `go test -race ./internal/store/...`. | data-engineer | `internal/store/migrate_test.go` | `go test -race ./internal/store/...` passes; both assertions (highest version, empty → `""`) hold. | 02-01-02 |
|
||||
|
||||
### Wave 2: doctor DB check + network check + CLI wiring + tests
|
||||
|
||||
Wave 2 depends on Wave 1 (`certpaths.DBPath` + `store.MigrationVersion`). It replaces both stubs with real checks, updates `All()` and the CLI subcommands, rewrites the broken stub test, and adds per-check tests. After this wave `orca doctor`, `orca doctor network`, and `orca doctor db` are fully functional.
|
||||
|
||||
| Task ID | Description | Persona | Files | Must-have | Deps |
|
||||
|---------|-------------|---------|-------|-----------|------|
|
||||
| 02-02-01 | Replace `DBStub()` with `DB()` in `internal/doctor/doctor.go` (D-027, D-034, D-035). The `Check.Run` closure: `path := certpaths.DBPath()`; `db, err := store.Open(path)` (defer `db.Close()`); `PRAGMA integrity_check` via `db.QueryRowContext(ctx, "PRAGMA integrity_check").Scan(&integrity)`; FAIL if not `"ok"` (first line of message); then `store.MigrationVersion(ctx, db)` — WARN if `""` (fresh/never-migrated), else PASS with `"... migrations up to <ver>"`. New imports: `strings`, `internal/store`, `internal/certpaths`. | data-engineer | `internal/doctor/doctor.go` | `go build ./internal/doctor/...` succeeds; `doctor.DB()` returns a `Check` with `Name=="db"`; `DBStub` still present (removed in 02-02-04 lockstep). | 02-01-01, 02-01-02 |
|
||||
| 02-02-02 | Add `probeHealthz(ctx, caPath, certPath, keyPath, serverName, addr) error` helper in `internal/doctor/doctor.go` (network-engineer territory — connection lifecycle). Builds an mTLS client via `transport.NewMTLSClient(caPath, serverName, certPath, keyPath)` (D-037: `serverName = node.Name`), `http.NewRequestWithContext(ctx, GET, "https://"+addr+"/healthz", nil)`, `client.Do(req)`, defer `resp.Body.Close()`, FAIL if status != 200. New imports: `net/http`, `time`, `internal/transport`. | network-engineer | `internal/doctor/doctor.go` | `go build ./internal/doctor/...` succeeds; `probeHealthz` exists with the specified signature; reuses `transport.NewMTLSClient` (no new TLS code — security-engineer territory respected). | 02-01-01 |
|
||||
| 02-02-03 | Replace `NetworkStub()` with `Network()` in `internal/doctor/doctor.go` (D-036, D-037, D-038). The `Check.Run` closure: `path := certpaths.DBPath()`; `db, err := store.Open(path)` (defer close); `nodes, err := store.NewNodeRepo(db).List(ctx)`; filter `state != model.NodeStateLeft` into `live`; if `len(live)==0` → `ResultWarn` ("no peers registered; network check skipped (single-node?)"). Else per-peer: `probeCtx, cancel := context.WithTimeout(ctx, 3*time.Second)`; `probeHealthz(...)`; collect PASS/FAIL lines; aggregate — FAIL if any peer failed, PASS if all OK. `serverName = n.Name`, `caPath = certpaths.CACertPath()`, `certPath/keyPath = certpaths.ServerCertPath()/ServerKeyPath()`. New imports: `internal/model`. | network-engineer | `internal/doctor/doctor.go` | `go build ./internal/doctor/...` succeeds; `doctor.Network()` returns a `Check` with `Name=="network"`; zero-peer → WARN; per-probe 3s timeout enforced. | 02-02-02 |
|
||||
| 02-02-04 | Update `All()` in `internal/doctor/doctor.go` to use `Network()` and `DB()` instead of the stubs (D-040). **Delete** `NetworkStub` and `DBStub` (no backward-compat shims — internal only). Update `internal/cli/doctor.go`: `doctorNetworkCmd.RunE` calls `doctor.Network()` (was `NetworkStub()`); `doctorDBCmd.RunE` calls `doctor.DB()` (was `DBStub()`). Ensure per-subcommand render honors `jsonOutput` (minor enhancement, in scope). | cli-engineer | `internal/doctor/doctor.go`, `internal/cli/doctor.go` | `go build ./...` succeeds; `grep -r "NetworkStub\|DBStub" internal/` returns nothing; `orca doctor`, `orca doctor network`, `orca doctor db` run without referencing stubs. | 02-02-01, 02-02-03 |
|
||||
| 02-02-05 | Rewrite + add doctor tests in `internal/doctor/doctor_test.go`. (a) Rewrite `TestRunAllChecksWithNoCA` — set `ORCA_HOME` to a temp dir with no CA. Assert per-check by name: `cert.ca` FAIL, `cert.server` FAIL, `cert.expiry` FAIL, `cert.fingerprint` FAIL, `db` PASS (store.Open runs migrations → version 0005), `network` WARN (no peers). Remove the stale "expects WARN (stubs)" comment. (b) `TestDBCheck_IntegrityOK` — fresh db via `store.Open` in temp, run `doctor.DB().Run(ctx)`, expect PASS, message contains "0005". (c) `TestNetworkCheck_NoPeers` — fresh db, no nodes, run `doctor.Network().Run(ctx)`, expect WARN. (d) `TestNetworkCheck_PeerUnreachable` — insert a node with `Address = "127.0.0.1:1"` (nothing listening), run, expect FAIL with the peer name in the message. (e) `TestNetworkCheck_PeerReachable` (integration) — bootstrap a CA via `security.CAInit`-equivalent, generate+sign a server cert with SAN `localhost`, start an `httptest.NewUnstartedServer` with TLS + `ClientAuth=RequireAndVerifyClientCert` (mirror `security/integration_test.go` pattern), insert a node row with `Address = ts.Listener.Addr().String()` and `Name = "localhost"`, set `ORCA_HOME`, run `doctor.Network().Run(ctx)`, expect PASS. Run `go test -race ./internal/doctor/...`. | network-engineer (network tests), data-engineer (db test), cli-engineer (All() rewrite test) | `internal/doctor/doctor_test.go` | `go test -race ./internal/doctor/...` passes; all 5 test cases pass; the stale stub assertion is gone. | 02-02-04 |
|
||||
|
||||
### Test strategy (P02)
|
||||
|
||||
- **Store layer:** `internal/store/migrate_test.go` — `MigrationVersion` returns highest applied migration (`0005_node_capacity.sql`) and `""` on empty. (`-race`.)
|
||||
- **Doctor db check:** `TestDBCheck_IntegrityOK` — fresh db → PASS with version in message. (Optional brittle `TestDBCheck_Corrupt` may be added if a reliable corruption method is found; otherwise rely on the integrity-string parsing logic via the PASS/FAIL branch coverage.)
|
||||
- **Doctor network check:** `TestNetworkCheck_NoPeers` (WARN), `TestNetworkCheck_PeerUnreachable` (FAIL, peer name in message), `TestNetworkCheck_PeerReachable` (integration: real mTLS `httptest` server → PASS). The reachable test reuses the proven `TestEndToEndMTLS` pattern from `security/integration_test.go`.
|
||||
- **Doctor `All()` regression:** rewritten `TestRunAllChecksWithNoCA` asserts per-check results (certs FAIL, db PASS, network WARN) — no more global "hasWarn" stub assertion.
|
||||
- **Race:** `go test -race ./...` is the gate.
|
||||
|
||||
### Vertical slice integrity (P02)
|
||||
|
||||
- **Wave 1** produces a runnable, testable artifact: `certpaths.DBPath()` (cycle broken) + `store.MigrationVersion` (tested). `go test -race ./internal/store/... ./internal/certpaths/...` passes. No doctor code depends on the stubs being changed yet.
|
||||
- **Wave 2** builds on Wave 1 to deliver the real `DB()` and `Network()` checks, wires the CLI, and replaces the stub tests. After Wave 2 an operator can demo `orca doctor` showing real PASS/WARN/FAIL for db and network.
|
||||
|
||||
---
|
||||
|
||||
## Phase P03 (final): review + ship + audit
|
||||
|
||||
**Goal:** Review the v0.3 milestone for completeness against REQ-022/030/032, audit the codebase for leftover stubs/dead code, run the full CI gate (`go test -race ./...`, `gosec`, `govulncheck`, `gitleaks`), tag the milestone release, and ship.
|
||||
|
||||
**Requirements:** REQ-022 (verify complete), REQ-030 (verify complete), REQ-032 (verify complete)
|
||||
**Milestone:** v0.3
|
||||
**Phase tag:** v0.3.3 (= milestone release; target milestone tag `v0.4.0` per ROADMAP next-minor rule)
|
||||
|
||||
### Wave 1: Review + audit + ship (single wave)
|
||||
|
||||
| Task ID | Description | Persona | Files | Must-have | Deps |
|
||||
|---------|-------------|---------|-------|-----------|------|
|
||||
| 03-01-01 | Verify REQ coverage: confirm REQ-022 (iter.Seq streaming job lists), REQ-030 (--watch table/JSON modes), REQ-032 (doctor network + db) are fully implemented. Update `REQUIREMENTS.md` status for REQ-022/030/032 from Pending/Partial → **Complete**. Cross-check against the plan's must-have criteria. | lead-developer | `.ciagent/REQUIREMENTS.md` | All three REQs marked Complete with phase references; no remaining "stub" or "Pending" status for v0.3 scope. | P01, P02 complete |
|
||||
| 03-01-02 | Codebase audit: `grep -r "NetworkStub\|DBStub" internal/` returns nothing; `grep -r "TODO\|FIXME" internal/` reviewed (no v0.3 leftovers); confirm no `dbPath` duplication remains in `internal/cli`; confirm `iter` import is used (no unused imports); run `go vet ./...`. | lead-developer | (read-only audit; edits only if cleanup needed) | `go vet ./...` clean; no stub references; no leftover TODOs for v0.3 scope. | 03-01-01 |
|
||||
| 03-01-03 | Full CI gate: `go build ./...`, `go test -race ./...`, `gosec` (vs baseline JSON), `govulncheck ./...` (offline mode per REQ-027), `gitleaks` (vs baseline per REQ-029). Fix any new findings. | lead-developer | (fixes if needed) | All gates green; no new gosec findings beyond baseline; govulncheck exit 0; gitleaks clean vs baseline. | 03-01-02 |
|
||||
| 03-01-04 | Tag + ship: per `.ciagent/RELEASE_POLICY.md`, tag `v0.3.3` (phase tag) and the milestone tag (next-minor per ROADMAP). Produce Gitea release. Update `ROADMAP.md` v0.3 section to mark P01/P02/P03 complete. | lead-developer | `.ciagent/ROADMAP.md` | `v0.3.3` tag exists; Gitea release published; ROADMAP v0.3 checkboxes updated. | 03-01-03 |
|
||||
|
||||
### Test strategy (P03)
|
||||
|
||||
- No new tests; this phase is review + audit + release.
|
||||
- The gate is the existing test suite + security scans all passing under CI.
|
||||
|
||||
---
|
||||
|
||||
## Cross-phase notes
|
||||
|
||||
- **Phase ordering / parallelism (D-042, revised by G-003):** P01 Wave 1 and P02 Wave 1 may run in parallel (file-disjoint: store repos vs certpaths+migrate). **P02 Wave 1 (02-01-01) MUST complete before P01 Wave 2 (01-02-04)** because both modify `internal/cli/node.go` (P01 adds `--watch`, P02 removes old `dbPath`). P01 Wave 2 tasks 01-02-01..01-02-03 (job.go only) are not blocked by P02. Recommended order: P01 W1 + P02 W1 in parallel → P02 W1 02-01-01 completes → P01 W2 (job.go tasks) + P02 W2 in parallel → P01 W2 01-02-04 (node.go) after 02-01-01. P03 is strictly sequential after both phases complete.
|
||||
- **No new dependencies (D-041):** `iter` (P01) and the mTLS health probe (P02) use stdlib + existing internal packages only. The 4 direct `go.mod` deps (cobra, hcl/v2, modernc/sqlite, uuid) are unchanged.
|
||||
- **Decisions logged during planning (new, this plan):**
|
||||
- **D-043** — `watchInterval` test hook: an unexported package var in `internal/store` (default `1 * time.Second`) referenced by both `Watch` methods, overridable from `_test.go`. Avoids a public `WatchWithInterval` constructor that would leak test-only API into production. Confidence 0.90.
|
||||
- **D-044** — P01 Wave 1 / Wave 2 split: Wave 1 = store-layer `Watch` methods + tests (data-engineer only, fully isolated); Wave 2 = CLI `--watch` flag + renders + CLI tests (cli-engineer). This keeps the `iter.Seq` contract verifiable without the CLI and matches persona territories. Confidence 0.93.
|
||||
- **D-045** — P02 Wave 1 / Wave 2 split: Wave 1 = `certpaths.DBPath()` relocation + `store.MigrationVersion` + tests (breaks the import cycle, data-engineer + lead-developer); Wave 2 = real `DB()`/`Network()` checks + CLI wiring + doctor tests. Wave 1 is the unblock for both checks. Confidence 0.91.
|
||||
- **D-046** — `--watch --json` event wrapper shape: `{"event":"update","job":{...}}` / `{"event":"init","job":{...}}` (and `node` analog). `"init"` on first sighting of an ID, `"update"` on subsequent change. Unchanged IDs on a tick emit nothing. Confidence 0.80 (matches D-030's per-event interpretation of REQ-030).
|
||||
|
||||
## Grill amendments (binding ACCEPT verdicts applied)
|
||||
|
||||
Three binding changes from `.ciagent/GRILL_v0.3.md` have been applied to this plan:
|
||||
|
||||
- **G-001 [CRITICAL]** — `Watch` element type changed from `iter.Seq[*model.Job]` (per-row) to `iter.Seq[[]*model.Job]` (full snapshot per tick). Each tick yields the complete snapshot as a single slice. This makes table render mode correct (clear-screen + re-render needs full snapshot) and enables natural `"delete"` events in JSON mode. Applied to tasks 01-01-01, 01-01-02, 01-02-02, 01-02-03, 01-02-04, 01-01-04.
|
||||
- **G-002 [CRITICAL]** — `Watch` must yield immediately on the first iteration, then `select` on the ticker for subsequent ticks. Prevents a 1s blank-screen UX bug in production (tests with 10ms interval missed this). Applied to tasks 01-01-01, 01-01-02; new test `TestWatch_ImmediateFirstYield` added to 01-01-04.
|
||||
- **G-003 [HIGH]** — D-042's "file-disjoint" claim corrected: both P01 (01-02-04) and P02 (02-01-01) modify `internal/cli/node.go`. P02 Wave 1 task 02-01-01 is now a dependency of P01 Wave 2 task 01-02-04. Cross-phase ordering updated.
|
||||
|
||||
DEFER items (G-004, G-006, G-007, G-009, G-012) are noted in the grill report and will be addressed during execution.
|
||||
|
||||
## Summary
|
||||
|
||||
- **Phases:** 3 (P01, P02 execution; P03 final review/ship)
|
||||
- **Waves:** P01 = 2 waves (4 + 5 tasks); P02 = 2 waves (3 + 5 tasks); P03 = 1 wave (4 tasks). Total = 5 waves.
|
||||
- **Tasks:** P01 = 9, P02 = 8, P03 = 4. Total = 21 tasks.
|
||||
- **Grill amendments:** G-001 (snapshot-per-tick), G-002 (immediate first yield), G-003 (serialize P02 W1 → P01 W2 on node.go). 3 ACCEPT verdicts applied.
|
||||
- **New planning decisions:** D-043 (watchInterval test hook), D-044 (P01 wave split), D-045 (P02 wave split), D-046 (JSON event wrapper shape).
|
||||
- **Requirements closed:** REQ-022, REQ-030 (P01); REQ-032 (P02, completion).
|
||||
- **No new go.mod dependencies.** No source code written in this plan — implementation begins at P01 Wave 1.
|
||||
@@ -1,175 +0,0 @@
|
||||
---
|
||||
milestone: v0.5
|
||||
milestone_slug: distribution
|
||||
type: feature
|
||||
phase_count: 4
|
||||
---
|
||||
|
||||
# Plan: Orca v0.5 — Distribution
|
||||
|
||||
Vertical-slice plan for the v0.5 Distribution milestone. Each phase is a
|
||||
vertical slice that ships independently as a patch on the v0.4.x line.
|
||||
The final phase (P4) is the milestone release (promoted to v0.5.0).
|
||||
|
||||
## Requirement → Phase Mapping
|
||||
|
||||
| REQ | Phase | Priority |
|
||||
|-----|-------|----------|
|
||||
| REQ-045 (public releases) | P0 ship (operational) | High |
|
||||
| REQ-041 (ORCA_HOME unified namespace) | P1 | High |
|
||||
| REQ-042 (--system flag) | P1 | High |
|
||||
| REQ-043 (install.sh 1-liner) | P2 | High |
|
||||
| REQ-044 (in-place update) | P2 | High |
|
||||
| REQ-046 (docker release) | P3 | Medium |
|
||||
| REQ-016 (README quickstart) | P2 | Medium (completion) |
|
||||
|
||||
## Phase 1 — Namespace Unification (REQ-041, REQ-042)
|
||||
|
||||
**Goal**: Single `ORCA_HOME` env var as namespace root for all
|
||||
on-disk state; `--system` flag selects `/root/.orca`.
|
||||
|
||||
**Persona**: backend-engineer (store/certpaths routing) + cli-engineer
|
||||
(`--system` flag).
|
||||
|
||||
**Wave 1** (single wave — no inter-task dependencies):
|
||||
|
||||
| Task | File(s) | Persona | REQ |
|
||||
|------|---------|---------|-----|
|
||||
| T1.1: Route `store.Open("")` through `certpaths.DBPath()` | `internal/store/store.go` | backend-engineer | REQ-041 |
|
||||
| T1.2: Route `init` command through `certpaths.Dir()` | `internal/cli/init.go` | backend-engineer | REQ-041 |
|
||||
| T1.3: Add `--system` persistent flag on `rootCmd` + `PersistentPreRunE` that sets `ORCA_HOME=/root/.orca` | `internal/cli/root.go` | cli-engineer | REQ-042 |
|
||||
| T1.4: Add `namespace_test.go` covering user-level, `ORCA_HOME` override, `--system` | `internal/cli/namespace_test.go` | cli-engineer | REQ-041/042 |
|
||||
| T1.5: Update `docs/namespace.md` (paths reference) | `docs/namespace.md` | backend-engineer | REQ-041 |
|
||||
|
||||
**Must-haves**:
|
||||
- `go test ./...` passes (including new namespace_test.go).
|
||||
- `ORCA_HOME=/tmp/x orca init` creates `/tmp/x` (not `~/.orca`).
|
||||
- `orca --system init` creates `/root/.orca` (when run as root).
|
||||
- Empty `ORCA_HOME` + no `--system` → `~/.orca` (backward compat).
|
||||
|
||||
**Verification**: 4-layer (structural: gofmt/vet; behavioral: namespace_test
|
||||
+ existing doctor_test; security: no new secret surface; quality: no
|
||||
regression in existing tests).
|
||||
|
||||
**Ship**: tag `v0.4.2`.
|
||||
|
||||
## Phase 2 — install.sh + In-Place Update (REQ-043, REQ-044, REQ-016)
|
||||
|
||||
**Goal**: 1-liner installer from public Gitea releases; idempotent
|
||||
update-in-place; README quickstart.
|
||||
|
||||
**Persona**: devops-engineer.
|
||||
|
||||
**Wave 1**:
|
||||
|
||||
| Task | File(s) | Persona | REQ |
|
||||
|------|---------|---------|-----|
|
||||
| T2.1: Write `scripts/install.sh` (curl 1-liner, user/system, latest/pinned, in-place update) | `scripts/install.sh` | devops-engineer | REQ-043/044 |
|
||||
| T2.2: Write `scripts/install_test.sh` (mocked download, path verification, update-in-place) | `scripts/install_test.sh` | devops-engineer | REQ-043/044 |
|
||||
| T2.3: Update README quickstart with 1-liner install + `--system` variant | `README.md` | devops-engineer | REQ-016 |
|
||||
| T2.4: Write `docs/install.md` (full install reference, troubleshooting, ORCA_HOME) | `docs/install.md` | devops-engineer | REQ-043 |
|
||||
|
||||
**install.sh spec** (per R-006):
|
||||
- Default: user-level. Binary → `~/.local/bin/orca`. Namespace → `~/.orca`.
|
||||
- `--system`: binary → `/usr/local/bin/orca`, namespace → `/root/.orca`. Requires root (uid 0).
|
||||
- `--version vX.Y.Z`: pin version. Default: query `/api/v1/repos/coreci/orca/releases/latest`.
|
||||
- Download `orca-{tag}-linux-{arch}.tar.gz` from the release asset.
|
||||
- In-place update: if `orca` exists at install path, run `orca version --json`,
|
||||
parse `version`, print "updated from X to Y". Overwrite binary. **Never**
|
||||
touch the namespace dir.
|
||||
- Detect arch: `amd64` (x86_64), `arm64` (aarch64).
|
||||
- Idempotent: re-running with same version is a no-op (or reinstalls).
|
||||
|
||||
**Must-haves**:
|
||||
- `bash scripts/install_test.sh` passes (mocked).
|
||||
- `curl -fsSL <url> | bash` works on a fresh system (verified in P4 e2e).
|
||||
- `curl -fsSL <url> | bash -s -- --system` installs to `/usr/local/bin` (as root).
|
||||
- Re-running updates the binary; `~/.orca/orca.db` preserved.
|
||||
|
||||
**Verification**: 4-layer (structural: shellcheck; behavioral:
|
||||
install_test.sh; security: no secret in script, no eval of remote
|
||||
content beyond the script itself; quality: idempotent).
|
||||
|
||||
**Ship**: tag `v0.4.3`.
|
||||
|
||||
## Phase 3 — Docker Release (REQ-046)
|
||||
|
||||
**Goal**: Multi-stage Dockerfile; publish to Gitea container registry
|
||||
per release.
|
||||
|
||||
**Persona**: devops-engineer.
|
||||
|
||||
**Wave 1**:
|
||||
|
||||
| Task | File(s) | Persona | REQ |
|
||||
|------|---------|---------|-----|
|
||||
| T3.1: Write `Dockerfile` (multi-stage: golang:1.25 → distroless/static-debian12) | `Dockerfile` | devops-engineer | REQ-046 |
|
||||
| T3.2: Extend `scripts/release.sh` with docker build + login + push | `scripts/release.sh` | devops-engineer | REQ-046 |
|
||||
| T3.3: Add `container-publish` step to `.coreci.yml` release pipeline | `.coreci.yml` | devops-engineer | REQ-046 |
|
||||
| T3.4: Write `docs/docker.md` (docker run quickstart, volume mounts, ORCA_HOME) | `docs/docker.md` | devops-engineer | REQ-046 |
|
||||
| T3.5: Add `.dockerignore` (exclude .git, bin, .env, *.tar.gz) | `.dockerignore` | devops-engineer | REQ-046 |
|
||||
|
||||
**Dockerfile spec** (per R-005):
|
||||
- Stage 1 (`golang:1.25`): `CGO_ENABLED=0 go build -trimpath -ldflags=... -o /orca ./cmd/orca`.
|
||||
- Stage 2 (`gcr.io/distroless/static-debian12:nonroot`): `COPY --from=builder /orca /orca`, `ENV ORCA_HOME=/var/lib/orca`, `ENTRYPOINT ["/orca"]`.
|
||||
- `ARG VERSION` + `ARG GIT_COMMIT` + `ARG BUILD_TIME` for ldflags injection.
|
||||
- Image runs as `nonroot` user (distroless default) — `ORCA_HOME=/var/lib/orca` must be volume-mounted.
|
||||
|
||||
**release.sh extension**:
|
||||
- After Gitea release: `docker build --build-arg VERSION=$VERSION ... -t git.cloudinit.dev/coreci/orca:$VERSION -t git.cloudinit.dev/coreci/orca:latest .`
|
||||
- `echo "$GITEA_TOKEN" | docker login git.cloudinit.dev -u cloudinit-bot --password-stdin`
|
||||
- `docker push git.cloudinit.dev/coreci/orca:$VERSION` + `docker push git.cloudinit.dev/coreci/orca:latest`
|
||||
- Skip gracefully if `docker` not on PATH (local dev without docker).
|
||||
|
||||
**.coreci.yml extension**:
|
||||
- New step `container-publish` in the `release` pipeline, using an image with docker CLI (e.g., `docker:24-cli` with docker-in-docker service, or a custom image). Per P-001 pitfall.
|
||||
|
||||
**Must-haves**:
|
||||
- `docker build -t orca-test .` succeeds locally.
|
||||
- `docker run --rm orca-test version` prints the version.
|
||||
- `scripts/release.sh vX.Y.Z` publishes both the Gitea release AND the container image.
|
||||
- `.coreci.yml` release pipeline includes the container-publish step.
|
||||
|
||||
**Verification**: 4-layer (structural: Dockerfile lint; behavioral: docker
|
||||
build + run; security: no secret in image, .env excluded; quality:
|
||||
reproducible build via ARGs).
|
||||
|
||||
**Ship**: tag `v0.4.4`.
|
||||
|
||||
## Phase 4 — Final Review + Ship + Audit (Milestone Release)
|
||||
|
||||
**Goal**: Multi-persona review, audit, milestone ship.
|
||||
|
||||
**Tasks**:
|
||||
| Task | Persona | Detail |
|
||||
|------|---------|--------|
|
||||
| T4.1: `ciagent-review` | all | Review P1-P3 changes across personas |
|
||||
| T4.2: `ciagent-audit` | lead-developer | Reconstruction test, file/branch/commit discipline |
|
||||
| T4.3: End-to-end verification | lead-developer | Unauth curl to releases API (REQ-045 ✓), fresh install.sh (REQ-043 ✓), `--system` (REQ-042 ✓), update-in-place (REQ-044 ✓), docker pull+run (REQ-046 ✓) |
|
||||
| T4.4: Milestone ship | lead-developer | Merge phase/04 → milestone/v0.5 → main, tag v0.4.5, create milestone release, build + upload all artifacts |
|
||||
| T4.5: Complete milestone | lead-developer | Update REQUIREMENTS.md (REQ-041..046 complete), ROADMAP.md (v0.5 complete), clear CHECKPOINT.json |
|
||||
|
||||
**Ship**: tag `v0.4.5` (the milestone release, promoted to `v0.5.0`).
|
||||
|
||||
## Wave Ordering Summary
|
||||
|
||||
All 4 phases are single-wave (no inter-phase dependencies within a
|
||||
phase). Phases execute strictly sequentially: P1 → P2 → P3 → P4.
|
||||
|
||||
- **P1** (Wave 1): T1.1..T1.5 — namespace unification.
|
||||
- **P2** (Wave 1): T2.1..T2.4 — install.sh.
|
||||
- **P3** (Wave 1): T3.1..T3.5 — docker.
|
||||
- **P4** (Wave 1): T4.1..T4.5 — review + ship.
|
||||
|
||||
## Versioning
|
||||
|
||||
- P0 ship: `v0.4.1` (first patch on v0.4.x line after v0.4.0 milestone tag).
|
||||
- P1 ship: `v0.4.2`.
|
||||
- P2 ship: `v0.4.3`.
|
||||
- P3 ship: `v0.4.4`.
|
||||
- P4 ship: `v0.4.5` (final phase = milestone release, promoted to `v0.5.0`).
|
||||
|
||||
Tags run on the v0.4.x line (previous minor). The milestone branch label
|
||||
is `milestone/v0.5-distribution`. No separate minor tag — the final
|
||||
phase's patch IS the milestone release per `run.md` versioning logic
|
||||
for feature milestones.
|
||||
@@ -1,236 +0,0 @@
|
||||
# Phase Plans: Orca v0.6 — Node Bootstrap & Proxmox
|
||||
|
||||
All 3 execution phases + final review with vertical-slice structure,
|
||||
wave ordering, and REQ-ID mapping. v0.6 scope: **Node Bootstrap &
|
||||
Proxmox** — `orca init` full bootstrap, Proxmox SSH join, doctor
|
||||
extensions.
|
||||
|
||||
Branching: branches numbered from phase 12 onward (v0.1 used 01-07,
|
||||
v0.2 used 08-11, v0.3 used 00+01-03, v0.5 used 00+01-04). v0.6 uses
|
||||
`phase/01-*`..`phase/04-*` on the `milestone/v0.6-node-bootstrap-proxmox`
|
||||
branch (numbering restarts per milestone per branch-strategy.md).
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: `orca init` Full Bootstrap + Schema 0006 (Wave 1)
|
||||
|
||||
**Branch**: `phase/01-init-bootstrap`
|
||||
**REQ Coverage**: REQ-047, REQ-048, REQ-049
|
||||
**Persona leads**: data-engineer (schema), backend-engineer (init orchestration), cli-engineer (output UX)
|
||||
|
||||
### Must-Haves
|
||||
|
||||
#### data-engineer territory
|
||||
- [ ] `internal/store/migrations/0006_node_kind_os.sql` — `ALTER TABLE nodes ADD COLUMN kind TEXT; ALTER TABLE nodes ADD COLUMN os TEXT;` (nullable, backward-compatible)
|
||||
- [ ] `internal/model/node.go` — add `Kind string `json:"kind,omitempty"`` + `OS string `json:"os,omitempty"`` fields; add `NodeKind` constants (`NodeKindLocalhost`, `NodeKindLinux`, `NodeKindProxmox`)
|
||||
- [ ] `internal/store/node_repo.go` — extend `Insert`/`Get`/`List`/`Watch`/`scanNode` for `kind, os` columns (use `sql.NullString`, map NULL → `""`); add `GetByName(ctx, name) (*Node, error)` and `UpdateLastSeenAndOS(ctx, id, os string) error` helpers
|
||||
- [ ] `internal/store/node_repo_test.go` — extend tests for new columns + helpers; assert NULL → `""` mapping; assert `GetByName` returns `ErrNotFound` for missing; assert `UpdateLastSeenAndOS` refreshes `last_seen` + `os` without changing `id`/`joined_at`
|
||||
|
||||
#### backend-engineer territory
|
||||
- [ ] `internal/cli/init.go` — full bootstrap sequence (replace current 35-line mkdir-only impl):
|
||||
- [ ] MkdirAll(certpaths.Dir(), 0o755) — keep
|
||||
- [ ] store.Open(certpaths.DBPath()) — runs migrations 0001..0006
|
||||
- [ ] security.CAInit(certpaths.Dir(), "orca-internal-ca") — idempotent (existing fast-path)
|
||||
- [ ] if !exists(certpaths.ServerCertPath()): GenerateCSR("localhost", ["localhost","127.0.0.1"]) → ca.SignCSR → WriteCert + WriteKey
|
||||
- [ ] detectOS() from /etc/os-release (see cli-engineer territory)
|
||||
- [ ] localhost node upsert: GetByName("localhost") → if found UpdateLastSeenAndOS; else Insert with kind=localhost, os=<detected>, name="localhost", addr="localhost:8443"
|
||||
- [ ] print summary (CA fp, server cert fp, os, node id, db path)
|
||||
- [ ] `internal/cli/init_test.go` — idempotency test: run init twice, assert no duplicate localhost node, last_seen refreshed, os unchanged; assert CA/cert not regenerated on re-run; assert doctor passes after init
|
||||
|
||||
#### cli-engineer territory
|
||||
- [ ] `internal/cli/osdetect.go` (NEW) — `detectOS() string`: read `/etc/os-release` then fall back to `/usr/lib/os-release`; parse `KEY=VALUE` lines via bufio.Scanner + strings.SplitN; strip surrounding quotes; return `ID` value or `"linux"` fallback. Map ubuntu/debian/alpine → verbatim; unknown values stored verbatim (not masked).
|
||||
- [ ] `internal/cli/osdetect_test.go` — test parsing with sample os-release content (ubuntu, debian, alpine, missing file, missing ID=, unknown ID, quoted values)
|
||||
- [ ] `internal/cli/init.go` output UX — multi-step progress lines: "✓ Namespace dir: ...", "✓ Database initialized: ...", "✓ CA provisioned: ... (fp=...)", "✓ Server cert provisioned: ... (fp=...)", "✓ OS detected: ubuntu", "✓ Localhost node registered: <id>"; `--json` outputs a single JSON summary object
|
||||
|
||||
### Verification
|
||||
|
||||
- `go build ./...` PASS
|
||||
- `go test ./internal/store/... ./internal/cli/... ./internal/model/...` PASS
|
||||
- `go test -race ./...` PASS
|
||||
- `orca init` on a fresh namespace → creates dir, db, CA, server cert, localhost node; `orca doctor` passes with zero FAILs
|
||||
- `orca init` re-run → no duplicate localhost node, last_seen refreshed, CA/cert not regenerated (idempotent, D-036)
|
||||
- `orca init --json` → valid JSON summary
|
||||
- `orca node list` shows the localhost node with kind=localhost, os=<detected>
|
||||
- Migration 0006 applies cleanly on existing dbs (existing rows get NULL kind/os → scanned as `""`)
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Proxmox SSH Join (Wave 1)
|
||||
|
||||
**Branch**: `phase/02-proxmox-join`
|
||||
**REQ Coverage**: REQ-050, REQ-051
|
||||
**Persona leads**: security-engineer (SSH key, TOFU, sudoers, PVE role), backend-engineer (SSH session orchestration), cli-engineer (flag wiring)
|
||||
**Depends on**: Phase 1 (migration 0006 + Node.Kind/OS fields)
|
||||
|
||||
### Must-Haves
|
||||
|
||||
#### dependency + security-engineer territory
|
||||
- [ ] `go.mod` / `go.sum` — add `golang.org/x/crypto v0.54.0`; bump `golang.org/x/sys` to v0.47.0; add `golang.org/x/term v0.45.0` (indirect). Run `go mod tidy`.
|
||||
- [ ] `internal/certpaths/certpaths.go` — add `SSHKeyPath() → Dir()/orca_ssh_key`, `SSHPubPath() → Dir()/orca_ssh_key.pub`, `KnownHostsPath() → Dir()/known_hosts`
|
||||
- [ ] `internal/security/sshkey.go` (NEW) — `GenerateOrLoadSSHKey(dir string) (keyPEM, pubLine []byte, err error)`:
|
||||
- [ ] If `orca_ssh_key` + `.pub` exist → load + return (idempotent)
|
||||
- [ ] Else: `ed25519.GenerateKey(rand.Reader)` → `x509.MarshalPKCS8PrivateKey` → PEM encode → `writeAtomic(keyPath, 0600, keyPEM)`; `ssh.NewPublicKey(pub)` → `ssh.MarshalAuthorizedKey` → `writeAtomic(pubPath, 0644, pubLine)`
|
||||
- [ ] Return keyPEM (for `ssh.ParsePrivateKey`) + pubLine (authorized_keys line)
|
||||
- [ ] `internal/security/sshkey_test.go` — test generate → load round-trip; test idempotent re-load; test file modes (0600/0644); test `ssh.ParsePrivateKey` accepts the PKCS8 PEM
|
||||
|
||||
#### backend-engineer territory (with security-engineer co-own)
|
||||
- [ ] `internal/proxmox/bootstrap.go` (NEW package) — `BootstrapProxmox(ctx context.Context, opts Options) (*Result, error)`:
|
||||
- **Options**: `Host, SSHUser, Password, ProxmoxUser (default "orca"), ProxmoxRole (default "OrcaOperator"), Port (default 22)`, `Logger *slog.Logger`
|
||||
- **Step 1**: `security.GenerateOrLoadSSHKey(certpaths.Dir())` → keyPEM, pubLine
|
||||
- **Step 2**: Build `ssh.ClientConfig` with `ssh.Password(opts.Password)` auth + `knownhosts.New(certpaths.KnownHostsPath())` HostKeyCallback (TOFU: captures on first connect, verifies on subsequent)
|
||||
- **Step 3**: `ssh.Dial("tcp", host:port, config)` with 10s timeout
|
||||
- **Step 4**: Deploy pubkey — `session.CombinedOutput("mkdir -p ~orca/.ssh && touch ~orca/.ssh/authorized_keys && chmod 0700 ~orca/.ssh && chmod 0600 ~orca/.ssh/authorized_keys && grep -qF '<publine>' ~orca/.ssh/authorized_keys || echo '<publine>' >> ~orca/.ssh/authorized_keys")` (idempotent append)
|
||||
- **Step 5**: Create orca system user — `session.CombinedOutput("id -u orca 2>/dev/null || useradd -m -s /bin/bash orca")` (idempotent)
|
||||
- **Step 6**: Create PVE role — `session.CombinedOutput("pveum role list 2>/dev/null | grep -q '^OrcaOperator' || pveum role add OrcaOperator --privs 'VM.Audit Datastore.AllocateSpace SDN.Use'")` (idempotent; use opts.ProxmoxRole for the name)
|
||||
- [ ] Step 7: Create PVE user — `session.CombinedOutput("pveum user list 2>/dev/null | grep -q 'orca@pam' || pveum user add orca@pam -comment 'Orca automation user'")` (idempotent; use opts.ProxmoxUser)
|
||||
- [ ] Step 8: Assign ACL — `session.CombinedOutput("pveum acl modify / -user orca@pam -role OrcaOperator")` (idempotent)
|
||||
- [ ] Step 9: Write sudoers — resolve binary paths via `command -v pct` etc.; write `/etc/sudoers.d/orca` (mode 0440) with NOEXEC on pct/qm, no NOEXEC on apt-get/dpkg; exclude pvesh (AD-020)
|
||||
- [ ] Step 10: Validate sudoers — `session.CombinedOutput("visudo -cf /etc/sudoers.d/orca")`; abort + cleanup if validation fails
|
||||
- [ ] Step 11: Audit log — `logger.Info("proxmox.bootstrap_ok", slog.String("host", opts.Host), slog.String("user", opts.ProxmoxUser), slog.String("role", opts.ProxmoxRole))`
|
||||
- [ ] **Result**: `Node{Kind: "proxmox", OS: "pve", Name: opts.Host, Address: opts.Host + ":8443"}`
|
||||
- [ ] `internal/proxmox/bootstrap_test.go` — unit tests with a mock SSH server (`httptest`-style or `net.Pipe` + manual SSH handshake) OR test the command-builder functions in isolation (probe commands, sudoers content, idempotency checks). Integration test against a real Proxmox host is out of scope for unit tests (flagged as `// +build integration`).
|
||||
|
||||
#### cli-engineer territory
|
||||
- [ ] `internal/cli/node.go` — extend `nodeJoinCmd`:
|
||||
- [ ] Add `--type` flag (values: `localhost` default, `linux`, `proxmox`)
|
||||
- [ ] Add `--host`, `--ssh-user` (default `root`), `--password`, `--proxmox-user` (default `orca`), `--proxmox-role` (default `OrcaOperator`), `--ssh-port` (default `22`) flags
|
||||
- [ ] When `--type proxmox`: validate `--host` + (`--password` or `$ORCA_PROXMOX_PASSWORD`) are set; call `proxmox.BootstrapProxmox(ctx, opts)`; insert the returned node via `NodeRepo.Insert`; print summary
|
||||
- [ ] When `--type localhost` (default): existing flow (fingerprint check + registry.Join)
|
||||
- [ ] Password from `--password` flag OR `$ORCA_PROXMOX_PASSWORD` env var (prefer env var per D-031; never log the password; zero the byte slice after use)
|
||||
- [ ] `internal/cli/node_test.go` — test flag wiring; test `--type proxmox` validation (missing host/password → error); test env var fallback
|
||||
|
||||
### Verification
|
||||
|
||||
- `go build ./...` PASS
|
||||
- `go test ./internal/proxmox/... ./internal/security/... ./internal/cli/...` PASS
|
||||
- `go test -race ./...` PASS
|
||||
- `go mod tidy` leaves no unused deps; `go.sum` has `golang.org/x/crypto v0.54.0`
|
||||
- `orca node join --type proxmox --host <pve-host> --password <pw>` on a real Proxmox 8/9 host:
|
||||
- Creates orcaOperator role, orca@pam user, ACL, sudoers file
|
||||
- `orca@pam` can `sudo pct list`, `sudo qm list`, `sudo apt-get update` without password
|
||||
- `orca@pam` CANNOT `sudo pvesh` (not in sudoers)
|
||||
- `orca@pam` CANNOT `sudo bash` (not in sudoers)
|
||||
- `visudo -cf /etc/sudoers.d/orca` passes
|
||||
- Re-running the join command is idempotent (no duplicate role/user/ACL/sudoers/key)
|
||||
- `orca node list` shows the proxmox node with kind=proxmox, os=pve
|
||||
- Audit log contains `proxmox.bootstrap_ok` entry with host, user, role
|
||||
- `~/.orca/orca_ssh_key` is 0600, `.pub` is 0644, `known_hosts` contains the PVE host key
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Doctor Extensions + Audit Logging (Wave 2)
|
||||
|
||||
**Branch**: `phase/03-doctor-extensions`
|
||||
**REQ Coverage**: REQ-052
|
||||
**Persona leads**: cli-engineer (subcommand wiring), backend-engineer (check logic), security-engineer (audit logging)
|
||||
**Depends on**: Phase 1 (localhost node + os field), Phase 2 (proxmox nodes + SSH client)
|
||||
|
||||
### Must-Haves
|
||||
|
||||
#### backend-engineer territory
|
||||
- [ ] `internal/doctor/doctor.go` — add `OS()` check:
|
||||
- Re-run `detectOS()` (from `internal/cli/osdetect.go` — extract to shared package or pass as param)
|
||||
- Load localhost node via `NodeRepo.GetByName("localhost")`
|
||||
- Compare detected OS to stored `node.OS`; drift → WARN ("OS drift: init=ubuntu, now=debian — re-run `orca init` to refresh"); match → PASS
|
||||
- Missing localhost node → FAIL ("no localhost node — run `orca init`")
|
||||
- [ ] `internal/doctor/doctor.go` — add `Proxmox()` check (clone `Network()` pattern):
|
||||
- List nodes from `NodeRepo`, filter `kind == "proxmox"`
|
||||
- Zero proxmox nodes → WARN ("no proxmox nodes registered (single-node?)")
|
||||
- Per node: load orca SSH key, build `ssh.ClientConfig` with `ssh.PublicKeys(signer)` + `knownhosts.New`, dial with 3s timeout, run `pveversion` via session
|
||||
- PASS = reachable + pveversion exits 0; FAIL = unreachable or pveversion fails
|
||||
- Accumulate per-node lines (clone `Network()`'s `lines []string` pattern)
|
||||
- [ ] `internal/doctor/doctor.go` — extend `All()` to include `OS()` and `Proxmox()`
|
||||
- [ ] `internal/doctor/doctor_test.go` — test `OS()` with mock node repo (drift, match, missing); test `Proxmox()` with mock nodes (zero nodes → WARN, reachable → PASS, unreachable → FAIL)
|
||||
|
||||
#### cli-engineer territory
|
||||
- [ ] `internal/cli/doctor.go` — add `doctorOSCmd` + `doctorProxmoxCmd` subcommands wired to `doctor.OS()` / `doctor.Proxmox()`; add to `doctorCmd.AddCommand(...)`
|
||||
- [ ] `internal/cli/doctor.go` — `doctor os` and `doctor proxmox` honor `--json` flag (reuse existing pattern)
|
||||
|
||||
#### security-engineer territory
|
||||
- [ ] `internal/audit/audit.go` (extend) — emit `proxmox.bootstrap_ok`, `proxmox.bootstrap_fail`, `node.os_drift` events with structured slog fields
|
||||
- [ ] Audit log entries for all bootstrap + join actions (REQ-052): `orca init` emits `init.bootstrap_ok` (os, node_id, ca_fp); `orca node join --type proxmox` emits `proxmox.bootstrap_ok` (host, user, role); `doctor os` drift emits `node.os_drift` (init_os, current_os)
|
||||
|
||||
### Verification
|
||||
|
||||
- `go build ./...` PASS
|
||||
- `go test ./internal/doctor/... ./internal/cli/...` PASS
|
||||
- `go test -race ./...` PASS
|
||||
- `orca doctor` (after `orca init`) → all checks PASS (cert, db, os, network=zero peers WARN, proxmox=zero nodes WARN)
|
||||
- `orca doctor os` → PASS (OS matches)
|
||||
- `orca doctor proxmox` (no proxmox nodes) → WARN ("no proxmox nodes registered")
|
||||
- `orca doctor proxmox` (after joining a PVE host) → PASS per node
|
||||
- `orca doctor proxmox` (PVE host down) → FAIL per node with error message
|
||||
- Audit log contains `init.bootstrap_ok` and `proxmox.bootstrap_ok` entries
|
||||
- `--json` output for `doctor os` and `doctor proxmox` is valid JSON
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Final Review + Ship + Audit (Wave 3)
|
||||
|
||||
**Branch**: `phase/04-final-review-ship`
|
||||
**REQ Coverage**: REQ-047, REQ-048, REQ-049, REQ-050, REQ-051, REQ-052 (all)
|
||||
**Persona leads**: lead-developer (review + audit), all personas (post-hoc review)
|
||||
|
||||
### Must-Haves
|
||||
|
||||
- [ ] **Review** (delegate to `ciagent-review`): multi-persona code review across P01-P03
|
||||
- Auto-apply P0 fixes; flag P1+ for post-hoc review
|
||||
- Review territory discipline (warn mode)
|
||||
- Review test coverage for all 6 REQs
|
||||
- [ ] **Audit** (delegate to `ciagent-audit`):
|
||||
- Reconstruction test: git log matches `.ciagent/` files
|
||||
- Branch hygiene: phase branches merged cleanly to milestone
|
||||
- Commit discipline: all commits have `---ci---` blocks
|
||||
- File discipline: no stale `.ciagent/` files
|
||||
- [ ] **Ship** (delegate to `ciagent-ship`):
|
||||
- Merge `phase/04` → `milestone/v0.6`
|
||||
- Merge `milestone/v0.6` → `main` (rebase-then-fast-forward per config.json)
|
||||
- Tag `v0.5.4` (final phase patch = milestone release per feature-milestone promotion)
|
||||
- Create Gitea release with full milestone summary (all phases, all REQs)
|
||||
- [ ] **Complete milestone**:
|
||||
- Update `.ciagent/REQUIREMENTS.md` — mark REQ-047..052 as Complete
|
||||
- Update `.ciagent/ROADMAP.md` — mark v0.6 as complete
|
||||
- Update `.ciagent/CHECKPOINT.json` — `milestone_complete: true`
|
||||
- Commit: `docs(milestone): complete node-bootstrap-proxmox`
|
||||
|
||||
### Verification
|
||||
|
||||
- `git log --oneline main..milestone/v0.6` shows all phase commits in order
|
||||
- `git tag --list v0.5.*` shows v0.5.0..v0.5.4
|
||||
- `main` branch contains all v0.6 work (fast-forward merge)
|
||||
- `orca init && orca doctor` on a fresh checkout passes end-to-end
|
||||
- Gitea release `v0.5.4` exists with milestone summary
|
||||
|
||||
---
|
||||
|
||||
## Wave Ordering
|
||||
|
||||
- **Wave 1** (Phases 1-2): Schema + init bootstrap (P01) is a hard
|
||||
prerequisite for Proxmox join (P02) — P02 depends on the `Node.Kind`/
|
||||
`OS` fields + migration 0006 from P01. `parallelization.enabled=false`
|
||||
→ sequential.
|
||||
- **Wave 2** (Phase 3): Doctor extensions depend on both P01 (localhost
|
||||
node + os field for `doctor os`) and P02 (proxmox nodes + SSH client
|
||||
for `doctor proxmox`).
|
||||
- **Wave 3** (Phase 4): Final review + ship + audit — covers all
|
||||
execution phases.
|
||||
|
||||
For v0.6, `parallelization.enabled=false` — phases run sequentially.
|
||||
|
||||
## Versioning
|
||||
|
||||
- **Milestone type**: `feature` (P01/P02/P03 ship `feat` phases)
|
||||
- **Patch per phase**: `v0.5.0` (P0), `v0.5.1` (P01), `v0.5.2` (P02), `v0.5.3` (P03), `v0.5.4` (P04 final = milestone release)
|
||||
- Tags run on the previous minor's patch line (v0.5.x) per branch-strategy.md
|
||||
- Milestone branch label: `milestone/v0.6-node-bootstrap-proxmox` (uses milestone number, not tag line)
|
||||
|
||||
## Requirement Coverage Matrix
|
||||
|
||||
| REQ | Phase | Persona lead | Must-haves |
|
||||
|-----|-------|-------------|------------|
|
||||
| REQ-047 | P01 | backend-engineer | init.go full bootstrap (CA + cert + db + localhost node, idempotent) |
|
||||
| REQ-048 | P01 | backend-engineer + cli-engineer | detectOS() from /etc/os-release + localhost node registration |
|
||||
| REQ-049 | P01 | data-engineer | migration 0006 + Node.Kind/OS + NodeRepo schema extension |
|
||||
| REQ-050 | P02 | security-engineer + backend-engineer | proxmox.BootstrapProxmox SSH dance + sshkey.go + certpaths SSH paths |
|
||||
| REQ-051 | P02 | security-engineer | OrcaOperator PVE role + orca@pam user + sudoers NOEXEC design |
|
||||
| REQ-052 | P03 | backend-engineer + security-engineer | doctor OS() + Proxmox() + audit logging of all bootstrap/join actions |
|
||||
@@ -1,250 +0,0 @@
|
||||
# Phase Plans: Orca v0.7 — Hardening & Completion
|
||||
|
||||
All 4 execution phases + final review with vertical-slice structure, wave
|
||||
ordering, and REQ-ID mapping. v0.7 scope: **Hardening & Completion** —
|
||||
register the unreachable `orca cert` command, add HCL config file parsing,
|
||||
uplift test coverage in core packages, and add the long-deferred pprof
|
||||
endpoint.
|
||||
|
||||
Branching: `phase/01-cert-register`..`phase/05-final-review-ship` on the
|
||||
`milestone/v0.7-hardening-completion` branch (numbering restarts per
|
||||
milestone per branch-strategy.md).
|
||||
|
||||
Milestone type: **NFR** (all phases are fix/test/chore; no `feat` phases).
|
||||
Tags run on the v0.6.x patch line: `v0.6.0` (P0) … `v0.6.5` (P05 =
|
||||
milestone release).
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Register `orca cert` Command Tree + cert_repo Tests (Wave 1)
|
||||
|
||||
**Branch**: `phase/01-cert-register`
|
||||
**REQ Coverage**: REQ-053
|
||||
**Persona leads**: lead-developer (cert registration + smoke test), data-engineer (cert_repo tests)
|
||||
**Source ideas**: I-401, I-402, I-412
|
||||
|
||||
### Must-Haves
|
||||
|
||||
#### lead-developer territory
|
||||
- [ ] `internal/cli/cert.go` — add `init()` that calls `rootCmd.AddCommand(NewCommand(slog.Default()))`. This is the one-line fix that makes the entire `cert ca-init | gen | show | renew | fingerprint` tree reachable. (AD-022)
|
||||
- [ ] `internal/cli/cert_test.go` (NEW) — regression test asserting `rootCmd.Commands()` contains a child whose `Use == "cert"`; assert each subcommand (`ca-init`, `gen`, `show`, `renew`, `fingerprint`) is present on the cert child.
|
||||
- [ ] `internal/cli/cert_smoke_test.go` (NEW) — end-to-end smoke test against a temp `ORCA_HOME`:
|
||||
- [ ] `orca cert ca-init --cn test-ca` → succeeds, `ca.crt` + `ca.key` exist with modes 0644/0600
|
||||
- [ ] `orca cert gen --cn test-server --san localhost --san 127.0.0.1` → succeeds, `server.crt` + `server.key` exist with modes 0644/0600
|
||||
- [ ] `orca cert show` → outputs PEM with no `PRIVATE KEY` blocks (REQ-035 redaction)
|
||||
- [ ] `orca cert fingerprint --which ca` → outputs a 64-char hex SHA-256
|
||||
- [ ] `orca cert fingerprint --which server` → outputs a 64-char hex SHA-256
|
||||
- [ ] `orca cert renew` → succeeds, server cert file mtime updates
|
||||
- [ ] `internal/cli/root_test.go` — extend the existing root test to assert `orca cert` is in the command tree (belt-and-suspenders with cert_test.go)
|
||||
|
||||
#### data-engineer territory
|
||||
- [ ] `internal/store/cert_repo_test.go` (NEW) — table-driven tests for `CertRepo`:
|
||||
- [ ] `Insert` a cert row → `Get` by serial returns matching row
|
||||
- [ ] `Insert` duplicate `serial_hex` → returns error (UNIQUE constraint, I-107)
|
||||
- [ ] `List` returns certs ordered by `issued_at desc`
|
||||
- [ ] Rotation history: Insert 4 certs for the same node → only last N=3 retained (REQ-025); oldest is pruned
|
||||
- [ ] `GetActive` returns the most-recent cert for a node
|
||||
- [ ] `Delete` removes a cert by serial
|
||||
|
||||
### Verification
|
||||
|
||||
- `go build ./...` PASS
|
||||
- `go vet ./...` PASS
|
||||
- `go test ./internal/cli/... ./internal/store/...` PASS
|
||||
- `go test -race ./...` PASS
|
||||
- `./bin/orca cert` → prints help (no longer "unknown command")
|
||||
- `./bin/orca cert ca-init` on a temp `ORCA_HOME` → succeeds
|
||||
- `./bin/orca cert show` → no private key material in output (REQ-035)
|
||||
- cert_repo_test.go covers Insert/Get/List/rotation-prune/duplicate-serial
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: HCL Config File Parsing (Wave 1)
|
||||
|
||||
**Branch**: `phase/02-config-parser`
|
||||
**REQ Coverage**: REQ-054
|
||||
**Persona leads**: backend-engineer (config package), lead-developer (root command --config flag wiring)
|
||||
**Source ideas**: I-406, I-408
|
||||
**Depends on**: Phase 1 (cert registration lands first so the CLI surface is complete before config extends it)
|
||||
|
||||
### Must-Haves
|
||||
|
||||
#### backend-engineer territory
|
||||
- [ ] `internal/config/config.go` (NEW package) — `Config` struct with HCL tags:
|
||||
- [ ] `DBPath string `hcl:"db_path,optional"``
|
||||
- [ ] `ListenAddr string `hcl:"listen_addr,optional"``
|
||||
- [ ] `CAPath string `hcl:"ca_path,optional"``
|
||||
- [ ] `ServerCertPath string `hcl:"server_cert_path,optional"``
|
||||
- [ ] `ServerKeyPath string `hcl:"server_key_path,optional"``
|
||||
- [ ] `NodeCapacity *CapacityConfig `hcl:"node_capacity,block"` (optional block)
|
||||
- [ ] `Load(paths ...string) (*Config, error)` — loads the first existing file from `paths` via `hclsimple.Decode` (reuse the jobspec pattern, `internal/jobspec/spec.go:40`); returns a zero-value `Config` if no file exists (no error)
|
||||
- [ ] `(*Config).MergeOverrides(flags Flags, env Environ) *Config` — applies precedence flag > env > file > default (D-039). Only non-zero flag values override; only set env vars override; file values are the base; missing fields fall back to `certpaths.*` defaults.
|
||||
- [ ] No package-level state (AD-023). `Load` is a pure function.
|
||||
- [ ] `internal/config/config_test.go` (NEW) — table-driven tests:
|
||||
- [ ] Load from a valid HCL file → all fields populated
|
||||
- [ ] Load from a missing file → zero Config, no error
|
||||
- [ ] Load from a malformed HCL file → error
|
||||
- [ ] MergeOverrides: flag wins over env wins over file wins over default (all 4 layers exercised)
|
||||
- [ ] MergeOverrides: empty flag does NOT override a set env value
|
||||
- [ ] MergeOverrides: empty env does NOT override a set file value
|
||||
- [ ] Optional `node_capacity` block parsed correctly
|
||||
|
||||
#### lead-developer territory
|
||||
- [ ] `internal/cli/root.go` — add `--config string` persistent flag (default `""`). In `PersistentPreRunE`, if `--config` is set, call `config.Load(flag)` and stash the `*Config` in `cmd.Context()` via a context key. If `--config` is empty, `config.Load` is not called (zero overhead; existing flag/env behavior unchanged).
|
||||
- [ ] `internal/cli/daemon.go` — in the daemon command, if a `*Config` is present in the context, use `cfg.ListenAddr` as the default addr (flag still overrides per D-039).
|
||||
- [ ] `internal/cli/root_test.go` — extend with `--config <tmpfile>` test: pass a config file, assert the merged values reach the daemon command.
|
||||
- [ ] `testdata/config.hcl` (NEW) — example config file for tests:
|
||||
```hcl
|
||||
db_path = "/tmp/orca/test.db"
|
||||
listen_addr = "127.0.0.1:9999"
|
||||
ca_path = "/tmp/orca/ca.crt"
|
||||
server_cert_path = "/tmp/orca/server.crt"
|
||||
server_key_path = "/tmp/orca/server.key"
|
||||
```
|
||||
|
||||
### Verification
|
||||
|
||||
- `go build ./...` PASS
|
||||
- `go vet ./...` PASS
|
||||
- `go test ./internal/config/... ./internal/cli/...` PASS
|
||||
- `go test -race ./...` PASS
|
||||
- `./bin/orca --config testdata/config.hcl daemon --help` → no error
|
||||
- Precedence test: flag value overrides config file value for the same key
|
||||
- No new direct deps (`hashicorp/hcl/v2` already in go.mod)
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Test Coverage Uplift (Wave 1)
|
||||
|
||||
**Branch**: `phase/03-coverage-uplift`
|
||||
**REQ Coverage**: REQ-055
|
||||
**Persona leads**: lead-developer (engine/transport/audit tests), data-engineer (store coverage)
|
||||
**Source ideas**: I-403, I-404, I-405, I-410
|
||||
**Depends on**: Phase 1 + Phase 2 (tests build on the now-reachable cert tree + config package)
|
||||
|
||||
### Must-Haves
|
||||
|
||||
#### lead-developer territory — internal/engine
|
||||
- [ ] `internal/engine/executor_test.go` (NEW) — test `Executor.Start`/`Wait` lifecycle:
|
||||
- [ ] Start a command (`/bin/echo hello`) → Wait → exit code 0, stdout captured
|
||||
- [ ] Start a failing command (`/bin/false`) → exit code non-zero
|
||||
- [ ] Cancel via ctx → process killed, `WaitDelay` honored (REQ-021)
|
||||
- [ ] Env propagation: `Env=["FOO=bar"]` → child process sees `FOO=bar`
|
||||
- [ ] `internal/engine/dispatcher_test.go` (NEW) — test `Dispatcher.Submit`/`Dispatch`:
|
||||
- [ ] Submit a job → dispatched to the correct peer (mock peer client)
|
||||
- [ ] Idempotency key present → retry on transient failure (mock returns error twice then succeeds)
|
||||
- [ ] Idempotency key absent → no retry (REQ-037)
|
||||
- [ ] Bounded queue backpressure: fill the channel → Submit blocks (with timeout assertion)
|
||||
- [ ] `internal/engine/peer_test.go` (NEW) — test the peer HTTP client:
|
||||
- [ ] `httptest.NewTLSServer` mock → peer client POSTs a dispatch request
|
||||
- [ ] TLS handshake failure → structured error with `peer` + `err` fields
|
||||
|
||||
#### lead-developer territory — internal/transport
|
||||
- [ ] `internal/transport/mtls_test.go` (NEW) — test mTLS handshake:
|
||||
- [ ] `httptest.NewTLSServer` with a test CA → client with valid cert handshakes OK
|
||||
- [ ] Client with expired cert → handshake fails with `event=mtls.handshake` log assertion
|
||||
- [ ] Client with wrong CA → handshake fails
|
||||
- [ ] `internal/transport/dispatch_test.go` (NEW) — test `Dispatch` RPC:
|
||||
- [ ] Successful dispatch → 200 OK
|
||||
- [ ] Dispatch with `X-Orca-Idempotency-Key` → idempotent
|
||||
- [ ] Dispatch without key → 400 (per REQ-037)
|
||||
- [ ] `internal/transport/handshake_log_test.go` (NEW) — assert `LogHandshakeOK`/`LogHandshakeFailed` emit the correct slog fields (`event`, `peer`, `cert_fp`, `err`)
|
||||
|
||||
#### lead-developer territory — internal/audit
|
||||
- [ ] `internal/audit/audit_test.go` (NEW) — test the `Audit` wrapper:
|
||||
- [ ] `Emit` with `ActionCertIssued` + `ResultSuccess` → `engine.Record` called with correct args (mock `engine.Audit`)
|
||||
- [ ] `EmitWithErr` → `engine.Record` called with `result=failure` + err in metadata
|
||||
- [ ] `LogHandshakeOK` → slog output contains `event=mtls.handshake`, `result=ok`, `peer`, `cert_fp`
|
||||
- [ ] `LogHandshakeFailed` → slog output contains `result=failed` + `err`
|
||||
- [ ] Nil-safe: `(*Audit)(nil).Emit(...)` → no panic
|
||||
|
||||
#### data-engineer territory — internal/proxmox
|
||||
- [ ] `internal/proxmox/bootstrap_test.go` — extend the existing test:
|
||||
- [ ] Mock the `sshDialer` interface (already present at `bootstrap.go:211`) → assert the full bootstrap sequence calls the right shell commands in order (user create, role create, role assign, sudoers drop, pubkey deploy)
|
||||
- [ ] Idempotent re-run: mock returns "already exists" for user create → bootstrap succeeds without re-creating
|
||||
- [ ] SSH auth failure → bootstrap returns wrapped error
|
||||
- [ ] Assert no password is logged (D-031)
|
||||
|
||||
#### CI gate (I-410)
|
||||
- [ ] `.coreci.yml` — add a `coverage-gate` step in the `test` pipeline that runs `go test -cover ./internal/engine ./internal/transport ./internal/proxmox ./internal/audit` and fails if any package < 50% (AD-025). Use a small shell snippet + `awk`/`grep` to parse coverage percentages.
|
||||
|
||||
### Verification
|
||||
|
||||
- `go build ./...` PASS
|
||||
- `go test -race ./...` PASS
|
||||
- `go test -cover ./internal/engine` → ≥ 50% (was 8.3%)
|
||||
- `go test -cover ./internal/transport` → ≥ 50% (was 26.3%)
|
||||
- `go test -cover ./internal/proxmox` → ≥ 50% (was 5.1%)
|
||||
- `go test -cover ./internal/audit` → ≥ 50% (was 0%)
|
||||
- CI coverage gate step passes
|
||||
- Any races uncovered by `-race` are fixed in this phase (not deferred)
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: `--pprof` Opt-in on `orca daemon` (Wave 1)
|
||||
|
||||
**Branch**: `phase/04-pprof-daemon`
|
||||
**REQ Coverage**: REQ-056
|
||||
**Persona leads**: lead-developer (daemon flag + pprof server)
|
||||
**Source ideas**: I-407, I-409
|
||||
**Depends on**: Phase 3 (daemon tests exist; pprof adds a new daemon path)
|
||||
|
||||
### Must-Haves
|
||||
|
||||
#### lead-developer territory
|
||||
- [ ] `internal/daemon/pprof.go` (NEW) — `StartPprof(addr string, log *slog.Logger) (*http.Server, error)`:
|
||||
- [ ] Create a dedicated `*http.ServeMux` (NOT `http.DefaultServeMux`)
|
||||
- [ ] `import _ "net/http/pprof"` → register `pprof.Index`, `pprof.Cmdline`, `pprof.Profile`, `pprof.Symbol`, `pprof.Trace`, `pprof.Handler` on the dedicated mux
|
||||
- [ ] Return a `*http.Server` listening on `addr` with the dedicated mux
|
||||
- [ ] Log a WARN: `pprof endpoint exposed unauthenticated on <addr> — operator-only, do not expose publicly`
|
||||
- [ ] Never touch the mTLS daemon listener (AD-024)
|
||||
- [ ] `internal/daemon/server.go` — add a `pprofAddr string` field to `Options` (default `""` = disabled). In `Start`, if `pprofAddr != ""`, call `StartPprof` and store the `*http.Server` for `Shutdown`.
|
||||
- [ ] `internal/daemon/pprof_test.go` (NEW) — test:
|
||||
- [ ] `StartPprof("127.0.0.1:0", ...)` → server starts, GET `/debug/pprof/` returns 200
|
||||
- [ ] GET `/debug/pprof/cmdline` returns the cmdline
|
||||
- [ ] `Shutdown` stops the pprof server
|
||||
- [ ] The mTLS daemon server (if running) is unaffected by pprof start/stop
|
||||
- [ ] `internal/cli/daemon.go` — add `--pprof string` flag (default `""` = disabled). Pass it into `daemon.Options.PprofAddr`. Document in `--help`: "enable pprof endpoint on <addr> (e.g. :6060); unauthenticated, operator-only".
|
||||
- [ ] `internal/cli/daemon_test.go` — extend: `--pprof 127.0.0.1:0` → daemon starts with pprof; flag absent → no pprof server.
|
||||
|
||||
### Verification
|
||||
|
||||
- `go build ./...` PASS
|
||||
- `go vet ./...` PASS
|
||||
- `go test -race ./internal/daemon/...` PASS
|
||||
- `./bin/orca daemon --pprof 127.0.0.1:0` (in background) → `curl http://127.0.0.1:<port>/debug/pprof/` returns 200
|
||||
- `./bin/orca daemon` (no `--pprof`) → no pprof listener, `/debug/pprof/` not reachable on the daemon port
|
||||
- pprof mux is separate from the mTLS daemon mux (asserted in test)
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: Final Review + Ship + Audit (Wave 1)
|
||||
|
||||
**Branch**: `phase/05-final-review-ship`
|
||||
**REQ Coverage**: all (REQ-053..056)
|
||||
**Persona leads**: lead-developer (review + audit + ship)
|
||||
|
||||
### Must-Haves
|
||||
|
||||
- [ ] Multi-persona code review across all v0.7 phases (ciagent-review)
|
||||
- [ ] Audit: reconstruction test (git log matches `.ciagent/` files), branch hygiene, commit discipline (ciagent-audit)
|
||||
- [ ] Fix any P0 issues found by review; record P1+ in `.ciagent/` for post-hoc
|
||||
- [ ] Merge `phase/05` → `milestone/v0.7-hardening-completion`
|
||||
- [ ] Merge `milestone/v0.7` → `main` (rebase-then-fast-forward per config)
|
||||
- [ ] Tag `v0.6.5` (final phase patch = milestone release)
|
||||
- [ ] Create Gitea release with full milestone summary (all phases, all REQs)
|
||||
- [ ] Update `.ciagent/REQUIREMENTS.md` — mark REQ-053..056 complete
|
||||
- [ ] Update `.ciagent/ROADMAP.md` — mark v0.7 complete
|
||||
- [ ] Write checkpoint: `{phase: 5, stage: "complete", phase_role: "final", milestone_complete: true}`
|
||||
- [ ] Clear checkpoint (milestone complete; next run starts a new milestone)
|
||||
|
||||
### Verification
|
||||
|
||||
- `make build` PASS
|
||||
- `make test` PASS
|
||||
- `make lint` PASS
|
||||
- `go vet ./...` PASS
|
||||
- `git log` on main shows all v0.7 phase commits
|
||||
- `git tag --list 'v0.6.*'` shows v0.6.0..v0.6.5
|
||||
- REQUIREMENTS.md shows REQ-053..056 as Complete
|
||||
- ROADMAP.md shows v0.7 as COMPLETE
|
||||
@@ -1,347 +0,0 @@
|
||||
# Phase Plans: Orca v0.8 — Coverage & Trust Hardening
|
||||
|
||||
All 4 execution phases + final review with vertical-slice structure, wave
|
||||
ordering, persona assignment, and REQ-ID mapping. v0.8 scope: **Coverage &
|
||||
Trust Hardening** — round-2 test coverage uplift across 9 packages (tiered
|
||||
floor: ≥70% for 6 retested, ≥50% for 3 zero-test per D-047), SSH trust
|
||||
hardening (`--host-key-fingerprint` pre-pin + `orca node key-reset` + latent
|
||||
TOFU capture-fix + `Result.HostKeyFingerprint` population), and a
|
||||
requirements-hygiene gate (`make verify-reqs`).
|
||||
|
||||
Branching: `phase/01-coverage-round2`..`phase/04-final-review-ship` on the
|
||||
`milestone/v0.8-coverage-trust-hardening` branch (numbering restarts per
|
||||
milestone per branch-strategy.md).
|
||||
|
||||
Milestone type: **NFR** (P01 test, P02 chore on the trust surface per D-043,
|
||||
P03 chore, P04 docs/review). Tags run on the v0.7.x patch line: `v0.7.0`
|
||||
(P0) … `v0.7.4` (P04 = milestone release).
|
||||
|
||||
**Vertical-slice integrity**: each phase is independently shippable.
|
||||
- **P01** ships tests-only (no production code changes except the proxmox
|
||||
`sessionRunner` seam, a backward-compatible interface extraction, and the
|
||||
engine `peerDispatcher` seam per RESEARCH §1.3).
|
||||
- **P02** ships the SSH trust features + TOFI bugfix + `Result` population.
|
||||
- **P03** ships the hygiene gate (Go program + Makefile + CI hook).
|
||||
- **P04** is review + ship + audit (no new REQs).
|
||||
|
||||
**Out of scope for v0.8** (candidate for v0.9, noted not added):
|
||||
- Lifting the 3 zero-test packages from 50% → 70% (D-047 explicitly
|
||||
toes-holds them; v0.9 can raise the floor).
|
||||
- A `peerDispatcher` interface seam in engine beyond what P01 needs for 70%
|
||||
coverage (httptest.NewTLSServer suffices; the seam is only added if
|
||||
coverage cannot otherwise hit 70%).
|
||||
- Pre-populating `known_hosts` from a remote keyscan API (TOFU + manual
|
||||
`--host-key-fingerprint` cover the v0.8 trust surface).
|
||||
- `verify-reqs` reverse-direction check (REQUIREMENTS Complete ↔ ROADMAP
|
||||
COMPLETE both ways) — forward direction (ROADMAP-shipped → REQUIREMENTS
|
||||
Complete) is the priority per the v0.7 drift that motivated REQ-060.
|
||||
|
||||
**Carried-forward research findings** (RESEARCH_v0.8.md, must incorporate):
|
||||
- §1.1 per-package coverage strategies + tiered floors (D-047).
|
||||
- §1.3 injected seams: reuse `sshDialer` (proxmox), `LocalExecutor` (engine),
|
||||
`Dispatcher` (transport), `watchInterval` (store), `openTestDB`/`withFastWatch`/`initTestEnv`/`resetRootFlags`/`stubDispatcher` helpers.
|
||||
- §1.4 realism flags: cli excludes `daemon.go`; `cmd/orca` 50% toe-hold only;
|
||||
proxmox needs the `sessionRunner` seam to hit 70%.
|
||||
- §2.1 latent TOFU capture bug (knownhosts.New returns KeyError{Want:[]} on
|
||||
first connect and does NOT auto-write — current BootstrapProxmox treats it
|
||||
as a dial failure).
|
||||
- §2.2 `Result.HostKeyFingerprint` is declared but never populated (always
|
||||
`""`); P02 must add `ssh.FingerprintSHA256` computation.
|
||||
- §2.3 `--host-key-fingerprint` plugs in at `internal/cli/node.go` (flag) +
|
||||
`internal/proxmox/bootstrap.go` (pinned callback).
|
||||
- §2.4 `key-reset` is local-known_hosts-only (D-046), atomic rewrite (AD-029).
|
||||
- §3 verify-reqs is a Go program at `cmd/verify-reqs/main.go` (~80 LOC,
|
||||
stdlib only, AD-030) + `make verify-reqs` + `.coreci.yml` validate hook.
|
||||
- §4 AD-025..AD-030 (renumbered AD-027..AD-030 in research for SSH/trust;
|
||||
AD-025/AD-026 from earlier milestones are stable).
|
||||
- §5 10 pitfalls carried into the risk register at the end of this file.
|
||||
|
||||
**Dependencies (RESEARCH §6)**: v0.8 adds **zero** new direct dependencies.
|
||||
`ssh.FingerprintSHA256`, `knownhosts.Line`/`Normalize`/`KeyError` are in the
|
||||
existing `golang.org/x/crypto` v0.54.0 dep. `verify-reqs` is stdlib-only.
|
||||
`go.mod` is unchanged by v0.8.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Coverage Uplift Round 2 (REQ-057)
|
||||
|
||||
**Branch**: `phase/01-coverage-round2`
|
||||
**REQ Coverage**: REQ-057
|
||||
**Tag**: `v0.7.1`
|
||||
**Depends on**: Phase 0 (this plan + clarify + research)
|
||||
**Source research**: RESEARCH_v0.8.md §1 (per-package strategies, helpers, seams)
|
||||
|
||||
### Tiered floor (D-047)
|
||||
|
||||
| Package | Current | Floor | Owner persona |
|
||||
|---------|---------|-------|---------------|
|
||||
| `internal/engine` | 8.3% | ≥ 70% | backend-engineer |
|
||||
| `internal/proxmox` | 5.1% | ≥ 70% | backend-engineer |
|
||||
| `internal/cli` | 27.6% | ≥ 70% (excluding `daemon.go`) | lead-developer |
|
||||
| `internal/transport` | 26.3% | ≥ 70% | backend-engineer |
|
||||
| `internal/store` | 47.2% | ≥ 70% | data-engineer |
|
||||
| `internal/jobspec` | 47.6% | ≥ 70% | data-engineer |
|
||||
| `internal/audit` | 0% (no tests) | ≥ 50% toe-hold | data-engineer |
|
||||
| `internal/certpaths` | 0% (no tests) | ≥ 50% toe-hold | data-engineer |
|
||||
| `cmd/orca` | 0% (no tests) | ≥ 50% toe-hold | lead-developer |
|
||||
|
||||
### Wave 1 — Seams + foundational test helpers (no production logic changes)
|
||||
|
||||
These are backward-compatible interface extractions that unlock the bulk of
|
||||
coverage in Wave 2. They are the only production-code changes in P01; all
|
||||
other P01 tasks add `_test.go` files only.
|
||||
|
||||
| Task ID | Owner | Wave | Must | Title | Files touched | Acceptance criterion |
|
||||
|---------|-------|------|------|-------|---------------|----------------------|
|
||||
| T01.1 | backend-engineer | 1 | Y | Add `sessionRunner` interface seam to proxmox | `internal/proxmox/bootstrap.go` | Extract a `sessionRunner` interface (`CombinedOutput(cmd string) ([]byte, error)`) ~10 LOC; default impl wraps `*ssh.Client.NewSession().CombinedOutput(...)`; `runRemote`/`deployPubKey`/`createLinuxUser`/`createPVERole`/`createPVEUser`/`assignPVEACL`/`writeSudoers`/`validateSudoers` use the seam. Backward compatible: existing callers unchanged. `go build ./internal/proxmox` PASS. (RESEARCH §1.3 gap #1, §5 pitfall #3) |
|
||||
| T01.2 | backend-engineer | 1 | N | Add `peerDispatcher` seam to engine (only if needed for 70%) | `internal/engine/dispatcher.go` | Extract a `peerDispatcher` interface (`Submit(ctx, spec, key) (*SubmitResponse, error)`) so `dispatchToPeer` is testable without `httptest.NewTLSServer`. **Only add if T01.5 cannot otherwise hit 70% via `httptest.NewTLSServer` alone.** If added, backward compatible. (RESEARCH §1.3 gap #2, §5 pitfall #8) |
|
||||
|
||||
### Wave 2 — Per-package coverage tests (build on Wave 1 seams)
|
||||
|
||||
| Task ID | Owner | Wave | Must | Title | Files touched | Acceptance criterion |
|
||||
|---------|-------|------|------|-------|---------------|----------------------|
|
||||
| T01.3 | backend-engineer | 2 | Y | `internal/transport` tests → ≥ 70% | `internal/transport/mtls_test.go` (NEW), `internal/transport/dispatch_test.go` (NEW), `internal/transport/handshake_log_test.go` (NEW), `internal/transport/retry_test.go` (NEW, extend) | `httptest.NewTLSServer` with a test CA (reuse `security.CAInit`/`GenerateCSR`/`SignCSR` per RESEARCH §1.2) for mTLS handshake paths; `stubDispatcher` (daemon/dispatch_test.go:24) pattern for Dispatch RPC; capture slog via a test `slog.Handler` for handshake_log. `go test -cover ./internal/transport` → ≥ 70% (was 26.3%). |
|
||||
| T01.4 | backend-engineer | 2 | Y | `internal/engine` tests → ≥ 70% | `internal/engine/executor_test.go` (NEW), `internal/engine/dispatcher_test.go` (NEW), `internal/engine/peer_test.go` (NEW), `internal/engine/scheduler_test.go` (extend), `internal/engine/registry_test.go` (NEW, if registry exists) | `Executor.Start`/`Wait` lifecycle (echo/false/ctx-cancel/Env propagation per REQ-021); `Dispatcher.Submit` with stubbed `LocalExecutor` + (if T01.2 added) stubbed `peerDispatcher` OR `httptest.NewTLSServer`; `PeerRegistry` in-memory Add/Remove/All/Get. Reuse `openTestDB` (node_repo_test.go:12). `go test -cover ./internal/engine` → ≥ 70% (was 8.3%). |
|
||||
| T01.5 | backend-engineer | 2 | Y | `internal/proxmox` tests → ≥ 70% | `internal/proxmox/bootstrap_test.go` (extend) | Swap `sshDialer` (existing seam) for a fake returning a mock `*ssh.Client`; swap `sessionRunner` (T01.1 seam) for a fake that returns canned `CombinedOutput` bytes. Assert full bootstrap sequence calls the right shell commands in order; idempotent re-run ("already exists" → no-op); SSH auth failure → wrapped error; no password logged (D-031). `go test -cover ./internal/proxmox` → ≥ 70% (was 5.1%). |
|
||||
| T01.6 | lead-developer | 2 | Y | `internal/cli` tests → ≥ 70% (excluding daemon.go) with GRILL condition #3 escape valve | `internal/cli/node_test.go` (NEW), `internal/cli/job_test.go` (NEW), `internal/cli/cert_test.go` (NEW), `internal/cli/doctor_test.go` (NEW), `internal/cli/audit_test.go` (NEW), `internal/cli/status_test.go` (NEW), `internal/cli/version_test.go` (NEW), `internal/cli/node_capacity_test.go` (NEW) | Table-driven `rootCmd.Execute()` against temp `ORCA_HOME` per subcommand (reuse `initTestEnv`/`resetRootFlags`/`discardWriter` per RESEARCH §1.2). Mock the proxmox path via `sshDialer` + `sessionRunner` seams. `daemon.go` is excluded — covered by `internal/daemon/server_test.go`. `go test -cover ./internal/cli` → ≥ 70% of non-daemon files (document the exclusion in a test-file comment). **GRILL condition #3 escape valve**: if 70% is not reached after Wave 2 effort and ≥ 65% is achieved (RESEARCH §1.4 flags 55-65% as realistic for one phase), ship cli at 65% and do NOT block P02/P03 on the last 5%; record the shortfall + rationale in the P01 verification commit. |
|
||||
| T01.7 | data-engineer | 2 | Y | `internal/store` tests → ≥ 70% (incl. missing `cert_repo_test.go`) | `internal/store/cert_repo_test.go` (NEW — v0.7 P01 leftover, RESEARCH §1.1), `internal/store/node_repo_test.go` (extend), `internal/store/job_task_repo_test.go` (extend), `internal/store/audit_repo_test.go` (extend), `internal/store/capacity_repo_test.go` (extend) | `cert_repo_test.go`: Insert/Get/List/ListByNode/LatestForKind/PruneOlderThan/Delete + N=3 rotation history per REQ-025 + duplicate-serial error. Reuse `openTestDB`/`withFastWatch` (RESEARCH §1.2). `go test -cover ./internal/store` → ≥ 70% (was 47.2%). |
|
||||
| T01.8 | data-engineer | 2 | Y | `internal/jobspec` tests → ≥ 70% | `internal/jobspec/spec_test.go` (extend), `internal/jobspec/testdata/*.hcl` (NEW golden fixtures) | Golden-file HCL fixtures (multi-task, env vars, args) + error-path table (missing job, no tasks, missing command, malformed HCL, empty file, nonexistent file for `ParseFile`). `go test -cover ./internal/jobspec` → ≥ 70% (was 47.6%). |
|
||||
| T01.9 | data-engineer | 2 | Y | `internal/audit` first tests → ≥ 50% toe-hold | `internal/audit/audit_test.go` (NEW) | Construct `Audit` with real `engine.Audit` backed by `:memory:` sqlite (via `store.NewAuditRepo` + `engine.NewAudit`); assert rows in `audit_log` table; capture slog via a test `slog.Handler` for `LogHandshakeOK`/`LogHandshakeFailed`. `go test -cover ./internal/audit` → ≥ 50% (was 0%). |
|
||||
| T01.10 | data-engineer | 2 | Y | `internal/certpaths` first tests → ≥ 50% toe-hold | `internal/certpaths/certpaths_test.go` (NEW) | Temp dir + `t.Setenv("ORCA_HOME", dir)` + assert each `*Path()` returns `filepath.Join(dir, <file>)`; test `ORCA_DB` override; test default-to-`~/.orca` fallback. Model on `namespace_test.go` (cli). `go test -cover ./internal/certpaths` → ≥ 50% (was 0%). |
|
||||
| T01.11 | lead-developer | 2 | Y | `cmd/orca` smoke test → ≥ 50% toe-hold | `cmd/orca/main_test.go` (NEW), possibly `cmd/orca/main.go` (refactor `main()` into `run() int` for testability) | Refactor `main()` to `run() int` (returns exit code; `main()` calls `os.Exit(run())`) so the test can call `run()` directly with a forced error path and assert non-zero exit + stderr contains "error:". Low-effort toe-hold — do NOT over-invest (RESEARCH §1.1, §5 pitfall #6). `go test -cover ./cmd/orca` → ≥ 50% (was 0%). |
|
||||
|
||||
### Wave 3 — Coverage gate verification
|
||||
|
||||
| Task ID | Owner | Wave | Must | Title | Files touched | Acceptance criterion |
|
||||
|---------|-------|------|------|-------|---------------|----------------------|
|
||||
| T01.12 | lead-developer | 3 | Y | Coverage-gate verification (all 9 packages hit tiered floor) | none (verification only) | `go test -cover ./internal/engine ./internal/proxmox ./internal/cli ./internal/transport ./internal/store ./internal/jobspec` → each ≥ 70%; `go test -cover ./internal/audit ./internal/certpaths ./cmd/orca` → each ≥ 50%. `go test -race ./...` PASS. Any races fixed in-phase (not deferred). |
|
||||
|
||||
### Phase 1 Must-Haves (summary)
|
||||
|
||||
All 9 packages hit their tiered floor (D-047): T01.1, T01.3, T01.4, T01.5,
|
||||
T01.6, T01.7, T01.8, T01.9, T01.10, T01.11, T01.12. T01.2 is conditional
|
||||
(only if needed for engine 70%).
|
||||
|
||||
### Phase 1 Verification
|
||||
|
||||
- `go build ./...` PASS
|
||||
- `go vet ./...` PASS
|
||||
- `go test -race ./...` PASS
|
||||
- Per-package coverage hits the tiered floor (T01.12)
|
||||
- The proxmox `sessionRunner` seam is backward compatible (existing
|
||||
`BootstrapProxmox` callers unchanged)
|
||||
- No new direct deps (`go.mod` unchanged)
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: SSH Trust Hardening (REQ-058, REQ-059)
|
||||
|
||||
**Branch**: `phase/02-ssh-trust-hardening`
|
||||
**REQ Coverage**: REQ-058, REQ-059
|
||||
**Tag**: `v0.7.2`
|
||||
**Depends on**: Phase 1 (proxmox `sessionRunner` seam from T01.1 is in place;
|
||||
the trust-surface code is now testable)
|
||||
**Source research**: RESEARCH_v0.8.md §2 (TOFU bug, fingerprint computation,
|
||||
flag wiring, key-reset atomic rewrite) + §4 AD-027..AD-029
|
||||
**Phase type**: chore (trust-surface hardening per D-043 — refines existing
|
||||
`orca node join --type proxmox` flow + existing TOFU `known_hosts` store; no
|
||||
new orchestration capability)
|
||||
|
||||
### Wave 1 — Trust-surface foundations (security helpers + flag declarations)
|
||||
|
||||
| Task ID | Owner | Wave | Must | Title | Files touched | Acceptance criterion |
|
||||
|---------|-------|------|------|-------|---------------|----------------------|
|
||||
| T02.1 | backend-engineer | 1 | Y | Add `security.SSHFingerprintSHA256` helper (AD-027) | `internal/security/sshkey.go` (extend) OR `internal/security/fingerprint.go` (extend) | Thin wrapper over `ssh.FingerprintSHA256(pubKey ssh.PublicKey) string` returning the canonical `SHA256:base64` string. Do NOT reuse `security.Fingerprint` (X.509 hex — different domain per RESEARCH §2.2). Unit test: known Ed25519 pub key → known `SHA256:` string. |
|
||||
| T02.2 | backend-engineer | 1 | Y | Export `security.WriteAtomic` (AD-029 enabler) | `internal/security/ca.go` | Rename `writeAtomic` → `WriteAtomic` (export) + update existing in-package callers. The `key-reset` atomic known_hosts rewrite (T02.7) needs it. Alternatively copy the ~20-LOC pattern into `proxmox` if export is undesirable — **recommend export** (RESEARCH §5 pitfall #10). `go build ./internal/security` PASS. |
|
||||
| T02.3 | backend-engineer | 1 | Y | Add `--host-key-fingerprint` flag on `orca node join` (D-044) | `internal/cli/node.go` | `nodeJoinCmd.Flags().StringVar(&joinHostKeyFP, "host-key-fingerprint", "", "SSH host key SHA256:base64 fingerprint (pre-pin; supersedes TOFU for --type proxmox)")` in the flag-registration block (node.go:344-354). Add `joinHostKeyFP string` to the var block (node.go:47-60). Validation in `RunE`: if `joinHostKeyFP != ""` and `--type != proxmox`, emit a clear error ("--host-key-fingerprint requires --type proxmox today"). Flag is generic for future SSH-joined kinds (D-044). |
|
||||
| T02.4 | backend-engineer | 1 | Y | Add `HostKeyFingerprint` field to `proxmox.Options` | `internal/proxmox/bootstrap.go` | Add `HostKeyFingerprint string` to the `Options` struct (bootstrap.go:55). Pass-through from `internal/cli/node.go` joinProxmox (node.go:158-166): `HostKeyFingerprint: joinHostKeyFP`. |
|
||||
|
||||
### Wave 2 — Trust features + bugfix (build on Wave 1)
|
||||
|
||||
| Task ID | Owner | Wave | Must | Title | Files touched | Acceptance criterion |
|
||||
|---------|-------|------|------|-------|---------------|----------------------|
|
||||
| T02.5 | backend-engineer | 2 | Y | Implement `pinnedHostKeyCallback` (REQ-058, AD-028) | `internal/proxmox/bootstrap.go` | `pinnedHostKeyCallback(expectedSHA256Base64 string) (ssh.HostKeyCallback, error)`: validate `SHA256:` prefix up front (reject raw hex with a clear error per D-045); callback receives server's `ssh.PublicKey`, computes `ssh.FingerprintSHA256(key)` (via T02.1 helper or inline), compares full strings to the operator-supplied value; returns `nil` on match, `error` on mismatch (fail closed). In `BootstrapProxmox`: if `opts.HostKeyFingerprint != ""` use `pinnedHostKeyCallback`, else fall back to the TOFU callback (T02.6). Unit test: match → callback returns nil; mismatch → returns error mentioning REQ-058; non-`SHA256:`-prefixed input → constructor returns error. |
|
||||
| T02.6 | backend-engineer | 2 | Y | **BUGFIX (v0.6 ship-defect)**: FIX the latent TOFU capture bug (RESEARCH §2.1, §5 pitfall #1, GRILL condition #1) | `internal/proxmox/bootstrap.go` | Wrap `knownhosts.New(...)` with a custom callback that: on `*knownhosts.KeyError{Want: []}` (host unknown) captures the server-presented `ssh.PublicKey`, writes a line via `knownhosts.Line([]string{knownhosts.Normalize(addr)}, key)` to `certpaths.KnownHostsPath()` using `security.WriteAtomic` (T02.2, AD-029), and returns `nil` (allow the dial to proceed). On `*knownhosts.KeyError{Want: [knownKey]}` (mismatch) returns the error (MITM detection). On `nil` (host present + match) returns `nil`. This fixes the v0.6 latent ship-defect where first-connect Proxmox join always failed (verified against `golang.org/x/crypto@v0.54.0/ssh/knownhosts/knownhosts.go:370-385`). P04 audit must record this as ship-defect closure. Unit test: first-connect captures the key + writes known_hosts; second-connect matches; mismatch-connect fails. |
|
||||
| T02.7 | backend-engineer | 2 | Y | Populate `Result.HostKeyFingerprint` (RESEARCH §2.2, §5 pitfall #2) | `internal/proxmox/bootstrap.go` | In the capture path (T02.6) and the pinned path (T02.5), set `Result.HostKeyFingerprint = ssh.FingerprintSHA256(hostKey)` (via T02.1). The field is currently declared (bootstrap.go:83-85) but always `""`. After T02.7, `orca node join --type proxmox` output includes the real fingerprint. Unit test: `Result.HostKeyFingerprint` is non-empty + `SHA256:`-prefixed after a successful bootstrap. |
|
||||
| T02.8 | backend-engineer | 2 | Y | Implement `orca node key-reset <node>` (REQ-059, D-046, AD-029) | `internal/cli/node.go`, `internal/proxmox/bootstrap.go` (new `ResetHostKey` helper OR inline in cli) | New `nodeKeyResetCmd` (`&cobra.Command{Use: "key-reset <node>", Args: cobra.ExactArgs(1), RunE: ...}`) registered via `nodeCmd.AddCommand(nodeKeyResetCmd)` (node.go:358-360). `RunE`: (1) resolve `<node>` arg via `nodeRegistry()` (node.go:37) → get node row → use `node.Name` (the host address for proxmox nodes) as the `known_hosts` match key; (2) call `proxmox.ResetHostKey(host) error` which reads `certpaths.KnownHostsPath()`, filters lines whose host field (before first whitespace, normalized via `knownhosts.Normalize`) matches, rewrites via `security.WriteAtomic` (T02.2); (3) audit-log `event=node.key_reset` with `actor`+`node`+`host` via `engine.Audit.Record`; (4) print `✓ Host key reset for <node> (next connect will re-pin via TOFU or --host-key-fingerprint)`. **Local only — do NOT revoke remote authorized_keys** (D-046). Unit test: known_hosts with 2 entries for the target host + 1 for another host → after reset, target's 2 lines removed, other host's line intact; audit row inserted. |
|
||||
| T02.9 | backend-engineer | 2 | Y | Apply the TOFU capture-fix to `doctor proxmox` probe (GRILL condition #2 — doctor parity with bootstrap) | `internal/doctor/doctor.go` | The doctor proxmox probe (doctor.go:412-415) uses the same `knownhosts.New(...)` callback pattern as bootstrap. Apply the same capture-fix wrapper (T02.6) so `doctor proxmox` on a first-connect node doesn't fail. **P02 is not complete until both bootstrap (T02.6) and doctor (T02.9) callbacks use the capture-fix wrapper — GRILL condition #2 binding parity check.** (If the doctor probe already relies on a prior `node join` having populated `known_hosts`, the fix is still correct — it makes the doctor robust to a missing entry.) |
|
||||
|
||||
### Wave 3 — End-to-end integration + verification
|
||||
|
||||
| Task ID | Owner | Wave | Must | Title | Files touched | Acceptance criterion |
|
||||
|---------|-------|------|------|-------|---------------|----------------------|
|
||||
| T02.10 | backend-engineer | 3 | Y | End-to-end trust-surface integration tests | `internal/proxmox/bootstrap_test.go` (extend), `internal/cli/node_test.go` (extend) | (1) `--host-key-fingerprint` with a correct pin → bootstrap succeeds + `Result.HostKeyFingerprint` matches the pin; (2) `--host-key-fingerprint` with a wrong pin → bootstrap fails fast with the REQ-058 mismatch error; (3) no `--host-key-fingerprint` + first connect (empty known_hosts) → TOFU captures the key + writes known_hosts + bootstrap succeeds; (4) no flag + second connect (known_hosts has the key) → matches + succeeds; (5) no flag + mismatch (known_hosts has a different key) → fails with MITM error; (6) `orca node key-reset <node>` → known_hosts entry removed + audit row inserted + next connect re-pins; (7) known_hosts pre-populated (v0.6→v0.8 migration path: existing entry from a prior join) → second-connect matches without re-capture, covering the upgrade path. |
|
||||
| T02.11 | backend-engineer | 3 | Y | `--host-key-fingerprint` non-proxmox type validation test | `internal/cli/node_test.go` (extend) | `orca node join --type linux --host-key-fingerprint SHA256:...` → clear error ("--host-key-fingerprint requires --type proxmox today"). Validates D-044 RunE check from T02.3. |
|
||||
|
||||
### Phase 2 Must-Haves (summary)
|
||||
|
||||
- T02.1, T02.2, T02.3, T02.4 (Wave 1 foundations)
|
||||
- T02.5 (`--host-key-fingerprint` pinned callback — REQ-058)
|
||||
- T02.6 (TOFU capture-fix — latent bug)
|
||||
- T02.7 (`Result.HostKeyFingerprint` populated)
|
||||
- T02.8 (`orca node key-reset` — REQ-059)
|
||||
- T02.9 (doctor proxmox TOFU fix)
|
||||
- T02.10, T02.11 (integration + validation)
|
||||
|
||||
### Phase 2 Verification
|
||||
|
||||
- `go build ./...` PASS
|
||||
- `go vet ./...` PASS
|
||||
- `go test -race ./internal/proxmox/... ./internal/cli/... ./internal/doctor/... ./internal/security/...` PASS
|
||||
- `./bin/orca node join --help` shows `--host-key-fingerprint` flag
|
||||
- `./bin/orca node key-reset --help` shows the key-reset subcommand
|
||||
- Pinned mismatch → fail closed (T02.10 case 2)
|
||||
- TOFU first-connect → captures + succeeds (T02.10 case 3)
|
||||
- `Result.HostKeyFingerprint` is non-empty after bootstrap (T02.7)
|
||||
- `key-reset` removes only the target host's known_hosts lines + audit-logs (T02.8)
|
||||
- No new direct deps
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Requirements-Hygiene Gate (REQ-060)
|
||||
|
||||
**Branch**: `phase/03-verify-reqs`
|
||||
**REQ Coverage**: REQ-060
|
||||
**Tag**: `v0.7.3`
|
||||
**Depends on**: Phase 2 (P03 is independent of P02 code, but ships after per
|
||||
ROADMAP ordering; the verify-reqs program parses the `.ciagent/` markdown
|
||||
which is stable by P03)
|
||||
**Source research**: RESEARCH_v0.8.md §3 (Makefile, .coreci.yml, parsing
|
||||
approach, AD-030) + §4 AD-030
|
||||
|
||||
### Wave 1 — Go program
|
||||
|
||||
| Task ID | Owner | Wave | Must | Title | Files touched | Acceptance criterion |
|
||||
|---------|-------|------|------|-------|---------------|----------------------|
|
||||
| T03.1 | lead-developer | 1 | Y | `cmd/verify-reqs/main.go` — Go program (~80 LOC, stdlib only, AD-030, GRILL condition #4 regex + reverse direction) | `cmd/verify-reqs/main.go` (NEW) | Parses `.ciagent/ROADMAP.md` + `.ciagent/REQUIREMENTS.md` using `regexp` (stdlib). **Forward assertion**: for every REQ-ID in REQUIREMENTS.md whose `Phase` column references a milestone that ROADMAP marks COMPLETE (substring-match `COMPLETE` within the bold span — NOT exact `\*\*COMPLETE\*\*` which misses v0.2's `**COMPLETE (merged to main via v0.3)**` header at ROADMAP.md:23), the REQUIREMENTS `Status` must be `Complete`. **Reverse assertion (GRILL condition #4)**: for every REQ-ID in REQUIREMENTS.md marked `Complete`, the corresponding milestone in ROADMAP.md must be marked COMPLETE. Regex: REQUIREMENTS row `^\|\s*(REQ-\d+)\s*\|.*?\|\s*\*\*(Complete\|Pending)\*\*\s*\|`; ROADMAP milestone-complete `^##\s*Milestone\s+v0\.\d+:.*—\s*\*\*COMPLETE[^\*]*\*\*` (substring tolerant); map milestone → REQs via the REQUIREMENTS `Phase` column (e.g. `v0.7 P1` → milestone `v0.7`). Exit 0 on consistency; exit 1 with a diff listing (REQ-ID + current status + expected status + direction) on drift. CLI: `go run ./cmd/verify-reqs .ciagent/ROADMAP.md .ciagent/REQUIREMENTS.md` (args optional; defaults to those paths). **Scope note (GRILL)**: REQ-060 catches doc-vs-doc drift only; code-vs-doc drift (e.g. the REQ-053 `cert_repo_test.go` omission — verified missing) is out of scope for this gate and handled by P04 `ciagent-audit`. |
|
||||
| T03.2 | lead-developer | 1 | Y | `cmd/verify-reqs/main_test.go` — golden-file tests | `cmd/verify-reqs/main_test.go` (NEW), `cmd/verify-reqs/testdata/` (NEW: `roadmap_clean.md`, `requirements_clean.md`, `roadmap_drift.md`, `requirements_drift.md`) | (1) Clean pair (ROADMAP v0.X COMPLETE + REQUIREMENTS REQ-XXX Complete) → exit 0, no diff; (2) Drift pair (ROADMAP v0.X COMPLETE + REQUIREMENTS REQ-XXX Pending) → exit 1 + diff lists the stale REQ; (3) Multiple drifts → all reported; (4) Missing args → uses defaults; (5) Malformed markdown → clear error (not a silent pass). |
|
||||
|
||||
### Wave 2 — Makefile + CI hook
|
||||
|
||||
| Task ID | Owner | Wave | Must | Title | Files touched | Acceptance criterion |
|
||||
|---------|-------|------|------|-------|---------------|----------------------|
|
||||
| T03.3 | lead-developer | 2 | Y | `make verify-reqs` target | `Makefile` | Add `verify-reqs` target: `go run ./cmd/verify-reqs .ciagent/ROADMAP.md .ciagent/REQUIREMENTS.md`. Add to `.PHONY`. `make verify-reqs` exits 0 on the current repo (REQUIREMENTS was corrected during v0.8 SPECIFY). |
|
||||
| T03.4 | lead-developer | 2 | Y | `.coreci.yml` validate-pipeline hook | `.coreci.yml` | Add a `verify-reqs` step to the `validate` pipeline (after `go-version`, alongside `gosec`/`govulncheck`/`gitleaks` per RESEARCH §3.2): `image: golang:1.25`, `commands: [make verify-reqs]`. Pipeline fails on drift. |
|
||||
|
||||
### Wave 3 — Synthetic drift verification
|
||||
|
||||
| Task ID | Owner | Wave | Must | Title | Files touched | Acceptance criterion |
|
||||
|---------|-------|------|------|-------|---------------|----------------------|
|
||||
| T03.5 | lead-developer | 3 | Y | Synthetic drift verification (REQ-060 acceptance) | none (verification only; temporarily flip a REQUIREMENTS row to Pending in a scratch commit, run `make verify-reqs`, assert exit 1 + diff, then revert) | (1) `make verify-reqs` on the current repo → exit 0; (2) flip one v0.7 REQ row to `Pending` in a scratch edit → `make verify-reqs` → exit 1 + diff lists that REQ-ID; (3) revert the scratch edit → exit 0. This is the REQ-060 acceptance criterion ("passes on current repo + fails on synthetic drift"). |
|
||||
|
||||
### Phase 3 Must-Haves (summary)
|
||||
|
||||
T03.1, T03.2, T03.3, T03.4, T03.5 — all must complete for the hygiene gate to
|
||||
ship.
|
||||
|
||||
### Phase 3 Verification
|
||||
|
||||
- `go build ./cmd/verify-reqs` PASS
|
||||
- `go test ./cmd/verify-reqs/...` PASS (golden-file tests)
|
||||
- `make verify-reqs` → exit 0 on the current repo
|
||||
- Synthetic drift → `make verify-reqs` exit 1 + diff (T03.5)
|
||||
- `.coreci.yml` validate pipeline includes the `verify-reqs` step
|
||||
- No new direct deps (stdlib only)
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Final Review + Ship + Audit (no new REQs)
|
||||
|
||||
**Branch**: `phase/04-final-review-ship`
|
||||
**REQ Coverage**: all (REQ-057..060)
|
||||
**Tag**: `v0.7.4` (milestone release)
|
||||
**Depends on**: Phase 1 + Phase 2 + Phase 3
|
||||
**Source**: milestone-release checklist (matches PLAN_v0.7 P05 structure)
|
||||
|
||||
### Wave 1 — Review + audit
|
||||
|
||||
| Task ID | Owner | Wave | Must | Title | Files touched | Acceptance criterion |
|
||||
|---------|-------|------|------|-------|---------------|----------------------|
|
||||
| T04.1 | lead-developer | 1 | Y | Multi-persona code review across all v0.8 phases | none (review only) | ciagent-review across P01..P03; P0 issues fixed in-phase; P1+ recorded in `.ciagent/` for post-hoc. |
|
||||
| T04.2 | lead-developer | 1 | Y | Audit: reconstruction test + branch hygiene + commit discipline | none (audit only) | ciagent-audit: git log matches `.ciagent/` files; branch hygiene clean; commit discipline enforced. |
|
||||
|
||||
### Wave 2 — Ship
|
||||
|
||||
| Task ID | Owner | Wave | Must | Title | Files touched | Acceptance criterion |
|
||||
|---------|-------|------|------|-------|---------------|----------------------|
|
||||
| T04.3 | lead-developer | 2 | Y | Merge phase/04 → milestone/v0.8-coverage-trust-hardening | none | Fast-forward merge (or rebase-then-fast-forward per config). |
|
||||
| T04.4 | lead-developer | 2 | Y | Merge milestone/v0.8 → main | none | Rebase-then-fast-forward per config. |
|
||||
| T04.5 | lead-developer | 2 | Y | Tag `v0.7.4` (milestone release) | none | `git tag v0.7.4` on the merged main HEAD. Per-phase tags `v0.7.0`..`v0.7.4` all present. |
|
||||
| T04.6 | lead-developer | 2 | Y | Create Gitea release `v0.7.4` with milestone summary | none | Release notes cover all 4 phases + REQ-057..060 + coverage deltas + trust-surface additions. |
|
||||
|
||||
### Wave 3 — Post-ship bookkeeping
|
||||
|
||||
| Task ID | Owner | Wave | Must | Title | Files touched | Acceptance criterion |
|
||||
|---------|-------|------|------|-------|---------------|----------------------|
|
||||
| T04.7 | lead-developer | 3 | Y | Update REQUIREMENTS.md — mark REQ-057..060 Complete | `.ciagent/REQUIREMENTS.md` | All 4 v0.8 REQ rows show `**Complete**` with phase + ship tag. `make verify-reqs` still passes (self-consistency). |
|
||||
| T04.8 | lead-developer | 3 | Y | Update ROADMAP.md — mark v0.8 COMPLETE | `.ciagent/ROADMAP.md` | v0.8 milestone section shows `**COMPLETE**`; all phase checkboxes `[x]`. `make verify-reqs` still passes. |
|
||||
| T04.9 | lead-developer | 3 | Y | Write + clear checkpoint | `.ciagent/` checkpoint | `{phase: 4, stage: "complete", phase_role: "final", milestone_complete: true}`; then clear checkpoint (milestone complete; next run starts a new milestone). |
|
||||
|
||||
### Phase 4 Must-Haves (summary)
|
||||
|
||||
All tasks (T04.1..T04.9) are must-haves — the final-review phase has no
|
||||
optional work.
|
||||
|
||||
### Phase 4 Verification
|
||||
|
||||
- `make build` PASS
|
||||
- `make test` PASS
|
||||
- `make lint` PASS
|
||||
- `make verify-reqs` PASS
|
||||
- `go vet ./...` PASS
|
||||
- `git log` on main shows all v0.8 phase commits
|
||||
- `git tag --list 'v0.7.*'` shows v0.7.0..v0.7.4
|
||||
- REQUIREMENTS.md shows REQ-057..060 as Complete
|
||||
- ROADMAP.md shows v0.8 as COMPLETE
|
||||
- Gitea release `v0.7.4` published with milestone summary
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: Final Review (next milestone, not part of v0.8 execution)
|
||||
|
||||
Per the v0.8 ROADMAP, there are 4 execution phases (P01..P04). P04 IS the
|
||||
final review + ship + audit phase. There is no separate P05 in v0.8 (unlike
|
||||
v0.7 which had P05). The orchestrator's next-milestone P0 begins after
|
||||
T04.9 clears the checkpoint.
|
||||
|
||||
---
|
||||
|
||||
## Risk Register (carried forward from RESEARCH_v0.8.md §5)
|
||||
|
||||
| # | Pitfall | Phase(s) affected | Mitigation |
|
||||
|---|---------|-------------------|------------|
|
||||
| 1 | TOFU capture is currently BROKEN: `knownhosts.New` returns `KeyError{Want:[]}` on first connect and does NOT auto-write; current `BootstrapProxmox` treats it as a dial failure. | P02 | T02.6 wraps the callback to capture-and-persist on `KeyError{Want:[]}` via `knownhosts.Line` + `security.WriteAtomic`. This is a v0.6 latent bug that P02 closes. |
|
||||
| 2 | `Result.HostKeyFingerprint` is declared but never populated (always `""`). D-045's rationale references "existing output" that doesn't exist. | P02 | T02.7 adds `ssh.FingerprintSHA256(hostKey)` computation in both the capture and pinned paths. 1-line addition once the host key is available. |
|
||||
| 3 | No `sessionRunner` seam in proxmox — testing the SSH command sequence without a real SSH server is impossible. | P01 | T01.1 adds a 1-interface ~10-LOC `sessionRunner` seam in Wave 1. Unlocks ~40% of proxmox coverage. Backward compatible. |
|
||||
| 4 | `internal/store/cert_repo.go` has NO test — v0.7 P01 REQ-053 was supposed to add `cert_repo_test.go` but it's missing (v0.7 leftover). | P01 | T01.7 adds `cert_repo_test.go` (Insert/Get/List/ListByNode/LatestForKind/PruneOlderThan/Delete + N=3 rotation). Directly lifts store coverage toward 70%. |
|
||||
| 5 | `internal/cli/daemon.go` starts a long-running mTLS server — testing it in cli requires a lifecycle harness; it's already covered by `internal/daemon/server_test.go`. | P01 | T01.6 excludes `daemon.go` from the cli 70% target; documents the exclusion in a test-file comment. Avoids double-testing. |
|
||||
| 6 | `cmd/orca` 50% toe-hold is low-value (15 LOC of glue; effort:coverage ratio is poor). | P01 | T01.11 keeps it at the 50% toe-hold per D-047; does NOT over-invest. A small `run() int` refactor enables a smoke test. |
|
||||
| 7 | `go: no such tool "covdata"` for zero-test packages — a Go toolchain quirk when a package has no test files; NOT a real 0% number. | P01 | T01.9, T01.10, T01.11 each add a `_test.go` file, which makes coverage computable. Don't treat the tooling error as a measurement. |
|
||||
| 8 | `transport.dispatchToPeer` has no seam — testing the remote-dispatch branch requires a new interface OR `httptest.NewTLSServer`. | P01 | T01.3 uses `httptest.NewTLSServer` (no refactor needed). T01.2 (conditional `peerDispatcher` seam) is only added if engine cannot otherwise hit 70%. |
|
||||
| 9 | `knownhosts.Line` + `knownhosts.Normalize` are the helpers for the TOFU-capture fix and `key-reset` matching. | P02 | T02.6 + T02.8 use `Normalize` to match host strings consistently (handles `host:22` vs `host`). |
|
||||
| 10 | `security.writeAtomic` is unexported (ca.go:305); `key-reset`'s atomic known_hosts rewrite needs it. | P02 | T02.2 exports `WriteAtomic` (recommended) OR copies the ~20-LOC pattern. Export is preferred — it's already used across ca.go + sshkey.go. |
|
||||
|
||||
---
|
||||
|
||||
## REQ-ID → Task mapping (traceability)
|
||||
|
||||
| REQ-ID | Phase | Tasks |
|
||||
|--------|-------|-------|
|
||||
| REQ-057 | P01 | T01.1, T01.2 (conditional), T01.3, T01.4, T01.5, T01.6, T01.7, T01.8, T01.9, T01.10, T01.11, T01.12 |
|
||||
| REQ-058 | P02 | T02.1, T02.3, T02.4, T02.5, T02.7, T02.10, T02.11 |
|
||||
| REQ-059 | P02 | T02.2, T02.8, T02.10 |
|
||||
| REQ-060 | P03 | T03.1, T03.2, T03.3, T03.4, T03.5 |
|
||||
| (latent TOFU bug) | P02 | T02.6, T02.9 (not a REQ — closes a v0.6 gap surfaced by RESEARCH §2.1) |
|
||||
| (milestone release) | P04 | T04.1..T04.9 |
|
||||
|
||||
---
|
||||
|
||||
## Task counts
|
||||
|
||||
| Phase | Tasks | Must-haves | Waves |
|
||||
|-------|-------|------------|-------|
|
||||
| P01 | 12 | 11 (T01.2 conditional) | 3 |
|
||||
| P02 | 11 | 11 | 3 |
|
||||
| P03 | 5 | 5 | 3 |
|
||||
| P04 | 9 | 9 | 3 |
|
||||
| **Total** | **37** | **36** | — |
|
||||
@@ -1,41 +0,0 @@
|
||||
# PRD v0.11 Extension: Production Hardening
|
||||
|
||||
**Status**: This file EXTENDS (does not supersede) `PRD_v0.9.md`. The
|
||||
16 load-bearing rules R-001…R-016 remain in effect; this file adds
|
||||
R-017…R-020, adopted per operator decision Q1=A (2026-08-07) after
|
||||
ingestion of 5 research documents covering ingress hardening, drift
|
||||
detection, platform-engineer positioning, strategic framing, and the
|
||||
systemd Path unit implementation.
|
||||
|
||||
## New load-bearing rules (R-017…R-020)
|
||||
|
||||
| ID | Rule |
|
||||
|---|---|
|
||||
| R-017 | Cluster ingress default is the hybrid: Traefik binds on `127.0.0.1:8443` (and `127.0.0.1:8080` for HTTP). Public `:443` / `:80` traffic is DNAT'd via nftables to Traefik. Cross-node cluster mesh stays on the cluster-internal private IP. Operators can opt out with `orca cluster config --public-binding=...`. Workloads can opt in to pure iptables with `service { ingress: native }`. In all cases, mTLS termination is unchanged: Traefik holds the certs. |
|
||||
| R-018 | Default drift detection cadence is 60s. Operators can tune per-path: critical_paths (5s default, systemd Path units enabled), standard_paths (30s default), file_watch_paths (systemd Path units, event-driven, R-001-clean). |
|
||||
| R-019 | Drift detector is a BACKSTOP. Primary failure detection is: systemd (`Type=notify`) for process state, Traefik health checks for routing state, step-ca cert notifications for cert expiry, Syncthing completion events for replication state. Drift detector exists to catch config divergence, not workload failures. |
|
||||
| R-020 | Hard gate: applier refuses new txns if pre-flight consistency check fails. Drift must be resolved before new state is committed. Auto-remediation is enabled by default for critical config paths but disabled for systemd unit files (require operator approval). Override: `--force` flag + per-namespace scoping (a drifted peer in ns-A does not block ns-B). |
|
||||
|
||||
## Relationship to R-001…R-016
|
||||
|
||||
R-017…R-020 are *extensions*, not reversals. They are compatible with:
|
||||
- R-001 (no Orca binary on servers) — systemd Path units are OS-native; nftables is OS-native; no Orca daemon introduced.
|
||||
- R-006 (mTLS by default; Traefik load-bearing) — R-017 preserves Traefik as the mTLS termination point; only the binding address changes.
|
||||
- R-007 (sockets by default) — unchanged; R-017 is about the public-ingress edge, not inter-workload sockets.
|
||||
- R-010 (transactional control plane) — R-018/R-019/R-020 refine the txn plane's drift-detection contract (C-09).
|
||||
|
||||
## New D-series (D-215…D-237)
|
||||
|
||||
D-215…D-226 (ingress hybrid, doc 1) and D-227…D-237 (drift detection, doc 5)
|
||||
are recorded in `PROJECT.md` § v0.11 Clarified Decisions. No collisions with
|
||||
existing D-series (ends at D-206).
|
||||
|
||||
## Milestone scope
|
||||
|
||||
v0.11 "Production Hardening" — 23 phases (P00…P16). Research adds scope to
|
||||
P09 (drift-event aggregation), P10 (drift detection + transactional plane),
|
||||
and P15.5 (ingress hybrid + threat model). Five net-new CLI commands
|
||||
(`orca cluster rotate-lead`, `orca upgrade`, `orca job migrate`,
|
||||
`orca logs --all-nodes`, `orca doctor mTLS`) are folded into existing
|
||||
phases per operator decision Q2=C. No new phases added (Q3=A folds ingress
|
||||
into P15.5).
|
||||
@@ -1,92 +0,0 @@
|
||||
# Orca — Comprehensive Product Requirements Document (v0.9/v0.10)
|
||||
|
||||
**Audience:** Operators, AI agents, downstream tooling authors
|
||||
|
||||
> This PRD SUPERSEDES the shipped v0.1–v0.8 architecture. The v0.9 and v0.10
|
||||
> milestones implement a re-architecture whose load-bearing rules (R-001…R-016)
|
||||
> and decisions (D-068…D-206) replace or demote several earlier documented
|
||||
> decisions. See §22 decision-trace and the Supersession Table in
|
||||
> `ARCHITECTURE.md` for the recorded reversals and their evidence basis.
|
||||
|
||||
## Status
|
||||
|
||||
| Item | Status |
|
||||
|---|---|
|
||||
| Spec lock-in | ✅ R-001…R-016 + D-001…D-206 settled |
|
||||
| v0.1–v0.8 implementation | ✅ shipped (REQ-001..060, D-001..D-047) |
|
||||
| v0.9 implementation | ⬜ Phase 0 pre-execution (this file is the spec input) |
|
||||
| v0.10 implementation | ⬜ planning (post-PRD) |
|
||||
| v1.x multi-host state | ⬜ parked (post-v1.0) |
|
||||
| v2.x full Nomad-HCL | ⬜ parked (post-v1.x) |
|
||||
|
||||
## Override justification (recorded for the grill supersession)
|
||||
|
||||
The v0.9/v0.10 re-architecture is justified on six independent grounds rather
|
||||
than preference. Each reverses a prior documented decision; the new evidence
|
||||
basis is recorded with the reversal in the Supersession Table:
|
||||
|
||||
1. **The v0.8 daemon model is operationally failing** in the target environment
|
||||
— R-001 ("no orca binary on any server") is a response to measured pain, not
|
||||
preference.
|
||||
2. **step-ca is externally mandated** (D-101) — the operator environment requires
|
||||
an external CA; AD-010's "too heavyweight" rationale is no longer operative.
|
||||
3. **Multi-tenancy is a hard product requirement** (R-002) — real multi-tenant
|
||||
use cases cannot be served by the single-namespace layout; the
|
||||
"no multi-tenancy" anti-pattern is obsolete.
|
||||
4. **WASM is a hard workload requirement** (D-088) — workloads are WASM, not
|
||||
processes; `os/exec` is insufficient; the "no container runtime" anti-pattern
|
||||
is reversed.
|
||||
5. **SSH-push is the only viable deployment target** for the operator's
|
||||
bare-Linux/Proxmox environment — installing/maintaining an orca daemon on
|
||||
every peer is operationally infeasible.
|
||||
6. **Simplicity/vision correction** — the v0.1-v0.8 daemon model was a wrong
|
||||
turn against the original CLI-first vision; the re-architecture corrects the
|
||||
vision.
|
||||
|
||||
## Canonical references
|
||||
|
||||
The full PRD text was provided by the operator and adopted wholesale. The
|
||||
load-bearing rules (R-001…R-016), the concept model (§4), the architecture
|
||||
(§5), the milestone plan (§23), and the decision trace (§22) are reproduced
|
||||
in the operator's original document. This file is the auditable pointer to
|
||||
that source; the substantive planning artifacts live in:
|
||||
|
||||
- `IDEATION_v0.9.md` — 30 ideas (REQ-061..REQ-090), three tiers
|
||||
- `GRILL_v0.9.md` — 9-axis adversarial review, 19 binding conditions, 10 phase challenges
|
||||
- `REQUIREMENTS.md` — REQ-061..REQ-090 appended
|
||||
- `ROADMAP.md` — v0.9 (13 phases) + v0.10 (19 phases) appended
|
||||
- `PERSONAS.md` — security/network/devops reactivated
|
||||
- `ARCHITECTURE.md` — v0.9 banners + Supersession Table
|
||||
|
||||
## The 16 load-bearing rules (invariants)
|
||||
|
||||
| ID | Rule |
|
||||
|---|---|
|
||||
| R-001 | No Orca Go binary runs on any server. The `orca` CLI on the operator's host is the only Orca software. Servers run Linux + systemd + apt-managed packages + config files written by the CLI. |
|
||||
| R-002 | Filesystem paths are namespaces. `ORCA_HOME` hosts many namespaces; each is a dir with `db/`, `.env`, `.env.secrets`, `jobs/`, `alloc/`, `ns.md`. `_defaults/` always exists. No `namespace` column in SQLite. |
|
||||
| R-003 | Cluster lead is always bare Linux; Proxmox can never be lead. |
|
||||
| R-004 | Workload migration Linux↔Proxmox supported; runtime can change at migration; SPIFFE identity preserved. |
|
||||
| R-005 | Storage replication enables migration; a Service's `count` replicas share one `runtime {}` block. |
|
||||
| R-006 | mTLS on by default; cluster CA = step-ca; Traefik + `LoadCredential=` are load-bearing. |
|
||||
| R-007 | Sockets by default (`/run/orca/alloc-<id>/port-<name>.sock`); `127.0.0.1` opt-in. |
|
||||
| R-008 | CLI results cached locally with per-class TTLs (`orca_cache` SQLite). |
|
||||
| R-009 | CLI host SPOF mitigated by external shared state in v1.x; v0.10 ships the abstractions + cache layer. |
|
||||
| R-010 | Control plane updates are transactional (ArgoCD-style desired-state/lead-applier). |
|
||||
| R-011 | Each namespace has `.env` (plaintext) and `.env.secrets` (AES-256-GCM, per-line nonce); master key per `ORCA_HOME` at `cluster/master.key`. |
|
||||
| R-012 | Workload kinds are `Job`, `Service`, `DaemonSet`; schema-separated by `kind:` in frontmatter. |
|
||||
| R-013 | Jobspec format is Markdown with YAML frontmatter (`.md` preferred); `.yaml` and `.hcl` accepted by parser dispatcher. |
|
||||
| R-014 | All user-facing config is Markdown with YAML frontmatter; body preserved verbatim. |
|
||||
| R-015 | Body of every `.md` config file is preserved verbatim and surfaced in `inspect`, `history`, diffs. |
|
||||
| R-016 | `.env` and `.env.secrets` are exempt from R-014 — standard dotenv format retained. |
|
||||
|
||||
## Milestone summary (§23, reordered per grill PC-01..PC-10)
|
||||
|
||||
### v0.9 — Workloads + Re-architecture Foundation (13 phases)
|
||||
P00 (deprecation sweep + migration-ordering + txn-design spike + test-infra bootstrap + persona reactivation + doc banners), P0a1 (path resolver + config demotion), P0a2 (namespace CRUD + inheritance), P0b (Markdown jobspec parser + fuzz), P0c (schemas + emitter interface), P01 (SSH-push transport + host-path volumes), P02 (service + Traefik emitter), P03 (update stanza), P04 (lifecycle hooks), P05 (constraints + CLI-side scheduler), P06 (task groups), P07a/P07b/P07c (process+podman / wasmtime [C-01 gated] / pve-vm+ct runtimes), P08 (sockets), P09 (Syncthing [C-02 gated]), P10 (lead rules + migration), P0X (ship + audit).
|
||||
|
||||
### v0.10 — Production Hardening (19 phases)
|
||||
P00 (CLI cache), P01 (metrics), P01.5 (SPIFFE spike [C-08 gated]), P02 (ACL), P03 (secrets), P04 (backup/restore), P05 (drain + daemon drain-and-stop), P06 (alloc history), P07 (recovery), P08 (integration tests), P09 (collector+aggregator), P10 (transactional plane [C-09 gated]), P11 (job lint), P12 (job verify), P13 (ns subcommands), P14a/P14b/P14c (data / daemon cutover / mixed-version tolerance), P15 (README), P15.5 (threat model [C-19 gated]), P16 (final review + ship — v1.0.0 release).
|
||||
|
||||
See `ROADMAP.md` for the full reordered plan and `GRILL_v0.9.md` for the 19
|
||||
binding conditions (C-01..C-19) and 10 phase challenges (PC-01..PC-10) that
|
||||
gate specific phases.
|
||||
@@ -36,7 +36,6 @@ Build a lightweight system to manage and execute workloads across a set of nodes
|
||||
| D-004 | Scheduling algorithm for v0.1? | **Single-node only (no scheduling)** | Multi-node scheduling is out of scope for v0.1. Tasks run on the node they're submitted to. | 0.90 |
|
||||
| D-005 | CLI output format? | **Human-readable by default, `--json` flag for machine consumption** | Serves both humans and AI agents. | 0.95 |
|
||||
| D-006 | Job/task definition format? | **HCL or YAML in `.hcl`/`.yaml` files** | Familiar to Nomad/HashiCorp users; simpler than JSON for humans. | 0.88 |
|
||||
| D-186 | Bash scripts coverage gate: count toward Go gate or exempt? | **Exempt from Go coverage gate; compensating control: bats tests (C-15) + shellcheck + shfmt in CI; every script must have >=1 happy-path and >=1 failure-path bats test** | Bash is a different language surface from Go; the 70%/50% Go coverage gate (D-042/D-047) is Go-specific. Forcing bash into the Go gate would require a coverage tool that does not exist for bash. The compensating control (bats + shellcheck + shfmt) provides equivalent discipline. | 0.82 |
|
||||
| D-007 | Authentication? | **mTLS for v0.1, token-based deferred** | mTLS is the most secure default. Tokens can be added later if needed. | 0.80 |
|
||||
| D-008 | Container runtime? | **Direct process execution (no container runtime) for v0.1** | Avoids the Docker/container dependency. Pure process management. | 0.85 |
|
||||
| D-009 | Configuration file location? | **`~/.orca/config.hcl` and `/etc/orca/orca.hcl`** | Standard XDG-style paths. | 0.90 |
|
||||
@@ -105,672 +104,3 @@ auto-resolved under full autonomy and are summarized here:
|
||||
(pull-based, ctx cancellation, ctrl-c via `signal.NotifyContext`).
|
||||
- **D-018: Bin-packing by CPU/memory with FIFO within node; JSON-over-HTTP
|
||||
orca.v1.Dispatch for cross-node** (no ConnectRPC dep).
|
||||
|
||||
## v0.3 Clarified Decisions (D-series, full autonomy)
|
||||
|
||||
v0.3 is a lean 2-execution-phase milestone completing the streaming and
|
||||
doctor work deferred from v0.2. The 6 v0.3 decisions (D-019..D-024)
|
||||
were auto-resolved under full autonomy:
|
||||
|
||||
| ID | Question | Decision | Rationale | Confidence |
|
||||
|----|----------|----------|-----------|------------|
|
||||
| D-019 | Watch refresh mechanism? | **Poll-based, 1s ticker** | Simpler than event channel; no daemon coupling for CLI; matches offline-first. | 0.90 |
|
||||
| D-020 | Watch output format (REQ-030)? | **Table by default; `--watch --json` streams one-line JSON per event** | Consistent with D-005 `--json` convention; serves humans + AI agents. | 0.92 |
|
||||
| D-021 | Doctor network check scope? | **Probe configured peer addresses via mTLS `/healthz` handshake; PASS/WARN/FAIL per peer** | Reuses existing transport client; read-only. | 0.85 |
|
||||
| D-022 | Doctor db check scope? | **`PRAGMA integrity_check` + migration version query** | Already specced in ARCHITECTURE.md §5; minimal surface. | 0.95 |
|
||||
| D-023 | iter.Seq cancellation? | **`signal.NotifyContext` on SIGINT/SIGTERM** | Per D-017 + ARCHITECTURE Flow 4. | 0.95 |
|
||||
| D-024 | `--watch` applies to job list only, or node list too? | **Both `orca job list --watch` and `orca node list --watch`** | Per ARCHITECTURE.md CLI layer + D-017. | 0.92 |
|
||||
|
||||
## v0.3 Scope Summary
|
||||
|
||||
v0.3 is a focused 2-execution-phase milestone completing the work
|
||||
deferred from v0.2 that was NOT already shipped in P08-P10. A codebase
|
||||
audit during re-init SPECIFY confirmed that REQ-014, REQ-027, REQ-028,
|
||||
REQ-029, REQ-031, REQ-037, REQ-039, REQ-040 all shipped in P08-P10
|
||||
despite stale REQUIREMENTS.md marking them Pending. The remaining work:
|
||||
|
||||
- **P01 — `iter.Seq` streaming for `--watch` flags.** Go 1.25+
|
||||
range-over-func semantics, pull-based `iter.Seq[Job]` /
|
||||
`iter.Seq[Node]`, `context.Context` cancellation,
|
||||
`signal.NotifyContext` on ctrl-c. Applies to both `orca job list
|
||||
--watch` and `orca node list --watch`. Covers REQ-022, REQ-030.
|
||||
- **P02 — `orca doctor` network + db full implementation.** Replaces
|
||||
the P01 stubs (`NetworkStub`, `DBStub`) with real checks: peer
|
||||
reachability via mTLS `/healthz` probe; SQLite `PRAGMA
|
||||
integrity_check` + migration version. Covers REQ-032 (completion).
|
||||
|
||||
The vision ("minimalist, offline-first, CLI-first orchestration
|
||||
engine") is unchanged. v0.3 is a completion milestone, not a direction
|
||||
change.
|
||||
|
||||
## v0.5 Scope Summary — Distribution
|
||||
|
||||
v0.5 is a 3-execution-phase milestone that makes Orca installable,
|
||||
distributable, and containerized. The engine functionality from
|
||||
v0.1–v0.3 is unchanged; this milestone is purely about **delivery
|
||||
surface**:
|
||||
|
||||
- **P01 — Namespace unification.** A single `ORCA_HOME` environment
|
||||
variable becomes the namespace root for *all* on-disk state (db,
|
||||
certs, init, daemon). A `--system` flag on the root command selects
|
||||
the system-level namespace root `/root/.orca`. Backward compatible:
|
||||
empty `ORCA_HOME` → `~/.orca`. Covers REQ-041, REQ-042.
|
||||
- **P02 — `install.sh` + in-place update.** A 1-liner installer pulls
|
||||
the release binary from the public Gitea release URL, installs at
|
||||
user level by default (`~/.local/bin/orca`) or system level
|
||||
(`/usr/local/bin/orca`) with `--system`. Re-running updates the
|
||||
binary in place while preserving config/db/certs in the namespace
|
||||
dir. Idempotent. Covers REQ-043, REQ-044. Also updates README
|
||||
quickstart (REQ-016 completion).
|
||||
- **P03 — Docker release.** A multi-stage `Dockerfile` builds a
|
||||
distroless image; `scripts/release.sh` and `.coreci.yml` publish the
|
||||
image to the Gitea container registry per release. Covers REQ-046.
|
||||
- **P04 — Final review + ship + audit.** Milestone release.
|
||||
|
||||
The vision ("minimalist, offline-first, CLI-first orchestration
|
||||
engine") is unchanged. v0.5 is a distribution milestone, not a
|
||||
direction change.
|
||||
|
||||
## v0.5 Clarified Decisions (D-series, full autonomy)
|
||||
|
||||
The 5 v0.5 decisions (D-025..D-029) were auto-resolved under full
|
||||
autonomy during the CLARIFY stage:
|
||||
|
||||
| ID | Question | Decision | Rationale | Confidence |
|
||||
|----|----------|----------|-----------|------------|
|
||||
| D-025 | System-level namespace path layout? | **`/root/.orca`** (mirror of user-level `~/.orca`) | Consistent shape with user-level; just a different root. Matches the user's "starts at /root" wording. Single dir keeps it simple. | 0.90 |
|
||||
| D-026 | Namespace override mechanism at runtime? | **Unify on `ORCA_HOME`** as single namespace root for all components (db, certs, init, daemon). Add `--system` flag that sets root to `/root/.orca`. | `ORCA_HOME` already exists for certs; extend to all components. Backward compatible (empty → `~/.orca`). One knob, not many. | 0.92 |
|
||||
| D-027 | Docker registry target? | **Gitea built-in container registry** (`git.cloudinit.dev/coreci/orca`) | Keeps everything in one forge; uses Gitea's native registry. Consistent with REQ-045 (public repo → public image pulls). | 0.88 |
|
||||
| D-028 | How to make releases publicly accessible (REQ-045)? | **Flip repo visibility to public** via `tea repos edit coreci/orca --private=false` during P0 ship | Simplest path to anonymous downloads; enables both install.sh pulls and docker pulls. Pre-existing `.env` leak already suppressed via gitleaks baseline + rotate-forward (commit 00127ce). | 0.85 |
|
||||
| D-029 | install.sh default version? | **Latest release** (query Gitea releases API), optional `--version vX.Y.Z` to pin | Matches typical 1-liner installer UX; users get newest by default, can pin for reproducibility. | 0.92 |
|
||||
|
||||
### v0.5 Operational prerequisite (P0 ship)
|
||||
|
||||
The Gitea repo `coreci/orca` is currently **private** (returns 404
|
||||
unauthenticated). P0 ship flips visibility to public via `tea repos
|
||||
edit coreci/orca --private=false` so that `install.sh` can pull
|
||||
release binaries unauthenticated (REQ-045). This is an operational
|
||||
step performed during the P0 ship, verified by an unauth `curl`
|
||||
against the releases API.
|
||||
|
||||
## v0.6 Scope Summary — Node Bootstrap & Proxmox
|
||||
|
||||
v0.6 is a 3-execution-phase milestone that turns `orca init` from a
|
||||
bare `mkdir` into a full single-node cluster bootstrap, and adds
|
||||
Proxmox 8 & 9 as a first-class remote node type joined over SSH with
|
||||
least-privilege role delegation. The engine functionality from
|
||||
v0.1–v0.5 is unchanged; this milestone is about **bootstrap
|
||||
ergonomics** and **heterogeneous node support**:
|
||||
|
||||
- **P01 — `orca init` full bootstrap.** A single `orca init` call now:
|
||||
(a) creates the namespace dir (`~/.orca` or `/root/.orca` with
|
||||
`--system`); (b) runs all DB migrations including the new 0006
|
||||
(`nodes.kind`, `nodes.os` — backward-compatible nullable columns);
|
||||
(c) bootstraps the internal CA via `security.CAInit` if `ca.crt` is
|
||||
absent; (d) generates the server cert via `security.GenerateCSR` +
|
||||
`ca.SignCSR` if `server.crt` is absent; (e) auto-detects the local
|
||||
OS via `/etc/os-release` `ID=` field (ubuntu/debian/alpine); (f)
|
||||
registers a `localhost` node with `kind=localhost`, `os=<detected>`,
|
||||
`addr=localhost:8443` if no localhost node exists yet. After
|
||||
`orca init`, `orca doctor` MUST pass with zero FAILs. Idempotent:
|
||||
re-running `orca init` is a no-op (or refresh) for already-provisioned
|
||||
artifacts. Covers REQ-047, REQ-048, REQ-049.
|
||||
- **P02 — Proxmox SSH join.** `orca node join --type proxmox --host
|
||||
<addr> --user root --password <pw>` (password via flag or
|
||||
`$ORCA_PROXMOX_PASSWORD`, **never persisted**) bootstraps a remote
|
||||
Proxmox 8/9 host via `golang.org/x/crypto/ssh` (new direct dep).
|
||||
Steps: (1) SSH password-auth; (2) generate or load orca's SSH
|
||||
keypair (`~/.orca/orca_ssh_key` / `.pub`, 0600/0644); (3) deploy
|
||||
pubkey to remote `~orca/.ssh/authorized_keys`; (4) create `orca`
|
||||
user (config-overridable name via `--proxmox-user`, default `orca`);
|
||||
(5) create PVE custom role `OrcaOperator` (config-overridable via
|
||||
`--proxmox-role`) with privileges `VM.Audit`,
|
||||
`Datastore.AllocateSpace`, `SDN.Use`; (6) assign role to `orca`
|
||||
user on `/`; (7) drop `/etc/sudoers.d/orca` allowlist (`pct`, `qm`,
|
||||
`pvesh`, `apt-get`, `dpkg` — no shell-escape commands); (8) record
|
||||
node row `kind=proxmox`, `os=pve`, audit log. Idempotent re-run.
|
||||
Covers REQ-050, REQ-051.
|
||||
- **P03 — `doctor os` + `doctor proxmox`.** Extends `orca doctor`
|
||||
with two new checks: `doctor os` re-runs `/etc/os-release` detection
|
||||
and verifies it matches the stored localhost node row's `os` field
|
||||
(drift = WARN); `doctor proxmox` iterates `kind=proxmox` nodes and
|
||||
SSH-probes each with `pveversion` / `pvecmd status` (3s timeout per
|
||||
peer per D-038 pattern), reporting PASS/WARN/FAIL per node. All
|
||||
bootstrap + join actions emit structured audit-log entries. Covers
|
||||
REQ-052.
|
||||
- **P04 — Final review + ship + audit.** Milestone release.
|
||||
|
||||
The vision ("minimalist, offline-first, CLI-first orchestration
|
||||
engine") is unchanged. v0.6 is a bootstrap-ergonomics + heterogeneous-
|
||||
nodes milestone, not a direction change.
|
||||
|
||||
## v0.6 Clarified Decisions (D-series, full autonomy)
|
||||
|
||||
The 8 v0.6 decisions (D-030..D-037) were resolved during the CLARIFY
|
||||
stage — D-030..D-034 confirmed by the operator in plan mode, D-035..D-037
|
||||
auto-resolved at full autonomy within the `clarify_budget`:
|
||||
|
||||
| ID | Question | Decision | Rationale | Confidence |
|
||||
|----|----------|----------|-----------|------------|
|
||||
| D-030 | SSH library for Proxmox join? | **`golang.org/x/crypto/ssh`** | Stdlib-adjacent, well-maintained, single new direct dep. Matches orca's minimal-deps ethos. Shell-out to `/usr/bin/ssh` would require openssh-client on the orca host and complicate password-auth + idempotent pubkey deploy. | 0.92 (operator-confirmed) |
|
||||
| D-031 | Proxmox join password handling? | **Flag/env only, never persisted** | `--password` flag or `$ORCA_PROXMOX_PASSWORD` is used once to deploy the orca pubkey + create the `orca` user; the password is never written to SQLite. Subsequent orca→Proxmox access uses the deployed SSH key. | 0.95 (operator-confirmed) |
|
||||
| D-032 | Localhost OS auto-detect signal? | **`/etc/os-release` `ID=` field** | Parse `ID=` from `/etc/os-release`; map `ubuntu`/`debian`/`alpine` → node `os`. Falls back to `linux` (unknown) if none match. Simplest reliable signal across the three target distros. | 0.93 (operator-confirmed) |
|
||||
| D-033 | Least-privilege Proxmox role granularity? | **Custom PVE role `OrcaOperator`** with `VM.Audit`, `Datastore.AllocateSpace`, `SDN.Use` + `/etc/sudoers.d/orca` allowlist (`pct`, `qm`, `pvesh`, `apt-get`, `dpkg`) | Config-overridable role + user names. Sufficient for "manage the host, VMs/CTs, storage, packages" without granting root shell. Built-in `PVEAuditor` is too read-only; full `Administrator` is too broad. | 0.88 (operator-confirmed) |
|
||||
| D-034 | Node kind/os schema? | **Add `nodes.kind` + `nodes.os` columns via migration 0006** | Schema-first, queryable, doctor can branch on kind. Nullable with `localhost`/`""` defaults for existing rows (backward-compatible). data-engineer owns the migration. | 0.94 (operator-confirmed) |
|
||||
| D-035 | SSH host-key verification on first Proxmox connect? | **TOFU: pin on first connect, refuse on mismatch thereafter** | First connect uses `ssh.InsecureIgnoreHostKey` to capture the host key; it is then persisted to `~/.orca/known_hosts` (or the nodes metadata) and all subsequent connects require a match. Balances first-run ergonomics against MITM risk on subsequent runs. Switching to pre-pinned keys is a future enhancement. | 0.82 (auto) |
|
||||
| D-036 | `orca init` idempotency semantics for already-provisioned artifacts? | **Skip-and-refresh, never overwrite** | If `ca.crt` exists → load it (no regen). If `server.crt` exists → keep it (no reissue). If a localhost node row exists → update `last_seen` + re-detect `os`, never insert a duplicate. If DB migrations are ahead → no-op. If `~/.orca` exists → MkdirAll is a no-op. Idempotent re-run is a hard requirement (REQ-047). | 0.95 (auto) |
|
||||
| D-037 | orca SSH keypair location + algorithm? | **`~/.orca/orca_ssh_key` (0600) + `~/.orca/orca_ssh_key.pub` (0644), Ed25519** | Ed25519 keys are smaller, faster, and more secure than RSA for SSH auth. Stored in the orca namespace dir alongside ca.crt/server.crt so `ORCA_HOME` relocation works. File modes mirror the cert file-mode discipline (REQ-033 spirit). Generated lazily on first `orca node join --type proxmox`, not at `orca init` (localhost doesn't need SSH). | 0.90 (auto) |
|
||||
|
||||
### v0.6 clarification notes
|
||||
|
||||
- **D-035 TOFU caveat**: TOFU (trust-on-first-use) is the standard SSH
|
||||
UX and matches the operator-mediated model from D-012 (CA cert
|
||||
distribution). The operator is expected to verify the host key
|
||||
fingerprint out-of-band on first connect if the network is
|
||||
untrusted. A future milestone may add `--host-key-fingerprint` pin
|
||||
flag to `orca node join --type proxmox` for pre-pinned deployments.
|
||||
- **D-036 idempotency**: re-running `orca init` on a node that already
|
||||
has a localhost row updates `last_seen` and re-detects `os` (in case
|
||||
the host OS was upgraded) but does NOT change the node `ID` or
|
||||
`joined_at`. This makes `orca init` safe to put in a systemd
|
||||
ExecStartPre or a config-management runbook.
|
||||
- **D-037 Ed25519**: `golang.org/x/crypto/ssh` + `golang.org/x/crypto/ed25519`
|
||||
are in the same module; no additional direct dep beyond D-030.
|
||||
|
||||
## v0.7 Clarified Decisions (D-series, full autonomy)
|
||||
|
||||
The 5 v0.7 decisions (D-038..D-042) were auto-resolved at full autonomy
|
||||
within the `clarify_budget` (10):
|
||||
|
||||
| ID | Question | Decision | Rationale | Confidence |
|
||||
|----|----------|----------|-----------|------------|
|
||||
| D-038 | Config file format — HCL or YAML? | **HCL** | D-009 already specced `config.hcl`. HCL is already a direct dep (hashicorp/hcl/v2 for jobspec). Adding YAML would introduce a second parser dep — violates minimal-deps. Use the existing `hclparse` pkg from jobspec. | 0.93 |
|
||||
| D-039 | Config precedence order (flag vs env vs file vs default)? | **flag > env > file > default** | Standard layered config: the most explicit (flag) wins, then the runtime (env), then the persisted (file), then the built-in default. Matches cobra/viper convention without the viper dep. | 0.92 |
|
||||
| D-040 | pprof security — bind to localhost only, or operator-chosen addr? | **Operator-chosen `--pprof <addr>` (default disabled)** | Default disabled keeps the minimalist posture. Operator picks the addr — localhost for dev, unix socket for prod. Separate mux so it never touches the mTLS daemon listener. No auth (pprof is operator-only, addr is the gate). | 0.85 |
|
||||
| D-041 | cert command registration — where in root command order? | **After `cert` is unreachable today, append after `node` in rootCmd.AddCommand order** | Alphabetical-ish with the existing cluster (audit, daemon, doctor, init, job, node, cert, status, version). No behavior change to existing commands. | 0.88 |
|
||||
| D-042 | Coverage target — 50% floor or higher? | **50% floor per package, 70% target for new packages** | 50% is achievable for the concurrent packages (engine, transport) without heroic mock effort; 70% is the floor for new code in P02/P04. Avoids a "raise coverage everywhere" rathole. | 0.85 |
|
||||
|
||||
## v0.7 Scope Summary — Hardening & Completion
|
||||
|
||||
v0.7 is a 4-execution-phase **NFR milestone** that closes out gaps
|
||||
surfaced by the v0.7 IDEATE stage: an unreachable command tree, a
|
||||
missing config file layer, low test coverage in core packages, and the
|
||||
long-deferred pprof endpoint. The engine functionality from v0.1–v0.6
|
||||
is unchanged; this milestone is purely about **correctness, coverage,
|
||||
and operability**:
|
||||
|
||||
- **P01 — Register `orca cert` command tree + cert_repo tests.** The
|
||||
`internal/cli/cert.go` command (`cert ca-init`, `cert gen`, `cert
|
||||
show`, `cert renew`, `cert fingerprint`) is fully implemented but
|
||||
never wired into `rootCmd`. This phase adds the missing
|
||||
`rootCmd.AddCommand(newCertCmd(...))` and adds the missing
|
||||
`internal/store/cert_repo_test.go`. Covers REQ-053.
|
||||
- **P02 — HCL config file parsing (`config.hcl`).** D-009 specified
|
||||
`~/.orca/config.hcl` and `/etc/orca/orca.hcl` as config locations,
|
||||
but no HCL config-file parser exists — the CLI relies entirely on
|
||||
flags and env vars. This phase adds a minimal `internal/config`
|
||||
package that loads `config.hcl` (keys: `db_path`, `listen_addr`,
|
||||
`ca_path`, `server_cert_path`, `server_key_path`, `node_capacity`),
|
||||
merges with env/flag overrides (flag > env > file > default), and
|
||||
surfaces it via `--config` flag on the root command. Covers
|
||||
REQ-054.
|
||||
- **P03 — Test coverage uplift.** Adds tests for the lowest-coverage
|
||||
packages: `internal/engine` (executor, dispatcher, peer — currently
|
||||
8.3%), `internal/transport` (mtls, dispatch, handshake_log —
|
||||
currently 26.3%), `internal/proxmox` (bootstrap SSH path —
|
||||
currently 5.1%), and `internal/audit` (no tests). Target: every
|
||||
package ≥ 50% coverage. Covers REQ-055.
|
||||
- **P04 — `--pprof` opt-in on `orca daemon`.** Adds the long-deferred
|
||||
I-308 pprof endpoint behind an opt-in `--pprof <addr>` flag (default
|
||||
disabled). `net/http/pprof` mounted on a separate mux so it never
|
||||
touches the mTLS daemon listener. Covers REQ-056.
|
||||
- **P05 — Final review + ship + audit.** Milestone release.
|
||||
|
||||
The vision ("minimalist, offline-first, CLI-first orchestration
|
||||
engine") is unchanged. v0.7 is a hardening milestone, not a direction
|
||||
change. Milestone type: NFR (all phases are fix/test/chore); the final
|
||||
phase's progressive patch IS the deliverable per `run.md` versioning
|
||||
logic. Tags run on the v0.6.x patch line: `v0.6.0` (P0) … `v0.6.5` (P05
|
||||
= milestone release).
|
||||
|
||||
## v0.8 Scope Summary — Coverage & Trust Hardening
|
||||
|
||||
v0.8 is a 3-execution-phase **NFR milestone** that continues the
|
||||
hardening theme opened by v0.7. v0.7 P03 (REQ-055) lifted four
|
||||
packages to ≥ 50%, but a coverage re-baseline after v0.7 ship shows
|
||||
the floor was insufficient: `internal/engine` regressed to 8.3%,
|
||||
`internal/proxmox` to 5.1%, and four more packages sit between 26% and
|
||||
48%. Three packages (`internal/audit`, `internal/certpaths`,
|
||||
`cmd/orca`) still have **no test files at all**. v0.8 also closes the
|
||||
two "future enhancement" hooks explicitly deferred in v0.6 — SSH
|
||||
host-key pre-pinning (D-035 caveat) and `orca node key-reset`
|
||||
(RESEARCH_v0.6 §80) — and adds a requirements-hygiene gate so the
|
||||
stale-REQ-status drift seen in REQUIREMENTS.md after v0.7 ship cannot
|
||||
recur:
|
||||
|
||||
- **P01 — Coverage uplift round 2.** Raise six under-50% packages to
|
||||
≥ 70% and add first tests for the three zero-test packages. Covers
|
||||
REQ-057.
|
||||
- **P02 — SSH trust hardening.** `--host-key-fingerprint` pre-pin flag
|
||||
on `orca node join --type proxmox` + `orca node key-reset <node>`
|
||||
command. Covers REQ-058, REQ-059.
|
||||
- **P03 — Requirements-hygiene gate.** `make verify-reqs` target +
|
||||
verify-stage assertion that ROADMAP `Complete` ↔ REQUIREMENTS
|
||||
`Complete`. Covers REQ-060.
|
||||
- **P04 — Final review + ship + audit.** Milestone release.
|
||||
|
||||
The vision is unchanged. v0.8 is a hardening milestone, not a
|
||||
direction change. Milestone type: NFR (all phases are test/feat-chore
|
||||
on the trust surface — see CLARIFY D-043 for the `feat` vs `chore`
|
||||
classification of P02); the final phase's progressive patch IS the
|
||||
deliverable per `run.md` versioning logic. Tags run on the **v0.7.x**
|
||||
patch line: `v0.7.0` (P0) … `v0.7.4` (P04 = milestone release).
|
||||
|
||||
## v0.8 Clarified Decisions (D-series, full autonomy)
|
||||
|
||||
The 5 v0.8 decisions (D-043..D-047) were auto-resolved at full autonomy
|
||||
within the `clarify_budget` (10):
|
||||
|
||||
| ID | Question | Decision | Rationale | Confidence |
|
||||
|----|----------|----------|-----------|------------|
|
||||
| D-043 | Is P02 (SSH trust hardening) a `feat` phase or a `chore` phase? It adds a new flag + a new subcommand. | **`chore` (trust-surface hardening), not `feat`** | Both `--host-key-fingerprint` and `orca node key-reset` refine the *existing* `orca node join --type proxmox` flow and the existing TOFU `known_hosts` store (D-035). No new orchestration capability, no new node kind, no new API. They close a security gap explicitly deferred in v0.6, not open new surface area. Per `run.md` versioning logic this keeps v0.8 NFR (all phases fix/test/chore/perf/refactor). | 0.84 |
|
||||
| D-044 | Where does `--host-key-fingerprint` live — on `orca node join` or only on `--type proxmox`? | **On `orca node join` (root of the join subcommand), validated when `--type proxmox`** | The flag is generic (any future SSH-joined node kind will use it); gating it to `--type proxmox` only would require re-adding it later. Validation (`flag requires --type proxmox today`) happens in `RunE`, not in the flag declaration, so the flag is declared once on `node join` and the type check emits a clear error for non-proxmox types until other SSH-joined kinds exist. | 0.86 |
|
||||
| D-045 | `--host-key-fingerprint` format — raw hex, `sha256:`-prefixed, or OpenSSH `SHA256:base64`? | **OpenSSH `SHA256:base64` (the format `ssh-keyscan -E sha256 -D -` emits and operators expect)** | Matches the fingerprint format operators already see from `ssh-keyscan` and `orca node join`'s own `Result.HostKeyFingerprint` output. Accept only `SHA256:`-prefixed base64; reject raw hex with a clear error. Internally decode base64 → compare against `ssh.PublicKey` Marshal + sha256. | 0.88 |
|
||||
| D-046 | Does `orca node key-reset <node>` also revoke the orca pubkey on the remote host, or only clear the local `known_hosts` entry? | **Local `known_hosts` entry only** | Revoking the remote authorized_keys entry would orphan a working node (next dispatch would fail auth). `key-reset` is the local "forget this host's key" operation (mirrors `ssh-keygen -R host`); re-establishing trust is a separate `orca node join` re-run. Audit-log the reset with `actor`, `node`, `event=node.key_reset`. | 0.90 |
|
||||
| D-047 | Coverage target for P01 — 70% floor or higher? | **70% floor for the 6 under-50% packages; 50% floor for the 3 zero-test packages (`internal/audit`, `internal/certpaths`, `cmd/orca`) as a first-toe-hold** | 70% across the board for the already-tested packages matches D-042's "70% target for new packages" and is achievable without heroic mock effort. For the zero-test packages, going 0→50% is the realistic single-phase step (0→70% risks a coverage rathole on `cmd/orca` which is glue code); a future milestone can lift them to 70%. | 0.82 |
|
||||
|
||||
---
|
||||
|
||||
# v0.9/v0.10 — Re-architecture Scope Summary (Supersedes v0.1–v0.8 architecture)
|
||||
|
||||
v0.9 is the first DIRECTION-CHANGE milestone in the project's history.
|
||||
It supersedes the shipped v0.1–v0.8 architecture per the adopted PRD
|
||||
(`.ciagent/PRD_v0.9.md`). The re-architecture deprecates the daemon/
|
||||
transport/internal-CA/HCL/single-namespace stack and builds a CLI-only/
|
||||
SSH-push/step-ca/Markdown-frontmatter/multi-namespace stack plus 8
|
||||
net-new subsystems.
|
||||
|
||||
## Override Justification (Re-architecture Justification axis)
|
||||
|
||||
The ci-griller returned REPLAN (0.70) on the Re-architecture Justification
|
||||
axis, noting the PRD reverses 6 documented decisions without new evidence
|
||||
and that the incremental-additive path was not evaluated. The user reviewed
|
||||
the fork and overrode the *direction* with a six-part evidence basis. The
|
||||
override is recorded verbatim below; each part addresses a reversal that
|
||||
the grill flagged as unjustified.
|
||||
|
||||
1. **The v0.8 daemon model is operationally failing** in the target
|
||||
environment — R-001 ("no orca binary on any server") is a response to
|
||||
measured pain, not preference.
|
||||
2. **step-ca is externally mandated** (D-101) — the operator environment
|
||||
requires an external CA; AD-010's "too heavyweight" rationale is no
|
||||
longer operative.
|
||||
3. **Multi-tenancy is a hard product requirement** (R-002) — real
|
||||
multi-tenant use cases cannot be served by the single-namespace layout;
|
||||
the "no multi-tenancy" anti-pattern is obsolete.
|
||||
4. **WASM is a hard workload requirement** (D-088) — workloads are WASM, not
|
||||
processes; `os/exec` is insufficient; the "no container runtime"
|
||||
anti-pattern is reversed.
|
||||
5. **SSH-push is the only viable deployment target** for the operator's
|
||||
bare-Linux/Proxmox environment — installing/maintaining an orca daemon
|
||||
on every peer is operationally infeasible.
|
||||
6. **Simplicity/vision correction** — the v0.1-v0.8 daemon model was a
|
||||
wrong turn against the original CLI-first vision; the re-architecture
|
||||
corrects the vision.
|
||||
|
||||
## Supersession Table (AD-series reversals, recorded per grill PC-09)
|
||||
|
||||
| Old decision | Was | Superseded by | Evidence basis |
|
||||
|---|---|---|---|
|
||||
| AD-010 (ARCHITECTURE.md:463) | step-ca/cfssl/vault-pki "too heavyweight" | **D-101** (step-ca) | Override ground 2 (external mandate) |
|
||||
| SPIFFE rejection (PROJECT.md:94) | internal CA chosen over SPIFFE | **D-068** (SPIFFE SVIDs) | Override ground 3 (multi-tenancy requires per-workload identity) |
|
||||
| No-container-runtime (ARCHITECTURE.md:477) | explicit anti-pattern | **D-088** (5 runtimes; wasmtime primary) | Override ground 4 (WASM is the workload profile) |
|
||||
| No-multi-tenancy (ARCHITECTURE.md:478) | explicit anti-pattern | **D-158 / R-002** (many namespaces under ORCA_HOME) | Override ground 3 (hard multi-tenant product req) |
|
||||
| AD-007 (HCL canonical) | HCL for jobspec | **R-013 / R-014** (Markdown canonical; HCL legacy) | PRD §8 (Markdown + body preservation is the operator-facing format) |
|
||||
| Daemon-on-every-node | `orca daemon` on all peers | **R-001** (no orca binary on any server) | Override grounds 1 + 5 (daemon failing; SSH-push only viable target) |
|
||||
|
||||
The 19 binding conditions (C-01..C-19) and 10 phase challenges
|
||||
(PC-01..PC-10) from `GRILL_v0.9.md` are adopted as execution gates.
|
||||
The 30 net-new requirements (REQ-061..REQ-090) from `IDEATION_v0.9.md`
|
||||
are recorded in `REQUIREMENTS.md`. The reordered phase plan is in
|
||||
`ROADMAP.md`.
|
||||
|
||||
## v0.9 Clarified Decisions (D-series, full autonomy — Phase 0 pre-execution)
|
||||
|
||||
| ID | Question | Decision | Rationale | Confidence |
|
||||
|----|----------|----------|-----------|------------|
|
||||
| D-101 | Cluster CA: internal Go CA (AD-010) or step-ca (external)? | **step-ca (apt-installed)** | Externally mandated per override ground 2; AD-010's "too heavyweight" rationale reversed. CLI wraps `step` CLI via SSH (no Go step-ca client library — keep zero-new-dep posture if possible, or add `github.com/smallstep/cli` as a dep). **Gated by C-07** (CA migration spec). | 0.74 |
|
||||
| D-068 | Workload identity: internal X.509 CA or SPIFFE SVIDs? | **SPIFFE SVIDs minted at submit time via step-ca** | Multi-tenancy (override ground 3) requires per-workload identity model; SPIFFE is the standard. SPIFFE ID `spiffe://orca/ns/<ns>/job/<name>/alloc/<id>` as SAN. **Gated by C-08** (mint spike in v0.10-P01.5; fallback to mTLS identity if spike fails). | 0.72 |
|
||||
| D-088 | Runtime: direct os/exec only (D-008) or multi-runtime? | **5 runtimes: wasm (wasmtime primary), podman, process, pve-vm, pve-ct** | WASM is the primary workload (override ground 4). `processRuntime` wraps existing `executor.go`; others are net-new. Split P07a/b/c per grill PC-10. **P07b gated by C-01** (wasmtime/CGO eval). | 0.82 |
|
||||
| D-158 | Namespace model: single flat root or multi-namespace? | **Multi-namespace under ORCA_HOME (R-002)** | Hard multi-tenant product requirement (override ground 3). `_defaults/` implicit root; `cluster/` for cluster-wide; per-namespace `db/`, `.env`, `.env.secrets`, `jobs/`, `alloc/`, `ns.md`. No namespace column in SQLite. | 0.84 |
|
||||
| D-179 | Jobspec format: HCL canonical (AD-007) or Markdown? | **Markdown with YAML frontmatter canonical (R-013); HCL legacy** | PRD §8 — Markdown + body preservation is the operator-facing format. HCL adapter (REQ-064) preserves `orca job run old-spec.hcl` during migration. | 0.85 |
|
||||
| D-185 | Re-architecture justification: incremental additive or full re-architecture? | **Full re-architecture (overridden by user)** | Six-part evidence basis above; the grill's REPLAN mechanics (PC-01..PC-10, C-01..C-19) adopted as gates. The incremental-additive path was evaluated and rejected on grounds 1 + 5 (daemon failing; SSH-push only viable). | 0.88 |
|
||||
| D-187 | wasmtime Go binding (bytecodealliance/wasmtime-go) is CGO-based — does adopting it revoke D-002 (modernc/sqlite CGO-free cross-compile story)? | **Use the wasmtime CLI (apt-installed on peer) via SSH exec; do NOT import wasmtime-go.** | The Go binding links libwasmtime via cgo and would revoke D-002's CGO-free cross-compile story. The CLI-via-SSH approach (same pattern as podman/qm/pct) avoids CGO entirely. `internal/runtime/wasm.go` imports only stdlib + sshpush. `CGO_ENABLED=0 go build ./...` succeeds. C-01 grill gate SATISFIED; D-002 NOT revoked. Full evaluation in `internal/runtime/C01_WASMTIME_CGO_EVAL.md`. | 0.90 |
|
||||
|
||||
---
|
||||
|
||||
# v0.10 Docs & Install Milestone — Scope Summary
|
||||
|
||||
v0.10 is a focused milestone that closes the documentation gap left by
|
||||
the v0.9 re-architecture and fixes the release/install pipeline bug that
|
||||
caused `install.sh` to resolve to v0.4.5 instead of the latest release.
|
||||
The v0.9 re-architecture shipped a complete CLI surface (markdown
|
||||
jobspec, `orca ns`, `orca node capacity`, CLI-side scheduler, emitters,
|
||||
Traefik ingress) but no operator-facing reference documentation. This
|
||||
milestone ships that documentation plus a worked full-stack example
|
||||
with ingress configured, and hardens the release pipeline so every
|
||||
Gitea release carries a Linux binary asset.
|
||||
|
||||
## Root cause of the v0.4.5 install
|
||||
|
||||
The v0.8.x releases (v0.8.0 through v0.8.15) shipped with **zero binary
|
||||
assets attached** to their Gitea releases. `scripts/install.sh` resolves
|
||||
"latest" by hitting `/releases/latest` (returns v0.8.15), then looks for
|
||||
`orca-v0.8.15-linux-amd64.tar.gz` in that release's assets. Since the
|
||||
asset is missing, install.sh errors out — there is no fallback walk to
|
||||
older releases that DO carry a binary. The user's v0.4.5 install came
|
||||
from an earlier run or a pinned `--version`. The fix is forward: harden
|
||||
`scripts/release.sh` to cross-build the amd64 tarball and verify the
|
||||
asset attached post-create; harden `scripts/install.sh` to walk
|
||||
backward through releases if the latest lacks the asset.
|
||||
|
||||
## v0.10 Phases
|
||||
|
||||
- **Phase 0 (pre-execution)**: specify → clarify → research → ideate → plan → grill. Tag `v0.9.0`.
|
||||
- **Phase P1 — release/install fix** (REQ-097, REQ-098): cross-build amd64 tarball in release.sh, post-create asset verification, install.sh fallback walk. Tag `v0.9.1`.
|
||||
- **Phase P2 — CLI + jobspec + ingress docs** (REQ-091, REQ-092, REQ-093): `docs/cli.md`, `docs/jobspec.md`, `docs/ingress.md`. Tag `v0.9.2`.
|
||||
- **Phase P3 — full-stack examples** (REQ-094): `examples/full-stack/` with 5 valid jobspecs + rendered artifacts + walkthrough README. Tag `v0.9.3`.
|
||||
- **Phase P4 — README + namespace.md refresh** (REQ-095, REQ-096): README subcommand table + install example + docs/examples sections; `docs/namespace.md` v0.9 layout. Tag `v0.9.4`.
|
||||
- **Phase P5 — final review + ship + audit** (milestone release). Tag `v0.9.5` = v0.10.0 milestone release.
|
||||
|
||||
**Milestone type**: feature (P1 ships `fix` phases; P2/P3/P4 ship `docs`
|
||||
phases; at least one non-docs phase makes this a feature milestone per
|
||||
the versioning logic). Tags run on the v0.9.x patch line. The milestone
|
||||
branch label is `milestone/v0.10-docs-cli-examples`.
|
||||
|
||||
The vision ("minimalist, offline-first, CLI-first orchestration
|
||||
engine") is unchanged. v0.10 is a documentation + install-hardening
|
||||
milestone, not a direction change. It builds on the v0.9
|
||||
re-architecture foundation without modifying any Go orchestration code.
|
||||
|
||||
## v0.10 Clarified Decisions (D-series, full autonomy — Phase 0 pre-execution)
|
||||
|
||||
| ID | Question | Decision | Rationale | Confidence |
|
||||
|----|----------|----------|-----------|------------|
|
||||
| D-188 | Should the CLI docs be a single `docs/cli.md` reference or a per-command `docs/cli/` subdirectory? | **Single `docs/cli.md` reference** | Mirrors the existing flat `docs/` pattern (install.md, docker.md, namespace.md, security-scanning.md). One file is more discoverable for a CLI tool and avoids navigation overhead. A per-command subdirectory diverges from the established layout. | 0.92 |
|
||||
| D-189 | Should the examples live in `examples/full-stack/` or in `testdata/`? | **`examples/full-stack/` as a new top-level directory** | `testdata/` holds legacy HCL fixtures (`hello.hcl`, `fail.hcl`) used by Go tests; mixing operator-facing examples with test fixtures conflates audiences. A new `examples/` directory is the conventional location for worked examples and is what an operator expects to find. | 0.93 |
|
||||
| D-190 | How deep should the ingress/Traefik documentation go? | **Dedicated `docs/ingress.md` plus a worked example in `examples/full-stack/`** | Ingress is the user's explicit ask ("full stack with ingress configured") and the Traefik/service-block model (R-007 socket vs TCP, atomic reload, drain, TLS) is non-trivial. A dedicated doc is the clearest answer; a section buried in `docs/cli.md` would be less discoverable. | 0.90 |
|
||||
| D-191 | Should the docs frame the v0.9 canonical path or document both v0.8 and v0.9 equally? | **Document the v0.9 canonical path; flag deprecated surface with callout boxes** | The v0.8 daemon/mTLS/HCL path is deprecated and scheduled for removal in v0.10-P14. Documenting it as primary misleads new operators; documenting both equally doubles the surface and risks documenting soon-removed code. Callout boxes with "deprecated in v0.9, removed in v0.10" point operators to the canonical path. | 0.91 |
|
||||
| D-192 | Should the existing v0.8.15 release be backfilled with a binary asset, or only fix the pipeline forward? | **Fix forward only; no backfill** | Backfilling a past release is an ops task, not a docs milestone deliverable. The next tagged phase (this milestone's P1 ship at v0.9.1) will be the first correctly-asseted release; install.sh's new fallback walk handles the gap until then. | 0.88 |
|
||||
| D-193 | Should `release.sh` build only `linux-amd64` or also `linux-arm64`? | **Cross-build `linux-amd64` explicitly (host-arch-independent); arm64 deferred to a follow-up** | The install.sh user base is amd64 today (the `.coreci.yml` release step hardcodes `--asset orca-${VERSION}-linux-amd64.tar.gz`). Building amd64 regardless of host arch (via `GOOS=linux GOARCH=amd64 go build`) guarantees the asset the install script expects. arm64 support is a separate enhancement. | 0.85 |
|
||||
| D-194 | Should `install.sh` add a `--check` dry-run mode? | **Yes, lightweight** | A dry-run mode (`--check`) that prints the version + asset URL + install path without writing is cheap to add and useful for debugging the "which release will I get?" question that the v0.4.5 incident surfaced. | 0.80 |
|
||||
|
||||
## v0.11 Clarified Decisions (D-series, full autonomy — Phase 0 pre-execution)
|
||||
|
||||
The following 23 decisions (D-215…D-237) extend the locked D-series
|
||||
(ends at D-206). They derive from 5 research documents ingested
|
||||
2026-08-07 covering ingress hardening, drift detection, platform-engineer
|
||||
positioning, strategic framing, and the systemd Path unit implementation.
|
||||
Operator decisions Q1=A, Q2=C, Q3=A, Q4=A, Q5=A are adopted.
|
||||
|
||||
### Ingress hybrid (D-215…D-226, from research doc 1)
|
||||
|
||||
| ID | Question | Decision | Rationale | Confidence |
|
||||
|----|----------|----------|-----------|------------|
|
||||
| D-215 | Public-binding default? | **Hybrid: nft DNAT → Traefik on `127.0.0.1:8443`** | Defense-in-depth (kernel + app layer); mature pattern (kube-proxy, Linkerd2-proxy, F5/HAProxy+nginx). Smaller Traefik attack surface. R-017. | 0.93 |
|
||||
| D-216 | Opt-out? | **`orca cluster config --public-binding=traefik-on-public-ip` for the simple case** | Operators who want simplicity get it with a one-line config change. | 0.94 |
|
||||
| D-217 | nftables emitter? | **Yes; renders `/etc/nftables.d/orca.nft`; idempotent `nft -f` apply** | Same emitter pattern as Traefik/systemd emitters (R-001-clean). | 0.93 |
|
||||
| D-218 | nftables tool vs iptables? | **`nft` (modern) over legacy `iptables`** | Atomic rule-set swap; modern kernel API. | 0.96 |
|
||||
| D-219 | Cross-node cluster mesh? | **Stays bound on private IP `192.168.x.x:8443`; unchanged** | Avoids adding iptables rules for cross-node mesh; keeps mesh logic unchanged. | 0.94 |
|
||||
| D-220 | Traefik `address` in static config? | **`127.0.0.1:8443` in default, `:443` in opt-out** | Single line change; certs/mTLS/dynamic config unchanged. | 0.97 |
|
||||
| D-221 | `orca doctor nft`? | **Yes; checks table, expected rules, file hash; drift detection via hash comparison** | Parity with `orca doctor traefik`; integrates with R-018 critical_paths. | 0.95 |
|
||||
| D-222 | Rate-limit meter? | **`ora_rl` set as part of the default rule set; configurable via `orca nft rate limit set`** | Kernel-level line-rate rate limiting; defense against SYN floods. | 0.91 |
|
||||
| D-223 | GeoIP blocking? | **Operator-opt-in via `orca nft country block add`**; cli + ipset extension | Not a default; operators opt in. | 0.88 |
|
||||
| D-224 | `nftables` not `iptables` in `.coreci.yml` pipelines? | **Yes; integration tests use `nft` exclusively** | Matches D-218. | 0.94 |
|
||||
| D-225 | Per-workload `ingress: native` coexists with hybrid default? | **Yes; `service { ingress: native }` opts into pure iptables + stunnel sidecars** | Workload-level opt-in; doesn't affect cluster default. | 0.93 |
|
||||
| D-226 | `nftables` rule hash baseline? | **`cluster/state/baseline.nft.hash` per peer; drift detection per §17** | Integrates with R-018 drift detection. | 0.90 |
|
||||
|
||||
### Drift detection (D-227…D-237, from research doc 5)
|
||||
|
||||
| ID | Question | Decision | Rationale | Confidence |
|
||||
|----|----------|----------|-----------|------------|
|
||||
| D-227 | Drift detection architecture? | **systemd Path units for critical paths + 60s polling backstop + auto-remediation** | R-001-clean (systemd is OS, not Orca); ~10s event-driven latency on critical paths. R-018/R-019. | 0.94 |
|
||||
| D-228 | Path unit event payload? | **Oneshot service; receives path via `%f`; computes sha256; writes event JSON to `/etc/orca/state/drift-events/`** | Stateless, self-contained, idempotent. | 0.93 |
|
||||
| D-229 | Lead-side pickup? | **Aggregator timer reads each peer's drift-events/, validates against applied txn hashes, triggers remediation** | Reuses existing 10s aggregator cadence (C-11); single SSH pull per tick. | 0.94 |
|
||||
| D-230 | Critical path polling cadence? | **5s backstop; systemd Path unit provides ~10s event-driven latency** | Closes the gap to K8s-comparable drift detection on critical paths. | 0.92 |
|
||||
| D-231 | Auto-remediation policy? | **Per-path config; critical paths default to auto; systemd units default to require-approval** | Config files are safe to re-push; service units may need careful ordering (don't restart serving workloads). | 0.93 |
|
||||
| D-232 | Remediation rate limit? | **5-minute cooldown per path; applies only on SUCCESSFUL remediation; transient failures retry on next aggregator tick** | Prevents loops from buggy external actors; avoids a 30s network blip blocking re-remediation for 5 min (refined per CLARIFY C4). | 0.92 |
|
||||
| D-233 | NFS path detection? | **`orca node setup` detects NFS mounts; falls back to polling for affected paths** | systemd Path units use inotify which doesn't work across NFS. | 0.90 |
|
||||
| D-234 | Secrets path exclusion? | **`/etc/orca/credentials/*` excluded from drift detection** | Re-remediating secrets might clobber intentional out-of-band rotation. | 0.95 |
|
||||
| D-235 | EnvironmentFile drift? | **Triggers `orca job restart <name>` instead of file-level remediation** | Workload already running won't pick up env changes without a restart. | 0.89 |
|
||||
| D-236 | `orca drift watch` semantics? | **`iter.Seq2[Event, error]` per D-017; `signal.NotifyContext` per D-023; default 2s poll** | Consistent with existing `--watch` pattern (D-017/D-023). | 0.95 |
|
||||
| D-237 | Aggregator timer changes? | **Existing 10s cadence; extended to also pull drift-events/ and remediate** | Reuses C-11 aggregator; no new timer. | 0.95 |
|
||||
|
||||
### P01.5 — SPIFFE SVID minting spike (gate C-08, D-068)
|
||||
|
||||
**C-08 SPIFFE mint spike: PASS.** The `step` CLI (smallstep step-ca)
|
||||
accepts a `spiffe://` URI in `--san` and emits a cert whose URI SAN
|
||||
(x509 subjectAltName URI entry) carries the SPIFFE URI. The fallback to
|
||||
mTLS identity (per D-068 / C-08) is NOT needed; D-068 stands.
|
||||
|
||||
- **SPIFFE URI format (locked):**
|
||||
`spiffe://orca.local/ns/<namespace>/sa/<service-account>/<alloc-id>`
|
||||
— trust domain `orca.local`; `ns/<ns>` scopes the workload to an
|
||||
Orca namespace (R-002); `sa/<sa>` is the service-account; `<alloc-id>`
|
||||
makes the SVID unique per allocation.
|
||||
- **step CLI command (locked):**
|
||||
`step ca certificate <spiffe-id> <cert> <key> --san <spiffe-id> --not-after 24h --provisioner orca-admin --password-file /dev/stdin --force`
|
||||
- **Cert parsing (locked):** `pem.Decode` → `x509.ParseCertificate` →
|
||||
iterate `cert.URIs` and match the expected SPIFFE URI (parsed as
|
||||
`*url.URL`, compared by canonical string). Missing URI SAN →
|
||||
`ErrSpiffeURIMissing` (cert rejected before reaching the workload).
|
||||
- **Implementation:** `internal/identity/spiffe.go` — `SpiffeURI`,
|
||||
`MintSVID`, `VerifySVID`, `SpiffeIDFromCert`, `SubjectFromSpiffe`.
|
||||
- **Tests:** `internal/identity/spiffe_test.go` — mock transport
|
||||
(`execer`) returns a self-signed cert minted in-process via
|
||||
`crypto/x509.CreateCertificate` with `URIs: []*url.URL{spiffeURI}`,
|
||||
exercising the exact production parsing path. 15 tests, all pass.
|
||||
- **Spike result record:** `internal/identity/SPIFFE_SPIKE_RESULT.md`.
|
||||
|
||||
## v0.12 Scope Summary — Security Hardening (Zero-Trust Identity)
|
||||
|
||||
v0.12 is a 27-execution-phase feature milestone dedicated to
|
||||
comprehensive security hardening across the entire attack surface,
|
||||
**including the operating system itself**. The threat-model review
|
||||
(v0.11 closeout + Phase 0 RESEARCH) surfaced 25 distinct findings
|
||||
(F1..F25) spanning injection, traversal, ACL, audit, crypto, OS
|
||||
scripts, emitters, sudoers, system users, file modes, daemon auth,
|
||||
backup, SQLite, install.sh, and migration. v0.12 closes all of them
|
||||
and adopts a **zero-trust identity model** as the load-bearing
|
||||
architectural change.
|
||||
|
||||
### Load-bearing rule adopted in Phase 0
|
||||
|
||||
**R-021**: *Orca never issues, stores, or accepts human-identity
|
||||
credentials. Human identity is exclusively external (OIDC). Machine
|
||||
identity is exclusively mTLS/SPIFFE. No passwords, no Orca-issued
|
||||
tokens, no CA-key passphrases.*
|
||||
|
||||
### Zero-trust identity model
|
||||
|
||||
Two identity layers, zero overlap:
|
||||
|
||||
- **Human operators** → OIDC (external IdP, BYO) OR the **bundled Dex**
|
||||
with a **WebAuthn (passkeys) connector** as the default
|
||||
password-free authenticator. `orca auth login` / `orca auth register`
|
||||
open the default browser to the Dex WebAuthn endpoint via OIDC
|
||||
authorization-code + PKCE + local loopback redirect. After the
|
||||
WebAuthn ceremony (biometric/security key), Dex redirects back with
|
||||
an auth code; CLI exchanges for a short-lived ID token (1h) +
|
||||
refresh. Headless/CI fallback: device-code flow.
|
||||
- **Machine-to-machine** → mTLS + SPIFFE SVIDs (unchanged from v0.11).
|
||||
|
||||
The "no Orca credentials" invariant holds: passkeys are public-key
|
||||
credentials (the private key never leaves the authenticator); the
|
||||
WebAuthn credential DB stores only public keys + credential IDs +
|
||||
sign counts. No passwords, no Orca-issued tokens, no CA-key
|
||||
passphrases anywhere in the system.
|
||||
|
||||
### Master key sealing
|
||||
|
||||
The secrets master key (32 random bytes) is **sealed to OIDC** —
|
||||
wrapped by a key derived from an OIDC token exchange at unseal time.
|
||||
`orca cluster unseal` (operator authenticates via OIDC → token
|
||||
exchange → unwrap master key into memory → zeroed on shutdown). The
|
||||
raw master key never touches disk. **Shamir 3-of-5 recovery**: at seal
|
||||
time, 5 shards are printed and the operator stores them offline. If
|
||||
the IdP is permanently lost AND a quorum of shards is unavailable, the
|
||||
cluster is unrecoverable by design (documented residual risk; no
|
||||
backdoor).
|
||||
|
||||
### New requirements (REQ-119..REQ-148)
|
||||
|
||||
30 net-new requirements derived from the threat-model findings and the
|
||||
zero-trust identity model. See REQUIREMENTS.md and ROADMAP.md for the
|
||||
full mapping. Highlights:
|
||||
|
||||
- REQ-119..121: command injection, path traversal, txn path allowlist
|
||||
- REQ-144: OIDC client + bundled Dex (BYO-IdP override)
|
||||
- REQ-145: ACL rewrite (remove KindToken, add KindOidc, enforce)
|
||||
- REQ-146: remove all password/token paths (breaking)
|
||||
- REQ-147: master key seal-to-OIDC + Shamir recovery
|
||||
- REQ-148: WebAuthn connector for Dex (passkeys, browser auth+register)
|
||||
- REQ-122..143: integrity, crypto, OS scripts, emitters, sudoers,
|
||||
system users, SQLite, migration, dual-write closure, transport,
|
||||
drift auth, integration tests, docs, final review
|
||||
|
||||
### v0.12 Clarified Decisions (D-series, full autonomy)
|
||||
|
||||
The 10 v0.12 decisions (D-238..D-247) were resolved during CLARIFY
|
||||
under full autonomy (autonomy.level=full, workflow.no_hitl=true):
|
||||
|
||||
| ID | Question | Decision | Rationale | Confidence |
|
||||
|----|----------|----------|-----------|------------|
|
||||
| D-238 | Milestone version? | **v0.12 (minor, not v1.0)** | v1.0.0 stays deferred for post-UAT per v0.11 PRD; v0.12 is a minor feature milestone. Tags on v0.11.x patch line. | 0.95 |
|
||||
| D-239 | OIDC provider model? | **Bundled Dex by default + BYO external IdP override** | Zero-trust out of the box without external setup; `oidc.issuer` repoint switches to BYO. | 0.90 |
|
||||
| D-240 | Bundled Dex upstream authenticator (password-free)? | **WebAuthn (passkeys) connector** | Public-key credentials; private key never leaves authenticator; reinforces "no passwords" invariant (R-021). | 0.88 |
|
||||
| D-241 | Master key sealing model? | **Seal to OIDC + Shamir 3-of-5 recovery** | No password anywhere; quorum recovery if IdP lost; no backdoor. | 0.85 |
|
||||
| D-242 | CLI browser flow? | **OIDC auth-code + PKCE + local loopback redirect** | Standard OIDC browser flow; secure for public clients; headless fallback via device-code. | 0.92 |
|
||||
| D-243 | WebAuthn RP ID / secure context? | **Traefik-served cluster domain (step-ca cert, R-017)** | WebAuthn requires HTTPS; Traefik already provides it; RP ID configurable via `orca auth init-idp`. | 0.90 |
|
||||
| D-244 | Passkey storage? | **SQLite at ClusterDir()/webauthn-credentials.db (0600); public keys only** | Public keys are not secrets; 0600 file mode for integrity; no passphrase wrapping needed. | 0.92 |
|
||||
| D-245 | Headless/CI auth fallback? | **Device-code flow** | No browser in CI; device-code is the standard OIDC headless path. | 0.90 |
|
||||
| D-246 | Token storage at rest? | **~/.orca/credentials.json (0600); short-lived (1h) + refresh** | Standard OIDC token storage; 0600; refresh handles rotation; no long-lived Orca-issued tokens. | 0.92 |
|
||||
| D-247 | Breaking-change handling for password/token removal? | **`orca upgrade` refuses v0.11 clusters using --password/bare-tokens without --accept-identity-migration** | No silent breakage; explicit migration gate; documented cutover. | 0.90 |
|
||||
|
||||
### v0.12 is a HARDENING + IDENTITY milestone, not a direction change
|
||||
|
||||
The vision ("minimalist, offline-first, CLI-first orchestration
|
||||
engine inspired by HashiCorp Nomad") is unchanged. v0.12 closes the
|
||||
security-surface gaps surfaced by the v0.11 threat model and adopts a
|
||||
zero-trust identity model. The offline-first principle (R-003) is
|
||||
preserved: the bundled Dex can run on the lead (offline), and the
|
||||
mTLS-only path remains for the single-operator fully-offline case (no
|
||||
human authn needed — the operator holds the pre-staged SSH key + mTLS
|
||||
cert; no password, no token).
|
||||
|
||||
### v0.13: Production Hardening Round 2 + UAT Plan (IN PROGRESS)
|
||||
|
||||
v0.12 (Security Hardening) is COMPLETE. v0.13 is the **final hardening round before UAT validation**. The UAT will likely surface 3-7 issues requiring a patch release. v1.0.0 is deferred until UAT passes.
|
||||
|
||||
v0.13 is the **final hardening
|
||||
round** before the v1.0.0 production-ready tag. Three deep codebase
|
||||
sweeps (security, reliability, feature/doc claims) surfaced ~60 gaps
|
||||
beyond v0.12. The most critical:
|
||||
|
||||
1. **`orca job run` runs locally** via `exec.CommandContext` — the
|
||||
scheduler/emitter/SSH-push pipeline is dead code. The documented
|
||||
deployment model (deploy to Proxmox/Ubuntu worker) is non-functional.
|
||||
**R-022** fixes this.
|
||||
2. **jobspec `schedule:`/`timeout:` silently dropped** by the markdown
|
||||
parser — DaemonSet is fundamentally broken (parser defaults Count=1,
|
||||
validator rejects Count!=0, schedule never parsed).
|
||||
3. **`acl.Check` called zero times** — v0.12's headline zero-trust
|
||||
feature is library-complete but not wired into any request path.
|
||||
**R-023** fixes this.
|
||||
4. **Command injection vectors** — `orca logs --job` backtick RCE via
|
||||
`%q` (bash executes command substitution in double quotes), tar-slip
|
||||
in backup restore, sudoers injection via `--proxmox-user`/`--role`,
|
||||
`txn rollback` shell injection, and 7 more.
|
||||
5. **Go toolchain 1.25.0** — 24 stdlib vulns with call traces in orca
|
||||
(archive/tar, crypto/tls, crypto/x509, net/http, encoding/pem...).
|
||||
6. **Concurrency hazards** — audit hash-chain race (concurrent appends
|
||||
corrupt tamper-evidence), concurrent `secrets set` silently loses
|
||||
data (no flock), no SQLite `busy_timeout` (database is locked),
|
||||
concurrent `orca upgrade` races on Traefik cutover.
|
||||
7. **Cache never invalidated by writes** — stale reads for 10–60s
|
||||
after `node join`/`ns create`/`job run`.
|
||||
8. **Massive doc drift** — README "mTLS by default" is false (SSH-push
|
||||
is canonical), `docs/cli.md` missing ~25 subcommands, CHANGELOG
|
||||
stale at v0.1, `verify-reqs` gate bypassed for v0.12.
|
||||
|
||||
v0.13 closes all critical/high/medium findings (15 new requirements,
|
||||
14 phases) and delivers the **UAT plan + signoff script** that gates
|
||||
the v1.0.0 cut.
|
||||
|
||||
### v0.14 Milestone: Ingress Bootstrap Completeness
|
||||
|
||||
**Scope**: ensure that linux & proxmox types are properly bootstrapped
|
||||
with traefik during cluster init or node join. All cluster endpoints are
|
||||
provisioned as sockets (R-007); routing between jobs and services
|
||||
depends on traefik being present and properly configured. v0.13 shipped
|
||||
traefik binary + systemd unit + empty dynamic dir but never wrote the
|
||||
static config nor applied nft rules — `orca-traefik.service` fails on a
|
||||
fresh `orca init` and `orca doctor nft` FAILs. v0.14 replaces the
|
||||
binary+systemd model with a **podman container** running a custom
|
||||
`orca-traefik` image, and completes the nft SNAT+DNAT ingress stack on
|
||||
every node type.
|
||||
|
||||
**New load-bearing rule**: **R-024** — Traefik runs exclusively as a
|
||||
podman container, deployed from the `orca-traefik` image published per
|
||||
release. Every orca-managed ingress surface bootstraps: nft DNAT
|
||||
(`:443→127.0.0.1:8443`, `:80→127.0.0.1:8080`) + SNAT/MASQUERADE
|
||||
postrouting + `podman run -d --restart=always --network host -v
|
||||
/etc/traefik/dynamic:/etc/traefik/dynamic:Z -v
|
||||
/etc/orca/step-ca-root.crt:/etc/orca/step-ca-root.crt:ro
|
||||
git.cloudinit.dev/coreci/orca-traefik:<tag>`. No node joins without a
|
||||
functional podman-traefik ingress data plane.
|
||||
|
||||
**Three topologies** (per operator constraints):
|
||||
1. **Linux**: host → nft → `podman run orca-traefik` (host network)
|
||||
2. **Proxmox Native**: host → nft → LXC (nesting=1) → `podman run orca-traefik`
|
||||
3. **Proxmox Floating-IP**: LXC (owns floating IP) → nft (inside LXC) → `podman run orca-traefik`
|
||||
|
||||
**Milestone type**: feature (multiple `feat` phases). Tags on v0.13.x
|
||||
patch line: `v0.13.0` (P0) ... `v0.13.8` (P8 final = v0.14 milestone
|
||||
release). 9 phases, 9 net-new requirements (REQ-171..REQ-179).
|
||||
|
||||
### v0.14 Decisions (D-series, full autonomy)
|
||||
|
||||
| ID | Question | Decision | Rationale | Confidence |
|
||||
|----|----------|----------|-----------|------------|
|
||||
| D-255 | Traefik deployment model? | **Podman container from custom `orca-traefik` image** | Operator constraint: traefik always deployed as a container. Replaces v0.13 binary+systemd. Image bakes static config. | 0.95 |
|
||||
| D-256 | Container network mode? | **`--network host`** | Binds 127.0.0.1:8080/8443 directly on host/LXC loopback; nft DNAT targets that. No port publishing complexity. | 0.92 |
|
||||
| D-257 | TLS cert resolver in image? | **No certResolver; `tls: {}` for v0.14, real mTLS deferred to v0.15** | Grill G-003 (confidence 0.55 < 0.60) auto-resolved to defer. Traefik v3.3 `certificatesResolvers` only supports acme/tailscale, not CA-file. Drop `certResolver: orca` (broken). Emit `tls: {}` in dynamic config. Real mTLS via dynamic `tls.certificates` + `tls.options.default.clientAuth.caFiles` lands in v0.15 when step-ca mints server certs. | 0.90 |
|
||||
| D-258 | Dynamic config volume? | **Mount `/etc/traefik/dynamic` from host** | Zero changes to existing `deployRemote` WriteFile path (`job_dispatch.go:243`). File provider watches it. | 0.95 |
|
||||
| D-259 | Floating-IP mode: register PVE host too? | **Yes — PVE host as `proxmox` + ingress LXC as `linux`** | PVE host needed in registry for `pct`/`qm` workload dispatch. Both register. | 0.92 |
|
||||
| D-260 | `--ingress-mode` persistence? | **Store `IngressMode` on `model.Node`** | `doctor ingress` needs to know which check path to run. Schema migration. | 0.90 |
|
||||
| D-261 | MAC generation when `--mac` omitted? | **Generate random `02:XX:...` in interactive mode; require `--mac` in `--json` mode** | Interactive: generate + confirm. Non-interactive: explicit required (no silent generation). | 0.88 |
|
||||
| D-262 | Proxmox native nft DNAT target? | **LXC bridge IP (not 127.0.0.1)** | LXC has its own network namespace; 127.0.0.1 on PVE host ≠ LXC loopback. `NftClusterConfig.DNATTarget` field (default `127.0.0.1:8443`; native mode = `<lxc-ip>:8443`). | 0.90 |
|
||||
| D-263 | LXC podman requirements? | **`--features nesting=1,keyctl=1` + `apt-get install podman`** | Ubuntu 24.04 LXC template has no podman preinstalled. Nesting+keyctl required for podman in unprivileged LXC. | 0.88 |
|
||||
|
||||
### v0.13 Decisions (D-series, full autonomy)
|
||||
|
||||
| ID | Question | Decision | Rationale | Confidence |
|
||||
|----|----------|----------|-----------|------------|
|
||||
| D-248 | Milestone version? | **v0.13 (minor, not v1.0)** | v1.0.0 stays deferred for UAT signoff; v0.13 is a minor feature milestone. Tags on v0.12.x patch line. | 0.95 |
|
||||
| D-249 | UAT validation mechanism? | **Operator-driven `docs/uat.md` + `scripts/uat-signoff.sh` assertions** | Operator builds real cluster (3 hosts), runs signoff script, pastes output. Exit 0 iff all ~35 assertions pass. | 0.92 |
|
||||
| D-250 | Hardening phase scope? | **All 8 themes, 14 phases** | "No limit on phases" per operator; comprehensive to avoid a round 3. | 0.90 |
|
||||
| D-251 | Ubuntu worker onboarding? | **Implement `--type linux` SSH-join** | `NodeKindLinux` is reserved but unimplemented; UAT plan needs first-class worker onboarding. Proxmox stays `--type proxmox`. | 0.88 |
|
||||
| D-252 | `job stop` semantics? | **Real `systemctl stop` via SSH** | Honest semantics matching `job restart` pattern; UAT assumes stop actually stops. | 0.90 |
|
||||
| D-253 | UAT cluster topology? | **3 hosts: lead Ubuntu + pve01 Proxmox + worker01 Ubuntu** | Minimal topology covering both node types + migrate-between-hosts. | 0.92 |
|
||||
| D-254 | UAT signoff script re-runnable? | **Idempotent — read + non-mutating assertions only** | Operator can iterate; no destructive ops. | 0.95 |
|
||||
|
||||
### v0.13 is the LAST hardening round
|
||||
|
||||
Three deep sweeps (security, reliability, feature/doc) were performed
|
||||
to ensure no gap is missed. 9 low-severity residual risks are
|
||||
documented and accepted (OIDC tokens plaintext at rest, HSTS on
|
||||
daemon, DNS timeout, temp file cleanup on SIGKILL, flock timeout on
|
||||
NFS, WASM-first aspirational, arm64 release, OIDC callback slowloris,
|
||||
pprof-allow-public flag). v0.13 closes everything else. The v1.0.0
|
||||
tag is cut only after the UAT signoff script passes.
|
||||
|
||||
+17
-372
@@ -21,7 +21,7 @@ earlier versions of this file.
|
||||
| REQ-011 | mTLS for inter-node communication | Medium | **v0.2 P01** | **Complete** (P01 shipped v0.2.1) |
|
||||
| REQ-012 | `~/.orca/config.hcl` and `/etc/orca/orca.hcl` config locations | Low | v0.1 P01 | **Complete** (CLI uses `~/.orca/` + `ORCA_DB` env) |
|
||||
| REQ-013 | Pre-push git hook triggers CoreCI on every push | High | v0.1 P01 | **Complete** |
|
||||
| REQ-014 | `gosec` + `govulncheck` in CI pipeline | High | v0.2 P03 | **Complete** (P10 shipped v0.2.3) |
|
||||
| REQ-014 | `gosec` + `govulncheck` in CI pipeline | High | v0.2 P03 | Pending (P03) |
|
||||
| REQ-015 | MIT LICENSE | Low | v0.1 P01 | **Complete** |
|
||||
| REQ-016 | README.md with quickstart | Medium | v0.1 P01 | **Complete** |
|
||||
| REQ-017 | `context.Context` propagation in all I/O | High | v0.1 | **Complete** |
|
||||
@@ -29,31 +29,25 @@ earlier versions of this file.
|
||||
| REQ-019 | Cobra CLI framework | High | v0.1 P01 | **Complete** |
|
||||
| REQ-020 | HCL parser integration (`hashicorp/hcl`) | Medium | v0.1 P03 | **Complete** |
|
||||
| REQ-021 | `os/exec` with `WaitDelay` (Go 1.25+) | Medium | v0.1 P03 | **Complete** |
|
||||
| REQ-022 | `iter.Seq` for streaming job lists (Go 1.25+) | Low | **v0.3 P01** | **Complete** (v0.3 P01 shipped v0.3.1) |
|
||||
| REQ-022 | `iter.Seq` for streaming job lists (Go 1.25+) | Low | v0.2 P04 | Pending (P04) |
|
||||
| REQ-023 | Self-signed mTLS cert generation | Medium | **v0.2 P01** | **Complete** (P01 shipped v0.2.1) |
|
||||
| REQ-024 | `Makefile` with standard targets | High | v0.1 P01 | **Complete** |
|
||||
| REQ-025 | Bounded cert rotation history: retain last N=3 server certs per node for rollback | Medium | **v0.2 P01** | **Complete** (P01 shipped v0.2.1) |
|
||||
| REQ-026 | Trusted-CA fingerprint pinned in config; daemon refuses to start on mismatch | High | **v0.2 P01** | **Complete** (P01 shipped v0.2.1) |
|
||||
| REQ-027 | `govulncheck` runs in offline mode in CI (no `vuln.go.dev` calls; pre-mirrored DB or `-format json` + `jq` gate) | High | v0.2 P03 | **Complete** (P10 shipped v0.2.3) |
|
||||
| REQ-028 | HCL/YAML schema for `NodeCapacity` declaration (`orca node join` flag and/or `~/.orca/node.hcl`) | High | v0.2 P02 | **Complete** (P09 shipped v0.2.2; `orca node capacity` CLI) |
|
||||
| REQ-029 | `gitleaks` baseline file committed to repo to suppress pre-existing `.env` SHA-1 leak in git history | Medium | v0.2 P03 | **Complete** (P10 shipped v0.2.3) |
|
||||
| REQ-030 | `--watch` output format mode: table (default) vs streaming one-line JSON per event | Low | **v0.3 P01** | **Complete** (v0.3 P01 shipped v0.3.1) |
|
||||
| REQ-031 | `go test -race` enabled in CI for all v0.2 packages | High | v0.2 P01–P04 | **Complete** (P10; `.coreci.yml` test pipeline runs `-race`) |
|
||||
| REQ-032 | `orca doctor` subcommand for diagnostics (CA/cert health, db integrity, peer reachability) | Medium | **v0.2 P01 / v0.3 P02** | **Complete** (cert checks P01 v0.2.1; network + db P02 v0.3.2) |
|
||||
| REQ-027 | `govulncheck` runs in offline mode in CI (no `vuln.go.dev` calls; pre-mirrored DB or `-format json` + `jq` gate) | High | v0.2 P03 | Pending (P03) |
|
||||
| REQ-028 | HCL/YAML schema for `NodeCapacity` declaration (`orca node join` flag and/or `~/.orca/node.hcl`) | High | v0.2 P02 | Pending (P02) |
|
||||
| REQ-029 | `gitleaks` baseline file committed to repo to suppress pre-existing `.env` SHA-1 leak in git history | Medium | v0.2 P03 | Pending (P03) |
|
||||
| REQ-030 | `--watch` output format mode: table (default) vs streaming one-line JSON per event | Low | v0.2 P04 | Pending (P04) |
|
||||
| REQ-031 | `go test -race` enabled in CI for all v0.2 packages | High | v0.2 P01–P04 | **Complete** for P01 (cross-cutting, verified P01); P02–P04 ongoing |
|
||||
| REQ-032 | `orca doctor` subcommand for diagnostics (CA/cert health, db integrity, peer reachability) | Medium | **v0.2 P01** | **Complete** for cert checks (P01); network/db are stubs, full impl in P02 |
|
||||
| REQ-033 | Cert file mode enforcement: 0600 for keys, 0644 for certs (refuses to start on violation) | High | **v0.2 P01** | **Complete** (P01 shipped v0.2.1) |
|
||||
| REQ-034 | Cert proactive rotation alarm: structured slog WARN 30 days before `not_after` | Medium | **v0.2 P01** | **Complete** (P01 shipped v0.2.1) |
|
||||
| REQ-035 | `orca cert show` redacts private key material from default and `--json` output | High | **v0.2 P01** | **Complete** (P01 shipped v0.2.1) |
|
||||
| REQ-036 | Server cert SAN validation: SAN entries (DNS + IP) populated at sign-time; refuses to sign a CSR without them | High | **v0.2 P01** | **Complete** (P01 shipped v0.2.1) |
|
||||
| REQ-037 | `X-Orca-Idempotency-Key` header on cross-node POST; dispatcher retries only when header is present | Medium | v0.2 P02 | **Complete** (P09 shipped v0.2.2; `internal/transport/idempotency.go`) |
|
||||
| REQ-037 | `X-Orca-Idempotency-Key` header on cross-node POST; dispatcher retries only when header is present | Medium | v0.2 P02 | Pending (P02) |
|
||||
| REQ-038 | Structured slog fields for mTLS failures: `event=mtls.handshake`, `peer`, `cert_fp`, `err` | Medium | **v0.2 P01** | **Complete** (P01 shipped v0.2.1) |
|
||||
| REQ-039 | `.gitleaks.toml` extended with stopwords for test data paths and CA cert PEM blocks | Medium | v0.2 P03 | **Complete** (P10 shipped v0.2.3) |
|
||||
| REQ-040 | `.golangci.yml` unified lint config superseding per-tool invocations | Low | v0.2 P03 | **Complete** (P10 shipped v0.2.3) |
|
||||
| REQ-041 | Unified namespace root via `ORCA_HOME` for all components (db, certs, init, daemon) | High | **v0.5 P1** | **Complete** (P1 shipped v0.4.2) |
|
||||
| REQ-042 | `--system` flag selects system-level namespace root `/root/.orca` | High | **v0.5 P1** | **Complete** (P1 shipped v0.4.2) |
|
||||
| REQ-043 | `install.sh` 1-liner pulling release binary from public Gitea URL; user-level default, `--system` for system-level | High | **v0.5 P2** | **Complete** (P2 shipped v0.4.3) |
|
||||
| REQ-044 | `install.sh` in-place update preserves config/state; idempotent re-run | High | **v0.5 P2** | **Complete** (P2 shipped v0.4.3) |
|
||||
| REQ-045 | Gitea repo + releases publicly accessible (unauthenticated download) | High | **v0.5 P0** | **Complete** (P0 ship: repo + org visibility public) |
|
||||
| REQ-046 | Docker image published to Gitea container registry per release | Medium | **v0.5 P3** | **Complete** (P3 shipped v0.4.4) |
|
||||
| REQ-039 | `.gitleaks.toml` extended with stopwords for test data paths and CA cert PEM blocks | Medium | v0.2 P03 | Pending (P03) |
|
||||
| REQ-040 | `.golangci.yml` unified lint config superseding per-tool invocations | Low | v0.2 P03 | Pending (P03) |
|
||||
|
||||
## v0.1 Milestone Summary
|
||||
|
||||
@@ -68,362 +62,13 @@ Plus REQ-025..REQ-040 (16 net-new) added by v0.2 IDEATE stage.
|
||||
|
||||
## v0.2 Milestone Summary
|
||||
|
||||
**Status: Functionally Complete (pending merge to main)** — P08 (mTLS),
|
||||
P09 (scheduling), P10 (security scan) all shipped to the
|
||||
`milestone/v0.2-networking-observability-security` branch as v0.2.1,
|
||||
v0.2.2, v0.2.3. The milestone branch has NOT been merged to main yet.
|
||||
REQ-022/030 (iter.Seq streaming) and REQ-032 (doctor network/db) were
|
||||
deferred to v0.3.
|
||||
**Status: In Progress** — P01 (mTLS) shipped (v0.2.1). 3 phases remain
|
||||
(P02 multi-node scheduling, P03 gosec+govulncheck+gitleaks, P04 iter.Seq).
|
||||
P01 covered REQ-011, REQ-023, REQ-025, REQ-026, REQ-031, REQ-032 (partial),
|
||||
REQ-033, REQ-034, REQ-035, REQ-036, REQ-038 (10 REQs complete; REQ-032
|
||||
complete for cert checks only).
|
||||
|
||||
## v0.3 Milestone Summary
|
||||
|
||||
**Status: Complete** — P01 (iter.Seq streaming, v0.3.1) and P02 (doctor
|
||||
network+db, v0.3.2) both shipped. REQ-022, REQ-030, REQ-032 all complete.
|
||||
Re-init SPECIFY audit confirmed all other v0.2-deferred REQs (014, 027,
|
||||
028, 029, 031, 037, 039, 040) already shipped in P08-P10.
|
||||
|
||||
## Deferred to v0.4
|
||||
## Deferred to v0.3
|
||||
|
||||
- pprof endpoint on `orca daemon` (idea I-308, 0.70 confidence): deferred
|
||||
to keep v0.2 lean; revisit in v0.3 once P02's dispatcher is stable.
|
||||
|
||||
## v0.5 Milestone Summary
|
||||
|
||||
**Status: Complete** — all 3 execution phases + final review shipped.
|
||||
P0 (v0.4.1), P1 (v0.4.2), P2 (v0.4.3), P3 (v0.4.4), P4 final (v0.4.5).
|
||||
REQ-041..046 all complete. Repo + releases publicly accessible (REQ-045).
|
||||
Docker image published to Gitea container registry (REQ-046).
|
||||
|
||||
- **P0** (v0.4.1): pre-execution + repo visibility flipped to public (REQ-045).
|
||||
- **P1** (v0.4.2): namespace unification — `ORCA_HOME` + `--system` (REQ-041/042).
|
||||
- **P2** (v0.4.3): `install.sh` 1-liner + in-place update (REQ-043/044) + README quickstart (REQ-016).
|
||||
- **P3** (v0.4.4): Docker release — distroless image + Gitea container registry (REQ-046).
|
||||
- **P4** (v0.4.5): final review + audit + milestone release.
|
||||
|
||||
## v0.6 Requirements — Node Bootstrap & Proxmox
|
||||
|
||||
| ID | Requirement | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-047 | `orca init` auto-provisions CA + server cert + DB migrations + localhost node (idempotent; safe re-run) | High | **v0.6 P1** | **Complete** (P1 shipped v0.5.1) |
|
||||
| REQ-048 | `orca init` registers a default `localhost` node with auto-detected OS via `/etc/os-release ID` | High | **v0.6 P1** | **Complete** (P1 shipped v0.5.1) |
|
||||
| REQ-049 | Node schema extension: `nodes.kind` (localhost\|linux\|proxmox) + `nodes.os` columns (migration 0006, backward-compatible) | High | **v0.6 P1** | **Complete** (P1 shipped v0.5.1) |
|
||||
| REQ-050 | `orca node join --type proxmox` SSH bootstrap via `golang.org/x/crypto/ssh` (new direct dep); password auth, deploy orca pubkey, create `orca` user (config-overridable), assign PVE role, drop sudoers allowlist; idempotent | High | **v0.6 P2** | **Complete** (P2 shipped v0.5.2) |
|
||||
| REQ-051 | Proxmox least-privilege `OrcaOperator` PVE role (VM.Audit, Datastore.AllocateSpace, SDN.Use) + `orca` user + `/etc/sudoers.d/orca` allowlist (pct, qm, pvesh, apt-get, dpkg); config-overridable user/role names | High | **v0.6 P2** | **Complete** (P2 shipped v0.5.2; refined: pvesh excluded per AD-020, orca@pam per AD-019) |
|
||||
| REQ-052 | `orca doctor` extensions: `doctor os` (verify localhost OS detection matches stored node row) + `doctor proxmox` (SSH-probe each `kind=proxmox` node with `pveversion`/`pvecmd status`, 3s timeout, PASS/WARN/FAIL); audit log all bootstrap + join actions | Medium | **v0.6 P3** | **Complete** (P3 shipped v0.5.3) |
|
||||
|
||||
## v0.6 Milestone Summary
|
||||
|
||||
**Status: Complete** — all 3 execution phases + final review shipped.
|
||||
P0 (v0.5.0), P1 (v0.5.1), P2 (v0.5.2), P3 (v0.5.3), P4 final (v0.5.4).
|
||||
REQ-047..052 all complete.
|
||||
|
||||
- **P0** (v0.5.0): pre-execution (specify → clarify → research → plan). 8 decisions (D-030..D-037).
|
||||
- **P1** (v0.5.1): `orca init` full bootstrap + schema 0006 (REQ-047/048/049).
|
||||
- **P2** (v0.5.2): Proxmox SSH join + OrcaOperator role + sudoers (REQ-050/051).
|
||||
- **P3** (v0.5.3): `doctor os` + `doctor proxmox` + audit logging (REQ-052).
|
||||
- **P4** (v0.5.4): final review + audit + milestone release.
|
||||
|
||||
## v0.7 Requirements — Hardening & Completion
|
||||
|
||||
| ID | Requirement | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-053 | `orca cert` command tree registered on root command (`cert ca-init`, `cert gen`, `cert show`, `cert renew`, `cert fingerprint`) — code exists in `internal/cli/cert.go` but is never AddCommand'd; unreachable today | High | **v0.7 P1** | **Complete** (P1 shipped v0.6.1) |
|
||||
| REQ-054 | HCL config file parsing: `internal/config` package loads `~/.orca/config.hcl` / `/etc/orca/orca.hcl` (keys: db_path, listen_addr, ca_path, server_cert_path, server_key_path, node_capacity); merge precedence flag > env > file > default; `--config` flag on root command | High | **v0.7 P2** | **Complete** (P2 shipped v0.6.2) |
|
||||
| REQ-055 | Test coverage uplift: every package ≥ 50% — adds tests for `internal/engine` (executor, dispatcher, peer), `internal/transport` (mtls, dispatch, handshake_log), `internal/proxmox` (bootstrap SSH path), `internal/audit` | Medium | **v0.7 P3** | **Complete** (P3 shipped v0.6.3) |
|
||||
| REQ-056 | `--pprof <addr>` opt-in flag on `orca daemon` (default disabled); `net/http/pprof` mounted on a separate mux, never on the mTLS daemon listener | Low | **v0.7 P4** | **Complete** (P4 shipped v0.6.4) |
|
||||
|
||||
## v0.8 Requirements — Coverage & Trust Hardening
|
||||
|
||||
| ID | Requirement | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-057 | Test coverage uplift round 2: raise `internal/engine` (8.3%), `internal/proxmox` (5.1%), `internal/cli` (27.6%), `internal/transport` (26.3%), `internal/store` (46.7%), `internal/jobspec` (47.6%) to ≥ 70%; add first tests for `internal/audit`, `internal/certpaths`, `cmd/orca` (currently 0%) to ≥ 50% (D-047 tiered floor) | High | **v0.8 P1** | **Complete** (P1 shipped v0.7.1; all 9 packages exceeded floor) |
|
||||
| REQ-058 | `--host-key-fingerprint <SHA256:base64>` pre-pin flag on `orca node join` (validated when `--type proxmox`): when supplied, join fails fast if the SSH host key's OpenSSH SHA-256 fingerprint does not match; supersedes TOFU (D-035) for pre-pinned deployments (D-044, D-045) | Medium | **v0.8 P2** | **Complete** (P2 shipped v0.7.2) |
|
||||
| REQ-059 | `orca node key-reset <node>` command: clears the persisted SSH host key entry for the node from `~/.orca/known_hosts` only (local, not remote authorized_keys — D-046); audit-logs `event=node.key_reset`; next `doctor proxmox`/dispatch re-pins via TOFU or `--host-key-fingerprint` | Low | **v0.8 P2** | **Complete** (P2 shipped v0.7.2) |
|
||||
| REQ-060 | Requirement-status hygiene sweep: REQUIREMENTS.md v0.7 rows were stale ("Pending" after ship); add a verify-stage assertion that every REQ listed as `Complete` in ROADMAP.md has a matching `Complete` row in REQUIREMENTS.md, enforced by `make verify-reqs` | Medium | **v0.8 P3** | **Complete** (P3 shipped v0.7.3) |
|
||||
|
||||
## v0.9/v0.10 Requirements — Re-architecture Foundation & Production Hardening
|
||||
|
||||
The v0.9/v0.10 milestones supersede the shipped v0.1–v0.8 architecture per the
|
||||
adopted PRD (`.ciagent/PRD_v0.9.md`). The re-architecture is justified on six
|
||||
grounds recorded in the PROJECT.md Supersession Table. 30 net-new requirements
|
||||
(REQ-061..REQ-090) derive from the v0.9 IDEATION; their phase placement and
|
||||
binding grill conditions (C-01..C-19) are documented in `IDEATION_v0.9.md`
|
||||
and `GRILL_v0.9.md`.
|
||||
|
||||
| ID | Requirement | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-061 | `orca daemon` deprecation command and build-tag removal path: v0.9 emits deprecation warning + still runs (dual-write window); v1.0 repurposes to `orca daemon drain-and-stop` (stops v0.8 daemons on peers via SSH, confirms workloads survive via systemd); post-v1.0 the command and `internal/daemon/` are deleted. `// Deprecated` Go doc comments + `slog.Warn` on every run (I-M-001) | High | **v0.11 P14b** (drain-and-stop + rotate-lead) | **Complete** |
|
||||
| REQ-062 | Coverage follow-ups: 3 zero-test packages (`internal/audit`, `internal/certpaths`, `cmd/orca`) + `internal/cli` to 70% floor; once `daemon.go` is deprecated/removed the exclusion reason disappears and the floor applies to the whole package; all net-new subsystems carry a 70% floor from their first phase (I-M-002) | Medium | **v0.9 P0X** + each new pkg | Complete |
|
||||
| REQ-063 | `known_hosts` flock concurrency gap (deferred P1 from REVIEW_v0.8 A2): add `flock`-style advisory lock (stdlib `syscall.Flock` wrapper) around the read-modify-write in `TOFUHostKeyCallback` capture path (`bootstrap.go:290-302`) and `ResetHostKey` (`bootstrap.go:479-523`); lock file at `cluster/known_hosts.lock` (R-002) (I-M-003) | Medium | **v0.9 P0a1** | Complete |
|
||||
| REQ-064 | HCL→Markdown jobspec adapter/bridge layer: keep `internal/jobspec/spec.go` as legacy HCL path behind `// Deprecated`; add `internal/jobspec/markdown.go` (canonical) + `internal/jobspec/dispatch.go` (extension-based dispatcher: `.md`→Markdown, `.hcl`→legacy, `.yaml`→Markdown-with-empty-body); unified `*WorkloadSpec` populated via adapter; preserves `orca job run old-spec.hcl` during migration window (I-M-004) | High | **v0.9 P0b** | Complete |
|
||||
| REQ-065 | `orca doctor --legacy-paths` detection: detects v0.8 residue (orca.db at ORCA_HOME root, ca.crt/ca.key, config.hcl, flat server.crt, namespace column in any *.db); outputs list of legacy artifacts with migration recommendations; the detection half of v0.10-P14 (I-M-005) | Medium | **v0.11 P14c** | **Complete** |
|
||||
| REQ-066 | Legacy CA state migration to step-ca: `orca upgrade --to-v1.0 --import-ca` reads `~/.orca/ca.key`, initializes step-ca with it, re-issues workload SVIDs; preserves audit history even if live trust root changes (I-M-006). **Gated by C-07** | High | **v0.11 P14a** | **Complete** |
|
||||
| REQ-067 | Fuzz test harness for Markdown frontmatter parser: `testing.F` fuzz target in `internal/jobspec/markdown_test.go` round-trips random frontmatter+body through `ParseMarkdown` asserting byte-exact body preservation; corpus of adversarial fixtures (CRLF, BOM, no-frontmatter, empty-frontmatter, frontmatter-with-only-separator) (I-M-007) | Medium | **v0.9 P0b** | Complete |
|
||||
| REQ-068 | Deprecation warnings on removed/repurposed CLI subcommands: each removed/changed command (`orca cert`, `orca node join` mTLS semantics, `orca job run <spec.hcl>`) emits `slog.Warn` deprecation banner with v1.0 replacement except under `orca upgrade`; `--no-deprecation-warnings` global flag via `root.go` `PersistentPreRunE` (I-M-008) | Low | **v0.9 P0X** + v0.10 P13 | Complete |
|
||||
| REQ-069 | `internal/config/config.go` HCL config demotion via adapter: keep `internal/config/` as `legacy_config.go` with `// Deprecated`; add `internal/config/markdown.go` for new Markdown-frontmatter loader (R-014); `root.go` dispatches on file extension (`.hcl`→legacy, `.md`→new); `--config` semantics: `.hcl` read-only legacy, `.md` canonical (I-M-009) | High | **v0.9 P0a1** | Complete |
|
||||
| REQ-070 | `internal/certpaths/` replacement with multi-namespace path resolver: new `internal/paths` package with `paths.NamespaceDir(ns)`, `paths.ClusterDir()`, `paths.CacheDB()`, `paths.MasterKey()`, `paths.NSDb(ns)`, `paths.NSEnv(ns)`, `paths.NSSecrets(ns)`; keep `certpaths` as thin shim for v0.8 compat then remove post-v1.0 (R-002) (I-M-010) — highest blast radius | High | **v0.9 P0a1** | Complete |
|
||||
| REQ-071 | `internal/store/` schema: per-namespace DBs, drop namespace column: `store.Open` gains namespace parameter (or caller passes `paths.NSDb(ns)`); `migrate.go` runs migrations per namespace DB; `cert_repo` (0004) removed (step-ca handles certs); audit_log moves to CLI-side cache DB (R-008) (I-M-011) | High | **v0.9 P0a1** + v0.10 P06 | Complete |
|
||||
| REQ-072 | `internal/transport/` deletion + SSH-push package: delete `mtls.go`, `dispatch.go`, `handshake_log.go`; extract retry/idempotency patterns into `internal/sshpush/`; existing `transport.IdempotencyStore` directly reusable (I-M-012). Deletion deferred to v0.10-P14 to keep dual-write window open | High | **v0.9 P00** (delete v0.10 P14) | Complete |
|
||||
| REQ-073 | SSH-push transport layer design: connection pooling (reuse `*ssh.Client` per peer), idempotency (content-addressed filenames), retry (exponential backoff 100ms×2 cap 5s max 5), timeout (30s SCP, 10s exec), fan-out (errgroup bounded concurrency default 8), known_hosts reuse `proxmox.TOFUHostKeyCallback` (I-B-001) | High | **v0.9 P01** (design P0a1) | Complete |
|
||||
| REQ-074 | Emitter template system (Layer 4): `internal/emitter/` package with `Emitter` interface `Render(spec *WorkloadSpec, node *Node) ([]File, error)`; implementations systemdEmitter/traefikEmitter/syncthingEmitter/socketEmitter; SSH-push SCPs `[]File` atomically (write-to-tmp + rename); emitters registered per kind + runtime (I-B-002) | High | **v0.9 P0c** | Complete |
|
||||
| REQ-075 | Lead applier execution model: CLI renders transaction bundle (tarball + apply.sh + verify.sh) on operator host, SCPs to lead's `/run/orca/txns/<txn-id>/`, lead's systemd timer runs `apply.sh` idempotently, CLI polls txn status via SSH; bash scripts generated by emitter not hand-written (I-B-003). **Gated by C-09** | High | **v0.11 P10a** | **Complete** |
|
||||
| REQ-076 | step-ca integration: `orca init` runs `step ca init` on lead; CLI SSHs to lead, installs step-ca via apt, stores step-ca.json; workload SVIDs via `step ca token` (JWE minted by CLI) → `step ca certificate`; SPIFFE ID as SAN; new `internal/stepca/` package wraps `step` CLI via SSH (I-B-004). Reverses AD-010 per override justification ground 2 | High | **v0.9 P07** + v0.10 P02 | Complete |
|
||||
| REQ-077 | Traefik dynamic config generation + atomic reload: Traefik emitter renders `/etc/traefik/dynamic/orca-<ns>-<svc>.yaml` with backends (socket paths R-007), health checks, mTLS config pointing at step-ca root; atomic reload via tmpfile+fsync+rename triggering fsnotify; drain writes `weight=0` or removes backend (I-B-005). **Gated by C-10** | High | **v0.9 P02** | Complete |
|
||||
| REQ-078 | Runtime abstraction interface (5 backends): `Runtime` interface in `internal/runtime/` with Prepare/Start/Stop/Status; processRuntime (wraps existing executor.go), wasmRuntime (wasmtime via SSH), podmanRuntime, pveVMRuntime (qm via proxmox SSH), pveCTRuntime (pct); runtimeRegistry keyed by `runtime:` frontmatter value; Alloc carries runtime field changeable on migration (I-B-006). Split P07a/b/c per PC-10. **P07b gated by C-01** | High | **v0.9 P07a/b/c** | Complete |
|
||||
| REQ-079 | Transaction bundle format + N-peer atomicity: bundle = tarball with desired-state.json + apply.sh + verify.sh + rollback.sh + manifest.sig (signed with master.key); content-addressed `<txn-id>=sha256(desired-state.json)` stored in `cluster/txns/<txn-id>/`; lead applies to self first then fans out; failure on any peer runs rollback.sh on applied peers (I-B-007). **Gated by C-09** | High | **v0.11 P10a** | **Complete** |
|
||||
| REQ-080 | Master key management + HKDF-SHA256 per-line .env.secrets encryption: `cluster/master.key` 32-byte random (generated at `orca init` using WriteAtomic pattern); each line `base64(nonce||ciphertext||tag)`, nonce=random(12 bytes), AES-256-GCM with AAD=line-number (prevents line-swap); HKDF-SHA256 derives per-namespace sub-keys; `orca secrets set/get`; v0.8 `internal/security/redact.go` reusable (I-B-008). **Gated by C-19** | High | **v0.11 P03** | **Complete** |
|
||||
| REQ-081 | Syncthing config rendering + folder-ID content-addressing: per-namespace Syncthing folder `orca-<ns>` with content-addressed folder ID `sha256(ns + master-key-fingerprint)`; CLI renders config.xml per peer; Syncthing runs as systemd unit (emitted by systemd emitter); CLI discovers peers via `cluster/peers/`; migration works because new node joins folder and syncs before workload starts (I-B-009). **Gated by C-02 + C-14** | Medium | **v0.9 P09** (spike v0.9 P00) | Complete |
|
||||
| REQ-082 | Namespace inheritance resolver algorithm: DFS parent walker with visited set for cycle detection; `_defaults/` implicit root (always exists, no parent); merge semantics: child overrides parent for scalars, arrays unioned (child adds to parent); pure function (no I/O) taking `map[nsName→*NSConfig]` returning `map[nsName→*ResolvedNS]` (I-B-010) | High | **v0.9 P0a2** | Complete |
|
||||
| REQ-083 | CLI-side scheduler redesign: `Score(node, workload) (score int, fits bool)` where `fits` checks runtime compatibility + constraints, `score` is bin-packing (most free capacity = highest); Services pick `count` distinct nodes (anti-affinity default); DaemonSets pick all matching nodes; Job = one-shot; CLI-side not daemon-side (R-001) (I-B-011) | High | **v0.9 P05** (skeleton P0c) | Complete |
|
||||
| REQ-084 | `orca job lint` category-driven lint engine: `Linter` runs `Rule` checks returning `Finding{Category, Severity, Message, Explanation}`; categories schema/runtime/security/migration/best-practice; `--explain` prints rationale; pure (no I/O) checks against static rules (I-B-012) | Medium | **v0.11 P11** | **Complete** |
|
||||
| REQ-085 | v0.8→v1.0 migration ordering: v0.9 ships new parser + kinds + runtime + SSH-push alongside old daemon (dual-write window); `orca job run` dispatches on extension (`.md`→SSH-push, `.hcl`→old daemon); v0.10-P05 drains old daemons; v0.10-P14 converts remaining `.hcl` specs and removes daemon (I-C-001). **Most important cross-cutting idea** | High | **v0.9 P00** → v0.10 P14 | Complete |
|
||||
| REQ-086 | "No orca on server" enforcement: `orca doctor no-orca-on-server` SSHs to each peer verifying no `orca` binary in PATH, no `orca` systemd service, no `orca` process, no `/etc/orca/` directory; runs after v0.10-P05 before v0.10-P16; reuses v0.8 `proxmox` SSH session infrastructure (I-C-002). Implements grill C-13 | High | **v0.11 P14c** | **Complete** |
|
||||
| REQ-087 | Test infrastructure: hermetic 3-linux + 1-proxmox cluster pipeline: `test/integration/` with docker-compose/vagrant creating 4 containers/VMs; Go test harness SSHes to each, runs CLI, asserts end-to-end workflows (ns create → workload submit → migrate → drain); proxmox simulated via mock pct/qm; v0.8 e2e tests (bootstrapE2ESetup) are foundation (I-C-003) | Medium | **v0.11 P08** | **Complete** |
|
||||
| REQ-088 | Security-engineer + network-engineer persona reactivation: reactivate security-engineer (step-ca provisioner model, SSH-push blast radius, Traefik edge, .env.secrets crypto) and network-engineer (socket exposure R-007, Syncthing P2P ports, Traefik routing); cross-cutting review not single phase (I-C-004). Implements grill C-05 | High | **v0.9 P00** → v0.10 P16 | Complete |
|
||||
| REQ-089 | Documentation rewrite: ARCHITECTURE.md/PROJECT.md/README + AD-010 supersession: v0.9-P00 adds "v0.9 Architecture (Supersedes v0.8)" section + banners + Superseded Decisions table; v0.10-P15 rewrites README quickstart for new curl|sh + orca init + orca ns create flow (I-C-005) | Medium | **v0.9 P00** + v0.10 P15/P16 | Complete |
|
||||
| REQ-090 | Dual-write window: v0.9 `orca job run` dispatches on extension (`.md`→SSH-push new path, `.hcl`→old daemon path) via parser dispatcher (REQ-064); daemon not removed until v0.10-P05; SSH-push path writes to separate systemd unit namespace (`orca-v1-<alloc>.service`) while daemon uses `orca-<job>.service` — no unit name overlap = no conflict (I-C-006) | High | **v0.9 P00** | Complete |
|
||||
|
||||
## v0.10 Docs & Install Milestone Requirements
|
||||
|
||||
The following requirements are scoped to the v0.10 docs/cli-examples
|
||||
milestone. They cover the CLI reference documentation, jobspec
|
||||
reference, ingress guide, full-stack example jobspecs, README refresh,
|
||||
namespace.md v0.9 layout update, and the release/install pipeline fix
|
||||
that guarantees every Gitea release carries a Linux binary asset.
|
||||
|
||||
| ID | Requirement | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-091 | `docs/cli.md` comprehensive CLI reference: every command/subcommand with synopsis, flags (name/type/default/description), and one-line example; global flags (`--json`, `--system`, `--config`, `--no-deprecation-warnings`); output modes (text vs `--json`, `--watch` table vs NDJSON); exit codes; deprecated surface (`orca daemon`, `orca cert`, `orca node join` mTLS path, legacy `.hcl` jobspec) flagged with callout boxes pointing to v0.10 removal | High | **v0.10 P2** | **Complete** |
|
||||
| REQ-092 | `docs/jobspec.md` markdown frontmatter schema reference: all top-level keys, block reference (runtime, ports, env/secrets, volumes, restart, update, service, health, lifecycle, constraints, affinity, tasks), kinds matrix (Job/Service/DaemonSet required vs allowed), CEL subset grammar, body byte-exact preservation (R-015), deprecated HCL form callout | High | **v0.10 P2** | **Complete** |
|
||||
| REQ-093 | `docs/ingress.md` Traefik ingress reference: `kind: Service` implies Traefik route (D-175), R-007 socket-vs-TCP-bind semantics, generated Traefik YAML shape (routers/services/healthCheck), atomic reload (C-10), drain (`weight: 0`), TLS (certResolver, trust domain, step-ca), worked-example pointer to `examples/full-stack/`, v0.10 forward limitations (socket activation, transactional update) | High | **v0.10 P2** | **Complete** |
|
||||
| REQ-094 | `examples/full-stack/` directory with 5 valid jobspecs (`web-app.md`, `api.md`, `worker.md`, `log-shipper.md`, `postgres.md`) exercising ports/service/health/restart/update/constraints/affinity/lifecycle/task-groups/volumes/replication/DaemonSet; `rendered/` subdir showing the Traefik dynamic YAML + systemd units orca generates; `README.md` walkthrough (init → node join → capacity set → ns create → job run → list --watch → inspect rendered) | High | **v0.10 P3** | **Complete** |
|
||||
| REQ-095 | README.md refresh: status line (v0.9 complete, v0.10 in progress), install `--version` example updated to current tag, subcommand table expanded to all commands with deprecation markers, update-in-place example updated, development targets complete (`verify-reqs`, `security-scan`, `test-race`, `changelog`), new Documentation + Examples sections linking all `docs/*.md` and `examples/` | High | **v0.10 P4** | **Complete** |
|
||||
| REQ-096 | `docs/namespace.md` v0.9 multi-namespace layout update: replace v0.8 flat path table with v0.9 layout (`cluster/`, `_defaults/`, per-ns `db/jobs/alloc/ns.md`), `ORCA_HOME`/`--system` resolution, `orca ns` subcommand cross-link, v0.8 flat layout flagged deprecated | Medium | **v0.10 P4** | **Complete** |
|
||||
| REQ-097 | `scripts/release.sh` release pipeline fix: cross-build `linux-amd64` tarball regardless of host arch (`GOOS=linux GOARCH=amd64 go build`); post-create asset verification (query `/releases/tags/$VERSION`, assert the tarball in attachments, retry/fail loudly if missing). Guarantees every Gitea release carries the Linux binary asset (root cause of v0.4.5 install) | High | **v0.10 P1** | **Complete** |
|
||||
| REQ-098 | `scripts/install.sh` asset fallback walk: if the latest/pinned release lacks the matching `orca-<ver>-<os>-<arch>.tar.gz`, walk backward through `/releases?limit=20` to the most recent release that has it, with a clear warning. Keeps pulling from releases (not main). Optional `--check` dry-run mode | High | **v0.10 P1** | **Complete** |
|
||||
|
||||
## v0.11 Production Hardening Milestone Requirements
|
||||
|
||||
The following requirements (REQ-099…REQ-NN) are scoped to the v0.11
|
||||
production-hardening milestone. They cover the ingress hybrid default
|
||||
(R-017), drift detection (R-018/R-019/R-020), the systemd Path unit
|
||||
implementation (D-227…D-237), and five net-new CLI commands added per
|
||||
operator decision Q2=C.
|
||||
|
||||
### Ingress hybrid (R-017, D-215…D-226)
|
||||
|
||||
| ID | Requirement | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-099 | `internal/emitter/nft.go`: nftables emitter renders `/etc/nftables.d/orca.nft` with DNAT (`:443`→`127.0.0.1:8443`, `:80`→`127.0.0.1:8080`), SYN-flood `tcp-flags` filter, `ora_rl` rate-limit meter (default 100/s burst 200), `orca_trusted_probes` set; idempotent `nft -f` apply; atomic rule-set swap (R-017, D-217, D-218, D-222) | High | **v0.11 P15.5** | **Complete** |
|
||||
| REQ-100 | Traefik static config emitter update: `entryPoints.websecure.address` changes from `:443` to `127.0.0.1:8443` (default); `entryPoints.web.address` changes to `127.0.0.1:8080`; `--public-binding=traefik-on-public-ip` opt-out emits `:443`/`:80` instead; certs/mTLS/dynamic config unchanged (R-017, D-220, D-216) | High | **v0.11 P15.5** | **Complete** |
|
||||
| REQ-101 | `orca doctor nft`: checks `table inet orca-ingress` exists, expected DNAT rules present, rate-limit meter present, `/etc/nftables.d/orca.nft` parses cleanly (`nft -c -f`), file hash matches latest applied txn; drift detection via hash comparison (R-018 critical_paths, D-221, D-226) | High | **v0.11 P15.5** | **Complete** |
|
||||
| REQ-102 | `orca nft` CLI: `show [--peer]`, `diff --against <txn-id>`, `doctor` (alias for `orca doctor nft`), `country block add <cc-list>` (opt-in GeoIP), `rate limit set --rate N/s`; all Layer-5 orchestrators that SSH into peers and parse `nft` output (D-223, D-222) | Medium | **v0.11 P15.5** | **Complete** |
|
||||
|
||||
### Drift detection (R-018/R-019/R-020, D-227…D-237)
|
||||
|
||||
| ID | Requirement | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-103 | `internal/drift` package: `Detector` interface (`Watch`, `Aggregate`, `Remediate`, `Acknowledge`), `Event`, `Config`, `PathSpec`, `RemediationPolicy` types; `iter.Seq2[Event, error]` per D-017; `signal.NotifyContext` per D-023 (R-018, D-236) | High | **v0.11 P10** | **Complete** |
|
||||
| REQ-104 | `orca drift` CLI tree: `watch [--interval=2s] [--paths=...] [--json]`, `show [--peer]`, `acknowledge <peer> <path>`, `remediate <peer> <path> [--force]`, `config show`, `config validate`; uses `iter.Seq2` + `signal.NotifyContext` (D-236) | High | **v0.11 P10** | **Complete** |
|
||||
| REQ-105 | systemd Path unit emitter: for each critical path, emit `orca-drift-<name>.path` (`PathChanged=`, `RateLimitIntervalSec=1s`, `RateLimitBurst=5`) + `orca-drift-<name>.service` (`Type=oneshot`, `ExecStart=/usr/local/bin/orca-drift-notify.sh %f`, `User=orca`, security hardening: `NoNewPrivileges`, `ProtectSystem=strict`); R-001-clean (R-018, D-227, D-228) | High | **v0.11 P10** | **Complete** |
|
||||
| REQ-106 | `scripts/orca-drift-notify.sh`: receives changed path as `$1`, computes sha256 (or "DELETED"), writes event JSON to `/etc/orca/state/drift-events/<event-id>.json` (event_id, ts, host, path, status, new_sha256, latest_txn, triggered_by); stateless, idempotent; `flock` for serialization (D-228) | High | **v0.11 P10** | **Complete** |
|
||||
| REQ-107 | `scripts/orca-aggregate.sh` extension: existing 10s aggregator cadence (C-11) now also rsyncs each peer's `/etc/orca/state/drift-events/`, validates event hashes against `/etc/orca/state/applied/<txn>/manifest.json`, triggers `orca-remediate.sh` for auto-remediable paths, consumes (deletes) event files on peers (D-229, D-237) | High | **v0.11 P09** | **Complete** |
|
||||
| REQ-108 | `scripts/orca-remediate.sh`: re-pushes latest applied txn's per-peer render tree via rsync, runs peer-side applier; 5-min cooldown per path applies ONLY on successful remediation (transient failures retry next tick); cooldown state at `/etc/orca/state/remediation-cooldown/` (D-231, D-232 refined per CLARIFY C4) | High | **v0.11 P10** | **Complete** |
|
||||
| REQ-109 | Drift cadence config in `config.md` (`kind: ClusterConfig`): `drift.polling.{enabled,default_interval,max_concurrent_peers}`, `drift.paths.{critical,standard,excluded}` (each with `systemd_path_unit`, `interval`, `paths` list), `drift.remediate.{auto,auto_paths,require_approval_paths,notify_on_remediation}`; critical defaults: Traefik dynamic, nftables, sudoers, orca-alloc services; secrets + `/run/orca/*` + drift-events dir excluded (R-018, D-231, D-234) | High | **v0.11 P10** | **Complete** |
|
||||
| REQ-110 | Pre-flight consistency gate in applier: `orca-pull.sh` (C-09) refuses new txns if drift detected on the target peer/namespace; `--force` flag overrides; per-namespace scoping means a drifted peer in ns-A does not block ns-B (R-020, Q4=A) | High | **v0.11 P10** | **Complete** |
|
||||
| REQ-111 | `orca` system user on peers: peer-setup emits `useradd -r orca` (system account, no login shell); `orca-drift-*.service` runs as `User=orca Group=orca`; SSH key access to lead for aggregator; idempotent at peer setup (net-new operational requirement from doc 5) | High | **v0.11 P10** | **Complete** |
|
||||
| REQ-112 | NFS detection at peer setup: `orca node join` / peer-setup detects NFS mounts on orca state dirs; if `/etc/orca` is on NFS, systemd Path units are disabled for those paths and polling is the only detection; logs a warning (D-233) | Medium | **v0.11 P10** | **Complete** |
|
||||
| REQ-113 | `orca job restart <name>`: restarts an allocation to pick up EnvironmentFile drift; goes through normal allocation lifecycle (not file-level remediation); triggers on drift of `/etc/orca/allocs/<id>/env` (D-235) | Medium | **v0.11 P10** | **Complete** |
|
||||
|
||||
### Net-new CLI surface (Q2=C — all five commands added to v0.11)
|
||||
|
||||
| ID | Requirement | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-114 | `orca cluster rotate-lead`: moves cluster CA + lead state to a new bare-Linux peer (R-003 enforces bare-Linux-only lead); workloads keep running (certs already distributed); SSH key rotation; idempotent (Q2=C, folds into P14b daemon cutover) | High | **v0.11 P14b** | **Complete** |
|
||||
| REQ-115 | `orca upgrade --to-vX`: thin wrapper around `install.sh` + `orca restore` (binary upgrade only, not full cluster rolling upgrade); handles Traefik binding cutover from `:443` to `127.0.0.1:8443` for existing v0.9/v0.10 clusters (R-017 migration path, CLARIFY C1, C2=a thin wrapper); full cluster-rolling-upgrade defers to v1.x (Q2=C) | High | **v0.11 P14a** | **Complete** |
|
||||
| REQ-116 | `orca job migrate <name> --to <node>`: drain+reschedule composite (uses P05 drain + P06 alloc history); live-migrate with storage replication defers to v1.x (CLARIFY C3=a); idempotent (Q2=C) | Medium | **v0.11 P05** | **Complete** |
|
||||
| REQ-117 | `orca logs --all-nodes --since 5m`: aggregates journald logs across peers via SSH; uses P06 alloc-history cache DB; `iter.Seq` streaming per D-017; `--since` duration flag; `--all-nodes` fans out (Q2=C, folds into P06) | Medium | **v0.11 P06** | **Complete** |
|
||||
| REQ-118 | `orca doctor mTLS`: verifies trust chain (CA → server cert → workload SVIDs exist + not expired) AND live mTLS handshake probe to each peer (reuses P01 metrics endpoint + P01.5 SPIFFE spike infra); both chain verification + live probe (CLARIFY C5, Q2=C, folds into P15.5) | High | **v0.11 P15.5** | **Complete** |
|
||||
|
||||
### Scope notes
|
||||
|
||||
- REQ-099…REQ-118 = 20 net-new requirements (REQ count grows 98→118).
|
||||
- No new phases added (Q3=A folds ingress into P15.5; Q2=C folds CLI commands into existing phases).
|
||||
- P09 expands (REQ-107 aggregator extension); P10 expands (REQ-103…REQ-113, the largest phase); P15.5 expands (REQ-099…REQ-102 ingress + REQ-118 mTLS doctor).
|
||||
- P05 gains REQ-116 (migrate); P06 gains REQ-117 (logs --all-nodes); P14a gains REQ-115 (upgrade); P14b gains REQ-114 (rotate-lead).
|
||||
|
||||
## v0.12 Milestone Summary — Security Hardening (Zero-Trust Identity)
|
||||
|
||||
**Status**: complete (shipped as v0.11.x tags; milestone release v0.11.28). 30 net-new requirements (REQ-119..REQ-148)
|
||||
derived from the v0.12 threat-model review (25 findings F1..F25) and the
|
||||
zero-trust identity model (R-021). See ROADMAP.md for the 29-phase plan
|
||||
(P0 + P01..P27 + P28 final) and RESEARCH_v0.12.md for the full threat model.
|
||||
|
||||
### Wave A — Critical injection & traversal
|
||||
|
||||
| ID | Requirement | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-119 | Command injection fix in `internal/runtime/podman.go` & `wasm.go`: shell-quote `cmdStr` via `shellQuote` in SSH exec interpolation (`podman.go:57`, `wasm.go:39`); add injection regression tests (bats + Go) covering `;`, `\|`, `$()`, backticks, newline injection (F3) | High | **v0.12 P01** | complete |
|
||||
| REQ-120 | Namespace path traversal fix: `validateNamespaceName` in `internal/ns/` rejects `..`, `/`, leading `-`, null bytes, control chars in `ns create`/`ns inherit`/`ns set-constraint`; add fuzz test (F4) | High | **v0.12 P02** | complete |
|
||||
| REQ-121 | Txn apply path allowlist: `apply.sh` python heredoc validates every `path` in `desired-state.json` against a prefix allowlist (`/etc/orca/`, `/etc/traefik/orca*`, `/etc/systemd/system/orca-*`, `/etc/nftables.d/orca*`, `/etc/syncthing/orca*`); rejects otherwise; HMAC-signed manifest unchanged (F5) | High | **v0.12 P03** | complete |
|
||||
|
||||
### Wave B — Zero-trust identity
|
||||
|
||||
| ID | Requirement | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-122 | ACL enforcement wiring: `acl.Check` invoked in daemon handlers (read/write/admin by route) and SSH-push applier (validates `ORCA_OIDC_TOKEN` env var against JWKS before applying any txn); deny-by-default enforced; actor recorded in audit (F1, foundational for REQ-145) | High | **v0.12 P06** | complete |
|
||||
| REQ-123 | Daemon auth hardening: mandatory mTLS (remove plaintext mode entirely); OIDC bearer accepted as second factor on human-facing endpoints; `MaxBytesReader` body limits; pprof loopback-only by default, refuse non-loopback without `--pprof-allow-public` confirmation (F6, F24) | High | **v0.12 P09** | complete |
|
||||
| REQ-124 | HTTP request body size limits: `http.MaxBytesReader` on all JSON-decoding handlers; `MaxHeaderBytes` set; rejects oversized bodies (F24) | Medium | **v0.12 P09** | complete |
|
||||
| REQ-125 | Audit log tamper-evidence: hash-chained entries (`prev_hash = sha256(prev_row \|\| payload)`), HMAC-SHA256 under master key on the chain head; `orca doctor audit` verifies the chain; append-only enforcement via SQLite trigger blocking UPDATE/DELETE; actor field carries OIDC `sub` or SPIFFE SVID (F2) | High | **v0.12 P10** | complete |
|
||||
| REQ-126 | SVID chain validation: `VerifySVID` validates the full cert chain against the CA pool, not just the URI SAN; reject certs signed by unknown CAs even with correct URI (F9) | High | **v0.12 P11** | complete |
|
||||
| REQ-127 | Backup symlink validation: `Restore` rejects `Linkname` that's absolute, contains `..`, or points outside `ORCA_HOME`; add regression test with crafted tarball (F7) | High | **v0.12 P12** | complete |
|
||||
| REQ-128 | step-ca /tmp hardening: `step ca certificate` writes to 0600 temp under `ClusterDir()/step-tmp/` (or `TMPDIR` override), not world-readable `/tmp`; cleanup in `defer` (F10) | High | **v0.12 P13** | complete |
|
||||
| REQ-129 | Master key rotation: `orca secrets rotate-master` re-encrypts all namespace secrets under a new master key; new master key re-sealed to OIDC as part of the same operation; `--dry-run` + atomic + automatic rollback to old sealed key on any ns failure; no passphrase (R-021) (F12) | High | **v0.12 P14** | complete |
|
||||
| REQ-130 | File-mode audit expansion: `EnforceFileModes` extended to SSH key, master key (sealed blob), server cert/key, known_hosts; `orca doctor modes` checks all; startup refuses to run on violation (F13) | Medium | **v0.12 P15** | complete |
|
||||
| REQ-131 | aggregate.sh JSON injection fix + drift-gate parse fix: replace `printf` interpolation with `jq`-based JSON construction (or Go-side aggregator emitting JSON); fix `orca-pull.sh` R-020 parsing to use `jq` instead of grep (F11, F18) | High | **v0.12 P16** | complete |
|
||||
| REQ-132 | install.sh checksum+GPG verification: release.sh publishes `SHA256SUMS` + `SHA256SUMS.asc` (GPG-signed) alongside tarball; install.sh verifies before `tar -xzf`; fail closed on mismatch (F14) | High | **v0.12 P17** | complete |
|
||||
| REQ-133 | nftables ruleset hardening: add conntrack bounds (`ct state established,related accept`), input default-deny on orca chain, drop invalid packets; `orca doctor nft` audits live ruleset against emitted one (F21) | Medium | **v0.12 P18** | complete |
|
||||
| REQ-134 | sudoers hardening: add NOEXEC to `apt-get`/`dpkg` (or remove if unused); `orca doctor proxmox` audits sudoers file against expected allowlist (F22) | Medium | **v0.12 P19** | complete |
|
||||
| REQ-135 | System user consistency: Proxmox bootstrap creates `nologin` system user (`-r -s /usr/sbin/nologin`), matching peer-setup; `orca doctor` flags inconsistency on existing peers; `orca upgrade` migrates (F23) | Medium | **v0.12 P20** | complete |
|
||||
| REQ-136 | SQLite file-mode + at-rest encryption: `store.Open` sets DB file mode 0600; optional `--encrypt-db` (CGO-free fallback per C-31: file-mode 0600 + documented threat if SQLCipher needs CGO); no CGO (F8) | High | **v0.12 P21** | complete |
|
||||
| REQ-137 | Migration safety: `copyFile` -> atomic temp+rename; `migrateDBSchema` runs in transaction with `foreign_keys(ON)`; pre-migration backup step (uses `internal/backup`); document manual rollback; v0.11->v0.12 identity migration: `orca upgrade` refuses clusters using `--password`/bare-tokens without `--accept-identity-migration` (F19, C-34) | High | **v0.12 P22** | complete |
|
||||
| REQ-138 | Legacy CA/mTLS/daemon + step-ca password-provisioner deletion: remove `internal/security/ca.go` legacy CA, `internal/transport/mtls.go` deprecated path, daemon plaintext mode; migrate `orca init`/`orca cert *` to step-ca exclusively; `certpaths` (v0.8 layout) removed; delete step-ca `--password-file` provisioner (replaced by OIDC provisioner); **gate: P06/P08/P09/P11 all shipped** (F16) | High | **v0.12 P23** | complete |
|
||||
| REQ-139 | known_hosts tightening + transport hardening: `Flock` tightens pre-existing looser perms to 0600; `classifyDialErr` switched from substring to typed errors; add SSH-exec rate limiting (token bucket per peer) (F15, F25) | Medium | **v0.12 P24** | complete |
|
||||
| REQ-140 | Drift event authentication: drift events signed with per-peer HMAC key (derived from master key); aggregator rejects unsigned/forged events; `orca-drift-notify.sh` reads key from 0600 file owned by `orca` (F18) | Medium | **v0.12 P25** | complete |
|
||||
| REQ-141 | Security integration test suite: hermetic harness exercising injection, traversal, symlink, drift-forgery, audit-tamper, daemon-auth-negative, OIDC mock-IdP flow, ACL-with-OIDC-claims negative tests, unseal/seal, WebAuthn virtual-authenticator ceremony, password-removal regression (assert `--password` is rejected); gates in `.coreci.yml` `validate` (C-33) | High | **v0.12 P26** | complete |
|
||||
| REQ-142 | Zero-trust + OIDC + WebAuthn + threat-model docs: `docs/threat-model.md` (STRIDE + zero-trust model + OIDC data-flow), `docs/oidc.md` (configure your IdP, Dex offline quickstart, claim-to-namespace mapping), `docs/webauthn.md` (passkey registration, RP ID, secure context), `docs/security-runbook.md` (unseal/seal, master key rotation, incident response, sudoers audit, nft audit); README security section names "no orca credentials" as an invariant | Medium | **v0.12 P27** | complete |
|
||||
| REQ-143 | Final review + ship + audit: multi-persona review across all phases, `ciagent-audit` reconstruction test, milestone merge to main, tag `v0.11.29` (= v0.12 milestone release per feature-milestone rule) | High | **v0.12 P28** | complete |
|
||||
| REQ-144 | OIDC client + bundled Dex: `orca auth login`/`logout`/`status`/`init-idp`; OIDC config block (`oidc.issuer`, `client_id`, `client_secret`, `scopes`); bundled Dex systemd unit + Traefik route on the lead; BYO external IdP override via `oidc.issuer` repoint; JWKS caching + refresh; token storage at `~/.orca/credentials.json` (0600); `--oidc` flag on commands requiring identity; browser auth-code + PKCE + local loopback redirect; headless device-code fallback (D-238..D-247) | High | **v0.12 P04** | complete |
|
||||
| REQ-145 | ACL rewrite to OIDC claims: remove `KindToken` entirely; `KindSpiffe` stays for machine identity; new `KindOidc` maps `sub`+`groups` -> namespace permissions; `acl.Check` takes OIDC claims struct; deny-by-default enforced in daemon + SSH-push applier; `acl.json` mode tightened to 0600 (F1) | High | **v0.12 P06** | complete |
|
||||
| REQ-146 | Remove all password/token paths (breaking): delete `--password`/`$ORCA_PROXMOX_PASSWORD` from Proxmox join (replace with pre-staged-key-only or `step ssh` OIDC cert exchange); delete step-ca `--password-file` provisioner (migrate to OIDC provisioner); delete any bare-token CLI paths; documented in migration guide (R-021, C-34) | High | **v0.12 P07** | complete |
|
||||
| REQ-147 | Master key seal-to-OIDC + Shamir recovery: master key encrypted with key derived from OIDC token exchange at unseal; `orca cluster unseal`/`seal`; sealed blob at `ClusterDir()/master.key.sealed` (0600); raw key never on disk; Shamir 3-of-5 shards printed at seal time; recovery via `--recovery` + 3 shards; mTLS-only offline path derives seal key from cluster CA (D-241, C-35) | High | **v0.12 P08** | complete |
|
||||
| REQ-148 | WebAuthn connector for Dex (passkeys): `orca-webauthn-connector` (~300 LoC Go, `go-webauthn`); register/login ceremonies at `/orca/webauthn/{register,login}` behind Traefik; `orca auth register` browser flow; passkey storage SQLite `ClusterDir()/webauthn-credentials.db` (0600, public keys only); RP ID = cluster Traefik domain; secure context via step-ca cert; headless device-code fallback; virtual-authenticator integration tests (D-240, D-243, D-244, C-38) | High | **v0.12 P05** | complete |
|
||||
|
||||
### Scope notes (v0.12)
|
||||
|
||||
- REQ-119..REQ-148 = 30 net-new requirements (REQ count grows 118 -> 148).
|
||||
- 29 phases (P0 + P01..P27 + P28 final); GRILL may split/merge.
|
||||
- P04 (OIDC+Dex) and P05 (WebAuthn) are the new `feat` phases; the rest are `fix`/`chore`/`test`/`docs`/`refactor`. Milestone type = feature (at least one `feat`).
|
||||
- Tags on v0.11.x patch line: `v0.11.0` (P0) ... `v0.11.29` (P28 final = v0.12 milestone release).
|
||||
- v1.0.0 production-ready tag stays deferred for post-v0.12 UAT (per v0.11 PRD).
|
||||
|
||||
---
|
||||
|
||||
## Milestone v0.13: Production Hardening Round 2 + UAT Plan
|
||||
|
||||
**Status**: complete (2026-08-10). v0.12 (Security Hardening) is
|
||||
COMPLETE; v0.13 is the final hardening round before the v1.0.0
|
||||
production-ready tag. v1.0.0 is gated on the UAT signoff script
|
||||
(`scripts/uat-signoff.sh`) delivered by this milestone.
|
||||
|
||||
### Wave A — Toolchain & injection hardening
|
||||
|
||||
| ID | Requirement | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-149 | Go toolchain bump to 1.25.12+ (closes 24 stdlib vulns: archive/tar GO-2025-4014/GO-2026-4869, crypto/tls GO-2026-5856/GO-2025-4008, crypto/x509 GO-2026-5037/4947/4946/GO-2025-4175/4155/4013, net/http GO-2026-4918/GO-2025-4012, net/url GO-2026-4601/4341/GO-2025-4010, encoding/pem GO-2025-4009, os GO-2026-4602); `govulncheck -show verbose` triage of 6 imported third-party vulns; bump deps with reachable traces | High | **v0.13 P01** | complete |
|
||||
| REQ-150 | Input validation & injection hardening: (a) `orca logs --job` validate against `^[A-Za-z0-9_-]+$`, use `shellQuote` not `%q` (critical: backtick RCE via SSH fanout); (b) pprof `isLoopback(":6060")` treat empty host as non-loopback/bind-all, reject unless explicit public-allow flag wired; remove phantom `--pprof-allow-public` references, make loopback-only a hard invariant; (c) backup restore tar-slip fix: use `filepath.Rel(target, dest)` containment check instead of `HasPrefix(name, "..")`; (d) `orca txn rollback` validate txn ID against `^T-[0-9a-f]{16}$`; (e) `orca nft diff --against` validate txn ID before `filepath.Join`; (f) `drain stopAlloc` validate `allocID` against `^[A-Za-z0-9_-]+$` before `systemctl stop`; (g) `cluster_compat` `shellQuote(first)` for peer dir name; (h) `runtime/podman.go` use `shellQuote(image)` not `%q`; (i) nft `TrustedProbes` validate each entry with `net.ParseIP`/`net.ParseCIDR`; (j) sudoers: validate `--proxmox-user`/`--proxmox-role` against `^[a-z_][a-z0-9_-]{0,31}$`; write to fixed `/etc/sudoers.d/orca`; `shellQuote` all pveum/useradd; `validateSudoers` check the actual file written; (k) `nft country block add` validate `^[A-Z]{2}$` | Critical | **v0.13 P02** | complete |
|
||||
|
||||
### Wave B — Scheduler wiring & jobspec parser (architectural)
|
||||
|
||||
| ID | Requirement | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-151 | Scheduler/deployment wiring: wire `internal/scheduler.Schedule()` into `orca job run` — replace local `exec.CommandContext` path with: evaluate constraints/capacity/affinity via scheduler → render systemd units via `internal/emitter` → SSH-push to target via `internal/sshpush`; `--target` overrides scheduler selection; capacity enforced (reject job if no node fits); CEL constraints evaluated; affinity weighted scoring; `systemd-analyze verify` on rendered unit before deploy; `job run` without `--target` uses scheduler bin-packing across registered nodes | Critical | **v0.13 P03** | complete |
|
||||
| REQ-152 | jobspec parser fixes: add `case "schedule":` and `case "timeout":` to top-level switch in `internal/jobspec/markdown.go` (currently silently dropped); fix DaemonSet — parser must not default Count to 1 for DaemonSet (validator rejects Count!=0); DaemonSet schedule block actually parsed and stored; `timeout:` on Jobs parsed and enforced (kill after duration); `restart:` policy translated to systemd `Restart=`/`StartLimitBurst` in emitter; add `job lint` warnings for advisory-only fields (cron, health, update, affinity) with honest "not enforced in this version" message | Critical | **v0.13 P03** | complete |
|
||||
|
||||
### Wave C — Zero-trust enforcement wiring
|
||||
|
||||
| ID | Requirement | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-153 | ACL enforcement + WebAuthn registration auth: (a) wire `acl.Check` into all 5 daemon handlers (`dispatch`/`jobs`/`nodes`/`tasks`/`health`) — extract OIDC sub/SPIFFE SVID from mTLS peer cert, check against ACL for namespace+verb, deny-by-default; (b) wire `acl.Check` into sshpush applier + txn apply path (validate `ORCA_OIDC_TOKEN` bearer against JWKS); (c) thread OIDC sub/SVID into audit `actor` field (replaces "cli"/"daemon"); (d) fix `acl.json` mode 0644→0600; (e) fix WebAuthn unauthenticated registration — `/orca/webauthn/register` requires existing authenticated session or admin bootstrap token; do not allow overwriting existing credentials without re-auth; (f) add flock on `acl.json` for concurrent grant/revoke | Critical | **v0.13 P04** | complete |
|
||||
| REQ-154 | Seal/audit CLI + chain race + key zeroing: (a) implement `orca cluster seal`/`unseal` (OIDC token exchange→unwrap master key→zeroed on shutdown; Shamir 3-of-5 shards printed at seal time; sealed blob at `ClusterDir()/master.key.sealed` 0600); (b) implement `orca doctor audit` (invokes `AuditRepo.VerifyChain`); (c) implement `orca doctor modes` (invokes `EnforceFileModes` across ORCA_HOME); (d) fix audit hash-chain race — `Append` uses `BEGIN IMMEDIATE` transaction; (e) fix `secrets rotate-master` to actually re-seal to OIDC; (f) zero master key / namespace keys / SVID private keys after use (defense-in-depth against pprof heap extraction) | High | **v0.13 P05** | complete |
|
||||
| REQ-155 | auth init-idp real + auth register: (a) implement `orca auth init-idp` — render Dex systemd unit + config template + Traefik dynamic route from `internal/webauthn/` connector at `https://<cluster>/orca/webauthn/{register,login}`; RP ID = cluster Traefik domain (C-38); HTTPS secure context via step-ca cert; atomic deploy with rollback; (b) implement `orca auth register` (browser flow to WebAuthn registration endpoint); (c) `loadOIDCConfig` config-file loading (`oidc.issuer` in config, not flags-only); (d) `orca doctor oidc` health check | High | **v0.13 P06** | complete |
|
||||
|
||||
### Wave D — Concurrency, transport, migration safety
|
||||
|
||||
| ID | Requirement | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-156 | Concurrency safety: (a) SQLite `busy_timeout(5000)` + `SetMaxOpenConns(1)` on all DSNs (store, cache, recovery, webauthn); (b) secrets file flock (concurrent `secrets set` on same ns no longer loses data); (c) upgrade lock file (refuse concurrent `orca upgrade`); (d) backup lock file; (e) cache invalidation by write commands (`node join`/`leave`, `ns create`/`delete`, `job run`/`stop` invalidate relevant cache class — read-after-write consistency); (f) `Executor.Run` mutex scope fix (hold only for DB inserts, not whole job duration); (g) `ns create` atomic dir+ns.md write; (h) `writeCurrentLead` atomic write; (i) consolidate 3 divergent `writeAtomic` impls onto `security.WriteAtomic`; (j) WebAuthn session stores guarded with `sync.Mutex` | High | **v0.13 P07** | complete |
|
||||
| REQ-157 | Transport & SSH safety: (a) replace substring matching in `transport.IsTransient` AND `sshpush.isTransient` with typed sentinels (`errors.Is`); (b) `rotateSSHKeys` 2-phase atomic swap (stage new key on all peers → atomic swap → verify → cleanup old); (c) `known_hosts` flock field actually read by `dial()` (TOFU callback uses new field, not v0.8 `certpaths.KnownHostsPath()`); (d) IPv6 `net.JoinHostPort` in proxmox SSH dial + drain `splitHostPort`; (e) explicit timeouts for all SSH commands (peer-setup, drift remediate/ack, txn rollback, job restart — use `context.WithTimeout`); (f) `verifyCutover` use `security.ClientTLSConfig` with orca CA pool; (g) OIDC callback server `ReadHeaderTimeout: 5s`; (h) root SIGINT/SIGTERM handler for non-watch commands (clean SSH session + temp file cleanup) | High | **v0.13 P08** | complete |
|
||||
| REQ-158 | Migration & operational safety: (a) migration transaction + torn-write fix — `migrateDBSchema` wraps ALTER TABLE in transaction; crash after `os.Rename` but before schema fixup is recoverable; (b) `job stop` real `systemctl stop` via SSH (matches `job restart` pattern; honest semantics); (c) DB retention/compaction for `jobs`/`tasks`/`audit_log` tables (retention policy + `orca doctor db` compaction check); (d) `orca logs --lines` cap + `--since` upper bound (prevent OOM from unbounded journalctl output); (e) cache DB mode 0600 (matches `store.Open`); (f) `upgrade.go` cutover backup-file + atomic-rename (replace direct `sed -i`) | High | **v0.13 P09** | complete |
|
||||
|
||||
### Wave E — Observability, docs, UAT
|
||||
|
||||
| ID | Requirement | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-159 | Observability expansion: metrics add `orca_jobs_by_state` histogram, `orca_drift_events_total` counter, `orca_ssh_errors_total` counter, `orca_txn_apply_total`/`orca_txn_rollback_total` counters, `orca_acl_denials_total` counter, `orca_audit_chain_head` gauge; new `docs/metrics.md` with Prometheus scrape config; security headers middleware on daemon (`X-Content-Type-Options`, `X-Frame-Options`) | Medium | **v0.13 P10** | complete |
|
||||
| REQ-160 | Doc drift round 2: (a) README — update status banner (v0.12+v0.13 complete), latest tag, subcommand table (add `auth`/`nft`/`peer-setup`/`secrets rotate-master`), correct "mTLS by default" claim (SSH-push is canonical, mTLS deprecated), add missing docs to table; (b) `docs/cli.md` — complete rewrite covering all ~40 subcommands; (c) CHANGELOG regen; (d) help text fixes (`job run` HCL→markdown, `job stop` daemon→SSH-push); (e) `docs/webauthn.md` add `auth register`; (f) `docs/namespace.md` add `inherit`/`set-constraint`; (g) `docs/install.md`+`docker.md` update version refs; (h) `docs/security-runbook.md` match P05 reality; (i) fix `verify-reqs` bold-format regex (currently bypasses v0.12); (j) fix ROADMAP/REQUIREMENTS v0.12 status hygiene; (k) `docs/security-scanning.md` gosec.json; (l) `internal/proxmox/bootstrap.go` comments (password→key auth); (m) deprecate `orca status` stub; (n) `make verify-docs` target (cli.md ↔ `orca --help` consistency) | High | **v0.13 P11** | complete |
|
||||
| REQ-161 | `--type linux` SSH-join: implement `NodeKindLinux` path (reserved at `model/node.go:29`); new `internal/linux/bootstrap.go` mirroring Proxmox pattern — orca pubkey deploy → `orca` system user → drift-events dir → no PVE role; key-auth only (R-021); `orca node join --type linux --host <ip> --ssh-user root --ssh-key <path>`; `peer-setup.go` kept as documented fallback | High | **v0.13 P12** | complete |
|
||||
| REQ-162 | UAT plan: `docs/uat.md` — 3-host topology (lead Ubuntu 22.04 + pve01 Proxmox VE 8/9 + worker01 Ubuntu 22.04); step-by-step with exact commands (bootstrap→onboard Proxmox→onboard Ubuntu worker→capacity→namespace→deploy full stack→migrate between hosts→exercise every claim); claim matrix mapping ~35 feature claims to UAT steps; signoff procedure (run `scripts/uat-signoff.sh`, paste output) | Critical | **v0.13 P12** | complete |
|
||||
| REQ-163 | UAT signoff script: `scripts/uat-signoff.sh` — idempotent, `set -euo pipefail`, ~35 named assertions covering all feature claims; read + non-mutating only (doctor, list, --dry-run); exit 0 iff all pass; `scripts/uat-smoke.sh` — pure-CLI subset for CI `validate` (version, acl file mode, doctor modes, no-password grep, metrics shape); tests for both scripts | Critical | **v0.13 P12** | complete |
|
||||
|
||||
### Scope notes (v0.13)
|
||||
|
||||
- REQ-149..REQ-163 = 15 net-new requirements (REQ count grows 148 -> 163).
|
||||
- 14 phases (P0 + P01..P12 + P13 final); "no limit on phases" per operator.
|
||||
- P03 (scheduler wiring) and P12 (`--type linux` + UAT) are the `feat` phases; the rest are `fix`/`chore`/`test`/`docs`/`refactor`. Milestone type = feature (at least one `feat`).
|
||||
- Tags on v0.12.x patch line: `v0.12.0` (P0) ... `v0.12.13` (P13 final = v0.13 milestone release).
|
||||
- v1.0.0 production-ready tag stays deferred for post-v0.13 UAT signoff (operator runs `scripts/uat-signoff.sh`, paste output back).
|
||||
|
||||
### Accepted residual risks (documented in threat-model, not fixed)
|
||||
|
||||
- OIDC tokens plaintext at rest (0600) — sealing on every CLI invocation conflicts with "no orca binary on servers" model
|
||||
- HSTS on daemon — mTLS-only API, no browser-facing surface on daemon itself
|
||||
- DNS resolution timeout — bounded by `net.Dialer{Timeout: 15s}`
|
||||
- Temp file cleanup on SIGKILL — orphaned temp files, operator-visible, low impact
|
||||
- Flock timeout on NFS — stuck holder is rare; `tryFlockEx` exists if needed later
|
||||
- "WASM-first" pillar aspirational — document as "WASM runtime available, process is default"
|
||||
- arm64/armv7 release — D-193 deferred; install.sh detection is forward-looking
|
||||
- OIDC callback slowloris — loopback, short-lived, single CLI invocation
|
||||
|
||||
## Milestone v0.14: Ingress Bootstrap Completeness
|
||||
|
||||
**Scope**: ensure that linux & proxmox types are properly bootstrapped with
|
||||
traefik during cluster init or node join. All cluster endpoints are
|
||||
provisioned as sockets (R-007); routing between jobs and services depends on
|
||||
traefik being present on the host and properly configured. The v0.13 traefik
|
||||
deployment shipped only a binary + systemd unit + empty dynamic dir — it
|
||||
never wrote the static config nor applied nft rules, so `orca-traefik.service`
|
||||
fails to start on a fresh `orca init` and `orca doctor nft` FAILs. v0.14
|
||||
replaces the binary+systemd model with a **podman container** running a custom
|
||||
`orca-traefik` image, and completes the nft SNAT+DNAT ingress stack on every
|
||||
node type.
|
||||
|
||||
**New load-bearing rule**:
|
||||
|
||||
- **R-024** — Traefik runs exclusively as a podman container, deployed from
|
||||
the `orca-traefik` image published per release. Every orca-managed ingress
|
||||
surface bootstraps: nft DNAT (`:443→127.0.0.1:8443`,
|
||||
`:80→127.0.0.1:8080`) + SNAT/MASQUERADE postrouting + `podman run -d
|
||||
--restart=unless-stopped --network host -v /etc/traefik/traefik.yml:/etc/traefik/traefik.yml:ro
|
||||
-v /etc/traefik/dynamic:/etc/traefik/dynamic:ro
|
||||
-v /etc/orca/step-ca-root.crt:/etc/orca/step-ca-root.crt:ro
|
||||
git.cloudinit.dev/coreci/orca-traefik:<tag>`. No node joins without a
|
||||
functional podman-traefik ingress data plane. The image's baked static
|
||||
config is a default; host-side `traefik.yml` mounted `:ro` overrides it
|
||||
(preserves `traefik-on-public-ip` opt-out, REQ-100).
|
||||
|
||||
**Three topologies** (per operator constraints):
|
||||
|
||||
- **Linux**: host → nft → `podman run orca-traefik` (host network)
|
||||
- **Proxmox Native**: host → nft → LXC (nesting=1) → `podman run orca-traefik`
|
||||
- **Proxmox Floating-IP**: LXC (owns floating IP) → nft (inside LXC) →
|
||||
`podman run orca-traefik`
|
||||
|
||||
| ID | Requirement | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-171 | `Dockerfile.traefik` + release pipeline: build + publish `git.cloudinit.dev/coreci/orca-traefik:<version>` alongside the orca image per release; `.coreci.yml` `container-publish-traefik` step; image bakes default `traefik.yml` (entrypoints `websecure` 127.0.0.1:8443, `web` 127.0.0.1:8080, `traefik` 127.0.0.1:8081 + file provider watching `/etc/traefik/dynamic` + json log/accessLog); host-side `/etc/traefik/traefik.yml` mounted `:ro` overrides baked config (preserves `traefik-on-public-ip` opt-out REQ-100); no `certificatesResolvers` (traefik v3.3 only supports acme/tailscale); `tls: {}` in dynamic config for v0.14 (real mTLS deferred to v0.15) | Critical | **v0.14 P1** | complete |
|
||||
| REQ-172 | Replace `internal/traefik/install.go` binary+systemd install with a podman-container reconciler: `podman pull orca-traefik:<tag>` + `podman run -d --restart=unless-stopped --network host --name orca-traefik -v /etc/traefik/traefik.yml:/etc/traefik/traefik.yml:ro -v /etc/traefik/dynamic:/etc/traefik/dynamic:ro -v /etc/orca/step-ca-root.crt:/etc/orca/step-ca-root.crt:ro <image>`; idempotent (pull+run if absent, start if stopped); install podman if absent (C-50); v0.13→v0.14 upgrade: detect+stop+disable+remove legacy `orca-traefik.service` + `/usr/local/bin/traefik` (C-57); works locally + over SSH-push; enable `podman-restart.service`; remove systemd unit generation | Critical | **v0.14 P2** | complete |
|
||||
| REQ-173 | nft SNAT+DNAT ruleset render+apply: extend `internal/emitter/nft.go` with postrouting masquerade chain; new `internal/ingress/bootstrap.go` renders `orca.nft` + applies `nft -f` + ensures `/etc/traefik/dynamic` dir + pushes step-ca root CA + invokes podman traefik reconciler; wired into `orca init` (localhost lead) | Critical | **v0.14 P3** | complete |
|
||||
| REQ-174 | Remote ingress bootstrap via SSH-push for `orca node join --type linux`: push step-ca root CA, render+apply nft remotely, invoke podman traefik reconciler remotely; register node as `linux` | Critical | **v0.14 P4** | complete |
|
||||
| REQ-175 | Proxmox native ingress mode (`--ingress-mode native`, default): on PVE host, render+apply nft (vmbr-compatible, separate `orca-ingress` table avoids pve-firewall conflict); create unprivileged LXC with `--features nesting=1,keyctl=1` running podman+orca-traefik; push step-ca root CA into LXC; nft DNAT target = LXC bridge IP; register PVE host as `proxmox` node; add `IngressMode` field to `model.Node` + schema migration | Critical | **v0.14 P5** | complete |
|
||||
| REQ-176 | Proxmox floating-IP mode (`--ingress-mode floating-ip --floating-ip --gateway --mac [--net-prefix]`): `pct create` Ubuntu LXC named `ingress` with `net0 bridge=vmbr0,hwaddr=<mac>,ip=<floating-ip>/<prefix>,gw=<gateway> --features nesting=1,keyctl=1 --onboot 1`; install podman + run orca-traefik inside LXC; apply nft DNAT+SNAT inside LXC; register LXC as managed `linux` node (name=`ingress`, addr=`<floating-ip>:8443`); interactive prompt for params when flags absent + not `--json`; validate IP/MAC/gateway; PVE host also registered as `proxmox` for workload dispatch | Critical | **v0.14 P6** | complete |
|
||||
| REQ-177 | `orca doctor ingress [--peer]`: verify orca-traefik container running (`podman inspect`), nft DNAT+SNAT applied, `/etc/traefik/dynamic` exists, step-ca root CA mounted; extend `scripts/uat-signoff.sh` with ingress assertions (40 podman_traefik, 41 nft_dnat_snat, 42 linux_worker, 43 proxmox_native_lxc / floating_ip_lxc) | High | **v0.14 P7** | complete |
|
||||
| REQ-178 | Docs: `docs/cli.md` (`--ingress-mode` + floating-IP flags + `doctor ingress`), `docs/uat.md` (native + floating-IP topologies), `docs/ingress.md` (podman-traefik image + volume mounts + certResolver), `docs/docker.md` (orca-traefik image), `ARCHITECTURE.md` (R-024 + ingress bootstrap section) | High | **v0.14 P7** | complete |
|
||||
| REQ-179 | Integration tests: hermetic harness fakes SSH; asserts init→podman traefik running + nft applied; linux join→remote podman+nft; proxmox native→LXC created with nesting + podman traefik; floating-ip→`pct create` with correct net0 args + LXC registered as linux node; release.sh builds orca-traefik image (Dockerfile.traefik parses) | Critical | **v0.14 P7** | complete |
|
||||
|
||||
### Scope notes (v0.14)
|
||||
|
||||
- REQ-171..REQ-179 = 9 net-new requirements (REQ count grows 163 -> 172).
|
||||
- 9 phases (P0 + P1..P7 + P8 final); feature milestone (multiple `feat` phases).
|
||||
- Tags on v0.13.x patch line: `v0.13.0` (P0) ... `v0.13.8` (P8 final = v0.14 milestone release).
|
||||
- Milestone branch: `milestone/v0.14-ingress-bootstrap`.
|
||||
|
||||
@@ -1,140 +0,0 @@
|
||||
# Research: v0.10 Docs & Install Milestone
|
||||
|
||||
## Documentation landscape in the orca tree
|
||||
|
||||
### What exists today
|
||||
|
||||
The `docs/` directory contains four files:
|
||||
|
||||
- `docs/install.md` — install guide (user-level, system-level, version
|
||||
pinning, in-place update, troubleshooting). Accurate for v0.5-v0.8
|
||||
but does not mention the v0.9 multi-namespace layout, `--config`, or
|
||||
`--no-deprecation-warnings`.
|
||||
- `docs/docker.md` — Docker image guide. Still documents `orca daemon`
|
||||
(deprecated in v0.9).
|
||||
- `docs/namespace.md` — namespace and paths. Documents the **v0.8 flat
|
||||
layout** (`~/.orca/orca.db`, `ca.crt`, `ca.key`, `server.crt`,
|
||||
`server.key`). Does NOT document the v0.9 multi-namespace layout
|
||||
(`cluster/`, `_defaults/`, per-ns `db/jobs/alloc/ns.md`), `orca ns`
|
||||
subcommands, or the `_defaults` implicit root (D-159/D-185/D-187).
|
||||
- `docs/security-scanning.md` — gosec + govulncheck + gitleaks guide.
|
||||
Accurate; no v0.9 drift.
|
||||
|
||||
### What's missing (the gap this milestone closes)
|
||||
|
||||
1. **No CLI reference doc.** The entire CLI command surface (init, job,
|
||||
node, ns, cert, daemon, doctor, status, audit, version) is
|
||||
undocumented in `docs/`. The README subcommand table is stale (lists
|
||||
only version/init/status/node/job with fake "Phase N" statuses,
|
||||
missing cert/daemon/doctor/audit/ns/node-capacity/node-key-reset).
|
||||
2. **No jobspec reference doc.** The markdown frontmatter schema (kinds,
|
||||
blocks, CEL subset, validation rules, body semantics) is
|
||||
undocumented. Operators must read `internal/jobspec/markdown.go` and
|
||||
`internal/spec/schema/schema.go` source.
|
||||
3. **No ingress/Traefik doc.** The service→Traefik mapping, R-007
|
||||
socket-vs-TCP-bind, atomic reload, drain, TLS — all undocumented.
|
||||
4. **No examples directory.** `testdata/` holds legacy HCL fixtures
|
||||
(`hello.hcl`, `fail.hcl`) for Go tests, not operator-facing
|
||||
examples. No worked full-stack demo exists.
|
||||
5. **README is stale.** Status line says "v0.1: Foundation".
|
||||
Subcommand table missing 5 commands. Install `--version` example
|
||||
pins v0.4.2. Update-in-place example references v0.4.1→v0.4.2.
|
||||
Development section omits 4 make targets.
|
||||
|
||||
### Prior art for CLI reference docs
|
||||
|
||||
- **Nomad**: `nomad job` / `nomad node` / `nomad agent` reference pages,
|
||||
one per subcommand, with flag tables and JSON examples. Orca's
|
||||
single-file `docs/cli.md` is simpler (one file vs a subdirectory) but
|
||||
follows the same flag-table + example convention.
|
||||
- **kubectl**: `kubectl reference` + per-command pages. Too heavy for
|
||||
orca; the single-file model fits the minimalist ethos.
|
||||
- **Docker CLI**: `docker run` reference with flag tables. Matches the
|
||||
shape orca's `docs/cli.md` will take.
|
||||
|
||||
### Prior art for example jobspecs
|
||||
|
||||
- **Nomad example jobs**: `nomad-job-spec.example` files in the Nomad
|
||||
repo showing service + job + sysbatch patterns. Orca's
|
||||
`examples/full-stack/` mirrors this with 5 markdown jobspecs covering
|
||||
Service/Job/DaemonSet + task groups + volumes + replication.
|
||||
- **Kubernetes examples**: `examples/` directory with yaml
|
||||
deployments/services/ingress. Orca's equivalent is the 5 jobspecs +
|
||||
rendered Traefik/systemd artifacts.
|
||||
|
||||
## Release/install pipeline research
|
||||
|
||||
### Root cause of the v0.4.5 install
|
||||
|
||||
Verified via the Gitea API:
|
||||
|
||||
```
|
||||
GET /api/v1/repos/coreci/orca/releases/latest
|
||||
→ tag_name: "v0.8.15"
|
||||
|
||||
GET /api/v1/repos/coreci/orca/releases/tags/v0.8.15
|
||||
→ attachments: [] (zero binary assets)
|
||||
```
|
||||
|
||||
The v0.8.x releases (v0.8.0 through v0.8.15) all shipped with **zero
|
||||
binary assets attached**. Only `v0.4.5` carries a tarball
|
||||
(`orca-v0.4.5-linux-amd64.tar.gz`).
|
||||
|
||||
`scripts/install.sh:70-78` resolves "latest" → v0.8.15, then
|
||||
`install.sh:96-104` looks for `orca-v0.8.15-linux-amd64.tar.gz` in
|
||||
v0.8.15's assets. Since the asset is missing, install.sh errors out
|
||||
(`could not find asset ... in release v0.8.15`). The v0.4.5 install
|
||||
came from an earlier run or a pinned `--version`.
|
||||
|
||||
### Why v0.8.x releases have no assets
|
||||
|
||||
`scripts/release.sh:132-136` calls `tea releases create "$VERSION" ...
|
||||
--asset "$TARBALL"`. The script builds the tarball (line 98) and passes
|
||||
it to `tea`. Two likely failure modes:
|
||||
|
||||
1. **Host arch mismatch**: `release.sh:89-95` builds for the host arch
|
||||
(`uname -m`). If the CI runner or dev machine is arm64, it produces
|
||||
`orca-v0.8.15-linux-arm64.tar.gz`, but `install.sh` looks for
|
||||
`linux-amd64`. The `.coreci.yml:121` release step hardcodes
|
||||
`--asset orca-${VERSION}-linux-amd64.tar.gz`, so the CI runner must
|
||||
be amd64 — but `release.sh` run locally on an arm64 dev machine
|
||||
produces the wrong arch.
|
||||
2. **Silent asset drop**: `tea releases create` has been observed to
|
||||
succeed (exit 0) without attaching the asset in some tea versions.
|
||||
The script treats `tea`'s exit code as success without verifying the
|
||||
asset actually appears in the release.
|
||||
|
||||
### Fix approach (REQ-097, REQ-098)
|
||||
|
||||
**release.sh**:
|
||||
- Cross-build `linux-amd64` explicitly via
|
||||
`GOOS=linux GOARCH=amd64 go build`, regardless of host arch.
|
||||
- After `tea releases create`, query
|
||||
`/api/v1/repos/$OWNER/$REPO/releases/tags/$VERSION` and assert the
|
||||
tarball appears in `attachments`. If not, retry once, then fail
|
||||
loudly with a clear error.
|
||||
|
||||
**install.sh**:
|
||||
- Add an asset fallback walk: if the resolved release (latest or
|
||||
pinned) lacks the matching tarball, query
|
||||
`/releases?limit=20`, walk backward, and use the most recent release
|
||||
that carries the `orca-<ver>-<os>-<arch>.tar.gz` asset. Print a
|
||||
clear warning.
|
||||
- Add `--check` dry-run mode (D-194) that prints the version + asset URL
|
||||
+ install path without writing.
|
||||
|
||||
## Persona assessment (PERSONAS.md)
|
||||
|
||||
This milestone touches two territories:
|
||||
|
||||
1. **`scripts/` (release.sh, install.sh)** — bash scripts, not Go.
|
||||
Backend-engineer territory (API-adjacent tooling). The fix is
|
||||
cross-build + API verification + fallback walk.
|
||||
2. **`docs/` + `examples/` + `README.md`** — markdown documentation.
|
||||
Lead-developer territory (coordination + cross-cutting docs).
|
||||
|
||||
No data-engineer work (no schema/migration changes). No
|
||||
frontend-engineer work (no UI). The data-engineer persona is
|
||||
deactivated for this milestone. A docs-engineer custom persona is
|
||||
created for P2/P3/P4 (markdown authoring with codebase-grounded
|
||||
factual claims).
|
||||
@@ -1,164 +0,0 @@
|
||||
# Research: v0.11 Production Hardening
|
||||
|
||||
## Source material
|
||||
|
||||
Five research documents were ingested 2026-08-07 as directional input
|
||||
(not verbatim) for v0.11 Phase 0. The current ciagent files
|
||||
(R-001…R-016, D-001…D-206) are authoritative and take precedence; where
|
||||
research conflicted, the ciagent files won. The research drove the
|
||||
adoption of R-017…R-020 and D-215…D-237 (see PROJECT.md, PRD_v0.11.md).
|
||||
|
||||
| Doc | Theme | Adopted as |
|
||||
|-----|-------|------------|
|
||||
| 1 | Ingress hybrid (nft DNAT → Traefik on 127.0.0.1:8443) | R-017, D-215..D-226, REQ-099..REQ-102 |
|
||||
| 2 | Platform-engineer playbook (8 differentiators, TCO, honest trade-offs) | README positioning (Q5=A), CLI surface gap analysis |
|
||||
| 3 | Strategic positioning ("be Proxmox-for-bare-metal, not K8s-without-K8s") | README framing (Q5=A: Nomad-inspired, honest trade-offs table from doc 3, not Proxmox-first lead) |
|
||||
| 4 | Drift detection cadence (R-018/R-019/R-020, tiered cadence, hard gate) | R-018, R-019, R-020, D-227..D-237, REQ-103..REQ-113 |
|
||||
| 5 | Drift detection concrete impl (systemd Path units, orca-drift-notify.sh, orca-remediate.sh) | D-227..D-237 detail, REQ-103..REQ-113 |
|
||||
|
||||
## Thread A — Ingress hardening (doc 1)
|
||||
|
||||
### What changes vs v0.9/v0.10
|
||||
|
||||
Traefik static config gains `entryPoints.websecure.address: 127.0.0.1:8443`
|
||||
(default) instead of `:443`. A new nftables emitter renders
|
||||
`/etc/nftables.d/orca.nft` with DNAT rules. Certs, mTLS, dynamic config,
|
||||
and the workload SPIFFE validation path are **completely unchanged**.
|
||||
Only the `address` line shifts + one new emitter + `orca doctor nft` +
|
||||
`orca nft ...` CLI.
|
||||
|
||||
### Defense in depth
|
||||
|
||||
Two layers: kernel (nftables: SYN flood, rate limit, GeoIP, conntrack)
|
||||
and application (Traefik: mTLS, SNI, ACL, dynamic routing, health
|
||||
checks). Neither can replace the other; they catch different attack
|
||||
classes.
|
||||
|
||||
### Codebase reality (verified 2026-08-07)
|
||||
|
||||
- `internal/emitter/traefik.go` + `traefik_atomic.go` exist (v0.9 P02).
|
||||
The static-config emitter is where the `address:` line change lands.
|
||||
- `internal/emitter/systemd.go` exists. New `.path`/`.service` unit
|
||||
types extend this emitter pattern (shared with drift detection, doc 5).
|
||||
- `internal/emitter/nft.go` does **not** exist — greenfield, ~200 LoC.
|
||||
- `scripts/` has `orca-verify-render.sh` but **not** `orca-aggregate.sh`,
|
||||
`orca-pull.sh`, `orca-apply-render.sh`, `orca-remediate.sh` — all are
|
||||
v0.11 P09/P10 scope.
|
||||
|
||||
## Thread B — Drift detection + transactional plane (docs 4 + 5)
|
||||
|
||||
### Architecture
|
||||
|
||||
systemd Path units (R-001-clean; systemd is OS, not Orca) watch critical
|
||||
paths via inotify. On change, a oneshot service computes sha256 and
|
||||
writes an event JSON to `/etc/orca/state/drift-events/`. The lead's
|
||||
aggregator timer (10s, C-11) rsyncs these events, validates against the
|
||||
applied txn manifest, and triggers remediation for auto-remediable
|
||||
paths.
|
||||
|
||||
### Tiered cadence
|
||||
|
||||
| Tier | Detection | Auto-remediate | Latency |
|
||||
|------|-----------|----------------|--------|
|
||||
| Critical | Path unit + 5s polling backstop | yes (config files only) | ~10s |
|
||||
| Standard | 30s polling | optional (systemd units: require approval) | 30s |
|
||||
| Default | 60s polling | no | 60s |
|
||||
|
||||
### R-020 hard gate
|
||||
|
||||
Applier refuses new txns if pre-flight consistency check fails. Override:
|
||||
`--force` flag + per-namespace scoping (Q4=A) — a drifted peer in ns-A
|
||||
does not block ns-B.
|
||||
|
||||
### Codebase reality (verified 2026-08-07)
|
||||
|
||||
- `internal/store/node_repo.go:80` and `internal/store/job_task_repo.go:82`
|
||||
already use `iter.Seq[T]`. Doc 5's `iter.Seq2[Event, error]` is the
|
||||
natural extension per D-017 (settled, shipped v0.3).
|
||||
- `internal/paths/paths.go:86` has `TxnDir()` — the txn staging dir the
|
||||
drift detector hooks into.
|
||||
- `internal/emit/contract.go` has the Go↔bash render-contract anti-drift
|
||||
(C-16). The *runtime* drift detector (doc 5) is net-new.
|
||||
- `internal/drift/` package does **not** exist — greenfield, ~500 LoC.
|
||||
- No `orca` system user creation in code — net-new operational
|
||||
requirement (REQ-111).
|
||||
- No NFS detection at peer setup — net-new (REQ-112, D-233).
|
||||
- `doctor.go` has an OS-drift *check* (one-shot, on-demand) but **not**
|
||||
a 60s runtime drift-polling loop. Doc 5's design is net-new scope.
|
||||
|
||||
### Alignment with existing gates
|
||||
|
||||
- **C-09** (`orca-pull.sh` failure contract) — R-020 refines
|
||||
"deterministic state" into an explicit refusal contract.
|
||||
- **C-11** (lead-side watchdog meta-timer) — doc 5's aggregator
|
||||
extension is the input C-11 monitors.
|
||||
- **REQ-075** (lead applier execution model) — doc 5's `orca-remediate.sh`
|
||||
is literally the same code path as a normal txn-apply, triggered by
|
||||
drift instead of a new submission.
|
||||
|
||||
## Thread C — Positioning/messaging (docs 2 + 3)
|
||||
|
||||
### Consistent with locked vision
|
||||
|
||||
The vision is *"A minimalist, offline-first, CLI-first orchestration
|
||||
engine inspired by HashiCorp Nomad"* — explicitly Nomad-inspired, not
|
||||
K8s. Doc 3's recommendation ("be Proxmox-for-bare-metal, not
|
||||
K8s-without-the-complexity") is consistent with the locked vision.
|
||||
|
||||
### Where doc 3 diverges (resolved per Q5=A)
|
||||
|
||||
Doc 3 recommends "leading with Proxmox positioning." But R-003 says
|
||||
"Proxmox can never be lead." Leading the *project identity* with a node
|
||||
type that can't be the lead is subtly contradictory. **Q5=A decision**:
|
||||
README uses the Nomad-inspired, OS-as-cluster framing (locked vision),
|
||||
mentions Proxmox as one node type, and incorporates doc 3's "honest
|
||||
trade-offs" table but not its Proxmox-first lead-positioning advice.
|
||||
|
||||
### CLI surface gap analysis (doc 2)
|
||||
|
||||
Doc 2's playbook cites ~10 CLI commands. Verified against the live
|
||||
codebase (`internal/cli/*.go`):
|
||||
|
||||
**Exist today**: `orca init`, `orca node {join,leave,list,key-reset,
|
||||
capacity}`, `orca job {run,list,stop,logs}`, `orca ns {list,create,
|
||||
delete,inspect,validate}`, `orca cert {ca-init,gen,show,renew,
|
||||
fingerprint}`, `orca doctor {cert,network,db,os,proxmox}`, `orca audit
|
||||
list`, `orca status`, `orca version`, `orca daemon` (deprecated).
|
||||
|
||||
**Not in v0.11 ROADMAP, added per Q2=C**: `orca cluster rotate-lead`
|
||||
(REQ-114, P14b), `orca upgrade` (REQ-115, P14a), `orca job migrate`
|
||||
(REQ-116, P05), `orca logs --all-nodes --since` (REQ-117, P06),
|
||||
`orca doctor mTLS` (REQ-118, P15.5).
|
||||
|
||||
**Already in v0.11 ROADMAP**: `orca node drain` (P05), `orca job lint`
|
||||
(P11), `orca job verify` (P12), `orca restore` (P07), `orca backup`
|
||||
(P04).
|
||||
|
||||
### Unverified performance claims in doc 3
|
||||
|
||||
Doc 3's "10s applier timer = 10,000x slower than K8s informers" and
|
||||
"60s drift polling" are **forward-looking design constraints**, not
|
||||
current-state limitations — no applier timer or drift-polling loop
|
||||
exists in the codebase. These are answered by R-018/R-019/R-020
|
||||
(doc 4 + doc 5): the drift detector is a backstop, not the primary
|
||||
detector, and critical paths get ~10s latency via systemd Path units.
|
||||
|
||||
## Persona assessment
|
||||
|
||||
v0.11 touches these territories:
|
||||
|
||||
| Territory | Persona | Phases |
|
||||
|-----------|---------|--------|
|
||||
| `internal/cli/**`, `internal/drift/**`, `internal/nft/**` | backend-engineer | P10, P15.5, P05, P06, P14a, P14b |
|
||||
| `internal/emitter/**`, `internal/sshpush/**` | backend-engineer + lead-developer | P09, P10, P15.5 |
|
||||
| `internal/store/**`, migrations | data-engineer | P14a (data migration) |
|
||||
| `scripts/orca-*.sh` | backend-engineer (bash tooling) | P09, P10 |
|
||||
| `docs/**`, `README.md`, `examples/**` | lead-developer + docs-engineer (phase-specific) | P15, P08 |
|
||||
| Threat model, security review, mTLS, secrets | security-engineer | P03, P15.5 |
|
||||
| nftables, Traefik binding, cluster mesh | network-engineer | P15.5, P09 |
|
||||
| Test coverage, integration harness | devops-engineer (phase-specific) | P08 |
|
||||
|
||||
No frontend-engineer work (no UI). The data-engineer persona is
|
||||
reactivated for P14a (v0.8→v1.0 data migration). A docs-engineer custom
|
||||
persona is created for P15 (README) and P08 (integration test docs).
|
||||
See `PERSONAS.md` for the updated roster.
|
||||
@@ -1,231 +0,0 @@
|
||||
# Research: v0.12 Security Hardening (Zero-Trust Identity)
|
||||
|
||||
## Source material
|
||||
|
||||
The v0.12 threat model was produced by a comprehensive security-surface
|
||||
review (Phase 0 RESEARCH, 2026-08-07) covering the entire Orca codebase
|
||||
AND the operating-system-level surface it touches. The review ingested:
|
||||
|
||||
- v0.11 closeout (CHECKPOINT.json: milestone_complete=true, 24 phases
|
||||
shipped, threat model produced in P15.5).
|
||||
- The 12-area security-surface inventory (see "Threat model findings"
|
||||
below), produced by deep code exploration of every `internal/` package,
|
||||
every `scripts/` file, the emitter surface, the OS-touching CLI
|
||||
commands, and the dual-write window.
|
||||
- The operator's locked decisions (D-238..D-247) on zero-trust identity:
|
||||
bundled Dex + WebAuthn, master key seal-to-OIDC + Shamir, no Orca
|
||||
credentials (R-021).
|
||||
|
||||
## Load-bearing rule adopted
|
||||
|
||||
**R-021**: *Orca never issues, stores, or accepts human-identity
|
||||
credentials. Human identity is exclusively external (OIDC). Machine
|
||||
identity is exclusively mTLS/SPIFFE. No passwords, no Orca-issued
|
||||
tokens, no CA-key passphrases.*
|
||||
|
||||
## Threat model findings (F1..F25)
|
||||
|
||||
| # | Area | Finding | Severity | Phase | REQ |
|
||||
|---|------|---------|----------|-------|-----|
|
||||
| F1 | ACL | `acl.ACL.Check` exists but no caller enforces it -- daemon & SSH-push have zero authz | Critical | P06 | REQ-145 |
|
||||
| F2 | Audit | Audit log is plain SQLite INSERT -- no hash chain, no MAC, not tamper-evident | Critical | P10 | REQ-125 |
|
||||
| F3 | Runtime | `podman.go:57` & `wasm.go:39` interpolate cmdStr unquoted into SSH exec -> command injection | Critical | P01 | REQ-119 |
|
||||
| F4 | Namespace | `ns create` doesn't reject `..`/`/` -> path traversal | Critical | P02 | REQ-120 |
|
||||
| F5 | Txn | `apply.sh` python heredoc writes to arbitrary paths from desired-state.json -- no allowlist | Critical | P03 | REQ-121 |
|
||||
| F6 | Daemon | Plaintext mode (default) has no auth on read endpoints; `--pprof` unauthenticated | High | P09 | REQ-123/124 |
|
||||
| F7 | Backup | `Restore` creates symlinks without validating Linkname -> symlink-to-/etc/shadow | High | P12 | REQ-127 |
|
||||
| F8 | SQLite | DBs unencrypted, no explicit file mode (defaults to umask 0644) | High | P21 | REQ-136 |
|
||||
| F9 | SPIFFE | `VerifySVID` checks URI SAN but not the cert chain against the CA | High | P11 | REQ-126 |
|
||||
| F10 | step-ca | `step ca certificate` writes SVID privkey to /tmp/orca-* world-readable | High | P13 | REQ-128 |
|
||||
| F11 | Scripts | `orca-aggregate.sh:64` interpolates raw peer output into JSON -> JSON injection | High | P16 | REQ-131 |
|
||||
| F12 | Secrets | No master.key rotation; no passphrase/KDF wrapping (raw 32 bytes, 0600-only) | High | P14 | REQ-129 |
|
||||
| F13 | File modes | `EnforceFileModes` only checks ca.{crt,key} -- SSH key, master key, server cert not re-verified | Medium | P15 | REQ-130 |
|
||||
| F14 | install.sh | curl|bash with no checksum/signature verification of the tarball | High | P17 | REQ-132 |
|
||||
| F15 | known_hosts | `Flock` creates 0600 if missing but doesn't tighten pre-existing looser perms | Medium | P24 | REQ-139 |
|
||||
| F16 | Dual-write | Legacy CA/mTLS/daemon marked Deprecated but still load-bearing -- expanded attack surface | Medium | P23 | REQ-138 |
|
||||
| F17 | History | Real GITEA_TOKEN committed in 0cba1aa, still in git history | High (human-gated) | P28 (gate) | -- |
|
||||
| F18 | Drift | `orca-pull.sh` R-020 grep-based JSON parsing fragile; drift events unauthenticated | Medium | P16/P25 | REQ-131/140 |
|
||||
| F19 | Migration | `ALTER TABLE DROP COLUMN` irreversible; `copyFile` non-atomic; no rollback | Medium | P22 | REQ-137 |
|
||||
| F20 | OS scripts | `orca-aggregate.sh`/`orca-remediate.sh` run as root with TOFU SSH (accept-new) | Medium | P16/P24 | REQ-131/139 |
|
||||
| F21 | nftables | Emitted ruleset has SYN-flood + rate-limit but no conntrack bounds, no input default-deny | Medium | P18 | REQ-133 |
|
||||
| F22 | sudoers | `OrcaOperator` sudoers has NOEXEC on pct/qm but allows apt-get/dpkg without NOEXEC | Medium | P19 | REQ-134 |
|
||||
| F23 | system user | Proxmox creates login user (-m -s /bin/bash); peer-setup creates nologin -- inconsistent privilege | Medium | P20 | REQ-135 |
|
||||
| F24 | Dispatch | No request body size limits (json.Decode with no MaxBytesReader) | Low | P09 | REQ-124 |
|
||||
| F25 | Transport | `classifyDialErr` is substring-based; no SSH-exec rate limiting | Low | P24 | REQ-139 |
|
||||
|
||||
## Zero-trust identity model (NEW in v0.12)
|
||||
|
||||
### Two identity layers, zero overlap
|
||||
|
||||
- **Human operators** -> OIDC (external IdP, BYO) OR the bundled Dex
|
||||
with a WebAuthn (passkeys) connector as the default password-free
|
||||
authenticator. `orca auth login` / `orca auth register` open the
|
||||
default browser to the Dex WebAuthn endpoint via OIDC
|
||||
authorization-code + PKCE + local loopback redirect. After the
|
||||
WebAuthn ceremony (biometric/security key), Dex redirects back with
|
||||
an auth code; CLI exchanges for a short-lived ID token (1h) +
|
||||
refresh. Headless/CI fallback: device-code flow.
|
||||
- **Machine-to-machine** -> mTLS + SPIFFE SVIDs (unchanged from v0.11).
|
||||
|
||||
### Why WebAuthn satisfies "no passwords anywhere"
|
||||
|
||||
Passkeys are **public-key credentials**. The private key is generated
|
||||
on the authenticator (TPM/security key/phone Secure Enclave) and never
|
||||
leaves it. The server (Dex) stores only the **public key** + credential
|
||||
ID + sign count. There is no password, no shared secret, no replayable
|
||||
credential. This is the strongest authentication primitive available
|
||||
and directly satisfies R-021.
|
||||
|
||||
### Bundled Dex architecture
|
||||
|
||||
- **Dex** (github.com/dexidp/dex) is the OIDC frontend. Orca bundles a
|
||||
Dex binary + config template, deployed via `orca auth init-idp` as a
|
||||
systemd unit on the lead, fronted by Traefik (R-017, step-ca cert).
|
||||
- **`orca-webauthn-connector`** is a custom Dex connector (~300 LoC Go,
|
||||
using `github.com/go-webauthn/webauthn`). It serves:
|
||||
- `GET /orca/webauthn/register` -- registration HTML/JS page.
|
||||
- `POST /orca/webauthn/register/begin` -- WebAuthn registration
|
||||
challenge (random nonce, user info).
|
||||
- `POST /orca/webauthn/register/finish` -- attestation verification,
|
||||
credential storage.
|
||||
- `GET /orca/webauthn/login` -- login HTML/JS page.
|
||||
- `POST /orca/webauthn/login/begin` -- assertion challenge.
|
||||
- `POST /orca/webauthn/login/finish` -- assertion verification, OIDC
|
||||
`sub` extraction, redirect with auth code.
|
||||
- **Passkey storage**: SQLite at `ClusterDir()/webauthn-credentials.db`
|
||||
(0600). Schema: `credentials(user_id TEXT PRIMARY KEY, credential_id
|
||||
BLOB, public_key BLOB, sign_count INTEGER, aaguid TEXT, created_at
|
||||
TEXT)`. Public keys only; no private keys, no secrets.
|
||||
- **BYO external IdP override**: `oidc.issuer` in config repoints to
|
||||
an external IdP. The bundled Dex + WebAuthn connector is bypassed;
|
||||
the external IdP's authenticators (including its own WebAuthn) are
|
||||
used. Orca never sees the upstream credentials.
|
||||
|
||||
### RQ-1 resolution (RESEARCH binding question)
|
||||
|
||||
**RQ-1**: How does the bundled Dex bootstrap an upstream identity
|
||||
without any password, given the mTLS-only constraint?
|
||||
|
||||
**Answer (resolved by C3/D-240)**: The bundled Dex's upstream
|
||||
authenticator IS the WebAuthn connector. No external password source
|
||||
is needed for the bundled path. The WebAuthn connector serves the
|
||||
registration + login ceremonies directly; Dex maps the credential ID
|
||||
to an OIDC `sub`. BYO-IdP covers password-based upstreams (LDAP/AD)
|
||||
if an operator insists -- but those never flow through Orca.
|
||||
|
||||
**C-37 fallback** (kept if WebAuthn proves infeasible): bundled Dex
|
||||
ships mTLS-client-cert-only (Traefik `X-Forwarded-Client-Cert` header
|
||||
-> Dex `typed-external-connector`). Password-based upstreams require
|
||||
BYO external IdP. The "no Orca credentials" invariant holds regardless.
|
||||
|
||||
### Master key sealing architecture
|
||||
|
||||
- **Seal**: at `orca cluster seal`, the in-memory master key is
|
||||
encrypted with a key derived from the operator's OIDC ID token
|
||||
(HKDF-SHA256 of the token's `sub` + a fresh 32-byte salt). The
|
||||
sealed blob (`salt || ciphertext`) is stored at
|
||||
`ClusterDir()/master.key.sealed` (0600). The raw key is zeroed from
|
||||
memory. Shamir 3-of-5 shards are printed for offline recovery.
|
||||
- **Unseal**: at `orca cluster unseal`, the operator authenticates via
|
||||
OIDC (WebAuthn ceremony). The resulting ID token's `sub` + the
|
||||
stored salt derive the unwrapping key. The master key is unwrapped
|
||||
into memory and held for the cluster's lifetime. Zeroed on shutdown.
|
||||
- **Recovery**: if the IdP is lost, the operator presents 3 of 5
|
||||
Shamir shards to `orca cluster unseal --recovery`. The shards
|
||||
reconstruct the seal key; the master key is unwrapped. No backdoor.
|
||||
- **mTLS-only offline path**: for the single-operator fully-offline
|
||||
case (no OIDC), the seal key is derived from the cluster's own CA.
|
||||
The operator holds the CA (a cert, not a password). Shamir recovery
|
||||
applies to the OIDC-sealed mode only.
|
||||
|
||||
### Offline-first reconciliation (R-003)
|
||||
|
||||
The OIDC provider must be reachable to unseal the master key and to
|
||||
authenticate operators. For offline/air-gapped clusters, the operator
|
||||
runs the **bundled Dex on the lead** (offline). For the
|
||||
single-operator fully-offline case, the operator can skip OIDC and
|
||||
rely on mTLS-only machine identity (no human authn needed -- the
|
||||
operator holds the pre-staged SSH key + mTLS cert; no password, no
|
||||
token). Orca stays minimal (no bundled IdP beyond Dex); it validates
|
||||
tokens against whatever issuer the operator configures.
|
||||
|
||||
## Dependency posture (new in v0.12)
|
||||
|
||||
v0.12 adds these dependencies (all CGO-free, audited):
|
||||
|
||||
- `github.com/coreos/go-oidc/v3` -- OIDC client (token verification,
|
||||
JWKS, ID token parsing). Pure Go.
|
||||
- `github.com/go-webauthn/webauthn` -- WebAuthn library (registration,
|
||||
login, attestation/assertion verification). Pure Go.
|
||||
- `github.com/dexidp/dex` -- bundled Dex binary (vendored, not a Go
|
||||
import; deployed as a separate systemd unit). Apache-2.0.
|
||||
- `golang.org/x/crypto/ssh/...` -- already a dependency (sshpush).
|
||||
|
||||
No CGO. No gRPC. No ConnectRPC. No YAML parser. The "stdlib + minimal
|
||||
deps" posture (D-008) is preserved.
|
||||
|
||||
## Codebase reality (verified 2026-08-07)
|
||||
|
||||
- `internal/acl/acl.go` -- ACL exists but is unenforced (F1). P06
|
||||
rewrites it (remove KindToken, add KindOidc, wire enforcement).
|
||||
- `internal/runtime/podman.go:57`, `internal/runtime/wasm.go:39` --
|
||||
unquoted cmdStr interpolation (F3). P01 fixes via shellQuote.
|
||||
- `internal/cli/ns.go:nsCreateCmd` -- no `..`/`/` rejection (F4). P02
|
||||
adds `validateNamespaceName`.
|
||||
- `internal/txn/txn.go:renderApplyScript` -- arbitrary path writes
|
||||
(F5). P03 adds prefix allowlist.
|
||||
- `internal/security/ca.go` -- legacy CA, deprecated but load-bearing
|
||||
(F16). P23 deletes it (gated on P06/P08/P09/P11).
|
||||
- `internal/secrets/secrets.go` -- master key raw file, no rotation
|
||||
(F12). P08 seals it to OIDC; P14 adds rotation.
|
||||
- `internal/audit/audit.go` -- plain SQLite INSERT (F2). P10 adds
|
||||
hash-chain + HMAC.
|
||||
- `internal/emitter/nft.go` -- no conntrack/default-deny (F21). P18
|
||||
hardens the ruleset.
|
||||
- `internal/proxmox/bootstrap.go:29` -- `--password` bootstrap (F23,
|
||||
R-021 violation). P07 removes it.
|
||||
- `internal/identity/spiffe.go:95` -- no chain validation (F9). P11
|
||||
fixes.
|
||||
- `scripts/install.sh` -- no checksum verification (F14). P17 adds
|
||||
SHA256SUMS + GPG signature.
|
||||
- `scripts/orca-aggregate.sh:64` -- raw JSON interpolation (F11). P16
|
||||
replaces with jq/Go.
|
||||
|
||||
## Alignment with existing gates
|
||||
|
||||
- **C-19** (threat model) -- v0.11 P15.5 produced the initial threat
|
||||
model; v0.12 is the comprehensive expansion (full OS surface).
|
||||
- **C-08** (SPIFFE spike) -- passed; v0.12 P11 hardens the verification
|
||||
path.
|
||||
- **R-001..R-020** -- unchanged; R-021 is an extension, not a reversal.
|
||||
- **D-008** (no CGO) -- preserved; all new deps are pure Go.
|
||||
|
||||
## Risks (for GRILL to pressure-test)
|
||||
|
||||
- **P07 (password removal) is breaking** -- mitigation: C-34 migration
|
||||
gate (`--accept-identity-migration`).
|
||||
- **P08 (master key seal) is the riskiest phase** -- a bug corrupts all
|
||||
secrets at rest. Mitigation: `--dry-run`, atomic re-encryption,
|
||||
automatic rollback to old sealed key on any failure.
|
||||
- **P21 (SQLite encryption) may need CGO** -- C-31 fallback to
|
||||
file-mode 0600 + documented threat if SQLCipher needs CGO. No CGO.
|
||||
- **P23 (dual-write closure) is high-impact** -- removing the legacy
|
||||
CA breaks `orca init`/`orca cert` if step-ca isn't fully wired.
|
||||
Mitigation: gate on P06/P08/P09/P11, full test coverage before
|
||||
deletion.
|
||||
- **P05 (WebAuthn connector) is new ground** -- ~300 LoC custom Dex
|
||||
connector. Mitigation: C-37 fallback (mTLS-client-cert-only) if
|
||||
WebAuthn proves infeasible; virtual-authenticator integration tests
|
||||
(P26) using `go-webauthn` test helpers.
|
||||
- **Bundled Dex is a new systemd unit + Traefik route** -- operational
|
||||
surface growth. Mitigation: `orca doctor oidc` checks Dex health,
|
||||
JWKS reachability, WebAuthn endpoint TLS.
|
||||
- **C-32 human gate** (leaked GITEA_TOKEN) could stall the final ship.
|
||||
Escalation path: ship as `v0.11.29-rc1` if rotation pending,
|
||||
`v0.11.29` when confirmed.
|
||||
|
||||
## Next steps
|
||||
|
||||
Phase 0 proceeds to IDEATE (produce the 30 net-new requirements
|
||||
REQ-119..REQ-148), then PLAN (29 phases, wave ordering, persona
|
||||
assignments), then GRILL (ratify C-29..C-38).
|
||||
@@ -1,163 +0,0 @@
|
||||
# RESEARCH v0.13: Production Hardening Round 2 — Threat Model & Gap Analysis
|
||||
|
||||
**Status**: complete (2026-08-07). Three deep codebase sweeps (security,
|
||||
reliability, feature/doc claims) performed via parallel sub-agents.
|
||||
~60 gaps surfaced beyond v0.12. Findings drive the 15 new requirements
|
||||
(REQ-149..REQ-163) and 14-phase plan.
|
||||
|
||||
## Methodology
|
||||
|
||||
Three parallel `explore` agents investigated the codebase:
|
||||
1. **Security sweep** — input validation, injection, SSH, crypto, TLS,
|
||||
race conditions, SQL, secrets, backup, pprof, rate limiting, memory,
|
||||
dependencies, toolchain vulns.
|
||||
2. **Reliability sweep** — idempotency, concurrency, SQLite, partial
|
||||
failure, SSH fanout, timeouts, systemd, journald, cache, watch
|
||||
streams, scheduler, capacity, namespace isolation, DB growth, time,
|
||||
signals, temp files, flock.
|
||||
3. **Feature/doc sweep** — README claims, docs/*, examples/*, Makefile,
|
||||
.coreci.yml, CHANGELOG, REQUIREMENTS/ROADMAP consistency, help text,
|
||||
deprecation warnings, WASM claim.
|
||||
|
||||
Each agent produced a structured report with file:line evidence. This
|
||||
document synthesizes the findings into the v0.13 plan.
|
||||
|
||||
## Threat Model Round 3 — Findings
|
||||
|
||||
### Critical (must fix in v0.13)
|
||||
|
||||
| ID | Finding | file:line | REQ |
|
||||
|----|---------|-----------|-----|
|
||||
| F26 | `orca job run` runs locally via `exec.CommandContext` — scheduler/emitter/SSH-push are dead code; documented deployment model non-functional | `internal/cli/job.go:352-372`, `internal/engine/executor.go:150-180` | REQ-151 |
|
||||
| F27 | jobspec `schedule:` and `timeout:` silently dropped by markdown parser — DaemonSet fundamentally broken | `internal/jobspec/markdown.go:480-573` | REQ-152 |
|
||||
| F28 | `verify-reqs` gate bypassed for v0.12 (bold-format regex mismatch) | `cmd/verify-reqs/main.go:29` | REQ-160 |
|
||||
| F29 | Command injection in `orca logs --job` via `%q`+backtick (RCE via SSH fanout) | `internal/cli/logs.go:283,289` | REQ-150 |
|
||||
| F30 | pprof loopback bypass via `:6060` (empty host = bind-all) | `internal/daemon/pprof.go:21-29` | REQ-150 |
|
||||
| F31 | Tar-slip in backup restore (`a/../../etc/passwd` bypasses `HasPrefix(name,"..")`) | `internal/backup/backup.go:302-304` | REQ-150 |
|
||||
| F32 | Unauthenticated WebAuthn registration (account takeover) | `internal/webauthn/connector.go:85,120` | REQ-153 |
|
||||
| F33 | ROADMAP marks v0.12 COMPLETE but seal/unseal/init-idp/auth-register don't exist | `.ciagent/ROADMAP.md:403` | REQ-154,155 |
|
||||
|
||||
### High (must fix in v0.13)
|
||||
|
||||
| ID | Finding | file:line | REQ |
|
||||
|----|---------|-----------|-----|
|
||||
| F34 | nft ruleset injection via unvalidated `TrustedProbes` IPs | `internal/emitter/nft.go:101-107` | REQ-150 |
|
||||
| F35 | sudoers/shell injection via `--proxmox-user`/`--proxmox-role` | `internal/proxmox/bootstrap.go:445-452` | REQ-150 |
|
||||
| F36 | `validateSudoers` checks wrong filename when `ProxmoxUser != "orca"` | `internal/proxmox/bootstrap.go:474` | REQ-150 |
|
||||
| F37 | `orca txn rollback` shell injection via unvalidated txn ID | `internal/cli/txn.go:240-241` | REQ-150 |
|
||||
| F38 | `orca nft diff --against` path traversal | `internal/cli/nft.go:225` | REQ-150 |
|
||||
| F39 | `drain stopAlloc` stored injection from compromised peer | `internal/cli/drain.go:132` | REQ-150 |
|
||||
| F40 | `cluster_compat` stored injection from peer | `internal/cli/cluster_compat.go:399` | REQ-150 |
|
||||
| F41 | podman `image` `%q` backtick injection | `internal/runtime/podman.go:67` | REQ-150 |
|
||||
| F42 | Go toolchain 1.25.0 — 24 stdlib vulns (tar, tls, x509, http, pem...) | `go.mod:3` | REQ-149 |
|
||||
| F43 | No SQLite `busy_timeout` — "database is locked" under concurrency | `internal/store/store.go:21` | REQ-156 |
|
||||
| F44 | Audit hash-chain race — concurrent appends corrupt tamper-evidence | `internal/store/audit_repo.go:908-919` | REQ-154 |
|
||||
| F45 | Concurrent `secrets set` silently loses data (no flock) | `internal/cli/secrets.go:135-148` | REQ-156 |
|
||||
| F46 | Concurrent `orca upgrade` races on Traefik cutover + binary install | `internal/cli/upgrade.go:111` | REQ-156 |
|
||||
| F47 | Cache never invalidated by writes — stale reads after join/create/run | `internal/cli/cache.go:763-770` | REQ-156 |
|
||||
| F48 | `acl.Check` called zero times — v0.12 zero-trust not wired | `internal/daemon/`, `internal/sshpush/` | REQ-153 |
|
||||
| F49 | `acl.json` mode 0644 (should be 0600 per REQ-145) | `internal/cli/acl.go:152` | REQ-153 |
|
||||
| F50 | README "mTLS by default" is false — SSH-push is canonical, mTLS deprecated | `README.md`, `internal/cli/node.go:93-98` | REQ-160 |
|
||||
| F51 | `docs/cli.md` missing ~25 subcommands; CHANGELOG stale at v0.1 | `docs/cli.md:4`, `CHANGELOG.md:9-32` | REQ-160 |
|
||||
| F52 | `docs/security-runbook.md` documents seal/unseal/doctor audit that don't exist | `docs/security-runbook.md:5-11,23` | REQ-160 |
|
||||
| F53 | `docs/webauthn.md` documents `orca auth register` that doesn't exist | `docs/webauthn.md:13` | REQ-155,160 |
|
||||
| F54 | `auth init-idp` is a stub — v0.12 R-021 load-bearing change has no working IdP | `internal/cli/auth.go:151-155` | REQ-155 |
|
||||
| F55 | `secrets rotate-master` writes raw key, doesn't re-seal to OIDC | `internal/cli/secrets.go:358` | REQ-154 |
|
||||
| F56 | `orca cluster seal`/`unseal` documented but not implemented | `docs/security-runbook.md:3-9` | REQ-154 |
|
||||
| F57 | `orca doctor audit` documented but not implemented | `docs/security-runbook.md:18` | REQ-154 |
|
||||
| F58 | `orca doctor modes` not implemented (REQ-130) | `internal/security/ca.go:236` | REQ-154 |
|
||||
| F59 | Audit actor field is "cli"/"daemon" not OIDC sub/SVID | `internal/cli/drain.go`, `internal/daemon/server.go` | REQ-153 |
|
||||
| F60 | `Executor.Run` holds mutex for whole job duration | `internal/engine/executor.go:101-103` | REQ-156 |
|
||||
| F61 | `splitHostPort` in drain.go breaks IPv6 addresses | `internal/cli/drain.go:68-74` | REQ-157 |
|
||||
| F62 | `transport.IsTransient` + `sshpush.isTransient` both use substring matching | `internal/transport/retry.go:44`, `internal/sshpush/transport.go:395-414` | REQ-157 |
|
||||
| F63 | `rotateSSHKeys` partial-result window (old key overwritten before all peers updated) | `internal/cli/rotate_lead.go:132` | REQ-157 |
|
||||
| F64 | `known_hosts` flock field stored but not read by `dial()` | `internal/sshpush/transport.go:60-63` | REQ-157 |
|
||||
| F65 | `verifyCutover` uses default http.Client against orca CA (will fail TLS verification) | `internal/cli/upgrade.go:313-314` | REQ-157 |
|
||||
| F66 | v0.8→v0.11 migration torn-write window (crash after rename, before schema fixup) | `internal/migration/migrate.go:135-140` | REQ-158 |
|
||||
| F67 | `job stop` is soft-stop only (doesn't signal process) | `internal/cli/job.go:266` | REQ-158 |
|
||||
| F68 | `upgrade.go` cutover uses direct `sed -i` (no backup file) | `internal/cli/upgrade.go:performCutover` | REQ-158 |
|
||||
| F69 | `nft country block add` validates length but not content; uses `%q` | `internal/cli/nft.go:136,259` | REQ-150 |
|
||||
| F70 | `--type linux` reserved but unimplemented | `internal/model/node.go:29` | REQ-161 |
|
||||
| F71 | No UAT/E2E test doc exists | repo-wide | REQ-162,163 |
|
||||
|
||||
### Medium (fix in v0.13)
|
||||
|
||||
| ID | Finding | file:line | REQ |
|
||||
|----|---------|-----------|-----|
|
||||
| F72 | Master/SVID keys never zeroed from memory after use | throughout `internal/secrets/`, `internal/seal/` | REQ-154 |
|
||||
| F73 | Cache DB mode 0644 (not 0600) | `internal/cache/cache.go:61-64` | REQ-158 |
|
||||
| F74 | `writeAtomic0600`/collector: predictable tmp, no cleanup, leaks | `internal/identity/oidc.go:134`, `internal/cli/collector.go:179` | REQ-156 |
|
||||
| F75 | `cli/acl.go writeAtomicFile` no fsync (durability gap) | `internal/cli/acl.go:161-181` | REQ-156 |
|
||||
| F76 | WebAuthn session stores unsynchronized global maps (data race) | `internal/webauthn/connector.go:67,171` | REQ-156 |
|
||||
| F77 | `loadOIDCConfig` TODO for config-file loading | `internal/cli/auth.go:168` | REQ-155 |
|
||||
| F78 | No retention/compaction for jobs/tasks/audit_log tables | `internal/store/` | REQ-158 |
|
||||
| F79 | `orca logs` no `--lines` cap, `--since` unbounded (OOM risk) | `internal/cli/logs.go:173-185` | REQ-158 |
|
||||
| F80 | `ns create` non-atomic (partial dir creation on mid-failure) | `internal/cli/ns.go:906-918` | REQ-156 |
|
||||
| F81 | `writeCurrentLead` non-atomic `os.WriteFile` | `internal/cli/rotate_lead.go:315-322` | REQ-156 |
|
||||
| F82 | `secrets set` doesn't validate namespace exists (creates phantom ns) | `internal/cli/secrets.go:130` | REQ-156 |
|
||||
| F83 | `backup` has no lock; concurrent backups may clobber | `internal/cli/backup.go:42-68` | REQ-156 |
|
||||
| F84 | Root command has no SIGINT/SIGTERM handler for non-watch commands | `cmd/orca/main.go:17-22` | REQ-157 |
|
||||
| F85 | SSH commands without explicit timeouts (peer-setup, drift, txn rollback, job restart) | various | REQ-157 |
|
||||
| F86 | Rendered systemd units never validated (`systemd-analyze verify`) before deploy | `internal/emitter/systemd.go:80-98` | REQ-151 |
|
||||
| F87 | OIDC callback HTTP server has no timeouts (slowloris) | `internal/identity/oidc.go:244` | REQ-157 |
|
||||
| F88 | No security headers on daemon TLS surface | `internal/daemon/health.go:93` | REQ-159 |
|
||||
| F89 | `orca status` returns hardcoded v0.1 stub, not deprecated | `internal/cli/status.go:22` | REQ-160 |
|
||||
| F90 | `job run` help text says "HCL spec file" but HCL is deprecated | `internal/cli/job.go:47-48` | REQ-160 |
|
||||
| F91 | README subcommand table omits `auth`, `nft`, `peer-setup` | `README.md` | REQ-160 |
|
||||
| F92 | `docs/namespace.md` omits `inherit`/`set-constraint` | `docs/namespace.md:114-134` | REQ-160 |
|
||||
| F93 | README "latest tag: v0.10.19" is stale (actual: v0.11.29) | `README.md:30,39` | REQ-160 |
|
||||
| F94 | `docs/install.md`+`docker.md` reference stale v0.4.x and deprecated daemon | `docs/install.md:42,62`, `docs/docker.md:21,43` | REQ-160 |
|
||||
| F95 | IPv6 host not bracketed in proxmox SSH dial | `internal/proxmox/bootstrap.go:140` | REQ-157 |
|
||||
|
||||
### Low (fix in v0.13 where cheap, document otherwise)
|
||||
|
||||
| ID | Finding | file:line | REQ |
|
||||
|----|---------|-----------|-----|
|
||||
| F96 | `--pprof-allow-public` documented but never implemented | `internal/daemon/pprof.go:37,42,43` | REQ-150 |
|
||||
| F97 | `nft country block add` weak code validation | `internal/cli/nft.go:136` | REQ-150 |
|
||||
| F98 | `cert show`/`fingerprint` don't emit deprecation warnings | `internal/cli/cert.go` | REQ-160 |
|
||||
| F99 | `docs/namespace.md` references `orca doctor --legacy-paths` that doesn't exist | `docs/namespace.md:165` | REQ-160 |
|
||||
| F100 | `release.sh` only builds linux-amd64; install.sh advertises arm64 | `scripts/release.sh:94-102` | accepted (D-193) |
|
||||
| F101 | `docs/cli.md` version example shows "v0.9.1" but default is "0.1.0-dev" | `docs/cli.md:253` | REQ-160 |
|
||||
|
||||
## CLEAN categories (verified, no new findings)
|
||||
|
||||
- **SQL injection in `internal/store/`** — all queries use `?` placeholders
|
||||
- **TLS version/cipher policy** — TLS 1.3 only, AEAD cipher allowlist
|
||||
- **SSH key generation** — Ed25519, `crypto/rand`, PKCS8, 0600
|
||||
- **TOFU host-key pinning** — fail-closed on mismatch, constant-time comparison
|
||||
- **Self-signed cert generation** — RSA 3072, 128-bit serial, correct KeyUsage
|
||||
- **Nonce reuse in secrets** — fresh 12-byte nonce per line from `crypto/rand`
|
||||
- **Gitleaks / secrets in git history** — only test fixtures
|
||||
- **Secrets logged in errors** — only keys/namespaces logged, never values
|
||||
- **CSRF on HTTP surfaces** — daemon is GET-only, no state-changing GETs
|
||||
- **Watch streams (iter.Seq)** — pull-based, defer cleanup, no goroutine leak
|
||||
- **DNS resolution** — bounded by `net.Dialer{Timeout: 15s}`
|
||||
- **Multi-namespace DB isolation** — per-ns file layout
|
||||
|
||||
## Accepted residual risks (documented, not fixed)
|
||||
|
||||
1. OIDC tokens plaintext at rest (0600) — sealing on every CLI invocation conflicts with "no orca binary on servers" model
|
||||
2. HSTS on daemon — mTLS-only API, no browser-facing surface
|
||||
3. DNS resolution timeout — bounded by `net.Dialer{Timeout: 15s}`
|
||||
4. Temp file cleanup on SIGKILL — orphaned temp files, operator-visible
|
||||
5. Flock timeout on NFS — stuck holder is rare; `tryFlockEx` exists
|
||||
6. "WASM-first" pillar aspirational — document as "WASM runtime available, process is default"
|
||||
7. arm64/armv7 release — D-193 deferred; install.sh detection is forward-looking
|
||||
8. OIDC callback slowloris — loopback, short-lived, single CLI invocation
|
||||
9. `--pprof-allow-public` flag — remove references, make loopback-only a hard invariant
|
||||
|
||||
## Architecture updates (for ARCHITECTURE.md)
|
||||
|
||||
- **R-022**: `orca job run` deploys via scheduler → emitter → SSH-push (local exec path removed)
|
||||
- **R-023**: Zero-trust enforcement wired (`acl.Check` on every request path)
|
||||
- New component: `internal/linux/bootstrap.go` (Ubuntu/Debian SSH-join, mirrors Proxmox pattern)
|
||||
- New artifact: `docs/uat.md` + `scripts/uat-signoff.sh` (v1.0 gate)
|
||||
- New artifact: `docs/metrics.md` (expanded Prometheus metric set)
|
||||
|
||||
## Conclusion
|
||||
|
||||
Three deep sweeps found ~60 gaps. v0.13 closes all critical/high/medium
|
||||
(REQ-149..REQ-163, 14 phases). 9 low-severity residual risks are
|
||||
documented and accepted. This is the last hardening round. v1.0.0 is
|
||||
gated on the UAT signoff script delivered by P12.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,506 +0,0 @@
|
||||
# Research: Orca v0.3 — scheduling-streaming
|
||||
|
||||
Phase: 0 (research) for milestone v0.3 (scheduling-streaming).
|
||||
Branch: `phase/00-pre-execution` (cut from `milestone/v0.3-scheduling-streaming`).
|
||||
Go toolchain: `go1.25.0` (confirmed via `go version`; `go.mod` declares `go 1.25.0`).
|
||||
|
||||
This document provides concrete, file-level implementation guidance for the
|
||||
two v0.3 execution phases:
|
||||
|
||||
- **P01** — `iter.Seq` streaming for `--watch` flags (REQ-022, REQ-030)
|
||||
- **P02** — `orca doctor` network + db full implementation (REQ-032 completion)
|
||||
|
||||
All assumptions are logged as decisions (D-025..D-038) with confidence scores.
|
||||
Full autonomy mode — no items flagged for human validation.
|
||||
|
||||
---
|
||||
|
||||
## Codebase Audit Summary
|
||||
|
||||
### Current state (commit ba5ffd7 + phase docs)
|
||||
|
||||
| Area | File | Key finding |
|
||||
|------|------|-------------|
|
||||
| CLI `job list` | `internal/cli/job.go:112-143` | `jobListCmd.RunE` calls `store.NewJobRepo(db).List(ctx)`, prints a fixed-width table; `--json` via `printJSON(jobs)`. No `--watch` flag exists. |
|
||||
| CLI `node list` | `internal/cli/node.go:155-186` | `nodeListCmd.RunE` calls `registry.List(ctx)` → `repo.List(ctx)`. Table + `--json`. No `--watch` flag. |
|
||||
| CLI root | `internal/cli/root.go` | `jsonOutput` is a package-level `bool` set by `--json` persistent flag. `printJSON` uses `json.NewEncoder` with 2-space indent. |
|
||||
| Job repo | `internal/store/job_task_repo.go` | `JobRepo.List(ctx) ([]*model.Job, error)` — single-shot query, closes rows. `scanJob` helper is reusable. |
|
||||
| Node repo | `internal/store/node_repo.go` | `NodeRepo.List(ctx) ([]*model.Node, error)`. `scanNode` helper is reusable. `scanner` interface defined here (`Scan(dest ...any) error`) — shared by `*sql.Row` and `*sql.Rows`. |
|
||||
| Store open | `internal/store/store.go` | `store.Open(path)` opens with `?_pragma=journal_mode(WAL)&_pragma=foreign_keys(ON)` and runs `migrate(db)`. |
|
||||
| Migrations | `internal/store/migrate.go` | `migrate` is unexported, runs at `Open` time. `schema_migrations` table tracks applied migrations by filename. Migrations are embedded via `//go:embed migrations/*.sql`. No public API to query migration version. |
|
||||
| Migrations on disk | `internal/store/migrations/` | `0001_nodes.sql`, `0002_jobs_tasks.sql`, `0003_audit_log.sql`, `0004_certs.sql`, `0005_node_capacity.sql`. Highest = 0005. |
|
||||
| Doctor | `internal/doctor/doctor.go` | `NetworkStub()` and `DBStub()` return WARN stubs. `All()` aggregates 6 checks. `Run(ctx)` iterates checks. `Check.Run` signature: `func(ctx context.Context) (Result, string)`. `Result` is `PASS|WARN|FAIL`. No DB or transport imports — cert-only. |
|
||||
| Doctor CLI | `internal/cli/doctor.go` | `doctorNetworkCmd`/`doctorDBCmd` call `doctor.NetworkStub()`/`doctor.DBStub()` directly. |
|
||||
| Doctor tests | `internal/doctor/doctor_test.go` | Two tests: `TestRunAllChecksWithNoCA` (expects FAIL + WARN for stubs), `TestRunWithCAAndServerCert` (cert checks PASS). Uses `t.Setenv("ORCA_HOME", dir)`. **The "expects WARN" assertion will break when stubs become real checks** — must be updated in P02. |
|
||||
| Transport mTLS client | `internal/transport/mtls.go` | `NewMTLSClient(caPath, serverName, certPath, keyPath)` builds an `http.Client` with a TLS-1.3-only config from `security.ClientTLSConfig`. `MTLSClient.Do(req)`. `DialContext` for low-level TLS dial. |
|
||||
| Transport dispatch client | `internal/transport/dispatch.go` | `NewDispatchClient(caPath, serverName, peerAddr)` wraps `MTLSClient`. `PeerAddr` is `http://` or `https://`. `Submit`/`Status` POST to `/orca.v1.Dispatch/*`. No `/healthz` GET helper. |
|
||||
| Daemon health | `internal/daemon/health.go` | `handleHealthz` → 200 `{"status":"alive"}`. `handleReadyz` → 200/503 with db ping. Mounted at `mux.HandleFunc("/healthz", ...)` in `server.go:104`. |
|
||||
| Daemon TLS | `internal/daemon/tls.go` | `StartMTLS(state)` sets `httpServer.TLSConfig` with `ClientAuth = RequireAndVerifyClientCert`. The daemon **requires client certs** in mTLS mode. |
|
||||
| Peer registry | `internal/engine/peer.go` | `PeerRegistry` is **in-memory only** (`map[string]*Peer` under `sync.RWMutex`). `NewPeerRegistry()` returns empty. `Peer` has `NodeID, Address, ServerName, CAPath, LastSeen, Capacity`. **Not persisted to SQLite.** |
|
||||
| Peer registry usage | `internal/cli/job.go:70`, `internal/cli/daemon.go:47` | Both create a **fresh empty** `NewPeerRegistry()` per process. No code ever calls `peers.Add(...)`. The registry is currently a structural placeholder. |
|
||||
| Node registry | `internal/engine/registry.go` | `NodeRegistry` wraps `store.NodeRepo` + `Audit`. `List(ctx)` → `repo.List(ctx)`. Persisted to `nodes` table. |
|
||||
| Node model | `internal/model/node.go` | `Node{ID, Name, Address, State, JoinedAt, LastSeen, Metadata}`. **No `ServerName` or `CAPath` field** — `model.Node` differs from `engine.Peer`. |
|
||||
| Cert paths | `internal/certpaths/certpaths.go` | `Dir()` honors `ORCA_HOME`; `CACertPath()`, `ServerCertPath()`, `ServerKeyPath()`. |
|
||||
| Security client TLS | `internal/security/tls_config.go:106` | `ClientTLSConfig(caPath, serverName, certPath, keyPath)` — TLS 1.3 only, AEAD allowlist, `RootCAs` = single CA. Both-or-neither for cert/key. |
|
||||
| Go version | `go.mod` + `go version` | `go 1.25.0` — `iter` package and range-over-func are stable stdlib. |
|
||||
| Deps | `go.mod` | cobra, hcl/v2, modernc/sqlite, uuid. **No new deps needed for v0.3.** `iter` is stdlib. |
|
||||
|
||||
### Critical gap analysis
|
||||
|
||||
1. **`PeerRegistry` is non-persistent and always empty at CLI time.** The
|
||||
doctor network check cannot rely on it — there is no code path that populates
|
||||
it. The `nodes` table IS persisted and has `Address`, but lacks the
|
||||
`ServerName`/`CAPath` needed for an mTLS probe. **Resolution: doctor network
|
||||
reads the `nodes` table via `NodeRepo.List`, and derives `ServerName` +
|
||||
`CAPath` from local config (`certpaths.CACertPath()` + node name/addr).**
|
||||
See D-029.
|
||||
|
||||
2. **`doctor.Run` / `Check.Run` do not plumb a `*sql.DB` or transport client.**
|
||||
The cert checks are filesystem-only. P02 must extend the check constructors
|
||||
to accept a DB handle and (for network) a transport client factory. The
|
||||
`Check.Run` signature (`func(ctx) (Result, string)`) is preserved by
|
||||
closure-capturing the handles in the constructor. See D-027, D-031.
|
||||
|
||||
3. **No public migration-version query.** `migrate()` is unexported and writes
|
||||
to `schema_migrations(name, applied_at)`. P02 adds a public
|
||||
`store.MigrationVersion(ctx, db)` (or method on a repo) that selects the max
|
||||
applied migration name. See D-033.
|
||||
|
||||
4. **Doctor test `TestRunAllChecksWithNoCA` asserts a WARN from stubs.** This
|
||||
will break when stubs become real (the db check will PASS with a fresh test
|
||||
DB, and the network check will WARN/FAIL on zero peers). Must be updated.
|
||||
See D-036.
|
||||
|
||||
---
|
||||
|
||||
## P01: iter.Seq Streaming for `--watch` Flags
|
||||
|
||||
Covers REQ-022 (`iter.Seq` for streaming job lists), REQ-030 (`--watch` output
|
||||
format: table default vs streaming one-line JSON per event).
|
||||
|
||||
### Decisions
|
||||
|
||||
| ID | Decision | Confidence |
|
||||
|----|----------|------------|
|
||||
| **D-025** | `iter.Seq` lives on the store repos, not the engine registry. `JobRepo.Watch(ctx) iter.Seq[*model.Job]` and `NodeRepo.Watch(ctx) iter.Seq[*model.Node]`. Rationale: repos already own the `*sql.DB` and the `scanJob`/`scanNode` helpers; engine.Registry.List just delegates to repo. Keeping Watch in the store layer matches the data-engineer territory and avoids a new engine→store iter dependency. | 0.90 |
|
||||
| **D-026** | Element type is `*model.Job` / `*model.Node` (pointer), matching the existing `[]*model.Job` return of `List`. This keeps `printJSON` and table rendering identical between one-shot and watch paths. | 0.88 |
|
||||
| **D-027** | The `Check.Run` signature in `doctor` is unchanged; P02 captures DB/transport handles in closure at constructor time (`Network(db)`, `DB(db)`). This is the established pattern (cert checks already closure-capture `certpaths`). | 0.92 |
|
||||
| **D-028** | Watch refresh = poll-based 1s ticker (per D-019). No event channel, no daemon coupling. Each tick re-runs the existing `List` query and yields the **full current snapshot** (one element per row). The CLI dedupes by detecting snapshot equality before re-rendering (see D-030). Rationale: simpler than NOTIFY/LISTEN, no daemon dependency, matches offline-first. | 0.90 |
|
||||
| **D-029** | `--watch` output: default = re-print the table on every changed snapshot (clear screen via ANSI `\033[2J\033[H` then table); `--watch --json` = one compact JSON line **per snapshot** (an array on each line, OR one line per element — see D-030). The CLI tracks the previous snapshot's hash to avoid spamming identical frames. | 0.85 |
|
||||
| **D-030** | `--watch --json` emits **one JSON object per element per tick where the element changed**, i.e. streaming one-line JSON per event (per REQ-030 wording). Implementation: on each tick, for each element, if its JSON bytes differ from the previous snapshot's bytes for that ID, print `{"event":"update","job":{...}}\n`. On first tick, print all as `{"event":"init",...}`. This is the most useful for AI agents tailing the stream. Confidence lower because REQ-030 is ambiguous between "array per tick" and "object per event"; the per-event interpretation matches "streaming one-line JSON per event" literally. | 0.72 |
|
||||
| **D-031** | Cancellation: `signal.NotifyContext(ctx, os.Interrupt, syscall.SIGTERM)` at the CLI command layer, **replacing** the current `context.WithTimeout(cmd.Context(), 5*time.Second)` for the watch path only. The non-watch `list` path keeps its 5s timeout. The `iter.Seq` receives this ctx and stops yielding on `ctx.Done()`. | 0.93 |
|
||||
| **D-032** | The `iter.Seq` implementation **must not leak goroutines**: the polling loop runs **inline in the yield callback's caller goroutine** (the `range` loop), not a separate goroutine. `for range seq { ... }` drives the pull; inside `Watch`, we loop `for { select { <-ticker.C: query+yield each; <-ctx.Done(): return } }` and call `yield(item)` directly. When `yield` returns false (consumer broke the loop), we stop and return. **No goroutine is spawned by Watch.** This is the cleanest Go 1.25 iter pattern and avoids leak surface entirely. | 0.95 |
|
||||
|
||||
### Implementation approach — store layer
|
||||
|
||||
**File: `internal/store/job_task_repo.go`** — add method:
|
||||
|
||||
```go
|
||||
// Watch yields the current snapshot of jobs on a 1-second ticker until
|
||||
// ctx is cancelled or the consumer stops pulling (yield returns false).
|
||||
// It does not spawn a goroutine; the polling loop runs in the caller's
|
||||
// goroutine via the range-over-func pull protocol.
|
||||
//
|
||||
// Each tick re-runs the List query and yields one *model.Job per row.
|
||||
// The caller is responsible for deduping across ticks if desired.
|
||||
func (r *JobRepo) Watch(ctx context.Context) iter.Seq[*model.Job] {
|
||||
return func(yield func(*model.Job) bool) {
|
||||
ticker := time.NewTicker(1 * time.Second)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
}
|
||||
// Reuse the existing List query + scanJob helper.
|
||||
rows, err := r.db.QueryContext(ctx,
|
||||
`SELECT id, name, spec, status, exit_code, created_at, started_at, ended_at
|
||||
FROM jobs ORDER BY created_at DESC`)
|
||||
if err != nil {
|
||||
// Surfacing errors from inside iter.Seq is awkward; the
|
||||
// CLI layer cannot receive a returned error. Log via slog
|
||||
// (the repo doesn't hold a logger today — see D-034) and
|
||||
// continue to next tick rather than terminating the
|
||||
// stream. A transient DB blip should not kill the watch.
|
||||
continue
|
||||
}
|
||||
for rows.Next() {
|
||||
j, err := scanJob(rows)
|
||||
if err != nil {
|
||||
rows.Close()
|
||||
return
|
||||
}
|
||||
if !yield(j) {
|
||||
rows.Close()
|
||||
return // consumer stopped
|
||||
}
|
||||
}
|
||||
rows.Close()
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**File: `internal/store/node_repo.go`** — add analogous `Watch`:
|
||||
|
||||
```go
|
||||
func (r *NodeRepo) Watch(ctx context.Context) iter.Seq[*model.Node] {
|
||||
return func(yield func(*model.Node) bool) {
|
||||
ticker := time.NewTicker(1 * time.Second)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
}
|
||||
rows, err := r.db.QueryContext(ctx,
|
||||
`SELECT id, name, address, state, joined_at, last_seen, metadata
|
||||
FROM nodes ORDER BY joined_at ASC`)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
for rows.Next() {
|
||||
n, err := scanNode(rows)
|
||||
if err != nil {
|
||||
rows.Close()
|
||||
return
|
||||
}
|
||||
if !yield(n) {
|
||||
rows.Close()
|
||||
return
|
||||
}
|
||||
}
|
||||
rows.Close()
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Imports: add `"iter"` and `"time"` (time already present in both files). The
|
||||
`iter` package is imported only for the return type; `yield` is the callback.
|
||||
|
||||
**Note on the existing `scanner` interface:** `scanJob`/`scanNode` accept the
|
||||
`scanner` interface (`Scan(dest ...any) error`) satisfied by both `*sql.Row`
|
||||
and `*sql.Rows`, so they are directly reusable in `Watch` — no refactor needed.
|
||||
|
||||
### Implementation approach — CLI layer
|
||||
|
||||
**File: `internal/cli/job.go`** — modify `jobListCmd`:
|
||||
|
||||
1. Add a package var `jobWatch bool` and register `jobListCmd.Flags().BoolVar(&jobWatch, "watch", false, "stream jobs until Ctrl-C")`.
|
||||
2. In `RunE`, branch on `jobWatch`:
|
||||
- If `!jobWatch`: keep the existing 5s-timeout `List` path.
|
||||
- If `jobWatch`:
|
||||
- `ctx, cancel := signal.NotifyContext(cmd.Context(), os.Interrupt, syscall.SIGTERM); defer cancel()` (drop the 5s timeout).
|
||||
- `seq := store.NewJobRepo(db).Watch(ctx)`
|
||||
- If `jsonOutput`: stream one-line JSON per event (D-030). Maintain a `map[string][]byte` of last-seen JSON per job ID. On each yielded job, marshal compact JSON; if it differs from the stored bytes (or ID unseen), print `{"event":"update","job":{...}}\n` and update the map.
|
||||
- Else (table): on each tick, after collecting the full snapshot, compare against the previous snapshot (by hashing the rendered table string or by comparing the slice of `[]*model.Job` via reflect/cmp). If changed, emit `"\033[2J\033[H"` (clear) then the table header + rows. This gives a "top-like" refresh.
|
||||
|
||||
Because `iter.Seq` does not return an error, the watch path swallows
|
||||
per-tick query errors inside `Watch` (D-034). The CLI relies on `ctx.Done()`
|
||||
for termination.
|
||||
|
||||
**File: `internal/cli/node.go`** — analogous change to `nodeListCmd`:
|
||||
- Add `nodeWatch bool`, register `--watch` flag.
|
||||
- Branch in `RunE`; `seq := store.NewNodeRepo(db).Watch(ctx)` (open a fresh `openDB()` for the watch path; `registry.List` is not used for watch — go straight to the repo to get the iter).
|
||||
|
||||
**Note:** `nodeListCmd` currently goes through `nodeRegistry()` which wraps
|
||||
`NodeRepo` in `engine.NodeRegistry`. For watch, bypass the registry and use
|
||||
`store.NewNodeRepo(db).Watch(ctx)` directly — the registry adds no value for a
|
||||
read-only stream and would require a `Watch` passthrough method. This keeps the
|
||||
iter boundary clean in the store layer (D-025).
|
||||
|
||||
### Pitfalls & mitigations (P01)
|
||||
|
||||
| Pitfall | Mitigation |
|
||||
|---------|------------|
|
||||
| **Goroutine leak** if Watch spawned a goroutine. | It doesn't — the polling loop is inline in the pull callback (D-032). `defer ticker.Stop()` + `rows.Close()` on every exit path. |
|
||||
| **Rows cursor held open across yield** — if `yield` blocks (e.g. slow consumer), the `*sql.Rows` stays open and holds a SQLite read lock. | Yield is called per-row inside the `rows.Next()` loop; the consumer (`range`) is fast (prints to stdout). For safety, close rows immediately after the loop or on `yield==false`. The 1s tick cadence bounds how long a cursor is held. WAL mode (set in `store.Open`) allows concurrent reads, so this does not block writers. |
|
||||
| **No error channel from iter.Seq** — a transient DB error is invisible to the CLI. | Log inside Watch via a package-level slog default (`slog.Default().Warn(...)`) since the repo has no logger field today (D-034: add an optional `logger *slog.Logger` to JobRepo/NodeRepo, defaulting to `slog.Default()` in the constructors — minimal change). Continue to next tick rather than terminating. |
|
||||
| **ctx cancellation mid-query** — `QueryContext` returns an error; `rows.Next()` returns false. | Handled: the `select` on `ctx.Done()` returns before the next tick; an in-flight query is cancelled by the ctx. |
|
||||
| **Rapid re-render flicker** in table mode. | Clear-screen + full re-render on changed snapshot only (hash compare). Unchanged snapshots produce no output. |
|
||||
| **`--watch` + `--json` interleaving with slog stderr.** | slog writes to stderr; CLI output to stdout — no interleaving on stdout. Safe. |
|
||||
| **Test determinism** — ticker is 1s, tests would be slow/flaky. | Provide a test-only constructor `WatchWithInterval(ctx, d time.Duration)` OR make the interval a field on the repo set via an unexported option. Preferred: an unexported `watchInterval` package var defaulting to 1s, overridable from `internal/store` tests. See D-035. |
|
||||
| **Signal handling clobbers root signal handler.** | `signal.NotifyContext` with `os.Interrupt` returns a fresh ctx; the root `cobra.Command` does not install its own SIGINT handler, so no conflict. `defer cancel()` restores default behavior on exit. |
|
||||
|
||||
### Test strategy (P01)
|
||||
|
||||
**`internal/store/job_task_repo_test.go` (new file or appended):**
|
||||
- `TestJobRepoWatch_YieldsSnapshots`: insert 1 job, call `Watch` with a 10ms interval (via test hook), range over `seq` collecting into a slice, insert a 2nd job from a goroutine after 30ms, cancel ctx after 80ms, assert the 2nd job appeared in the collected slice. Use `context.WithTimeout` for cancellation.
|
||||
- `TestJobRepoWatch_StopsOnConsumerBreak`: range over `seq` and `break` after the first yield; assert the function returns (no hang) within a short deadline. This validates the `yield==false` path.
|
||||
- `TestJobRepoWatch_StopsOnCtxCancel`: cancel ctx; assert the range loop exits within 50ms.
|
||||
- `TestNodeRepoWatch_*`: mirror the above for nodes.
|
||||
|
||||
**`internal/cli/job_test.go` (new) / `node_test.go` (new) — if CLI tests exist; otherwise add:**
|
||||
- `TestJobListWatch_JSONStreaming`: spin a temp DB, insert a job, invoke the `jobListCmd.RunE` with `--watch --json` in a goroutine, insert a 2nd job, capture stdout for ~200ms, assert two JSON lines appear. Cancel via ctx.
|
||||
- `TestJobListWatch_TableRefresh`: assert clear-screen escape + table re-render on change.
|
||||
- These CLI tests are harder to make deterministic; prefer testing the store-layer Watch thoroughly and keep CLI watch tests to a smoke-level "produces output, exits on ctx.Done".
|
||||
|
||||
### Dependency check (P01)
|
||||
|
||||
- `iter` — Go 1.25 stdlib (`go.mod` declares `go 1.25.0`). ✅ no new dep.
|
||||
- `time`, `context`, `os/signal`, `syscall`, `encoding/json` — stdlib. ✅
|
||||
- No new go.mod dependencies required.
|
||||
|
||||
---
|
||||
|
||||
## P02: `orca doctor` network + db full implementation
|
||||
|
||||
Covers REQ-032 completion (network + db checks replacing `NetworkStub`/`DBStub`).
|
||||
|
||||
### Decisions
|
||||
|
||||
| ID | Decision | Confidence |
|
||||
|----|----------|------------|
|
||||
| **D-033** | Add a public `store.MigrationVersion(ctx, db) (string, error)` function (in `migrate.go` or a new `internal/store/migrate_query.go`) that returns the **highest applied migration filename** from `schema_migrations`. SQL: `SELECT name FROM schema_migrations ORDER BY name DESC LIMIT 1`. This is the source of truth for "migration version" — it reflects what `migrate()` actually applied. Returns `("", nil)` if no migrations applied (fresh empty table) and `("", sql.ErrNoRows)` is treated as empty. | 0.90 |
|
||||
| **D-034** | The doctor DB check opens its own `*sql.DB` via `store.Open(dbPath())` (reusing the CLI's `dbPath`) inside the check constructor closure, rather than receiving a shared handle. Rationale: doctor should be runnable whether the daemon is up or down; `store.Open` uses WAL so a concurrent daemon is fine. The check `defer db.Close()`. This avoids threading a `*sql.DB` through `doctor.Run`/`All()` and keeps the `Check.Run` signature stable. | 0.86 |
|
||||
| **D-035** | The doctor DB check runs `PRAGMA integrity_check` via `db.QueryRow("PRAGMA integrity_check")`. SQLite returns a single row with a TEXT value: `"ok"` on success, or a multi-line error description on failure. PASS if the value is `"ok"`; FAIL otherwise (with the first line of the message). Plus query the migration version (D-033); WARN if `schema_migrations` is empty (fresh/never-migrated db) — this is suspicious but not corrupt. | 0.92 |
|
||||
| **D-036** | Doctor network check sources peer addresses from the **`nodes` table** (`NodeRepo.List`), NOT from `engine.PeerRegistry` (which is in-memory and always empty at CLI time — see gap #1). For each node with `state != 'left'`, probe `https://<address>/healthz` over mTLS. | 0.88 |
|
||||
| **D-037** | For each peer, the network check builds an mTLS client via `transport.NewMTLSClient(certpaths.CACertPath(), serverName, certpaths.ServerCertPath(), certpaths.ServerKeyPath())`. `serverName` is derived as the node's `Name` (the SAN on a peer's server cert is its node name, per `security.GenerateCSR(nodeName, sans)` — confirmed in `integration_test.go:41` `GenerateCSR("test-server", ...)`. If the SAN uses a different value the probe will fail handshake, which is itself a useful diagnostic). `CAPath` is the local `ca.crt` (all peers share one CA per D-011). This presents the local node's client cert, satisfying the daemon's `RequireAndVerifyClientCert`. | 0.80 |
|
||||
| **D-038** | Network check result semantics: **zero peers registered** → `WARN` ("no peers registered; network check skipped") — not FAIL, because a single-node install legitimately has no peers. **A peer unreachable / handshake failed** → `FAIL` for that peer, aggregated to a single `network` check result that is FAIL if any peer failed, PASS if all peers probed OK, WARN if zero peers. Each peer's per-line outcome is folded into the message string (e.g. `PASS — 2/2 peers reachable; FAIL — peer node-b (host:port): tls handshake error`). | 0.85 |
|
||||
|
||||
### Implementation approach — db check
|
||||
|
||||
**File: `internal/store/migrate.go`** — add:
|
||||
|
||||
```go
|
||||
// MigrationVersion returns the filename of the most recently applied
|
||||
// migration, or "" if no migrations have been applied (empty db or
|
||||
// schema_migrations table missing). Used by `orca doctor db`.
|
||||
func MigrationVersion(ctx context.Context, db *sql.DB) (string, error) {
|
||||
var name string
|
||||
err := db.QueryRowContext(ctx,
|
||||
`SELECT name FROM schema_migrations ORDER BY name DESC LIMIT 1`).Scan(&name)
|
||||
if err == sql.ErrNoRows {
|
||||
return "", nil
|
||||
}
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("query migration version: %w", err)
|
||||
}
|
||||
return name, nil
|
||||
}
|
||||
```
|
||||
|
||||
(Add `"context"` import — already imported in migrate.go.)
|
||||
|
||||
**File: `internal/doctor/doctor.go`** — replace `DBStub()` with `DB()`:
|
||||
|
||||
```go
|
||||
// DB checks SQLite integrity and migration version (REQ-032).
|
||||
// It opens its own *sql.DB so it can run whether or not the daemon is up.
|
||||
func DB() Check {
|
||||
return Check{
|
||||
Name: "db",
|
||||
Description: "SQLite PRAGMA integrity_check + migration version",
|
||||
Run: func(ctx context.Context) (Result, string) {
|
||||
path := dbPath() // dbPath currently lives in internal/cli; see D-039
|
||||
db, err := store.Open(path)
|
||||
if err != nil {
|
||||
return ResultFail, fmt.Sprintf("open %s: %v", path, err)
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
// 1. integrity_check
|
||||
var integrity string
|
||||
if err := db.QueryRowContext(ctx, "PRAGMA integrity_check").Scan(&integrity); err != nil {
|
||||
return ResultFail, fmt.Sprintf("integrity_check query: %v", err)
|
||||
}
|
||||
if integrity != "ok" {
|
||||
first := strings.SplitN(integrity, "\n", 2)[0]
|
||||
return ResultFail, fmt.Sprintf("integrity_check: %s", first)
|
||||
}
|
||||
|
||||
// 2. migration version
|
||||
ver, err := store.MigrationVersion(ctx, db)
|
||||
if err != nil {
|
||||
return ResultFail, fmt.Sprintf("migration version: %v", err)
|
||||
}
|
||||
if ver == "" {
|
||||
return ResultWarn, "integrity ok; no migrations applied (fresh db?)"
|
||||
}
|
||||
return ResultPass, fmt.Sprintf("integrity ok; migrations up to %s", ver)
|
||||
},
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**D-039 (assumption, confidence 0.78):** `dbPath()` currently lives in
|
||||
`internal/cli/node.go` and is unexported. The `doctor` package cannot import
|
||||
`internal/cli` (would create a cycle: `cli` imports `doctor`). **Resolution:**
|
||||
move `dbPath()` (and the `ORCA_DB` env logic) into `certpaths` (rename the
|
||||
package conceptually, or add a sibling `internal/paths` package) OR duplicate
|
||||
the ~5-line `dbPath` function inside `internal/doctor`. The cleanest is to add
|
||||
`func DBPath() string` to `internal/certpaths/certpaths.go` (it already owns
|
||||
`Dir()` honoring `ORCA_HOME`) and have both `cli` and `doctor` call it.
|
||||
`cli.dbPath` becomes a thin wrapper or is replaced. This is a small refactor
|
||||
within P02's scope. Logged as D-039, confidence 0.78 (territory overlap between
|
||||
cli-engineer and the doctor package; lead-developer adjudicates).
|
||||
|
||||
### Implementation approach — network check
|
||||
|
||||
**File: `internal/doctor/doctor.go`** — replace `NetworkStub()` with `Network()`:
|
||||
|
||||
```go
|
||||
// Network probes each registered peer's /healthz over mTLS (REQ-032).
|
||||
// Peers are sourced from the nodes table. Zero peers => WARN (single-node
|
||||
// install is legitimate). Any peer unreachable => FAIL.
|
||||
func Network() Check {
|
||||
return Check{
|
||||
Name: "network",
|
||||
Description: "peer reachability via mTLS /healthz probe",
|
||||
Run: func(ctx context.Context) (Result, string) {
|
||||
// 1. Load registered nodes (skip 'left').
|
||||
path := certpaths.DBPath() // same resolution as D-039
|
||||
db, err := store.Open(path)
|
||||
if err != nil {
|
||||
return ResultFail, fmt.Sprintf("open db for node list: %v", err)
|
||||
}
|
||||
defer db.Close()
|
||||
nodes, err := store.NewNodeRepo(db).List(ctx)
|
||||
if err != nil {
|
||||
return ResultFail, fmt.Sprintf("list nodes: %v", err)
|
||||
}
|
||||
// filter out left nodes
|
||||
var live []*model.Node
|
||||
for _, n := range nodes {
|
||||
if n.State != model.NodeStateLeft {
|
||||
live = append(live, n)
|
||||
}
|
||||
}
|
||||
if len(live) == 0 {
|
||||
return ResultWarn, "no peers registered; network check skipped (single-node?)"
|
||||
}
|
||||
|
||||
caPath := certpaths.CACertPath()
|
||||
certPath := certpaths.ServerCertPath()
|
||||
keyPath := certpaths.ServerKeyPath()
|
||||
// Short per-probe timeout so one slow peer doesn't stall doctor.
|
||||
var lines []string
|
||||
overall := ResultPass
|
||||
for _, n := range live {
|
||||
probeCtx, cancel := context.WithTimeout(ctx, 3*time.Second)
|
||||
err := probeHealthz(probeCtx, caPath, certPath, keyPath, n.Name, n.Address)
|
||||
cancel()
|
||||
if err != nil {
|
||||
overall = ResultFail
|
||||
lines = append(lines, fmt.Sprintf("FAIL %s (%s): %v", n.Name, n.Address, err))
|
||||
} else {
|
||||
lines = append(lines, fmt.Sprintf("PASS %s (%s)", n.Name, n.Address))
|
||||
}
|
||||
}
|
||||
if overall == ResultPass {
|
||||
return ResultPass, fmt.Sprintf("%d/%d peers reachable: %s", len(live), len(live), strings.Join(lines, "; "))
|
||||
}
|
||||
return ResultFail, strings.Join(lines, "; ")
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// probeHealthz does a GET https://addr/healthz over mTLS.
|
||||
func probeHealthz(ctx context.Context, caPath, certPath, keyPath, serverName, addr string) error {
|
||||
client, err := transport.NewMTLSClient(caPath, serverName, certPath, keyPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("build mTLS client: %w", err)
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, "https://"+addr+"/healthz", nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("build request: %w", err)
|
||||
}
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return fmt.Errorf("probe: %w", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return fmt.Errorf("healthz status %d", resp.StatusCode)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
New imports in `doctor.go`: `net/http`, `strings`, `time`, `git.cloudinit.dev/coreci/orca/internal/store`, `git.cloudinit.dev/coreci/orca/internal/transport`, `git.cloudinit.dev/coreci/orca/internal/model`.
|
||||
|
||||
**File: `internal/doctor/doctor.go`** — update `All()`:
|
||||
```go
|
||||
func All() []Check {
|
||||
return []Check{
|
||||
CertCA(), CertServer(), CertExpiry(), CertFingerprint(),
|
||||
Network(), // was NetworkStub()
|
||||
DB(), // was DBStub()
|
||||
}
|
||||
}
|
||||
```
|
||||
Keep `NetworkStub`/`DBStub` exported functions for one release as thin
|
||||
wrappers that call the new ones? **No** — delete them; the CLI doctor.go
|
||||
references them and must be updated in lockstep (they are internal). See D-040.
|
||||
|
||||
**File: `internal/cli/doctor.go`** — update `doctorNetworkCmd` and `doctorDBCmd`:
|
||||
```go
|
||||
doctorNetworkCmd.RunE: c := doctor.Network() // was doctor.NetworkStub()
|
||||
doctorDBCmd.RunE: c := doctor.DB() // was doctor.DBStub()
|
||||
```
|
||||
Also: the per-subcommand render should honor `jsonOutput` (currently it only
|
||||
prints text). Minor enhancement, in scope.
|
||||
|
||||
### Pitfalls & mitigations (P02)
|
||||
|
||||
| Pitfall | Mitigation |
|
||||
|---------|------------|
|
||||
| **`model.Node` has no `ServerName`/`CAPath`** — mTLS needs `ServerName` to match the cert SAN. | Derive `ServerName = node.Name` (D-037). This assumes peer server certs are issued with SAN = node name, which matches `GenerateCSR(nodeName, sans)`. If a deployment uses DNS SANs instead, the probe fails — which is itself a diagnostic. Document this assumption in the check message. |
|
||||
| **No local client cert/key** — doctor can't present a client cert if `server.crt`/`server.key` are missing. | The check should FAIL with a clear message if `certpaths.ServerCertPath()` doesn't exist, BEFORE attempting probes. Reuse `os.Stat`. This also covers the single-node-never-joined case. |
|
||||
| **Daemon down** — peer's `/healthz` unreachable. | Per-probe 3s timeout (D-038). Surfaces as FAIL per peer with the dial/handshake error in the message. Doctor is designed to run with daemon up or down, so this is expected behavior, not a crash. |
|
||||
| **Self-probe** — the local node is likely in the `nodes` table too. Doctor will probe itself over mTLS. This is fine (validates the local daemon's mTLS stack) but requires the local daemon to be running. If the daemon is down, the self-probe fails → FAIL, which is the correct signal. | Document; no special-casing. |
|
||||
| **DB file doesn't exist** — `store.Open` creates the dir + file + runs migrations (so a missing db becomes a fresh empty db). The db check would then PASS with "no migrations applied" WARN. | This is acceptable: `store.Open` is idempotent. If the operator expected an existing db, the WARN surfaces the surprise. Could additionally `os.Stat` the path before `Open` and WARN if it didn't exist pre-open — optional refinement. |
|
||||
| **`PRAGMA integrity_check` can return multiple rows** in rare cases (when there are multiple errors). `QueryRow` only reads the first. | For `integrity_check`, a single row containing `"ok"` or the first error is the documented SQLite behavior for the common case. Use `QueryRow` + `Scan`; if it's not `"ok"`, that's already a FAIL. Acceptable. |
|
||||
| **Doctor test `TestRunAllChecksWithNoCA`** asserts WARN from stubs. | Update the test: with real checks, a no-CA scenario yields FAIL on cert.ca (unchanged) AND FAIL on db (open succeeds, integrity ok, but no migrations if fresh — actually WARN) AND WARN on network (no peers). Rewrite assertions to check each check by name rather than "hasWarn globally". See test strategy. |
|
||||
| **Import cycle:** `doctor` → `cli` (for `dbPath`). | Resolved by D-039: move `dbPath` to `certpaths` (or a new `internal/paths`); both `cli` and `doctor` import it. No cycle. |
|
||||
| **`store.Open` runs migrations on every open** — doctor opening the db to run integrity_check would also (re)migrate. | `migrate()` is idempotent (checks `schema_migrations` per name). Re-opening is safe; no-op if already migrated. Acceptable. |
|
||||
|
||||
### Test strategy (P02)
|
||||
|
||||
**`internal/store/migrate_test.go` (new or appended):**
|
||||
- `TestMigrationVersion`: open a fresh test db (which runs migrate), call `MigrationVersion`, assert it returns `0005_node_capacity.sql` (the highest current migration). Then manually delete all rows from `schema_migrations`, assert returns `""` with nil error.
|
||||
|
||||
**`internal/doctor/doctor_test.go` (update):**
|
||||
- Update `TestRunAllChecksWithNoCA`: set `ORCA_HOME` to temp dir (no CA). Expect: cert.ca FAIL, cert.server FAIL, cert.expiry FAIL, cert.fingerprint FAIL, **db WARN** (fresh db, no migrations — actually `store.Open` runs migrations, so db will PASS with version 0005; adjust: db PASS), **network WARN** (no peers). Rewrite to assert per-check rather than "hasWarn/hasFail globally". Remove the stale "expected WARN (stubs)" comment.
|
||||
- New `TestDBCheck_IntegrityOK`: open a fresh db via `store.Open` in temp, run `doctor.DB().Run(ctx)`, expect PASS and message contains "0005".
|
||||
- New `TestDBCheck_Corrupt`: open db, manually `db.Exec("DROP TABLE jobs")` to introduce inconsistency, run integrity_check — but `integrity_check` mostly detects corruption, not missing tables. More reliable: write garbage to the db file via raw file write, then open — `store.Open` may fail at Ping. Assert FAIL. (This test is brittle; prefer a unit test on the integrity string-parsing logic with a stub.)
|
||||
- New `TestNetworkCheck_NoPeers`: fresh db, no nodes, run `doctor.Network().Run(ctx)`, expect WARN.
|
||||
- New `TestNetworkCheck_PeerReachable`: this is an integration test — bootstrap a CA (`security.CAInit`), generate+sign a server cert with SAN `localhost`, start an `httptest.NewUnstartedServer` with `ts.TLS = serverTLS` and `ClientAuth = RequireAndVerifyClientCert` (mirror `security/integration_test.go:88-108`), generate+sign a client cert, insert a node row with `Address = ts.Listener.Addr().String()` and `Name = "localhost"`, set `ORCA_HOME` to the temp dir holding the CA + client cert, run `doctor.Network().Run(ctx)`, expect PASS. This reuses the proven pattern from `TestEndToEndMTLS`.
|
||||
- New `TestNetworkCheck_PeerUnreachable`: insert a node with `Address = "127.0.0.1:1"` (nothing listening), run, expect FAIL with the peer name in the message.
|
||||
|
||||
### Dependency check (P02)
|
||||
|
||||
- `net/http`, `crypto/tls` (via transport), `strings`, `time`, `context` — stdlib. ✅
|
||||
- `internal/transport`, `internal/store`, `internal/model`, `internal/certpaths` — existing internal packages. ✅
|
||||
- No new go.mod dependencies required.
|
||||
|
||||
---
|
||||
|
||||
## Cross-cutting decisions
|
||||
|
||||
| ID | Decision | Confidence |
|
||||
|----|----------|------------|
|
||||
| **D-039** | Move `dbPath()` (the `ORCA_DB`-honoring path resolver) from `internal/cli` to `internal/certpaths` as `DBPath()`, to break the would-be `doctor→cli` import cycle. Both `cli` and `doctor` then import `certpaths`. `certpaths` already owns the `ORCA_HOME`-honoring `Dir()`. Territory: this is a shared infra concern; `lead-developer` adjudicates. | 0.78 |
|
||||
| **D-040** | Delete `doctor.NetworkStub` and `doctor.DBStub` (no backward-compat shims). They are internal, referenced only by `internal/cli/doctor.go` which is updated in the same phase. Keeping dead stub code violates `no-redundant-implementations`. | 0.95 |
|
||||
| **D-041** | No new go.mod dependencies for v0.3. `iter` (P01) and mTLS health probe (P02) use stdlib + existing internal packages only. The 4 existing direct deps (cobra, hcl, modernc/sqlite, uuid) are unchanged. | 0.97 |
|
||||
| **D-042** | Phase ordering: P01 (iter.Seq) and P02 (doctor) are **independent** — no file is modified by both (P01 touches cli/job.go, cli/node.go, store repos; P02 touches doctor.go, cli/doctor.go, store/migrate.go, certpaths). They can be developed in either order or in parallel. Recommend P01 first only because it's the lower-risk change. | 0.85 |
|
||||
|
||||
---
|
||||
|
||||
## Summary of assumptions logged
|
||||
|
||||
All assumptions below are logged as decisions with confidence scores; none are
|
||||
flagged for human validation (full autonomy). Low-confidence (<0.80) items that
|
||||
warrant normal decision-flow attention:
|
||||
|
||||
- **D-030** (0.72): `--watch --json` emits one JSON object per changed element
|
||||
per tick (vs. one array per tick). REQ-030 wording is ambiguous; this
|
||||
interpretation matches "streaming one-line JSON per event" literally.
|
||||
- **D-037** (0.80): peer `ServerName` = node `Name` (SAN convention).
|
||||
- **D-039** (0.78): `dbPath` relocation to `certpaths` — territory overlap.
|
||||
|
||||
These three are escalated through the normal decision flow (DecisionEngine) per
|
||||
the researcher protocol, NOT flagged for human validation.
|
||||
@@ -1,161 +0,0 @@
|
||||
# Research: Orca v0.5 — Distribution
|
||||
|
||||
Research findings for the v0.5 Distribution milestone (install, namespace,
|
||||
docker, public releases). Conducted during P0 RESEARCH under full autonomy.
|
||||
|
||||
## R-001: Gitea Container Registry
|
||||
|
||||
**Source**: https://docs.gitea.com/usage/packages/container (Gitea 1.27.1 docs)
|
||||
|
||||
**Findings**:
|
||||
- Gitea ships a built-in OCI-compliant container registry.
|
||||
- Image naming convention: `{registry}/{owner}/{image}:{tag}`.
|
||||
For orca: `git.cloudinit.dev/coreci/orca:{tag}`.
|
||||
- Auth: `docker login git.cloudinit.dev` with username + personal access
|
||||
token (or password if no 2FA). The `GITEA_TOKEN` env var already used
|
||||
for release publishing works as the password.
|
||||
- Push: `docker push git.cloudinit.dev/coreci/orca:v0.4.4`.
|
||||
- Pull: anonymous pull works **if the repo is public** (REQ-045 flips
|
||||
this). For private repos, pull requires auth.
|
||||
- Tags are case-insensitive — use lowercase image names.
|
||||
- The registry supports multi-arch manifests via `docker buildx`.
|
||||
|
||||
**Implication for P03**: `scripts/release.sh` must add a `docker build`
|
||||
+ `docker login` + `docker push` step. The `.coreci.yml` release
|
||||
pipeline needs a `container-publish` step. Credential is `GITEA_TOKEN`
|
||||
(reused from the existing release flow — no new secret needed).
|
||||
|
||||
## R-002: `tea repos edit` — Repo Visibility
|
||||
|
||||
**Source**: `tea repos edit --help` (tea 0.14.1 installed locally)
|
||||
|
||||
**Findings**:
|
||||
- Command: `tea repos edit --private false --repo coreci/orca`
|
||||
- The `--private` flag accepts `true`/`false` (string, not bool).
|
||||
- Default login `bot` (cloudinit-bot) is already configured and is the
|
||||
default login. No extra auth needed.
|
||||
- The change is immediate and reversible (re-run with `--private true`).
|
||||
|
||||
**Implication for P0 ship**: Run this as an operational step during the
|
||||
P0 ship. Verify with unauth `curl` against the releases API afterward.
|
||||
|
||||
## R-003: Gitea Releases API — Asset Download URLs
|
||||
|
||||
**Source**: `/api/v1/repos/coreci/orca/releases/latest` (authed probe)
|
||||
|
||||
**Findings**:
|
||||
- Auth header format: `Authorization: token <GITEA_TOKEN>` (NOT basic
|
||||
auth — basic auth returns "invalid username, password or token").
|
||||
- Latest release endpoint: `GET /api/v1/repos/coreci/orca/releases/latest`
|
||||
→ JSON with `tag_name`, `name`, `body`, `assets[]`.
|
||||
- Each asset has `browser_download_url` — the direct download URL.
|
||||
- **Public access**: once the repo is public (R-002), the releases API
|
||||
and asset downloads work **without authentication**. This is what
|
||||
`install.sh` relies on (REQ-043).
|
||||
- Asset naming convention from existing releases:
|
||||
`orca-{version}-linux-amd64.tar.gz` (per `scripts/release.sh`).
|
||||
|
||||
**Implication for P02 install.sh**:
|
||||
1. Query `GET /api/v1/repos/coreci/orca/releases/latest` (unauth, post-R-002).
|
||||
2. Parse `tag_name` for the version.
|
||||
3. Find the asset with `name` matching `orca-{tag}-linux-{arch}.tar.gz`.
|
||||
4. Download `browser_download_url` with `curl -fsSL`.
|
||||
5. Extract and install.
|
||||
|
||||
## R-004: ORCA_HOME Propagation Points (Codebase Audit)
|
||||
|
||||
**Source**: `grep` for `UserHomeDir|os.Getenv("ORCA|\.orca` across `*.go`
|
||||
|
||||
**Findings** — exactly 3 production code sites determine the namespace
|
||||
root today:
|
||||
|
||||
| File | Current behavior | Needs change? |
|
||||
|------|-----------------|----------------|
|
||||
| `internal/certpaths/certpaths.go:21-26` | `Dir()` honors `ORCA_HOME` → `~/.orca` | **No** — this is the single source of truth. Already correct. |
|
||||
| `internal/store/store.go:13-19` | `Open("")` hardcodes `~/.orca/orca.db` (ignores `ORCA_HOME`) | **Yes** — route through `certpaths.DBPath()` instead. |
|
||||
| `internal/cli/init.go:16-22` | Hardcodes `~/.orca` via `os.UserHomeDir()` | **Yes** — route through `certpaths.Dir()`. |
|
||||
|
||||
All other call sites (`node.go:openDB`, `daemon.go`, `job.go`, `doctor.go`,
|
||||
`cert.go`) already go through `certpaths.DBPath()` or `certpaths.Dir()`
|
||||
indirectly. **No other files need changes for REQ-041.**
|
||||
|
||||
**For REQ-042 (`--system`)**: Add a `--system` persistent flag on
|
||||
`rootCmd`. When set, `rootCmd.PersistentPreRunE` sets
|
||||
`os.Setenv("ORCA_HOME", "/root/.orca")` before any subcommand runs.
|
||||
This is the minimal-touch approach — all downstream code already
|
||||
honors `ORCA_HOME`. The flag is a CLI convenience that maps to the
|
||||
env var, not a parallel mechanism.
|
||||
|
||||
**Backward compatibility**: empty `ORCA_HOME` + no `--system` →
|
||||
`~/.orca` (unchanged). Existing tests that `t.Setenv("ORCA_HOME", ...)`
|
||||
continue to work.
|
||||
|
||||
## R-005: Distroless Base Image for CGO-free Go Binaries
|
||||
|
||||
**Source**: Go module audit — `modernc.org/sqlite` (pure Go, CGO-free),
|
||||
`go.mod` has no CGO dependencies.
|
||||
|
||||
**Findings**:
|
||||
- `gcr.io/distroless/static-debian12` is the correct base for static
|
||||
Go binaries with no CGO and no libc dependency. ~2MB image.
|
||||
- orca uses `modernc.org/sqlite` (pure Go) — no CGO, no libc. ✓
|
||||
- Multi-stage Dockerfile:
|
||||
- Stage 1 (`golang:1.25`): build with `-trimpath -ldflags` (same as
|
||||
Makefile), output `bin/orca`.
|
||||
- Stage 2 (`gcr.io/distroless/static-debian12`): `COPY bin/orca /orca`,
|
||||
`ENTRYPOINT ["/orca"]`.
|
||||
- `CGO_ENABLED=0` must be set in the build stage to guarantee a static
|
||||
binary (Go defaults to CGO_ENABLED=1 on platforms with a C compiler).
|
||||
- The image runs as `nonroot` user by default in distroless — but orca
|
||||
writes to `~/.orca` (or `/root/.orca` for `--system`). For the
|
||||
container image, default `ORCA_HOME=/var/lib/orca` and document
|
||||
volume mount at that path.
|
||||
|
||||
**Implication for P03**: Dockerfile is ~15 lines. The `.coreci.yml`
|
||||
release pipeline adds a `docker build --build-arg VERSION=$VERSION -t
|
||||
git.cloudinit.dev/coreci/orca:$VERSION .` step + login + push.
|
||||
|
||||
## R-006: install.sh Conventions (curl|sh pattern)
|
||||
|
||||
**Source**: Common patterns from deno, rustup, homebrew installers.
|
||||
|
||||
**Findings**:
|
||||
- 1-liner: `curl -fsSL <url> | bash` (or `| bash -s -- --system`).
|
||||
- The script must be downloadable from a stable URL. orca's script
|
||||
lives at `scripts/install.sh` in the repo, accessible via
|
||||
`https://git.cloudinit.dev/coreci/orca/raw/branch/main/scripts/install.sh`
|
||||
(once repo is public per R-002).
|
||||
- Args passed via `bash -s -- --system --version v0.4.4`.
|
||||
- In-place update: detect existing binary at install path, read its
|
||||
version via `orca version --json` (parse `version` field), print
|
||||
"updated from X to Y", overwrite binary. **Never** touch the
|
||||
namespace dir (`~/.orca` or `/root/.orca`) — that's user state.
|
||||
- User-level default: `~/.local/bin/orca` (XDG-ish, on PATH on most
|
||||
modern distros). System-level: `/usr/local/bin/orca` (requires root).
|
||||
|
||||
**Implication for P02**: install.sh is ~80-100 lines of bash. Idempotent.
|
||||
Tested via a `scripts/install_test.sh` that mocks the download and
|
||||
verifies path selection + update-in-place.
|
||||
|
||||
## Pitfalls (P-001..P-003)
|
||||
|
||||
- **P-001**: `docker` may not be available in the CoreCI release
|
||||
pipeline container. The `.coreci.yml` release step uses
|
||||
`image: golang:1.25` which does NOT include docker. **Mitigation**:
|
||||
the release pipeline must use a `docker:dind` sidecar or a step image
|
||||
that has the docker CLI. Alternatively, `scripts/release.sh` handles
|
||||
docker publish only when run locally or in a CI step that has docker.
|
||||
The `.coreci.yml` container step must use an image with docker CLI
|
||||
(e.g., `catthehacker/docker:docker-latest` or a custom image).
|
||||
|
||||
- **P-002**: Making the repo public exposes git history including the
|
||||
pre-existing `.env` SHA-1 leak (commit `00127ce` documented the
|
||||
rotate-forward decision; `.gitleaks-baseline.json` suppresses it for
|
||||
scanning). The leak is a **non-secret** (the token was rotated). This
|
||||
is an accepted risk per the existing decision — no new action needed,
|
||||
but document it in the P0 ship commit.
|
||||
|
||||
- **P-003**: `CGO_ENABLED=0` must be explicit in the Dockerfile build
|
||||
stage. Without it, `go build` in `golang:1.25` may produce a
|
||||
dynamically-linked binary that won't run in distroless. Verified:
|
||||
orca has no CGO deps, but `CGO_ENABLED=0` is belt-and-suspenders.
|
||||
@@ -1,250 +0,0 @@
|
||||
# Research: Orca v0.6 — Node Bootstrap & Proxmox
|
||||
|
||||
Findings grounded in codebase analysis (8 key files read) + verified
|
||||
against `golang.org/x/crypto` v0.54.0 (probe built clean), Proxmox VE
|
||||
9.2.3 admin guide (§14.7-14.8 pveum + privileges), sudoers(5) man
|
||||
page (NOEXEC/NOPASSWD), and freedesktop.org os-release spec.
|
||||
|
||||
## A. SSH library — `golang.org/x/crypto/ssh`
|
||||
|
||||
### A.1 go.mod addition
|
||||
|
||||
```
|
||||
require golang.org/x/crypto v0.54.0
|
||||
```
|
||||
|
||||
Latest available, compatible with go 1.25. Transitive deps (verified
|
||||
by probe build):
|
||||
- `golang.org/x/crypto v0.54.0` (direct)
|
||||
- `golang.org/x/sys v0.47.0` (indirect — bumps from v0.42.0)
|
||||
- `golang.org/x/term v0.45.0` (indirect — pulled by ssh for PTY)
|
||||
|
||||
**3 module entries, 0 new heavy deps.** Matches D-030 minimal-deps
|
||||
rationale. `go.sum` gains ~6 lines.
|
||||
|
||||
### A.2 Minimal API surface
|
||||
|
||||
```go
|
||||
import (
|
||||
"crypto/ed25519"
|
||||
"crypto/rand"
|
||||
"crypto/x509"
|
||||
"encoding/pem"
|
||||
"net"
|
||||
"time"
|
||||
"golang.org/x/crypto/ssh"
|
||||
"golang.org/x/crypto/ssh/knownhosts"
|
||||
)
|
||||
```
|
||||
|
||||
Key functions:
|
||||
- `ssh.Dial(network, addr, config) (*ssh.Client, error)` — high-level dialer
|
||||
- `(*ssh.Client).NewSession() (*ssh.Session, error)`
|
||||
- `(*ssh.Session).CombinedOutput(cmd) ([]byte, error)` — run + capture
|
||||
- `ssh.ClientConfig{User, Auth, HostKeyCallback, Timeout}`
|
||||
- `ssh.Password(secret) ssh.AuthMethod` — password auth
|
||||
- `ssh.PublicKeys(signer) ssh.AuthMethod` — pubkey auth
|
||||
- `ssh.ParsePrivateKey(pem) (ssh.Signer, error)` — parse PKCS8 PEM (works with orca's existing key format)
|
||||
- `ssh.NewPublicKey(pub) (ssh.PublicKey, error)` + `ssh.MarshalAuthorizedKey(pub) []byte` — authorized_keys line
|
||||
- `ssh.FixedHostKey(key) ssh.HostKeyCallback` — strict pin (subsequent connects)
|
||||
- `knownhosts.New(path) (ssh.HostKeyCallback, error)` — TOFU via known_hosts file (cleaner than custom callback; avoids deprecated `InsecureIgnoreHostKey`)
|
||||
|
||||
### A.3 Ed25519 keygen (D-037)
|
||||
|
||||
Verified end-to-end: `ed25519.GenerateKey(rand.Reader)` →
|
||||
`x509.MarshalPKCS8PrivateKey(priv)` → PEM encode → `ssh.ParsePrivateKey`
|
||||
round-trips cleanly. `ssh.MarshalAuthorizedKey` produces valid
|
||||
`ssh-ed25519 AAAA...` line. **PKCS8 PEM (orca's existing format)
|
||||
parses with `ssh.ParsePrivateKey` — no OpenSSH-format marshaller
|
||||
needed.** Reuse `security.WriteKey`/`writeAtomic` for persistence.
|
||||
|
||||
### A.4 File upload — `cat > file` via session, NOT SFTP
|
||||
|
||||
SFTP lives in separate module `github.com/pkg/sftp` — would add a 4th
|
||||
direct dep beyond D-030. The only files orca uploads are:
|
||||
- `~orca/.ssh/authorized_keys` (1-line append)
|
||||
- `/etc/sudoers.d/orca` (few lines)
|
||||
|
||||
Both are text. Use `session.CombinedOutput` with heredoc / `tee -a`.
|
||||
Keeps everything within `x/crypto/ssh`.
|
||||
|
||||
### A.5 TOFU host-key handling (D-035)
|
||||
|
||||
Use `golang.org/x/crypto/ssh/knownhosts.New(path)` as the
|
||||
`HostKeyCallback`. On first connect, the callback writes the host key
|
||||
to `~/.orca/known_hosts` (OpenSSH format). On subsequent connects, it
|
||||
verifies and returns an error on mismatch. **Avoids
|
||||
`ssh.InsecureIgnoreHostKey` deprecation** — `knownhosts.New` handles
|
||||
both capture and verify in one callback. On host-key change
|
||||
(reinstall), fail closed with a clear error; operator runs
|
||||
`orca node key-reset <node>` (future) or manually edits `known_hosts`.
|
||||
|
||||
## B. `/etc/os-release` parsing (D-032)
|
||||
|
||||
### B.1 Confirmed `ID=` values
|
||||
|
||||
| Distro | `ID=` | `ID_LIKE=` | Verified |
|
||||
|--------|-------|-----------|----------|
|
||||
| Ubuntu | `ubuntu` | `debian` | ✅ (this host: Ubuntu 24.04) |
|
||||
| Debian | `debian` | — | ✅ (freedesktop spec) |
|
||||
| Alpine | `alpine` | — | ✅ (Alpine policy) |
|
||||
| Proxmox VE | `pve` | `debian` | ✅ (PVE ships own os-release) |
|
||||
|
||||
`VARIANT_ID` absent on all four target distros — not worth capturing
|
||||
for v0.6.
|
||||
|
||||
### B.2 Parsing approach
|
||||
|
||||
No Go stdlib helper. Trivial: `bufio.Scanner` +
|
||||
`strings.SplitN(line, "=", 2)` + strip surrounding quotes. ~15 lines.
|
||||
Returns `map[string]string`; read `ID` field. Fallback `"linux"` if
|
||||
file missing or `ID` absent (D-032). Read `/etc/os-release` first;
|
||||
fall back to `/usr/lib/os-release` for minimal containers. Unknown `ID`
|
||||
values stored verbatim (not masked) — `doctor os` can warn.
|
||||
|
||||
## C. Proxmox VE role & user management
|
||||
|
||||
### C.1 Realm: `orca@pam` (NOT `orca@pve`)
|
||||
|
||||
Confirmed by both researchers + PVE User Management docs: since
|
||||
`orca node join` SSHes in and creates a Linux system user via
|
||||
`useradd`, the PVE user must be `orca@pam` (PAM realm maps to host
|
||||
system users). `orca@pve` would require a separate PVE-internal
|
||||
password and interactive `-password` prompt over non-PTY SSH (hangs).
|
||||
`@pam` sidesteps both issues. **D-033 refined: `orca@pam`.**
|
||||
|
||||
### C.2 OrcaOperator PVE role — privilege set
|
||||
|
||||
Per D-033 (operator-confirmed): `VM.Audit`, `Datastore.AllocateSpace`,
|
||||
`SDN.Use`. This is a **minimal API-level role** — the actual management
|
||||
capability comes from the sudoers allowlist (sudo runs as root, bypassing
|
||||
PVE RBAC). The PVE role governs non-sudo API access (future REST client).
|
||||
|
||||
**Refinement from research**: `VM.Audit` covers containers (CTs) as well
|
||||
as VMs (both live under `/vms/{vmid}` path; no separate `CT.*` family).
|
||||
PVE 8→9: privilege set valid on both (no breaking changes to pveum or
|
||||
the core privilege names).
|
||||
|
||||
Researcher 2 proposed an expanded 21-privilege set for fuller API-level
|
||||
management. **Decision: keep D-033's 3-priv minimal set for v0.6** — the
|
||||
operator explicitly confirmed it, and the sudoers allowlist is the
|
||||
primary management path. The expanded set is noted as a v0.7+
|
||||
enhancement option if orca adds a direct PVE REST client.
|
||||
|
||||
### C.3 pveum command sequence (idempotent)
|
||||
|
||||
```bash
|
||||
# 1. Role — probe-then-add (pveum role add fails if exists)
|
||||
pveum role list | grep -q '^OrcaOperator' || \
|
||||
pveum role add OrcaOperator --privs "VM.Audit Datastore.AllocateSpace SDN.Use"
|
||||
|
||||
# 2. User — probe-then-add (maps to existing Linux system user)
|
||||
pveum user list | grep -q 'orca@pam' || \
|
||||
pveum user add orca@pam -comment "Orca automation user"
|
||||
|
||||
# 3. ACL — modify is idempotent (creates or updates)
|
||||
pveum acl modify / -user orca@pam -role OrcaOperator
|
||||
```
|
||||
|
||||
Flag syntax: both `-privs` and `--privs` work (Perl Getopt::Long). Use
|
||||
`--privs` (canonical). Privs are **space-separated** inside quotes
|
||||
(NOT comma-separated).
|
||||
|
||||
### C.4 sudoers file `/etc/sudoers.d/orca` (D-033 refined)
|
||||
|
||||
**Research refinement**: exclude `pvesh` from sudoers — `pvesh` can
|
||||
reach the `/nodes/{node}/execute` API endpoint which spawns shell
|
||||
commands server-side, bypassing sudo's `NOEXEC` tag. Keep `pct`/`qm`
|
||||
with `NOEXEC`; `apt-get`/`dpkg` without `NOEXEC` (they need to spawn
|
||||
child processes for maintainer scripts).
|
||||
|
||||
```
|
||||
# /etc/sudoers.d/orca — mode 0440, owner root:root
|
||||
# Orca automation: VM/CT management + package management, no shell escape
|
||||
orca ALL=(root) NOPASSWD: NOEXEC: /usr/bin/pct, /usr/bin/qm
|
||||
orca ALL=(root) NOPASSWD: /usr/bin/apt-get, /usr/bin/dpkg
|
||||
```
|
||||
|
||||
`NOEXEC` works via Linux seccomp (sudoers man page). `pct`/`qm` are
|
||||
Perl scripts run via dynamically-linked `/usr/bin/perl` → NOEXEC
|
||||
effective. `apt-get`/`dpkg` need exec for postinst scripts → no
|
||||
NOEXEC. File mode **0440** or sudo refuses to load. Validate with
|
||||
`visudo -cf /etc/sudoers.d/orca` after writing; abort bootstrap on
|
||||
validation failure.
|
||||
|
||||
**Resolve binary paths at runtime** via `command -v pct` etc. before
|
||||
writing the sudoers file (cheap insurance against non-standard installs).
|
||||
|
||||
### C.5 PVE 8 vs 9
|
||||
|
||||
No breaking changes to pveum, privilege names, or sudo defaults
|
||||
between 8 and 9. `VM.Monitor` removed in 9.0 (OrcaOperator doesn't
|
||||
use it). Privileged container creation needs `Sys.Modify` in 9.0
|
||||
(OrcaOperator doesn't have it → intended). Both versions: `orca@pam`
|
||||
flow identical. Binary paths identical (`/usr/bin/{pct,qm,pvesh}`).
|
||||
|
||||
## D. Codebase integration points (confirmed by reading files)
|
||||
|
||||
### D.1 Files to modify/create per requirement
|
||||
|
||||
| File | Change | REQ |
|
||||
|------|--------|-----|
|
||||
| `go.mod` / `go.sum` | Add `golang.org/x/crypto v0.54.0`; bump sys, add term | REQ-050 |
|
||||
| `internal/model/node.go` | Add `Kind`, `OS` string fields + `NodeKind` constants | REQ-049 |
|
||||
| `internal/store/migrations/0006_node_kind_os.sql` | **NEW**: `ALTER TABLE nodes ADD COLUMN kind TEXT; ADD COLUMN os TEXT;` (nullable, backward-compatible) | REQ-049 |
|
||||
| `internal/store/node_repo.go` | Extend INSERT/SELECT/scanNode for `kind, os`; add `GetByName`, `UpdateLastSeenAndOS` helpers | REQ-049 |
|
||||
| `internal/cli/init.go` | Full bootstrap: MkdirAll → store.Open (runs migrations) → CAInit → server cert gen (if absent) → detectOS → localhost node upsert | REQ-047,048 |
|
||||
| `internal/cli/node.go` | Add `--type`, `--host`, `--user`, `--password`, `--proxmox-user`, `--proxmox-role` flags; `bootstrapProxmox` branch | REQ-050,051 |
|
||||
| `internal/security/sshkey.go` | **NEW**: `GenerateOrLoadSSHKey(dir)` — Ed25519 keygen, PKCS8 PEM, 0600/0644 modes | REQ-050 |
|
||||
| `internal/proxmox/bootstrap.go` | **NEW package**: `BootstrapProxmox(ctx, opts)` — SSH dial, pubkey deploy, useradd, pveum role/user/acl, sudoers write, visudo validate | REQ-050,051 |
|
||||
| `internal/doctor/doctor.go` | Add `OS()` and `Proxmox()` checks; extend `All()` | REQ-052 |
|
||||
| `internal/cli/doctor.go` | Add `doctor os` + `doctor proxmox` subcommands | REQ-052 |
|
||||
| `internal/certpaths/certpaths.go` | Add `SSHKeyPath`, `SSHPubPath`, `KnownHostsPath` | REQ-050 |
|
||||
|
||||
### D.2 Reuse opportunities (confirmed)
|
||||
|
||||
- `security.CAInit` (ca.go:63) — **already idempotent** (fast-path loads existing). `orca init` calls it directly.
|
||||
- `security.GenerateCSR` (csr.go) — signature fits: `GenerateCSR("localhost", []string{"localhost","127.0.0.1"})`.
|
||||
- `security.WriteCert`/`WriteKey` (ca.go:292) — enforce 0644/0600 via `writeAtomic`; reuse for SSH key.
|
||||
- `store.Open` (migrate.go) — runs migrations on open; calling it in `orca init` auto-applies 0006.
|
||||
- Migration runner — FS-embedded, sorts lexicographically, idempotent per-file. Adding `0006_*.sql` is the entire change.
|
||||
- `doctor.Network()` (doctor.go:222) — exact pattern to clone for `doctor.Proxmox()` (list nodes, filter by kind, 3s timeout per peer, PASS/WARN/FAIL).
|
||||
|
||||
### D.3 No changes needed
|
||||
|
||||
- `internal/security/ca.go`, `csr.go` — idempotent already, signatures fit.
|
||||
- `internal/store/migrate.go` — runner is generic.
|
||||
- `internal/transport/*` — mTLS transport not involved in SSH bootstrap.
|
||||
- `internal/engine/*` — NodeRegistry.Join works; new fields are metadata.
|
||||
|
||||
## E. Pitfalls & gotchas
|
||||
|
||||
1. **`pveum` flag is `--privs` (space-separated)**, not `--privs "a,b,c"`. Confirmed by both researchers + official docs.
|
||||
2. **`orca@pam` not `orca@pve`** — PVE-internal realm requires interactive password prompt over non-PTY SSH (hangs). PAM realm maps to the Linux system user orca creates.
|
||||
3. **Exclude `pvesh` from sudoers** — `pvesh` can trigger API `execute` endpoint spawning shell commands server-side, bypassing `NOEXEC`. Use PVE API via OrcaOperator role for API access instead.
|
||||
4. **`NOEXEC` only on dynamically-linked binaries** — `pct`/`qm` are Perl scripts via dynamically-linked `/usr/bin/perl` → effective. `apt-get`/`dpkg` need exec → no NOEXEC.
|
||||
5. **sudoers file mode 0440** — or sudo silently refuses to load it. `chmod 0440` + `visudo -cf` validate after write.
|
||||
6. **Migration 0006 NULL handling** — `scanNode` must use `sql.NullString` for `kind`/`os` and map NULL → `""` (Go struct fields are `string`, not `*string`).
|
||||
7. **localhost node idempotency** — `NodeRepo.Insert` fails on UNIQUE constraint if `orca init` re-runs. Need `GetByName("localhost")` check first; if found, `UpdateLastSeenAndOS` instead of `Insert`. Don't change `id` or `joined_at` (D-036).
|
||||
8. **`orca init` must not regenerate server cert** (D-036) — check `certpaths.ServerCertPath()` existence before `GenerateCSR`. `CAInit` has a fast-path; server cert gen needs an explicit existence check.
|
||||
9. **Password handling (D-031)** — `--password` flag visible in `ps`/`/proc` briefly. Prefer `$ORCA_PROXMOX_PASSWORD` env var. Never log the password (slog redaction). Zero the byte slice after use.
|
||||
10. **`knownhosts.New` for TOFU** — avoids deprecated `ssh.InsecureIgnoreHostKey`. Handles both capture and verify in one callback.
|
||||
11. **PKCS8 PEM parses with `ssh.ParsePrivateKey`** — no need for OpenSSH-format marshaller. Consistent with `ca.key`/`server.key` format.
|
||||
12. **`/etc/os-release` is a symlink** on most distros → `os.ReadFile` follows it. Fall back to `/usr/lib/os-release` for minimal containers.
|
||||
|
||||
## F. Persona recommendations (v0.6 roster)
|
||||
|
||||
| Persona | Active | Reason |
|
||||
|---------|--------|--------|
|
||||
| `lead-developer` | ✅ | Coordination across P01/P02/P03; SSH/bootstrap touches security + cli + store + doctor |
|
||||
| `backend-engineer` | ✅ | Owns `internal/cli/init.go` full-bootstrap orchestration + `internal/proxmox/bootstrap.go` SSH logic |
|
||||
| `cli-engineer` | ✅ | Owns `--type`/`--host`/`--password` flag wiring, `doctor os`/`doctor proxmox` subcommands, init output UX |
|
||||
| `data-engineer` | ✅ **REACTIVATE** | Owns migration 0006 + `NodeRepo` schema extension (kind/os columns, new helpers) |
|
||||
| `security-engineer` | ✅ **REACTIVATE** | Owns `internal/security/sshkey.go`, TOFU host-key, sudoers design, password redaction, audit logging |
|
||||
| `devops-engineer` | ❌ **DEACTIVATE** | No install.sh/Dockerfile/.coreci.yml surface in v0.6 |
|
||||
| `network-engineer` | ❌ | No transport/mTLS surface (SSH is point-to-point bootstrap, not mesh) |
|
||||
| `frontend-engineer` | ❌ | No web UI |
|
||||
|
||||
**Territory overlaps to adjudicate (lead-developer)**:
|
||||
- `internal/proxmox/bootstrap.go` (security-engineer SSH/sudoers logic) vs `internal/cli/node.go` (cli-engineer flag wiring) — boundary: security package exposes `BootstrapProxmox(ctx, opts) error`, CLI just calls it.
|
||||
- `internal/doctor/doctor.go` `Proxmox()` reuses SSH client from `internal/proxmox` (security) but check scaffolding clones `doctor.Network()` pattern (backend adjudicates since network-engineer deactivated).
|
||||
@@ -1,161 +0,0 @@
|
||||
# Research: Orca v0.7 — Hardening & Completion
|
||||
|
||||
## 1. Codebase audit findings (RESEARCH stage)
|
||||
|
||||
A full codebase audit surfaced the gaps that define the v0.7 scope.
|
||||
Each finding is grounded in a specific file/coverage measurement.
|
||||
|
||||
### 1.1 `orca cert` command tree is unreachable (critical)
|
||||
|
||||
- `internal/cli/cert.go:44` exports `NewCommand(log *slog.Logger)
|
||||
*cobra.Command` which builds the full `cert ca-init | gen | show |
|
||||
renew | fingerprint` tree (5 subcommands, all implemented, all
|
||||
spec-compliant per REQ-033/035/036).
|
||||
- **No file in the repo calls `NewCommand` or registers it on
|
||||
`rootCmd`.** `grep -rn "rootCmd.AddCommand" internal/cli/` lists
|
||||
daemon, init, audit, version, job, node, doctor, status — `cert` is
|
||||
absent. `./bin/orca cert` returns `error: unknown command "cert"`.
|
||||
- The function is named `NewCommand` (not `newCertCmd`), so it is not
|
||||
picked up by any init-based registration convention.
|
||||
- **Impact**: every cert operation the spec promises (REQ-023, REQ-025,
|
||||
REQ-033, REQ-035, REQ-036) is unreachable from the CLI. Operators
|
||||
cannot bootstrap a CA, issue a server cert, or rotate one without
|
||||
hand-crafting calls into the `security` package. This is the single
|
||||
highest-impact bug in the v0.1–v0.6 line.
|
||||
- **Fix**: one-line `rootCmd.AddCommand(NewCommand(log))` in
|
||||
`internal/cli/cert.go` (or a new `init()`), plus a regression test
|
||||
that asserts `rootCmd.Commands()` contains a child whose `Use ==
|
||||
"cert"`.
|
||||
|
||||
### 1.2 `internal/store/cert_repo.go` has no test file
|
||||
|
||||
- `internal/store/cert_repo.go` exists (the `certs` table from
|
||||
migration 0004) but `internal/store/cert_repo_test.go` does not.
|
||||
- Every other repo in `internal/store/` has a `_test.go`:
|
||||
`node_repo_test.go`, `job_task_repo_test.go`, `capacity_repo_test.go`,
|
||||
`audit_repo_test.go`, `migrate_test.go`.
|
||||
- **Fix**: add `cert_repo_test.go` covering Insert/Get/List/rotation
|
||||
history (N=3 per REQ-025) + serial_hex uniqueness.
|
||||
|
||||
### 1.3 Low test coverage in core packages
|
||||
|
||||
| Package | Coverage | Missing tests for |
|
||||
|---------|----------|-------------------|
|
||||
| `internal/engine` | 8.3% | `executor.go`, `dispatcher.go`, `peer.go` (only `scheduler_test.go` exists) |
|
||||
| `internal/transport` | 26.3% | `mtls.go`, `dispatch.go`, `handshake_log.go` (only `idempotency_test.go` exists) |
|
||||
| `internal/proxmox` | 5.1% | `bootstrap.go` SSH path (only `bootstrap_test.go` exists, exercises the no-op dry-run) |
|
||||
| `internal/audit` | no test files | `audit.go` (Emit, EmitWithErr, LogHandshake*) |
|
||||
|
||||
- Target per D-042: 50% floor per package, 70% for new code in P02/P04.
|
||||
- Strategy: table-driven tests + `httptest.NewTLSServer` for transport;
|
||||
interface-based mocks for the SSH dialer (already an interface in
|
||||
`proxmox/bootstrap.go:211` `defaultSSHDialer` with `DialContext`).
|
||||
|
||||
### 1.4 No HCL config file parser
|
||||
|
||||
- D-009 specified `~/.orca/config.hcl` and `/etc/orca/orca.hcl` as
|
||||
config locations. `find . -name "*.hcl"` returns only testdata
|
||||
(`testdata/hello.hcl`, `testdata/fail.hcl`) used by jobspec tests.
|
||||
- The CLI relies entirely on flags + env vars (`ORCA_HOME`,
|
||||
`ORCA_DB`, `ORCA_PROXMOX_PASSWORD`). There is no `internal/config`
|
||||
package.
|
||||
- `internal/jobspec/spec.go:40` already uses
|
||||
`hclsimple.Decode(filename, data, nil, &spec)` — the exact same
|
||||
pattern works for a `Config` struct. No new dep required (hashicorp/hcl/v2
|
||||
is already a direct dep).
|
||||
- **Fix**: new `internal/config` package with a `Config` struct (HCL
|
||||
tags: `db_path`, `listen_addr`, `ca_path`, `server_cert_path`,
|
||||
`server_key_path`, `node_capacity`), a `Load(paths ...string)`
|
||||
function, and a `--config` flag on the root command. Precedence per
|
||||
D-039: flag > env > file > default.
|
||||
|
||||
### 1.5 pprof endpoint (I-308, deferred since v0.2)
|
||||
|
||||
- I-308 was deferred in v0.2 IDEATE ("keep v0.2 lean") and never
|
||||
revisited. The daemon (`internal/daemon/server.go`) has no pprof
|
||||
surface today.
|
||||
- `net/http/pprof` is stdlib — zero new deps. Mount on a separate
|
||||
`*http.ServeMux` so it never touches the mTLS daemon listener.
|
||||
- **Fix**: `--pprof <addr>` flag on `orca daemon` (default disabled).
|
||||
If set, start a second `http.Server` on `<addr>` with
|
||||
`pprof.Index`/`pprof.Cmdline`/etc. registered. Log a WARN that the
|
||||
endpoint is unauthenticated + operator-only.
|
||||
|
||||
## 2. Prior art & patterns
|
||||
|
||||
### 2.1 HCL config in HashiCorp tools
|
||||
|
||||
Nomad, Consul, and Terraform all use HCL for config with the same
|
||||
`hclsimple.Decode` + struct-tag pattern. The precedence model (flag >
|
||||
env > file > default) is the de-facto standard; Viper implements it but
|
||||
adds a large dep. Orca's `internal/config` will implement the 4-layer
|
||||
merge by hand (~80 LOC) to stay minimal-deps.
|
||||
|
||||
### 2.2 pprof in Go daemons
|
||||
|
||||
Standard pattern: `import _ "net/http/pprof"` registers handlers on
|
||||
`http.DefaultServeMux`. Best practice for production daemons is a
|
||||
**separate listener** (not DefaultServeMux) so pprof is never exposed
|
||||
on the public port. Orca will use a dedicated `*http.ServeMux` +
|
||||
`http.Server` on the `--pprof` addr, default disabled.
|
||||
|
||||
### 2.3 Test coverage for concurrent Go
|
||||
|
||||
`internal/engine` (executor, dispatcher) and `internal/transport`
|
||||
(mtls, dispatch) are concurrent. Coverage strategy:
|
||||
- `httptest.NewTLSServer` for transport — exercise real TLS handshakes
|
||||
against an in-process server.
|
||||
- Interface-based mocks for the SSH dialer (proxmox) and the peer
|
||||
client (transport) — both already have interface seams.
|
||||
- `sync.WaitGroup` + channel assertions for executor/dispatcher
|
||||
lifecycle.
|
||||
- `-race` is already on in CI (REQ-031) — new tests inherit it.
|
||||
|
||||
## 3. v0.7 Architectural Decisions (AD-022..AD-026)
|
||||
|
||||
| ID | Decision | Rationale |
|
||||
|----|----------|-----------|
|
||||
| AD-022 | `orca cert` registered via `init()` in `cert.go` calling `rootCmd.AddCommand(NewCommand(slog.Default()))` | Keeps registration co-located with the command definition; matches the pattern in `daemon.go`/`audit.go` where each command file self-registers. Avoids a central registration function that would drift. |
|
||||
| AD-023 | `internal/config` package: `Config` struct + `Load(paths ...string) (*Config, error)`; no global singleton | Config is passed explicitly to `daemon.NewServer`, `cli` commands, etc. No package-level state — testable, no init-order surprises. |
|
||||
| AD-024 | pprof on a separate `*http.Server` + `*http.ServeMux`, default disabled | Never co-mingles with the mTLS daemon listener. Operator opts in via `--pprof :6060`. Matches Go daemon best practice. |
|
||||
| AD-025 | Coverage floor measured per-package via `go test -cover ./<pkg>` | No aggregate threshold (aggregates hide low-coverage packages). CI gate added in P03: `go test -cover ./internal/engine ./internal/transport ./internal/proxmox ./internal/audit` and assert each ≥ 50%. |
|
||||
| AD-026 | No new direct dependencies in v0.7 | `net/http/pprof` (stdlib), `hashicorp/hcl/v2` (already direct). v0.7 preserves the minimal-deps ethos. |
|
||||
|
||||
## 4. PERSONAS assessment
|
||||
|
||||
v0.7 is an NFR milestone touching CLI, config, tests, and daemon. The
|
||||
default 3-persona roster (lead-developer, backend-engineer,
|
||||
data-engineer) is sufficient:
|
||||
|
||||
- **lead-developer**: owns P01 (cert registration) + P04 (pprof) — CLI/
|
||||
daemon territory.
|
||||
- **backend-engineer**: owns P02 (config package) — internal/config +
|
||||
CLI integration.
|
||||
- **data-engineer**: owns P01 cert_repo tests + P03 store coverage —
|
||||
`internal/store` territory.
|
||||
- **lead-developer** also owns P03 engine/transport/proxmox/audit
|
||||
coverage (test-only phase, no schema changes).
|
||||
|
||||
No new personas needed. No phase-specific personas. Territory
|
||||
enforcement stays `warn`. See `.ciagent/PERSONAS.md` (updated).
|
||||
|
||||
## 5. Dependencies
|
||||
|
||||
v0.7 adds **zero** new direct dependencies:
|
||||
- HCL parsing: `hashicorp/hcl/v2` (already direct, used by jobspec).
|
||||
- pprof: `net/http/pprof` (stdlib).
|
||||
- Tests: `net/http/httptest` (stdlib), existing interfaces.
|
||||
|
||||
`go.mod` is unchanged by v0.7.
|
||||
|
||||
## 6. Risks
|
||||
|
||||
- **P01 cert registration** may surface latent bugs in the cert
|
||||
subcommands (they've never been exercised end-to-end). Mitigation:
|
||||
P01 includes a smoke test that runs `cert ca-init` + `cert gen` +
|
||||
`cert show` + `cert fingerprint` against a temp `ORCA_HOME`.
|
||||
- **P02 config precedence** is easy to get wrong (flag/env/file/default
|
||||
merge order). Mitigation: table-driven test covering all 4 layers.
|
||||
- **P03 coverage** on concurrent packages may reveal race conditions
|
||||
(already hidden by the 8.3% coverage). Mitigation: `-race` is on; P03
|
||||
fixes any races it uncovers as part of the same phase.
|
||||
@@ -1,285 +0,0 @@
|
||||
# Research: Orca v0.8 — Coverage & Trust Hardening
|
||||
|
||||
Findings grounded in codebase analysis (44 source/test files read, coverage
|
||||
re-measured for all 9 target packages) + `golang.org/x/crypto` v0.54.0 API
|
||||
verification (`ssh.FingerprintSHA256`, `knownhosts.Line`/`Normalize`/`KeyError`).
|
||||
|
||||
## 1. Coverage analysis (P01 — REQ-057)
|
||||
|
||||
### 1.1 Re-measured coverage (confirmed via `go test ./<pkg>/... -cover`)
|
||||
|
||||
| Package | Coverage | Tier (D-047) | Notes |
|
||||
|---------|----------|--------------|-------|
|
||||
| `internal/engine` | **8.3%** | ≥ 70% floor | Only `scheduler_test.go` (4 tests, 66 LOC); executor/dispatcher/peer/registry/audit untested |
|
||||
| `internal/proxmox` | **5.1%** | ≥ 70% floor | Only `bootstrap_test.go` (4 tests, validation + sudoersContent string asserts); SSH dial path untested |
|
||||
| `internal/cli` | **27.6%** | ≥ 70% floor | 5 test files (root, init, namespace, osdetect, watch); node/job/cert/doctor/audit/cmds untested |
|
||||
| `internal/transport` | **26.3%** | ≥ 70% floor | Only `idempotency_test.go` (7 tests); mtls/dispatch/retry/handshake_log untested |
|
||||
| `internal/store` | **47.2%** | ≥ 70% floor | node_repo + job_task + capacity + audit + migrate tested; **cert_repo has NO test** (REQ-053 leftover — v0.7 P01 was supposed to add it but it's missing) |
|
||||
| `internal/jobspec` | **47.6%** | ≥ 70% floor | Only `spec_test.go` (4 tests); `Validate()`, `ParseFile` (file I/O), edge cases untested |
|
||||
| `internal/audit` | **0%** (no test files) | ≥ 50% toe-hold | `go: no such tool "covdata"` is a known tooling gap, NOT a real number — the package simply has no `_test.go` |
|
||||
| `internal/certpaths` | **0%** (no test files) | ≥ 50% toe-hold | Same `covdata` tooling gap; no `_test.go` exists |
|
||||
| `cmd/orca` | **0%** (no test files) | ≥ 50% toe-hold | Same; `main.go` is 15 LOC of glue (`cli.Execute()` + error print) |
|
||||
|
||||
**Coverage-floor achievability assessment (per package):**
|
||||
|
||||
- **engine → 70% REALISTIC.** The package has clean seams: `LocalExecutor` interface (dispatcher.go:39), `PeerRegistry` is in-memory with `Add`/`Remove`/`All`/`Get` (peer.go), `Executor.Submit/Status` take a `*store.JobRepo`+`*store.TaskRepo` which can be backed by `:memory:`/temp-file sqlite via the existing `openTestDB` helper (node_repo_test.go:12). The `sshDialer` seam pattern (proxmox) has an analogue here: `transport.NewDispatchClient` is called inside `dispatchToPeer` (dispatcher.go:158) — to test dispatch-to-peer without a real mTLS server, either (a) inject a fake `DispatchClient` via a new interface seam, or (b) use `httptest.NewTLSServer` with a self-signed CA. Option (a) is lower-effort and aligns with the `LocalExecutor` pattern. Recommendation: extract a `peerDispatcher` interface (`Submit(ctx, spec, key) (*SubmitResponse, error)`) and inject it, OR test via `LocalSubmit`/`LocalStatus` paths (which only need a stubbed `LocalExecutor`) — the latter covers ~60% of dispatcher.go without a new seam. **Flag: 70% may require a small refactor to inject the dispatch client; 60-65% is achievable without one. Plan should decide whether to add the seam or accept 65%.**
|
||||
- **proxmox → 70% REALISTIC.** The `sshDialer` seam already exists (bootstrap.go:201-213, `sshDialerType` interface + `defaultSSHDialer` struct, overridable package-level var). A fake SSH dialer returning a mock `*ssh.Client` is the path. **However:** `*ssh.Client.NewSession()` + `session.CombinedOutput()` are concrete methods on the real `*ssh.Client` — there's no `sshSession` interface seam. To test `runRemote`/`deployPubKey`/`createLinuxUser`/`createPVERole`/etc. without a real SSH server, EITHER (a) introduce a `sessionRunner` interface seam (small refactor), OR (b) use `httptest.NewTLSServer` is wrong (it's SSH not HTTP) — instead use a real in-process SSH server via `golang.org/x/crypto/ssh` `NewServerConn` (more code but no new dep). **Flag: 70% likely requires either a `sessionRunner` interface refactor OR an in-process SSH server fixture. 50-55% is achievable with just the existing `sshDialer` seam + testing validation paths + `sudoersContent` string asserts (already done). Plan should add the `sessionRunner` seam — it's a 1-interface, ~10-LOC change that unlocks the bulk of the package.**
|
||||
- **cli → 70% AMBITIOUS but realistic.** The package is the largest (17 source files, ~2000 LOC). The existing tests use `rootCmd.SetArgs()` + `rootCmd.Execute()` + `t.TempDir()` + `ORCA_HOME` env (namespace_test.go:46-53 — `TestInitHonorsORCAHOME` is the template). The untested commands are `node join/leave/list`, `job run/list/stop/logs`, `cert *`, `doctor *`, `audit list`, `status`, `version`, `daemon`. Many touch the DB + certpaths + (for `node join --type proxmox`) the SSH dialer. **Strategy:** table-driven `rootCmd.Execute()` against a temp `ORCA_HOME` for each subcommand; mock the proxmox path via the existing `sshDialer` seam; capture stdout via `rootCmd.SetOut(&buf)`. **Flag: 70% across the whole package is a lot of test code; 55-65% is more realistic for one phase. The `daemon` command (background server) is hard to test without a lifecycle harness — recommend excluding it from the 70% target and documenting why.**
|
||||
- **transport → 70% REALISTIC.** `httptest.NewTLSServer` is the standard seam (already used in `internal/daemon/dispatch_test.go:59` and `server_test.go`). The `Dispatcher` interface (dispatch.go:49) is already mockable (`stubDispatcher` in dispatch_test.go:24 is the template). `MTLSClient.Do` wraps `http.Client.Do` — testable via `httptest.NewTLSServer` with a CA + client cert. `retry.go` `Do[T]` is generic + already partly tested via `idempotency_test.go` (TestRetrySucceedsAfterTransient etc.) — extend with backoff-timing asserts. `handshake_log.go` is pure slog calls — trivial to test by capturing into a `slog.Handler`. **No new seams needed; 70% achievable.**
|
||||
- **store → 70% REALISTIC.** The existing `openTestDB` helper (node_repo_test.go:12) + `withFastWatch` (job_task_repo_test.go:36) are reusable. **Critical gap:** `cert_repo.go` has NO test file despite v0.7 P01 REQ-053 claiming it was added — this is a v0.7 leftover bug. Adding `cert_repo_test.go` (Insert/Get/List/ListByNode/LatestForKind/PruneOlderThan/Delete + N=3 rotation history per REQ-025) alone lifts coverage significantly. Job/Task repo `Watch` is tested; `ListRecent`, error paths, scan-edge cases need coverage. **No new seams; 70% achievable.**
|
||||
- **jobspec → 70% REALISTIC.** `Parse` + `Validate` + `ParseFile` are pure functions over HCL bytes. Add golden-file HCL fixtures (multi-task, env vars, args) + error-path table (missing job, no tasks, missing command, malformed HCL, empty file, nonexistent file for `ParseFile`). `testdata/` dir doesn't exist yet — create it. **No new seams; 70% achievable, likely the easiest of the six.**
|
||||
- **audit → 50% toe-hold REALISTIC.** Package is 125 LOC, 4 exported funcs (`New`, `Emit`, `EmitWithErr`, `LogHandshakeOK`, `LogHandshakeFailed`, `FormatAction`, `Action.String`, `Result.String`). Strategy: construct `Audit` with a real `engine.Audit` backed by `:memory:` sqlite (via `store.NewAuditRepo` + `engine.NewAudit`) + assert rows in `audit_log` table; capture slog output via a test `slog.Handler`. **No new seams; 50% easily achievable, 70% achievable if desired.**
|
||||
- **certpaths → 50% toe-hold TRIVIAL.** Package is 64 LOC, pure path-join functions honoring `ORCA_HOME`/`ORCA_DB` env. Strategy: temp dir + `t.Setenv("ORCA_HOME", dir)` + assert each `*Path()` returns `filepath.Join(dir, <file>)`; test `ORCA_DB` override; test default-to-`~/.orca` fallback. Model the test on `namespace_test.go` (cli). **No new seams; 50%+ trivially achievable.**
|
||||
- **cmd/orca → 50% toe-hold REALISTIC but LOW VALUE.** `main.go` is 15 LOC: `cli.Execute()` + `fmt.Fprintf(os.Stderr, "error: %v")` + `os.Exit(1)`. The only testable behavior is "main() calls Execute and exits non-zero on error." A smoke test that calls `main()` in a subprocess (or refactors main into a `run() int` for testability) is the path. **Flag: 50% on a 15-LOC glue file is ~7 lines of covered code — the effort:coverage ratio is poor. D-047 explicitly called this out ("0→70% risks a coverage rathole on `cmd/orca` which is glue code"). Recommend the plan keep this at the 50% toe-hold and not over-invest.**
|
||||
|
||||
### 1.2 Existing test-helper utilities (reuse, do NOT re-create)
|
||||
|
||||
| Helper | Location | Reuse for |
|
||||
|--------|----------|-----------|
|
||||
| `openTestDB(t)` | `internal/store/node_repo_test.go:12` | engine, audit, store tests — returns `(*NodeRepo, func())` backed by temp-file sqlite; adapt to return `*sql.DB` for JobRepo/TaskRepo/AuditRepo/CapacityRepo/CertRepo |
|
||||
| `withFastWatch(t, d)` | `internal/store/job_task_repo_test.go:36` | store Watch tests — overrides `watchInterval` for deterministic ticks |
|
||||
| `initTestEnv(t)` | `internal/cli/init_test.go:17` | cli tests — sets `ORCA_HOME` to temp dir + returns cleanup |
|
||||
| `resetRootFlags(t)` | `internal/cli/namespace_test.go:13` | cli tests — resets `rootCmd` args/out/json/system flags between subtests |
|
||||
| `discardWriter` | `internal/cli/init_test.go:33` | cli tests — `io.Writer` that discards stdout |
|
||||
| `stubDispatcher` | `internal/daemon/dispatch_test.go:24` | transport/engine tests — implements `transport.Dispatcher` (`LocalSubmit`/`LocalStatus`); reusable as a `LocalExecutor` too since the signatures match |
|
||||
| `insertNode(t, repo, ctx, id, name)` | `internal/store/node_repo_test.go:217` | store/doctor tests — inserts a minimal node |
|
||||
| `security.CAInit`/`LoadCA`/`GenerateCSR`/`SignCSR`/`WriteCert`/`WriteKey` | `internal/security/ca.go` | transport mTLS tests — bootstrap a real CA + server cert into a temp dir (pattern in `doctor_test.go:69-94`) |
|
||||
| `t.Setenv("ORCA_HOME", dir)` + `t.Setenv("ORCA_DB", ...)` | `internal/doctor/doctor_test.go:23-24` | any test needing the orca namespace — preferred over manual `os.Setenv` (auto-cleanup) |
|
||||
|
||||
### 1.3 Injected seams already present in the codebase (confirm by reading)
|
||||
|
||||
1. **`sshDialer` (proxmox)** — `internal/proxmox/bootstrap.go:201-213`: package-level `var sshDialer sshDialerType = defaultSSHDialer{}`; interface `sshDialerType{ DialContext(ctx, network, addr, *ssh.ClientConfig) (*ssh.Client, error) }`. Tests can swap `sshDialer` for a fake. **GAP:** no `sessionRunner` seam — `runRemote` (line 217) calls `conn.NewSession()` + `session.CombinedOutput(cmd)` directly on the concrete `*ssh.Client`. Recommend P01 plan add a `sessionRunner` interface (`CombinedOutput(cmd) ([]byte, error)`) so `deployPubKey`/`createLinuxUser`/`createPVERole`/`createPVEUser`/`assignPVEACL`/`writeSudoers`/`validateSudoers` become testable without a real SSH endpoint.
|
||||
2. **`LocalExecutor` (engine dispatcher)** — `internal/engine/dispatcher.go:39`: interface `Submit(ctx, []byte) (string, error)` + `Status(ctx, string) (string, error)`. `Dispatcher` depends on it; tests inject a stub. **GAP:** `dispatchToPeer` (line 154) calls `transport.NewDispatchClient` directly (no seam) — to test the remote-dispatch branch, either add a `peerDispatcher` interface or test via `httptest.NewTLSServer`.
|
||||
3. **`PeerPersister` (engine peer)** — `internal/engine/peer.go:39`: optional persist callback; unused in production but available as a seam.
|
||||
4. **`Dispatcher` (transport)** — `internal/transport/dispatch.go:49`: `LocalSubmit`/`LocalStatus` interface; `stubDispatcher` in `daemon/dispatch_test.go:24` is the template stub.
|
||||
5. **`watchInterval` (store)** — `internal/store/job_task_repo.go:20`: unexported `var watchInterval = 1 * time.Second`; tests override via `withFastWatch`.
|
||||
|
||||
### 1.4 Packages where 70% is unrealistic in a single phase (with evidence)
|
||||
|
||||
- **`internal/cli` — 70% is ambitious.** 17 source files, ~2000 LOC. The `daemon` command (`internal/cli/daemon.go`) starts a long-running mTLS server — testing it requires a lifecycle harness (start, probe, shutdown) and is better covered by `internal/daemon/server_test.go` (already exists, 150 LOC). Recommend the P01 plan **exclude `daemon.go` from the cli 70% target** (document it as covered by the daemon package's own tests) and aim for 70% of the *remaining* cli files. Even so, 55-65% is the realistic single-phase outcome for the rest.
|
||||
- **`cmd/orca` — 70% is explicitly out of scope per D-047.** 15 LOC of glue; 50% toe-hold is the right call.
|
||||
- **`internal/proxmox` — 70% likely requires the `sessionRunner` seam refactor.** Without it, only the validation paths + `sudoersContent` string asserts are testable (~50-55%). The plan should add the seam; with it, 70% is achievable.
|
||||
|
||||
---
|
||||
|
||||
## 2. SSH trust hardening research (P02 — REQ-058, REQ-059)
|
||||
|
||||
### 2.1 Current TOFU `knownhosts.New()` callback — how it works
|
||||
|
||||
**Location:** `internal/proxmox/bootstrap.go:125-128` (bootstrap) + `internal/doctor/doctor.go:412-415` (doctor proxmox probe).
|
||||
|
||||
```go
|
||||
hostKeyCallback, err := knownhosts.New(certpaths.KnownHostsPath())
|
||||
// ...
|
||||
sshConfig := &ssh.ClientConfig{
|
||||
HostKeyCallback: hostKeyCallback,
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
**Mechanism (`golang.org/x/crypto/ssh/knownhosts`):**
|
||||
- `knownhosts.New(files ...string)` returns an `ssh.HostKeyCallback` that reads the OpenSSH-format `known_hosts` file at `certpaths.KnownHostsPath()` (= `$ORCA_HOME/known_hosts`, see `internal/certpaths/certpaths.go:62`).
|
||||
- **First connect (host absent from file):** the callback returns a `*knownhosts.KeyError` with `Want: []` (empty). This is a "host unknown" signal. **IMPORTANT:** `knownhosts.New` does NOT auto-write the key on first connect — it returns an error. The current orca code at `bootstrap.go:140` treats ANY dial error as a failure (`return nil, fmt.Errorf("ssh dial %s: %w", sshAddr, err)`). **This means the current TOFU flow is INCOMPLETE:** on a truly first connect, `knownhosts.New` returns `KeyError{Want:[]}` and the dial fails — there is no capture-and-persist step. The v0.6 RESEARCH_v0.6.md §A.5 claimed `knownhosts.New` "handles both capture and verify in one callback" but the actual `golang.org/x/crypto` API does NOT auto-capture; it only verifies. **This is a latent bug OR the operator is expected to pre-populate `known_hosts` manually (which contradicts the TOFU UX).** P02 must address this: either (a) wrap `knownhosts.New` with a custom callback that captures on `KeyError{Want:[]}` and writes via `knownhosts.Line`, or (b) accept that `--host-key-fingerprint` (REQ-058) becomes the *required* path for first connect and TOFU capture is a separate enhancement. **Flag for plan: the current TOFU capture is broken; P02 should fix it as part of the trust-hardening work (the `--host-key-fingerprint` path is actually simpler than TOFU because it doesn't need capture).**
|
||||
- **Subsequent connects (host present, key matches):** callback returns `nil` → dial proceeds.
|
||||
- **Subsequent connects (host present, key MISMATCH):** callback returns `*knownhosts.KeyError{Want: [knownKey]}` → dial fails with a clear error. This is the MITM-detection path.
|
||||
|
||||
**File format:** OpenSSH `known_hosts` — one line per host: `[host]:port ssh-key-type base64-key` (or hashed-host form via `knownhosts.HashHostname`). `knownhosts.Line(addresses []string, key ssh.PublicKey) string` produces the line; `knownhosts.Normalize(address)` normalizes the host:port.
|
||||
|
||||
### 2.2 `Result.HostKeyFingerprint` — current computation (CRITICAL FINDING)
|
||||
|
||||
**Location:** `internal/proxmox/bootstrap.go:83-85` (field declaration) + `bootstrap.go:195-198` (return statement).
|
||||
|
||||
```go
|
||||
type Result struct {
|
||||
NodeName string
|
||||
NodeAddress string
|
||||
HostKeyFingerprint string // field EXISTS
|
||||
}
|
||||
// ...
|
||||
return &Result{
|
||||
NodeName: opts.Host,
|
||||
NodeAddress: opts.Host + ":8443",
|
||||
// HostKeyFingerprint is NOT SET — always empty string
|
||||
}, nil
|
||||
```
|
||||
|
||||
**Finding:** `Result.HostKeyFingerprint` is **declared but never populated**. The current `BootstrapProxmox` returns it as `""`. There is **no fingerprint computation today** — no `ssh.FingerprintSHA256` call, no hex digest, nothing. D-045's rationale ("matches the fingerprint format operators already see from `orca node join`'s own `Result.HostKeyFingerprint` output") is based on a field that is currently always empty.
|
||||
|
||||
**Implication for P02:** The plan must ADD the fingerprint computation. The correct function is `ssh.FingerprintSHA256(pubKey ssh.PublicKey) string` (verified via `go doc`), which returns the **OpenSSH `SHA256:base64` format** (unpadded base64, exactly what `ssh-keyscan -E sha256` emits and what D-045 specifies). So D-045's format choice is correct *by intent* but the code doesn't produce it yet — P02 populates `Result.HostKeyFingerprint = ssh.FingerprintSHA256(hostKey)` during the capture path, and `--host-key-fingerprint` compares against `ssh.FingerprintSHA256` of the server-presented key.
|
||||
|
||||
**No existing fingerprint-comparison utility in `internal/security/`.** `security.Fingerprint` (fingerprint.go:17) computes SHA-256 **hex** of an X.509 cert's DER — a DIFFERENT format (hex, not base64; X.509, not SSH). `security.FingerprintOf` (fingerprint.go:34) is the same. **Do NOT reuse these for SSH host-key comparison** — they're for the mTLS CA pin (`--ca-fingerprint`). P02 needs a new SSH-specific helper, e.g. `security.SSHFingerprintSHA256(pubKey ssh.PublicKey) string` (thin wrapper over `ssh.FingerprintSHA256`) or inline in `proxmox/bootstrap.go`.
|
||||
|
||||
### 2.3 Where `--host-key-fingerprint` plugs in (REQ-058)
|
||||
|
||||
**CLI seam:** `internal/cli/node.go:344-354` — the `init()` registers flags on `nodeJoinCmd`. Add:
|
||||
```go
|
||||
nodeJoinCmd.Flags().StringVar(&joinHostKeyFP, "host-key-fingerprint", "", "SSH host key SHA256:base64 fingerprint (pre-pin; supersedes TOFU for --type proxmox)")
|
||||
```
|
||||
Per D-044, the flag lives on `orca node join` (not just `--type proxmox`); validation in `RunE` (`node.go:78-83`) emits a clear error if the flag is set for a non-proxmox type.
|
||||
|
||||
**Transport seam:** `internal/proxmox/bootstrap.go:131-136` — `ssh.ClientConfig.HostKeyCallback`. Currently `knownhosts.New(...)`. When `--host-key-fingerprint` is supplied, replace the callback with a `ssh.FixedHostKey`-style verifier that:
|
||||
1. Parses the operator-supplied `SHA256:base64` string (strip `SHA256:` prefix, base64-decode → 32 bytes).
|
||||
2. In the callback, receives the server's `ssh.PublicKey`, computes `ssh.FingerprintSHA256(key)`, compares to the operator string.
|
||||
3. Returns `nil` on match, `error` on mismatch (fail closed).
|
||||
|
||||
**Recommended callback shape (concrete):**
|
||||
```go
|
||||
func pinnedHostKeyCallback(expectedSHA256Base64 string) (ssh.HostKeyCallback, error) {
|
||||
// Validate format: must start with "SHA256:".
|
||||
if !strings.HasPrefix(expectedSHA256Base64, "SHA256:") {
|
||||
return nil, fmt.Errorf("host-key-fingerprint: must be OpenSSH SHA256:base64 format (got %q)", expectedSHA256Base64)
|
||||
}
|
||||
expected := expectedSHA256Base64 // store full string for direct compare
|
||||
return func(_ string, _ net.Addr, key ssh.PublicKey) error {
|
||||
got := ssh.FingerprintSHA256(key)
|
||||
if got != expected {
|
||||
return fmt.Errorf("host key fingerprint mismatch: got %s, want %s — refusing to connect (REQ-058)", got, expected)
|
||||
}
|
||||
return nil
|
||||
}, nil
|
||||
}
|
||||
```
|
||||
**Why compare full strings (not base64-decoded bytes):** `ssh.FingerprintSHA256` returns the canonical `SHA256:base64` string; comparing it directly to the operator-supplied string is simplest and avoids a base64-decode step. Reject non-`SHA256:`-prefixed input up front with a clear error (D-045: "Accept only `SHA256:`-prefixed base64; reject raw hex with a clear error").
|
||||
|
||||
**Pass-through to proxmox:** `internal/cli/node.go:158-166` — add `HostKeyFingerprint string` to `proxmox.Options` (bootstrap.go:55) and pass `joinHostKeyFP` through. `BootstrapProxmox` selects the callback: if `opts.HostKeyFingerprint != ""` use `pinnedHostKeyCallback`, else fall back to the TOFU `knownhosts.New` (with the capture-fix from §2.1).
|
||||
|
||||
### 2.4 `orca node key-reset <node>` (REQ-059, D-046 — local known_hosts only)
|
||||
|
||||
**Scope (D-046):** clear the local `~/.orca/known_hosts` entry for the node ONLY; do NOT revoke the remote authorized_keys entry (would orphan a working node). Audit-log `event=node.key_reset` with `actor` + `node`.
|
||||
|
||||
**`known_hosts` line format written by `golang.org/x/crypto/ssh/knownhosts`:**
|
||||
- `knownhosts.Line(addresses []string, key ssh.PublicKey) string` → `"[host]:port ssh-ed25519 AAAA...\n"` (or `host ssh-ed25519 AAAA...` if port 22 — `knownhosts.Normalize` handles the `:22` vs bare-host normalization).
|
||||
- The file is plain text, one entry per line, `#`-prefixed comments allowed.
|
||||
|
||||
**No library function to remove a host's entries.** `knownhosts.New` only reads. The reset must be implemented manually:
|
||||
1. Read `certpaths.KnownHostsPath()` (`internal/certpaths/certpaths.go:62`).
|
||||
2. Filter lines: keep lines whose host field (before the first whitespace) does NOT match `knownhosts.Normalize(nodeName)` (or the node's address). **Edge:** a host may have multiple entries (one per key type); remove all matching lines.
|
||||
3. Write the filtered content back via **atomic rewrite** (temp file in same dir + `os.Rename`) — reuse `security.writeAtomic` (ca.go:305) OR implement inline (it's unexported in `security`; either export it or copy the ~20-LOC pattern). **Recommend atomic rewrite, NOT in-place truncation** — in-place rewrite via `os.OpenFile(O_TRUNC|O_WRONLY)` risks data loss on crash mid-write.
|
||||
|
||||
**CLI registration seam:** `internal/cli/node.go:358-360` — the `init()` does `nodeCmd.AddCommand(nodeJoinCmd)`, `nodeLeaveCmd`, `nodeListCmd`. Add:
|
||||
```go
|
||||
nodeCmd.AddCommand(nodeKeyResetCmd)
|
||||
```
|
||||
where `nodeKeyResetCmd` is a new `&cobra.Command{Use: "key-reset <node>", Args: cobra.ExactArgs(1), RunE: ...}`. The `RunE`:
|
||||
1. Resolve `<node>` arg → look up the node in the registry (`nodeRegistry()` at node.go:37) to get its address (for matching `known_hosts` lines) — OR accept the raw host string directly. **Recommend:** accept the node NAME (consistent with `doctor proxmox` which iterates `node.Name`), look up the node row, use `node.Name` (which is the host address for proxmox nodes per `bootstrap.go:196`) as the `known_hosts` match key.
|
||||
2. Call a new `proxmox.ResetHostKey(host string) error` (or inline in cli) that does the atomic rewrite.
|
||||
3. Audit-log via `engine.Audit.Record(ctx, "cli", "node.key_reset", nodeID, "success", nil, map[string]any{"host": host})`.
|
||||
4. Print `✓ Host key reset for <node> (next connect will re-pin via TOFU or --host-key-fingerprint)`.
|
||||
|
||||
**Reusability:** the `nodeRegistry()` helper (node.go:37) + `openDB()` (node.go:25) + `newLogger()` (node.go:33) are all available for the key-reset command.
|
||||
|
||||
### 2.5 CLI registration seam summary (P02)
|
||||
|
||||
| Addition | File:line | Change |
|
||||
|----------|-----------|--------|
|
||||
| `--host-key-fingerprint` flag | `internal/cli/node.go:344-354` (init) | `nodeJoinCmd.Flags().StringVar(&joinHostKeyFP, "host-key-fingerprint", "", "...")` |
|
||||
| `joinHostKeyFP` var | `internal/cli/node.go:47-60` (var block) | add `joinHostKeyFP string` |
|
||||
| Pass-through to proxmox | `internal/cli/node.go:158-166` (joinProxmox) | add `HostKeyFingerprint: joinHostKeyFP` to `proxmox.Options` |
|
||||
| `HostKeyFingerprint` field | `internal/proxmox/bootstrap.go:55` (Options) | add field |
|
||||
| Pinned callback | `internal/proxmox/bootstrap.go:131-136` | branch: if `opts.HostKeyFingerprint != ""` use pinned callback else TOFU |
|
||||
| Populate `Result.HostKeyFingerprint` | `internal/proxmox/bootstrap.go:195-198` | set `HostKeyFingerprint: ssh.FingerprintSHA256(hostKey)` during capture |
|
||||
| `key-reset` subcommand | `internal/cli/node.go:358-360` (init) | `nodeCmd.AddCommand(nodeKeyResetCmd)` + new cmd var |
|
||||
| `ResetHostKey` helper | `internal/proxmox/bootstrap.go` (new) OR `internal/security/sshkey.go` | atomic known_hosts rewrite |
|
||||
|
||||
---
|
||||
|
||||
## 3. Requirements-hygiene gate research (P03 — REQ-060)
|
||||
|
||||
### 3.1 Current Makefile targets
|
||||
|
||||
`Makefile` has 11 targets: `build`, `test`, `test-race`, `lint`, `fmt`, `clean`, `run`, `version`, `changelog`, `release`, `security-scan` (Makefile:1-100). **No `verify-reqs` target exists.** The `.PHONY` list at line 1 must be extended.
|
||||
|
||||
### 3.2 Current `.coreci.yml` pipeline structure
|
||||
|
||||
4 pipelines (`.coreci.yml:19-134`):
|
||||
- **validate** (line 20): 4 steps — `go-version` (gofmt+vet), `gosec`, `govulncheck`, `gitleaks`.
|
||||
- **build** (line 53): 1 step — version-injected `go build`.
|
||||
- **test** (line 72): 1 step — `go test -race -coverprofile=coverage.out ./...` + `go tool cover -func | tail -1`.
|
||||
- **release** (line 81): gated on `refs/tags/v*`; 3 steps — build-artifact, gitea-release, container-publish.
|
||||
|
||||
**Hook for `verify-reqs`:** add a 5th step to the `validate` pipeline (after `go-version`, before/after `gosec`) OR add it to the `test` pipeline. **Recommend `validate` pipeline** — requirements hygiene is a static check (no test run needed), belongs alongside gofmt/vet/lint. Step shape:
|
||||
```yaml
|
||||
- name: verify-reqs
|
||||
image: golang:1.25
|
||||
commands:
|
||||
- make verify-reqs
|
||||
```
|
||||
|
||||
### 3.3 `verify-reqs` implementation recommendation
|
||||
|
||||
**Assertion (REQ-060):** every REQ row in `ROADMAP.md` marked `[x]`/Complete must have a matching REQ-ID row in `REQUIREMENTS.md` with `Complete` status. (Reverse direction — every REQUIREMENTS `Complete` has a ROADMAP `[x]` — is also worth checking but the drift that motivated this was ROADMAP-shipped-but-REQUIREMENTS-Pending, so the forward direction is the priority.)
|
||||
|
||||
**Approach: small Go program in `cmd/verify-reqs` OR a shell+awk script?**
|
||||
|
||||
- **Go program** (~80 LOC): parse both markdown tables with `regexp`, build two `map[string]string` (REQ-ID → status), diff. Pros: type-safe, testable, consistent with the Go toolchain; can be a `cmd/verify-reqs/main.go` with its own `_test.go`. Cons: adds a binary target.
|
||||
- **Shell+awk** (~30 LOC): `awk` over the markdown tables. Pros: no new Go package; minimal. Cons: fragile parsing, hard to test, shell-quoting issues.
|
||||
|
||||
**Recommendation: Go program at `cmd/verify-reqs/main.go`.** Reasons: (1) testable with golden-file fixtures (parse a sample ROADMAP+REQUIREMENTS pair, assert diff); (2) consistent with the project's Go-only tooling ethos (no shell-awk fragility); (3) the `make verify-reqs` target just calls `go run ./cmd/verify-reqs`; (4) CoreCI's `golang:1.25` image has `go` available — no extra dep.
|
||||
|
||||
**Parsing approach (concrete):**
|
||||
1. ROADMAP.md: regex `^\s*-\s*\[(x|X| )\]\s*Phase.*—.*tag` is NOT the right pattern (that's phase lines, not REQ rows). The REQ coverage is in per-phase bullet lists under "### Per-phase REQ coverage" (ROADMAP.md:161-180) AND in the milestone section bodies. **Simpler:** the ROADMAP uses `- [x] Phase N: ...` for completed phases. The authoritative REQ↔status mapping lives in **REQUIREMENTS.md** (the single table at lines 9-56 + per-milestone tables at 103-142). **Re-interpret REQ-060:** the assertion is really "ROADMAP milestone sections marked COMPLETE ↔ REQUIREMENTS rows for that milestone marked Complete." The drift was: v0.7 ROADMAP said "COMPLETE" (line 116) but REQUIREMENTS v0.7 rows (REQ-053..056) were "Pending" (now corrected to "Complete" in SPECIFY).
|
||||
2. **Refined assertion:** parse REQUIREMENTS.md table rows (`| REQ-XXX | ... | ... | ... | **Complete** |` or `| Pending |`); for each REQ-ID, record status. Then parse ROADMAP.md for milestone-level "COMPLETE" markers (`## Milestone v0.X: ... — **COMPLETE**`) AND phase-level `- [x]` markers. For each milestone marked COMPLETE in ROADMAP, assert every REQ-ID belonging to that milestone (per the REQUIREMENTS milestone column) is `Complete` in REQUIREMENTS. **OR (simpler, matches the SPECIFY wording):** for every REQ-ID in REQUIREMENTS.md whose `Phase` column references a milestone that ROADMAP marks COMPLETE, the Status must be `Complete`. This catches the exact drift (ROADMAP-shipped, REQUIREMENTS-stale).
|
||||
|
||||
**Concrete regex:**
|
||||
- REQUIREMENTS row: `^\|\s*(REQ-\d+)\s*\|.*?\|\s*\*\*(Complete|Pending)\*\*\s*\|` (capture ID + status).
|
||||
- ROADMAP milestone-complete: `^##\s*Milestone\s+v0\.\d+:.*—\s*\*\*COMPLETE\*\*` (capture milestone label).
|
||||
- Map milestone → REQs via the REQUIREMENTS `Phase` column (e.g. `v0.7 P1` → milestone `v0.7`).
|
||||
|
||||
**Where it hooks in:** `make verify-reqs` runs `go run ./cmd/verify-reqs .ciagent/ROADMAP.md .ciagent/REQUIREMENTS.md`; `.coreci.yml` validate pipeline adds the step. Exit 0 on consistency, exit 1 with a diff listing on drift.
|
||||
|
||||
### 3.4 The drift that motivated REQ-060
|
||||
|
||||
After v0.7 ship, REQUIREMENTS.md rows REQ-053..056 were "Pending" despite ROADMAP.md marking milestone v0.7 COMPLETE and all phases `[x]`. This was corrected during v0.8 SPECIFY (the rows now read `**Complete**`). REQ-060 ensures the drift cannot recur: the CI validate pipeline fails if ROADMAP says COMPLETE but REQUIREMENTS says Pending.
|
||||
|
||||
---
|
||||
|
||||
## 4. Architectural decisions surfaced (AD-027..AD-030)
|
||||
|
||||
| ID | Decision | Rationale |
|
||||
|----|----------|-----------|
|
||||
| AD-027 | `ssh.FingerprintSHA256` (OpenSSH `SHA256:base64`) as the SSH host-key fingerprint format | Matches D-045 + `ssh-keyscan -E sha256` output. The existing `security.Fingerprint` (hex, X.509) is NOT reused — different domain. P02 adds a thin SSH-specific helper. |
|
||||
| AD-028 | `--host-key-fingerprint` callback compares full `SHA256:base64` strings, not decoded bytes | `ssh.FingerprintSHA256` returns the canonical string; direct string compare avoids a base64-decode step and is less error-prone. Validate `SHA256:` prefix up front. |
|
||||
| AD-029 | `orca node key-reset` rewrites `known_hosts` via atomic temp-file + rename | Prevents data loss on crash mid-write. Reuse the `writeAtomic` pattern from `security/ca.go:305` (export it or copy the ~20 LOC). |
|
||||
| AD-030 | `verify-reqs` implemented as `cmd/verify-reqs/main.go` (Go program), not shell+awk | Testable, type-safe, consistent with Go-only tooling. `make verify-reqs` runs `go run ./cmd/verify-reqs`. Hooked into `.coreci.yml` validate pipeline. |
|
||||
|
||||
---
|
||||
|
||||
## 5. Pitfalls, gaps, and flags for the plan
|
||||
|
||||
1. **TOFU capture is currently BROKEN (§2.1).** `knownhosts.New` returns `KeyError{Want:[]}` on first connect and does NOT auto-write the key. The current `BootstrapProxmox` treats this as a dial failure. P02 must either (a) wrap the callback to capture-and-persist on `KeyError{Want:[]}` via `knownhosts.Line` + atomic write, or (b) make `--host-key-fingerprint` the required first-connect path. **Recommend (a) — fix TOFU + add pre-pin as superset.** This is a v0.6 latent bug that P02 closes.
|
||||
2. **`Result.HostKeyFingerprint` is never populated (§2.2).** D-045's rationale references "existing output" that doesn't exist. P02 must ADD the computation (`ssh.FingerprintSHA256`). Low risk — it's a 1-line addition once the host key is available.
|
||||
3. **No `sessionRunner` seam in proxmox (§1.3).** Testing the SSH command sequence (deployPubKey, createLinuxUser, pveum, sudoers, visudo) without a real SSH server requires a new interface seam. **Recommend P01 plan add it** — 1 interface, ~10 LOC, unlocks ~40% of proxmox coverage.
|
||||
4. **`internal/store/cert_repo.go` has NO test (§1.1).** v0.7 P01 REQ-053 was supposed to add `cert_repo_test.go` but it's missing — `internal/store/` glob shows no `cert_repo_test.go`. This is a v0.7 leftover. P01 should add it (it directly lifts store coverage toward 70%).
|
||||
5. **`internal/cli/daemon.go` excluded from cli 70% target (§1.4).** The daemon command starts a long-running server; it's covered by `internal/daemon/server_test.go` (150 LOC). Don't double-test in cli.
|
||||
6. **`cmd/orca` 50% toe-hold is low-value (§1.1).** 15 LOC of glue; the test effort:coverage ratio is poor. D-047 already called this out. Don't over-invest.
|
||||
7. **`go: no such tool "covdata"` for zero-test packages (§1.1).** This is a Go toolchain quirk when a package has no test files — `go test -cover` can't compute coverage without a test binary. It's NOT a real 0% number (it's "undefined"). Adding any `_test.go` file makes the number computable. Don't treat the error as a coverage measurement.
|
||||
8. **`transport.dispatchToPeer` has no seam (§1.3).** Testing the remote-dispatch branch of `Dispatcher.Submit` requires either a new `peerDispatcher` interface OR `httptest.NewTLSServer`. The latter is already used in `daemon/dispatch_test.go`; recommend the plan use `httptest.NewTLSServer` (no refactor needed) for transport coverage.
|
||||
9. **`knownhosts.Line` + `knownhosts.Normalize` are the helpers for the TOFU-capture fix and for `key-reset` matching (§2.1, §2.4).** Use `Normalize` to match host strings consistently (handles `host:22` vs `host`).
|
||||
10. **`security.writeAtomic` is unexported (ca.go:305).** `key-reset`'s atomic known_hosts rewrite needs it. Either export `WriteAtomic` from `security`, or copy the ~20-LOC pattern into `proxmox`/`cli`. **Recommend export** — it's already used across ca.go + sshkey.go and is generally useful.
|
||||
|
||||
---
|
||||
|
||||
## 6. Dependencies
|
||||
|
||||
v0.8 adds **zero** new direct dependencies:
|
||||
- SSH host-key fingerprint: `ssh.FingerprintSHA256` (already in `golang.org/x/crypto/ssh` v0.54.0, direct dep since v0.6).
|
||||
- `knownhosts.Line`/`Normalize`/`KeyError`: same `golang.org/x/crypto` module.
|
||||
- `verify-reqs`: stdlib only (`regexp`, `os`, `fmt`).
|
||||
- Tests: `net/http/httptest` (stdlib), existing interfaces.
|
||||
|
||||
`go.mod` is unchanged by v0.8.
|
||||
|
||||
---
|
||||
|
||||
## 7. PERSONAS assessment (v0.8)
|
||||
|
||||
v0.8 is an NFR milestone touching tests (9 packages), SSH trust surface (proxmox + cli/node + security), and a requirements-hygiene Go program. The 3-persona roster from config.json (lead-developer, backend-engineer, data-engineer) is sufficient — no phase-specific personas needed.
|
||||
|
||||
**Roster confirmation:**
|
||||
- **lead-developer** — owns coordination + `cmd/orca` smoke test + `internal/cli` coverage (cert/doctor/audit/status/version subcommands) + the `verify-reqs` Go program (coordination territory).
|
||||
- **backend-engineer** — owns `internal/transport` tests (httptest.NewTLSServer) + `internal/engine` tests (LocalExecutor stubs, PeerRegistry) + SSH trust-surface in `internal/proxmox/bootstrap.go` (pinned callback, TOFU capture fix, sessionRunner seam) + `internal/cli/node.go` (`--host-key-fingerprint` flag, `key-reset` subcommand).
|
||||
- **data-engineer** — owns `internal/store` tests (cert_repo_test.go gap + coverage uplift) + `internal/audit` tests (sqlite-backed audit_log asserts) + `internal/certpaths` tests (path-join asserts) + `internal/jobspec` tests (golden HCL fixtures).
|
||||
|
||||
No frontend persona (no UI). No devops persona (no packaging/distribution — `verify-reqs` is a Go program, not a CI config change; the `.coreci.yml` edit is a 3-line hook, lead-developer territory). No security-engineer persona (the SSH trust work is backend-engineer territory — the security-engineer was deactivated in v0.7 and v0.8 doesn't re-add it; the trust-surface hardening is a refinement of the existing `proxmox` package, not new security architecture).
|
||||
|
||||
See `.ciagent/PERSONAS.md` (updated with v0.8 YAML frontmatter + territory globs matching the actual file structure).
|
||||
@@ -1,321 +0,0 @@
|
||||
# Review: Orca v0.8 — Coverage & Trust Hardening (final-phase)
|
||||
|
||||
**Reviewer**: ci-code-reviewer (multi-persona: correctness, testing, security, performance, maintainability, adversarial)
|
||||
**Branch**: `phase/04-final-review-ship` (review HEAD = P03 ship `70c5718`)
|
||||
**Diff scope**: `main...milestone/v0.8-coverage-trust-hardening` (all v0.8 work, 59 files, +6550/-173)
|
||||
**Date**: 2026-08-04
|
||||
**Verdict**: **PASS-WITH-FOLLOWUPS** (0 P0, 2 P1, 2 P2)
|
||||
|
||||
## Methodology
|
||||
|
||||
Read the full diff (`internal/`, `cmd/`, `Makefile`, `.coreci.yml`), all 3 phase
|
||||
verification reports, PLAN/RESEARCH/GRILL/PERSONAS, and the critical production
|
||||
files directly (`internal/proxmox/bootstrap.go`, `internal/security/ca.go`,
|
||||
`internal/security/sshkey.go`, `internal/doctor/doctor.go:400-454`,
|
||||
`cmd/verify-reqs/main.go`). Re-ran `go build ./...`, `go vet ./...`,
|
||||
`go test -race` on proxmox/security/doctor/cli/verify-reqs/cmd-orca, and
|
||||
`make verify-reqs` (all PASS). Re-verified the verify-reqs regex against the
|
||||
real ROADMAP.md (matches v0.1..v0.7 COMPLETE incl. v0.2 parenthetical; v0.8
|
||||
correctly not matched). Confirmed all 7 T02.10 e2e cases are present and
|
||||
exercised through a real in-process SSH server.
|
||||
|
||||
---
|
||||
|
||||
## Per-Axis Findings
|
||||
|
||||
### 1. Correctness (lead-developer)
|
||||
|
||||
**C1 — `sessionRunner` seam backward-compat** ✅
|
||||
`internal/proxmox/bootstrap.go:170-172,322-339`. The seam is a package-level
|
||||
`var sessionRunner sessionRunnerType` (line 326) initialized lazily inside
|
||||
`BootstrapProxmox` from the dialed `*ssh.Client` (`if sessionRunner == nil {
|
||||
sessionRunner = &sshSessionRunner{client: conn} }`). Existing callers are
|
||||
unchanged — the default `sshSessionRunner` wraps the real
|
||||
`conn.NewSession().CombinedOutput(...)`. Tests reset `sessionRunner = nil`
|
||||
between runs (bootstrap_test.go:910, 1010, 1035) to avoid cross-test leakage.
|
||||
Backward compatible as required by P01 verification. No issue.
|
||||
|
||||
**C2 — `verify-reqs` parser correctness** ✅
|
||||
`cmd/verify-reqs/main.go:18-26`. The `reqRowRe` uses a greedy `.*` for the
|
||||
Requirement+Priority cells and anchors the Phase+Status match at the END of
|
||||
the line, where those two columns always live. This correctly handles
|
||||
escaped pipes inside the Requirement cell (e.g. REQ-049
|
||||
`localhost\|linux\|proxmox` — verified by the passing `make verify-reqs`
|
||||
which reports 60 consistent rows, matching the 60 REQ rows in
|
||||
REQUIREMENTS.md). The status token is optionally bold-wrapped
|
||||
(`\*{0,2}(Complete|Pending)\*{0,2}`) with `[^|]*` for trailing notes —
|
||||
handles `**Complete** (P01 shipped v0.2.1)`. The milestone-complete regex
|
||||
`^##\s*Milestone\s+(v0\.\d+):.*—\s*\*\*[^*]*\bCOMPLETE\b[^*]*\*\*` is
|
||||
substring-tolerant (GRILL #4) — verified against the real ROADMAP: matches
|
||||
v0.1..v0.7 incl. v0.2's `**COMPLETE (merged to main via v0.3)**` and v0.6's
|
||||
duplicate header (line 94 matched; line 92 without COMPLETE ignored). v0.8
|
||||
(line 136, not yet COMPLETE) correctly not matched — so the v0.8 REQ rows
|
||||
being `Pending` is NOT flagged as drift (correct: milestone not shipped
|
||||
yet). No issue.
|
||||
|
||||
**C3 — TOFU capture-fix logic** ✅
|
||||
`internal/proxmox/bootstrap.go:275-310`. On `*knownhosts.KeyError` with
|
||||
empty `Want` (host unknown), captures the key, reads existing known_hosts
|
||||
(create-if-missing), ensures trailing newline, appends
|
||||
`knownhosts.Line([]string{knownhosts.Normalize(addr)}, key)`, writes via
|
||||
`security.WriteAtomic`, returns nil (dial proceeds). On non-empty `Want`
|
||||
(mismatch) returns the error (MITM detection preserved). On `nil` (match)
|
||||
records key + returns nil. Correct against x/crypto v0.54.0 `checkAddr`
|
||||
semantics. No issue.
|
||||
|
||||
**C4 — `ResetHostKey` line matching** ✅
|
||||
`internal/proxmox/bootstrap.go:479-523`. Matches a line when its first
|
||||
whitespace-delimited field, normalized via `knownhosts.Normalize`, equals
|
||||
the normalized target. Handles `[host]:22` vs bare `host` (Normalize
|
||||
brackets ports). Preserves comments/blanks. Atomic rewrite via
|
||||
`security.WriteAtomic`. Edge cases handled: empty host errors, missing
|
||||
file is a no-op, no matching lines is a no-op. Test
|
||||
`TestResetHostKey_RemovesTargetLines` (bootstrap_test.go:718) seeds 2 lines
|
||||
for target + 1 for another host, asserts target's 2 removed + other
|
||||
intact. No issue.
|
||||
|
||||
**C5 — `--host-key-fingerprint` non-proxmox validation** ✅
|
||||
`internal/cli/node.go:79-81`. `RunE` checks `joinHostKeyFP != "" &&
|
||||
joinType != "proxmox"` → clear error. Test
|
||||
`TestNodeJoinHostKeyFingerprintRequiresProxmox` (node_test.go:445) asserts
|
||||
the error; `TestNodeJoinHostKeyFingerprintProxmoxAccepted` (node_test.go:477)
|
||||
asserts the negative-space (proxmox type accepts the flag). No issue.
|
||||
|
||||
### 2. Testing (all personas)
|
||||
|
||||
**T1 — Coverage held post-P02** ✅
|
||||
PHASE2 verification: proxmox 86.5% (was 87.1%), cli 76.7% (was 76.2%),
|
||||
doctor 70.4%. Marginal changes from new code paths — no coverage regression.
|
||||
Re-ran `go test -race ./internal/proxmox/... ./internal/cli/...` PASS.
|
||||
|
||||
**T2 — Race tests pass** ✅
|
||||
`go test -race -count=1 ./internal/proxmox/... ./internal/security/...
|
||||
./internal/doctor/... ./cmd/verify-reqs/... ./cmd/orca/...` all PASS.
|
||||
`./internal/cli/...` PASS (77s, dominated by watch tests). No races.
|
||||
|
||||
**T3 — 7 T02.10 integration cases** ✅
|
||||
All 7 present in `internal/proxmox/bootstrap_test.go`, exercised
|
||||
end-to-end through `BootstrapProxmox` with a real in-process SSH server
|
||||
(`bootstrapE2ESetup`):
|
||||
- Case 1: `TestBootstrapE2E_PinnedFingerprintCorrect` (815)
|
||||
- Case 2: `TestBootstrapE2E_PinnedFingerprintWrong` (842)
|
||||
- Case 3: `TestBootstrapE2E_TOFUFirstConnectCapturesKey` (865)
|
||||
- Case 4: `TestBootstrapE2E_TOFUSecondConnectMatches` (905)
|
||||
- Case 5: `TestBootstrapE2E_TOFUMismatchFails` (925)
|
||||
- Case 6: `TestBootstrapE2E_KeyResetThenRePin` (1002)
|
||||
- Case 7: `TestBootstrapE2E_PrePopulatedKnownHostsMatches` (966, v0.6→v0.8 migration)
|
||||
|
||||
**T4 — Golden tests for verify-reqs** ✅
|
||||
`cmd/verify-reqs/main_test.go` has 7 tests covering: clean pair, multi-drift
|
||||
(both directions), default-args subprocess, malformed (no REQ rows → error),
|
||||
missing file → error, v0.2 substring-tolerant header regression guard
|
||||
(`TestVerify_v02SubstringTolerantHeader`), and real-repo regression guard.
|
||||
The substring-tolerant regex is explicitly exercised — the drift fixture's
|
||||
ROADMAP uses `**COMPLETE (merged to main via v0.3)**` on v0.2 and the test
|
||||
asserts REQ-002 (v0.2 P1, Pending) is flagged forward-drift (would be
|
||||
silently skipped if the regex regressed). No issue.
|
||||
|
||||
**T5 — Doctor parity test** ✅
|
||||
`internal/doctor/doctor_test.go` extended with 94 LOC covering
|
||||
`probeProxmoxPVEVersion` paths. The doctor callback now uses the shared
|
||||
`proxmox.TOFUHostKeyCallback` (doctor.go:424) — GRILL #2 parity verified by
|
||||
reading both call sites. No issue.
|
||||
|
||||
### 3. Security (backend-engineer)
|
||||
|
||||
**S1 — `--host-key-fingerprint` fails closed** ✅
|
||||
`internal/proxmox/bootstrap.go:141-153,245-258`. The pinned/TOFU branch is
|
||||
mutually exclusive (`if opts.HostKeyFingerprint != "" { ... } else { ... }`).
|
||||
The pinned callback (245-258) validates `SHA256:` prefix up front (rejects
|
||||
raw hex per D-045), computes `ssh.FingerprintSHA256(key)`, returns an error
|
||||
on any mismatch — no fallback to TOFU. The dial (164) aborts on callback
|
||||
error before any SSH session command runs. Cannot be bypassed: the pin is
|
||||
compared as a full string against the canonical fingerprint of the
|
||||
server-presented key; a mismatch returns before `*capturedKey` is set. No
|
||||
issue.
|
||||
|
||||
**S2 — `key-reset` is local-only (D-046)** ✅
|
||||
`internal/proxmox/bootstrap.go:479-523` + `internal/cli/node.go:348-407`.
|
||||
`ResetHostKey` only reads/writes `certpaths.KnownHostsPath()`. No SSH dial,
|
||||
no remote authorized_keys touch. Audit-logs `node.key_reset` with
|
||||
actor+node+host (node.go:396-400). Verified by `TestNodeKeyReset`
|
||||
(node_test.go:326) which asserts the audit row. No issue.
|
||||
|
||||
**S3 — TOFU capture-fix doesn't weaken MITM detection** ✅
|
||||
See C3 — the fix ONLY captures on `KeyError{Want:[]}` (host unknown); a
|
||||
non-empty `Want` (key mismatch / MITM) returns the error. The capture path
|
||||
writes the server-presented key, so a subsequent different key fails. No
|
||||
issue.
|
||||
|
||||
**S4 — `WriteAtomic` is actually atomic** ✅
|
||||
`internal/security/ca.go:308-337`. Temp file in same dir
|
||||
(`os.CreateTemp(dir, ".tmp-*")`), `Write`, `Chmod`, `Sync`, `Close`, then
|
||||
`os.Rename` (atomic on POSIX same-filesystem). `defer os.Remove(tmpName)`
|
||||
cleans up on failure. Genuine atomic-write pattern. No issue.
|
||||
|
||||
### 4. Performance (all)
|
||||
|
||||
**P1 — ResetHostKey is O(n) in file size** ✅
|
||||
`internal/proxmox/bootstrap.go:479-523`: one `os.ReadFile` (O(n)), one
|
||||
`strings.Split` + linear filter loop (O(n)), one `security.WriteAtomic`
|
||||
(O(n)). No nested loops, no O(n²). For a known_hosts file (typically tens
|
||||
of lines), this is negligible. No issue.
|
||||
|
||||
**P2 — Unnecessary allocations** (P2 — nit)
|
||||
`bootstrap.go:494-510`: `strings.Split(string(existing), "\n")` allocates a
|
||||
slice of all lines + `append(kept, []byte(line+"\n")...)` reallocates the
|
||||
kept buffer. For known_hosts (small file) this is fine; a `bufio.Scanner`
|
||||
over `bytes.NewReader(existing)` with a `strings.Builder` would be leaner,
|
||||
but the current shape is clear and the file is tiny. Not worth changing.
|
||||
Flagged P2 (nit, no action).
|
||||
|
||||
### 5. Maintainability (lead-developer)
|
||||
|
||||
**M1 — `TOFUHostKeyCallback` extraction** ✅
|
||||
`internal/proxmox/bootstrap.go:261-310` is exported and shared by bootstrap
|
||||
(148) and doctor (doctor.go:424) via
|
||||
`proxmox.TOFUHostKeyCallback(sshAddr, &capturedHostKey)`. Clean coupling:
|
||||
doctor imports proxmox (one-way), no duplication, no circular dep. The
|
||||
GRILL #2 parity requirement (both call sites use the same wrapper) is
|
||||
satisfied by construction. No issue.
|
||||
|
||||
**M2 — `verify()` testable** ✅
|
||||
`cmd/verify-reqs/main.go:98-148`: the core logic is a pure function
|
||||
`verify(roadmapPath, reqsPath string) (diff []string, count int, err error)`
|
||||
with `main()` as a thin wrapper. Golden-file tests call `verify()` directly
|
||||
(no subprocess). Mirrors the T01.11 `main()→run()` pattern. No issue.
|
||||
|
||||
**M3 — cli tests follow conventions** ✅
|
||||
`internal/cli/namespace_test.go:21-35` adds `resetCommandFlags()` to zero
|
||||
the package-level flag-bound vars between subtests (cobra parses into
|
||||
globals; without reset a prior test's value persists). Called from
|
||||
`resetRootFlags`. This is a sound convention — the test isolation is
|
||||
correct. No issue.
|
||||
|
||||
**M4 — `resetCommandFlags` completeness** (P2 — nit)
|
||||
`namespace_test.go:28-34` resets the join/leave/cap/audit/run/stop flags but
|
||||
NOT `joinHostKeyFP`. A test that sets `--host-key-fingerprint` without
|
||||
calling `resetRootFlags` could leak the value to a later test. In practice
|
||||
all node tests call `resetRootFlags` which calls `resetCommandFlags`, so
|
||||
this is a latent risk only. Recommend adding `joinHostKeyFP = ""` to
|
||||
`resetCommandFlags` for completeness. Flagged P2 (nit).
|
||||
|
||||
### 6. Adversarial (backend-engineer)
|
||||
|
||||
**A1 — Can `--host-key-fingerprint` be bypassed?** ✅
|
||||
No. The pinned callback (bootstrap.go:249-258) returns an error before
|
||||
recording the key or allowing the dial to proceed on any mismatch. There is
|
||||
no code path where a supplied pin is ignored — the branch at 141-153 is
|
||||
`if opts.HostKeyFingerprint != ""` (pinned) `else` (TOFU); once pinned is
|
||||
chosen, TOFU is not consulted. No bypass.
|
||||
|
||||
**A2 — Can `key-reset` corrupt known_hosts under concurrent write?** (P1 — important, low likelihood)
|
||||
`proxmox.ResetHostKey` (bootstrap.go:479-523) and `TOFUHostKeyCallback`
|
||||
(bootstrap.go:290-302) both do read-modify-write on
|
||||
`certpaths.KnownHostsPath()` WITHOUT a lock. Two concurrent operations
|
||||
(e.g. `orca node join --type proxmox hostA` + `orca node key-reset hostB`,
|
||||
or two simultaneous joins to different hosts) could interleave:
|
||||
- T1 reads known_hosts (empty), T2 reads known_hosts (empty)
|
||||
- T1 writes hostA line, T2 writes hostB line
|
||||
- Last rename wins → one line lost.
|
||||
|
||||
The `security.WriteAtomic` (temp+rename) prevents corruption (the file is
|
||||
always valid OpenSSH format), but a captured line can be silently lost. This
|
||||
is a **last-writer-wins race on a flat file with no lock**. Severity is low
|
||||
because orca is a single-operator CLI (concurrent joins are unusual) and
|
||||
the lost line is recoverable (re-connect re-pins via TOFU). But it is a
|
||||
real correctness gap for the trust surface. Recommend either (a) a
|
||||
file-lock around the read-modify-write, or (b) documenting the
|
||||
single-operator assumption explicitly. Flagged P1 (important, post-hoc).
|
||||
|
||||
**A3 — Can verify-reqs be fooled by a crafted markdown table?** ✅
|
||||
No. The `reqRowRe` anchors on `^\|\s*(REQ-\d+)\s*\|` and the status column
|
||||
at end-of-line. A crafted row with a fake status would have to match the
|
||||
regex exactly. The "malformed" fixture (`requirements_malformed.md`)
|
||||
exercises the no-REQ-rows path → clear error. A row like
|
||||
`| REQ-999 | missing status cell | High | v0.1 |` (no final `|...|`) does
|
||||
NOT match `reqRowRe` (the trailing `\|\s*$` requires the status cell) — it
|
||||
is silently skipped, which `verify` reports as "no REQ rows" only if ALL
|
||||
rows are malformed. If some rows are valid + one malformed, the malformed
|
||||
row is skipped without error — a minor blind spot, but acceptable (the
|
||||
gate catches drift, not typos). No blocking issue.
|
||||
|
||||
---
|
||||
|
||||
## GRILL Conditions Verification
|
||||
|
||||
### #1 — T02.6 labeled as v0.6 ship-defect bugfix ✅
|
||||
Commit `8b0cbe1` summary: "fix(proxmox): TOFU capture bug — **v0.6
|
||||
ship-defect** first-connect join always failed (T02.6)". The commit message
|
||||
explicitly labels it as a v0.6 ship-defect bugfix, not a v0.8 feature.
|
||||
PHASE2 verification report records it as "TOFU bugfix (T02.6, v0.6
|
||||
ship-defect)". **Satisfied.**
|
||||
|
||||
### #2 — T02.9 doctor parity (bootstrap + doctor use capture-fix wrapper) ✅
|
||||
- Bootstrap: `internal/proxmox/bootstrap.go:148` calls
|
||||
`TOFUHostKeyCallback(sshAddr, &capturedHostKey)`.
|
||||
- Doctor: `internal/doctor/doctor.go:424` calls
|
||||
`proxmox.TOFUHostKeyCallback(sshAddr, nil)`.
|
||||
Both use the SAME exported wrapper (`proxmox.TOFUHostKeyCallback`,
|
||||
bootstrap.go:275). No duplication. The doctor diff
|
||||
(`internal/doctor/doctor.go`) removes the direct `knownhosts.New` call and
|
||||
replaces it with the shared wrapper. **Satisfied.**
|
||||
|
||||
### #3 — T01.6 cli escape valve (was it needed?) ✅
|
||||
PHASE1 verification: cli hit **76.2%** (above the 70% floor, excluding
|
||||
daemon.go). The escape valve (ship at 65% if 70% not reached) was **NOT
|
||||
needed**. The plan's conditional was correctly conservative; the actual
|
||||
result exceeded the floor. **Satisfied (not invoked).**
|
||||
|
||||
### #4 — T03.1 verify-reqs regex substring-tolerant + reverse direction ✅
|
||||
- **Substring-tolerant**: `cmd/verify-reqs/main.go:30`:
|
||||
`^##\s*Milestone\s+(v0\.\d+):.*—\s*\*\*[^*]*\bCOMPLETE\b[^*]*\*\*`.
|
||||
Verified against the real ROADMAP: matches v0.2's
|
||||
`**COMPLETE (merged to main via v0.3)**` and all other COMPLETE
|
||||
milestones. Golden test `TestVerify_v02SubstringTolerantHeader`
|
||||
(main_test.go:140) guards against regression.
|
||||
- **Reverse direction**: `cmd/verify-reqs/main.go:135-139`: a REQ marked
|
||||
`Complete` whose referenced milestones are ALL not-C_COMPLETE in ROADMAP
|
||||
is flagged as `direction=reverse` drift. Golden test `TestVerify_drift`
|
||||
asserts REQ-003 is reverse-drift.
|
||||
- **Scope note**: PLAN + commit `fc2b020` document that REQ-060 catches
|
||||
doc-vs-doc drift only (code-vs-doc like the REQ-053 cert_repo_test.go
|
||||
case is out of scope; P04 audit is the backstop). **Satisfied.**
|
||||
|
||||
---
|
||||
|
||||
## P0 Fixes Applied
|
||||
|
||||
**None.** No P0 issues (correctness bugs, security holes, broken build/test)
|
||||
were found. `go build ./...`, `go vet ./...`, `go test -race` (all key
|
||||
packages), and `make verify-reqs` all PASS. The milestone is shippable as-is.
|
||||
|
||||
---
|
||||
|
||||
## P1+ Issues Flagged (post-hoc review)
|
||||
|
||||
| ID | Severity | File:line | Issue | Recommendation |
|
||||
|----|----------|-----------|-------|----------------|
|
||||
| A2 | P1 (important, low likelihood) | `internal/proxmox/bootstrap.go:290-302, 479-523` | `TOFUHostKeyCallback` capture path and `ResetHostKey` both do read-modify-write on `known_hosts` with no lock; concurrent operations can lose a captured line (last-writer-wins via atomic rename — no corruption, but data loss). | Add a file-lock (`flock` on a `.known_hosts.lock` sibling, or `github.com/gofrs/flock` if a dep is acceptable) around the RMW in both paths; OR document the single-operator assumption in the key-reset help text. Defer to v0.9. |
|
||||
| M4 | P2 (nit) | `internal/cli/namespace_test.go:28-34` | `resetCommandFlags()` does not reset `joinHostKeyFP`; a test setting `--host-key-fingerprint` without `resetRootFlags` could leak the value. | Add `joinHostKeyFP = ""` to `resetCommandFlags`. Trivial. |
|
||||
| P2 | P2 (nit) | `internal/proxmox/bootstrap.go:494-510` | `ResetHostKey` uses `strings.Split` + repeated `append` (minor allocation churn). | Optional: use `bufio.Scanner` + `strings.Builder`. Not worth changing for a small file. |
|
||||
|
||||
---
|
||||
|
||||
## Overall Verdict
|
||||
|
||||
**PASS-WITH-FOLLOWUPS**
|
||||
|
||||
The v0.8 milestone is correct, secure, tested, and shippable. All 4 GRILL
|
||||
binding conditions are satisfied. Zero P0 issues. The single P1 (concurrent
|
||||
`known_hosts` write race, A2) is a real but low-likelihood gap appropriate
|
||||
for post-hoc follow-up — it does not block the milestone ship because orca
|
||||
is a single-operator CLI and the atomic-rename guarantees the file is never
|
||||
corrupted (only a captured line can be lost, recoverable on re-connect).
|
||||
The 2 P2 nits are cosmetic. Coverage held post-P02 (86.5%/76.7%/70.4% for
|
||||
proxmox/cli/doctor), race tests pass, all 7 T02.10 e2e cases are present and
|
||||
exercised through a real in-process SSH server, and `make verify-reqs`
|
||||
passes on the current repo (60 consistent rows).
|
||||
|
||||
Recommend proceeding to P04 ship (T04.7/T04.8: mark REQ-057..060 Complete +
|
||||
ROADMAP v0.8 COMPLETE, then tag v0.7.4).
|
||||
+51
-701
@@ -20,712 +20,62 @@
|
||||
- `iter.Seq` streaming job lists (REQ-022)
|
||||
- Frontend / devops personas (no web UI; CoreCI handles release)
|
||||
|
||||
## Milestone v0.2: Networking, Observability, Security Hardening — **COMPLETE (merged to main via v0.3)**
|
||||
## Milestone v0.2: Networking, Observability, Security Hardening — **IN PROGRESS**
|
||||
|
||||
Scope: extend v0.1 with secure cross-node transport, multi-node scheduling,
|
||||
richer CI security scanning, and streaming I/O.
|
||||
|
||||
- [x] Phase 8: mTLS handshake + internal CA with CSR join (Wave 1) — shipped v0.2.1
|
||||
- [x] Phase 9: Multi-node scheduling & job dispatch (Wave 1) — shipped v0.2.2
|
||||
- [x] Phase 10: `gosec` + `govulncheck` + gitleaks in CI (Wave 2) — shipped v0.2.3
|
||||
- [x] Phase 11: `iter.Seq` streaming job/node lists (Wave 2) — **completed in v0.3 P01** (shipped v0.3.1)
|
||||
- [ ] Phase 8: mTLS handshake + internal CA with CSR join (Wave 1)
|
||||
- [ ] Phase 9: Multi-node scheduling & job dispatch (Wave 1)
|
||||
- [ ] Phase 10: `gosec` + `govulncheck` + gitleaks in CI (Wave 2)
|
||||
- [ ] Phase 11: `iter.Seq` streaming job/node lists (Wave 2)
|
||||
|
||||
**Milestone tag**: `v0.4.0` (shipped — v0.2 work merged to main via v0.3 milestone).
|
||||
**Target milestone tag**: `v0.3.0` (next-minor per feature-milestone promotion rule).
|
||||
|
||||
Per-phase tags: `v0.2.1` (P01), `v0.2.2` (P02), `v0.2.3` (P03) — all shipped.
|
||||
|
||||
## Milestone v0.3: Scheduling & Streaming Completion — **COMPLETE**
|
||||
|
||||
Scope: complete the two work items deferred from v0.2 that were not
|
||||
already shipped in P08-P10. A re-init SPECIFY codebase audit confirmed
|
||||
that REQ-014/027/028/029/031/037/039/040 all shipped in P08-P10 despite
|
||||
stale REQUIREMENTS.md marking them Pending. The remaining work is lean:
|
||||
|
||||
- [x] Phase 0: Pre-execution (specify → clarify → research → plan → grill) — shipped v0.3.0
|
||||
- [x] Phase 1: `iter.Seq` streaming for `--watch` flags (REQ-022, REQ-030) — shipped v0.3.1
|
||||
- [x] Phase 2: `orca doctor` network + db full implementation (REQ-032 completion) — shipped v0.3.2
|
||||
- [x] Phase 3: Final review + ship + audit (milestone release) — shipped v0.3.3
|
||||
|
||||
**Milestone tag**: `v0.4.0` (next-minor per feature-milestone promotion rule).
|
||||
|
||||
Per-phase tags: `v0.3.0` (P0), `v0.3.1` (P01), `v0.3.2` (P02), `v0.3.3` (P03 final = milestone release).
|
||||
Per-phase tags: `v0.2.1` (P01), `v0.2.2` (P02), `v0.2.3` (P03), `v0.2.4` (P04).
|
||||
Per `.ciagent/RELEASE_POLICY.md`, every phase tag produces a Gitea release.
|
||||
|
||||
### Per-phase REQ coverage
|
||||
|
||||
- **P01 — `iter.Seq` streaming for `--watch` flags**
|
||||
- REQ-022 (`iter.Seq` for streaming job lists, Go 1.25+)
|
||||
- REQ-030 (`--watch` output format mode: table default vs streaming JSON per event)
|
||||
- Applies to both `orca job list --watch` and `orca node list --watch`
|
||||
(D-024, per ARCHITECTURE.md CLI layer + D-017)
|
||||
|
||||
- **P02 — `orca doctor` network + db full implementation**
|
||||
- REQ-032 (completion: network reachability via mTLS `/healthz` probe,
|
||||
db integrity via `PRAGMA integrity_check` + migration version)
|
||||
- Replaces `NetworkStub` and `DBStub` from v0.2 P01
|
||||
|
||||
### v0.3 is a completion milestone, not a direction change
|
||||
|
||||
The vision ("minimalist, offline-first, CLI-first orchestration
|
||||
engine") is unchanged. v0.3 closes out the v0.2 deferrals and merges
|
||||
the accumulated v0.2 work to main.
|
||||
|
||||
## Milestone v0.5: Distribution — **COMPLETE**
|
||||
|
||||
Scope: make Orca installable, distributable, and containerized. The
|
||||
engine functionality from v0.1–v0.3 is unchanged; this milestone is
|
||||
purely about delivery surface.
|
||||
|
||||
- [x] Phase 0: Pre-execution (specify → clarify → research → plan) — shipped `v0.4.1` (+ repo public)
|
||||
- [x] Phase 1: Namespace unification (`ORCA_HOME` + `--system`) (REQ-041, REQ-042) — shipped `v0.4.2`
|
||||
- [x] Phase 2: `install.sh` + in-place update + README quickstart (REQ-043, REQ-044) — shipped `v0.4.3`
|
||||
- [x] Phase 3: Docker release (Dockerfile + Gitea container registry) (REQ-046) — shipped `v0.4.4`
|
||||
- [x] Phase 4: Final review + ship + audit (milestone release) — shipped `v0.4.5`
|
||||
|
||||
**Operational prerequisite (P0 ship)**: repo + org visibility flipped to
|
||||
public (REQ-045) — unauth releases API + asset download + docker pull all
|
||||
verified HTTP 200.
|
||||
|
||||
**Milestone tag**: `v0.4.5` (final phase patch = milestone release per
|
||||
feature-milestone promotion rule). Per-phase tags: `v0.4.1`…`v0.4.5`.
|
||||
|
||||
## Milestone v0.6: Node Bootstrap & Proxmox
|
||||
|
||||
## Milestone v0.6: Node Bootstrap & Proxmox — **COMPLETE**
|
||||
|
||||
Scope: make `orca init` produce a fully working single-node cluster
|
||||
(CA + server cert + DB + localhost node registered with auto-detected
|
||||
OS), and add Proxmox 8 & 9 as a first-class remote node type joined
|
||||
over SSH with least-privilege role delegation.
|
||||
|
||||
- [x] Phase 0: Pre-execution (specify → clarify → research → plan → grill) — tag `v0.5.0`
|
||||
- [x] Phase 1: `orca init` full bootstrap + localhost node + schema 0006 (REQ-047, REQ-048, REQ-049) — tag `v0.5.1`
|
||||
- [x] Phase 2: Proxmox SSH join + OrcaOperator role + sudoers allowlist (REQ-050, REQ-051) — tag `v0.5.2`
|
||||
- [x] Phase 3: `doctor os` + `doctor proxmox` SSH probe + audit logging (REQ-052) — tag `v0.5.3`
|
||||
- [x] Phase 4: Final review + ship + audit (milestone release) — tag `v0.5.4`
|
||||
|
||||
**Milestone type**: feature (P1/P2/P3 ship `feat` phases).
|
||||
**Milestone tag**: `v0.5.4` (final phase patch = milestone release per
|
||||
feature-milestone promotion rule). Per-phase tags: `v0.5.0`…`v0.5.4`.
|
||||
|
||||
Tags run on the previous minor's patch line (v0.5.x) per
|
||||
branch-strategy.md. The milestone branch label uses the milestone
|
||||
number (`milestone/v0.6-node-bootstrap-proxmox`); no separate minor
|
||||
tag is created.
|
||||
|
||||
## Milestone v0.7: Hardening & Completion — **COMPLETE**
|
||||
|
||||
Scope: NFR milestone closing gaps surfaced by the v0.7 IDEATE stage —
|
||||
an unreachable command tree, a missing config file layer, low test
|
||||
coverage in core packages, and the long-deferred pprof endpoint.
|
||||
|
||||
- [x] Phase 0: Pre-execution (specify → clarify → research → ideate → plan) — tag `v0.6.0` (shipped)
|
||||
- [x] Phase 1: Register `orca cert` command tree + cert_repo tests (REQ-053) — tag `v0.6.1` (shipped)
|
||||
- [x] Phase 2: HCL config file parsing — `internal/config` package (REQ-054) — tag `v0.6.2` (shipped)
|
||||
- [x] Phase 3: Test coverage uplift — engine/transport/proxmox/audit ≥ 50% (REQ-055) — tag `v0.6.3` (shipped)
|
||||
- [x] Phase 4: `--pprof` opt-in on `orca daemon` (REQ-056) — tag `v0.6.4` (shipped)
|
||||
- [x] Phase 5: Final review + ship + audit (milestone release) — tag `v0.6.5` (shipped)
|
||||
|
||||
**Milestone type**: NFR (all phases are fix/test/chore; no `feat` phases).
|
||||
**Milestone tag**: `v0.6.5` (final phase patch = milestone release per
|
||||
NFR-milestone progressive-patch rule). Per-phase tags: `v0.6.0`…`v0.6.5`.
|
||||
Tags run on the previous minor's patch line (v0.6.x) per
|
||||
branch-strategy.md. The milestone branch label uses the milestone
|
||||
number (`milestone/v0.7-hardening-completion`); no separate minor tag.
|
||||
|
||||
## Milestone v0.8: Coverage & Trust Hardening — **COMPLETE**
|
||||
|
||||
Scope: continue the v0.7 hardening theme. v0.7 P03's ≥ 50% floor left
|
||||
six packages still under 50% (engine 8.3%, proxmox 5.1%, cli 27.6%,
|
||||
transport 26.3%, store 46.7%, jobspec 47.6%) and three packages with
|
||||
no tests at all (`internal/audit`, `internal/certpaths`, `cmd/orca`).
|
||||
v0.8 also closes the two SSH-trust "future enhancement" hooks deferred
|
||||
in v0.6 (D-035 `--host-key-fingerprint` pre-pin, RESEARCH_v0.6 §80
|
||||
`orca node key-reset`) and adds a requirements-hygiene gate to prevent
|
||||
the stale-REQ-status drift seen after v0.7 ship.
|
||||
|
||||
- [x] Phase 0: Pre-execution (specify → clarify → research → plan → grill) — tag `v0.7.0` (shipped)
|
||||
- [x] Phase 1: Test coverage uplift round 2 — 6 packages to ≥ 70%, 3 zero-test packages to first tests (REQ-057) — tag `v0.7.1` (shipped)
|
||||
- [x] Phase 2: SSH trust hardening — `--host-key-fingerprint` pre-pin + `orca node key-reset` + TOFU bugfix + `HostKeyFingerprint` population (REQ-058, REQ-059) — tag `v0.7.2` (shipped)
|
||||
- [x] Phase 3: Requirements-hygiene gate — `make verify-reqs` + verify assertion (REQ-060) — tag `v0.7.3` (shipped)
|
||||
- [x] Phase 4: Final review + ship + audit (milestone release) — tag `v0.7.4` (shipped)
|
||||
|
||||
**Milestone type**: NFR (P01 test, P02 chore on trust surface per
|
||||
D-043, P03 chore, P04 docs/review). Final phase patch IS the milestone
|
||||
release per NFR-milestone progressive-patch rule. Per-phase tags:
|
||||
`v0.7.0`…`v0.7.4`. Tags run on the previous minor's patch line (v0.7.x)
|
||||
per branch-strategy.md. The milestone branch label uses the milestone
|
||||
number (`milestone/v0.8-coverage-trust-hardening`); no separate minor
|
||||
tag.
|
||||
|
||||
### Per-phase REQ coverage
|
||||
|
||||
- **P01 — Coverage uplift round 2**
|
||||
- REQ-057 (raise `internal/engine`, `internal/proxmox`,
|
||||
`internal/cli`, `internal/transport`, `internal/store`,
|
||||
`internal/jobspec` to ≥ 70%; add first tests for `internal/audit`,
|
||||
`internal/certpaths`, `cmd/orca`)
|
||||
|
||||
- **P02 — SSH trust hardening**
|
||||
- REQ-058 (`--host-key-fingerprint <sha256>` pre-pin flag on
|
||||
`orca node join --type proxmox`; fail fast on mismatch; supersedes
|
||||
TOFU for pre-pinned deployments)
|
||||
- REQ-059 (`orca node key-reset <node>` clears persisted SSH host
|
||||
key so next `doctor proxmox`/dispatch re-pins via TOFU or
|
||||
`--host-key-fingerprint`)
|
||||
|
||||
- **P03 — Requirements-hygiene gate**
|
||||
- REQ-060 (`make verify-reqs` target + verify-stage assertion:
|
||||
every REQ `Complete` in ROADMAP.md has matching `Complete` row in
|
||||
REQUIREMENTS.md; enforced in CI `validate` pipeline)
|
||||
|
||||
### v0.8 is a continuation milestone, not a direction change
|
||||
|
||||
The vision ("minimalist, offline-first, CLI-first orchestration
|
||||
engine") is unchanged. v0.8 closes the coverage debt left by v0.7's
|
||||
50% floor and the trust-surface gaps explicitly deferred in v0.6.
|
||||
|
||||
## Milestone v0.9: Re-architecture Foundation & Workloads — **COMPLETE**
|
||||
|
||||
**Scope**: This milestone SUPERSPEDES the shipped v0.1–v0.8 architecture per
|
||||
the adopted PRD (`.ciagent/PRD_v0.9.md`). The re-architecture is justified on
|
||||
six grounds recorded in the PROJECT.md Supersession Table: (1) the v0.8 daemon
|
||||
model is operationally failing, (2) step-ca is externally mandated, (3)
|
||||
multi-tenancy is a hard product requirement, (4) WASM is a hard workload
|
||||
requirement, (5) SSH-push is the only viable deployment target, (6) vision
|
||||
correction. The 16 load-bearing rules (R-001…R-016) are invariants. The
|
||||
ci-griller reviewed the re-architecture adversarially; the user overrode the
|
||||
Re-architecture Justification REPLAN with the six-part evidence basis; the
|
||||
19 binding conditions (C-01..C-19) and 10 phase challenges (PC-01..PC-10)
|
||||
from `GRILL_v0.9.md` are adopted as execution gates. 30 net-new requirements
|
||||
(REQ-061..REQ-090) derive from `IDEATION_v0.9.md`.
|
||||
|
||||
**Milestone type**: feature (P01..P10 ship `feat` phases; P00/P0X are
|
||||
chore/docs).
|
||||
|
||||
- [ ] Phase 0: Pre-execution (specify → clarify → research → ideate → plan → grill) — tag `v0.8.0` (shipped; this is the phase you are reading)
|
||||
- [x] Phase P00: Deprecation sweep + bash tooling gate + render contract + doc banners (REQ-068,072,088,089,090; gates C-03,C-05,C-06,C-15..C-18) — tag `v0.8.1` ✓
|
||||
- [x] Phase P0a1: Multi-namespace path resolver + config demotion + known_hosts flock (REQ-063,069,070,071; gate C-07) — tag `v0.8.2` ✓
|
||||
- [x] Phase P0a2: Namespace CRUD + inheritance engine (REQ-082) — tag `v0.8.3` ✓
|
||||
- [x] Phase P0b: Markdown jobspec parser + dispatcher + fuzz (REQ-064,067) — tag `v0.8.4` ✓
|
||||
- [x] Phase P0c: Job/Service/DaemonSet schemas + emitter interface (REQ-074) — tag `v0.8.5` ✓
|
||||
- [x] Phase P01: SSH-push transport (REQ-073) — tag `v0.8.6` ✓
|
||||
- [x] Phase P02: Service block + Traefik emitter (REQ-077; gate C-10) — tag `v0.8.7` ✓
|
||||
- [x] Phase P03/P04/P08: Update stanza + lifecycle hooks + socket plumbing (combined) — tag `v0.8.8` ✓
|
||||
- [x] Phase P05: CLI-side scheduler + CEL constraints (REQ-083) — tag `v0.8.9` ✓
|
||||
- [x] Phase P06: Task groups (multi-process services) — tag `v0.8.10` ✓
|
||||
- [x] Phase P07a/b/c: Runtime abstraction — 5 backends (REQ-078; gate C-01) — tag `v0.8.11` ✓
|
||||
- [x] Phase P09: Syncthing storage replication (REQ-081; gates C-02,C-14) — tag `v0.8.12` ✓
|
||||
- [x] Phase P10: Lead rules + step-ca (REQ-076) — tag `v0.8.13` ✓
|
||||
- [x] Phase P0X: Ship + audit (REQ-062,068) — tag `v0.8.14` ✓
|
||||
|
||||
**Milestone tag**: `v0.8.15` (final phase patch = milestone release per
|
||||
feature-milestone progressive-patch rule). Per-phase tags: `v0.8.1`…`v0.8.14`.
|
||||
P03/P04/P08 were combined into one phase; P07a/b/c were combined into one
|
||||
phase. Actual execution: 14 tagged phases. Tags run on the previous minor's
|
||||
patch line (v0.8.x) per branch-strategy.md. The milestone branch label uses
|
||||
the milestone number (`milestone/v0.9-rearchitecture`); no separate minor tag.
|
||||
|
||||
### Per-phase REQ coverage (v0.9)
|
||||
|
||||
- **P00** — Deprecation/migration/test-infra/persona/docs foundation (REQ-072, REQ-085, REQ-088, REQ-089, REQ-090)
|
||||
- **P0a1** — Path resolver + config demotion + known_hosts flock (REQ-063, REQ-069, REQ-070, REQ-071)
|
||||
- **P0a2** — Namespace inheritance resolver (REQ-082)
|
||||
- **P0b** — Markdown parser + adapter + fuzz (REQ-064, REQ-067)
|
||||
- **P0c** — Schemas + emitter interface (REQ-074)
|
||||
- **P01** — SSH-push transport (REQ-073)
|
||||
- **P02** — Service + Traefik emitter (REQ-077)
|
||||
- **P05** — CLI-side scheduler (REQ-083)
|
||||
- **P07a/b/c** — Runtime abstraction (REQ-078) + step-ca integration (REQ-076)
|
||||
- **P09** — Syncthing replication (REQ-081)
|
||||
- **P0X** — Coverage gate (REQ-062) + deprecation warnings (REQ-068)
|
||||
|
||||
### v0.9 is a DIRECTION CHANGE — first in the project's history
|
||||
|
||||
Every prior milestone (v0.1–v0.8) explicitly said "the vision is unchanged;
|
||||
this milestone is not a direction change." v0.9 is the first milestone that
|
||||
reverses the vision's anti-patterns (daemon-on-every-node, internal CA,
|
||||
HCL-canonical, single-namespace, no-container-runtime, no-SPIFFE). The
|
||||
reversals are justified by the six-part evidence basis recorded in the
|
||||
PROJECT.md Supersession Table.
|
||||
|
||||
## Milestone v0.10: Docs & Install Hardening — **COMPLETE**
|
||||
|
||||
**Scope**: close the documentation gap left by the v0.9 re-architecture
|
||||
and fix the release/install pipeline bug that caused `install.sh` to
|
||||
resolve to v0.4.5 instead of the latest release. The v0.9
|
||||
re-architecture shipped a complete CLI surface (markdown jobspec,
|
||||
`orca ns`, `orca node capacity`, CLI-side scheduler, emitters, Traefik
|
||||
ingress) but no operator-facing reference documentation. This milestone
|
||||
ships that documentation plus a worked full-stack example with ingress
|
||||
configured, and hardens the release pipeline so every Gitea release
|
||||
carries a Linux binary asset.
|
||||
|
||||
**Milestone type**: feature (P1 ships `fix` phases; P2/P3/P4 ship `docs`
|
||||
phases; at least one non-docs phase makes this a feature milestone per
|
||||
the versioning logic).
|
||||
|
||||
- [x] Phase 0: Pre-execution (specify → clarify → research → ideate → plan → grill) — tag `v0.9.0`
|
||||
- [x] Phase P1: release.sh + install.sh fix (REQ-097, REQ-098) — tag `v0.9.1`
|
||||
- [x] Phase P2: docs/cli.md + docs/jobspec.md + docs/ingress.md (REQ-091, REQ-092, REQ-093) — tag `v0.9.2`
|
||||
- [x] Phase P3: examples/full-stack/ (REQ-094) — tag `v0.9.3`
|
||||
- [x] Phase P4: README.md + docs/namespace.md refresh (REQ-095, REQ-096) — tag `v0.9.4`
|
||||
- [x] Phase P5: Final review + ship + audit (milestone release) — tag `v0.9.5` = v0.10.0 milestone release
|
||||
|
||||
**Milestone tag**: `v0.9.5` (final phase patch = milestone release per
|
||||
feature-milestone progressive-patch rule). Per-phase tags: `v0.9.0`…`v0.9.5`.
|
||||
Tags run on the previous minor's patch line (v0.9.x) per
|
||||
branch-strategy.md. The milestone branch label uses the milestone
|
||||
number (`milestone/v0.10-docs-cli-examples`); no separate minor tag.
|
||||
|
||||
### Per-phase REQ coverage (v0.10 docs milestone)
|
||||
|
||||
- **P1** — release.sh cross-build + asset verification (REQ-097); install.sh fallback walk (REQ-098)
|
||||
- **P2** — CLI reference (REQ-091); jobspec reference (REQ-092); ingress guide (REQ-093)
|
||||
- **P3** — full-stack examples (REQ-094)
|
||||
- **P4** — README refresh (REQ-095); namespace.md v0.9 layout (REQ-096)
|
||||
|
||||
### Root cause of the v0.4.5 install (documented in RESEARCH_v0.10.md)
|
||||
|
||||
The v0.8.x releases (v0.8.0–v0.8.15) shipped with zero binary assets
|
||||
attached to their Gitea releases. `install.sh` resolves "latest" →
|
||||
v0.8.15, looks for `orca-v0.8.15-linux-amd64.tar.gz`, finds nothing, and
|
||||
errors out. The v0.4.5 install came from an earlier run or a pinned
|
||||
`--version`. The fix is forward: release.sh cross-builds amd64 and
|
||||
verifies the asset post-create; install.sh walks backward through
|
||||
releases if the latest lacks the asset.
|
||||
|
||||
## Milestone v0.11: Production Hardening — **COMPLETE**
|
||||
|
||||
**Scope**: ship a cluster that operators can run. Builds on the v0.9
|
||||
re-architecture foundation with the production-grade subsystems:
|
||||
secrets, transactions, ACL/SPIFFE, backup/restore, drain, recovery, and
|
||||
the v0.8→v1.0 migration. **Phase 0 adopts 4 new load-bearing rules
|
||||
(R-017…R-020) and 23 new decisions (D-215…D-237) from 5 research docs
|
||||
covering ingress hardening, drift detection, platform-engineer
|
||||
positioning, strategic framing, and the systemd Path unit
|
||||
implementation.** No new phases added; scope is folded into existing
|
||||
phases per operator decisions Q2=C (add 5 CLI commands), Q3=A (fold
|
||||
ingress into P15.5).
|
||||
|
||||
**Milestone type**: feature (multiple `feat` phases).
|
||||
|
||||
- [x] Phase 0: Pre-execution (specify → clarify → research → plan → grill) — tag `v0.10.0`
|
||||
- [x] Phase P00: CLI cache layer (REQ-062 cache floor; R-008) — tag `v0.10.1`
|
||||
- [x] Phase P01: Metrics endpoint (hand-rolled text exposition) — tag `v0.10.2`
|
||||
- [x] Phase P01.5: SPIFFE SVID minting spike (REQ-076; **gate C-08** — if spike fails, fall back to mTLS identity) — tag `v0.10.3`
|
||||
- [x] Phase P02: ACL (SPIFFE + token identities) — tag `v0.10.4`
|
||||
- [x] Phase P03: Secrets subsystem (REQ-080; **gate C-19** threat model) — tag `v0.10.5`
|
||||
- [x] Phase P04: Backup/restore (tar + signed) — tag `v0.10.6`
|
||||
- [x] Phase P05: Drain + daemon drain-and-stop (REQ-061) + **`orca job migrate` (REQ-116)** — tag `v0.10.7`
|
||||
- [x] Phase P06: Alloc history (CLI-side SQLite retention; REQ-071 cache DB) + **`orca logs --all-nodes --since` (REQ-117)** — tag `v0.10.8`
|
||||
- [x] Phase P07: Recovery (`orca restore`) — tag `v0.10.9`
|
||||
- [x] Phase P08: Integration tests — expand hermetic harness (REQ-087) + **drift-detection integration tests (auto-remediation, NFS fallback, cooldown, secret exclusion)** — tag `v0.10.10`
|
||||
- [x] Phase P09: Collector + aggregator (opt-in; **gates C-11, C-12, C-14**) + **drift-event aggregation extension (REQ-107, D-237)** — tag `v0.10.11`
|
||||
- [x] Phase P10a: Transactional plane (REQ-075, REQ-079; **gate C-09**; **gate C-23** cluster-wide vs ns-scoped txn distinction) — tag `v0.10.12`
|
||||
- [x] Phase P10b: Drift detection (R-018/R-019/R-020; REQ-103..REQ-113; `orca drift` CLI, systemd Path unit emitter, `orca-drift-notify.sh`, `orca-remediate.sh`, cadence config, `--force`+per-ns gate, `orca` system user, NFS detection) — depends on P10a — tag `v0.10.13`
|
||||
- [x] Phase P11: `orca job lint` (REQ-084) — tag `v0.10.14`
|
||||
- [x] Phase P12: `orca job verify` (dry-run txn through lead) — tag `v0.10.15`
|
||||
- [x] Phase P13: `orca ns` subcommands (full surface) + deprecation warnings (REQ-068) — tag `v0.10.16`
|
||||
- [x] Phase P14a: v0.8→v1.0 data migration (REQ-066; **gate C-07**; **gate C-25** post-cutover verification + rollback; **gate C-27** orca user creation) + **`orca upgrade --to-vX` (REQ-115, thin wrapper, handles R-017 binding cutover)** — tag `v0.10.17`
|
||||
- [x] Phase P14b: Daemon cutover + running-allocation adoption + **`orca cluster rotate-lead` (REQ-114)** — tag `v0.10.18`
|
||||
- [x] Phase P14c: Mixed-version tolerance + no-orca-on-server enforcement (REQ-065, REQ-086; implements C-13) — tag `v0.10.19`
|
||||
- [x] Phase P15: README quickstart (REQ-089; **Nomad-inspired framing per Q5=A, honest-trade-offs table from research doc 3**) — tag `v0.10.20`
|
||||
- [x] Phase P15.5: Threat model + security review (**gate C-19**; **gate C-28** two sub-waves) + **ingress hybrid (R-017; nft emitter REQ-099, Traefik binding REQ-100, `orca doctor nft` REQ-101, `orca nft` CLI REQ-102) + `orca doctor mTLS` (REQ-118)** — tag `v0.10.21`
|
||||
- [x] Phase P16: Final review + ship + audit — **v0.11.0 milestone release** — tag `v0.10.22` (v1.0.0 cut separately after UAT sign-off)
|
||||
|
||||
**Milestone tag**: `v0.11.0` (the v0.11 milestone release tag; v1.0.0 is
|
||||
UAT-gated and cut separately after v0.11 completion per operator decision —
|
||||
the v1.0.0 tag marks production-ready sign-off, not a separate milestone).
|
||||
Per-phase patches run on the v0.10.x line per branch-strategy.md. Per-phase
|
||||
tags: `v0.10.0`…`v0.10.21`.
|
||||
|
||||
### Per-phase REQ coverage (v0.11)
|
||||
|
||||
- **P00** — CLI cache (R-008)
|
||||
- **P01.5** — SPIFFE spike (REQ-076; C-08)
|
||||
- **P03** — Secrets (REQ-080; C-19)
|
||||
- **P05** — Drain + daemon stop (REQ-061) + `orca job migrate` (REQ-116)
|
||||
- **P06** — Alloc history (REQ-071 cache DB) + `orca logs --all-nodes --since` (REQ-117)
|
||||
- **P08** — Integration tests (REQ-087) + drift-detection integration tests
|
||||
- **P09** — Collector + aggregator (C-11, C-12, C-14) + drift-event aggregation (REQ-107, D-237)
|
||||
- **P10a** — Transactional plane (REQ-075, REQ-079; C-09; C-23)
|
||||
- **P10b** — Drift detection (R-018/R-019/R-020; REQ-103..REQ-113)
|
||||
- **P11** — Job lint (REQ-084)
|
||||
- **P13** — ns subcommands + deprecation warnings (REQ-068)
|
||||
- **P14a/b/c** — Migration (REQ-066, REQ-065, REQ-086; C-07, C-13) + `orca upgrade` (REQ-115) + `orca cluster rotate-lead` (REQ-114)
|
||||
- **P15** — README (REQ-089; Q5=A framing)
|
||||
- **P15.5** — Threat model (C-19) + ingress hybrid (R-017; REQ-099..REQ-102) + `orca doctor mTLS` (REQ-118)
|
||||
|
||||
### New load-bearing rules adopted in Phase 0
|
||||
|
||||
- **R-017** — Ingress hybrid: nft DNAT → Traefik on `127.0.0.1:8443`; opt-out via `--public-binding`; `service { ingress: native }` per-workload opt-in
|
||||
- **R-018** — Drift cadence: default 60s; critical 5s + systemd Path units; standard 30s
|
||||
- **R-019** — Drift detector is a BACKSTOP; primary = systemd/Traefik/step-ca/Syncthing
|
||||
- **R-020** — Hard gate: applier refuses txns on pre-flight drift; `--force` + per-ns scoping override
|
||||
|
||||
### Risk register (from grill + research, for ongoing monitoring)
|
||||
|
||||
- **step-ca single-instance SPOF** (mitigation: C-12 doc; v1.x HA via systemd failover)
|
||||
- **master.key passphrase-less 0600** (mitigation: C-19 threat model; consider OS keyring in v1.x)
|
||||
- **wasmtime CGO breaks cross-compile** (mitigation: C-01 spike; fallback to podman/process primary)
|
||||
- **bash control plane drift** (mitigation: C-15..C-18 render-format contract + bats gate)
|
||||
- **daemon cutover orphans running allocs** (mitigation: P14b split; test adoption)
|
||||
- **27→35+ phase scope** (mitigation: C-04 resolved — operator accepted 40 phases; v0.11 grows to 24 phases per grill C-24 split of P10→P10a/P10b; scope folded in, no other new phases)
|
||||
- **R-020 deadlock** (mitigation: `--force` flag + per-namespace scoping per Q4=A; drifted peer in ns-A doesn't block ns-B)
|
||||
- **P10 sizing** (mitigation: P10 is the largest phase — drift detection + txn plane; grill may split into P10a/P10b if vertical slice is too large)
|
||||
- **Ingress default migration** (mitigation: `orca upgrade` [REQ-115] handles Traefik binding cutover from `:443` to `127.0.0.1:8443` for existing v0.9/v0.10 clusters)
|
||||
- **`orca` system user on peers** (mitigation: net-new operational requirement; peer-setup emits `useradd -r orca` idempotently; documented in P10)
|
||||
|
||||
## Deferred to v1.x (out of scope for v0.11)
|
||||
|
||||
- `sqlite-wal-shared` state backend (R-009 abstractions ship in v1.0; backend in v1.x)
|
||||
- `git` state backend
|
||||
- `file+flock` state backend
|
||||
- `orca cluster setup-shared` UX
|
||||
- HA `step-ca` (active/passive via systemd)
|
||||
- Journald log shipping (optional centralized audit)
|
||||
- Network policy (`nftables` snippets)
|
||||
- GPU / TPU constraints
|
||||
|
||||
## Deferred to v2.x (out of scope for v1.x)
|
||||
|
||||
- Full Nomad-HCL parser with no conversion round-trip
|
||||
- Nomad-API subset for migrating existing Nomad fleets
|
||||
- Nomad driver bridge
|
||||
- Helm-equivalent templating (probably never)
|
||||
- Service mesh beyond Traefik
|
||||
- CRDs / Operators / Plugin model
|
||||
- Leader-elected Raft coordinator
|
||||
- External CA / Let's Encrypt / cert transparency
|
||||
- Online-only features (HSTS, OCSP stapling, telemetry)
|
||||
|
||||
## Milestone v0.12: Security Hardening (Zero-Trust Identity) — **COMPLETE**
|
||||
|
||||
**Scope**: comprehensive security hardening across the entire attack
|
||||
surface, **including the operating system itself**, plus adoption of a
|
||||
zero-trust identity model. The v0.12 threat-model review (Phase 0
|
||||
RESEARCH) surfaced 25 distinct findings (F1..F25) spanning injection,
|
||||
traversal, ACL, audit, crypto, OS scripts, emitters, sudoers, system
|
||||
users, file modes, daemon auth, backup, SQLite, install.sh, and
|
||||
migration. v0.12 closes all of them and adopts **R-021** (no Orca
|
||||
credentials) as the load-bearing architectural change: human identity is
|
||||
exclusively external (OIDC), machine identity is exclusively
|
||||
mTLS/SPIFFE, and no passwords/Orca-issued-tokens/CA-key-passphrases
|
||||
exist anywhere in the system.
|
||||
|
||||
The operator locked two architectural decisions: **(1) bundled Dex by
|
||||
default + BYO external IdP override** (D-239), and **(2) master key
|
||||
seal-to-OIDC + Shamir 3-of-5 recovery** (D-241). A third decision added
|
||||
**WebAuthn (passkeys) as the bundled password-free authenticator** for
|
||||
Dex (D-240) -- passkeys are public-key credentials (private key never
|
||||
leaves the authenticator), directly satisfying R-021.
|
||||
|
||||
**Milestone type**: feature (P04 OIDC+Dex and P05 WebAuthn ship `feat`
|
||||
phases; the rest are `fix`/`chore`/`test`/`docs`/`refactor`).
|
||||
|
||||
- [x] Phase 0: Pre-execution (specify -> clarify -> research -> ideate -> plan -> grill) -- tag `v0.11.0`
|
||||
- [x] Phase P0[0-9]: Command injection fix (podman/wasm shellQuote) (REQ-119, F3) -- tag `v0.11.1`
|
||||
- [x] Phase P0[0-9]: Namespace path traversal fix (REQ-120, F4) -- tag `v0.11.2`
|
||||
- [x] Phase P0[0-9]: Txn apply path allowlist (REQ-121, F5) -- tag `v0.11.3`
|
||||
- [x] Phase P0[0-9]: OIDC client + bundled Dex (REQ-144; BYO-IdP override) -- tag `v0.11.4`
|
||||
- [x] Phase P0[0-9]: WebAuthn connector for Dex (REQ-148; passkeys, browser auth+register) -- tag `v0.11.5`
|
||||
- [x] Phase P0[0-9]: ACL rewrite to OIDC claims + enforcement (REQ-145, REQ-122, F1) -- tag `v0.11.6`
|
||||
- [x] Phase P0[0-9]: Remove all password/token paths (breaking; REQ-146, R-021, C-34) -- tag `v0.11.7`
|
||||
- [x] Phase P0[0-9]: Master key seal-to-OIDC + Shamir 3-of-5 (REQ-147, C-35) -- tag `v0.11.8`
|
||||
- [x] Phase P0[0-9]: Daemon auth hardening (REQ-123, REQ-124, F6, F24) -- tag `v0.11.9`
|
||||
- [x] Phase P0+: Audit log tamper-evidence (REQ-125, F2) -- tag `v0.11.10`
|
||||
- [x] Phase P0+: SVID chain validation (REQ-126, F9) -- tag `v0.11.11`
|
||||
- [x] Phase P0+: Backup symlink validation (REQ-127, F7) -- tag `v0.11.12`
|
||||
- [x] Phase P0+: step-ca /tmp hardening (REQ-128, F10) -- tag `v0.11.13`
|
||||
- [x] Phase P0+: Master key rotation (re-seal to OIDC; REQ-129, F12, C-30) -- tag `v0.11.14`
|
||||
- [x] Phase P0+: File-mode audit expansion (REQ-130, F13) -- tag `v0.11.15`
|
||||
- [x] Phase P0+: aggregate.sh JSON injection + drift-gate parse fix (REQ-131, F11, F18) -- tag `v0.11.16`
|
||||
- [x] Phase P0+: install.sh checksum+GPG verification (REQ-132, F14) -- tag `v0.11.17`
|
||||
- [x] Phase P0+: nftables ruleset hardening (REQ-133, F21) -- tag `v0.11.18`
|
||||
- [x] Phase P0+: sudoers hardening (REQ-134, F22) -- tag `v0.11.19`
|
||||
- [x] Phase P0+: System user consistency (REQ-135, F23) -- tag `v0.11.20`
|
||||
- [x] Phase P0+: SQLite file-mode + at-rest encryption (REQ-136, F8, C-31) -- tag `v0.11.21`
|
||||
- [x] Phase P0+: Migration safety + identity migration (REQ-137, F19, C-34) -- tag `v0.11.22`
|
||||
- [x] Phase P0+: Legacy CA/mTLS/daemon + step-ca password-provisioner deletion (REQ-138, F16; **gate C-29: P06/P08/P09/P11**) -- tag `v0.11.23`
|
||||
- [x] Phase P0+: known_hosts tightening + transport hardening (REQ-139, F15, F25) -- tag `v0.11.24`
|
||||
- [x] Phase P0+: Drift event authentication (REQ-140, F18) -- tag `v0.11.25`
|
||||
- [x] Phase P0+: Security integration test suite (REQ-141, C-33) -- tag `v0.11.26`
|
||||
- [x] Phase P0+: Zero-trust + OIDC + WebAuthn + threat-model docs (REQ-142) -- tag `v0.11.27`
|
||||
- [x] Phase P0+: Final review + ship + audit (milestone release) -- tag `v0.11.28` = **v0.12 milestone release**
|
||||
|
||||
**Milestone tag**: `v0.11.28` (final phase patch = milestone release per
|
||||
feature-milestone progressive-patch rule; no separate `v0.12.0` tag).
|
||||
Per-phase tags: `v0.11.0`..`v0.11.28` (29 tags). Tags run on the
|
||||
previous minor's patch line (v0.11.x) per branch-strategy.md. The
|
||||
milestone branch label uses the milestone number
|
||||
(`milestone/v0.12-security-hardening`); no separate minor tag.
|
||||
|
||||
The v1.0.0 production-ready tag stays deferred for post-v0.12 UAT
|
||||
(per v0.11 PRD; v0.12 is a minor feature milestone, not the v1.0 cut).
|
||||
|
||||
### Per-phase REQ coverage (v0.12)
|
||||
|
||||
- **P01** -- Command injection (REQ-119, F3)
|
||||
- **P02** -- Namespace path traversal (REQ-120, F4)
|
||||
- **P03** -- Txn apply path allowlist (REQ-121, F5)
|
||||
- **P04** -- OIDC client + bundled Dex (REQ-144; D-239, D-242, D-246)
|
||||
- **P05** -- WebAuthn connector (REQ-148; D-240, D-243, D-244, C-38)
|
||||
- **P06** -- ACL rewrite + enforcement (REQ-145, REQ-122, F1)
|
||||
- **P07** -- Remove password/token paths (REQ-146, R-021, C-34)
|
||||
- **P08** -- Master key seal-to-OIDC + Shamir (REQ-147, D-241, C-35)
|
||||
- **P09** -- Daemon auth (REQ-123, REQ-124, F6, F24)
|
||||
- **P10** -- Audit tamper-evidence (REQ-125, F2)
|
||||
- **P11** -- SVID chain validation (REQ-126, F9)
|
||||
- **P12** -- Backup symlink validation (REQ-127, F7)
|
||||
- **P13** -- step-ca /tmp hardening (REQ-128, F10)
|
||||
- **P14** -- Master key rotation (REQ-129, F12, C-30)
|
||||
- **P15** -- File-mode audit expansion (REQ-130, F13)
|
||||
- **P16** -- aggregate.sh JSON injection + drift-gate (REQ-131, F11, F18)
|
||||
- **P17** -- install.sh checksum+GPG (REQ-132, F14)
|
||||
- **P18** -- nftables ruleset hardening (REQ-133, F21)
|
||||
- **P19** -- sudoers hardening (REQ-134, F22)
|
||||
- **P20** -- System user consistency (REQ-135, F23)
|
||||
- **P21** -- SQLite file-mode + encryption (REQ-136, F8, C-31)
|
||||
- **P22** -- Migration safety + identity migration (REQ-137, F19, C-34)
|
||||
- **P23** -- Dual-write closure (REQ-138, F16; **gate C-29**)
|
||||
- **P24** -- known_hosts + transport hardening (REQ-139, F15, F25)
|
||||
- **P25** -- Drift event authentication (REQ-140, F18)
|
||||
- **P26** -- Security integration test suite (REQ-141, C-33)
|
||||
- **P27** -- Zero-trust + OIDC + WebAuthn + threat-model docs (REQ-142)
|
||||
- **P28** -- Final review + ship + audit (REQ-143)
|
||||
|
||||
### New load-bearing rule adopted in Phase 0
|
||||
|
||||
- **R-021** -- Orca never issues, stores, or accepts human-identity
|
||||
credentials. Human identity is exclusively external (OIDC). Machine
|
||||
identity is exclusively mTLS/SPIFFE. No passwords, no Orca-issued
|
||||
tokens, no CA-key passphrases.
|
||||
|
||||
### Binding conditions (for GRILL ratification; C-29..C-38)
|
||||
|
||||
- **C-29**: P23 (dual-write closure) gated on P06/P08/P09/P11 all shipped.
|
||||
- **C-30**: P14 (master key rotation) reversible; `--dry-run` mandatory; auto-rollback to old sealed key on any ns failure.
|
||||
- **C-31**: P21 (SQLite encryption): CGO-free fallback to file-mode 0600 + documented threat if SQLCipher needs CGO. No CGO.
|
||||
- **C-32**: **Human-gate**: leaked GITEA_TOKEN (F17) rotated + `.env` re-seeded before P28 ships. History-scrub best-effort, non-blocking. Escalation hook in `---ci---`.
|
||||
- **C-33**: P26 (security integration tests) in `.coreci.yml` `validate`, gates merges -- not opt-in.
|
||||
- **C-34**: P07 (password/token removal) breaking. `orca upgrade` (P22) refuses v0.11 clusters using `--password`/bare-tokens without `--accept-identity-migration`. No silent breakage.
|
||||
- **C-35**: P08 (Shamir recovery): 3-of-5 shards printed at seal time, operator stores offline. If IdP lost AND quorum unavailable -> cluster unrecoverable by design (documented residual risk). No backdoor.
|
||||
- **C-36**: OIDC client secret (confidential clients) at `ClusterDir()/oidc-client-secret` (0600), rotatable via `orca auth rotate-client-secret`, never committed. Public PKCE clients avoid even this.
|
||||
- **C-37**: P04 (bundled Dex): if WebAuthn proves infeasible, bundled Dex ships mTLS-client-cert-only; password-based upstreams require BYO external IdP. The "no Orca credentials" invariant holds regardless. *(Largely moot -- WebAuthn solves it.)*
|
||||
- **C-38**: P05 (WebAuthn): RP ID must match the cluster's Traefik-served domain; `orca auth init-idp` configures it. HTTPS secure context via Traefik (step-ca cert). P26 integration tests use the WebAuthn virtual-authenticator API -- no hardware key required in CI.
|
||||
|
||||
### Risk register (from grill + research, for ongoing monitoring)
|
||||
|
||||
- **P07 breaking change** (mitigation: C-34 migration gate)
|
||||
- **P08 master key seal is riskiest** (mitigation: `--dry-run`, atomic, auto-rollback, C-35 Shamir recovery)
|
||||
- **P21 SQLite encryption may need CGO** (mitigation: C-31 fallback to file-mode 0600)
|
||||
- **P23 dual-write closure high-impact** (mitigation: gate C-29; full test coverage before deletion)
|
||||
- **P05 WebAuthn connector is new ground** (mitigation: C-37 mTLS-client-cert fallback; virtual-authenticator tests in P26)
|
||||
- **Bundled Dex is a new systemd unit + Traefik route** (mitigation: `orca doctor oidc` health check)
|
||||
- **C-32 human gate could stall final ship** (mitigation: ship as `v0.11.28-rc1` if rotation pending)
|
||||
- **29 phases is large** (mitigation: grill may split/merge; operator accepted "more than 20 if warranted")
|
||||
|
||||
### Deferred to v1.x (out of scope for v0.12)
|
||||
|
||||
- HA step-ca (active/passive via systemd)
|
||||
- `sqlite-wal-shared` / `git` / `file+flock` state backends
|
||||
- OS keyring integration for master key (v0.12 uses OIDC seal instead)
|
||||
- Full cluster-rolling-upgrade orchestrator (v0.12 ships the thin `orca upgrade` wrapper only)
|
||||
- Live-migrate with storage replication (v0.12 ships drain+reschedule only)
|
||||
- Journald log shipping (optional centralized audit)
|
||||
- Network policy (`nftables` snippets beyond the ingress ruleset)
|
||||
- GPU / TPU constraints
|
||||
|
||||
### Deferred to v2.x (out of scope for v1.x)
|
||||
|
||||
- Full Nomad-HCL parser with no conversion round-trip
|
||||
- Nomad-API subset for migrating existing Nomad fleets
|
||||
- Nomad driver bridge
|
||||
- Helm-equivalent templating (probably never)
|
||||
- Service mesh beyond Traefik
|
||||
- CRDs / Operators / Plugin model
|
||||
- Leader-elected Raft coordinator
|
||||
- External CA / Let's Encrypt / cert transparency
|
||||
- Online-only features (HSTS, OCSP stapling, telemetry)
|
||||
|
||||
## Milestone v0.13: Production Hardening Round 2 + UAT Plan — **COMPLETE**
|
||||
|
||||
**Scope**: final production hardening round before the v1.0.0
|
||||
production-ready tag. Three deep codebase sweeps (security, reliability,
|
||||
feature/doc claims) surfaced ~60 gaps beyond v0.12 — the most critical
|
||||
being that `orca job run` runs locally via `exec.CommandContext` and
|
||||
never invokes the scheduler/emitter/SSH-push path (the documented
|
||||
deployment model is non-functional), jobspec `schedule:`/`timeout:` are
|
||||
silently dropped by the markdown parser (DaemonSet is fundamentally
|
||||
broken), `acl.Check` is called zero times in the codebase (v0.12's
|
||||
headline zero-trust feature is library-complete but not wired), and
|
||||
several command-injection vectors remain (`orca logs --job` backtick
|
||||
RCE via `%q`, tar-slip in restore, sudoers injection, etc.). v0.13
|
||||
closes all critical/high/medium findings and delivers the UAT plan +
|
||||
signoff script that gates the v1.0.0 cut.
|
||||
|
||||
**Load-bearing architectural changes**:
|
||||
- **R-022** — `orca job run` deploys to remote nodes via the scheduler
|
||||
→ emitter → SSH-push pipeline. The local `exec.CommandContext` path
|
||||
is removed. Constraints/capacity/affinity are enforced. This makes
|
||||
the documented deployment model functional and is the prerequisite
|
||||
for the UAT plan.
|
||||
- **R-023** — Zero-trust enforcement is operationally wired:
|
||||
`acl.Check` is invoked on every daemon handler + sshpush + txn apply
|
||||
path; `acl.json` is 0600; audit `actor` carries OIDC sub/SVID;
|
||||
WebAuthn registration requires auth; `cluster seal`/`unseal` +
|
||||
`doctor audit`/`doctor modes` CLI commands exist.
|
||||
|
||||
### Phases (14 total: P0 + P01..P12 + P13 final)
|
||||
|
||||
- [x] Phase P0: Pre-execution (SPECIFY→CLARIFY→RESEARCH→IDEATE→PLAN→GRILL) — tag `v0.12.0`
|
||||
- [x] Phase P01: Toolchain & dependency vulns (REQ-149) — tag `v0.12.1`
|
||||
- [x] Phase P02: Input validation & injection hardening (REQ-150) — tag `v0.12.2`
|
||||
- [x] Phase P03: Scheduler/deployment wiring + jobspec parser (REQ-151, REQ-152) — tag `v0.12.3`
|
||||
- [x] Phase P04: ACL enforcement + WebAuthn registration auth (REQ-153) — tag `v0.12.4`
|
||||
- [x] Phase P05: Seal/audit CLI + chain race + key zeroing (REQ-154) — tag `v0.12.5`
|
||||
- [x] Phase P06: auth init-idp real + auth register (REQ-155) — tag `v0.12.6`
|
||||
- [x] Phase P07: Concurrency safety (REQ-156) — tag `v0.12.7`
|
||||
- [x] Phase P08: Transport & SSH safety (REQ-157) — tag `v0.12.8`
|
||||
- [x] Phase P09: Migration & operational safety (REQ-158) — tag `v0.12.9`
|
||||
- [x] Phase P10: Observability & metrics (REQ-159) — tag `v0.12.10`
|
||||
- [x] Phase P11: Doc drift round 2 (REQ-160) — tag `v0.12.11`
|
||||
- [x] Phase P12: `--type linux` + UAT plan + signoff script (REQ-161, REQ-162, REQ-163) — tag `v0.12.12`
|
||||
- [x] Phase P13: Final review + ship + audit (milestone release) — tag `v0.12.13` = **v0.13 milestone release**
|
||||
|
||||
**Milestone tag**: `v0.12.13` (final phase patch = milestone release per
|
||||
feature-milestone rule; no separate `v0.13.0` tag). Per-phase tags:
|
||||
`v0.12.0`..`v0.12.13` (14 tags). Tags run on the previous minor's patch
|
||||
line (v0.12.x). The milestone branch label uses the milestone number
|
||||
(`milestone/v0.13-production-hardening-2`); no separate minor tag.
|
||||
|
||||
The v1.0.0 production-ready tag stays deferred for post-v0.13 UAT
|
||||
signoff (operator runs `scripts/uat-signoff.sh`, pastes output back;
|
||||
CI agent verifies and cuts v1.0.0).
|
||||
|
||||
### Per-phase REQ coverage (v0.13)
|
||||
|
||||
- **P01** — Toolchain bump (REQ-149)
|
||||
- **P02** — Injection hardening (REQ-150)
|
||||
- **P03** — Scheduler wiring + jobspec parser (REQ-151, REQ-152)
|
||||
- **P04** — ACL enforcement + WebAuthn reg auth (REQ-153)
|
||||
- **P05** — Seal/audit CLI + chain race + key zeroing (REQ-154)
|
||||
- **P06** — auth init-idp real + auth register (REQ-155)
|
||||
- **P07** — Concurrency safety (REQ-156)
|
||||
- **P08** — Transport & SSH safety (REQ-157)
|
||||
- **P09** — Migration & operational safety (REQ-158)
|
||||
- **P10** — Observability & metrics (REQ-159)
|
||||
- **P11** — Doc drift round 2 (REQ-160)
|
||||
- **P12** — `--type linux` + UAT plan + signoff (REQ-161, REQ-162, REQ-163)
|
||||
- **P13** — Final review + ship + audit
|
||||
|
||||
### New load-bearing rules adopted in Phase 0
|
||||
|
||||
- **R-022** — `orca job run` deploys to remote nodes via scheduler →
|
||||
emitter → SSH-push. Local exec path removed. Constraints/capacity/
|
||||
affinity enforced.
|
||||
- **R-023** — Zero-trust enforcement is operationally wired:
|
||||
`acl.Check` on every request path; `acl.json` 0600; audit actor =
|
||||
OIDC sub/SVID; WebAuthn registration requires auth.
|
||||
|
||||
### Binding conditions (for GRILL ratification — C-39..C-49)
|
||||
|
||||
- **C-39**: P03 (scheduler wiring) is the riskiest phase — changes the
|
||||
core `job run` path. Must not break existing `job run` (local
|
||||
fallback if no remote nodes registered). Full test coverage before
|
||||
P04 ships.
|
||||
- **C-40**: P04 (ACL enforcement) is deny-by-default — must not lock
|
||||
out the operator. Bootstrap ACL grants `cluster-admin` to the init
|
||||
cert's SPIFFE SVID. Staged rollout: log-only mode for first run,
|
||||
enforce after bootstrap ACL verified.
|
||||
- **C-41**: P05 (seal) — C-35 residual risk still applies (IdP lost +
|
||||
Shamir quorum unavailable → cluster unrecoverable). No backdoor.
|
||||
- **C-42**: P12 (UAT plan + signoff) is the v1.0 gate artifact. If
|
||||
P01..P11 slip, P12 still ships (honest signal via failing
|
||||
assertions). The signoff script is idempotent and read-only.
|
||||
- **C-43**: `verify-reqs` bold-format regex must be fixed in P11 so
|
||||
- **C-44**: P03 MUST fail-closed when scheduler selects a node but SSH-push fails. Local fallback only when `len(registeredNodes)==0`. Test case mandatory.
|
||||
- **C-45**: P04 MUST implement log-only/dry-run mode as default for first invocation after ACL wiring. Enforce mode after bootstrap ACL verified.
|
||||
- **C-46**: P12 dependency table MUST include P05 (seal) and P06 (auth init-idp) in addition to P03 and P04.
|
||||
- **C-47**: P12 `uat-signoff.sh` MUST include explicit assertions for: (a) job deployed to remote node, (b) ACL deny-by-default, (c) seal/unseal round-trip, (d) OIDC health check.
|
||||
- **C-48**: P12 `docs/uat.md` MUST document hardware prerequisites (Proxmox VE 8/9 host required). Alternative UAT path (3x Ubuntu, Proxmox claims skipped) MUST be documented.
|
||||
- **C-49**: Plan narrative MUST soften "last hardening round" to "last hardening round before UAT validation." UAT will likely surface 3-7 issues requiring patch release.
|
||||
the consistency gate works for v0.12 AND v0.13.
|
||||
|
||||
### Risk register (for grill + research, for ongoing monitoring)
|
||||
|
||||
- **P03 scheduler wiring is riskiest** (mitigation: C-39 local fallback)
|
||||
- **P04 ACL deny-by-default could lock out operator** (mitigation: C-40 bootstrap ACL + staged rollout)
|
||||
- **P05 seal residual risk** (mitigation: C-41 documented, no backdoor)
|
||||
- **P02 injection hardening is high-count** (11 sub-fixes; mitigation: each is small and independently testable)
|
||||
- **14 phases is large** (mitigation: operator accepted "no limit on phases"; many phases are small fix bundles)
|
||||
- **UAT plan depends on P03 (scheduler) being functional** (mitigation: P12 ships regardless; failing assertions are honest signal)
|
||||
|
||||
### Deferred to v1.x (out of scope for v0.13) — unchanged from v0.12
|
||||
|
||||
- HA step-ca (active/passive via systemd)
|
||||
- `sqlite-wal-shared` / `git` / `file+flock` state backends
|
||||
- OS keyring integration for master key
|
||||
- Full cluster-rolling-upgrade orchestrator (v0.13 ships the thin `orca upgrade` wrapper only)
|
||||
- Live-migrate with storage replication (v0.13 ships drain+reschedule only)
|
||||
- Journald log shipping (optional centralized audit)
|
||||
- Network policy (`nftables` snippets beyond the ingress ruleset)
|
||||
- GPU / TPU constraints
|
||||
- jobspec `health` prober (v0.13 adds lint warning; enforcement deferred)
|
||||
- jobspec `update` rolling/canary controller (v0.13 adds lint warning; enforcement deferred)
|
||||
- jobspec `schedule.cron` scheduler loop (v0.13 adds lint warning; enforcement deferred)
|
||||
|
||||
## Milestone v0.14: Ingress Bootstrap Completeness — **COMPLETE**
|
||||
|
||||
**Scope**: ensure that linux & proxmox types are properly bootstrapped with
|
||||
traefik during cluster init or node join. All cluster endpoints are
|
||||
provisioned as sockets (R-007); routing between jobs and services depends on
|
||||
traefik being present and properly configured. v0.13 shipped traefik binary +
|
||||
systemd unit + empty dynamic dir but never wrote the static config nor applied
|
||||
nft rules. v0.14 replaces the binary+systemd model with a **podman container**
|
||||
running a custom `orca-traefik` image, and completes the nft SNAT+DNAT ingress
|
||||
stack on every node type.
|
||||
|
||||
**New load-bearing rule**: **R-024** — Traefik runs exclusively as a podman
|
||||
container from the `orca-traefik` image published per release. Every
|
||||
orca-managed ingress surface bootstraps nft DNAT + SNAT/MASQUERADE +
|
||||
`podman run --restart=always --network host` with dynamic-config + step-ca
|
||||
root CA volume mounts.
|
||||
|
||||
**Three topologies**:
|
||||
1. **Linux**: host → nft → `podman run orca-traefik` (host network)
|
||||
2. **Proxmox Native**: host → nft → LXC (nesting=1) → `podman run orca-traefik`
|
||||
3. **Proxmox Floating-IP**: LXC (owns floating IP) → nft (inside LXC) → `podman run orca-traefik`
|
||||
|
||||
**Milestone type**: feature (multiple `feat` phases). Tags on v0.13.x patch
|
||||
line: `v0.13.0` (P0) ... `v0.13.8` (P8 final = v0.14 milestone release).
|
||||
|
||||
- [x] Phase 0: Pre-execution (SPECIFY→CLARIFY→RESEARCH→PLAN→GRILL) — tag `v0.13.0`
|
||||
- [x] Phase 1: `orca-traefik` container image + release pipeline (REQ-171) — tag `v0.13.1`
|
||||
- [x] Phase 2: Podman traefik reconciler — replace binary+systemd install (REQ-172) — tag `v0.13.2`
|
||||
- [x] Phase 3: nft SNAT+DNAT + `orca init` ingress bootstrap (REQ-173) — tag `v0.13.3`
|
||||
- [x] Phase 4: `orca node join --type linux` remote ingress bootstrap (REQ-174) — tag `v0.13.4`
|
||||
- [x] Phase 5: Proxmox native ingress mode — LXC + podman traefik (REQ-175) — tag `v0.13.5`
|
||||
- [x] Phase 6: Proxmox floating-IP LXC ingress + interactive prompt (REQ-176) — tag `v0.13.6`
|
||||
- [x] Phase 7: `doctor ingress` + docs + integration tests (REQ-177,178,179) — tag `v0.13.7`
|
||||
- [x] Phase 8: Final review + ship + audit (milestone release) — tag `v0.13.8` = **v0.14 milestone release**
|
||||
|
||||
### Per-phase REQ coverage (v0.14)
|
||||
|
||||
- **P1** — `Dockerfile.traefik` + release pipeline (REQ-171)
|
||||
- **P2** — Podman traefik reconciler (REQ-172)
|
||||
- **P3** — nft SNAT+DNAT + init bootstrap (REQ-173)
|
||||
- **P4** — Linux node join remote ingress (REQ-174)
|
||||
- **P5** — Proxmox native ingress — LXC + podman (REQ-175)
|
||||
- **P6** — Proxmox floating-IP LXC ingress (REQ-176)
|
||||
- **P7** — doctor ingress + docs + tests (REQ-177,178,179)
|
||||
- **P8** — Final review + ship + audit
|
||||
|
||||
### v0.14 is a continuation milestone, not a direction change
|
||||
|
||||
The vision ("minimalist, offline-first, CLI-first orchestration engine")
|
||||
is unchanged. v0.14 completes the ingress bootstrap that v0.13 left
|
||||
non-functional (binary installed but no config, no nft applied). The
|
||||
podman-container model is the operator's constraint; the architecture's
|
||||
socket+traefik routing design (R-007, R-017) is unchanged.
|
||||
### Per-phase REQ coverage (post-IDEATE)
|
||||
|
||||
- **P01 — mTLS handshake + internal CA with CSR join** (Wave 1)
|
||||
- REQ-011, REQ-023 (carried over from v0.1)
|
||||
- REQ-025 (cert rotation history), REQ-026 (CA fingerprint pinning),
|
||||
REQ-033 (file mode enforcement), REQ-034 (rotation alarm),
|
||||
REQ-035 (cert show redaction), REQ-036 (SAN validation),
|
||||
REQ-038 (mTLS failure log fields)
|
||||
- REQ-032 (orca doctor — initial implementation; checks CA/cert state)
|
||||
|
||||
- **P02 — Multi-node scheduling & job dispatch** (Wave 1)
|
||||
- REQ-028 (NodeCapacity HCL schema — P02 enabler; lands first)
|
||||
- REQ-037 (X-Orca-Idempotency-Key on cross-node POST)
|
||||
|
||||
- **P03 — `gosec` + `govulncheck` + gitleaks in CI** (Wave 2)
|
||||
- REQ-014 (carried over)
|
||||
- REQ-027 (govulncheck offline mode — new in v0.2 IDEATE, per REQ-cand-C;
|
||||
this changes P03's scope: CI must not call `vuln.go.dev` by default;
|
||||
resolve via pre-mirrored DB or `-format json` + `jq` wrapper. PLAN
|
||||
stage decides between the two options.)
|
||||
- REQ-029 (gitleaks baseline for pre-existing `.env` leak in history,
|
||||
per REQ-cand-E)
|
||||
- REQ-039 (`.gitleaks.toml` stopwords), REQ-040 (`.golangci.yml`)
|
||||
|
||||
- **P04 — `iter.Seq` streaming job/node lists** (Wave 2)
|
||||
- REQ-022 (carried over)
|
||||
- REQ-030 (`--watch --json` streaming output mode, per REQ-cand-F)
|
||||
|
||||
- **Cross-cutting (P01–P04)**
|
||||
- REQ-031 (`go test -race` enabled in CI for all v0.2 packages)
|
||||
|
||||
### P03 scope change (vs. pre-IDEATE plan)
|
||||
|
||||
REQ-027 (govulncheck offline mode) adds explicit work to P03: the CI
|
||||
job must be configured to NOT make outbound calls to `vuln.go.dev`
|
||||
(default `govulncheck` behavior). Two implementation paths are viable;
|
||||
PLAN chooses:
|
||||
- Pre-mirror the vulnerability database inside the CoreCI image
|
||||
(`GOVULNCHECK_DB=/path/to/local.db`).
|
||||
- Use `govulncheck -format json` (which always exits 0) and gate
|
||||
merges via a wrapper that parses the JSON and returns non-zero on
|
||||
unsuppressed findings.
|
||||
|
||||
Either path keeps the offline-first invariant (REQ-003) intact.
|
||||
|
||||
+19
-106
@@ -4,19 +4,15 @@
|
||||
{
|
||||
"slug": "orca",
|
||||
"name": "Orca",
|
||||
"description": "Offline/CLI-first orchestration engine (Orca) \u2014 Nomad-inspired, far simpler than Kubernetes",
|
||||
"milestone": "v0.14",
|
||||
"description": "Offline/CLI-first orchestration engine (Orca) — Nomad-inspired, far simpler than Kubernetes",
|
||||
"milestone": "v0.1",
|
||||
"phase": 0,
|
||||
"milestone_type": "feature",
|
||||
"default_branch": "main",
|
||||
"tech_stack": {
|
||||
"language": "go",
|
||||
"version": "1.25+",
|
||||
"frameworks": [
|
||||
"cobra",
|
||||
"connectrpc",
|
||||
"modernc/sqlite"
|
||||
],
|
||||
"frameworks": ["cobra", "connectrpc", "modernc/sqlite"],
|
||||
"build_cmd": "make build",
|
||||
"test_cmd": "make test",
|
||||
"typecheck_cmd": "go vet ./...",
|
||||
@@ -27,38 +23,20 @@
|
||||
}
|
||||
],
|
||||
"active_project": "orca",
|
||||
"active_projects": [
|
||||
"orca"
|
||||
],
|
||||
"ship": {
|
||||
"per_phase": true,
|
||||
"allow_skip": false,
|
||||
"max_release_retries": 3
|
||||
},
|
||||
"active_projects": ["orca"],
|
||||
"autonomy": {
|
||||
"level": "full",
|
||||
"decision_confidence_threshold": 0.6,
|
||||
"decision_confidence_threshold": 0.60,
|
||||
"max_revision_iterations": 3,
|
||||
"max_verification_retries": 2,
|
||||
"clarify_budget": 10,
|
||||
"escalation_hooks": [
|
||||
"deploy",
|
||||
"delete_data",
|
||||
"merge_to_main"
|
||||
]
|
||||
"escalation_hooks": ["delete", "drop", "force", "reset --hard"]
|
||||
},
|
||||
"workflow": {
|
||||
"no_hitl": true,
|
||||
"release_flow_per_phase": true,
|
||||
"merge_strategy": {
|
||||
"allowed": [
|
||||
"fast-forward",
|
||||
"rebase-then-fast-forward"
|
||||
],
|
||||
"forbidden": [
|
||||
"merge-commit-no-ff",
|
||||
"squash"
|
||||
],
|
||||
"allowed": ["fast-forward", "rebase-then-fast-forward"],
|
||||
"forbidden": ["merge-commit-no-ff", "squash"],
|
||||
"phase_to_milestone": "fast-forward",
|
||||
"milestone_to_main": "rebase-then-fast-forward"
|
||||
},
|
||||
@@ -76,59 +54,25 @@
|
||||
{
|
||||
"name": "lead-developer",
|
||||
"domain": "coordination",
|
||||
"frameworks": [
|
||||
"cobra"
|
||||
],
|
||||
"constraints": [
|
||||
"boundary-enforcement",
|
||||
"offline-first",
|
||||
"no-redundant-implementations"
|
||||
],
|
||||
"territory": [
|
||||
"**/*.go",
|
||||
"cmd/**",
|
||||
"internal/**"
|
||||
],
|
||||
"frameworks": ["cobra"],
|
||||
"constraints": ["boundary-enforcement", "offline-first", "no-redundant-implementations"],
|
||||
"territory": ["**/*.go", "cmd/**", "internal/**"],
|
||||
"active": true
|
||||
},
|
||||
{
|
||||
"name": "backend-engineer",
|
||||
"domain": "backend",
|
||||
"frameworks": [
|
||||
"cobra",
|
||||
"connectrpc"
|
||||
],
|
||||
"constraints": [
|
||||
"API-first",
|
||||
"error-handling",
|
||||
"minimal-dependencies",
|
||||
"security-first"
|
||||
],
|
||||
"territory": [
|
||||
"**/api/**",
|
||||
"**/*_handler*",
|
||||
"**/*_handler.go",
|
||||
"internal/cli/**"
|
||||
],
|
||||
"frameworks": ["cobra", "connectrpc"],
|
||||
"constraints": ["API-first", "error-handling", "minimal-dependencies", "security-first"],
|
||||
"territory": ["**/api/**", "**/*_handler*", "**/*_handler.go", "internal/cli/**"],
|
||||
"active": true
|
||||
},
|
||||
{
|
||||
"name": "data-engineer",
|
||||
"domain": "data",
|
||||
"frameworks": [
|
||||
"modernc/sqlite"
|
||||
],
|
||||
"constraints": [
|
||||
"schema-first",
|
||||
"migration-safe",
|
||||
"local-storage-only"
|
||||
],
|
||||
"territory": [
|
||||
"**/database/**",
|
||||
"**/model.go",
|
||||
"**/migration*",
|
||||
"migrations/**"
|
||||
],
|
||||
"frameworks": ["modernc/sqlite"],
|
||||
"constraints": ["schema-first", "migration-safe", "local-storage-only"],
|
||||
"territory": ["**/database/**", "**/model.go", "**/migration*", "migrations/**"],
|
||||
"active": true
|
||||
}
|
||||
]
|
||||
@@ -142,9 +86,7 @@
|
||||
},
|
||||
"ci": {
|
||||
"provider": "coreci",
|
||||
"allowed_providers": [
|
||||
"coreci"
|
||||
],
|
||||
"allowed_providers": ["coreci"],
|
||||
"gitea": {
|
||||
"url": "https://git.cloudinit.dev",
|
||||
"owner": "coreci",
|
||||
@@ -172,35 +114,6 @@
|
||||
"url": "https://git.cloudinit.dev/coreci/orca.git",
|
||||
"main_branch": "main"
|
||||
},
|
||||
"release": {
|
||||
"forge": "gitea",
|
||||
"gitea": {
|
||||
"base_url": "https://git.cloudinit.dev",
|
||||
"owner": "coreci",
|
||||
"repo": "orca",
|
||||
"token_env": "GITEA_TOKEN"
|
||||
},
|
||||
"container_registry": {
|
||||
"forge": "gitea",
|
||||
"registry": "git.cloudinit.dev",
|
||||
"owner": "coreci",
|
||||
"image": "orca",
|
||||
"credential_env": "GITEA_TOKEN",
|
||||
"images": ["orca", "orca-traefik"]
|
||||
}
|
||||
},
|
||||
"secrets": {
|
||||
"scopes": [
|
||||
{
|
||||
"name": "gitea",
|
||||
"vars": [
|
||||
"GITEA_TOKEN",
|
||||
"GITEA_USER"
|
||||
],
|
||||
"env_file": ".env"
|
||||
}
|
||||
]
|
||||
},
|
||||
"commands": {
|
||||
"test": "make test",
|
||||
"build": "make build",
|
||||
@@ -208,4 +121,4 @@
|
||||
"lint": "make lint",
|
||||
"format": "gofmt -w ."
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+10
-81
@@ -13,8 +13,6 @@ description: Orca — offline/CLI-first orchestration engine. Full release flow
|
||||
# - gosec (REQ-014, REQ-040) Static analysis for Go security smells
|
||||
# - govulncheck (REQ-014, REQ-027) Offline vuln scan of dependencies
|
||||
# - gitleaks (REQ-039) Pre-commit-style secret scan
|
||||
# v0.8 P03 added a requirements-hygiene stage:
|
||||
# - verify-reqs (REQ-060) ROADMAP COMPLETE ↔ REQUIREMENTS Complete
|
||||
# The `test` pipeline runs with -race (REQ-031).
|
||||
# See docs/security-scanning.md for operator-facing details.
|
||||
|
||||
@@ -23,25 +21,20 @@ pipelines:
|
||||
description: Validate Go toolchain, formatting, and security scans
|
||||
steps:
|
||||
- name: go-version
|
||||
image: golang:1.25.12
|
||||
image: golang:1.25
|
||||
commands:
|
||||
- go version
|
||||
- gofmt -l .
|
||||
- go vet ./...
|
||||
|
||||
- name: verify-reqs
|
||||
image: golang:1.25.12
|
||||
commands:
|
||||
- make verify-reqs
|
||||
|
||||
- name: gosec
|
||||
image: golang:1.25.12
|
||||
image: golang:1.25
|
||||
commands:
|
||||
- go install github.com/securego/gosec/v2/cmd/gosec@v2.18.2
|
||||
- gosec -fmt text -quiet ./...
|
||||
|
||||
- name: govulncheck
|
||||
image: golang:1.25.12
|
||||
image: golang:1.25
|
||||
env:
|
||||
# REQ-027: offline mode. GOFLAGS=-mod=mod ensures module mode;
|
||||
# GOVULNCHECK_DB (when present) overrides the bundled DB.
|
||||
@@ -51,7 +44,7 @@ pipelines:
|
||||
- govulncheck -mode binary ./...
|
||||
|
||||
- name: gitleaks
|
||||
image: golang:1.25.12
|
||||
image: golang:1.25
|
||||
commands:
|
||||
- apk add --no-cache curl
|
||||
- sh -c "$(curl -fsSL https://github.com/gitleaks/gitleaks/releases/latest/download/install.sh)"
|
||||
@@ -61,7 +54,7 @@ pipelines:
|
||||
description: Build the orca binary with version injection
|
||||
steps:
|
||||
- name: build
|
||||
image: golang:1.25.12
|
||||
image: golang:1.25
|
||||
env:
|
||||
VERSION: ${CI_COMMIT_TAG:-dev}
|
||||
GIT_COMMIT: ${CI_COMMIT_SHA}
|
||||
@@ -80,7 +73,7 @@ pipelines:
|
||||
description: Run all tests with race detection and coverage (REQ-031)
|
||||
steps:
|
||||
- name: test
|
||||
image: golang:1.25.12
|
||||
image: golang:1.25
|
||||
commands:
|
||||
- go test -race -coverprofile=coverage.out ./...
|
||||
- go tool cover -func=coverage.out | tail -1
|
||||
@@ -91,7 +84,7 @@ pipelines:
|
||||
ref: "refs/tags/v*"
|
||||
steps:
|
||||
- name: build-artifact
|
||||
image: golang:1.25.12
|
||||
image: golang:1.25
|
||||
env:
|
||||
VERSION: ${CI_COMMIT_TAG}
|
||||
GIT_COMMIT: ${CI_COMMIT_SHA}
|
||||
@@ -105,81 +98,17 @@ pipelines:
|
||||
go build -trimpath -ldflags="${LDFLAGS}" -o bin/orca ./cmd/orca
|
||||
- make changelog
|
||||
- tar -czf orca-${VERSION}-linux-amd64.tar.gz -C bin orca
|
||||
- sha256sum orca-${VERSION}-linux-amd64.tar.gz > SHA256SUMS
|
||||
- ls -lh orca-${VERSION}-linux-amd64.tar.gz SHA256SUMS
|
||||
- cat SHA256SUMS
|
||||
- ls -lh orca-${VERSION}-linux-amd64.tar.gz
|
||||
- name: gitea-release
|
||||
image: golang:1.25.12
|
||||
image: golang:1.25
|
||||
env:
|
||||
GITEA_TOKEN: ${GITEA_TOKEN}
|
||||
VERSION: ${CI_COMMIT_TAG}
|
||||
commands:
|
||||
- apk add --no-cache curl tar python3
|
||||
- apk add --no-cache curl tar
|
||||
- sh -c "$(curl -fsSL https://gitea.com/gitea/tea/releases/latest/download/install.sh)"
|
||||
- tea releases create ${VERSION}
|
||||
--repo coreci/orca
|
||||
--title "Orca ${VERSION}"
|
||||
--note-file CHANGELOG.md
|
||||
--asset orca-${VERSION}-linux-amd64.tar.gz
|
||||
--asset SHA256SUMS
|
||||
- |
|
||||
# Verify assets are actually attached (REQ-097, gate C-21).
|
||||
# tea releases create has been observed to exit 0 without
|
||||
# attaching the asset in some versions. Verify via the API.
|
||||
ASSET_COUNT=$(curl -fsSL \
|
||||
"https://git.cloudinit.dev/api/v1/repos/coreci/orca/releases/tags/${VERSION}" \
|
||||
| python3 -c "import json,sys; r=json.load(sys.stdin); print(len(r.get('assets',[])))")
|
||||
echo "Release ${VERSION} has ${ASSET_COUNT} assets"
|
||||
if [ "${ASSET_COUNT}" -lt 2 ]; then
|
||||
echo "ERROR: Expected at least 2 assets (tarball + SHA256SUMS), got ${ASSET_COUNT}"
|
||||
echo "Attempting to attach assets manually..."
|
||||
TARBALL_URL=$(curl -fsSL \
|
||||
"https://git.cloudinit.dev/api/v1/repos/coreci/orca/releases/tags/${VERSION}" \
|
||||
| python3 -c "import json,sys; r=json.load(sys.stdin); print(r.get('id',''))")
|
||||
if [ -n "${TARBALL_URL}" ]; then
|
||||
curl -fsSL -X "POST" \
|
||||
"https://git.cloudinit.dev/api/v1/repos/coreci/orca/releases/${TARBALL_URL}/assets?name=orca-${VERSION}-linux-amd64.tar.gz" \
|
||||
-H "Authorization: token ${GITEA_TOKEN}" \
|
||||
-F "attachment=@orca-${VERSION}-linux-amd64.tar.gz"
|
||||
curl -fsSL -X "POST" \
|
||||
"https://git.cloudinit.dev/api/v1/repos/coreci/orca/releases/${TARBALL_URL}/assets?name=SHA256SUMS" \
|
||||
-H "Authorization: token ${GITEA_TOKEN}" \
|
||||
-F "attachment=@SHA256SUMS"
|
||||
fi
|
||||
fi
|
||||
- name: container-publish
|
||||
description: Build and publish OCI image to Gitea container registry (REQ-046)
|
||||
image: docker:24-cli
|
||||
env:
|
||||
GITEA_TOKEN: ${GITEA_TOKEN}
|
||||
VERSION: ${CI_COMMIT_TAG}
|
||||
GIT_COMMIT: ${CI_COMMIT_SHA}
|
||||
BUILD_TIME: ${CI_BUILD_TIME}
|
||||
commands:
|
||||
- docker build
|
||||
--build-arg VERSION=${VERSION}
|
||||
--build-arg GIT_COMMIT=${GIT_COMMIT}
|
||||
--build-arg BUILD_TIME=${BUILD_TIME}
|
||||
-t git.cloudinit.dev/coreci/orca:${VERSION}
|
||||
-t git.cloudinit.dev/coreci/orca:latest
|
||||
.
|
||||
- echo "${GITEA_TOKEN}" | docker login git.cloudinit.dev -u cloudinit-bot --password-stdin
|
||||
- docker push git.cloudinit.dev/coreci/orca:${VERSION}
|
||||
- docker push git.cloudinit.dev/coreci/orca:latest
|
||||
- docker logout git.cloudinit.dev
|
||||
- name: container-publish-traefik
|
||||
description: Build and publish orca-traefik OCI image (REQ-171, R-024)
|
||||
image: docker:24-cli
|
||||
env:
|
||||
GITEA_TOKEN: ${GITEA_TOKEN}
|
||||
VERSION: ${CI_COMMIT_TAG}
|
||||
commands:
|
||||
- docker build
|
||||
-f Dockerfile.traefik
|
||||
-t git.cloudinit.dev/coreci/orca-traefik:${VERSION}
|
||||
-t git.cloudinit.dev/coreci/orca-traefik:latest
|
||||
.
|
||||
- echo "${GITEA_TOKEN}" | docker login git.cloudinit.dev -u cloudinit-bot --password-stdin
|
||||
- docker push git.cloudinit.dev/coreci/orca-traefik:${VERSION}
|
||||
- docker push git.cloudinit.dev/coreci/orca-traefik:latest
|
||||
- docker logout git.cloudinit.dev
|
||||
|
||||
@@ -1,20 +0,0 @@
|
||||
.git
|
||||
.githooks
|
||||
.bin
|
||||
bin/
|
||||
*.tar.gz
|
||||
*.tar.gz.asc
|
||||
.env
|
||||
.env.*
|
||||
.gitleaks-baseline.json
|
||||
.gitleaks.toml
|
||||
.golangci.yml
|
||||
.ciagent/
|
||||
testdata/
|
||||
docs/
|
||||
*.md
|
||||
!README.md
|
||||
LICENSE
|
||||
coverage.out
|
||||
orca
|
||||
orca-v*
|
||||
@@ -10,6 +10,4 @@ orca
|
||||
*.db-shm
|
||||
.env
|
||||
.env.local
|
||||
.env.secrets
|
||||
.env.*
|
||||
*.tar.gz
|
||||
|
||||
@@ -1,2 +0,0 @@
|
||||
disable=SC2086
|
||||
external-sources=true
|
||||
+26
-132
@@ -5,137 +5,31 @@ All notable changes to orca are documented in this file.
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## v0.13 milestone (in progress) — tag line v0.12.x
|
||||
|
||||
The v0.13 milestone is **Production Hardening Round 2 + UAT Plan**.
|
||||
Three deep codebase sweeps (security, reliability, feature/doc claims)
|
||||
surfaced ~60 gaps beyond v0.12. v0.13 closes all critical/high/medium
|
||||
findings and delivers the UAT plan + signoff script that gates the
|
||||
v1.0.0 cut.
|
||||
|
||||
**Load-bearing architectural changes**:
|
||||
- **R-022** — `orca job run` deploys to remote nodes via the scheduler →
|
||||
emitter → SSH-push pipeline. The local `exec.CommandContext` path is
|
||||
removed (P03).
|
||||
- **R-023** — Zero-trust enforcement is operationally wired: `acl.Check`
|
||||
is invoked on every daemon handler + sshpush + txn apply path;
|
||||
`acl.json` is 0600; audit `actor` carries OIDC sub/SVID; WebAuthn
|
||||
registration requires auth; `cluster seal`/`unseal` + `doctor audit`/
|
||||
`doctor modes` CLI commands exist (P04, P05).
|
||||
|
||||
### v0.13 phase commits (v0.11.29..HEAD)
|
||||
|
||||
- `ed91d68` — feat(P10): observability expansion — metrics + security headers (REQ-159)
|
||||
- `531b369` — fix(P09): migration + operational safety — job stop, retention, logs cap (REQ-158)
|
||||
- `3a3ea74` — fix(P08): transport + SSH safety — typed errors, IPv6, timeouts, signal (REQ-157)
|
||||
- `0358efe` — fix(P07): concurrency safety — SQLite, flock, cache, atomic writes (REQ-156)
|
||||
- `978334a` — feat(P06): auth init-idp real + auth register + doctor oidc (REQ-155)
|
||||
- `9e83238` — feat(P05): seal/audit CLI + chain race fix + key zeroing (REQ-154)
|
||||
- `5232fcb` — fix(P04): wire ACL enforcement + WebAuthn reg auth + audit actor (REQ-153)
|
||||
- `cf3d98e` — feat(P03): wire scheduler into job run + fix jobspec parser (REQ-151, REQ-152)
|
||||
- `4b70e31` — fix(P02): input validation + injection hardening — 11 vectors (REQ-150)
|
||||
- `b0158c9` — fix(P01): bump go toolchain to 1.25.12 + fix pre-existing test bugs (REQ-149)
|
||||
- `7479cd1` — docs(checkpoint): P0 shipped — v0.12.0 tagged
|
||||
- `1a2dd1a` — docs(P00): incorporate grill binding conditions C-44..C-49
|
||||
- `437d9b2` — docs(P00): grill v0.13 — CONDITIONAL PROCEED (6 binding conditions C-44..C-49)
|
||||
- `82bfab1` — docs(P00): create phase plans — 14 phases, 15 REQs, vertical slices
|
||||
- `a2a651e` — docs(P00): ideation results — 15 accepted (REQ-149..REQ-163), 0 skipped
|
||||
- `3f5e5de` — docs(P00): research findings — threat model round 3 (~60 gaps, F26-F101)
|
||||
- `7a60b35` — docs(P00): clarify v0.13 — 7 decisions resolved (D-248..D-254)
|
||||
- `8071793` — docs(init): validate specification — v0.13 Production Hardening Round 2 + UAT Plan
|
||||
|
||||
### v0.13 phase summary
|
||||
|
||||
- **P0** — Pre-execution: specify → clarify → research → ideate → plan → grill (tag `v0.12.0`)
|
||||
- **P01** — Toolchain & dependency vulns: Go 1.25.12 bump, 24 stdlib vulns closed, govulncheck triage (REQ-149)
|
||||
- **P02** — Input validation & injection hardening: 11 vectors closed (`orca logs --job` RCE, tar-slip, sudoers injection, pprof loopback, txn/nft ID validation, drain allocID, cluster_compat, podman image, nft TrustedProbes, sudoers user/role) (REQ-150)
|
||||
- **P03** — Scheduler/deployment wiring + jobspec parser: `orca job run` wires scheduler → emitter → SSH-push; `schedule:`/`timeout:` parsed by markdown jobspec (REQ-151, REQ-152)
|
||||
- **P04** — ACL enforcement + WebAuthn registration auth: `acl.Check` wired into daemon + sshpush + txn apply; WebAuthn registration requires auth; audit actor carries OIDC sub/SVID (REQ-153)
|
||||
- **P05** — Seal/audit CLI + chain race fix + key zeroing: `orca cluster seal`/`unseal`, `orca doctor audit`, `orca doctor modes` CLI commands; audit hash-chain race fix; master key zeroed on exit (REQ-154)
|
||||
- **P06** — auth init-idp real + auth register + doctor oidc: real Dex deployment, `orca auth register` browser flow, `orca doctor oidc` health check (REQ-155)
|
||||
- **P07** — Concurrency safety: SQLite WAL, flock on known_hosts, cache thread-safety, atomic writes (REQ-156)
|
||||
- **P08** — Transport & SSH safety: typed dial errors, IPv6 support, connect timeouts, signal handling (REQ-157)
|
||||
- **P09** — Migration & operational safety: `orca job stop` via SSH, DB retention check, logs cap (REQ-158)
|
||||
- **P10** — Observability & metrics: metrics endpoint expansion, security headers (REQ-159)
|
||||
- **P11** — Doc drift round 2 (this phase, REQ-160)
|
||||
|
||||
## v0.12 milestone — COMPLETE (tag line v0.11.x)
|
||||
|
||||
The v0.12 milestone is **Security Hardening (Zero-Trust Identity)**.
|
||||
Comprehensive security hardening across the entire attack surface
|
||||
including the OS, plus adoption of a zero-trust identity model. 25
|
||||
threat-model findings (F1..F25) closed. R-021 adopted: no Orca-issued
|
||||
credentials — human identity is exclusively external (OIDC), machine
|
||||
identity is exclusively mTLS/SPIFFE.
|
||||
|
||||
**Milestone release**: `v0.11.28` (29 phases, tags `v0.11.0`..`v0.11.28`).
|
||||
|
||||
### v0.12 phase highlights
|
||||
|
||||
- Command injection fix (REQ-119, F3)
|
||||
- Namespace path traversal fix (REQ-120, F4)
|
||||
- Txn apply path allowlist (REQ-121, F5)
|
||||
- OIDC client + bundled Dex (REQ-144; BYO-IdP override)
|
||||
- WebAuthn connector for Dex / passkeys (REQ-148)
|
||||
- ACL rewrite to OIDC claims + enforcement (REQ-145, REQ-122, F1)
|
||||
- Remove all password/token paths (REQ-146, R-021, C-34)
|
||||
- Master key seal-to-OIDC + Shamir 3-of-5 recovery (REQ-147, C-35)
|
||||
- Daemon auth hardening (REQ-123, REQ-124, F6, F24)
|
||||
- Audit log tamper-evidence (REQ-125, F2)
|
||||
- SVID chain validation (REQ-126, F9)
|
||||
- Backup symlink validation (REQ-127, F7)
|
||||
- step-ca /tmp hardening (REQ-128, F10)
|
||||
- Master key rotation (REQ-129, F12, C-30)
|
||||
- File-mode audit expansion (REQ-130, F13)
|
||||
- aggregate.sh JSON injection + drift-gate fix (REQ-131, F11, F18)
|
||||
- install.sh checksum+GPG verification (REQ-132, F14)
|
||||
- nftables ruleset hardening (REQ-133, F21)
|
||||
- sudoers hardening (REQ-134, F22)
|
||||
- System user consistency (REQ-135, F23)
|
||||
- SQLite file-mode + at-rest encryption (REQ-136, F8, C-31)
|
||||
- Migration safety + identity migration (REQ-137, F19, C-34)
|
||||
- Legacy CA/mTLS/daemon + step-ca password-provisioner deletion (REQ-138, F16)
|
||||
- known_hosts tightening + transport hardening (REQ-139, F15, F25)
|
||||
- Drift event authentication (REQ-140, F18)
|
||||
- Security integration test suite (REQ-141, C-33)
|
||||
- Zero-trust + OIDC + WebAuthn + threat-model docs (REQ-142)
|
||||
- Final review + ship + audit (REQ-143)
|
||||
|
||||
## v0.11 milestone — COMPLETE (tag line v0.10.x)
|
||||
|
||||
The v0.11 milestone is **Production Hardening**. See the git log and
|
||||
ROADMAP for the full phase list.
|
||||
|
||||
## v0.1 milestone — COMPLETE
|
||||
|
||||
Initial CLI skeleton, node management, task execution, state
|
||||
persistence, audit log, health checks, and CoreCI release flow.
|
||||
|
||||
- `e1b5385` — feat(P06): CoreCI release flow with .coreci.yml and tea integration
|
||||
- `07b8ad2` — ship(P05): health checks merged into milestone
|
||||
- `b06458d` — docs(P05): verification - 4 layers pass
|
||||
- `708d983` — feat(P05): health check daemon with /healthz, /readyz, /v1/* handlers
|
||||
- `30c523c` — ship(P04): state persistence merged into milestone
|
||||
- `759b1b5` — docs(P04): verification - 4 layers pass
|
||||
- `b25e074` — feat(P04): audit log + persistence hardening
|
||||
- `bb6b5b3` — ship(P03): task exec merged into milestone
|
||||
- `857f756` — docs(P03): verification - 4 layers pass
|
||||
- `f9a9873` — feat(P03): task execution engine with HCL specs, jobs, tasks, WaitDelay
|
||||
- `78334f1` — ship(P02): node mgmt merged into milestone
|
||||
- `c7dbcef` — docs(P02): verification - 4 layers pass
|
||||
- `9580f34` — feat(P02): node management with SQLite-backed registry
|
||||
- `46e929e` — chore(P01): source .env in trigger_coreci.sh for GITEA_TOKEN
|
||||
- `503923b` — ship(P01): cli skeleton merged into milestone
|
||||
- `e3f6e1d` — docs(P01): verification - 4 layers pass
|
||||
- `aa3ccce` — feat(P01): CLI skeleton with Cobra, subcommand stubs, pre-push hook
|
||||
- `c203895` — docs(P00): create 6 phase plans with wave ordering
|
||||
- `65eb2e6` — docs(P00): research findings - architecture + personas
|
||||
- `6f34f17` — docs(P00): ideation - 30 ideas accepted (3 tiers)
|
||||
- `bc7ce1c` — docs(P00): clarify ambiguities (full autonomy, 10 decisions)
|
||||
- `55aae53` — chore(P00): rename orch-engine to orca, configure gitea + coreci (v0.1)
|
||||
- `0cba1aa` — chore(P00): set autonomy level to full
|
||||
- `e2e77e7` — chore(P00): quick task - populate config.json with backlog reference
|
||||
- `8c086de` — chore(config): populate ciagent config with standard settings
|
||||
- `8774008` — docs(init): validate specification
|
||||
- `e1b538575c57158c7a6661d5919b103f6c7932fc` — feat(P06): CoreCI release flow with .coreci.yml and tea integration
|
||||
- `07b8ad2ceaba7ca303dfe91876930d33b76c633e` — ship(P05): health checks merged into milestone
|
||||
- `b06458d31370750417a3b239dac61c6e2fdf5329` — docs(P05): verification - 4 layers pass
|
||||
- `708d9834296271094667700e88e80bfa27db7bdd` — feat(P05): health check daemon with /healthz, /readyz, /v1/* handlers
|
||||
- `30c523c0c7a8e75a2e97b42f1c8a39802febcbdc` — ship(P04): state persistence merged into milestone
|
||||
- `759b1b519d7fadf3d91d3070952d9ad2051a0eba` — docs(P04): verification - 4 layers pass
|
||||
- `b25e074e1d3518f175478184ff8d002ec0d8412c` — feat(P04): audit log + persistence hardening
|
||||
- `bb6b5b3e8342c16601a8503223c7186ecdbb00df` — ship(P03): task exec merged into milestone
|
||||
- `857f7563190e7703f97c607a50d6b0a897d250e9` — docs(P03): verification - 4 layers pass
|
||||
- `f9a98733411cfa8657e82636e0c55671086ebe46` — feat(P03): task execution engine with HCL specs, jobs, tasks, WaitDelay
|
||||
- `78334f1f74f0c185c6d38014c796aac4903b8141` — ship(P02): node mgmt merged into milestone
|
||||
- `c7dbcef9587596786a541a7566479d9fb93fcf0a` — docs(P02): verification - 4 layers pass
|
||||
- `9580f347c68e395dccfbe83b27a857d52bf21075` — feat(P02): node management with SQLite-backed registry
|
||||
- `46e929e4c6539bd604539ba27d5ed0c606e87bb9` — chore(P01): source .env in trigger_coreci.sh for GITEA_TOKEN
|
||||
- `503923bf1ee2c60f8375acc7eb9608d346368e1c` — ship(P01): cli skeleton merged into milestone
|
||||
- `e3f6e1df825d39f73933c9996bd2cc4717ff1061` — docs(P01): verification - 4 layers pass
|
||||
- `aa3cccead503a37dfec75873d06d2d396a2876f2` — feat(P01): CLI skeleton with Cobra, subcommand stubs, pre-push hook
|
||||
- `c2038952c74f7c242ba3be65d2f4269b23685f5a` — docs(P00): create 6 phase plans with wave ordering
|
||||
- `65eb2e601b741b36388598b9f8adddd7bd8dd3a8` — docs(P00): research findings - architecture + personas
|
||||
- `6f34f1794b9f526c06a1dc139d4a74371599502e` — docs(P00): ideation - 30 ideas accepted (3 tiers)
|
||||
- `bc7ce1caf672e87774455a6cd6cc0db986cd09b3` — docs(P00): clarify ambiguities (full autonomy, 10 decisions)
|
||||
- `55aae5347ec09bce9ef7697ea0c9c9ee158bc040` — chore(P00): rename orch-engine to orca, configure gitea + coreci (v0.1)
|
||||
- `0cba1aa5feef9564f8b9a2a97ae735dc859a8a84` — chore(P00): set autonomy level to full
|
||||
- `e2e77e79b9cbfb462044662543845476f843161b` — chore(P00): quick task - populate config.json with backlog reference
|
||||
- `8c086def698bf0af31e8e820b6b7a2783af06f43` — chore(config): populate ciagent config with standard settings
|
||||
- `8774008c3e47e4ca4711f4fef164531006d16216` — docs(init): validate specification
|
||||
|
||||
Generated by make changelog. Do not edit by hand.
|
||||
|
||||
-56
@@ -1,56 +0,0 @@
|
||||
# Dockerfile — multi-stage build for orca
|
||||
#
|
||||
# Stage 1: build the static binary with golang:1.25
|
||||
# Stage 2: distroless static runtime (CGO-free, ~2MB image)
|
||||
#
|
||||
# Build args:
|
||||
# VERSION — semver tag injected via -ldflags (e.g. v0.4.4)
|
||||
# GIT_COMMIT — short commit hash
|
||||
# BUILD_TIME — ISO 8601 build timestamp
|
||||
#
|
||||
# Build:
|
||||
# docker build --build-arg VERSION=v0.4.4 -t git.cloudinit.dev/coreci/orca:v0.4.4 .
|
||||
#
|
||||
# Run:
|
||||
# docker run --rm git.cloudinit.dev/coreci/orca:v0.4.4 version
|
||||
# docker run --rm -v orca-data:/var/lib/orca git.cloudinit.dev/coreci/orca:v0.4.4 init
|
||||
|
||||
ARG VERSION=dev
|
||||
ARG GIT_COMMIT=unknown
|
||||
ARG BUILD_TIME=unknown
|
||||
|
||||
# --- Stage 1: build -------------------------------------------------------
|
||||
|
||||
FROM golang:1.25.12 AS builder
|
||||
|
||||
ARG VERSION
|
||||
ARG GIT_COMMIT
|
||||
ARG BUILD_TIME
|
||||
|
||||
WORKDIR /src
|
||||
|
||||
# Cache module downloads — copy go.mod/go.sum first, download, then copy source.
|
||||
COPY go.mod go.sum ./
|
||||
RUN go mod download
|
||||
|
||||
COPY . .
|
||||
|
||||
# CGO_ENABLED=0 guarantees a static binary (modernc/sqlite is pure Go).
|
||||
RUN CGO_ENABLED=0 go build -trimpath \
|
||||
-ldflags="-s -w \
|
||||
-X git.cloudinit.dev/coreci/orca/internal/cli.version=${VERSION} \
|
||||
-X git.cloudinit.dev/coreci/orca/internal/cli.gitCommit=${GIT_COMMIT} \
|
||||
-X git.cloudinit.dev/coreci/orca/internal/cli.buildTime=${BUILD_TIME}" \
|
||||
-o /orca ./cmd/orca
|
||||
|
||||
# --- Stage 2: runtime -----------------------------------------------------
|
||||
|
||||
FROM gcr.io/distroless/static-debian12:nonroot
|
||||
|
||||
# ORCA_HOME points to a volume-mountable path inside the container.
|
||||
# Mount a volume at /var/lib/orca to persist state across container restarts.
|
||||
ENV ORCA_HOME=/var/lib/orca
|
||||
|
||||
COPY --from=builder /orca /orca
|
||||
|
||||
ENTRYPOINT ["/orca"]
|
||||
@@ -1,33 +0,0 @@
|
||||
# Dockerfile.traefik — custom orca-traefik image (R-024)
|
||||
#
|
||||
# Extends the official traefik:v3.3.0 image with a baked default static
|
||||
# config. The host-side /etc/traefik/traefik.yml (rendered by
|
||||
# emitter.RenderTraefikStaticConfig) is mounted :ro at runtime to
|
||||
# override this default — preserving the traefik-on-public-ip opt-out
|
||||
# (REQ-100) and any site-local customisation.
|
||||
#
|
||||
# Dynamic config (routers, services, certs) is mounted from
|
||||
# /etc/traefik/dynamic on the host — orca writes to it atomically via
|
||||
# the SSH-push transport (C-10 protocol).
|
||||
#
|
||||
# Build:
|
||||
# docker build -f Dockerfile.traefik -t git.cloudinit.dev/coreci/orca-traefik:v0.13.1 .
|
||||
#
|
||||
# Run (hybrid R-017 mode — nft DNATs :443/:80 to loopback):
|
||||
# podman run -d --name orca-traefik --restart=unless-stopped \
|
||||
# --network host \
|
||||
# -v /etc/traefik/traefik.yml:/etc/traefik/traefik.yml:ro \
|
||||
# -v /etc/traefik/dynamic:/etc/traefik/dynamic:ro \
|
||||
# -v /etc/orca/step-ca-root.crt:/etc/orca/step-ca-root.crt:ro \
|
||||
# git.cloudinit.dev/coreci/orca-traefik:v0.13.1
|
||||
|
||||
FROM traefik:v3.3.0
|
||||
|
||||
LABEL org.opencontainers.image.title="orca-traefik"
|
||||
LABEL org.opencontainers.image.description="Custom Traefik image for Orca ingress (R-024)"
|
||||
LABEL org.opencontainers.image.source="https://git.cloudinit.dev/coreci/orca"
|
||||
|
||||
COPY docker/orca-traefik/traefik.yml /etc/traefik/traefik.yml
|
||||
COPY docker/orca-traefik/step-ca-root.crt /etc/orca/step-ca-root.crt
|
||||
|
||||
CMD ["--configFile=/etc/traefik/traefik.yml"]
|
||||
@@ -1,4 +1,4 @@
|
||||
.PHONY: build test test-race lint fmt clean run release version changelog help security-scan verify-reqs verify-docs
|
||||
.PHONY: build test test-race lint fmt clean run release version changelog help security-scan
|
||||
|
||||
BINARY := bin/orca
|
||||
GOFLAGS := -trimpath
|
||||
@@ -30,8 +30,6 @@ help:
|
||||
@echo " changelog Generate CHANGELOG.md from ---ci--- commit blocks"
|
||||
@echo " release Run scripts/release.sh [VERSION] — build, tar, publish"
|
||||
@echo " security-scan Run gosec+govulncheck+gitleaks (P03, REQ-014/027/039)"
|
||||
@echo " verify-reqs Assert ROADMAP COMPLETE ↔ REQUIREMENTS Complete (REQ-060)"
|
||||
@echo " verify-docs Assert docs/cli.md ↔ orca --help consistency (REQ-160)"
|
||||
|
||||
build:
|
||||
@mkdir -p bin
|
||||
@@ -40,42 +38,19 @@ build:
|
||||
|
||||
test:
|
||||
go test -coverprofile=coverage.out ./...
|
||||
$(MAKE) test-bash
|
||||
|
||||
# test-race runs the full test suite under the race detector (REQ-031).
|
||||
# Wired into the .coreci.yml `test` pipeline as well.
|
||||
test-race:
|
||||
go test -race -coverprofile=coverage.out ./...
|
||||
$(MAKE) test-bash
|
||||
|
||||
lint:
|
||||
gofmt -l .
|
||||
go vet ./...
|
||||
$(MAKE) lint-bash
|
||||
|
||||
fmt:
|
||||
gofmt -w .
|
||||
|
||||
# test-bash runs bats tests for shell scripts (grill C-15). Skips gracefully
|
||||
# if bats is not installed.
|
||||
test-bash:
|
||||
@command -v bats >/dev/null 2>&1 && { \
|
||||
echo "→ bats scripts/tests/*.bash"; \
|
||||
bats scripts/tests/*.bash; \
|
||||
} || echo "bats not installed; skipping bash tests (see scripts/tests/README.md)"
|
||||
|
||||
# lint-bash runs shellcheck + shfmt on shell scripts (grill C-15). Skips
|
||||
# gracefully if the tools are not installed.
|
||||
lint-bash:
|
||||
@command -v shellcheck >/dev/null 2>&1 && { \
|
||||
echo "→ shellcheck scripts/"; \
|
||||
shellcheck scripts/*.sh scripts/lib/*.sh scripts/tests/*.bash || true; \
|
||||
} || echo "shellcheck not installed; skipping (see scripts/tests/README.md)"
|
||||
@command -v shfmt >/dev/null 2>&1 && { \
|
||||
echo "→ shfmt -d scripts/"; \
|
||||
shfmt -d scripts/; \
|
||||
} || echo "shfmt not installed; skipping (see scripts/tests/README.md)"
|
||||
|
||||
clean:
|
||||
rm -rf bin coverage.out *.tar.gz
|
||||
|
||||
@@ -123,15 +98,3 @@ release:
|
||||
# in a developer's local environment; CI requires all three).
|
||||
security-scan:
|
||||
./scripts/security_scan.sh
|
||||
|
||||
# verify-reqs asserts ROADMAP milestone COMPLETE ↔ REQUIREMENTS row Complete
|
||||
# consistency (REQ-060). Catches doc-vs-doc drift; code-vs-doc drift is out
|
||||
# of scope (P04 audit). Exits 0 on consistency, 1 with a diff on drift.
|
||||
verify-reqs:
|
||||
go run ./cmd/verify-reqs .ciagent/ROADMAP.md .ciagent/REQUIREMENTS.md
|
||||
|
||||
# verify-docs asserts that every top-level subcommand in docs/cli.md
|
||||
# exists in `orca --help` output (and vice versa). Catches doc drift
|
||||
# (REQ-160). Requires the binary to be built first (`make build`).
|
||||
verify-docs: build
|
||||
./scripts/verify-docs.sh ./bin/orca docs/cli.md
|
||||
|
||||
@@ -1,163 +1,58 @@
|
||||
# Orca
|
||||
|
||||
A minimalist, offline-first, CLI-first orchestration engine inspired by
|
||||
HashiCorp Nomad. Proxmox is one supported node type — not the project's
|
||||
identity.
|
||||
Offline/CLI-first orchestration engine inspired by HashiCorp Nomad, far simpler than Kubernetes.
|
||||
|
||||
## Status
|
||||
|
||||
**v0.12: Security Hardening (Zero-Trust Identity) — COMPLETE** | **v0.13: Production Hardening Round 2 + UAT Plan — IN PROGRESS** | **v1.0: UAT-gated** (cut
|
||||
separately after v0.13 completion per operator decision)
|
||||
|
||||
See [.ciagent/ROADMAP.md](.ciagent/ROADMAP.md) for the full roadmap.
|
||||
**v0.1: Foundation** — see [.ciagent/ROADMAP.md](.ciagent/ROADMAP.md) for the 6-phase plan.
|
||||
|
||||
## Pillars
|
||||
|
||||
- **Simplicity** — single binary, minimal dependencies, no daemon on the
|
||||
critical path
|
||||
- **Offline-first** — no cloud dependencies; the cluster is the OS
|
||||
- **CLI-first** — the command line is the primary interface (humans and
|
||||
AI agents)
|
||||
- **Security before features** — SSH-push is the canonical transport
|
||||
(mTLS available for daemon mode); NFRs ship before new functionality
|
||||
- **WASM-first** — workloads target OS primitives (systemd units,
|
||||
journald), not a container runtime shim
|
||||
- **Simplicity** — single binary, minimal dependencies
|
||||
- **AI-first** — CLI designed for both humans and AI agents
|
||||
- **Offline-first** — no cloud dependencies
|
||||
- **CLI-first** — primary interface is the command line
|
||||
- **Security before features** — NFRs ship before new functionality
|
||||
- **Bug fixes before features** — stability is paramount
|
||||
- **NFRs before features** — observability and auditability first
|
||||
|
||||
## Quickstart
|
||||
|
||||
### Install (1-liner)
|
||||
|
||||
```bash
|
||||
# User-level install (binary at ~/.local/bin/orca, state at ~/.orca)
|
||||
curl -fsSL https://git.cloudinit.dev/coreci/orca/raw/main/scripts/install.sh | bash
|
||||
# Build
|
||||
make build
|
||||
|
||||
# System-level install (binary at /usr/local/bin/orca, state at /root/.orca)
|
||||
curl -fsSL https://git.cloudinit.dev/coreci/orca/raw/main/scripts/install.sh | sudo bash -s -- --system
|
||||
# Run
|
||||
./bin/orca version
|
||||
./bin/orca --help
|
||||
|
||||
# Pin a specific version (latest tag: v0.12.10)
|
||||
curl -fsSL https://git.cloudinit.dev/coreci/orca/raw/main/scripts/install.sh | bash -s -- --version v0.12.10
|
||||
|
||||
# Dry-run: check what would be installed without writing
|
||||
curl -fsSL https://git.cloudinit.dev/coreci/orca/raw/main/scripts/install.sh | bash -s -- --check
|
||||
```
|
||||
|
||||
Then initialize local state and verify:
|
||||
|
||||
```bash
|
||||
orca init # creates ~/.orca/ (or /root/.orca with --system)
|
||||
orca version # prints version info
|
||||
orca --help # show all subcommands
|
||||
```
|
||||
|
||||
### Build from source
|
||||
|
||||
```bash
|
||||
make build # Build binary to ./bin/orca
|
||||
./bin/orca init # Initialize local state
|
||||
./bin/orca version # Verify
|
||||
```
|
||||
|
||||
### Update in place
|
||||
|
||||
Re-running the installer updates the binary while preserving your
|
||||
config, database, and certificates in the namespace dir:
|
||||
|
||||
```bash
|
||||
curl -fsSL https://git.cloudinit.dev/coreci/orca/raw/main/scripts/install.sh | bash
|
||||
# → "updated orca from v0.11.28 to v0.12.10"
|
||||
# Initialize local state
|
||||
./bin/orca init
|
||||
```
|
||||
|
||||
## Subcommands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `orca init` | Initialize local orca state with full bootstrap |
|
||||
| `orca status` | **(deprecated v0.1 stub)** Show orca daemon status — use `orca node list` + `orca metrics /healthz` |
|
||||
| `orca version` | Print version information |
|
||||
| `orca daemon` | **(deprecated)** Run the orca daemon (HTTP API + health checks) |
|
||||
| `orca metrics` | Start metrics endpoint (Prometheus text exposition) |
|
||||
| `orca logs` | Aggregate journald logs across nodes (`--all-nodes --since`) |
|
||||
| `orca backup` | Create a signed tar.gz backup of ORCA_HOME |
|
||||
| `orca restore` | Restore ORCA_HOME from a verified signed backup |
|
||||
| `orca upgrade` | Upgrade orca to a new version (thin wrapper; R-017 cutover) |
|
||||
| `orca node` | Manage orca nodes: `join`, `leave`, `list`, `key-reset`, `drain`, `capacity` |
|
||||
| `orca job` | Manage orca jobs: `run`, `list`, `stop`, `logs`, `lint`, `verify`, `migrate`, `restart` |
|
||||
| `orca ns` | Manage orca namespaces: `list`, `create`, `delete`, `inspect`, `validate`, `inherit`, `set-constraint` |
|
||||
| `orca cert` | **(deprecated)** Manage orca certificates: `ca-init`, `gen`, `show`, `renew`, `fingerprint` |
|
||||
| `orca doctor` | Run self-checks: `cert`, `network`, `db`, `os`, `proxmox`, `no-orca-on-server`, `nft`, `audit`, `modes`, `oidc`, `db-retention` |
|
||||
| `orca audit` | View orca audit log (`list`) |
|
||||
| `orca cache` | CLI cache management: `show`, `invalidate`, `invalidate-all` |
|
||||
| `orca acl` | ACL management: `grant`, `revoke`, `list`, `check` |
|
||||
| `orca secrets` | Secrets management: `set`, `get`, `list`, `rotate`, `delete`, `rotate-master` |
|
||||
| `orca drift` | Drift detection: `show`, `watch`, `acknowledge`, `remediate`, `config` |
|
||||
| `orca txn` | Transaction management: `apply`, `list`, `show`, `rollback` |
|
||||
| `orca nft` | nftables ingress management: `show`, `diff`, `doctor`, `country block`, `rate limit` |
|
||||
| `orca collector` | Collector/aggregator management: `start`, `stop`, `status` |
|
||||
| `orca cluster` | Cluster management: `cutover`, `rotate-lead`, `compat-check`, `seal`, `unseal` |
|
||||
| `orca auth` | OIDC authentication: `login`, `logout`, `status`, `init-idp`, `register` |
|
||||
| `orca peer-setup` | Create the orca system user + drift-events dir on a peer (REQ-111) |
|
||||
|
||||
See [docs/cli.md](docs/cli.md) for the full CLI reference with all flags
|
||||
and examples.
|
||||
|
||||
## Honest trade-offs
|
||||
|
||||
Orca is not a Kubernetes replacement for every workload. This table is
|
||||
the honest comparison — K8s wins in several dimensions, and that is
|
||||
acknowledged rather than papered over.
|
||||
|
||||
| Dimension | Kubernetes wins | Orca wins |
|
||||
|-----------|-----------------|-----------|
|
||||
| Ecosystem | Mature CNCF ecosystem; vast operator, controller, plugin surface | — |
|
||||
| Talent pool | Large pool of K8s-experienced engineers | — |
|
||||
| Multi-cloud | Portable across all major clouds; control plane is cloud-agnostic | — |
|
||||
| Stateful operators | Rich operator pattern (CRD + controller) for stateful workloads | — |
|
||||
| Service mesh | First-class service mesh (Istio, Linkerd) | — |
|
||||
| Auto-scaling | Cluster autoscaler, HPA/VPA, deep integrations | — |
|
||||
| Daemon footprint | — | No daemon on the critical path; the cluster is the OS |
|
||||
| OS-native | — | Workloads are systemd units + journald; no container runtime shim |
|
||||
| Transport | — | SSH-push is canonical (no daemon needed); mTLS available for daemon mode |
|
||||
| Offline-first | — | No cloud dependencies; fully air-gapped operation |
|
||||
| WASM-first | — | Workloads target OS primitives, not a container runtime |
|
||||
| Proxmox | — | First-class Proxmox node type (`--type proxmox`) via SSH-push |
|
||||
|
||||
## Documentation
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [docs/cli.md](docs/cli.md) | CLI reference — every command, flag, and example |
|
||||
| [docs/jobspec.md](docs/jobspec.md) | Jobspec reference — markdown frontmatter schema |
|
||||
| [docs/ingress.md](docs/ingress.md) | Ingress guide — Traefik configuration |
|
||||
| [docs/namespace.md](docs/namespace.md) | Namespace and path layout |
|
||||
| [docs/install.md](docs/install.md) | Installation guide |
|
||||
| [docs/security-scanning.md](docs/security-scanning.md) | Security scanning tools |
|
||||
| [docs/security-runbook.md](docs/security-runbook.md) | Security runbook — seal/unseal, rotation, incident response |
|
||||
| [docs/webauthn.md](docs/webauthn.md) | WebAuthn / passkeys registration and login |
|
||||
| [docs/threat-model.md](docs/threat-model.md) | STRIDE threat model + zero-trust architecture |
|
||||
| [docs/oidc.md](docs/oidc.md) | OIDC configuration — Dex quickstart, BYO IdP |
|
||||
|
||||
## Examples
|
||||
|
||||
| Example | Description |
|
||||
|---------|-------------|
|
||||
| [examples/full-stack/](examples/full-stack/) | Full-stack deployment with ingress (5 services + rendered artifacts) |
|
||||
| Command | Description | Status |
|
||||
|---------|-------------|--------|
|
||||
| `orca version` | Print version info | ✅ Phase 1 |
|
||||
| `orca init` | Initialize local orca state | ✅ Phase 1 (stub) |
|
||||
| `orca status` | Show orca daemon status | ✅ Phase 1 (stub) |
|
||||
| `orca node` | Node management (`join`, `leave`, `list`) | Phase 2 |
|
||||
| `orca job` | Job management (`run`, `list`, `stop`, `logs`) | Phase 3 |
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
make build # Build binary to ./bin/orca
|
||||
make test # Run tests
|
||||
go vet ./... # Vet all packages
|
||||
make lint # Run gofmt + go vet + shellcheck
|
||||
make verify-reqs # Assert ROADMAP ↔ REQUIREMENTS consistency
|
||||
make verify-docs # Assert docs/cli.md ↔ `orca --help` consistency
|
||||
make build # Build binary to ./bin/orca
|
||||
make test # Run tests with race detection
|
||||
make lint # Run golangci-lint
|
||||
make fmt # Format code
|
||||
make release # Build + create Gitea release (Phase 6)
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
See [.ciagent/ARCHITECTURE.md](.ciagent/ARCHITECTURE.md) for full
|
||||
architecture details.
|
||||
See [.ciagent/ARCHITECTURE.md](.ciagent/ARCHITECTURE.md) for full architecture details.
|
||||
|
||||
## License
|
||||
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
15ee8f02ef938496ce9baae35e2971a5fcfb2d55b2c6e35e49c11285f67aeb53 orca-v0.12.18-linux-amd64.tar.gz
|
||||
+1
-9
@@ -8,16 +8,8 @@ import (
|
||||
)
|
||||
|
||||
func main() {
|
||||
os.Exit(run())
|
||||
}
|
||||
|
||||
// run executes the orca CLI and returns the process exit code. It is
|
||||
// extracted from main so tests can exercise the error path without
|
||||
// os.Exit terminating the test process.
|
||||
func run() int {
|
||||
if err := cli.Execute(); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "error: %v\n", err)
|
||||
return 1
|
||||
os.Exit(1)
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
@@ -1,41 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"io"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestRunSuccess(t *testing.T) {
|
||||
orig := os.Args
|
||||
t.Cleanup(func() { os.Args = orig })
|
||||
os.Args = []string{"orca", "version"}
|
||||
if code := run(); code != 0 {
|
||||
t.Errorf("run() = %d, want 0", code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunError(t *testing.T) {
|
||||
origArgs := os.Args
|
||||
t.Cleanup(func() { os.Args = origArgs })
|
||||
os.Args = []string{"orca", "job", "run", "/nonexistent/spec.hcl"}
|
||||
|
||||
r, w, err := os.Pipe()
|
||||
if err != nil {
|
||||
t.Fatalf("pipe: %v", err)
|
||||
}
|
||||
origStderr := os.Stderr
|
||||
os.Stderr = w
|
||||
t.Cleanup(func() { os.Stderr = origStderr })
|
||||
|
||||
code := run()
|
||||
w.Close()
|
||||
out, _ := io.ReadAll(r)
|
||||
if code != 1 {
|
||||
t.Errorf("run() = %d, want 1", code)
|
||||
}
|
||||
if !strings.Contains(string(out), "error:") {
|
||||
t.Errorf("stderr missing 'error:' prefix: %s", out)
|
||||
}
|
||||
}
|
||||
@@ -1,197 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"fmt"
|
||||
"os"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// reqRowRe captures a REQUIREMENTS.md table row's REQ-ID, Phase cell, and
|
||||
// status in one pass. The leading .* is greedy so it consumes the
|
||||
// Requirement and Priority cells (which may contain markdown-escaped pipes
|
||||
// like `localhost\|linux\|proxmox` — see REQ-049) and backtracks to anchor
|
||||
// the Phase + Status match at the END of the line, where those two columns
|
||||
// always live. The status token is optionally wrapped in markdown bold
|
||||
// (real rows use `**Complete**`; synthetic/future rows may use bare
|
||||
// `pending` or `complete` in any case), and may carry trailing notes
|
||||
// (e.g. "**Complete** (P01 shipped v0.2.1)") matched by [^|]* before
|
||||
// the closing pipe. The (?i) flag makes the match case-insensitive so
|
||||
// lowercase `pending` (used by v0.12/v0.13 REQ rows) is captured;
|
||||
// normalizeStatus canonicalizes the captured value to title case.
|
||||
var reqRowRe = regexp.MustCompile(`(?i)^\|\s*(REQ-\d+)\s*\|.*\|\s*([^|]*?)\s*\|\s*\*{0,2}(Complete|Pending)\*{0,2}[^|]*\|\s*$`)
|
||||
|
||||
// milestoneCompleteRe matches a ROADMAP.md milestone header that is marked
|
||||
// COMPLETE. The bold markers are optional (GRILL #4 + REQ-160 T11): it
|
||||
// matches `**COMPLETE**`, `**COMPLETE (merged to main via v0.3)**`, and
|
||||
// bare `COMPLETE` (as used by the v0.12 milestone header). The word
|
||||
// COMPLETE may be preceded or followed by non-asterisk text. The milestone
|
||||
// version (v0.X) is captured.
|
||||
var milestoneCompleteRe = regexp.MustCompile(`^##\s*Milestone\s+(v0\.\d+):.*—\s*\*{0,2}[^*]*\bCOMPLETE\b[^*]*\*{0,2}`)
|
||||
|
||||
// phaseRe extracts the milestone version from a REQUIREMENTS Phase cell such
|
||||
// as `v0.7 P1`, `**v0.2 P01**`, `v0.2 P01–P04`, or bare `v0.7`. The cell may
|
||||
// contain multiple milestone refs separated by `/` or `–`; each is extracted.
|
||||
var phaseTokenRe = regexp.MustCompile(`v0\.\d+`)
|
||||
|
||||
// reqRow holds a parsed REQUIREMENTS.md row.
|
||||
type reqRow struct {
|
||||
id string
|
||||
phase string // raw Phase cell (e.g. "v0.7 P1", "**v0.2 P01 / v0.3 P02**")
|
||||
status string // "Complete" or "Pending"
|
||||
}
|
||||
|
||||
// normalizeStatus canonicalizes a captured status token to the title-case
|
||||
// form ("Complete" or "Pending") so that case-insensitive matches like
|
||||
// "pending" or "complete" compare correctly against the drift assertions.
|
||||
func normalizeStatus(s string) string {
|
||||
s = strings.TrimSpace(s)
|
||||
if s == "" {
|
||||
return s
|
||||
}
|
||||
return strings.ToUpper(s[:1]) + strings.ToLower(s[1:])
|
||||
}
|
||||
|
||||
// milestoneVersions returns the distinct v0.X milestones referenced in the
|
||||
// phase cell (e.g. "v0.7 P1" → ["v0.7"]; "v0.2 P01 / v0.3 P02" →
|
||||
// ["v0.2","v0.3"]).
|
||||
func (r reqRow) milestoneVersions() []string {
|
||||
matches := phaseTokenRe.FindAllString(r.phase, -1)
|
||||
seen := map[string]bool{}
|
||||
var out []string
|
||||
for _, m := range matches {
|
||||
if !seen[m] {
|
||||
seen[m] = true
|
||||
out = append(out, m)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func main() {
|
||||
roadmapPath := ".ciagent/ROADMAP.md"
|
||||
reqsPath := ".ciagent/REQUIREMENTS.md"
|
||||
if len(os.Args) > 1 {
|
||||
roadmapPath = os.Args[1]
|
||||
}
|
||||
if len(os.Args) > 2 {
|
||||
reqsPath = os.Args[2]
|
||||
}
|
||||
diff, count, err := verify(roadmapPath, reqsPath)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "verify-reqs: %v\n", err)
|
||||
os.Exit(2)
|
||||
}
|
||||
if len(diff) > 0 {
|
||||
fmt.Fprintf(os.Stderr, "requirements drift detected (%d):\n", len(diff))
|
||||
for _, line := range diff {
|
||||
fmt.Fprintln(os.Stderr, line)
|
||||
}
|
||||
os.Exit(1)
|
||||
}
|
||||
fmt.Printf("✓ %d requirements consistent with roadmap\n", count)
|
||||
}
|
||||
|
||||
// verify parses the ROADMAP and REQUIREMENTS markdown and returns a diff
|
||||
// listing of any drift. On success diff is nil and count is the number of
|
||||
// consistent REQ rows. A non-nil error signals a parse/read failure (not
|
||||
// drift); drift is reported via the diff slice.
|
||||
func verify(roadmapPath, reqsPath string) (diff []string, count int, err error) {
|
||||
completeMilestones, err := parseRoadmap(roadmapPath)
|
||||
if err != nil {
|
||||
return nil, 0, fmt.Errorf("parse roadmap %q: %w", roadmapPath, err)
|
||||
}
|
||||
rows, err := parseRequirements(reqsPath)
|
||||
if err != nil {
|
||||
return nil, 0, fmt.Errorf("parse requirements %q: %w", reqsPath, err)
|
||||
}
|
||||
if len(rows) == 0 {
|
||||
return nil, 0, fmt.Errorf("no REQ rows found in %s", reqsPath)
|
||||
}
|
||||
|
||||
type driftEntry struct {
|
||||
id string
|
||||
current string
|
||||
want string
|
||||
dir string // "forward" or "reverse"
|
||||
}
|
||||
var drifts []driftEntry
|
||||
|
||||
for _, r := range rows {
|
||||
milestones := r.milestoneVersions()
|
||||
anyComplete := false
|
||||
for _, m := range milestones {
|
||||
if completeMilestones[m] {
|
||||
anyComplete = true
|
||||
break
|
||||
}
|
||||
}
|
||||
// Forward assertion: a REQ whose milestone is COMPLETE in ROADMAP
|
||||
// must be marked Complete in REQUIREMENTS.
|
||||
if anyComplete && r.status != "Complete" {
|
||||
drifts = append(drifts, driftEntry{r.id, r.status, "Complete", "forward"})
|
||||
}
|
||||
// Reverse assertion (GRILL #4): a REQ marked Complete in
|
||||
// REQUIREMENTS must reference at least one milestone ROADMAP marks
|
||||
// COMPLETE. If all referenced milestones are NOT complete (or no
|
||||
// milestone is referenced), that is premature-Complete drift.
|
||||
if r.status == "Complete" && !anyComplete {
|
||||
drifts = append(drifts, driftEntry{r.id, r.status, "Pending (milestone not COMPLETE in ROADMAP)", "reverse"})
|
||||
}
|
||||
}
|
||||
|
||||
sort.Slice(drifts, func(i, j int) bool { return drifts[i].id < drifts[j].id })
|
||||
for _, d := range drifts {
|
||||
diff = append(diff, fmt.Sprintf(" %s: status=%s, expected=%s (direction=%s)", d.id, d.current, d.want, d.dir))
|
||||
}
|
||||
|
||||
consistent := len(rows) - len(drifts)
|
||||
return diff, consistent, nil
|
||||
}
|
||||
|
||||
func parseRoadmap(path string) (map[string]bool, error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer f.Close()
|
||||
complete := map[string]bool{}
|
||||
sc := bufio.NewScanner(f)
|
||||
sc.Buffer(make([]byte, 1024*1024), 1024*1024)
|
||||
for sc.Scan() {
|
||||
line := sc.Text()
|
||||
m := milestoneCompleteRe.FindStringSubmatch(line)
|
||||
if m != nil {
|
||||
complete[m[1]] = true
|
||||
}
|
||||
}
|
||||
if err := sc.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return complete, nil
|
||||
}
|
||||
|
||||
func parseRequirements(path string) ([]reqRow, error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer f.Close()
|
||||
var rows []reqRow
|
||||
sc := bufio.NewScanner(f)
|
||||
sc.Buffer(make([]byte, 1024*1024), 1024*1024)
|
||||
for sc.Scan() {
|
||||
line := sc.Text()
|
||||
m := reqRowRe.FindStringSubmatch(line)
|
||||
if m == nil {
|
||||
continue
|
||||
}
|
||||
rows = append(rows, reqRow{id: m[1], phase: strings.TrimSpace(m[2]), status: normalizeStatus(m[3])})
|
||||
}
|
||||
if err := sc.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return rows, nil
|
||||
}
|
||||
@@ -1,170 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Tests run with cwd = package dir (cmd/verify-reqs), so `testdata/...`
|
||||
// paths resolve relative to the package. The real-repo tests use a
|
||||
// repoRoot helper to locate `.ciagent/...`.
|
||||
|
||||
func repoRoot(t *testing.T) string {
|
||||
t.Helper()
|
||||
wd, err := os.Getwd()
|
||||
if err != nil {
|
||||
t.Fatalf("getwd: %v", err)
|
||||
}
|
||||
return filepath.Join(wd, "..", "..")
|
||||
}
|
||||
|
||||
// case (1): clean pair → no drift.
|
||||
func TestVerify_clean(t *testing.T) {
|
||||
diff, count, err := verify(
|
||||
filepath.Join("testdata", "roadmap_clean.md"),
|
||||
filepath.Join("testdata", "requirements_clean.md"),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if len(diff) != 0 {
|
||||
t.Fatalf("expected no drift, got:\n%s", strings.Join(diff, "\n"))
|
||||
}
|
||||
if count != 5 {
|
||||
t.Fatalf("expected 5 consistent rows, got %d", count)
|
||||
}
|
||||
}
|
||||
|
||||
// case (2)+(3): drift pair → all drifts reported, both directions.
|
||||
func TestVerify_drift(t *testing.T) {
|
||||
diff, _, err := verify(
|
||||
filepath.Join("testdata", "roadmap_drift.md"),
|
||||
filepath.Join("testdata", "requirements_drift.md"),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if len(diff) != 4 {
|
||||
t.Fatalf("expected 4 drifts (REQ-002 forward, REQ-003 reverse, REQ-004 forward, REQ-005 forward), got %d:\n%s",
|
||||
len(diff), strings.Join(diff, "\n"))
|
||||
}
|
||||
joined := strings.Join(diff, "\n")
|
||||
for _, want := range []string{"REQ-002", "REQ-003", "REQ-004", "REQ-005"} {
|
||||
if !strings.Contains(joined, want) {
|
||||
t.Errorf("diff missing %s:\n%s", want, joined)
|
||||
}
|
||||
}
|
||||
if !strings.Contains(joined, "direction=reverse") {
|
||||
t.Errorf("expected a reverse-direction drift, got:\n%s", joined)
|
||||
}
|
||||
if !strings.Contains(joined, "direction=forward") {
|
||||
t.Errorf("expected a forward-direction drift, got:\n%s", joined)
|
||||
}
|
||||
}
|
||||
|
||||
// case (4): missing args → defaults resolve to the repo's .ciagent/ files.
|
||||
// Runs the program as a subprocess from the repo root.
|
||||
func TestVerify_defaultArgs(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("subprocess test skipped in -short mode")
|
||||
}
|
||||
root := repoRoot(t)
|
||||
cmd := exec.Command("go", "run", "./cmd/verify-reqs")
|
||||
cmd.Dir = root
|
||||
out := &strings.Builder{}
|
||||
cmd.Stdout = out
|
||||
cmd.Stderr = out
|
||||
if err := cmd.Run(); err != nil {
|
||||
if exitErr, ok := err.(*exec.ExitError); ok {
|
||||
t.Fatalf("verify-reqs on real repo exited %d (should be 0): %s",
|
||||
exitErr.ExitCode(), out.String())
|
||||
}
|
||||
t.Fatalf("go run failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// case (5): malformed markdown (no REQ rows) → clear error, not silent pass.
|
||||
func TestVerify_malformed(t *testing.T) {
|
||||
_, _, err := verify(
|
||||
filepath.Join("testdata", "roadmap_clean.md"),
|
||||
filepath.Join("testdata", "requirements_malformed.md"),
|
||||
)
|
||||
if err == nil {
|
||||
t.Fatal("expected error on malformed (no REQ rows) requirements, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "no REQ rows") {
|
||||
t.Errorf("expected 'no REQ rows' error, got: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// case (6): missing file → clear error, not silent pass.
|
||||
func TestVerify_missingFile(t *testing.T) {
|
||||
_, _, err := verify(
|
||||
filepath.Join("testdata", "roadmap_clean.md"),
|
||||
filepath.Join("testdata", "does_not_exist.md"),
|
||||
)
|
||||
if err == nil {
|
||||
t.Fatal("expected error on missing file, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "does_not_exist.md") {
|
||||
t.Errorf("expected error to mention the missing file, got: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// GRILL #4 golden test: the v0.2-style
|
||||
// `**COMPLETE (merged to main via v0.3)**` header MUST be recognized as a
|
||||
// complete milestone by the substring-tolerant regex. The drift fixture's
|
||||
// roadmap carries exactly this header on its v0.2 line, and the drift
|
||||
// fixture's REQ-002 (v0.2 P1, Pending) would be silently skipped if the
|
||||
// regex regressed to the exact `\*\*COMPLETE\*\*` form. TestVerify_drift
|
||||
// already asserts REQ-002 is flagged forward-drift; this test makes the
|
||||
// intent explicit and guards against a regex regression.
|
||||
func TestVerify_v02SubstringTolerantHeader(t *testing.T) {
|
||||
diff, _, err := verify(
|
||||
filepath.Join("testdata", "roadmap_drift.md"),
|
||||
filepath.Join("testdata", "requirements_drift.md"),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
// REQ-002 references milestone v0.2, whose header uses the
|
||||
// `**COMPLETE (merged to main via v0.3)**` variant. If the regex
|
||||
// regressed, v0.2 would not be marked complete and REQ-002 (Pending)
|
||||
// would NOT be flagged as forward drift.
|
||||
found := false
|
||||
for _, d := range diff {
|
||||
if strings.Contains(d, "REQ-002") && strings.Contains(d, "direction=forward") {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Fatalf("REQ-002 forward drift not reported — substring-tolerant regex may have regressed; diff:\n%s",
|
||||
strings.Join(diff, "\n"))
|
||||
}
|
||||
}
|
||||
|
||||
// Verify the actual repo passes (regression guard for the real docs).
|
||||
func TestVerify_realRepo(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("real-repo test skipped in -short mode")
|
||||
}
|
||||
root := repoRoot(t)
|
||||
diff, count, err := verify(
|
||||
filepath.Join(root, ".ciagent", "ROADMAP.md"),
|
||||
filepath.Join(root, ".ciagent", "REQUIREMENTS.md"),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("verify on real repo errored: %v", err)
|
||||
}
|
||||
if len(diff) != 0 {
|
||||
t.Fatalf("real repo has requirements drift (should be clean after SPECIFY):\n%s",
|
||||
strings.Join(diff, "\n"))
|
||||
}
|
||||
if count <= 0 {
|
||||
t.Fatalf("real repo reported %d consistent rows (expected > 0)", count)
|
||||
}
|
||||
}
|
||||
@@ -1,9 +0,0 @@
|
||||
# Requirements: Clean Fixture
|
||||
|
||||
| ID | Requirement | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-001 | Foundation req | High | v0.1 P1 | **Complete** |
|
||||
| REQ-002 | Multi-node with escaped pipes (kind\|os) | Medium | **v0.2 P1** | **Complete** (shipped v0.2.1) |
|
||||
| REQ-003 | Scheduling req | Low | v0.3 P1 | **Complete** |
|
||||
| REQ-004 | Future req spanning phases | Low | v0.2 P1 / v0.3 P2 | **Complete** (multi-phase) |
|
||||
| REQ-005 | Pending future req | Low | v0.9 P1 | Pending |
|
||||
@@ -1,9 +0,0 @@
|
||||
# Requirements: Drift Fixture
|
||||
|
||||
| ID | Requirement | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-001 | Foundation req (clean) | High | v0.1 P1 | **Complete** |
|
||||
| REQ-002 | Multi-node req (forward drift: milestone COMPLETE but row Pending) | Medium | **v0.2 P1** | Pending |
|
||||
| REQ-003 | Scheduling req (reverse drift: row Complete but milestone NOT complete) | Low | v0.9 P1 | **Complete** |
|
||||
| REQ-004 | Multi-phase with range (forward drift: spans COMPLETE v0.2 + non-COMPLETE v0.9) | Low | v0.2 P1–P2 | Pending |
|
||||
| REQ-005 | Second forward drift (v0.3 COMPLETE, row Pending) | Low | v0.3 P1 | Pending |
|
||||
@@ -1,8 +0,0 @@
|
||||
# Requirements: Malformed Fixture
|
||||
|
||||
This file has no valid REQ rows, just prose and a broken table.
|
||||
|
||||
| ID | Requirement | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| NOT-A-REQ | broken row | High | v0.1 | **Complete** |
|
||||
| REQ-999 | missing status cell | High | v0.1 |
|
||||
-20
@@ -1,20 +0,0 @@
|
||||
# Roadmap: Clean Fixture
|
||||
|
||||
## Milestone v0.1: Foundation — **COMPLETE**
|
||||
|
||||
- [x] Phase 1
|
||||
|
||||
## Milestone v0.2: Networking — **COMPLETE (merged to main via v0.3)**
|
||||
|
||||
- [x] Phase 8
|
||||
- [x] Phase 9
|
||||
|
||||
## Milestone v0.3: Scheduling — **COMPLETE**
|
||||
|
||||
- [x] Phase 1
|
||||
|
||||
## Milestone v0.9: Future Work
|
||||
|
||||
Not yet shipped.
|
||||
|
||||
- [ ] Phase 1
|
||||
-20
@@ -1,20 +0,0 @@
|
||||
# Roadmap: Drift Fixture
|
||||
|
||||
## Milestone v0.1: Foundation — **COMPLETE**
|
||||
|
||||
- [x] Phase 1
|
||||
|
||||
## Milestone v0.2: Networking — **COMPLETE (merged to main via v0.3)**
|
||||
|
||||
- [x] Phase 8
|
||||
- [x] Phase 9
|
||||
|
||||
## Milestone v0.3: Scheduling — **COMPLETE**
|
||||
|
||||
- [x] Phase 1
|
||||
|
||||
## Milestone v0.9: Future Work
|
||||
|
||||
Not yet shipped.
|
||||
|
||||
- [ ] Phase 1
|
||||
@@ -1,19 +0,0 @@
|
||||
entryPoints:
|
||||
websecure:
|
||||
address: "127.0.0.1:8443"
|
||||
web:
|
||||
address: "127.0.0.1:8080"
|
||||
traefik:
|
||||
address: "127.0.0.1:8081"
|
||||
|
||||
providers:
|
||||
file:
|
||||
directory: "/etc/traefik/dynamic"
|
||||
watch: true
|
||||
|
||||
log:
|
||||
level: INFO
|
||||
format: json
|
||||
|
||||
accessLog:
|
||||
format: json
|
||||
-1465
File diff suppressed because it is too large
Load Diff
@@ -1,96 +0,0 @@
|
||||
# Docker Guide
|
||||
|
||||
Orca is available as a container image on the Gitea container registry.
|
||||
The image is a minimal distroless static build (~2MB runtime layer)
|
||||
that runs the orca binary directly.
|
||||
|
||||
## Image
|
||||
|
||||
```
|
||||
git.cloudinit.dev/coreci/orca:<version>
|
||||
git.cloudinit.dev/coreci/orca:latest
|
||||
```
|
||||
|
||||
The image is built from the `Dockerfile` in the repo root:
|
||||
- **Build stage**: `golang:1.25` — compiles a static binary with
|
||||
`CGO_ENABLED=0` (modernc/sqlite is pure Go, no CGO).
|
||||
- **Runtime stage**: `gcr.io/distroless/static-debian12:nonroot` —
|
||||
~2MB, no shell, runs as `nonroot` user.
|
||||
|
||||
## Pull
|
||||
|
||||
```bash
|
||||
docker pull git.cloudinit.dev/coreci/orca:latest
|
||||
# or pin a version
|
||||
docker pull git.cloudinit.dev/coreci/orca:v0.4.4
|
||||
```
|
||||
|
||||
The repo is public (REQ-045), so anonymous pull works without login.
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
# Print version
|
||||
docker run --rm git.cloudinit.dev/coreci/orca:v0.4.4 version
|
||||
|
||||
# Initialize state (creates /var/lib/orca/ inside the container)
|
||||
docker run --rm -v orca-data:/var/lib/orca git.cloudinit.dev/coreci/orca:v0.4.4 init
|
||||
|
||||
# Run the daemon (persist state via volume)
|
||||
docker run -d --name orca \
|
||||
-p 8080:8080 \
|
||||
-v orca-data:/var/lib/orca \
|
||||
git.cloudinit.dev/coreci/orca:v0.4.4 daemon --addr=:8080
|
||||
```
|
||||
|
||||
## State Persistence
|
||||
|
||||
The image sets `ENV ORCA_HOME=/var/lib/orca`. All orca state (SQLite
|
||||
database, CA certs, server certs) is written under this path. To
|
||||
persist state across container restarts, mount a volume:
|
||||
|
||||
```bash
|
||||
docker volume create orca-data
|
||||
docker run --rm -v orca-data:/var/lib/orca git.cloudinit.dev/coreci/orca:v0.4.4 init
|
||||
docker run -d --name orca -p 8080:8080 -v orca-data:/var/lib/orca git.cloudinit.dev/coreci/orca:v0.4.4 daemon
|
||||
```
|
||||
|
||||
Without a volume, state is lost when the container exits.
|
||||
|
||||
## System-Level Namespace Inside Containers
|
||||
|
||||
The `--system` flag is not needed inside containers — the image already
|
||||
sets `ORCA_HOME=/var/lib/orca`. Use `--system` only if you want a
|
||||
different namespace root (e.g., `/root/.orca`), which requires running
|
||||
as root (the distroless image runs as `nonroot` by default).
|
||||
|
||||
## Build Locally
|
||||
|
||||
```bash
|
||||
docker build --build-arg VERSION=v0.4.4 -t orca-local:v0.4.4 .
|
||||
docker run --rm orca-local:v0.4.4 version
|
||||
```
|
||||
|
||||
Build args:
|
||||
- `VERSION` — semver tag (injected via `-ldflags`)
|
||||
- `GIT_COMMIT` — short commit hash
|
||||
- `BUILD_TIME` — ISO 8601 build timestamp
|
||||
|
||||
## Publish (for maintainers)
|
||||
|
||||
The `.coreci.yml` release pipeline includes a `container-publish` step
|
||||
that builds and pushes the image on every tag release. To publish
|
||||
manually:
|
||||
|
||||
```bash
|
||||
export GITEA_TOKEN=<token>
|
||||
docker build --build-arg VERSION=v0.4.4 -t git.cloudinit.dev/coreci/orca:v0.4.4 -t git.cloudinit.dev/coreci/orca:latest .
|
||||
echo "$GITEA_TOKEN" | docker login git.cloudinit.dev -u cloudinit-bot --password-stdin
|
||||
docker push git.cloudinit.dev/coreci/orca:v0.4.4
|
||||
docker push git.cloudinit.dev/coreci/orca:latest
|
||||
```
|
||||
|
||||
## See Also
|
||||
|
||||
- [Install Guide](install.md) — binary install (alternative to Docker).
|
||||
- [Namespace and Paths](namespace.md) — `ORCA_HOME` and `--system` flag.
|
||||
-273
@@ -1,273 +0,0 @@
|
||||
# Orca Ingress & Traefik Guide
|
||||
|
||||
This document explains how Orca configures ingress via Traefik dynamic
|
||||
configuration. It covers the service→Traefik mapping, the R-007
|
||||
socket-vs-TCP-bind model, atomic reload, drain, TLS, and a worked
|
||||
example.
|
||||
|
||||
> **Canonical path (v0.9)**: Orca generates Traefik dynamic
|
||||
> configuration files via the `TraefikEmitter`. The `kind: Service`
|
||||
> workload implies a Traefik route. The emitter renders one YAML file
|
||||
> per Service; Traefik watches the dynamic config directory and reloads
|
||||
> atomically on change.
|
||||
|
||||
## The model
|
||||
|
||||
A `kind: Service` jobspec **implies** a Traefik route (D-175). `Job`
|
||||
and `DaemonSet` do **not** carry a Traefik route by default — a
|
||||
`service:` block on a `Job` is rejected by the validator.
|
||||
|
||||
When `orca job run` submits a `kind: Service` workload, the
|
||||
`TraefikEmitter` renders a Traefik dynamic config file at:
|
||||
|
||||
```
|
||||
/etc/traefik/dynamic/orca-<service-name>.yaml
|
||||
```
|
||||
|
||||
This file contains:
|
||||
- One **router** (`orca-<name>`) with a `PathPrefix` rule and TLS config.
|
||||
- One **service** (`orca-<name>`) as a `loadBalancer` with one **server**
|
||||
per port, pointing at the workload's Unix socket (or TCP port).
|
||||
- A **healthCheck** stanza when the `health:` block is present.
|
||||
|
||||
Traefik watches `/etc/traefik/dynamic/` via `fsnotify` and reloads
|
||||
whenever a file changes. Orca writes config atomically (write-tmp +
|
||||
rename) so Traefik sees a single `IN_MOVED_TO` event and never observes
|
||||
a half-written file.
|
||||
|
||||
## R-007: socket vs TCP bind
|
||||
|
||||
Orca workloads bind to a **Unix socket** by default, not a TCP port.
|
||||
This is the R-007 security model: loopback-only by default, no network
|
||||
exposure.
|
||||
|
||||
### Default: Unix socket
|
||||
|
||||
When `service.bind` is empty (default), the workload binds a Unix
|
||||
socket at:
|
||||
|
||||
```
|
||||
/run/orca/alloc-<alloc-id>/port-<port-name>.sock
|
||||
```
|
||||
|
||||
systemd creates `/run/orca/alloc-<alloc-id>/` via
|
||||
`RuntimeDirectory=orca/alloc-<alloc-id>` (mode 0750, owned by
|
||||
`orca:orca`). The Traefik backend server URL is:
|
||||
|
||||
```yaml
|
||||
servers:
|
||||
- url: "unix:///run/orca/alloc-<alloc-id>/port-<port-name>.sock"
|
||||
```
|
||||
|
||||
### TCP opt-in: `service.bind: 127.0.0.1`
|
||||
|
||||
When `service.bind: 127.0.0.1` is set, the workload binds a TCP port
|
||||
directly (loopback only). The emitter adds an `ExecStartPre` marker to
|
||||
the systemd unit so the bind mode is visible:
|
||||
|
||||
```ini
|
||||
ExecStartPre=/bin/echo orca: bind 127.0.0.1 port <name> (tcp, R-007 opt-in)
|
||||
```
|
||||
|
||||
`service.bind` must be a valid IP address. Empty (socket default) or
|
||||
`127.0.0.1` (TCP opt-in) are the documented values; any other valid IP
|
||||
is accepted but the bind happens in the process, not the emitter.
|
||||
|
||||
## Generated Traefik YAML
|
||||
|
||||
For a Service named `web` with port `http`:
|
||||
|
||||
```yaml
|
||||
http:
|
||||
routers:
|
||||
orca-web:
|
||||
rule: PathPrefix("/web")
|
||||
service: orca-web
|
||||
tls:
|
||||
certResolver: orca
|
||||
domains:
|
||||
- main: "cluster.orca.local"
|
||||
services:
|
||||
orca-web:
|
||||
loadBalancer:
|
||||
servers:
|
||||
- url: "unix:///run/orca/alloc-<alloc-id>/port-http.sock"
|
||||
healthCheck:
|
||||
path: /healthz
|
||||
interval: 5s
|
||||
timeout: 1s
|
||||
```
|
||||
|
||||
- One router per Service, named `orca-<service-name>`.
|
||||
- Router rule: `PathPrefix("/<service-name>")`.
|
||||
- TLS: `certResolver: orca`, trust domain `cluster.orca.local`
|
||||
(placeholder; step-ca provisioner overrides in v0.11).
|
||||
- One service per Service, named `orca-<service-name>`.
|
||||
- One server per port, URL is `unix://<socket-path>`.
|
||||
- `healthCheck` stanza present when `health:` block is set (required
|
||||
for Service). Path is `/healthz`; interval and timeout come from the
|
||||
`health:` block.
|
||||
|
||||
## Atomic reload (gate C-10)
|
||||
|
||||
Orca writes Traefik config atomically to avoid Traefik observing a
|
||||
half-written file:
|
||||
|
||||
1. Write to `<path>.tmp` via `WriteFileIdempotent` (write + fsync).
|
||||
2. `mv -f <path>.tmp <path>` (atomic POSIX rename).
|
||||
|
||||
Traefik's `fsnotify` watcher sees a single `IN_MOVED_TO` event and
|
||||
reloads. If the new config is malformed, Traefik logs an error and
|
||||
**holds last-good config** — the cluster keeps serving traffic on the
|
||||
previous config.
|
||||
|
||||
## Drain
|
||||
|
||||
`RenderDrain` produces the same Traefik YAML with `weight: 0` on every
|
||||
server in the load balancer:
|
||||
|
||||
```yaml
|
||||
servers:
|
||||
- url: "unix:///run/orca/alloc-<alloc-id>/port-http.sock"
|
||||
weight: 0
|
||||
```
|
||||
|
||||
Traefik stops sending traffic to the drained backend. The workload
|
||||
keeps running; drain is reversible (re-submit the normal config to
|
||||
restore traffic).
|
||||
|
||||
## TLS
|
||||
|
||||
- **v0.14 model**: `tls: {}` in dynamic config (no certResolver).
|
||||
Traefik v3.3 `certificatesResolvers` only supports `acme` and
|
||||
`tailscale` — not CA-file-based. The `certResolver: orca` reference
|
||||
from v0.11 was broken (research finding). v0.14 emits `tls: {}`
|
||||
(traefik uses its default self-signed cert). Real mTLS via dynamic
|
||||
`tls.certificates` + `tls.options.default.clientAuth.caFiles` is
|
||||
deferred to v0.15.
|
||||
- **Step-ca root CA**: mounted at `/etc/orca/step-ca-root.crt` in the
|
||||
traefik container. v0.14 does not use it for TLS termination (it's
|
||||
a placeholder for v0.15 mTLS).
|
||||
|
||||
## R-024: Podman Traefik Container (v0.14)
|
||||
|
||||
As of v0.14, Traefik runs as a **podman container** from the custom
|
||||
`orca-traefik` image (published per release). The v0.13 binary+systemd
|
||||
install is replaced.
|
||||
|
||||
### Three topologies
|
||||
|
||||
1. **Linux**: host → nft DNAT → `podman run orca-traefik` (`--network host`)
|
||||
2. **Proxmox Native** (`--ingress-mode native`, default): PVE host →
|
||||
nft DNAT → LXC (nesting=1,keyctl=1,fuse=1) → `podman run orca-traefik`
|
||||
3. **Proxmox Floating-IP** (`--ingress-mode floating-ip`): LXC owns
|
||||
the floating IP → nft inside LXC → `podman run orca-traefik`
|
||||
|
||||
### Container configuration
|
||||
|
||||
```bash
|
||||
podman run -d --name orca-traefik --restart=unless-stopped \
|
||||
--network host \
|
||||
-v /etc/traefik/traefik.yml:/etc/traefik/traefik.yml:ro \
|
||||
-v /etc/traefik/dynamic:/etc/traefik/dynamic:ro \
|
||||
-v /etc/orca/step-ca-root.crt:/etc/orca/step-ca-root.crt:ro \
|
||||
git.cloudinit.dev/coreci/orca-traefik:<version>
|
||||
```
|
||||
|
||||
- `--network host`: traefik binds 127.0.0.1:8080/8443 on host/LXC loopback
|
||||
- `--restart=unless-stopped`: survives reboot via `podman-restart.service`
|
||||
- No `:Z` SELinux flag (research Topic 7)
|
||||
- Static config mounted `:ro` (overrides baked image default, preserves
|
||||
`traefik-on-public-ip` opt-out, REQ-100)
|
||||
|
||||
### nft ruleset
|
||||
|
||||
The nft emitter (`internal/emitter/nft.go`) renders `/etc/nftables.d/orca.nft`:
|
||||
|
||||
- DNAT `:443` → `<DNATTarget>:8443` (default 127.0.0.1; LXC IP for native)
|
||||
- DNAT `:80` → `<DNATTarget>:8080`
|
||||
- SNAT/MASQUERADE: `ip saddr 127.0.0.0/8 oifname != "lo" masquerade`
|
||||
- Input/forward chains at priority -10 (pve-firewall coexistence)
|
||||
|
||||
### `orca doctor ingress`
|
||||
|
||||
```bash
|
||||
orca doctor ingress # check localhost
|
||||
orca doctor ingress --peer <name> # check remote peer
|
||||
```
|
||||
|
||||
Verifies: podman container running, nft DNAT+SNAT, dynamic dir exists,
|
||||
step-ca root CA present.
|
||||
|
||||
### Dockerfile.traefik
|
||||
|
||||
```dockerfile
|
||||
FROM traefik:v3.3.0
|
||||
COPY docker/orca-traefik/traefik.yml /etc/traefik/traefik.yml
|
||||
CMD ["--configFile=/etc/traefik/traefik.yml"]
|
||||
```
|
||||
|
||||
Built + published per release alongside the orca image
|
||||
(`scripts/release.sh` + `.coreci.yml container-publish-traefik`).
|
||||
|
||||
## Health checks
|
||||
|
||||
The `health:` block (required for `Service`) maps to the Traefik
|
||||
`healthCheck` stanza:
|
||||
|
||||
```yaml
|
||||
health:
|
||||
check_type: http
|
||||
interval: 5s
|
||||
timeout: 1s
|
||||
unhealthy_threshold: 2
|
||||
```
|
||||
|
||||
→
|
||||
|
||||
```yaml
|
||||
healthCheck:
|
||||
path: /healthz
|
||||
interval: 5s
|
||||
timeout: 1s
|
||||
```
|
||||
|
||||
Traefik polls each backend's `/healthz` at the configured interval. An
|
||||
unhealthy backend is removed from the load balancer pool until it
|
||||
passes the health check again.
|
||||
|
||||
## Worked example
|
||||
|
||||
See [examples/full-stack/](../examples/full-stack/) for a complete
|
||||
multi-service stack with ingress configured:
|
||||
- `web-app.md` — frontend Service (socket bind, PathPrefix route)
|
||||
- `api.md` — backend API Service (TCP opt-in, `127.0.0.1` bind)
|
||||
- `examples/full-stack/rendered/traefik-dynamic-web-app.yaml` — the
|
||||
Traefik config Orca generates
|
||||
|
||||
## v0.11 forward (limitations)
|
||||
|
||||
The following are not yet implemented in v0.9 and will land in v0.11:
|
||||
|
||||
- **`service.host` / `service.route_id`**: stored on the `ServiceBlock`
|
||||
but not yet consumed by the `TraefikEmitter`. The router rule is
|
||||
hardcoded `PathPrefix("/<name>")`. Custom host-based routing lands in
|
||||
v0.11.
|
||||
- **Socket activation**: real socket-activation (socket unit files, fd
|
||||
passing) lands in v0.11-P08. The current emitter renders the
|
||||
`RuntimeDirectory` + socket path comments but does not create socket
|
||||
units.
|
||||
- **Transactional update execution**: the `update:` block's rolling/
|
||||
canary/blue-green plan is computed by the emitter but not yet
|
||||
executed transactionally. Transactional execution lands in
|
||||
v0.11-P10.
|
||||
- **SPIFFE SVID minting**: workload identity via step-ca SVIDs lands in
|
||||
v0.11-P01.5 (gate C-08).
|
||||
- **Secrets in env**: `env: { KEY: { from: "secret:..." } }` resolution
|
||||
to `EnvironmentFile=`/`LoadCredential=` lands in v0.11-P03.
|
||||
|
||||
## See also
|
||||
|
||||
- [docs/cli.md](cli.md) — CLI reference
|
||||
- [docs/jobspec.md](jobspec.md) — Jobspec reference (`service:`, `health:`, `ports:` blocks)
|
||||
- [examples/full-stack/](../examples/full-stack/) — Full-stack example with ingress
|
||||
-139
@@ -1,139 +0,0 @@
|
||||
# Install Guide
|
||||
|
||||
Orca is distributed as a single binary via a 1-liner installer that
|
||||
pulls from the public Gitea release artifacts. This guide covers
|
||||
user-level install, system-level install, in-place updates, version
|
||||
pinning, and troubleshooting.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- A Linux system with `curl` and `tar` installed.
|
||||
- For user-level install: write access to `~/.local/bin/`.
|
||||
- For system-level install: root (`sudo`) access.
|
||||
|
||||
## User-Level Install (Default)
|
||||
|
||||
```bash
|
||||
curl -fsSL https://git.cloudinit.dev/coreci/orca/raw/branch/main/scripts/install.sh | bash
|
||||
```
|
||||
|
||||
This installs:
|
||||
- Binary: `~/.local/bin/orca`
|
||||
- Namespace root: `~/.orca/` (created by `orca init`)
|
||||
|
||||
If `~/.local/bin` is not on your `PATH`, add it:
|
||||
```bash
|
||||
echo 'export PATH="$PATH:$HOME/.local/bin"' >> ~/.bashrc
|
||||
source ~/.bashrc
|
||||
```
|
||||
|
||||
## System-Level Install
|
||||
|
||||
```bash
|
||||
curl -fsSL https://git.cloudinit.dev/coreci/orca/raw/branch/main/scripts/install.sh | sudo bash -s -- --system
|
||||
```
|
||||
|
||||
This installs:
|
||||
- Binary: `/usr/local/bin/orca`
|
||||
- Namespace root: `/root/.orca/` (created by `orca --system init`)
|
||||
|
||||
The `--system` flag requires root (uid 0). It errors if `ORCA_HOME` is
|
||||
already set to a conflicting value.
|
||||
|
||||
## Initialize State
|
||||
|
||||
After installing, initialize the local state directory:
|
||||
|
||||
```bash
|
||||
# User-level
|
||||
orca init
|
||||
|
||||
# System-level
|
||||
orca --system init
|
||||
```
|
||||
|
||||
This creates the namespace root directory (`~/.orca` or `/root/.orca`).
|
||||
|
||||
## Version Pinning
|
||||
|
||||
By default, the installer fetches the **latest** release. To pin a
|
||||
specific version:
|
||||
|
||||
```bash
|
||||
curl -fsSL https://git.cloudinit.dev/coreci/orca/raw/branch/main/scripts/install.sh | bash -s -- --version v0.4.2
|
||||
```
|
||||
|
||||
## In-Place Update
|
||||
|
||||
Re-running the installer updates the binary in place while **preserving**
|
||||
your config, database, and certificates in the namespace dir:
|
||||
|
||||
```bash
|
||||
curl -fsSL https://git.cloudinit.dev/coreci/orca/raw/branch/main/scripts/install.sh | bash
|
||||
```
|
||||
|
||||
Output:
|
||||
```
|
||||
install: ✓ updated orca from v0.4.1 to v0.4.2 at /home/user/.local/bin/orca
|
||||
```
|
||||
|
||||
The installer:
|
||||
1. Detects the existing binary at the install path.
|
||||
2. Reads its version via `orca version --json`.
|
||||
3. Downloads the new release.
|
||||
4. Overwrites the binary.
|
||||
5. **Never touches** the namespace dir (`~/.orca` or `/root/.orca`).
|
||||
|
||||
## Uninstall
|
||||
|
||||
```bash
|
||||
# Remove the binary
|
||||
rm ~/.local/bin/orca # user-level
|
||||
sudo rm /usr/local/bin/orca # system-level
|
||||
|
||||
# Optionally remove state (THIS DELETES YOUR DATABASE + CERTS)
|
||||
rm -rf ~/.orca # user-level
|
||||
sudo rm -rf /root/.orca # system-level
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### `install: error: --system requires root`
|
||||
|
||||
The `--system` flag requires root. Re-run with `sudo`:
|
||||
```bash
|
||||
curl -fsSL ... | sudo bash -s -- --system
|
||||
```
|
||||
|
||||
### `install: error: --system conflicts with ORCA_HOME=...`
|
||||
|
||||
`ORCA_HOME` is set to a non-system path. Either unset it or drop `--system`:
|
||||
```bash
|
||||
unset ORCA_HOME
|
||||
curl -fsSL ... | sudo bash -s -- --system
|
||||
```
|
||||
|
||||
### `install: error: could not find asset orca-vX.Y.Z-linux-amd64.tar.gz`
|
||||
|
||||
The requested version does not have a Linux release asset. Check
|
||||
available releases at
|
||||
`https://git.cloudinit.dev/coreci/orca/releases`.
|
||||
|
||||
### `install: error: unsupported architecture: ...`
|
||||
|
||||
The installer supports `amd64` (x86_64), `arm64` (aarch64), and `armv7`.
|
||||
Contact the maintainers if you need another architecture.
|
||||
|
||||
### `~/.local/bin is not on your PATH`
|
||||
|
||||
Add it to your shell profile:
|
||||
```bash
|
||||
echo 'export PATH="$PATH:$HOME/.local/bin"' >> ~/.bashrc
|
||||
source ~/.bashrc
|
||||
```
|
||||
|
||||
## See Also
|
||||
|
||||
- [Namespace and Paths](namespace.md) — `ORCA_HOME`, `--system`, path layout.
|
||||
- [Docker Guide](docker.md) — running orca in a container.
|
||||
- [Development](../README.md#development) — building from source.
|
||||
-442
@@ -1,442 +0,0 @@
|
||||
# Orca Jobspec Reference
|
||||
|
||||
This document is the complete reference for the Orca jobspec format —
|
||||
the Markdown-with-frontmatter specification that describes workloads.
|
||||
|
||||
> **Canonical format (v0.9)**: Orca uses Markdown with YAML frontmatter
|
||||
> as the canonical jobspec format (R-013/R-014). The legacy HCL format
|
||||
> is supported via an adapter during the migration window but is
|
||||
> deprecated (see [HCL jobspec](#deprecated-hcl-jobspec) below).
|
||||
|
||||
## File formats
|
||||
|
||||
The `orca job run` command dispatches by file extension:
|
||||
|
||||
| Extension | Parser | Body |
|
||||
|-----------|--------|------|
|
||||
| `.md` | `ParseMarkdown` (canonical) | Verbatim after closing `---` (R-015 byte-exact) |
|
||||
| `.yaml` / `.yml` | `parseYAMLFile` | Whole file as frontmatter; body empty |
|
||||
| `.hcl` | `ParseHCL` (legacy adapter) | Empty (deprecated) |
|
||||
|
||||
## Minimal example
|
||||
|
||||
```yaml
|
||||
---
|
||||
kind: Job
|
||||
name: my-job
|
||||
runtime:
|
||||
one_of: process
|
||||
command: /bin/echo hello
|
||||
---
|
||||
# My Job
|
||||
|
||||
This body is preserved byte-exact and carried to the target node.
|
||||
```
|
||||
|
||||
## Top-level keys
|
||||
|
||||
| Key | Type | Default | Required | Notes |
|
||||
|-----|------|---------|----------|-------|
|
||||
| `orca-spec-version` | string | `""` | no | Free-form version tag (e.g. `"1"`) |
|
||||
| `kind` | enum | — | **yes** | One of `Job`, `Service`, `DaemonSet` |
|
||||
| `name` | string | — | **yes** | Workload name (trimmed, non-empty) |
|
||||
| `count` | int | `1` | no | Job: must be 1; Service: ≥1; DaemonSet: not allowed |
|
||||
| `runtime` | block | nil | see kinds | Runtime block (or per-task runtimes in a task group) |
|
||||
| `ports` | block list | nil | Service: **yes** | Array of port mappings |
|
||||
| `env` | block map | nil | no | Environment variables |
|
||||
| `secrets` | inline/block list | nil | no | Secret names (resolution in v0.11) |
|
||||
| `volumes` | block list | nil | no | Volume mounts |
|
||||
| `restart` | block | nil | Service/DaemonSet: **yes** | Restart policy |
|
||||
| `update` | block | nil | Service: **yes** | Update strategy |
|
||||
| `service` | block | nil | no | Traefik route definition (implied for Service; not allowed for Job/DaemonSet) |
|
||||
| `health` | block | nil | Service: **yes** | Health check |
|
||||
| `lifecycle` | block | nil | no | Pre-stop / post-start hooks |
|
||||
| `constraints` | list | nil | no | CEL expressions (node selection) |
|
||||
| `affinity` | block list | nil | no | Co-location / anti-affinity rules |
|
||||
| `tasks` | block list | nil | no | Task group (multi-process alloc) |
|
||||
| `timeout` | duration string | `""` | no | Job timeout |
|
||||
| `schedule` | block | nil | DaemonSet: **yes** | Schedule mode |
|
||||
|
||||
## Kinds
|
||||
|
||||
### `Job`
|
||||
|
||||
A one-shot batch task. Runs once and exits.
|
||||
|
||||
- `count` must be 1 (or unset). Use `Service` for replicas.
|
||||
- `service` block is **not allowed** (no Traefik route for Jobs).
|
||||
- `restart` optional (defaults to `never` / `on-failure`).
|
||||
- `timeout` optional.
|
||||
|
||||
**Example**:
|
||||
```yaml
|
||||
---
|
||||
kind: Job
|
||||
name: data-migration
|
||||
runtime:
|
||||
one_of: process
|
||||
command: /usr/bin/python3 migrate.py
|
||||
timeout: 300s
|
||||
env:
|
||||
DB_URL: postgres://localhost/mydb
|
||||
---
|
||||
```
|
||||
|
||||
### `Service`
|
||||
|
||||
A long-running, load-balanced workload with a Traefik route.
|
||||
|
||||
- `count` ≥ 1 (number of replicas).
|
||||
- `ports` required (at least one).
|
||||
- `restart` required; `mode` one of `service`, `on-failure`, `never`.
|
||||
- `update` required; `strategy` one of `rolling`, `canary`, `blue-green`.
|
||||
- `runtime` required (or a task group with per-task runtimes).
|
||||
- `health` required (Traefik routing requires health checks).
|
||||
- `service` block optional (implied for Service; use for `bind` override).
|
||||
- `service.bind` if present must be a valid IP (`127.0.0.1` = TCP opt-in;
|
||||
default = Unix socket).
|
||||
|
||||
**Example**:
|
||||
```yaml
|
||||
---
|
||||
kind: Service
|
||||
name: web
|
||||
count: 3
|
||||
runtime:
|
||||
one_of: process
|
||||
command: /usr/bin/httpd
|
||||
ports:
|
||||
- name: http
|
||||
port: 8080
|
||||
restart:
|
||||
mode: service
|
||||
attempts: 5
|
||||
delay: 2s
|
||||
update:
|
||||
strategy: rolling
|
||||
max_parallel: 1
|
||||
health:
|
||||
check_type: http
|
||||
interval: 5s
|
||||
timeout: 1s
|
||||
unhealthy_threshold: 2
|
||||
constraints:
|
||||
- node.role == "web"
|
||||
---
|
||||
```
|
||||
|
||||
### `DaemonSet`
|
||||
|
||||
A workload that runs on every matching node.
|
||||
|
||||
- `schedule` required; `mode` one of `every-node`, `matching`, `mandatory`.
|
||||
- `ports` **not allowed** (no Traefik route by default).
|
||||
- `count` **not allowed** (implicit = matching nodes).
|
||||
- `restart` required.
|
||||
|
||||
**Example**:
|
||||
```yaml
|
||||
---
|
||||
kind: DaemonSet
|
||||
name: log-shipper
|
||||
schedule:
|
||||
mode: every-node
|
||||
runtime:
|
||||
one_of: process
|
||||
command: /usr/bin/fluent-bit
|
||||
restart:
|
||||
mode: service
|
||||
---
|
||||
```
|
||||
|
||||
## Block reference
|
||||
|
||||
### `runtime`
|
||||
|
||||
The runtime backend for the workload.
|
||||
|
||||
| Field | Key | Type | Default | Notes |
|
||||
|-------|-----|------|---------|-------|
|
||||
| `one_of` | `one_of` | string | — | Runtime type (see below) |
|
||||
| `image` | `image` | string | `""` | Container image (for `podman`) |
|
||||
| `command` | `command` | string | — | ExecStart command |
|
||||
|
||||
**Supported runtime types** (`one_of`):
|
||||
|
||||
| Type | Description | Requires |
|
||||
|------|-------------|----------|
|
||||
| `process` | Direct process execution via systemd (default) | systemd on target |
|
||||
| `wasm` / `wasmtime` | WASM via wasmtime CLI (apt-installed on peer, SSH exec) | wasmtime on target |
|
||||
| `podman` | Container via podman | podman on target |
|
||||
| `pve-vm` | Proxmox VM via `qm` | Proxmox node |
|
||||
| `pve-ct` | Proxmox container via `pct` | Proxmox node |
|
||||
| `proxmox` | Alias for Proxmox runtime | Proxmox node |
|
||||
|
||||
An empty/missing `Runtime` or `OneOf` is runtime-agnostic (always fits
|
||||
the runtime axis in the scheduler).
|
||||
|
||||
### `ports`
|
||||
|
||||
Array of port mappings. Required for `Service`.
|
||||
|
||||
| Field | Key | Type | Default | Notes |
|
||||
|-------|-----|------|---------|-------|
|
||||
| `name` | `name` | string | — | Port name (used in socket path) |
|
||||
| `port` | `port` | int | — | Container port |
|
||||
| `host_port` | `host_port` | int | `0` | Host port |
|
||||
| `protocol` | `protocol` | string | `""` | Protocol (e.g. `tcp`) |
|
||||
| `host_ip` | `host_ip` | string | `""` | Host IP |
|
||||
|
||||
**Example**:
|
||||
```yaml
|
||||
ports:
|
||||
- name: http
|
||||
port: 8080
|
||||
host_port: 80
|
||||
protocol: tcp
|
||||
- name: https
|
||||
port: 8443
|
||||
host_port: 443
|
||||
```
|
||||
|
||||
### `env`
|
||||
|
||||
Environment variables. Scalar values or secret references.
|
||||
|
||||
```yaml
|
||||
env:
|
||||
FOO: bar
|
||||
BAZ: "qux"
|
||||
SECRET_REF:
|
||||
from: "secret:db-password"
|
||||
INLINE: {from: "secret:token"}
|
||||
```
|
||||
|
||||
> Secret resolution (`from: "secret:..."`) lands in v0.11-P03. The
|
||||
> parser stores the reference; the emitter will emit
|
||||
> `EnvironmentFile=`/`LoadCredential=` in v0.11.
|
||||
|
||||
### `secrets`
|
||||
|
||||
List of secret names. Inline array or block list.
|
||||
|
||||
```yaml
|
||||
secrets: ["db-password", "api-token"]
|
||||
# or
|
||||
secrets:
|
||||
- db-password
|
||||
- api-token
|
||||
```
|
||||
|
||||
### `volumes`
|
||||
|
||||
Array of volume mounts.
|
||||
|
||||
| Field | Key | Type | Default | Notes |
|
||||
|-------|-----|------|---------|-------|
|
||||
| `name` | `name` | string | — | Volume name |
|
||||
| `type` | `type` | string | — | Volume type (e.g. `host`) |
|
||||
| `source` | `source` | string | — | Source path (or `replicate:<peer>,<peer>` for Syncthing) |
|
||||
| `target` | `target` | string | — | Mount target |
|
||||
| `read_only` | `read_only` | bool | `false` | Read-only mount (`true`/`yes`/`on`/`1`) |
|
||||
|
||||
**Example**:
|
||||
```yaml
|
||||
volumes:
|
||||
- name: data
|
||||
type: host
|
||||
source: /data
|
||||
target: /data
|
||||
read_only: true
|
||||
```
|
||||
|
||||
### `restart`
|
||||
|
||||
Restart policy.
|
||||
|
||||
| Field | Key | Type | Default | Notes |
|
||||
|-------|-----|------|---------|-------|
|
||||
| `mode` | `mode` | enum | — | `never`, `on-failure`, `service` |
|
||||
| `attempts` / `max_retries` | `attempts` or `max_retries` | int | `0` | Max retries (both keys accepted) |
|
||||
| `delay` | `delay` | duration string | `""` | Retry delay (e.g. `2s`) |
|
||||
|
||||
### `update`
|
||||
|
||||
Update strategy. Required for `Service`.
|
||||
|
||||
| Field | Key | Type | Default | Notes |
|
||||
|-------|-----|------|---------|-------|
|
||||
| `strategy` | `strategy` | enum | — | `rolling`, `canary`, `blue-green` |
|
||||
| `max_surge` | `max_surge` | int | `0` | Max surge |
|
||||
| `max_parallel` | `max_parallel` | int | `1` (clamped to `count`) | Max parallel updates |
|
||||
| `min_healthy_time` | `min_healthy_time` | duration | `""` | Min time healthy before next batch |
|
||||
| `healthy_deadline` | `healthy_deadline` | duration | `""` | Deadline for health |
|
||||
| `canary` | `canary` | int or `"<n>%"` | — | Canary size (int count or percentage) |
|
||||
| `auto_promote` | `auto_promote` | bool | `false` | Auto-promote canary (`true`/`yes`/`on`/`1`) |
|
||||
|
||||
**Strategies**:
|
||||
- **rolling**: batches of `max_parallel`, each batch waits for healthy.
|
||||
- **canary**: canary batch first, then `promote` (manual or `auto_promote`), then remaining in `max_parallel` batches.
|
||||
- **blue-green**: all new allocs start in parallel, wait healthy, then `cutover`.
|
||||
|
||||
> Transactional update execution lands in v0.11-P10. The current
|
||||
> emitter computes the plan; execution is a v0.11 deliverable.
|
||||
|
||||
### `service`
|
||||
|
||||
Traefik route definition. Implied for `Service`; not allowed for
|
||||
`Job`/`DaemonSet`. See [docs/ingress.md](ingress.md) for details.
|
||||
|
||||
| Field | Key | Type | Default | Notes |
|
||||
|-------|-----|------|---------|-------|
|
||||
| `name` | `name` | string | — | Service name |
|
||||
| `port` | `port` | int | — | Service port |
|
||||
| `bind` | `bind` | string (IP) | `""` | Bind mode: empty = Unix socket (default); `127.0.0.1` = TCP opt-in (R-007) |
|
||||
| `host` | `host` | string | `""` | Host (stored, not yet consumed by emitter) |
|
||||
| `route_id` | `route_id` | string | `""` | Route ID (stored, not yet consumed by emitter) |
|
||||
|
||||
### `health`
|
||||
|
||||
Health check. Required for `Service`.
|
||||
|
||||
| Field | Key | Type | Default | Notes |
|
||||
|-------|-----|------|---------|-------|
|
||||
| `check_type` | `check_type` | string | — | Check type (e.g. `http`) |
|
||||
| `interval` | `interval` | duration string | — | Check interval (e.g. `5s`) |
|
||||
| `timeout` | `timeout` | duration string | — | Check timeout |
|
||||
| `unhealthy_threshold` | `unhealthy_threshold` | int | `0` | Failures before unhealthy |
|
||||
|
||||
Maps to Traefik `healthCheck` stanza (`path: /healthz`).
|
||||
|
||||
### `lifecycle`
|
||||
|
||||
Lifecycle hooks. Maps to systemd `ExecStartPost` / `ExecStop`.
|
||||
|
||||
| Field | Key | Type | Default | systemd mapping |
|
||||
|-------|-----|------|---------|-----------------|
|
||||
| `post_start` | `post_start` | string list | nil | `ExecStartPost=` (runs after main starts) |
|
||||
| `pre_stop` | `pre_stop` | string list | nil | `ExecStop=` (runs before kill) |
|
||||
|
||||
**Example**:
|
||||
```yaml
|
||||
lifecycle:
|
||||
pre_stop:
|
||||
- /bin/sh -c 'sleep 5'
|
||||
- /usr/local/bin/drain.sh
|
||||
post_start:
|
||||
- /usr/local/bin/warm-cache.sh
|
||||
```
|
||||
|
||||
### `constraints`
|
||||
|
||||
CEL-subset expressions for node selection. Inline array or block list.
|
||||
|
||||
```yaml
|
||||
constraints:
|
||||
- node.role == "web"
|
||||
- region == "us"
|
||||
# or inline
|
||||
constraints: ['node.role == "web"', 'region == "us"']
|
||||
```
|
||||
|
||||
**CEL subset grammar** (hand-rolled, no CEL dependency):
|
||||
- Node attributes: `node.hostname`, `node.kind`, `node.cpus`,
|
||||
`node.memory`, `node.tags`, `node.runtimes`
|
||||
- Bare identifiers: equivalent to `node.<name>`
|
||||
- Literals: string (`"..."`), int
|
||||
- Comparisons: `==`, `!=`, `>=`, `<=`, `>`, `<`
|
||||
- Membership: `in`, `not in`
|
||||
- Boolean: `and`, `or`, `not`, parentheses
|
||||
- Anything outside the subset returns an error (node skipped, not
|
||||
silently mis-evaluated)
|
||||
|
||||
### `affinity`
|
||||
|
||||
Co-location / anti-affinity rules.
|
||||
|
||||
```yaml
|
||||
affinity:
|
||||
- target: zone == "a"
|
||||
weight: 80
|
||||
- target: web
|
||||
weight: -50 # anti-affinity (negative weight)
|
||||
```
|
||||
|
||||
- `target`: CEL expression or bare workload name (for name-based
|
||||
co-location).
|
||||
- `weight`: positive = co-locate, negative = anti-affinity.
|
||||
- Affinity is a **hint** (not a gate); evaluation failures are ignored.
|
||||
|
||||
### `tasks` (task group)
|
||||
|
||||
Multi-process alloc (P06). When `tasks` is non-empty, the alloc runs
|
||||
multiple processes, each as its own systemd unit, grouped under a
|
||||
systemd target.
|
||||
|
||||
```yaml
|
||||
tasks:
|
||||
- name: app
|
||||
runtime:
|
||||
one_of: process
|
||||
command: /usr/bin/httpd -f
|
||||
env:
|
||||
LOG_LEVEL: debug
|
||||
- name: sidecar
|
||||
runtime:
|
||||
one_of: wasm
|
||||
command: /bin/wasm-runner sidecar.wasm
|
||||
```
|
||||
|
||||
- A task with no `runtime:` inherits the top-level `spec.Runtime`.
|
||||
- Each task can have its own `env:` overlay.
|
||||
- `command` falls back: `task.Command` → `task.Runtime.Command` →
|
||||
`spec.Runtime.Command`.
|
||||
- Task names must be unique within the group.
|
||||
|
||||
## Kinds matrix
|
||||
|
||||
| Feature | Job | Service | DaemonSet |
|
||||
|---------|-----|---------|-----------|
|
||||
| `count` | must be 1 | ≥ 1 | not allowed |
|
||||
| `ports` | optional | **required** | not allowed |
|
||||
| `service` block | not allowed | optional (implied) | not allowed |
|
||||
| `restart` | optional | **required** | **required** |
|
||||
| `update` | optional | **required** | optional |
|
||||
| `health` | optional | **required** | optional |
|
||||
| `runtime` | optional | **required** (or task group) | optional |
|
||||
| `schedule` | optional | optional | **required** |
|
||||
| `tasks` | optional | optional | optional |
|
||||
| Traefik route | no | yes (implied) | no (by default) |
|
||||
|
||||
## Body semantics
|
||||
|
||||
The body after the closing `---` is preserved **byte-exact** (R-015) —
|
||||
including trailing newlines, CRLF, BOM in body, and `---` inside code
|
||||
fences. The body is carried verbatim to the target node. It is not
|
||||
interpreted as commands/scripts by the parser today.
|
||||
|
||||
## Deprecated: HCL jobspec
|
||||
|
||||
The legacy HCL jobspec format is supported via an adapter during the
|
||||
migration window. It is deprecated in v0.9 and will be removed in
|
||||
v0.11.
|
||||
|
||||
```hcl
|
||||
job "hello-orca" {
|
||||
}
|
||||
|
||||
task "greet" {
|
||||
command = "/bin/echo"
|
||||
args = ["hello", "from", "orca"]
|
||||
}
|
||||
```
|
||||
|
||||
The adapter converts this to a `*WorkloadSpec{Kind: "Job", Name:
|
||||
"hello-orca", Count: 1, Runtime: {OneOf: "process", Command:
|
||||
"/bin/echo"}}`. Use `.md` for all new jobspecs.
|
||||
|
||||
## See also
|
||||
|
||||
- [docs/cli.md](cli.md) — CLI reference (including `orca job run`)
|
||||
- [docs/ingress.md](ingress.md) — Traefik ingress configuration
|
||||
- [examples/full-stack/](../examples/full-stack/) — Full-stack example jobspecs
|
||||
@@ -1,52 +0,0 @@
|
||||
# Orca Metrics Reference
|
||||
|
||||
Orca exposes Prometheus text-exposition metrics at `/metrics` on the
|
||||
metrics endpoint (default `:9100`, configurable via `--addr`).
|
||||
|
||||
## Running the metrics endpoint
|
||||
|
||||
```sh
|
||||
orca metrics --addr :9100
|
||||
```
|
||||
|
||||
## Prometheus scrape config
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: orca
|
||||
static_configs:
|
||||
- targets: ['localhost:9100']
|
||||
scrape_interval: 15s
|
||||
```
|
||||
|
||||
## Metric reference
|
||||
|
||||
| Metric | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `nodes_total` | Gauge | Total number of registered nodes |
|
||||
| `allocs_total` | Gauge | Total number of job allocations |
|
||||
| `orca_jobs_by_state{state}` | Gauge | Jobs grouped by status (running, complete, failed, etc.) |
|
||||
| `orca_audit_chain_head` | Gauge | Audit chain integrity (1 = chain head verified, 0 = error) |
|
||||
|
||||
## Counter metrics (incremented by CLI operations)
|
||||
|
||||
The following counters are incremented during normal operations and
|
||||
are available when the metrics endpoint polls the DB:
|
||||
|
||||
| Metric | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `orca_drift_events_total` | Counter | Total drift events detected |
|
||||
| `orca_ssh_errors_total` | Counter | Total SSH connection/exec errors |
|
||||
| `orca_txn_apply_total` | Counter | Total transaction applies |
|
||||
| `orca_txn_rollback_total` | Counter | Total transaction rollbacks |
|
||||
| `orca_acl_denials_total` | Counter | Total ACL denials (enforce mode) |
|
||||
|
||||
## Security headers
|
||||
|
||||
The metrics endpoint sets the following security headers on all responses:
|
||||
- `X-Content-Type-Options: nosniff`
|
||||
- `X-Frame-Options: DENY`
|
||||
|
||||
## Health check
|
||||
|
||||
The endpoint also exposes `/healthz` returning `200 ok` for liveness probes.
|
||||
@@ -1,226 +0,0 @@
|
||||
# Namespace and Paths
|
||||
|
||||
Orca stores all on-disk state under a single **namespace root**
|
||||
directory. The v0.9 re-architecture introduced a multi-namespace
|
||||
layout (R-002) where each namespace is a self-contained directory tree
|
||||
with its own database, jobs, allocs, env, and secrets. A `cluster/`
|
||||
directory holds cluster-wide artifacts shared across namespaces.
|
||||
|
||||
> **v0.9 layout (canonical)**: This document describes the v0.9
|
||||
> multi-namespace layout. The v0.8 flat layout (`orca.db`, `ca.crt`,
|
||||
> `server.crt` at the root) is deprecated and removed in v0.12
|
||||
> (REQ-138).
|
||||
|
||||
## Namespace root resolution
|
||||
|
||||
The namespace root is resolved in this order:
|
||||
|
||||
1. If `--system` flag is passed → root is `/root/.orca` (errors if
|
||||
`ORCA_HOME` is set to a conflicting value).
|
||||
2. Else if `ORCA_HOME` is set → root is `$ORCA_HOME`.
|
||||
3. Else → root is `~/.orca` (`$HOME/.orca`).
|
||||
|
||||
### `ORCA_HOME` (REQ-041)
|
||||
|
||||
Set the `ORCA_HOME` environment variable to change the namespace root
|
||||
for all orca components:
|
||||
|
||||
```bash
|
||||
export ORCA_HOME=/var/lib/orca
|
||||
orca init # creates /var/lib/orca/
|
||||
orca ns create prod
|
||||
```
|
||||
|
||||
### `--system` (REQ-042)
|
||||
|
||||
The `--system` persistent flag selects the system-level namespace root
|
||||
`/root/.orca`:
|
||||
|
||||
```bash
|
||||
sudo orca --system init # creates /root/.orca/
|
||||
sudo orca --system ns list
|
||||
```
|
||||
|
||||
If `ORCA_HOME` is already set to a different value, `--system` returns
|
||||
an error (to avoid silent namespace mismatches).
|
||||
|
||||
## v0.9 multi-namespace layout (R-002)
|
||||
|
||||
```
|
||||
$ORCA_HOME/
|
||||
├── cluster/ # cluster-wide (NOT a workload namespace)
|
||||
│ ├── ca.crt, ca.key # step-ca root (R-006, D-101)
|
||||
│ ├── master.key # AES-256-GCM root (R-011, mode 0600)
|
||||
│ ├── master.key.sealed # sealed master key (REQ-147, mode 0600)
|
||||
│ ├── config.md # Markdown frontmatter config (R-014)
|
||||
│ ├── known_hosts # SSH known_hosts (D-035)
|
||||
│ ├── orca_ssh_key # orca SSH private key (D-037)
|
||||
│ ├── orca_ssh_key.pub # orca SSH public key
|
||||
│ ├── peers/<host>/ # per-peer directory
|
||||
│ ├── txns/ # cluster transaction log (R-016)
|
||||
│ ├── acl.json # ACL state (mode 0600)
|
||||
│ ├── oidc-client-secret # OIDC client secret (mode 0600, C-36)
|
||||
│ ├── webauthn-credentials.db # WebAuthn public keys (mode 0600)
|
||||
│ └── state/ # cluster state
|
||||
├── _defaults/ # implicit root namespace (always exists)
|
||||
│ ├── ns.md # namespace frontmatter (kind: Namespace)
|
||||
│ ├── .env # per-namespace env
|
||||
│ ├── .env.secrets # encrypted secrets
|
||||
│ ├── db/orca.db # per-namespace SQLite database
|
||||
│ ├── jobs/ # submitted jobspecs
|
||||
│ └── alloc/ # allocation state
|
||||
├── <explicit-namespace>/ # operator-created (e.g., prod, staging)
|
||||
│ ├── ns.md
|
||||
│ ├── .env, .env.secrets
|
||||
│ ├── db/orca.db
|
||||
│ ├── jobs/, alloc/
|
||||
│ └── syncthing/ # Syncthing config (if replicated volumes)
|
||||
└── orca_cache.db # CLI-side cache (R-008)
|
||||
```
|
||||
|
||||
### Key points
|
||||
|
||||
- **`_defaults/`** is the implicit root namespace (D-159). It always
|
||||
exists. Every namespace inherits from `_defaults` and cannot opt out
|
||||
(D-185, D-187).
|
||||
- **`cluster/`** is NOT a workload namespace — it holds cluster-wide
|
||||
artifacts (CA, master key, SSH keys, known_hosts, peers, txns, ACL,
|
||||
OIDC secrets, WebAuthn credentials).
|
||||
- **Per-namespace DBs**: each namespace has its own
|
||||
`db/orca.db` (R-002). No namespace column in SQLite.
|
||||
- **Namespace inheritance**: child namespaces inherit env and
|
||||
constraints from parents (via `ns.md` frontmatter `parents:` field).
|
||||
`_defaults` is always appended last in the inheritance chain.
|
||||
- **`orca ns` subcommands**: `list`, `create`, `delete`, `inspect`,
|
||||
`validate`, `inherit`, `set-constraint` — see below and
|
||||
[docs/cli.md](cli.md#orca-ns).
|
||||
|
||||
### Path reference (`internal/paths/`)
|
||||
|
||||
| Function | Path | Contents |
|
||||
|----------|------|----------|
|
||||
| `Root()` | `$ORCA_HOME` | Namespace root |
|
||||
| `ClusterDir()` | `Root()/cluster` | Cluster-wide artifacts |
|
||||
| `NamespaceDir(ns)` | `Root()/ns` | Per-namespace directory |
|
||||
| `NSDb(ns)` | `Root()/ns/db/orca.db` | Per-namespace SQLite DB |
|
||||
| `NSEnv(ns)` | `Root()/ns/.env` | Per-namespace env |
|
||||
| `NSSecrets(ns)` | `Root()/ns/.env.secrets` | Encrypted secrets |
|
||||
| `NSJobs(ns)` | `Root()/ns/jobs` | Jobs dir |
|
||||
| `NSAlloc(ns)` | `Root()/ns/alloc` | Alloc dir |
|
||||
| `NSMd(ns)` | `Root()/ns/ns.md` | Namespace frontmatter |
|
||||
| `DefaultNamespace()` | `_defaults` | Implicit root (D-159) |
|
||||
| `CACertPath()` | `ClusterDir()/ca.crt` | step-ca root (D-101) |
|
||||
| `MasterKeyPath()` | `ClusterDir()/master.key` | AES-256-GCM root key |
|
||||
| `KnownHostsPath()` | `ClusterDir()/known_hosts` | SSH known_hosts |
|
||||
| `SSHKeyPath()` | `ClusterDir()/orca_ssh_key` | orca SSH private key |
|
||||
| `ConfigPath()` | `ClusterDir()/config.md` | Markdown config (R-014) |
|
||||
| `CacheDB()` | `Root()/orca_cache.db` | CLI-side cache (R-008) |
|
||||
| `PeersDir()` | `ClusterDir()/peers` | Peers directory |
|
||||
| `TxnDir()` | `ClusterDir()/txns` | Transaction log (R-016) |
|
||||
|
||||
## Creating and managing namespaces
|
||||
|
||||
```bash
|
||||
# List all namespaces
|
||||
orca ns list
|
||||
|
||||
# Create a namespace (inherits from _defaults)
|
||||
orca ns create prod
|
||||
|
||||
# Create a namespace with an explicit parent
|
||||
orca ns create staging --parent prod
|
||||
|
||||
# Inspect the effective inheritance chain + merged env
|
||||
orca ns inspect prod
|
||||
|
||||
# Validate a namespace's inheritance chain
|
||||
orca ns validate prod
|
||||
|
||||
# Delete an empty namespace (refuses if jobs/ or alloc/ non-empty)
|
||||
orca ns delete staging
|
||||
```
|
||||
|
||||
### `orca ns inherit` — set parent namespace (R-002)
|
||||
|
||||
Set the parent namespace for a namespace. Updates `ns.md` frontmatter
|
||||
(`parents` field) and validates the new chain has no cycles. The
|
||||
implicit root `_defaults` is always appended last (D-185).
|
||||
|
||||
```bash
|
||||
orca ns inherit <name> --parent <parent-namespace>
|
||||
```
|
||||
|
||||
**Example**:
|
||||
```bash
|
||||
# Make staging inherit from prod (chain: staging -> prod -> _defaults)
|
||||
orca ns inherit staging --parent prod
|
||||
```
|
||||
|
||||
The child cannot inherit from itself transitively — the resolver
|
||||
validates the chain before writing. If a cycle is detected, the
|
||||
command exits 1 with an error.
|
||||
|
||||
### `orca ns set-constraint` — set a constraint (R-002)
|
||||
|
||||
Set a constraint on a namespace. Constraints are `key=value` strings
|
||||
(e.g., `max-allocs=10`) stored in `ns.md` frontmatter and unioned
|
||||
across the inheritance chain by the resolver.
|
||||
|
||||
```bash
|
||||
orca ns set-constraint <name> <key>=<value>
|
||||
```
|
||||
|
||||
**Example**:
|
||||
```bash
|
||||
# Limit prod to 10 concurrent allocations
|
||||
orca ns set-constraint prod max-allocs=10
|
||||
|
||||
# Set a required node affinity
|
||||
orca ns set-constraint prod require-label=ssd
|
||||
```
|
||||
|
||||
Constraints are unioned (not overridden) across the inheritance chain:
|
||||
if `_defaults` sets `max-allocs=50` and `prod` sets `max-allocs=10`,
|
||||
the effective constraint is the most restrictive one (CEL evaluation
|
||||
determines precedence per constraint key).
|
||||
|
||||
See [docs/cli.md](cli.md#orca-ns) for the full `orca ns` reference.
|
||||
|
||||
## `ORCA_DB` override
|
||||
|
||||
For finer-grained control, `ORCA_DB` overrides only the database path
|
||||
(not the cert/namespace paths). This is primarily a testing affordance.
|
||||
|
||||
```bash
|
||||
export ORCA_DB=/tmp/test.db
|
||||
orca init # uses /tmp/test.db for the DB, ~/.orca/ for everything else
|
||||
```
|
||||
|
||||
## Deprecated: v0.8 flat layout
|
||||
|
||||
> **Removed in v0.12** (REQ-138): The v0.8 flat layout (`orca.db`,
|
||||
> `ca.crt`, `ca.key`, `server.crt`, `server.key` at the namespace root)
|
||||
> is superseded by the v0.9 multi-namespace layout (R-002) and the
|
||||
> dual-write window is closed.
|
||||
|
||||
The v0.8 flat layout stored all state at the namespace root:
|
||||
|
||||
| Path | Contents |
|
||||
|------|----------|
|
||||
| `~/.orca/orca.db` | SQLite database |
|
||||
| `~/.orca/ca.crt` | CA certificate |
|
||||
| `~/.orca/ca.key` | CA private key |
|
||||
| `~/.orca/server.crt` | Server certificate |
|
||||
| `~/.orca/server.key` | Server private key |
|
||||
|
||||
The v0.9 re-architecture moved these to `cluster/` (CA, SSH keys) and
|
||||
per-namespace `db/` (SQLite) to support multi-tenancy (R-002). The
|
||||
`internal/certpaths` shim that supported the dual-write window is
|
||||
removed in v0.12.
|
||||
|
||||
## See also
|
||||
|
||||
- [Install Guide](install.md) — 1-liner install with `install.sh`.
|
||||
- [Docker Guide](docker.md) — running orca in a container.
|
||||
- [CLI Reference](cli.md#orca-ns) — `orca ns` subcommands.
|
||||
- [Jobspec Reference](jobspec.md) — markdown frontmatter schema.
|
||||
@@ -1,31 +0,0 @@
|
||||
# OIDC Configuration (v0.12)
|
||||
|
||||
## Bundled Dex (default)
|
||||
|
||||
`orca auth init-idp --rp-id <cluster-domain>` bootstraps a local Dex
|
||||
on the lead, fronted by Traefik (step-ca cert). The WebAuthn connector
|
||||
provides password-free passkey registration + login.
|
||||
|
||||
## BYO External IdP
|
||||
|
||||
Set `oidc.issuer` in config to repoint to Keycloak/Authentik/Google/etc.
|
||||
The bundled Dex is bypassed; the external IdP's authenticators are used.
|
||||
|
||||
## Claim-to-Namespace Mapping
|
||||
|
||||
OIDC `sub` (subject) maps to an ACL entry. Groups (`groups` claim) map
|
||||
to group-based grants. `orca acl grant <ns> --oidc-sub <sub> --perm read`
|
||||
or `orca acl grant <ns> --oidc-group <group> --perm admin`.
|
||||
|
||||
## Offline / Air-Gapped
|
||||
|
||||
Run the bundled Dex on the lead (offline). For the single-operator
|
||||
fully-offline case, skip OIDC and rely on mTLS-only machine identity
|
||||
(no human authn needed; the operator holds the pre-staged SSH key +
|
||||
mTLS cert; no password, no token).
|
||||
|
||||
## Credentials Storage
|
||||
|
||||
`~/.orca/credentials.json` (0600). Short-lived ID token (1h) + refresh.
|
||||
The IdP issues tokens; Orca only stores them. No long-lived
|
||||
Orca-issued tokens (R-021).
|
||||
@@ -1,148 +0,0 @@
|
||||
# Security Runbook (v0.13)
|
||||
|
||||
This runbook documents the operational security procedures for orca's
|
||||
zero-trust identity model (R-021): human identity is exclusively
|
||||
external (OIDC), machine identity is exclusively mTLS/SPIFFE, and no
|
||||
passwords / Orca-issued tokens / CA-key passphrases exist anywhere in
|
||||
the system. The v0.12 milestone shipped these capabilities; the v0.13
|
||||
milestone wired them operationally (R-023).
|
||||
|
||||
## Master Key Seal/Unseal (REQ-147, P05)
|
||||
|
||||
The cluster master key (`ClusterDir()/master.key`, mode 0600) encrypts
|
||||
all namespace `.env.secrets` via per-namespace HKDF-SHA256 sub-keys
|
||||
(AES-256-GCM). The master key can be **sealed** (encrypted at rest) and
|
||||
**unsealed** (unwrapped into memory for use).
|
||||
|
||||
### Seal
|
||||
|
||||
```bash
|
||||
orca cluster seal
|
||||
```
|
||||
|
||||
Encrypts the raw master key with a key derived from either:
|
||||
- the OIDC ID token subject (if `orca auth login` has been run), or
|
||||
- the cluster CA fingerprint (mTLS-only offline path, D-241).
|
||||
|
||||
The sealed blob is written to `ClusterDir()/master.key.sealed` (0600).
|
||||
**Five Shamir shards (3-of-5 recovery)** are printed to stdout — store
|
||||
them offline. The raw master key is then deleted from disk so the
|
||||
cluster is sealed at rest.
|
||||
|
||||
### Unseal
|
||||
|
||||
```bash
|
||||
orca cluster unseal
|
||||
```
|
||||
|
||||
Reads the sealed blob and unwraps the master key using the OIDC ID
|
||||
token subject or the cluster CA fingerprint. The unwrapped key is
|
||||
written back to `ClusterDir()/master.key` (0600) and zeroed from
|
||||
memory on process exit.
|
||||
|
||||
### Recovery (IdP lost)
|
||||
|
||||
```bash
|
||||
orca cluster unseal --recovery
|
||||
```
|
||||
|
||||
If the IdP is permanently lost, the operator is prompted for 3 of the
|
||||
5 Shamir shards printed at seal time. With quorum, the master key is
|
||||
reconstructed and written back to disk. If quorum is unavailable, the
|
||||
cluster is unrecoverable by design (C-35: no backdoor).
|
||||
|
||||
## Master Key Rotation (REQ-129, C-30)
|
||||
|
||||
```bash
|
||||
orca secrets rotate-master [--dry-run]
|
||||
```
|
||||
|
||||
Generates a new master key, re-encrypts every namespace's
|
||||
`.env.secrets` under the new key, and re-seals the master key to OIDC.
|
||||
With `--dry-run`, reports affected namespaces without writing.
|
||||
|
||||
- **Atomic per-namespace**: each namespace is re-encrypted independently.
|
||||
- **Automatic rollback**: on any namespace failure, the old sealed key
|
||||
is restored (C-30).
|
||||
- **No passphrase** (R-021): the master key is sealed to OIDC, not to a
|
||||
human-typed passphrase.
|
||||
|
||||
## File-Mode Audit (REQ-033, REQ-130, F13)
|
||||
|
||||
```bash
|
||||
orca doctor modes
|
||||
```
|
||||
|
||||
Verifies file modes on security-sensitive files across `ORCA_HOME`:
|
||||
- private keys / secrets: `0600`
|
||||
- certs / public keys: `0644`
|
||||
|
||||
Exits 0 if all files have correct modes; exits 1 if any violation is
|
||||
found. Missing files are not counted as violations.
|
||||
|
||||
Checks: SSH key, master key (sealed blob), server cert/key,
|
||||
known_hosts, `acl.json`, OIDC client secret.
|
||||
|
||||
## Audit Log Tamper-Evidence (REQ-125, F2)
|
||||
|
||||
```bash
|
||||
orca doctor audit
|
||||
```
|
||||
|
||||
Verifies the audit log hash chain. Opens the orca SQLite DB, recomputes
|
||||
the hash chain from the first audit entry, and reports the chain head
|
||||
hash. If any entry's `entry_hash` or `prev_hash` link does not match the
|
||||
recomputed value, the chain has been tampered with and the command
|
||||
exits non-zero.
|
||||
|
||||
The audit log is append-only (SQLite trigger blocks
|
||||
UPDATE/DELETE). Each entry's `actor` field carries the OIDC `sub` or
|
||||
SPIFFE SVID. Run this after any suspected intrusion or as part of a
|
||||
regular audit cadence.
|
||||
|
||||
## Sudoers Audit (REQ-134, F22)
|
||||
|
||||
```bash
|
||||
orca doctor proxmox
|
||||
```
|
||||
|
||||
Audits the `/etc/sudoers.d/orca` file against the expected allowlist:
|
||||
- `pct` + `qm` with NOEXEC
|
||||
- `apt-get` / `dpkg` excluded (or NOEXEC'd)
|
||||
- `pvesh` EXCLUDED (AD-020: pvesh can bypass NOEXEC via the API execute
|
||||
endpoint)
|
||||
|
||||
## nft Audit (REQ-133, F21)
|
||||
|
||||
```bash
|
||||
orca doctor nft
|
||||
```
|
||||
|
||||
Audits the live nftables ingress ruleset against the on-disk
|
||||
`/etc/nftables.d/orca.nft` hash (recorded at the latest applied txn).
|
||||
Reports drift if the live ruleset does not match. Also verifies:
|
||||
- table exists
|
||||
- DNAT `:443 → 127.0.0.1:8443` and `:80 → 127.0.0.1:8080` present
|
||||
- rate-limit meter present
|
||||
- `/etc/nftables.d/orca.nft` parses
|
||||
|
||||
## Incident Response
|
||||
|
||||
1. **Revoke the compromised identity** (OIDC user/group or SPIFFE SVID).
|
||||
2. **Rotate the master key** (`orca secrets rotate-master`).
|
||||
3. **Review the audit log** (`orca doctor audit` verifies the hash
|
||||
chain; `orca audit list` shows entries).
|
||||
4. **Check file modes** (`orca doctor modes` detects permission drift).
|
||||
5. If the master key is compromised, **all historical secrets are
|
||||
compromised** (no forward secrecy — documented residual risk).
|
||||
6. **Re-seal** the master key after rotation (`orca cluster seal`).
|
||||
|
||||
## OIDC Provider Health (P06)
|
||||
|
||||
```bash
|
||||
orca doctor oidc
|
||||
```
|
||||
|
||||
Checks the bundled Dex OIDC provider health. Verifies the Dex systemd
|
||||
unit is running and the `/.well-known/openid-configuration` endpoint
|
||||
responds. Run after `orca auth init-idp` or after a Dex config change.
|
||||
@@ -1,47 +0,0 @@
|
||||
# Orca Threat Model (v0.12)
|
||||
|
||||
## Overview
|
||||
|
||||
Orca is a minimalist, offline-first, CLI-first orchestration engine.
|
||||
v0.12 adopts a **zero-trust identity model** (R-021): no Orca-issued
|
||||
credentials. Human identity is exclusively OIDC; machine identity is
|
||||
exclusively mTLS/SPIFFE.
|
||||
|
||||
## R-021 — No Orca Credentials
|
||||
|
||||
Orca never issues, stores, or accepts human-identity credentials.
|
||||
- Human identity: OIDC (external IdP or bundled Dex + WebAuthn)
|
||||
- Machine identity: mTLS + SPIFFE SVIDs
|
||||
- No passwords, no Orca-issued tokens, no CA-key passphrases
|
||||
|
||||
## STRIDE Analysis
|
||||
|
||||
| Component | Spoofing | Tampering | Repudiation | Info Disclosure | DoS | Elevation |
|
||||
|-----------|----------|-----------|-------------|-----------------|-----|-----------|
|
||||
| OIDC client | mitigated by JWKS verification | — | mitigated by ID token | — | — | — |
|
||||
| WebAuthn connector | mitigated by public-key auth | — | mitigated by signed assertions | — | — | — |
|
||||
| ACL | mitigated by deny-by-default + OIDC claims | — | mitigated by audit log | — | — | mitigated by least-privilege perms |
|
||||
| Master key seal | — | mitigated by AES-256-GCM + Shamir | — | mitigated by 0600 + sealing | — | — |
|
||||
| SSH-push transport | mitigated by key auth + TOFU/pin | — | mitigated by audit | — | mitigated by rate limiting (v1.x) | — |
|
||||
| Daemon (deprecated) | mitigated by mandatory mTLS | — | mitigated by audit | mitigated by body limits | mitigated by body limits | mitigated by ACL |
|
||||
| Backup/restore | — | mitigated by HMAC signature | — | mitigated by symlink validation | — | — |
|
||||
| Audit log | — | mitigated by hash chain + append-only trigger | — | — | — | — |
|
||||
| Drift detection | mitigated by per-peer HMAC | — | — | — | — | — |
|
||||
| nftables ingress | — | — | — | — | mitigated by conntrack + rate limit | — |
|
||||
| sudoers | — | — | — | — | — | mitigated by NOEXEC + least-privilege |
|
||||
|
||||
## OS Surface
|
||||
|
||||
Orca writes to: `/etc/orca/`, `/etc/traefik/orca*`, `/etc/systemd/system/orca-*`,
|
||||
`/etc/nftables.d/orca*`, `/etc/syncthing/orca*`, `/etc/sudoers.d/orca`.
|
||||
All via SSH-push (key auth, no passwords). The `orca` system user is
|
||||
`nologin` (no shell access). Scripts run as root only for file writes
|
||||
to `/etc/` (the operator pre-stages the SSH key; no password flows).
|
||||
|
||||
## Residual Risks
|
||||
|
||||
- Legacy CA/mTLS/daemon dual-write window (v1.x closure)
|
||||
- SQLite unencrypted at rest (0600 file mode; CGO-free SQLCipher is v1.x)
|
||||
- Master key compromise compromises all historical secrets (no forward secrecy)
|
||||
- IdP loss: Shamir 3-of-5 recovery; if quorum unavailable, unrecoverable by design
|
||||
- Transport rate limiting + typed errors (v1.x)
|
||||
-360
@@ -1,360 +0,0 @@
|
||||
# Orca User Acceptance Testing (UAT) Plan
|
||||
|
||||
**Version**: v0.13 (production hardening round 2)
|
||||
**Gate**: v1.0.0 production-ready tag is deferred until this UAT passes
|
||||
**Signoff**: run `scripts/uat-signoff.sh` on the lead node and paste the output back
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Hardware
|
||||
|
||||
| Role | OS | Requirements |
|
||||
|------|-----|-------------|
|
||||
| **lead** | Ubuntu 22.04 LTS | Operator laptop or VM; SSH key; `orca` binary (built from v0.13 tag) |
|
||||
| **pve01** | Proxmox VE 8/9 | Bare-metal or nested; SSH root access; orca SSH key pre-staged |
|
||||
| **worker01** | Ubuntu 22.04 LTS | VM or bare-metal; SSH root access; orca SSH key pre-staged |
|
||||
|
||||
### Alternative topology (3x Ubuntu, no Proxmox)
|
||||
|
||||
If a Proxmox host is unavailable, run the UAT with 3x Ubuntu hosts.
|
||||
Use `--type linux` for all remote nodes. Proxmox-specific claims
|
||||
(`doctor proxmox`, PVE role, sudoers) are **skipped** in this path.
|
||||
The signoff script reports exercised vs. skipped claims.
|
||||
|
||||
## Step-by-step UAT
|
||||
|
||||
### Step 1: Install orca + initialize the cluster
|
||||
|
||||
Install orca (1-liner):
|
||||
```sh
|
||||
curl -fsSL https://git.cloudinit.dev/coreci/orca/raw/branch/main/scripts/install.sh | bash
|
||||
```
|
||||
|
||||
Initialize the cluster:
|
||||
```sh
|
||||
export ORCA_HOME=~/orca-uat
|
||||
orca init
|
||||
```
|
||||
|
||||
**Expected**: orca init creates:
|
||||
- CA cert + server cert
|
||||
- SSH keypair (orca_ssh_key + orca_ssh_key.pub)
|
||||
- known_hosts file (empty, for TOFU capture)
|
||||
- Master key (for secrets encryption)
|
||||
- Traefik data-plane ingress (binary + systemd unit + config)
|
||||
- Localhost node registered
|
||||
|
||||
**Pre-staging remote nodes**: `orca init` interactively prompts for remote
|
||||
host addresses and runs `ssh-copy-id` automatically (password prompt passes
|
||||
through). Enter each host (pve01, worker01) when prompted, or press Enter to
|
||||
skip. The orca public key is deployed to each host; TOFU host-key capture is
|
||||
automatic on the first `orca node join` — no manual fingerprint pinning needed.
|
||||
|
||||
### Step 2: Onboard the Proxmox host
|
||||
|
||||
```sh
|
||||
orca node join --type proxmox \
|
||||
--host pve01 \
|
||||
--ssh-user root
|
||||
```
|
||||
|
||||
**Expected**: SSH bootstrap succeeds, orca user created, PVE role assigned, node registered as `ready` with `kind=proxmox`.
|
||||
|
||||
### Step 3: Onboard the Ubuntu worker
|
||||
|
||||
```sh
|
||||
orca node join --type linux \
|
||||
--host worker01 \
|
||||
--ssh-user root
|
||||
```
|
||||
|
||||
**Expected**: SSH bootstrap succeeds, orca user created, drift-events dir created, node registered as `ready` with `kind=linux`.
|
||||
|
||||
### Step 4: Verify nodes
|
||||
|
||||
```sh
|
||||
orca node list
|
||||
orca node list --json
|
||||
```
|
||||
|
||||
**Expected**: 3 nodes listed (localhost + pve01 + worker01), all `ready`.
|
||||
|
||||
### Step 5: Set capacity on remote nodes
|
||||
|
||||
```sh
|
||||
orca node capacity set --node pve01 --cpu 4 --memory 8192 --disk 100000
|
||||
orca node capacity set --node worker01 --cpu 2 --memory 4096 --disk 50000
|
||||
orca node capacity list
|
||||
```
|
||||
|
||||
**Expected**: capacity shown for both remote nodes.
|
||||
|
||||
### Step 6: Create a namespace
|
||||
|
||||
```sh
|
||||
orca ns create prod
|
||||
orca ns list
|
||||
```
|
||||
|
||||
**Expected**: `prod` namespace listed.
|
||||
|
||||
### Step 7: Deploy the full stack
|
||||
|
||||
Deploy each service from `examples/full-stack/`:
|
||||
|
||||
```sh
|
||||
orca job run examples/full-stack/web-app.md --target pve01
|
||||
orca job run examples/full-stack/api.md --target pve01
|
||||
orca job run examples/full-stack/worker.md --target worker01
|
||||
orca job run examples/full-stack/postgres.md --target pve01
|
||||
orca job run examples/full-stack/log-shipper.md --target worker01
|
||||
```
|
||||
|
||||
**Expected**: each job is scheduled on the target, systemd unit deployed via SSH-push, job status `running` or `complete`.
|
||||
|
||||
### Step 8: Verify deployment
|
||||
|
||||
```sh
|
||||
orca job list
|
||||
orca job list --json
|
||||
```
|
||||
|
||||
**Expected**: all 5 jobs listed, with correct target nodes.
|
||||
|
||||
On each remote node:
|
||||
```sh
|
||||
ssh root@pve01 systemctl status 'orca-alloc-*'
|
||||
ssh root@worker01 systemctl status 'orca-alloc-*'
|
||||
```
|
||||
|
||||
### Step 9: Verify Traefik routes
|
||||
|
||||
```sh
|
||||
ssh root@pve01 ls /etc/traefik/dynamic/
|
||||
ssh root@worker01 ls /etc/traefik/dynamic/
|
||||
```
|
||||
|
||||
**Expected**: `traefik-dynamic-*.yaml` files present on nodes where jobs were deployed.
|
||||
|
||||
### Step 10: Migrate between hosts
|
||||
|
||||
Migrate `web-app` from pve01 to worker01:
|
||||
|
||||
```sh
|
||||
orca job migrate web-app --to worker01
|
||||
```
|
||||
|
||||
**Expected**: job drained on pve01, rescheduled on worker01, new systemd unit deployed.
|
||||
|
||||
Verify:
|
||||
```sh
|
||||
orca job list
|
||||
ssh root@worker01 systemctl status 'orca-alloc-*web-app*'
|
||||
ssh root@pve01 systemctl status 'orca-alloc-*web-app*' # should be stopped
|
||||
```
|
||||
|
||||
### Step 11: Aggregate logs
|
||||
|
||||
```sh
|
||||
orca logs --all-nodes --job web-app --since 5m
|
||||
```
|
||||
|
||||
**Expected**: log entries from multiple nodes.
|
||||
|
||||
### Step 12: ACL enforcement
|
||||
|
||||
```sh
|
||||
orca acl grant operator-1 --namespace prod --permissions read,write
|
||||
orca acl check operator-1 --namespace prod --permission read
|
||||
orca acl check operator-1 --namespace prod --permission admin
|
||||
```
|
||||
|
||||
**Expected**: read+write allowed, admin denied (not granted).
|
||||
|
||||
### Step 12b: Initialize the OIDC provider (for seal)
|
||||
|
||||
```sh
|
||||
orca auth init-idp --rp-id orca.local
|
||||
```
|
||||
|
||||
**Expected**: Dex config + systemd unit + Traefik route rendered. (Dex binary must be installed separately.)
|
||||
|
||||
### Step 13: Seal/unseal
|
||||
|
||||
```sh
|
||||
orca cluster seal
|
||||
orca cluster unseal
|
||||
orca secrets set prod TEST_KEY=test-value
|
||||
orca secrets get prod TEST_KEY
|
||||
```
|
||||
|
||||
**Expected**: seal succeeds, unseal succeeds, secrets readable post-unseal.
|
||||
|
||||
### Step 14: Audit chain
|
||||
|
||||
```sh
|
||||
orca doctor audit
|
||||
```
|
||||
|
||||
**Expected**: chain head reported, no tamper detected.
|
||||
|
||||
### Step 15: Doctor modes
|
||||
|
||||
```sh
|
||||
orca doctor modes
|
||||
```
|
||||
|
||||
**Expected**: all file modes correct, exit 0.
|
||||
|
||||
### Step 16: OIDC health
|
||||
|
||||
```sh
|
||||
orca doctor oidc
|
||||
```
|
||||
|
||||
**Expected**: Dex unit active, issuer reachable (or WARN if Dex not installed).
|
||||
|
||||
### Step 17: Backup and restore
|
||||
|
||||
```sh
|
||||
orca backup --out /tmp/uat-backup.tar.gz
|
||||
orca restore --in /tmp/uat-backup.tar.gz --dry-run
|
||||
```
|
||||
|
||||
**Expected**: backup succeeds, restore dry-run succeeds.
|
||||
|
||||
### Step 18: Drift detection
|
||||
|
||||
```sh
|
||||
orca drift show
|
||||
```
|
||||
|
||||
**Expected**: no error (empty drift is fine).
|
||||
|
||||
### Step 19: Transaction idempotency
|
||||
|
||||
```sh
|
||||
orca txn apply <some-txn-dir>
|
||||
orca txn apply <some-txn-dir> # re-run
|
||||
```
|
||||
|
||||
**Expected**: second apply is idempotent (exit 5 or "already applied").
|
||||
|
||||
### Step 20: Metrics
|
||||
|
||||
```sh
|
||||
orca metrics --addr :9100 &
|
||||
sleep 3
|
||||
curl -s http://localhost:9100/metrics | grep orca_
|
||||
```
|
||||
|
||||
**Expected**: expanded metric set present (`orca_jobs_running`, `orca_audit_chain_head`, etc.).
|
||||
|
||||
### Step 21: Compat check
|
||||
|
||||
```sh
|
||||
orca cluster compat-check
|
||||
```
|
||||
|
||||
**Expected**: exit 0, all nodes compatible.
|
||||
|
||||
### Step 22: Run the signoff script
|
||||
|
||||
```sh
|
||||
scripts/uat-signoff.sh
|
||||
```
|
||||
|
||||
**Expected**: `UAT SIGNOFF: N/35 assertions passed`, exit 0 iff N==35.
|
||||
|
||||
## Claim Matrix
|
||||
|
||||
| # | Claim | UAT Step | Signoff Assertion |
|
||||
|---|-------|----------|-------------------|
|
||||
| 1 | Cluster initializes from scratch | Step 1 | `assert_orca_version` |
|
||||
| 2 | Proxmox host onboards via SSH | Step 2 | `assert_proxmox_onboarded` |
|
||||
| 3 | Ubuntu worker onboards via `--type linux` | Step 3 | `assert_linux_worker_onboarded` |
|
||||
| 4 | Node list shows all nodes | Step 4 | `assert_cluster_initialized` |
|
||||
| 5 | Capacity is set on remote nodes | Step 5 | `assert_capacity_set` |
|
||||
| 6 | Namespace created | Step 6 | `assert_namespace_created` |
|
||||
| 7 | Full stack deploys to remote nodes | Step 7 | `assert_full_stack_running` |
|
||||
| 8 | Scheduler deploys to remote (not local) | Step 7 | `assert_job_deploys_to_remote` |
|
||||
| 9 | Traefik routes present | Step 9 | `assert_traefik_routes` |
|
||||
| 10 | Job migrates between hosts | Step 10 | `assert_migrate_worked` |
|
||||
| 11 | Logs aggregate from multiple nodes | Step 11 | `assert_logs_aggregate` |
|
||||
| 12 | ACL grant/check works | Step 12 | `assert_acl_enforced` |
|
||||
| 13 | ACL deny-by-default | Step 12 | `assert_acl_deny_default` |
|
||||
| 14 | acl.json mode 0600 | Step 12 | `assert_acl_file_mode` |
|
||||
| 15 | Seal/unseal round-trip | Step 13 | `assert_seal_unseal_roundtrip` |
|
||||
| 16 | Audit chain intact | Step 14 | `assert_audit_chain_intact` |
|
||||
| 17 | Doctor modes passes | Step 15 | `assert_doctor_modes` |
|
||||
| 18 | OIDC health check | Step 16 | `assert_oidc_health` |
|
||||
| 19 | Backup works | Step 17 | `assert_backup_restore_dryrun` |
|
||||
| 20 | Drift visible | Step 18 | `assert_drift_visible` |
|
||||
| 21 | Txn idempotent | Step 19 | `assert_txn_idempotent` |
|
||||
| 22 | Metrics expanded | Step 20 | `assert_metrics_expanded` |
|
||||
| 23 | Compat check passes | Step 21 | `assert_compat_check_passes` |
|
||||
| 24 | No `--password` in docs/examples | — | `assert_no_password_in_docs` |
|
||||
| 25 | Go toolchain current | — | `assert_go_toolchain_current` |
|
||||
| 26 | cli.md matches `orca --help` | — | `assert_cli_md_complete` |
|
||||
| 27 | pprof not on all interfaces | — | `assert_no_pprof_on_all_interfaces` |
|
||||
| 28 | WebAuthn registration requires auth | — | `assert_webauthn_reg_requires_auth` |
|
||||
| 29 | Audit chain survives concurrency | — | `assert_audit_chain_concurrent` |
|
||||
| 30 | Concurrent secrets no data loss | — | `assert_concurrent_secrets_no_loss` |
|
||||
| 31 | Cache invalidated after write | — | `assert_cache_invalidated_after_write` |
|
||||
| 32 | SQLite no lock under concurrency | — | `assert_sqlite_no_lock` |
|
||||
| 33 | No injection in logs --job | — | `assert_no_injection_in_logs` |
|
||||
| 34 | `--type linux` exists as subcommand | Step 3 | `assert_type_linux_available` |
|
||||
| 35 | `orca status` deprecated | — | `assert_status_deprecated` |
|
||||
|
||||
## Signoff procedure
|
||||
|
||||
1. Run all steps above on the 3-host cluster
|
||||
2. Run `scripts/uat-signoff.sh` on the lead
|
||||
3. Paste the output back to the CI agent
|
||||
4. The CI agent verifies `35/35 PASS` and cuts `v1.0.0`
|
||||
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### ORCA_HOME not set
|
||||
All orca commands use `$ORCA_HOME` (default `~/.orca`). If commands fail
|
||||
with "no such file or directory", verify:
|
||||
```sh
|
||||
echo $ORCA_HOME
|
||||
ls $ORCA_HOME/orca.db $ORCA_HOME/orca_ssh_key $ORCA_HOME/known_hosts $ORCA_HOME/cluster/master.key
|
||||
```
|
||||
|
||||
### known_hosts missing
|
||||
If SSH operations fail with "open .../known_hosts: no such file", the
|
||||
known_hosts file was not created during `orca init`. Fix:
|
||||
```sh
|
||||
touch $ORCA_HOME/known_hosts
|
||||
chmod 600 $ORCA_HOME/known_hosts
|
||||
```
|
||||
|
||||
### Traefik not running
|
||||
If Traefik routes are not deployed, verify Traefik is running:
|
||||
```sh
|
||||
systemctl status orca-traefik
|
||||
ls /etc/traefik/dynamic/
|
||||
```
|
||||
If not installed, `orca init` should have installed it. Re-run `orca init`
|
||||
or install manually from https://github.com/traefik/traefik/releases.
|
||||
|
||||
### SSH connection refused
|
||||
If the orca SSH key is not pre-staged on the remote host:
|
||||
```sh
|
||||
ssh-copy-id -i ~/.orca/orca_ssh_key.pub root@<host>
|
||||
```
|
||||
|
||||
### Job deployed but not visible in `job list`
|
||||
The remote dispatch path now inserts a DB record (v0.12.16). If you
|
||||
still don't see it, check:
|
||||
```sh
|
||||
orca job list --json
|
||||
```
|
||||
Look for the `"node"` field — it shows which node the job deployed to.
|
||||
|
||||
### Proxmox: process runtime rejected
|
||||
Proxmox nodes require `one_of: pve-ct` or `one_of: pve-vm` in the
|
||||
jobspec. `one_of: process` (systemd) is for Linux/Ubuntu workers only.
|
||||
@@ -1,97 +0,0 @@
|
||||
# WebAuthn / Passkeys (v0.13)
|
||||
|
||||
## Overview
|
||||
|
||||
The bundled Dex uses a custom WebAuthn connector (`orca-webauthn-connector`,
|
||||
REQ-148) for password-free authentication. Passkeys are public-key
|
||||
credentials — the private key never leaves the authenticator (TPM /
|
||||
security key / phone Secure Enclave). This directly satisfies R-021
|
||||
(no Orca-issued credentials): the authenticator proves possession of
|
||||
the private key without ever exposing it.
|
||||
|
||||
The WebAuthn connector ships as part of the v0.12 milestone (P05) and
|
||||
is operationally wired in v0.13 (P04: registration requires auth; P06:
|
||||
real Dex deployment).
|
||||
|
||||
## Registration
|
||||
|
||||
```bash
|
||||
orca auth register [--no-browser]
|
||||
```
|
||||
|
||||
Opens the browser to the Dex WebAuthn registration page at
|
||||
`https://<cluster>/orca/webauthn/register`. The operator authenticates
|
||||
via an existing session or admin bootstrap token, then performs the
|
||||
WebAuthn ceremony (biometric or security key). After the ceremony,
|
||||
Dex maps the credential ID to an OIDC `sub`.
|
||||
|
||||
- **`--no-browser`**: print the registration URL instead of opening a
|
||||
browser (useful for headless operators or remote SSH sessions — copy
|
||||
the URL into a local browser).
|
||||
|
||||
Credentials are stored at `ClusterDir()/webauthn-credentials.db`
|
||||
(mode 0600, public keys only — private keys never leave the
|
||||
authenticator and are never stored by orca).
|
||||
|
||||
**Example**:
|
||||
```bash
|
||||
# Interactive (opens browser)
|
||||
orca auth register
|
||||
|
||||
# Headless / remote SSH (print URL)
|
||||
orca auth register --no-browser
|
||||
# → https://orca.local/orca/webauthn/register
|
||||
```
|
||||
|
||||
## RP ID
|
||||
|
||||
The relying-party ID is the cluster's Traefik-served domain, set via
|
||||
`--rp-id` on `orca auth init-idp` (C-38). The RP ID **must** match the
|
||||
cluster's Traefik domain — WebAuthn enforces that the RP ID is a
|
||||
registrable domain suffix of the current origin.
|
||||
|
||||
HTTPS secure context is provided by Traefik (step-ca cert, R-017).
|
||||
WebAuthn requires a secure context (HTTPS or localhost); the step-ca
|
||||
cert behind Traefik satisfies this.
|
||||
|
||||
## Bootstrap Sequence
|
||||
|
||||
1. `orca init` bootstraps the cluster CA (step-ca, mTLS-only).
|
||||
2. `orca auth init-idp --rp-id <cluster-domain>` deploys Dex behind
|
||||
Traefik (step-ca cert) with the WebAuthn connector configured.
|
||||
3. First operator authenticates via an existing session or admin
|
||||
bootstrap token, then registers a passkey:
|
||||
|
||||
```bash
|
||||
orca auth register
|
||||
```
|
||||
|
||||
4. Subsequent operators use WebAuthn login (`orca auth login` opens
|
||||
the browser to the Dex login page; the WebAuthn ceremony is one of
|
||||
the available upstreams).
|
||||
|
||||
## Health Check
|
||||
|
||||
```bash
|
||||
orca doctor oidc
|
||||
```
|
||||
|
||||
Verifies the bundled Dex OIDC provider is running and the
|
||||
`/.well-known/openid-configuration` endpoint responds. Run after
|
||||
`orca auth init-idp` or after a Dex config change.
|
||||
|
||||
## Security properties
|
||||
|
||||
- **No passwords**: WebAuthn is password-free. No password is ever
|
||||
sent to or stored by orca (R-021).
|
||||
- **Phishing-resistant**: the WebAuthn protocol cryptographically binds
|
||||
the ceremony to the RP ID, defeating credential phishing.
|
||||
- **Private key never leaves the authenticator**: orca stores only
|
||||
public keys.
|
||||
- **Secure context required**: HTTPS via step-ca / Traefik (C-38).
|
||||
|
||||
## See also
|
||||
|
||||
- [docs/oidc.md](oidc.md) — OIDC configuration (Dex quickstart, BYO IdP)
|
||||
- [docs/security-runbook.md](security-runbook.md) — security runbook
|
||||
- [docs/cli.md](cli.md#orca-auth) — `orca auth` CLI reference
|
||||
@@ -1,191 +0,0 @@
|
||||
# Full-Stack Example with Ingress
|
||||
|
||||
This directory contains a complete multi-service stack deployed with
|
||||
Orca, including Traefik ingress configuration. Each file is a valid
|
||||
Orca jobspec (`.md` frontmatter) that passes the v0.9 parser and schema
|
||||
validators.
|
||||
|
||||
> **Runnable out-of-the-box**: The `runtime.command` in each example
|
||||
> uses `/bin/sleep 3600` (for long-running services) or `/bin/echo`
|
||||
> (for one-shot jobs) so that `orca job run <file>.md` succeeds on any
|
||||
> Linux machine without installing any software. Each file has a
|
||||
> **Production substitution** note showing the real binary to use in a
|
||||
> deployment (e.g. `/usr/bin/httpd`,
|
||||
> `/usr/lib/postgresql/16/bin/postgres`).
|
||||
|
||||
## Stack overview
|
||||
|
||||
| File | Kind | Runtime | Ingress | Description |
|
||||
|------|------|---------|---------|-------------|
|
||||
| `web-app.md` | Service | process | Unix socket (default) | Frontend HTTP server, 3 replicas, rolling update |
|
||||
| `api.md` | Service | process | TCP `127.0.0.1:9090` (R-007 opt-in) | Backend API, 2 replicas, canary update |
|
||||
| `worker.md` | Job | process | none | One-shot batch worker with lifecycle hooks |
|
||||
| `log-shipper.md` | Service | process | Unix socket (metrics) | Log shipper on a dedicated node |
|
||||
| `postgres.md` | Service | process | Unix socket | Database with volume replication, blue-green update |
|
||||
|
||||
## Rendered artifacts
|
||||
|
||||
The `rendered/` directory shows what Orca generates on the target nodes
|
||||
when you submit these jobspecs:
|
||||
|
||||
| File | Description |
|
||||
|------|-------------|
|
||||
| `traefik-dynamic-web-app.yaml` | Traefik dynamic config for the web-app Service |
|
||||
| `traefik-dynamic-api.yaml` | Traefik dynamic config for the api Service (TCP bind) |
|
||||
| `systemd-web-app.service` | Systemd unit for the web-app alloc |
|
||||
| `systemd-api.service` | Systemd unit for the api alloc (with TCP bind marker) |
|
||||
| `systemd-log-shipper.service` | Systemd unit for the log-shipper alloc |
|
||||
|
||||
## Walkthrough
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Orca installed (`orca version` works)
|
||||
- 2+ Linux nodes reachable over SSH (for multi-node scheduling)
|
||||
- Traefik installed on the lead node (watches `/etc/traefik/dynamic/`)
|
||||
|
||||
### Step 1: Initialize the cluster
|
||||
|
||||
```bash
|
||||
# On the operator laptop
|
||||
orca init
|
||||
```
|
||||
|
||||
This creates `~/.orca/` (or `/root/.orca` with `--system`), bootstraps
|
||||
the CA, generates the server cert, auto-detects the OS, and registers
|
||||
a localhost node.
|
||||
|
||||
### Step 2: Join remote nodes
|
||||
|
||||
```bash
|
||||
# Join a Proxmox node (v0.9 canonical SSH-push path)
|
||||
orca node join --type proxmox --host 192.168.1.100 --ssh-user root
|
||||
|
||||
# Join a second node
|
||||
orca node join --type proxmox --host 192.168.1.101 --ssh-key ~/.ssh/orca_ed25519
|
||||
```
|
||||
|
||||
### Step 3: Declare node capacity
|
||||
|
||||
The CLI-side scheduler uses capacity declarations for bin-packing:
|
||||
|
||||
```bash
|
||||
orca node capacity set --cpu 4000 --memory 8192 --disk 100000 --node 192.168.1.100
|
||||
orca node capacity set --cpu 4000 --memory 8192 --disk 100000 --node 192.168.1.101
|
||||
```
|
||||
|
||||
### Step 4: Create a namespace
|
||||
|
||||
```bash
|
||||
orca ns create prod --parent _defaults
|
||||
```
|
||||
|
||||
This creates `~/.orca/prod/` with `db/`, `jobs/`, `alloc/`, and `ns.md`.
|
||||
|
||||
### Step 5: Submit the stack
|
||||
|
||||
```bash
|
||||
orca job run web-app.md
|
||||
orca job run api.md
|
||||
orca job run worker.md
|
||||
orca job run log-shipper.md
|
||||
orca job run postgres.md
|
||||
```
|
||||
|
||||
Each `orca job run` parses the `.md` jobspec, validates it against the
|
||||
schema, schedules it via the CLI-side bin-packing scheduler, and
|
||||
generates the systemd + Traefik artifacts on the target node via
|
||||
SSH-push.
|
||||
|
||||
### Step 6: Observe placements
|
||||
|
||||
```bash
|
||||
orca job list --watch
|
||||
|
||||
# Output:
|
||||
# ID NAME STATUS EXIT
|
||||
# abc-123... web-app running 0
|
||||
# def-456... api running 0
|
||||
# ghi-789... worker complete 0
|
||||
# jkl-012... log-shipper running 0
|
||||
# mno-345... postgres running 0
|
||||
```
|
||||
|
||||
### Step 7: Inspect rendered artifacts
|
||||
|
||||
After submission, the target nodes have:
|
||||
|
||||
```
|
||||
/etc/systemd/system/orca-v1-web-app.service # systemd unit
|
||||
/etc/systemd/system/orca-v1-api.service # systemd unit (TCP bind)
|
||||
/etc/traefik/dynamic/orca-web-app.yaml # Traefik dynamic config
|
||||
/etc/traefik/dynamic/orca-api.yaml # Traefik dynamic config
|
||||
/run/orca/alloc-web-app-0/port-http.sock # Unix socket (R-007 default)
|
||||
```
|
||||
|
||||
See the `rendered/` directory in this example for the exact file
|
||||
contents.
|
||||
|
||||
### Step 8: Verify ingress
|
||||
|
||||
Traefik watches `/etc/traefik/dynamic/` and atomically reloads when a
|
||||
file changes (write-tmp + rename, gate C-10). The web-app is reachable
|
||||
at `https://<cluster-domain>/web-app` and the API at
|
||||
`https://<cluster-domain>/api`.
|
||||
|
||||
Health checks (`/healthz` on each backend) ensure Traefik only routes
|
||||
to healthy instances.
|
||||
|
||||
### Step 9: Drain and rollback
|
||||
|
||||
To drain a service (stop traffic, keep the workload running):
|
||||
|
||||
```bash
|
||||
# Orca writes a Traefik config with weight:0 on every backend
|
||||
# (RenderDrain). Traefik stops sending traffic.
|
||||
```
|
||||
|
||||
To roll back, re-submit the normal jobspec — Orca writes the
|
||||
non-drained Traefik config and Traefik resumes routing.
|
||||
|
||||
## Ingress model
|
||||
|
||||
See [docs/ingress.md](../../docs/ingress.md) for the full Traefik
|
||||
ingress reference. Key points:
|
||||
|
||||
- `kind: Service` **implies** a Traefik route (D-175).
|
||||
- Default bind is a **Unix socket** at
|
||||
`/run/orca/alloc-<id>/port-<name>.sock` (R-007).
|
||||
- `service.bind: 127.0.0.1` opts in to **TCP** (loopback only).
|
||||
- One Traefik dynamic file per Service at
|
||||
`/etc/traefik/dynamic/orca-<name>.yaml`.
|
||||
- Atomic reload via write-tmp + rename (gate C-10).
|
||||
- Drain sets `weight: 0` per backend.
|
||||
|
||||
## Validation
|
||||
|
||||
All jobspecs in this directory are validated by a Go test:
|
||||
|
||||
```bash
|
||||
go test ./examples/full-stack/ -v -run TestExamplesValidate
|
||||
```
|
||||
|
||||
This test parses each `.md` file with `jobspec.ParseFile` and validates
|
||||
it against `schema.ValidatorFor(kind)` — ensuring every field used in
|
||||
the examples exists in the current `WorkloadSpec` struct and passes the
|
||||
per-kind validators (gate C-20).
|
||||
|
||||
## v0.11 forward
|
||||
|
||||
The following are not yet implemented in v0.9 and will land in v0.11:
|
||||
|
||||
- **DaemonSet `schedule:` block**: the parser does not yet populate the
|
||||
`schedule:` frontmatter block (v0.9 parser gap). The `log-shipper`
|
||||
example uses `kind: Service` with `count: 1` and a `node.role`
|
||||
constraint as a workaround.
|
||||
- **Secret resolution**: `env: { KEY: { from: "secret:..." } }` is
|
||||
parsed but not resolved to `EnvironmentFile=`/`LoadCredential=` until
|
||||
v0.11-P03.
|
||||
- **Transactional update execution**: the `update:` block's plan is
|
||||
computed but not executed transactionally until v0.11-P10.
|
||||
- **Socket activation**: real socket unit files land in v0.11-P08.
|
||||
@@ -1,46 +0,0 @@
|
||||
---
|
||||
kind: Service
|
||||
name: api
|
||||
count: 2
|
||||
runtime:
|
||||
one_of: process
|
||||
command: /bin/sleep 3600
|
||||
ports:
|
||||
- name: api
|
||||
port: 9090
|
||||
restart:
|
||||
mode: service
|
||||
attempts: 3
|
||||
delay: 5s
|
||||
update:
|
||||
strategy: canary
|
||||
canary: 1
|
||||
max_parallel: 1
|
||||
auto_promote: false
|
||||
min_healthy_time: 30s
|
||||
healthy_deadline: 5m
|
||||
service:
|
||||
name: api
|
||||
port: 9090
|
||||
bind: 127.0.0.1
|
||||
health:
|
||||
check_type: http
|
||||
interval: 10s
|
||||
timeout: 2s
|
||||
unhealthy_threshold: 3
|
||||
constraints:
|
||||
- node.role == "api"
|
||||
- node.cpus >= 2
|
||||
env:
|
||||
DB_HOST: postgres
|
||||
DB_PORT: "5432"
|
||||
LOG_LEVEL: info
|
||||
---
|
||||
# API Server
|
||||
|
||||
Backend API service binding to 127.0.0.1:9090 (TCP opt-in, R-007).
|
||||
Canary update strategy with manual promote. Two replicas with CPU
|
||||
constraint (>= 2 vCPUs) and API-role node selection.
|
||||
|
||||
> **Production substitution**: replace `runtime.command` with your
|
||||
> actual API binary, e.g. `/usr/bin/api-server --listen 127.0.0.1:9090`.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user