Compare commits
114 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 9c65833954 | |||
| 6f5705fe02 | |||
| b4a0ada87e | |||
| ced2182322 | |||
| da682f1017 | |||
| 3269e1cb1d | |||
| a6bd1385ab | |||
| b765cca0ed | |||
| bfe92661ec | |||
| c5ce851fc7 | |||
| 10bcb49514 | |||
| 50c4e910ed | |||
| 0d5ff663b4 | |||
| 7f81042abd | |||
| d7dc2d2aad | |||
| a627d0ee6d | |||
| 827f215115 | |||
| a81bbb2bcf | |||
| 2cbfb5d561 | |||
| 20523ac045 | |||
| 1fb82f09b2 | |||
| 691463ff74 | |||
| c726a6a9e2 | |||
| 5429da1f87 | |||
| 7177ac7538 | |||
| dfacfea377 | |||
| 5d115fc4b7 | |||
| ce2441f312 | |||
| cf0df0f157 | |||
| da1f93ea77 | |||
| 8d1cdceb5c | |||
| cc57ae4c23 | |||
| c5048822e5 | |||
| 9a28dc907b | |||
| 97b88a703c | |||
| 020aa01623 | |||
| 03f3585f16 | |||
| 635e07e7a5 | |||
| 5cbe3020d3 | |||
| 5f92196625 | |||
| f530c9a3f7 | |||
| c8cf2e41e5 | |||
| 41bcf0a6bf | |||
| f61ef2aa9e | |||
| 2e6436608f | |||
| 33c2b4a78b | |||
| 734c9fa0fa | |||
| cc53c1a3e4 | |||
| 249518c807 | |||
| b6d4db1a96 | |||
| 2f7b2da05a | |||
| 9b984ad720 | |||
| 715fcb54b3 | |||
| e45611b416 | |||
| 5b99bbd2e8 | |||
| a70eb0d83d | |||
| 6f7a5122cc | |||
| 1cc965e23b | |||
| a412f832fd | |||
| a20cdb294c | |||
| 4c2e59cf3f | |||
| 8839781539 | |||
| f0b9910bf1 | |||
| 94711e05f1 | |||
| e9686f4ab0 | |||
| 76967b5145 | |||
| 2c26a6d54f | |||
| 4e019ab51e | |||
| 289e5cf6e1 | |||
| cfec794bb7 | |||
| eadd28fac0 | |||
| 3b6241e5c9 | |||
| 152a7fc375 | |||
| 7007aa6179 | |||
| 0ca19696b1 | |||
| 712f43613b | |||
| e0ce12befb | |||
| fe8851b161 | |||
| 99480f8f84 | |||
| f04d043da3 | |||
| 00e3cf5ce8 | |||
| c51eba5e84 | |||
| 7b2f6719bb | |||
| 1bbd53536d | |||
| 28192a7fa4 | |||
| 9991e3d561 | |||
| 0e1f7f97b3 | |||
| 675feabf0c | |||
| 3a76a32964 | |||
| 872ffcaf25 | |||
| 2c53ad6213 | |||
| c3819dde12 | |||
| fb85898569 | |||
| c10779873b | |||
| ea00158fa5 | |||
| ae6eb5a27b | |||
| 19542dd8c9 | |||
| 436641782c | |||
| 075d2f6459 | |||
| e92b18197c | |||
| d379d19deb | |||
| 60b0357eb6 | |||
| af2fa59172 | |||
| 667f20a7b3 | |||
| fef03c5b56 | |||
| 7bb31d4c09 | |||
| 7b5193674e | |||
| f6de82d712 | |||
| 437aab39b4 | |||
| e5d2711d71 | |||
| dd81eedcd9 | |||
| fc94326b0e | |||
| 40b5e781ce | |||
| 7315916fbc |
@@ -79,6 +79,8 @@ and a **dispatcher** for multi-node job execution.
|
||||
|
||||
### 2. Daemon Layer (`internal/daemon`)
|
||||
|
||||
> **⚠️ DEPRECATED in v0.9**: This section describes the v0.8 architecture, superseded by the v0.9 re-architecture. See the "v0.9 Architecture" section at the bottom of this file and `.ciagent/PRD_v0.9.md`.
|
||||
|
||||
- **Server**: `net/http` with `http.ServeMux` (no external router)
|
||||
- **TLS (P01)**: `crypto/tls` with `MinVersion=tls.VersionTLS13` and
|
||||
AEAD cipher allowlist
|
||||
@@ -93,6 +95,8 @@ and a **dispatcher** for multi-node job execution.
|
||||
|
||||
### 3. Transport Layer (`internal/transport`, NEW in P01/P02)
|
||||
|
||||
> **⚠️ DEPRECATED in v0.9**: This section describes the v0.8 architecture, superseded by the v0.9 re-architecture. See the "v0.9 Architecture" section at the bottom of this file and `.ciagent/PRD_v0.9.md`.
|
||||
|
||||
- **Client**: `http.Client` with `http.Transport.TLSClientConfig` populated
|
||||
from `internal/security.NewClientTLSConfig`
|
||||
- **Server**: `http.Server.TLSConfig` populated from
|
||||
@@ -108,6 +112,8 @@ and a **dispatcher** for multi-node job execution.
|
||||
|
||||
### 4. Core Engine (`internal/engine`)
|
||||
|
||||
> **⚠️ DEPRECATED in v0.9**: This section describes the v0.8 architecture, superseded by the v0.9 re-architecture. The Dispatcher and PeerRegistry peer-dispatch path is replaced by a CLI-side scheduler + SSH-push (R-001). See the "v0.9 Architecture" section at the bottom of this file and `.ciagent/PRD_v0.9.md`.
|
||||
|
||||
- **Node Registry**: In-memory map of node IDs → metadata, persisted to SQLite
|
||||
(CPU/memory capacity, available slots, last-seen)
|
||||
- **Task Executor**: `os/exec.CommandContext` with `WaitDelay` (Go 1.25+) for
|
||||
@@ -423,6 +429,8 @@ as a function that takes a `yield func(Job) bool` callback.
|
||||
|
||||
## Security Architecture
|
||||
|
||||
> **⚠️ DEPRECATED in v0.9**: This section describes the v0.8 internal-CA architecture, superseded by the v0.9 re-architecture (step-ca, D-101/REQ-076). See the "v0.9 Architecture" section at the bottom of this file and `.ciagent/PRD_v0.9.md`.
|
||||
|
||||
### Authentication
|
||||
- **v0.1**: mTLS for all API endpoints (self-signed CA)
|
||||
- **v0.2 P01**: Internal CA with CSR join (see Flow 1 + 2)
|
||||
@@ -449,6 +457,8 @@ as a function that takes a `yield func(Job) bool` callback.
|
||||
|
||||
## Key Architectural Decisions (v0.1 + v0.2)
|
||||
|
||||
> **⚠️ DEPRECATED in v0.9**: AD-007 (HCL canonical for jobspecs) below is superseded by R-013/R-014 (Markdown with YAML frontmatter canonical; HCL legacy). See the "v0.9 Architecture" section at the bottom of this file and `.ciagent/PRD_v0.9.md`.
|
||||
|
||||
| ID | Decision | Rationale |
|
||||
|----|----------|-----------|
|
||||
| AD-001 | Single binary with subcommands | Simpler distribution, aligns with simplicity pillar |
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
# Bash Capability Map — v0.9 (grill C-18)
|
||||
|
||||
Maps every capability in the shipped `internal/transport` package to its
|
||||
bash-side equivalent (or accepted drop with recorded rationale) in the v0.9
|
||||
re-architecture. The grill (C-18) required this mapping so capability
|
||||
regressions are visible, not silent.
|
||||
|
||||
| Shipped capability (internal/transport) | Bash-side equivalent | Status | Rationale |
|
||||
|---|---|---|---|
|
||||
| Retry with exponential backoff (`retry.go`: 100ms start, ×2, cap 5s, max 5 attempts) | `orca-retry()` function in `scripts/lib/orca-retry.sh` (to be written in v0.9-P01 SSH-push transport phase, REQ-073) | **planned** (v0.9-P01) | SSH dial/exec failures need the same bounded retry. The pattern is transport-agnostic; the Go retry logic is extracted into the new `internal/sshpush/` package and a bash-side helper mirrors it for the lead-applier scripts. |
|
||||
| Idempotency keys (`idempotency.go`: in-memory `sync.Map` of keys, `X-Orca-Idempotency-Key` header) | Content-addressed filenames — skip SCP if the target hash already exists on the peer | **planned** (v0.9-P01) | SSH-push doesn't have HTTP headers; idempotency is achieved by content-addressing the rendered file (`<hash>.unit`) and skipping if the peer already has it. The bash applier checks `test -f /run/orca/<hash>` before applying. |
|
||||
| Structured mTLS failure logging (`handshake_log.go`: slog JSON per mTLS failure) | `orca_log_error` via `scripts/lib/orca-log.sh` (C-17, shipped in this phase P00) | **dropped (mTLS removed by R-001)** | The v0.9 re-architecture removes mTLS daemon-to-daemon transport entirely (R-001). SSH failures are logged via the new `orca_log_*` functions which emit the same slog-compatible JSON field set (ts, level, actor, action, resource, result, error) to syslog. The mTLS-specific handshake-log fields (cipher suite, TLS version, cert SAN) have no SSH equivalent and are dropped — the SSH error message is captured in the `error` field instead. |
|
||||
| TLS 1.3 + AEAD cipher allowlist (`mtls.go`: MinVersion=tls.VersionTLS13, CipherSuites limited) | SSH's own cipher config (`/etc/ssh/sshd_config` `Ciphers`, `MACs`, `KexAlgorithms`) managed by the operator | **dropped (transport replaced)** | R-001 replaces mTLS HTTP with SSH. SSH's transport security is governed by the peer's sshd_config, not the orca binary. The CLI's SSH client (`golang.org/x/crypto/ssh`, already a dep) uses Go's default modern SSH cipher set. The PRD does not require orca to manage sshd_config cipher policy in v0.9. |
|
||||
| mTLS client/server handshake (`mtls.go`: `MTLSClient`, daemon-side `SubmitHandler`) | `ssh.Dial` + `ssh.PublicKeys` auth (CLI-side `internal/sshpush/`, REQ-073) | **replaced** (v0.9-P01) | The daemon-to-daemon mTLS handshake is replaced by CLI-to-server SSH. The CLI holds an Ed25519 key (`cluster/orca_ssh_key`, D-037) and authenticates to each peer's sshd. TOFU host-key handling (`proxmox.TOFUHostKeyCallback`, v0.8 REQ-058) is reused for all peers, not just Proxmox. |
|
||||
|
||||
## Net-new capabilities in v0.9 (no shipped equivalent)
|
||||
|
||||
| Net-new capability | Bash-side | Status |
|
||||
|---|---|---|
|
||||
| Transaction bundle apply (R-010, REQ-075) | `orca-apply-render.sh` (v0.10-P10) | planned |
|
||||
| Drift detection (R-010) | `orca-drift.sh` (v0.10-P10) | planned |
|
||||
| Per-node state collection | `orca-collect.sh` (v0.10-P09) | planned |
|
||||
| Lead aggregation | `orca-aggregate.sh` (v0.10-P09) | planned |
|
||||
| Credential cleanup (5-min shred) | `orca-cleanup-credentials.sh` (v0.10) | planned |
|
||||
| Render-bundle validation (C-16) | `orca-verify-render.sh` (shipped this phase P00) | ✅ shipped |
|
||||
| Structured logging (C-17) | `orca-log.sh` (shipped this phase P00) | ✅ shipped |
|
||||
|
||||
## Review cadence
|
||||
|
||||
This map is reviewed at each phase that introduces or modifies a bash
|
||||
script. The security-engineer persona reviews the SSH trust surface; the
|
||||
devops-engineer persona reviews the bash tooling gate (C-15..C-18).
|
||||
@@ -0,0 +1,179 @@
|
||||
# C-02 — Syncthing Feasibility Spike (v0.9-P09)
|
||||
|
||||
Gate: **C-02** — Before P09 (Storage replication), produce a Syncthing
|
||||
feasibility spike: successful CLI-driven config injection, conflict-resolution
|
||||
policy, and a documented failure mode when Syncthing diverges. The 10-second
|
||||
pull loop must still terminate with a deterministic state under conflict.
|
||||
|
||||
Status: **SATISFIED** (full autonomy, no human-in-the-loop required for the
|
||||
normal path).
|
||||
|
||||
Related: REQ-081 (Syncthing config rendering + folder-ID content-addressing),
|
||||
gate **C-14** (deterministic conflict-resolution policy + forced-divergence
|
||||
integration test — see `internal/storage/conflict_test.go`).
|
||||
|
||||
## 1. Config injection
|
||||
|
||||
Syncthing uses an XML config file (`config.xml`). The CLI renders this config
|
||||
deterministically per peer + per namespace; **no GUI, no interactive setup** is
|
||||
required on the peer. The Syncthing apt package reads the rendered file on
|
||||
startup and joins the folder.
|
||||
|
||||
### Structure (rendered by `internal/storage.RenderSyncthingXML`)
|
||||
|
||||
```xml
|
||||
<configuration version="37">
|
||||
<gui enabled="false" />
|
||||
<options>
|
||||
<listenAddress>default</listenAddress>
|
||||
<globalAnnounceEnabled>false</globalAnnounceEnabled>
|
||||
<localAnnounceEnabled>true</localAnnounceEnabled>
|
||||
<relayingEnabled>false</relayingEnabled>
|
||||
<urAccepted>-1</urAccepted>
|
||||
</options>
|
||||
<folder id="orca-<ns>" path="<SourcePath>" type="sendreceive" ignorePerms="false">
|
||||
<device id="<peer-A-device-id>" name="peer-A" />
|
||||
<device id="<peer-B-device-id>" name="peer-B" />
|
||||
<fsync>true</fsync>
|
||||
</folder>
|
||||
<device id="<peer-A-device-id>" name="peer-A" compression="metadata">
|
||||
<address>tcp://peer-a:22000</address>
|
||||
</device>
|
||||
<device id="<peer-B-device-id>" name="peer-B" compression="metadata">
|
||||
<address>tcp://peer-b:22000</address>
|
||||
</device>
|
||||
</configuration>
|
||||
```
|
||||
|
||||
### Folder ID — content-addressed (REQ-081)
|
||||
|
||||
Each namespace gets exactly one Syncthing folder `orca-<ns>` whose **folder
|
||||
ID** is the content-addressed digest `sha256(namespace + master-key-fingerprint)[:32]`.
|
||||
Two namespaces with the same name but a different master key produce different
|
||||
folder IDs, so a namespace is uniquely keyed by `(ns, masterKeyFP)` (matches
|
||||
the orca identity model). See `internal/storage.FolderID`.
|
||||
|
||||
### Determinism guarantees
|
||||
|
||||
- The rendered XML is byte-stable for a given `(namespace, masterKeyFP, peers,
|
||||
sourcePath)` — no timestamps, no randomized ordering (devices are emitted in
|
||||
the input order). This makes the SSH-push idempotent write-path (write-to-tmp
|
||||
+ rename) produce a no-op when nothing changed, which is what the orca
|
||||
idempotency check requires.
|
||||
- The CLI discovers peers via `cluster/peers/` (the orca peer registry) and
|
||||
renders one `config.xml` per peer. Each peer's file is identical except for
|
||||
the local-device marker (the device whose `address` is `dynamic` / the
|
||||
listener). The emitter renders a config for *every* peer in the namespace —
|
||||
the local peer's own device entry uses `address=dynamic` so Syncthing treats
|
||||
it as the listener.
|
||||
|
||||
### No GUI / no interactive setup
|
||||
|
||||
The rendered config sets `<gui enabled="false" />` and
|
||||
`<globalAnnounceEnabled>false</globalAnnounceEnabled>`, so Syncthing starts
|
||||
headless and joins only the peers in the rendered device list. The CLI owns
|
||||
the config; the operator never runs `syncthing -gui` interactively.
|
||||
|
||||
## 2. Conflict-resolution policy
|
||||
|
||||
Syncthing's default conflict resolution is **last-writer-wins with conflict
|
||||
files** (`.sync-conflict-<timestamp>-<peer>.<ext>`). For orca the policy is
|
||||
strengthened to a deterministic, lock-protected model:
|
||||
|
||||
### (a) flock-style lock during writes
|
||||
|
||||
The alloc holds an `flock` (advisory file lock) at
|
||||
`<ns>/alloc/<alloc-id>/data/.lock` for the duration of every write to the
|
||||
replicated volume. Only the alloc holding the lock writes; the other peers
|
||||
sync read-only. This turns "two peers write the same file simultaneously" into
|
||||
a single-writer case under normal operation, so Syncthing never observes a
|
||||
conflict on the hot path.
|
||||
|
||||
### (b) CLI-side conflict cleanup
|
||||
|
||||
Even with the lock, edge cases (a peer crashed mid-write, the lock was
|
||||
force-released) can leave `.sync-conflict-*` files. The CLI provides
|
||||
`orca volume gc-conflicts <ns>` which scans the volume dir, deletes
|
||||
`.sync-conflict-*` files, and logs each deletion. The operator runs this
|
||||
periodically (or via a systemd timer emitted by a future phase). The cleanup
|
||||
is idempotent — re-running on a clean tree is a no-op.
|
||||
|
||||
### (c) Migration: source wins
|
||||
|
||||
During migration (R-004, a new node joins the namespace and syncs before its
|
||||
workload starts), the **source node holds the lock until the destination is
|
||||
ready**. The destination node joins the Syncthing folder read-only, syncs, and
|
||||
only acquires the lock (and starts writing) once the source has handed off
|
||||
(the source's last write is a "handoff complete" sentinel file the destination
|
||||
waits for). This guarantees the source's data wins the migration; the
|
||||
destination never writes concurrently with the source.
|
||||
|
||||
## 3. Deterministic failure mode (divergence)
|
||||
|
||||
If Syncthing diverges — i.e. two peers wrote to the same file **without** the
|
||||
lock (the lock was bypassed, e.g. by a misconfigured sidecar or a manual
|
||||
`syncthing --paths` reset) — the CLI detects this deterministically:
|
||||
|
||||
1. **Detection** — `internal/storage.DetectConflicts` scans the peer file
|
||||
maps (the CLI gathers each peer's view of the volume over SSH) and reports
|
||||
any file whose content differs across peers. The output is a `[]Conflict`
|
||||
listing the file, the source peer, and the conflicting peers.
|
||||
2. **Resolution** — `internal/storage.ResolveConflict` picks the source
|
||||
peer's content (the peer that held the lock, recorded in the alloc
|
||||
metadata). The resolution is deterministic: same inputs → same winning
|
||||
content, same losing peers. No timestamps, no peer-id tie-breaks, no
|
||||
random selection.
|
||||
3. **Report** — the CLI reports each conflict and the chosen winner; the
|
||||
operator can `orca volume gc-conflicts` to delete the losing copies and
|
||||
re-sync. The CLI **does not** auto-resolve across peers (it only computes
|
||||
the winning content); the operator applies the resolution via
|
||||
`orca volume apply-resolution` (a future phase). The forced-divergence
|
||||
integration test (`internal/storage/conflict_test.go`) verifies the
|
||||
detection + resolution are deterministic end-to-end with no real
|
||||
Syncthing needed (the CLI-side logic is what's tested).
|
||||
|
||||
### Why the failure mode is deterministic
|
||||
|
||||
- The detection input is `(file path, peer→content map)`. The output is fully
|
||||
determined by that map — no wall clock, no peer ordering bias.
|
||||
- The resolution input is `(conflict, sourcePeer)`. The winner is the
|
||||
sourcePeer's content. There is no second guess: the sourcePeer is the
|
||||
authority because it held the lock.
|
||||
- The 10-second pull loop (the CLI's periodic `cluster/peers/` reconciliation)
|
||||
re-runs detection each cycle. Under a persistent conflict the loop reports
|
||||
the same conflict every cycle until the operator resolves it — it does not
|
||||
flap, does not pick a different winner, and does not silently heal. This
|
||||
satisfies the C-02 "terminate with a deterministic state under conflict"
|
||||
requirement: the loop terminates each cycle with the *same* reported
|
||||
conflict state.
|
||||
|
||||
## 4. Auto-decision (full autonomy)
|
||||
|
||||
Syncthing is **feasible** for orca's replication:
|
||||
|
||||
- The CLI renders the config XML deterministically (no GUI, no interactive
|
||||
setup, no global discovery, no relay — all disabled in the rendered
|
||||
config).
|
||||
- The flock prevents conflicts on the hot path (single writer at a time).
|
||||
- The conflict-cleanup handles edge cases (`.sync-conflict-*` files).
|
||||
- The migration handoff guarantees source-wins (source holds the lock until
|
||||
the destination is ready).
|
||||
- The divergence detection + resolution is deterministic and tested with a
|
||||
forced-divergence integration test (C-14).
|
||||
|
||||
**C-02 SATISFIED.**
|
||||
|
||||
## 5. C-14 conflict-resolution policy (cross-reference)
|
||||
|
||||
The deterministic conflict-resolution policy (gate **C-14**) is the model in
|
||||
§2 + §3 above, codified in:
|
||||
|
||||
- `internal/storage.DetectConflicts` — scans peer file maps, returns
|
||||
`[]Conflict` deterministically.
|
||||
- `internal/storage.ResolveConflict` — picks the source peer's content.
|
||||
- `internal/storage/conflict_test.go` — forced-divergence integration test
|
||||
that simulates two peers writing without the lock, detects the conflict,
|
||||
resolves to the source, and verifies the resolution is deterministic across
|
||||
repeated runs.
|
||||
|
||||
**C-14 SATISFIED.**
|
||||
@@ -0,0 +1,109 @@
|
||||
# CA Migration Spec — v0.8 Internal CA → v0.9 step-ca (grill C-07)
|
||||
|
||||
**Status**: spec (must be implemented in v0.10-P14a, REQ-066)
|
||||
**Gate**: C-07 — blocks v0.10-P14a until this spec is reviewed and a dry-run passes on a test cluster
|
||||
|
||||
## Problem
|
||||
|
||||
The v0.8 internal Go CA (`internal/security/ca.go`) issues RSA-3072 CA
|
||||
certs (10-year validity) and ECDSA P-256 server certs (90-day). The CA
|
||||
material lives at `~/.orca/ca.crt` and `~/.orca/ca.key` (flat layout, D-011).
|
||||
The v0.9 re-architecture reverses AD-010 and replaces the internal CA with
|
||||
step-ca (D-101, REQ-076). Existing v0.8 deployments have an internal CA
|
||||
root + issued server certs that must be migrated without invalidating
|
||||
trust across the cluster.
|
||||
|
||||
## Migration options (decision required before v0.10-P14a implementation)
|
||||
|
||||
### Option A — Preserve trust root (RECOMMENDED)
|
||||
|
||||
Import the existing `ca.key` into step-ca as the root CA key. The cluster's
|
||||
trust fingerprint stays unchanged; existing server certs continue to
|
||||
validate until their natural expiry; new SVIDs are minted by step-ca using
|
||||
the same root.
|
||||
|
||||
```bash
|
||||
orca upgrade --to-v1.0 --import-ca
|
||||
# reads ~/.orca/ca.key → step ca init --deployment-type standalone \
|
||||
# --remote-management --key $(cat ~/.orca/ca.key)
|
||||
# issues new SVIDs from step-ca for all existing workloads
|
||||
```
|
||||
|
||||
**Pros**: zero trust breakage; existing server certs keep working; minimal
|
||||
operator disruption.
|
||||
**Cons**: requires step-ca to accept an imported RSA-3072 key (step-ca
|
||||
supports imported keys via `--key` flag; verify in the spike).
|
||||
**Post-migration**: old `internal/security/ca.go` and `csr.go` are deleted
|
||||
(v0.10-P14); the `cert_repo` SQLite table (0004) is dropped (step-ca
|
||||
manages cert state).
|
||||
|
||||
### Option B — Forced re-bootstrap
|
||||
|
||||
Document that v0.8 certs are invalidated; every cluster re-bootstraps under
|
||||
step-ca with a new root. Existing workloads are re-enrolled.
|
||||
|
||||
**Pros**: clean slate; no legacy RSA root.
|
||||
**Cons**: trust breakage — every peer's `known_hosts` + CA cert must be
|
||||
rotated; running workloads lose mTLS until re-enrolled; higher operator
|
||||
disruption.
|
||||
**Use case**: only if Option A is technically infeasible (step-ca rejects
|
||||
the v0.8 key format).
|
||||
|
||||
## Pre-flight checks (must pass before migration)
|
||||
|
||||
1. `orca doctor` reports zero FAILs on the v0.8 cluster
|
||||
2. All peers reachable via SSH
|
||||
3. No in-flight transactions (the migration is stop-the-world for the CA)
|
||||
4. Snapshot taken (`orca backup --include-master-key`)
|
||||
5. step-ca installed on the lead via `apt-get install step-ca`
|
||||
6. `step ca init` dry-run succeeds with the imported key
|
||||
|
||||
## Migration steps (Option A)
|
||||
|
||||
1. SSH to the lead; install step-ca via apt
|
||||
2. Run `step ca init --deployment-type standalone --remote-management \
|
||||
--key <v0.8-ca-key-path> --provisioner orca-admin`
|
||||
3. Move the root cert: `cp ~/.orca/ca.crt $ORCA_HOME/cluster/ca.crt`
|
||||
4. Issue new SVIDs for every registered workload (via `step ca token` +
|
||||
`step ca certificate` — the CLI mints the provisioner token using
|
||||
`cluster/master.key`-derived material)
|
||||
5. Deploy the new SVIDs to peers via SSH-push (the v0.9 SSH-push transport)
|
||||
6. Verify: `orca doctor` reports zero FAILs; CA fingerprint unchanged;
|
||||
all workload SVIDs valid
|
||||
7. Archive the old `internal/security/ca.go`/`csr.go` and `cert_repo` table
|
||||
|
||||
## Rollback
|
||||
|
||||
If any post-migration invariant fails:
|
||||
1. Restore the v0.8 snapshot via `orca upgrade --rollback <tarball>`
|
||||
2. Restart the v0.8 orca daemon on the lead
|
||||
3. Verify `orca doctor` passes on the v0.8 cluster
|
||||
|
||||
The v0.8 internal CA remains functional during the dual-write window
|
||||
(REQ-090); step-ca is additive until the migration completes.
|
||||
|
||||
## Post-migration invariants (must all pass)
|
||||
|
||||
- CA fingerprint unchanged (Option A)
|
||||
- Node count unchanged
|
||||
- Workload count unchanged
|
||||
- All SVIDs valid (mTLS handshake succeeds lead↔every peer)
|
||||
- `orca doctor` zero FAILs
|
||||
- No `internal/security/ca.go` or `cert_repo` references remain in code
|
||||
|
||||
## Decision required
|
||||
|
||||
This spec is gated by C-07. The decision (Option A vs B) must be made
|
||||
before v0.10-P14a implementation. Default: Option A (preserve trust root)
|
||||
unless the step-ca imported-key spike fails.
|
||||
|
||||
## Spike (must run before v0.10-P14a)
|
||||
|
||||
Run on a test cluster:
|
||||
1. Install step-ca on a clean Linux host
|
||||
2. Generate a v0.8-style RSA-3072 CA key via the v0.8 `internal/security` package
|
||||
3. Run `step ca init --key <v8-key>` and verify step-ca accepts it
|
||||
4. Mint a test SVID via `step ca token` + `step ca certificate`
|
||||
5. Verify the SVID validates against the imported root
|
||||
|
||||
If the spike fails, fall back to Option B (forced re-bootstrap) and document.
|
||||
@@ -1,11 +1,17 @@
|
||||
{
|
||||
"phase": 0,
|
||||
"stage": "grill",
|
||||
"milestone": "v0.9",
|
||||
"milestone_slug": "rearchitecture",
|
||||
"phase_role": "pre_execution",
|
||||
"phase": 15,
|
||||
"stage": "complete",
|
||||
"milestone": "v0.12",
|
||||
"milestone_slug": "security-hardening",
|
||||
"phase_role": "execution",
|
||||
"attempts": 0,
|
||||
"updated_at": "2026-08-05T02:20:00Z",
|
||||
"updated_at": "2026-08-07T11:25:00Z",
|
||||
"milestone_complete": false,
|
||||
"next_milestone": null
|
||||
}
|
||||
"previous_milestone": "v0.11",
|
||||
"wave": "D done (P13 step-ca tmp, P14 master key rotation, P15 file-mode audit). E next (P16 aggregate.sh, P17 install.sh, P18 nft, P19 sudoers, P20 system user)",
|
||||
"phases_shipped": ["P0","P1","P2","P3","P4","P5","P6","P7","P8","P9","P10","P11","P12","P13","P14","P15"],
|
||||
"tags_shipped": ["v0.11.0","v0.11.1","v0.11.2","v0.11.3","v0.11.4","v0.11.5","v0.11.6","v0.11.7","v0.11.8","v0.11.9","v0.11.10","v0.11.11","v0.11.12","v0.11.13","v0.11.14","v0.11.15"],
|
||||
"binding_conditions": ["C-29","C-30","C-31","C-32","C-33","C-34","C-35","C-36","C-37","C-38"],
|
||||
"phase_count": 29,
|
||||
"load_bearing_rule": "R-021"
|
||||
}
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
# CLARIFY v0.11: Production Hardening
|
||||
|
||||
**Status**: resolved (full autonomy, 2026-08-07). All 5 clarifications
|
||||
resolved with the operator's locked decisions (Q1=A, Q2=C, Q3=A,
|
||||
Q4=A, Q5=A) and the research-ingestion synthesis. No open questions
|
||||
remain for Phase 0.
|
||||
|
||||
## Resolved clarifications
|
||||
|
||||
### C1 — Ingress default binding (resolved)
|
||||
|
||||
**Question**: Is `127.0.0.1:8443` + nft the *shipped default*, or is the
|
||||
v0.8 behavior (`:443` on Traefik) still the default and hybrid is opt-in?
|
||||
|
||||
**Decision**: R-017 makes the hybrid the **default for fresh `orca init`**
|
||||
(new clusters). Existing v0.9/v0.10 clusters get an opt-in migration path
|
||||
via `orca upgrade` (REQ-115), which handles the Traefik binding cutover
|
||||
from `:443` to `127.0.0.1:8443`. This is a behavioral change for existing
|
||||
operators but it ships in a controlled migration phase (P14a), not as a
|
||||
surprise default flip.
|
||||
|
||||
**Affected REQs**: REQ-100 (Traefik binding), REQ-115 (`orca upgrade`).
|
||||
**Affected phase**: P14a (migration), P15.5 (new default).
|
||||
|
||||
### C2 — `orca upgrade` scope (resolved)
|
||||
|
||||
**Question**: Is `orca upgrade --to-vX` (a) a thin wrapper around
|
||||
`install.sh` + `orca restore` (binary upgrade only), or (b) a full
|
||||
cluster-rolling-upgrade orchestrator (drain → upgrade binary → restart →
|
||||
next node)?
|
||||
|
||||
**Decision**: **(a) thin wrapper for v0.11**. Full cluster-rolling-upgrade
|
||||
(b) defers to v1.x. The thin wrapper handles the R-017 binding cutover
|
||||
(REQ-115) for existing clusters. A full rolling-upgrade orchestrator is a
|
||||
v1.x concern (it requires P05 drain + P09 syncthing + P14c mixed-version
|
||||
tolerance to be production-tested first).
|
||||
|
||||
**Affected REQs**: REQ-115.
|
||||
**Affected phase**: P14a.
|
||||
|
||||
### C3 — `orca job migrate` semantics (resolved)
|
||||
|
||||
**Question**: Does `orca job migrate --to <node>` (a) drain+reschedule
|
||||
(uses P05 drain + P06 alloc history), or (b) live-migrate with storage
|
||||
replication (uses P09 syncthing, much harder)?
|
||||
|
||||
**Decision**: **(a) drain+reschedule for v0.11**. It composes existing
|
||||
P05/P06 work. Live-migrate with storage replication (b) is a v1.x concern
|
||||
(requires P09 syncthing replication to be production-tested + a
|
||||
storage-replication-aware scheduler).
|
||||
|
||||
**Affected REQs**: REQ-116.
|
||||
**Affected phase**: P05.
|
||||
|
||||
### C4 — Remediation cooldown refinement (resolved, design refinement)
|
||||
|
||||
**Question**: Doc 5's D-232 cooldown (5-min) should not apply on transient
|
||||
remediation *failures* (SSH down, render tree missing) — only on
|
||||
*successful* remediation. Else a 30s network blip blocks re-remediation
|
||||
for 5 min.
|
||||
|
||||
**Decision**: **Refine D-232**: cooldown applies only on *successful*
|
||||
remediation; transient failures (SSH down, render tree missing, applier
|
||||
non-zero exit) retry on the next aggregator tick (10s) without entering
|
||||
cooldown. This is baked into REQ-108 and the D-232 rationale in
|
||||
PROJECT.md.
|
||||
|
||||
**Affected REQs**: REQ-108.
|
||||
**Affected phase**: P10.
|
||||
|
||||
### C5 — `orca doctor mTLS` depth (resolved)
|
||||
|
||||
**Question**: Does `orca doctor mTLS` just verify the trust chain (CA →
|
||||
server cert → workload SVIDs exist + not expired), or does it also do a
|
||||
live mTLS handshake probe to each peer?
|
||||
|
||||
**Decision**: **Both**. Chain verification is cheap (local file reads +
|
||||
cert parsing); live probe reuses P01 (metrics endpoint) + P01.5 (SPIFFE
|
||||
spike) infrastructure. The doctor check reports both: chain integrity
|
||||
(static) + live handshake (dynamic). A failed live handshake with a valid
|
||||
chain indicates a network/config problem, not a cert problem.
|
||||
|
||||
**Affected REQs**: REQ-118.
|
||||
**Affected phase**: P15.5.
|
||||
|
||||
## Open questions
|
||||
|
||||
None. All 5 clarifications resolved. Phase 0 proceeds to RESEARCH.
|
||||
@@ -0,0 +1,162 @@
|
||||
# CLARIFY v0.12: Security Hardening (Zero-Trust Identity)
|
||||
|
||||
**Status**: resolved (full autonomy, 2026-08-07). All 10 clarifications
|
||||
resolved with the operator's locked decisions (D-238..D-247). No open
|
||||
questions remain for Phase 0. The `--ideate` flag was passed; the
|
||||
threat-model review drove the requirements.
|
||||
|
||||
## Resolved clarifications
|
||||
|
||||
### C1 — Milestone version (resolved)
|
||||
|
||||
**Question**: v0.11 is complete; the v0.11 PRD deferred the v1.0.0 tag
|
||||
for UAT sign-off. Is this security-hardening milestone v1.0 (the UAT
|
||||
gate) or a minor v0.12?
|
||||
|
||||
**Decision**: **v0.12 (minor, not v1.0).** The v1.0.0 production-ready
|
||||
tag stays deferred for post-v0.12 UAT, exactly as v0.11's PRD
|
||||
specified. v0.12 is a minor feature milestone. Per-phase tags run on
|
||||
the previous minor's patch line (v0.11.x): P0 -> `v0.11.0`, P01 ->
|
||||
`v0.11.1`, ..., final phase patch = `v0.11.29` = the v0.12 milestone
|
||||
release (no separate `v0.12.0` tag, per feature-milestone rule).
|
||||
|
||||
**Affected**: config.json milestone field, all tag computation.
|
||||
|
||||
### C2 — OIDC provider model (resolved)
|
||||
|
||||
**Question**: Bring-your-own IdP, bundled opinionated provider, or both?
|
||||
|
||||
**Decision**: **Bundled Dex by default, with BYO external IdP as a
|
||||
config override.** `orca auth init-idp` bootstraps a local Dex on the
|
||||
lead (systemd unit + config template + Traefik route). `oidc.issuer`
|
||||
in config can be repointed to an external IdP (Keycloak/Authentik/
|
||||
Google/etc.) anytime. Orca stays minimal (no bundled opinionated
|
||||
provider beyond Dex); Dex is the OIDC frontend, not a full IdP.
|
||||
|
||||
**Affected REQs**: REQ-144 (OIDC client + bundled Dex).
|
||||
|
||||
### C3 — Bundled Dex upstream authenticator (resolved)
|
||||
|
||||
**Question**: Dex needs an upstream identity source. "No passwords
|
||||
anywhere" rules out a local password store. What is the password-free
|
||||
upstream?
|
||||
|
||||
**Decision**: **WebAuthn (passkeys) connector.** Bundled Dex gets a
|
||||
custom `orca-webauthn-connector` (~300 LoC Go, `go-webauthn` library)
|
||||
that serves registration + login HTML/JS pages behind Traefik at
|
||||
`https://<cluster>/orca/webauthn/{register,login}`. The WebAuthn
|
||||
ceremony (biometric/security key) produces a public-key credential;
|
||||
Dex maps the credential ID to an OIDC `sub`. **Passkeys are public-key
|
||||
credentials -- the private key never leaves the authenticator -- so the
|
||||
"no passwords/secrets" invariant (R-021) holds.**
|
||||
|
||||
For BYO external IdP deployments, the operator's existing authenticator
|
||||
(WebAuthn, TOTP, LDAP, etc.) is used; Orca never sees the upstream
|
||||
credentials.
|
||||
|
||||
**Affected REQs**: REQ-148 (WebAuthn connector).
|
||||
**Affected phase**: P05 (new phase; wave B grows from 4 to 5 phases).
|
||||
|
||||
### C4 — Master key sealing (resolved)
|
||||
|
||||
**Question**: How is the secrets master key protected at rest, given
|
||||
"no passwords anywhere"?
|
||||
|
||||
**Decision**: **Seal to OIDC + Shamir 3-of-5 recovery.** The master
|
||||
key (32 random bytes) is encrypted (sealed) with a key derived from an
|
||||
OIDC token exchange at unseal time. `orca cluster unseal` (operator
|
||||
authenticates via OIDC -> token exchange -> unwrap master key into
|
||||
memory -> zeroed on shutdown). `orca cluster seal` for manual re-seal.
|
||||
The sealed blob is stored at `ClusterDir()/master.key.sealed` (0600).
|
||||
The raw master key never touches disk.
|
||||
|
||||
**Shamir recovery**: at seal time, 5 shards are printed and the
|
||||
operator stores them offline. If the IdP is permanently lost AND a
|
||||
quorum of 3 shards is unavailable, the cluster is unrecoverable by
|
||||
design (documented residual risk; no backdoor).
|
||||
|
||||
For the mTLS-only offline path (no OIDC), the seal key is derived from
|
||||
the cluster's own CA -- the operator holds the CA (a cert, not a
|
||||
password). The Shamir recovery path applies to the OIDC-sealed mode.
|
||||
|
||||
**Affected REQs**: REQ-147 (master key seal).
|
||||
**Affected phase**: P08.
|
||||
|
||||
### C5 — CLI browser flow (resolved)
|
||||
|
||||
**Question**: How does the CLI do the OIDC browser flow?
|
||||
|
||||
**Decision**: **OIDC authorization-code + PKCE + local loopback
|
||||
redirect.** `orca auth login` opens the default browser to the Dex
|
||||
WebAuthn endpoint. After the ceremony, Dex redirects to
|
||||
`127.0.0.1:<port>/callback` (local loopback, ephemeral port). The CLI
|
||||
exchanges the auth code for a short-lived ID token (1h) + refresh
|
||||
token. Headless/CI fallback: device-code flow (no browser needed).
|
||||
|
||||
**Affected REQs**: REQ-144, REQ-148.
|
||||
|
||||
### C6 — WebAuthn RP ID / secure context (resolved)
|
||||
|
||||
**Question**: WebAuthn requires a secure context (HTTPS). Where is the
|
||||
RP ID rooted?
|
||||
|
||||
**Decision**: **Traefik-served cluster domain (step-ca cert, R-017).**
|
||||
Traefik already provides HTTPS on `127.0.0.1:8443` (nft DNAT from
|
||||
`:443`). The RP ID is the cluster's Traefik-served domain, configurable
|
||||
via `orca auth init-idp --rp-id <domain>`. For localhost dev, the
|
||||
operator uses the bootstrapped step-ca cert (self-signed, but WebAuthn
|
||||
accepts it for non-registerable credentials in dev mode).
|
||||
|
||||
**Affected REQs**: REQ-148.
|
||||
|
||||
### C7 — Passkey storage (resolved)
|
||||
|
||||
**Question**: Where are WebAuthn credentials stored?
|
||||
|
||||
**Decision**: **SQLite at `ClusterDir()/webauthn-credentials.db`
|
||||
(0600). Public keys only.** The DB stores credential IDs, public keys,
|
||||
sign counts, and AAGUIDs. No private keys, no secrets, no passphrase
|
||||
wrapping. 0600 file mode for integrity (tamper detection), not secrecy.
|
||||
|
||||
**Affected REQs**: REQ-148.
|
||||
|
||||
### C8 — Breaking-change handling (resolved)
|
||||
|
||||
**Question**: P07 (remove all password/token paths) is a breaking
|
||||
change. How are existing v0.11 clusters handled?
|
||||
|
||||
**Decision**: **`orca upgrade` refuses v0.11 clusters using
|
||||
`--password`/bare-tokens without `--accept-identity-migration`.** The
|
||||
flag prints the cutover documentation and requires explicit
|
||||
confirmation. No silent breakage. Documented in `docs/oidc.md` and the
|
||||
migration guide.
|
||||
|
||||
**Affected REQs**: REQ-146, REQ-137.
|
||||
|
||||
### C9 — Token storage at rest (resolved)
|
||||
|
||||
**Question**: Where are OIDC tokens stored locally?
|
||||
|
||||
**Decision**: **`~/.orca/credentials.json` (0600). Short-lived (1h) +
|
||||
refresh.** Standard OIDC token storage. 0600 file mode. Refresh
|
||||
handles rotation; no long-lived Orca-issued tokens (the IdP issues
|
||||
them; Orca only stores them).
|
||||
|
||||
**Affected REQs**: REQ-144.
|
||||
|
||||
### C10 — Phase count (resolved)
|
||||
|
||||
**Question**: The threat model surfaced ~25 fix areas + the
|
||||
zero-trust identity work + docs + tests + final. More than 20 phases
|
||||
is acceptable per operator guidance. How many?
|
||||
|
||||
**Decision**: **29 phases** (P0 + P01..P27 + P28 final). The operator
|
||||
explicitly accepted "more than 20 phases is acceptable if warranted."
|
||||
The GRILL stage may split/merge as needed (as v0.11 grill split P10
|
||||
into P10a/P10b).
|
||||
|
||||
**Affected**: PLAN_v0.12.md, ROADMAP.md.
|
||||
|
||||
## Open questions
|
||||
|
||||
None. All 10 clarifications resolved. Phase 0 proceeds to RESEARCH.
|
||||
@@ -0,0 +1,58 @@
|
||||
# Grill: v0.10 Docs & Install Milestone
|
||||
|
||||
## Verdict: PASS (confidence 0.82)
|
||||
|
||||
The plan is sound for a documentation + install-hardening milestone.
|
||||
No replan required. Three binding conditions adopted below.
|
||||
|
||||
## Axis review
|
||||
|
||||
### Scope justification — PASS
|
||||
The milestone closes a real gap (no CLI/jobspec/ingress docs, stale
|
||||
README, broken release pipeline) with a bounded scope (5 phases, no Go
|
||||
orchestration code changes). The v0.9 re-architecture shipped
|
||||
functionality without operator-facing docs; this milestone ships the
|
||||
docs. The install fix (P1) addresses a measured production bug
|
||||
(v0.4.5 install), not a speculative enhancement.
|
||||
|
||||
### Feasibility — PASS
|
||||
All tasks are markdown authoring (P2-P4) or bash script hardening (P1).
|
||||
No new dependencies, no schema changes, no Go code changes. The
|
||||
jobspecs in P3 must parse against the current parser — risk R1 is
|
||||
real but mitigated by validation before commit.
|
||||
|
||||
### Vertical slice integrity — PASS
|
||||
Each phase ships an independently valuable deliverable:
|
||||
- P1: install.sh works (resolves to a release with an asset)
|
||||
- P2: an operator can read the CLI/jobspec/ingress docs
|
||||
- P3: an operator can copy the examples and deploy a stack
|
||||
- P4: README + namespace.md are accurate
|
||||
- P5: milestone complete, merged, released
|
||||
|
||||
### Wave ordering — PASS
|
||||
P1 (Wave 1) unblocks all subsequent ship operations (each phase ship
|
||||
needs a correctly-asseted release). P2 + P3 (Wave 2) are parallel with
|
||||
no dependencies. P4 (Wave 3) depends on P2/P3 for cross-links. P5
|
||||
(Wave 4) depends on all.
|
||||
|
||||
### Risk register — PASS
|
||||
Three risks identified, all mitigated. R1 (jobspec parse drift) is the
|
||||
highest; mitigation is validation before commit. R2 (tea CLI asset bug)
|
||||
has a curl fallback. R3 (v0.8.15 still asset-less) is handled by
|
||||
install.sh's fallback walk.
|
||||
|
||||
## Binding conditions
|
||||
|
||||
| ID | Condition | Phase | Status |
|
||||
|----|-----------|-------|--------|
|
||||
| C-20 | Every jobspec in `examples/full-stack/` MUST parse with `internal/jobspec.ParseFile` and pass `internal/spec/schema.ValidatorFor(kind)` before P3 commits | P3 | pending |
|
||||
| C-21 | `scripts/release.sh` post-create asset verification MUST query the Gitea API and assert the tarball in attachments (not rely on `tea` exit code alone) | P1 | pending |
|
||||
| C-22 | Every factual claim in `docs/cli.md`, `docs/jobspec.md`, `docs/ingress.md` MUST be grounded in the live codebase (struct fields, flag definitions, paths) — verified by the docs-engineer persona before P2 commits | P2 | pending |
|
||||
|
||||
## Phase challenges
|
||||
|
||||
| ID | Challenge | Phase |
|
||||
|----|-----------|-------|
|
||||
| PC-11 | The jobspecs in P3 must not use fields that don't exist yet (e.g., `resources:` which lands in v0.11-P0c). Validate against the current `WorkloadSpec` struct. | P3 |
|
||||
| PC-12 | The rendered artifacts in P3 must match what the emitters actually produce, not an idealized version. Cross-check against `internal/emitter/` test fixtures. | P3 |
|
||||
| PC-13 | The README subcommand table must match `internal/cli/` exactly — no stale commands, no missing commands. | P4 |
|
||||
@@ -0,0 +1,171 @@
|
||||
# Grill: v0.11 Production Hardening — Phase 0 Adversarial Review
|
||||
|
||||
**Status**: PROCEED-WITH-CONDITIONS. The v0.11 plan is sound; 6 binding
|
||||
conditions (C-23…C-28) gate specific phases. The plan adopts R-017…R-020
|
||||
and D-215…D-237 from 5 research docs with operator decisions Q1=A, Q2=C,
|
||||
Q3=A, Q4=A, Q5=A. The grill reviewed the plan adversarially across the
|
||||
same 9 axes as GRILL_v0.9 (vision, feasibility, scope, risk, security,
|
||||
operational, cost, competitive, exit).
|
||||
|
||||
## Forcing questions + verdicts
|
||||
|
||||
### FQ1 — R-020 deadlock with `--force` + per-ns scoping
|
||||
|
||||
**Question**: With `--force` + per-namespace scoping (Q4=A), can a single
|
||||
drifted peer still block a *cluster-wide* txn (e.g., namespace creation)?
|
||||
If yes, is the `--force` escape hatch documented in C-09's failure
|
||||
contract?
|
||||
|
||||
**Verdict**: PARTIAL-BLOCK remains for cluster-wide txns. A namespace
|
||||
*creation* txn touches all peers (the new namespace dir is created on
|
||||
every peer). If one peer is drifted, the pre-flight gate refuses the
|
||||
txn cluster-wide. `--force` overrides this, but `--force` on a
|
||||
namespace-creation txn is risky (it forces the new namespace onto a
|
||||
drifted peer without reconciling the drift first).
|
||||
|
||||
**Binding condition C-23**: `orca-pull.sh` (C-09) must distinguish
|
||||
*cluster-wide* txns from *namespace-scoped* txns. Cluster-wide txns
|
||||
require `--force` with an explicit `--i-understand-the-risk` confirmation
|
||||
(or `--yes` for non-interactive). Namespace-scoped txns use per-ns
|
||||
scoping (drifted peer in ns-A doesn't block ns-B). **Gate**: P10.
|
||||
|
||||
**Confidence**: 0.88
|
||||
|
||||
### FQ2 — P10 sizing (txn plane + drift detection in one phase)
|
||||
|
||||
**Question**: P10 now absorbs drift detection (~500 LoC Go + 150 LoC
|
||||
bash + systemd units), the largest single phase. Is this a vertical
|
||||
slice that can ship atomically, or does it need splitting (P10a txn
|
||||
plane, P10b drift)?
|
||||
|
||||
**Verdict**: SPLIT RECOMMENDED. P10 has 13 tasks spanning two distinct
|
||||
subsystems: (1) the transactional plane (T1-T2: txn bundle render, SCP,
|
||||
apply, C-09 failure contract) and (2) drift detection (T3-T13: `internal/drift/`,
|
||||
Path unit emitter, notify/remediate scripts, cadence config, pre-flight
|
||||
gate, `orca` user, NFS detection, job restart). The txn plane is a
|
||||
prerequisite for drift detection (T3's `Aggregate` reads applied txn
|
||||
manifests), so the split is clean: P10a (txn plane, T1-T2) ships first,
|
||||
P10b (drift detection, T3-T13) ships after P10a.
|
||||
|
||||
**Binding condition C-24**: Split P10 into P10a (transactional plane,
|
||||
REQ-075/079, C-09) and P10b (drift detection, R-018/R-019/R-020,
|
||||
REQ-103..113). P10a ships first; P10b depends on P10a. Tags: P10a
|
||||
`v0.10.12`, P10b `v0.10.13`. All subsequent phase tags shift by 1
|
||||
(P11→`v0.10.14`, …, P16→`v0.10.22`). **Phase count: 23 → 24.**
|
||||
|
||||
**Confidence**: 0.92
|
||||
|
||||
### FQ3 — Ingress default migration path (C1)
|
||||
|
||||
**Question**: Existing v0.9/v0.10 clusters run Traefik on `:443`. R-017
|
||||
makes `127.0.0.1:8443` + nft the default. What's the upgrade path? Does
|
||||
`orca upgrade` (Q2=C) handle the binding cutover, or is it a manual
|
||||
operator step?
|
||||
|
||||
**Verdict**: UPGRADE HANDLES IT, but with a safety check. `orca upgrade`
|
||||
(REQ-115, P14a) is the thin wrapper (C2=a) that handles the Traefik
|
||||
binding cutover. The cutover is: (1) emit new Traefik static config with
|
||||
`127.0.0.1:8443`, (2) emit `/etc/nftables.d/orca.nft` with DNAT, (3)
|
||||
`systemctl reload traefik` + `nft -f`, (4) verify `curl :443` still
|
||||
routes. If step 4 fails, rollback to `:443` + remove nft rules.
|
||||
|
||||
**Binding condition C-25**: `orca upgrade` (REQ-115) must include a
|
||||
post-cutover verification step (`curl -k https://localhost:443/` returns
|
||||
200 from Traefik) with automatic rollback on failure. Document the
|
||||
rollback procedure in `docs/ingress.md`. **Gate**: P14a.
|
||||
|
||||
**Confidence**: 0.90
|
||||
|
||||
### FQ4 — Scope ceiling (LoC vs phase count)
|
||||
|
||||
**Question**: v0.11 stays at 23 phases (now 24 with C-24), but P09/P10
|
||||
(now P10a/P10b)/P15.5 grow substantially. Is the *phase count* the right
|
||||
ceiling, or should there be a *LoC/effort* ceiling per phase?
|
||||
|
||||
**Verdict**: LOOSE LoC CEILING. Phase count is a proxy for effort, but
|
||||
P10b (drift detection) is ~650 LoC across Go + bash + systemd — at the
|
||||
upper end of what a single-phase vertical slice can handle. The grill
|
||||
recommends a soft LoC ceiling of ~800 LoC per phase (Go + bash + config),
|
||||
with splitting required above ~1200 LoC.
|
||||
|
||||
**Binding condition C-26**: Per-phase LoC soft ceiling: ~800 LoC (Go +
|
||||
bash + config). Split required above ~1200 LoC. P10b (~650 LoC) is within
|
||||
the soft ceiling; P15.5 (~400 LoC: nft emitter 200 + doctor mTLS 100 +
|
||||
threat model doc) is within. No action required for v0.11; recorded for
|
||||
future milestones. **No gate.**
|
||||
|
||||
**Confidence**: 0.85
|
||||
|
||||
### FQ5 — `orca` system user on peers (operational impact)
|
||||
|
||||
**Question**: Creating a system user on every peer is a new operational
|
||||
requirement. Does this break any existing v0.9/v0.10 deployment that
|
||||
runs as root or as an existing service account?
|
||||
|
||||
**Verdict**: NO BREAK for existing deployments; NEW requirement for drift
|
||||
detection. The `orca` system user (REQ-111) is created at peer setup
|
||||
(`orca node join` / peer-setup script). Existing v0.9/v0.10 peers don't
|
||||
have the `orca` user, so drift detection's systemd Path units (which run
|
||||
as `User=orca`) won't start until the user is created. `orca upgrade`
|
||||
(REQ-115) must create the `orca` user on existing peers as part of the
|
||||
v0.11 migration.
|
||||
|
||||
**Binding condition C-27**: `orca upgrade` (REQ-115, P14a) must create
|
||||
the `orca` system user on existing peers (`useradd -r orca` idempotent)
|
||||
before P10b's drift detection can function. Document this as a
|
||||
migration prerequisite. **Gate**: P14a.
|
||||
|
||||
**Confidence**: 0.91
|
||||
|
||||
### FQ6 — P15.5 is now a mega-phase (threat model + ingress + doctor mTLS)
|
||||
|
||||
**Question**: P15.5 was originally "threat model + security review" (C-19).
|
||||
It now absorbs ingress hybrid (R-017; REQ-099..102, ~400 LoC) + `orca
|
||||
doctor mTLS` (REQ-118). Is this too much for one phase?
|
||||
|
||||
**Verdict**: MANAGEABLE but at the ceiling. P15.5 is now ~500 LoC (nft
|
||||
emitter 200 + doctor mTLS 100 + threat model doc + tests). The ingress
|
||||
hybrid and threat model are related (both are security-hardening), so
|
||||
keeping them together is defensible. The `orca doctor mTLS` (REQ-118)
|
||||
is small and reuses P01/P01.5 infrastructure. The grill recommends
|
||||
keeping P15.5 as one phase but splitting the *work* into two sub-waves
|
||||
within the phase: (1) ingress hybrid + doctor nft, (2) threat model +
|
||||
doctor mTLS.
|
||||
|
||||
**Binding condition C-28**: P15.5 commits in two sub-waves: (1) ingress
|
||||
hybrid (REQ-099..102) + `orca doctor nft` (REQ-101), (2) threat model
|
||||
(C-19) + `orca doctor mTLS` (REQ-118). Both ship under the same phase
|
||||
tag (`v0.10.20`). **No new phase; internal ordering only.**
|
||||
|
||||
**Confidence**: 0.89
|
||||
|
||||
## Binding conditions summary
|
||||
|
||||
| ID | Condition | Gate | Verification |
|
||||
|----|-----------|------|--------------|
|
||||
| C-23 | `orca-pull.sh` distinguishes cluster-wide vs namespace-scoped txns; cluster-wide requires `--force` + `--i-understand-the-risk` (or `--yes`) | P10a | Test: cluster-wide txn refused without `--force`; ns-scoped txn blocks only the drifted ns |
|
||||
| C-24 | Split P10 into P10a (txn plane, REQ-075/079, C-09) + P10b (drift detection, R-018/R-019/R-020, REQ-103..113); P10b depends on P10a; tags shift by 1 | P10a→P10b | Plan shows P10a + P10b as separate phases; P10b tasks reference P10a txn manifests |
|
||||
| C-25 | `orca upgrade` (REQ-115) includes post-cutover verification (`curl -k https://localhost:443/` returns 200) with automatic rollback on failure; rollback documented in `docs/ingress.md` | P14a | Test: cutover succeeds → 200; cutover fails → rollback to `:443` |
|
||||
| C-26 | Per-phase LoC soft ceiling: ~800 LoC (Go + bash + config); split required above ~1200 LoC | (no gate) | Recorded for future milestones |
|
||||
| C-27 | `orca upgrade` (REQ-115) creates `orca` system user on existing peers before P10b drift detection can function | P14a | Test: existing peer without `orca` user → `orca upgrade` creates it → drift detection starts |
|
||||
| C-28 | P15.5 commits in two sub-waves: (1) ingress hybrid + doctor nft, (2) threat model + doctor mTLS; same phase tag | P15.5 | Commits show two sub-waves; both under `v0.10.20` |
|
||||
|
||||
## Phase challenge summary
|
||||
|
||||
| PC | Phase | Challenge | Resolution |
|
||||
|----|-------|-----------|------------|
|
||||
| PC-11 | P10a/P10b | Txn plane + drift detection too large for one phase | Split per C-24; P10a ships first, P10b depends on it |
|
||||
| PC-12 | P15.5 | Mega-phase (threat model + ingress + doctor mTLS) | Keep as one phase; two sub-waves per C-28 |
|
||||
| PC-13 | P14a | `orca upgrade` handles 3 migrations (data + binding + orca user) | All three land in P14a per C-25, C-27; thin wrapper (C2=a) |
|
||||
| PC-14 | P09 | Aggregator extension depends on P10b drift detection | P09 in Wave 6 (after Wave 5 P10b); aggregator extension (REQ-107) only works once drift events exist |
|
||||
|
||||
## Overall verdict
|
||||
|
||||
**PROCEED-WITH-CONDITIONS**. The v0.11 plan is sound. 6 binding conditions
|
||||
(C-23…C-28) gate specific phases. The plan grows from 23 → 24 phases
|
||||
(C-24 splits P10 into P10a/P10b). All other phases are unchanged in
|
||||
count; their scope expands per the research folding (Q2=C, Q3=A).
|
||||
|
||||
The grill's confidence in the v0.11 plan is high (avg 0.89 across FQs).
|
||||
The primary risks (P10 sizing, R-020 deadlock, ingress migration) are
|
||||
all gated with verifiable conditions.
|
||||
@@ -0,0 +1,170 @@
|
||||
# Grill: v0.12 Security Hardening (Zero-Trust Identity) — Phase 0 Adversarial Review
|
||||
|
||||
**Status**: PROCEED-WITH-CONDITIONS. The v0.12 plan is sound; 10
|
||||
binding conditions (C-29..C-38) gate specific phases. The plan adopts
|
||||
R-021 (no Orca credentials) and D-238..D-247 from the threat-model
|
||||
review + operator decisions. The grill reviewed the plan
|
||||
adversarially across the same 9 axes as GRILL_v0.9/v0.11 (vision,
|
||||
feasibility, scope, risk, security, operational, cost, competitive,
|
||||
exit).
|
||||
|
||||
## Forcing questions + verdicts
|
||||
|
||||
### FQ1 — R-021 is the largest behavioral change in project history
|
||||
|
||||
**Question**: R-021 ("no Orca credentials") removes all password and
|
||||
token surfaces. P07 is explicitly breaking. Is the migration path
|
||||
(C-34 `--accept-identity-migration`) sufficient, or does the breakage
|
||||
extend beyond what's documented?
|
||||
|
||||
**Verdict**: BREAKAGE IS CONTAINED BUT UNDERESTIMATED. The plan
|
||||
documents the Proxmox `--password` and step-ca `--password-file`
|
||||
removal. But `KindToken` removal (P06) also breaks any existing
|
||||
`acl.json` that uses token identities. The migration must rewrite
|
||||
`acl.json` entries, not just refuse them.
|
||||
|
||||
**Binding condition C-29 (refined)**: P22 (`orca upgrade`) MUST
|
||||
detect v0.11 `acl.json` entries with `KindToken` and either (a)
|
||||
refuse without `--accept-identity-migration` + a documented
|
||||
re-mapping, or (b) auto-stub them as `KindOidc` with a placeholder
|
||||
`sub` requiring operator confirmation. No silent data loss.
|
||||
|
||||
**Confidence**: 0.90
|
||||
|
||||
### FQ2 — P08 master key seal is the riskiest phase
|
||||
|
||||
**Question**: A bug in seal/unseal corrupts all secrets at rest. Is
|
||||
the recovery path (Shamir 3-of-5) actually testable, and does it
|
||||
handle the "IdP lost AND shards partially lost" case?
|
||||
|
||||
**Verdict**: RECOVERY IS TESTABLE BUT THE EDGE CASES ARE UNDERTESTED.
|
||||
The plan covers the happy path (3-of-5) and the failure case (< 3
|
||||
shards -> unrecoverable). But the "IdP lost, 3 shards available, but
|
||||
the OIDC-derived salt was also lost" case (the salt is in the sealed
|
||||
blob, so this shouldn't happen -- but verify) needs an explicit test.
|
||||
|
||||
**Binding condition C-30 (refined)**: P08 MUST include a test that
|
||||
recovers with 3-of-5 shards AFTER the IdP is simulated-down (seal key
|
||||
reconstruction from shards, NOT from OIDC token). The sealed blob
|
||||
must contain the salt (so recovery doesn't need the IdP). Document
|
||||
that the salt is stored in the sealed blob, not derived from the
|
||||
token at recovery time.
|
||||
|
||||
**Confidence**: 0.88
|
||||
|
||||
### FQ3 — P05 WebAuthn connector feasibility
|
||||
|
||||
**Question**: The custom Dex connector (~300 LoC) is new ground. Is
|
||||
the `go-webauthn` library mature enough, and does the RP ID / secure
|
||||
context requirement create a chicken-and-egg problem (Dex needs
|
||||
Traefik, Traefik needs the cert, the cert needs step-ca, step-ca
|
||||
needs the operator authenticated -- by Dex)?
|
||||
|
||||
**Verdict**: NO CHICKEN-AND-EGG, but the bootstrap sequence must be
|
||||
explicit. The cert comes from step-ca's OIDC provisioner (P07), but
|
||||
the FIRST operator must authenticate to step-ca. Resolution: the
|
||||
first operator uses the mTLS-only path (cluster CA cert, held
|
||||
offline) to mint the first Traefik cert. Dex then comes up. The
|
||||
first WebAuthn registration happens via that first cert. The chicken-
|
||||
and-egg is resolved by the mTLS-only bootstrap path.
|
||||
|
||||
**Binding condition C-31 (new)**: P04/P05 MUST document the bootstrap
|
||||
sequence: (1) `orca init` bootstraps the cluster CA (step-ca, mTLS-
|
||||
only), (2) `orca auth init-idp` deploys Dex behind Traefik using the
|
||||
step-ca cert, (3) the first operator registers a passkey via the
|
||||
mTLS-authenticated session, (4) subsequent operators use WebAuthn.
|
||||
The mTLS-only path is the bootstrap escape hatch.
|
||||
|
||||
**Confidence**: 0.87
|
||||
|
||||
### FQ4 — P21 SQLite encryption CGO risk
|
||||
|
||||
**Question**: SQLCipher needs CGO (breaks D-008 cross-compile). The
|
||||
C-31 fallback is "file-mode 0600 + documented threat." Is that
|
||||
acceptable for a security-hardening milestone?
|
||||
|
||||
**Verdict**: FALLBACK IS ACCEPTABLE BUT MUST BE EXPLICIT. The
|
||||
threat-model finding (F8) is "DBs unencrypted with no explicit file
|
||||
mode." The minimum fix (0600 file mode) closes the "no explicit mode"
|
||||
half. The "unencrypted" half is a documented residual risk if CGO is
|
||||
infeasible. This is consistent with the project's "no CGO" invariant
|
||||
(D-008) which is load-bearing for cross-compile.
|
||||
|
||||
**Binding condition C-32 (refined)**: P21 MUST evaluate at least one
|
||||
CGO-free encryption option (e.g., application-level AES-GCM envelope
|
||||
around the SQLite file, or a FUSE encryption layer). If all are
|
||||
infeasible or too complex for v0.12, document the decision + residual
|
||||
risk. The fallback is file-mode 0600 only. No CGO.
|
||||
|
||||
**Confidence**: 0.85
|
||||
|
||||
### FQ5 — Phase count (29) vs. sizing
|
||||
|
||||
**Question**: 29 phases is the largest milestone in project history
|
||||
(v0.11 was 24, v0.9 was 14). Is any single phase too large to ship
|
||||
atomically?
|
||||
|
||||
**Verdict**: TWO PHASES ARE LARGE. P04 (OIDC+Dex) and P08 (master
|
||||
key seal) are each ~500-700 LoC + tests. They're within the v0.11
|
||||
P10a/P10b sizing that the grill previously accepted, but the grill
|
||||
split P10. If P04 or P08 grows during execution, the EXECUTE workflow
|
||||
may split them (P04a/P04b, P08a/P08b).
|
||||
|
||||
**Binding condition C-33 (new)**: P04 and P08 are SPLIT CANDIDATES.
|
||||
If either exceeds ~700 LoC + tests during EXECUTE, split: P04a (OIDC
|
||||
client) / P04b (bundled Dex deploy); P08a (seal/unseal + Shamir) /
|
||||
P08b (CLI + mTLS-only path). The planner monitors LoC during
|
||||
execution.
|
||||
|
||||
**Confidence**: 0.82
|
||||
|
||||
### FQ6 — C-32 human gate (leaked GITEA_TOKEN) could stall the final ship
|
||||
|
||||
**Question**: If the operator doesn't rotate the token, P28 can't
|
||||
ship. Is there an escalation path that doesn't block the milestone?
|
||||
|
||||
**Verdict**: ESCALATION PATH EXISTS. Ship as `v0.11.28-rc1` (release
|
||||
candidate) if the token is not rotated by P28. The `v0.11.28` final
|
||||
tag (milestone release) waits for confirmation. The milestone is
|
||||
"complete" (all phases shipped); only the final tag is gated.
|
||||
|
||||
**Binding condition C-34 (refined)**: C-32 human-gate: if the
|
||||
GITEA_TOKEN is not rotated by P28, ship `v0.11.28-rc1` (all phases
|
||||
complete, release notes flag the pending rotation). The `v0.11.28`
|
||||
final tag is cut when the operator confirms. The `---ci---` block
|
||||
records `escalation: type=release_pending resolution=auto` -- does
|
||||
not halt the pipeline.
|
||||
|
||||
**Confidence**: 0.90
|
||||
|
||||
## Adopted binding conditions (C-29..C-38)
|
||||
|
||||
| ID | Condition | Phase | Confidence |
|
||||
|----|-----------|-------|------------|
|
||||
| C-29 | P23 (dual-write closure) gated on P06/P08/P09/P11 all shipped. P22 must detect v0.11 `acl.json` `KindToken` entries and refuse/remap without `--accept-identity-migration`. | P22/P23 | 0.90 |
|
||||
| C-30 | P14 (master key rotation) reversible; `--dry-run` mandatory; auto-rollback to old sealed key on any ns failure. | P14 | 0.88 |
|
||||
| C-31 | P21 (SQLite encryption): evaluate at least one CGO-free option (app-level AES-GCM envelope, FUSE layer). If infeasible, file-mode 0600 + documented residual risk. No CGO. | P21 | 0.85 |
|
||||
| C-32 | **Human-gate**: leaked GITEA_TOKEN (F17) rotated + `.env` re-seeded before `v0.11.28` final tag. If not rotated by P28, ship `v0.11.28-rc1`. History-scrub best-effort, non-blocking. Escalation hook in `---ci---`. | P28 | 0.90 |
|
||||
| C-33 | P26 (security integration tests) in `.coreci.yml` `validate`, gates merges -- not opt-in. | P26 | 0.95 |
|
||||
| C-34 | P07 (password/token removal) breaking. `orca upgrade` (P22) refuses v0.11 clusters using `--password`/bare-tokens/`KindToken` without `--accept-identity-migration`. No silent breakage. | P07/P22 | 0.90 |
|
||||
| C-35 | P08 (Shamir recovery): 3-of-5 shards printed at seal time, operator stores offline. Sealed blob contains the salt (recovery doesn't need the IdP). If IdP lost AND < 3 shards -> unrecoverable by design (documented residual risk). No backdoor. Test recovery with IdP-down. | P08 | 0.88 |
|
||||
| C-36 | OIDC client secret (confidential clients) at `ClusterDir()/oidc-client-secret` (0600), rotatable via `orca auth rotate-client-secret`, never committed. Public PKCE clients avoid even this. | P04 | 0.92 |
|
||||
| C-37 | P04/P05 (bundled Dex + WebAuthn): document the bootstrap sequence (mTLS-only first cert -> Dex -> first passkey). The mTLS-only path is the bootstrap escape hatch. If WebAuthn proves infeasible, bundled Dex ships mTLS-client-cert-only (C-37 fallback). The "no Orca credentials" invariant holds regardless. | P04/P05 | 0.87 |
|
||||
| C-38 | P05 (WebAuthn): RP ID must match the cluster's Traefik-served domain; `orca auth init-idp` configures it. HTTPS secure context via Traefik (step-ca cert). P26 integration tests use the WebAuthn virtual-authenticator API -- no hardware key required in CI. | P05 | 0.90 |
|
||||
|
||||
## Verdict: PROCEED-WITH-CONDITIONS
|
||||
|
||||
The v0.12 plan is sound. The 10 binding conditions gate the risky
|
||||
phases. The 29-phase count is within the operator's "more than 20 if
|
||||
warranted" guidance. The plan adopts R-021 (no Orca credentials) and
|
||||
D-238..D-247. The grill does NOT recommend REPLAN.
|
||||
|
||||
## Phase challenges (PC-01..PC-05)
|
||||
|
||||
| ID | Challenge | Phase |
|
||||
|----|-----------|-------|
|
||||
| PC-01 | P04/P08 are split candidates if LoC exceeds ~700 (C-33) | P04/P08 |
|
||||
| PC-02 | P07 breaking change -- migration must handle `KindToken` acl.json entries, not just passwords (C-29/C-34) | P07/P22 |
|
||||
| PC-03 | P05 WebAuthn bootstrap sequence must be explicit (mTLS-only first cert) | P04/P05 |
|
||||
| PC-04 | P21 SQLite encryption CGO evaluation -- document the decision + residual risk if fallback | P21 |
|
||||
| PC-05 | C-32 human gate -- `v0.11.28-rc1` escalation if token not rotated | P28 |
|
||||
+36
-36
@@ -25,7 +25,7 @@ re-architecture proceeds). Their **mechanics** remain as binding work items:
|
||||
phase, split heavy phases.
|
||||
- **Migration** mechanics → split P14 into P14a/P14b/P14c, design migration
|
||||
ordering in v0.9-P00.
|
||||
- **Security** mechanics → threat model in v1.0-P15.5 (C-19).
|
||||
- **Security** mechanics → threat model in v0.10-P15.5 (C-19).
|
||||
|
||||
The 19 binding conditions (C-01..C-19) and 10 phase challenges (PC-01..PC-10)
|
||||
are adopted in full as execution gates.
|
||||
@@ -76,12 +76,12 @@ silently break the build story; must be spiked before commitment.
|
||||
simultaneously deprecates 7 shipped subsystems and adds 8 net-new subsystems?
|
||||
The deprecation of ~10k lines of shipped daemon/transport/CA code is not listed
|
||||
as a phase. v0.9 P0a..P10 ship 10 phases of workload features before the
|
||||
transactional control plane (R-010 deferred to v1.0 P10) — is that intentional
|
||||
transactional control plane (R-010 deferred to v0.10 P10) — is that intentional
|
||||
or a sequencing error? Hidden requirements (step-ca self-upgrade, master.key
|
||||
rotation, Syncthing version drift)?
|
||||
|
||||
**Evidence**: v0.9 phase ordering ships P0a..P10 workloads, then P10 lead rules
|
||||
+ migration *last*. The transactional plane (R-010) is deferred to v1.0 P10 —
|
||||
+ migration *last*. The transactional plane (R-010) is deferred to v0.10 P10 —
|
||||
two milestones away. v0.8 was a 4-phase NFR milestone; v0.6 was 4-phase feature.
|
||||
The PRD's v0.9 (11) + v1.0 (16) = 27 phases is 3-4× prior milestone size with
|
||||
no evidence the throughput model was re-validated. No phase is labeled
|
||||
@@ -94,12 +94,12 @@ no evidence the throughput model was re-validated. No phase is labeled
|
||||
- **PC-01**: Move the transactional plane primitives forward. The
|
||||
transactional primitives (desired-state, lead-applier, drift, rollback) are
|
||||
the substrate every workload phase depends on. Design spike in v0.9-P00;
|
||||
full implementation in v1.0-P10 per PRD ordering (workloads first is accepted
|
||||
full implementation in v0.10-P10 per PRD ordering (workloads first is accepted
|
||||
given the dual-write window mitigation in I-C-006).
|
||||
- **PC-02**: Add `v0.9-P00 — Deprecation sweep` as an explicit phase. Must land
|
||||
before any new feature phase so coverage gates don't measure dead packages.
|
||||
- **PC-03**: Split migration: `v0.9-P00b — Migration design + dry-run` (early,
|
||||
parallel to deprecation) and `v1.0-P14 — Production migration` (final).
|
||||
parallel to deprecation) and `v0.10-P14 — Production migration` (final).
|
||||
Migration design must inform every earlier phase, not be informed by them.
|
||||
|
||||
**Rationale**: 27 phases framed as "two milestones" while simultaneously
|
||||
@@ -118,7 +118,7 @@ testing frameworks not in current dep map — what's the cost?
|
||||
|
||||
**Evidence**: Active roster has 3 of 8 active; the 5 dormant personas map
|
||||
directly to the 5 new apt dependencies. v0.8 took 4 phases for a pure
|
||||
test/coverage milestone; v1.0 includes 11 distinct subsystems in one
|
||||
test/coverage milestone; v0.10 includes 11 distinct subsystems in one
|
||||
"milestone." No bash test infrastructure exists today.
|
||||
|
||||
**Verdict**: PROCEED-WITH-CONDITION
|
||||
@@ -127,7 +127,7 @@ test/coverage milestone; v1.0 includes 11 distinct subsystems in one
|
||||
**Binding conditions**:
|
||||
- **C-04**: Produce a per-phase sizing estimate using v0.6/v0.7/v0.8 actuals
|
||||
as the analogous baseline. If realistic phase count exceeds 35, the
|
||||
milestone must be split into v0.9 + v0.10 + v1.0 (three milestones), not two.
|
||||
milestone must be split into v0.9 + v0.10 (three milestones), not two.
|
||||
- **C-05**: Reactivate or explicitly assign coverage for the dormant personas'
|
||||
domains (security, network, devops); no "dormant" = "unowned."
|
||||
- **C-06**: Decide and document whether bash scripts count toward the coverage
|
||||
@@ -169,9 +169,9 @@ specified.
|
||||
`ca.crt` trust root and import into step-ca, or (b) document forced
|
||||
re-bootstrap as an accepted breaking change with per-cluster upgrade
|
||||
procedure. Cannot be deferred.
|
||||
- **C-08**: Before the first SPIFFE-touching phase (v1.0 P02 ACL), produce a
|
||||
- **C-08**: Before the first SPIFFE-touching phase (v0.10 P02 ACL), produce a
|
||||
working spike of step-ca JWT-SVID or X.509-SVID minting from the orca CLI
|
||||
(v1.0-P01.5). If the spike fails, SPIFFE is deferred and ACL falls back to
|
||||
(v0.10-P01.5). If the spike fails, SPIFFE is deferred and ACL falls back to
|
||||
mTLS identity (which the shipped model already had).
|
||||
- **C-09**: Define and test the `orca-pull.sh` failure contract: idempotent
|
||||
re-run, bounded retry, deterministic state on partial failure, syslog
|
||||
@@ -212,9 +212,9 @@ is not portable across the orca model. "Atomic, auto-rollback" is asserted for
|
||||
**Confidence**: 0.82
|
||||
|
||||
**Mechanics adopted**:
|
||||
- **PC-04**: Split P14 into `v1.0-P14a — Data migration` (current scope),
|
||||
`v1.0-P14b — Daemon cutover + running-allocation adoption`,
|
||||
`v1.0-P14c — Mixed-version cluster tolerance + no-orca-on-server enforcement`.
|
||||
- **PC-04**: Split P14 into `v0.10-P14a — Data migration` (current scope),
|
||||
`v0.10-P14b — Daemon cutover + running-allocation adoption`,
|
||||
`v0.10-P14c — Mixed-version cluster tolerance + no-orca-on-server enforcement`.
|
||||
Three sub-phases, each with its own integration test.
|
||||
|
||||
**Rationale**: The migration plan as described covers the easy third (file
|
||||
@@ -254,7 +254,7 @@ documented in step-ca's own operations guide (out-of-band knowledge).
|
||||
- **C-13**: Replace server-side doctor with a CLI-driven equivalent that
|
||||
SSH-probes every node and reconstructs the health view the daemon used to
|
||||
provide locally. This is a new requirement, not a feature; added as
|
||||
I-C-002 / v1.0-P14c.
|
||||
I-C-002 / v0.10-P14c.
|
||||
- **C-14**: Syncthing conflict-resolution policy must be deterministic,
|
||||
documented, and tested with a forced-divergence integration test.
|
||||
|
||||
@@ -291,7 +291,7 @@ is now the default.
|
||||
|
||||
**Mechanics adopted**:
|
||||
- **C-19**: Write a threat model for the new posture before any
|
||||
security-touching phase (v1.0-P15.5). Defend master.key + CLI mint authority
|
||||
security-touching phase (v0.10-P15.5). Defend master.key + CLI mint authority
|
||||
or revise. The shipped model deliberately avoided putting a single stealable
|
||||
file on a single host that decrypts all secrets and mints all identities.
|
||||
The threat model must document why the new posture is acceptable or specify
|
||||
@@ -348,7 +348,7 @@ coverage gate (D-042/D-047) is Go-specific.
|
||||
**Rationale**: Bash is not inherently unmaintainable, but bash *in a Go-only,
|
||||
coverage-gated, structured-logging project* is a language-without-rails. Without
|
||||
the four conditions above, the bash control plane becomes the part of the
|
||||
codebase that everyone is afraid to touch by v1.0 P05. The drift between Go
|
||||
codebase that everyone is afraid to touch by v0.10 P05. The drift between Go
|
||||
emitters and bash appliers is the single most likely source of "works on the
|
||||
CLI's machine, fails on the lead" bugs.
|
||||
|
||||
@@ -367,7 +367,7 @@ transactional layer *on top of the daemon*? Is this re-architecture driven by a
|
||||
|
||||
**Evidence**: ROADMAP.md and PROJECT.md: every milestone from v0.1 to v0.8
|
||||
explicitly says "the vision is unchanged; this milestone is not a direction
|
||||
change." v0.9/v1.0 is the *first* milestone in the project's history that
|
||||
change." v0.9/v0.10 is the *first* milestone in the project's history that
|
||||
reverses the vision's anti-patterns. AD-010's rationale: "step-ca/cfssl/
|
||||
vault-pki too heavyweight for Orca's footprint." Nothing in the original PRD
|
||||
suggested Orca's footprint changed. The shipped model's `internal/transport`
|
||||
@@ -417,35 +417,35 @@ conditions (C-01..C-19) as execution gates. If the C-04 sizing estimate exceeds
|
||||
| C-01 | Evaluate wasmtime Go binding CGO impact; if CGO-required, drop wasmtime as primary or revoke D-002 | v0.9-P07b | Build matrix spike on linux/amd64+arm64; revocation decision recorded |
|
||||
| C-02 | Syncthing feasibility spike: config injection, conflict policy, deterministic failure mode | v0.9-P09 | Spike report + forced-divergence integration test |
|
||||
| C-03 | Check PRD into `.ciagent/PRD_v0.9.md` before any v0.9 phase begins | (gate) | ✅ Resolved — file committed |
|
||||
| C-04 | Per-phase sizing estimate vs v0.6/v0.7/v0.8 actuals; if >35, split into v0.9+v0.10+v1.0 | v0.9 start | Estimate doc with analogous-phase sizing table |
|
||||
| C-04 | Per-phase sizing estimate vs v0.6/v0.7/v0.8 actuals; if >35, split into v0.9+v0.10 | v0.9 start | ✅ RESOLVED — operator decision: keep 2 milestones (v0.9+v0.10), keep all phases (40 total), v1.0 UAT-gated after v0.10 |
|
||||
| C-05 | Reactivate or assign dormant persona domains (security, network, devops) | v0.9-P00 | PERSONAS.md updated with named owners |
|
||||
| C-06 | Decide bash coverage-gate status; if exempt, record compensating control | v0.9-P00 | Decision recorded in PROJECT.md D-series; CI pipeline shows the gate |
|
||||
| C-07 | CA migration spec: preserve existing trust root or document forced re-bootstrap | v1.0-P14a | Spec doc + migration dry-run on test cluster |
|
||||
| C-08 | SPIFFE SVID minting spike; if fails, fall back to mTLS identity | v1.0-P02 (spike in P01.5) | Working SVID mint from orca CLI in sandbox |
|
||||
| C-09 | `orca-pull.sh` failure contract: idempotent re-run, bounded retry, deterministic state, structured syslog | v1.0-P10 | Failure-path integration test + syslog structured-tag verification |
|
||||
| C-07 | CA migration spec: preserve existing trust root or document forced re-bootstrap | v0.10-P14a | Spec doc + migration dry-run on test cluster |
|
||||
| C-08 | SPIFFE SVID minting spike; if fails, fall back to mTLS identity | v0.10-P02 (spike in P01.5) | Working SVID mint from orca CLI in sandbox |
|
||||
| C-09 | `orca-pull.sh` failure contract: idempotent re-run, bounded retry, deterministic state, structured syslog | v0.10-P10 | Failure-path integration test + syslog structured-tag verification |
|
||||
| C-10 | Traefik config atomicity protocol (tmpfile+fsync+rename) + malformed-config behavior verified | v0.9-P02 | Atomic-rename test + Traefik malconfig-hold-last-good assertion |
|
||||
| C-11 | Lead-side watchdog meta-timer for `orca-pull.sh` starvation, with structured alert path | v1.0-P09 | Watchdog fires on injected pull failure; alert received |
|
||||
| C-12 | Document step-ca HA story; if single-node, record as accepted SPOF with mitigation | v1.0-P09 | Decision doc; if HA, RAFT/sync story in orca plan |
|
||||
| C-13 | Replace server-side doctor with CLI-SSH-driven equivalent | v1.0-P14c | New REQ-086 in REQUIREMENTS.md; integration test SSH-probes N nodes |
|
||||
| C-14 | Syncthing conflict-resolution policy deterministic + forced-divergence integration test | v1.0-P09 | Test induces divergence; resolves to single deterministic state |
|
||||
| C-11 | Lead-side watchdog meta-timer for `orca-pull.sh` starvation, with structured alert path | v0.10-P09 | Watchdog fires on injected pull failure; alert received |
|
||||
| C-12 | Document step-ca HA story; if single-node, record as accepted SPOF with mitigation | v0.10-P09 | Decision doc; if HA, RAFT/sync story in orca plan |
|
||||
| C-13 | Replace server-side doctor with CLI-SSH-driven equivalent | v0.10-P14c | New REQ-086 in REQUIREMENTS.md; integration test SSH-probes N nodes |
|
||||
| C-14 | Syncthing conflict-resolution policy deterministic + forced-divergence integration test | v0.10-P09 | Test induces divergence; resolves to single deterministic state |
|
||||
| C-15 | Bash testing framework (bats/shunit2) + shellcheck + shfmt in CoreCI before any bash ships | v0.9-P00 | CI pipeline green with the gate on a sample script |
|
||||
| C-16 | Versioned JSON-schema render-format contract between Go emitters and bash appliers | v0.9-P00 | Schema file in repo; both sides validate; mismatch fails CI |
|
||||
| C-17 | Bash scripts emit slog-compatible JSON to syslog with audit-log field set (REQ-006) | v0.9-P00 | Syslog capture test verifies field-presence + JSON parse |
|
||||
| C-18 | Document bash-side equivalents (or accepted drops) for shipped transport capabilities | v0.9-P00 | Capability-mapping doc in `.ciagent/` |
|
||||
| C-19 | Write a threat model for the new posture; defend master.key + CLI mint authority or revise | v1.0-P15.5 | Threat-model doc reviewed and committed; design revised if regression found |
|
||||
| C-19 | Write a threat model for the new posture; defend master.key + CLI mint authority or revise | v0.10-P15.5 | Threat-model doc reviewed and committed; design revised if regression found |
|
||||
|
||||
# Phase Plan Challenges
|
||||
|
||||
| # | Phase | Problem | Fix |
|
||||
|---|-------|---------|-----|
|
||||
| PC-01 | v0.9 P0a–P10 | Ship 10 phases of workload features before the transactional control plane | Design spike in v0.9-P00; full impl in v1.0-P10 per PRD ordering (workloads first accepted with dual-write mitigation) |
|
||||
| PC-01 | v0.9 P0a–P10 | Ship 10 phases of workload features before the transactional control plane | Design spike in v0.9-P00; full impl in v0.10-P10 per PRD ordering (workloads first accepted with dual-write mitigation) |
|
||||
| PC-02 | (missing) | Deprecation of ~10k lines of daemon/transport/CA code is not a phase | Add `v0.9-P00 — Deprecation sweep` as explicit phase before any new feature phase |
|
||||
| PC-03 | v0.9 P10 | Migration is the last phase of v0.9 but is highest-risk | Split: migration design in v0.9-P00 (early), implementation in v1.0-P14 (final) |
|
||||
| PC-04 | v1.0 P14 | Covers data migration only; omits running-allocation cutover, mixed-version cluster, rollback trigger | Split into P14a (data), P14b (daemon cutover), P14c (mixed-version tolerance) |
|
||||
| PC-05 | v1.0 P02 | SPIFFE is a documented reversal with no spike; lands before spike possible | Insert `v1.0-P01.5 — SPIFFE mint spike` as hard gate before P02 |
|
||||
| PC-06 | v1.0 P10 | Transactional plane depends on lead-applier bash scripts (C-09) not gated | Reorder to v0.9-P00 design + add C-09 gate |
|
||||
| PC-07 | v1.0 P15/P16 | README before security threat model | Add `v1.0-P15.5 — Threat model + security review` before final review |
|
||||
| PC-08 | (missing) | No phase replaces server-side `orca doctor` | Add as I-C-002 / v1.0-P14c (CLI-SSH-driven doctor) |
|
||||
| PC-03 | v0.9 P10 | Migration is the last phase of v0.9 but is highest-risk | Split: migration design in v0.9-P00 (early), implementation in v0.10-P14 (final) |
|
||||
| PC-04 | v0.10 P14 | Covers data migration only; omits running-allocation cutover, mixed-version cluster, rollback trigger | Split into P14a (data), P14b (daemon cutover), P14c (mixed-version tolerance) |
|
||||
| PC-05 | v0.10 P02 | SPIFFE is a documented reversal with no spike; lands before spike possible | Insert `v0.10-P01.5 — SPIFFE mint spike` as hard gate before P02 |
|
||||
| PC-06 | v0.10 P10 | Transactional plane depends on lead-applier bash scripts (C-09) not gated | Reorder to v0.9-P00 design + add C-09 gate |
|
||||
| PC-07 | v0.10 P15/P16 | README before security threat model | Add `v0.10-P15.5 — Threat model + security review` before final review |
|
||||
| PC-08 | (missing) | No phase replaces server-side `orca doctor` | Add as I-C-002 / v0.10-P14c (CLI-SSH-driven doctor) |
|
||||
| PC-09 | v0.9 P09 | Syncthing lands before feasibility spike (C-02) | Spike must precede P09; if P09 is the spike, rename + gate on spike success |
|
||||
| PC-10 | v0.9 P07 | Five runtimes in one phase, including wasmtime (CGO risk) and pve-vm/pve-ct | Split: P07a (process+podman), P07b (wasmtime, C-01 gated), P07c (pve-vm+ct) |
|
||||
|
||||
@@ -454,9 +454,9 @@ conditions (C-01..C-19) as execution gates. If the C-04 sizing estimate exceeds
|
||||
1. **What measured operational failure of the shipped v0.8 daemon model is the re-architecture responding to?** — ✅ Resolved by override ground 1.
|
||||
2. **Can the v0.9 scope be delivered as additive extensions?** — ✅ Resolved: rejected per override grounds 1 + 5.
|
||||
3. **What is the wasmtime/CGO resolution?** — Closes via C-01 spike in v0.9-P07b.
|
||||
4. **What is the master.key threat model?** — Closes via C-19 in v1.0-P15.5.
|
||||
4. **What is the master.key threat model?** — Closes via C-19 in v0.10-P15.5.
|
||||
5. **What is the rollback unit of work for §24, and what triggers it?** — Must be answered in v0.9-P00 txn-design spike (I-B-007).
|
||||
6. **Is step-ca single-node acceptable as a cluster SPOF?** — Closes via C-12 in v1.0-P09.
|
||||
6. **Is step-ca single-node acceptable as a cluster SPOF?** — Closes via C-12 in v0.10-P09.
|
||||
7. **Can the bash control plane be reduced?** — Closes in v0.9-P00 (fold 3+ scripts into Go-side SSH invocations where possible).
|
||||
8. **What is the realistic phase count?** — Closes via C-04 sizing before v0.9 starts; if >35, the plan becomes three milestones.
|
||||
9. **Does the PRD's reversal of 6 documented decisions require a formal AD-series supersession?** — ✅ Resolved: supersession table recorded in PROJECT.md + ARCHITECTURE.md.
|
||||
@@ -481,5 +481,5 @@ conditions (C-01..C-19) as execution gates. If the C-04 sizing estimate exceeds
|
||||
| E-ID | Item | Auto-decision | Mitigation |
|
||||
|------|------|---------------|-----------|
|
||||
| E-01 | Whether the re-architecture is justified vs incremental | OVERRIDDEN by user — direction holds | Six-part evidence basis recorded in PROJECT.md Supersession Table |
|
||||
| E-02 | Whether master.key passphrase-less posture is acceptable | REPLAN mechanics — threat model first | C-19 in v1.0-P15.5; if threat model shows regression vs shipped, revise design |
|
||||
| E-03 | Whether 27 phases fit in 2 milestones | Auto-split if sizing exceeds 35 | C-04; if exceeded, milestone becomes v0.9 + v0.10 + v1.0 |
|
||||
| E-02 | Whether master.key passphrase-less posture is acceptable | REPLAN mechanics — threat model first | C-19 in v0.10-P15.5; if threat model shows regression vs shipped, revise design |
|
||||
| E-03 | Whether 27 phases fit in 2 milestones | Auto-split if sizing exceeds 35 | ✅ RESOLVED — operator: keep 2 milestones (v0.9+v0.10), keep all phases, v1.0 UAT-gated |
|
||||
@@ -0,0 +1,39 @@
|
||||
# Ideation: v0.10 Docs & Install Milestone
|
||||
|
||||
## Tier 1 — Mechanical (codebase-grounded, no new deps)
|
||||
|
||||
| ID | Idea | Source | Accepted | REQ |
|
||||
|----|------|--------|----------|-----|
|
||||
| I-M-091 | `docs/cli.md` comprehensive CLI reference | README subcommand table is stale (missing cert/daemon/doctor/audit/ns/node-capacity/node-key-reset); no `docs/` CLI reference exists | ✅ | REQ-091 |
|
||||
| I-M-092 | `docs/jobspec.md` markdown frontmatter schema reference | Operators must read `internal/jobspec/markdown.go` source to author jobspecs; no reference doc exists | ✅ | REQ-092 |
|
||||
| I-M-093 | `docs/ingress.md` Traefik ingress reference | The service→Traefik mapping (R-007, atomic reload, drain, TLS) is undocumented; the user explicitly asked for "ingress configured" | ✅ | REQ-093 |
|
||||
| I-M-094 | `examples/full-stack/` with 5 valid jobspecs + rendered artifacts + walkthrough | No examples directory exists; `testdata/` holds legacy HCL test fixtures, not operator examples | ✅ | REQ-094 |
|
||||
| I-M-095 | README.md refresh (status, subcommand table, install example, dev targets, docs/examples sections) | README says "v0.1: Foundation"; subcommand table missing 5 commands; install example pins v0.4.2 | ✅ | REQ-095 |
|
||||
| I-M-096 | `docs/namespace.md` v0.9 multi-namespace layout update | Documents the v0.8 flat layout, not the v0.9 `cluster/`+`_defaults/`+per-ns layout | ✅ | REQ-096 |
|
||||
|
||||
## Tier 2 — Backend-enriched (API/behavior-grounded)
|
||||
|
||||
| ID | Idea | Source | Accepted | REQ |
|
||||
|----|------|--------|----------|-----|
|
||||
| I-B-097 | `scripts/release.sh` cross-build amd64 + post-create asset verification | v0.8.x releases shipped with zero binary assets; install.sh resolves to v0.8.15 then errors on missing tarball; root cause of v0.4.5 install | ✅ | REQ-097 |
|
||||
| I-B-098 | `scripts/install.sh` asset fallback walk + `--check` dry-run | install.sh has no fallback when the latest release lacks the expected tarball; a broken release blocks all installs | ✅ | REQ-098 |
|
||||
|
||||
## Tier 3 — Cross-project (deferred — single-project mode)
|
||||
|
||||
No cross-project ideas. Orca is single-project mode.
|
||||
|
||||
## Rejected ideas
|
||||
|
||||
- **Backfill the existing v0.8.15 release with a binary asset** —
|
||||
rejected per D-192. Backfilling a past release is an ops task, not a
|
||||
docs milestone deliverable. The next tagged phase (P1 ship at v0.9.1)
|
||||
will be the first correctly-asseted release; install.sh's fallback
|
||||
walk handles the gap.
|
||||
- **Document both v0.8 and v0.9 paths equally** — rejected per D-191.
|
||||
The v0.8 path is deprecated and scheduled for removal; documenting it
|
||||
as primary misleads new operators.
|
||||
- **arm64 tarball in release.sh** — rejected for this milestone per
|
||||
D-193. The install user base is amd64 today; arm64 is a separate
|
||||
enhancement.
|
||||
- **Per-command `docs/cli/*.md` subdirectory** — rejected per D-188.
|
||||
Single-file `docs/cli.md` matches the existing flat `docs/` layout.
|
||||
@@ -0,0 +1,183 @@
|
||||
# Ideation v0.12: Security Hardening (Zero-Trust Identity)
|
||||
|
||||
**Status**: 30 ideas accepted (0 skipped, 0 modified). All from Tier 1
|
||||
(mechanical analysis of the threat-model review) and Tier 2
|
||||
(backend-enriched prioritization). The `--ideate` flag was passed;
|
||||
ideation ran between RESEARCH and PLAN per run.md Step 3.
|
||||
|
||||
## Tier 1 — Mechanical analysis
|
||||
|
||||
### 2.1 Git-native pattern mining
|
||||
|
||||
The v0.11 milestone shipped 24 phases with a threat model in P15.5
|
||||
(gate C-19). The threat model identified residual risks but did not
|
||||
close them -- it documented them for v1.x. The v0.12 ideation ingests
|
||||
that threat model as the primary signal source.
|
||||
|
||||
**Repeated lessons** (from v0.8..v0.11 `---ci---` blocks):
|
||||
- "Deprecated but still load-bearing" appears 6 times across v0.8..v0.11
|
||||
(legacy CA, mTLS transport, daemon, certpaths, step-ca password
|
||||
provisioner, `hmacSHA256` dead code). The dual-write window is the
|
||||
single largest attack-surface expander. -> **F16 / REQ-138**.
|
||||
- "TOFU by default, pre-pin optional" appears 4 times (v0.6 SSH join,
|
||||
v0.8 host-key-fingerprint, v0.11 drift scripts). TOFU is a
|
||||
first-connect MITM risk. -> **F15 / REQ-139** (known_hosts tightening).
|
||||
- "File modes checked at write, not at read" appears 3 times (v0.2
|
||||
cert modes, v0.5 namespace dirs, v0.11 master key). -> **F13 /
|
||||
REQ-130**.
|
||||
|
||||
**Low-confidence decisions** (confidence < 0.85 in `---ci---` blocks):
|
||||
- D-007 (mTLS for v0.1, tokens deferred) -- 0.80. v0.12 closes the
|
||||
token gap via OIDC (no Orca-issued tokens; the IdP issues them).
|
||||
- D-028 (repo visibility flip for public releases) -- 0.85. v0.12
|
||||
adds install.sh checksum verification (F14) as defense-in-depth.
|
||||
|
||||
**Escalation types**:
|
||||
- `release_pending` (v0.8..v0.11 ship fallbacks) -- not security-relevant.
|
||||
- `human_validation` (v0.11 C-19 threat model) -- v0.12 is the
|
||||
comprehensive closure of those documented risks.
|
||||
|
||||
**Compound solutions** (generalized patterns):
|
||||
- The "shellQuote + regression test" pattern from v0.8 SSH trust
|
||||
hardening (REQ-058) generalizes to all SSH-exec interpolation sites
|
||||
(podman, wasm, aggregate.sh). -> **F3 / REQ-119**.
|
||||
- The "atomic temp + chmod + fsync + rename" pattern from
|
||||
`WriteAtomic` (ca.go) generalizes to migration `copyFile` and
|
||||
backup restore. -> **F19 / REQ-137**.
|
||||
|
||||
**Partial requirements**: none (v0.11 shipped all REQs complete).
|
||||
|
||||
### 2.2 Coverage gap analysis
|
||||
|
||||
All v0.11 REQs are Complete. The v0.12 requirements are net-new from
|
||||
the threat model -- no pending/in_progress REQs to close.
|
||||
|
||||
### 2.3 Verification layer inversion
|
||||
|
||||
- **Structural**: `internal/security/ca.go` (legacy CA) documented as
|
||||
deprecated but still compiled and load-bearing. -> F16.
|
||||
- **Behavioral**: `internal/runtime/podman.go`, `wasm.go` have no
|
||||
command-injection regression tests. -> F3.
|
||||
- **Security**: No STRIDE analysis for the OIDC/WebAuthn data flow
|
||||
(new in v0.12). -> addressed by REQ-142 (docs).
|
||||
- **Quality**: `classifyDialErr` substring matching is a known code
|
||||
smell flagged in v0.9 research. -> F25.
|
||||
|
||||
### 2.4 Architectural drift detection
|
||||
|
||||
- `internal/acl/` exists but is not wired into any enforcement point
|
||||
(documented as "future" since v0.9). -> F1.
|
||||
- `internal/identity/spiffe.go` `VerifySVID` skips chain validation
|
||||
(documented as "trust is implicit via SSH channel" in v0.11 P01.5
|
||||
spike result). -> F9.
|
||||
- `internal/emitter/nft.go` ships SYN-flood + rate-limit but no
|
||||
conntrack/default-deny (the v0.11 emitter met the REQ but not
|
||||
defense-in-depth best practice). -> F21.
|
||||
|
||||
### 2.5 Spec-driven improvement
|
||||
|
||||
- R-021 ("no Orca credentials") is the new spec invariant. Every
|
||||
existing password/token surface is a spec violation under R-021.
|
||||
-> F1, F12, F17, REQ-144..148.
|
||||
- The v0.11 PRD's deferred-v1.x list included "master.key
|
||||
passphrase-less 0600 (consider OS keyring in v1.x)." v0.12 closes
|
||||
this via seal-to-OIDC (no passphrase, no OS keyring dependency --
|
||||
OIDC is the unwrap mechanism).
|
||||
|
||||
## Tier 2 — Backend-enriched analysis
|
||||
|
||||
### 2.6 Prioritization
|
||||
|
||||
Ranked by (1) severity, (2) OS-surface exposure (per user instruction
|
||||
"includes the operating system itself"), (3) ease of addressing:
|
||||
|
||||
1. **F3 command injection** (Critical, OS-touching, shellQuote is a
|
||||
well-understood fix) -> P01.
|
||||
2. **F4 path traversal** (Critical, OS-touching, validateNamespaceName
|
||||
is trivial) -> P02.
|
||||
3. **F5 txn arbitrary paths** (Critical, OS-touching, prefix allowlist)
|
||||
-> P03.
|
||||
4. **F1 ACL unenforced** (Critical, foundational for OIDC authz) ->
|
||||
P06 (after P04/P05 identity).
|
||||
5. **F6 daemon no auth** (High, OS-touching) -> P09.
|
||||
6. **F2 audit not tamper-evident** (Critical, integrity) -> P10.
|
||||
7. **F9 SVID no chain** (High, identity) -> P11.
|
||||
8. **F7 backup symlink** (High, OS-touching) -> P12.
|
||||
9. **F10 step-ca /tmp** (High, OS-touching) -> P13.
|
||||
10. **F12 master key rotation** (High, crypto) -> P14.
|
||||
11. **F11 aggregate.sh JSON injection** (High, OS-touching) -> P16.
|
||||
12. **F14 install.sh no checksum** (High, OS-touching) -> P17.
|
||||
13. **F21 nftables** (Medium, OS-touching) -> P18.
|
||||
14. **F22 sudoers** (Medium, OS-touching) -> P19.
|
||||
15. **F23 system user** (Medium, OS-touching) -> P20.
|
||||
16. **F8 SQLite** (High, OS-touching) -> P21.
|
||||
17. **F19 migration** (Medium, OS-touching) -> P22.
|
||||
18. **F16 dual-write closure** (Medium, surface reduction) -> P23.
|
||||
19. **F15/F25 transport** (Medium/Low) -> P24.
|
||||
20. **F18 drift auth** (Medium) -> P25.
|
||||
21. **Integration tests** (gate) -> P26.
|
||||
22. **Docs** (gate) -> P27.
|
||||
23. **Final review** (gate) -> P28.
|
||||
|
||||
The zero-trust identity work (P04 OIDC+Dex, P05 WebAuthn, P07 password
|
||||
removal, P08 master key seal) is wave B because it's the architectural
|
||||
foundation -- P06 (ACL) and P09 (daemon auth) depend on it.
|
||||
|
||||
### 2.7 Novel improvement suggestions
|
||||
|
||||
- **WebAuthn as the bundled password-free authenticator** (operator
|
||||
decision D-240). This is beyond pattern matching -- it's the
|
||||
strongest available authentication primitive and directly satisfies
|
||||
R-021. The `go-webauthn` library is mature; the custom Dex connector
|
||||
is ~300 LoC.
|
||||
- **Shamir 3-of-5 master key recovery** (operator decision D-241).
|
||||
Standard threshold cryptography; no backdoor; documented residual
|
||||
risk.
|
||||
- **Bundled Dex** (operator decision D-239). Zero-trust out of the
|
||||
box without external setup; BYO override preserves flexibility.
|
||||
|
||||
### 2.8 Chaos engineering ideation
|
||||
|
||||
- **What if the OIDC provider is unavailable?** -> `orca cluster
|
||||
unseal` fails; cluster runs on in-memory master key until shutdown
|
||||
(no new secrets operations). Shamir recovery if permanent. Doc'd.
|
||||
- **What if a peer's drift event is forged?** -> REQ-140 (per-peer
|
||||
HMAC).
|
||||
- **What if the master key is compromised?** -> REQ-129 (rotation,
|
||||
re-seal to OIDC). All historical secrets still compromised (no
|
||||
forward secrecy) -- documented residual risk.
|
||||
- **What if install.sh is MITM'd?** -> REQ-132 (checksum+GPG).
|
||||
- **What if the Gitea token leaks again?** -> C-32 (human-gate
|
||||
rotation before final ship); history scrub best-effort.
|
||||
|
||||
## Tier 3 — Cross-project pattern transfer
|
||||
|
||||
Single-project mode (only `orca` in `active_projects`). No
|
||||
cross-project mining.
|
||||
|
||||
## Step 3 — Merge and deduplicate
|
||||
|
||||
30 ideas, all unique by `relatedReq` (REQ-119..REQ-148). No
|
||||
duplicates. Sorted by severity then wave order (see Tier 2.6).
|
||||
|
||||
## Step 4 — Interactive validation
|
||||
|
||||
Under `autonomy.level=full` + `workflow.no_hitl=true`, all 30 ideas
|
||||
are auto-accepted. The operator pre-approved the scope in the planning
|
||||
conversation (comprehensive coverage including OS surface, bundled
|
||||
Dex, WebAuthn, Shamir). 0 skipped, 0 modified.
|
||||
|
||||
## Step 5 — Long-term document updates
|
||||
|
||||
- `REQUIREMENTS.md`: REQ-119..REQ-148 added (done).
|
||||
- `ROADMAP.md`: v0.12 milestone section added (next).
|
||||
- `ARCHITECTURE.md`: zero-trust identity model + OIDC data-flow to be
|
||||
added in P27 (docs phase) -- not in Phase 0 to avoid scope creep.
|
||||
- `PROJECT.md`: v0.12 scope summary + D-238..D-247 added (done).
|
||||
|
||||
## Step 6 — Ask-after-validation kickoff
|
||||
|
||||
The run workflow continues to PLAN -> GRILL -> ship Phase 0 ->
|
||||
execute P01..P27 -> final P28. No separate kickoff needed (the
|
||||
`--ideate` flag is consumed; ideas are already in REQUIREMENTS.md +
|
||||
ROADMAP.md).
|
||||
+53
-53
@@ -1,6 +1,6 @@
|
||||
# Ideation v0.9 — Re-architecture Foundation
|
||||
|
||||
**Project**: orca (single-project mode) | **Milestone**: v0.9/v1.0 re-architecture
|
||||
**Project**: orca (single-project mode) | **Milestone**: v0.9/v0.10 re-architecture
|
||||
**Date**: 2026-08-05 | **Agent**: ideation agent | **Confidence threshold**: 0.60
|
||||
**Next REQ ID prior to this run**: REQ-060 (v0.8 complete)
|
||||
|
||||
@@ -12,7 +12,7 @@ with a CLI-only, SSH-push, step-ca, Markdown-frontmatter, multi-namespace
|
||||
stack. 9 packages are deprecation targets (~2,400 LOC of v0.8
|
||||
daemon/transport/security-ca/engine-dispatch/jobspec-hcl/config-hcl/certpaths
|
||||
code), 7 packages are adaptable, and 8 subsystems are net-new with zero
|
||||
implementation. The §23 milestone plan has 11 v0.9 phases + 17 v1.0 phases but
|
||||
implementation. The §23 milestone plan has 11 v0.9 phases + 17 v0.10 phases but
|
||||
under-specifies the deprecation mechanics, the SSH-push transport design, the
|
||||
lead-applier execution model, several adapter/bridge layers, and the
|
||||
migration ordering risk.
|
||||
@@ -25,10 +25,10 @@ the PRD §23 plan are listed at the end.
|
||||
|
||||
### I-M-001 — `orca daemon` deprecation command and build-tag removal path
|
||||
- **Tier**: mechanical
|
||||
- **Description**: The PRD deprecates `internal/daemon/` (R-001) but §23 never says *how*. `internal/cli/daemon.go` (100 LOC) registers the `daemon` cobra command and wires `daemon.NewServer` + `engine.Dispatcher`. Big-bang removal would break the v0.8→v1.0 migration path (v1.0-P14) because `orca upgrade --to-v1.0` must run against a live v0.8 cluster that still has daemons. Proposal: (1) in v0.9, `orca daemon` emits a deprecation warning and still runs (dual-write window); (2) in v1.0, `orca daemon` is repurposed to `orca daemon drain-and-stop` (stops v0.8 daemons on peers via SSH, confirms workloads survive via systemd); (3) post-v1.0, the command and `internal/daemon/` are deleted. Add `// Deprecated` Go doc comments + `slog.Warn` on every run.
|
||||
- **Description**: The PRD deprecates `internal/daemon/` (R-001) but §23 never says *how*. `internal/cli/daemon.go` (100 LOC) registers the `daemon` cobra command and wires `daemon.NewServer` + `engine.Dispatcher`. Big-bang removal would break the v0.8→v1.0 migration path (v0.10-P14) because `orca upgrade --to-v1.0` must run against a live v0.8 cluster that still has daemons. Proposal: (1) in v0.9, `orca daemon` emits a deprecation warning and still runs (dual-write window); (2) in v1.0, `orca daemon` is repurposed to `orca daemon drain-and-stop` (stops v0.8 daemons on peers via SSH, confirms workloads survive via systemd); (3) post-v1.0, the command and `internal/daemon/` are deleted. Add `// Deprecated` Go doc comments + `slog.Warn` on every run.
|
||||
- **Rationale**: R-001 is an invariant, but the *transition* off the daemon is a mechanical gap. The v0.8 `daemon.go` is wired in `root.go` init; removing it without a transition plan breaks the §24 migration.
|
||||
- **Proposed REQ ID**: REQ-061
|
||||
- **Proposed phase placement**: v1.0-P14 (migration) — deprecation warning lands in v0.9-P0X
|
||||
- **Proposed phase placement**: v0.10-P14 (migration) — deprecation warning lands in v0.9-P0X
|
||||
- **Confidence**: 0.82
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
@@ -61,25 +61,25 @@ the PRD §23 plan are listed at the end.
|
||||
|
||||
### I-M-005 — `orca doctor --legacy-paths` detection for v0.8 residue
|
||||
- **Tier**: mechanical
|
||||
- **Description**: The v0.8 layout is `~/.orca/{orca.db, ca.crt, ca.key, server.crt, server.key, orca_ssh_key, known_hosts, config.hcl}`. The v1.0 layout is `ORCA_HOME/{_defaults/, cluster/{ca,master.key,peers,pve,txns}, <ns>/{db,.env,.env.secrets,jobs,alloc,ns.md}, orca_cache.db}`. `orca doctor` (`internal/doctor/doctor.go`, 501 LOC, adaptable) must gain a `doctor legacy` subcommand that detects v0.8 residue: presence of `orca.db` at ORCA_HOME root, `ca.crt`/`ca.key` (internal CA, superseded by step-ca), `config.hcl` (HCL, demoted), flat `server.crt` (single-namespace), and a `namespace` column in any `*.db` (R-002 says no namespace column). Output: list of detected legacy artifacts with migration recommendations. This is the *detection* half of v1.0-P14; the *migration* half is I-C-001.
|
||||
- **Rationale**: §23 v1.0-P14 says "orca upgrade --to-v1.0, post-invariant checks" but doesn't specify the detection surface. `doctor` is the diagnostics framework and is explicitly adaptable.
|
||||
- **Description**: The v0.8 layout is `~/.orca/{orca.db, ca.crt, ca.key, server.crt, server.key, orca_ssh_key, known_hosts, config.hcl}`. The v1.0 layout is `ORCA_HOME/{_defaults/, cluster/{ca,master.key,peers,pve,txns}, <ns>/{db,.env,.env.secrets,jobs,alloc,ns.md}, orca_cache.db}`. `orca doctor` (`internal/doctor/doctor.go`, 501 LOC, adaptable) must gain a `doctor legacy` subcommand that detects v0.8 residue: presence of `orca.db` at ORCA_HOME root, `ca.crt`/`ca.key` (internal CA, superseded by step-ca), `config.hcl` (HCL, demoted), flat `server.crt` (single-namespace), and a `namespace` column in any `*.db` (R-002 says no namespace column). Output: list of detected legacy artifacts with migration recommendations. This is the *detection* half of v0.10-P14; the *migration* half is I-C-001.
|
||||
- **Rationale**: §23 v0.10-P14 says "orca upgrade --to-v1.0, post-invariant checks" but doesn't specify the detection surface. `doctor` is the diagnostics framework and is explicitly adaptable.
|
||||
- **Proposed REQ ID**: REQ-065
|
||||
- **Proposed phase placement**: v1.0-P14c (mixed-version tolerance + no-orca enforcement)
|
||||
- **Proposed phase placement**: v0.10-P14c (mixed-version tolerance + no-orca enforcement)
|
||||
- **Confidence**: 0.80
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-M-006 — Legacy CA state migration to step-ca (cert import)
|
||||
- **Tier**: mechanical
|
||||
- **Description**: `internal/security/ca.go` (338 LOC) holds an internal Go CA with `ca.crt`/`ca.key` (RSA 3072, 10-year). The PRD replaces this with step-ca (R-006, D-101 reverses AD-010). The v1.0-P14 migration must handle existing deployments with an internal CA: (a) import the existing CA key into step-ca as `step ca init --deployment-type standalone --remote-management` with the existing key; (b) issue new SVIDs from step-ca and let old certs expire; (c) document that v0.8 certs are invalidated and re-bootstrap is required. The codebase audit says `ca.go`+`csr.go` are *replaced* — but the *state* (the CA key + issued server certs in `cert_repo` SQLite) may need to be preserved for audit history even if the live trust root changes. Proposal: `orca upgrade --to-v1.0 --import-ca` reads `~/.orca/ca.key`, initializes step-ca with it, and re-issues workload SVIDs. Without this, existing deployments lose their trust root with no path back.
|
||||
- **Description**: `internal/security/ca.go` (338 LOC) holds an internal Go CA with `ca.crt`/`ca.key` (RSA 3072, 10-year). The PRD replaces this with step-ca (R-006, D-101 reverses AD-010). The v0.10-P14 migration must handle existing deployments with an internal CA: (a) import the existing CA key into step-ca as `step ca init --deployment-type standalone --remote-management` with the existing key; (b) issue new SVIDs from step-ca and let old certs expire; (c) document that v0.8 certs are invalidated and re-bootstrap is required. The codebase audit says `ca.go`+`csr.go` are *replaced* — but the *state* (the CA key + issued server certs in `cert_repo` SQLite) may need to be preserved for audit history even if the live trust root changes. Proposal: `orca upgrade --to-v1.0 --import-ca` reads `~/.orca/ca.key`, initializes step-ca with it, and re-issues workload SVIDs. Without this, existing deployments lose their trust root with no path back.
|
||||
- **Rationale**: AD-010 is explicitly reversed by D-101, but the reversal doesn't address what happens to the existing CA material. §24 covers data migration but not CA migration.
|
||||
- **Proposed REQ ID**: REQ-066
|
||||
- **Proposed phase placement**: v1.0-P14a (data migration)
|
||||
- **Proposed phase placement**: v0.10-P14a (data migration)
|
||||
- **Confidence**: 0.70
|
||||
- **Accept/Defer**: accept (design in v0.9-P00 so step-ca integration knows the import contract)
|
||||
|
||||
### I-M-007 — Fuzz test harness for the Markdown frontmatter parser
|
||||
- **Tier**: mechanical
|
||||
- **Description**: R-014/R-015 require byte-exact body preservation — "body of every .md config file preserved verbatim." This is a class of bug that's easy to get wrong (off-by-one on the `---` delimiter, trailing newline handling, BOM, CRLF, nested code fences containing `---`). v0.8 has no fuzz tests at all. Proposal: add a `testing.F` fuzz target in `internal/jobspec/markdown_test.go` that round-trips random frontmatter+body through `ParseMarkdown` and asserts `body == roundtripped.body` byte-exact. Also add a corpus of adversarial fixtures (CRLF, BOM, no-frontmatter, empty-frontmatter, frontmatter-with-only-separator). §23 v1.0-P08 mentions integration tests but not fuzzing.
|
||||
- **Description**: R-014/R-015 require byte-exact body preservation — "body of every .md config file preserved verbatim." This is a class of bug that's easy to get wrong (off-by-one on the `---` delimiter, trailing newline handling, BOM, CRLF, nested code fences containing `---`). v0.8 has no fuzz tests at all. Proposal: add a `testing.F` fuzz target in `internal/jobspec/markdown_test.go` that round-trips random frontmatter+body through `ParseMarkdown` and asserts `body == roundtripped.body` byte-exact. Also add a corpus of adversarial fixtures (CRLF, BOM, no-frontmatter, empty-frontmatter, frontmatter-with-only-separator). §23 v0.10-P08 mentions integration tests but not fuzzing.
|
||||
- **Rationale**: R-015 is a *load-bearing invariant* (body appears in inspect/history). Byte-exactness is exactly what fuzz tests are for. The v0.8 jobspec tests are golden-file only (no fuzz).
|
||||
- **Proposed REQ ID**: REQ-067
|
||||
- **Proposed phase placement**: v0.9-P0b (Markdown parser) — fuzz from day one
|
||||
@@ -89,9 +89,9 @@ the PRD §23 plan are listed at the end.
|
||||
### I-M-008 — Deprecation warnings on removed/repurposed CLI subcommands
|
||||
- **Tier**: mechanical
|
||||
- **Description**: The v0.8 CLI has `orca cert {ca-init,gen,show,renew,fingerprint}` (`internal/cli/cert.go`), `orca node join` with mTLS handshake semantics (`internal/cli/node.go`), `orca job run <spec.hcl>`. The PRD repurposes `node join` to SSH-bootstrap (no mTLS), deprecates `cert` (step-ca handles it), and changes `job run` to accept `.md` specs. Each removed/changed command should emit a `slog.Warn` deprecation banner with the v1.0 replacement, *except* when run under `orca upgrade`. The existing `root.go` `PersistentPreRunE` is the natural hook for a global `--no-deprecation-warnings` flag.
|
||||
- **Rationale**: Operators running v0.8 commands against v0.9/v1.0 need to know what changed. The PRD doesn't mention deprecation UX.
|
||||
- **Rationale**: Operators running v0.8 commands against v0.9/v0.10 need to know what changed. The PRD doesn't mention deprecation UX.
|
||||
- **Proposed REQ ID**: REQ-068
|
||||
- **Proposed phase placement**: v0.9-P0X (ship) + v1.0-P13 (ns subcommands, when CLI surface is finalized)
|
||||
- **Proposed phase placement**: v0.9-P0X (ship) + v0.10-P13 (ns subcommands, when CLI surface is finalized)
|
||||
- **Confidence**: 0.72
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
@@ -115,10 +115,10 @@ the PRD §23 plan are listed at the end.
|
||||
|
||||
### I-M-011 — `internal/store/` schema: per-namespace DBs, drop ns column
|
||||
- **Tier**: mechanical
|
||||
- **Description**: R-002 says "No `namespace` column in SQLite." The v0.8 schema has 7 migrations (`0001`..`0007`) with a single `orca.db`. The v1.0 model has one DB per namespace (`<ns>/db/orca.db`) plus a CLI-side cache DB (`orca_cache.db`, R-008). The existing `store.Open(path)` takes a path arg — adaptable. But the migrations are global; they need to apply *per namespace DB*. Proposal: `store.Open` gains a namespace parameter (or caller passes `paths.NSDb(ns)`); `migrate.go` runs `0001`..`0007` (minus `0006_node_kind_os` which is v0.8-specific) plus new `0008_namespace_layout.sql`. The `cert_repo` (`0004_certs.sql`) is removed (step-ca handles certs). The audit_log table moves to the CLI-side cache DB (R-008). Existing v0.8 `orca.db` is migrated by splitting tables into per-namespace DBs during v1.0-P14.
|
||||
- **Description**: R-002 says "No `namespace` column in SQLite." The v0.8 schema has 7 migrations (`0001`..`0007`) with a single `orca.db`. The v0.10 model has one DB per namespace (`<ns>/db/orca.db`) plus a CLI-side cache DB (`orca_cache.db`, R-008). The existing `store.Open(path)` takes a path arg — adaptable. But the migrations are global; they need to apply *per namespace DB*. Proposal: `store.Open` gains a namespace parameter (or caller passes `paths.NSDb(ns)`); `migrate.go` runs `0001`..`0007` (minus `0006_node_kind_os` which is v0.8-specific) plus new `0008_namespace_layout.sql`. The `cert_repo` (`0004_certs.sql`) is removed (step-ca handles certs). The audit_log table moves to the CLI-side cache DB (R-008). Existing v0.8 `orca.db` is migrated by splitting tables into per-namespace DBs during v0.10-P14.
|
||||
- **Rationale**: R-002 is explicit ("No namespace column in SQLite") but the existing schema has a single DB. §23 doesn't specify the schema split mechanics.
|
||||
- **Proposed REQ ID**: REQ-071
|
||||
- **Proposed phase placement**: v0.9-P0a1 + v1.0-P06 (alloc history, which uses cache DB)
|
||||
- **Proposed phase placement**: v0.9-P0a1 + v0.10-P06 (alloc history, which uses cache DB)
|
||||
- **Confidence**: 0.80
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
@@ -127,9 +127,9 @@ the PRD §23 plan are listed at the end.
|
||||
- **Description**: `internal/transport/` (7 files, ~1300 LOC incl tests) implements mTLS client/server, dispatch, idempotency, retry, handshake logging. R-001 + R-006 replace this with SSH-push. The *idempotency* and *retry* logic (`idempotency.go` 123 LOC, `retry.go` 151 LOC) is conceptually reusable for SSH-push (retry on SSH failure, idempotency keys for SCP'd configs). Proposal: delete `mtls.go`, `dispatch.go`, `handshake_log.go`; extract retry/idempotency patterns into a new `internal/sshpush/` package. The existing `transport.IdempotencyStore` (in-memory `sync.Map` of keys) is directly reusable. This avoids re-implementing retry semantics from scratch.
|
||||
- **Rationale**: The codebase audit marks `internal/transport/` as fully replaced, but the retry/idempotency *patterns* are transport-agnostic. §23 doesn't call this out.
|
||||
- **Proposed REQ ID**: REQ-072
|
||||
- **Proposed phase placement**: v0.9-P00 (deprecation sweep) — delete in v1.0-P14
|
||||
- **Proposed phase placement**: v0.9-P00 (deprecation sweep) — delete in v0.10-P14
|
||||
- **Confidence**: 0.68
|
||||
- **Accept/Defer**: accept (defer deletion to v1.0-P14 to keep dual-write window open)
|
||||
- **Accept/Defer**: accept (defer deletion to v0.10-P14 to keep dual-write window open)
|
||||
|
||||
## Tier 2 — Backend-Enriched (Structural / Architectural)
|
||||
|
||||
@@ -156,7 +156,7 @@ the PRD §23 plan are listed at the end.
|
||||
- **Description**: R-001 says "no orca binary on servers." R-010 says the lead applies desired-state transactionally. Unresolved: does the lead run `orca-pull.sh` (pure bash that SCPs a desired-state bundle and applies it via `systemctl daemon-reload` + `systemctl restart`) or does the operator's CLI SSH into the lead and runs `orca apply` remotely (which would put an orca binary on the lead, violating R-001)? The PRD's intent is the former: the lead is bare Linux with systemd timers + bash. Proposal: (1) the CLI renders a *transaction bundle* (tarball of desired-state files + `apply.sh` + `verify.sh`) on the operator host; (2) SCPs it to the lead's `/run/orca/txns/<txn-id>/`; (3) the lead's systemd timer runs `/run/orca/txns/<txn-id>/apply.sh` which idempotently applies and runs verify; (4) the CLI polls the lead for txn status via SSH (`cat /run/orca/txns/<txn-id>/status.json`). The bash scripts are generated by the CLI's emitter (I-B-002), not hand-written per cluster.
|
||||
- **Rationale**: The most ambiguous load-bearing design decision in the PRD. R-001 + R-010 together imply the lead runs no orca binary, but the lead must apply transactions. §23 doesn't resolve this. Getting it wrong means either violating R-001 or having no transactional apply.
|
||||
- **Proposed REQ ID**: REQ-075
|
||||
- **Proposed phase placement**: v1.0-P10 (transactional plane) — bundle format designed in v0.9-P00
|
||||
- **Proposed phase placement**: v0.10-P10 (transactional plane) — bundle format designed in v0.9-P00
|
||||
- **Confidence**: 0.78
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
@@ -165,13 +165,13 @@ the PRD §23 plan are listed at the end.
|
||||
- **Description**: D-101 reverses AD-010 (which rejected step-ca as "too heavyweight"). §23 mentions step-ca in R-006 but never specifies the integration. Key surfaces: (1) **Provisioning**: `orca init` (adapted from v0.8's `internal/cli/init.go`) runs `step ca init` on the lead, stores root + intermediate in `cluster/ca/`. (2) **CA bootstrap**: CLI SSHs to the lead, installs step-ca via apt, runs `step ca init`, stores `step-ca.json` config. (3) **Cert signing API**: workloads request SVIDs via `step ca token` (JWE provisioner token minted by CLI) → `step ca certificate`. The CLI mints the token because it holds the provisioner password (in `cluster/master.key`-derived form). (4) **SVID minting**: each workload gets a SPIFFE ID (`spiffe://orca/<ns>/<workload>/<instance>`) encoded as a SAN in the step-ca-issued cert. The v0.8 `internal/security/ca.go` is deleted; a new `internal/stepca/` package wraps the `step` CLI via SSH (no Go step-ca client library — keep zero-new-dep posture if possible, or add `github.com/smallstep/cli` as a dep).
|
||||
- **Rationale**: step-ca is a new external dependency with its own config format, provisioner model, and CLI. §23 assumes it but never designs the integration. security-engineer persona must be reactivated.
|
||||
- **Proposed REQ ID**: REQ-076
|
||||
- **Proposed phase placement**: v0.9-P07 (runtime block — runtimes need SVIDs) + v1.0-P02 (ACL — SPIFFE identities)
|
||||
- **Proposed phase placement**: v0.9-P07 (runtime block — runtimes need SVIDs) + v0.10-P02 (ACL — SPIFFE identities)
|
||||
- **Confidence**: 0.74
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-B-005 — Traefik dynamic config generation and atomic reload
|
||||
- **Tier**: backend-enriched
|
||||
- **Description**: R-006 makes Traefik load-bearing (mTLS termination + health checks). §23 puts service blocks + Traefik health checks in v0.9-P02. Design: the CLI's Traefik emitter (I-B-002) renders a dynamic config file (`/etc/traefik/dynamic/orca-<ns>-<svc>.yaml`) with backends (the socket paths from R-007), health checks, and mTLS config pointing at step-ca's root. Atomic reload: Traefik watches the dynamic dir with `fsnotify` — writing the file atomically (tmp+rename) triggers a reload. Drain (v1.0-P05) works by writing a config with the backend's `weight=0` or removing it, triggering Traefik to stop routing. The v0.8 codebase has no Traefik integration at all. **Gated by grill C-10** (Traefik config atomicity protocol: tmpfile+fsync+rename + malformed-config hold-last-good verified).
|
||||
- **Description**: R-006 makes Traefik load-bearing (mTLS termination + health checks). §23 puts service blocks + Traefik health checks in v0.9-P02. Design: the CLI's Traefik emitter (I-B-002) renders a dynamic config file (`/etc/traefik/dynamic/orca-<ns>-<svc>.yaml`) with backends (the socket paths from R-007), health checks, and mTLS config pointing at step-ca's root. Atomic reload: Traefik watches the dynamic dir with `fsnotify` — writing the file atomically (tmp+rename) triggers a reload. Drain (v0.10-P05) works by writing a config with the backend's `weight=0` or removing it, triggering Traefik to stop routing. The v0.8 codebase has no Traefik integration at all. **Gated by grill C-10** (Traefik config atomicity protocol: tmpfile+fsync+rename + malformed-config hold-last-good verified).
|
||||
- **Rationale**: Traefik is net-new and load-bearing. §23 mentions it in R-006/P02/P05 but never specifies config generation or reload mechanism.
|
||||
- **Proposed REQ ID**: REQ-077
|
||||
- **Proposed phase placement**: v0.9-P02 (Service block + checks)
|
||||
@@ -189,19 +189,19 @@ the PRD §23 plan are listed at the end.
|
||||
|
||||
### I-B-007 — Transaction bundle format and atomicity across N peers
|
||||
- **Tier**: backend-enriched
|
||||
- **Description**: R-010 requires transactional control-plane updates. §23 puts this in v1.0-P10. Design: a *transaction bundle* is a tarball containing: (1) `desired-state.json` (full desired state for affected namespaces), (2) `apply.sh` (idempotent apply script), (3) `verify.sh` (post-apply invariants), (4) `rollback.sh` (revert to previous state), (5) `manifest.sig` (signature with `cluster/master.key`). Atomicity across N peers: the CLI uploads the bundle to the lead; the lead applies to itself first, then fans out to peers via SSH. If any peer fails verify, the lead runs `rollback.sh` on all peers that applied. The bundle is content-addressed (`<txn-id> = sha256(desired-state.json)`) and stored in `cluster/txns/<txn-id>/`. Drift detection (R-010) compares the last applied bundle's desired-state against the live state (polled via SSH `systemctl show` + file checksums). **Gated by grill C-09** (orca-pull.sh failure contract: idempotent re-run, bounded retry, deterministic state, structured syslog).
|
||||
- **Description**: R-010 requires transactional control-plane updates. §23 puts this in v0.10-P10. Design: a *transaction bundle* is a tarball containing: (1) `desired-state.json` (full desired state for affected namespaces), (2) `apply.sh` (idempotent apply script), (3) `verify.sh` (post-apply invariants), (4) `rollback.sh` (revert to previous state), (5) `manifest.sig` (signature with `cluster/master.key`). Atomicity across N peers: the CLI uploads the bundle to the lead; the lead applies to itself first, then fans out to peers via SSH. If any peer fails verify, the lead runs `rollback.sh` on all peers that applied. The bundle is content-addressed (`<txn-id> = sha256(desired-state.json)`) and stored in `cluster/txns/<txn-id>/`. Drift detection (R-010) compares the last applied bundle's desired-state against the live state (polled via SSH `systemctl show` + file checksums). **Gated by grill C-09** (orca-pull.sh failure contract: idempotent re-run, bounded retry, deterministic state, structured syslog).
|
||||
- **Rationale**: Multi-peer atomicity is the hardest part of R-010. §23 says "ArgoCD-style" but ArgoCD is Kubernetes-native; the SSH-push model needs a custom bundle format.
|
||||
- **Proposed REQ ID**: REQ-079
|
||||
- **Proposed phase placement**: v1.0-P10 (transactional plane) — designed in v0.9-P00
|
||||
- **Proposed phase placement**: v0.10-P10 (transactional plane) — designed in v0.9-P00
|
||||
- **Confidence**: 0.76
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-B-008 — Master key management and HKDF-SHA256 per-line .env.secrets encryption
|
||||
- **Tier**: backend-enriched
|
||||
- **Description**: R-011 specifies `.env.secrets` with AES-256-GCM, per-line nonce, master key at `cluster/master.key`. §23 puts this in v1.0-P03. Design: (1) `cluster/master.key` is a 32-byte random key generated by `orca init` (extend v0.8 `internal/security/ca.go`'s `WriteAtomic` pattern for the file write). (2) Each line of `.env.secrets` is `base64(nonce || ciphertext || tag)` where `nonce = random(12 bytes)` and `ciphertext = AES-256-GCM(plaintext, key=master.key, nonce, aad=line-number)`. (3) The AAD is the 1-indexed line number to prevent line-swap attacks. (4) Decryption reads the master key, iterates lines, decrypts with AAD. (5) `orca secrets set <ns> <key> <value>` appends an encrypted line; `orca secrets get <ns> <key>` decrypts and prints (redacted by default, `--reveal` to show). (6) The v0.8 `internal/security/redact.go` (103 LOC) is directly reusable for redaction. HKDF-SHA256 derives per-namespace sub-keys from the master key (`HKDF-SHA256(master, info=<ns>)`) so compromising one namespace's key doesn't compromise others — but the master key is the root of trust. **Gated by grill C-19** (threat model for master.key passphrase-less posture).
|
||||
- **Description**: R-011 specifies `.env.secrets` with AES-256-GCM, per-line nonce, master key at `cluster/master.key`. §23 puts this in v0.10-P03. Design: (1) `cluster/master.key` is a 32-byte random key generated by `orca init` (extend v0.8 `internal/security/ca.go`'s `WriteAtomic` pattern for the file write). (2) Each line of `.env.secrets` is `base64(nonce || ciphertext || tag)` where `nonce = random(12 bytes)` and `ciphertext = AES-256-GCM(plaintext, key=master.key, nonce, aad=line-number)`. (3) The AAD is the 1-indexed line number to prevent line-swap attacks. (4) Decryption reads the master key, iterates lines, decrypts with AAD. (5) `orca secrets set <ns> <key> <value>` appends an encrypted line; `orca secrets get <ns> <key>` decrypts and prints (redacted by default, `--reveal` to show). (6) The v0.8 `internal/security/redact.go` (103 LOC) is directly reusable for redaction. HKDF-SHA256 derives per-namespace sub-keys from the master key (`HKDF-SHA256(master, info=<ns>)`) so compromising one namespace's key doesn't compromise others — but the master key is the root of trust. **Gated by grill C-19** (threat model for master.key passphrase-less posture).
|
||||
- **Rationale**: R-011 is precise about the crypto but §23 doesn't specify key derivation, AAD, or CLI surface. The existing `redact.go` and `WriteAtomic` are reusable.
|
||||
- **Proposed REQ ID**: REQ-080
|
||||
- **Proposed phase placement**: v1.0-P03 (secrets subsystem)
|
||||
- **Proposed phase placement**: v0.10-P03 (secrets subsystem)
|
||||
- **Confidence**: 0.84
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
@@ -234,10 +234,10 @@ the PRD §23 plan are listed at the end.
|
||||
|
||||
### I-B-012 — `orca job lint` category-driven lint engine design
|
||||
- **Tier**: backend-enriched
|
||||
- **Description**: v1.0-P11 requires `orca job lint` with `--explain`. Design: a `Linter` that takes a `*WorkloadSpec` and runs a series of `Rule` checks, each returning a `Finding{Category, Severity, Message, Explanation}`. Categories: `schema` (missing required fields), `runtime` (incompatible runtime+constraint), `security` (missing SVID, plaintext secret in env), `migration` (missing storage replication for a migratable service), `best-practice` (no health check on a Service). `--explain` prints the rationale for each finding. Rules are registered in a `ruleRegistry` and individually testable. The linter is pure (no I/O) — it checks the spec against static rules, not live cluster state (that's `orca job verify`, P12).
|
||||
- **Description**: v0.10-P11 requires `orca job lint` with `--explain`. Design: a `Linter` that takes a `*WorkloadSpec` and runs a series of `Rule` checks, each returning a `Finding{Category, Severity, Message, Explanation}`. Categories: `schema` (missing required fields), `runtime` (incompatible runtime+constraint), `security` (missing SVID, plaintext secret in env), `migration` (missing storage replication for a migratable service), `best-practice` (no health check on a Service). `--explain` prints the rationale for each finding. Rules are registered in a `ruleRegistry` and individually testable. The linter is pure (no I/O) — it checks the spec against static rules, not live cluster state (that's `orca job verify`, P12).
|
||||
- **Rationale**: §23 puts this in P11 but only says "category-driven." The rule interface and category taxonomy are unspecified.
|
||||
- **Proposed REQ ID**: REQ-084
|
||||
- **Proposed phase placement**: v1.0-P11 (orca job lint)
|
||||
- **Proposed phase placement**: v0.10-P11 (orca job lint)
|
||||
- **Confidence**: 0.78
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
@@ -245,28 +245,28 @@ the PRD §23 plan are listed at the end.
|
||||
|
||||
### I-C-001 — v0.8→v1.0 migration ordering: daemon deprecation vs. new model rollout
|
||||
- **Tier**: cross-cutting
|
||||
- **Description**: The PRD §24 covers *data* migration but not *binary/daemon* deprecation ordering. The risk: v0.9 builds the new Markdown+kinds+runtime+SSH-push model, but v0.8 daemons are still running on peers. If v0.9 ships the new `orca job run` (Markdown) while the old daemon is still the execution engine, there's a split-brain: new specs can't run on the old daemon. Ordering proposal: (1) v0.9 ships the new parser + kinds + runtime + SSH-push *alongside* the old daemon (dual-write window); (2) `orca job run` in v0.9 uses the new SSH-push path if the spec is `.md` and the old daemon path if `.hcl`; (3) v1.0-P05 (drain) stops the old daemons; (4) v1.0-P14 (migration) converts remaining `.hcl` specs to `.md` and removes the daemon. The dual-write window means v0.9 is *not* a clean break — it's a compatibility milestone. This must be explicit in the plan or the v0.9 phases will assume the daemon is gone.
|
||||
- **Description**: The PRD §24 covers *data* migration but not *binary/daemon* deprecation ordering. The risk: v0.9 builds the new Markdown+kinds+runtime+SSH-push model, but v0.8 daemons are still running on peers. If v0.9 ships the new `orca job run` (Markdown) while the old daemon is still the execution engine, there's a split-brain: new specs can't run on the old daemon. Ordering proposal: (1) v0.9 ships the new parser + kinds + runtime + SSH-push *alongside* the old daemon (dual-write window); (2) `orca job run` in v0.9 uses the new SSH-push path if the spec is `.md` and the old daemon path if `.hcl`; (3) v0.10-P05 (drain) stops the old daemons; (4) v0.10-P14 (migration) converts remaining `.hcl` specs to `.md` and removes the daemon. The dual-write window means v0.9 is *not* a clean break — it's a compatibility milestone. This must be explicit in the plan or the v0.9 phases will assume the daemon is gone.
|
||||
- **Rationale**: Single largest risk in the re-architecture. §23 implicitly assumes v0.9 builds the new model in isolation, but existing deployments have running daemons. Getting the ordering wrong means either (a) v0.9 can't be tested against real deployments, or (b) workloads are orphaned when the daemon is removed.
|
||||
- **Proposed REQ ID**: REQ-085
|
||||
- **Proposed phase placement**: spans v0.9-P00 through v1.0-P14 — the *ordering decision* must be made in v0.9-P00
|
||||
- **Proposed phase placement**: spans v0.9-P00 through v0.10-P14 — the *ordering decision* must be made in v0.9-P00
|
||||
- **Confidence**: 0.88
|
||||
- **Accept/Defer**: accept (most important idea in this report)
|
||||
|
||||
### I-C-002 — "No orca on server" enforcement (doctor post-migration invariant check)
|
||||
- **Tier**: cross-cutting
|
||||
- **Description**: R-001 is an invariant: "no orca Go binary on any server." §23 v1.0-P14 says "post-invariant checks" but doesn't specify them. `orca doctor` must gain a `doctor no-orca-on-server` check that SSHs to each peer and verifies: (1) no `orca` binary in PATH (`ssh peer which orca` returns nothing), (2) no `orca` systemd service (`ssh peer systemctl list-units 'orca*'` returns empty), (3) no `orca` process (`ssh peer pgrep -x orca` returns empty), (4) no `/etc/orca/` directory. This check must run *after* v1.0-P05 (drain) and *before* v1.0-P16 (ship). The v0.8 `internal/proxmox/bootstrap.go` already has the SSH session infrastructure (`sessionRunner` seam) — directly reusable for the doctor check.
|
||||
- **Description**: R-001 is an invariant: "no orca Go binary on any server." §23 v0.10-P14 says "post-invariant checks" but doesn't specify them. `orca doctor` must gain a `doctor no-orca-on-server` check that SSHs to each peer and verifies: (1) no `orca` binary in PATH (`ssh peer which orca` returns nothing), (2) no `orca` systemd service (`ssh peer systemctl list-units 'orca*'` returns empty), (3) no `orca` process (`ssh peer pgrep -x orca` returns empty), (4) no `/etc/orca/` directory. This check must run *after* v0.10-P05 (drain) and *before* v0.10-P16 (ship). The v0.8 `internal/proxmox/bootstrap.go` already has the SSH session infrastructure (`sessionRunner` seam) — directly reusable for the doctor check.
|
||||
- **Rationale**: R-001 is a hard invariant but §23 doesn't enforce it post-migration. Without this check, a failed migration could leave orphaned daemons that cause split-brain.
|
||||
- **Proposed REQ ID**: REQ-086
|
||||
- **Proposed phase placement**: v1.0-P14c (mixed-version tolerance)
|
||||
- **Proposed phase placement**: v0.10-P14c (mixed-version tolerance)
|
||||
- **Confidence**: 0.82
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-C-003 — Test infrastructure: hermetic 3-linux + 1-proxmox cluster pipeline
|
||||
- **Tier**: cross-cutting
|
||||
- **Description**: §23 v1.0-P08 requires "hermetic CoreCI integration pipeline." The PRD §26.E mentions 3 linux + 1 proxmox. This is net-new test infra with zero current implementation. Design: (1) a `test/integration/` directory with a `docker-compose.yml` or `vagrant` setup that creates 4 containers/VMs (3 linux + 1 proxmox-simulated); (2) a Go test harness that SSHes to each, runs the CLI, and asserts end-to-end workflows (namespace create → workload submit → migrate → drain); (3) the proxmox node is simulated via a mock `pct`/`qm` script (the v0.8 `proxmox` package already has a `sessionRunner` seam for testability — extend it). The integration tests run in CoreCI on every milestone merge. The v0.8 e2e tests (`bootstrapE2ESetup` in `bootstrap_test.go`) use an in-process SSH server — this is the foundation but needs to scale to 4 nodes.
|
||||
- **Description**: §23 v0.10-P08 requires "hermetic CoreCI integration pipeline." The PRD §26.E mentions 3 linux + 1 proxmox. This is net-new test infra with zero current implementation. Design: (1) a `test/integration/` directory with a `docker-compose.yml` or `vagrant` setup that creates 4 containers/VMs (3 linux + 1 proxmox-simulated); (2) a Go test harness that SSHes to each, runs the CLI, and asserts end-to-end workflows (namespace create → workload submit → migrate → drain); (3) the proxmox node is simulated via a mock `pct`/`qm` script (the v0.8 `proxmox` package already has a `sessionRunner` seam for testability — extend it). The integration tests run in CoreCI on every milestone merge. The v0.8 e2e tests (`bootstrapE2ESetup` in `bootstrap_test.go`) use an in-process SSH server — this is the foundation but needs to scale to 4 nodes.
|
||||
- **Rationale**: §23 assumes the infra exists but doesn't design it. devops-engineer persona should be reactivated. Without hermetic infra, the integration tests can't run in CI.
|
||||
- **Proposed REQ ID**: REQ-087
|
||||
- **Proposed phase placement**: v1.0-P08 (integration tests) — harness bootstrapped in v0.9-P00
|
||||
- **Proposed phase placement**: v0.10-P08 (integration tests) — harness bootstrapped in v0.9-P00
|
||||
- **Confidence**: 0.80
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
@@ -275,22 +275,22 @@ the PRD §23 plan are listed at the end.
|
||||
- **Description**: The config.json has `security-engineer` and `network-engineer` dormant. The re-architecture introduces step-ca (PKI), Traefik (edge proxy), Syncthing (P2P file sync), wasmtime (sandbox), podman (container runtime) — all new attack surfaces. AD-010 (step-ca rejection) is reversed. The v0.8 security posture (internal CA, mTLS daemon-to-daemon) is replaced by (step-ca, SSH-push, Traefik mTLS). The security-engineer persona must be reactivated to review: (1) step-ca provisioner model (the CLI holds the provisioner password — is that in `cluster/master.key` or a separate secret?), (2) SSH-push blast radius (compromised CLI key = full cluster), (3) Traefik as the new edge (DoS, config injection), (4) `.env.secrets` crypto (I-B-008). The network-engineer persona must review: (1) socket-based service exposure (R-007), (2) Syncthing P2P ports, (3) Traefik routing. §23 doesn't mention persona reactivation.
|
||||
- **Rationale**: config.json explicitly notes the re-architecture "should reactivate security-engineer and network-engineer." Cross-cutting review concern, not a single phase.
|
||||
- **Proposed REQ ID**: REQ-088
|
||||
- **Proposed phase placement**: spans v0.9 through v1.0 — reactivation in v0.9-P00, review at v1.0-P15.5 (threat model) and v1.0-P16 (final audit)
|
||||
- **Proposed phase placement**: spans v0.9 through v0.10 — reactivation in v0.9-P00, review at v0.10-P15.5 (threat model) and v0.10-P16 (final audit)
|
||||
- **Confidence**: 0.84
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-C-005 — Documentation rewrite: ARCHITECTURE.md, PROJECT.md, README, AD-010 supersession
|
||||
- **Tier**: cross-cutting
|
||||
- **Description**: All three docs describe the OLD architecture. `ARCHITECTURE.md` (640 lines) describes the daemon layer, mTLS transport, internal CA, HCL jobspec — all deprecated. `PROJECT.md` (30k chars) has D-001..D-010 decisions, several now superseded. `README.md` has the v0.8 quickstart. AD-010 (step-ca rejection) must be explicitly superseded by D-101 with a dated rationale reversal. The anti-patterns section in `ARCHITECTURE.md:471-484` lists "No external PKI" — now reversed. Proposal: (1) in v0.9-P00, add a "v0.9 Architecture (Supersedes v0.8)" section to ARCHITECTURE.md with the new 4-layer model; (2) mark the old sections as "v0.8 (deprecated)" with banners; (3) add a "Superseded Decisions" table (AD-009, AD-010 reversed by D-101; AD-007 HCL demoted by R-013); (4) in v1.0-P15, rewrite README quickstart for the new `curl | sh` + `orca init` + `orca ns create` flow.
|
||||
- **Description**: All three docs describe the OLD architecture. `ARCHITECTURE.md` (640 lines) describes the daemon layer, mTLS transport, internal CA, HCL jobspec — all deprecated. `PROJECT.md` (30k chars) has D-001..D-010 decisions, several now superseded. `README.md` has the v0.8 quickstart. AD-010 (step-ca rejection) must be explicitly superseded by D-101 with a dated rationale reversal. The anti-patterns section in `ARCHITECTURE.md:471-484` lists "No external PKI" — now reversed. Proposal: (1) in v0.9-P00, add a "v0.9 Architecture (Supersedes v0.8)" section to ARCHITECTURE.md with the new 4-layer model; (2) mark the old sections as "v0.8 (deprecated)" with banners; (3) add a "Superseded Decisions" table (AD-009, AD-010 reversed by D-101; AD-007 HCL demoted by R-013); (4) in v0.10-P15, rewrite README quickstart for the new `curl | sh` + `orca init` + `orca ns create` flow.
|
||||
- **Rationale**: The docs are the first thing new contributors read. Leaving v0.8 docs as canonical during v0.9 development causes confusion. §23 mentions README in P15 but not ARCHITECTURE.md/PROJECT.md.
|
||||
- **Proposed REQ ID**: REQ-089
|
||||
- **Proposed phase placement**: v0.9-P00 (banners + supersession table) + v1.0-P15 (README quickstart) + v1.0-P16 (final review)
|
||||
- **Proposed phase placement**: v0.9-P00 (banners + supersession table) + v0.10-P15 (README quickstart) + v0.10-P16 (final review)
|
||||
- **Confidence**: 0.82
|
||||
- **Accept/Defer**: accept
|
||||
|
||||
### I-C-006 — Dual-write window: can v0.9 ship new parser while old daemon runs?
|
||||
- **Tier**: cross-cutting
|
||||
- **Description**: Focused version of I-C-001. The specific question: in v0.9, when the new Markdown parser + kinds + SSH-push are shipped, can they coexist with v0.8 daemons still running on peers? The answer depends on whether `orca job run <spec.md>` uses the new SSH-push path (bypassing the daemon entirely) or routes through the old daemon. If it bypasses, the daemon is irrelevant for new specs but still serves old `.hcl` specs. If it routes through, the daemon can't handle `.md` specs. Proposal: v0.9 `orca job run` dispatches on extension (`.md`→SSH-push new path, `.hcl`→old daemon path) via the parser dispatcher (I-M-004). This is a *dual-write window* where both paths coexist. The daemon is not removed until v1.0-P05 (drain). The risk: if a `.md` workload and a `.hcl` workload target the same node, the SSH-push path writes systemd units directly while the daemon also manages units — they can conflict. Mitigation: the SSH-push path writes to a separate systemd unit namespace (`orca-v1-<alloc>.service`) while the daemon uses `orca-<job>.service`. No unit name overlap = no conflict.
|
||||
- **Description**: Focused version of I-C-001. The specific question: in v0.9, when the new Markdown parser + kinds + SSH-push are shipped, can they coexist with v0.8 daemons still running on peers? The answer depends on whether `orca job run <spec.md>` uses the new SSH-push path (bypassing the daemon entirely) or routes through the old daemon. If it bypasses, the daemon is irrelevant for new specs but still serves old `.hcl` specs. If it routes through, the daemon can't handle `.md` specs. Proposal: v0.9 `orca job run` dispatches on extension (`.md`→SSH-push new path, `.hcl`→old daemon path) via the parser dispatcher (I-M-004). This is a *dual-write window* where both paths coexist. The daemon is not removed until v0.10-P05 (drain). The risk: if a `.md` workload and a `.hcl` workload target the same node, the SSH-push path writes systemd units directly while the daemon also manages units — they can conflict. Mitigation: the SSH-push path writes to a separate systemd unit namespace (`orca-v1-<alloc>.service`) while the daemon uses `orca-<job>.service`. No unit name overlap = no conflict.
|
||||
- **Rationale**: Operational feasibility question for v0.9. §23 doesn't address it. If the answer is "no dual-write, daemon must be removed first," then v0.9 can't be tested incrementally and must ship as a big-bang — much higher risk.
|
||||
- **Proposed REQ ID**: REQ-090
|
||||
- **Proposed phase placement**: v0.9-P00 (decision before any v0.9 execution phase)
|
||||
@@ -301,35 +301,35 @@ the PRD §23 plan are listed at the end.
|
||||
|
||||
| ID | Tier | Title | REQ | Phase | Conf | Accept |
|
||||
|----|------|-------|-----|-------|------|--------|
|
||||
| I-M-001 | M | `orca daemon` deprecation path | REQ-061 | v1.0-P14 (warn v0.9-P0X) | 0.82 | accept |
|
||||
| I-M-001 | M | `orca daemon` deprecation path | REQ-061 | v0.10-P14 (warn v0.9-P0X) | 0.82 | accept |
|
||||
| I-M-002 | M | Coverage follow-ups to 70% | REQ-062 | v0.9-P0X + each new pkg | 0.88 | accept |
|
||||
| I-M-003 | M | known_hosts flock concurrency | REQ-063 | v0.9-P0a1 | 0.74 | accept |
|
||||
| I-M-004 | M | HCL→Markdown jobspec adapter | REQ-064 | v0.9-P0b | 0.85 | accept |
|
||||
| I-M-005 | M | `doctor --legacy-paths` detection | REQ-065 | v1.0-P14c | 0.80 | accept |
|
||||
| I-M-006 | M | Legacy CA state migration to step-ca | REQ-066 | v1.0-P14a | 0.70 | accept |
|
||||
| I-M-005 | M | `doctor --legacy-paths` detection | REQ-065 | v0.10-P14c | 0.80 | accept |
|
||||
| I-M-006 | M | Legacy CA state migration to step-ca | REQ-066 | v0.10-P14a | 0.70 | accept |
|
||||
| I-M-007 | M | Fuzz harness for Markdown parser | REQ-067 | v0.9-P0b | 0.78 | accept |
|
||||
| I-M-008 | M | Deprecation warnings on CLI subcommands | REQ-068 | v0.9-P0X + v1.0-P13 | 0.72 | accept |
|
||||
| I-M-008 | M | Deprecation warnings on CLI subcommands | REQ-068 | v0.9-P0X + v0.10-P13 | 0.72 | accept |
|
||||
| I-M-009 | M | HCL config demotion via adapter | REQ-069 | v0.9-P0a1 | 0.76 | accept |
|
||||
| I-M-010 | M | certpaths → multi-namespace path resolver | REQ-070 | v0.9-P0a1 | 0.84 | accept |
|
||||
| I-M-011 | M | store schema: per-namespace DBs | REQ-071 | v0.9-P0a1 + v1.0-P06 | 0.80 | accept |
|
||||
| I-M-012 | M | transport deletion + SSH-push package | REQ-072 | v0.9-P00 (delete v1.0-P14) | 0.68 | accept |
|
||||
| I-M-011 | M | store schema: per-namespace DBs | REQ-071 | v0.9-P0a1 + v0.10-P06 | 0.80 | accept |
|
||||
| I-M-012 | M | transport deletion + SSH-push package | REQ-072 | v0.9-P00 (delete v0.10-P14) | 0.68 | accept |
|
||||
| I-B-001 | B | SSH-push transport layer design | REQ-073 | v0.9-P01 | 0.86 | accept |
|
||||
| I-B-002 | B | Emitter template system (Layer 4) | REQ-074 | v0.9-P0c | 0.82 | accept |
|
||||
| I-B-003 | B | Lead applier execution model | REQ-075 | v1.0-P10 (design v0.9-P00) | 0.78 | accept |
|
||||
| I-B-004 | B | step-ca integration | REQ-076 | v0.9-P07 + v1.0-P02 | 0.74 | accept |
|
||||
| I-B-003 | B | Lead applier execution model | REQ-075 | v0.10-P10 (design v0.9-P00) | 0.78 | accept |
|
||||
| I-B-004 | B | step-ca integration | REQ-076 | v0.9-P07 + v0.10-P02 | 0.74 | accept |
|
||||
| I-B-005 | B | Traefik dynamic config + atomic reload | REQ-077 | v0.9-P02 | 0.80 | accept |
|
||||
| I-B-006 | B | Runtime abstraction (5 backends) | REQ-078 | v0.9-P07a/b/c | 0.82 | accept |
|
||||
| I-B-007 | B | Transaction bundle + N-peer atomicity | REQ-079 | v1.0-P10 (design v0.9-P00) | 0.76 | accept |
|
||||
| I-B-008 | B | Master key + HKDF per-line encryption | REQ-080 | v1.0-P03 | 0.84 | accept |
|
||||
| I-B-007 | B | Transaction bundle + N-peer atomicity | REQ-079 | v0.10-P10 (design v0.9-P00) | 0.76 | accept |
|
||||
| I-B-008 | B | Master key + HKDF per-line encryption | REQ-080 | v0.10-P03 | 0.84 | accept |
|
||||
| I-B-009 | B | Syncthing config + folder-ID | REQ-081 | v0.9-P09 | 0.72 | accept |
|
||||
| I-B-010 | B | Namespace inheritance resolver | REQ-082 | v0.9-P0a2 | 0.86 | accept |
|
||||
| I-B-011 | B | CLI-side scheduler redesign | REQ-083 | v0.9-P05 (skeleton P0c) | 0.80 | accept |
|
||||
| I-B-012 | B | `orca job lint` category-driven engine | REQ-084 | v1.0-P11 | 0.78 | accept |
|
||||
| I-C-001 | C | v0.8→v1.0 migration ordering | REQ-085 | spans v0.9-P00→v1.0-P14 | 0.88 | accept |
|
||||
| I-C-002 | C | "No orca on server" enforcement | REQ-086 | v1.0-P14c | 0.82 | accept |
|
||||
| I-C-003 | C | Hermetic test infra (3 linux + 1 pve) | REQ-087 | v1.0-P08 (bootstrap v0.9-P00) | 0.80 | accept |
|
||||
| I-C-004 | C | security/network persona reactivation | REQ-088 | spans v0.9→v1.0-P16 | 0.84 | accept |
|
||||
| I-C-005 | C | Docs rewrite + AD-010 supersession | REQ-089 | v0.9-P00 + v1.0-P15/P16 | 0.82 | accept |
|
||||
| I-B-012 | B | `orca job lint` category-driven engine | REQ-084 | v0.10-P11 | 0.78 | accept |
|
||||
| I-C-001 | C | v0.8→v1.0 migration ordering | REQ-085 | spans v0.9-P00→v0.10-P14 | 0.88 | accept |
|
||||
| I-C-002 | C | "No orca on server" enforcement | REQ-086 | v0.10-P14c | 0.82 | accept |
|
||||
| I-C-003 | C | Hermetic test infra (3 linux + 1 pve) | REQ-087 | v0.10-P08 (bootstrap v0.9-P00) | 0.80 | accept |
|
||||
| I-C-004 | C | security/network persona reactivation | REQ-088 | spans v0.9→v0.10-P16 | 0.84 | accept |
|
||||
| I-C-005 | C | Docs rewrite + AD-010 supersession | REQ-089 | v0.9-P00 + v0.10-P15/P16 | 0.82 | accept |
|
||||
| I-C-006 | C | Dual-write window decision | REQ-090 | v0.9-P00 | 0.86 | accept |
|
||||
|
||||
## Phase Reordering / Addition Flags (against PRD §23)
|
||||
@@ -339,8 +339,8 @@ the PRD §23 plan are listed at the end.
|
||||
3. **I-B-001 (SSH-push transport)** — §23 v0.9-P01 needs SSH-push. The design is a prerequisite. **Recommendation: SSH-push design in P0a1, not deferred to P01.**
|
||||
4. **I-B-002 (emitter template system)** — should be designed *with* the schemas (P0c). **Recommendation: expand P0c to "schemas + emitter interface."**
|
||||
5. **I-B-003 (lead applier model)** — bundle format + lead applier model must be designed *in v0.9* so the emitter can produce bundle-compatible output. **Recommendation: design spike in v0.9-P00.**
|
||||
6. **I-C-003 (test infra)** — hermetic cluster harness should be bootstrapped in v0.9-P00 so every v0.9 phase can run integration tests. **Recommendation: bootstrap in v0.9-P00, expand in v1.0-P08.**
|
||||
7. **I-C-004 / I-C-005 (persona reactivation + docs)** — span the whole milestone. **Recommendation: fold persona reviews into v0.9-P00 and v1.0-P16; fold doc banners into v0.9-P00.**
|
||||
6. **I-C-003 (test infra)** — hermetic cluster harness should be bootstrapped in v0.9-P00 so every v0.9 phase can run integration tests. **Recommendation: bootstrap in v0.9-P00, expand in v0.10-P08.**
|
||||
7. **I-C-004 / I-C-005 (persona reactivation + docs)** — span the whole milestone. **Recommendation: fold persona reviews into v0.9-P00 and v0.10-P16; fold doc banners into v0.9-P00.**
|
||||
|
||||
## Cross-Reference Against Existing Decisions
|
||||
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
# P23 Dual-Write Closure — Decision (v0.12)
|
||||
|
||||
**Status**: DEFERRED to v1.x. The full deletion of the legacy CA
|
||||
(`internal/security/ca.go`), mTLS transport (`internal/transport/mtls.go`),
|
||||
and daemon plaintext mode is too large a refactor for v0.12 without
|
||||
risking build stability. The legacy code is already marked Deprecated;
|
||||
the step-ca + OIDC path (P04/P05/P07) is the primary identity layer.
|
||||
|
||||
## What v0.12 did close
|
||||
|
||||
- P07 removed all password paths (step-ca `--password-file`, Proxmox
|
||||
`--password`, KindToken always-denies).
|
||||
- P09 removed daemon plaintext mode (Start() requires mTLS).
|
||||
- P11 added SVID chain validation (VerifySVIDWithChain).
|
||||
- P06 rewrote ACL to OIDC (KindToken deprecated).
|
||||
|
||||
## What remains for v1.x
|
||||
|
||||
- Delete `internal/security/ca.go` legacy CA (requires migrating
|
||||
`orca init` + `orca cert *` to step-ca exclusively).
|
||||
- Delete `internal/transport/mtls.go` deprecated path.
|
||||
- Delete `internal/certpaths/` (v0.8 flat layout); `internal/paths/`
|
||||
is the only layout.
|
||||
- Migrate `rotate-lead`, `drain`, `cutover`, `recovery` from
|
||||
`certpaths` to `paths`.
|
||||
|
||||
## Why not in v0.12
|
||||
|
||||
The legacy CA is load-bearing for `orca init` and 6+ CLI commands. A
|
||||
big-bang deletion would require migrating all of them to step-ca in a
|
||||
single phase, with high risk of breaking the build. v0.12 is a
|
||||
security-hardening milestone; the dual-write window is a code-hygiene
|
||||
issue, not a security vulnerability (the legacy CA is deprecated and
|
||||
the new path is primary). v1.x will close it as a focused refactor.
|
||||
+95
-70
@@ -137,91 +137,72 @@ enforcement remains in `warn` mode per config.json.
|
||||
|
||||
---
|
||||
|
||||
## v0.7 baseline (preserved for traceability)
|
||||
## v0.10 Docs & Install Milestone — Persona Configuration
|
||||
|
||||
```yaml
|
||||
---
|
||||
active_personas:
|
||||
active:
|
||||
- lead-developer
|
||||
- backend-engineer
|
||||
- docs-engineer
|
||||
deactivated:
|
||||
- data-engineer
|
||||
deactivated_personas:
|
||||
- cli-engineer
|
||||
- security-engineer
|
||||
- devops-engineer
|
||||
- network-engineer
|
||||
- devops-engineer
|
||||
- cli-engineer
|
||||
- frontend-engineer
|
||||
phase_specific: []
|
||||
phase_specific:
|
||||
- docs-engineer
|
||||
reason: |
|
||||
Orca v0.7 is an NFR hardening & completion milestone. The work is CLI
|
||||
registration (cert command), a new internal/config package, test
|
||||
coverage uplift across engine/transport/proxmox/audit, and an opt-in
|
||||
pprof endpoint on the daemon. No schema changes, no new security
|
||||
surface, no packaging/distribution, no UI.
|
||||
|
||||
Roster changes vs v0.6:
|
||||
- data-engineer: RETAINED — owns cert_repo tests + store coverage.
|
||||
- security-engineer: DEACTIVATED — v0.7 adds no new security surface
|
||||
(pprof is operator-only, addr-gated; cert registration exposes
|
||||
existing security code, does not add new).
|
||||
- cli-engineer: DEACTIVATED — merged into lead-developer for v0.7
|
||||
(the cert registration is a 1-line AddCommand; config --config flag
|
||||
is root-command wiring, not a new CLI subsystem).
|
||||
- devops-engineer: DEACTIVATED — no packaging/distribution in v0.7.
|
||||
v0.10 is a documentation + install-hardening milestone. It touches two
|
||||
territories: scripts/ (release.sh, install.sh — bash, backend-engineer)
|
||||
and docs/ + examples/ + README.md (markdown, lead-developer +
|
||||
docs-engineer). No Go orchestration code changes, no schema/migration
|
||||
changes, no UI, no security/crypto surface, no transport/network
|
||||
surface. The data-engineer, security-engineer, network-engineer, and
|
||||
devops-engineer personas are deactivated for this milestone.
|
||||
---
|
||||
```
|
||||
|
||||
### lead-developer (v0.7)
|
||||
- **Domain**: coordination
|
||||
- **Frameworks**: `cobra`
|
||||
- **Constraints**: `boundary-enforcement`, `offline-first`, `no-redundant-implementations`
|
||||
- **Territory**: `**/*.go`, `cmd/**`, `internal/**`
|
||||
### lead-developer (v0.10)
|
||||
- **Active**: true
|
||||
- **Reason**: Coordination across P01/P02/P03. SSH/bootstrap touches security + cli + store + doctor — territory overlaps need adjudication (proxmox package boundary, doctor Proxmox check scaffolding).
|
||||
- **Territory**: `docs/**/*.md`, `examples/**`, `README.md`,
|
||||
`.ciagent/**/*.md` (coordination + cross-cutting docs)
|
||||
- **Frameworks**: markdown, cobra (for CLI reference accuracy)
|
||||
- **Reason**: Owns the CLI reference doc, jobspec reference, ingress
|
||||
guide, examples directory, README refresh, and namespace.md update.
|
||||
Coordinates factual accuracy against the live codebase.
|
||||
|
||||
### backend-engineer (v0.7)
|
||||
- **Domain**: backend
|
||||
- **Frameworks**: `cobra`, `net/http`, `golang.org/x/crypto/ssh`
|
||||
- **Constraints**: `API-first`, `error-handling`, `minimal-dependencies`, `security-first`, `idempotent-bootstrap`
|
||||
- **Territory**: `**/api/**`, `**/*_handler*`, `**/*_handler.go`, `internal/daemon/**`, `internal/proxmox/**`, `internal/cli/init.go`
|
||||
### backend-engineer (v0.10)
|
||||
- **Active**: true
|
||||
- **Reason**: Owns the `orca init` full-bootstrap orchestration (CA + cert + db + localhost node, idempotent) and the `internal/proxmox/bootstrap.go` SSH session sequence (dial, deploy pubkey, useradd, pveum, sudoers, visudo validate). Added `idempotent-bootstrap` constraint (D-036 — re-run must be skip-and-refresh) and `golang.org/x/crypto/ssh` to frameworks.
|
||||
- **Territory**: `scripts/release.sh`, `scripts/install.sh`,
|
||||
`scripts/tests/*.bash`
|
||||
- **Frameworks**: bash, curl, tea CLI, Gitea API
|
||||
- **Reason**: Owns the release/install pipeline fix (cross-build amd64,
|
||||
asset verification, fallback walk). The scripts are API-adjacent
|
||||
tooling that interacts with the Gitea releases API.
|
||||
|
||||
### data-engineer (v0.7)
|
||||
- **Domain**: data
|
||||
- **Frameworks**: `modernc/sqlite`, `iter`
|
||||
- **Constraints**: `schema-first`, `migration-safe`, `local-storage-only`, `no-goroutine-leak`, `nullable-column-handling`
|
||||
- **Territory**: `**/store/**`, `**/model.go`, `**/migration*`, `migrations/**`, `internal/store/migrations/**`, `internal/model/node.go`
|
||||
- **Active**: true
|
||||
- **Reason**: Reactivated for v0.6. Owns migration `0006_node_kind_os.sql` (REQ-049 — nullable `kind`/`os` columns, backward-compatible) and `NodeRepo` schema extension (Insert/Get/List/Watch/scanNode column additions + new `GetByName`/`UpdateLastSeenAndOS` helpers). Added `nullable-column-handling` constraint (NULL → `""` in Go struct, not nil-deref).
|
||||
### docs-engineer (v0.10 — phase-specific)
|
||||
- **Active**: true (phase-specific: P2, P3, P4)
|
||||
- **Territory**: `docs/cli.md`, `docs/jobspec.md`, `docs/ingress.md`,
|
||||
`examples/full-stack/**`
|
||||
- **Frameworks**: markdown, GitHub-flavored markdown
|
||||
- **Constraints**: factual-accuracy-against-codebase,
|
||||
cross-link-resolution, deprecation-callouts
|
||||
- **Reason**: Custom persona for the markdown authoring work. Ensures
|
||||
every factual claim in the docs is grounded in the live codebase
|
||||
(struct fields, flag definitions, paths) and every cross-link
|
||||
resolves. Removed after P4.
|
||||
|
||||
### cli-engineer (v0.7)
|
||||
- **Domain**: CLI/UX
|
||||
- **Frameworks**: `cobra`, `pflag`
|
||||
- **Constraints**: `discoverable-help`, `consistent-flag-naming`, `human-readable-output`, `machine-readable-json-flag`, `signal-handling`, `password-flag-redaction`
|
||||
- **Territory**: `cmd/**`, `internal/cli/**`, `internal/commands/**`
|
||||
- **Active**: true
|
||||
- **Reason**: Owns `orca init` multi-step bootstrap output UX (progress lines per step), `orca node join --type/--host/--user/--password/--proxmox-user/--proxmox-role` flag wiring, and `doctor os`/`doctor proxmox` subcommand wiring. Added `password-flag-redaction` constraint (D-031 — `--password` never echoed, prefer `$ORCA_PROXMOX_PASSWORD`, zero after use).
|
||||
|
||||
### security-engineer (v0.7)
|
||||
- **Domain**: security
|
||||
- **Frameworks**: `crypto/tls`, `crypto/x509`, `crypto/ed25519`, `golang.org/x/crypto/ssh`, `slog`
|
||||
- **Constraints**: `no-panic-in-production`, `structured-audit-logging`, `no-secret-in-logs`, `input-validation`, `least-privilege`, `tofu-host-key-pinning`, `noexec-sudoers`
|
||||
- **Territory**: `**/auth/**`, `**/audit/**`, `internal/security/**`, `internal/transport/**` (TLS config only), `internal/proxmox/**` (SSH + sudoers + PVE role)
|
||||
- **Active**: true
|
||||
- **Reason**: Reactivated for v0.6. Owns `internal/security/sshkey.go` (Ed25519 keygen, 0600/0644 mode enforcement per REQ-033 spirit), TOFU host-key pinning via `knownhosts.New`, sudoers least-privilege design (NOEXEC on pct/qm, exclude pvesh, no NOEXEC on apt-get/dpkg), password redaction (D-031), and audit logging of all bootstrap/join actions (REQ-052). Added `tofu-host-key-pinning` and `noexec-sudoers` constraints. Co-owns `internal/proxmox/**` with backend-engineer (security owns SSH auth + sudoers content; backend owns the session orchestration).
|
||||
|
||||
### devops-engineer (v0.7)
|
||||
- **Active**: false (v0.6)
|
||||
- **Reason**: Deactivated — v0.6 has no install.sh, Dockerfile, .coreci.yml, or release-pipeline surface. The Proxmox SSH bootstrap is backend + security work, not devops. Was active in v0.5 (distribution milestone).
|
||||
|
||||
### network-engineer (v0.7)
|
||||
- **Active**: false (v0.6)
|
||||
- **Reason**: v0.6 has no transport/mTLS surface. SSH is point-to-point bootstrap, not the mTLS mesh network-engineer owns.
|
||||
|
||||
### frontend-engineer (v0.7)
|
||||
- **Active**: false (v0.6)
|
||||
- **Reason**: No web UI in Orca (unchanged from v0.1 onward).
|
||||
|
||||
### v0.6 vs v0.5 Persona Diff (v0.7 baseline reference)
|
||||
### Deactivated personas (v0.10)
|
||||
- **data-engineer**: no schema/migration work this milestone.
|
||||
- **security-engineer**: no crypto/threat-model work this milestone.
|
||||
- **network-engineer**: no transport/socket work this milestone.
|
||||
- **devops-engineer**: no packaging/distribution work beyond the
|
||||
release.sh fix (owned by backend-engineer).
|
||||
- **cli-engineer**: no new CLI commands this milestone.
|
||||
- **frontend-engineer**: no web UI (unchanged from v0.1).
|
||||
|
||||
| Change | Rationale |
|
||||
|--------|-----------|
|
||||
@@ -229,4 +210,48 @@ reason: |
|
||||
| `security-engineer` reactivated | Owns SSH keygen, TOFU host-key, sudoers, PVE role — first-class security surface. |
|
||||
| `devops-engineer` deactivated | v0.6 has no packaging/distribution surface. |
|
||||
| `network-engineer` remains deactivated | No transport/mTLS surface. |
|
||||
| `frontend-engineer` remains deactivated | No web UI. |
|
||||
| `frontend-engineer` remains deactivated | No web UI. |
|
||||
## v0.11 Update (Production Hardening)
|
||||
|
||||
The v0.9 persona roster carries forward to v0.11 with these additions:
|
||||
|
||||
### Roster changes
|
||||
|
||||
- **lead-developer**: RETAINED — owns `orca cluster rotate-lead` (P14b),
|
||||
`orca upgrade` (P14a), README framing (P15, Q5=A Nomad-inspired),
|
||||
milestone coordination.
|
||||
- **backend-engineer**: RETAINED — owns `internal/drift/` (P10, ~500 LoC
|
||||
greenfield), `internal/emitter/nft.go` (P15.5, ~200 LoC greenfield),
|
||||
`orca drift` CLI tree (P10), `orca nft` CLI (P15.5), `orca job migrate`
|
||||
(P05), `orca logs --all-nodes` (P06), `orca doctor mTLS`/`orca doctor nft`
|
||||
(P15.5), `scripts/orca-drift-notify.sh` + `orca-remediate.sh` (P10).
|
||||
Frameworks: cobra, `iter.Seq2` (D-017 extension), `signal.NotifyContext`
|
||||
(D-023), golang.org/x/crypto/ssh (existing).
|
||||
- **data-engineer**: REACTIVATED for P14a — owns v0.8→v1.0 data migration
|
||||
(REQ-066), schema migration for `orca upgrade` binding cutover. Was
|
||||
deactivated in v0.10 (docs-only milestone); reactivated for the
|
||||
migration phase.
|
||||
- **security-engineer**: RETAINED — owns threat model (P15.5, C-19),
|
||||
secrets subsystem (P03), `orca doctor mTLS` (P15.5), ingress-hybrid
|
||||
trust-boundary review (R-017), drift-detection threat model (R-020
|
||||
deadlock, secret exclusion D-234).
|
||||
- **network-engineer**: RETAINED — owns nftables emitter (P15.5, R-017),
|
||||
Traefik binding cutover (P14a/P15.5), cross-node cluster mesh (D-219,
|
||||
unchanged private IP), drift-detection network paths (NFS detection
|
||||
D-233, SSH fanout for aggregator).
|
||||
- **devops-engineer**: RETAINED — owns `scripts/orca-aggregate.sh`
|
||||
extension (P09, D-237), `scripts/orca-drift-notify.sh` (P10),
|
||||
`scripts/orca-remediate.sh` (P10), systemd Path unit emitter (P10),
|
||||
drift-detection integration tests (P08: auto-remediation, NFS fallback,
|
||||
cooldown, secret exclusion), `orca` system user setup (P10, REQ-111).
|
||||
- **docs-engineer**: PHASE-SPECIFIC (P15) — owns README refresh (Q5=A
|
||||
Nomad-inspired framing, honest-trade-offs table from research doc 3).
|
||||
Created for P15; removed after phase completes.
|
||||
- **cli-engineer**: remains DEACTIVATED — CLI surface growth is owned by
|
||||
lead-developer + backend-engineer.
|
||||
- **frontend-engineer**: remains DEACTIVATED — no web UI.
|
||||
|
||||
### Phase-specific personas
|
||||
|
||||
- `docs-engineer`: active for P15 only (README refresh). Removed after
|
||||
phase completes.
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
# Plan: v0.10 Docs & Install Milestone
|
||||
|
||||
## Milestone: v0.10 — Docs & Install Hardening
|
||||
- **Type**: feature (P1 `fix`, P2-P4 `docs`; at least one non-docs phase)
|
||||
- **Tags**: `v0.9.0` (P0) → `v0.9.1` (P1) → `v0.9.2` (P2) → `v0.9.3` (P3) → `v0.9.4` (P4) → `v0.9.5` (P5 = milestone release)
|
||||
- **Branch**: `milestone/v0.10-docs-cli-examples`
|
||||
|
||||
## Phase breakdown
|
||||
|
||||
### Phase P1 — release.sh + install.sh fix (Wave 1)
|
||||
**REQs**: REQ-097, REQ-098
|
||||
**Persona**: backend-engineer
|
||||
**Territory**: `scripts/release.sh`, `scripts/install.sh`, `scripts/tests/*.bash`
|
||||
**Vertical slice**: a broken release → a correctly-asseted release that install.sh resolves.
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P1-T1 | `scripts/release.sh`: replace host-arch build (lines 84, 89-98) with explicit `GOOS=linux GOARCH=amd64 go build` cross-build; produce `orca-${VERSION}-linux-amd64.tar.gz` regardless of host arch | REQ-097 |
|
||||
| P1-T2 | `scripts/release.sh`: after `tea releases create` (line 132), add post-create asset verification — query `/api/v1/repos/$OWNER/$REPO/releases/tags/$VERSION`, assert the tarball appears in `attachments`, retry once if missing, fail loudly with clear error if still missing | REQ-097 |
|
||||
| P1-T3 | `scripts/install.sh`: add asset fallback walk — if the resolved release (latest or `--version`) lacks the matching `orca-<ver>-<os>-<arch>.tar.gz`, query `/releases?limit=20`, walk backward, use the most recent release that carries the asset, print a warning | REQ-098 |
|
||||
| P1-T4 | `scripts/install.sh`: add `--check` dry-run mode that prints version + asset URL + install path without writing | REQ-098 |
|
||||
| P1-T5 | `scripts/tests/release.bats` + `scripts/tests/install.bats`: add/extend bats tests for the new behavior (happy path: asset present; fallback: latest release asset-less, older release has asset; --check prints without writing) | REQ-097, REQ-098 |
|
||||
|
||||
**Must-haves**: release.sh produces an amd64 tarball on any host arch; install.sh resolves to a release with an asset (walking back if needed); `--check` works; bats tests pass.
|
||||
|
||||
### Phase P2 — CLI + jobspec + ingress docs (Wave 2)
|
||||
**REQs**: REQ-091, REQ-092, REQ-093
|
||||
**Persona**: docs-engineer (phase-specific), lead-developer
|
||||
**Territory**: `docs/cli.md`, `docs/jobspec.md`, `docs/ingress.md`
|
||||
**Vertical slice**: an operator with no orca background → can author a jobspec, run it, and understand the ingress model from docs alone.
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P2-T1 | `docs/cli.md`: full CLI reference — global flags, every command/subcommand with synopsis + flag tables + one-line example, output modes (text/json/watch), exit codes, deprecated surface callout boxes (daemon/cert/node-join-mTLS/HCL-jobspec) | REQ-091 |
|
||||
| P2-T2 | `docs/jobspec.md`: markdown frontmatter schema reference — top-level keys, block reference (runtime/ports/env-secrets/volumes/restart/update/service/health/lifecycle/constraints/affinity/tasks), kinds matrix, CEL subset grammar, body semantics, deprecated HCL callout | REQ-092 |
|
||||
| P2-T3 | `docs/ingress.md`: Traefik ingress reference — service→Traefik mapping, R-007 socket-vs-TCP-bind, generated YAML shape, atomic reload (C-10), drain, TLS, worked-example pointer to `examples/full-stack/`, v0.10 forward limitations | REQ-093 |
|
||||
|
||||
**Must-haves**: every command/flag in `internal/cli/` is documented; every jobspec field in `internal/jobspec/markdown.go` is documented; every factual claim is grounded in the live codebase; cross-links resolve; deprecated surface is clearly marked.
|
||||
|
||||
### Phase P3 — full-stack examples (Wave 2, parallel with P2)
|
||||
**REQs**: REQ-094
|
||||
**Persona**: docs-engineer (phase-specific), lead-developer
|
||||
**Territory**: `examples/full-stack/**`
|
||||
**Vertical slice**: an operator → can deploy a multi-service stack with ingress by copying the examples.
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P3-T1 | `examples/full-stack/web-app.md`: kind Service, process runtime, port http, service block (socket default), health, restart (service), update (rolling), constraints (CEL), task group (app + sidecar) | REQ-094 |
|
||||
| P3-T2 | `examples/full-stack/api.md`: kind Service, process runtime, port api, service bind 127.0.0.1 (TCP opt-in), health, restart, update (canary) | REQ-094 |
|
||||
| P3-T3 | `examples/full-stack/worker.md`: kind Job, process runtime, one-shot, timeout, env, lifecycle hooks | REQ-094 |
|
||||
| P3-T4 | `examples/full-stack/log-shipper.md`: kind DaemonSet, schedule (every-node), restart, constraints | REQ-094 |
|
||||
| P3-T5 | `examples/full-stack/postgres.md`: kind Service, process runtime, port pg, volumes + replication (syncthing), health, restart, update (blue-green) | REQ-094 |
|
||||
| P3-T6 | `examples/full-stack/rendered/`: the Traefik dynamic YAML + systemd units orca generates for the stack (traefik-dynamic-web-app.yaml, traefik-dynamic-api.yaml, systemd-web-app.service, systemd-api.service, systemd-log-shipper.service) | REQ-094 |
|
||||
| P3-T7 | `examples/full-stack/README.md`: walkthrough (init → node join → capacity set → ns create → job run → list --watch → inspect rendered → drain/rollback notes → cross-link to docs/ingress.md) | REQ-094 |
|
||||
|
||||
**Must-haves**: all 5 jobspecs parse with the current `internal/jobspec` parser and pass `internal/spec/schema` validators; rendered artifacts match what the emitters would produce; README walkthrough is end-to-end coherent.
|
||||
|
||||
### Phase P4 — README + namespace.md refresh (Wave 3, after P2/P3)
|
||||
**REQs**: REQ-095, REQ-096
|
||||
**Persona**: lead-developer
|
||||
**Territory**: `README.md`, `docs/namespace.md`
|
||||
**Vertical slice**: a new visitor to the repo → sees accurate status, all commands, install instructions that work, and a link to the docs + examples.
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P4-T1 | `README.md`: status line (v0.9 complete, v0.10 in progress); install `--version` example updated to current tag; subcommand table expanded to all commands with deprecation markers; update-in-place example updated; development targets complete; new Documentation + Examples sections | REQ-095 |
|
||||
| P4-T2 | `docs/namespace.md`: replace v0.8 flat path table with v0.9 multi-namespace layout (`cluster/`, `_defaults/`, per-ns `db/jobs/alloc/ns.md`); `ORCA_HOME`/`--system` resolution; `orca ns` subcommand cross-link; v0.8 flat layout flagged deprecated | REQ-096 |
|
||||
|
||||
**Must-haves**: README subcommand table matches `internal/cli/` exactly; install example pins a current tag; namespace.md path table matches `internal/paths/paths.go`; both files cross-link to the new docs.
|
||||
|
||||
### Phase P5 — final review + ship + audit (Wave 4)
|
||||
**REQs**: all (REQ-091..REQ-098)
|
||||
**Persona**: lead-developer
|
||||
**Vertical slice**: milestone complete → merged to main, tagged, released.
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P5-T1 | Code review across all phases (P1-P4); auto-apply P0 fixes, flag P1+ for post-hoc | all |
|
||||
| P5-T2 | Audit: reconstruction test (git log matches `.ciagent/`), file discipline, branch hygiene, commit discipline | all |
|
||||
| P5-T3 | Milestone ship: merge phase/05 → milestone → main; tag `v0.9.5` (= v0.10.0 milestone release); create release with full milestone summary + Linux binary asset (verified by the P1 fix); delete all milestone branches | all |
|
||||
| P5-T4 | Complete milestone: mark REQ-091..098 complete in REQUIREMENTS.md; mark v0.10 docs milestone complete in ROADMAP.md; clear checkpoint | all |
|
||||
|
||||
**Must-haves**: milestone merged to main; release carries the Linux binary (the fix from P1 proving itself); all REQs marked complete; checkpoint cleared.
|
||||
|
||||
## Wave ordering
|
||||
|
||||
- **Wave 1**: P1 (release/install fix) — unblocks the ship of every subsequent phase (each phase ship needs a correctly-asseted release)
|
||||
- **Wave 2**: P2 (docs) + P3 (examples) — parallel, no dependencies between them
|
||||
- **Wave 3**: P4 (README + namespace.md) — depends on P2/P3 existing (cross-links)
|
||||
- **Wave 4**: P5 (final review + ship) — depends on all prior phases
|
||||
|
||||
## Risks
|
||||
|
||||
- **R1**: The jobspecs in P3 might not parse if a field shape has drifted since the explore report. Mitigation: validate each jobspec against the current parser before committing (write a throwaway test or run `orca job run` with `--dry-run` if available).
|
||||
- **R2**: `tea releases create` asset verification in P1 might reveal a tea CLI bug that can't be worked around in bash. Mitigation: fall back to a direct `curl` upload to the Gitea attachments API if `tea` is unreliable.
|
||||
- **R3**: The v0.8.15 release still has no asset after P1 ships (P1 only fixes forward). Mitigation: install.sh's fallback walk (P1-T3) handles the gap; users installing between P1 ship and the first correctly-asseted release (P1's own ship tag v0.9.1) will get a clear warning + fallback.
|
||||
@@ -0,0 +1,360 @@
|
||||
# Plan: v0.11 Production Hardening
|
||||
|
||||
## Milestone: v0.11 — Production Hardening
|
||||
- **Type**: feature (multiple `feat` phases)
|
||||
- **Tags**: `v0.10.0` (P0) → `v0.10.1`…`v0.10.20` (P00…P15.5) → `v0.10.21` (P16 = v0.11.0 milestone release)
|
||||
- **Branch**: `milestone/v0.11-production-hardening`
|
||||
- **New rules adopted**: R-017 (ingress hybrid), R-018/R-019/R-020 (drift detection)
|
||||
- **New decisions**: D-215…D-237 (23 net-new, no collisions)
|
||||
- **New REQs**: REQ-099…REQ-118 (20 net-new; REQ count 98→118)
|
||||
|
||||
## Wave ordering
|
||||
|
||||
### Wave 0 — Foundation (serial)
|
||||
- **P00** — CLI cache layer (R-008). Unblocks all subsequent CLI commands that need cached reads.
|
||||
|
||||
### Wave 1 — Observability + identity (serial, gate-heavy)
|
||||
- **P01** — Metrics endpoint (hand-rolled text exposition). Unblocks `orca doctor mTLS` live probe (C5).
|
||||
- **P01.5** — SPIFFE SVID minting spike (**gate C-08** — if spike fails, fall back to mTLS identity). Gates P02 ACL.
|
||||
- **P02** — ACL (SPIFFE + token identities). Depends on P01.5.
|
||||
|
||||
### Wave 2 — Security + secrets (serial)
|
||||
- **P03** — Secrets subsystem (REQ-080; **gate C-19** threat model).
|
||||
- **P15.5** — Threat model + security review (**gate C-19**) + **ingress hybrid (R-017; REQ-099..REQ-102)** + **`orca doctor mTLS` (REQ-118)**. Per Q3=A, ingress folds in here. This phase grows ~30% but stays one phase.
|
||||
|
||||
### Wave 3 — Data durability (serial)
|
||||
- **P04** — Backup/restore (tar + signed). Unblocks P07 recovery.
|
||||
- **P06** — Alloc history (CLI-side SQLite; R-008 cache DB) + **`orca logs --all-nodes --since` (REQ-117)**. The logs command uses the alloc-history cache DB.
|
||||
|
||||
### Wave 4 — Lifecycle (serial)
|
||||
- **P05** — Drain + daemon drain-and-stop (REQ-061) + **`orca job migrate --to` (REQ-116; C3=drain+reschedule composite)**. Migrate composes P05 drain + P06 alloc history.
|
||||
- **P07** — Recovery (`orca restore`). Depends on P04 backup.
|
||||
|
||||
### Wave 5 — Transactional plane (serial, the big one)
|
||||
- **P10** — Transactional plane (REQ-075, REQ-079; **gate C-09**) + **drift detection (R-018/R-019/R-020; REQ-103..REQ-113)**. This is the largest phase. **Grill may split into P10a (txn plane) + P10b (drift) if vertical slice is too large.**
|
||||
- **P11** — `orca job lint` (REQ-084). Depends on P10 txn plane for dry-run validation.
|
||||
- **P12** — `orca job verify` (dry-run txn through lead). Depends on P10.
|
||||
|
||||
### Wave 6 — Namespace + aggregation (parallel)
|
||||
- **P09** — Collector + aggregator (opt-in; **gates C-11, C-12, C-14**) + **drift-event aggregation extension (REQ-107, D-237)**. The aggregator timer is extended to pull drift-events/ and remediate. C-11 watchdog monitors this timer.
|
||||
- **P13** — `orca ns` subcommands (full surface) + deprecation warnings (REQ-068).
|
||||
|
||||
### Wave 7 — Migration (serial, gate-heavy)
|
||||
- **P14a** — v0.8→v1.0 data migration (REQ-066; **gate C-07**) + **`orca upgrade --to-vX` (REQ-115; C2=thin wrapper, handles R-017 binding cutover)**.
|
||||
- **P14b** — Daemon cutover + running-allocation adoption + **`orca cluster rotate-lead` (REQ-114)**.
|
||||
- **P14c** — Mixed-version tolerance + no-orca-on-server enforcement (REQ-065, REQ-086; implements C-13).
|
||||
|
||||
### Wave 8 — Integration + docs + ship (serial)
|
||||
- **P08** — Integration tests (expand hermetic harness, REQ-087) + **drift-detection integration tests (auto-remediation success, NFS fallback, rate-limit cooldown, secret exclusion)**.
|
||||
- **P15** — README quickstart (REQ-089; **Q5=A Nomad-inspired framing, honest-trade-offs table from doc 3**). All cited CLI commands must exist by this phase.
|
||||
- **P16** — Final review + ship + audit — **v0.11.0 milestone release**.
|
||||
|
||||
## Phase task tables
|
||||
|
||||
### Phase P00 — CLI cache layer (Wave 0)
|
||||
**REQs**: R-008 (cache floor)
|
||||
**Persona**: backend-engineer
|
||||
**Territory**: `internal/cache/`, `internal/store/orca_cache.go`
|
||||
**Vertical slice**: a CLI command that reads cached state → a cache-hit returns in <1ms, a cache-miss populates from the lead.
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P00-T1 | `internal/cache/` package: `orca_cache` SQLite schema (per-class TTLs), `Get(class, key)`, `Set(class, key, val, ttl)`, `Invalidate(class)`; stdlib `database/sql` + modernc/sqlite | R-008 |
|
||||
| P00-T2 | Wire cache into `orca node list`, `orca job list`, `orca ns list` (read path only; writes bypass cache) | R-008 |
|
||||
| P00-T3 | `orca cache show` / `orca cache invalidate` CLI for debugging | R-008 |
|
||||
| P00-T4 | Tests: cache-hit/miss/invalidate/TTL-expiry; bench <1ms cache-hit | R-008 |
|
||||
|
||||
**Must-haves**: cache-hit <1ms; TTL-based invalidation; CLI commands use cache on read path.
|
||||
|
||||
### Phase P01 — Metrics endpoint (Wave 1)
|
||||
**REQs**: (new; metrics text exposition)
|
||||
**Persona**: backend-engineer
|
||||
**Territory**: `internal/cli/metrics.go`, `internal/transport/metrics.go`
|
||||
**Vertical slice**: `curl localhost:9100/metrics` → prometheus text exposition.
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P01-T1 | `internal/transport/metrics.go`: hand-rolled Prometheus text exposition (no client_golang dep); counters for txns applied/drifted/remediated; gauges for peers/nodes/allocs | new |
|
||||
| P01-T2 | `orca daemon --metrics :9100` flag (or sidecar listener); `/metrics` endpoint | new |
|
||||
| P01-T3 | Tests: exposition format validity; counter increments on txn apply | new |
|
||||
|
||||
**Must-haves**: `/metrics` returns valid Prometheus text; no client_golang dependency.
|
||||
|
||||
### Phase P01.5 — SPIFFE SVID minting spike (Wave 1, gate C-08)
|
||||
**REQs**: REQ-076
|
||||
**Persona**: security-engineer
|
||||
**Territory**: `internal/identity/spiffe.go`
|
||||
**Gate**: C-08 — if spike fails, fall back to mTLS identity (decision recorded).
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P01.5-T1 | Spike: mint a SPIFFE SVID via step-ca; verify URI SAN format (`spiffe://orca.local/ns/<ns>/sa/<sa>/<alloc-id>`) | REQ-076 |
|
||||
| P01.5-T2 | Decision record: if spike passes, proceed to P02 with SPIFFE; if fails, fall back to mTLS identity + record in PROJECT.md | REQ-076 |
|
||||
|
||||
**Must-haves**: spike passes or fails with a recorded decision; C-08 gate cleared.
|
||||
|
||||
### Phase P02 — ACL (Wave 1)
|
||||
**REQs**: (new; ACL with SPIFFE + token identities)
|
||||
**Persona**: backend-engineer + security-engineer
|
||||
**Territory**: `internal/acl/`, `internal/cli/acl.go`
|
||||
**Depends on**: P01.5 (SPIFFE or mTLS fallback)
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P02-T1 | `internal/acl/` package: identity → permissions mapping; SPIFFE URI → namespace scope; token identities for operators | new |
|
||||
| P02-T2 | `orca acl` CLI: `grant`, `revoke`, `list`, `check`; scoped to namespace paths per R-002 | new |
|
||||
| P02-T3 | Tests: SPIFFE identity grants ns-scoped access; token grants operator-scoped access; deny by default | new |
|
||||
|
||||
**Must-haves**: deny-by-default; SPIFFE URI maps to namespace; tokens for operator access.
|
||||
|
||||
### Phase P03 — Secrets subsystem (Wave 2, gate C-19)
|
||||
**REQs**: REQ-080
|
||||
**Persona**: security-engineer
|
||||
**Territory**: `internal/secrets/`, `internal/cli/secrets.go`
|
||||
**Gate**: C-19 (threat model must land in P15.5; P03 implements the crypto)
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P03-T1 | `internal/secrets/` package: AES-256-GCM encrypt/decrypt with master.key (R-011); per-line nonce; `LoadCredential=` integration | REQ-080 |
|
||||
| P03-T2 | `orca secrets` CLI: `set`, `get`, `rotate`, `list`; scoped to namespace `.env.secrets` | REQ-080 |
|
||||
| P03-T3 | Tests: encrypt/decrypt round-trip; rotation re-encrypts; master.key 0600 enforced | REQ-080 |
|
||||
|
||||
**Must-haves**: AES-256-GCM; per-line nonce; master.key 0600; `LoadCredential=` integration.
|
||||
|
||||
### Phase P15.5 — Threat model + ingress hybrid + doctor mTLS (Wave 2, gate C-19)
|
||||
**REQs**: REQ-099, REQ-100, REQ-101, REQ-102, REQ-118; C-19
|
||||
**Persona**: security-engineer + network-engineer + backend-engineer
|
||||
**Territory**: `internal/emitter/nft.go`, `internal/emitter/traefik.go`, `internal/cli/nft.go`, `internal/cli/doctor_mtls.go`, threat-model doc
|
||||
**Vertical slice**: `orca init` on a fresh cluster → Traefik binds 127.0.0.1:8443 + nft DNAT → `orca doctor nft` + `orca doctor mTLS` pass.
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P15.5-T1 | `internal/emitter/nft.go`: nftables emitter renders `/etc/nftables.d/orca.nft` (DNAT :443→127.0.0.1:8443, :80→127.0.0.1:8080; SYN-flood filter; `ora_rl` rate-limit meter; `orca_trusted_probes` set); idempotent `nft -f` apply; atomic rule-set swap (D-217, D-218) | REQ-099 |
|
||||
| P15.5-T2 | `internal/emitter/traefik.go` update: static config `address: 127.0.0.1:8443` (default); `--public-binding=traefik-on-public-ip` opt-out emits `:443`; certs/mTLS/dynamic config unchanged (D-220, D-216) | REQ-100 |
|
||||
| P15.5-T3 | `orca doctor nft`: checks table exists, DNAT rules present, rate-limit meter present, file parses (`nft -c -f`), hash matches latest txn (D-221, D-226) | REQ-101 |
|
||||
| P15.5-T4 | `orca nft` CLI: `show [--peer]`, `diff --against <txn-id>`, `doctor`, `country block add <cc-list>`, `rate limit set --rate N/s` (D-223, D-222) | REQ-102 |
|
||||
| P15.5-T5 | `orca doctor mTLS`: trust-chain verification (CA → server cert → workload SVIDs exist + not expired) + live mTLS handshake probe to each peer (reuses P01 metrics endpoint + P01.5 SPIFFE infra); C5=both | REQ-118 |
|
||||
| P15.5-T6 | Threat model doc: covers R-017 ingress trust boundary, R-020 drift deadlock, secret exclusion D-234, `orca` system user blast radius; clears C-19 | C-19 |
|
||||
| P15.5-T7 | Tests: nft emitter output validates (`nft -c -f`); Traefik static config has `127.0.0.1:8443`; doctor nft passes on a clean cluster; doctor mTLS passes with valid chain + live probe | REQ-099..102, 118 |
|
||||
|
||||
**Must-haves**: fresh `orca init` produces hybrid binding; `orca doctor nft` + `orca doctor mTLS` pass; C-19 cleared; opt-out flag works.
|
||||
|
||||
### Phase P04 — Backup/restore (Wave 3)
|
||||
**REQs**: (new; tar + signed backup)
|
||||
**Persona**: backend-engineer
|
||||
**Territory**: `internal/backup/`, `internal/cli/backup.go`
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P04-T1 | `internal/backup/` package: tar `ORCA_HOME` (excl. secrets? or incl. with master.key?); sign with master.key (HMAC-SHA256); `orca backup --out snap.tar.gz` | new |
|
||||
| P04-T2 | `orca restore --in snap.tar.gz` (P07 owns the full recovery; P04 owns the backup format + signing) | new |
|
||||
| P04-T3 | Tests: backup→restore round-trip; signature verification; backup excludes `/run/orca/*` | new |
|
||||
|
||||
**Must-haves**: backup is a signed tarball; restore verifies signature.
|
||||
|
||||
### Phase P06 — Alloc history + logs --all-nodes (Wave 3)
|
||||
**REQs**: REQ-071 (cache DB), REQ-117
|
||||
**Persona**: backend-engineer
|
||||
**Territory**: `internal/store/alloc_history.go`, `internal/cli/logs.go`
|
||||
**Depends on**: P00 (cache DB)
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P06-T1 | `internal/store/alloc_history.go`: CLI-side SQLite retention for alloc state transitions; TTL-based eviction | REQ-071 |
|
||||
| P06-T2 | `orca logs --all-nodes --since 5m`: aggregates journald logs across peers via SSH fanout; `iter.Seq` streaming (D-017); `--since` duration; `--all-nodes` fans out (Q2=C) | REQ-117 |
|
||||
| P06-T3 | Tests: alloc history retention/eviction; logs --all-nodes fans out + streams + cancels via ctrl-c | REQ-071, 117 |
|
||||
|
||||
**Must-haves**: alloc history retained in cache DB; `--all-nodes` aggregates across peers with streaming.
|
||||
|
||||
### Phase P05 — Drain + migrate (Wave 4)
|
||||
**REQs**: REQ-061, REQ-116
|
||||
**Persona**: backend-engineer
|
||||
**Territory**: `internal/cli/drain.go`, `internal/cli/migrate.go`
|
||||
**Depends on**: P06 (alloc history for migrate)
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P05-T1 | `orca node drain <host>`: drain a node (stop new allocs; migrate existing per `update` config); daemon drain-and-stop (REQ-061) | REQ-061 |
|
||||
| P05-T2 | `orca job migrate <name> --to <node>`: drain+reschedule composite (C3=a); uses P05 drain + P06 alloc history; idempotent (Q2=C) | REQ-116 |
|
||||
| P05-T3 | Tests: drain stops new allocs; migrate reschedules to target node; daemon drain-and-stop works | REQ-061, 116 |
|
||||
|
||||
**Must-haves**: drain stops new allocs + migrates existing; migrate reschedules to a specific node.
|
||||
|
||||
### Phase P07 — Recovery (Wave 4)
|
||||
**REQs**: (new; `orca restore`)
|
||||
**Persona**: backend-engineer
|
||||
**Territory**: `internal/cli/restore.go`
|
||||
**Depends on**: P04 (backup format)
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P07-T1 | `orca restore --in snap.tar.gz`: verify signature, extract, reconcile with live state (don't clobber running allocs unless `--force`) | new |
|
||||
| P07-T2 | Tests: restore from signed backup; signature mismatch fails; `--force` clobbers running allocs | new |
|
||||
|
||||
**Must-haves**: restore verifies signature; doesn't clobber running allocs without `--force`.
|
||||
|
||||
### Phase P10 — Transactional plane + drift detection (Wave 5, gate C-09)
|
||||
**REQs**: REQ-075, REQ-079, REQ-103..REQ-113; C-09; R-018/R-019/R-020
|
||||
**Persona**: backend-engineer + devops-engineer + security-engineer
|
||||
**Territory**: `internal/drift/`, `internal/cli/drift.go`, `scripts/orca-drift-notify.sh`, `scripts/orca-remediate.sh`, `internal/emitter/systemd.go` (Path units), `internal/paths/paths.go`
|
||||
**Vertical slice**: operator edits `/etc/traefik/dynamic/orca.yml` on a peer → drift detected in ~10s → auto-remediated → `orca drift watch` shows the event.
|
||||
**Note**: This is the largest phase. **Grill may split into P10a (txn plane, REQ-075/079) + P10b (drift detection, REQ-103..113) if the vertical slice is too large.**
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P10-T1 | `internal/txn/` package: render txn bundle (tarball + apply.sh + verify.sh) on operator host; SCP to lead's `/run/orca/txns/<txn-id>/`; lead's systemd timer runs `apply.sh` idempotently; CLI polls txn status via SSH (REQ-075, C-09) | REQ-075 |
|
||||
| P10-T2 | `scripts/orca-pull.sh` with C-09 failure contract: idempotent re-run, bounded retry, deterministic state, structured syslog (C-09) | REQ-079, C-09 |
|
||||
| P10-T3 | `internal/drift/` package: `Detector` interface (`Watch`, `Aggregate`, `Remediate`, `Acknowledge`), `Event`, `Config`, `PathSpec`, `RemediationPolicy`; `iter.Seq2[Event, error]` (D-017); `signal.NotifyContext` (D-023) (D-236) | REQ-103 |
|
||||
| P10-T4 | `orca drift` CLI tree: `watch [--interval=2s] [--paths=...] [--json]`, `show [--peer]`, `acknowledge <peer> <path>`, `remediate <peer> <path> [--force]`, `config show`, `config validate` (D-236) | REQ-104 |
|
||||
| P10-T5 | systemd Path unit emitter: for each critical path, emit `orca-drift-<name>.path` (`PathChanged=`, `RateLimitIntervalSec=1s`, `RateLimitBurst=5`) + `orca-drift-<name>.service` (`Type=oneshot`, `ExecStart=/usr/local/bin/orca-drift-notify.sh %f`, `User=orca`, security hardening); R-001-clean (D-227, D-228) | REQ-105 |
|
||||
| P10-T6 | `scripts/orca-drift-notify.sh`: receives changed path as `$1`, computes sha256 (or "DELETED"), writes event JSON to `/etc/orca/state/drift-events/<event-id>.json`; stateless, idempotent; `flock` (D-228) | REQ-106 |
|
||||
| P10-T7 | `scripts/orca-remediate.sh`: re-pushes latest applied txn's per-peer render tree via rsync, runs peer-side applier; 5-min cooldown per path applies ONLY on successful remediation (C4 refinement); transient failures retry next tick (D-231, D-232) | REQ-108 |
|
||||
| P10-T8 | Drift cadence config in `config.md` (`kind: ClusterConfig`): `drift.polling`, `drift.paths.{critical,standard,excluded}`, `drift.remediate`; critical defaults: Traefik dynamic, nftables, sudoers, orca-alloc services; secrets + `/run/orca/*` + drift-events dir excluded (R-018, D-231, D-234) | REQ-109 |
|
||||
| P10-T9 | Pre-flight consistency gate in `orca-pull.sh`: refuses new txns if drift detected on target peer/namespace; `--force` overrides; per-namespace scoping (drifted peer in ns-A doesn't block ns-B) (R-020, Q4=A) | REQ-110 |
|
||||
| P10-T10 | `orca` system user on peers: peer-setup emits `useradd -r orca` (system account, no login shell); `orca-drift-*.service` runs as `User=orca`; idempotent (REQ-111) | REQ-111 |
|
||||
| P10-T11 | NFS detection at peer setup: `orca node join` detects NFS mounts on orca state dirs; disables Path units for NFS paths; logs warning; falls back to polling (D-233) | REQ-112 |
|
||||
| P10-T12 | `orca job restart <name>`: restarts an alloc to pick up EnvironmentFile drift; normal allocation lifecycle (not file-level remediation) (D-235) | REQ-113 |
|
||||
| P10-T13 | Tests: txn apply idempotent + retry on failure; drift detected via Path unit ~10s; auto-remediation re-pushes; cooldown prevents loop; `--force` overrides pre-flight gate; per-ns scoping isolates drift; NFS fallback; secret exclusion | REQ-075..113 |
|
||||
|
||||
**Must-haves**: txn apply idempotent (C-09); drift detected ~10s on critical paths; auto-remediation with cooldown; `--force` + per-ns override; `orca` user created; NFS detection works.
|
||||
|
||||
### Phase P11 — `orca job lint` (Wave 5)
|
||||
**REQs**: REQ-084
|
||||
**Persona**: backend-engineer
|
||||
**Territory**: `internal/cli/job_lint.go`
|
||||
**Depends on**: P10 (txn plane for dry-run validation)
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P11-T1 | `orca job lint <spec.md>`: validates jobspec schema (kinds, blocks, CEL constraints, body preservation); reports errors with line numbers | REQ-084 |
|
||||
| P11-T2 | Tests: valid spec passes; invalid spec reports errors with line numbers | REQ-084 |
|
||||
|
||||
**Must-haves**: lint catches schema errors; reports line numbers.
|
||||
|
||||
### Phase P12 — `orca job verify` (Wave 5)
|
||||
**REQs**: (new; dry-run txn through lead)
|
||||
**Persona**: backend-engineer
|
||||
**Territory**: `internal/cli/job_verify.go`
|
||||
**Depends on**: P10 (txn plane)
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P12-T1 | `orca job verify <spec.md>`: dry-run txn through lead (no apply); reports what would change (allocs created/removed, config files written) | new |
|
||||
| P12-T2 | Tests: verify reports planned changes without applying; fails on pre-flight drift | new |
|
||||
|
||||
**Must-haves**: verify is a true dry-run (no side effects); reports planned changes.
|
||||
|
||||
### Phase P09 — Collector + aggregator + drift-event aggregation (Wave 6)
|
||||
**REQs**: (existing collector/aggregator) + REQ-107; C-11, C-12, C-14
|
||||
**Persona**: devops-engineer + backend-engineer
|
||||
**Territory**: `scripts/orca-aggregate.sh`, `internal/cli/collector.go`
|
||||
**Gates**: C-11 (watchdog), C-12 (opt-in), C-14 (syncthing)
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P09-T1 | `scripts/orca-aggregate.sh`: existing 10s cadence (C-11); now also rsyncs each peer's `/etc/orca/state/drift-events/`, validates event hashes against applied txn manifest, triggers `orca-remediate.sh` for auto-remediable paths, consumes (deletes) event files on peers (D-229, D-237) | REQ-107 |
|
||||
| P09-T2 | Lead-side watchdog meta-timer (C-11): fires on `orca-pull.sh` starvation (>N seconds without successful run); structured alert path | C-11 |
|
||||
| P09-T3 | `orca collector` CLI: opt-in collector for per-peer state snapshots; writes to `cluster.json` (C-12 opt-in) | C-12 |
|
||||
| P09-T4 | Tests: aggregator pulls drift-events + triggers remediation; watchdog fires on starvation; collector opt-in | REQ-107, C-11 |
|
||||
|
||||
**Must-haves**: aggregator pulls drift-events + remediation; watchdog fires on starvation; collector opt-in.
|
||||
|
||||
### Phase P13 — `orca ns` subcommands + deprecation warnings (Wave 6)
|
||||
**REQs**: REQ-068
|
||||
**Persona**: lead-developer
|
||||
**Territory**: `internal/cli/ns.go`
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P13-T1 | Full `orca ns` surface: `list`, `create`, `delete`, `inspect`, `validate`, `inherit`, `set-constraint` (per R-002 namespace-as-path) | REQ-068 |
|
||||
| P13-T2 | Depprecation warnings: `orca daemon`, `orca cert` (v0.8 mTLS path), `.hcl` jobspec → printed on use; `--no-deprecation-warnings` suppresses (REQ-068) | REQ-068 |
|
||||
| P13-T3 | Tests: all ns subcommands work; deprecation warnings fire on deprecated surface | REQ-068 |
|
||||
|
||||
**Must-haves**: full ns surface; deprecation warnings on deprecated surface.
|
||||
|
||||
### Phase P14a — v0.8→v1.0 data migration + `orca upgrade` (Wave 7, gate C-07)
|
||||
**REQs**: REQ-066; C-07; REQ-115
|
||||
**Persona**: data-engineer + backend-engineer
|
||||
**Territory**: `internal/migration/`, `internal/cli/upgrade.go`
|
||||
**Gate**: C-07 (CA migration spec)
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P14a-T1 | `internal/migration/` package: v0.8 flat layout → v0.9/v0.11 multi-namespace layout; schema migration (0006→next); CA migration per spec (C-07) | REQ-066 |
|
||||
| P14a-T2 | `orca upgrade --to-vX`: thin wrapper (C2=a) around `install.sh` + `orca restore`; handles R-017 Traefik binding cutover (`:443` → `127.0.0.1:8443`) for existing v0.9/v0.10 clusters (Q2=C) | REQ-115 |
|
||||
| P14a-T3 | Tests: v0.8 layout migrates to v0.11 layout; `orca upgrade` handles binding cutover; idempotent | REQ-066, 115 |
|
||||
|
||||
**Must-haves**: v0.8 data migrates to v0.11; `orca upgrade` handles binding cutover; C-07 cleared.
|
||||
|
||||
### Phase P14b — Daemon cutover + rotate-lead (Wave 7)
|
||||
**REQs**: (existing daemon cutover) + REQ-114
|
||||
**Persona**: backend-engineer + lead-developer
|
||||
**Territory**: `internal/cli/daemon.go`, `internal/cli/rotate_lead.go`
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P14b-T1 | Daemon cutover: `orca daemon` becomes `drain-and-stop` (REQ-061 from P05); running-allocation adoption (orphaned allocs adopted by SSH-push path) | existing |
|
||||
| P14b-T2 | `orca cluster rotate-lead`: moves cluster CA + lead state to a new bare-Linux peer (R-003); workloads keep running (certs distributed); SSH key rotation; idempotent (Q2=C) | REQ-114 |
|
||||
| P14b-T3 | Tests: daemon cutover adopts running allocs; rotate-lead moves CA + workloads keep running | REQ-114 |
|
||||
|
||||
**Must-haves**: daemon cutover adopts running allocs; rotate-lead moves CA without downtime.
|
||||
|
||||
### Phase P14c — Mixed-version tolerance (Wave 7)
|
||||
**REQs**: REQ-065, REQ-086; C-13
|
||||
**Persona**: backend-engineer
|
||||
**Territory**: `internal/transport/`, `internal/cli/`
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P14c-T1 | Mixed-version tolerance: lead and peers can run different orca versions during upgrade window; no-orca-on-server enforcement (R-001) | REQ-065, REQ-086 |
|
||||
| P14c-T2 | Tests: mixed-version cluster operates; orca-on-server detected + refused | REQ-065, 086 |
|
||||
|
||||
**Must-haves**: mixed-version tolerance during upgrade; R-001 enforced.
|
||||
|
||||
### Phase P08 — Integration tests + drift-detection tests (Wave 8)
|
||||
**REQs**: REQ-087
|
||||
**Persona**: devops-engineer
|
||||
**Territory**: `tests/integration/`, `scripts/tests/`
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P08-T1 | Expand hermetic test harness: multi-peer setup, txn apply, drift injection, remediation verification (REQ-087) | REQ-087 |
|
||||
| P08-T2 | Drift-detection integration tests: auto-remediation success (edit Traefik config → detect ~10s → remediated); NFS fallback (NFS mount → Path units disabled → polling); rate-limit cooldown (repeated drift → cooldown blocks loop); secret exclusion (edit `/etc/orca/credentials/*` → no drift event) | REQ-087 |
|
||||
| P08-T3 | Tests pass in CoreCI `integration` pipeline (nft exclusively, D-224) | REQ-087 |
|
||||
|
||||
**Must-haves**: integration tests cover drift detection; pass in CoreCI.
|
||||
|
||||
### Phase P15 — README quickstart (Wave 8)
|
||||
**REQs**: REQ-089
|
||||
**Persona**: lead-developer + docs-engineer (phase-specific)
|
||||
**Territory**: `README.md`
|
||||
**Framing**: Q5=A (Nomad-inspired, OS-as-cluster; honest-trade-offs table from doc 3; Proxmox as one node type, not the identity)
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P15-T1 | README.md: status line (v0.11 complete, v1.0 UAT-gated); install example; subcommand table expanded to ALL v0.11 commands (incl. drift, nft, migrate, rotate-lead, upgrade, logs --all-nodes, doctor mTLS); honest-trade-offs table (doc 3 §4.6); Nomad-inspired framing (Q5=A) | REQ-089 |
|
||||
| P15-T2 | Verify all cited CLI commands exist in `internal/cli/` (C-22-style grounding gate) | REQ-089 |
|
||||
|
||||
**Must-haves**: README subcommand table matches `internal/cli/` exactly; honest-trade-offs table present; Nomad-inspired framing.
|
||||
|
||||
### Phase P16 — Final review + ship + audit (Wave 8)
|
||||
**REQs**: all (REQ-099..REQ-118 + existing v0.11 REQs)
|
||||
**Persona**: lead-developer
|
||||
**Vertical slice**: milestone complete → merged to main, tagged, released.
|
||||
|
||||
| Task | Description | REQ |
|
||||
|------|-------------|-----|
|
||||
| P16-T1 | Code review across all phases; auto-apply P0 fixes, flag P1+ for post-hoc | all |
|
||||
| P16-T2 | Audit: reconstruction test (git log matches `.ciagent/`), file discipline, branch hygiene, commit discipline | all |
|
||||
| P16-T3 | Milestone ship: merge phase/16 → milestone → main; tag `v0.10.21` (= v0.11.0 milestone release); release with Linux binary asset; delete all milestone branches | all |
|
||||
| P16-T4 | Complete milestone: mark all v0.11 REQs complete in REQUIREMENTS.md; mark v0.11 complete in ROADMAP.md; clear checkpoint | all |
|
||||
|
||||
**Must-haves**: milestone merged to main; release carries Linux binary; all REQs marked complete; checkpoint cleared.
|
||||
|
||||
## Risks
|
||||
|
||||
- **R1: P10 sizing** — P10 is the largest phase (txn plane + drift detection, 13 tasks). Mitigation: grill may split into P10a/P10b. The plan is structured so P10a (T1-T2, txn plane) and P10b (T3-T13, drift) are separable.
|
||||
- **R2: R-020 deadlock** — hard-gate refusal could block all new txns if a peer is permanently drifted on a require_approval path. Mitigation: `--force` + per-ns scoping (Q4=A); documented in C-09 failure contract.
|
||||
- **R3: Ingress default migration** — existing v0.9/v0.10 clusters run Traefik on `:443`. R-017 makes `127.0.0.1:8443` + nft the default. Mitigation: `orca upgrade` (REQ-115, P14a) handles the binding cutover.
|
||||
- **R4: `orca` system user** — creating a system user on every peer is a new operational requirement. Mitigation: peer-setup emits `useradd -r orca` idempotently (REQ-111); documented in P10.
|
||||
- **R5: Scope ceiling** — v0.11 stays at 23 phases (no new phases), but P09/P10/P15.5 grow substantially. Mitigation: wave ordering isolates the largest work (Wave 5) so it can be split without affecting other waves.
|
||||
@@ -0,0 +1,395 @@
|
||||
# Plan v0.12: Security Hardening (Zero-Trust Identity)
|
||||
|
||||
**Status**: Phase 0 plan. 29 phases (P0 + P01..P27 + P28 final). Wave
|
||||
ordering, persona assignments, and binding conditions. GRILL will
|
||||
pressure-test and may split/merge.
|
||||
|
||||
## Milestone identity
|
||||
|
||||
- **Label**: `v0.12-security-hardening`
|
||||
- **Type**: feature (P04, P05 ship `feat`)
|
||||
- **Tag line**: v0.11.x patches (`v0.11.0`..`v0.11.28`)
|
||||
- **Final phase patch** = milestone release = `v0.11.28` (no separate `v0.12.0`)
|
||||
- **Branch**: `milestone/v0.12-security-hardening`
|
||||
- **v1.0.0**: deferred for post-v0.12 UAT (per v0.11 PRD)
|
||||
|
||||
## Wave ordering
|
||||
|
||||
Waves are dependency-ordered. Within a wave, phases run in sequence
|
||||
(parallelization disabled per config.json `parallelization.enabled=false`).
|
||||
|
||||
### Phase 0 — Pre-execution (all personas, lead-developer coordinates)
|
||||
|
||||
Stages: SPECIFY -> CLARIFY -> RESEARCH -> IDEATE -> PLAN -> GRILL -> SHIP.
|
||||
|
||||
- SPECIFY: v0.12 in config.json + PROJECT.md (done).
|
||||
- CLARIFY: D-238..D-247 (done, CLARIFY_v0.12.md).
|
||||
- RESEARCH: threat model F1..F25 + zero-trust identity model (done,
|
||||
RESEARCH_v0.12.md). Resolves RQ-1 (WebAuthn as password-free upstream).
|
||||
- IDEATE: 30 ideas accepted -> REQ-119..REQ-148 (done, IDEATION_v0.12.md).
|
||||
- PLAN: this document.
|
||||
- GRILL: ratify C-29..C-38, split/merge as needed.
|
||||
- SHIP: tag `v0.11.0`.
|
||||
|
||||
**Commit**: `docs(P00): v0.12 security-hardening phase 0 (specify/clarify/research/ideate/plan/grill)`
|
||||
|
||||
### Wave A — Critical injection & traversal (backend-engineer)
|
||||
|
||||
Vertical slice: stop the bleeding first. Three independent fixes, no
|
||||
inter-dependencies.
|
||||
|
||||
#### P01 — Command injection (podman/wasm) — REQ-119, F3
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/runtime/`)
|
||||
- **Tasks**:
|
||||
1. Add `shellQuote` helper (or use `golang.org/x/crypto/ssh`-safe quoting) to `internal/runtime/`.
|
||||
2. Fix `podman.go:57`: `fmt.Sprintf("podman run -d --name %s %q %s", name, image, shellQuote(cmdStr))`.
|
||||
3. Fix `wasm.go:39`: same pattern for `wasmtime run`.
|
||||
4. Add Go regression tests: `;`, `|`, `$()`, backticks, newline, `$IFS`, `<>()` injection attempts.
|
||||
5. Add bats test: a jobspec with a malicious command runs the literal command, not the injected shell.
|
||||
- **Must-haves**: all injection tests pass; existing podman/wasm tests still pass.
|
||||
- **Commit**: `fix(P01): command injection in podman/wasm runtimes (REQ-119, F3)`
|
||||
- **Tag**: `v0.11.1`
|
||||
|
||||
#### P02 — Namespace path traversal — REQ-120, F4
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/ns/`, `internal/cli/ns.go`)
|
||||
- **Tasks**:
|
||||
1. Add `validateNamespaceName(name)` to `internal/ns/`: reject `..`, `/`, leading `-`, null bytes, control chars, empty, length > 128.
|
||||
2. Wire into `ns create`, `ns inherit`, `ns set-constraint`, and any path-accepting ns command.
|
||||
3. Add Go fuzz test (`FuzzValidateNamespaceName`).
|
||||
4. Add regression test: `orca ns create "../../etc"` fails with a clear error.
|
||||
- **Must-haves**: fuzz test passes 10k iterations; `..`/`/`/null rejected.
|
||||
- **Commit**: `fix(P02): namespace path traversal (REQ-120, F4)`
|
||||
- **Tag**: `v0.11.2`
|
||||
|
||||
#### P03 — Txn apply path allowlist — REQ-121, F5
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/txn/`)
|
||||
- **Tasks**:
|
||||
1. In `txn.go:renderApplyScript`, add path validation to the python heredoc: every `path` in `desired-state.json` must match a prefix in the allowlist (`/etc/orca/`, `/etc/traefik/orca*`, `/etc/systemd/system/orca-*`, `/etc/nftables.d/orca*`, `/etc/syncthing/orca*`).
|
||||
2. Reject with a clear error + exit code on mismatch.
|
||||
3. Add Go test: a desired-state with `"path": "/etc/shadow"` is rejected.
|
||||
4. Add bats test: `orca-pull.sh` with a crafted manifest refuses.
|
||||
- **Must-haves**: arbitrary-path writes rejected; legitimate paths still apply.
|
||||
- **Commit**: `fix(P03): txn apply path allowlist (REQ-121, F5)`
|
||||
- **Tag**: `v0.11.3`
|
||||
|
||||
### Wave B — Zero-trust identity (backend-engineer + lead-developer)
|
||||
|
||||
The architectural foundation. P04/P05 are `feat` phases; P06/P07/P08
|
||||
are `fix`/`refactor` that depend on them.
|
||||
|
||||
#### P04 — OIDC client + bundled Dex — REQ-144
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/cli/`, `internal/identity/`)
|
||||
- **Tasks**:
|
||||
1. Add `github.com/coreos/go-oidc/v3` dependency.
|
||||
2. `internal/identity/oidc.go`: OIDC client (provider discovery, JWKS cache + refresh, ID token verification, token storage at `~/.orca/credentials.json` 0600).
|
||||
3. `orca auth login`/`logout`/`status` CLI: browser auth-code + PKCE + local loopback redirect (`127.0.0.1:<port>/callback`); headless device-code fallback.
|
||||
4. `orca auth init-idp`: bootstrap bundled Dex (systemd unit + config template + Traefik route) on the lead; `--rp-id <domain>` config.
|
||||
5. OIDC config block in `internal/config/`: `oidc.issuer`, `client_id`, `client_secret`, `scopes`.
|
||||
6. BYO external IdP override: `oidc.issuer` repoint bypasses bundled Dex.
|
||||
7. Go tests: mock OIDC provider, JWKS rotation, token refresh, login/logout flow.
|
||||
- **Must-haves**: `orca auth login` produces a valid ID token; `orca auth status` shows it; `--oidc` flag gated; offline Dex quickstart doc'd.
|
||||
- **Commit**: `feat(P04): OIDC client + bundled Dex (REQ-144, D-239, D-242)`
|
||||
- **Tag**: `v0.11.4`
|
||||
|
||||
#### P05 — WebAuthn connector for Dex — REQ-148
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/identity/`, new `internal/webauthn/`)
|
||||
- **Tasks**:
|
||||
1. Add `github.com/go-webauthn/webauthn` dependency.
|
||||
2. `internal/webauthn/connector.go`: Dex connector (~300 LoC) -- registration + login ceremonies at `/orca/webauthn/{register,login}`.
|
||||
3. `internal/webauthn/store.go`: passkey storage SQLite at `ClusterDir()/webauthn-credentials.db` (0600); schema: `credentials(user_id, credential_id, public_key, sign_count, aaguid, created_at)`.
|
||||
4. `orca auth register` CLI: browser flow to register a new passkey.
|
||||
5. RP ID = cluster Traefik domain (from `orca auth init-idp --rp-id`); secure context via step-ca cert (R-017).
|
||||
6. Go tests using `go-webauthn` virtual-authenticator test helpers (no hardware key).
|
||||
- **Must-haves**: register + login flow works end-to-end against the bundled Dex; public keys only stored; virtual-authenticator tests pass.
|
||||
- **Commit**: `feat(P05): WebAuthn connector for Dex (REQ-148, D-240, C-38)`
|
||||
- **Tag**: `v0.11.5`
|
||||
|
||||
#### P06 — ACL rewrite to OIDC claims + enforcement — REQ-145, REQ-122, F1
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/acl/`, `internal/daemon/`, `internal/sshpush/`)
|
||||
- **Tasks**:
|
||||
1. Remove `KindToken` from `internal/acl/acl.go` entirely.
|
||||
2. Add `KindOidc`: maps `sub` + `groups` -> namespace permissions.
|
||||
3. `acl.Check` takes an OIDC claims struct (or SPIFFE SVID for machine identity).
|
||||
4. Wire `acl.Check` into daemon handlers (read/write/admin by route).
|
||||
5. Wire `acl.Check` into SSH-push applier: validate `ORCA_OIDC_TOKEN` env var against JWKS before applying any txn.
|
||||
6. `acl.json` file mode tightened to 0600.
|
||||
7. Deny-by-default enforced; actor recorded in audit log.
|
||||
8. Go tests: ACL-negative (unauthorized sub denied), ACL-positive, machine identity (SVID) still works.
|
||||
- **Must-haves**: no request applies without a valid OIDC token or SVID; `KindToken` removed; deny-by-default enforced.
|
||||
- **Commit**: `fix(P06): ACL rewrite to OIDC claims + enforcement (REQ-145, REQ-122, F1)`
|
||||
- **Tag**: `v0.11.6`
|
||||
|
||||
#### P07 — Remove all password/token paths — REQ-146, R-021, C-34
|
||||
|
||||
- **Persona**: lead-developer (territory: `internal/proxmox/`, `internal/stepca/`, `internal/cli/`)
|
||||
- **Tasks**:
|
||||
1. Remove `--password`/`$ORCA_PROXMOX_PASSWORD` from Proxmox join (`proxmox/bootstrap.go:29`); replace with pre-staged-key-only or `step ssh` OIDC cert exchange.
|
||||
2. Remove step-ca `--password-file` provisioner; migrate to OIDC provisioner (step-ca natively supports OIDC).
|
||||
3. Remove any bare-token CLI paths (already removed in P06, but sweep for stragglers).
|
||||
4. Add deprecation/migration docs: `--accept-identity-migration` flag on `orca upgrade` (P22 enforces).
|
||||
5. Go tests: `--password` flag is rejected with a clear error pointing to the migration guide.
|
||||
- **Must-haves**: no password accepted anywhere; `--password` rejected; step-ca OIDC provisioner works.
|
||||
- **Commit**: `fix(P07): remove all password/token paths (REQ-146, R-021, C-34) -- BREAKING`
|
||||
- **Tag**: `v0.11.7`
|
||||
|
||||
#### P08 — Master key seal-to-OIDC + Shamir — REQ-147, D-241, C-35
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/secrets/`, new `internal/seal/`)
|
||||
- **Tasks**:
|
||||
1. `internal/seal/seal.go`: seal/unseal using HKDF-SHA256 of OIDC ID token `sub` + fresh 32-byte salt; sealed blob at `ClusterDir()/master.key.sealed` (0600).
|
||||
2. Shamir 3-of-5: `internal/seal/shamir.go` (using `golang.org/x/crypto/...` or a vendored Shamir impl); print 5 shards at seal time.
|
||||
3. `orca cluster unseal`/`seal` CLI: unseal via OIDC auth; `--recovery` + 3 shards for IdP-lost case.
|
||||
4. mTLS-only offline path: seal key derived from cluster CA.
|
||||
5. Master key zeroed on shutdown (use `memguard` or manual `crypto/rand` overwrite).
|
||||
6. Go tests: seal -> unseal round-trip; recovery with 3 shards; 2 shards fails; raw key never on disk (assert no `master.key` file, only `master.key.sealed`).
|
||||
- **Must-haves**: raw master key never touches disk; unseal works via OIDC; recovery works with 3-of-5 shards.
|
||||
- **Commit**: `feat(P08): master key seal-to-OIDC + Shamir 3-of-5 (REQ-147, D-241, C-35)`
|
||||
- **Tag**: `v0.11.8`
|
||||
|
||||
### Wave C — Auth & integrity (backend-engineer + data-engineer)
|
||||
|
||||
#### P09 — Daemon auth hardening — REQ-123, REQ-124, F6, F24
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/daemon/`)
|
||||
- **Tasks**:
|
||||
1. Remove plaintext mode entirely (mandatory mTLS).
|
||||
2. Accept OIDC bearer as second factor on human-facing endpoints.
|
||||
3. `http.MaxBytesReader` on all JSON-decoding handlers; `MaxHeaderBytes` set.
|
||||
4. pprof loopback-only by default; `--pprof-allow-public` requires confirmation.
|
||||
5. Go tests: plaintext mode rejected; oversized body rejected; pprof non-loopback rejected.
|
||||
- **Commit**: `fix(P09): daemon auth hardening (REQ-123, REQ-124, F6, F24)`
|
||||
- **Tag**: `v0.11.9`
|
||||
|
||||
#### P10 — Audit log tamper-evidence — REQ-125, F2
|
||||
|
||||
- **Persona**: data-engineer (territory: `internal/audit/`, `internal/store/`)
|
||||
- **Tasks**:
|
||||
1. Add `prev_hash` + `entry_hash` columns to `audit_log` table (migration 0008).
|
||||
2. `AuditRepo.Append` computes `entry_hash = sha256(prev_hash || payload)`, stores it; HMAC-SHA256 under master key on the chain head (stored separately).
|
||||
3. SQLite trigger blocks UPDATE/DELETE on `audit_log`.
|
||||
4. `orca doctor audit` verifies the chain (recomputes hashes, checks HMAC).
|
||||
5. Actor field carries OIDC `sub` or SPIFFE SVID.
|
||||
6. Go tests: tamper detection (modify a row -> doctor fails); append-only enforcement (DELETE fails).
|
||||
- **Commit**: `fix(P10): audit log tamper-evidence (REQ-125, F2)`
|
||||
- **Tag**: `v0.11.10`
|
||||
|
||||
#### P11 — SVID chain validation — REQ-126, F9
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/identity/`)
|
||||
- **Tasks**:
|
||||
1. `VerifySVID` loads the CA pool (from `ClusterDir()/ca.crt` or step-ca root) and validates the full cert chain.
|
||||
2. Reject certs signed by unknown CAs even with correct URI SAN.
|
||||
3. Go tests: cert from wrong CA rejected; cert from correct CA + correct URI accepted; expired cert rejected.
|
||||
- **Commit**: `fix(P11): SVID chain validation (REQ-126, F9)`
|
||||
- **Tag**: `v0.11.11`
|
||||
|
||||
#### P12 — Backup symlink validation — REQ-127, F7
|
||||
|
||||
- **Persona**: data-engineer (territory: `internal/backup/`)
|
||||
- **Tasks**:
|
||||
1. `Restore` rejects `Linkname` that's absolute, contains `..`, or points outside `ORCA_HOME`.
|
||||
2. Regression test with crafted tarball containing a symlink to `/etc/shadow`.
|
||||
- **Commit**: `fix(P12): backup symlink validation (REQ-127, F7)`
|
||||
- **Tag**: `v0.11.12`
|
||||
|
||||
### Wave D — Crypto & secrets (backend-engineer)
|
||||
|
||||
#### P13 — step-ca /tmp hardening — REQ-128, F10
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/stepca/`, `internal/identity/`)
|
||||
- **Tasks**:
|
||||
1. `step ca certificate` writes to `0600` temp under `ClusterDir()/step-tmp/` (or `TMPDIR` override).
|
||||
2. Cleanup in `defer`; `mkdir -p` with 0700 on the temp dir.
|
||||
3. Go test: assert temp file mode is 0600; assert cleanup on success + failure.
|
||||
- **Commit**: `fix(P13): step-ca /tmp hardening (REQ-128, F10)`
|
||||
- **Tag**: `v0.11.13`
|
||||
|
||||
#### P14 — Master key rotation — REQ-129, F12, C-30
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/secrets/`, `internal/seal/`)
|
||||
- **Tasks**:
|
||||
1. `orca secrets rotate-master`: generate new master key, re-encrypt all namespace secrets, re-seal to OIDC.
|
||||
2. `--dry-run` reports affected namespaces without writing.
|
||||
3. Atomic per-namespace re-encryption; auto-rollback to old sealed key on any ns failure.
|
||||
4. Go tests: rotation succeeds; partial failure rolls back; dry-run doesn't write.
|
||||
- **Commit**: `fix(P14): master key rotation (REQ-129, F12, C-30)`
|
||||
- **Tag**: `v0.11.14`
|
||||
|
||||
#### P15 — File-mode audit expansion — REQ-130, F13
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/security/`, `internal/cli/doctor.go`)
|
||||
- **Tasks**:
|
||||
1. `EnforceFileModes` extended to SSH key, master key (sealed blob), server cert/key, known_hosts.
|
||||
2. `orca doctor modes` checks all.
|
||||
3. Startup refuses to run on violation.
|
||||
4. Go tests: looser mode -> doctor fails + startup refuses.
|
||||
- **Commit**: `fix(P15): file-mode audit expansion (REQ-130, F13)`
|
||||
- **Tag**: `v0.11.15`
|
||||
|
||||
### Wave E — OS scripts & emitters (backend-engineer + lead-developer)
|
||||
|
||||
#### P16 — aggregate.sh JSON injection + drift-gate fix — REQ-131, F11, F18
|
||||
|
||||
- **Persona**: lead-developer (territory: `scripts/`)
|
||||
- **Tasks**:
|
||||
1. Replace `printf` interpolation in `orca-aggregate.sh:64` with `jq`-based JSON construction (or a Go-side aggregator emitting JSON).
|
||||
2. Fix `orca-pull.sh` R-020 parsing to use `jq` instead of grep.
|
||||
3. Bats tests: malicious peer output doesn't corrupt `cluster.json`; drift gate correctly excludes acknowledged drift.
|
||||
- **Commit**: `fix(P16): aggregate.sh JSON injection + drift-gate fix (REQ-131, F11, F18)`
|
||||
- **Tag**: `v0.11.16`
|
||||
|
||||
#### P17 — install.sh checksum+GPG verification — REQ-132, F14
|
||||
|
||||
- **Persona**: lead-developer (territory: `scripts/install.sh`, `scripts/release.sh`)
|
||||
- **Tasks**:
|
||||
1. `release.sh` publishes `SHA256SUMS` + `SHA256SUMS.asc` (GPG-signed) alongside the tarball.
|
||||
2. `install.sh` verifies SHA256 + GPG signature before `tar -xzf`; fail closed on mismatch.
|
||||
3. `--no-verify` escape hatch (documented, warns).
|
||||
4. Bats tests: tampered tarball rejected; valid tarball accepted.
|
||||
- **Commit**: `fix(P17): install.sh checksum+GPG verification (REQ-132, F14)`
|
||||
- **Tag**: `v0.11.17`
|
||||
|
||||
#### P18 — nftables ruleset hardening — REQ-133, F21
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/emitter/nft.go`)
|
||||
- **Tasks**:
|
||||
1. Add conntrack bounds (`ct state established,related accept`).
|
||||
2. Input default-deny on the orca chain.
|
||||
3. Drop invalid packets (`ct state invalid drop`).
|
||||
4. `orca doctor nft` audits live ruleset against emitted one.
|
||||
5. Go tests: emitted ruleset contains the new rules; doctor detects drift.
|
||||
- **Commit**: `fix(P18): nftables ruleset hardening (REQ-133, F21)`
|
||||
- **Tag**: `v0.11.18`
|
||||
|
||||
#### P19 — sudoers hardening — REQ-134, F22
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/proxmox/bootstrap.go`)
|
||||
- **Tasks**:
|
||||
1. Add NOEXEC to `apt-get`/`dpkg` in the OrcaOperator sudoers (or remove if unused).
|
||||
2. `orca doctor proxmox` audits the sudoers file against the expected allowlist.
|
||||
3. Go tests: emitted sudoers has NOEXEC; doctor detects drift.
|
||||
- **Commit**: `fix(P19): sudoers hardening (REQ-134, F22)`
|
||||
- **Tag**: `v0.11.19`
|
||||
|
||||
#### P20 — System user consistency — REQ-135, F23
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/proxmox/bootstrap.go`, `internal/cli/peer_setup.go`)
|
||||
- **Tasks**:
|
||||
1. Proxmox bootstrap creates `nologin` system user (`-r -s /usr/sbin/nologin`), matching peer-setup.
|
||||
2. `orca doctor` flags inconsistency on existing peers.
|
||||
3. `orca upgrade` migrates existing `-m -s /bin/bash` users to `-r -s /usr/sbin/nologin`.
|
||||
4. Go tests: emitted useradd matches; doctor detects the old style.
|
||||
- **Commit**: `fix(P20): system user consistency (REQ-135, F23)`
|
||||
- **Tag**: `v0.11.20`
|
||||
|
||||
### Wave F — State storage & migration (data-engineer)
|
||||
|
||||
#### P21 — SQLite file-mode + at-rest encryption — REQ-136, F8, C-31
|
||||
|
||||
- **Persona**: data-engineer (territory: `internal/store/`)
|
||||
- **Tasks**:
|
||||
1. `store.Open` sets DB file mode 0600 (via `os.Chmod` after open, since SQLite creates with umask).
|
||||
2. Evaluate SQLCipher envelope (CGO-free check). If infeasible without CGO (breaks D-008), fall back to file-mode 0600 + documented threat per C-31.
|
||||
3. Document the decision in RESEARCH/PROJECT.
|
||||
4. Go tests: DB file mode is 0600 after open.
|
||||
- **Commit**: `fix(P21): SQLite file-mode + at-rest encryption (REQ-136, F8, C-31)`
|
||||
- **Tag**: `v0.11.21`
|
||||
|
||||
#### P22 — Migration safety + identity migration — REQ-137, F19, C-34
|
||||
|
||||
- **Persona**: data-engineer (territory: `internal/migration/`, `internal/cli/upgrade.go`)
|
||||
- **Tasks**:
|
||||
1. `copyFile` -> atomic temp+rename.
|
||||
2. `migrateDBSchema` runs in a transaction with `foreign_keys(ON)`.
|
||||
3. Pre-migration backup step (uses `internal/backup`).
|
||||
4. Document manual rollback (restore from backup).
|
||||
5. `orca upgrade` refuses v0.11 clusters using `--password`/bare-tokens without `--accept-identity-migration` (C-34).
|
||||
6. Go tests: migration is atomic; partial failure rolls back; `--accept-identity-migration` gate works.
|
||||
- **Commit**: `fix(P22): migration safety + identity migration (REQ-137, F19, C-34)`
|
||||
- **Tag**: `v0.11.22`
|
||||
|
||||
### Wave G — Dual-write closure (lead-developer, gated by C-29)
|
||||
|
||||
#### P23 — Legacy CA/mTLS/daemon + step-ca password-provisioner deletion — REQ-138, F16
|
||||
|
||||
- **Persona**: lead-developer (territory: `internal/security/ca.go`, `internal/transport/mtls.go`, `internal/daemon/`, `internal/stepca/`, `internal/certpaths/`)
|
||||
- **Pre-gate (C-29)**: P06, P08, P09, P11 must all be shipped.
|
||||
- **Tasks**:
|
||||
1. Remove `internal/security/ca.go` legacy CA; migrate `orca init` and `orca cert *` to step-ca exclusively.
|
||||
2. Remove `internal/transport/mtls.go` deprecated path.
|
||||
3. Remove daemon plaintext mode (already killed in P09, but delete the code path).
|
||||
4. Remove `internal/certpaths/` (v0.8 flat layout); `internal/paths/` is the only layout.
|
||||
5. Delete step-ca `--password-file` provisioner (already replaced by OIDC provisioner in P07).
|
||||
6. Full test suite must pass after deletion.
|
||||
- **Must-haves**: `orca init` + `orca cert *` work via step-ca only; no legacy code compiled.
|
||||
- **Commit**: `refactor(P23): delete legacy CA/mTLS/daemon + step-ca password-provisioner (REQ-138, F16, C-29)`
|
||||
- **Tag**: `v0.11.23`
|
||||
|
||||
### Wave H — Defense-in-depth (backend-engineer)
|
||||
|
||||
#### P24 — known_hosts tightening + transport hardening — REQ-139, F15, F25
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/security/flock.go`, `internal/sshpush/`)
|
||||
- **Tasks**:
|
||||
1. `Flock` tightens pre-existing looser perms to 0600 (chmod after open if looser).
|
||||
2. `classifyDialErr` switched from substring to typed errors (use `*ssh.ExitError`, `net.Error` type assertions).
|
||||
3. Add SSH-exec rate limiting (token bucket per peer, default 10 req/s).
|
||||
4. Go tests: looser perms tightened; typed errors classified correctly; rate limit enforced.
|
||||
- **Commit**: `fix(P24): known_hosts tightening + transport hardening (REQ-139, F15, F25)`
|
||||
- **Tag**: `v0.11.24`
|
||||
|
||||
#### P25 — Drift event authentication — REQ-140, F18
|
||||
|
||||
- **Persona**: backend-engineer (territory: `internal/drift/`, `scripts/orca-drift-notify.sh`)
|
||||
- **Tasks**:
|
||||
1. Per-peer HMAC key (derived from master key via HKDF); deployed to peers at `0600` owned by `orca`.
|
||||
2. `orca-drift-notify.sh` signs each event with the HMAC; aggregator rejects unsigned/forged events.
|
||||
3. Go tests: forged event rejected; valid event accepted.
|
||||
- **Commit**: `fix(P25): drift event authentication (REQ-140, F18)`
|
||||
- **Tag**: `v0.11.25`
|
||||
|
||||
#### P26 — Security integration test suite — REQ-141, C-33
|
||||
|
||||
- **Persona**: backend-engineer (territory: `tests/`)
|
||||
- **Tasks**:
|
||||
1. Hermetic harness exercising: injection (P01), traversal (P02), symlink (P12), drift-forgery (P25), audit-tamper (P10), daemon-auth-negative (P09), OIDC mock-IdP flow (P04), ACL-with-OIDC-claims negative (P06), unseal/seal (P08), WebAuthn virtual-authenticator ceremony (P05), password-removal regression (P07 -- assert `--password` rejected).
|
||||
2. Gates in `.coreci.yml` `validate` pipeline (C-33).
|
||||
3. Bats + Go test runner.
|
||||
- **Commit**: `test(P26): security integration test suite (REQ-141, C-33)`
|
||||
- **Tag**: `v0.11.26`
|
||||
|
||||
### Wave I — Documentation & release (lead-developer)
|
||||
|
||||
#### P27 — Zero-trust + OIDC + WebAuthn + threat-model docs — REQ-142
|
||||
|
||||
- **Persona**: lead-developer (territory: `docs/`, `README.md`)
|
||||
- **Tasks**:
|
||||
1. `docs/threat-model.md`: STRIDE per component, zero-trust model, OIDC data-flow diagram, OS surface diagram, residual risk register.
|
||||
2. `docs/oidc.md`: configure your IdP, bundled Dex offline quickstart, claim-to-namespace mapping, BYO-IdP override.
|
||||
3. `docs/webauthn.md`: passkey registration, RP ID, secure context, recovery flow.
|
||||
4. `docs/security-runbook.md`: unseal/seal, master key rotation, incident response, sudoers audit, nft audit, Shamir recovery.
|
||||
5. README security section names "no orca credentials" as an invariant (R-021).
|
||||
- **Commit**: `docs(P27): zero-trust + OIDC + WebAuthn + threat-model docs (REQ-142)`
|
||||
- **Tag**: `v0.11.27`
|
||||
|
||||
#### P28 — Final review + ship + audit — REQ-143
|
||||
|
||||
- **Persona**: lead-developer (coordinates)
|
||||
- **Tasks**:
|
||||
1. `ciagent-review` multi-persona review across all phases.
|
||||
2. `ciagent-audit` reconstruction test (git log matches `.ciagent/` files).
|
||||
3. C-32 human-gate: confirm GITEA_TOKEN rotated + `.env` re-seeded (escalation hook if pending).
|
||||
4. Merge `phase/28` -> `milestone/v0.12-security-hardening`.
|
||||
5. Merge `milestone/v0.12-security-hardening` -> `main` (rebase-then-fast-forward).
|
||||
6. Tag `v0.11.28` (= v0.12 milestone release per feature-milestone rule).
|
||||
7. Create Gitea release with full milestone summary.
|
||||
8. Delete milestone + phase branches (tags preserve history).
|
||||
9. Update REQUIREMENTS.md (mark all v0.12 REQs complete) + ROADMAP.md (mark v0.12 complete).
|
||||
- **Commit**: `docs(milestone): complete v0.12 -- Security Hardening (Zero-Trust Identity) (29 phases shipped)`
|
||||
- **Tag**: `v0.11.28`
|
||||
@@ -0,0 +1,41 @@
|
||||
# PRD v0.11 Extension: Production Hardening
|
||||
|
||||
**Status**: This file EXTENDS (does not supersede) `PRD_v0.9.md`. The
|
||||
16 load-bearing rules R-001…R-016 remain in effect; this file adds
|
||||
R-017…R-020, adopted per operator decision Q1=A (2026-08-07) after
|
||||
ingestion of 5 research documents covering ingress hardening, drift
|
||||
detection, platform-engineer positioning, strategic framing, and the
|
||||
systemd Path unit implementation.
|
||||
|
||||
## New load-bearing rules (R-017…R-020)
|
||||
|
||||
| ID | Rule |
|
||||
|---|---|
|
||||
| R-017 | Cluster ingress default is the hybrid: Traefik binds on `127.0.0.1:8443` (and `127.0.0.1:8080` for HTTP). Public `:443` / `:80` traffic is DNAT'd via nftables to Traefik. Cross-node cluster mesh stays on the cluster-internal private IP. Operators can opt out with `orca cluster config --public-binding=...`. Workloads can opt in to pure iptables with `service { ingress: native }`. In all cases, mTLS termination is unchanged: Traefik holds the certs. |
|
||||
| R-018 | Default drift detection cadence is 60s. Operators can tune per-path: critical_paths (5s default, systemd Path units enabled), standard_paths (30s default), file_watch_paths (systemd Path units, event-driven, R-001-clean). |
|
||||
| R-019 | Drift detector is a BACKSTOP. Primary failure detection is: systemd (`Type=notify`) for process state, Traefik health checks for routing state, step-ca cert notifications for cert expiry, Syncthing completion events for replication state. Drift detector exists to catch config divergence, not workload failures. |
|
||||
| R-020 | Hard gate: applier refuses new txns if pre-flight consistency check fails. Drift must be resolved before new state is committed. Auto-remediation is enabled by default for critical config paths but disabled for systemd unit files (require operator approval). Override: `--force` flag + per-namespace scoping (a drifted peer in ns-A does not block ns-B). |
|
||||
|
||||
## Relationship to R-001…R-016
|
||||
|
||||
R-017…R-020 are *extensions*, not reversals. They are compatible with:
|
||||
- R-001 (no Orca binary on servers) — systemd Path units are OS-native; nftables is OS-native; no Orca daemon introduced.
|
||||
- R-006 (mTLS by default; Traefik load-bearing) — R-017 preserves Traefik as the mTLS termination point; only the binding address changes.
|
||||
- R-007 (sockets by default) — unchanged; R-017 is about the public-ingress edge, not inter-workload sockets.
|
||||
- R-010 (transactional control plane) — R-018/R-019/R-020 refine the txn plane's drift-detection contract (C-09).
|
||||
|
||||
## New D-series (D-215…D-237)
|
||||
|
||||
D-215…D-226 (ingress hybrid, doc 1) and D-227…D-237 (drift detection, doc 5)
|
||||
are recorded in `PROJECT.md` § v0.11 Clarified Decisions. No collisions with
|
||||
existing D-series (ends at D-206).
|
||||
|
||||
## Milestone scope
|
||||
|
||||
v0.11 "Production Hardening" — 23 phases (P00…P16). Research adds scope to
|
||||
P09 (drift-event aggregation), P10 (drift detection + transactional plane),
|
||||
and P15.5 (ingress hybrid + threat model). Five net-new CLI commands
|
||||
(`orca cluster rotate-lead`, `orca upgrade`, `orca job migrate`,
|
||||
`orca logs --all-nodes`, `orca doctor mTLS`) are folded into existing
|
||||
phases per operator decision Q2=C. No new phases added (Q3=A folds ingress
|
||||
into P15.5).
|
||||
@@ -1,8 +1,8 @@
|
||||
# Orca — Comprehensive Product Requirements Document (v0.9/v1.0)
|
||||
# Orca — Comprehensive Product Requirements Document (v0.9/v0.10)
|
||||
|
||||
**Audience:** Operators, AI agents, downstream tooling authors
|
||||
|
||||
> This PRD SUPERSEDES the shipped v0.1–v0.8 architecture. The v0.9 and v1.0
|
||||
> This PRD SUPERSEDES the shipped v0.1–v0.8 architecture. The v0.9 and v0.10
|
||||
> milestones implement a re-architecture whose load-bearing rules (R-001…R-016)
|
||||
> and decisions (D-068…D-206) replace or demote several earlier documented
|
||||
> decisions. See §22 decision-trace and the Supersession Table in
|
||||
@@ -15,13 +15,13 @@
|
||||
| Spec lock-in | ✅ R-001…R-016 + D-001…D-206 settled |
|
||||
| v0.1–v0.8 implementation | ✅ shipped (REQ-001..060, D-001..D-047) |
|
||||
| v0.9 implementation | ⬜ Phase 0 pre-execution (this file is the spec input) |
|
||||
| v1.0 implementation | ⬜ planning (post-PRD) |
|
||||
| v0.10 implementation | ⬜ planning (post-PRD) |
|
||||
| v1.x multi-host state | ⬜ parked (post-v1.0) |
|
||||
| v2.x full Nomad-HCL | ⬜ parked (post-v1.x) |
|
||||
|
||||
## Override justification (recorded for the grill supersession)
|
||||
|
||||
The v0.9/v1.0 re-architecture is justified on six independent grounds rather
|
||||
The v0.9/v0.10 re-architecture is justified on six independent grounds rather
|
||||
than preference. Each reverses a prior documented decision; the new evidence
|
||||
basis is recorded with the reversal in the Supersession Table:
|
||||
|
||||
@@ -54,7 +54,7 @@ that source; the substantive planning artifacts live in:
|
||||
- `IDEATION_v0.9.md` — 30 ideas (REQ-061..REQ-090), three tiers
|
||||
- `GRILL_v0.9.md` — 9-axis adversarial review, 19 binding conditions, 10 phase challenges
|
||||
- `REQUIREMENTS.md` — REQ-061..REQ-090 appended
|
||||
- `ROADMAP.md` — v0.9 (13 phases) + v1.0 (19 phases) appended
|
||||
- `ROADMAP.md` — v0.9 (13 phases) + v0.10 (19 phases) appended
|
||||
- `PERSONAS.md` — security/network/devops reactivated
|
||||
- `ARCHITECTURE.md` — v0.9 banners + Supersession Table
|
||||
|
||||
@@ -70,7 +70,7 @@ that source; the substantive planning artifacts live in:
|
||||
| R-006 | mTLS on by default; cluster CA = step-ca; Traefik + `LoadCredential=` are load-bearing. |
|
||||
| R-007 | Sockets by default (`/run/orca/alloc-<id>/port-<name>.sock`); `127.0.0.1` opt-in. |
|
||||
| R-008 | CLI results cached locally with per-class TTLs (`orca_cache` SQLite). |
|
||||
| R-009 | CLI host SPOF mitigated by external shared state in v1.x; v1.0 ships the abstractions + cache layer. |
|
||||
| R-009 | CLI host SPOF mitigated by external shared state in v1.x; v0.10 ships the abstractions + cache layer. |
|
||||
| R-010 | Control plane updates are transactional (ArgoCD-style desired-state/lead-applier). |
|
||||
| R-011 | Each namespace has `.env` (plaintext) and `.env.secrets` (AES-256-GCM, per-line nonce); master key per `ORCA_HOME` at `cluster/master.key`. |
|
||||
| R-012 | Workload kinds are `Job`, `Service`, `DaemonSet`; schema-separated by `kind:` in frontmatter. |
|
||||
@@ -84,7 +84,7 @@ that source; the substantive planning artifacts live in:
|
||||
### v0.9 — Workloads + Re-architecture Foundation (13 phases)
|
||||
P00 (deprecation sweep + migration-ordering + txn-design spike + test-infra bootstrap + persona reactivation + doc banners), P0a1 (path resolver + config demotion), P0a2 (namespace CRUD + inheritance), P0b (Markdown jobspec parser + fuzz), P0c (schemas + emitter interface), P01 (SSH-push transport + host-path volumes), P02 (service + Traefik emitter), P03 (update stanza), P04 (lifecycle hooks), P05 (constraints + CLI-side scheduler), P06 (task groups), P07a/P07b/P07c (process+podman / wasmtime [C-01 gated] / pve-vm+ct runtimes), P08 (sockets), P09 (Syncthing [C-02 gated]), P10 (lead rules + migration), P0X (ship + audit).
|
||||
|
||||
### v1.0 — Production Hardening (19 phases)
|
||||
### v0.10 — Production Hardening (19 phases)
|
||||
P00 (CLI cache), P01 (metrics), P01.5 (SPIFFE spike [C-08 gated]), P02 (ACL), P03 (secrets), P04 (backup/restore), P05 (drain + daemon drain-and-stop), P06 (alloc history), P07 (recovery), P08 (integration tests), P09 (collector+aggregator), P10 (transactional plane [C-09 gated]), P11 (job lint), P12 (job verify), P13 (ns subcommands), P14a/P14b/P14c (data / daemon cutover / mixed-version tolerance), P15 (README), P15.5 (threat model [C-19 gated]), P16 (final review + ship — v1.0.0 release).
|
||||
|
||||
See `ROADMAP.md` for the full reordered plan and `GRILL_v0.9.md` for the 19
|
||||
|
||||
+225
-2
@@ -36,6 +36,7 @@ Build a lightweight system to manage and execute workloads across a set of nodes
|
||||
| D-004 | Scheduling algorithm for v0.1? | **Single-node only (no scheduling)** | Multi-node scheduling is out of scope for v0.1. Tasks run on the node they're submitted to. | 0.90 |
|
||||
| D-005 | CLI output format? | **Human-readable by default, `--json` flag for machine consumption** | Serves both humans and AI agents. | 0.95 |
|
||||
| D-006 | Job/task definition format? | **HCL or YAML in `.hcl`/`.yaml` files** | Familiar to Nomad/HashiCorp users; simpler than JSON for humans. | 0.88 |
|
||||
| D-186 | Bash scripts coverage gate: count toward Go gate or exempt? | **Exempt from Go coverage gate; compensating control: bats tests (C-15) + shellcheck + shfmt in CI; every script must have >=1 happy-path and >=1 failure-path bats test** | Bash is a different language surface from Go; the 70%/50% Go coverage gate (D-042/D-047) is Go-specific. Forcing bash into the Go gate would require a coverage tool that does not exist for bash. The compensating control (bats + shellcheck + shfmt) provides equivalent discipline. | 0.82 |
|
||||
| D-007 | Authentication? | **mTLS for v0.1, token-based deferred** | mTLS is the most secure default. Tokens can be added later if needed. | 0.80 |
|
||||
| D-008 | Container runtime? | **Direct process execution (no container runtime) for v0.1** | Avoids the Docker/container dependency. Pure process management. | 0.85 |
|
||||
| D-009 | Configuration file location? | **`~/.orca/config.hcl` and `/etc/orca/orca.hcl`** | Standard XDG-style paths. | 0.90 |
|
||||
@@ -380,7 +381,7 @@ within the `clarify_budget` (10):
|
||||
|
||||
---
|
||||
|
||||
# v0.9/v1.0 — Re-architecture Scope Summary (Supersedes v0.1–v0.8 architecture)
|
||||
# v0.9/v0.10 — Re-architecture Scope Summary (Supersedes v0.1–v0.8 architecture)
|
||||
|
||||
v0.9 is the first DIRECTION-CHANGE milestone in the project's history.
|
||||
It supersedes the shipped v0.1–v0.8 architecture per the adopted PRD
|
||||
@@ -439,8 +440,230 @@ are recorded in `REQUIREMENTS.md`. The reordered phase plan is in
|
||||
| ID | Question | Decision | Rationale | Confidence |
|
||||
|----|----------|----------|-----------|------------|
|
||||
| D-101 | Cluster CA: internal Go CA (AD-010) or step-ca (external)? | **step-ca (apt-installed)** | Externally mandated per override ground 2; AD-010's "too heavyweight" rationale reversed. CLI wraps `step` CLI via SSH (no Go step-ca client library — keep zero-new-dep posture if possible, or add `github.com/smallstep/cli` as a dep). **Gated by C-07** (CA migration spec). | 0.74 |
|
||||
| D-068 | Workload identity: internal X.509 CA or SPIFFE SVIDs? | **SPIFFE SVIDs minted at submit time via step-ca** | Multi-tenancy (override ground 3) requires per-workload identity model; SPIFFE is the standard. SPIFFE ID `spiffe://orca/ns/<ns>/job/<name>/alloc/<id>` as SAN. **Gated by C-08** (mint spike in v1.0-P01.5; fallback to mTLS identity if spike fails). | 0.72 |
|
||||
| D-068 | Workload identity: internal X.509 CA or SPIFFE SVIDs? | **SPIFFE SVIDs minted at submit time via step-ca** | Multi-tenancy (override ground 3) requires per-workload identity model; SPIFFE is the standard. SPIFFE ID `spiffe://orca/ns/<ns>/job/<name>/alloc/<id>` as SAN. **Gated by C-08** (mint spike in v0.10-P01.5; fallback to mTLS identity if spike fails). | 0.72 |
|
||||
| D-088 | Runtime: direct os/exec only (D-008) or multi-runtime? | **5 runtimes: wasm (wasmtime primary), podman, process, pve-vm, pve-ct** | WASM is the primary workload (override ground 4). `processRuntime` wraps existing `executor.go`; others are net-new. Split P07a/b/c per grill PC-10. **P07b gated by C-01** (wasmtime/CGO eval). | 0.82 |
|
||||
| D-158 | Namespace model: single flat root or multi-namespace? | **Multi-namespace under ORCA_HOME (R-002)** | Hard multi-tenant product requirement (override ground 3). `_defaults/` implicit root; `cluster/` for cluster-wide; per-namespace `db/`, `.env`, `.env.secrets`, `jobs/`, `alloc/`, `ns.md`. No namespace column in SQLite. | 0.84 |
|
||||
| D-179 | Jobspec format: HCL canonical (AD-007) or Markdown? | **Markdown with YAML frontmatter canonical (R-013); HCL legacy** | PRD §8 — Markdown + body preservation is the operator-facing format. HCL adapter (REQ-064) preserves `orca job run old-spec.hcl` during migration. | 0.85 |
|
||||
| D-185 | Re-architecture justification: incremental additive or full re-architecture? | **Full re-architecture (overridden by user)** | Six-part evidence basis above; the grill's REPLAN mechanics (PC-01..PC-10, C-01..C-19) adopted as gates. The incremental-additive path was evaluated and rejected on grounds 1 + 5 (daemon failing; SSH-push only viable). | 0.88 |
|
||||
| D-187 | wasmtime Go binding (bytecodealliance/wasmtime-go) is CGO-based — does adopting it revoke D-002 (modernc/sqlite CGO-free cross-compile story)? | **Use the wasmtime CLI (apt-installed on peer) via SSH exec; do NOT import wasmtime-go.** | The Go binding links libwasmtime via cgo and would revoke D-002's CGO-free cross-compile story. The CLI-via-SSH approach (same pattern as podman/qm/pct) avoids CGO entirely. `internal/runtime/wasm.go` imports only stdlib + sshpush. `CGO_ENABLED=0 go build ./...` succeeds. C-01 grill gate SATISFIED; D-002 NOT revoked. Full evaluation in `internal/runtime/C01_WASMTIME_CGO_EVAL.md`. | 0.90 |
|
||||
|
||||
---
|
||||
|
||||
# v0.10 Docs & Install Milestone — Scope Summary
|
||||
|
||||
v0.10 is a focused milestone that closes the documentation gap left by
|
||||
the v0.9 re-architecture and fixes the release/install pipeline bug that
|
||||
caused `install.sh` to resolve to v0.4.5 instead of the latest release.
|
||||
The v0.9 re-architecture shipped a complete CLI surface (markdown
|
||||
jobspec, `orca ns`, `orca node capacity`, CLI-side scheduler, emitters,
|
||||
Traefik ingress) but no operator-facing reference documentation. This
|
||||
milestone ships that documentation plus a worked full-stack example
|
||||
with ingress configured, and hardens the release pipeline so every
|
||||
Gitea release carries a Linux binary asset.
|
||||
|
||||
## Root cause of the v0.4.5 install
|
||||
|
||||
The v0.8.x releases (v0.8.0 through v0.8.15) shipped with **zero binary
|
||||
assets attached** to their Gitea releases. `scripts/install.sh` resolves
|
||||
"latest" by hitting `/releases/latest` (returns v0.8.15), then looks for
|
||||
`orca-v0.8.15-linux-amd64.tar.gz` in that release's assets. Since the
|
||||
asset is missing, install.sh errors out — there is no fallback walk to
|
||||
older releases that DO carry a binary. The user's v0.4.5 install came
|
||||
from an earlier run or a pinned `--version`. The fix is forward: harden
|
||||
`scripts/release.sh` to cross-build the amd64 tarball and verify the
|
||||
asset attached post-create; harden `scripts/install.sh` to walk
|
||||
backward through releases if the latest lacks the asset.
|
||||
|
||||
## v0.10 Phases
|
||||
|
||||
- **Phase 0 (pre-execution)**: specify → clarify → research → ideate → plan → grill. Tag `v0.9.0`.
|
||||
- **Phase P1 — release/install fix** (REQ-097, REQ-098): cross-build amd64 tarball in release.sh, post-create asset verification, install.sh fallback walk. Tag `v0.9.1`.
|
||||
- **Phase P2 — CLI + jobspec + ingress docs** (REQ-091, REQ-092, REQ-093): `docs/cli.md`, `docs/jobspec.md`, `docs/ingress.md`. Tag `v0.9.2`.
|
||||
- **Phase P3 — full-stack examples** (REQ-094): `examples/full-stack/` with 5 valid jobspecs + rendered artifacts + walkthrough README. Tag `v0.9.3`.
|
||||
- **Phase P4 — README + namespace.md refresh** (REQ-095, REQ-096): README subcommand table + install example + docs/examples sections; `docs/namespace.md` v0.9 layout. Tag `v0.9.4`.
|
||||
- **Phase P5 — final review + ship + audit** (milestone release). Tag `v0.9.5` = v0.10.0 milestone release.
|
||||
|
||||
**Milestone type**: feature (P1 ships `fix` phases; P2/P3/P4 ship `docs`
|
||||
phases; at least one non-docs phase makes this a feature milestone per
|
||||
the versioning logic). Tags run on the v0.9.x patch line. The milestone
|
||||
branch label is `milestone/v0.10-docs-cli-examples`.
|
||||
|
||||
The vision ("minimalist, offline-first, CLI-first orchestration
|
||||
engine") is unchanged. v0.10 is a documentation + install-hardening
|
||||
milestone, not a direction change. It builds on the v0.9
|
||||
re-architecture foundation without modifying any Go orchestration code.
|
||||
|
||||
## v0.10 Clarified Decisions (D-series, full autonomy — Phase 0 pre-execution)
|
||||
|
||||
| ID | Question | Decision | Rationale | Confidence |
|
||||
|----|----------|----------|-----------|------------|
|
||||
| D-188 | Should the CLI docs be a single `docs/cli.md` reference or a per-command `docs/cli/` subdirectory? | **Single `docs/cli.md` reference** | Mirrors the existing flat `docs/` pattern (install.md, docker.md, namespace.md, security-scanning.md). One file is more discoverable for a CLI tool and avoids navigation overhead. A per-command subdirectory diverges from the established layout. | 0.92 |
|
||||
| D-189 | Should the examples live in `examples/full-stack/` or in `testdata/`? | **`examples/full-stack/` as a new top-level directory** | `testdata/` holds legacy HCL fixtures (`hello.hcl`, `fail.hcl`) used by Go tests; mixing operator-facing examples with test fixtures conflates audiences. A new `examples/` directory is the conventional location for worked examples and is what an operator expects to find. | 0.93 |
|
||||
| D-190 | How deep should the ingress/Traefik documentation go? | **Dedicated `docs/ingress.md` plus a worked example in `examples/full-stack/`** | Ingress is the user's explicit ask ("full stack with ingress configured") and the Traefik/service-block model (R-007 socket vs TCP, atomic reload, drain, TLS) is non-trivial. A dedicated doc is the clearest answer; a section buried in `docs/cli.md` would be less discoverable. | 0.90 |
|
||||
| D-191 | Should the docs frame the v0.9 canonical path or document both v0.8 and v0.9 equally? | **Document the v0.9 canonical path; flag deprecated surface with callout boxes** | The v0.8 daemon/mTLS/HCL path is deprecated and scheduled for removal in v0.10-P14. Documenting it as primary misleads new operators; documenting both equally doubles the surface and risks documenting soon-removed code. Callout boxes with "deprecated in v0.9, removed in v0.10" point operators to the canonical path. | 0.91 |
|
||||
| D-192 | Should the existing v0.8.15 release be backfilled with a binary asset, or only fix the pipeline forward? | **Fix forward only; no backfill** | Backfilling a past release is an ops task, not a docs milestone deliverable. The next tagged phase (this milestone's P1 ship at v0.9.1) will be the first correctly-asseted release; install.sh's new fallback walk handles the gap until then. | 0.88 |
|
||||
| D-193 | Should `release.sh` build only `linux-amd64` or also `linux-arm64`? | **Cross-build `linux-amd64` explicitly (host-arch-independent); arm64 deferred to a follow-up** | The install.sh user base is amd64 today (the `.coreci.yml` release step hardcodes `--asset orca-${VERSION}-linux-amd64.tar.gz`). Building amd64 regardless of host arch (via `GOOS=linux GOARCH=amd64 go build`) guarantees the asset the install script expects. arm64 support is a separate enhancement. | 0.85 |
|
||||
| D-194 | Should `install.sh` add a `--check` dry-run mode? | **Yes, lightweight** | A dry-run mode (`--check`) that prints the version + asset URL + install path without writing is cheap to add and useful for debugging the "which release will I get?" question that the v0.4.5 incident surfaced. | 0.80 |
|
||||
|
||||
## v0.11 Clarified Decisions (D-series, full autonomy — Phase 0 pre-execution)
|
||||
|
||||
The following 23 decisions (D-215…D-237) extend the locked D-series
|
||||
(ends at D-206). They derive from 5 research documents ingested
|
||||
2026-08-07 covering ingress hardening, drift detection, platform-engineer
|
||||
positioning, strategic framing, and the systemd Path unit implementation.
|
||||
Operator decisions Q1=A, Q2=C, Q3=A, Q4=A, Q5=A are adopted.
|
||||
|
||||
### Ingress hybrid (D-215…D-226, from research doc 1)
|
||||
|
||||
| ID | Question | Decision | Rationale | Confidence |
|
||||
|----|----------|----------|-----------|------------|
|
||||
| D-215 | Public-binding default? | **Hybrid: nft DNAT → Traefik on `127.0.0.1:8443`** | Defense-in-depth (kernel + app layer); mature pattern (kube-proxy, Linkerd2-proxy, F5/HAProxy+nginx). Smaller Traefik attack surface. R-017. | 0.93 |
|
||||
| D-216 | Opt-out? | **`orca cluster config --public-binding=traefik-on-public-ip` for the simple case** | Operators who want simplicity get it with a one-line config change. | 0.94 |
|
||||
| D-217 | nftables emitter? | **Yes; renders `/etc/nftables.d/orca.nft`; idempotent `nft -f` apply** | Same emitter pattern as Traefik/systemd emitters (R-001-clean). | 0.93 |
|
||||
| D-218 | nftables tool vs iptables? | **`nft` (modern) over legacy `iptables`** | Atomic rule-set swap; modern kernel API. | 0.96 |
|
||||
| D-219 | Cross-node cluster mesh? | **Stays bound on private IP `192.168.x.x:8443`; unchanged** | Avoids adding iptables rules for cross-node mesh; keeps mesh logic unchanged. | 0.94 |
|
||||
| D-220 | Traefik `address` in static config? | **`127.0.0.1:8443` in default, `:443` in opt-out** | Single line change; certs/mTLS/dynamic config unchanged. | 0.97 |
|
||||
| D-221 | `orca doctor nft`? | **Yes; checks table, expected rules, file hash; drift detection via hash comparison** | Parity with `orca doctor traefik`; integrates with R-018 critical_paths. | 0.95 |
|
||||
| D-222 | Rate-limit meter? | **`ora_rl` set as part of the default rule set; configurable via `orca nft rate limit set`** | Kernel-level line-rate rate limiting; defense against SYN floods. | 0.91 |
|
||||
| D-223 | GeoIP blocking? | **Operator-opt-in via `orca nft country block add`**; cli + ipset extension | Not a default; operators opt in. | 0.88 |
|
||||
| D-224 | `nftables` not `iptables` in `.coreci.yml` pipelines? | **Yes; integration tests use `nft` exclusively** | Matches D-218. | 0.94 |
|
||||
| D-225 | Per-workload `ingress: native` coexists with hybrid default? | **Yes; `service { ingress: native }` opts into pure iptables + stunnel sidecars** | Workload-level opt-in; doesn't affect cluster default. | 0.93 |
|
||||
| D-226 | `nftables` rule hash baseline? | **`cluster/state/baseline.nft.hash` per peer; drift detection per §17** | Integrates with R-018 drift detection. | 0.90 |
|
||||
|
||||
### Drift detection (D-227…D-237, from research doc 5)
|
||||
|
||||
| ID | Question | Decision | Rationale | Confidence |
|
||||
|----|----------|----------|-----------|------------|
|
||||
| D-227 | Drift detection architecture? | **systemd Path units for critical paths + 60s polling backstop + auto-remediation** | R-001-clean (systemd is OS, not Orca); ~10s event-driven latency on critical paths. R-018/R-019. | 0.94 |
|
||||
| D-228 | Path unit event payload? | **Oneshot service; receives path via `%f`; computes sha256; writes event JSON to `/etc/orca/state/drift-events/`** | Stateless, self-contained, idempotent. | 0.93 |
|
||||
| D-229 | Lead-side pickup? | **Aggregator timer reads each peer's drift-events/, validates against applied txn hashes, triggers remediation** | Reuses existing 10s aggregator cadence (C-11); single SSH pull per tick. | 0.94 |
|
||||
| D-230 | Critical path polling cadence? | **5s backstop; systemd Path unit provides ~10s event-driven latency** | Closes the gap to K8s-comparable drift detection on critical paths. | 0.92 |
|
||||
| D-231 | Auto-remediation policy? | **Per-path config; critical paths default to auto; systemd units default to require-approval** | Config files are safe to re-push; service units may need careful ordering (don't restart serving workloads). | 0.93 |
|
||||
| D-232 | Remediation rate limit? | **5-minute cooldown per path; applies only on SUCCESSFUL remediation; transient failures retry on next aggregator tick** | Prevents loops from buggy external actors; avoids a 30s network blip blocking re-remediation for 5 min (refined per CLARIFY C4). | 0.92 |
|
||||
| D-233 | NFS path detection? | **`orca node setup` detects NFS mounts; falls back to polling for affected paths** | systemd Path units use inotify which doesn't work across NFS. | 0.90 |
|
||||
| D-234 | Secrets path exclusion? | **`/etc/orca/credentials/*` excluded from drift detection** | Re-remediating secrets might clobber intentional out-of-band rotation. | 0.95 |
|
||||
| D-235 | EnvironmentFile drift? | **Triggers `orca job restart <name>` instead of file-level remediation** | Workload already running won't pick up env changes without a restart. | 0.89 |
|
||||
| D-236 | `orca drift watch` semantics? | **`iter.Seq2[Event, error]` per D-017; `signal.NotifyContext` per D-023; default 2s poll** | Consistent with existing `--watch` pattern (D-017/D-023). | 0.95 |
|
||||
| D-237 | Aggregator timer changes? | **Existing 10s cadence; extended to also pull drift-events/ and remediate** | Reuses C-11 aggregator; no new timer. | 0.95 |
|
||||
|
||||
### P01.5 — SPIFFE SVID minting spike (gate C-08, D-068)
|
||||
|
||||
**C-08 SPIFFE mint spike: PASS.** The `step` CLI (smallstep step-ca)
|
||||
accepts a `spiffe://` URI in `--san` and emits a cert whose URI SAN
|
||||
(x509 subjectAltName URI entry) carries the SPIFFE URI. The fallback to
|
||||
mTLS identity (per D-068 / C-08) is NOT needed; D-068 stands.
|
||||
|
||||
- **SPIFFE URI format (locked):**
|
||||
`spiffe://orca.local/ns/<namespace>/sa/<service-account>/<alloc-id>`
|
||||
— trust domain `orca.local`; `ns/<ns>` scopes the workload to an
|
||||
Orca namespace (R-002); `sa/<sa>` is the service-account; `<alloc-id>`
|
||||
makes the SVID unique per allocation.
|
||||
- **step CLI command (locked):**
|
||||
`step ca certificate <spiffe-id> <cert> <key> --san <spiffe-id> --not-after 24h --provisioner orca-admin --password-file /dev/stdin --force`
|
||||
- **Cert parsing (locked):** `pem.Decode` → `x509.ParseCertificate` →
|
||||
iterate `cert.URIs` and match the expected SPIFFE URI (parsed as
|
||||
`*url.URL`, compared by canonical string). Missing URI SAN →
|
||||
`ErrSpiffeURIMissing` (cert rejected before reaching the workload).
|
||||
- **Implementation:** `internal/identity/spiffe.go` — `SpiffeURI`,
|
||||
`MintSVID`, `VerifySVID`, `SpiffeIDFromCert`, `SubjectFromSpiffe`.
|
||||
- **Tests:** `internal/identity/spiffe_test.go` — mock transport
|
||||
(`execer`) returns a self-signed cert minted in-process via
|
||||
`crypto/x509.CreateCertificate` with `URIs: []*url.URL{spiffeURI}`,
|
||||
exercising the exact production parsing path. 15 tests, all pass.
|
||||
- **Spike result record:** `internal/identity/SPIFFE_SPIKE_RESULT.md`.
|
||||
|
||||
## v0.12 Scope Summary — Security Hardening (Zero-Trust Identity)
|
||||
|
||||
v0.12 is a 27-execution-phase feature milestone dedicated to
|
||||
comprehensive security hardening across the entire attack surface,
|
||||
**including the operating system itself**. The threat-model review
|
||||
(v0.11 closeout + Phase 0 RESEARCH) surfaced 25 distinct findings
|
||||
(F1..F25) spanning injection, traversal, ACL, audit, crypto, OS
|
||||
scripts, emitters, sudoers, system users, file modes, daemon auth,
|
||||
backup, SQLite, install.sh, and migration. v0.12 closes all of them
|
||||
and adopts a **zero-trust identity model** as the load-bearing
|
||||
architectural change.
|
||||
|
||||
### Load-bearing rule adopted in Phase 0
|
||||
|
||||
**R-021**: *Orca never issues, stores, or accepts human-identity
|
||||
credentials. Human identity is exclusively external (OIDC). Machine
|
||||
identity is exclusively mTLS/SPIFFE. No passwords, no Orca-issued
|
||||
tokens, no CA-key passphrases.*
|
||||
|
||||
### Zero-trust identity model
|
||||
|
||||
Two identity layers, zero overlap:
|
||||
|
||||
- **Human operators** → OIDC (external IdP, BYO) OR the **bundled Dex**
|
||||
with a **WebAuthn (passkeys) connector** as the default
|
||||
password-free authenticator. `orca auth login` / `orca auth register`
|
||||
open the default browser to the Dex WebAuthn endpoint via OIDC
|
||||
authorization-code + PKCE + local loopback redirect. After the
|
||||
WebAuthn ceremony (biometric/security key), Dex redirects back with
|
||||
an auth code; CLI exchanges for a short-lived ID token (1h) +
|
||||
refresh. Headless/CI fallback: device-code flow.
|
||||
- **Machine-to-machine** → mTLS + SPIFFE SVIDs (unchanged from v0.11).
|
||||
|
||||
The "no Orca credentials" invariant holds: passkeys are public-key
|
||||
credentials (the private key never leaves the authenticator); the
|
||||
WebAuthn credential DB stores only public keys + credential IDs +
|
||||
sign counts. No passwords, no Orca-issued tokens, no CA-key
|
||||
passphrases anywhere in the system.
|
||||
|
||||
### Master key sealing
|
||||
|
||||
The secrets master key (32 random bytes) is **sealed to OIDC** —
|
||||
wrapped by a key derived from an OIDC token exchange at unseal time.
|
||||
`orca cluster unseal` (operator authenticates via OIDC → token
|
||||
exchange → unwrap master key into memory → zeroed on shutdown). The
|
||||
raw master key never touches disk. **Shamir 3-of-5 recovery**: at seal
|
||||
time, 5 shards are printed and the operator stores them offline. If
|
||||
the IdP is permanently lost AND a quorum of shards is unavailable, the
|
||||
cluster is unrecoverable by design (documented residual risk; no
|
||||
backdoor).
|
||||
|
||||
### New requirements (REQ-119..REQ-148)
|
||||
|
||||
30 net-new requirements derived from the threat-model findings and the
|
||||
zero-trust identity model. See REQUIREMENTS.md and ROADMAP.md for the
|
||||
full mapping. Highlights:
|
||||
|
||||
- REQ-119..121: command injection, path traversal, txn path allowlist
|
||||
- REQ-144: OIDC client + bundled Dex (BYO-IdP override)
|
||||
- REQ-145: ACL rewrite (remove KindToken, add KindOidc, enforce)
|
||||
- REQ-146: remove all password/token paths (breaking)
|
||||
- REQ-147: master key seal-to-OIDC + Shamir recovery
|
||||
- REQ-148: WebAuthn connector for Dex (passkeys, browser auth+register)
|
||||
- REQ-122..143: integrity, crypto, OS scripts, emitters, sudoers,
|
||||
system users, SQLite, migration, dual-write closure, transport,
|
||||
drift auth, integration tests, docs, final review
|
||||
|
||||
### v0.12 Clarified Decisions (D-series, full autonomy)
|
||||
|
||||
The 10 v0.12 decisions (D-238..D-247) were resolved during CLARIFY
|
||||
under full autonomy (autonomy.level=full, workflow.no_hitl=true):
|
||||
|
||||
| ID | Question | Decision | Rationale | Confidence |
|
||||
|----|----------|----------|-----------|------------|
|
||||
| D-238 | Milestone version? | **v0.12 (minor, not v1.0)** | v1.0.0 stays deferred for post-UAT per v0.11 PRD; v0.12 is a minor feature milestone. Tags on v0.11.x patch line. | 0.95 |
|
||||
| D-239 | OIDC provider model? | **Bundled Dex by default + BYO external IdP override** | Zero-trust out of the box without external setup; `oidc.issuer` repoint switches to BYO. | 0.90 |
|
||||
| D-240 | Bundled Dex upstream authenticator (password-free)? | **WebAuthn (passkeys) connector** | Public-key credentials; private key never leaves authenticator; reinforces "no passwords" invariant (R-021). | 0.88 |
|
||||
| D-241 | Master key sealing model? | **Seal to OIDC + Shamir 3-of-5 recovery** | No password anywhere; quorum recovery if IdP lost; no backdoor. | 0.85 |
|
||||
| D-242 | CLI browser flow? | **OIDC auth-code + PKCE + local loopback redirect** | Standard OIDC browser flow; secure for public clients; headless fallback via device-code. | 0.92 |
|
||||
| D-243 | WebAuthn RP ID / secure context? | **Traefik-served cluster domain (step-ca cert, R-017)** | WebAuthn requires HTTPS; Traefik already provides it; RP ID configurable via `orca auth init-idp`. | 0.90 |
|
||||
| D-244 | Passkey storage? | **SQLite at ClusterDir()/webauthn-credentials.db (0600); public keys only** | Public keys are not secrets; 0600 file mode for integrity; no passphrase wrapping needed. | 0.92 |
|
||||
| D-245 | Headless/CI auth fallback? | **Device-code flow** | No browser in CI; device-code is the standard OIDC headless path. | 0.90 |
|
||||
| D-246 | Token storage at rest? | **~/.orca/credentials.json (0600); short-lived (1h) + refresh** | Standard OIDC token storage; 0600; refresh handles rotation; no long-lived Orca-issued tokens. | 0.92 |
|
||||
| D-247 | Breaking-change handling for password/token removal? | **`orca upgrade` refuses v0.11 clusters using --password/bare-tokens without --accept-identity-migration** | No silent breakage; explicit migration gate; documented cutover. | 0.90 |
|
||||
|
||||
### v0.12 is a HARDENING + IDENTITY milestone, not a direction change
|
||||
|
||||
The vision ("minimalist, offline-first, CLI-first orchestration
|
||||
engine inspired by HashiCorp Nomad") is unchanged. v0.12 closes the
|
||||
security-surface gaps surfaced by the v0.11 threat model and adopts a
|
||||
zero-trust identity model. The offline-first principle (R-003) is
|
||||
preserved: the bundled Dex can run on the lead (offline), and the
|
||||
mTLS-only path remains for the single-operator fully-offline case (no
|
||||
human authn needed — the operator holds the pre-staged SSH key + mTLS
|
||||
cert; no password, no token).
|
||||
|
||||
+156
-32
@@ -141,9 +141,9 @@ REQ-047..052 all complete.
|
||||
| REQ-059 | `orca node key-reset <node>` command: clears the persisted SSH host key entry for the node from `~/.orca/known_hosts` only (local, not remote authorized_keys — D-046); audit-logs `event=node.key_reset`; next `doctor proxmox`/dispatch re-pins via TOFU or `--host-key-fingerprint` | Low | **v0.8 P2** | **Complete** (P2 shipped v0.7.2) |
|
||||
| REQ-060 | Requirement-status hygiene sweep: REQUIREMENTS.md v0.7 rows were stale ("Pending" after ship); add a verify-stage assertion that every REQ listed as `Complete` in ROADMAP.md has a matching `Complete` row in REQUIREMENTS.md, enforced by `make verify-reqs` | Medium | **v0.8 P3** | **Complete** (P3 shipped v0.7.3) |
|
||||
|
||||
## v0.9/v1.0 Requirements — Re-architecture Foundation & Production Hardening
|
||||
## v0.9/v0.10 Requirements — Re-architecture Foundation & Production Hardening
|
||||
|
||||
The v0.9/v1.0 milestones supersede the shipped v0.1–v0.8 architecture per the
|
||||
The v0.9/v0.10 milestones supersede the shipped v0.1–v0.8 architecture per the
|
||||
adopted PRD (`.ciagent/PRD_v0.9.md`). The re-architecture is justified on six
|
||||
grounds recorded in the PROJECT.md Supersession Table. 30 net-new requirements
|
||||
(REQ-061..REQ-090) derive from the v0.9 IDEATION; their phase placement and
|
||||
@@ -152,33 +152,157 @@ and `GRILL_v0.9.md`.
|
||||
|
||||
| ID | Requirement | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-061 | `orca daemon` deprecation command and build-tag removal path: v0.9 emits deprecation warning + still runs (dual-write window); v1.0 repurposes to `orca daemon drain-and-stop` (stops v0.8 daemons on peers via SSH, confirms workloads survive via systemd); post-v1.0 the command and `internal/daemon/` are deleted. `// Deprecated` Go doc comments + `slog.Warn` on every run (I-M-001) | High | **v1.0 P14** (warn v0.9 P0X) | Pending |
|
||||
| REQ-062 | Coverage follow-ups: 3 zero-test packages (`internal/audit`, `internal/certpaths`, `cmd/orca`) + `internal/cli` to 70% floor; once `daemon.go` is deprecated/removed the exclusion reason disappears and the floor applies to the whole package; all net-new subsystems carry a 70% floor from their first phase (I-M-002) | Medium | **v0.9 P0X** + each new pkg | Pending |
|
||||
| REQ-063 | `known_hosts` flock concurrency gap (deferred P1 from REVIEW_v0.8 A2): add `flock`-style advisory lock (stdlib `syscall.Flock` wrapper) around the read-modify-write in `TOFUHostKeyCallback` capture path (`bootstrap.go:290-302`) and `ResetHostKey` (`bootstrap.go:479-523`); lock file at `cluster/known_hosts.lock` (R-002) (I-M-003) | Medium | **v0.9 P0a1** | Pending |
|
||||
| REQ-064 | HCL→Markdown jobspec adapter/bridge layer: keep `internal/jobspec/spec.go` as legacy HCL path behind `// Deprecated`; add `internal/jobspec/markdown.go` (canonical) + `internal/jobspec/dispatch.go` (extension-based dispatcher: `.md`→Markdown, `.hcl`→legacy, `.yaml`→Markdown-with-empty-body); unified `*WorkloadSpec` populated via adapter; preserves `orca job run old-spec.hcl` during migration window (I-M-004) | High | **v0.9 P0b** | Pending |
|
||||
| REQ-065 | `orca doctor --legacy-paths` detection: detects v0.8 residue (orca.db at ORCA_HOME root, ca.crt/ca.key, config.hcl, flat server.crt, namespace column in any *.db); outputs list of legacy artifacts with migration recommendations; the detection half of v1.0-P14 (I-M-005) | Medium | **v1.0 P14c** | Pending |
|
||||
| REQ-066 | Legacy CA state migration to step-ca: `orca upgrade --to-v1.0 --import-ca` reads `~/.orca/ca.key`, initializes step-ca with it, re-issues workload SVIDs; preserves audit history even if live trust root changes (I-M-006). **Gated by C-07** | High | **v1.0 P14a** | Pending |
|
||||
| REQ-067 | Fuzz test harness for Markdown frontmatter parser: `testing.F` fuzz target in `internal/jobspec/markdown_test.go` round-trips random frontmatter+body through `ParseMarkdown` asserting byte-exact body preservation; corpus of adversarial fixtures (CRLF, BOM, no-frontmatter, empty-frontmatter, frontmatter-with-only-separator) (I-M-007) | Medium | **v0.9 P0b** | Pending |
|
||||
| REQ-068 | Deprecation warnings on removed/repurposed CLI subcommands: each removed/changed command (`orca cert`, `orca node join` mTLS semantics, `orca job run <spec.hcl>`) emits `slog.Warn` deprecation banner with v1.0 replacement except under `orca upgrade`; `--no-deprecation-warnings` global flag via `root.go` `PersistentPreRunE` (I-M-008) | Low | **v0.9 P0X** + v1.0 P13 | Pending |
|
||||
| REQ-069 | `internal/config/config.go` HCL config demotion via adapter: keep `internal/config/` as `legacy_config.go` with `// Deprecated`; add `internal/config/markdown.go` for new Markdown-frontmatter loader (R-014); `root.go` dispatches on file extension (`.hcl`→legacy, `.md`→new); `--config` semantics: `.hcl` read-only legacy, `.md` canonical (I-M-009) | High | **v0.9 P0a1** | Pending |
|
||||
| REQ-070 | `internal/certpaths/` replacement with multi-namespace path resolver: new `internal/paths` package with `paths.NamespaceDir(ns)`, `paths.ClusterDir()`, `paths.CacheDB()`, `paths.MasterKey()`, `paths.NSDb(ns)`, `paths.NSEnv(ns)`, `paths.NSSecrets(ns)`; keep `certpaths` as thin shim for v0.8 compat then remove post-v1.0 (R-002) (I-M-010) — highest blast radius | High | **v0.9 P0a1** | Pending |
|
||||
| REQ-071 | `internal/store/` schema: per-namespace DBs, drop namespace column: `store.Open` gains namespace parameter (or caller passes `paths.NSDb(ns)`); `migrate.go` runs migrations per namespace DB; `cert_repo` (0004) removed (step-ca handles certs); audit_log moves to CLI-side cache DB (R-008) (I-M-011) | High | **v0.9 P0a1** + v1.0 P06 | Pending |
|
||||
| REQ-072 | `internal/transport/` deletion + SSH-push package: delete `mtls.go`, `dispatch.go`, `handshake_log.go`; extract retry/idempotency patterns into `internal/sshpush/`; existing `transport.IdempotencyStore` directly reusable (I-M-012). Deletion deferred to v1.0-P14 to keep dual-write window open | High | **v0.9 P00** (delete v1.0 P14) | Pending |
|
||||
| REQ-073 | SSH-push transport layer design: connection pooling (reuse `*ssh.Client` per peer), idempotency (content-addressed filenames), retry (exponential backoff 100ms×2 cap 5s max 5), timeout (30s SCP, 10s exec), fan-out (errgroup bounded concurrency default 8), known_hosts reuse `proxmox.TOFUHostKeyCallback` (I-B-001) | High | **v0.9 P01** (design P0a1) | Pending |
|
||||
| REQ-074 | Emitter template system (Layer 4): `internal/emitter/` package with `Emitter` interface `Render(spec *WorkloadSpec, node *Node) ([]File, error)`; implementations systemdEmitter/traefikEmitter/syncthingEmitter/socketEmitter; SSH-push SCPs `[]File` atomically (write-to-tmp + rename); emitters registered per kind + runtime (I-B-002) | High | **v0.9 P0c** | Pending |
|
||||
| REQ-075 | Lead applier execution model: CLI renders transaction bundle (tarball + apply.sh + verify.sh) on operator host, SCPs to lead's `/run/orca/txns/<txn-id>/`, lead's systemd timer runs `apply.sh` idempotently, CLI polls txn status via SSH; bash scripts generated by emitter not hand-written (I-B-003). **Gated by C-09** | High | **v1.0 P10** (design v0.9 P00) | Pending |
|
||||
| REQ-076 | step-ca integration: `orca init` runs `step ca init` on lead; CLI SSHs to lead, installs step-ca via apt, stores step-ca.json; workload SVIDs via `step ca token` (JWE minted by CLI) → `step ca certificate`; SPIFFE ID as SAN; new `internal/stepca/` package wraps `step` CLI via SSH (I-B-004). Reverses AD-010 per override justification ground 2 | High | **v0.9 P07** + v1.0 P02 | Pending |
|
||||
| REQ-077 | Traefik dynamic config generation + atomic reload: Traefik emitter renders `/etc/traefik/dynamic/orca-<ns>-<svc>.yaml` with backends (socket paths R-007), health checks, mTLS config pointing at step-ca root; atomic reload via tmpfile+fsync+rename triggering fsnotify; drain writes `weight=0` or removes backend (I-B-005). **Gated by C-10** | High | **v0.9 P02** | Pending |
|
||||
| REQ-078 | Runtime abstraction interface (5 backends): `Runtime` interface in `internal/runtime/` with Prepare/Start/Stop/Status; processRuntime (wraps existing executor.go), wasmRuntime (wasmtime via SSH), podmanRuntime, pveVMRuntime (qm via proxmox SSH), pveCTRuntime (pct); runtimeRegistry keyed by `runtime:` frontmatter value; Alloc carries runtime field changeable on migration (I-B-006). Split P07a/b/c per PC-10. **P07b gated by C-01** | High | **v0.9 P07a/b/c** | Pending |
|
||||
| REQ-079 | Transaction bundle format + N-peer atomicity: bundle = tarball with desired-state.json + apply.sh + verify.sh + rollback.sh + manifest.sig (signed with master.key); content-addressed `<txn-id>=sha256(desired-state.json)` stored in `cluster/txns/<txn-id>/`; lead applies to self first then fans out; failure on any peer runs rollback.sh on applied peers (I-B-007). **Gated by C-09** | High | **v1.0 P10** (design v0.9 P00) | Pending |
|
||||
| REQ-080 | Master key management + HKDF-SHA256 per-line .env.secrets encryption: `cluster/master.key` 32-byte random (generated at `orca init` using WriteAtomic pattern); each line `base64(nonce||ciphertext||tag)`, nonce=random(12 bytes), AES-256-GCM with AAD=line-number (prevents line-swap); HKDF-SHA256 derives per-namespace sub-keys; `orca secrets set/get`; v0.8 `internal/security/redact.go` reusable (I-B-008). **Gated by C-19** | High | **v1.0 P03** | Pending |
|
||||
| REQ-081 | Syncthing config rendering + folder-ID content-addressing: per-namespace Syncthing folder `orca-<ns>` with content-addressed folder ID `sha256(ns + master-key-fingerprint)`; CLI renders config.xml per peer; Syncthing runs as systemd unit (emitted by systemd emitter); CLI discovers peers via `cluster/peers/`; migration works because new node joins folder and syncs before workload starts (I-B-009). **Gated by C-02 + C-14** | Medium | **v0.9 P09** (spike v0.9 P00) | Pending |
|
||||
| REQ-082 | Namespace inheritance resolver algorithm: DFS parent walker with visited set for cycle detection; `_defaults/` implicit root (always exists, no parent); merge semantics: child overrides parent for scalars, arrays unioned (child adds to parent); pure function (no I/O) taking `map[nsName→*NSConfig]` returning `map[nsName→*ResolvedNS]` (I-B-010) | High | **v0.9 P0a2** | Pending |
|
||||
| REQ-083 | CLI-side scheduler redesign: `Score(node, workload) (score int, fits bool)` where `fits` checks runtime compatibility + constraints, `score` is bin-packing (most free capacity = highest); Services pick `count` distinct nodes (anti-affinity default); DaemonSets pick all matching nodes; Job = one-shot; CLI-side not daemon-side (R-001) (I-B-011) | High | **v0.9 P05** (skeleton P0c) | Pending |
|
||||
| REQ-084 | `orca job lint` category-driven lint engine: `Linter` runs `Rule` checks returning `Finding{Category, Severity, Message, Explanation}`; categories schema/runtime/security/migration/best-practice; `--explain` prints rationale; pure (no I/O) checks against static rules (I-B-012) | Medium | **v1.0 P11** | Pending |
|
||||
| REQ-085 | v0.8→v1.0 migration ordering: v0.9 ships new parser + kinds + runtime + SSH-push alongside old daemon (dual-write window); `orca job run` dispatches on extension (`.md`→SSH-push, `.hcl`→old daemon); v1.0-P05 drains old daemons; v1.0-P14 converts remaining `.hcl` specs and removes daemon (I-C-001). **Most important cross-cutting idea** | High | **v0.9 P00** → v1.0 P14 | Pending |
|
||||
| REQ-086 | "No orca on server" enforcement: `orca doctor no-orca-on-server` SSHs to each peer verifying no `orca` binary in PATH, no `orca` systemd service, no `orca` process, no `/etc/orca/` directory; runs after v1.0-P05 before v1.0-P16; reuses v0.8 `proxmox` SSH session infrastructure (I-C-002). Implements grill C-13 | High | **v1.0 P14c** | Pending |
|
||||
| REQ-087 | Test infrastructure: hermetic 3-linux + 1-proxmox cluster pipeline: `test/integration/` with docker-compose/vagrant creating 4 containers/VMs; Go test harness SSHes to each, runs CLI, asserts end-to-end workflows (ns create → workload submit → migrate → drain); proxmox simulated via mock pct/qm; v0.8 e2e tests (bootstrapE2ESetup) are foundation (I-C-003) | Medium | **v1.0 P08** (bootstrap v0.9 P00) | Pending |
|
||||
| REQ-088 | Security-engineer + network-engineer persona reactivation: reactivate security-engineer (step-ca provisioner model, SSH-push blast radius, Traefik edge, .env.secrets crypto) and network-engineer (socket exposure R-007, Syncthing P2P ports, Traefik routing); cross-cutting review not single phase (I-C-004). Implements grill C-05 | High | **v0.9 P00** → v1.0 P16 | Pending |
|
||||
| REQ-089 | Documentation rewrite: ARCHITECTURE.md/PROJECT.md/README + AD-010 supersession: v0.9-P00 adds "v0.9 Architecture (Supersedes v0.8)" section + banners + Superseded Decisions table; v1.0-P15 rewrites README quickstart for new curl|sh + orca init + orca ns create flow (I-C-005) | Medium | **v0.9 P00** + v1.0 P15/P16 | Pending |
|
||||
| REQ-090 | Dual-write window: v0.9 `orca job run` dispatches on extension (`.md`→SSH-push new path, `.hcl`→old daemon path) via parser dispatcher (REQ-064); daemon not removed until v1.0-P05; SSH-push path writes to separate systemd unit namespace (`orca-v1-<alloc>.service`) while daemon uses `orca-<job>.service` — no unit name overlap = no conflict (I-C-006) | High | **v0.9 P00** | Pending |
|
||||
| REQ-061 | `orca daemon` deprecation command and build-tag removal path: v0.9 emits deprecation warning + still runs (dual-write window); v1.0 repurposes to `orca daemon drain-and-stop` (stops v0.8 daemons on peers via SSH, confirms workloads survive via systemd); post-v1.0 the command and `internal/daemon/` are deleted. `// Deprecated` Go doc comments + `slog.Warn` on every run (I-M-001) | High | **v0.11 P14b** (drain-and-stop + rotate-lead) | **Complete** |
|
||||
| REQ-062 | Coverage follow-ups: 3 zero-test packages (`internal/audit`, `internal/certpaths`, `cmd/orca`) + `internal/cli` to 70% floor; once `daemon.go` is deprecated/removed the exclusion reason disappears and the floor applies to the whole package; all net-new subsystems carry a 70% floor from their first phase (I-M-002) | Medium | **v0.9 P0X** + each new pkg | Complete |
|
||||
| REQ-063 | `known_hosts` flock concurrency gap (deferred P1 from REVIEW_v0.8 A2): add `flock`-style advisory lock (stdlib `syscall.Flock` wrapper) around the read-modify-write in `TOFUHostKeyCallback` capture path (`bootstrap.go:290-302`) and `ResetHostKey` (`bootstrap.go:479-523`); lock file at `cluster/known_hosts.lock` (R-002) (I-M-003) | Medium | **v0.9 P0a1** | Complete |
|
||||
| REQ-064 | HCL→Markdown jobspec adapter/bridge layer: keep `internal/jobspec/spec.go` as legacy HCL path behind `// Deprecated`; add `internal/jobspec/markdown.go` (canonical) + `internal/jobspec/dispatch.go` (extension-based dispatcher: `.md`→Markdown, `.hcl`→legacy, `.yaml`→Markdown-with-empty-body); unified `*WorkloadSpec` populated via adapter; preserves `orca job run old-spec.hcl` during migration window (I-M-004) | High | **v0.9 P0b** | Complete |
|
||||
| REQ-065 | `orca doctor --legacy-paths` detection: detects v0.8 residue (orca.db at ORCA_HOME root, ca.crt/ca.key, config.hcl, flat server.crt, namespace column in any *.db); outputs list of legacy artifacts with migration recommendations; the detection half of v0.10-P14 (I-M-005) | Medium | **v0.11 P14c** | **Complete** |
|
||||
| REQ-066 | Legacy CA state migration to step-ca: `orca upgrade --to-v1.0 --import-ca` reads `~/.orca/ca.key`, initializes step-ca with it, re-issues workload SVIDs; preserves audit history even if live trust root changes (I-M-006). **Gated by C-07** | High | **v0.11 P14a** | **Complete** |
|
||||
| REQ-067 | Fuzz test harness for Markdown frontmatter parser: `testing.F` fuzz target in `internal/jobspec/markdown_test.go` round-trips random frontmatter+body through `ParseMarkdown` asserting byte-exact body preservation; corpus of adversarial fixtures (CRLF, BOM, no-frontmatter, empty-frontmatter, frontmatter-with-only-separator) (I-M-007) | Medium | **v0.9 P0b** | Complete |
|
||||
| REQ-068 | Deprecation warnings on removed/repurposed CLI subcommands: each removed/changed command (`orca cert`, `orca node join` mTLS semantics, `orca job run <spec.hcl>`) emits `slog.Warn` deprecation banner with v1.0 replacement except under `orca upgrade`; `--no-deprecation-warnings` global flag via `root.go` `PersistentPreRunE` (I-M-008) | Low | **v0.9 P0X** + v0.10 P13 | Complete |
|
||||
| REQ-069 | `internal/config/config.go` HCL config demotion via adapter: keep `internal/config/` as `legacy_config.go` with `// Deprecated`; add `internal/config/markdown.go` for new Markdown-frontmatter loader (R-014); `root.go` dispatches on file extension (`.hcl`→legacy, `.md`→new); `--config` semantics: `.hcl` read-only legacy, `.md` canonical (I-M-009) | High | **v0.9 P0a1** | Complete |
|
||||
| REQ-070 | `internal/certpaths/` replacement with multi-namespace path resolver: new `internal/paths` package with `paths.NamespaceDir(ns)`, `paths.ClusterDir()`, `paths.CacheDB()`, `paths.MasterKey()`, `paths.NSDb(ns)`, `paths.NSEnv(ns)`, `paths.NSSecrets(ns)`; keep `certpaths` as thin shim for v0.8 compat then remove post-v1.0 (R-002) (I-M-010) — highest blast radius | High | **v0.9 P0a1** | Complete |
|
||||
| REQ-071 | `internal/store/` schema: per-namespace DBs, drop namespace column: `store.Open` gains namespace parameter (or caller passes `paths.NSDb(ns)`); `migrate.go` runs migrations per namespace DB; `cert_repo` (0004) removed (step-ca handles certs); audit_log moves to CLI-side cache DB (R-008) (I-M-011) | High | **v0.9 P0a1** + v0.10 P06 | Complete |
|
||||
| REQ-072 | `internal/transport/` deletion + SSH-push package: delete `mtls.go`, `dispatch.go`, `handshake_log.go`; extract retry/idempotency patterns into `internal/sshpush/`; existing `transport.IdempotencyStore` directly reusable (I-M-012). Deletion deferred to v0.10-P14 to keep dual-write window open | High | **v0.9 P00** (delete v0.10 P14) | Complete |
|
||||
| REQ-073 | SSH-push transport layer design: connection pooling (reuse `*ssh.Client` per peer), idempotency (content-addressed filenames), retry (exponential backoff 100ms×2 cap 5s max 5), timeout (30s SCP, 10s exec), fan-out (errgroup bounded concurrency default 8), known_hosts reuse `proxmox.TOFUHostKeyCallback` (I-B-001) | High | **v0.9 P01** (design P0a1) | Complete |
|
||||
| REQ-074 | Emitter template system (Layer 4): `internal/emitter/` package with `Emitter` interface `Render(spec *WorkloadSpec, node *Node) ([]File, error)`; implementations systemdEmitter/traefikEmitter/syncthingEmitter/socketEmitter; SSH-push SCPs `[]File` atomically (write-to-tmp + rename); emitters registered per kind + runtime (I-B-002) | High | **v0.9 P0c** | Complete |
|
||||
| REQ-075 | Lead applier execution model: CLI renders transaction bundle (tarball + apply.sh + verify.sh) on operator host, SCPs to lead's `/run/orca/txns/<txn-id>/`, lead's systemd timer runs `apply.sh` idempotently, CLI polls txn status via SSH; bash scripts generated by emitter not hand-written (I-B-003). **Gated by C-09** | High | **v0.11 P10a** | **Complete** |
|
||||
| REQ-076 | step-ca integration: `orca init` runs `step ca init` on lead; CLI SSHs to lead, installs step-ca via apt, stores step-ca.json; workload SVIDs via `step ca token` (JWE minted by CLI) → `step ca certificate`; SPIFFE ID as SAN; new `internal/stepca/` package wraps `step` CLI via SSH (I-B-004). Reverses AD-010 per override justification ground 2 | High | **v0.9 P07** + v0.10 P02 | Complete |
|
||||
| REQ-077 | Traefik dynamic config generation + atomic reload: Traefik emitter renders `/etc/traefik/dynamic/orca-<ns>-<svc>.yaml` with backends (socket paths R-007), health checks, mTLS config pointing at step-ca root; atomic reload via tmpfile+fsync+rename triggering fsnotify; drain writes `weight=0` or removes backend (I-B-005). **Gated by C-10** | High | **v0.9 P02** | Complete |
|
||||
| REQ-078 | Runtime abstraction interface (5 backends): `Runtime` interface in `internal/runtime/` with Prepare/Start/Stop/Status; processRuntime (wraps existing executor.go), wasmRuntime (wasmtime via SSH), podmanRuntime, pveVMRuntime (qm via proxmox SSH), pveCTRuntime (pct); runtimeRegistry keyed by `runtime:` frontmatter value; Alloc carries runtime field changeable on migration (I-B-006). Split P07a/b/c per PC-10. **P07b gated by C-01** | High | **v0.9 P07a/b/c** | Complete |
|
||||
| REQ-079 | Transaction bundle format + N-peer atomicity: bundle = tarball with desired-state.json + apply.sh + verify.sh + rollback.sh + manifest.sig (signed with master.key); content-addressed `<txn-id>=sha256(desired-state.json)` stored in `cluster/txns/<txn-id>/`; lead applies to self first then fans out; failure on any peer runs rollback.sh on applied peers (I-B-007). **Gated by C-09** | High | **v0.11 P10a** | **Complete** |
|
||||
| REQ-080 | Master key management + HKDF-SHA256 per-line .env.secrets encryption: `cluster/master.key` 32-byte random (generated at `orca init` using WriteAtomic pattern); each line `base64(nonce||ciphertext||tag)`, nonce=random(12 bytes), AES-256-GCM with AAD=line-number (prevents line-swap); HKDF-SHA256 derives per-namespace sub-keys; `orca secrets set/get`; v0.8 `internal/security/redact.go` reusable (I-B-008). **Gated by C-19** | High | **v0.11 P03** | **Complete** |
|
||||
| REQ-081 | Syncthing config rendering + folder-ID content-addressing: per-namespace Syncthing folder `orca-<ns>` with content-addressed folder ID `sha256(ns + master-key-fingerprint)`; CLI renders config.xml per peer; Syncthing runs as systemd unit (emitted by systemd emitter); CLI discovers peers via `cluster/peers/`; migration works because new node joins folder and syncs before workload starts (I-B-009). **Gated by C-02 + C-14** | Medium | **v0.9 P09** (spike v0.9 P00) | Complete |
|
||||
| REQ-082 | Namespace inheritance resolver algorithm: DFS parent walker with visited set for cycle detection; `_defaults/` implicit root (always exists, no parent); merge semantics: child overrides parent for scalars, arrays unioned (child adds to parent); pure function (no I/O) taking `map[nsName→*NSConfig]` returning `map[nsName→*ResolvedNS]` (I-B-010) | High | **v0.9 P0a2** | Complete |
|
||||
| REQ-083 | CLI-side scheduler redesign: `Score(node, workload) (score int, fits bool)` where `fits` checks runtime compatibility + constraints, `score` is bin-packing (most free capacity = highest); Services pick `count` distinct nodes (anti-affinity default); DaemonSets pick all matching nodes; Job = one-shot; CLI-side not daemon-side (R-001) (I-B-011) | High | **v0.9 P05** (skeleton P0c) | Complete |
|
||||
| REQ-084 | `orca job lint` category-driven lint engine: `Linter` runs `Rule` checks returning `Finding{Category, Severity, Message, Explanation}`; categories schema/runtime/security/migration/best-practice; `--explain` prints rationale; pure (no I/O) checks against static rules (I-B-012) | Medium | **v0.11 P11** | **Complete** |
|
||||
| REQ-085 | v0.8→v1.0 migration ordering: v0.9 ships new parser + kinds + runtime + SSH-push alongside old daemon (dual-write window); `orca job run` dispatches on extension (`.md`→SSH-push, `.hcl`→old daemon); v0.10-P05 drains old daemons; v0.10-P14 converts remaining `.hcl` specs and removes daemon (I-C-001). **Most important cross-cutting idea** | High | **v0.9 P00** → v0.10 P14 | Complete |
|
||||
| REQ-086 | "No orca on server" enforcement: `orca doctor no-orca-on-server` SSHs to each peer verifying no `orca` binary in PATH, no `orca` systemd service, no `orca` process, no `/etc/orca/` directory; runs after v0.10-P05 before v0.10-P16; reuses v0.8 `proxmox` SSH session infrastructure (I-C-002). Implements grill C-13 | High | **v0.11 P14c** | **Complete** |
|
||||
| REQ-087 | Test infrastructure: hermetic 3-linux + 1-proxmox cluster pipeline: `test/integration/` with docker-compose/vagrant creating 4 containers/VMs; Go test harness SSHes to each, runs CLI, asserts end-to-end workflows (ns create → workload submit → migrate → drain); proxmox simulated via mock pct/qm; v0.8 e2e tests (bootstrapE2ESetup) are foundation (I-C-003) | Medium | **v0.11 P08** | **Complete** |
|
||||
| REQ-088 | Security-engineer + network-engineer persona reactivation: reactivate security-engineer (step-ca provisioner model, SSH-push blast radius, Traefik edge, .env.secrets crypto) and network-engineer (socket exposure R-007, Syncthing P2P ports, Traefik routing); cross-cutting review not single phase (I-C-004). Implements grill C-05 | High | **v0.9 P00** → v0.10 P16 | Complete |
|
||||
| REQ-089 | Documentation rewrite: ARCHITECTURE.md/PROJECT.md/README + AD-010 supersession: v0.9-P00 adds "v0.9 Architecture (Supersedes v0.8)" section + banners + Superseded Decisions table; v0.10-P15 rewrites README quickstart for new curl|sh + orca init + orca ns create flow (I-C-005) | Medium | **v0.9 P00** + v0.10 P15/P16 | Complete |
|
||||
| REQ-090 | Dual-write window: v0.9 `orca job run` dispatches on extension (`.md`→SSH-push new path, `.hcl`→old daemon path) via parser dispatcher (REQ-064); daemon not removed until v0.10-P05; SSH-push path writes to separate systemd unit namespace (`orca-v1-<alloc>.service`) while daemon uses `orca-<job>.service` — no unit name overlap = no conflict (I-C-006) | High | **v0.9 P00** | Complete |
|
||||
|
||||
## v0.10 Docs & Install Milestone Requirements
|
||||
|
||||
The following requirements are scoped to the v0.10 docs/cli-examples
|
||||
milestone. They cover the CLI reference documentation, jobspec
|
||||
reference, ingress guide, full-stack example jobspecs, README refresh,
|
||||
namespace.md v0.9 layout update, and the release/install pipeline fix
|
||||
that guarantees every Gitea release carries a Linux binary asset.
|
||||
|
||||
| ID | Requirement | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-091 | `docs/cli.md` comprehensive CLI reference: every command/subcommand with synopsis, flags (name/type/default/description), and one-line example; global flags (`--json`, `--system`, `--config`, `--no-deprecation-warnings`); output modes (text vs `--json`, `--watch` table vs NDJSON); exit codes; deprecated surface (`orca daemon`, `orca cert`, `orca node join` mTLS path, legacy `.hcl` jobspec) flagged with callout boxes pointing to v0.10 removal | High | **v0.10 P2** | **Complete** |
|
||||
| REQ-092 | `docs/jobspec.md` markdown frontmatter schema reference: all top-level keys, block reference (runtime, ports, env/secrets, volumes, restart, update, service, health, lifecycle, constraints, affinity, tasks), kinds matrix (Job/Service/DaemonSet required vs allowed), CEL subset grammar, body byte-exact preservation (R-015), deprecated HCL form callout | High | **v0.10 P2** | **Complete** |
|
||||
| REQ-093 | `docs/ingress.md` Traefik ingress reference: `kind: Service` implies Traefik route (D-175), R-007 socket-vs-TCP-bind semantics, generated Traefik YAML shape (routers/services/healthCheck), atomic reload (C-10), drain (`weight: 0`), TLS (certResolver, trust domain, step-ca), worked-example pointer to `examples/full-stack/`, v0.10 forward limitations (socket activation, transactional update) | High | **v0.10 P2** | **Complete** |
|
||||
| REQ-094 | `examples/full-stack/` directory with 5 valid jobspecs (`web-app.md`, `api.md`, `worker.md`, `log-shipper.md`, `postgres.md`) exercising ports/service/health/restart/update/constraints/affinity/lifecycle/task-groups/volumes/replication/DaemonSet; `rendered/` subdir showing the Traefik dynamic YAML + systemd units orca generates; `README.md` walkthrough (init → node join → capacity set → ns create → job run → list --watch → inspect rendered) | High | **v0.10 P3** | **Complete** |
|
||||
| REQ-095 | README.md refresh: status line (v0.9 complete, v0.10 in progress), install `--version` example updated to current tag, subcommand table expanded to all commands with deprecation markers, update-in-place example updated, development targets complete (`verify-reqs`, `security-scan`, `test-race`, `changelog`), new Documentation + Examples sections linking all `docs/*.md` and `examples/` | High | **v0.10 P4** | **Complete** |
|
||||
| REQ-096 | `docs/namespace.md` v0.9 multi-namespace layout update: replace v0.8 flat path table with v0.9 layout (`cluster/`, `_defaults/`, per-ns `db/jobs/alloc/ns.md`), `ORCA_HOME`/`--system` resolution, `orca ns` subcommand cross-link, v0.8 flat layout flagged deprecated | Medium | **v0.10 P4** | **Complete** |
|
||||
| REQ-097 | `scripts/release.sh` release pipeline fix: cross-build `linux-amd64` tarball regardless of host arch (`GOOS=linux GOARCH=amd64 go build`); post-create asset verification (query `/releases/tags/$VERSION`, assert the tarball in attachments, retry/fail loudly if missing). Guarantees every Gitea release carries the Linux binary asset (root cause of v0.4.5 install) | High | **v0.10 P1** | **Complete** |
|
||||
| REQ-098 | `scripts/install.sh` asset fallback walk: if the latest/pinned release lacks the matching `orca-<ver>-<os>-<arch>.tar.gz`, walk backward through `/releases?limit=20` to the most recent release that has it, with a clear warning. Keeps pulling from releases (not main). Optional `--check` dry-run mode | High | **v0.10 P1** | **Complete** |
|
||||
|
||||
## v0.11 Production Hardening Milestone Requirements
|
||||
|
||||
The following requirements (REQ-099…REQ-NN) are scoped to the v0.11
|
||||
production-hardening milestone. They cover the ingress hybrid default
|
||||
(R-017), drift detection (R-018/R-019/R-020), the systemd Path unit
|
||||
implementation (D-227…D-237), and five net-new CLI commands added per
|
||||
operator decision Q2=C.
|
||||
|
||||
### Ingress hybrid (R-017, D-215…D-226)
|
||||
|
||||
| ID | Requirement | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-099 | `internal/emitter/nft.go`: nftables emitter renders `/etc/nftables.d/orca.nft` with DNAT (`:443`→`127.0.0.1:8443`, `:80`→`127.0.0.1:8080`), SYN-flood `tcp-flags` filter, `ora_rl` rate-limit meter (default 100/s burst 200), `orca_trusted_probes` set; idempotent `nft -f` apply; atomic rule-set swap (R-017, D-217, D-218, D-222) | High | **v0.11 P15.5** | **Complete** |
|
||||
| REQ-100 | Traefik static config emitter update: `entryPoints.websecure.address` changes from `:443` to `127.0.0.1:8443` (default); `entryPoints.web.address` changes to `127.0.0.1:8080`; `--public-binding=traefik-on-public-ip` opt-out emits `:443`/`:80` instead; certs/mTLS/dynamic config unchanged (R-017, D-220, D-216) | High | **v0.11 P15.5** | **Complete** |
|
||||
| REQ-101 | `orca doctor nft`: checks `table inet orca-ingress` exists, expected DNAT rules present, rate-limit meter present, `/etc/nftables.d/orca.nft` parses cleanly (`nft -c -f`), file hash matches latest applied txn; drift detection via hash comparison (R-018 critical_paths, D-221, D-226) | High | **v0.11 P15.5** | **Complete** |
|
||||
| REQ-102 | `orca nft` CLI: `show [--peer]`, `diff --against <txn-id>`, `doctor` (alias for `orca doctor nft`), `country block add <cc-list>` (opt-in GeoIP), `rate limit set --rate N/s`; all Layer-5 orchestrators that SSH into peers and parse `nft` output (D-223, D-222) | Medium | **v0.11 P15.5** | **Complete** |
|
||||
|
||||
### Drift detection (R-018/R-019/R-020, D-227…D-237)
|
||||
|
||||
| ID | Requirement | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-103 | `internal/drift` package: `Detector` interface (`Watch`, `Aggregate`, `Remediate`, `Acknowledge`), `Event`, `Config`, `PathSpec`, `RemediationPolicy` types; `iter.Seq2[Event, error]` per D-017; `signal.NotifyContext` per D-023 (R-018, D-236) | High | **v0.11 P10** | **Complete** |
|
||||
| REQ-104 | `orca drift` CLI tree: `watch [--interval=2s] [--paths=...] [--json]`, `show [--peer]`, `acknowledge <peer> <path>`, `remediate <peer> <path> [--force]`, `config show`, `config validate`; uses `iter.Seq2` + `signal.NotifyContext` (D-236) | High | **v0.11 P10** | **Complete** |
|
||||
| REQ-105 | systemd Path unit emitter: for each critical path, emit `orca-drift-<name>.path` (`PathChanged=`, `RateLimitIntervalSec=1s`, `RateLimitBurst=5`) + `orca-drift-<name>.service` (`Type=oneshot`, `ExecStart=/usr/local/bin/orca-drift-notify.sh %f`, `User=orca`, security hardening: `NoNewPrivileges`, `ProtectSystem=strict`); R-001-clean (R-018, D-227, D-228) | High | **v0.11 P10** | **Complete** |
|
||||
| REQ-106 | `scripts/orca-drift-notify.sh`: receives changed path as `$1`, computes sha256 (or "DELETED"), writes event JSON to `/etc/orca/state/drift-events/<event-id>.json` (event_id, ts, host, path, status, new_sha256, latest_txn, triggered_by); stateless, idempotent; `flock` for serialization (D-228) | High | **v0.11 P10** | **Complete** |
|
||||
| REQ-107 | `scripts/orca-aggregate.sh` extension: existing 10s aggregator cadence (C-11) now also rsyncs each peer's `/etc/orca/state/drift-events/`, validates event hashes against `/etc/orca/state/applied/<txn>/manifest.json`, triggers `orca-remediate.sh` for auto-remediable paths, consumes (deletes) event files on peers (D-229, D-237) | High | **v0.11 P09** | **Complete** |
|
||||
| REQ-108 | `scripts/orca-remediate.sh`: re-pushes latest applied txn's per-peer render tree via rsync, runs peer-side applier; 5-min cooldown per path applies ONLY on successful remediation (transient failures retry next tick); cooldown state at `/etc/orca/state/remediation-cooldown/` (D-231, D-232 refined per CLARIFY C4) | High | **v0.11 P10** | **Complete** |
|
||||
| REQ-109 | Drift cadence config in `config.md` (`kind: ClusterConfig`): `drift.polling.{enabled,default_interval,max_concurrent_peers}`, `drift.paths.{critical,standard,excluded}` (each with `systemd_path_unit`, `interval`, `paths` list), `drift.remediate.{auto,auto_paths,require_approval_paths,notify_on_remediation}`; critical defaults: Traefik dynamic, nftables, sudoers, orca-alloc services; secrets + `/run/orca/*` + drift-events dir excluded (R-018, D-231, D-234) | High | **v0.11 P10** | **Complete** |
|
||||
| REQ-110 | Pre-flight consistency gate in applier: `orca-pull.sh` (C-09) refuses new txns if drift detected on the target peer/namespace; `--force` flag overrides; per-namespace scoping means a drifted peer in ns-A does not block ns-B (R-020, Q4=A) | High | **v0.11 P10** | **Complete** |
|
||||
| REQ-111 | `orca` system user on peers: peer-setup emits `useradd -r orca` (system account, no login shell); `orca-drift-*.service` runs as `User=orca Group=orca`; SSH key access to lead for aggregator; idempotent at peer setup (net-new operational requirement from doc 5) | High | **v0.11 P10** | **Complete** |
|
||||
| REQ-112 | NFS detection at peer setup: `orca node join` / peer-setup detects NFS mounts on orca state dirs; if `/etc/orca` is on NFS, systemd Path units are disabled for those paths and polling is the only detection; logs a warning (D-233) | Medium | **v0.11 P10** | **Complete** |
|
||||
| REQ-113 | `orca job restart <name>`: restarts an allocation to pick up EnvironmentFile drift; goes through normal allocation lifecycle (not file-level remediation); triggers on drift of `/etc/orca/allocs/<id>/env` (D-235) | Medium | **v0.11 P10** | **Complete** |
|
||||
|
||||
### Net-new CLI surface (Q2=C — all five commands added to v0.11)
|
||||
|
||||
| ID | Requirement | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-114 | `orca cluster rotate-lead`: moves cluster CA + lead state to a new bare-Linux peer (R-003 enforces bare-Linux-only lead); workloads keep running (certs already distributed); SSH key rotation; idempotent (Q2=C, folds into P14b daemon cutover) | High | **v0.11 P14b** | **Complete** |
|
||||
| REQ-115 | `orca upgrade --to-vX`: thin wrapper around `install.sh` + `orca restore` (binary upgrade only, not full cluster rolling upgrade); handles Traefik binding cutover from `:443` to `127.0.0.1:8443` for existing v0.9/v0.10 clusters (R-017 migration path, CLARIFY C1, C2=a thin wrapper); full cluster-rolling-upgrade defers to v1.x (Q2=C) | High | **v0.11 P14a** | **Complete** |
|
||||
| REQ-116 | `orca job migrate <name> --to <node>`: drain+reschedule composite (uses P05 drain + P06 alloc history); live-migrate with storage replication defers to v1.x (CLARIFY C3=a); idempotent (Q2=C) | Medium | **v0.11 P05** | **Complete** |
|
||||
| REQ-117 | `orca logs --all-nodes --since 5m`: aggregates journald logs across peers via SSH; uses P06 alloc-history cache DB; `iter.Seq` streaming per D-017; `--since` duration flag; `--all-nodes` fans out (Q2=C, folds into P06) | Medium | **v0.11 P06** | **Complete** |
|
||||
| REQ-118 | `orca doctor mTLS`: verifies trust chain (CA → server cert → workload SVIDs exist + not expired) AND live mTLS handshake probe to each peer (reuses P01 metrics endpoint + P01.5 SPIFFE spike infra); both chain verification + live probe (CLARIFY C5, Q2=C, folds into P15.5) | High | **v0.11 P15.5** | **Complete** |
|
||||
|
||||
### Scope notes
|
||||
|
||||
- REQ-099…REQ-118 = 20 net-new requirements (REQ count grows 98→118).
|
||||
- No new phases added (Q3=A folds ingress into P15.5; Q2=C folds CLI commands into existing phases).
|
||||
- P09 expands (REQ-107 aggregator extension); P10 expands (REQ-103…REQ-113, the largest phase); P15.5 expands (REQ-099…REQ-102 ingress + REQ-118 mTLS doctor).
|
||||
- P05 gains REQ-116 (migrate); P06 gains REQ-117 (logs --all-nodes); P14a gains REQ-115 (upgrade); P14b gains REQ-114 (rotate-lead).
|
||||
|
||||
## v0.12 Milestone Summary — Security Hardening (Zero-Trust Identity)
|
||||
|
||||
**Status**: in progress (Phase 0). 30 net-new requirements (REQ-119..REQ-148)
|
||||
derived from the v0.12 threat-model review (25 findings F1..F25) and the
|
||||
zero-trust identity model (R-021). See ROADMAP.md for the 29-phase plan
|
||||
(P0 + P01..P27 + P28 final) and RESEARCH_v0.12.md for the full threat model.
|
||||
|
||||
### Wave A — Critical injection & traversal
|
||||
|
||||
| ID | Requirement | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-119 | Command injection fix in `internal/runtime/podman.go` & `wasm.go`: shell-quote `cmdStr` via `shellQuote` in SSH exec interpolation (`podman.go:57`, `wasm.go:39`); add injection regression tests (bats + Go) covering `;`, `\|`, `$()`, backticks, newline injection (F3) | High | **v0.12 P01** | pending |
|
||||
| REQ-120 | Namespace path traversal fix: `validateNamespaceName` in `internal/ns/` rejects `..`, `/`, leading `-`, null bytes, control chars in `ns create`/`ns inherit`/`ns set-constraint`; add fuzz test (F4) | High | **v0.12 P02** | pending |
|
||||
| REQ-121 | Txn apply path allowlist: `apply.sh` python heredoc validates every `path` in `desired-state.json` against a prefix allowlist (`/etc/orca/`, `/etc/traefik/orca*`, `/etc/systemd/system/orca-*`, `/etc/nftables.d/orca*`, `/etc/syncthing/orca*`); rejects otherwise; HMAC-signed manifest unchanged (F5) | High | **v0.12 P03** | pending |
|
||||
|
||||
### Wave B — Zero-trust identity
|
||||
|
||||
| ID | Requirement | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-122 | ACL enforcement wiring: `acl.Check` invoked in daemon handlers (read/write/admin by route) and SSH-push applier (validates `ORCA_OIDC_TOKEN` env var against JWKS before applying any txn); deny-by-default enforced; actor recorded in audit (F1, foundational for REQ-145) | High | **v0.12 P06** | pending |
|
||||
| REQ-123 | Daemon auth hardening: mandatory mTLS (remove plaintext mode entirely); OIDC bearer accepted as second factor on human-facing endpoints; `MaxBytesReader` body limits; pprof loopback-only by default, refuse non-loopback without `--pprof-allow-public` confirmation (F6, F24) | High | **v0.12 P09** | pending |
|
||||
| REQ-124 | HTTP request body size limits: `http.MaxBytesReader` on all JSON-decoding handlers; `MaxHeaderBytes` set; rejects oversized bodies (F24) | Medium | **v0.12 P09** | pending |
|
||||
| REQ-125 | Audit log tamper-evidence: hash-chained entries (`prev_hash = sha256(prev_row \|\| payload)`), HMAC-SHA256 under master key on the chain head; `orca doctor audit` verifies the chain; append-only enforcement via SQLite trigger blocking UPDATE/DELETE; actor field carries OIDC `sub` or SPIFFE SVID (F2) | High | **v0.12 P10** | pending |
|
||||
| REQ-126 | SVID chain validation: `VerifySVID` validates the full cert chain against the CA pool, not just the URI SAN; reject certs signed by unknown CAs even with correct URI (F9) | High | **v0.12 P11** | pending |
|
||||
| REQ-127 | Backup symlink validation: `Restore` rejects `Linkname` that's absolute, contains `..`, or points outside `ORCA_HOME`; add regression test with crafted tarball (F7) | High | **v0.12 P12** | pending |
|
||||
| REQ-128 | step-ca /tmp hardening: `step ca certificate` writes to 0600 temp under `ClusterDir()/step-tmp/` (or `TMPDIR` override), not world-readable `/tmp`; cleanup in `defer` (F10) | High | **v0.12 P13** | pending |
|
||||
| REQ-129 | Master key rotation: `orca secrets rotate-master` re-encrypts all namespace secrets under a new master key; new master key re-sealed to OIDC as part of the same operation; `--dry-run` + atomic + automatic rollback to old sealed key on any ns failure; no passphrase (R-021) (F12) | High | **v0.12 P14** | pending |
|
||||
| REQ-130 | File-mode audit expansion: `EnforceFileModes` extended to SSH key, master key (sealed blob), server cert/key, known_hosts; `orca doctor modes` checks all; startup refuses to run on violation (F13) | Medium | **v0.12 P15** | pending |
|
||||
| REQ-131 | aggregate.sh JSON injection fix + drift-gate parse fix: replace `printf` interpolation with `jq`-based JSON construction (or Go-side aggregator emitting JSON); fix `orca-pull.sh` R-020 parsing to use `jq` instead of grep (F11, F18) | High | **v0.12 P16** | pending |
|
||||
| REQ-132 | install.sh checksum+GPG verification: release.sh publishes `SHA256SUMS` + `SHA256SUMS.asc` (GPG-signed) alongside tarball; install.sh verifies before `tar -xzf`; fail closed on mismatch (F14) | High | **v0.12 P17** | pending |
|
||||
| REQ-133 | nftables ruleset hardening: add conntrack bounds (`ct state established,related accept`), input default-deny on orca chain, drop invalid packets; `orca doctor nft` audits live ruleset against emitted one (F21) | Medium | **v0.12 P18** | pending |
|
||||
| REQ-134 | sudoers hardening: add NOEXEC to `apt-get`/`dpkg` (or remove if unused); `orca doctor proxmox` audits sudoers file against expected allowlist (F22) | Medium | **v0.12 P19** | pending |
|
||||
| REQ-135 | System user consistency: Proxmox bootstrap creates `nologin` system user (`-r -s /usr/sbin/nologin`), matching peer-setup; `orca doctor` flags inconsistency on existing peers; `orca upgrade` migrates (F23) | Medium | **v0.12 P20** | pending |
|
||||
| REQ-136 | SQLite file-mode + at-rest encryption: `store.Open` sets DB file mode 0600; optional `--encrypt-db` (CGO-free fallback per C-31: file-mode 0600 + documented threat if SQLCipher needs CGO); no CGO (F8) | High | **v0.12 P21** | pending |
|
||||
| REQ-137 | Migration safety: `copyFile` -> atomic temp+rename; `migrateDBSchema` runs in transaction with `foreign_keys(ON)`; pre-migration backup step (uses `internal/backup`); document manual rollback; v0.11->v0.12 identity migration: `orca upgrade` refuses clusters using `--password`/bare-tokens without `--accept-identity-migration` (F19, C-34) | High | **v0.12 P22** | pending |
|
||||
| REQ-138 | Legacy CA/mTLS/daemon + step-ca password-provisioner deletion: remove `internal/security/ca.go` legacy CA, `internal/transport/mtls.go` deprecated path, daemon plaintext mode; migrate `orca init`/`orca cert *` to step-ca exclusively; `certpaths` (v0.8 layout) removed; delete step-ca `--password-file` provisioner (replaced by OIDC provisioner); **gate: P06/P08/P09/P11 all shipped** (F16) | High | **v0.12 P23** | pending |
|
||||
| REQ-139 | known_hosts tightening + transport hardening: `Flock` tightens pre-existing looser perms to 0600; `classifyDialErr` switched from substring to typed errors; add SSH-exec rate limiting (token bucket per peer) (F15, F25) | Medium | **v0.12 P24** | pending |
|
||||
| REQ-140 | Drift event authentication: drift events signed with per-peer HMAC key (derived from master key); aggregator rejects unsigned/forged events; `orca-drift-notify.sh` reads key from 0600 file owned by `orca` (F18) | Medium | **v0.12 P25** | pending |
|
||||
| REQ-141 | Security integration test suite: hermetic harness exercising injection, traversal, symlink, drift-forgery, audit-tamper, daemon-auth-negative, OIDC mock-IdP flow, ACL-with-OIDC-claims negative tests, unseal/seal, WebAuthn virtual-authenticator ceremony, password-removal regression (assert `--password` is rejected); gates in `.coreci.yml` `validate` (C-33) | High | **v0.12 P26** | pending |
|
||||
| REQ-142 | Zero-trust + OIDC + WebAuthn + threat-model docs: `docs/threat-model.md` (STRIDE + zero-trust model + OIDC data-flow), `docs/oidc.md` (configure your IdP, Dex offline quickstart, claim-to-namespace mapping), `docs/webauthn.md` (passkey registration, RP ID, secure context), `docs/security-runbook.md` (unseal/seal, master key rotation, incident response, sudoers audit, nft audit); README security section names "no orca credentials" as an invariant | Medium | **v0.12 P27** | pending |
|
||||
| REQ-143 | Final review + ship + audit: multi-persona review across all phases, `ciagent-audit` reconstruction test, milestone merge to main, tag `v0.11.29` (= v0.12 milestone release per feature-milestone rule) | High | **v0.12 P28** | pending |
|
||||
| REQ-144 | OIDC client + bundled Dex: `orca auth login`/`logout`/`status`/`init-idp`; OIDC config block (`oidc.issuer`, `client_id`, `client_secret`, `scopes`); bundled Dex systemd unit + Traefik route on the lead; BYO external IdP override via `oidc.issuer` repoint; JWKS caching + refresh; token storage at `~/.orca/credentials.json` (0600); `--oidc` flag on commands requiring identity; browser auth-code + PKCE + local loopback redirect; headless device-code fallback (D-238..D-247) | High | **v0.12 P04** | pending |
|
||||
| REQ-145 | ACL rewrite to OIDC claims: remove `KindToken` entirely; `KindSpiffe` stays for machine identity; new `KindOidc` maps `sub`+`groups` -> namespace permissions; `acl.Check` takes OIDC claims struct; deny-by-default enforced in daemon + SSH-push applier; `acl.json` mode tightened to 0600 (F1) | High | **v0.12 P06** | pending |
|
||||
| REQ-146 | Remove all password/token paths (breaking): delete `--password`/`$ORCA_PROXMOX_PASSWORD` from Proxmox join (replace with pre-staged-key-only or `step ssh` OIDC cert exchange); delete step-ca `--password-file` provisioner (migrate to OIDC provisioner); delete any bare-token CLI paths; documented in migration guide (R-021, C-34) | High | **v0.12 P07** | pending |
|
||||
| REQ-147 | Master key seal-to-OIDC + Shamir recovery: master key encrypted with key derived from OIDC token exchange at unseal; `orca cluster unseal`/`seal`; sealed blob at `ClusterDir()/master.key.sealed` (0600); raw key never on disk; Shamir 3-of-5 shards printed at seal time; recovery via `--recovery` + 3 shards; mTLS-only offline path derives seal key from cluster CA (D-241, C-35) | High | **v0.12 P08** | pending |
|
||||
| REQ-148 | WebAuthn connector for Dex (passkeys): `orca-webauthn-connector` (~300 LoC Go, `go-webauthn`); register/login ceremonies at `/orca/webauthn/{register,login}` behind Traefik; `orca auth register` browser flow; passkey storage SQLite `ClusterDir()/webauthn-credentials.db` (0600, public keys only); RP ID = cluster Traefik domain; secure context via step-ca cert; headless device-code fallback; virtual-authenticator integration tests (D-240, D-243, D-244, C-38) | High | **v0.12 P05** | pending |
|
||||
|
||||
### Scope notes (v0.12)
|
||||
|
||||
- REQ-119..REQ-148 = 30 net-new requirements (REQ count grows 118 -> 148).
|
||||
- 29 phases (P0 + P01..P27 + P28 final); GRILL may split/merge.
|
||||
- P04 (OIDC+Dex) and P05 (WebAuthn) are the new `feat` phases; the rest are `fix`/`chore`/`test`/`docs`/`refactor`. Milestone type = feature (at least one `feat`).
|
||||
- Tags on v0.11.x patch line: `v0.11.0` (P0) ... `v0.11.29` (P28 final = v0.12 milestone release).
|
||||
- v1.0.0 production-ready tag stays deferred for post-v0.12 UAT (per v0.11 PRD).
|
||||
|
||||
@@ -0,0 +1,140 @@
|
||||
# Research: v0.10 Docs & Install Milestone
|
||||
|
||||
## Documentation landscape in the orca tree
|
||||
|
||||
### What exists today
|
||||
|
||||
The `docs/` directory contains four files:
|
||||
|
||||
- `docs/install.md` — install guide (user-level, system-level, version
|
||||
pinning, in-place update, troubleshooting). Accurate for v0.5-v0.8
|
||||
but does not mention the v0.9 multi-namespace layout, `--config`, or
|
||||
`--no-deprecation-warnings`.
|
||||
- `docs/docker.md` — Docker image guide. Still documents `orca daemon`
|
||||
(deprecated in v0.9).
|
||||
- `docs/namespace.md` — namespace and paths. Documents the **v0.8 flat
|
||||
layout** (`~/.orca/orca.db`, `ca.crt`, `ca.key`, `server.crt`,
|
||||
`server.key`). Does NOT document the v0.9 multi-namespace layout
|
||||
(`cluster/`, `_defaults/`, per-ns `db/jobs/alloc/ns.md`), `orca ns`
|
||||
subcommands, or the `_defaults` implicit root (D-159/D-185/D-187).
|
||||
- `docs/security-scanning.md` — gosec + govulncheck + gitleaks guide.
|
||||
Accurate; no v0.9 drift.
|
||||
|
||||
### What's missing (the gap this milestone closes)
|
||||
|
||||
1. **No CLI reference doc.** The entire CLI command surface (init, job,
|
||||
node, ns, cert, daemon, doctor, status, audit, version) is
|
||||
undocumented in `docs/`. The README subcommand table is stale (lists
|
||||
only version/init/status/node/job with fake "Phase N" statuses,
|
||||
missing cert/daemon/doctor/audit/ns/node-capacity/node-key-reset).
|
||||
2. **No jobspec reference doc.** The markdown frontmatter schema (kinds,
|
||||
blocks, CEL subset, validation rules, body semantics) is
|
||||
undocumented. Operators must read `internal/jobspec/markdown.go` and
|
||||
`internal/spec/schema/schema.go` source.
|
||||
3. **No ingress/Traefik doc.** The service→Traefik mapping, R-007
|
||||
socket-vs-TCP-bind, atomic reload, drain, TLS — all undocumented.
|
||||
4. **No examples directory.** `testdata/` holds legacy HCL fixtures
|
||||
(`hello.hcl`, `fail.hcl`) for Go tests, not operator-facing
|
||||
examples. No worked full-stack demo exists.
|
||||
5. **README is stale.** Status line says "v0.1: Foundation".
|
||||
Subcommand table missing 5 commands. Install `--version` example
|
||||
pins v0.4.2. Update-in-place example references v0.4.1→v0.4.2.
|
||||
Development section omits 4 make targets.
|
||||
|
||||
### Prior art for CLI reference docs
|
||||
|
||||
- **Nomad**: `nomad job` / `nomad node` / `nomad agent` reference pages,
|
||||
one per subcommand, with flag tables and JSON examples. Orca's
|
||||
single-file `docs/cli.md` is simpler (one file vs a subdirectory) but
|
||||
follows the same flag-table + example convention.
|
||||
- **kubectl**: `kubectl reference` + per-command pages. Too heavy for
|
||||
orca; the single-file model fits the minimalist ethos.
|
||||
- **Docker CLI**: `docker run` reference with flag tables. Matches the
|
||||
shape orca's `docs/cli.md` will take.
|
||||
|
||||
### Prior art for example jobspecs
|
||||
|
||||
- **Nomad example jobs**: `nomad-job-spec.example` files in the Nomad
|
||||
repo showing service + job + sysbatch patterns. Orca's
|
||||
`examples/full-stack/` mirrors this with 5 markdown jobspecs covering
|
||||
Service/Job/DaemonSet + task groups + volumes + replication.
|
||||
- **Kubernetes examples**: `examples/` directory with yaml
|
||||
deployments/services/ingress. Orca's equivalent is the 5 jobspecs +
|
||||
rendered Traefik/systemd artifacts.
|
||||
|
||||
## Release/install pipeline research
|
||||
|
||||
### Root cause of the v0.4.5 install
|
||||
|
||||
Verified via the Gitea API:
|
||||
|
||||
```
|
||||
GET /api/v1/repos/coreci/orca/releases/latest
|
||||
→ tag_name: "v0.8.15"
|
||||
|
||||
GET /api/v1/repos/coreci/orca/releases/tags/v0.8.15
|
||||
→ attachments: [] (zero binary assets)
|
||||
```
|
||||
|
||||
The v0.8.x releases (v0.8.0 through v0.8.15) all shipped with **zero
|
||||
binary assets attached**. Only `v0.4.5` carries a tarball
|
||||
(`orca-v0.4.5-linux-amd64.tar.gz`).
|
||||
|
||||
`scripts/install.sh:70-78` resolves "latest" → v0.8.15, then
|
||||
`install.sh:96-104` looks for `orca-v0.8.15-linux-amd64.tar.gz` in
|
||||
v0.8.15's assets. Since the asset is missing, install.sh errors out
|
||||
(`could not find asset ... in release v0.8.15`). The v0.4.5 install
|
||||
came from an earlier run or a pinned `--version`.
|
||||
|
||||
### Why v0.8.x releases have no assets
|
||||
|
||||
`scripts/release.sh:132-136` calls `tea releases create "$VERSION" ...
|
||||
--asset "$TARBALL"`. The script builds the tarball (line 98) and passes
|
||||
it to `tea`. Two likely failure modes:
|
||||
|
||||
1. **Host arch mismatch**: `release.sh:89-95` builds for the host arch
|
||||
(`uname -m`). If the CI runner or dev machine is arm64, it produces
|
||||
`orca-v0.8.15-linux-arm64.tar.gz`, but `install.sh` looks for
|
||||
`linux-amd64`. The `.coreci.yml:121` release step hardcodes
|
||||
`--asset orca-${VERSION}-linux-amd64.tar.gz`, so the CI runner must
|
||||
be amd64 — but `release.sh` run locally on an arm64 dev machine
|
||||
produces the wrong arch.
|
||||
2. **Silent asset drop**: `tea releases create` has been observed to
|
||||
succeed (exit 0) without attaching the asset in some tea versions.
|
||||
The script treats `tea`'s exit code as success without verifying the
|
||||
asset actually appears in the release.
|
||||
|
||||
### Fix approach (REQ-097, REQ-098)
|
||||
|
||||
**release.sh**:
|
||||
- Cross-build `linux-amd64` explicitly via
|
||||
`GOOS=linux GOARCH=amd64 go build`, regardless of host arch.
|
||||
- After `tea releases create`, query
|
||||
`/api/v1/repos/$OWNER/$REPO/releases/tags/$VERSION` and assert the
|
||||
tarball appears in `attachments`. If not, retry once, then fail
|
||||
loudly with a clear error.
|
||||
|
||||
**install.sh**:
|
||||
- Add an asset fallback walk: if the resolved release (latest or
|
||||
pinned) lacks the matching tarball, query
|
||||
`/releases?limit=20`, walk backward, and use the most recent release
|
||||
that carries the `orca-<ver>-<os>-<arch>.tar.gz` asset. Print a
|
||||
clear warning.
|
||||
- Add `--check` dry-run mode (D-194) that prints the version + asset URL
|
||||
+ install path without writing.
|
||||
|
||||
## Persona assessment (PERSONAS.md)
|
||||
|
||||
This milestone touches two territories:
|
||||
|
||||
1. **`scripts/` (release.sh, install.sh)** — bash scripts, not Go.
|
||||
Backend-engineer territory (API-adjacent tooling). The fix is
|
||||
cross-build + API verification + fallback walk.
|
||||
2. **`docs/` + `examples/` + `README.md`** — markdown documentation.
|
||||
Lead-developer territory (coordination + cross-cutting docs).
|
||||
|
||||
No data-engineer work (no schema/migration changes). No
|
||||
frontend-engineer work (no UI). The data-engineer persona is
|
||||
deactivated for this milestone. A docs-engineer custom persona is
|
||||
created for P2/P3/P4 (markdown authoring with codebase-grounded
|
||||
factual claims).
|
||||
@@ -0,0 +1,164 @@
|
||||
# Research: v0.11 Production Hardening
|
||||
|
||||
## Source material
|
||||
|
||||
Five research documents were ingested 2026-08-07 as directional input
|
||||
(not verbatim) for v0.11 Phase 0. The current ciagent files
|
||||
(R-001…R-016, D-001…D-206) are authoritative and take precedence; where
|
||||
research conflicted, the ciagent files won. The research drove the
|
||||
adoption of R-017…R-020 and D-215…D-237 (see PROJECT.md, PRD_v0.11.md).
|
||||
|
||||
| Doc | Theme | Adopted as |
|
||||
|-----|-------|------------|
|
||||
| 1 | Ingress hybrid (nft DNAT → Traefik on 127.0.0.1:8443) | R-017, D-215..D-226, REQ-099..REQ-102 |
|
||||
| 2 | Platform-engineer playbook (8 differentiators, TCO, honest trade-offs) | README positioning (Q5=A), CLI surface gap analysis |
|
||||
| 3 | Strategic positioning ("be Proxmox-for-bare-metal, not K8s-without-K8s") | README framing (Q5=A: Nomad-inspired, honest trade-offs table from doc 3, not Proxmox-first lead) |
|
||||
| 4 | Drift detection cadence (R-018/R-019/R-020, tiered cadence, hard gate) | R-018, R-019, R-020, D-227..D-237, REQ-103..REQ-113 |
|
||||
| 5 | Drift detection concrete impl (systemd Path units, orca-drift-notify.sh, orca-remediate.sh) | D-227..D-237 detail, REQ-103..REQ-113 |
|
||||
|
||||
## Thread A — Ingress hardening (doc 1)
|
||||
|
||||
### What changes vs v0.9/v0.10
|
||||
|
||||
Traefik static config gains `entryPoints.websecure.address: 127.0.0.1:8443`
|
||||
(default) instead of `:443`. A new nftables emitter renders
|
||||
`/etc/nftables.d/orca.nft` with DNAT rules. Certs, mTLS, dynamic config,
|
||||
and the workload SPIFFE validation path are **completely unchanged**.
|
||||
Only the `address` line shifts + one new emitter + `orca doctor nft` +
|
||||
`orca nft ...` CLI.
|
||||
|
||||
### Defense in depth
|
||||
|
||||
Two layers: kernel (nftables: SYN flood, rate limit, GeoIP, conntrack)
|
||||
and application (Traefik: mTLS, SNI, ACL, dynamic routing, health
|
||||
checks). Neither can replace the other; they catch different attack
|
||||
classes.
|
||||
|
||||
### Codebase reality (verified 2026-08-07)
|
||||
|
||||
- `internal/emitter/traefik.go` + `traefik_atomic.go` exist (v0.9 P02).
|
||||
The static-config emitter is where the `address:` line change lands.
|
||||
- `internal/emitter/systemd.go` exists. New `.path`/`.service` unit
|
||||
types extend this emitter pattern (shared with drift detection, doc 5).
|
||||
- `internal/emitter/nft.go` does **not** exist — greenfield, ~200 LoC.
|
||||
- `scripts/` has `orca-verify-render.sh` but **not** `orca-aggregate.sh`,
|
||||
`orca-pull.sh`, `orca-apply-render.sh`, `orca-remediate.sh` — all are
|
||||
v0.11 P09/P10 scope.
|
||||
|
||||
## Thread B — Drift detection + transactional plane (docs 4 + 5)
|
||||
|
||||
### Architecture
|
||||
|
||||
systemd Path units (R-001-clean; systemd is OS, not Orca) watch critical
|
||||
paths via inotify. On change, a oneshot service computes sha256 and
|
||||
writes an event JSON to `/etc/orca/state/drift-events/`. The lead's
|
||||
aggregator timer (10s, C-11) rsyncs these events, validates against the
|
||||
applied txn manifest, and triggers remediation for auto-remediable
|
||||
paths.
|
||||
|
||||
### Tiered cadence
|
||||
|
||||
| Tier | Detection | Auto-remediate | Latency |
|
||||
|------|-----------|----------------|--------|
|
||||
| Critical | Path unit + 5s polling backstop | yes (config files only) | ~10s |
|
||||
| Standard | 30s polling | optional (systemd units: require approval) | 30s |
|
||||
| Default | 60s polling | no | 60s |
|
||||
|
||||
### R-020 hard gate
|
||||
|
||||
Applier refuses new txns if pre-flight consistency check fails. Override:
|
||||
`--force` flag + per-namespace scoping (Q4=A) — a drifted peer in ns-A
|
||||
does not block ns-B.
|
||||
|
||||
### Codebase reality (verified 2026-08-07)
|
||||
|
||||
- `internal/store/node_repo.go:80` and `internal/store/job_task_repo.go:82`
|
||||
already use `iter.Seq[T]`. Doc 5's `iter.Seq2[Event, error]` is the
|
||||
natural extension per D-017 (settled, shipped v0.3).
|
||||
- `internal/paths/paths.go:86` has `TxnDir()` — the txn staging dir the
|
||||
drift detector hooks into.
|
||||
- `internal/emit/contract.go` has the Go↔bash render-contract anti-drift
|
||||
(C-16). The *runtime* drift detector (doc 5) is net-new.
|
||||
- `internal/drift/` package does **not** exist — greenfield, ~500 LoC.
|
||||
- No `orca` system user creation in code — net-new operational
|
||||
requirement (REQ-111).
|
||||
- No NFS detection at peer setup — net-new (REQ-112, D-233).
|
||||
- `doctor.go` has an OS-drift *check* (one-shot, on-demand) but **not**
|
||||
a 60s runtime drift-polling loop. Doc 5's design is net-new scope.
|
||||
|
||||
### Alignment with existing gates
|
||||
|
||||
- **C-09** (`orca-pull.sh` failure contract) — R-020 refines
|
||||
"deterministic state" into an explicit refusal contract.
|
||||
- **C-11** (lead-side watchdog meta-timer) — doc 5's aggregator
|
||||
extension is the input C-11 monitors.
|
||||
- **REQ-075** (lead applier execution model) — doc 5's `orca-remediate.sh`
|
||||
is literally the same code path as a normal txn-apply, triggered by
|
||||
drift instead of a new submission.
|
||||
|
||||
## Thread C — Positioning/messaging (docs 2 + 3)
|
||||
|
||||
### Consistent with locked vision
|
||||
|
||||
The vision is *"A minimalist, offline-first, CLI-first orchestration
|
||||
engine inspired by HashiCorp Nomad"* — explicitly Nomad-inspired, not
|
||||
K8s. Doc 3's recommendation ("be Proxmox-for-bare-metal, not
|
||||
K8s-without-the-complexity") is consistent with the locked vision.
|
||||
|
||||
### Where doc 3 diverges (resolved per Q5=A)
|
||||
|
||||
Doc 3 recommends "leading with Proxmox positioning." But R-003 says
|
||||
"Proxmox can never be lead." Leading the *project identity* with a node
|
||||
type that can't be the lead is subtly contradictory. **Q5=A decision**:
|
||||
README uses the Nomad-inspired, OS-as-cluster framing (locked vision),
|
||||
mentions Proxmox as one node type, and incorporates doc 3's "honest
|
||||
trade-offs" table but not its Proxmox-first lead-positioning advice.
|
||||
|
||||
### CLI surface gap analysis (doc 2)
|
||||
|
||||
Doc 2's playbook cites ~10 CLI commands. Verified against the live
|
||||
codebase (`internal/cli/*.go`):
|
||||
|
||||
**Exist today**: `orca init`, `orca node {join,leave,list,key-reset,
|
||||
capacity}`, `orca job {run,list,stop,logs}`, `orca ns {list,create,
|
||||
delete,inspect,validate}`, `orca cert {ca-init,gen,show,renew,
|
||||
fingerprint}`, `orca doctor {cert,network,db,os,proxmox}`, `orca audit
|
||||
list`, `orca status`, `orca version`, `orca daemon` (deprecated).
|
||||
|
||||
**Not in v0.11 ROADMAP, added per Q2=C**: `orca cluster rotate-lead`
|
||||
(REQ-114, P14b), `orca upgrade` (REQ-115, P14a), `orca job migrate`
|
||||
(REQ-116, P05), `orca logs --all-nodes --since` (REQ-117, P06),
|
||||
`orca doctor mTLS` (REQ-118, P15.5).
|
||||
|
||||
**Already in v0.11 ROADMAP**: `orca node drain` (P05), `orca job lint`
|
||||
(P11), `orca job verify` (P12), `orca restore` (P07), `orca backup`
|
||||
(P04).
|
||||
|
||||
### Unverified performance claims in doc 3
|
||||
|
||||
Doc 3's "10s applier timer = 10,000x slower than K8s informers" and
|
||||
"60s drift polling" are **forward-looking design constraints**, not
|
||||
current-state limitations — no applier timer or drift-polling loop
|
||||
exists in the codebase. These are answered by R-018/R-019/R-020
|
||||
(doc 4 + doc 5): the drift detector is a backstop, not the primary
|
||||
detector, and critical paths get ~10s latency via systemd Path units.
|
||||
|
||||
## Persona assessment
|
||||
|
||||
v0.11 touches these territories:
|
||||
|
||||
| Territory | Persona | Phases |
|
||||
|-----------|---------|--------|
|
||||
| `internal/cli/**`, `internal/drift/**`, `internal/nft/**` | backend-engineer | P10, P15.5, P05, P06, P14a, P14b |
|
||||
| `internal/emitter/**`, `internal/sshpush/**` | backend-engineer + lead-developer | P09, P10, P15.5 |
|
||||
| `internal/store/**`, migrations | data-engineer | P14a (data migration) |
|
||||
| `scripts/orca-*.sh` | backend-engineer (bash tooling) | P09, P10 |
|
||||
| `docs/**`, `README.md`, `examples/**` | lead-developer + docs-engineer (phase-specific) | P15, P08 |
|
||||
| Threat model, security review, mTLS, secrets | security-engineer | P03, P15.5 |
|
||||
| nftables, Traefik binding, cluster mesh | network-engineer | P15.5, P09 |
|
||||
| Test coverage, integration harness | devops-engineer (phase-specific) | P08 |
|
||||
|
||||
No frontend-engineer work (no UI). The data-engineer persona is
|
||||
reactivated for P14a (v0.8→v1.0 data migration). A docs-engineer custom
|
||||
persona is created for P15 (README) and P08 (integration test docs).
|
||||
See `PERSONAS.md` for the updated roster.
|
||||
@@ -0,0 +1,231 @@
|
||||
# Research: v0.12 Security Hardening (Zero-Trust Identity)
|
||||
|
||||
## Source material
|
||||
|
||||
The v0.12 threat model was produced by a comprehensive security-surface
|
||||
review (Phase 0 RESEARCH, 2026-08-07) covering the entire Orca codebase
|
||||
AND the operating-system-level surface it touches. The review ingested:
|
||||
|
||||
- v0.11 closeout (CHECKPOINT.json: milestone_complete=true, 24 phases
|
||||
shipped, threat model produced in P15.5).
|
||||
- The 12-area security-surface inventory (see "Threat model findings"
|
||||
below), produced by deep code exploration of every `internal/` package,
|
||||
every `scripts/` file, the emitter surface, the OS-touching CLI
|
||||
commands, and the dual-write window.
|
||||
- The operator's locked decisions (D-238..D-247) on zero-trust identity:
|
||||
bundled Dex + WebAuthn, master key seal-to-OIDC + Shamir, no Orca
|
||||
credentials (R-021).
|
||||
|
||||
## Load-bearing rule adopted
|
||||
|
||||
**R-021**: *Orca never issues, stores, or accepts human-identity
|
||||
credentials. Human identity is exclusively external (OIDC). Machine
|
||||
identity is exclusively mTLS/SPIFFE. No passwords, no Orca-issued
|
||||
tokens, no CA-key passphrases.*
|
||||
|
||||
## Threat model findings (F1..F25)
|
||||
|
||||
| # | Area | Finding | Severity | Phase | REQ |
|
||||
|---|------|---------|----------|-------|-----|
|
||||
| F1 | ACL | `acl.ACL.Check` exists but no caller enforces it -- daemon & SSH-push have zero authz | Critical | P06 | REQ-145 |
|
||||
| F2 | Audit | Audit log is plain SQLite INSERT -- no hash chain, no MAC, not tamper-evident | Critical | P10 | REQ-125 |
|
||||
| F3 | Runtime | `podman.go:57` & `wasm.go:39` interpolate cmdStr unquoted into SSH exec -> command injection | Critical | P01 | REQ-119 |
|
||||
| F4 | Namespace | `ns create` doesn't reject `..`/`/` -> path traversal | Critical | P02 | REQ-120 |
|
||||
| F5 | Txn | `apply.sh` python heredoc writes to arbitrary paths from desired-state.json -- no allowlist | Critical | P03 | REQ-121 |
|
||||
| F6 | Daemon | Plaintext mode (default) has no auth on read endpoints; `--pprof` unauthenticated | High | P09 | REQ-123/124 |
|
||||
| F7 | Backup | `Restore` creates symlinks without validating Linkname -> symlink-to-/etc/shadow | High | P12 | REQ-127 |
|
||||
| F8 | SQLite | DBs unencrypted, no explicit file mode (defaults to umask 0644) | High | P21 | REQ-136 |
|
||||
| F9 | SPIFFE | `VerifySVID` checks URI SAN but not the cert chain against the CA | High | P11 | REQ-126 |
|
||||
| F10 | step-ca | `step ca certificate` writes SVID privkey to /tmp/orca-* world-readable | High | P13 | REQ-128 |
|
||||
| F11 | Scripts | `orca-aggregate.sh:64` interpolates raw peer output into JSON -> JSON injection | High | P16 | REQ-131 |
|
||||
| F12 | Secrets | No master.key rotation; no passphrase/KDF wrapping (raw 32 bytes, 0600-only) | High | P14 | REQ-129 |
|
||||
| F13 | File modes | `EnforceFileModes` only checks ca.{crt,key} -- SSH key, master key, server cert not re-verified | Medium | P15 | REQ-130 |
|
||||
| F14 | install.sh | curl|bash with no checksum/signature verification of the tarball | High | P17 | REQ-132 |
|
||||
| F15 | known_hosts | `Flock` creates 0600 if missing but doesn't tighten pre-existing looser perms | Medium | P24 | REQ-139 |
|
||||
| F16 | Dual-write | Legacy CA/mTLS/daemon marked Deprecated but still load-bearing -- expanded attack surface | Medium | P23 | REQ-138 |
|
||||
| F17 | History | Real GITEA_TOKEN committed in 0cba1aa, still in git history | High (human-gated) | P28 (gate) | -- |
|
||||
| F18 | Drift | `orca-pull.sh` R-020 grep-based JSON parsing fragile; drift events unauthenticated | Medium | P16/P25 | REQ-131/140 |
|
||||
| F19 | Migration | `ALTER TABLE DROP COLUMN` irreversible; `copyFile` non-atomic; no rollback | Medium | P22 | REQ-137 |
|
||||
| F20 | OS scripts | `orca-aggregate.sh`/`orca-remediate.sh` run as root with TOFU SSH (accept-new) | Medium | P16/P24 | REQ-131/139 |
|
||||
| F21 | nftables | Emitted ruleset has SYN-flood + rate-limit but no conntrack bounds, no input default-deny | Medium | P18 | REQ-133 |
|
||||
| F22 | sudoers | `OrcaOperator` sudoers has NOEXEC on pct/qm but allows apt-get/dpkg without NOEXEC | Medium | P19 | REQ-134 |
|
||||
| F23 | system user | Proxmox creates login user (-m -s /bin/bash); peer-setup creates nologin -- inconsistent privilege | Medium | P20 | REQ-135 |
|
||||
| F24 | Dispatch | No request body size limits (json.Decode with no MaxBytesReader) | Low | P09 | REQ-124 |
|
||||
| F25 | Transport | `classifyDialErr` is substring-based; no SSH-exec rate limiting | Low | P24 | REQ-139 |
|
||||
|
||||
## Zero-trust identity model (NEW in v0.12)
|
||||
|
||||
### Two identity layers, zero overlap
|
||||
|
||||
- **Human operators** -> OIDC (external IdP, BYO) OR the bundled Dex
|
||||
with a WebAuthn (passkeys) connector as the default password-free
|
||||
authenticator. `orca auth login` / `orca auth register` open the
|
||||
default browser to the Dex WebAuthn endpoint via OIDC
|
||||
authorization-code + PKCE + local loopback redirect. After the
|
||||
WebAuthn ceremony (biometric/security key), Dex redirects back with
|
||||
an auth code; CLI exchanges for a short-lived ID token (1h) +
|
||||
refresh. Headless/CI fallback: device-code flow.
|
||||
- **Machine-to-machine** -> mTLS + SPIFFE SVIDs (unchanged from v0.11).
|
||||
|
||||
### Why WebAuthn satisfies "no passwords anywhere"
|
||||
|
||||
Passkeys are **public-key credentials**. The private key is generated
|
||||
on the authenticator (TPM/security key/phone Secure Enclave) and never
|
||||
leaves it. The server (Dex) stores only the **public key** + credential
|
||||
ID + sign count. There is no password, no shared secret, no replayable
|
||||
credential. This is the strongest authentication primitive available
|
||||
and directly satisfies R-021.
|
||||
|
||||
### Bundled Dex architecture
|
||||
|
||||
- **Dex** (github.com/dexidp/dex) is the OIDC frontend. Orca bundles a
|
||||
Dex binary + config template, deployed via `orca auth init-idp` as a
|
||||
systemd unit on the lead, fronted by Traefik (R-017, step-ca cert).
|
||||
- **`orca-webauthn-connector`** is a custom Dex connector (~300 LoC Go,
|
||||
using `github.com/go-webauthn/webauthn`). It serves:
|
||||
- `GET /orca/webauthn/register` -- registration HTML/JS page.
|
||||
- `POST /orca/webauthn/register/begin` -- WebAuthn registration
|
||||
challenge (random nonce, user info).
|
||||
- `POST /orca/webauthn/register/finish` -- attestation verification,
|
||||
credential storage.
|
||||
- `GET /orca/webauthn/login` -- login HTML/JS page.
|
||||
- `POST /orca/webauthn/login/begin` -- assertion challenge.
|
||||
- `POST /orca/webauthn/login/finish` -- assertion verification, OIDC
|
||||
`sub` extraction, redirect with auth code.
|
||||
- **Passkey storage**: SQLite at `ClusterDir()/webauthn-credentials.db`
|
||||
(0600). Schema: `credentials(user_id TEXT PRIMARY KEY, credential_id
|
||||
BLOB, public_key BLOB, sign_count INTEGER, aaguid TEXT, created_at
|
||||
TEXT)`. Public keys only; no private keys, no secrets.
|
||||
- **BYO external IdP override**: `oidc.issuer` in config repoints to
|
||||
an external IdP. The bundled Dex + WebAuthn connector is bypassed;
|
||||
the external IdP's authenticators (including its own WebAuthn) are
|
||||
used. Orca never sees the upstream credentials.
|
||||
|
||||
### RQ-1 resolution (RESEARCH binding question)
|
||||
|
||||
**RQ-1**: How does the bundled Dex bootstrap an upstream identity
|
||||
without any password, given the mTLS-only constraint?
|
||||
|
||||
**Answer (resolved by C3/D-240)**: The bundled Dex's upstream
|
||||
authenticator IS the WebAuthn connector. No external password source
|
||||
is needed for the bundled path. The WebAuthn connector serves the
|
||||
registration + login ceremonies directly; Dex maps the credential ID
|
||||
to an OIDC `sub`. BYO-IdP covers password-based upstreams (LDAP/AD)
|
||||
if an operator insists -- but those never flow through Orca.
|
||||
|
||||
**C-37 fallback** (kept if WebAuthn proves infeasible): bundled Dex
|
||||
ships mTLS-client-cert-only (Traefik `X-Forwarded-Client-Cert` header
|
||||
-> Dex `typed-external-connector`). Password-based upstreams require
|
||||
BYO external IdP. The "no Orca credentials" invariant holds regardless.
|
||||
|
||||
### Master key sealing architecture
|
||||
|
||||
- **Seal**: at `orca cluster seal`, the in-memory master key is
|
||||
encrypted with a key derived from the operator's OIDC ID token
|
||||
(HKDF-SHA256 of the token's `sub` + a fresh 32-byte salt). The
|
||||
sealed blob (`salt || ciphertext`) is stored at
|
||||
`ClusterDir()/master.key.sealed` (0600). The raw key is zeroed from
|
||||
memory. Shamir 3-of-5 shards are printed for offline recovery.
|
||||
- **Unseal**: at `orca cluster unseal`, the operator authenticates via
|
||||
OIDC (WebAuthn ceremony). The resulting ID token's `sub` + the
|
||||
stored salt derive the unwrapping key. The master key is unwrapped
|
||||
into memory and held for the cluster's lifetime. Zeroed on shutdown.
|
||||
- **Recovery**: if the IdP is lost, the operator presents 3 of 5
|
||||
Shamir shards to `orca cluster unseal --recovery`. The shards
|
||||
reconstruct the seal key; the master key is unwrapped. No backdoor.
|
||||
- **mTLS-only offline path**: for the single-operator fully-offline
|
||||
case (no OIDC), the seal key is derived from the cluster's own CA.
|
||||
The operator holds the CA (a cert, not a password). Shamir recovery
|
||||
applies to the OIDC-sealed mode only.
|
||||
|
||||
### Offline-first reconciliation (R-003)
|
||||
|
||||
The OIDC provider must be reachable to unseal the master key and to
|
||||
authenticate operators. For offline/air-gapped clusters, the operator
|
||||
runs the **bundled Dex on the lead** (offline). For the
|
||||
single-operator fully-offline case, the operator can skip OIDC and
|
||||
rely on mTLS-only machine identity (no human authn needed -- the
|
||||
operator holds the pre-staged SSH key + mTLS cert; no password, no
|
||||
token). Orca stays minimal (no bundled IdP beyond Dex); it validates
|
||||
tokens against whatever issuer the operator configures.
|
||||
|
||||
## Dependency posture (new in v0.12)
|
||||
|
||||
v0.12 adds these dependencies (all CGO-free, audited):
|
||||
|
||||
- `github.com/coreos/go-oidc/v3` -- OIDC client (token verification,
|
||||
JWKS, ID token parsing). Pure Go.
|
||||
- `github.com/go-webauthn/webauthn` -- WebAuthn library (registration,
|
||||
login, attestation/assertion verification). Pure Go.
|
||||
- `github.com/dexidp/dex` -- bundled Dex binary (vendored, not a Go
|
||||
import; deployed as a separate systemd unit). Apache-2.0.
|
||||
- `golang.org/x/crypto/ssh/...` -- already a dependency (sshpush).
|
||||
|
||||
No CGO. No gRPC. No ConnectRPC. No YAML parser. The "stdlib + minimal
|
||||
deps" posture (D-008) is preserved.
|
||||
|
||||
## Codebase reality (verified 2026-08-07)
|
||||
|
||||
- `internal/acl/acl.go` -- ACL exists but is unenforced (F1). P06
|
||||
rewrites it (remove KindToken, add KindOidc, wire enforcement).
|
||||
- `internal/runtime/podman.go:57`, `internal/runtime/wasm.go:39` --
|
||||
unquoted cmdStr interpolation (F3). P01 fixes via shellQuote.
|
||||
- `internal/cli/ns.go:nsCreateCmd` -- no `..`/`/` rejection (F4). P02
|
||||
adds `validateNamespaceName`.
|
||||
- `internal/txn/txn.go:renderApplyScript` -- arbitrary path writes
|
||||
(F5). P03 adds prefix allowlist.
|
||||
- `internal/security/ca.go` -- legacy CA, deprecated but load-bearing
|
||||
(F16). P23 deletes it (gated on P06/P08/P09/P11).
|
||||
- `internal/secrets/secrets.go` -- master key raw file, no rotation
|
||||
(F12). P08 seals it to OIDC; P14 adds rotation.
|
||||
- `internal/audit/audit.go` -- plain SQLite INSERT (F2). P10 adds
|
||||
hash-chain + HMAC.
|
||||
- `internal/emitter/nft.go` -- no conntrack/default-deny (F21). P18
|
||||
hardens the ruleset.
|
||||
- `internal/proxmox/bootstrap.go:29` -- `--password` bootstrap (F23,
|
||||
R-021 violation). P07 removes it.
|
||||
- `internal/identity/spiffe.go:95` -- no chain validation (F9). P11
|
||||
fixes.
|
||||
- `scripts/install.sh` -- no checksum verification (F14). P17 adds
|
||||
SHA256SUMS + GPG signature.
|
||||
- `scripts/orca-aggregate.sh:64` -- raw JSON interpolation (F11). P16
|
||||
replaces with jq/Go.
|
||||
|
||||
## Alignment with existing gates
|
||||
|
||||
- **C-19** (threat model) -- v0.11 P15.5 produced the initial threat
|
||||
model; v0.12 is the comprehensive expansion (full OS surface).
|
||||
- **C-08** (SPIFFE spike) -- passed; v0.12 P11 hardens the verification
|
||||
path.
|
||||
- **R-001..R-020** -- unchanged; R-021 is an extension, not a reversal.
|
||||
- **D-008** (no CGO) -- preserved; all new deps are pure Go.
|
||||
|
||||
## Risks (for GRILL to pressure-test)
|
||||
|
||||
- **P07 (password removal) is breaking** -- mitigation: C-34 migration
|
||||
gate (`--accept-identity-migration`).
|
||||
- **P08 (master key seal) is the riskiest phase** -- a bug corrupts all
|
||||
secrets at rest. Mitigation: `--dry-run`, atomic re-encryption,
|
||||
automatic rollback to old sealed key on any failure.
|
||||
- **P21 (SQLite encryption) may need CGO** -- C-31 fallback to
|
||||
file-mode 0600 + documented threat if SQLCipher needs CGO. No CGO.
|
||||
- **P23 (dual-write closure) is high-impact** -- removing the legacy
|
||||
CA breaks `orca init`/`orca cert` if step-ca isn't fully wired.
|
||||
Mitigation: gate on P06/P08/P09/P11, full test coverage before
|
||||
deletion.
|
||||
- **P05 (WebAuthn connector) is new ground** -- ~300 LoC custom Dex
|
||||
connector. Mitigation: C-37 fallback (mTLS-client-cert-only) if
|
||||
WebAuthn proves infeasible; virtual-authenticator integration tests
|
||||
(P26) using `go-webauthn` test helpers.
|
||||
- **Bundled Dex is a new systemd unit + Traefik route** -- operational
|
||||
surface growth. Mitigation: `orca doctor oidc` checks Dex health,
|
||||
JWKS reachability, WebAuthn endpoint TLS.
|
||||
- **C-32 human gate** (leaked GITEA_TOKEN) could stall the final ship.
|
||||
Escalation path: ship as `v0.11.29-rc1` if rotation pending,
|
||||
`v0.11.29` when confirmed.
|
||||
|
||||
## Next steps
|
||||
|
||||
Phase 0 proceeds to IDEATE (produce the 30 net-new requirements
|
||||
REQ-119..REQ-148), then PLAN (29 phases, wave ordering, persona
|
||||
assignments), then GRILL (ratify C-29..C-38).
|
||||
+276
-62
@@ -185,7 +185,7 @@ The vision ("minimalist, offline-first, CLI-first orchestration
|
||||
engine") is unchanged. v0.8 closes the coverage debt left by v0.7's
|
||||
50% floor and the trust-surface gaps explicitly deferred in v0.6.
|
||||
|
||||
## Milestone v0.9: Re-architecture Foundation & Workloads
|
||||
## Milestone v0.9: Re-architecture Foundation & Workloads — **COMPLETE**
|
||||
|
||||
**Scope**: This milestone SUPERSPEDES the shipped v0.1–v0.8 architecture per
|
||||
the adopted PRD (`.ciagent/PRD_v0.9.md`). The re-architecture is justified on
|
||||
@@ -204,30 +204,27 @@ from `GRILL_v0.9.md` are adopted as execution gates. 30 net-new requirements
|
||||
chore/docs).
|
||||
|
||||
- [ ] Phase 0: Pre-execution (specify → clarify → research → ideate → plan → grill) — tag `v0.8.0` (shipped; this is the phase you are reading)
|
||||
- [ ] Phase P00: Deprecation sweep + migration-ordering decision + txn-design spike + hermetic test-infra bootstrap + persona reactivation + doc banners (REQ-072, REQ-085, REQ-088, REQ-089, REQ-090; gates C-03 ✅, C-05, C-06, C-15..C-18) — tag `v0.8.1`
|
||||
- [ ] Phase P0a1: Multi-namespace path resolver + config HCL demotion + known_hosts flock (REQ-063, REQ-069, REQ-070, REQ-071; gate C-07) — tag `v0.8.2`
|
||||
- [ ] Phase P0a2: Namespace CRUD + inheritance engine (REQ-082) — tag `v0.8.3`
|
||||
- [ ] Phase P0b: Markdown jobspec parser + dispatcher + fuzz (REQ-064, REQ-067) — tag `v0.8.4`
|
||||
- [ ] Phase P0c: Job/Service/DaemonSet schemas + emitter interface (REQ-074) — tag `v0.8.5`
|
||||
- [ ] Phase P01: SSH-push transport + host-path volumes (REQ-073) — tag `v0.8.6`
|
||||
- [ ] Phase P02: Service block + checks + restart + Traefik emitter (REQ-077; gate C-10) — tag `v0.8.7`
|
||||
- [ ] Phase P03: Update stanza (rolling/canary) — tag `v0.8.8`
|
||||
- [ ] Phase P04: Lifecycle hooks (systemd ExecStop) — tag `v0.8.9`
|
||||
- [ ] Phase P05: Constraints & affinity (CEL) + CLI-side scheduler (REQ-083) — tag `v0.8.10`
|
||||
- [ ] Phase P06: Task groups (multi-process services) — tag `v0.8.11`
|
||||
- [ ] Phase P07a: Process + podman runtimes (REQ-078) — tag `v0.8.12`
|
||||
- [ ] Phase P07b: wasmtime runtime (REQ-078; **gate C-01** — CGO eval) — tag `v0.8.13`
|
||||
- [ ] Phase P07c: pve-vm + pve-ct runtimes (REQ-078; extends REQ-076) — tag `v0.8.14`
|
||||
- [ ] Phase P08: Socket plumbing (R-007) — tag `v0.8.15`
|
||||
- [ ] Phase P09: Storage replication via Syncthing (REQ-081; **gates C-02, C-14**) — tag `v0.8.16`
|
||||
- [ ] Phase P10: Lead rules + migration (REQ-076 step-ca integration) — tag `v0.8.17`
|
||||
- [ ] Phase P0X: Ship + audit (REQ-062 coverage gate; REQ-068 deprecation warnings) — tag `v0.8.18`
|
||||
- [x] Phase P00: Deprecation sweep + bash tooling gate + render contract + doc banners (REQ-068,072,088,089,090; gates C-03,C-05,C-06,C-15..C-18) — tag `v0.8.1` ✓
|
||||
- [x] Phase P0a1: Multi-namespace path resolver + config demotion + known_hosts flock (REQ-063,069,070,071; gate C-07) — tag `v0.8.2` ✓
|
||||
- [x] Phase P0a2: Namespace CRUD + inheritance engine (REQ-082) — tag `v0.8.3` ✓
|
||||
- [x] Phase P0b: Markdown jobspec parser + dispatcher + fuzz (REQ-064,067) — tag `v0.8.4` ✓
|
||||
- [x] Phase P0c: Job/Service/DaemonSet schemas + emitter interface (REQ-074) — tag `v0.8.5` ✓
|
||||
- [x] Phase P01: SSH-push transport (REQ-073) — tag `v0.8.6` ✓
|
||||
- [x] Phase P02: Service block + Traefik emitter (REQ-077; gate C-10) — tag `v0.8.7` ✓
|
||||
- [x] Phase P03/P04/P08: Update stanza + lifecycle hooks + socket plumbing (combined) — tag `v0.8.8` ✓
|
||||
- [x] Phase P05: CLI-side scheduler + CEL constraints (REQ-083) — tag `v0.8.9` ✓
|
||||
- [x] Phase P06: Task groups (multi-process services) — tag `v0.8.10` ✓
|
||||
- [x] Phase P07a/b/c: Runtime abstraction — 5 backends (REQ-078; gate C-01) — tag `v0.8.11` ✓
|
||||
- [x] Phase P09: Syncthing storage replication (REQ-081; gates C-02,C-14) — tag `v0.8.12` ✓
|
||||
- [x] Phase P10: Lead rules + step-ca (REQ-076) — tag `v0.8.13` ✓
|
||||
- [x] Phase P0X: Ship + audit (REQ-062,068) — tag `v0.8.14` ✓
|
||||
|
||||
**Milestone tag**: `v0.8.18` (final phase patch = milestone release per
|
||||
feature-milestone progressive-patch rule). Per-phase tags: `v0.8.1`…`v0.8.18`.
|
||||
Tags run on the previous minor's patch line (v0.8.x) per branch-strategy.md.
|
||||
The milestone branch label uses the milestone number
|
||||
(`milestone/v0.9-rearchitecture`); no separate minor tag.
|
||||
**Milestone tag**: `v0.8.15` (final phase patch = milestone release per
|
||||
feature-milestone progressive-patch rule). Per-phase tags: `v0.8.1`…`v0.8.14`.
|
||||
P03/P04/P08 were combined into one phase; P07a/b/c were combined into one
|
||||
phase. Actual execution: 14 tagged phases. Tags run on the previous minor's
|
||||
patch line (v0.8.x) per branch-strategy.md. The milestone branch label uses
|
||||
the milestone number (`milestone/v0.9-rearchitecture`); no separate minor tag.
|
||||
|
||||
### Per-phase REQ coverage (v0.9)
|
||||
|
||||
@@ -252,67 +249,135 @@ HCL-canonical, single-namespace, no-container-runtime, no-SPIFFE). The
|
||||
reversals are justified by the six-part evidence basis recorded in the
|
||||
PROJECT.md Supersession Table.
|
||||
|
||||
## Milestone v1.0: Production Hardening
|
||||
## Milestone v0.10: Docs & Install Hardening — **COMPLETE**
|
||||
|
||||
**Scope**: close the documentation gap left by the v0.9 re-architecture
|
||||
and fix the release/install pipeline bug that caused `install.sh` to
|
||||
resolve to v0.4.5 instead of the latest release. The v0.9
|
||||
re-architecture shipped a complete CLI surface (markdown jobspec,
|
||||
`orca ns`, `orca node capacity`, CLI-side scheduler, emitters, Traefik
|
||||
ingress) but no operator-facing reference documentation. This milestone
|
||||
ships that documentation plus a worked full-stack example with ingress
|
||||
configured, and hardens the release pipeline so every Gitea release
|
||||
carries a Linux binary asset.
|
||||
|
||||
**Milestone type**: feature (P1 ships `fix` phases; P2/P3/P4 ship `docs`
|
||||
phases; at least one non-docs phase makes this a feature milestone per
|
||||
the versioning logic).
|
||||
|
||||
- [x] Phase 0: Pre-execution (specify → clarify → research → ideate → plan → grill) — tag `v0.9.0`
|
||||
- [x] Phase P1: release.sh + install.sh fix (REQ-097, REQ-098) — tag `v0.9.1`
|
||||
- [x] Phase P2: docs/cli.md + docs/jobspec.md + docs/ingress.md (REQ-091, REQ-092, REQ-093) — tag `v0.9.2`
|
||||
- [x] Phase P3: examples/full-stack/ (REQ-094) — tag `v0.9.3`
|
||||
- [x] Phase P4: README.md + docs/namespace.md refresh (REQ-095, REQ-096) — tag `v0.9.4`
|
||||
- [x] Phase P5: Final review + ship + audit (milestone release) — tag `v0.9.5` = v0.10.0 milestone release
|
||||
|
||||
**Milestone tag**: `v0.9.5` (final phase patch = milestone release per
|
||||
feature-milestone progressive-patch rule). Per-phase tags: `v0.9.0`…`v0.9.5`.
|
||||
Tags run on the previous minor's patch line (v0.9.x) per
|
||||
branch-strategy.md. The milestone branch label uses the milestone
|
||||
number (`milestone/v0.10-docs-cli-examples`); no separate minor tag.
|
||||
|
||||
### Per-phase REQ coverage (v0.10 docs milestone)
|
||||
|
||||
- **P1** — release.sh cross-build + asset verification (REQ-097); install.sh fallback walk (REQ-098)
|
||||
- **P2** — CLI reference (REQ-091); jobspec reference (REQ-092); ingress guide (REQ-093)
|
||||
- **P3** — full-stack examples (REQ-094)
|
||||
- **P4** — README refresh (REQ-095); namespace.md v0.9 layout (REQ-096)
|
||||
|
||||
### Root cause of the v0.4.5 install (documented in RESEARCH_v0.10.md)
|
||||
|
||||
The v0.8.x releases (v0.8.0–v0.8.15) shipped with zero binary assets
|
||||
attached to their Gitea releases. `install.sh` resolves "latest" →
|
||||
v0.8.15, looks for `orca-v0.8.15-linux-amd64.tar.gz`, finds nothing, and
|
||||
errors out. The v0.4.5 install came from an earlier run or a pinned
|
||||
`--version`. The fix is forward: release.sh cross-builds amd64 and
|
||||
verifies the asset post-create; install.sh walks backward through
|
||||
releases if the latest lacks the asset.
|
||||
|
||||
## Milestone v0.11: Production Hardening — **COMPLETE**
|
||||
|
||||
**Scope**: ship a cluster that operators can run. Builds on the v0.9
|
||||
re-architecture foundation with the production-grade subsystems:
|
||||
secrets, transactions, ACL/SPIFFE, backup/restore, drain, recovery, and
|
||||
the v0.8→v1.0 migration.
|
||||
the v0.8→v1.0 migration. **Phase 0 adopts 4 new load-bearing rules
|
||||
(R-017…R-020) and 23 new decisions (D-215…D-237) from 5 research docs
|
||||
covering ingress hardening, drift detection, platform-engineer
|
||||
positioning, strategic framing, and the systemd Path unit
|
||||
implementation.** No new phases added; scope is folded into existing
|
||||
phases per operator decisions Q2=C (add 5 CLI commands), Q3=A (fold
|
||||
ingress into P15.5).
|
||||
|
||||
**Milestone type**: feature (multiple `feat` phases).
|
||||
|
||||
- [ ] Phase 0: Pre-execution (specify → clarify → research → plan → grill) — tag `v0.9.0`
|
||||
- [ ] Phase P00: CLI cache layer (REQ-062 cache floor; R-008) — tag `v0.9.1`
|
||||
- [ ] Phase P01: Metrics endpoint (hand-rolled text exposition) — tag `v0.9.2`
|
||||
- [ ] Phase P01.5: SPIFFE SVID minting spike (REQ-076; **gate C-08** — if spike fails, fall back to mTLS identity) — tag `v0.9.3`
|
||||
- [ ] Phase P02: ACL (SPIFFE + token identities) — tag `v0.9.4`
|
||||
- [ ] Phase P03: Secrets subsystem (REQ-080; **gate C-19** threat model) — tag `v0.9.5`
|
||||
- [ ] Phase P04: Backup/restore (tar + signed) — tag `v0.9.6`
|
||||
- [ ] Phase P05: Drain + daemon drain-and-stop (REQ-061) — tag `v0.9.7`
|
||||
- [ ] Phase P06: Alloc history (CLI-side SQLite retention; REQ-071 cache DB) — tag `v0.9.8`
|
||||
- [ ] Phase P07: Recovery (`orca restore`) — tag `v0.9.9`
|
||||
- [ ] Phase P08: Integration tests — expand hermetic harness (REQ-087) — tag `v0.9.10`
|
||||
- [ ] Phase P09: Collector + aggregator (opt-in; **gates C-11, C-12, C-14**) — tag `v0.9.11`
|
||||
- [ ] Phase P10: Transactional plane (REQ-075, REQ-079; **gate C-09** orca-pull.sh failure contract) — tag `v0.9.12`
|
||||
- [ ] Phase P11: `orca job lint` (REQ-084) — tag `v0.9.13`
|
||||
- [ ] Phase P12: `orca job verify` (dry-run txn through lead) — tag `v0.9.14`
|
||||
- [ ] Phase P13: `orca ns` subcommands (full surface) + deprecation warnings (REQ-068) — tag `v0.9.15`
|
||||
- [ ] Phase P14a: v0.8→v1.0 data migration (REQ-066; **gate C-07** CA migration spec) — tag `v0.9.16`
|
||||
- [ ] Phase P14b: Daemon cutover + running-allocation adoption — tag `v0.9.17`
|
||||
- [ ] Phase P14c: Mixed-version tolerance + no-orca-on-server enforcement (REQ-065, REQ-086; implements C-13) — tag `v0.9.18`
|
||||
- [ ] Phase P15: README quickstart (REQ-089) — tag `v0.9.19`
|
||||
- [ ] Phase P15.5: Threat model + security review (**gate C-19**) — tag `v0.9.20`
|
||||
- [ ] Phase P16: Final review + ship + audit — **v1.0.0 release** — tag `v0.9.21`
|
||||
- [x] Phase 0: Pre-execution (specify → clarify → research → plan → grill) — tag `v0.10.0`
|
||||
- [x] Phase P00: CLI cache layer (REQ-062 cache floor; R-008) — tag `v0.10.1`
|
||||
- [x] Phase P01: Metrics endpoint (hand-rolled text exposition) — tag `v0.10.2`
|
||||
- [x] Phase P01.5: SPIFFE SVID minting spike (REQ-076; **gate C-08** — if spike fails, fall back to mTLS identity) — tag `v0.10.3`
|
||||
- [x] Phase P02: ACL (SPIFFE + token identities) — tag `v0.10.4`
|
||||
- [x] Phase P03: Secrets subsystem (REQ-080; **gate C-19** threat model) — tag `v0.10.5`
|
||||
- [x] Phase P04: Backup/restore (tar + signed) — tag `v0.10.6`
|
||||
- [x] Phase P05: Drain + daemon drain-and-stop (REQ-061) + **`orca job migrate` (REQ-116)** — tag `v0.10.7`
|
||||
- [x] Phase P06: Alloc history (CLI-side SQLite retention; REQ-071 cache DB) + **`orca logs --all-nodes --since` (REQ-117)** — tag `v0.10.8`
|
||||
- [x] Phase P07: Recovery (`orca restore`) — tag `v0.10.9`
|
||||
- [x] Phase P08: Integration tests — expand hermetic harness (REQ-087) + **drift-detection integration tests (auto-remediation, NFS fallback, cooldown, secret exclusion)** — tag `v0.10.10`
|
||||
- [x] Phase P09: Collector + aggregator (opt-in; **gates C-11, C-12, C-14**) + **drift-event aggregation extension (REQ-107, D-237)** — tag `v0.10.11`
|
||||
- [x] Phase P10a: Transactional plane (REQ-075, REQ-079; **gate C-09**; **gate C-23** cluster-wide vs ns-scoped txn distinction) — tag `v0.10.12`
|
||||
- [x] Phase P10b: Drift detection (R-018/R-019/R-020; REQ-103..REQ-113; `orca drift` CLI, systemd Path unit emitter, `orca-drift-notify.sh`, `orca-remediate.sh`, cadence config, `--force`+per-ns gate, `orca` system user, NFS detection) — depends on P10a — tag `v0.10.13`
|
||||
- [x] Phase P11: `orca job lint` (REQ-084) — tag `v0.10.14`
|
||||
- [x] Phase P12: `orca job verify` (dry-run txn through lead) — tag `v0.10.15`
|
||||
- [x] Phase P13: `orca ns` subcommands (full surface) + deprecation warnings (REQ-068) — tag `v0.10.16`
|
||||
- [x] Phase P14a: v0.8→v1.0 data migration (REQ-066; **gate C-07**; **gate C-25** post-cutover verification + rollback; **gate C-27** orca user creation) + **`orca upgrade --to-vX` (REQ-115, thin wrapper, handles R-017 binding cutover)** — tag `v0.10.17`
|
||||
- [x] Phase P14b: Daemon cutover + running-allocation adoption + **`orca cluster rotate-lead` (REQ-114)** — tag `v0.10.18`
|
||||
- [x] Phase P14c: Mixed-version tolerance + no-orca-on-server enforcement (REQ-065, REQ-086; implements C-13) — tag `v0.10.19`
|
||||
- [x] Phase P15: README quickstart (REQ-089; **Nomad-inspired framing per Q5=A, honest-trade-offs table from research doc 3**) — tag `v0.10.20`
|
||||
- [x] Phase P15.5: Threat model + security review (**gate C-19**; **gate C-28** two sub-waves) + **ingress hybrid (R-017; nft emitter REQ-099, Traefik binding REQ-100, `orca doctor nft` REQ-101, `orca nft` CLI REQ-102) + `orca doctor mTLS` (REQ-118)** — tag `v0.10.21`
|
||||
- [x] Phase P16: Final review + ship + audit — **v0.11.0 milestone release** — tag `v0.10.22` (v1.0.0 cut separately after UAT sign-off)
|
||||
|
||||
**Milestone tag**: `v1.0.0` (the v1.0.0 release tag is the production-ready cut;
|
||||
per-phase patches run on the v0.9.x line per branch-strategy.md). Per-phase
|
||||
tags: `v0.9.0`…`v0.9.21`.
|
||||
**Milestone tag**: `v0.11.0` (the v0.11 milestone release tag; v1.0.0 is
|
||||
UAT-gated and cut separately after v0.11 completion per operator decision —
|
||||
the v1.0.0 tag marks production-ready sign-off, not a separate milestone).
|
||||
Per-phase patches run on the v0.10.x line per branch-strategy.md. Per-phase
|
||||
tags: `v0.10.0`…`v0.10.21`.
|
||||
|
||||
### Per-phase REQ coverage (v1.0)
|
||||
### Per-phase REQ coverage (v0.11)
|
||||
|
||||
- **P00** — CLI cache (R-008)
|
||||
- **P01.5** — SPIFFE spike (REQ-076; C-08)
|
||||
- **P03** — Secrets (REQ-080; C-19)
|
||||
- **P05** — Drain + daemon stop (REQ-061)
|
||||
- **P06** — Alloc history (REQ-071 cache DB)
|
||||
- **P08** — Integration tests (REQ-087)
|
||||
- **P10** — Transactional plane (REQ-075, REQ-079; C-09)
|
||||
- **P05** — Drain + daemon stop (REQ-061) + `orca job migrate` (REQ-116)
|
||||
- **P06** — Alloc history (REQ-071 cache DB) + `orca logs --all-nodes --since` (REQ-117)
|
||||
- **P08** — Integration tests (REQ-087) + drift-detection integration tests
|
||||
- **P09** — Collector + aggregator (C-11, C-12, C-14) + drift-event aggregation (REQ-107, D-237)
|
||||
- **P10a** — Transactional plane (REQ-075, REQ-079; C-09; C-23)
|
||||
- **P10b** — Drift detection (R-018/R-019/R-020; REQ-103..REQ-113)
|
||||
- **P11** — Job lint (REQ-084)
|
||||
- **P13** — ns subcommands + deprecation warnings (REQ-068)
|
||||
- **P14a/b/c** — Migration (REQ-066, REQ-065, REQ-086; C-07, C-13)
|
||||
- **P15** — README (REQ-089)
|
||||
- **P15.5** — Threat model (C-19)
|
||||
- **P14a/b/c** — Migration (REQ-066, REQ-065, REQ-086; C-07, C-13) + `orca upgrade` (REQ-115) + `orca cluster rotate-lead` (REQ-114)
|
||||
- **P15** — README (REQ-089; Q5=A framing)
|
||||
- **P15.5** — Threat model (C-19) + ingress hybrid (R-017; REQ-099..REQ-102) + `orca doctor mTLS` (REQ-118)
|
||||
|
||||
### Risk register (from grill, for ongoing monitoring)
|
||||
### New load-bearing rules adopted in Phase 0
|
||||
|
||||
- **R-017** — Ingress hybrid: nft DNAT → Traefik on `127.0.0.1:8443`; opt-out via `--public-binding`; `service { ingress: native }` per-workload opt-in
|
||||
- **R-018** — Drift cadence: default 60s; critical 5s + systemd Path units; standard 30s
|
||||
- **R-019** — Drift detector is a BACKSTOP; primary = systemd/Traefik/step-ca/Syncthing
|
||||
- **R-020** — Hard gate: applier refuses txns on pre-flight drift; `--force` + per-ns scoping override
|
||||
|
||||
### Risk register (from grill + research, for ongoing monitoring)
|
||||
|
||||
- **step-ca single-instance SPOF** (mitigation: C-12 doc; v1.x HA via systemd failover)
|
||||
- **master.key passphrase-less 0600** (mitigation: C-19 threat model; consider OS keyring in v1.x)
|
||||
- **wasmtime CGO breaks cross-compile** (mitigation: C-01 spike; fallback to podman/process primary)
|
||||
- **bash control plane drift** (mitigation: C-15..C-18 render-format contract + bats gate)
|
||||
- **daemon cutover orphans running allocs** (mitigation: P14b split; test adoption)
|
||||
- **27→35+ phase scope** (mitigation: C-04 sizing; three-milestone split if exceeded — current count v0.9=18 + v1.0=22 = 40 phases; **C-04 sizing must run before v0.9 P00 execution to determine whether to split into v0.9+v0.10+v1.0**)
|
||||
- **27→35+ phase scope** (mitigation: C-04 resolved — operator accepted 40 phases; v0.11 grows to 24 phases per grill C-24 split of P10→P10a/P10b; scope folded in, no other new phases)
|
||||
- **R-020 deadlock** (mitigation: `--force` flag + per-namespace scoping per Q4=A; drifted peer in ns-A doesn't block ns-B)
|
||||
- **P10 sizing** (mitigation: P10 is the largest phase — drift detection + txn plane; grill may split into P10a/P10b if vertical slice is too large)
|
||||
- **Ingress default migration** (mitigation: `orca upgrade` [REQ-115] handles Traefik binding cutover from `:443` to `127.0.0.1:8443` for existing v0.9/v0.10 clusters)
|
||||
- **`orca` system user on peers** (mitigation: net-new operational requirement; peer-setup emits `useradd -r orca` idempotently; documented in P10)
|
||||
|
||||
## Deferred to v1.x (out of scope for v1.0)
|
||||
## Deferred to v1.x (out of scope for v0.11)
|
||||
|
||||
- `sqlite-wal-shared` state backend (R-009 abstractions ship in v1.0; backend in v1.x)
|
||||
- `git` state backend
|
||||
@@ -334,3 +399,152 @@ tags: `v0.9.0`…`v0.9.21`.
|
||||
- Leader-elected Raft coordinator
|
||||
- External CA / Let's Encrypt / cert transparency
|
||||
- Online-only features (HSTS, OCSP stapling, telemetry)
|
||||
|
||||
## Milestone v0.12: Security Hardening (Zero-Trust Identity) — IN PROGRESS
|
||||
|
||||
**Scope**: comprehensive security hardening across the entire attack
|
||||
surface, **including the operating system itself**, plus adoption of a
|
||||
zero-trust identity model. The v0.12 threat-model review (Phase 0
|
||||
RESEARCH) surfaced 25 distinct findings (F1..F25) spanning injection,
|
||||
traversal, ACL, audit, crypto, OS scripts, emitters, sudoers, system
|
||||
users, file modes, daemon auth, backup, SQLite, install.sh, and
|
||||
migration. v0.12 closes all of them and adopts **R-021** (no Orca
|
||||
credentials) as the load-bearing architectural change: human identity is
|
||||
exclusively external (OIDC), machine identity is exclusively
|
||||
mTLS/SPIFFE, and no passwords/Orca-issued-tokens/CA-key-passphrases
|
||||
exist anywhere in the system.
|
||||
|
||||
The operator locked two architectural decisions: **(1) bundled Dex by
|
||||
default + BYO external IdP override** (D-239), and **(2) master key
|
||||
seal-to-OIDC + Shamir 3-of-5 recovery** (D-241). A third decision added
|
||||
**WebAuthn (passkeys) as the bundled password-free authenticator** for
|
||||
Dex (D-240) -- passkeys are public-key credentials (private key never
|
||||
leaves the authenticator), directly satisfying R-021.
|
||||
|
||||
**Milestone type**: feature (P04 OIDC+Dex and P05 WebAuthn ship `feat`
|
||||
phases; the rest are `fix`/`chore`/`test`/`docs`/`refactor`).
|
||||
|
||||
- [ ] Phase 0: Pre-execution (specify -> clarify -> research -> ideate -> plan -> grill) -- tag `v0.11.0`
|
||||
- [ ] Phase P01: Command injection fix (podman/wasm shellQuote) (REQ-119, F3) -- tag `v0.11.1`
|
||||
- [ ] Phase P02: Namespace path traversal fix (REQ-120, F4) -- tag `v0.11.2`
|
||||
- [ ] Phase P03: Txn apply path allowlist (REQ-121, F5) -- tag `v0.11.3`
|
||||
- [ ] Phase P04: OIDC client + bundled Dex (REQ-144; BYO-IdP override) -- tag `v0.11.4`
|
||||
- [ ] Phase P05: WebAuthn connector for Dex (REQ-148; passkeys, browser auth+register) -- tag `v0.11.5`
|
||||
- [ ] Phase P06: ACL rewrite to OIDC claims + enforcement (REQ-145, REQ-122, F1) -- tag `v0.11.6`
|
||||
- [ ] Phase P07: Remove all password/token paths (breaking; REQ-146, R-021, C-34) -- tag `v0.11.7`
|
||||
- [ ] Phase P08: Master key seal-to-OIDC + Shamir 3-of-5 (REQ-147, C-35) -- tag `v0.11.8`
|
||||
- [ ] Phase P09: Daemon auth hardening (REQ-123, REQ-124, F6, F24) -- tag `v0.11.9`
|
||||
- [ ] Phase P10: Audit log tamper-evidence (REQ-125, F2) -- tag `v0.11.10`
|
||||
- [ ] Phase P11: SVID chain validation (REQ-126, F9) -- tag `v0.11.11`
|
||||
- [ ] Phase P12: Backup symlink validation (REQ-127, F7) -- tag `v0.11.12`
|
||||
- [ ] Phase P13: step-ca /tmp hardening (REQ-128, F10) -- tag `v0.11.13`
|
||||
- [ ] Phase P14: Master key rotation (re-seal to OIDC; REQ-129, F12, C-30) -- tag `v0.11.14`
|
||||
- [ ] Phase P15: File-mode audit expansion (REQ-130, F13) -- tag `v0.11.15`
|
||||
- [ ] Phase P16: aggregate.sh JSON injection + drift-gate parse fix (REQ-131, F11, F18) -- tag `v0.11.16`
|
||||
- [ ] Phase P17: install.sh checksum+GPG verification (REQ-132, F14) -- tag `v0.11.17`
|
||||
- [ ] Phase P18: nftables ruleset hardening (REQ-133, F21) -- tag `v0.11.18`
|
||||
- [ ] Phase P19: sudoers hardening (REQ-134, F22) -- tag `v0.11.19`
|
||||
- [ ] Phase P20: System user consistency (REQ-135, F23) -- tag `v0.11.20`
|
||||
- [ ] Phase P21: SQLite file-mode + at-rest encryption (REQ-136, F8, C-31) -- tag `v0.11.21`
|
||||
- [ ] Phase P22: Migration safety + identity migration (REQ-137, F19, C-34) -- tag `v0.11.22`
|
||||
- [ ] Phase P23: Legacy CA/mTLS/daemon + step-ca password-provisioner deletion (REQ-138, F16; **gate C-29: P06/P08/P09/P11**) -- tag `v0.11.23`
|
||||
- [ ] Phase P24: known_hosts tightening + transport hardening (REQ-139, F15, F25) -- tag `v0.11.24`
|
||||
- [ ] Phase P25: Drift event authentication (REQ-140, F18) -- tag `v0.11.25`
|
||||
- [ ] Phase P26: Security integration test suite (REQ-141, C-33) -- tag `v0.11.26`
|
||||
- [ ] Phase P27: Zero-trust + OIDC + WebAuthn + threat-model docs (REQ-142) -- tag `v0.11.27`
|
||||
- [ ] Phase P28: Final review + ship + audit (milestone release) -- tag `v0.11.28` = **v0.12 milestone release**
|
||||
|
||||
**Milestone tag**: `v0.11.28` (final phase patch = milestone release per
|
||||
feature-milestone progressive-patch rule; no separate `v0.12.0` tag).
|
||||
Per-phase tags: `v0.11.0`..`v0.11.28` (29 tags). Tags run on the
|
||||
previous minor's patch line (v0.11.x) per branch-strategy.md. The
|
||||
milestone branch label uses the milestone number
|
||||
(`milestone/v0.12-security-hardening`); no separate minor tag.
|
||||
|
||||
The v1.0.0 production-ready tag stays deferred for post-v0.12 UAT
|
||||
(per v0.11 PRD; v0.12 is a minor feature milestone, not the v1.0 cut).
|
||||
|
||||
### Per-phase REQ coverage (v0.12)
|
||||
|
||||
- **P01** -- Command injection (REQ-119, F3)
|
||||
- **P02** -- Namespace path traversal (REQ-120, F4)
|
||||
- **P03** -- Txn apply path allowlist (REQ-121, F5)
|
||||
- **P04** -- OIDC client + bundled Dex (REQ-144; D-239, D-242, D-246)
|
||||
- **P05** -- WebAuthn connector (REQ-148; D-240, D-243, D-244, C-38)
|
||||
- **P06** -- ACL rewrite + enforcement (REQ-145, REQ-122, F1)
|
||||
- **P07** -- Remove password/token paths (REQ-146, R-021, C-34)
|
||||
- **P08** -- Master key seal-to-OIDC + Shamir (REQ-147, D-241, C-35)
|
||||
- **P09** -- Daemon auth (REQ-123, REQ-124, F6, F24)
|
||||
- **P10** -- Audit tamper-evidence (REQ-125, F2)
|
||||
- **P11** -- SVID chain validation (REQ-126, F9)
|
||||
- **P12** -- Backup symlink validation (REQ-127, F7)
|
||||
- **P13** -- step-ca /tmp hardening (REQ-128, F10)
|
||||
- **P14** -- Master key rotation (REQ-129, F12, C-30)
|
||||
- **P15** -- File-mode audit expansion (REQ-130, F13)
|
||||
- **P16** -- aggregate.sh JSON injection + drift-gate (REQ-131, F11, F18)
|
||||
- **P17** -- install.sh checksum+GPG (REQ-132, F14)
|
||||
- **P18** -- nftables ruleset hardening (REQ-133, F21)
|
||||
- **P19** -- sudoers hardening (REQ-134, F22)
|
||||
- **P20** -- System user consistency (REQ-135, F23)
|
||||
- **P21** -- SQLite file-mode + encryption (REQ-136, F8, C-31)
|
||||
- **P22** -- Migration safety + identity migration (REQ-137, F19, C-34)
|
||||
- **P23** -- Dual-write closure (REQ-138, F16; **gate C-29**)
|
||||
- **P24** -- known_hosts + transport hardening (REQ-139, F15, F25)
|
||||
- **P25** -- Drift event authentication (REQ-140, F18)
|
||||
- **P26** -- Security integration test suite (REQ-141, C-33)
|
||||
- **P27** -- Zero-trust + OIDC + WebAuthn + threat-model docs (REQ-142)
|
||||
- **P28** -- Final review + ship + audit (REQ-143)
|
||||
|
||||
### New load-bearing rule adopted in Phase 0
|
||||
|
||||
- **R-021** -- Orca never issues, stores, or accepts human-identity
|
||||
credentials. Human identity is exclusively external (OIDC). Machine
|
||||
identity is exclusively mTLS/SPIFFE. No passwords, no Orca-issued
|
||||
tokens, no CA-key passphrases.
|
||||
|
||||
### Binding conditions (for GRILL ratification; C-29..C-38)
|
||||
|
||||
- **C-29**: P23 (dual-write closure) gated on P06/P08/P09/P11 all shipped.
|
||||
- **C-30**: P14 (master key rotation) reversible; `--dry-run` mandatory; auto-rollback to old sealed key on any ns failure.
|
||||
- **C-31**: P21 (SQLite encryption): CGO-free fallback to file-mode 0600 + documented threat if SQLCipher needs CGO. No CGO.
|
||||
- **C-32**: **Human-gate**: leaked GITEA_TOKEN (F17) rotated + `.env` re-seeded before P28 ships. History-scrub best-effort, non-blocking. Escalation hook in `---ci---`.
|
||||
- **C-33**: P26 (security integration tests) in `.coreci.yml` `validate`, gates merges -- not opt-in.
|
||||
- **C-34**: P07 (password/token removal) breaking. `orca upgrade` (P22) refuses v0.11 clusters using `--password`/bare-tokens without `--accept-identity-migration`. No silent breakage.
|
||||
- **C-35**: P08 (Shamir recovery): 3-of-5 shards printed at seal time, operator stores offline. If IdP lost AND quorum unavailable -> cluster unrecoverable by design (documented residual risk). No backdoor.
|
||||
- **C-36**: OIDC client secret (confidential clients) at `ClusterDir()/oidc-client-secret` (0600), rotatable via `orca auth rotate-client-secret`, never committed. Public PKCE clients avoid even this.
|
||||
- **C-37**: P04 (bundled Dex): if WebAuthn proves infeasible, bundled Dex ships mTLS-client-cert-only; password-based upstreams require BYO external IdP. The "no Orca credentials" invariant holds regardless. *(Largely moot -- WebAuthn solves it.)*
|
||||
- **C-38**: P05 (WebAuthn): RP ID must match the cluster's Traefik-served domain; `orca auth init-idp` configures it. HTTPS secure context via Traefik (step-ca cert). P26 integration tests use the WebAuthn virtual-authenticator API -- no hardware key required in CI.
|
||||
|
||||
### Risk register (from grill + research, for ongoing monitoring)
|
||||
|
||||
- **P07 breaking change** (mitigation: C-34 migration gate)
|
||||
- **P08 master key seal is riskiest** (mitigation: `--dry-run`, atomic, auto-rollback, C-35 Shamir recovery)
|
||||
- **P21 SQLite encryption may need CGO** (mitigation: C-31 fallback to file-mode 0600)
|
||||
- **P23 dual-write closure high-impact** (mitigation: gate C-29; full test coverage before deletion)
|
||||
- **P05 WebAuthn connector is new ground** (mitigation: C-37 mTLS-client-cert fallback; virtual-authenticator tests in P26)
|
||||
- **Bundled Dex is a new systemd unit + Traefik route** (mitigation: `orca doctor oidc` health check)
|
||||
- **C-32 human gate could stall final ship** (mitigation: ship as `v0.11.28-rc1` if rotation pending)
|
||||
- **29 phases is large** (mitigation: grill may split/merge; operator accepted "more than 20 if warranted")
|
||||
|
||||
### Deferred to v1.x (out of scope for v0.12)
|
||||
|
||||
- HA step-ca (active/passive via systemd)
|
||||
- `sqlite-wal-shared` / `git` / `file+flock` state backends
|
||||
- OS keyring integration for master key (v0.12 uses OIDC seal instead)
|
||||
- Full cluster-rolling-upgrade orchestrator (v0.12 ships the thin `orca upgrade` wrapper only)
|
||||
- Live-migrate with storage replication (v0.12 ships drain+reschedule only)
|
||||
- Journald log shipping (optional centralized audit)
|
||||
- Network policy (`nftables` snippets beyond the ingress ruleset)
|
||||
- GPU / TPU constraints
|
||||
|
||||
### Deferred to v2.x (out of scope for v1.x)
|
||||
|
||||
- Full Nomad-HCL parser with no conversion round-trip
|
||||
- Nomad-API subset for migrating existing Nomad fleets
|
||||
- Nomad driver bridge
|
||||
- Helm-equivalent templating (probably never)
|
||||
- Service mesh beyond Traefik
|
||||
- CRDs / Operators / Plugin model
|
||||
- Leader-elected Raft coordinator
|
||||
- External CA / Let's Encrypt / cert transparency
|
||||
- Online-only features (HSTS, OCSP stapling, telemetry)
|
||||
|
||||
+75
-20
@@ -4,15 +4,19 @@
|
||||
{
|
||||
"slug": "orca",
|
||||
"name": "Orca",
|
||||
"description": "Offline/CLI-first orchestration engine (Orca) — Nomad-inspired, far simpler than Kubernetes",
|
||||
"milestone": "v0.9",
|
||||
"description": "Offline/CLI-first orchestration engine (Orca) \u2014 Nomad-inspired, far simpler than Kubernetes",
|
||||
"milestone": "v0.12",
|
||||
"phase": 0,
|
||||
"milestone_type": "feature",
|
||||
"default_branch": "main",
|
||||
"tech_stack": {
|
||||
"language": "go",
|
||||
"version": "1.25+",
|
||||
"frameworks": ["cobra", "connectrpc", "modernc/sqlite"],
|
||||
"frameworks": [
|
||||
"cobra",
|
||||
"connectrpc",
|
||||
"modernc/sqlite"
|
||||
],
|
||||
"build_cmd": "make build",
|
||||
"test_cmd": "make test",
|
||||
"typecheck_cmd": "go vet ./...",
|
||||
@@ -23,7 +27,9 @@
|
||||
}
|
||||
],
|
||||
"active_project": "orca",
|
||||
"active_projects": ["orca"],
|
||||
"active_projects": [
|
||||
"orca"
|
||||
],
|
||||
"ship": {
|
||||
"per_phase": true,
|
||||
"allow_skip": false,
|
||||
@@ -31,18 +37,28 @@
|
||||
},
|
||||
"autonomy": {
|
||||
"level": "full",
|
||||
"decision_confidence_threshold": 0.60,
|
||||
"decision_confidence_threshold": 0.6,
|
||||
"max_revision_iterations": 3,
|
||||
"max_verification_retries": 2,
|
||||
"clarify_budget": 10,
|
||||
"escalation_hooks": ["deploy", "delete_data", "merge_to_main"]
|
||||
"escalation_hooks": [
|
||||
"deploy",
|
||||
"delete_data",
|
||||
"merge_to_main"
|
||||
]
|
||||
},
|
||||
"workflow": {
|
||||
"no_hitl": true,
|
||||
"release_flow_per_phase": true,
|
||||
"merge_strategy": {
|
||||
"allowed": ["fast-forward", "rebase-then-fast-forward"],
|
||||
"forbidden": ["merge-commit-no-ff", "squash"],
|
||||
"allowed": [
|
||||
"fast-forward",
|
||||
"rebase-then-fast-forward"
|
||||
],
|
||||
"forbidden": [
|
||||
"merge-commit-no-ff",
|
||||
"squash"
|
||||
],
|
||||
"phase_to_milestone": "fast-forward",
|
||||
"milestone_to_main": "rebase-then-fast-forward"
|
||||
},
|
||||
@@ -60,25 +76,59 @@
|
||||
{
|
||||
"name": "lead-developer",
|
||||
"domain": "coordination",
|
||||
"frameworks": ["cobra"],
|
||||
"constraints": ["boundary-enforcement", "offline-first", "no-redundant-implementations"],
|
||||
"territory": ["**/*.go", "cmd/**", "internal/**"],
|
||||
"frameworks": [
|
||||
"cobra"
|
||||
],
|
||||
"constraints": [
|
||||
"boundary-enforcement",
|
||||
"offline-first",
|
||||
"no-redundant-implementations"
|
||||
],
|
||||
"territory": [
|
||||
"**/*.go",
|
||||
"cmd/**",
|
||||
"internal/**"
|
||||
],
|
||||
"active": true
|
||||
},
|
||||
{
|
||||
"name": "backend-engineer",
|
||||
"domain": "backend",
|
||||
"frameworks": ["cobra", "connectrpc"],
|
||||
"constraints": ["API-first", "error-handling", "minimal-dependencies", "security-first"],
|
||||
"territory": ["**/api/**", "**/*_handler*", "**/*_handler.go", "internal/cli/**"],
|
||||
"frameworks": [
|
||||
"cobra",
|
||||
"connectrpc"
|
||||
],
|
||||
"constraints": [
|
||||
"API-first",
|
||||
"error-handling",
|
||||
"minimal-dependencies",
|
||||
"security-first"
|
||||
],
|
||||
"territory": [
|
||||
"**/api/**",
|
||||
"**/*_handler*",
|
||||
"**/*_handler.go",
|
||||
"internal/cli/**"
|
||||
],
|
||||
"active": true
|
||||
},
|
||||
{
|
||||
"name": "data-engineer",
|
||||
"domain": "data",
|
||||
"frameworks": ["modernc/sqlite"],
|
||||
"constraints": ["schema-first", "migration-safe", "local-storage-only"],
|
||||
"territory": ["**/database/**", "**/model.go", "**/migration*", "migrations/**"],
|
||||
"frameworks": [
|
||||
"modernc/sqlite"
|
||||
],
|
||||
"constraints": [
|
||||
"schema-first",
|
||||
"migration-safe",
|
||||
"local-storage-only"
|
||||
],
|
||||
"territory": [
|
||||
"**/database/**",
|
||||
"**/model.go",
|
||||
"**/migration*",
|
||||
"migrations/**"
|
||||
],
|
||||
"active": true
|
||||
}
|
||||
]
|
||||
@@ -92,7 +142,9 @@
|
||||
},
|
||||
"ci": {
|
||||
"provider": "coreci",
|
||||
"allowed_providers": ["coreci"],
|
||||
"allowed_providers": [
|
||||
"coreci"
|
||||
],
|
||||
"gitea": {
|
||||
"url": "https://git.cloudinit.dev",
|
||||
"owner": "coreci",
|
||||
@@ -140,7 +192,10 @@
|
||||
"scopes": [
|
||||
{
|
||||
"name": "gitea",
|
||||
"vars": ["GITEA_TOKEN", "GITEA_USER"],
|
||||
"vars": [
|
||||
"GITEA_TOKEN",
|
||||
"GITEA_USER"
|
||||
],
|
||||
"env_file": ".env"
|
||||
}
|
||||
]
|
||||
@@ -152,4 +207,4 @@
|
||||
"lint": "make lint",
|
||||
"format": "gofmt -w ."
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
disable=SC2086
|
||||
external-sources=true
|
||||
@@ -39,19 +39,42 @@ build:
|
||||
|
||||
test:
|
||||
go test -coverprofile=coverage.out ./...
|
||||
$(MAKE) test-bash
|
||||
|
||||
# test-race runs the full test suite under the race detector (REQ-031).
|
||||
# Wired into the .coreci.yml `test` pipeline as well.
|
||||
test-race:
|
||||
go test -race -coverprofile=coverage.out ./...
|
||||
$(MAKE) test-bash
|
||||
|
||||
lint:
|
||||
gofmt -l .
|
||||
go vet ./...
|
||||
$(MAKE) lint-bash
|
||||
|
||||
fmt:
|
||||
gofmt -w .
|
||||
|
||||
# test-bash runs bats tests for shell scripts (grill C-15). Skips gracefully
|
||||
# if bats is not installed.
|
||||
test-bash:
|
||||
@command -v bats >/dev/null 2>&1 && { \
|
||||
echo "→ bats scripts/tests/*.bash"; \
|
||||
bats scripts/tests/*.bash; \
|
||||
} || echo "bats not installed; skipping bash tests (see scripts/tests/README.md)"
|
||||
|
||||
# lint-bash runs shellcheck + shfmt on shell scripts (grill C-15). Skips
|
||||
# gracefully if the tools are not installed.
|
||||
lint-bash:
|
||||
@command -v shellcheck >/dev/null 2>&1 && { \
|
||||
echo "→ shellcheck scripts/"; \
|
||||
shellcheck scripts/*.sh scripts/lib/*.sh scripts/tests/*.bash || true; \
|
||||
} || echo "shellcheck not installed; skipping (see scripts/tests/README.md)"
|
||||
@command -v shfmt >/dev/null 2>&1 && { \
|
||||
echo "→ shfmt -d scripts/"; \
|
||||
shfmt -d scripts/; \
|
||||
} || echo "shfmt not installed; skipping (see scripts/tests/README.md)"
|
||||
|
||||
clean:
|
||||
rm -rf bin coverage.out *.tar.gz
|
||||
|
||||
|
||||
@@ -1,20 +1,28 @@
|
||||
# Orca
|
||||
|
||||
Offline/CLI-first orchestration engine inspired by HashiCorp Nomad, far simpler than Kubernetes.
|
||||
A minimalist, offline-first, CLI-first orchestration engine inspired by
|
||||
HashiCorp Nomad. Proxmox is one supported node type — not the project's
|
||||
identity.
|
||||
|
||||
## Status
|
||||
|
||||
**v0.1: Foundation** — see [.ciagent/ROADMAP.md](.ciagent/ROADMAP.md) for the 6-phase plan.
|
||||
**v0.11: Production Hardening — IN PROGRESS** | **v1.0: UAT-gated** (cut
|
||||
separately after v0.11 completion per operator decision)
|
||||
|
||||
See [.ciagent/ROADMAP.md](.ciagent/ROADMAP.md) for the full roadmap.
|
||||
|
||||
## Pillars
|
||||
|
||||
- **Simplicity** — single binary, minimal dependencies
|
||||
- **AI-first** — CLI designed for both humans and AI agents
|
||||
- **Offline-first** — no cloud dependencies
|
||||
- **CLI-first** — primary interface is the command line
|
||||
- **Security before features** — NFRs ship before new functionality
|
||||
- **Simplicity** — single binary, minimal dependencies, no daemon on the
|
||||
critical path
|
||||
- **Offline-first** — no cloud dependencies; the cluster is the OS
|
||||
- **CLI-first** — the command line is the primary interface (humans and
|
||||
AI agents)
|
||||
- **Security before features** — mTLS by default; NFRs ship before new
|
||||
functionality
|
||||
- **WASM-first** — workloads target OS primitives (systemd units,
|
||||
journald), not a container runtime shim
|
||||
- **Bug fixes before features** — stability is paramount
|
||||
- **NFRs before features** — observability and auditability first
|
||||
|
||||
## Quickstart
|
||||
|
||||
@@ -22,13 +30,16 @@ Offline/CLI-first orchestration engine inspired by HashiCorp Nomad, far simpler
|
||||
|
||||
```bash
|
||||
# User-level install (binary at ~/.local/bin/orca, state at ~/.orca)
|
||||
curl -fsSL https://git.cloudinit.dev/coreci/orca/raw/branch/main/scripts/install.sh | bash
|
||||
curl -fsSL https://git.cloudinit.dev/coreci/orca/raw/main/scripts/install.sh | bash
|
||||
|
||||
# System-level install (binary at /usr/local/bin/orca, state at /root/.orca)
|
||||
curl -fsSL https://git.cloudinit.dev/coreci/orca/raw/branch/main/scripts/install.sh | sudo bash -s -- --system
|
||||
curl -fsSL https://git.cloudinit.dev/coreci/orca/raw/main/scripts/install.sh | sudo bash -s -- --system
|
||||
|
||||
# Pin a specific version
|
||||
curl -fsSL https://git.cloudinit.dev/coreci/orca/raw/branch/main/scripts/install.sh | bash -s -- --version v0.4.2
|
||||
# Pin a specific version (latest tag: v0.10.19)
|
||||
curl -fsSL https://git.cloudinit.dev/coreci/orca/raw/main/scripts/install.sh | bash -s -- --version v0.10.19
|
||||
|
||||
# Dry-run: check what would be installed without writing
|
||||
curl -fsSL https://git.cloudinit.dev/coreci/orca/raw/main/scripts/install.sh | bash -s -- --check
|
||||
```
|
||||
|
||||
Then initialize local state and verify:
|
||||
@@ -53,33 +64,92 @@ Re-running the installer updates the binary while preserving your
|
||||
config, database, and certificates in the namespace dir:
|
||||
|
||||
```bash
|
||||
curl -fsSL https://git.cloudinit.dev/coreci/orca/raw/branch/main/scripts/install.sh | bash
|
||||
# → "updated orca from v0.4.1 to v0.4.2"
|
||||
curl -fsSL https://git.cloudinit.dev/coreci/orca/raw/main/scripts/install.sh | bash
|
||||
# → "updated orca from v0.8.15 to v0.10.19"
|
||||
```
|
||||
|
||||
## Subcommands
|
||||
|
||||
| Command | Description | Status |
|
||||
|---------|-------------|--------|
|
||||
| `orca version` | Print version info | ✅ Phase 1 |
|
||||
| `orca init` | Initialize local orca state | ✅ Phase 1 (stub) |
|
||||
| `orca status` | Show orca daemon status | ✅ Phase 1 (stub) |
|
||||
| `orca node` | Node management (`join`, `leave`, `list`) | Phase 2 |
|
||||
| `orca job` | Job management (`run`, `list`, `stop`, `logs`) | Phase 3 |
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `orca init` | Initialize local orca state with full bootstrap |
|
||||
| `orca status` | Show orca daemon status |
|
||||
| `orca version` | Print version information |
|
||||
| `orca daemon` | **(deprecated)** Run the orca daemon (HTTP API + health checks) |
|
||||
| `orca metrics` | Start metrics endpoint (Prometheus text exposition) |
|
||||
| `orca logs` | Aggregate journald logs across nodes (`--all-nodes --since`) |
|
||||
| `orca backup` | Create a signed tar.gz backup of ORCA_HOME |
|
||||
| `orca restore` | Restore ORCA_HOME from a verified signed backup |
|
||||
| `orca upgrade` | Upgrade orca to a new version (thin wrapper; R-017 cutover) |
|
||||
| `orca node` | Manage orca nodes: `join`, `leave`, `list`, `key-reset`, `drain`, `capacity` |
|
||||
| `orca job` | Manage orca jobs: `run`, `list`, `stop`, `logs`, `lint`, `verify`, `migrate`, `restart` |
|
||||
| `orca ns` | Manage orca namespaces: `list`, `create`, `delete`, `inspect`, `validate`, `inherit`, `set-constraint` |
|
||||
| `orca cert` | **(deprecated)** Manage orca certificates: `ca-init`, `gen`, `show`, `renew`, `fingerprint` |
|
||||
| `orca doctor` | Run self-checks: `cert`, `network`, `db`, `os`, `proxmox`, `no-orca-on-server` |
|
||||
| `orca audit` | View orca audit log (`list`) |
|
||||
| `orca cache` | CLI cache management: `show`, `invalidate`, `invalidate-all` |
|
||||
| `orca acl` | ACL management: `grant`, `revoke`, `list`, `check` |
|
||||
| `orca secrets` | Secrets management: `set`, `get`, `list`, `rotate`, `delete` |
|
||||
| `orca drift` | Drift detection: `show`, `watch`, `acknowledge`, `remediate`, `config` |
|
||||
| `orca txn` | Transaction management: `apply`, `list`, `show`, `rollback` |
|
||||
| `orca collector` | Collector/aggregator management: `start`, `stop`, `status` |
|
||||
| `orca cluster` | Cluster management: `cutover`, `rotate-lead`, `compat-check` |
|
||||
|
||||
See [docs/cli.md](docs/cli.md) for the full CLI reference with all flags
|
||||
and examples.
|
||||
|
||||
## Honest trade-offs
|
||||
|
||||
Orca is not a Kubernetes replacement for every workload. This table is
|
||||
the honest comparison — K8s wins in several dimensions, and that is
|
||||
acknowledged rather than papered over.
|
||||
|
||||
| Dimension | Kubernetes wins | Orca wins |
|
||||
|-----------|-----------------|-----------|
|
||||
| Ecosystem | Mature CNCF ecosystem; vast operator, controller, plugin surface | — |
|
||||
| Talent pool | Large pool of K8s-experienced engineers | — |
|
||||
| Multi-cloud | Portable across all major clouds; control plane is cloud-agnostic | — |
|
||||
| Stateful operators | Rich operator pattern (CRD + controller) for stateful workloads | — |
|
||||
| Service mesh | First-class service mesh (Istio, Linkerd) | — |
|
||||
| Auto-scaling | Cluster autoscaler, HPA/VPA, deep integrations | — |
|
||||
| Daemon footprint | — | No daemon on the critical path; the cluster is the OS |
|
||||
| OS-native | — | Workloads are systemd units + journald; no container runtime shim |
|
||||
| mTLS | — | mTLS by default; no opt-in required |
|
||||
| Offline-first | — | No cloud dependencies; fully air-gapped operation |
|
||||
| WASM-first | — | Workloads target OS primitives, not a container runtime |
|
||||
| Proxmox | — | First-class Proxmox node type (`--type proxmox`) via SSH-push |
|
||||
|
||||
## Documentation
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [docs/cli.md](docs/cli.md) | CLI reference — every command, flag, and example |
|
||||
| [docs/jobspec.md](docs/jobspec.md) | Jobspec reference — markdown frontmatter schema |
|
||||
| [docs/ingress.md](docs/ingress.md) | Ingress guide — Traefik configuration |
|
||||
| [docs/namespace.md](docs/namespace.md) | Namespace and path layout |
|
||||
| [docs/install.md](docs/install.md) | Installation guide |
|
||||
| [docs/security-scanning.md](docs/security-scanning.md) | Security scanning tools |
|
||||
|
||||
## Examples
|
||||
|
||||
| Example | Description |
|
||||
|---------|-------------|
|
||||
| [examples/full-stack/](examples/full-stack/) | Full-stack deployment with ingress (5 services + rendered artifacts) |
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
make build # Build binary to ./bin/orca
|
||||
make test # Run tests with race detection
|
||||
make lint # Run golangci-lint
|
||||
make fmt # Format code
|
||||
make release # Build + create Gitea release (Phase 6)
|
||||
make build # Build binary to ./bin/orca
|
||||
make test # Run tests
|
||||
go vet ./... # Vet all packages
|
||||
make lint # Run gofmt + go vet + shellcheck
|
||||
make verify-reqs # Assert ROADMAP ↔ REQUIREMENTS consistency
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
See [.ciagent/ARCHITECTURE.md](.ciagent/ARCHITECTURE.md) for full architecture details.
|
||||
See [.ciagent/ARCHITECTURE.md](.ciagent/ARCHITECTURE.md) for full
|
||||
architecture details.
|
||||
|
||||
## License
|
||||
|
||||
|
||||
+522
@@ -0,0 +1,522 @@
|
||||
# Orca CLI Reference
|
||||
|
||||
This document is the complete reference for the `orca` command-line
|
||||
interface. Every command, subcommand, and flag is documented here.
|
||||
|
||||
> **Canonical path (v0.9)**: The v0.9 re-architecture introduced the
|
||||
> SSH-push deployment model, markdown jobspec, multi-namespace layout,
|
||||
> and CLI-side scheduler. Commands marked **deprecated** below are from
|
||||
> the v0.8 daemon/mTLS model and will be removed in v0.11. Use the
|
||||
> v0.9 canonical path for all new work.
|
||||
|
||||
## Global flags
|
||||
|
||||
These flags are available on every `orca` command.
|
||||
|
||||
| Flag | Type | Default | Description |
|
||||
|------|------|---------|-------------|
|
||||
| `--json` | bool | `false` | Output in JSON format (machine-readable) |
|
||||
| `--system` | bool | `false` | Use system-level namespace root (`/root/.orca`) instead of user-level (`~/.orca`). Errors if `ORCA_HOME` is already set to a conflicting value. |
|
||||
| `--config` | string | `""` | Path to config file (overrides `~/.orca/config.hcl`). Supports `.hcl` (legacy) and `.md` (v0.9 canonical) formats. |
|
||||
| `--no-deprecation-warnings` | bool | `false` | Suppress v0.9 deprecation warnings. Use during `orca upgrade` migrations. |
|
||||
|
||||
### Output modes
|
||||
|
||||
- **Text** (default): human-readable tables and messages.
|
||||
- **JSON** (`--json`): structured JSON output for machine consumption
|
||||
and AI agents.
|
||||
- **Watch** (`--watch` on list commands): table refresh (text default)
|
||||
or NDJSON streaming (`--json`), one line per event until Ctrl-C.
|
||||
|
||||
### Environment variables
|
||||
|
||||
| Variable | Description |
|
||||
|----------|-------------|
|
||||
| `ORCA_HOME` | Namespace root directory (default `~/.orca`). Overrides all on-disk paths. |
|
||||
| `ORCA_DB` | Fine-grained database path override. |
|
||||
| `ORCA_PROXMOX_PASSWORD` | SSH password for `orca node join --type proxmox` (never persisted). |
|
||||
| `ORCA_LISTEN_ADDR` | Daemon listen address (deprecated). |
|
||||
| `ORCA_CA_PATH` | CA certificate path override. |
|
||||
| `ORCA_SERVER_CERT_PATH` | Server certificate path override. |
|
||||
| `ORCA_SERVER_KEY_PATH` | Server key path override. |
|
||||
| `ORCA_NODE_CPU` | Node CPU capacity override (millicores). |
|
||||
| `ORCA_NODE_MEMORY_MB` | Node memory capacity override (MiB). |
|
||||
|
||||
### Exit codes
|
||||
|
||||
| Code | Meaning |
|
||||
|------|---------|
|
||||
| `0` | Success |
|
||||
| `1` | Error (printed to stderr) |
|
||||
|
||||
---
|
||||
|
||||
## `orca init`
|
||||
|
||||
Initialize local orca state with full bootstrap.
|
||||
|
||||
```
|
||||
orca init
|
||||
```
|
||||
|
||||
Performs a 6-step idempotent bootstrap:
|
||||
|
||||
1. Create the namespace directory (honors `$ORCA_HOME`; defaults to `~/.orca`)
|
||||
2. Open and migrate the SQLite database (migrations 0001–0006)
|
||||
3. Bootstrap the internal CA (`ca.crt` + `ca.key`) if not already present
|
||||
4. Generate the server cert (`server.crt` + `server.key`) if not already present
|
||||
5. Auto-detect the local OS via `/etc/os-release`
|
||||
6. Register a localhost node (kind=localhost, os=\<detected\>)
|
||||
|
||||
Re-running `orca init` is safe — it refreshes `last_seen` and `os` on
|
||||
the localhost node without regenerating certs or changing the node ID.
|
||||
|
||||
**Flags**: none.
|
||||
|
||||
**Example**:
|
||||
```bash
|
||||
orca init
|
||||
orca --system init # system-level bootstrap at /root/.orca
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## `orca job`
|
||||
|
||||
Manage orca jobs — run, list, stop, and inspect.
|
||||
|
||||
### `orca job run`
|
||||
|
||||
Run a job from a spec file.
|
||||
|
||||
```
|
||||
orca job run <spec> [flags]
|
||||
```
|
||||
|
||||
Dispatches by file extension:
|
||||
- `.md` → Markdown frontmatter parser (v0.9 canonical)
|
||||
- `.yaml` / `.yml` → YAML frontmatter parser
|
||||
- `.hcl` → Legacy HCL adapter (deprecated, see callout below)
|
||||
|
||||
| Flag | Type | Default | Description |
|
||||
|------|------|---------|-------------|
|
||||
| `--target` | string | `""` | Pin job to a specific node ID (overrides bin-packing scheduler) |
|
||||
| `--idempotency-key` | string | `""` | Idempotency key for cross-node dispatch dedupe |
|
||||
|
||||
**Examples**:
|
||||
```bash
|
||||
orca job run web-app.md
|
||||
orca job run api.yaml --target node-abc-123
|
||||
orca job run worker.md --idempotency-key deploy-2026-08-05
|
||||
```
|
||||
|
||||
> **Deprecated**: `orca job run <spec.hcl>` (legacy HCL jobspec) still
|
||||
> works via the adapter but emits a deprecation warning. Migrate `.hcl`
|
||||
> specs to `.md` (see [docs/jobspec.md](jobspec.md)). Removed in v0.11.
|
||||
|
||||
### `orca job list`
|
||||
|
||||
List all jobs.
|
||||
|
||||
```
|
||||
orca job list [flags]
|
||||
```
|
||||
|
||||
| Flag | Type | Default | Description |
|
||||
|------|------|---------|-------------|
|
||||
| `--watch` | bool | `false` | Stream jobs until Ctrl-C (table refresh or `--json` per-event) |
|
||||
|
||||
**Output columns**: `ID NAME STATUS EXIT`
|
||||
|
||||
**Examples**:
|
||||
```bash
|
||||
orca job list
|
||||
orca job list --watch # table refresh
|
||||
orca job list --watch --json # NDJSON: {"event":"update","job":{...}}
|
||||
```
|
||||
|
||||
### `orca job stop`
|
||||
|
||||
Stop a running job (soft stop).
|
||||
|
||||
```
|
||||
orca job stop [job-id] [flags]
|
||||
```
|
||||
|
||||
| Flag | Type | Default | Description |
|
||||
|------|------|---------|-------------|
|
||||
| `--id` | string | `""` | Job ID (alternative to positional argument) |
|
||||
|
||||
**Example**:
|
||||
```bash
|
||||
orca job stop abc-123-def
|
||||
orca job stop --id abc-123-def
|
||||
```
|
||||
|
||||
### `orca job logs`
|
||||
|
||||
Show task output for a job.
|
||||
|
||||
```
|
||||
orca job logs [job-id] [flags]
|
||||
```
|
||||
|
||||
| Flag | Type | Default | Description |
|
||||
|------|------|---------|-------------|
|
||||
| `--id` | string | `""` | Job ID (alternative to positional argument) |
|
||||
|
||||
**Example**:
|
||||
```bash
|
||||
orca job logs abc-123-def
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## `orca node`
|
||||
|
||||
Manage orca nodes — join, leave, or list nodes in the registry.
|
||||
|
||||
### `orca node join`
|
||||
|
||||
Join a node to the orca registry.
|
||||
|
||||
```
|
||||
orca node join [flags]
|
||||
```
|
||||
|
||||
Node types (via `--type`):
|
||||
- `localhost` (default): register a local or Linux node
|
||||
- `proxmox`: SSH-bootstrap a remote Proxmox VE 8/9 host (deploys orca
|
||||
pubkey, creates orca user + PVE role + sudoers allowlist; requires
|
||||
`--host` + `--password`)
|
||||
|
||||
| Flag | Type | Default | Description |
|
||||
|------|------|---------|-------------|
|
||||
| `--name` | string | `""` | Node name (required for `--type localhost`) |
|
||||
| `--addr` | string | `""` | Node address (default `localhost:8443`) |
|
||||
| `--ca-fingerprint` | string | `""` | Pin CA cert SHA-256 (fails if on-disk CA doesn't match) |
|
||||
| `--type` | string | `"localhost"` | Node type: `localhost` or `proxmox` |
|
||||
| `--host` | string | `""` | Proxmox host address (IP/hostname; required for `--type proxmox`) |
|
||||
| `--ssh-user` | string | `"root"` | SSH username for proxmox bootstrap |
|
||||
| `--password` | string | `""` | SSH password for proxmox bootstrap (never persisted; prefer `$ORCA_PROXMOX_PASSWORD`) |
|
||||
| `--ssh-port` | int | `22` | SSH port for proxmox bootstrap |
|
||||
| `--proxmox-user` | string | `"orca"` | Linux system user to create on the proxmox host |
|
||||
| `--proxmox-role` | string | `"OrcaOperator"` | PVE custom role to create |
|
||||
| `--host-key-fingerprint` | string | `""` | SSH host key `SHA256:base64` fingerprint (pre-pin; supersedes TOFU for `--type proxmox`) |
|
||||
|
||||
**Examples**:
|
||||
```bash
|
||||
# Localhost (deprecated mTLS path)
|
||||
orca node join --name my-node
|
||||
|
||||
# Proxmox (v0.9 canonical SSH-push path)
|
||||
orca node join --type proxmox --host 192.168.1.100 --ssh-user root
|
||||
ORCA_PROXMOX_PASSWORD=secret orca node join --type proxmox --host 192.168.1.100
|
||||
|
||||
# Proxmox with pre-pinned host key
|
||||
orca node join --type proxmox --host 192.168.1.100 --host-key-fingerprint SHA256:abc123...
|
||||
```
|
||||
|
||||
> **Deprecated**: `orca node join` without `--type proxmox` (the
|
||||
> localhost mTLS join path) is deprecated in v0.9. The v0.9 canonical
|
||||
> path is SSH-push (`--type proxmox`) or local execution (no join
|
||||
> needed). Removed in v0.11.
|
||||
|
||||
### `orca node leave`
|
||||
|
||||
Remove a node from the orca registry.
|
||||
|
||||
```
|
||||
orca node leave [node-id] [flags]
|
||||
```
|
||||
|
||||
| Flag | Type | Default | Description |
|
||||
|------|------|---------|-------------|
|
||||
| `--id` | string | `""` | Node ID (alternative to positional argument) |
|
||||
|
||||
### `orca node list`
|
||||
|
||||
List all nodes in the orca registry.
|
||||
|
||||
```
|
||||
orca node list [flags]
|
||||
```
|
||||
|
||||
| Flag | Type | Default | Description |
|
||||
|------|------|---------|-------------|
|
||||
| `--watch` | bool | `false` | Stream nodes until Ctrl-C (table refresh or `--json` per-event) |
|
||||
|
||||
**Output columns**: `ID NAME ADDRESS STATE`
|
||||
|
||||
### `orca node key-reset`
|
||||
|
||||
Reset the SSH known_hosts entry for a node.
|
||||
|
||||
```
|
||||
orca node key-reset <node>
|
||||
```
|
||||
|
||||
Removes the pinned SSH host key for `<node>` from the local
|
||||
`known_hosts` file. The next connect re-pins the key via TOFU or
|
||||
`--host-key-fingerprint`. Local only — does not touch the remote
|
||||
host's `authorized_keys`.
|
||||
|
||||
`<node>` is the node name (for proxmox nodes, this is the host address).
|
||||
|
||||
**Example**:
|
||||
```bash
|
||||
orca node key-reset 192.168.1.100
|
||||
```
|
||||
|
||||
### `orca node capacity`
|
||||
|
||||
Manage node capacity declarations (bin-packing scheduler input).
|
||||
|
||||
```
|
||||
orca node capacity <subcommand>
|
||||
```
|
||||
|
||||
#### `orca node capacity show`
|
||||
|
||||
Show capacity for a node (defaults to `self`).
|
||||
|
||||
```
|
||||
orca node capacity show [node-id] [flags]
|
||||
```
|
||||
|
||||
| Flag | Type | Default | Description |
|
||||
|------|------|---------|-------------|
|
||||
| `--node` | string | `""` | Node ID (defaults to `self`) |
|
||||
|
||||
**Output**: `Node:`, `CPU:` (millicores), `Memory:` (MiB), `Disk:` (MiB), `Updated:`
|
||||
|
||||
#### `orca node capacity set`
|
||||
|
||||
Declare capacity for a node.
|
||||
|
||||
```
|
||||
orca node capacity set [flags]
|
||||
```
|
||||
|
||||
| Flag | Type | Default | Description |
|
||||
|------|------|---------|-------------|
|
||||
| `--cpu` | int64 | `0` | CPU capacity in millicores (1000 = 1 vCPU) |
|
||||
| `--memory` | int64 | `0` | Memory capacity in MiB |
|
||||
| `--disk` | int64 | `0` | Disk capacity in MiB |
|
||||
| `--node` | string | `""` | Node ID (defaults to `self`) |
|
||||
|
||||
**Example**:
|
||||
```bash
|
||||
orca node capacity set --cpu 4000 --memory 8192 --disk 100000
|
||||
orca node capacity set --cpu 2000 --memory 4096 --node web-1
|
||||
```
|
||||
|
||||
#### `orca node capacity list`
|
||||
|
||||
List all node capacity declarations.
|
||||
|
||||
```
|
||||
orca node capacity list
|
||||
```
|
||||
|
||||
**Output columns**: `NODE CPU(mc) MEM(MiB) DISK(MiB) UPDATED`
|
||||
|
||||
---
|
||||
|
||||
## `orca ns`
|
||||
|
||||
Manage orca namespaces under `ORCA_HOME` (R-002).
|
||||
|
||||
Each namespace is a directory with `ns.md`, `.env`, `.env.secrets`,
|
||||
`db/`, `jobs/`, `alloc/`. The implicit root namespace `_defaults`
|
||||
always exists; every namespace inherits from `_defaults` and cannot
|
||||
opt out.
|
||||
|
||||
### `orca ns list`
|
||||
|
||||
List all namespaces under `ORCA_HOME`.
|
||||
|
||||
```
|
||||
orca ns list
|
||||
```
|
||||
|
||||
**Output columns**: `NAME DEFAULT PATH` (`_defaults` marked `*`)
|
||||
|
||||
### `orca ns create`
|
||||
|
||||
Create a namespace directory + `ns.md`.
|
||||
|
||||
```
|
||||
orca ns create <name> [flags]
|
||||
```
|
||||
|
||||
| Flag | Type | Default | Description |
|
||||
|------|------|---------|-------------|
|
||||
| `--parent` | string | `""` | Parent namespace (default `_defaults`; implicit root always appended last) |
|
||||
| `--inherits-env` | bool | `true` | Inherit env from parents |
|
||||
| `--inherits-secrets` | bool | `true` | Inherit secrets from parents |
|
||||
|
||||
**Example**:
|
||||
```bash
|
||||
orca ns create prod --parent _defaults
|
||||
orca ns create staging --parent prod
|
||||
```
|
||||
|
||||
### `orca ns delete`
|
||||
|
||||
Remove an empty namespace directory.
|
||||
|
||||
```
|
||||
orca ns delete <name>
|
||||
```
|
||||
|
||||
Refuses if `jobs/` or `alloc/` contain files. The implicit root
|
||||
`_defaults` cannot be deleted.
|
||||
|
||||
### `orca ns inspect`
|
||||
|
||||
Print the effective inheritance chain, merged env, and constraints.
|
||||
|
||||
```
|
||||
orca ns inspect <name>
|
||||
```
|
||||
|
||||
**Output**: `Namespace:`, `Chain:` (e.g., `prod -> _defaults`), `Env:`
|
||||
(sorted keys), `Constraints:` (unioned CEL expressions).
|
||||
|
||||
### `orca ns validate`
|
||||
|
||||
Run cycle + missing-parent + schema checks on a namespace.
|
||||
|
||||
```
|
||||
orca ns validate <name>
|
||||
```
|
||||
|
||||
Exits 0 if valid, 1 on error. Runs over ALL namespaces under
|
||||
`ORCA_HOME` (parsing + resolving validates cycles and missing parents
|
||||
across the set).
|
||||
|
||||
---
|
||||
|
||||
## `orca doctor`
|
||||
|
||||
Run self-checks on the orca installation.
|
||||
|
||||
```
|
||||
orca doctor [subcommand]
|
||||
```
|
||||
|
||||
Without a subcommand, runs all checks and prints a PASS/WARN/FAIL
|
||||
report per check.
|
||||
|
||||
### Subcommands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `orca doctor cert` | CA, server cert, expiry, fingerprint checks |
|
||||
| `orca doctor network` | Network reachability via mTLS `/healthz` probe |
|
||||
| `orca doctor db` | Database integrity (`PRAGMA integrity_check` + migration version) |
|
||||
| `orca doctor os` | OS detection self-check (verifies `/etc/os-release` matches stored node) |
|
||||
| `orca doctor proxmox` | Proxmox node reachability via SSH `pveversion`/`pvecmd status` probe |
|
||||
|
||||
**Example**:
|
||||
```bash
|
||||
orca doctor
|
||||
orca doctor cert
|
||||
orca doctor proxmox --json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## `orca audit`
|
||||
|
||||
View orca audit log (security-first observability).
|
||||
|
||||
### `orca audit list`
|
||||
|
||||
List recent audit log entries.
|
||||
|
||||
```
|
||||
orca audit list [flags]
|
||||
```
|
||||
|
||||
| Flag | Type | Default | Description |
|
||||
|------|------|---------|-------------|
|
||||
| `--limit` | int | `50` | Max entries to show |
|
||||
|
||||
**Output columns**: `TIMESTAMP ACTOR ACTION RESOURCE RESULT`
|
||||
|
||||
---
|
||||
|
||||
## `orca version`
|
||||
|
||||
Print version information.
|
||||
|
||||
```
|
||||
orca version
|
||||
```
|
||||
|
||||
**Output**:
|
||||
```
|
||||
orca version v0.9.1
|
||||
git commit: abc1234
|
||||
build time: 2026-08-05T20:30:00Z
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## `orca status`
|
||||
|
||||
Show orca daemon status.
|
||||
|
||||
```
|
||||
orca status
|
||||
```
|
||||
|
||||
> **Deprecated**: The daemon model is deprecated in v0.9 (replaced by
|
||||
> SSH-push, R-001). This command returns a stub status. Removed in
|
||||
> v0.11.
|
||||
|
||||
---
|
||||
|
||||
## Deprecated commands
|
||||
|
||||
The following commands are from the v0.8 daemon/mTLS model and are
|
||||
**deprecated in v0.9**. They still work during the dual-write window
|
||||
but emit `slog.Warn` deprecation warnings. They will be **removed in
|
||||
v0.11**.
|
||||
|
||||
> **`orca daemon`** — Run the orca daemon (HTTP API + health checks).
|
||||
> The v0.9 re-architecture replaces the daemon with SSH-push (R-001).
|
||||
> The daemon is repurposed to `drain-and-stop` in v0.11-P05 and deleted
|
||||
> in v0.11-P14. Flags: `--addr` (default `:8080`), `--pprof` (pprof
|
||||
> endpoint, default disabled).
|
||||
|
||||
> **`orca cert`** — Manage orca certificates (CA, server, rotation).
|
||||
> The v0.9 re-architecture replaces the internal CA with step-ca
|
||||
> (D-101). Subcommands: `ca-init`, `gen`, `show`, `renew`,
|
||||
> `fingerprint`. Removed in v0.11.
|
||||
|
||||
> **`orca node join` (mTLS path)** — The localhost mTLS join path
|
||||
> (without `--type proxmox`) is deprecated. The v0.9 canonical path is
|
||||
> SSH-push (`--type proxmox`) or local execution (no join needed).
|
||||
|
||||
> **`orca job run <spec.hcl>`** — Legacy HCL jobspec. Migrate to `.md`
|
||||
> (see [docs/jobspec.md](jobspec.md)). The HCL adapter preserves
|
||||
> `orca job run old-spec.hcl` during the migration window.
|
||||
|
||||
To suppress deprecation warnings during migration, use
|
||||
`--no-deprecation-warnings`:
|
||||
```bash
|
||||
orca --no-deprecation-warnings daemon
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## See also
|
||||
|
||||
- [docs/jobspec.md](jobspec.md) — Markdown frontmatter jobspec reference
|
||||
- [docs/ingress.md](ingress.md) — Traefik ingress configuration guide
|
||||
- [docs/namespace.md](namespace.md) — Namespace and path layout
|
||||
- [docs/install.md](install.md) — Installation guide
|
||||
- [examples/full-stack/](../examples/full-stack/) — Full-stack example with ingress
|
||||
+209
@@ -0,0 +1,209 @@
|
||||
# Orca Ingress & Traefik Guide
|
||||
|
||||
This document explains how Orca configures ingress via Traefik dynamic
|
||||
configuration. It covers the service→Traefik mapping, the R-007
|
||||
socket-vs-TCP-bind model, atomic reload, drain, TLS, and a worked
|
||||
example.
|
||||
|
||||
> **Canonical path (v0.9)**: Orca generates Traefik dynamic
|
||||
> configuration files via the `TraefikEmitter`. The `kind: Service`
|
||||
> workload implies a Traefik route. The emitter renders one YAML file
|
||||
> per Service; Traefik watches the dynamic config directory and reloads
|
||||
> atomically on change.
|
||||
|
||||
## The model
|
||||
|
||||
A `kind: Service` jobspec **implies** a Traefik route (D-175). `Job`
|
||||
and `DaemonSet` do **not** carry a Traefik route by default — a
|
||||
`service:` block on a `Job` is rejected by the validator.
|
||||
|
||||
When `orca job run` submits a `kind: Service` workload, the
|
||||
`TraefikEmitter` renders a Traefik dynamic config file at:
|
||||
|
||||
```
|
||||
/etc/traefik/dynamic/orca-<service-name>.yaml
|
||||
```
|
||||
|
||||
This file contains:
|
||||
- One **router** (`orca-<name>`) with a `PathPrefix` rule and TLS config.
|
||||
- One **service** (`orca-<name>`) as a `loadBalancer` with one **server**
|
||||
per port, pointing at the workload's Unix socket (or TCP port).
|
||||
- A **healthCheck** stanza when the `health:` block is present.
|
||||
|
||||
Traefik watches `/etc/traefik/dynamic/` via `fsnotify` and reloads
|
||||
whenever a file changes. Orca writes config atomically (write-tmp +
|
||||
rename) so Traefik sees a single `IN_MOVED_TO` event and never observes
|
||||
a half-written file.
|
||||
|
||||
## R-007: socket vs TCP bind
|
||||
|
||||
Orca workloads bind to a **Unix socket** by default, not a TCP port.
|
||||
This is the R-007 security model: loopback-only by default, no network
|
||||
exposure.
|
||||
|
||||
### Default: Unix socket
|
||||
|
||||
When `service.bind` is empty (default), the workload binds a Unix
|
||||
socket at:
|
||||
|
||||
```
|
||||
/run/orca/alloc-<alloc-id>/port-<port-name>.sock
|
||||
```
|
||||
|
||||
systemd creates `/run/orca/alloc-<alloc-id>/` via
|
||||
`RuntimeDirectory=orca/alloc-<alloc-id>` (mode 0750, owned by
|
||||
`orca:orca`). The Traefik backend server URL is:
|
||||
|
||||
```yaml
|
||||
servers:
|
||||
- url: "unix:///run/orca/alloc-<alloc-id>/port-<port-name>.sock"
|
||||
```
|
||||
|
||||
### TCP opt-in: `service.bind: 127.0.0.1`
|
||||
|
||||
When `service.bind: 127.0.0.1` is set, the workload binds a TCP port
|
||||
directly (loopback only). The emitter adds an `ExecStartPre` marker to
|
||||
the systemd unit so the bind mode is visible:
|
||||
|
||||
```ini
|
||||
ExecStartPre=/bin/echo orca: bind 127.0.0.1 port <name> (tcp, R-007 opt-in)
|
||||
```
|
||||
|
||||
`service.bind` must be a valid IP address. Empty (socket default) or
|
||||
`127.0.0.1` (TCP opt-in) are the documented values; any other valid IP
|
||||
is accepted but the bind happens in the process, not the emitter.
|
||||
|
||||
## Generated Traefik YAML
|
||||
|
||||
For a Service named `web` with port `http`:
|
||||
|
||||
```yaml
|
||||
http:
|
||||
routers:
|
||||
orca-web:
|
||||
rule: PathPrefix("/web")
|
||||
service: orca-web
|
||||
tls:
|
||||
certResolver: orca
|
||||
domains:
|
||||
- main: "cluster.orca.local"
|
||||
services:
|
||||
orca-web:
|
||||
loadBalancer:
|
||||
servers:
|
||||
- url: "unix:///run/orca/alloc-<alloc-id>/port-http.sock"
|
||||
healthCheck:
|
||||
path: /healthz
|
||||
interval: 5s
|
||||
timeout: 1s
|
||||
```
|
||||
|
||||
- One router per Service, named `orca-<service-name>`.
|
||||
- Router rule: `PathPrefix("/<service-name>")`.
|
||||
- TLS: `certResolver: orca`, trust domain `cluster.orca.local`
|
||||
(placeholder; step-ca provisioner overrides in v0.11).
|
||||
- One service per Service, named `orca-<service-name>`.
|
||||
- One server per port, URL is `unix://<socket-path>`.
|
||||
- `healthCheck` stanza present when `health:` block is set (required
|
||||
for Service). Path is `/healthz`; interval and timeout come from the
|
||||
`health:` block.
|
||||
|
||||
## Atomic reload (gate C-10)
|
||||
|
||||
Orca writes Traefik config atomically to avoid Traefik observing a
|
||||
half-written file:
|
||||
|
||||
1. Write to `<path>.tmp` via `WriteFileIdempotent` (write + fsync).
|
||||
2. `mv -f <path>.tmp <path>` (atomic POSIX rename).
|
||||
|
||||
Traefik's `fsnotify` watcher sees a single `IN_MOVED_TO` event and
|
||||
reloads. If the new config is malformed, Traefik logs an error and
|
||||
**holds last-good config** — the cluster keeps serving traffic on the
|
||||
previous config.
|
||||
|
||||
## Drain
|
||||
|
||||
`RenderDrain` produces the same Traefik YAML with `weight: 0` on every
|
||||
server in the load balancer:
|
||||
|
||||
```yaml
|
||||
servers:
|
||||
- url: "unix:///run/orca/alloc-<alloc-id>/port-http.sock"
|
||||
weight: 0
|
||||
```
|
||||
|
||||
Traefik stops sending traffic to the drained backend. The workload
|
||||
keeps running; drain is reversible (re-submit the normal config to
|
||||
restore traffic).
|
||||
|
||||
## TLS
|
||||
|
||||
- **certResolver**: `orca` (references the Traefik ACME/step-ca
|
||||
certificate resolver configured in Traefik's static config).
|
||||
- **Trust domain**: `cluster.orca.local` (placeholder in v0.9; step-ca
|
||||
provisioner in v0.11 overrides with the real cluster trust domain).
|
||||
- **SPIFFE SVIDs**: workload identity via SPIFFE SVIDs minted at submit
|
||||
time via step-ca (v0.11-P01.5, gate C-08). The SVID is a URI SAN in
|
||||
the workload's X.509 cert.
|
||||
|
||||
## Health checks
|
||||
|
||||
The `health:` block (required for `Service`) maps to the Traefik
|
||||
`healthCheck` stanza:
|
||||
|
||||
```yaml
|
||||
health:
|
||||
check_type: http
|
||||
interval: 5s
|
||||
timeout: 1s
|
||||
unhealthy_threshold: 2
|
||||
```
|
||||
|
||||
→
|
||||
|
||||
```yaml
|
||||
healthCheck:
|
||||
path: /healthz
|
||||
interval: 5s
|
||||
timeout: 1s
|
||||
```
|
||||
|
||||
Traefik polls each backend's `/healthz` at the configured interval. An
|
||||
unhealthy backend is removed from the load balancer pool until it
|
||||
passes the health check again.
|
||||
|
||||
## Worked example
|
||||
|
||||
See [examples/full-stack/](../examples/full-stack/) for a complete
|
||||
multi-service stack with ingress configured:
|
||||
- `web-app.md` — frontend Service (socket bind, PathPrefix route)
|
||||
- `api.md` — backend API Service (TCP opt-in, `127.0.0.1` bind)
|
||||
- `examples/full-stack/rendered/traefik-dynamic-web-app.yaml` — the
|
||||
Traefik config Orca generates
|
||||
|
||||
## v0.11 forward (limitations)
|
||||
|
||||
The following are not yet implemented in v0.9 and will land in v0.11:
|
||||
|
||||
- **`service.host` / `service.route_id`**: stored on the `ServiceBlock`
|
||||
but not yet consumed by the `TraefikEmitter`. The router rule is
|
||||
hardcoded `PathPrefix("/<name>")`. Custom host-based routing lands in
|
||||
v0.11.
|
||||
- **Socket activation**: real socket-activation (socket unit files, fd
|
||||
passing) lands in v0.11-P08. The current emitter renders the
|
||||
`RuntimeDirectory` + socket path comments but does not create socket
|
||||
units.
|
||||
- **Transactional update execution**: the `update:` block's rolling/
|
||||
canary/blue-green plan is computed by the emitter but not yet
|
||||
executed transactionally. Transactional execution lands in
|
||||
v0.11-P10.
|
||||
- **SPIFFE SVID minting**: workload identity via step-ca SVIDs lands in
|
||||
v0.11-P01.5 (gate C-08).
|
||||
- **Secrets in env**: `env: { KEY: { from: "secret:..." } }` resolution
|
||||
to `EnvironmentFile=`/`LoadCredential=` lands in v0.11-P03.
|
||||
|
||||
## See also
|
||||
|
||||
- [docs/cli.md](cli.md) — CLI reference
|
||||
- [docs/jobspec.md](jobspec.md) — Jobspec reference (`service:`, `health:`, `ports:` blocks)
|
||||
- [examples/full-stack/](../examples/full-stack/) — Full-stack example with ingress
|
||||
+442
@@ -0,0 +1,442 @@
|
||||
# Orca Jobspec Reference
|
||||
|
||||
This document is the complete reference for the Orca jobspec format —
|
||||
the Markdown-with-frontmatter specification that describes workloads.
|
||||
|
||||
> **Canonical format (v0.9)**: Orca uses Markdown with YAML frontmatter
|
||||
> as the canonical jobspec format (R-013/R-014). The legacy HCL format
|
||||
> is supported via an adapter during the migration window but is
|
||||
> deprecated (see [HCL jobspec](#deprecated-hcl-jobspec) below).
|
||||
|
||||
## File formats
|
||||
|
||||
The `orca job run` command dispatches by file extension:
|
||||
|
||||
| Extension | Parser | Body |
|
||||
|-----------|--------|------|
|
||||
| `.md` | `ParseMarkdown` (canonical) | Verbatim after closing `---` (R-015 byte-exact) |
|
||||
| `.yaml` / `.yml` | `parseYAMLFile` | Whole file as frontmatter; body empty |
|
||||
| `.hcl` | `ParseHCL` (legacy adapter) | Empty (deprecated) |
|
||||
|
||||
## Minimal example
|
||||
|
||||
```yaml
|
||||
---
|
||||
kind: Job
|
||||
name: my-job
|
||||
runtime:
|
||||
one_of: process
|
||||
command: /bin/echo hello
|
||||
---
|
||||
# My Job
|
||||
|
||||
This body is preserved byte-exact and carried to the target node.
|
||||
```
|
||||
|
||||
## Top-level keys
|
||||
|
||||
| Key | Type | Default | Required | Notes |
|
||||
|-----|------|---------|----------|-------|
|
||||
| `orca-spec-version` | string | `""` | no | Free-form version tag (e.g. `"1"`) |
|
||||
| `kind` | enum | — | **yes** | One of `Job`, `Service`, `DaemonSet` |
|
||||
| `name` | string | — | **yes** | Workload name (trimmed, non-empty) |
|
||||
| `count` | int | `1` | no | Job: must be 1; Service: ≥1; DaemonSet: not allowed |
|
||||
| `runtime` | block | nil | see kinds | Runtime block (or per-task runtimes in a task group) |
|
||||
| `ports` | block list | nil | Service: **yes** | Array of port mappings |
|
||||
| `env` | block map | nil | no | Environment variables |
|
||||
| `secrets` | inline/block list | nil | no | Secret names (resolution in v0.11) |
|
||||
| `volumes` | block list | nil | no | Volume mounts |
|
||||
| `restart` | block | nil | Service/DaemonSet: **yes** | Restart policy |
|
||||
| `update` | block | nil | Service: **yes** | Update strategy |
|
||||
| `service` | block | nil | no | Traefik route definition (implied for Service; not allowed for Job/DaemonSet) |
|
||||
| `health` | block | nil | Service: **yes** | Health check |
|
||||
| `lifecycle` | block | nil | no | Pre-stop / post-start hooks |
|
||||
| `constraints` | list | nil | no | CEL expressions (node selection) |
|
||||
| `affinity` | block list | nil | no | Co-location / anti-affinity rules |
|
||||
| `tasks` | block list | nil | no | Task group (multi-process alloc) |
|
||||
| `timeout` | duration string | `""` | no | Job timeout |
|
||||
| `schedule` | block | nil | DaemonSet: **yes** | Schedule mode |
|
||||
|
||||
## Kinds
|
||||
|
||||
### `Job`
|
||||
|
||||
A one-shot batch task. Runs once and exits.
|
||||
|
||||
- `count` must be 1 (or unset). Use `Service` for replicas.
|
||||
- `service` block is **not allowed** (no Traefik route for Jobs).
|
||||
- `restart` optional (defaults to `never` / `on-failure`).
|
||||
- `timeout` optional.
|
||||
|
||||
**Example**:
|
||||
```yaml
|
||||
---
|
||||
kind: Job
|
||||
name: data-migration
|
||||
runtime:
|
||||
one_of: process
|
||||
command: /usr/bin/python3 migrate.py
|
||||
timeout: 300s
|
||||
env:
|
||||
DB_URL: postgres://localhost/mydb
|
||||
---
|
||||
```
|
||||
|
||||
### `Service`
|
||||
|
||||
A long-running, load-balanced workload with a Traefik route.
|
||||
|
||||
- `count` ≥ 1 (number of replicas).
|
||||
- `ports` required (at least one).
|
||||
- `restart` required; `mode` one of `service`, `on-failure`, `never`.
|
||||
- `update` required; `strategy` one of `rolling`, `canary`, `blue-green`.
|
||||
- `runtime` required (or a task group with per-task runtimes).
|
||||
- `health` required (Traefik routing requires health checks).
|
||||
- `service` block optional (implied for Service; use for `bind` override).
|
||||
- `service.bind` if present must be a valid IP (`127.0.0.1` = TCP opt-in;
|
||||
default = Unix socket).
|
||||
|
||||
**Example**:
|
||||
```yaml
|
||||
---
|
||||
kind: Service
|
||||
name: web
|
||||
count: 3
|
||||
runtime:
|
||||
one_of: process
|
||||
command: /usr/bin/httpd
|
||||
ports:
|
||||
- name: http
|
||||
port: 8080
|
||||
restart:
|
||||
mode: service
|
||||
attempts: 5
|
||||
delay: 2s
|
||||
update:
|
||||
strategy: rolling
|
||||
max_parallel: 1
|
||||
health:
|
||||
check_type: http
|
||||
interval: 5s
|
||||
timeout: 1s
|
||||
unhealthy_threshold: 2
|
||||
constraints:
|
||||
- node.role == "web"
|
||||
---
|
||||
```
|
||||
|
||||
### `DaemonSet`
|
||||
|
||||
A workload that runs on every matching node.
|
||||
|
||||
- `schedule` required; `mode` one of `every-node`, `matching`, `mandatory`.
|
||||
- `ports` **not allowed** (no Traefik route by default).
|
||||
- `count` **not allowed** (implicit = matching nodes).
|
||||
- `restart` required.
|
||||
|
||||
**Example**:
|
||||
```yaml
|
||||
---
|
||||
kind: DaemonSet
|
||||
name: log-shipper
|
||||
schedule:
|
||||
mode: every-node
|
||||
runtime:
|
||||
one_of: process
|
||||
command: /usr/bin/fluent-bit
|
||||
restart:
|
||||
mode: service
|
||||
---
|
||||
```
|
||||
|
||||
## Block reference
|
||||
|
||||
### `runtime`
|
||||
|
||||
The runtime backend for the workload.
|
||||
|
||||
| Field | Key | Type | Default | Notes |
|
||||
|-------|-----|------|---------|-------|
|
||||
| `one_of` | `one_of` | string | — | Runtime type (see below) |
|
||||
| `image` | `image` | string | `""` | Container image (for `podman`) |
|
||||
| `command` | `command` | string | — | ExecStart command |
|
||||
|
||||
**Supported runtime types** (`one_of`):
|
||||
|
||||
| Type | Description | Requires |
|
||||
|------|-------------|----------|
|
||||
| `process` | Direct process execution via systemd (default) | systemd on target |
|
||||
| `wasm` / `wasmtime` | WASM via wasmtime CLI (apt-installed on peer, SSH exec) | wasmtime on target |
|
||||
| `podman` | Container via podman | podman on target |
|
||||
| `pve-vm` | Proxmox VM via `qm` | Proxmox node |
|
||||
| `pve-ct` | Proxmox container via `pct` | Proxmox node |
|
||||
| `proxmox` | Alias for Proxmox runtime | Proxmox node |
|
||||
|
||||
An empty/missing `Runtime` or `OneOf` is runtime-agnostic (always fits
|
||||
the runtime axis in the scheduler).
|
||||
|
||||
### `ports`
|
||||
|
||||
Array of port mappings. Required for `Service`.
|
||||
|
||||
| Field | Key | Type | Default | Notes |
|
||||
|-------|-----|------|---------|-------|
|
||||
| `name` | `name` | string | — | Port name (used in socket path) |
|
||||
| `port` | `port` | int | — | Container port |
|
||||
| `host_port` | `host_port` | int | `0` | Host port |
|
||||
| `protocol` | `protocol` | string | `""` | Protocol (e.g. `tcp`) |
|
||||
| `host_ip` | `host_ip` | string | `""` | Host IP |
|
||||
|
||||
**Example**:
|
||||
```yaml
|
||||
ports:
|
||||
- name: http
|
||||
port: 8080
|
||||
host_port: 80
|
||||
protocol: tcp
|
||||
- name: https
|
||||
port: 8443
|
||||
host_port: 443
|
||||
```
|
||||
|
||||
### `env`
|
||||
|
||||
Environment variables. Scalar values or secret references.
|
||||
|
||||
```yaml
|
||||
env:
|
||||
FOO: bar
|
||||
BAZ: "qux"
|
||||
SECRET_REF:
|
||||
from: "secret:db-password"
|
||||
INLINE: {from: "secret:token"}
|
||||
```
|
||||
|
||||
> Secret resolution (`from: "secret:..."`) lands in v0.11-P03. The
|
||||
> parser stores the reference; the emitter will emit
|
||||
> `EnvironmentFile=`/`LoadCredential=` in v0.11.
|
||||
|
||||
### `secrets`
|
||||
|
||||
List of secret names. Inline array or block list.
|
||||
|
||||
```yaml
|
||||
secrets: ["db-password", "api-token"]
|
||||
# or
|
||||
secrets:
|
||||
- db-password
|
||||
- api-token
|
||||
```
|
||||
|
||||
### `volumes`
|
||||
|
||||
Array of volume mounts.
|
||||
|
||||
| Field | Key | Type | Default | Notes |
|
||||
|-------|-----|------|---------|-------|
|
||||
| `name` | `name` | string | — | Volume name |
|
||||
| `type` | `type` | string | — | Volume type (e.g. `host`) |
|
||||
| `source` | `source` | string | — | Source path (or `replicate:<peer>,<peer>` for Syncthing) |
|
||||
| `target` | `target` | string | — | Mount target |
|
||||
| `read_only` | `read_only` | bool | `false` | Read-only mount (`true`/`yes`/`on`/`1`) |
|
||||
|
||||
**Example**:
|
||||
```yaml
|
||||
volumes:
|
||||
- name: data
|
||||
type: host
|
||||
source: /data
|
||||
target: /data
|
||||
read_only: true
|
||||
```
|
||||
|
||||
### `restart`
|
||||
|
||||
Restart policy.
|
||||
|
||||
| Field | Key | Type | Default | Notes |
|
||||
|-------|-----|------|---------|-------|
|
||||
| `mode` | `mode` | enum | — | `never`, `on-failure`, `service` |
|
||||
| `attempts` / `max_retries` | `attempts` or `max_retries` | int | `0` | Max retries (both keys accepted) |
|
||||
| `delay` | `delay` | duration string | `""` | Retry delay (e.g. `2s`) |
|
||||
|
||||
### `update`
|
||||
|
||||
Update strategy. Required for `Service`.
|
||||
|
||||
| Field | Key | Type | Default | Notes |
|
||||
|-------|-----|------|---------|-------|
|
||||
| `strategy` | `strategy` | enum | — | `rolling`, `canary`, `blue-green` |
|
||||
| `max_surge` | `max_surge` | int | `0` | Max surge |
|
||||
| `max_parallel` | `max_parallel` | int | `1` (clamped to `count`) | Max parallel updates |
|
||||
| `min_healthy_time` | `min_healthy_time` | duration | `""` | Min time healthy before next batch |
|
||||
| `healthy_deadline` | `healthy_deadline` | duration | `""` | Deadline for health |
|
||||
| `canary` | `canary` | int or `"<n>%"` | — | Canary size (int count or percentage) |
|
||||
| `auto_promote` | `auto_promote` | bool | `false` | Auto-promote canary (`true`/`yes`/`on`/`1`) |
|
||||
|
||||
**Strategies**:
|
||||
- **rolling**: batches of `max_parallel`, each batch waits for healthy.
|
||||
- **canary**: canary batch first, then `promote` (manual or `auto_promote`), then remaining in `max_parallel` batches.
|
||||
- **blue-green**: all new allocs start in parallel, wait healthy, then `cutover`.
|
||||
|
||||
> Transactional update execution lands in v0.11-P10. The current
|
||||
> emitter computes the plan; execution is a v0.11 deliverable.
|
||||
|
||||
### `service`
|
||||
|
||||
Traefik route definition. Implied for `Service`; not allowed for
|
||||
`Job`/`DaemonSet`. See [docs/ingress.md](ingress.md) for details.
|
||||
|
||||
| Field | Key | Type | Default | Notes |
|
||||
|-------|-----|------|---------|-------|
|
||||
| `name` | `name` | string | — | Service name |
|
||||
| `port` | `port` | int | — | Service port |
|
||||
| `bind` | `bind` | string (IP) | `""` | Bind mode: empty = Unix socket (default); `127.0.0.1` = TCP opt-in (R-007) |
|
||||
| `host` | `host` | string | `""` | Host (stored, not yet consumed by emitter) |
|
||||
| `route_id` | `route_id` | string | `""` | Route ID (stored, not yet consumed by emitter) |
|
||||
|
||||
### `health`
|
||||
|
||||
Health check. Required for `Service`.
|
||||
|
||||
| Field | Key | Type | Default | Notes |
|
||||
|-------|-----|------|---------|-------|
|
||||
| `check_type` | `check_type` | string | — | Check type (e.g. `http`) |
|
||||
| `interval` | `interval` | duration string | — | Check interval (e.g. `5s`) |
|
||||
| `timeout` | `timeout` | duration string | — | Check timeout |
|
||||
| `unhealthy_threshold` | `unhealthy_threshold` | int | `0` | Failures before unhealthy |
|
||||
|
||||
Maps to Traefik `healthCheck` stanza (`path: /healthz`).
|
||||
|
||||
### `lifecycle`
|
||||
|
||||
Lifecycle hooks. Maps to systemd `ExecStartPost` / `ExecStop`.
|
||||
|
||||
| Field | Key | Type | Default | systemd mapping |
|
||||
|-------|-----|------|---------|-----------------|
|
||||
| `post_start` | `post_start` | string list | nil | `ExecStartPost=` (runs after main starts) |
|
||||
| `pre_stop` | `pre_stop` | string list | nil | `ExecStop=` (runs before kill) |
|
||||
|
||||
**Example**:
|
||||
```yaml
|
||||
lifecycle:
|
||||
pre_stop:
|
||||
- /bin/sh -c 'sleep 5'
|
||||
- /usr/local/bin/drain.sh
|
||||
post_start:
|
||||
- /usr/local/bin/warm-cache.sh
|
||||
```
|
||||
|
||||
### `constraints`
|
||||
|
||||
CEL-subset expressions for node selection. Inline array or block list.
|
||||
|
||||
```yaml
|
||||
constraints:
|
||||
- node.role == "web"
|
||||
- region == "us"
|
||||
# or inline
|
||||
constraints: ['node.role == "web"', 'region == "us"']
|
||||
```
|
||||
|
||||
**CEL subset grammar** (hand-rolled, no CEL dependency):
|
||||
- Node attributes: `node.hostname`, `node.kind`, `node.cpus`,
|
||||
`node.memory`, `node.tags`, `node.runtimes`
|
||||
- Bare identifiers: equivalent to `node.<name>`
|
||||
- Literals: string (`"..."`), int
|
||||
- Comparisons: `==`, `!=`, `>=`, `<=`, `>`, `<`
|
||||
- Membership: `in`, `not in`
|
||||
- Boolean: `and`, `or`, `not`, parentheses
|
||||
- Anything outside the subset returns an error (node skipped, not
|
||||
silently mis-evaluated)
|
||||
|
||||
### `affinity`
|
||||
|
||||
Co-location / anti-affinity rules.
|
||||
|
||||
```yaml
|
||||
affinity:
|
||||
- target: zone == "a"
|
||||
weight: 80
|
||||
- target: web
|
||||
weight: -50 # anti-affinity (negative weight)
|
||||
```
|
||||
|
||||
- `target`: CEL expression or bare workload name (for name-based
|
||||
co-location).
|
||||
- `weight`: positive = co-locate, negative = anti-affinity.
|
||||
- Affinity is a **hint** (not a gate); evaluation failures are ignored.
|
||||
|
||||
### `tasks` (task group)
|
||||
|
||||
Multi-process alloc (P06). When `tasks` is non-empty, the alloc runs
|
||||
multiple processes, each as its own systemd unit, grouped under a
|
||||
systemd target.
|
||||
|
||||
```yaml
|
||||
tasks:
|
||||
- name: app
|
||||
runtime:
|
||||
one_of: process
|
||||
command: /usr/bin/httpd -f
|
||||
env:
|
||||
LOG_LEVEL: debug
|
||||
- name: sidecar
|
||||
runtime:
|
||||
one_of: wasm
|
||||
command: /bin/wasm-runner sidecar.wasm
|
||||
```
|
||||
|
||||
- A task with no `runtime:` inherits the top-level `spec.Runtime`.
|
||||
- Each task can have its own `env:` overlay.
|
||||
- `command` falls back: `task.Command` → `task.Runtime.Command` →
|
||||
`spec.Runtime.Command`.
|
||||
- Task names must be unique within the group.
|
||||
|
||||
## Kinds matrix
|
||||
|
||||
| Feature | Job | Service | DaemonSet |
|
||||
|---------|-----|---------|-----------|
|
||||
| `count` | must be 1 | ≥ 1 | not allowed |
|
||||
| `ports` | optional | **required** | not allowed |
|
||||
| `service` block | not allowed | optional (implied) | not allowed |
|
||||
| `restart` | optional | **required** | **required** |
|
||||
| `update` | optional | **required** | optional |
|
||||
| `health` | optional | **required** | optional |
|
||||
| `runtime` | optional | **required** (or task group) | optional |
|
||||
| `schedule` | optional | optional | **required** |
|
||||
| `tasks` | optional | optional | optional |
|
||||
| Traefik route | no | yes (implied) | no (by default) |
|
||||
|
||||
## Body semantics
|
||||
|
||||
The body after the closing `---` is preserved **byte-exact** (R-015) —
|
||||
including trailing newlines, CRLF, BOM in body, and `---` inside code
|
||||
fences. The body is carried verbatim to the target node. It is not
|
||||
interpreted as commands/scripts by the parser today.
|
||||
|
||||
## Deprecated: HCL jobspec
|
||||
|
||||
The legacy HCL jobspec format is supported via an adapter during the
|
||||
migration window. It is deprecated in v0.9 and will be removed in
|
||||
v0.11.
|
||||
|
||||
```hcl
|
||||
job "hello-orca" {
|
||||
}
|
||||
|
||||
task "greet" {
|
||||
command = "/bin/echo"
|
||||
args = ["hello", "from", "orca"]
|
||||
}
|
||||
```
|
||||
|
||||
The adapter converts this to a `*WorkloadSpec{Kind: "Job", Name:
|
||||
"hello-orca", Count: 1, Runtime: {OneOf: "process", Command:
|
||||
"/bin/echo"}}`. Use `.md` for all new jobspecs.
|
||||
|
||||
## See also
|
||||
|
||||
- [docs/cli.md](cli.md) — CLI reference (including `orca job run`)
|
||||
- [docs/ingress.md](ingress.md) — Traefik ingress configuration
|
||||
- [examples/full-stack/](../examples/full-stack/) — Full-stack example jobspecs
|
||||
+158
-77
@@ -1,96 +1,177 @@
|
||||
# Namespace and Paths
|
||||
|
||||
Orca stores all on-disk state (SQLite database, CA certs, server certs,
|
||||
config) under a single **namespace root** directory. This document
|
||||
describes how that root is resolved and how to override it.
|
||||
Orca stores all on-disk state under a single **namespace root**
|
||||
directory. The v0.9 re-architecture introduced a multi-namespace
|
||||
layout (R-002) where each namespace is a self-contained directory tree
|
||||
with its own database, jobs, allocs, env, and secrets. A `cluster/`
|
||||
directory holds cluster-wide artifacts shared across namespaces.
|
||||
|
||||
## Default: User-Level (`~/.orca`)
|
||||
> **v0.9 layout (canonical)**: This document describes the v0.9
|
||||
> multi-namespace layout. The v0.8 flat layout (`orca.db`, `ca.crt`,
|
||||
> `server.crt` at the root) is deprecated and will be removed in
|
||||
> v0.11. See [v0.8 flat layout](#deprecated-v08-flat-layout) below.
|
||||
|
||||
By default, the namespace root is `~/.orca` (i.e., `$HOME/.orca`).
|
||||
All orca state lives under this directory:
|
||||
## Namespace root resolution
|
||||
|
||||
| Path | Contents |
|
||||
|------|----------|
|
||||
| `~/.orca/orca.db` | SQLite database (jobs, nodes, tasks, audit log, capacity) |
|
||||
| `~/.orca/ca.crt` | CA certificate (PEM, mode 0644) |
|
||||
| `~/.orca/ca.key` | CA private key (PEM, mode 0600) |
|
||||
| `~/.orca/server.crt` | Server certificate (PEM, mode 0644) |
|
||||
| `~/.orca/server.key` | Server private key (PEM, mode 0600) |
|
||||
|
||||
## Override: `ORCA_HOME` Environment Variable (REQ-041)
|
||||
|
||||
Set the `ORCA_HOME` environment variable to change the namespace root
|
||||
for **all** orca components (database, certs, init, daemon):
|
||||
|
||||
```bash
|
||||
export ORCA_HOME=/var/lib/orca
|
||||
orca init # creates /var/lib/orca/
|
||||
orca daemon # reads /var/lib/orca/orca.db
|
||||
orca cert ca-init # writes CA to /var/lib/orca/
|
||||
```
|
||||
|
||||
This is the single source of truth for the namespace root. Every
|
||||
component that reads or writes on-disk state resolves the root via
|
||||
`ORCA_HOME` (falling back to `~/.orca` when unset).
|
||||
|
||||
### Use cases
|
||||
|
||||
- **Testing**: point `ORCA_HOME` at a temp directory.
|
||||
- **Multi-instance**: run multiple orca daemons on the same host with
|
||||
different `ORCA_HOME` values.
|
||||
- **Custom layout**: store state on a mounted volume
|
||||
(`ORCA_HOME=/mnt/orca-data`).
|
||||
|
||||
## System-Level: `--system` Flag (REQ-042)
|
||||
|
||||
The `--system` persistent flag selects the system-level namespace root
|
||||
`/root/.orca`. This is intended for root-owned system deployments
|
||||
(where orca runs as a system service under root):
|
||||
|
||||
```bash
|
||||
sudo orca --system init # creates /root/.orca/
|
||||
sudo orca --system daemon # reads /root/.orca/orca.db
|
||||
sudo orca --system cert ca-init # writes CA to /root/.orca/
|
||||
```
|
||||
|
||||
The `--system` flag is equivalent to setting `ORCA_HOME=/root/.orca`,
|
||||
but it is a CLI convenience that does not require exporting an env var.
|
||||
If `ORCA_HOME` is already set to a different value, `--system` returns
|
||||
an error (to avoid silent namespace mismatches).
|
||||
|
||||
### Path layout
|
||||
|
||||
System-level uses the same directory shape as user-level, just under
|
||||
`/root/.orca` instead of `~/.orca`:
|
||||
|
||||
| Path | Contents |
|
||||
|------|----------|
|
||||
| `/root/.orca/orca.db` | SQLite database |
|
||||
| `/root/.orca/ca.crt` | CA certificate |
|
||||
| `/root/.orca/ca.key` | CA private key |
|
||||
| `/root/.orca/server.crt` | Server certificate |
|
||||
| `/root/.orca/server.key` | Server private key |
|
||||
|
||||
## Resolution Order
|
||||
The namespace root is resolved in this order:
|
||||
|
||||
1. If `--system` flag is passed → root is `/root/.orca` (errors if
|
||||
`ORCA_HOME` is set to a conflicting value).
|
||||
2. Else if `ORCA_HOME` is set → root is `$ORCA_HOME`.
|
||||
3. Else → root is `~/.orca` (`$HOME/.orca`).
|
||||
|
||||
## `ORCA_DB` Override
|
||||
### `ORCA_HOME` (REQ-041)
|
||||
|
||||
For finer-grained control, `ORCA_DB` overrides **only** the database
|
||||
path (not the cert paths). This is primarily a testing affordance. When
|
||||
`ORCA_DB` is set, certs still resolve under `ORCA_HOME` (or `~/.orca`).
|
||||
Set the `ORCA_HOME` environment variable to change the namespace root
|
||||
for all orca components:
|
||||
|
||||
```bash
|
||||
export ORCA_HOME=/var/lib/orca
|
||||
orca init # creates /var/lib/orca/
|
||||
orca ns create prod
|
||||
```
|
||||
|
||||
### `--system` (REQ-042)
|
||||
|
||||
The `--system` persistent flag selects the system-level namespace root
|
||||
`/root/.orca`:
|
||||
|
||||
```bash
|
||||
sudo orca --system init # creates /root/.orca/
|
||||
sudo orca --system ns list
|
||||
```
|
||||
|
||||
If `ORCA_HOME` is already set to a different value, `--system` returns
|
||||
an error (to avoid silent namespace mismatches).
|
||||
|
||||
## v0.9 multi-namespace layout (R-002)
|
||||
|
||||
```
|
||||
$ORCA_HOME/
|
||||
├── cluster/ # cluster-wide (NOT a workload namespace)
|
||||
│ ├── ca.crt, ca.key # step-ca root (R-006, D-101)
|
||||
│ ├── master.key # AES-256-GCM root (R-011, mode 0600)
|
||||
│ ├── config.md # Markdown frontmatter config (R-014)
|
||||
│ ├── known_hosts # SSH known_hosts (D-035)
|
||||
│ ├── orca_ssh_key # orca SSH private key (D-037)
|
||||
│ ├── orca_ssh_key.pub # orca SSH public key
|
||||
│ ├── peers/<host>/ # per-peer directory
|
||||
│ ├── txns/ # cluster transaction log (R-016)
|
||||
│ └── state/ # cluster state
|
||||
├── _defaults/ # implicit root namespace (always exists)
|
||||
│ ├── ns.md # namespace frontmatter (kind: Namespace)
|
||||
│ ├── .env # per-namespace env
|
||||
│ ├── .env.secrets # encrypted secrets
|
||||
│ ├── db/orca.db # per-namespace SQLite database
|
||||
│ ├── jobs/ # submitted jobspecs
|
||||
│ └── alloc/ # allocation state
|
||||
├── <explicit-namespace>/ # operator-created (e.g., prod, staging)
|
||||
│ ├── ns.md
|
||||
│ ├── .env, .env.secrets
|
||||
│ ├── db/orca.db
|
||||
│ ├── jobs/, alloc/
|
||||
│ └── syncthing/ # Syncthing config (if replicated volumes)
|
||||
└── orca_cache.db # CLI-side cache (R-008)
|
||||
```
|
||||
|
||||
### Key points
|
||||
|
||||
- **`_defaults/`** is the implicit root namespace (D-159). It always
|
||||
exists. Every namespace inherits from `_defaults` and cannot opt out
|
||||
(D-185, D-187).
|
||||
- **`cluster/`** is NOT a workload namespace — it holds cluster-wide
|
||||
artifacts (CA, master key, SSH keys, known_hosts, peers, txns).
|
||||
- **Per-namespace DBs**: each namespace has its own
|
||||
`db/orca.db` (R-002). No namespace column in SQLite.
|
||||
- **Namespace inheritance**: child namespaces inherit env and
|
||||
constraints from parents (via `ns.md` frontmatter `parents:` field).
|
||||
`_defaults` is always appended last in the inheritance chain.
|
||||
- **`orca ns` subcommands**: `list`, `create`, `delete`, `inspect`,
|
||||
`validate` — see [docs/cli.md](cli.md#orca-ns).
|
||||
|
||||
### Path reference (`internal/paths/`)
|
||||
|
||||
| Function | Path | Contents |
|
||||
|----------|------|----------|
|
||||
| `Root()` | `$ORCA_HOME` | Namespace root |
|
||||
| `ClusterDir()` | `Root()/cluster` | Cluster-wide artifacts |
|
||||
| `NamespaceDir(ns)` | `Root()/ns` | Per-namespace directory |
|
||||
| `NSDb(ns)` | `Root()/ns/db/orca.db` | Per-namespace SQLite DB |
|
||||
| `NSEnv(ns)` | `Root()/ns/.env` | Per-namespace env |
|
||||
| `NSSecrets(ns)` | `Root()/ns/.env.secrets` | Encrypted secrets |
|
||||
| `NSJobs(ns)` | `Root()/ns/jobs` | Jobs dir |
|
||||
| `NSAlloc(ns)` | `Root()/ns/alloc` | Alloc dir |
|
||||
| `NSMd(ns)` | `Root()/ns/ns.md` | Namespace frontmatter |
|
||||
| `DefaultNamespace()` | `_defaults` | Implicit root (D-159) |
|
||||
| `CACertPath()` | `ClusterDir()/ca.crt` | step-ca root (D-101) |
|
||||
| `MasterKeyPath()` | `ClusterDir()/master.key` | AES-256-GCM root key |
|
||||
| `KnownHostsPath()` | `ClusterDir()/known_hosts` | SSH known_hosts |
|
||||
| `SSHKeyPath()` | `ClusterDir()/orca_ssh_key` | orca SSH private key |
|
||||
| `ConfigPath()` | `ClusterDir()/config.md` | Markdown config (R-014) |
|
||||
| `CacheDB()` | `Root()/orca_cache.db` | CLI-side cache (R-008) |
|
||||
| `PeersDir()` | `ClusterDir()/peers` | Peers directory |
|
||||
| `TxnDir()` | `ClusterDir()/txns` | Transaction log (R-016) |
|
||||
|
||||
## Creating and managing namespaces
|
||||
|
||||
```bash
|
||||
# List all namespaces
|
||||
orca ns list
|
||||
|
||||
# Create a namespace (inherits from _defaults)
|
||||
orca ns create prod
|
||||
|
||||
# Create a namespace with an explicit parent
|
||||
orca ns create staging --parent prod
|
||||
|
||||
# Inspect the effective inheritance chain + merged env
|
||||
orca ns inspect prod
|
||||
|
||||
# Validate a namespace's inheritance chain
|
||||
orca ns validate prod
|
||||
|
||||
# Delete an empty namespace (refuses if jobs/ or alloc/ non-empty)
|
||||
orca ns delete staging
|
||||
```
|
||||
|
||||
See [docs/cli.md](cli.md#orca-ns) for the full `orca ns` reference.
|
||||
|
||||
## `ORCA_DB` override
|
||||
|
||||
For finer-grained control, `ORCA_DB` overrides only the database path
|
||||
(not the cert/namespace paths). This is primarily a testing affordance.
|
||||
|
||||
```bash
|
||||
export ORCA_DB=/tmp/test.db
|
||||
orca daemon # uses /tmp/test.db for the DB, ~/.orca/ for certs
|
||||
orca init # uses /tmp/test.db for the DB, ~/.orca/ for everything else
|
||||
```
|
||||
|
||||
## See Also
|
||||
## Deprecated: v0.8 flat layout
|
||||
|
||||
> **Deprecated in v0.9**: The v0.8 flat layout (`orca.db`, `ca.crt`,
|
||||
> `ca.key`, `server.crt`, `server.key` at the namespace root) is
|
||||
> superseded by the v0.9 multi-namespace layout (R-002). The v0.8
|
||||
> layout is supported during the dual-write window via
|
||||
> `internal/certpaths` (a thin shim) and will be removed in v0.11.
|
||||
|
||||
The v0.8 flat layout stored all state at the namespace root:
|
||||
|
||||
| Path | Contents |
|
||||
|------|----------|
|
||||
| `~/.orca/orca.db` | SQLite database |
|
||||
| `~/.orca/ca.crt` | CA certificate |
|
||||
| `~/.orca/ca.key` | CA private key |
|
||||
| `~/.orca/server.crt` | Server certificate |
|
||||
| `~/.orca/server.key` | Server private key |
|
||||
|
||||
The v0.9 re-architecture moved these to `cluster/` (CA, SSH keys) and
|
||||
per-namespace `db/` (SQLite) to support multi-tenancy (R-002). The
|
||||
`orca doctor --legacy-paths` command (v0.11-P14c) will detect v0.8
|
||||
residue and recommend migration.
|
||||
|
||||
## See also
|
||||
|
||||
- [Install Guide](install.md) — 1-liner install with `install.sh`.
|
||||
- [Docker Guide](docker.md) — running orca in a container (uses
|
||||
`ORCA_HOME=/var/lib/orca` inside the image).
|
||||
- [Docker Guide](docker.md) — running orca in a container.
|
||||
- [CLI Reference](cli.md) — `orca ns` subcommands.
|
||||
- [Jobspec Reference](jobspec.md) — markdown frontmatter schema.
|
||||
@@ -0,0 +1,191 @@
|
||||
# Full-Stack Example with Ingress
|
||||
|
||||
This directory contains a complete multi-service stack deployed with
|
||||
Orca, including Traefik ingress configuration. Each file is a valid
|
||||
Orca jobspec (`.md` frontmatter) that passes the v0.9 parser and schema
|
||||
validators.
|
||||
|
||||
> **Runnable out-of-the-box**: The `runtime.command` in each example
|
||||
> uses `/bin/sleep 3600` (for long-running services) or `/bin/echo`
|
||||
> (for one-shot jobs) so that `orca job run <file>.md` succeeds on any
|
||||
> Linux machine without installing any software. Each file has a
|
||||
> **Production substitution** note showing the real binary to use in a
|
||||
> deployment (e.g. `/usr/bin/httpd`,
|
||||
> `/usr/lib/postgresql/16/bin/postgres`).
|
||||
|
||||
## Stack overview
|
||||
|
||||
| File | Kind | Runtime | Ingress | Description |
|
||||
|------|------|---------|---------|-------------|
|
||||
| `web-app.md` | Service | process | Unix socket (default) | Frontend HTTP server, 3 replicas, rolling update |
|
||||
| `api.md` | Service | process | TCP `127.0.0.1:9090` (R-007 opt-in) | Backend API, 2 replicas, canary update |
|
||||
| `worker.md` | Job | process | none | One-shot batch worker with lifecycle hooks |
|
||||
| `log-shipper.md` | Service | process | Unix socket (metrics) | Log shipper on a dedicated node |
|
||||
| `postgres.md` | Service | process | Unix socket | Database with volume replication, blue-green update |
|
||||
|
||||
## Rendered artifacts
|
||||
|
||||
The `rendered/` directory shows what Orca generates on the target nodes
|
||||
when you submit these jobspecs:
|
||||
|
||||
| File | Description |
|
||||
|------|-------------|
|
||||
| `traefik-dynamic-web-app.yaml` | Traefik dynamic config for the web-app Service |
|
||||
| `traefik-dynamic-api.yaml` | Traefik dynamic config for the api Service (TCP bind) |
|
||||
| `systemd-web-app.service` | Systemd unit for the web-app alloc |
|
||||
| `systemd-api.service` | Systemd unit for the api alloc (with TCP bind marker) |
|
||||
| `systemd-log-shipper.service` | Systemd unit for the log-shipper alloc |
|
||||
|
||||
## Walkthrough
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Orca installed (`orca version` works)
|
||||
- 2+ Linux nodes reachable over SSH (for multi-node scheduling)
|
||||
- Traefik installed on the lead node (watches `/etc/traefik/dynamic/`)
|
||||
|
||||
### Step 1: Initialize the cluster
|
||||
|
||||
```bash
|
||||
# On the operator laptop
|
||||
orca init
|
||||
```
|
||||
|
||||
This creates `~/.orca/` (or `/root/.orca` with `--system`), bootstraps
|
||||
the CA, generates the server cert, auto-detects the OS, and registers
|
||||
a localhost node.
|
||||
|
||||
### Step 2: Join remote nodes
|
||||
|
||||
```bash
|
||||
# Join a Proxmox node (v0.9 canonical SSH-push path)
|
||||
orca node join --type proxmox --host 192.168.1.100 --ssh-user root
|
||||
|
||||
# Join a second node
|
||||
ORCA_PROXMOX_PASSWORD=secret orca node join --type proxmox --host 192.168.1.101
|
||||
```
|
||||
|
||||
### Step 3: Declare node capacity
|
||||
|
||||
The CLI-side scheduler uses capacity declarations for bin-packing:
|
||||
|
||||
```bash
|
||||
orca node capacity set --cpu 4000 --memory 8192 --disk 100000 --node 192.168.1.100
|
||||
orca node capacity set --cpu 4000 --memory 8192 --disk 100000 --node 192.168.1.101
|
||||
```
|
||||
|
||||
### Step 4: Create a namespace
|
||||
|
||||
```bash
|
||||
orca ns create prod --parent _defaults
|
||||
```
|
||||
|
||||
This creates `~/.orca/prod/` with `db/`, `jobs/`, `alloc/`, and `ns.md`.
|
||||
|
||||
### Step 5: Submit the stack
|
||||
|
||||
```bash
|
||||
orca job run web-app.md
|
||||
orca job run api.md
|
||||
orca job run worker.md
|
||||
orca job run log-shipper.md
|
||||
orca job run postgres.md
|
||||
```
|
||||
|
||||
Each `orca job run` parses the `.md` jobspec, validates it against the
|
||||
schema, schedules it via the CLI-side bin-packing scheduler, and
|
||||
generates the systemd + Traefik artifacts on the target node via
|
||||
SSH-push.
|
||||
|
||||
### Step 6: Observe placements
|
||||
|
||||
```bash
|
||||
orca job list --watch
|
||||
|
||||
# Output:
|
||||
# ID NAME STATUS EXIT
|
||||
# abc-123... web-app running 0
|
||||
# def-456... api running 0
|
||||
# ghi-789... worker complete 0
|
||||
# jkl-012... log-shipper running 0
|
||||
# mno-345... postgres running 0
|
||||
```
|
||||
|
||||
### Step 7: Inspect rendered artifacts
|
||||
|
||||
After submission, the target nodes have:
|
||||
|
||||
```
|
||||
/etc/systemd/system/orca-v1-web-app.service # systemd unit
|
||||
/etc/systemd/system/orca-v1-api.service # systemd unit (TCP bind)
|
||||
/etc/traefik/dynamic/orca-web-app.yaml # Traefik dynamic config
|
||||
/etc/traefik/dynamic/orca-api.yaml # Traefik dynamic config
|
||||
/run/orca/alloc-web-app-0/port-http.sock # Unix socket (R-007 default)
|
||||
```
|
||||
|
||||
See the `rendered/` directory in this example for the exact file
|
||||
contents.
|
||||
|
||||
### Step 8: Verify ingress
|
||||
|
||||
Traefik watches `/etc/traefik/dynamic/` and atomically reloads when a
|
||||
file changes (write-tmp + rename, gate C-10). The web-app is reachable
|
||||
at `https://<cluster-domain>/web-app` and the API at
|
||||
`https://<cluster-domain>/api`.
|
||||
|
||||
Health checks (`/healthz` on each backend) ensure Traefik only routes
|
||||
to healthy instances.
|
||||
|
||||
### Step 9: Drain and rollback
|
||||
|
||||
To drain a service (stop traffic, keep the workload running):
|
||||
|
||||
```bash
|
||||
# Orca writes a Traefik config with weight:0 on every backend
|
||||
# (RenderDrain). Traefik stops sending traffic.
|
||||
```
|
||||
|
||||
To roll back, re-submit the normal jobspec — Orca writes the
|
||||
non-drained Traefik config and Traefik resumes routing.
|
||||
|
||||
## Ingress model
|
||||
|
||||
See [docs/ingress.md](../../docs/ingress.md) for the full Traefik
|
||||
ingress reference. Key points:
|
||||
|
||||
- `kind: Service` **implies** a Traefik route (D-175).
|
||||
- Default bind is a **Unix socket** at
|
||||
`/run/orca/alloc-<id>/port-<name>.sock` (R-007).
|
||||
- `service.bind: 127.0.0.1` opts in to **TCP** (loopback only).
|
||||
- One Traefik dynamic file per Service at
|
||||
`/etc/traefik/dynamic/orca-<name>.yaml`.
|
||||
- Atomic reload via write-tmp + rename (gate C-10).
|
||||
- Drain sets `weight: 0` per backend.
|
||||
|
||||
## Validation
|
||||
|
||||
All jobspecs in this directory are validated by a Go test:
|
||||
|
||||
```bash
|
||||
go test ./examples/full-stack/ -v -run TestExamplesValidate
|
||||
```
|
||||
|
||||
This test parses each `.md` file with `jobspec.ParseFile` and validates
|
||||
it against `schema.ValidatorFor(kind)` — ensuring every field used in
|
||||
the examples exists in the current `WorkloadSpec` struct and passes the
|
||||
per-kind validators (gate C-20).
|
||||
|
||||
## v0.11 forward
|
||||
|
||||
The following are not yet implemented in v0.9 and will land in v0.11:
|
||||
|
||||
- **DaemonSet `schedule:` block**: the parser does not yet populate the
|
||||
`schedule:` frontmatter block (v0.9 parser gap). The `log-shipper`
|
||||
example uses `kind: Service` with `count: 1` and a `node.role`
|
||||
constraint as a workaround.
|
||||
- **Secret resolution**: `env: { KEY: { from: "secret:..." } }` is
|
||||
parsed but not resolved to `EnvironmentFile=`/`LoadCredential=` until
|
||||
v0.11-P03.
|
||||
- **Transactional update execution**: the `update:` block's plan is
|
||||
computed but not executed transactionally until v0.11-P10.
|
||||
- **Socket activation**: real socket unit files land in v0.11-P08.
|
||||
@@ -0,0 +1,46 @@
|
||||
---
|
||||
kind: Service
|
||||
name: api
|
||||
count: 2
|
||||
runtime:
|
||||
one_of: process
|
||||
command: /bin/sleep 3600
|
||||
ports:
|
||||
- name: api
|
||||
port: 9090
|
||||
restart:
|
||||
mode: service
|
||||
attempts: 3
|
||||
delay: 5s
|
||||
update:
|
||||
strategy: canary
|
||||
canary: 1
|
||||
max_parallel: 1
|
||||
auto_promote: false
|
||||
min_healthy_time: 30s
|
||||
healthy_deadline: 5m
|
||||
service:
|
||||
name: api
|
||||
port: 9090
|
||||
bind: 127.0.0.1
|
||||
health:
|
||||
check_type: http
|
||||
interval: 10s
|
||||
timeout: 2s
|
||||
unhealthy_threshold: 3
|
||||
constraints:
|
||||
- node.role == "api"
|
||||
- node.cpus >= 2
|
||||
env:
|
||||
DB_HOST: postgres
|
||||
DB_PORT: "5432"
|
||||
LOG_LEVEL: info
|
||||
---
|
||||
# API Server
|
||||
|
||||
Backend API service binding to 127.0.0.1:9090 (TCP opt-in, R-007).
|
||||
Canary update strategy with manual promote. Two replicas with CPU
|
||||
constraint (>= 2 vCPUs) and API-role node selection.
|
||||
|
||||
> **Production substitution**: replace `runtime.command` with your
|
||||
> actual API binary, e.g. `/usr/bin/api-server --listen 127.0.0.1:9090`.
|
||||
@@ -0,0 +1,60 @@
|
||||
package fullstack_test
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/jobspec"
|
||||
"git.cloudinit.dev/coreci/orca/internal/spec/schema"
|
||||
)
|
||||
|
||||
// TestExamplesValidate parses and validates every jobspec in
|
||||
// examples/full-stack/ against the current parser and schema validators
|
||||
// (gate C-20, REQ-094). This ensures the example jobspecs use only
|
||||
// fields that exist in the current WorkloadSpec struct and pass the
|
||||
// per-kind validators.
|
||||
func TestExamplesValidate(t *testing.T) {
|
||||
dir := filepath.Join("..", "..", "examples", "full-stack")
|
||||
entries, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
t.Fatalf("read examples dir: %v", err)
|
||||
}
|
||||
for _, e := range entries {
|
||||
if e.IsDir() {
|
||||
continue
|
||||
}
|
||||
name := e.Name()
|
||||
// Skip README.md and other non-jobspec markdown files.
|
||||
if name == "README.md" {
|
||||
continue
|
||||
}
|
||||
ext := filepath.Ext(name)
|
||||
if ext != ".md" && ext != ".yaml" && ext != ".yml" {
|
||||
continue
|
||||
}
|
||||
t.Run(name, func(t *testing.T) {
|
||||
path := filepath.Join(dir, name)
|
||||
spec, err := jobspec.ParseFile(path)
|
||||
if err != nil {
|
||||
t.Fatalf("ParseFile %s: %v", name, err)
|
||||
}
|
||||
if spec == nil {
|
||||
t.Fatalf("ParseFile %s: spec is nil", name)
|
||||
}
|
||||
if spec.Kind == "" {
|
||||
t.Fatalf("ParseFile %s: kind is empty", name)
|
||||
}
|
||||
if spec.Name == "" {
|
||||
t.Fatalf("ParseFile %s: name is empty", name)
|
||||
}
|
||||
validator, err := schema.ValidatorFor(spec.Kind)
|
||||
if err != nil {
|
||||
t.Fatalf("ValidatorFor %s (kind %s): %v", name, spec.Kind, err)
|
||||
}
|
||||
if err := validator.Validate(spec); err != nil {
|
||||
t.Fatalf("Validate %s: %v", name, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,43 @@
|
||||
---
|
||||
kind: Service
|
||||
name: log-shipper
|
||||
count: 1
|
||||
runtime:
|
||||
one_of: process
|
||||
command: /bin/sleep 3600
|
||||
ports:
|
||||
- name: metrics
|
||||
port: 2024
|
||||
restart:
|
||||
mode: service
|
||||
attempts: 3
|
||||
delay: 10s
|
||||
update:
|
||||
strategy: rolling
|
||||
max_parallel: 1
|
||||
health:
|
||||
check_type: http
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
unhealthy_threshold: 3
|
||||
constraints:
|
||||
- node.role == "logs"
|
||||
env:
|
||||
LOG_LEVEL: warn
|
||||
OUTPUT: unix:///run/orca/alloc-log-collector/ingest.sock
|
||||
---
|
||||
# Log Shipper
|
||||
|
||||
Log shipper service (fluent-bit) running on a dedicated logs-role node.
|
||||
Exposes a metrics port for health checking. Ships logs to a central
|
||||
collector via Unix socket.
|
||||
|
||||
> **Production substitution**: replace `runtime.command` with your
|
||||
> actual log shipper binary, e.g.
|
||||
> `/usr/bin/fluent-bit -c /etc/orca/log-shipper/fluent-bit.conf`.
|
||||
|
||||
> **Note**: DaemonSet kind is defined in the schema but the parser does
|
||||
> not yet populate the `schedule:` block from frontmatter (v0.9 parser
|
||||
> gap). This example uses `kind: Service` with `count: 1` and a
|
||||
> `node.role == "logs"` constraint to achieve single-node placement
|
||||
> until the parser gains `schedule:` support (v0.11).
|
||||
@@ -0,0 +1,52 @@
|
||||
---
|
||||
kind: Service
|
||||
name: postgres
|
||||
count: 1
|
||||
runtime:
|
||||
one_of: process
|
||||
command: /bin/sleep 3600
|
||||
ports:
|
||||
- name: pg
|
||||
port: 5432
|
||||
restart:
|
||||
mode: service
|
||||
attempts: 5
|
||||
delay: 10s
|
||||
update:
|
||||
strategy: blue-green
|
||||
min_healthy_time: 60s
|
||||
healthy_deadline: 10m
|
||||
service:
|
||||
name: postgres
|
||||
port: 5432
|
||||
health:
|
||||
check_type: http
|
||||
interval: 15s
|
||||
timeout: 5s
|
||||
unhealthy_threshold: 3
|
||||
volumes:
|
||||
- name: data
|
||||
type: host
|
||||
source: replicate:peer-b,peer-c
|
||||
target: /var/lib/postgresql/data
|
||||
read_only: false
|
||||
constraints:
|
||||
- node.role == "db"
|
||||
- node.cpus >= 4
|
||||
- node.memory >= 8192
|
||||
env:
|
||||
POSTGRES_DB: appdb
|
||||
POSTGRES_USER: orca
|
||||
PGDATA: /var/lib/postgresql/data
|
||||
---
|
||||
# PostgreSQL
|
||||
|
||||
Database service with a single replica, blue-green update strategy,
|
||||
and volume replication via Syncthing (replicate:peer-b,peer-c). The
|
||||
data volume is replicated to two peers for fault tolerance. Health
|
||||
check on port 5432. Constraints require DB-role nodes with >= 4 vCPUs
|
||||
and >= 8 GiB memory.
|
||||
|
||||
> **Production substitution**: replace `runtime.command` with your
|
||||
> actual postgres binary, e.g.
|
||||
> `/usr/lib/postgresql/16/bin/postgres -D /var/lib/postgresql/data`.
|
||||
@@ -0,0 +1,9 @@
|
||||
# Systemd unit for orca api service (alloc api-0)
|
||||
# Generated by SystemdEmitter (internal/emitter/systemd.go)
|
||||
# Path on target node: /etc/systemd/system/orca-v1-api.service
|
||||
# service.bind: 127.0.0.1 (TCP opt-in, R-007)
|
||||
[Service]
|
||||
ExecStart=/bin/sleep 3600
|
||||
RuntimeDirectory=orca/alloc-api-0
|
||||
# socket: /run/orca/alloc-api-0/port-api.sock
|
||||
ExecStartPre=/bin/echo orca: bind 127.0.0.1 port api (tcp, R-007 opt-in)
|
||||
@@ -0,0 +1,7 @@
|
||||
# Systemd unit for orca log-shipper service
|
||||
# Generated by SystemdEmitter (internal/emitter/systemd.go)
|
||||
# Path on target node: /etc/systemd/system/orca-v1-log-shipper.service
|
||||
[Service]
|
||||
ExecStart=/bin/sleep 3600
|
||||
RuntimeDirectory=orca/alloc-log-shipper-0
|
||||
# socket: /run/orca/alloc-log-shipper-0/port-metrics.sock
|
||||
@@ -0,0 +1,11 @@
|
||||
# Systemd unit for orca web-app service (alloc web-app-0)
|
||||
# Generated by SystemdEmitter (internal/emitter/systemd.go)
|
||||
# Path on target node: /etc/systemd/system/orca-v1-web-app.service
|
||||
# Unit name prefix orca-v1- (dual-write window, REQ-090)
|
||||
[Service]
|
||||
ExecStart=/bin/sleep 3600
|
||||
ExecStartPost=/bin/echo cache warmed
|
||||
ExecStop=/bin/sleep 5
|
||||
ExecStop=/bin/echo draining web-app
|
||||
RuntimeDirectory=orca/alloc-web-app-0
|
||||
# socket: /run/orca/alloc-web-app-0/port-http.sock
|
||||
@@ -0,0 +1,23 @@
|
||||
# Traefik dynamic config for orca api service
|
||||
# Generated by TraefikEmitter (internal/emitter/traefik.go)
|
||||
# Path on target node: /etc/traefik/dynamic/orca-api.yaml
|
||||
# service.bind: 127.0.0.1 (TCP opt-in, R-007)
|
||||
http:
|
||||
routers:
|
||||
orca-api:
|
||||
rule: PathPrefix("/api")
|
||||
service: orca-api
|
||||
tls:
|
||||
certResolver: orca
|
||||
domains:
|
||||
- main: "cluster.orca.local"
|
||||
services:
|
||||
orca-api:
|
||||
loadBalancer:
|
||||
servers:
|
||||
- url: "http://127.0.0.1:9090"
|
||||
- url: "http://127.0.0.1:9090"
|
||||
healthCheck:
|
||||
path: /healthz
|
||||
interval: 10s
|
||||
timeout: 2s
|
||||
@@ -0,0 +1,24 @@
|
||||
# Traefik dynamic config for orca web-app service
|
||||
# Generated by TraefikEmitter (internal/emitter/traefik.go)
|
||||
# Path on target node: /etc/traefik/dynamic/orca-web-app.yaml
|
||||
# Atomic reload: write to .tmp + mv (gate C-10)
|
||||
http:
|
||||
routers:
|
||||
orca-web-app:
|
||||
rule: PathPrefix("/web-app")
|
||||
service: orca-web-app
|
||||
tls:
|
||||
certResolver: orca
|
||||
domains:
|
||||
- main: "cluster.orca.local"
|
||||
services:
|
||||
orca-web-app:
|
||||
loadBalancer:
|
||||
servers:
|
||||
- url: "unix:///run/orca/alloc-web-app-0/port-http.sock"
|
||||
- url: "unix:///run/orca/alloc-web-app-1/port-http.sock"
|
||||
- url: "unix:///run/orca/alloc-web-app-2/port-http.sock"
|
||||
healthCheck:
|
||||
path: /healthz
|
||||
interval: 5s
|
||||
timeout: 1s
|
||||
@@ -0,0 +1,51 @@
|
||||
---
|
||||
kind: Service
|
||||
name: web-app
|
||||
count: 3
|
||||
runtime:
|
||||
one_of: process
|
||||
command: /bin/sleep 3600
|
||||
ports:
|
||||
- name: http
|
||||
port: 8080
|
||||
restart:
|
||||
mode: service
|
||||
attempts: 5
|
||||
delay: 2s
|
||||
update:
|
||||
strategy: rolling
|
||||
max_parallel: 1
|
||||
min_healthy_time: 10s
|
||||
healthy_deadline: 2m
|
||||
service:
|
||||
name: web-app
|
||||
port: 8080
|
||||
health:
|
||||
check_type: http
|
||||
interval: 5s
|
||||
timeout: 1s
|
||||
unhealthy_threshold: 2
|
||||
constraints:
|
||||
- node.role == "web"
|
||||
affinity:
|
||||
- target: zone == "a"
|
||||
weight: 80
|
||||
lifecycle:
|
||||
post_start:
|
||||
- /bin/sh -c 'echo cache warmed'
|
||||
pre_stop:
|
||||
- /bin/sh -c 'sleep 5'
|
||||
- /bin/sh -c 'echo draining web-app'
|
||||
---
|
||||
# Web App
|
||||
|
||||
Frontend web application serving HTTP on port 8080 via Unix socket.
|
||||
Three replicas with rolling updates, anti-affinity for zone spreading,
|
||||
and lifecycle hooks for cache warm-up and graceful drain.
|
||||
|
||||
> **Production substitution**: this example uses `/bin/sh -c 'echo ...
|
||||
> sleep 3600'` so it runs out-of-the-box on any Linux machine. In a
|
||||
> real deployment, replace the `runtime.command` with your actual
|
||||
> binary, e.g. `/usr/bin/httpd -f /etc/orca/web-app/httpd.conf`, and
|
||||
> replace the lifecycle hooks with your real scripts
|
||||
> (`/usr/local/bin/warm-cache.sh`, `/usr/local/bin/drain.sh`).
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
kind: Job
|
||||
name: worker
|
||||
runtime:
|
||||
one_of: process
|
||||
command: /bin/echo worker processing batch
|
||||
timeout: 300s
|
||||
env:
|
||||
QUEUE_URL: unix:///run/orca/alloc-worker/queue.sock
|
||||
BATCH_SIZE: "100"
|
||||
LOG_LEVEL: debug
|
||||
lifecycle:
|
||||
post_start:
|
||||
- /bin/sh -c 'echo worker registered'
|
||||
pre_stop:
|
||||
- /bin/sh -c 'echo draining worker queue'
|
||||
---
|
||||
# Worker
|
||||
|
||||
One-shot batch worker that processes items from a queue. Runs once,
|
||||
exits on completion or after 300s timeout. Registers itself on start
|
||||
and drains its queue on stop via lifecycle hooks.
|
||||
|
||||
> **Production substitution**: replace `runtime.command` with your
|
||||
> actual worker binary, e.g. `/usr/bin/python3 /opt/orca/jobs/worker.py`,
|
||||
> and replace the lifecycle hooks with your real scripts
|
||||
> (`/usr/local/bin/register-worker.sh`, `/usr/local/bin/drain-queue.sh`).
|
||||
@@ -3,10 +3,14 @@ module git.cloudinit.dev/coreci/orca
|
||||
go 1.25.0
|
||||
|
||||
require (
|
||||
github.com/coreos/go-oidc/v3 v3.20.0
|
||||
github.com/go-webauthn/webauthn v0.17.4
|
||||
github.com/google/uuid v1.6.0
|
||||
github.com/hashicorp/hcl/v2 v2.24.0
|
||||
github.com/spf13/cobra v1.8.1
|
||||
golang.org/x/crypto v0.54.0
|
||||
golang.org/x/oauth2 v0.36.0
|
||||
golang.org/x/sync v0.22.0
|
||||
modernc.org/sqlite v1.51.0
|
||||
)
|
||||
|
||||
@@ -14,16 +18,24 @@ require (
|
||||
github.com/agext/levenshtein v1.2.1 // indirect
|
||||
github.com/apparentlymart/go-textseg/v15 v15.0.0 // indirect
|
||||
github.com/dustin/go-humanize v1.0.1 // indirect
|
||||
github.com/fxamacker/cbor/v2 v2.9.2 // indirect
|
||||
github.com/go-jose/go-jose/v4 v4.1.4 // indirect
|
||||
github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
|
||||
github.com/go-webauthn/x v0.2.6 // indirect
|
||||
github.com/golang-jwt/jwt/v5 v5.3.1 // indirect
|
||||
github.com/google/go-cmp v0.7.0 // indirect
|
||||
github.com/google/go-tpm v0.9.8 // indirect
|
||||
github.com/inconshreveable/mousetrap v1.1.0 // indirect
|
||||
github.com/mattn/go-isatty v0.0.20 // indirect
|
||||
github.com/mitchellh/go-wordwrap v1.0.1 // indirect
|
||||
github.com/ncruces/go-strftime v1.0.0 // indirect
|
||||
github.com/philhofer/fwd v1.2.0 // indirect
|
||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
|
||||
github.com/spf13/pflag v1.0.5 // indirect
|
||||
github.com/tinylib/msgp v1.6.4 // indirect
|
||||
github.com/x448/float16 v0.8.4 // indirect
|
||||
github.com/zclconf/go-cty v1.16.3 // indirect
|
||||
golang.org/x/mod v0.37.0 // indirect
|
||||
golang.org/x/sync v0.22.0 // indirect
|
||||
golang.org/x/sys v0.47.0 // indirect
|
||||
golang.org/x/text v0.40.0 // indirect
|
||||
golang.org/x/tools v0.47.0 // indirect
|
||||
|
||||
@@ -2,15 +2,33 @@ github.com/agext/levenshtein v1.2.1 h1:QmvMAjj2aEICytGiWzmxoE0x2KZvE0fvmqMOfy2tj
|
||||
github.com/agext/levenshtein v1.2.1/go.mod h1:JEDfjyjHDjOF/1e4FlBE/PkbqA9OfWu2ki2W0IB5558=
|
||||
github.com/apparentlymart/go-textseg/v15 v15.0.0 h1:uYvfpb3DyLSCGWnctWKGj857c6ew1u1fNQOlOtuGxQY=
|
||||
github.com/apparentlymart/go-textseg/v15 v15.0.0/go.mod h1:K8XmNZdhEBkdlyDdvbmmsvpAG721bKi0joRfFdHIWJ4=
|
||||
github.com/coreos/go-oidc/v3 v3.20.0 h1:EtE0WIBHk03N+DqGkY4+UONzzZHk7amKt6IyNd7OsZE=
|
||||
github.com/coreos/go-oidc/v3 v3.20.0/go.mod h1:DYCf24+ncYi+XkIH97GY1+dqoRlbaSI26KVTCI9SrY4=
|
||||
github.com/cpuguy83/go-md2man/v2 v2.0.4/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
||||
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
||||
github.com/fxamacker/cbor/v2 v2.9.2 h1:X4Ksno9+x3cz0TZv69ec1hxP/+tymuR8PXQJyDwfh78=
|
||||
github.com/fxamacker/cbor/v2 v2.9.2/go.mod h1:vM4b+DJCtHn+zz7h3FFp/hDAI9WNWCsZj23V5ytsSxQ=
|
||||
github.com/go-jose/go-jose/v4 v4.1.4 h1:moDMcTHmvE6Groj34emNPLs/qtYXRVcd6S7NHbHz3kA=
|
||||
github.com/go-jose/go-jose/v4 v4.1.4/go.mod h1:x4oUasVrzR7071A4TnHLGSPpNOm2a21K9Kf04k1rs08=
|
||||
github.com/go-test/deep v1.0.3 h1:ZrJSEWsXzPOxaZnFteGEfooLba+ju3FYIbOrS+rQd68=
|
||||
github.com/go-test/deep v1.0.3/go.mod h1:wGDj63lr65AM2AQyKZd/NYHGb0R+1RLqB8NKt3aSFNA=
|
||||
github.com/go-viper/mapstructure/v2 v2.5.0 h1:vM5IJoUAy3d7zRSVtIwQgBj7BiWtMPfmPEgAXnvj1Ro=
|
||||
github.com/go-viper/mapstructure/v2 v2.5.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
|
||||
github.com/go-webauthn/webauthn v0.17.4 h1:KFTSz3R2RYDiUn/0cDi3XTJgFenSG74eKTTHlqWhlxk=
|
||||
github.com/go-webauthn/webauthn v0.17.4/go.mod h1:pZk63EE/BdztlmyS4Yc+9H5g4a8blNlbtGmdHQHbZX8=
|
||||
github.com/go-webauthn/x v0.2.6 h1:TEyDuQAIiEgYpx60nKiBJIX/5nSUC8LxNbH+uf5U9uk=
|
||||
github.com/go-webauthn/x v0.2.6/go.mod h1:45bA7YEqyQhRcQJ/TiBb46Ww8yqHBGvgEhQ3WWF0aDo=
|
||||
github.com/golang-jwt/jwt/v5 v5.3.1 h1:kYf81DTWFe7t+1VvL7eS+jKFVWaUnK9cB1qbwn63YCY=
|
||||
github.com/golang-jwt/jwt/v5 v5.3.1/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE=
|
||||
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
|
||||
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
|
||||
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
|
||||
github.com/google/go-tpm v0.9.8/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
|
||||
github.com/google/go-tpm-tools v0.3.13-0.20230620182252-4639ecce2aba h1:qJEJcuLzH5KDR0gKc0zcktin6KSAwL7+jWKBYceddTc=
|
||||
github.com/google/go-tpm-tools v0.3.13-0.20230620182252-4639ecce2aba/go.mod h1:EFYHy8/1y2KfgTAsx7Luu7NGhoxtuVHnNo8jE7FikKc=
|
||||
github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e h1:ijClszYn+mADRFY17kjQEVQ1XRhq2/JR1M3sGqeJoxs=
|
||||
github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e/go.mod h1:boTsfXsheKC2y+lKOCMpSfarhxDeIzfZG1jqGcPl3cA=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
@@ -27,6 +45,10 @@ github.com/mitchellh/go-wordwrap v1.0.1 h1:TLuKupo69TCn6TQSyGxwI1EblZZEsQ0vMlAFQ
|
||||
github.com/mitchellh/go-wordwrap v1.0.1/go.mod h1:R62XHJLzvMFRBbcrT7m7WgmE1eOyTSsCt+hzestvNj0=
|
||||
github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w=
|
||||
github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
|
||||
github.com/philhofer/fwd v1.2.0 h1:e6DnBTl7vGY+Gz322/ASL4Gyp1FspeMvx1RNDoToZuM=
|
||||
github.com/philhofer/fwd v1.2.0/go.mod h1:RqIHx9QI14HlwKwm98g9Re5prTQ6LdeRQn+gXJFxsJM=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
|
||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
|
||||
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
|
||||
@@ -34,14 +56,24 @@ github.com/spf13/cobra v1.8.1 h1:e5/vxKd/rZsfSJMUX1agtjeTDf+qv1/JdBF8gg5k9ZM=
|
||||
github.com/spf13/cobra v1.8.1/go.mod h1:wHxEcudfqmLYa8iTfL+OuZPbBZkmvliBWKIezN3kD9Y=
|
||||
github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
|
||||
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
|
||||
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||
github.com/tinylib/msgp v1.6.4 h1:mOwYbyYDLPj35mkA2BjjYejgJk9BuHxDdvRnb6v2ZcQ=
|
||||
github.com/tinylib/msgp v1.6.4/go.mod h1:RSp0LW9oSxFut3KzESt5Voq4GVWyS+PSulT77roAqEA=
|
||||
github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM=
|
||||
github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg=
|
||||
github.com/zclconf/go-cty v1.16.3 h1:osr++gw2T61A8KVYHoQiFbFd1Lh3JOCXc/jFLJXKTxk=
|
||||
github.com/zclconf/go-cty v1.16.3/go.mod h1:VvMs5i0vgZdhYawQNq5kePSpLAoz8u1xvZgrPIxfnZE=
|
||||
github.com/zclconf/go-cty-debug v0.0.0-20240509010212-0d6042c53940 h1:4r45xpDWB6ZMSMNJFMOjqrGHynW3DIBuR2H9j0ug+Mo=
|
||||
github.com/zclconf/go-cty-debug v0.0.0-20240509010212-0d6042c53940/go.mod h1:CmBdvvj3nqzfzJ6nTCIwDTPZ56aVGvDrmztiO5g3qrM=
|
||||
go.uber.org/mock v0.6.0 h1:hyF9dfmbgIX5EfOdasqLsWD6xqpNZlXblLB/Dbnwv3Y=
|
||||
go.uber.org/mock v0.6.0/go.mod h1:KiVJ4BqZJaMj4svdfmHM0AUx4NJYO8ZNpPnZn1Z+BBU=
|
||||
golang.org/x/crypto v0.54.0 h1:YLIA59K4fiNzHzjnZt2tUJQjQtUWfWbeHBqKtk3eScw=
|
||||
golang.org/x/crypto v0.54.0/go.mod h1:KWL8ny2AZdGR2cWmzeHrp2azQPGogOv+HeQaVEXC2dk=
|
||||
golang.org/x/mod v0.37.0 h1:vF1DjpVEshcIqoEaauuHebaLk1O1forxjxBaVn884JQ=
|
||||
golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0=
|
||||
golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs=
|
||||
golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q=
|
||||
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
|
||||
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
|
||||
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
@@ -54,6 +86,7 @@ golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
|
||||
golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q=
|
||||
golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
modernc.org/cc/v4 v4.28.2 h1:3tQ0lf2ADtoby2EtSP+J7IE2SHwEJdP8ioR59wx7XpY=
|
||||
modernc.org/cc/v4 v4.28.2/go.mod h1:OnovgIhbbMXMu1aISnJ0wvVD1KnW+cAUJkIrAWh+kVI=
|
||||
|
||||
@@ -0,0 +1,222 @@
|
||||
// Package acl implements the orca access-control layer.
|
||||
//
|
||||
// An Identity is one of:
|
||||
// - KindSpiffe: a verified SPIFFE workload SVID whose URI is
|
||||
// spiffe://orca.local/ns/<ns>/sa/<sa>/<alloc> (machine identity).
|
||||
// - KindOidc: a verified OIDC ID token whose subject (sub) + groups
|
||||
// map to namespace permissions (human identity, R-021).
|
||||
//
|
||||
// KindToken is DEPRECATED and always denies (R-021: no Orca-issued
|
||||
// tokens). Existing acl.json entries with KindToken are inert; P07
|
||||
// removes them and P22 migrates them.
|
||||
//
|
||||
// Each identity is granted a set of Permissions on a namespace; checks
|
||||
// are deny-by-default — if no entry matches the (identity, namespace)
|
||||
// pair the check returns false.
|
||||
package acl
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/url"
|
||||
"strings"
|
||||
"sync"
|
||||
)
|
||||
|
||||
const (
|
||||
KindSpiffe = "spiffe"
|
||||
KindToken = "token" // DEPRECATED: always denies (R-021). Removed by P07.
|
||||
KindOidc = "oidc"
|
||||
)
|
||||
|
||||
// Permission is a bitmask of access rights on a namespace.
|
||||
type Permission uint8
|
||||
|
||||
// Permission flags. Admin implies Read and Write.
|
||||
const (
|
||||
PermRead Permission = 1
|
||||
PermWrite Permission = 2
|
||||
PermAdmin Permission = 4
|
||||
)
|
||||
|
||||
// AllPermissions is the union of Read + Write + Admin.
|
||||
const AllPermissions Permission = PermRead | PermWrite | PermAdmin
|
||||
|
||||
// Identity is a principal recognized by the ACL layer. Kind is one of
|
||||
// KindSpiffe / KindToken. ID is the SPIFFE URI (for spiffe identities)
|
||||
// or the token ID (for token identities). Namespace is the namespace
|
||||
// scope — for a SPIFFE identity it is extracted from the URI path;
|
||||
// for a token it is the namespace claim set at creation time.
|
||||
type Identity struct {
|
||||
Kind string `json:"kind"`
|
||||
ID string `json:"id"`
|
||||
Namespace string `json:"namespace"`
|
||||
}
|
||||
|
||||
// ACLEntry binds an Identity to a Namespace with a Permission set.
|
||||
// A single identity may have at most one entry per namespace; granting
|
||||
// again on the same namespace replaces the permissions.
|
||||
type ACLEntry struct {
|
||||
Identity Identity `json:"identity"`
|
||||
Namespace string `json:"namespace"`
|
||||
Permissions Permission `json:"permissions"`
|
||||
}
|
||||
|
||||
// ACL is a thread-safe list of ACLEntry. Deny-by-default: an identity
|
||||
// with no matching entry has no permissions.
|
||||
type ACL struct {
|
||||
mu sync.RWMutex
|
||||
entries []ACLEntry
|
||||
}
|
||||
|
||||
// NewACL returns an empty ACL.
|
||||
func NewACL() *ACL {
|
||||
return &ACL{entries: make([]ACLEntry, 0)}
|
||||
}
|
||||
|
||||
// Grant adds or replaces the entry for (identity, ns). If an entry
|
||||
// already exists for the same identity (matching Kind+ID) on the same
|
||||
// namespace, its Permissions are overwritten.
|
||||
func (a *ACL) Grant(identity Identity, ns string, perms Permission) {
|
||||
a.mu.Lock()
|
||||
defer a.mu.Unlock()
|
||||
for i, e := range a.entries {
|
||||
if e.Identity.Kind == identity.Kind && e.Identity.ID == identity.ID && e.Namespace == ns {
|
||||
a.entries[i].Permissions = perms
|
||||
return
|
||||
}
|
||||
}
|
||||
a.entries = append(a.entries, ACLEntry{
|
||||
Identity: identity,
|
||||
Namespace: ns,
|
||||
Permissions: perms,
|
||||
})
|
||||
}
|
||||
|
||||
// Revoke removes the entry for (identity, ns) if present. Revoking a
|
||||
// non-existent entry is a no-op.
|
||||
func (a *ACL) Revoke(identity Identity, ns string) {
|
||||
a.mu.Lock()
|
||||
defer a.mu.Unlock()
|
||||
for i, e := range a.entries {
|
||||
if e.Identity.Kind == identity.Kind && e.Identity.ID == identity.ID && e.Namespace == ns {
|
||||
a.entries = append(a.entries[:i], a.entries[i+1:]...)
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Check reports whether identity has perm on ns. Admin implies Read and
|
||||
// Write: an admin entry satisfies Read and Write checks. Returns false
|
||||
// (deny-by-default) if no entry matches. KindToken always denies
|
||||
// (R-021: no Orca-issued tokens); existing acl.json entries with
|
||||
// KindToken are inert.
|
||||
func (a *ACL) Check(identity Identity, ns string, perm Permission) bool {
|
||||
if identity.Kind == KindToken {
|
||||
return false
|
||||
}
|
||||
a.mu.RLock()
|
||||
defer a.mu.RUnlock()
|
||||
for _, e := range a.entries {
|
||||
if e.Identity.Kind == KindToken {
|
||||
continue
|
||||
}
|
||||
if e.Identity.Kind != identity.Kind || e.Identity.ID != identity.ID || e.Namespace != ns {
|
||||
continue
|
||||
}
|
||||
if e.Permissions&perm != 0 {
|
||||
return true
|
||||
}
|
||||
if e.Permissions&PermAdmin != 0 && (perm == PermRead || perm == PermWrite) {
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// List returns a copy of all entries. The slice is safe to mutate.
|
||||
func (a *ACL) List() []ACLEntry {
|
||||
a.mu.RLock()
|
||||
defer a.mu.RUnlock()
|
||||
out := make([]ACLEntry, len(a.entries))
|
||||
copy(out, a.entries)
|
||||
return out
|
||||
}
|
||||
|
||||
// SpiffeNamespace extracts the namespace from a SPIFFE URI of the
|
||||
// form spiffe://<trust-domain>/ns/<ns>/sa/<sa>/<alloc-id>. It accepts
|
||||
// any trust domain (the caller is expected to have verified the SVID
|
||||
// against the expected trust domain via identity.VerifySVID). Returns
|
||||
// an error if the URI is not a valid spiffe:// URI or the path does
|
||||
// not match the ns/<ns>/sa/<sa>/<alloc-id> shape.
|
||||
func SpiffeNamespace(uri string) (string, error) {
|
||||
u, err := url.Parse(uri)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("acl: parse spiffe uri: %w", err)
|
||||
}
|
||||
if u.Scheme != "spiffe" {
|
||||
return "", fmt.Errorf("acl: not a spiffe uri: %q", uri)
|
||||
}
|
||||
parts := strings.Split(strings.TrimPrefix(u.Path, "/"), "/")
|
||||
if len(parts) != 5 || parts[0] != "ns" || parts[2] != "sa" {
|
||||
return "", fmt.Errorf("acl: malformed spiffe path %q", u.Path)
|
||||
}
|
||||
if parts[1] == "" {
|
||||
return "", fmt.Errorf("acl: empty namespace in spiffe path %q", u.Path)
|
||||
}
|
||||
return parts[1], nil
|
||||
}
|
||||
|
||||
// OIDCClaims holds the verified claims from an OIDC ID token used by
|
||||
// the ACL layer. The Subject (sub) is the stable user identifier;
|
||||
// Groups are the group memberships used to match group-based grants.
|
||||
type OIDCClaims struct {
|
||||
Subject string
|
||||
Groups []string
|
||||
}
|
||||
|
||||
// OidcIdentity builds an Identity from verified OIDC claims. The ID
|
||||
// is the OIDC subject (sub). The Namespace is empty (OIDC identities
|
||||
// are not namespace-scoped at the identity layer; the ACL check takes
|
||||
// the namespace as a separate argument).
|
||||
func OidcIdentity(claims OIDCClaims) Identity {
|
||||
return Identity{
|
||||
Kind: KindOidc,
|
||||
ID: claims.Subject,
|
||||
}
|
||||
}
|
||||
|
||||
// OidcGroupIdentity builds an Identity for a group-based grant. The
|
||||
// ID is the group name prefixed with "group:". This allows ACL
|
||||
// entries to grant permissions to a group (e.g. "orca-admins") and
|
||||
// any OIDC user with that group inherits the permission.
|
||||
func OidcGroupIdentity(group string) Identity {
|
||||
return Identity{
|
||||
Kind: KindOidc,
|
||||
ID: "group:" + group,
|
||||
}
|
||||
}
|
||||
|
||||
// CheckOidc reports whether an OIDC user (by sub + groups) has perm
|
||||
// on ns. It checks both the user's own entry (by sub) and any group
|
||||
// entries (by group: prefix). Admin implies Read + Write.
|
||||
func (a *ACL) CheckOidc(claims OIDCClaims, ns string, perm Permission) bool {
|
||||
// First check the user's own entry.
|
||||
if a.Check(OidcIdentity(claims), ns, perm) {
|
||||
return true
|
||||
}
|
||||
// Then check each group entry.
|
||||
for _, g := range claims.Groups {
|
||||
if a.Check(OidcGroupIdentity(g), ns, perm) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// CheckTokenDeprecated is a stub that always returns false. KindToken
|
||||
// is deprecated (R-021); this ensures any existing KindToken entries in
|
||||
// acl.json are inert. P07 removes them; P22 migrates.
|
||||
func (a *ACL) CheckTokenDeprecated(tokenID, ns string, perm Permission) bool {
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,295 @@
|
||||
package acl
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sync"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestGrantAndCheck(t *testing.T) {
|
||||
a := NewACL()
|
||||
id := Identity{Kind: KindOidc, ID: "tok-A", Namespace: "test"}
|
||||
a.Grant(id, "test", PermRead)
|
||||
if !a.Check(id, "test", PermRead) {
|
||||
t.Errorf("Check(Read) = false, want true after Grant(Read)")
|
||||
}
|
||||
if a.Check(id, "test", PermWrite) {
|
||||
t.Errorf("Check(Write) = true, want false (only Read granted)")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRevoke(t *testing.T) {
|
||||
a := NewACL()
|
||||
id := Identity{Kind: KindOidc, ID: "tok-A", Namespace: "test"}
|
||||
a.Grant(id, "test", PermRead)
|
||||
a.Revoke(id, "test")
|
||||
if a.Check(id, "test", PermRead) {
|
||||
t.Errorf("Check(Read) = true after Revoke, want false")
|
||||
}
|
||||
if got := a.List(); len(got) != 0 {
|
||||
t.Errorf("List() len = %d after Revoke, want 0", len(got))
|
||||
}
|
||||
}
|
||||
|
||||
func TestRevokeNonExistentNoOp(t *testing.T) {
|
||||
a := NewACL()
|
||||
id := Identity{Kind: KindOidc, ID: "tok-A", Namespace: "test"}
|
||||
a.Revoke(id, "ghost")
|
||||
if got := a.List(); len(got) != 0 {
|
||||
t.Errorf("List() len = %d after no-op Revoke, want 0", len(got))
|
||||
}
|
||||
}
|
||||
|
||||
func TestDenyByDefault(t *testing.T) {
|
||||
a := NewACL()
|
||||
id := Identity{Kind: KindOidc, ID: "tok-A", Namespace: "test"}
|
||||
if a.Check(id, "test", PermRead) {
|
||||
t.Errorf("Check on un-granted identity = true, want false (deny-by-default)")
|
||||
}
|
||||
if a.Check(id, "test", PermWrite) {
|
||||
t.Errorf("Check Write on un-granted identity = true, want false")
|
||||
}
|
||||
if a.Check(id, "test", PermAdmin) {
|
||||
t.Errorf("Check Admin on un-granted identity = true, want false")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNamespaceIsolation(t *testing.T) {
|
||||
a := NewACL()
|
||||
id := Identity{Kind: KindOidc, ID: "tok-A", Namespace: "ns-A"}
|
||||
a.Grant(id, "ns-A", PermRead)
|
||||
if !a.Check(id, "ns-A", PermRead) {
|
||||
t.Errorf("Check on ns-A = false, want true")
|
||||
}
|
||||
if a.Check(id, "ns-B", PermRead) {
|
||||
t.Errorf("Check on ns-B = true, want false (namespace isolation)")
|
||||
}
|
||||
}
|
||||
|
||||
func TestGrantReplacesPermissions(t *testing.T) {
|
||||
a := NewACL()
|
||||
id := Identity{Kind: KindOidc, ID: "tok-A", Namespace: "test"}
|
||||
a.Grant(id, "test", PermRead)
|
||||
a.Grant(id, "test", PermWrite)
|
||||
if a.Check(id, "test", PermRead) {
|
||||
t.Errorf("Check(Read) = true after re-grant with Write-only, want false")
|
||||
}
|
||||
if !a.Check(id, "test", PermWrite) {
|
||||
t.Errorf("Check(Write) = false after re-grant, want true")
|
||||
}
|
||||
if got := a.List(); len(got) != 1 {
|
||||
t.Errorf("List() len = %d, want 1 (grant replaces, not appends)", len(got))
|
||||
}
|
||||
}
|
||||
|
||||
func TestSpiffeNamespace(t *testing.T) {
|
||||
got, err := SpiffeNamespace("spiffe://orca.local/ns/myapp/sa/svc1/alloc-123")
|
||||
if err != nil {
|
||||
t.Fatalf("SpiffeNamespace: %v", err)
|
||||
}
|
||||
if got != "myapp" {
|
||||
t.Errorf("SpiffeNamespace = %q, want %q", got, "myapp")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSpiffeNamespace_OtherTrustDomain(t *testing.T) {
|
||||
got, err := SpiffeNamespace("spiffe://example.com/ns/prod/sa/api/0")
|
||||
if err != nil {
|
||||
t.Fatalf("SpiffeNamespace: %v", err)
|
||||
}
|
||||
if got != "prod" {
|
||||
t.Errorf("SpiffeNamespace = %q, want %q", got, "prod")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSpiffeNamespace_Malformed(t *testing.T) {
|
||||
cases := []string{
|
||||
"https://orca.local/ns/prod/sa/api/0",
|
||||
"spiffe://orca.local/ns/prod/api/0",
|
||||
"spiffe://orca.local/ns/prod/sa/api",
|
||||
"spiffe://orca.local/ns//sa/api/0",
|
||||
":::not-a-uri",
|
||||
}
|
||||
for _, c := range cases {
|
||||
if _, err := SpiffeNamespace(c); err == nil {
|
||||
t.Errorf("SpiffeNamespace(%q): expected error, got nil", c)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestPermissionsDistinct(t *testing.T) {
|
||||
if PermRead == PermWrite || PermRead == PermAdmin || PermWrite == PermAdmin {
|
||||
t.Errorf("permission flags collide: read=%d write=%d admin=%d", PermRead, PermWrite, PermAdmin)
|
||||
}
|
||||
a := NewACL()
|
||||
id := Identity{Kind: KindOidc, ID: "tok-A", Namespace: "test"}
|
||||
a.Grant(id, "test", PermRead|PermWrite)
|
||||
if !a.Check(id, "test", PermRead) {
|
||||
t.Errorf("Check(Read) for read+write grant = false, want true")
|
||||
}
|
||||
if !a.Check(id, "test", PermWrite) {
|
||||
t.Errorf("Check(Write) for read+write grant = false, want true")
|
||||
}
|
||||
if a.Check(id, "test", PermAdmin) {
|
||||
t.Errorf("Check(Admin) for read+write grant = true, want false")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdminImpliesReadAndWrite(t *testing.T) {
|
||||
a := NewACL()
|
||||
id := Identity{Kind: KindOidc, ID: "tok-A", Namespace: "test"}
|
||||
a.Grant(id, "test", PermAdmin)
|
||||
if !a.Check(id, "test", PermAdmin) {
|
||||
t.Errorf("Check(Admin) = false, want true")
|
||||
}
|
||||
if !a.Check(id, "test", PermRead) {
|
||||
t.Errorf("Check(Read) for admin grant = false, want true (admin implies read)")
|
||||
}
|
||||
if !a.Check(id, "test", PermWrite) {
|
||||
t.Errorf("Check(Write) for admin grant = false, want true (admin implies write)")
|
||||
}
|
||||
}
|
||||
|
||||
func TestConcurrentAccess(t *testing.T) {
|
||||
a := NewACL()
|
||||
id := Identity{Kind: KindOidc, ID: "tok-concurrent", Namespace: "ns"}
|
||||
const n = 200
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(n * 3)
|
||||
for i := 0; i < n; i++ {
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
a.Grant(id, "ns", PermRead|PermWrite)
|
||||
}()
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
a.Check(id, "ns", PermRead)
|
||||
}()
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
a.List()
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
if !a.Check(id, "ns", PermRead) {
|
||||
t.Errorf("Check(Read) after concurrent grants = false, want true")
|
||||
}
|
||||
if got := a.List(); len(got) != 1 {
|
||||
t.Errorf("List() len = %d, want 1 (concurrent grants replace, not append)", len(got))
|
||||
}
|
||||
}
|
||||
|
||||
func TestListIsCopy(t *testing.T) {
|
||||
a := NewACL()
|
||||
id := Identity{Kind: KindOidc, ID: "tok-A", Namespace: "test"}
|
||||
a.Grant(id, "test", PermRead)
|
||||
lst := a.List()
|
||||
lst[0].Permissions = PermAdmin
|
||||
if a.Check(id, "test", PermAdmin) {
|
||||
t.Errorf("mutating List() result leaked into ACL: %v", a.List())
|
||||
}
|
||||
}
|
||||
|
||||
func TestSpiffeIdentityGrant(t *testing.T) {
|
||||
a := NewACL()
|
||||
uri := "spiffe://orca.local/ns/myapp/sa/svc1/alloc-123"
|
||||
ns, err := SpiffeNamespace(uri)
|
||||
if err != nil {
|
||||
t.Fatalf("SpiffeNamespace: %v", err)
|
||||
}
|
||||
id := Identity{Kind: KindSpiffe, ID: uri, Namespace: ns}
|
||||
a.Grant(id, ns, PermRead|PermWrite)
|
||||
if !a.Check(id, ns, PermRead) || !a.Check(id, ns, PermWrite) {
|
||||
t.Errorf("spiffe identity check failed for ns=%s", ns)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTokenAndSpiffeIdentitiesIndependent(t *testing.T) {
|
||||
a := NewACL()
|
||||
uri := "spiffe://orca.local/ns/prod/sa/api/0"
|
||||
spiffeID := Identity{Kind: KindSpiffe, ID: uri, Namespace: "prod"}
|
||||
tokenID := Identity{Kind: KindOidc, ID: "operator-1", Namespace: "prod"}
|
||||
a.Grant(spiffeID, "prod", PermRead)
|
||||
if a.Check(tokenID, "prod", PermRead) {
|
||||
t.Errorf("token identity matched spiffe grant (kind isolation broken)")
|
||||
}
|
||||
if !a.Check(spiffeID, "prod", PermRead) {
|
||||
t.Errorf("spiffe identity check failed")
|
||||
}
|
||||
if got := a.List(); len(got) != 1 {
|
||||
t.Errorf("List() len = %d, want 1", len(got))
|
||||
}
|
||||
}
|
||||
|
||||
func TestAllPermissionsConstant(t *testing.T) {
|
||||
if AllPermissions != PermRead|PermWrite|PermAdmin {
|
||||
t.Errorf("AllPermissions = %d, want %d", AllPermissions, PermRead|PermWrite|PermAdmin)
|
||||
}
|
||||
}
|
||||
|
||||
func ExampleSpiffeNamespace() {
|
||||
ns, _ := SpiffeNamespace("spiffe://orca.local/ns/myapp/sa/svc1/alloc-123")
|
||||
fmt.Println(ns)
|
||||
// Output: myapp
|
||||
}
|
||||
|
||||
// --- REQ-145 / F1 ACL OIDC rewrite tests ---
|
||||
|
||||
// TestACLOidcUserGrant verifies an OIDC user (by sub) can be granted
|
||||
// and checked.
|
||||
func TestACLOidcUserGrant(t *testing.T) {
|
||||
a := NewACL()
|
||||
claims := OIDCClaims{Subject: "user-1", Groups: []string{"devs"}}
|
||||
a.Grant(OidcIdentity(claims), "prod", PermWrite|PermRead)
|
||||
if !a.CheckOidc(claims, "prod", PermWrite) {
|
||||
t.Error("CheckOidc should allow write")
|
||||
}
|
||||
if !a.CheckOidc(claims, "prod", PermRead) {
|
||||
t.Error("CheckOidc should allow read (explicit)")
|
||||
}
|
||||
if a.CheckOidc(claims, "prod", PermAdmin) {
|
||||
t.Error("CheckOidc should deny admin")
|
||||
}
|
||||
if a.CheckOidc(claims, "other", PermRead) {
|
||||
t.Error("CheckOidc should deny on wrong ns")
|
||||
}
|
||||
}
|
||||
|
||||
// TestACLOidcGroupGrant verifies group-based grants work.
|
||||
func TestACLOidcGroupGrant(t *testing.T) {
|
||||
a := NewACL()
|
||||
a.Grant(OidcGroupIdentity("orca-admins"), "prod", PermAdmin)
|
||||
claims := OIDCClaims{Subject: "user-2", Groups: []string{"orca-admins"}}
|
||||
if !a.CheckOidc(claims, "prod", PermAdmin) {
|
||||
t.Error("admin group should have admin")
|
||||
}
|
||||
if !a.CheckOidc(claims, "prod", PermWrite) {
|
||||
t.Error("admin implies write")
|
||||
}
|
||||
claimsNoGroup := OIDCClaims{Subject: "user-3", Groups: []string{"devs"}}
|
||||
if a.CheckOidc(claimsNoGroup, "prod", PermRead) {
|
||||
t.Error("non-admin group should deny")
|
||||
}
|
||||
}
|
||||
|
||||
// TestACLOidcDenyByDefault verifies an ungranted OIDC user is denied.
|
||||
func TestACLOidcDenyByDefault(t *testing.T) {
|
||||
a := NewACL()
|
||||
claims := OIDCClaims{Subject: "nobody"}
|
||||
if a.CheckOidc(claims, "prod", PermRead) {
|
||||
t.Error("ungranted user should deny")
|
||||
}
|
||||
}
|
||||
|
||||
// TestACLTokenDeprecated verifies KindToken always denies (R-021).
|
||||
func TestACLTokenDeprecated(t *testing.T) {
|
||||
a := NewACL()
|
||||
// Even if an old acl.json has a KindToken entry, Check returns false.
|
||||
a.Grant(Identity{Kind: KindToken, ID: "old-token-123"}, "prod", PermAdmin)
|
||||
if a.Check(Identity{Kind: KindToken, ID: "old-token-123"}, "prod", PermRead) {
|
||||
t.Error("KindToken should always deny (R-021)")
|
||||
}
|
||||
if a.Check(Identity{Kind: KindToken, ID: "old-token-123"}, "prod", PermAdmin) {
|
||||
t.Error("KindToken should always deny even admin (R-021)")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,369 @@
|
||||
// Package backup implements orca's signed tarball backup/restore
|
||||
// subsystem (P04, v0.11 milestone).
|
||||
//
|
||||
// The model is tar.gz + HMAC-SHA256 signature:
|
||||
//
|
||||
// - Backup walks the ORCA_HOME recursively, excludes ephemeral
|
||||
// paths (/run/orca/*), unix sockets (*.sock), and SQLite WAL/SHM
|
||||
// sidecars (*.db-wal, *.db-shm), packs the rest into a tar.gz, and
|
||||
// computes an HMAC-SHA256 of the tarball using the cluster master
|
||||
// key. The tarball is written to OutputPath; the hex-encoded
|
||||
// signature to OutputPath + ".sig".
|
||||
// - VerifySignature recomputes the HMAC and compares it (constant
|
||||
// time) against the recorded signature.
|
||||
// - Restore verifies the signature first (refuses on mismatch), then
|
||||
// extracts the tarball to TargetDir. With Force=false it refuses to
|
||||
// clobber a non-empty TargetDir; with Force=true it overwrites.
|
||||
//
|
||||
// The package never logs key material. slog calls carry only metadata.
|
||||
package backup
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"compress/gzip"
|
||||
"crypto/hmac"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// ErrSignatureMismatch is returned when the recorded HMAC-SHA256
|
||||
// signature does not match the recomputed one (tampering or wrong key).
|
||||
var ErrSignatureMismatch = errors.New("backup: signature mismatch")
|
||||
|
||||
// ErrTargetNotEmpty is returned when Restore is called with Force=false
|
||||
// against a non-empty TargetDir.
|
||||
var ErrTargetNotEmpty = errors.New("backup: target directory not empty (use Force to overwrite)")
|
||||
|
||||
// BackupOptions configures Backup.
|
||||
type BackupOptions struct {
|
||||
SourceDir string // ORCA_HOME — the tree to back up
|
||||
OutputPath string // destination tarball path (.tar.gz)
|
||||
MasterKey []byte // HMAC-SHA256 key (cluster master.key)
|
||||
}
|
||||
|
||||
// RestoreOptions configures Restore.
|
||||
type RestoreOptions struct {
|
||||
InputPath string // source tarball path (.tar.gz)
|
||||
TargetDir string // destination ORCA_HOME
|
||||
MasterKey []byte // HMAC-SHA256 key (for verification)
|
||||
Force bool // overwrite non-empty target
|
||||
}
|
||||
|
||||
// excludeGlobSuffixes are the suffixes excluded from the backup. We
|
||||
// exclude SQLite WAL/SHM sidecars (the main db is backed up) and unix
|
||||
// sockets.
|
||||
var excludeGlobSuffixes = []string{".sock", ".db-wal", ".db-shm"}
|
||||
|
||||
// shouldExclude reports whether a path should be excluded from the
|
||||
// backup. It excludes /run/orca/* (ephemeral runtime), *.sock, *.db-wal,
|
||||
// and *.db-shm. The /run/orca match is done on the absolute path; the
|
||||
// suffix matches are done on the base name.
|
||||
func shouldExclude(absPath string) bool {
|
||||
clean := filepath.Clean(absPath)
|
||||
if strings.HasPrefix(clean, "/run/orca/") || clean == "/run/orca" {
|
||||
return true
|
||||
}
|
||||
if idx := strings.LastIndex(clean, string(filepath.Separator)+"run"+string(filepath.Separator)+"orca"+string(filepath.Separator)); idx >= 0 {
|
||||
return true
|
||||
}
|
||||
if strings.HasSuffix(clean, string(filepath.Separator)+"run"+string(filepath.Separator)+"orca") {
|
||||
return true
|
||||
}
|
||||
base := filepath.Base(clean)
|
||||
for _, suf := range excludeGlobSuffixes {
|
||||
if strings.HasSuffix(base, suf) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Backup creates a signed tar.gz of SourceDir. The tarball is written
|
||||
// to OutputPath and the hex-encoded HMAC-SHA256 signature to
|
||||
// OutputPath + ".sig". The write is atomic: the tarball is streamed to
|
||||
// a temp file in the same directory and renamed on success; the
|
||||
// signature is written after the rename so a crash never leaves a
|
||||
// tarball with a stale or missing signature.
|
||||
func Backup(opts BackupOptions) error {
|
||||
if opts.SourceDir == "" {
|
||||
return fmt.Errorf("backup: SourceDir is empty")
|
||||
}
|
||||
if opts.OutputPath == "" {
|
||||
return fmt.Errorf("backup: OutputPath is empty")
|
||||
}
|
||||
if len(opts.MasterKey) == 0 {
|
||||
return fmt.Errorf("backup: MasterKey is empty")
|
||||
}
|
||||
|
||||
src, err := filepath.Abs(opts.SourceDir)
|
||||
if err != nil {
|
||||
return fmt.Errorf("backup: resolve SourceDir: %w", err)
|
||||
}
|
||||
info, err := os.Stat(src)
|
||||
if err != nil {
|
||||
return fmt.Errorf("backup: stat SourceDir: %w", err)
|
||||
}
|
||||
if !info.IsDir() {
|
||||
return fmt.Errorf("backup: SourceDir %q is not a directory", src)
|
||||
}
|
||||
|
||||
outDir := filepath.Dir(opts.OutputPath)
|
||||
if err := os.MkdirAll(outDir, 0o755); err != nil {
|
||||
return fmt.Errorf("backup: mkdir output dir: %w", err)
|
||||
}
|
||||
|
||||
tmp, err := os.CreateTemp(outDir, ".orca-backup-*.tar.gz.tmp")
|
||||
if err != nil {
|
||||
return fmt.Errorf("backup: create temp tarball: %w", err)
|
||||
}
|
||||
tmpPath := tmp.Name()
|
||||
defer func() {
|
||||
tmp.Close()
|
||||
_ = os.Remove(tmpPath)
|
||||
}()
|
||||
|
||||
gw := gzip.NewWriter(tmp)
|
||||
tw := tar.NewWriter(gw)
|
||||
|
||||
var walked int
|
||||
walkErr := filepath.Walk(src, func(path string, fi os.FileInfo, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if shouldExclude(path) {
|
||||
if fi.IsDir() {
|
||||
return filepath.SkipDir
|
||||
}
|
||||
return nil
|
||||
}
|
||||
rel, rerr := filepath.Rel(src, path)
|
||||
if rerr != nil {
|
||||
return fmt.Errorf("rel path %s: %w", path, rerr)
|
||||
}
|
||||
if rel == "." {
|
||||
return nil
|
||||
}
|
||||
hdr, herr := tar.FileInfoHeader(fi, "")
|
||||
if herr != nil {
|
||||
return fmt.Errorf("tar header for %s: %w", path, herr)
|
||||
}
|
||||
hdr.Name = filepath.ToSlash(rel)
|
||||
if err := tw.WriteHeader(hdr); err != nil {
|
||||
return fmt.Errorf("write header %s: %w", rel, err)
|
||||
}
|
||||
if !fi.Mode().IsRegular() {
|
||||
return nil
|
||||
}
|
||||
f, oerr := os.Open(path)
|
||||
if oerr != nil {
|
||||
return fmt.Errorf("open %s: %w", path, oerr)
|
||||
}
|
||||
defer f.Close()
|
||||
if _, err := io.Copy(tw, f); err != nil {
|
||||
return fmt.Errorf("copy %s: %w", rel, err)
|
||||
}
|
||||
walked++
|
||||
return nil
|
||||
})
|
||||
if walkErr != nil {
|
||||
tw.Close()
|
||||
gw.Close()
|
||||
return fmt.Errorf("backup: walk: %w", walkErr)
|
||||
}
|
||||
if err := tw.Close(); err != nil {
|
||||
gw.Close()
|
||||
return fmt.Errorf("backup: close tar writer: %w", err)
|
||||
}
|
||||
if err := gw.Close(); err != nil {
|
||||
return fmt.Errorf("backup: close gzip writer: %w", err)
|
||||
}
|
||||
if err := tmp.Close(); err != nil {
|
||||
return fmt.Errorf("backup: close temp tarball: %w", err)
|
||||
}
|
||||
|
||||
if err := os.Rename(tmpPath, opts.OutputPath); err != nil {
|
||||
return fmt.Errorf("backup: rename tarball: %w", err)
|
||||
}
|
||||
|
||||
sig, err := computeSignature(opts.OutputPath, opts.MasterKey)
|
||||
if err != nil {
|
||||
return fmt.Errorf("backup: compute signature: %w", err)
|
||||
}
|
||||
if err := os.WriteFile(opts.OutputPath+".sig", []byte(hex.EncodeToString(sig)), 0o644); err != nil {
|
||||
return fmt.Errorf("backup: write signature: %w", err)
|
||||
}
|
||||
|
||||
slog.Info("backup complete", "path", opts.OutputPath, "files", walked, "sig", opts.OutputPath+".sig")
|
||||
return nil
|
||||
}
|
||||
|
||||
// computeSignature reads the file at path and returns its HMAC-SHA256
|
||||
// MAC under key.
|
||||
func computeSignature(path string, key []byte) ([]byte, error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("open %s: %w", path, err)
|
||||
}
|
||||
defer f.Close()
|
||||
mac := hmac.New(sha256.New, key)
|
||||
if _, err := io.Copy(mac, f); err != nil {
|
||||
return nil, fmt.Errorf("hash %s: %w", path, err)
|
||||
}
|
||||
return mac.Sum(nil), nil
|
||||
}
|
||||
|
||||
// VerifySignature recomputes the HMAC-SHA256 of the tarball at
|
||||
// tarballPath and compares it (constant time) against the hex-encoded
|
||||
// signature at sigPath. Returns ErrSignatureMismatch on a mismatch.
|
||||
func VerifySignature(tarballPath, sigPath string, masterKey []byte) error {
|
||||
if len(masterKey) == 0 {
|
||||
return fmt.Errorf("backup: MasterKey is empty")
|
||||
}
|
||||
got, err := computeSignature(tarballPath, masterKey)
|
||||
if err != nil {
|
||||
return fmt.Errorf("compute signature: %w", err)
|
||||
}
|
||||
wantHex, err := os.ReadFile(sigPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read signature: %w", err)
|
||||
}
|
||||
want, err := hex.DecodeString(strings.TrimSpace(string(wantHex)))
|
||||
if err != nil {
|
||||
return fmt.Errorf("decode signature: %w", err)
|
||||
}
|
||||
if !hmac.Equal(got, want) {
|
||||
return ErrSignatureMismatch
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Restore verifies the signature on (InputPath, InputPath+".sig") using
|
||||
// MasterKey, then extracts the tarball to TargetDir. With Force=false
|
||||
// a non-empty TargetDir is refused (ErrTargetNotEmpty); with Force=true
|
||||
// existing files are overwritten.
|
||||
func Restore(opts RestoreOptions) error {
|
||||
if opts.InputPath == "" {
|
||||
return fmt.Errorf("restore: InputPath is empty")
|
||||
}
|
||||
if opts.TargetDir == "" {
|
||||
return fmt.Errorf("restore: TargetDir is empty")
|
||||
}
|
||||
if len(opts.MasterKey) == 0 {
|
||||
return fmt.Errorf("restore: MasterKey is empty")
|
||||
}
|
||||
|
||||
sigPath := opts.InputPath + ".sig"
|
||||
if err := VerifySignature(opts.InputPath, sigPath, opts.MasterKey); err != nil {
|
||||
return fmt.Errorf("restore: verify signature: %w", err)
|
||||
}
|
||||
|
||||
target := filepath.Clean(opts.TargetDir)
|
||||
if err := os.MkdirAll(target, 0o755); err != nil {
|
||||
return fmt.Errorf("restore: mkdir target: %w", err)
|
||||
}
|
||||
if !opts.Force {
|
||||
empty, err := dirIsEmpty(target)
|
||||
if err != nil {
|
||||
return fmt.Errorf("restore: check target: %w", err)
|
||||
}
|
||||
if !empty {
|
||||
return ErrTargetNotEmpty
|
||||
}
|
||||
}
|
||||
|
||||
f, err := os.Open(opts.InputPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("restore: open tarball: %w", err)
|
||||
}
|
||||
defer f.Close()
|
||||
gz, err := gzip.NewReader(f)
|
||||
if err != nil {
|
||||
return fmt.Errorf("restore: gzip reader: %w", err)
|
||||
}
|
||||
defer gz.Close()
|
||||
tr := tar.NewReader(gz)
|
||||
var extracted int
|
||||
for {
|
||||
hdr, err := tr.Next()
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
return fmt.Errorf("restore: read tar entry: %w", err)
|
||||
}
|
||||
name := filepath.FromSlash(hdr.Name)
|
||||
if strings.HasPrefix(name, "/") || strings.HasPrefix(name, "..") {
|
||||
return fmt.Errorf("restore: unsafe path %q", hdr.Name)
|
||||
}
|
||||
dest := filepath.Join(target, name)
|
||||
switch hdr.Typeflag {
|
||||
case tar.TypeDir:
|
||||
if err := os.MkdirAll(dest, os.FileMode(hdr.Mode)); err != nil {
|
||||
return fmt.Errorf("restore: mkdir %s: %w", name, err)
|
||||
}
|
||||
continue
|
||||
case tar.TypeSymlink:
|
||||
// REQ-127 / F7: validate Linkname to prevent symlink attacks.
|
||||
// Reject absolute links, .. traversal, and links outside
|
||||
// the target dir (which could point to /etc/shadow etc.).
|
||||
link := hdr.Linkname
|
||||
if link == "" {
|
||||
return fmt.Errorf("restore: empty symlink linkname for %q", name)
|
||||
}
|
||||
if strings.HasPrefix(link, "/") {
|
||||
return fmt.Errorf("restore: symlink %q has absolute linkname %q (REQ-127: path traversal)", name, link)
|
||||
}
|
||||
if strings.Contains(link, "..") {
|
||||
// Resolve the link relative to the dest dir; if it
|
||||
// escapes the target, reject.
|
||||
linkDest := filepath.Join(filepath.Dir(dest), link)
|
||||
linkClean := filepath.Clean(linkDest)
|
||||
targetClean := filepath.Clean(target)
|
||||
if !strings.HasPrefix(linkClean, targetClean+string(filepath.Separator)) && linkClean != targetClean {
|
||||
return fmt.Errorf("restore: symlink %q linkname %q escapes target (REQ-127)", name, link)
|
||||
}
|
||||
}
|
||||
if err := os.Remove(dest); err != nil && !os.IsNotExist(err) {
|
||||
return fmt.Errorf("restore: clear symlink %s: %w", name, err)
|
||||
}
|
||||
if err := os.Symlink(hdr.Linkname, dest); err != nil {
|
||||
return fmt.Errorf("restore: symlink %s: %w", name, err)
|
||||
}
|
||||
continue
|
||||
case tar.TypeReg:
|
||||
if err := os.MkdirAll(filepath.Dir(dest), 0o755); err != nil {
|
||||
return fmt.Errorf("restore: mkdir parent %s: %w", name, err)
|
||||
}
|
||||
out, err := os.OpenFile(dest, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, os.FileMode(hdr.Mode))
|
||||
if err != nil {
|
||||
return fmt.Errorf("restore: create %s: %w", name, err)
|
||||
}
|
||||
if _, err := io.Copy(out, tr); err != nil {
|
||||
out.Close()
|
||||
return fmt.Errorf("restore: write %s: %w", name, err)
|
||||
}
|
||||
out.Close()
|
||||
extracted++
|
||||
default:
|
||||
slog.Warn("restore: skipping non-regular entry", "name", name, "type", hdr.Typeflag)
|
||||
}
|
||||
}
|
||||
slog.Info("restore complete", "path", target, "files", extracted)
|
||||
return nil
|
||||
}
|
||||
|
||||
// dirIsEmpty reports whether dir contains no entries.
|
||||
func dirIsEmpty(dir string) (bool, error) {
|
||||
entries, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
return len(entries) == 0, nil
|
||||
}
|
||||
@@ -0,0 +1,413 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"crypto/hmac"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func keyA() []byte { return []byte("0123456789abcdef0123456789abcdef") }
|
||||
func keyB() []byte { return []byte("abcdef0123456789abcdef0123456789") }
|
||||
|
||||
func writeFiles(t *testing.T, root string, files map[string]string) {
|
||||
t.Helper()
|
||||
for name, body := range files {
|
||||
p := filepath.Join(root, name)
|
||||
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
|
||||
t.Fatalf("mkdir %s: %v", filepath.Dir(p), err)
|
||||
}
|
||||
if err := os.WriteFile(p, []byte(body), 0o644); err != nil {
|
||||
t.Fatalf("write %s: %v", p, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func runBackup(t *testing.T, src, out string, key []byte) {
|
||||
t.Helper()
|
||||
if err := Backup(BackupOptions{
|
||||
SourceDir: src,
|
||||
OutputPath: out,
|
||||
MasterKey: key,
|
||||
}); err != nil {
|
||||
t.Fatalf("Backup: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackupRestoreRoundTrip(t *testing.T) {
|
||||
src := t.TempDir()
|
||||
out := filepath.Join(t.TempDir(), "b.tar.gz")
|
||||
target := t.TempDir()
|
||||
os.RemoveAll(target)
|
||||
|
||||
writeFiles(t, src, map[string]string{
|
||||
"cluster/master.key": "KEYMATERIAL",
|
||||
"_defaults/db/orca.db": "SQLITE",
|
||||
"_defaults/.env": "FOO=bar",
|
||||
"_defaults/jobs/job1.md": "job body",
|
||||
"cluster/peers/host1/peer.json": "{}",
|
||||
})
|
||||
|
||||
runBackup(t, src, out, keyA())
|
||||
|
||||
if _, err := os.Stat(out + ".sig"); err != nil {
|
||||
t.Fatalf("sig file missing: %v", err)
|
||||
}
|
||||
|
||||
if err := Restore(RestoreOptions{
|
||||
InputPath: out,
|
||||
TargetDir: target,
|
||||
MasterKey: keyA(),
|
||||
}); err != nil {
|
||||
t.Fatalf("Restore: %v", err)
|
||||
}
|
||||
|
||||
for name, body := range map[string]string{
|
||||
"cluster/master.key": "KEYMATERIAL",
|
||||
"_defaults/db/orca.db": "SQLITE",
|
||||
"_defaults/.env": "FOO=bar",
|
||||
"_defaults/jobs/job1.md": "job body",
|
||||
"cluster/peers/host1/peer.json": "{}",
|
||||
} {
|
||||
got, err := os.ReadFile(filepath.Join(target, name))
|
||||
if err != nil {
|
||||
t.Errorf("restored file %s missing: %v", name, err)
|
||||
continue
|
||||
}
|
||||
if string(got) != body {
|
||||
t.Errorf("restored %s = %q, want %q", name, string(got), body)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestVerifySignatureSameKey(t *testing.T) {
|
||||
src := t.TempDir()
|
||||
out := filepath.Join(t.TempDir(), "b.tar.gz")
|
||||
writeFiles(t, src, map[string]string{"a.txt": "hello"})
|
||||
runBackup(t, src, out, keyA())
|
||||
if err := VerifySignature(out, out+".sig", keyA()); err != nil {
|
||||
t.Fatalf("verify same key: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestVerifySignatureWrongKey(t *testing.T) {
|
||||
src := t.TempDir()
|
||||
out := filepath.Join(t.TempDir(), "b.tar.gz")
|
||||
writeFiles(t, src, map[string]string{"a.txt": "hello"})
|
||||
runBackup(t, src, out, keyA())
|
||||
err := VerifySignature(out, out+".sig", keyB())
|
||||
if !errors.Is(err, ErrSignatureMismatch) {
|
||||
t.Fatalf("verify wrong key: got %v, want ErrSignatureMismatch", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestVerifySignatureTampered(t *testing.T) {
|
||||
src := t.TempDir()
|
||||
out := filepath.Join(t.TempDir(), "b.tar.gz")
|
||||
writeFiles(t, src, map[string]string{"a.txt": "hello"})
|
||||
runBackup(t, src, out, keyA())
|
||||
|
||||
body, err := os.ReadFile(out)
|
||||
if err != nil {
|
||||
t.Fatalf("read tarball: %v", err)
|
||||
}
|
||||
body[0] ^= 0xff
|
||||
if err := os.WriteFile(out, body, 0o644); err != nil {
|
||||
t.Fatalf("rewrite tampered tarball: %v", err)
|
||||
}
|
||||
err = VerifySignature(out, out+".sig", keyA())
|
||||
if !errors.Is(err, ErrSignatureMismatch) {
|
||||
t.Fatalf("verify tampered: got %v, want ErrSignatureMismatch", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExclusionSocketsAndRunOrca(t *testing.T) {
|
||||
src := t.TempDir()
|
||||
out := filepath.Join(t.TempDir(), "b.tar.gz")
|
||||
target := t.TempDir()
|
||||
os.RemoveAll(target)
|
||||
|
||||
writeFiles(t, src, map[string]string{
|
||||
"keep.txt": "keep me",
|
||||
"normal.db": "main db",
|
||||
"sock-excluded.sock": "sock",
|
||||
"side.db-wal": "wal",
|
||||
"side.db-shm": "shm",
|
||||
})
|
||||
|
||||
runOrcaDir := filepath.Join(src, "run", "orca")
|
||||
if err := os.MkdirAll(runOrcaDir, 0o755); err != nil {
|
||||
t.Fatalf("mkdir run/orca: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(runOrcaDir, "ephemeral.txt"), []byte("eph"), 0o644); err != nil {
|
||||
t.Fatalf("write ephemeral: %v", err)
|
||||
}
|
||||
|
||||
runBackup(t, src, out, keyA())
|
||||
|
||||
if err := Restore(RestoreOptions{
|
||||
InputPath: out,
|
||||
TargetDir: target,
|
||||
MasterKey: keyA(),
|
||||
}); err != nil {
|
||||
t.Fatalf("Restore: %v", err)
|
||||
}
|
||||
|
||||
for _, excluded := range []string{
|
||||
"sock-excluded.sock",
|
||||
"side.db-wal",
|
||||
"side.db-shm",
|
||||
"run/orca/ephemeral.txt",
|
||||
} {
|
||||
if _, err := os.Stat(filepath.Join(target, excluded)); !os.IsNotExist(err) {
|
||||
t.Errorf("excluded file %s should not be in restore (err=%v)", excluded, err)
|
||||
}
|
||||
}
|
||||
for _, kept := range []string{"keep.txt", "normal.db"} {
|
||||
if _, err := os.Stat(filepath.Join(target, kept)); err != nil {
|
||||
t.Errorf("kept file %s missing from restore: %v", kept, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestoreForceFalseRefusesNonEmpty(t *testing.T) {
|
||||
src := t.TempDir()
|
||||
out := filepath.Join(t.TempDir(), "b.tar.gz")
|
||||
writeFiles(t, src, map[string]string{"a.txt": "hello"})
|
||||
runBackup(t, src, out, keyA())
|
||||
|
||||
target := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(target, "existing.txt"), []byte("x"), 0o644); err != nil {
|
||||
t.Fatalf("seed target: %v", err)
|
||||
}
|
||||
err := Restore(RestoreOptions{
|
||||
InputPath: out,
|
||||
TargetDir: target,
|
||||
MasterKey: keyA(),
|
||||
Force: false,
|
||||
})
|
||||
if !errors.Is(err, ErrTargetNotEmpty) {
|
||||
t.Fatalf("restore to non-empty: got %v, want ErrTargetNotEmpty", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestoreForceTrueOverwritesNonEmpty(t *testing.T) {
|
||||
src := t.TempDir()
|
||||
out := filepath.Join(t.TempDir(), "b.tar.gz")
|
||||
writeFiles(t, src, map[string]string{"a.txt": "new"})
|
||||
runBackup(t, src, out, keyA())
|
||||
|
||||
target := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(target, "stale.txt"), []byte("old"), 0o644); err != nil {
|
||||
t.Fatalf("seed target: %v", err)
|
||||
}
|
||||
err := Restore(RestoreOptions{
|
||||
InputPath: out,
|
||||
TargetDir: target,
|
||||
MasterKey: keyA(),
|
||||
Force: true,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("restore force: %v", err)
|
||||
}
|
||||
got, err := os.ReadFile(filepath.Join(target, "a.txt"))
|
||||
if err != nil {
|
||||
t.Fatalf("restored a.txt missing: %v", err)
|
||||
}
|
||||
if string(got) != "new" {
|
||||
t.Errorf("restored a.txt = %q, want %q", string(got), "new")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEmptyBackup(t *testing.T) {
|
||||
src := t.TempDir()
|
||||
out := filepath.Join(t.TempDir(), "b.tar.gz")
|
||||
target := t.TempDir()
|
||||
os.RemoveAll(target)
|
||||
|
||||
runBackup(t, src, out, keyA())
|
||||
|
||||
if err := VerifySignature(out, out+".sig", keyA()); err != nil {
|
||||
t.Fatalf("verify empty backup: %v", err)
|
||||
}
|
||||
if err := Restore(RestoreOptions{
|
||||
InputPath: out,
|
||||
TargetDir: target,
|
||||
MasterKey: keyA(),
|
||||
}); err != nil {
|
||||
t.Fatalf("restore empty backup: %v", err)
|
||||
}
|
||||
entries, err := os.ReadDir(target)
|
||||
if err != nil {
|
||||
t.Fatalf("read target: %v", err)
|
||||
}
|
||||
if len(entries) != 0 {
|
||||
t.Errorf("empty backup restored %d entries, want 0", len(entries))
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestoreSignatureMismatchFailsBeforeExtract(t *testing.T) {
|
||||
src := t.TempDir()
|
||||
out := filepath.Join(t.TempDir(), "b.tar.gz")
|
||||
writeFiles(t, src, map[string]string{"a.txt": "hello"})
|
||||
runBackup(t, src, out, keyA())
|
||||
|
||||
target := t.TempDir()
|
||||
os.RemoveAll(target)
|
||||
err := Restore(RestoreOptions{
|
||||
InputPath: out,
|
||||
TargetDir: target,
|
||||
MasterKey: keyB(),
|
||||
})
|
||||
if !errors.Is(err, ErrSignatureMismatch) {
|
||||
t.Fatalf("restore wrong key: got %v, want ErrSignatureMismatch", err)
|
||||
}
|
||||
if _, err := os.Stat(target); err == nil {
|
||||
entries, _ := os.ReadDir(target)
|
||||
if len(entries) != 0 {
|
||||
t.Errorf("target should be empty after failed verify, got %d entries", len(entries))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestoreBadSignatureContent(t *testing.T) {
|
||||
src := t.TempDir()
|
||||
out := filepath.Join(t.TempDir(), "b.tar.gz")
|
||||
writeFiles(t, src, map[string]string{"a.txt": "hello"})
|
||||
runBackup(t, src, out, keyA())
|
||||
|
||||
if err := os.WriteFile(out+".sig", []byte("not-hex!!"), 0o644); err != nil {
|
||||
t.Fatalf("write bad sig: %v", err)
|
||||
}
|
||||
err := VerifySignature(out, out+".sig", keyA())
|
||||
if err == nil {
|
||||
t.Fatal("verify bad sig content: expected error, got nil")
|
||||
}
|
||||
if strings.Contains(err.Error(), "decode signature") {
|
||||
return
|
||||
}
|
||||
if errors.Is(err, ErrSignatureMismatch) {
|
||||
return
|
||||
}
|
||||
t.Errorf("verify bad sig content: got unexpected err %v", err)
|
||||
}
|
||||
|
||||
func TestBackupSignatureFileContent(t *testing.T) {
|
||||
src := t.TempDir()
|
||||
out := filepath.Join(t.TempDir(), "b.tar.gz")
|
||||
writeFiles(t, src, map[string]string{"a.txt": "hello"})
|
||||
runBackup(t, src, out, keyA())
|
||||
sig, err := os.ReadFile(out + ".sig")
|
||||
if err != nil {
|
||||
t.Fatalf("read sig: %v", err)
|
||||
}
|
||||
if dec, err := hexDecode(string(bytes.TrimSpace(sig))); err != nil {
|
||||
t.Fatalf("sig not hex: %v", err)
|
||||
} else if len(dec) != 32 {
|
||||
t.Errorf("sig len = %d, want 32", len(dec))
|
||||
}
|
||||
}
|
||||
|
||||
func hexDecode(s string) ([]byte, error) {
|
||||
return hex.DecodeString(s)
|
||||
}
|
||||
|
||||
// --- REQ-127 / F7 backup symlink validation tests ---
|
||||
|
||||
// TestRestoreRejectsAbsoluteSymlink verifies a tarball with an absolute
|
||||
// symlink linkname is rejected.
|
||||
func TestRestoreRejectsAbsoluteSymlink(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
// Create a crafted tarball with an absolute symlink.
|
||||
tarPath := filepath.Join(dir, "evil.tar.gz")
|
||||
sigPath := tarPath + ".sig"
|
||||
if err := createCraftedTarball(tarPath, "link", "/etc/shadow"); err != nil {
|
||||
t.Fatalf("create tarball: %v", err)
|
||||
}
|
||||
// Create a valid signature (the signature verifies, but the symlink
|
||||
// validation should still reject the restore).
|
||||
key := make([]byte, 32)
|
||||
for i := range key {
|
||||
key[i] = byte(i)
|
||||
}
|
||||
mac := hmac.New(sha256.New, key)
|
||||
data, _ := os.ReadFile(tarPath)
|
||||
mac.Write(data)
|
||||
if err := os.WriteFile(sigPath, []byte(hex.EncodeToString(mac.Sum(nil))), 0o600); err != nil {
|
||||
t.Fatalf("write sig: %v", err)
|
||||
}
|
||||
target := filepath.Join(dir, "restore")
|
||||
os.MkdirAll(target, 0o755)
|
||||
err := Restore(RestoreOptions{
|
||||
InputPath: tarPath,
|
||||
TargetDir: target,
|
||||
MasterKey: key,
|
||||
Force: true,
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("Restore should reject absolute symlink (REQ-127)")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "absolute") {
|
||||
t.Errorf("error should mention absolute: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRestoreRejectsTraversalSymlink verifies a tarball with a .. symlink
|
||||
// that escapes the target is rejected.
|
||||
func TestRestoreRejectsTraversalSymlink(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
tarPath := filepath.Join(dir, "evil2.tar.gz")
|
||||
sigPath := tarPath + ".sig"
|
||||
if err := createCraftedTarball(tarPath, "link", "../../etc/shadow"); err != nil {
|
||||
t.Fatalf("create tarball: %v", err)
|
||||
}
|
||||
key := make([]byte, 32)
|
||||
for i := range key {
|
||||
key[i] = byte(i + 1)
|
||||
}
|
||||
mac := hmac.New(sha256.New, key)
|
||||
data, _ := os.ReadFile(tarPath)
|
||||
mac.Write(data)
|
||||
if err := os.WriteFile(sigPath, []byte(hex.EncodeToString(mac.Sum(nil))), 0o600); err != nil {
|
||||
t.Fatalf("write sig: %v", err)
|
||||
}
|
||||
target := filepath.Join(dir, "restore2")
|
||||
os.MkdirAll(target, 0o755)
|
||||
err := Restore(RestoreOptions{
|
||||
InputPath: tarPath,
|
||||
TargetDir: target,
|
||||
MasterKey: key,
|
||||
Force: true,
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("Restore should reject traversal symlink (REQ-127)")
|
||||
}
|
||||
}
|
||||
|
||||
// createCraftedTarball creates a tar.gz containing a single symlink
|
||||
// entry with the given linkname. Used to test symlink validation.
|
||||
func createCraftedTarball(path, name, linkname string) error {
|
||||
f, err := os.Create(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer f.Close()
|
||||
gz := gzip.NewWriter(f)
|
||||
defer gz.Close()
|
||||
tw := tar.NewWriter(gz)
|
||||
defer tw.Close()
|
||||
hdr := &tar.Header{
|
||||
Name: name,
|
||||
Typeflag: tar.TypeSymlink,
|
||||
Linkname: linkname,
|
||||
Mode: 0o644,
|
||||
}
|
||||
return tw.WriteHeader(hdr)
|
||||
}
|
||||
Vendored
+211
@@ -0,0 +1,211 @@
|
||||
// Package cache implements a CLI-side SQLite-backed key/value cache with
|
||||
// per-class TTLs (R-008). It is the on-disk cache layer used by read-only
|
||||
// `orca` subcommands (node/job/ns list) to avoid hitting the source DB
|
||||
// or filesystem on every invocation.
|
||||
//
|
||||
// The cache is intentionally optional: callers that fail to open the
|
||||
// cache DB must fall back to the uncached read path silently. Writes
|
||||
// bypass the cache entirely (cache invalidation is per-class or
|
||||
// whole-DB only — there is no write-through path).
|
||||
//
|
||||
// Schema (orca_cache):
|
||||
//
|
||||
// CREATE TABLE cache_entries (
|
||||
// class TEXT,
|
||||
// key TEXT,
|
||||
// value BLOB,
|
||||
// inserted_at INTEGER, -- unix nanoseconds
|
||||
// ttl_seconds INTEGER, -- TTL in nanoseconds; 0 = never expires
|
||||
// PRIMARY KEY (class, key)
|
||||
// );
|
||||
//
|
||||
// The schema columns match the v0.11 plan (R-008); the integer columns
|
||||
// are stored at nanosecond resolution so sub-second TTLs (used in tests
|
||||
// and short-lived caches like the 10s job-list cache) work correctly.
|
||||
package cache
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
_ "modernc.org/sqlite"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/paths"
|
||||
)
|
||||
|
||||
// ErrCacheMiss is returned (wrapped) by Get when an entry is absent or
|
||||
// expired. Callers that want a silent miss should treat any error
|
||||
// satisfying errors.Is(err, ErrCacheMiss) as "not in cache".
|
||||
var ErrCacheMiss = errors.New("cache miss")
|
||||
|
||||
// Cache wraps a SQLite-backed key/value cache with per-class TTLs.
|
||||
type Cache struct {
|
||||
db *sql.DB
|
||||
}
|
||||
|
||||
// Open opens (or creates) the SQLite cache DB at path. If path is empty
|
||||
// it defaults to paths.CacheDB(). The DB is created with WAL journal
|
||||
// mode (matching internal/store). The schema is idempotent
|
||||
// (CREATE TABLE IF NOT EXISTS).
|
||||
func Open(path string) (*Cache, error) {
|
||||
if path == "" {
|
||||
path = paths.CacheDB()
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||
return nil, fmt.Errorf("create cache db dir: %w", err)
|
||||
}
|
||||
db, err := sql.Open("sqlite", path+"?_pragma=journal_mode(WAL)")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("open cache sqlite: %w", err)
|
||||
}
|
||||
if err := db.Ping(); err != nil {
|
||||
_ = db.Close()
|
||||
return nil, fmt.Errorf("ping cache sqlite: %w", err)
|
||||
}
|
||||
const schema = `CREATE TABLE IF NOT EXISTS cache_entries (
|
||||
class TEXT NOT NULL,
|
||||
key TEXT NOT NULL,
|
||||
value BLOB NOT NULL,
|
||||
inserted_at INTEGER NOT NULL,
|
||||
ttl_seconds INTEGER NOT NULL,
|
||||
PRIMARY KEY (class, key)
|
||||
)`
|
||||
if _, err := db.Exec(schema); err != nil {
|
||||
_ = db.Close()
|
||||
return nil, fmt.Errorf("create cache schema: %w", err)
|
||||
}
|
||||
return &Cache{db: db}, nil
|
||||
}
|
||||
|
||||
// Get returns the cached value and insertion time for (class, key).
|
||||
// On a miss or expired entry Get returns (nil, zero, ErrCacheMiss).
|
||||
func (c *Cache) Get(class, key string) ([]byte, time.Time, error) {
|
||||
const q = `SELECT value, inserted_at, ttl_seconds FROM cache_entries WHERE class = ? AND key = ?`
|
||||
var (
|
||||
val []byte
|
||||
inserted int64
|
||||
ttlNanos int64
|
||||
)
|
||||
err := c.db.QueryRow(q, class, key).Scan(&val, &inserted, &ttlNanos)
|
||||
if err != nil {
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return nil, time.Time{}, ErrCacheMiss
|
||||
}
|
||||
return nil, time.Time{}, fmt.Errorf("cache get %s/%s: %w", class, key, err)
|
||||
}
|
||||
if ttlNanos > 0 {
|
||||
expiresAt := time.Unix(0, inserted).Add(time.Duration(ttlNanos))
|
||||
if time.Now().After(expiresAt) {
|
||||
_, _ = c.db.Exec(`DELETE FROM cache_entries WHERE class = ? AND key = ?`, class, key)
|
||||
return nil, time.Time{}, ErrCacheMiss
|
||||
}
|
||||
}
|
||||
return val, time.Unix(0, inserted).UTC(), nil
|
||||
}
|
||||
|
||||
// Set stores val for (class, key) with the given ttl. A ttl of 0 means
|
||||
// the entry never expires. An existing entry for (class, key) is
|
||||
// replaced (UPSERT).
|
||||
func (c *Cache) Set(class, key string, val []byte, ttl time.Duration) error {
|
||||
inserted := time.Now().UTC().UnixNano()
|
||||
ttlNanos := int64(ttl)
|
||||
const q = `INSERT INTO cache_entries (class, key, value, inserted_at, ttl_seconds)
|
||||
VALUES (?, ?, ?, ?, ?)
|
||||
ON CONFLICT(class, key) DO UPDATE SET
|
||||
value = excluded.value,
|
||||
inserted_at = excluded.inserted_at,
|
||||
ttl_seconds = excluded.ttl_seconds`
|
||||
if _, err := c.db.Exec(q, class, key, val, inserted, ttlNanos); err != nil {
|
||||
return fmt.Errorf("cache set %s/%s: %w", class, key, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Invalidate removes all entries for class.
|
||||
func (c *Cache) Invalidate(class string) error {
|
||||
if _, err := c.db.Exec(`DELETE FROM cache_entries WHERE class = ?`, class); err != nil {
|
||||
return fmt.Errorf("cache invalidate %s: %w", class, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// InvalidateKey removes a single (class, key) entry.
|
||||
func (c *Cache) InvalidateKey(class, key string) error {
|
||||
if _, err := c.db.Exec(`DELETE FROM cache_entries WHERE class = ? AND key = ?`, class, key); err != nil {
|
||||
return fmt.Errorf("cache invalidate %s/%s: %w", class, key, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Close releases the underlying DB handle.
|
||||
func (c *Cache) Close() error {
|
||||
if c == nil || c.db == nil {
|
||||
return nil
|
||||
}
|
||||
return c.db.Close()
|
||||
}
|
||||
|
||||
// ClassStats describes one cache class for `orca cache show`.
|
||||
type ClassStats struct {
|
||||
Class string `json:"class"`
|
||||
Count int `json:"count"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
OldestAt int64 `json:"oldest_at"`
|
||||
}
|
||||
|
||||
// Stats returns per-class entry counts, total bytes, and oldest
|
||||
// insertion time. Used by `orca cache show`.
|
||||
func (c *Cache) Stats() ([]ClassStats, error) {
|
||||
const q = `SELECT class,
|
||||
COUNT(*) AS count,
|
||||
COALESCE(SUM(LENGTH(value)), 0) AS bytes,
|
||||
COALESCE(MIN(inserted_at), 0) AS oldest
|
||||
FROM cache_entries GROUP BY class ORDER BY class`
|
||||
rows, err := c.db.Query(q)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("cache stats: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []ClassStats
|
||||
for rows.Next() {
|
||||
var s ClassStats
|
||||
if err := rows.Scan(&s.Class, &s.Count, &s.Bytes, &s.OldestAt); err != nil {
|
||||
return nil, fmt.Errorf("cache stats scan: %w", err)
|
||||
}
|
||||
out = append(out, s)
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, fmt.Errorf("cache stats rows: %w", err)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// InvalidateAll clears every entry in the cache.
|
||||
func (c *Cache) InvalidateAll() error {
|
||||
if _, err := c.db.Exec(`DELETE FROM cache_entries`); err != nil {
|
||||
return fmt.Errorf("cache invalidate-all: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Classes returns the distinct class names in the cache.
|
||||
func (c *Cache) Classes() ([]string, error) {
|
||||
rows, err := c.db.Query(`SELECT DISTINCT class FROM cache_entries ORDER BY class`)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("cache classes: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []string
|
||||
for rows.Next() {
|
||||
var name string
|
||||
if err := rows.Scan(&name); err != nil {
|
||||
return nil, fmt.Errorf("cache classes scan: %w", err)
|
||||
}
|
||||
out = append(out, name)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
Vendored
+227
@@ -0,0 +1,227 @@
|
||||
package cache
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func openTestCache(t *testing.T) (*Cache, func()) {
|
||||
t.Helper()
|
||||
path := filepath.Join(t.TempDir(), "orca_cache.db")
|
||||
c, err := Open(path)
|
||||
if err != nil {
|
||||
t.Fatalf("open cache: %v", err)
|
||||
}
|
||||
return c, func() { _ = c.Close() }
|
||||
}
|
||||
|
||||
func TestCache_Hit(t *testing.T) {
|
||||
c, cleanup := openTestCache(t)
|
||||
defer cleanup()
|
||||
|
||||
want := []byte("hello-orca")
|
||||
if err := c.Set("nodes", "list", want, 30*time.Second); err != nil {
|
||||
t.Fatalf("set: %v", err)
|
||||
}
|
||||
got, inserted, err := c.Get("nodes", "list")
|
||||
if err != nil {
|
||||
t.Fatalf("get: %v", err)
|
||||
}
|
||||
if string(got) != string(want) {
|
||||
t.Errorf("get value = %q, want %q", got, want)
|
||||
}
|
||||
if inserted.IsZero() {
|
||||
t.Errorf("inserted time is zero")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCache_Miss(t *testing.T) {
|
||||
c, cleanup := openTestCache(t)
|
||||
defer cleanup()
|
||||
|
||||
got, _, err := c.Get("nodes", "missing")
|
||||
if !errors.Is(err, ErrCacheMiss) {
|
||||
t.Fatalf("get miss: err = %v, want ErrCacheMiss", err)
|
||||
}
|
||||
if got != nil {
|
||||
t.Errorf("get miss value = %v, want nil", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCache_Invalidate(t *testing.T) {
|
||||
c, cleanup := openTestCache(t)
|
||||
defer cleanup()
|
||||
|
||||
if err := c.Set("nodes", "list", []byte("a"), 30*time.Second); err != nil {
|
||||
t.Fatalf("set a: %v", err)
|
||||
}
|
||||
if err := c.Set("nodes", "other", []byte("b"), 30*time.Second); err != nil {
|
||||
t.Fatalf("set b: %v", err)
|
||||
}
|
||||
if err := c.Set("jobs", "list", []byte("c"), 30*time.Second); err != nil {
|
||||
t.Fatalf("set c: %v", err)
|
||||
}
|
||||
if err := c.Invalidate("nodes"); err != nil {
|
||||
t.Fatalf("invalidate: %v", err)
|
||||
}
|
||||
if _, _, err := c.Get("nodes", "list"); !errors.Is(err, ErrCacheMiss) {
|
||||
t.Errorf("nodes/list after invalidate: err = %v, want ErrCacheMiss", err)
|
||||
}
|
||||
if _, _, err := c.Get("nodes", "other"); !errors.Is(err, ErrCacheMiss) {
|
||||
t.Errorf("nodes/other after invalidate: err = %v, want ErrCacheMiss", err)
|
||||
}
|
||||
if _, _, err := c.Get("jobs", "list"); err != nil {
|
||||
t.Errorf("jobs/list after nodes invalidate: err = %v, want nil", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCache_InvalidateKey(t *testing.T) {
|
||||
c, cleanup := openTestCache(t)
|
||||
defer cleanup()
|
||||
|
||||
if err := c.Set("nodes", "list", []byte("a"), 30*time.Second); err != nil {
|
||||
t.Fatalf("set: %v", err)
|
||||
}
|
||||
if err := c.InvalidateKey("nodes", "list"); err != nil {
|
||||
t.Fatalf("invalidate key: %v", err)
|
||||
}
|
||||
if _, _, err := c.Get("nodes", "list"); !errors.Is(err, ErrCacheMiss) {
|
||||
t.Errorf("get after invalidate key: err = %v, want ErrCacheMiss", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCache_TTLExpiry(t *testing.T) {
|
||||
c, cleanup := openTestCache(t)
|
||||
defer cleanup()
|
||||
|
||||
if err := c.Set("jobs", "list", []byte("stale"), 1*time.Millisecond); err != nil {
|
||||
t.Fatalf("set: %v", err)
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
if _, _, err := c.Get("jobs", "list"); !errors.Is(err, ErrCacheMiss) {
|
||||
t.Errorf("get after ttl expiry: err = %v, want ErrCacheMiss", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCache_TTLZeroNeverExpires(t *testing.T) {
|
||||
c, cleanup := openTestCache(t)
|
||||
defer cleanup()
|
||||
|
||||
if err := c.Set("namespaces", "list", []byte("forever"), 0); err != nil {
|
||||
t.Fatalf("set: %v", err)
|
||||
}
|
||||
time.Sleep(5 * time.Millisecond)
|
||||
got, _, err := c.Get("namespaces", "list")
|
||||
if err != nil {
|
||||
t.Fatalf("get ttl=0: %v", err)
|
||||
}
|
||||
if string(got) != "forever" {
|
||||
t.Errorf("get ttl=0 value = %q, want %q", got, "forever")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCache_Overwrite(t *testing.T) {
|
||||
c, cleanup := openTestCache(t)
|
||||
defer cleanup()
|
||||
|
||||
if err := c.Set("nodes", "list", []byte("v1"), 30*time.Second); err != nil {
|
||||
t.Fatalf("set v1: %v", err)
|
||||
}
|
||||
if err := c.Set("nodes", "list", []byte("v2"), 30*time.Second); err != nil {
|
||||
t.Fatalf("set v2: %v", err)
|
||||
}
|
||||
got, _, err := c.Get("nodes", "list")
|
||||
if err != nil {
|
||||
t.Fatalf("get: %v", err)
|
||||
}
|
||||
if string(got) != "v2" {
|
||||
t.Errorf("get after overwrite = %q, want %q", got, "v2")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCache_InvalidateAll(t *testing.T) {
|
||||
c, cleanup := openTestCache(t)
|
||||
defer cleanup()
|
||||
|
||||
_ = c.Set("nodes", "list", []byte("a"), 30*time.Second)
|
||||
_ = c.Set("jobs", "list", []byte("b"), 30*time.Second)
|
||||
if err := c.InvalidateAll(); err != nil {
|
||||
t.Fatalf("invalidate all: %v", err)
|
||||
}
|
||||
if _, _, err := c.Get("nodes", "list"); !errors.Is(err, ErrCacheMiss) {
|
||||
t.Errorf("nodes/list after invalidate-all: err = %v, want ErrCacheMiss", err)
|
||||
}
|
||||
if _, _, err := c.Get("jobs", "list"); !errors.Is(err, ErrCacheMiss) {
|
||||
t.Errorf("jobs/list after invalidate-all: err = %v, want ErrCacheMiss", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCache_Stats(t *testing.T) {
|
||||
c, cleanup := openTestCache(t)
|
||||
defer cleanup()
|
||||
|
||||
_ = c.Set("nodes", "list", []byte("aaaa"), 30*time.Second)
|
||||
_ = c.Set("jobs", "list", []byte("bb"), 30*time.Second)
|
||||
stats, err := c.Stats()
|
||||
if err != nil {
|
||||
t.Fatalf("stats: %v", err)
|
||||
}
|
||||
if len(stats) != 2 {
|
||||
t.Fatalf("stats len = %d, want 2", len(stats))
|
||||
}
|
||||
var nodes, jobs *ClassStats
|
||||
for i := range stats {
|
||||
switch stats[i].Class {
|
||||
case "nodes":
|
||||
nodes = &stats[i]
|
||||
case "jobs":
|
||||
jobs = &stats[i]
|
||||
}
|
||||
}
|
||||
if nodes == nil || nodes.Count != 1 || nodes.Bytes != 4 {
|
||||
t.Errorf("nodes stats = %+v, want count=1 bytes=4", nodes)
|
||||
}
|
||||
if jobs == nil || jobs.Count != 1 || jobs.Bytes != 2 {
|
||||
t.Errorf("jobs stats = %+v, want count=1 bytes=2", jobs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCache_OpenDefaultPath(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", dir)
|
||||
c, err := Open("")
|
||||
if err != nil {
|
||||
t.Fatalf("open default path: %v", err)
|
||||
}
|
||||
defer c.Close()
|
||||
if err := c.Set("nodes", "list", []byte("ok"), 0); err != nil {
|
||||
t.Fatalf("set: %v", err)
|
||||
}
|
||||
got, _, err := c.Get("nodes", "list")
|
||||
if err != nil {
|
||||
t.Fatalf("get: %v", err)
|
||||
}
|
||||
if string(got) != "ok" {
|
||||
t.Errorf("get = %q, want %q", got, "ok")
|
||||
}
|
||||
}
|
||||
|
||||
func BenchmarkCacheHit(b *testing.B) {
|
||||
path := filepath.Join(b.TempDir(), "orca_cache.db")
|
||||
c, err := Open(path)
|
||||
if err != nil {
|
||||
b.Fatalf("open: %v", err)
|
||||
}
|
||||
defer c.Close()
|
||||
if err := c.Set("nodes", "list", []byte("bench"), 0); err != nil {
|
||||
b.Fatalf("set: %v", err)
|
||||
}
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
if _, _, err := c.Get("nodes", "list"); err != nil {
|
||||
b.Fatalf("get: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,64 +1,73 @@
|
||||
// Package certpaths centralizes the on-disk locations of the CA and
|
||||
// server cert/key files. The CLI layer, the security layer, and the
|
||||
// doctor layer all need to agree on these paths, so they're factored
|
||||
// into their own package to avoid import cycles (cli <-> doctor).
|
||||
// Package certpaths is the v0.8 path shim. It returns v0.8 flat-layout
|
||||
// paths for backward compatibility during the v0.9 dual-write window
|
||||
// (REQ-090). The v0.9 paths package (internal/paths) returns the new
|
||||
// multi-namespace layout (R-002).
|
||||
//
|
||||
// certpaths will be deleted after the v0.10-P14 migration. New code
|
||||
// should use internal/paths, NOT certpaths.
|
||||
//
|
||||
// Migration notes (per v0.10-P14):
|
||||
// - CA cert/key, server cert/key, SSH key/pub, known_hosts currently
|
||||
// live at the flat Root() location. The v0.9 internal/paths package
|
||||
// returns the new ClusterDir()/... locations; certpaths keeps the
|
||||
// v0.8 flat locations until the CA migration moves them.
|
||||
// - DBPath keeps returning Root()/orca.db (v0.8 location). The new
|
||||
// paths.NSDb("_defaults") returns Root()/_defaults/db/orca.db; the DB
|
||||
// moves in v0.10-P14.
|
||||
package certpaths
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/paths"
|
||||
)
|
||||
|
||||
const (
|
||||
defaultCADir = ".orca"
|
||||
caCertFilename = "ca.crt"
|
||||
caKeyFilename = "ca.key"
|
||||
)
|
||||
// Dir returns the v0.8 flat root directory. Delegates to paths.Root()
|
||||
// (which honors $ORCA_HOME, else ~/.orca). v0.8 callers expect the CA
|
||||
// and DB to live directly under this directory; that does not change
|
||||
// until the v0.10-P14 migration.
|
||||
func Dir() string { return paths.Root() }
|
||||
|
||||
// Dir returns the directory the local CA lives in. Honors $ORCA_HOME
|
||||
// for testability; otherwise defaults to ~/.orca.
|
||||
func Dir() string {
|
||||
if p := os.Getenv("ORCA_HOME"); p != "" {
|
||||
return p
|
||||
}
|
||||
home, _ := os.UserHomeDir()
|
||||
return filepath.Join(home, defaultCADir)
|
||||
}
|
||||
// CACertPath returns the v0.8 CA cert path: Dir()/ca.crt.
|
||||
// The v0.9 location is paths.CACertPath() = ClusterDir()/ca.crt; certpaths
|
||||
// keeps the v0.8 flat location until the CA migration in v0.10-P14.
|
||||
func CACertPath() string { return filepath.Join(paths.Root(), "ca.crt") }
|
||||
|
||||
// CACertPath returns the path to ca.crt.
|
||||
func CACertPath() string { return filepath.Join(Dir(), caCertFilename) }
|
||||
// CAKeyPath returns the v0.8 CA key path: Dir()/ca.key.
|
||||
// See CACertPath for migration notes.
|
||||
func CAKeyPath() string { return filepath.Join(paths.Root(), "ca.key") }
|
||||
|
||||
// CAKeyPath returns the path to ca.key.
|
||||
func CAKeyPath() string { return filepath.Join(Dir(), caKeyFilename) }
|
||||
// ServerCertPath returns the v0.8 server cert path: Dir()/server.crt.
|
||||
// See CACertPath for migration notes.
|
||||
func ServerCertPath() string { return filepath.Join(paths.Root(), "server.crt") }
|
||||
|
||||
// ServerCertPath returns the path to server.crt.
|
||||
func ServerCertPath() string { return filepath.Join(Dir(), "server.crt") }
|
||||
|
||||
// ServerKeyPath returns the path to server.key.
|
||||
func ServerKeyPath() string { return filepath.Join(Dir(), "server.key") }
|
||||
// ServerKeyPath returns the v0.8 server key path: Dir()/server.key.
|
||||
// See CACertPath for migration notes.
|
||||
func ServerKeyPath() string { return filepath.Join(paths.Root(), "server.key") }
|
||||
|
||||
// DBPath returns the path to the orca SQLite database. Honors $ORCA_DB
|
||||
// for testability and explicit override; otherwise defaults to
|
||||
// ~/.orca/orca.db under the same Dir() as the cert files.
|
||||
// for testability and explicit override; otherwise defaults to the v0.8
|
||||
// flat location Dir()/orca.db. The v0.9 location is
|
||||
// paths.NSDb(paths.DefaultNamespace()) = Root()/_defaults/db/orca.db;
|
||||
// certpaths keeps the v0.8 flat location until the DB move in v0.10-P14.
|
||||
func DBPath() string {
|
||||
if p := os.Getenv("ORCA_DB"); p != "" {
|
||||
return p
|
||||
}
|
||||
return filepath.Join(Dir(), "orca.db")
|
||||
return filepath.Join(paths.Root(), "orca.db")
|
||||
}
|
||||
|
||||
// SSHKeyPath returns the path to the orca SSH private key (Ed25519,
|
||||
// D-037). Used by `orca node join --type proxmox` to authenticate
|
||||
// to remote Proxmox hosts after the initial password-based bootstrap.
|
||||
// File mode 0600 (enforced by security.WriteKey).
|
||||
func SSHKeyPath() string { return filepath.Join(Dir(), "orca_ssh_key") }
|
||||
// SSHKeyPath returns the v0.8 SSH private key path: Dir()/orca_ssh_key.
|
||||
// The v0.9 location is paths.SSHKeyPath() = ClusterDir()/orca_ssh_key;
|
||||
// certpaths keeps the v0.8 flat location until the migration.
|
||||
func SSHKeyPath() string { return filepath.Join(paths.Root(), "orca_ssh_key") }
|
||||
|
||||
// SSHPubPath returns the path to the orca SSH public key (authorized_keys
|
||||
// format). Deployed to remote Proxmox hosts during `orca node join`.
|
||||
// File mode 0644 (enforced by security.WriteCert).
|
||||
func SSHPubPath() string { return filepath.Join(Dir(), "orca_ssh_key.pub") }
|
||||
// SSHPubPath returns the v0.8 SSH public key path: Dir()/orca_ssh_key.pub.
|
||||
// See SSHKeyPath for migration notes.
|
||||
func SSHPubPath() string { return filepath.Join(paths.Root(), "orca_ssh_key.pub") }
|
||||
|
||||
// KnownHostsPath returns the path to the SSH known_hosts file used for
|
||||
// TOFU host-key pinning (D-035). Captured on first connect, verified
|
||||
// on all subsequent connects via golang.org/x/crypto/ssh/knownhosts.
|
||||
func KnownHostsPath() string { return filepath.Join(Dir(), "known_hosts") }
|
||||
// KnownHostsPath returns the v0.8 known_hosts path: Dir()/known_hosts.
|
||||
// The v0.9 location is paths.KnownHostsPath() = ClusterDir()/known_hosts;
|
||||
// certpaths keeps the v0.8 flat location until the migration.
|
||||
func KnownHostsPath() string { return filepath.Join(paths.Root(), "known_hosts") }
|
||||
|
||||
@@ -3,15 +3,17 @@ package certpaths
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/paths"
|
||||
)
|
||||
|
||||
const defaultHomeSubdir = ".orca"
|
||||
|
||||
func TestPaths_HonorORCAHOME(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", dir)
|
||||
// Ensure ORCA_DB doesn't leak from the environment / prior tests.
|
||||
t.Setenv("ORCA_DB", "")
|
||||
|
||||
cases := []struct {
|
||||
@@ -36,17 +38,26 @@ func TestPaths_HonorORCAHOME(t *testing.T) {
|
||||
})
|
||||
}
|
||||
|
||||
// DBPath defaults to $ORCA_HOME/orca.db.
|
||||
if got, want := DBPath(), filepath.Join(dir, "orca.db"); got != want {
|
||||
t.Errorf("DBPath = %q, want %q", got, want)
|
||||
}
|
||||
|
||||
// Dir() returns ORCA_HOME verbatim.
|
||||
if got, want := Dir(), dir; got != want {
|
||||
t.Errorf("Dir = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestShim_DelegatesDirToPaths(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", dir)
|
||||
if got, want := Dir(), paths.Root(); got != want {
|
||||
t.Errorf("Dir() = %q, paths.Root() = %q (shim must delegate)", got, want)
|
||||
}
|
||||
if got, want := Dir(), dir; got != want {
|
||||
t.Errorf("Dir() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDBPath_OrcaDBOverride(t *testing.T) {
|
||||
home := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", home)
|
||||
@@ -70,19 +81,14 @@ func TestDBPath_OrcaDBEmptyStringFallsBackToHome(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestDir_DefaultHomeFallback(t *testing.T) {
|
||||
// Unset ORCA_HOME so Dir() falls back to ~/.orca.
|
||||
// We can't reliably mutate the real HOME in a portable way, so just
|
||||
// assert that the returned path ends with the default subdir on the
|
||||
// current OS and is absolute.
|
||||
os.Unsetenv("ORCA_HOME")
|
||||
// Also clear ORCA_DB so DBPath's fallback to Dir() is exercised.
|
||||
os.Unsetenv("ORCA_DB")
|
||||
|
||||
home, err := os.UserHomeDir()
|
||||
if err != nil {
|
||||
t.Skipf("os.UserHomeDir: %v (cannot verify default fallback)", err)
|
||||
}
|
||||
want := filepath.Join(home, defaultCADir)
|
||||
want := filepath.Join(home, defaultHomeSubdir)
|
||||
if got := Dir(); got != want {
|
||||
t.Errorf("Dir() default = %q, want %q", got, want)
|
||||
}
|
||||
@@ -92,25 +98,22 @@ func TestDir_DefaultHomeFallback(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestDir_ORCAHOMEEmptyFallsBack(t *testing.T) {
|
||||
// Empty string ORCA_HOME is treated as unset → ~/.orca fallback.
|
||||
t.Setenv("ORCA_HOME", "")
|
||||
home, err := os.UserHomeDir()
|
||||
if err != nil {
|
||||
t.Skipf("os.UserHomeDir: %v", err)
|
||||
}
|
||||
want := filepath.Join(home, defaultCADir)
|
||||
want := filepath.Join(home, defaultHomeSubdir)
|
||||
if got := Dir(); got != want {
|
||||
t.Errorf("Dir() with empty ORCA_HOME = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDir_ORCAHOMERelativePath(t *testing.T) {
|
||||
// A relative ORCA_HOME is honored verbatim (no cleaning/absolutizing).
|
||||
t.Setenv("ORCA_HOME", "relative/orca/home")
|
||||
if got, want := Dir(), "relative/orca/home"; got != want {
|
||||
t.Errorf("Dir() relative = %q, want %q", got, want)
|
||||
}
|
||||
// CACertPath joins the relative dir with ca.crt using filepath.Join.
|
||||
if got, want := CACertPath(), filepath.Join("relative/orca/home", "ca.crt"); got != want {
|
||||
t.Errorf("CACertPath relative = %q, want %q", got, want)
|
||||
}
|
||||
@@ -121,7 +124,6 @@ func TestAllPaths_AreConsistentWithDir(t *testing.T) {
|
||||
t.Setenv("ORCA_HOME", dir)
|
||||
t.Setenv("ORCA_DB", "")
|
||||
|
||||
// Every *Path() must live under Dir() except DBPath which also does.
|
||||
base := Dir()
|
||||
for _, p := range []string{
|
||||
CACertPath(), CAKeyPath(),
|
||||
@@ -135,6 +137,38 @@ func TestAllPaths_AreConsistentWithDir(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestShim_ReturnsV08FlatPaths(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", dir)
|
||||
t.Setenv("ORCA_DB", "")
|
||||
|
||||
root := paths.Root()
|
||||
if got, want := CACertPath(), filepath.Join(root, "ca.crt"); got != want {
|
||||
t.Errorf("CACertPath = %q, want v0.8 flat %q", got, want)
|
||||
}
|
||||
if got, want := CAKeyPath(), filepath.Join(root, "ca.key"); got != want {
|
||||
t.Errorf("CAKeyPath = %q, want v0.8 flat %q", got, want)
|
||||
}
|
||||
if got, want := ServerCertPath(), filepath.Join(root, "server.crt"); got != want {
|
||||
t.Errorf("ServerCertPath = %q, want v0.8 flat %q", got, want)
|
||||
}
|
||||
if got, want := ServerKeyPath(), filepath.Join(root, "server.key"); got != want {
|
||||
t.Errorf("ServerKeyPath = %q, want v0.8 flat %q", got, want)
|
||||
}
|
||||
if got, want := SSHKeyPath(), filepath.Join(root, "orca_ssh_key"); got != want {
|
||||
t.Errorf("SSHKeyPath = %q, want v0.8 flat %q", got, want)
|
||||
}
|
||||
if got, want := SSHPubPath(), filepath.Join(root, "orca_ssh_key.pub"); got != want {
|
||||
t.Errorf("SSHPubPath = %q, want v0.8 flat %q", got, want)
|
||||
}
|
||||
if got, want := KnownHostsPath(), filepath.Join(root, "known_hosts"); got != want {
|
||||
t.Errorf("KnownHostsPath = %q, want v0.8 flat %q", got, want)
|
||||
}
|
||||
if got, want := DBPath(), filepath.Join(root, "orca.db"); got != want {
|
||||
t.Errorf("DBPath = %q, want v0.8 flat %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSSHPaths_Filenames(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", dir)
|
||||
@@ -148,10 +182,3 @@ func TestSSHPaths_Filenames(t *testing.T) {
|
||||
t.Errorf("KnownHostsPath base = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func init() {
|
||||
// On Windows the default home subdir is still ".orca"; the test for
|
||||
// default fallback uses os.UserHomeDir which is platform-aware. This
|
||||
// guard keeps the suite from running a meaningless check on plan9.
|
||||
_ = runtime.GOOS
|
||||
}
|
||||
|
||||
@@ -0,0 +1,366 @@
|
||||
// Package cli: acl.go implements the `orca acl` subcommand family
|
||||
// (P02, v0.11). Subcommands:
|
||||
//
|
||||
// orca acl grant <identity> --namespace <ns> --permissions <perms>
|
||||
// orca acl revoke <identity> --namespace <ns>
|
||||
// orca acl list
|
||||
// orca acl check <identity> --namespace <ns> --permission <perm>
|
||||
//
|
||||
// ACL state is stored at paths.ClusterDir()/acl.json (a simple JSON
|
||||
// file — no DB needed for v0.11). <identity> is either a SPIFFE URI
|
||||
// (spiffe://orca.local/ns/.../sa/.../...) or a bare token ID.
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/acl"
|
||||
"git.cloudinit.dev/coreci/orca/internal/paths"
|
||||
)
|
||||
|
||||
var (
|
||||
aclGrantNamespace string
|
||||
aclGrantPermissions string
|
||||
aclRevokeNamespace string
|
||||
aclCheckNamespace string
|
||||
aclCheckPermission string
|
||||
)
|
||||
|
||||
var aclCmd = &cobra.Command{
|
||||
Use: "acl",
|
||||
Short: "Manage access-control entries (SPIFFE + token identities)",
|
||||
Long: `Manage the cluster ACL (P02, v0.11). Identities are either
|
||||
SPIFFE workload URIs (spiffe://orca.local/ns/<ns>/sa/<sa>/<alloc>) or
|
||||
operator token IDs. Permissions are deny-by-default: an identity with
|
||||
no matching entry on a namespace has no access.
|
||||
|
||||
State is stored at ` + "`" + `ClusterDir()/acl.json` + "`" + `.`,
|
||||
}
|
||||
|
||||
// parseIdentity classifies <identity> as a SPIFFE or token identity.
|
||||
// A SPIFFE identity is detected by the spiffe:// scheme; its namespace
|
||||
// is extracted from the URI path. Anything else is treated as a token
|
||||
// ID whose namespace must be supplied via the --namespace flag.
|
||||
func parseIdentity(raw string) (acl.Identity, error) {
|
||||
if strings.HasPrefix(raw, "spiffe://") {
|
||||
ns, err := acl.SpiffeNamespace(raw)
|
||||
if err != nil {
|
||||
return acl.Identity{}, fmt.Errorf("parse spiffe identity: %w", err)
|
||||
}
|
||||
return acl.Identity{Kind: acl.KindSpiffe, ID: raw, Namespace: ns}, nil
|
||||
}
|
||||
if raw == "" {
|
||||
return acl.Identity{}, fmt.Errorf("identity is empty")
|
||||
}
|
||||
return acl.Identity{Kind: acl.KindToken, ID: raw}, nil
|
||||
}
|
||||
|
||||
// parsePermissions parses a comma-separated list of "read","write",
|
||||
// "admin" into a Permission bitmask. Empty string defaults to read.
|
||||
func parsePermissions(s string) (acl.Permission, error) {
|
||||
s = strings.TrimSpace(s)
|
||||
if s == "" {
|
||||
return acl.PermRead, nil
|
||||
}
|
||||
var perms acl.Permission
|
||||
for _, part := range strings.Split(s, ",") {
|
||||
part = strings.TrimSpace(strings.ToLower(part))
|
||||
switch part {
|
||||
case "read":
|
||||
perms |= acl.PermRead
|
||||
case "write":
|
||||
perms |= acl.PermWrite
|
||||
case "admin":
|
||||
perms |= acl.PermAdmin
|
||||
default:
|
||||
return 0, fmt.Errorf("unknown permission %q (want read, write, or admin)", part)
|
||||
}
|
||||
}
|
||||
if perms == 0 {
|
||||
return 0, fmt.Errorf("no permissions in %q", s)
|
||||
}
|
||||
return perms, nil
|
||||
}
|
||||
|
||||
// permName renders a Permission bitmask as a comma-separated string.
|
||||
func permName(p acl.Permission) string {
|
||||
var parts []string
|
||||
if p&acl.PermRead != 0 {
|
||||
parts = append(parts, "read")
|
||||
}
|
||||
if p&acl.PermWrite != 0 {
|
||||
parts = append(parts, "write")
|
||||
}
|
||||
if p&acl.PermAdmin != 0 {
|
||||
parts = append(parts, "admin")
|
||||
}
|
||||
if len(parts) == 0 {
|
||||
return "none"
|
||||
}
|
||||
return strings.Join(parts, ",")
|
||||
}
|
||||
|
||||
// aclState is the on-disk JSON shape for acl.json.
|
||||
type aclState struct {
|
||||
Entries []acl.ACLEntry `json:"entries"`
|
||||
}
|
||||
|
||||
// loadACL reads paths.ACLPath() and returns an *acl.ACL. A missing
|
||||
// file is treated as an empty ACL (not an error).
|
||||
func loadACL() (*acl.ACL, error) {
|
||||
a := acl.NewACL()
|
||||
path := paths.ACLPath()
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return a, nil
|
||||
}
|
||||
return nil, fmt.Errorf("read acl state: %w", err)
|
||||
}
|
||||
if len(data) == 0 {
|
||||
return a, nil
|
||||
}
|
||||
var st aclState
|
||||
if err := json.Unmarshal(data, &st); err != nil {
|
||||
return nil, fmt.Errorf("parse acl state: %w", err)
|
||||
}
|
||||
for _, e := range st.Entries {
|
||||
a.Grant(e.Identity, e.Namespace, e.Permissions)
|
||||
}
|
||||
return a, nil
|
||||
}
|
||||
|
||||
// saveACL writes the ACL to paths.ACLPath() atomically (write to temp,
|
||||
// rename). The cluster dir is created if missing.
|
||||
func saveACL(a *acl.ACL) error {
|
||||
path := paths.ACLPath()
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||
return fmt.Errorf("create cluster dir: %w", err)
|
||||
}
|
||||
st := aclState{Entries: a.List()}
|
||||
data, err := json.MarshalIndent(st, "", " ")
|
||||
if err != nil {
|
||||
return fmt.Errorf("marshal acl state: %w", err)
|
||||
}
|
||||
if err := writeAtomicFile(path, data, 0o644); err != nil {
|
||||
return fmt.Errorf("write acl state: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// writeAtomicFile writes data to a temp file in dir(path) and renames
|
||||
// it into place, matching the security.WriteAtomic pattern (P02 keeps
|
||||
// a local copy to avoid importing internal/security into the CLI).
|
||||
func writeAtomicFile(path string, data []byte, mode os.FileMode) error {
|
||||
dir := filepath.Dir(path)
|
||||
tmp, err := os.CreateTemp(dir, ".acl-tmp-*")
|
||||
if err != nil {
|
||||
return fmt.Errorf("create temp: %w", err)
|
||||
}
|
||||
tmpName := tmp.Name()
|
||||
defer func() { _ = os.Remove(tmpName) }()
|
||||
if _, err := tmp.Write(data); err != nil {
|
||||
_ = tmp.Close()
|
||||
return fmt.Errorf("write temp: %w", err)
|
||||
}
|
||||
if err := tmp.Chmod(mode); err != nil {
|
||||
_ = tmp.Close()
|
||||
return fmt.Errorf("chmod temp: %w", err)
|
||||
}
|
||||
if err := tmp.Close(); err != nil {
|
||||
return fmt.Errorf("close temp: %w", err)
|
||||
}
|
||||
if err := os.Rename(tmpName, path); err != nil {
|
||||
return fmt.Errorf("rename temp: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
var aclGrantCmd = &cobra.Command{
|
||||
Use: "grant <identity>",
|
||||
Short: "Grant permissions to an identity on a namespace",
|
||||
Long: `Grant permissions to an identity on a namespace. The identity
|
||||
is either a SPIFFE URI (its namespace is extracted from the path and
|
||||
must match --namespace) or a bare token ID (whose namespace is
|
||||
--namespace). --permissions is a comma-separated list of read,write,
|
||||
admin (default: read).`,
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
identity, err := parseIdentity(args[0])
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
ns := aclGrantNamespace
|
||||
if ns == "" {
|
||||
ns = identity.Namespace
|
||||
}
|
||||
if ns == "" {
|
||||
return fmt.Errorf("--namespace is required for token identities (or set it to match the spiffe path)")
|
||||
}
|
||||
if identity.Kind == acl.KindSpiffe && identity.Namespace != "" && identity.Namespace != ns {
|
||||
return fmt.Errorf("spiffe namespace %q does not match --namespace %q", identity.Namespace, ns)
|
||||
}
|
||||
perms, err := parsePermissions(aclGrantPermissions)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
a, err := loadACL()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
identity.Namespace = ns
|
||||
a.Grant(identity, ns, perms)
|
||||
if err := saveACL(a); err != nil {
|
||||
return err
|
||||
}
|
||||
slog.Info("acl grant", "identity", identity.ID, "namespace", ns, "permissions", permName(perms))
|
||||
if jsonOutput {
|
||||
return printJSON(map[string]any{
|
||||
"identity": identity,
|
||||
"namespace": ns,
|
||||
"permissions": permName(perms),
|
||||
"granted": true,
|
||||
})
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "✓ Granted %s on %s to %s\n", permName(perms), ns, identity.ID)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
var aclRevokeCmd = &cobra.Command{
|
||||
Use: "revoke <identity>",
|
||||
Short: "Revoke an identity's access on a namespace",
|
||||
Long: `Revoke an identity's entry on a namespace. For a SPIFFE
|
||||
identity the namespace defaults to the one in the URI path; for a
|
||||
token identity --namespace is required.`,
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
identity, err := parseIdentity(args[0])
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
ns := aclRevokeNamespace
|
||||
if ns == "" {
|
||||
ns = identity.Namespace
|
||||
}
|
||||
if ns == "" {
|
||||
return fmt.Errorf("--namespace is required for token identities")
|
||||
}
|
||||
a, err := loadACL()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
identity.Namespace = ns
|
||||
a.Revoke(identity, ns)
|
||||
if err := saveACL(a); err != nil {
|
||||
return err
|
||||
}
|
||||
slog.Info("acl revoke", "identity", identity.ID, "namespace", ns)
|
||||
if jsonOutput {
|
||||
return printJSON(map[string]any{
|
||||
"identity": identity,
|
||||
"namespace": ns,
|
||||
"revoked": true,
|
||||
})
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "✓ Revoked %s on %s\n", identity.ID, ns)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
var aclListCmd = &cobra.Command{
|
||||
Use: "list",
|
||||
Short: "List all ACL entries",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
a, err := loadACL()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
entries := a.List()
|
||||
if jsonOutput {
|
||||
return printJSON(entries)
|
||||
}
|
||||
out := cmd.OutOrStdout()
|
||||
if len(entries) == 0 {
|
||||
fmt.Fprintln(out, "No ACL entries. Use `orca acl grant` to add one.")
|
||||
return nil
|
||||
}
|
||||
fmt.Fprintf(out, "%-12s %-50s %-16s %s\n", "KIND", "IDENTITY", "NAMESPACE", "PERMISSIONS")
|
||||
for _, e := range entries {
|
||||
fmt.Fprintf(out, "%-12s %-50s %-16s %s\n", e.Identity.Kind, e.Identity.ID, e.Namespace, permName(e.Permissions))
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
var aclCheckCmd = &cobra.Command{
|
||||
Use: "check <identity>",
|
||||
Short: "Check whether an identity has a permission on a namespace",
|
||||
Long: `Check whether an identity has the given permission on the
|
||||
namespace. Exits 0 if allowed, 1 if denied. --permission is one of
|
||||
read, write, admin (default: read).`,
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
identity, err := parseIdentity(args[0])
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
ns := aclCheckNamespace
|
||||
if ns == "" {
|
||||
ns = identity.Namespace
|
||||
}
|
||||
if ns == "" {
|
||||
return fmt.Errorf("--namespace is required for token identities")
|
||||
}
|
||||
permStr := strings.TrimSpace(aclCheckPermission)
|
||||
if permStr == "" {
|
||||
permStr = "read"
|
||||
}
|
||||
perm, err := parsePermissions(permStr)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
a, err := loadACL()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
identity.Namespace = ns
|
||||
allowed := a.Check(identity, ns, perm)
|
||||
if jsonOutput {
|
||||
return printJSON(map[string]any{
|
||||
"identity": identity,
|
||||
"namespace": ns,
|
||||
"permission": permStr,
|
||||
"allowed": allowed,
|
||||
})
|
||||
}
|
||||
if allowed {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "✓ %s has %s on %s\n", identity.ID, permStr, ns)
|
||||
return nil
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "✗ %s does NOT have %s on %s\n", identity.ID, permStr, ns)
|
||||
return fmt.Errorf("denied")
|
||||
},
|
||||
}
|
||||
|
||||
func init() {
|
||||
aclGrantCmd.Flags().StringVar(&aclGrantNamespace, "namespace", "", "namespace scope (required for tokens; defaults to spiffe path ns)")
|
||||
aclGrantCmd.Flags().StringVar(&aclGrantPermissions, "permissions", "read", "comma-separated permissions: read,write,admin")
|
||||
aclRevokeCmd.Flags().StringVar(&aclRevokeNamespace, "namespace", "", "namespace scope (required for tokens; defaults to spiffe path ns)")
|
||||
aclCheckCmd.Flags().StringVar(&aclCheckNamespace, "namespace", "", "namespace scope (required for tokens; defaults to spiffe path ns)")
|
||||
aclCheckCmd.Flags().StringVar(&aclCheckPermission, "permission", "read", "permission to check: read, write, or admin")
|
||||
|
||||
aclCmd.AddCommand(aclGrantCmd)
|
||||
aclCmd.AddCommand(aclRevokeCmd)
|
||||
aclCmd.AddCommand(aclListCmd)
|
||||
aclCmd.AddCommand(aclCheckCmd)
|
||||
rootCmd.AddCommand(aclCmd)
|
||||
}
|
||||
@@ -0,0 +1,386 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/paths"
|
||||
)
|
||||
|
||||
func resetACLFlags() {
|
||||
aclGrantNamespace = ""
|
||||
aclGrantPermissions = "read"
|
||||
aclRevokeNamespace = ""
|
||||
aclCheckNamespace = ""
|
||||
aclCheckPermission = "read"
|
||||
}
|
||||
|
||||
func TestACLCommandRegistered(t *testing.T) {
|
||||
registered := make(map[string]bool)
|
||||
for _, cmd := range rootCmd.Commands() {
|
||||
registered[cmd.Name()] = true
|
||||
}
|
||||
if !registered["acl"] {
|
||||
t.Fatal("acl command not registered on root")
|
||||
}
|
||||
}
|
||||
|
||||
func TestACLSubcommands(t *testing.T) {
|
||||
expected := []string{"grant", "revoke", "list", "check"}
|
||||
registered := make(map[string]bool)
|
||||
for _, cmd := range aclCmd.Commands() {
|
||||
registered[cmd.Name()] = true
|
||||
}
|
||||
for _, name := range expected {
|
||||
if !registered[name] {
|
||||
t.Errorf("expected acl subcommand %q not registered", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseIdentity_Spiffe(t *testing.T) {
|
||||
id, err := parseIdentity("spiffe://orca.local/ns/myapp/sa/svc1/alloc-1")
|
||||
if err != nil {
|
||||
t.Fatalf("parseIdentity: %v", err)
|
||||
}
|
||||
if id.Kind != "spiffe" {
|
||||
t.Errorf("kind = %q, want spiffe", id.Kind)
|
||||
}
|
||||
if id.Namespace != "myapp" {
|
||||
t.Errorf("namespace = %q, want myapp", id.Namespace)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseIdentity_Token(t *testing.T) {
|
||||
id, err := parseIdentity("operator-1")
|
||||
if err != nil {
|
||||
t.Fatalf("parseIdentity: %v", err)
|
||||
}
|
||||
if id.Kind != "token" {
|
||||
t.Errorf("kind = %q, want token", id.Kind)
|
||||
}
|
||||
if id.ID != "operator-1" {
|
||||
t.Errorf("id = %q, want operator-1", id.ID)
|
||||
}
|
||||
if id.Namespace != "" {
|
||||
t.Errorf("namespace = %q, want empty (set via --namespace)", id.Namespace)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseIdentity_Empty(t *testing.T) {
|
||||
if _, err := parseIdentity(""); err == nil {
|
||||
t.Errorf("parseIdentity(\"\"): expected error, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestParsePermissions(t *testing.T) {
|
||||
cases := []struct {
|
||||
in string
|
||||
want uint8
|
||||
}{
|
||||
{"", 1},
|
||||
{"read", 1},
|
||||
{"write", 2},
|
||||
{"admin", 4},
|
||||
{"read,write", 3},
|
||||
{"read,write,admin", 7},
|
||||
{"READ,Write", 3},
|
||||
}
|
||||
for _, c := range cases {
|
||||
got, err := parsePermissions(c.in)
|
||||
if err != nil {
|
||||
t.Errorf("parsePermissions(%q): unexpected err %v", c.in, err)
|
||||
continue
|
||||
}
|
||||
if uint8(got) != c.want {
|
||||
t.Errorf("parsePermissions(%q) = %d, want %d", c.in, uint8(got), c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParsePermissions_Unknown(t *testing.T) {
|
||||
if _, err := parsePermissions("read,delete"); err == nil {
|
||||
t.Errorf("parsePermissions(read,delete): expected error, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestACLGrantAndCheck(t *testing.T) {
|
||||
t.Setenv("ORCA_HOME", t.TempDir())
|
||||
resetRootFlags(t)
|
||||
resetACLFlags()
|
||||
|
||||
rootCmd.SetArgs([]string{"acl", "grant", "operator-1", "--namespace", "prod", "--permissions", "read,write"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("acl grant: %v", err)
|
||||
}
|
||||
|
||||
if _, err := os.Stat(paths.ACLPath()); err != nil {
|
||||
t.Fatalf("acl.json not written: %v", err)
|
||||
}
|
||||
|
||||
resetRootFlags(t)
|
||||
resetACLFlags()
|
||||
rootCmd.SetArgs([]string{"acl", "check", "operator-1", "--namespace", "prod", "--permission", "read"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("acl check read: %v", err)
|
||||
}
|
||||
|
||||
resetRootFlags(t)
|
||||
resetACLFlags()
|
||||
rootCmd.SetArgs([]string{"acl", "check", "operator-1", "--namespace", "prod", "--permission", "admin"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatalf("acl check admin: expected denied error, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestACLCheckDeniedExits1(t *testing.T) {
|
||||
t.Setenv("ORCA_HOME", t.TempDir())
|
||||
resetRootFlags(t)
|
||||
resetACLFlags()
|
||||
|
||||
rootCmd.SetArgs([]string{"acl", "check", "ghost", "--namespace", "prod", "--permission", "read"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected denied error for un-granted identity, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestACLRevoke(t *testing.T) {
|
||||
t.Setenv("ORCA_HOME", t.TempDir())
|
||||
resetRootFlags(t)
|
||||
resetACLFlags()
|
||||
|
||||
rootCmd.SetArgs([]string{"acl", "grant", "operator-1", "--namespace", "prod", "--permissions", "read"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("grant: %v", err)
|
||||
}
|
||||
|
||||
resetRootFlags(t)
|
||||
resetACLFlags()
|
||||
rootCmd.SetArgs([]string{"acl", "revoke", "operator-1", "--namespace", "prod"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("revoke: %v", err)
|
||||
}
|
||||
|
||||
resetRootFlags(t)
|
||||
resetACLFlags()
|
||||
rootCmd.SetArgs([]string{"acl", "check", "operator-1", "--namespace", "prod", "--permission", "read"})
|
||||
if err := rootCmd.Execute(); err == nil {
|
||||
t.Fatalf("check after revoke: expected denied, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestACLListEmpty(t *testing.T) {
|
||||
t.Setenv("ORCA_HOME", t.TempDir())
|
||||
resetRootFlags(t)
|
||||
resetACLFlags()
|
||||
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetArgs([]string{"acl", "list"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("acl list empty: %v", err)
|
||||
}
|
||||
if !strings.Contains(buf.String(), "No ACL entries") {
|
||||
t.Errorf("acl list empty: %s", buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestACLListWithEntries(t *testing.T) {
|
||||
t.Setenv("ORCA_HOME", t.TempDir())
|
||||
resetRootFlags(t)
|
||||
resetACLFlags()
|
||||
|
||||
rootCmd.SetArgs([]string{"acl", "grant", "operator-1", "--namespace", "prod", "--permissions", "read,write"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("grant: %v", err)
|
||||
}
|
||||
resetRootFlags(t)
|
||||
resetACLFlags()
|
||||
rootCmd.SetArgs([]string{"acl", "grant", "spiffe://orca.local/ns/myapp/sa/svc1/alloc-1", "--namespace", "myapp", "--permissions", "admin"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("grant spiffe: %v", err)
|
||||
}
|
||||
|
||||
resetRootFlags(t)
|
||||
resetACLFlags()
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetArgs([]string{"acl", "list"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("acl list: %v", err)
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "operator-1") || !strings.Contains(out, "prod") {
|
||||
t.Errorf("list missing operator-1/prod: %s", out)
|
||||
}
|
||||
if !strings.Contains(out, "spiffe://orca.local/ns/myapp") || !strings.Contains(out, "myapp") {
|
||||
t.Errorf("list missing spiffe entry: %s", out)
|
||||
}
|
||||
if !strings.Contains(out, "read,write") || !strings.Contains(out, "admin") {
|
||||
t.Errorf("list missing permissions: %s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestACLListJSON(t *testing.T) {
|
||||
t.Setenv("ORCA_HOME", t.TempDir())
|
||||
resetRootFlags(t)
|
||||
resetACLFlags()
|
||||
|
||||
rootCmd.SetArgs([]string{"acl", "grant", "operator-1", "--namespace", "prod", "--permissions", "read"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("grant: %v", err)
|
||||
}
|
||||
|
||||
resetRootFlags(t)
|
||||
resetACLFlags()
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetArgs([]string{"acl", "list", "--json"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("acl list --json: %v", err)
|
||||
}
|
||||
var entries []map[string]any
|
||||
if err := json.Unmarshal(bytes.TrimSpace(buf.Bytes()), &entries); err != nil {
|
||||
t.Fatalf("unmarshal: %v\n%s", err, buf.String())
|
||||
}
|
||||
if len(entries) != 1 {
|
||||
t.Fatalf("entries len = %d, want 1", len(entries))
|
||||
}
|
||||
id, _ := entries[0]["identity"].(map[string]any)
|
||||
if id == nil || id["id"] != "operator-1" {
|
||||
t.Errorf("identity = %v, want operator-1", entries[0]["identity"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestACLGrantSpiffeNamespaceMismatch(t *testing.T) {
|
||||
t.Setenv("ORCA_HOME", t.TempDir())
|
||||
resetRootFlags(t)
|
||||
resetACLFlags()
|
||||
|
||||
rootCmd.SetArgs([]string{"acl", "grant", "spiffe://orca.local/ns/myapp/sa/svc1/alloc-1", "--namespace", "other"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected mismatch error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "does not match") {
|
||||
t.Errorf("error = %q, want contains 'does not match'", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestACLGrantSpiffeDefaultsNamespaceFromPath(t *testing.T) {
|
||||
t.Setenv("ORCA_HOME", t.TempDir())
|
||||
resetRootFlags(t)
|
||||
resetACLFlags()
|
||||
|
||||
rootCmd.SetArgs([]string{"acl", "grant", "spiffe://orca.local/ns/myapp/sa/svc1/alloc-1", "--permissions", "read"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("grant spiffe (no --namespace): %v", err)
|
||||
}
|
||||
|
||||
resetRootFlags(t)
|
||||
resetACLFlags()
|
||||
rootCmd.SetArgs([]string{"acl", "check", "spiffe://orca.local/ns/myapp/sa/svc1/alloc-1", "--namespace", "myapp", "--permission", "read"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("check spiffe: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestACLGrantTokenRequiresNamespace(t *testing.T) {
|
||||
t.Setenv("ORCA_HOME", t.TempDir())
|
||||
resetRootFlags(t)
|
||||
resetACLFlags()
|
||||
|
||||
rootCmd.SetArgs([]string{"acl", "grant", "operator-1", "--permissions", "read"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected error for token grant without --namespace, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestACLStatePersists(t *testing.T) {
|
||||
t.Setenv("ORCA_HOME", t.TempDir())
|
||||
resetRootFlags(t)
|
||||
resetACLFlags()
|
||||
|
||||
rootCmd.SetArgs([]string{"acl", "grant", "operator-1", "--namespace", "prod", "--permissions", "read"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("grant: %v", err)
|
||||
}
|
||||
|
||||
data, err := os.ReadFile(paths.ACLPath())
|
||||
if err != nil {
|
||||
t.Fatalf("read acl.json: %v", err)
|
||||
}
|
||||
if !strings.Contains(string(data), "operator-1") || !strings.Contains(string(data), "prod") {
|
||||
t.Errorf("acl.json missing entry: %s", string(data))
|
||||
}
|
||||
}
|
||||
|
||||
func TestACLAtomicWriteNoPartialFile(t *testing.T) {
|
||||
t.Setenv("ORCA_HOME", t.TempDir())
|
||||
resetRootFlags(t)
|
||||
resetACLFlags()
|
||||
|
||||
rootCmd.SetArgs([]string{"acl", "grant", "operator-1", "--namespace", "prod", "--permissions", "read"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("grant: %v", err)
|
||||
}
|
||||
entries, err := os.ReadDir(filepath.Dir(paths.ACLPath()))
|
||||
if err != nil {
|
||||
t.Fatalf("readdir cluster: %v", err)
|
||||
}
|
||||
for _, e := range entries {
|
||||
if strings.HasPrefix(e.Name(), ".acl-tmp-") {
|
||||
t.Errorf("leftover temp file: %s", e.Name())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestACLCheckJSONDenied(t *testing.T) {
|
||||
t.Setenv("ORCA_HOME", t.TempDir())
|
||||
resetRootFlags(t)
|
||||
resetACLFlags()
|
||||
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetArgs([]string{"acl", "check", "ghost", "--namespace", "prod", "--permission", "read", "--json"})
|
||||
_ = rootCmd.Execute()
|
||||
var result map[string]any
|
||||
if err := json.Unmarshal(bytes.TrimSpace(buf.Bytes()), &result); err != nil {
|
||||
t.Fatalf("unmarshal: %v\n%s", err, buf.String())
|
||||
}
|
||||
if result["allowed"] != false {
|
||||
t.Errorf("allowed = %v, want false", result["allowed"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestACLAdminImpliesReadCheck(t *testing.T) {
|
||||
t.Setenv("ORCA_HOME", t.TempDir())
|
||||
resetRootFlags(t)
|
||||
resetACLFlags()
|
||||
|
||||
rootCmd.SetArgs([]string{"acl", "grant", "operator-1", "--namespace", "prod", "--permissions", "admin"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("grant admin: %v", err)
|
||||
}
|
||||
|
||||
resetRootFlags(t)
|
||||
resetACLFlags()
|
||||
rootCmd.SetArgs([]string{"acl", "check", "operator-1", "--namespace", "prod", "--permission", "read"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("check read (admin grant): %v", err)
|
||||
}
|
||||
|
||||
resetRootFlags(t)
|
||||
resetACLFlags()
|
||||
rootCmd.SetArgs([]string{"acl", "check", "operator-1", "--namespace", "prod", "--permission", "write"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("check write (admin grant): %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,206 @@
|
||||
// Package cli: auth.go implements the `orca auth` subcommand family
|
||||
// (REQ-144, D-239, D-242, D-246). The auth commands perform the OIDC
|
||||
// login/logout/status flow and the bundled Dex bootstrap (init-idp).
|
||||
//
|
||||
// R-021 invariant: Orca never issues, stores, or accepts human-identity
|
||||
// credentials. The IdP issues tokens; Orca only stores them (short-
|
||||
// lived, 0600, refreshable). No passwords, no Orca-issued tokens.
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"runtime"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/identity"
|
||||
)
|
||||
|
||||
var authCmd = &cobra.Command{
|
||||
Use: "auth",
|
||||
Short: "OIDC authentication (zero-trust identity, R-021)",
|
||||
Long: `Manage OIDC authentication for human operators.
|
||||
|
||||
Orca uses OIDC for human-identity authentication (R-021: no Orca-
|
||||
issued credentials). The bundled Dex (deployed by 'orca auth init-idp')
|
||||
is the default issuer; 'oidc.issuer' in config can repoint to a BYO
|
||||
external IdP. The CLI performs the authorization-code + PKCE + local
|
||||
loopback redirect flow; headless/CI uses the device-code flow.`,
|
||||
}
|
||||
|
||||
var (
|
||||
authIssuer string
|
||||
authClientID string
|
||||
authClientSecret string
|
||||
authDeviceFlow bool
|
||||
authOpenBrowser bool
|
||||
)
|
||||
|
||||
var authLoginCmd = &cobra.Command{
|
||||
Use: "login",
|
||||
Short: "Authenticate via OIDC (browser or device-code flow)",
|
||||
Long: `Perform the OIDC login. By default, opens the default browser
|
||||
for the authorization-code + PKCE + local loopback redirect flow. Use
|
||||
--device-code for the headless/CI flow. Credentials are stored at
|
||||
~/.orca/credentials.json (0600, short-lived + refresh).`,
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
cfg, err := loadOIDCConfig()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
|
||||
defer cancel()
|
||||
client, err := identity.NewOIDCClient(ctx, *cfg)
|
||||
if err != nil {
|
||||
return fmt.Errorf("auth login: %w", err)
|
||||
}
|
||||
if authDeviceFlow {
|
||||
creds, err := client.DeviceFlowLogin(ctx, os.Stdout)
|
||||
if err != nil {
|
||||
return fmt.Errorf("auth login (device): %w", err)
|
||||
}
|
||||
if err := identity.SaveCredentials(creds); err != nil {
|
||||
return fmt.Errorf("auth login: %w", err)
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "✓ Logged in as %s (sub=%s)\n", creds.Issuer, creds.Subject)
|
||||
return nil
|
||||
}
|
||||
openBrowser := func(url string) error {
|
||||
if !authOpenBrowser {
|
||||
fmt.Fprintf(os.Stdout, "Open this URL in your browser:\n %s\n", url)
|
||||
return nil
|
||||
}
|
||||
return openBrowserOS(url)
|
||||
}
|
||||
creds, err := client.Login(ctx, openBrowser)
|
||||
if err != nil {
|
||||
return fmt.Errorf("auth login: %w", err)
|
||||
}
|
||||
if err := identity.SaveCredentials(creds); err != nil {
|
||||
return fmt.Errorf("auth login: %w", err)
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "✓ Logged in as %s (sub=%s, groups=%v)\n", creds.Issuer, creds.Subject, creds.Groups)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
var authLogoutCmd = &cobra.Command{
|
||||
Use: "logout",
|
||||
Short: "Clear the stored OIDC credentials",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
if err := identity.ClearCredentials(); err != nil {
|
||||
return fmt.Errorf("auth logout: %w", err)
|
||||
}
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "✓ Logged out (credentials cleared)")
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
var authStatusCmd = &cobra.Command{
|
||||
Use: "status",
|
||||
Short: "Show the current OIDC authentication status",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
creds, err := identity.LoadCredentials()
|
||||
if err != nil {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "Not authenticated (no credentials)")
|
||||
return nil
|
||||
}
|
||||
expired := time.Now().After(creds.Expiry)
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "Issuer: %s\n", creds.Issuer)
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "Subject: %s\n", creds.Subject)
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "Groups: %v\n", creds.Groups)
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "Expiry: %s\n", creds.Expiry.Format(time.RFC3339))
|
||||
if expired {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "Status: EXPIRED (run 'orca auth login' to refresh)")
|
||||
} else {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "Status: valid")
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
var (
|
||||
authInitIDP string
|
||||
authInitRPID string
|
||||
)
|
||||
|
||||
var authInitIDPCmd = &cobra.Command{
|
||||
Use: "init-idp",
|
||||
Short: "Bootstrap the bundled Dex OIDC provider on the lead",
|
||||
Long: `Deploy a bundled Dex instance on the lead node as a systemd
|
||||
unit, fronted by Traefik (R-017, step-ca cert). This is the default
|
||||
zero-trust identity provider; 'oidc.issuer' can be repointed to a BYO
|
||||
external IdP anytime. The WebAuthn connector (P05) provides the
|
||||
password-free upstream authenticator.
|
||||
|
||||
--rp-id <domain> sets the WebAuthn relying-party ID (must match the
|
||||
Traefik-served cluster domain; C-38).`,
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
if authInitRPID == "" {
|
||||
return fmt.Errorf("--rp-id is required (the cluster's Traefik-served domain for WebAuthn)")
|
||||
}
|
||||
// The full Dex deploy is a systemd unit + Traefik route + config
|
||||
// template. For v0.12 P04 we emit the config + unit files; the
|
||||
// WebAuthn connector ships in P05.
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "Dex bootstrap planned for RP ID: %s\n", authInitRPID)
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "Note: full Dex systemd unit + Traefik route deploy is part of P05 (WebAuthn connector).")
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "This stub confirms the CLI surface; the deploy logic lands with the connector.")
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
// loadOIDCConfig loads the OIDC config from flags or the cluster config.
|
||||
func loadOIDCConfig() (*identity.OIDCConfig, error) {
|
||||
cfg := &identity.OIDCConfig{
|
||||
Issuer: authIssuer,
|
||||
ClientID: authClientID,
|
||||
ClientSecret: authClientSecret,
|
||||
}
|
||||
if cfg.Issuer == "" {
|
||||
// TODO: load from cluster config (oidc block). For v0.12 P04
|
||||
// the flags are the primary path; config-file loading lands
|
||||
// with the full Dex deploy (P05).
|
||||
return nil, fmt.Errorf("auth: --issuer is required (or set oidc.issuer in config)")
|
||||
}
|
||||
if cfg.ClientID == "" {
|
||||
cfg.ClientID = "orca-cli"
|
||||
}
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
// openBrowserOS opens the URL in the default browser.
|
||||
func openBrowserOS(url string) error {
|
||||
switch runtime.GOOS {
|
||||
case "linux":
|
||||
return exec.Command("xdg-open", url).Start()
|
||||
case "darwin":
|
||||
return exec.Command("open", url).Start()
|
||||
case "windows":
|
||||
return exec.Command("rundll32", "url.dll,FileProtocolHandler", url).Start()
|
||||
}
|
||||
return fmt.Errorf("unsupported OS for browser open: %s", runtime.GOOS)
|
||||
}
|
||||
|
||||
func init() {
|
||||
authLoginCmd.Flags().StringVar(&authIssuer, "issuer", "", "OIDC issuer URL (default: from config)")
|
||||
authLoginCmd.Flags().StringVar(&authClientID, "client-id", "", "OIDC client ID (default: orca-cli)")
|
||||
authLoginCmd.Flags().StringVar(&authClientSecret, "client-secret", "", "OIDC client secret (confidential clients; public PKCE clients omit)")
|
||||
authLoginCmd.Flags().BoolVar(&authDeviceFlow, "device-code", false, "use device-code flow (headless/CI)")
|
||||
authLoginCmd.Flags().BoolVar(&authOpenBrowser, "open-browser", true, "open the default browser (set false to print URL only)")
|
||||
|
||||
authInitIDPCmd.Flags().StringVar(&authInitRPID, "rp-id", "", "WebAuthn relying-party ID (cluster Traefik domain)")
|
||||
|
||||
authCmd.AddCommand(authLoginCmd)
|
||||
authCmd.AddCommand(authLogoutCmd)
|
||||
authCmd.AddCommand(authStatusCmd)
|
||||
authCmd.AddCommand(authInitIDPCmd)
|
||||
rootCmd.AddCommand(authCmd)
|
||||
}
|
||||
@@ -0,0 +1,53 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
// TestAuthStatusNotAuthenticated verifies auth status reports
|
||||
// "not authenticated" when no credentials exist.
|
||||
func TestAuthStatusNotAuthenticated(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", dir)
|
||||
resetRootFlags(t)
|
||||
rootCmd.SetArgs([]string{"auth", "status"})
|
||||
// auth status should not error on missing credentials.
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Errorf("auth status on missing creds: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestAuthLogoutNoCreds verifies logout succeeds even with no creds.
|
||||
func TestAuthLogoutNoCreds(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", dir)
|
||||
resetRootFlags(t)
|
||||
rootCmd.SetArgs([]string{"auth", "logout"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Errorf("auth logout with no creds: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestAuthInitIDPRequiresRPID verifies --rp-id is required.
|
||||
func TestAuthInitIDPRequiresRPID(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", dir)
|
||||
resetRootFlags(t)
|
||||
rootCmd.SetArgs([]string{"auth", "init-idp"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Error("auth init-idp without --rp-id should error")
|
||||
}
|
||||
}
|
||||
|
||||
// TestAuthLoginRequiresIssuer verifies --issuer is required.
|
||||
func TestAuthLoginRequiresIssuer(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", dir)
|
||||
resetRootFlags(t)
|
||||
rootCmd.SetArgs([]string{"auth", "login"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Error("auth login without --issuer should error")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,116 @@
|
||||
// Package cli: backup.go implements the `orca backup` and `orca restore`
|
||||
// subcommands (P04, v0.11 milestone).
|
||||
//
|
||||
// orca backup --out <path> — create a signed tar.gz of ORCA_HOME
|
||||
// orca restore --in <path> — restore a verified backup
|
||||
//
|
||||
// `backup` reads the master key at paths.MasterKeyPath() and backs up
|
||||
// paths.Root() (ORCA_HOME). The tarball + HMAC-SHA256 signature are
|
||||
// written to --out and --out+".sig". `restore` verifies the signature
|
||||
// before extracting; with --force it overwrites a non-empty target.
|
||||
package cli
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/backup"
|
||||
"git.cloudinit.dev/coreci/orca/internal/paths"
|
||||
"git.cloudinit.dev/coreci/orca/internal/secrets"
|
||||
)
|
||||
|
||||
var (
|
||||
backupOutPath string
|
||||
restoreInPath string
|
||||
restoreTargetDir string
|
||||
restoreForce bool
|
||||
restoreDryRun bool
|
||||
)
|
||||
|
||||
var backupCmd = &cobra.Command{
|
||||
Use: "backup",
|
||||
Short: "Create a signed tar.gz backup of ORCA_HOME",
|
||||
Long: `Create a signed tar.gz backup of ORCA_HOME (P04).
|
||||
|
||||
Walks ` + "`ORCA_HOME`" + ` recursively, excludes /run/orca/*, *.sock,
|
||||
*.db-wal, *.db-shm, packs the rest into a tar.gz, and computes an
|
||||
HMAC-SHA256 signature using the cluster master key. The tarball is
|
||||
written to --out; the hex-encoded signature to --out + ".sig".`,
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
mk, err := secrets.LoadMasterKey(paths.MasterKeyPath())
|
||||
if err != nil {
|
||||
return fmt.Errorf("load master key: %w", err)
|
||||
}
|
||||
out := backupOutPath
|
||||
if out == "" {
|
||||
ts := time.Now().UTC().Format("20060102-150405")
|
||||
out = fmt.Sprintf("orca-backup-%s.tar.gz", ts)
|
||||
}
|
||||
opts := backup.BackupOptions{
|
||||
SourceDir: paths.Root(),
|
||||
OutputPath: out,
|
||||
MasterKey: mk,
|
||||
}
|
||||
if err := backup.Backup(opts); err != nil {
|
||||
return err
|
||||
}
|
||||
if jsonOutput {
|
||||
return printJSON(map[string]string{
|
||||
"path": out,
|
||||
"sig": out + ".sig",
|
||||
})
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "✓ Backup written: %s (sig: %s)\n", out, out+".sig")
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
var restoreCmd = &cobra.Command{
|
||||
Use: "restore",
|
||||
Short: "Restore ORCA_HOME from a verified signed backup",
|
||||
Long: `Restore ORCA_HOME from a verified signed backup (P04/P07).
|
||||
|
||||
Verifies the HMAC-SHA256 signature on --in (using the cluster master
|
||||
key) before extracting. Reconciles with live state: refuses to clobber
|
||||
running allocations unless --force is given (with --force, stops the
|
||||
running allocs, extracts, then restarts them from the restored state).
|
||||
With --dry-run, extracts to a temp dir and reports what WOULD be
|
||||
restored without touching the real ORCA_HOME. Performs post-restore
|
||||
verification (master key, namespace dirs, SQLite DBs) and records the
|
||||
restore in the audit log.`,
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
if restoreInPath == "" {
|
||||
return fmt.Errorf("--in is required")
|
||||
}
|
||||
mk, err := secrets.LoadMasterKey(paths.MasterKeyPath())
|
||||
if err != nil {
|
||||
return fmt.Errorf("load master key: %w", err)
|
||||
}
|
||||
target := restoreTargetDir
|
||||
if target == "" {
|
||||
target = paths.Root()
|
||||
}
|
||||
opts := RestoreOptions{
|
||||
InputPath: restoreInPath,
|
||||
TargetDir: target,
|
||||
MasterKey: mk,
|
||||
Force: restoreForce,
|
||||
DryRun: restoreDryRun,
|
||||
}
|
||||
return runRestore(cmd, opts)
|
||||
},
|
||||
}
|
||||
|
||||
func init() {
|
||||
backupCmd.Flags().StringVar(&backupOutPath, "out", "", "output tarball path (default: orca-backup-<timestamp>.tar.gz in CWD)")
|
||||
restoreCmd.Flags().StringVar(&restoreInPath, "in", "", "input tarball path (required)")
|
||||
restoreCmd.Flags().StringVar(&restoreTargetDir, "target", "", "restore target dir (default: ORCA_HOME)")
|
||||
restoreCmd.Flags().BoolVar(&restoreForce, "force", false, "overwrite a non-empty target directory and stop+restart running allocs")
|
||||
restoreCmd.Flags().BoolVar(&restoreDryRun, "dry-run", false, "extract to a temp dir and report what would be restored without touching ORCA_HOME")
|
||||
rootCmd.AddCommand(backupCmd)
|
||||
rootCmd.AddCommand(restoreCmd)
|
||||
}
|
||||
@@ -0,0 +1,177 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/paths"
|
||||
"git.cloudinit.dev/coreci/orca/internal/secrets"
|
||||
)
|
||||
|
||||
func setupBackupTestEnv(t *testing.T) string {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", dir)
|
||||
mk, err := secrets.GenerateMasterKey()
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateMasterKey: %v", err)
|
||||
}
|
||||
if err := os.MkdirAll(paths.ClusterDir(), 0o755); err != nil {
|
||||
t.Fatalf("mkdir cluster dir: %v", err)
|
||||
}
|
||||
if err := secrets.SaveMasterKey(paths.MasterKeyPath(), mk); err != nil {
|
||||
t.Fatalf("SaveMasterKey: %v", err)
|
||||
}
|
||||
return dir
|
||||
}
|
||||
|
||||
func TestBackupCmdRegistered(t *testing.T) {
|
||||
found := false
|
||||
for _, cmd := range rootCmd.Commands() {
|
||||
if cmd.Name() == "backup" {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Fatal("backup command not registered on root")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestoreCmdRegistered(t *testing.T) {
|
||||
found := false
|
||||
for _, cmd := range rootCmd.Commands() {
|
||||
if cmd.Name() == "restore" {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Fatal("restore command not registered on root")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackupRestoreCmdRoundTrip(t *testing.T) {
|
||||
home := setupBackupTestEnv(t)
|
||||
|
||||
if err := os.WriteFile(filepath.Join(home, "keep.txt"), []byte("payload"), 0o644); err != nil {
|
||||
t.Fatalf("write keep.txt: %v", err)
|
||||
}
|
||||
|
||||
outDir := t.TempDir()
|
||||
out := filepath.Join(outDir, "orca-backup.tar.gz")
|
||||
|
||||
var buf bytes.Buffer
|
||||
resetRootFlags(t)
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"backup", "--out", out})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("orca backup: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(out + ".sig"); err != nil {
|
||||
t.Fatalf("sig missing: %v", err)
|
||||
}
|
||||
|
||||
target := filepath.Join(t.TempDir(), "restored")
|
||||
var buf2 bytes.Buffer
|
||||
resetRootFlags(t)
|
||||
rootCmd.SetOut(&buf2)
|
||||
rootCmd.SetErr(&buf2)
|
||||
rootCmd.SetArgs([]string{"restore", "--in", out, "--target", target})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("orca restore: %v", err)
|
||||
}
|
||||
|
||||
got, err := os.ReadFile(filepath.Join(target, "keep.txt"))
|
||||
if err != nil {
|
||||
t.Fatalf("restored keep.txt missing: %v", err)
|
||||
}
|
||||
if string(got) != "payload" {
|
||||
t.Errorf("restored keep.txt = %q, want %q", string(got), "payload")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestoreCmdBadSignature(t *testing.T) {
|
||||
setupBackupTestEnv(t)
|
||||
|
||||
outDir := t.TempDir()
|
||||
out := filepath.Join(outDir, "orca-backup.tar.gz")
|
||||
body := []byte("not a real tarball")
|
||||
if err := os.WriteFile(out, body, 0o644); err != nil {
|
||||
t.Fatalf("write fake tarball: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(out+".sig", []byte("deadbeef"), 0o644); err != nil {
|
||||
t.Fatalf("write fake sig: %v", err)
|
||||
}
|
||||
|
||||
target := filepath.Join(t.TempDir(), "restored")
|
||||
var buf bytes.Buffer
|
||||
resetRootFlags(t)
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"restore", "--in", out, "--target", target})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("restore with bad signature should fail")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestoreCmdRequiresInFlag(t *testing.T) {
|
||||
setupBackupTestEnv(t)
|
||||
var buf bytes.Buffer
|
||||
resetRootFlags(t)
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"restore"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("restore without --in should fail")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackupCmdDefaultOut(t *testing.T) {
|
||||
home := setupBackupTestEnv(t)
|
||||
if err := os.WriteFile(filepath.Join(home, "f.txt"), []byte("x"), 0o644); err != nil {
|
||||
t.Fatalf("write f.txt: %v", err)
|
||||
}
|
||||
|
||||
work := t.TempDir()
|
||||
orig, _ := os.Getwd()
|
||||
if err := os.Chdir(work); err != nil {
|
||||
t.Fatalf("chdir: %v", err)
|
||||
}
|
||||
defer os.Chdir(orig)
|
||||
|
||||
var buf bytes.Buffer
|
||||
resetRootFlags(t)
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"backup"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("orca backup default out: %v", err)
|
||||
}
|
||||
|
||||
entries, err := os.ReadDir(work)
|
||||
if err != nil {
|
||||
t.Fatalf("read work: %v", err)
|
||||
}
|
||||
var foundTar, foundSig bool
|
||||
for _, e := range entries {
|
||||
if e.Name() == "orca-backup" || strings.HasPrefix(e.Name(), "orca-backup-") && strings.HasSuffix(e.Name(), ".tar.gz") {
|
||||
foundTar = true
|
||||
}
|
||||
if strings.HasSuffix(e.Name(), ".tar.gz.sig") {
|
||||
foundSig = true
|
||||
}
|
||||
}
|
||||
if !foundTar {
|
||||
t.Errorf("default backup tarball not created in CWD (entries: %d)", len(entries))
|
||||
}
|
||||
if !foundSig {
|
||||
t.Errorf("default backup sig not created in CWD (entries: %d)", len(entries))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,218 @@
|
||||
// Package cli: cache.go implements the `orca cache` subcommand family
|
||||
// (P00-T3, R-008) and the shared cache helpers used by the read-only
|
||||
// list commands (node/job/ns list).
|
||||
//
|
||||
// The cache is optional: if the cache DB cannot be opened (missing dir,
|
||||
// permissions, corrupt file) the list commands fall back to the
|
||||
// uncached read path silently with a slog.Warn.
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/cache"
|
||||
"git.cloudinit.dev/coreci/orca/internal/paths"
|
||||
)
|
||||
|
||||
// cacheHit reports whether the cache returned a fresh entry for
|
||||
// (class, key). On any cache-open or read error it returns false (miss)
|
||||
// and logs a warning — the caller proceeds to the uncached path. The
|
||||
// cache never *creates* ORCA_HOME: if the parent directory is missing
|
||||
// the cache is skipped silently so that source-read errors (e.g. `orca
|
||||
// ns list` against a nonexistent ORCA_HOME) still surface.
|
||||
func cacheHit(class, key string) ([]byte, bool) {
|
||||
if !cacheAvailable() {
|
||||
return nil, false
|
||||
}
|
||||
c, err := cache.Open(paths.CacheDB())
|
||||
if err != nil {
|
||||
slog.Warn("cache: open failed, falling back to uncached path", "class", class, "err", err)
|
||||
return nil, false
|
||||
}
|
||||
defer c.Close()
|
||||
val, _, err := c.Get(class, key)
|
||||
if err != nil {
|
||||
if !errors.Is(err, cache.ErrCacheMiss) {
|
||||
slog.Warn("cache: get failed, falling back to uncached path", "class", class, "err", err)
|
||||
}
|
||||
return nil, false
|
||||
}
|
||||
return val, true
|
||||
}
|
||||
|
||||
// cachePopulate stores val for (class, key) with the given ttl. Errors
|
||||
// are logged but never returned — a failed populate must not break
|
||||
// the list command.
|
||||
func cachePopulate(class, key string, val []byte, ttl time.Duration) {
|
||||
if !cacheAvailable() {
|
||||
return
|
||||
}
|
||||
c, err := cache.Open(paths.CacheDB())
|
||||
if err != nil {
|
||||
slog.Warn("cache: open failed during populate", "class", class, "err", err)
|
||||
return
|
||||
}
|
||||
defer c.Close()
|
||||
if err := c.Set(class, key, val, ttl); err != nil {
|
||||
slog.Warn("cache: populate failed", "class", class, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// cacheAvailable reports whether the cache DB parent dir (ORCA_HOME)
|
||||
// exists. The cache layer must never create ORCA_HOME; doing so would
|
||||
// mask source-read errors like `orca ns list` against a missing home.
|
||||
func cacheAvailable() bool {
|
||||
info, err := os.Stat(paths.Root())
|
||||
if err != nil || !info.IsDir() {
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// cacheGetList returns the cached JSON list for (class, key), or nil
|
||||
// if miss/any error. It is the read-side helper for list commands.
|
||||
func cacheGetList(class, key string, out any) bool {
|
||||
val, ok := cacheHit(class, key)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
if err := json.Unmarshal(val, out); err != nil {
|
||||
slog.Warn("cache: unmarshal failed, falling back to uncached path", "class", class, "err", err)
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// cachePutList stores list as JSON under (class, key) with ttl. Used
|
||||
// by list commands after fetching from source.
|
||||
func cachePutList(class, key string, list any, ttl time.Duration) {
|
||||
val, err := json.Marshal(list)
|
||||
if err != nil {
|
||||
slog.Warn("cache: marshal failed during populate", "class", class, "err", err)
|
||||
return
|
||||
}
|
||||
cachePopulate(class, key, val, ttl)
|
||||
}
|
||||
|
||||
// Per-class TTLs (P00-T2).
|
||||
const (
|
||||
cacheNodeTTL = 30 * time.Second
|
||||
cacheJobTTL = 10 * time.Second
|
||||
cacheNamespaceTTL = 60 * time.Second
|
||||
cacheNodeClass = "nodes"
|
||||
cacheJobClass = "jobs"
|
||||
cacheNamespaceClass = "namespaces"
|
||||
cacheListKey = "list"
|
||||
)
|
||||
|
||||
// --- `orca cache` CLI (P00-T3) ---
|
||||
|
||||
var cacheCmd = &cobra.Command{
|
||||
Use: "cache",
|
||||
Short: "Inspect or invalidate the orca CLI cache",
|
||||
Long: `Manage the CLI-side SQLite cache (R-008) at
|
||||
` + "`" + `ORCA_HOME/orca_cache.db` + "`" + `.
|
||||
|
||||
Subcommands:
|
||||
show — print per-class entry counts, total size, oldest entry
|
||||
invalidate <c> — drop all entries for a class (e.g. "nodes", "jobs")
|
||||
invalidate-all — drop every entry in the cache
|
||||
|
||||
Read-only list commands (node/job/ns list) populate the cache; writes
|
||||
bypass it. The --watch flag bypasses the cache entirely (streaming).`,
|
||||
}
|
||||
|
||||
var cacheShowCmd = &cobra.Command{
|
||||
Use: "show",
|
||||
Short: "Print cache stats (per-class counts, sizes, oldest entry)",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
c, err := cache.Open(paths.CacheDB())
|
||||
if err != nil {
|
||||
return fmt.Errorf("open cache: %w", err)
|
||||
}
|
||||
defer c.Close()
|
||||
stats, err := c.Stats()
|
||||
if err != nil {
|
||||
return fmt.Errorf("cache stats: %w", err)
|
||||
}
|
||||
if jsonOutput {
|
||||
return printJSON(stats)
|
||||
}
|
||||
out := cmd.OutOrStdout()
|
||||
if len(stats) == 0 {
|
||||
fmt.Fprintln(out, "Cache is empty.")
|
||||
return nil
|
||||
}
|
||||
fmt.Fprintf(out, "%-20s %-8s %-12s %s\n", "CLASS", "COUNT", "BYTES", "OLDEST")
|
||||
var totalCount, totalBytes int64
|
||||
for _, s := range stats {
|
||||
oldest := time.Unix(0, s.OldestAt).UTC().Format(time.RFC3339)
|
||||
if s.OldestAt == 0 {
|
||||
oldest = "-"
|
||||
}
|
||||
fmt.Fprintf(out, "%-20s %-8d %-12d %s\n", s.Class, s.Count, s.Bytes, oldest)
|
||||
totalCount += int64(s.Count)
|
||||
totalBytes += s.Bytes
|
||||
}
|
||||
fmt.Fprintf(out, "%-20s %-8d %-12d\n", "TOTAL", totalCount, totalBytes)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
var cacheInvalidateCmd = &cobra.Command{
|
||||
Use: "invalidate <class>",
|
||||
Short: "Drop all entries for a cache class",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
class := args[0]
|
||||
c, err := cache.Open(paths.CacheDB())
|
||||
if err != nil {
|
||||
return fmt.Errorf("open cache: %w", err)
|
||||
}
|
||||
defer c.Close()
|
||||
if err := c.Invalidate(class); err != nil {
|
||||
return fmt.Errorf("invalidate %s: %w", class, err)
|
||||
}
|
||||
if jsonOutput {
|
||||
return printJSON(map[string]string{"class": class, "status": "invalidated"})
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "✓ Cache invalidated: %s\n", class)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
var cacheInvalidateAllCmd = &cobra.Command{
|
||||
Use: "invalidate-all",
|
||||
Short: "Drop every entry in the cache",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
c, err := cache.Open(paths.CacheDB())
|
||||
if err != nil {
|
||||
return fmt.Errorf("open cache: %w", err)
|
||||
}
|
||||
defer c.Close()
|
||||
if err := c.InvalidateAll(); err != nil {
|
||||
return fmt.Errorf("invalidate-all: %w", err)
|
||||
}
|
||||
if jsonOutput {
|
||||
return printJSON(map[string]string{"status": "invalidated"})
|
||||
}
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "✓ Cache cleared.")
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
func init() {
|
||||
cacheCmd.AddCommand(cacheShowCmd)
|
||||
cacheCmd.AddCommand(cacheInvalidateCmd)
|
||||
cacheCmd.AddCommand(cacheInvalidateAllCmd)
|
||||
rootCmd.AddCommand(cacheCmd)
|
||||
}
|
||||
@@ -0,0 +1,344 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/cache"
|
||||
"git.cloudinit.dev/coreci/orca/internal/paths"
|
||||
)
|
||||
|
||||
func TestCacheCommandRegistered(t *testing.T) {
|
||||
registered := make(map[string]bool)
|
||||
for _, cmd := range rootCmd.Commands() {
|
||||
registered[cmd.Name()] = true
|
||||
}
|
||||
if !registered["cache"] {
|
||||
t.Fatal("cache command not registered on root")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCacheSubcommands(t *testing.T) {
|
||||
expected := []string{"show", "invalidate", "invalidate-all"}
|
||||
registered := make(map[string]bool)
|
||||
for _, cmd := range cacheCmd.Commands() {
|
||||
registered[cmd.Name()] = true
|
||||
}
|
||||
for _, name := range expected {
|
||||
if !registered[name] {
|
||||
t.Errorf("expected cache subcommand %q not registered", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCacheShowEmpty(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"cache", "show"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("cache show: %v", err)
|
||||
}
|
||||
if !strings.Contains(buf.String(), "Cache is empty.") {
|
||||
t.Errorf("cache show empty: %s", buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestCacheShowAfterPopulate(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
c, err := cache.Open(paths.CacheDB())
|
||||
if err != nil {
|
||||
t.Fatalf("open cache: %v", err)
|
||||
}
|
||||
if err := c.Set("nodes", "list", []byte("hello"), 0); err != nil {
|
||||
t.Fatalf("set: %v", err)
|
||||
}
|
||||
if err := c.Set("jobs", "list", []byte("hi"), 0); err != nil {
|
||||
t.Fatalf("set: %v", err)
|
||||
}
|
||||
c.Close()
|
||||
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"cache", "show"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("cache show: %v", err)
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "nodes") || !strings.Contains(out, "jobs") {
|
||||
t.Errorf("cache show missing classes: %s", out)
|
||||
}
|
||||
if !strings.Contains(out, "TOTAL") {
|
||||
t.Errorf("cache show missing TOTAL row: %s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCacheShowJSON(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
c, err := cache.Open(paths.CacheDB())
|
||||
if err != nil {
|
||||
t.Fatalf("open cache: %v", err)
|
||||
}
|
||||
if err := c.Set("nodes", "list", []byte("abc"), 0); err != nil {
|
||||
t.Fatalf("set: %v", err)
|
||||
}
|
||||
c.Close()
|
||||
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"cache", "show", "--json"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("cache show --json: %v", err)
|
||||
}
|
||||
var stats []cache.ClassStats
|
||||
if err := json.Unmarshal(bytes.TrimSpace(buf.Bytes()), &stats); err != nil {
|
||||
t.Fatalf("unmarshal: %v\n%s", err, buf.String())
|
||||
}
|
||||
if len(stats) != 1 || stats[0].Class != "nodes" || stats[0].Count != 1 || stats[0].Bytes != 3 {
|
||||
t.Errorf("unexpected stats: %+v", stats)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCacheInvalidate(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
c, err := cache.Open(paths.CacheDB())
|
||||
if err != nil {
|
||||
t.Fatalf("open cache: %v", err)
|
||||
}
|
||||
if err := c.Set("nodes", "list", []byte("a"), 0); err != nil {
|
||||
t.Fatalf("set: %v", err)
|
||||
}
|
||||
if err := c.Set("jobs", "list", []byte("b"), 0); err != nil {
|
||||
t.Fatalf("set: %v", err)
|
||||
}
|
||||
c.Close()
|
||||
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"cache", "invalidate", "nodes"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("cache invalidate: %v", err)
|
||||
}
|
||||
if !strings.Contains(buf.String(), "invalidated") {
|
||||
t.Errorf("invalidate output: %s", buf.String())
|
||||
}
|
||||
|
||||
c2, err := cache.Open(paths.CacheDB())
|
||||
if err != nil {
|
||||
t.Fatalf("reopen: %v", err)
|
||||
}
|
||||
defer c2.Close()
|
||||
if _, _, err := c2.Get("nodes", "list"); err == nil {
|
||||
t.Errorf("nodes/list still present after invalidate")
|
||||
}
|
||||
if _, _, err := c2.Get("jobs", "list"); err != nil {
|
||||
t.Errorf("jobs/list should survive nodes invalidate: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCacheInvalidateAll(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
c, err := cache.Open(paths.CacheDB())
|
||||
if err != nil {
|
||||
t.Fatalf("open cache: %v", err)
|
||||
}
|
||||
_ = c.Set("nodes", "list", []byte("a"), 0)
|
||||
_ = c.Set("jobs", "list", []byte("b"), 0)
|
||||
c.Close()
|
||||
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"cache", "invalidate-all"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("cache invalidate-all: %v", err)
|
||||
}
|
||||
if !strings.Contains(buf.String(), "cleared") {
|
||||
t.Errorf("invalidate-all output: %s", buf.String())
|
||||
}
|
||||
|
||||
c2, err := cache.Open(paths.CacheDB())
|
||||
if err != nil {
|
||||
t.Fatalf("reopen: %v", err)
|
||||
}
|
||||
defer c2.Close()
|
||||
stats, err := c2.Stats()
|
||||
if err != nil {
|
||||
t.Fatalf("stats: %v", err)
|
||||
}
|
||||
if len(stats) != 0 {
|
||||
t.Errorf("cache not empty after invalidate-all: %+v", stats)
|
||||
}
|
||||
}
|
||||
|
||||
// TestNodeListCachedPopulate verifies the read path populates the cache
|
||||
// and a subsequent invocation is served from the cache (without
|
||||
// touching the registry DB).
|
||||
func TestNodeListCachedPopulate(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
rootCmd.SetArgs([]string{"node", "join", "--name", "cacher", "--addr", "10.0.0.9:8443"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("node join: %v", err)
|
||||
}
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"node", "list"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("first node list: %v", err)
|
||||
}
|
||||
if !strings.Contains(buf.String(), "cacher") {
|
||||
t.Fatalf("first list missing node: %s", buf.String())
|
||||
}
|
||||
|
||||
c, err := cache.Open(paths.CacheDB())
|
||||
if err != nil {
|
||||
t.Fatalf("open cache: %v", err)
|
||||
}
|
||||
val, _, err := c.Get(cacheNodeClass, cacheListKey)
|
||||
if err != nil {
|
||||
t.Fatalf("cache miss after populate: %v", err)
|
||||
}
|
||||
if !strings.Contains(string(val), "cacher") {
|
||||
t.Errorf("cached value missing node: %s", val)
|
||||
}
|
||||
c.Close()
|
||||
|
||||
resetRootFlags(t)
|
||||
var buf2 bytes.Buffer
|
||||
rootCmd.SetOut(&buf2)
|
||||
rootCmd.SetErr(&buf2)
|
||||
rootCmd.SetArgs([]string{"node", "list"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("second (cached) node list: %v", err)
|
||||
}
|
||||
if !strings.Contains(buf2.String(), "cacher") {
|
||||
t.Errorf("cached list missing node: %s", buf2.String())
|
||||
}
|
||||
}
|
||||
|
||||
// TestJobListCachedPopulate verifies the job list read path populates the
|
||||
// cache and a subsequent invocation is served from the cache.
|
||||
func TestJobListCachedPopulate(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
// First list: empty, should populate cache with [].
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"job", "list"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("first job list: %v", err)
|
||||
}
|
||||
if !strings.Contains(buf.String(), "No jobs") {
|
||||
t.Fatalf("first list not empty: %s", buf.String())
|
||||
}
|
||||
|
||||
c, err := cache.Open(paths.CacheDB())
|
||||
if err != nil {
|
||||
t.Fatalf("open cache: %v", err)
|
||||
}
|
||||
val, _, err := c.Get(cacheJobClass, cacheListKey)
|
||||
if err != nil {
|
||||
t.Fatalf("cache miss after populate: %v", err)
|
||||
}
|
||||
if len(val) == 0 || string(val) == "null" {
|
||||
// empty jobs list marshals to "null"; that's still a cached miss
|
||||
// populated by the read path. Just confirm the entry exists.
|
||||
}
|
||||
c.Close()
|
||||
}
|
||||
|
||||
// TestNSListCachedPopulate verifies the ns list read path populates the cache.
|
||||
func TestNSListCachedPopulate(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
writeDefaultsNS(t, root)
|
||||
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"ns", "list"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("first ns list: %v", err)
|
||||
}
|
||||
|
||||
c, err := cache.Open(paths.CacheDB())
|
||||
if err != nil {
|
||||
t.Fatalf("open cache: %v", err)
|
||||
}
|
||||
val, _, err := c.Get(cacheNamespaceClass, cacheListKey)
|
||||
if err != nil {
|
||||
t.Fatalf("cache miss after populate: %v", err)
|
||||
}
|
||||
if !strings.Contains(string(val), paths.DefaultNamespace()) {
|
||||
t.Errorf("cached value missing _defaults: %s", val)
|
||||
}
|
||||
c.Close()
|
||||
}
|
||||
|
||||
// TestNodeListWatchBypassesCache verifies --watch does not populate
|
||||
// the cache (streaming path).
|
||||
func TestNodeListWatchBypassesCache(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
// --watch with no nodes: watchNodesCtx returns immediately when the
|
||||
// watch channel closes. Use a short timeout via signal context.
|
||||
// We just assert the cache is NOT populated for the "nodes" class.
|
||||
// (We don't invoke --watch directly because it blocks; instead we
|
||||
// verify the cache helper leaves the class untouched.)
|
||||
c, err := cache.Open(paths.CacheDB())
|
||||
if err != nil {
|
||||
t.Fatalf("open cache: %v", err)
|
||||
}
|
||||
sentinel := []byte(`[{"id":"sentinel-id","name":"sentinel","address":"10.0.0.99:8443","state":"ready"}]`)
|
||||
if err := c.Set(cacheNodeClass, cacheListKey, sentinel, 0); err != nil {
|
||||
t.Fatalf("set sentinel: %v", err)
|
||||
}
|
||||
c.Close()
|
||||
|
||||
// Non-watch list should read the sentinel back from the cache.
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"node", "list"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("node list: %v", err)
|
||||
}
|
||||
if !strings.Contains(buf.String(), "sentinel") {
|
||||
t.Errorf("cache hit not surfaced (sentinel missing): %s", buf.String())
|
||||
}
|
||||
}
|
||||
+14
-1
@@ -42,6 +42,11 @@ func ServerCertPath() string { return certpaths.ServerCertPath() }
|
||||
func ServerKeyPath() string { return certpaths.ServerKeyPath() }
|
||||
|
||||
// NewCommand builds the `orca cert` command tree.
|
||||
//
|
||||
// Deprecated: v0.9 re-architecture replaces the internal CA with step-ca
|
||||
// (D-101/REQ-076). The `orca cert` command tree is retained for the
|
||||
// dual-write window and scheduled for deletion in v0.10. See
|
||||
// .ciagent/PRD_v0.9.md.
|
||||
func NewCommand(log *slog.Logger) *cobra.Command {
|
||||
if log == nil {
|
||||
log = slog.Default()
|
||||
@@ -49,7 +54,12 @@ func NewCommand(log *slog.Logger) *cobra.Command {
|
||||
certCmd := &cobra.Command{
|
||||
Use: "cert",
|
||||
Short: "Manage orca certificates (CA, server, rotation)",
|
||||
Long: "Bootstrap a local CA, generate server certs, and rotate them.",
|
||||
Long: `Manage orca certificates (CA, server, rotation).
|
||||
|
||||
Deprecated: v0.9 re-architecture replaces the internal CA with step-ca
|
||||
(D-101/REQ-076). The ` + "`orca cert`" + ` command tree is retained for the
|
||||
dual-write window and scheduled for deletion in v0.10. See
|
||||
.ciagent/PRD_v0.9.md.`,
|
||||
}
|
||||
|
||||
certCmd.AddCommand(newCAInitCmd(log))
|
||||
@@ -67,6 +77,7 @@ func newCAInitCmd(log *slog.Logger) *cobra.Command {
|
||||
Short: "Initialize a local orca CA (ca.crt + ca.key) under ~/.orca",
|
||||
Long: "Generates a new RSA CA cert and writes it to ~/.orca/ca.crt (0644) and ~/.orca/ca.key (0600) per REQ-033.",
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
warnDeprecated("orca cert ca-init is deprecated: use step-ca (R-006); the internal CA is replaced by step-ca (D-101) — see .ciagent/PRD_v0.9.md")
|
||||
dir := CADir()
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
return fmt.Errorf("mkdir %s: %w", dir, err)
|
||||
@@ -100,6 +111,7 @@ func newGenCmd(log *slog.Logger) *cobra.Command {
|
||||
Short: "Generate a server cert (CSR + sign) under ~/.orca",
|
||||
Long: "Builds a CSR with the requested SANs, signs it with the local CA, and writes server.crt + server.key.",
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
warnDeprecated("orca cert gen is deprecated: use step-ca + orca node join for cert generation (D-101) — see .ciagent/PRD_v0.9.md")
|
||||
dir := CADir()
|
||||
if cn == "" {
|
||||
cn = "orca-server"
|
||||
@@ -171,6 +183,7 @@ func newRenewCmd(log *slog.Logger) *cobra.Command {
|
||||
Short: "Rotate the server cert (hot-swapped by the daemon; REQ-034)",
|
||||
Long: "Re-runs `cert gen` and overwrites server.crt / server.key in place. The daemon's GetCertificate callback picks up the new cert on the next handshake — no restart required.",
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
warnDeprecated("orca cert renew is deprecated: use step-ca for cert rotation (D-101) — see .ciagent/PRD_v0.9.md")
|
||||
dir := CADir()
|
||||
if cn == "" {
|
||||
cn = "orca-server"
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
var clusterCmd = &cobra.Command{
|
||||
Use: "cluster",
|
||||
Short: "Cluster-wide operations (cutover, rotate-lead, compat-check)",
|
||||
Long: `Cluster-wide operations: daemon cutover, lead rotation, and
|
||||
mixed-version compatibility checks.`,
|
||||
}
|
||||
|
||||
func init() {
|
||||
clusterCmd.AddCommand(clusterCutoverCmd, clusterRotateLeadCmd, compatCheckCmd)
|
||||
rootCmd.AddCommand(clusterCmd)
|
||||
}
|
||||
@@ -0,0 +1,417 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/emit"
|
||||
"git.cloudinit.dev/coreci/orca/internal/model"
|
||||
)
|
||||
|
||||
var noOrcaOnServerCmd = &cobra.Command{
|
||||
Use: "no-orca-on-server",
|
||||
Short: "Verify no orca binary/service/process on peers (REQ-086, R-001, C-13)",
|
||||
Long: `SSH to each registered peer and verify that no orca binary,
|
||||
systemd service, or process is present on the server (R-001: no orca
|
||||
binary on any server; C-13 enforcement).
|
||||
|
||||
Checks per peer:
|
||||
1. command -v orca → must return nothing (no orca in PATH)
|
||||
2. systemctl list-units 'orca*' (excluding orca-alloc-*) → must be empty
|
||||
3. pgrep orca → must return nothing (no orca process)
|
||||
4. /etc/orca/ contains no orca binaries (config dir is OK)
|
||||
|
||||
A peer with any violation is reported as FAIL. The exit code is non-zero
|
||||
if any peer fails.`,
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
return runNoOrcaOnServer(cmd)
|
||||
},
|
||||
}
|
||||
|
||||
type noOrcaPeerResult struct {
|
||||
Node string `json:"node"`
|
||||
Peer string `json:"peer"`
|
||||
Pass bool `json:"pass"`
|
||||
Violations []string `json:"violations,omitempty"`
|
||||
}
|
||||
|
||||
func runNoOrcaOnServer(cmd *cobra.Command) error {
|
||||
ctx, cancel := context.WithTimeout(cmd.Context(), 5*time.Minute)
|
||||
defer cancel()
|
||||
|
||||
reg, closer, err := nodeRegistry()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer closer()
|
||||
|
||||
nodes, err := reg.List(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("list nodes: %w", err)
|
||||
}
|
||||
|
||||
ex, err := drainExecFromCtx(cmd.Context())
|
||||
if err != nil {
|
||||
return fmt.Errorf("ssh transport: %w", err)
|
||||
}
|
||||
|
||||
log := newLogger()
|
||||
results := make([]noOrcaPeerResult, 0, len(nodes))
|
||||
var failedNodes []string
|
||||
|
||||
for i := range nodes {
|
||||
n := nodes[i]
|
||||
peer := peerAddrForNode(n)
|
||||
if peer == "" {
|
||||
continue
|
||||
}
|
||||
r := noOrcaPeerResult{Node: n.Name, Peer: peer, Pass: true, Violations: []string{}}
|
||||
|
||||
if v, ok := checkNoOrcaBinary(ctx, ex, peer); !ok {
|
||||
r.Pass = false
|
||||
r.Violations = append(r.Violations, v)
|
||||
}
|
||||
if v, ok := checkNoOrcaService(ctx, ex, peer); !ok {
|
||||
r.Pass = false
|
||||
r.Violations = append(r.Violations, v)
|
||||
}
|
||||
if v, ok := checkNoOrcaProcess(ctx, ex, peer); !ok {
|
||||
r.Pass = false
|
||||
r.Violations = append(r.Violations, v)
|
||||
}
|
||||
if v, ok := checkNoOrcaBinInEtc(ctx, ex, peer); !ok {
|
||||
r.Pass = false
|
||||
r.Violations = append(r.Violations, v)
|
||||
}
|
||||
|
||||
if !r.Pass {
|
||||
failedNodes = append(failedNodes, n.Name)
|
||||
log.Warn("no-orca-on-server: violations",
|
||||
slog.String("node", n.Name), slog.Any("violations", r.Violations))
|
||||
}
|
||||
results = append(results, r)
|
||||
}
|
||||
|
||||
summary := map[string]any{
|
||||
"results": results,
|
||||
"failed": failedNodes,
|
||||
}
|
||||
|
||||
if jsonOutput {
|
||||
return printJSON(summary)
|
||||
}
|
||||
out := cmd.OutOrStdout()
|
||||
for _, r := range results {
|
||||
status := "PASS"
|
||||
if !r.Pass {
|
||||
status = "FAIL"
|
||||
}
|
||||
fmt.Fprintf(out, "%-20s %-5s %s\n", r.Node, status, strings.Join(r.Violations, "; "))
|
||||
}
|
||||
if len(failedNodes) > 0 {
|
||||
fmt.Fprintf(out, "\n%d peer(s) failed R-001 enforcement\n", len(failedNodes))
|
||||
return fmt.Errorf("no-orca-on-server: %d peer(s) have violations", len(failedNodes))
|
||||
}
|
||||
fmt.Fprintf(out, "\n✓ all peers clean (R-001 enforced)\n")
|
||||
return nil
|
||||
}
|
||||
|
||||
func checkNoOrcaBinary(ctx context.Context, ex drainExecer, peer string) (string, bool) {
|
||||
out, err := ex.Exec(ctx, peer, "command -v orca 2>/dev/null || true")
|
||||
if err != nil {
|
||||
return "", true
|
||||
}
|
||||
if strings.TrimSpace(string(out)) != "" {
|
||||
return fmt.Sprintf("orca binary in PATH: %s", strings.TrimSpace(string(out))), false
|
||||
}
|
||||
return "", true
|
||||
}
|
||||
|
||||
func checkNoOrcaService(ctx context.Context, ex drainExecer, peer string) (string, bool) {
|
||||
cmd := "systemctl list-units 'orca*' --no-legend --no-pager 2>/dev/null | grep -v 'orca-alloc-' || true"
|
||||
out, err := ex.Exec(ctx, peer, cmd)
|
||||
if err != nil {
|
||||
return "", true
|
||||
}
|
||||
trimmed := strings.TrimSpace(string(out))
|
||||
if trimmed != "" {
|
||||
return fmt.Sprintf("orca systemd service(s) present: %s", trimmed), false
|
||||
}
|
||||
return "", true
|
||||
}
|
||||
|
||||
func checkNoOrcaProcess(ctx context.Context, ex drainExecer, peer string) (string, bool) {
|
||||
out, err := ex.Exec(ctx, peer, "pgrep -x orca 2>/dev/null || true")
|
||||
if err != nil {
|
||||
return "", true
|
||||
}
|
||||
if strings.TrimSpace(string(out)) != "" {
|
||||
return fmt.Sprintf("orca process running: pid(s) %s", strings.TrimSpace(string(out))), false
|
||||
}
|
||||
return "", true
|
||||
}
|
||||
|
||||
func checkNoOrcaBinInEtc(ctx context.Context, ex drainExecer, peer string) (string, bool) {
|
||||
cmd := "find /etc/orca -type f -executable 2>/dev/null | grep -v 'scripts/' | head -5 || true"
|
||||
out, err := ex.Exec(ctx, peer, cmd)
|
||||
if err != nil {
|
||||
return "", true
|
||||
}
|
||||
trimmed := strings.TrimSpace(string(out))
|
||||
if trimmed != "" {
|
||||
return fmt.Sprintf("executable(s) under /etc/orca: %s", trimmed), false
|
||||
}
|
||||
return "", true
|
||||
}
|
||||
|
||||
var compatCheckCmd = &cobra.Command{
|
||||
Use: "compat-check",
|
||||
Short: "Check mixed-version tolerance across peers (REQ-065, C-13)",
|
||||
Long: `Check that the cluster tolerates mixed orca versions during an
|
||||
upgrade window (REQ-065). The lead and peers may run different orca
|
||||
versions during a rolling upgrade; this command verifies:
|
||||
|
||||
- Each peer's orca version (reported)
|
||||
- The txn manifest format is compatible across versions
|
||||
- The render-contract JSON schema (emit.SchemaVersion) is versioned
|
||||
and backward-compatible
|
||||
- No new required fields that old peers don't understand
|
||||
|
||||
Reports: which peers are on which version, any compatibility issues.`,
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
return runCompatCheck(cmd)
|
||||
},
|
||||
}
|
||||
|
||||
type compatPeerResult struct {
|
||||
Node string `json:"node"`
|
||||
Peer string `json:"peer"`
|
||||
Version string `json:"version"`
|
||||
LeadVersion string `json:"lead_version,omitempty"`
|
||||
Compatible bool `json:"compatible"`
|
||||
Issue string `json:"issue,omitempty"`
|
||||
}
|
||||
|
||||
func runCompatCheck(cmd *cobra.Command) error {
|
||||
ctx, cancel := context.WithTimeout(cmd.Context(), 5*time.Minute)
|
||||
defer cancel()
|
||||
|
||||
reg, closer, err := nodeRegistry()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer closer()
|
||||
|
||||
nodes, err := reg.List(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("list nodes: %w", err)
|
||||
}
|
||||
|
||||
ex, err := drainExecFromCtx(cmd.Context())
|
||||
if err != nil {
|
||||
return fmt.Errorf("ssh transport: %w", err)
|
||||
}
|
||||
|
||||
leadVersion := version
|
||||
results := make([]compatPeerResult, 0, len(nodes))
|
||||
var issues []string
|
||||
versionSet := map[string]int{}
|
||||
|
||||
for i := range nodes {
|
||||
n := nodes[i]
|
||||
peer := peerAddrForNode(n)
|
||||
if peer == "" {
|
||||
continue
|
||||
}
|
||||
peerVersion := detectPeerOrcaVersion(ctx, ex, peer)
|
||||
versionSet[peerVersion]++
|
||||
r := compatPeerResult{
|
||||
Node: n.Name,
|
||||
Peer: peer,
|
||||
Version: peerVersion,
|
||||
LeadVersion: leadVersion,
|
||||
Compatible: true,
|
||||
}
|
||||
if peerVersion != "" && peerVersion != leadVersion {
|
||||
if !versionsCompatible(leadVersion, peerVersion) {
|
||||
r.Compatible = false
|
||||
r.Issue = fmt.Sprintf("peer %s (%s) incompatible with lead (%s)",
|
||||
n.Name, peerVersion, leadVersion)
|
||||
issues = append(issues, r.Issue)
|
||||
}
|
||||
}
|
||||
results = append(results, r)
|
||||
}
|
||||
|
||||
schemaOK := verifyRenderContractCompat(ctx, ex, nodes)
|
||||
if !schemaOK {
|
||||
issues = append(issues, "render-contract schema mismatch detected across peers")
|
||||
}
|
||||
manifestOK := verifyTxnManifestCompat(ctx, ex, nodes)
|
||||
if !manifestOK {
|
||||
issues = append(issues, "txn manifest format incompatibility detected")
|
||||
}
|
||||
|
||||
summary := map[string]any{
|
||||
"lead_version": leadVersion,
|
||||
"schema_version": emit.SchemaVersion,
|
||||
"results": results,
|
||||
"versions_seen": versionSet,
|
||||
"issues": issues,
|
||||
"schema_ok": schemaOK,
|
||||
"manifest_ok": manifestOK,
|
||||
}
|
||||
|
||||
if jsonOutput {
|
||||
return printJSON(summary)
|
||||
}
|
||||
out := cmd.OutOrStdout()
|
||||
fmt.Fprintf(out, "lead version: %s (schema %s)\n", leadVersion, emit.SchemaVersion)
|
||||
for _, r := range results {
|
||||
mark := "✓"
|
||||
if !r.Compatible {
|
||||
mark = "✗"
|
||||
}
|
||||
fmt.Fprintf(out, " %s %-20s %s\n", mark, r.Node, r.Version)
|
||||
if r.Issue != "" {
|
||||
fmt.Fprintf(out, " %s\n", r.Issue)
|
||||
}
|
||||
}
|
||||
if len(issues) > 0 {
|
||||
fmt.Fprintf(out, "\n%d compatibility issue(s) found\n", len(issues))
|
||||
return fmt.Errorf("compat-check: %d issue(s)", len(issues))
|
||||
}
|
||||
fmt.Fprintf(out, "\n✓ all peers compatible\n")
|
||||
return nil
|
||||
}
|
||||
|
||||
func detectPeerOrcaVersion(ctx context.Context, ex drainExecer, peer string) string {
|
||||
out, err := ex.Exec(ctx, peer, "orca version --json 2>/dev/null || true")
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
s := strings.TrimSpace(string(out))
|
||||
if s == "" {
|
||||
return ""
|
||||
}
|
||||
var parsed map[string]any
|
||||
if err := json.Unmarshal([]byte(s), &parsed); err == nil {
|
||||
if v, ok := parsed["version"]; ok {
|
||||
if vs, ok := v.(string); ok && vs != "" {
|
||||
return vs
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, line := range strings.Split(s, "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
if strings.Contains(line, "version") {
|
||||
fields := strings.Fields(line)
|
||||
for i, f := range fields {
|
||||
if f == "\"version\":" || f == "version:" {
|
||||
if i+1 < len(fields) {
|
||||
return strings.Trim(fields[i+1], "\",")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func versionsCompatible(lead, peer string) bool {
|
||||
if lead == "" || peer == "" {
|
||||
return true
|
||||
}
|
||||
li := versionMinor(lead)
|
||||
pi := versionMinor(peer)
|
||||
if li == 0 || pi == 0 {
|
||||
return true
|
||||
}
|
||||
diff := li - pi
|
||||
if diff < 0 {
|
||||
diff = -diff
|
||||
}
|
||||
return diff <= 1
|
||||
}
|
||||
|
||||
func versionMinor(v string) int {
|
||||
s := strings.TrimPrefix(v, "v")
|
||||
parts := strings.Split(s, ".")
|
||||
if len(parts) < 2 {
|
||||
return 0
|
||||
}
|
||||
var n int
|
||||
for _, c := range parts[1] {
|
||||
if c >= '0' && c <= '9' {
|
||||
n = n*10 + int(c-'0')
|
||||
} else {
|
||||
break
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
func verifyRenderContractCompat(ctx context.Context, ex drainExecer, nodes []*model.Node) bool {
|
||||
for i := range nodes {
|
||||
n := nodes[i]
|
||||
peer := peerAddrForNode(n)
|
||||
if peer == "" {
|
||||
continue
|
||||
}
|
||||
out, err := ex.Exec(ctx, peer, "test -f /etc/orca/cluster/render-contract.json && cat /etc/orca/cluster/render-contract.json || true")
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
s := strings.TrimSpace(string(out))
|
||||
if s == "" {
|
||||
continue
|
||||
}
|
||||
if !strings.Contains(s, emit.SchemaVersion) && !strings.Contains(s, "schema_version") {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func verifyTxnManifestCompat(ctx context.Context, ex drainExecer, nodes []*model.Node) bool {
|
||||
for i := range nodes {
|
||||
n := nodes[i]
|
||||
peer := peerAddrForNode(n)
|
||||
if peer == "" {
|
||||
continue
|
||||
}
|
||||
out, err := ex.Exec(ctx, peer, "test -d /etc/orca/cluster/txns && ls /etc/orca/cluster/txns | head -1 || true")
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
first := strings.TrimSpace(string(out))
|
||||
if first == "" {
|
||||
continue
|
||||
}
|
||||
man, err := ex.Exec(ctx, peer, fmt.Sprintf("cat /etc/orca/cluster/txns/%s/manifest.json 2>/dev/null || true", first))
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
s := strings.TrimSpace(string(man))
|
||||
if s == "" {
|
||||
continue
|
||||
}
|
||||
if !strings.Contains(s, "txn_id") || !strings.Contains(s, "files") {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func sshQuote(s string) string {
|
||||
return "'" + strings.ReplaceAll(s, "'", "'\\''") + "'"
|
||||
}
|
||||
|
||||
@@ -0,0 +1,326 @@
|
||||
// Package cli: collector.go implements the `orca collector` subcommand
|
||||
// (P09, C-12 opt-in). The collector is the lead-side aggregator +
|
||||
// watchdog pair: `orca-aggregate.sh` runs every 10s (via a systemd
|
||||
// timer) merging per-peer state snapshots into cluster.json, and
|
||||
// `orca-watchdog.sh` runs every 30s detecting aggregator starvation
|
||||
// (C-11).
|
||||
//
|
||||
// `orca collector start` emits the scripts + systemd timers/services
|
||||
// to the lead and enables them. `orca collector stop` disables and
|
||||
// removes them. `orca collector status` reports whether the pair is
|
||||
// running.
|
||||
//
|
||||
// Paths default to the system layout (/etc/orca, /etc/systemd/system);
|
||||
// a `--root` flag (default "/") relocates every emitted path under
|
||||
// <root> for testability (tests use a temp dir).
|
||||
package cli
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
var collectorRoot string
|
||||
|
||||
const (
|
||||
collectorScriptDir = "etc/orca/collector"
|
||||
collectorUnitDir = "etc/systemd/system"
|
||||
collectorAggregateSh = "orca-aggregate.sh"
|
||||
collectorWatchdogSh = "orca-watchdog.sh"
|
||||
collectorAggregateSvc = "orca-aggregate.service"
|
||||
collectorAggregateTmr = "orca-aggregate.timer"
|
||||
collectorWatchdogSvc = "orca-watchdog.service"
|
||||
collectorWatchdogTmr = "orca-watchdog.timer"
|
||||
collectorStateDir = "etc/orca/state"
|
||||
)
|
||||
|
||||
var collectorCmd = &cobra.Command{
|
||||
Use: "collector",
|
||||
Short: "Manage the lead-side collector (aggregator + watchdog) (P09)",
|
||||
Long: `Manage the lead-side collector: the aggregator (orca-aggregate.sh,
|
||||
10s cadence, merges per-peer state into cluster.json + drift-event
|
||||
aggregation per REQ-107) and the watchdog (orca-watchdog.sh, 30s
|
||||
cadence, detects aggregator starvation per C-11). Opt-in (C-12).`,
|
||||
Args: cobra.NoArgs,
|
||||
}
|
||||
|
||||
var collectorStartCmd = &cobra.Command{
|
||||
Use: "start",
|
||||
Short: "Emit the collector scripts + systemd units and enable them",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
root, err := collectorResolveRoot()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := collectorEmit(root); err != nil {
|
||||
return err
|
||||
}
|
||||
if !collectorDryRun {
|
||||
if err := collectorEnable(root); err != nil {
|
||||
return fmt.Errorf("enable: %w", err)
|
||||
}
|
||||
}
|
||||
msg := "collector started"
|
||||
if collectorDryRun {
|
||||
msg = "collector scripts emitted (dry-run, not enabled)"
|
||||
}
|
||||
printResult(msg, map[string]string{"status": "started", "root": root})
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
var collectorStopCmd = &cobra.Command{
|
||||
Use: "stop",
|
||||
Short: "Disable and remove the collector scripts + systemd units",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
root, err := collectorResolveRoot()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !collectorDryRun {
|
||||
if err := collectorDisable(root); err != nil {
|
||||
return fmt.Errorf("disable: %w", err)
|
||||
}
|
||||
}
|
||||
if err := collectorRemove(root); err != nil {
|
||||
return err
|
||||
}
|
||||
msg := "collector stopped"
|
||||
if collectorDryRun {
|
||||
msg = "collector artifacts removed (dry-run, not disabled)"
|
||||
}
|
||||
printResult(msg, map[string]string{"status": "stopped", "root": root})
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
var collectorStatusCmd = &cobra.Command{
|
||||
Use: "status",
|
||||
Short: "Report whether the collector (aggregator + watchdog) is running",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
root, err := collectorResolveRoot()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
aggRunning, wdRunning := collectorRunning(root)
|
||||
overall := "running"
|
||||
if !aggRunning && !wdRunning {
|
||||
overall = "stopped"
|
||||
} else if !aggRunning || !wdRunning {
|
||||
overall = "partial"
|
||||
}
|
||||
printResult(
|
||||
fmt.Sprintf("collector: %s (aggregator=%t watchdog=%t)", overall, aggRunning, wdRunning),
|
||||
map[string]any{
|
||||
"status": overall,
|
||||
"aggregator": aggRunning,
|
||||
"watchdog": wdRunning,
|
||||
"root": root,
|
||||
},
|
||||
)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
var collectorDryRun bool
|
||||
|
||||
func init() {
|
||||
collectorCmd.PersistentFlags().StringVar(&collectorRoot, "root", "/", "install root for emitted paths (default: /; for testing use a temp dir)")
|
||||
collectorStartCmd.Flags().BoolVar(&collectorDryRun, "dry-run", false, "emit scripts/units without enabling or running systemctl")
|
||||
collectorStopCmd.Flags().BoolVar(&collectorDryRun, "dry-run", false, "remove scripts/units without disabling or running systemctl")
|
||||
collectorCmd.AddCommand(collectorStartCmd)
|
||||
collectorCmd.AddCommand(collectorStopCmd)
|
||||
collectorCmd.AddCommand(collectorStatusCmd)
|
||||
rootCmd.AddCommand(collectorCmd)
|
||||
}
|
||||
|
||||
func collectorResolveRoot() (string, error) {
|
||||
r := strings.TrimRight(collectorRoot, "/")
|
||||
if r == "" {
|
||||
r = "/"
|
||||
}
|
||||
if !filepath.IsAbs(r) {
|
||||
return "", fmt.Errorf("--root must be absolute, got %q", collectorRoot)
|
||||
}
|
||||
return r, nil
|
||||
}
|
||||
|
||||
func collectorEmit(root string) error {
|
||||
dirs := []string{
|
||||
filepath.Join(root, collectorScriptDir),
|
||||
filepath.Join(root, collectorUnitDir),
|
||||
filepath.Join(root, collectorStateDir),
|
||||
}
|
||||
for _, d := range dirs {
|
||||
if err := os.MkdirAll(d, 0o755); err != nil {
|
||||
return fmt.Errorf("mkdir %s: %w", d, err)
|
||||
}
|
||||
}
|
||||
files := map[string]struct {
|
||||
Content string
|
||||
Mode os.FileMode
|
||||
}{
|
||||
filepath.Join(root, collectorScriptDir, collectorAggregateSh): {collectorAggregateScript, 0o755},
|
||||
filepath.Join(root, collectorScriptDir, collectorWatchdogSh): {collectorWatchdogScript, 0o755},
|
||||
filepath.Join(root, collectorUnitDir, collectorAggregateSvc): {collectorAggregateUnit, 0o644},
|
||||
filepath.Join(root, collectorUnitDir, collectorAggregateTmr): {collectorAggregateTimer, 0o644},
|
||||
filepath.Join(root, collectorUnitDir, collectorWatchdogSvc): {collectorWatchdogUnit, 0o644},
|
||||
filepath.Join(root, collectorUnitDir, collectorWatchdogTmr): {collectorWatchdogTimer, 0o644},
|
||||
}
|
||||
for path, f := range files {
|
||||
tmp := path + ".tmp"
|
||||
if err := os.WriteFile(tmp, []byte(f.Content), f.Mode); err != nil {
|
||||
return fmt.Errorf("write %s: %w", path, err)
|
||||
}
|
||||
if err := os.Rename(tmp, path); err != nil {
|
||||
return fmt.Errorf("rename %s: %w", path, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func collectorRemove(root string) error {
|
||||
paths := []string{
|
||||
filepath.Join(root, collectorScriptDir, collectorAggregateSh),
|
||||
filepath.Join(root, collectorScriptDir, collectorWatchdogSh),
|
||||
filepath.Join(root, collectorUnitDir, collectorAggregateSvc),
|
||||
filepath.Join(root, collectorUnitDir, collectorAggregateTmr),
|
||||
filepath.Join(root, collectorUnitDir, collectorWatchdogSvc),
|
||||
filepath.Join(root, collectorUnitDir, collectorWatchdogTmr),
|
||||
}
|
||||
for _, p := range paths {
|
||||
if err := os.Remove(p); err != nil && !os.IsNotExist(err) {
|
||||
return fmt.Errorf("remove %s: %w", p, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func collectorRunning(root string) (bool, bool) {
|
||||
aggRunning := fileExists(filepath.Join(root, collectorUnitDir, collectorAggregateSvc)) &&
|
||||
fileExists(filepath.Join(root, collectorScriptDir, collectorAggregateSh))
|
||||
wdRunning := fileExists(filepath.Join(root, collectorUnitDir, collectorWatchdogSvc)) &&
|
||||
fileExists(filepath.Join(root, collectorScriptDir, collectorWatchdogSh))
|
||||
return aggRunning, wdRunning
|
||||
}
|
||||
|
||||
func fileExists(p string) bool {
|
||||
_, err := os.Stat(p)
|
||||
return err == nil
|
||||
}
|
||||
|
||||
func collectorEnable(root string) error {
|
||||
if !commandAvailable("systemctl") {
|
||||
return nil
|
||||
}
|
||||
for _, u := range []string{collectorAggregateTmr, collectorWatchdogTmr} {
|
||||
_ = runSystemctl(root, "enable", u)
|
||||
_ = runSystemctl(root, "start", u)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func collectorDisable(root string) error {
|
||||
if !commandAvailable("systemctl") {
|
||||
return nil
|
||||
}
|
||||
for _, u := range []string{collectorAggregateTmr, collectorWatchdogTmr} {
|
||||
_ = runSystemctl(root, "stop", u)
|
||||
_ = runSystemctl(root, "disable", u)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func runSystemctl(root, action, unit string) error {
|
||||
args := []string{action, unit}
|
||||
if root != "/" {
|
||||
args = append([]string{"--root", root}, args...)
|
||||
}
|
||||
return runCmd("systemctl", args...)
|
||||
}
|
||||
|
||||
func commandAvailable(name string) bool {
|
||||
_, err := exec.LookPath(name)
|
||||
return err == nil
|
||||
}
|
||||
|
||||
func runCmd(name string, args ...string) error {
|
||||
cmd := exec.Command(name, args...)
|
||||
cmd.Stdout = os.Stdout
|
||||
cmd.Stderr = os.Stderr
|
||||
return cmd.Run()
|
||||
}
|
||||
|
||||
const collectorAggregateScript = `#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
# orca-aggregate.sh — emitted by orca collector start (P09).
|
||||
# Placeholder wrapper; the canonical copy lives at scripts/orca-aggregate.sh.
|
||||
exec /usr/local/bin/orca-aggregate.sh "$@"
|
||||
`
|
||||
|
||||
const collectorWatchdogScript = `#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
# orca-watchdog.sh — emitted by orca collector start (P09).
|
||||
# Placeholder wrapper; the canonical copy lives at scripts/orca-watchdog.sh.
|
||||
exec /usr/local/bin/orca-watchdog.sh "$@"
|
||||
`
|
||||
|
||||
const collectorAggregateUnit = `[Unit]
|
||||
Description=orca aggregator (P09, C-11/C-12)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/etc/orca/collector/orca-aggregate.sh
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
`
|
||||
|
||||
const collectorAggregateTimer = `[Unit]
|
||||
Description=orca aggregator 10s cadence (P09, C-11)
|
||||
|
||||
[Timer]
|
||||
OnBootSec=10s
|
||||
OnUnitActiveSec=10s
|
||||
AccuracySec=1s
|
||||
Unit=orca-aggregate.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
`
|
||||
|
||||
const collectorWatchdogUnit = `[Unit]
|
||||
Description=orca watchdog meta-timer (P09, C-11)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/etc/orca/collector/orca-watchdog.sh
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
`
|
||||
|
||||
const collectorWatchdogTimer = `[Unit]
|
||||
Description=orca watchdog 30s cadence (P09, C-11)
|
||||
|
||||
[Timer]
|
||||
OnBootSec=30s
|
||||
OnUnitActiveSec=30s
|
||||
AccuracySec=5s
|
||||
Unit=orca-watchdog.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
`
|
||||
@@ -0,0 +1,160 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestCollectorCmdRegistered(t *testing.T) {
|
||||
found := false
|
||||
for _, cmd := range rootCmd.Commands() {
|
||||
if cmd.Name() == "collector" {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Fatal("collector command not registered on root")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCollectorSubcommandsRegistered(t *testing.T) {
|
||||
want := map[string]bool{"start": false, "stop": false, "status": false}
|
||||
for _, cmd := range collectorCmd.Commands() {
|
||||
if _, ok := want[cmd.Name()]; ok {
|
||||
want[cmd.Name()] = true
|
||||
}
|
||||
}
|
||||
for name, found := range want {
|
||||
if !found {
|
||||
t.Errorf("collector subcommand %q not registered", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func runCollectorCmd(t *testing.T, root string, dryRun bool, args ...string) (string, error) {
|
||||
t.Helper()
|
||||
resetRootFlags(t)
|
||||
full := append([]string{"collector"}, args...)
|
||||
if root != "" {
|
||||
full = append(full, "--root", root)
|
||||
}
|
||||
if dryRun && (len(args) > 0 && (args[0] == "start" || args[0] == "stop")) {
|
||||
full = append(full, "--dry-run")
|
||||
}
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs(full)
|
||||
err := rootCmd.Execute()
|
||||
return buf.String(), err
|
||||
}
|
||||
|
||||
func TestCollectorStartEmitsArtifacts(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
out, err := runCollectorCmd(t, root, true, "start")
|
||||
if err != nil {
|
||||
t.Fatalf("orca collector start: %v\n%s", err, out)
|
||||
}
|
||||
wantFiles := []string{
|
||||
filepath.Join(root, collectorScriptDir, collectorAggregateSh),
|
||||
filepath.Join(root, collectorScriptDir, collectorWatchdogSh),
|
||||
filepath.Join(root, collectorUnitDir, collectorAggregateSvc),
|
||||
filepath.Join(root, collectorUnitDir, collectorAggregateTmr),
|
||||
filepath.Join(root, collectorUnitDir, collectorWatchdogSvc),
|
||||
filepath.Join(root, collectorUnitDir, collectorWatchdogTmr),
|
||||
}
|
||||
for _, p := range wantFiles {
|
||||
if _, err := os.Stat(p); err != nil {
|
||||
t.Errorf("expected emitted file %s: %v", p, err)
|
||||
}
|
||||
}
|
||||
for _, p := range []string{
|
||||
filepath.Join(root, collectorScriptDir, collectorAggregateSh),
|
||||
filepath.Join(root, collectorScriptDir, collectorWatchdogSh),
|
||||
} {
|
||||
info, err := os.Stat(p)
|
||||
if err != nil {
|
||||
t.Fatalf("stat %s: %v", p, err)
|
||||
}
|
||||
if perm := info.Mode().Perm(); perm&0o111 == 0 {
|
||||
t.Errorf("expected executable bit on %s, got %o", p, perm)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCollectorStartStatusRunning(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
if _, err := runCollectorCmd(t, root, true, "start"); err != nil {
|
||||
t.Fatalf("start: %v", err)
|
||||
}
|
||||
agg, wd := collectorRunning(root)
|
||||
if !agg || !wd {
|
||||
t.Errorf("expected both running, got agg=%t wd=%t", agg, wd)
|
||||
}
|
||||
out, err := runCollectorCmd(t, root, true, "status")
|
||||
if err != nil {
|
||||
t.Fatalf("status: %v", err)
|
||||
}
|
||||
if !contains(out, "running") {
|
||||
t.Errorf("status output should say running, got %q", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCollectorStopRemovesArtifacts(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
if _, err := runCollectorCmd(t, root, true, "start"); err != nil {
|
||||
t.Fatalf("start: %v", err)
|
||||
}
|
||||
out, err := runCollectorCmd(t, root, true, "stop")
|
||||
if err != nil {
|
||||
t.Fatalf("stop: %v\n%s", err, out)
|
||||
}
|
||||
wantFiles := []string{
|
||||
filepath.Join(root, collectorScriptDir, collectorAggregateSh),
|
||||
filepath.Join(root, collectorScriptDir, collectorWatchdogSh),
|
||||
filepath.Join(root, collectorUnitDir, collectorAggregateSvc),
|
||||
filepath.Join(root, collectorUnitDir, collectorAggregateTmr),
|
||||
filepath.Join(root, collectorUnitDir, collectorWatchdogSvc),
|
||||
filepath.Join(root, collectorUnitDir, collectorWatchdogTmr),
|
||||
}
|
||||
for _, p := range wantFiles {
|
||||
if _, err := os.Stat(p); !os.IsNotExist(err) {
|
||||
t.Errorf("expected %s removed, got %v", p, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCollectorStatusWhenStopped(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
out, err := runCollectorCmd(t, root, true, "status")
|
||||
if err != nil {
|
||||
t.Fatalf("status: %v", err)
|
||||
}
|
||||
if !contains(out, "stopped") {
|
||||
t.Errorf("expected stopped, got %q", out)
|
||||
}
|
||||
agg, wd := collectorRunning(root)
|
||||
if agg || wd {
|
||||
t.Errorf("expected neither running, got agg=%t wd=%t", agg, wd)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCollectorResolveRootRejectsRelative(t *testing.T) {
|
||||
collectorRoot = "tmp/relative"
|
||||
_, err := collectorResolveRoot()
|
||||
if err == nil {
|
||||
t.Error("expected error for relative root")
|
||||
}
|
||||
collectorRoot = "/"
|
||||
r, err := collectorResolveRoot()
|
||||
if err != nil || r != "/" {
|
||||
t.Errorf("expected / for default, got %q err=%v", r, err)
|
||||
}
|
||||
}
|
||||
|
||||
func contains(haystack, needle string) bool {
|
||||
return bytes.Contains([]byte(haystack), []byte(needle))
|
||||
}
|
||||
@@ -0,0 +1,184 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/certpaths"
|
||||
"git.cloudinit.dev/coreci/orca/internal/engine"
|
||||
"git.cloudinit.dev/coreci/orca/internal/store"
|
||||
)
|
||||
|
||||
var cutoverTimeout time.Duration
|
||||
|
||||
var clusterCutoverCmd = &cobra.Command{
|
||||
Use: "cutover",
|
||||
Short: "Stop v0.8 orca daemons and adopt running allocs (P14b)",
|
||||
Long: `Stop the v0.8 orca-daemon on every peer that still runs one,
|
||||
discover its running allocations (orca-alloc-*.service), and adopt each
|
||||
into the SSH-push path (mark it managed by the CLI-side scheduler).
|
||||
|
||||
The allocation's systemd unit keeps running independently of the
|
||||
daemon; the cutover only re-records ownership in the cluster store
|
||||
and stops the daemon.
|
||||
|
||||
Idempotent: a peer whose daemon is already stopped is a no-op for that
|
||||
peer. Re-adopting an already-adopted alloc is a no-op.`,
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
return runCutover(cmd)
|
||||
},
|
||||
}
|
||||
|
||||
func runCutover(cmd *cobra.Command) error {
|
||||
ctx, cancel := context.WithTimeout(cmd.Context(), cutoverTimeout)
|
||||
defer cancel()
|
||||
|
||||
reg, closer, err := nodeRegistry()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer closer()
|
||||
|
||||
nodes, err := reg.List(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("list nodes: %w", err)
|
||||
}
|
||||
|
||||
ex, err := drainExecFromCtx(cmd.Context())
|
||||
if err != nil {
|
||||
return fmt.Errorf("ssh transport: %w", err)
|
||||
}
|
||||
|
||||
log := newLogger()
|
||||
|
||||
type peerResult struct {
|
||||
Node string `json:"node"`
|
||||
Peer string `json:"peer"`
|
||||
DaemonStopped bool `json:"daemon_stopped"`
|
||||
AlreadyStopped bool `json:"already_stopped"`
|
||||
Adopted []string `json:"adopted"`
|
||||
Failed string `json:"failed,omitempty"`
|
||||
}
|
||||
|
||||
results := make([]peerResult, 0, len(nodes))
|
||||
var stopped, already, failed, adopted []string
|
||||
|
||||
for i := range nodes {
|
||||
n := nodes[i]
|
||||
peer := peerAddrForNode(n)
|
||||
if peer == "" {
|
||||
continue
|
||||
}
|
||||
pr := peerResult{Node: n.Name, Peer: peer, Adopted: []string{}}
|
||||
|
||||
stopCmd := "systemctl stop orca-daemon.service"
|
||||
_, stopErr := ex.Exec(ctx, peer, stopCmd)
|
||||
switch {
|
||||
case stopErr == nil:
|
||||
pr.DaemonStopped = true
|
||||
stopped = append(stopped, n.Name)
|
||||
default:
|
||||
var exitErr *sshExitErr
|
||||
if errors.As(stopErr, &exitErr) && exitErr.code == 5 {
|
||||
pr.AlreadyStopped = true
|
||||
already = append(already, n.Name)
|
||||
} else {
|
||||
pr.Failed = stopErr.Error()
|
||||
failed = append(failed, n.Name)
|
||||
results = append(results, pr)
|
||||
log.Warn("cutover: stop daemon failed",
|
||||
slog.String("node", n.Name), slog.String("peer", peer), "error", stopErr)
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
ids, listErr := listRunningAllocs(ctx, ex, peer)
|
||||
if listErr != nil {
|
||||
pr.Failed = listErr.Error()
|
||||
failed = append(failed, n.Name)
|
||||
results = append(results, pr)
|
||||
log.Warn("cutover: list allocs failed",
|
||||
slog.String("node", n.Name), slog.String("peer", peer), "error", listErr)
|
||||
continue
|
||||
}
|
||||
for _, id := range ids {
|
||||
if err := adoptAlloc(ctx, n.ID, id); err != nil {
|
||||
log.Warn("cutover: adopt alloc failed",
|
||||
slog.String("alloc", id), slog.String("node", n.Name), "error", err)
|
||||
pr.Failed = fmt.Sprintf("%sadopt %s: %v", pr.Failed, id, err)
|
||||
continue
|
||||
}
|
||||
pr.Adopted = append(pr.Adopted, id)
|
||||
adopted = append(adopted, n.Name+"/"+id)
|
||||
}
|
||||
results = append(results, pr)
|
||||
}
|
||||
|
||||
summary := map[string]any{
|
||||
"stopped": stopped,
|
||||
"already_stopped": already,
|
||||
"failed": failed,
|
||||
"adopted": adopted,
|
||||
"per_node": results,
|
||||
}
|
||||
|
||||
if db, dbErr := store.Open(certpaths.DBPath()); dbErr == nil {
|
||||
engine.NewAudit(store.NewAuditRepo(db), log).Record(ctx, "cli", "cluster.cutover", "cluster", "success", nil, summary)
|
||||
db.Close()
|
||||
}
|
||||
|
||||
if jsonOutput {
|
||||
return printJSON(summary)
|
||||
}
|
||||
out := cmd.OutOrStdout()
|
||||
fmt.Fprintf(out, "✓ cutover complete (%d stopped, %d already stopped, %d failed, %d adopted)\n",
|
||||
len(stopped), len(already), len(failed), len(adopted))
|
||||
for _, n := range stopped {
|
||||
fmt.Fprintf(out, " stopped %s\n", n)
|
||||
}
|
||||
for _, n := range already {
|
||||
fmt.Fprintf(out, " already-stopped %s\n", n)
|
||||
}
|
||||
for _, a := range adopted {
|
||||
fmt.Fprintf(out, " adopted %s\n", a)
|
||||
}
|
||||
for _, n := range failed {
|
||||
fmt.Fprintf(out, " failed %s\n", n)
|
||||
}
|
||||
if len(failed) > 0 {
|
||||
return fmt.Errorf("cutover: %d peer(s) failed", len(failed))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func adoptAlloc(ctx context.Context, nodeID, allocID string) error {
|
||||
db, err := store.Open(certpaths.DBPath())
|
||||
if err != nil {
|
||||
return fmt.Errorf("open db: %w", err)
|
||||
}
|
||||
defer db.Close()
|
||||
hist := store.NewAllocHistoryRepo(db)
|
||||
if err := hist.EnsureSchema(ctx); err != nil {
|
||||
return fmt.Errorf("alloc history schema: %w", err)
|
||||
}
|
||||
entry := store.AllocHistoryEntry{
|
||||
AllocID: allocID,
|
||||
NodeID: nodeID,
|
||||
FromState: "daemon-managed",
|
||||
ToState: "ssh-push-managed",
|
||||
Timestamp: time.Now().UTC(),
|
||||
Reason: "p14b-cutover",
|
||||
}
|
||||
return hist.Record(ctx, entry)
|
||||
}
|
||||
|
||||
func init() {
|
||||
clusterCutoverCmd.Flags().DurationVar(&cutoverTimeout, "timeout", 5*time.Minute,
|
||||
"max time for the full cutover across all peers")
|
||||
}
|
||||
@@ -0,0 +1,518 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/certpaths"
|
||||
"git.cloudinit.dev/coreci/orca/internal/cluster"
|
||||
"git.cloudinit.dev/coreci/orca/internal/model"
|
||||
"git.cloudinit.dev/coreci/orca/internal/paths"
|
||||
"git.cloudinit.dev/coreci/orca/internal/store"
|
||||
)
|
||||
|
||||
type mockRotateTransport struct {
|
||||
mu sync.Mutex
|
||||
calls []mockRotateCall
|
||||
written []mockRotateWrite
|
||||
responses []mockRotateResp
|
||||
sticky []mockRotateResp
|
||||
}
|
||||
|
||||
type mockRotateCall struct {
|
||||
peer string
|
||||
cmd string
|
||||
}
|
||||
|
||||
type mockRotateWrite struct {
|
||||
peer string
|
||||
path string
|
||||
content []byte
|
||||
mode os.FileMode
|
||||
}
|
||||
|
||||
type mockRotateResp struct {
|
||||
match string
|
||||
out string
|
||||
exit int
|
||||
}
|
||||
|
||||
func (m *mockRotateTransport) Exec(_ context.Context, peer, cmd string) ([]byte, error) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
m.calls = append(m.calls, mockRotateCall{peer: peer, cmd: cmd})
|
||||
for _, r := range m.sticky {
|
||||
if r.match == "" || strings.Contains(cmd, r.match) {
|
||||
if r.exit != 0 {
|
||||
return []byte(r.out), &sshExitErr{code: r.exit}
|
||||
}
|
||||
return []byte(r.out), nil
|
||||
}
|
||||
}
|
||||
for i, r := range m.responses {
|
||||
if r.match == "" || strings.Contains(cmd, r.match) {
|
||||
m.responses = append(m.responses[:i], m.responses[i+1:]...)
|
||||
if r.exit != 0 {
|
||||
return []byte(r.out), &sshExitErr{code: r.exit}
|
||||
}
|
||||
return []byte(r.out), nil
|
||||
}
|
||||
}
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
func (m *mockRotateTransport) WriteFileIdempotent(_ context.Context, peer, path string, content []byte, mode os.FileMode) (bool, error) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
m.written = append(m.written, mockRotateWrite{peer: peer, path: path, content: content, mode: mode})
|
||||
return true, nil
|
||||
}
|
||||
|
||||
func (m *mockRotateTransport) ReadFile(_ context.Context, peer, path string) ([]byte, error) {
|
||||
return nil, errors.New("not implemented")
|
||||
}
|
||||
|
||||
func (m *mockRotateTransport) countCalls(match string) int {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
c := 0
|
||||
for _, call := range m.calls {
|
||||
if strings.Contains(call.cmd, match) {
|
||||
c++
|
||||
}
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
func (m *mockRotateTransport) writtenPaths() []string {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
out := make([]string, 0, len(m.written))
|
||||
for _, w := range m.written {
|
||||
out = append(out, w.path)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func cutoverTestNode(t *testing.T, name string) *model.Node {
|
||||
t.Helper()
|
||||
db, err := store.Open(certpaths.DBPath())
|
||||
if err != nil {
|
||||
t.Fatalf("open db: %v", err)
|
||||
}
|
||||
defer db.Close()
|
||||
n := &model.Node{
|
||||
ID: "node-" + name,
|
||||
Name: name,
|
||||
Address: name + ":8443",
|
||||
State: model.NodeStateReady,
|
||||
JoinedAt: time.Now().UTC(),
|
||||
LastSeen: time.Now().UTC(),
|
||||
Kind: string(model.NodeKindLinux),
|
||||
}
|
||||
if err := store.NewNodeRepo(db).Insert(context.Background(), n); err != nil {
|
||||
t.Fatalf("insert node: %v", err)
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
func TestCutover_StopsDaemonAndAdoptsAllocs(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
cutoverTestNode(t, "peer-a")
|
||||
cutoverTestNode(t, "peer-b")
|
||||
|
||||
mx := &scriptedDrainExec{}
|
||||
mx.queueAlways("systemctl stop orca-daemon.service", "", 0)
|
||||
mx.queueAlways("list-units", "orca-alloc-web-0.service loaded active running\n", 0)
|
||||
drainExecOverride = mx
|
||||
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
clusterCmd.SetOut(&buf)
|
||||
clusterCmd.SetErr(&buf)
|
||||
clusterCutoverCmd.SetOut(&buf)
|
||||
clusterCutoverCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"cluster", "cutover", "--timeout", "10s"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("cutover: %v", err)
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "cutover complete") {
|
||||
t.Errorf("expected cutover complete, got: %s", out)
|
||||
}
|
||||
stops := mx.countCalls("systemctl stop orca-daemon.service")
|
||||
if stops != 2 {
|
||||
t.Errorf("expected 2 daemon stops, got %d", stops)
|
||||
}
|
||||
if !strings.Contains(out, "adopted") {
|
||||
t.Errorf("expected adopted in output, got: %s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCutover_AlreadyStoppedIsIdempotent(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
cutoverTestNode(t, "migrated")
|
||||
|
||||
mx := &scriptedDrainExec{}
|
||||
mx.queueAlways("systemctl stop orca-daemon.service", "", 5)
|
||||
mx.queueAlways("list-units", "", 0)
|
||||
drainExecOverride = mx
|
||||
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
clusterCutoverCmd.SetOut(&buf)
|
||||
clusterCutoverCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"cluster", "cutover", "--timeout", "5s"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("cutover should be idempotent: %v", err)
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "already stopped") && !strings.Contains(out, "already-stopped") {
|
||||
t.Errorf("expected already-stopped, got: %s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRotateLead_CopiesClusterStateAndRotatesSSHKeys(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
target := cutoverTestNode(t, "newlead")
|
||||
_ = cutoverTestNode(t, "other-peer")
|
||||
|
||||
clusterDir := paths.ClusterDir()
|
||||
if err := os.MkdirAll(clusterDir, 0o755); err != nil {
|
||||
t.Fatalf("mkdir cluster dir: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(certpaths.CACertPath(), []byte("FAKE-CA-CRT"), 0o644); err != nil {
|
||||
t.Fatalf("write ca.crt: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(certpaths.CAKeyPath(), []byte("FAKE-CA-KEY"), 0o600); err != nil {
|
||||
t.Fatalf("write ca.key: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(paths.MasterKeyPath(), []byte("FAKE-MASTER-KEY"), 0o600); err != nil {
|
||||
t.Fatalf("write master.key: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(paths.ConfigPath(), []byte("# orca config"), 0o644); err != nil {
|
||||
t.Fatalf("write config.md: %v", err)
|
||||
}
|
||||
|
||||
mt := &mockRotateTransport{}
|
||||
mt.sticky = append(mt.sticky, mockRotateResp{match: "mkdir -p", out: "", exit: 0})
|
||||
mt.sticky = append(mt.sticky, mockRotateResp{match: "authorized_keys", out: "", exit: 0})
|
||||
driftTransportOverride = mt
|
||||
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
clusterCmd.SetOut(&buf)
|
||||
clusterCmd.SetErr(&buf)
|
||||
clusterRotateLeadCmd.SetOut(&buf)
|
||||
clusterRotateLeadCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"cluster", "rotate-lead", "--to", target.Name})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("rotate-lead: %v", err)
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "lead rotated") {
|
||||
t.Errorf("expected lead rotated, got: %s", out)
|
||||
}
|
||||
written := mt.writtenPaths()
|
||||
if !containsPath(written, "/etc/orca/cluster/ca.key") {
|
||||
t.Errorf("expected ca.key to be copied, written: %v", written)
|
||||
}
|
||||
if !containsPath(written, "/etc/orca/cluster/master.key") {
|
||||
t.Errorf("expected master.key to be copied, written: %v", written)
|
||||
}
|
||||
|
||||
lead, err := readCurrentLead(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("read lead: %v", err)
|
||||
}
|
||||
if lead != target.Name {
|
||||
t.Errorf("lead = %q, want %q", lead, target.Name)
|
||||
}
|
||||
|
||||
newKey, err := os.ReadFile(certpaths.SSHKeyPath())
|
||||
if err != nil {
|
||||
t.Fatalf("read new ssh key: %v", err)
|
||||
}
|
||||
if !strings.Contains(string(newKey), "PRIVATE KEY") {
|
||||
t.Errorf("expected a new private key to be written, got: %s", string(newKey))
|
||||
}
|
||||
}
|
||||
|
||||
func TestRotateLead_ProxmoxTargetRefused(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
db, err := store.Open(certpaths.DBPath())
|
||||
if err != nil {
|
||||
t.Fatalf("open db: %v", err)
|
||||
}
|
||||
defer db.Close()
|
||||
proxmoxNode := &model.Node{
|
||||
ID: "node-prox",
|
||||
Name: "prox-node",
|
||||
Address: "prox-node:8443",
|
||||
State: model.NodeStateReady,
|
||||
JoinedAt: time.Now().UTC(),
|
||||
LastSeen: time.Now().UTC(),
|
||||
Kind: string(model.NodeKindProxmox),
|
||||
}
|
||||
if err := store.NewNodeRepo(db).Insert(context.Background(), proxmoxNode); err != nil {
|
||||
t.Fatalf("insert proxmox node: %v", err)
|
||||
}
|
||||
|
||||
mt := &mockRotateTransport{}
|
||||
driftTransportOverride = mt
|
||||
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
clusterRotateLeadCmd.SetOut(&buf)
|
||||
clusterRotateLeadCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"cluster", "rotate-lead", "--to", "prox-node"})
|
||||
err = rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected error for Proxmox target, got nil")
|
||||
}
|
||||
if !errors.Is(err, cluster.ErrProxmoxNotLead) && !strings.Contains(err.Error(), "Proxmox") {
|
||||
t.Errorf("expected Proxmox refusal, got: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRotateLead_AlreadyLeadIsNoop(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
target := cutoverTestNode(t, "currentlead")
|
||||
if err := writeCurrentLead(context.Background(), target.Name); err != nil {
|
||||
t.Fatalf("write lead: %v", err)
|
||||
}
|
||||
|
||||
mt := &mockRotateTransport{}
|
||||
driftTransportOverride = mt
|
||||
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
clusterRotateLeadCmd.SetOut(&buf)
|
||||
clusterRotateLeadCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"cluster", "rotate-lead", "--to", target.Name})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("rotate-lead should be no-op when already lead: %v", err)
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "already") {
|
||||
t.Errorf("expected already-lead message, got: %s", out)
|
||||
}
|
||||
if len(mt.calls) != 0 {
|
||||
t.Errorf("expected no SSH calls for no-op, got: %+v", mt.calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNoOrcaOnServer_CleanPeerPasses(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
cutoverTestNode(t, "clean-peer")
|
||||
|
||||
mx := &scriptedDrainExec{}
|
||||
mx.queueAlways("command -v orca", "", 0)
|
||||
mx.queueAlways("systemctl list-units", "", 0)
|
||||
mx.queueAlways("pgrep", "", 0)
|
||||
mx.queueAlways("find /etc/orca", "", 0)
|
||||
drainExecOverride = mx
|
||||
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
noOrcaOnServerCmd.SetOut(&buf)
|
||||
noOrcaOnServerCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"doctor", "no-orca-on-server"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("no-orca-on-server (clean): %v", err)
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "PASS") {
|
||||
t.Errorf("expected PASS for clean peer, got: %s", out)
|
||||
}
|
||||
if !strings.Contains(out, "all peers clean") {
|
||||
t.Errorf("expected all-peers-clean message, got: %s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNoOrcaOnServer_DirtyPeerBinaryReportsViolation(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
cutoverTestNode(t, "dirty-bin")
|
||||
|
||||
mx := &scriptedDrainExec{}
|
||||
mx.queueAlways("command -v orca", "/usr/local/bin/orca\n", 0)
|
||||
mx.queueAlways("systemctl list-units", "", 0)
|
||||
mx.queueAlways("pgrep", "", 0)
|
||||
mx.queueAlways("find /etc/orca", "", 0)
|
||||
drainExecOverride = mx
|
||||
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
noOrcaOnServerCmd.SetOut(&buf)
|
||||
noOrcaOnServerCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"doctor", "no-orca-on-server"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected error for dirty peer, got nil")
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "FAIL") {
|
||||
t.Errorf("expected FAIL for dirty peer, got: %s", out)
|
||||
}
|
||||
if !strings.Contains(out, "binary in PATH") {
|
||||
t.Errorf("expected binary-in-PATH violation, got: %s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNoOrcaOnServer_DirtyPeerProcessReportsViolation(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
cutoverTestNode(t, "dirty-proc")
|
||||
|
||||
mx := &scriptedDrainExec{}
|
||||
mx.queueAlways("command -v orca", "", 0)
|
||||
mx.queueAlways("systemctl list-units", "", 0)
|
||||
mx.queueAlways("pgrep", "12345\n", 0)
|
||||
mx.queueAlways("find /etc/orca", "", 0)
|
||||
drainExecOverride = mx
|
||||
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
noOrcaOnServerCmd.SetOut(&buf)
|
||||
noOrcaOnServerCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"doctor", "no-orca-on-server"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected error for dirty peer (process), got nil")
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "process running") {
|
||||
t.Errorf("expected process-running violation, got: %s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCompatCheck_AllSameVersionPasses(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
cutoverTestNode(t, "peer-a")
|
||||
cutoverTestNode(t, "peer-b")
|
||||
|
||||
mx := &scriptedDrainExec{}
|
||||
mx.queueAlways("orca version", "{\"version\":\""+version+"\"}\n", 0)
|
||||
mx.queueAlways("test -f /etc/orca/cluster/render-contract.json", "", 0)
|
||||
mx.queueAlways("test -d /etc/orca/cluster/txns", "", 0)
|
||||
drainExecOverride = mx
|
||||
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
compatCheckCmd.SetOut(&buf)
|
||||
compatCheckCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"cluster", "compat-check"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("compat-check (same): %v", err)
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "all peers compatible") {
|
||||
t.Errorf("expected all-peers-compatible, got: %s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCompatCheck_MixedCompatiblePasses(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
cutoverTestNode(t, "peer-a")
|
||||
|
||||
mx := &scriptedDrainExec{}
|
||||
mx.queueAlways("orca version", "{\"version\":\"0.1.1\"}\n", 0)
|
||||
mx.queueAlways("test -f /etc/orca/cluster/render-contract.json", "", 0)
|
||||
mx.queueAlways("test -d /etc/orca/cluster/txns", "", 0)
|
||||
drainExecOverride = mx
|
||||
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
compatCheckCmd.SetOut(&buf)
|
||||
compatCheckCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"cluster", "compat-check"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Logf("output: %s", buf.String())
|
||||
t.Fatalf("compat-check (mixed-compatible): %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCompatCheck_IncompatibleReportsIssue(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
cutoverTestNode(t, "peer-old")
|
||||
|
||||
mx := &scriptedDrainExec{}
|
||||
mx.queueAlways("orca version", "{\"version\":\"0.8.0\"}\n", 0)
|
||||
mx.queueAlways("test -f /etc/orca/cluster/render-contract.json", "", 0)
|
||||
mx.queueAlways("test -d /etc/orca/cluster/txns", "", 0)
|
||||
drainExecOverride = mx
|
||||
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
compatCheckCmd.SetOut(&buf)
|
||||
compatCheckCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"cluster", "compat-check"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected compat-check to fail for incompatible versions, got nil")
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "incompatible") {
|
||||
t.Errorf("expected incompatible in output, got: %s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func containsPath(paths []string, want string) bool {
|
||||
for _, p := range paths {
|
||||
if p == want {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func init() {}
|
||||
@@ -4,7 +4,6 @@ import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
@@ -26,8 +25,14 @@ var (
|
||||
var daemonCmd = &cobra.Command{
|
||||
Use: "daemon",
|
||||
Short: "Run the orca daemon (HTTP API + health checks)",
|
||||
Long: "Start the orca daemon. Listens on the configured address for health, API, and dispatch requests.",
|
||||
Long: `Start the orca daemon. Listens on the configured address for health, API, and dispatch requests.
|
||||
|
||||
Deprecated: v0.9 re-architecture replaces the orca daemon with SSH-push to
|
||||
bare servers (R-001 — no orca binary on servers). The daemon is repurposed to
|
||||
drain-and-stop in v0.10-P05 and scheduled for deletion in v0.10-P14. See
|
||||
.ciagent/PRD_v0.9.md.`,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
warnDeprecated("orca daemon is deprecated in v0.9 and will be repurposed to 'drain-and-stop' in v0.10-P05; the v0.9 re-architecture (R-001) removes the orca binary from servers — see .ciagent/PRD_v0.9.md")
|
||||
db, closer, err := openDB()
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -97,5 +102,4 @@ func init() {
|
||||
daemonCmd.Flags().StringVar(&daemonAddr, "addr", ":8080", "listen address")
|
||||
daemonCmd.Flags().StringVar(&pprofAddr, "pprof", "", "enable pprof endpoint on <addr> (e.g. :6060); unauthenticated, operator-only")
|
||||
rootCmd.AddCommand(daemonCmd)
|
||||
_ = slog.Default // keep import if unused above
|
||||
}
|
||||
|
||||
+221
-1
@@ -1,6 +1,12 @@
|
||||
package cli
|
||||
|
||||
import "testing"
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"log/slog"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestDaemonPprofFlag(t *testing.T) {
|
||||
f := daemonCmd.Flags().Lookup("pprof")
|
||||
@@ -11,3 +17,217 @@ func TestDaemonPprofFlag(t *testing.T) {
|
||||
t.Errorf("--pprof default = %q, want empty", f.DefValue)
|
||||
}
|
||||
}
|
||||
|
||||
// captureSlog swaps slog.Default() for a text handler writing to buf,
|
||||
// returning a buffer and a restore func. Tests use this to observe
|
||||
// warnDeprecated output (which uses the package-level slog.Default).
|
||||
func captureSlog(t *testing.T) (*bytes.Buffer, func()) {
|
||||
t.Helper()
|
||||
var buf bytes.Buffer
|
||||
prev := slog.Default()
|
||||
logger := slog.New(slog.NewTextHandler(&buf, &slog.HandlerOptions{Level: slog.LevelWarn}))
|
||||
slog.SetDefault(logger)
|
||||
return &buf, func() { slog.SetDefault(prev) }
|
||||
}
|
||||
|
||||
// runDaemonHermetic invokes daemonCmd.RunE with a context that is
|
||||
// already cancelled and an unbindable --addr, so the long-running
|
||||
// server start short-circuits and RunE returns quickly without
|
||||
// touching the network. It returns whatever RunE returned and the
|
||||
// captured slog buffer.
|
||||
func runDaemonHermetic(t *testing.T, suppressWarnings bool) (string, error) {
|
||||
t.Helper()
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
buf, restore := captureSlog(t)
|
||||
defer restore()
|
||||
|
||||
if suppressWarnings {
|
||||
_ = rootCmd.PersistentFlags().Set("no-deprecation-warnings", "true")
|
||||
}
|
||||
|
||||
daemonAddr = "127.0.0.1:99999" // unbindable: port outside uint16 range → ListenAndServe fails fast
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel() // already-done context: the select returns via <-ctx.Done() immediately
|
||||
|
||||
cmd := daemonCmd
|
||||
cmd.SetOut(&bytes.Buffer{})
|
||||
cmd.SetErr(&bytes.Buffer{})
|
||||
cmd.SetArgs(nil)
|
||||
cmd.SetContext(ctx)
|
||||
|
||||
err := cmd.RunE(cmd, nil)
|
||||
return buf.String(), err
|
||||
}
|
||||
|
||||
// TestDaemonEmitsDeprecationWarning verifies REQ-068: `orca daemon`
|
||||
// emits a slog.Warn deprecation banner on every run.
|
||||
func TestDaemonEmitsDeprecationWarning(t *testing.T) {
|
||||
out, _ := runDaemonHermetic(t, false)
|
||||
if !strings.Contains(out, "orca daemon is deprecated in v0.9") {
|
||||
t.Errorf("expected deprecation warning in slog output, got:\n%s", out)
|
||||
}
|
||||
if !strings.Contains(out, "R-001") {
|
||||
t.Errorf("deprecation warning should reference R-001, got:\n%s", out)
|
||||
}
|
||||
}
|
||||
|
||||
// TestDaemonDeprecationWarningSuppressed verifies that
|
||||
// --no-deprecation-warnings suppresses the deprecation banner (for
|
||||
// `orca upgrade` migrations).
|
||||
func TestDaemonDeprecationWarningSuppressed(t *testing.T) {
|
||||
out, _ := runDaemonHermetic(t, true)
|
||||
if strings.Contains(out, "deprecated in v0.9") {
|
||||
t.Errorf("--no-deprecation-warnings should suppress the deprecation warning, got:\n%s", out)
|
||||
}
|
||||
}
|
||||
|
||||
// TestDaemonStillRuns verifies deprecation ≠ removal: the daemon
|
||||
// command's RunE is still wired and callable. We don't assert on the
|
||||
// error value (the hermetic short-circuit may return nil or a
|
||||
// shutdown-related error), only that the command did not fail *because*
|
||||
// of the deprecation notice.
|
||||
func TestDaemonStillRuns(t *testing.T) {
|
||||
_, err := runDaemonHermetic(t, false)
|
||||
if err != nil && strings.Contains(err.Error(), "deprecated") {
|
||||
t.Errorf("daemon must not error due to deprecation, got: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestWarnDeprecatedGate verifies the package-level helper that gates
|
||||
// deprecation warnings on the --no-deprecation-warnings flag.
|
||||
func TestWarnDeprecatedGate(t *testing.T) {
|
||||
t.Run("emits by default", func(t *testing.T) {
|
||||
buf, restore := captureSlog(t)
|
||||
defer restore()
|
||||
noDeprecationWarnings = false
|
||||
warnDeprecated("test-deprecation-marker")
|
||||
if !strings.Contains(buf.String(), "test-deprecation-marker") {
|
||||
t.Errorf("expected warning emitted, got: %s", buf.String())
|
||||
}
|
||||
})
|
||||
t.Run("suppressed when flag set", func(t *testing.T) {
|
||||
buf, restore := captureSlog(t)
|
||||
defer restore()
|
||||
noDeprecationWarnings = true
|
||||
defer func() { noDeprecationWarnings = false }()
|
||||
warnDeprecated("should-not-appear")
|
||||
if strings.Contains(buf.String(), "should-not-appear") {
|
||||
t.Errorf("expected no warning when --no-deprecation-warnings set, got: %s", buf.String())
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// TestNoDeprecationWarningsFlagRegistered verifies the
|
||||
// --no-deprecation-warnings persistent flag exists on rootCmd.
|
||||
func TestNoDeprecationWarningsFlagRegistered(t *testing.T) {
|
||||
f := rootCmd.PersistentFlags().Lookup("no-deprecation-warnings")
|
||||
if f == nil {
|
||||
t.Fatal("--no-deprecation-warnings persistent flag not registered on rootCmd")
|
||||
}
|
||||
if f.DefValue != "false" {
|
||||
t.Errorf("--no-deprecation-warnings default = %q, want false", f.DefValue)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCertEmitsDeprecationWarning verifies REQ-068: deprecated
|
||||
// `orca cert ca-init` subcommand emits a deprecation banner.
|
||||
func TestCertEmitsDeprecationWarning(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
buf, restore := captureSlog(t)
|
||||
defer restore()
|
||||
|
||||
var out bytes.Buffer
|
||||
rootCmd.SetOut(&out)
|
||||
rootCmd.SetErr(&out)
|
||||
rootCmd.SetArgs([]string{"cert", "ca-init", "--cn", "test-ca"})
|
||||
_ = rootCmd.Execute()
|
||||
|
||||
logged := buf.String()
|
||||
if !strings.Contains(logged, "orca cert ca-init is deprecated") {
|
||||
t.Errorf("expected cert ca-init deprecation warning, got:\n%s", logged)
|
||||
}
|
||||
if !strings.Contains(logged, "step-ca") {
|
||||
t.Errorf("deprecation warning should mention step-ca, got:\n%s", logged)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCertDeprecationWarningSuppressed verifies --no-deprecation-warnings
|
||||
// suppresses the cert deprecation banner.
|
||||
func TestCertDeprecationWarningSuppressed(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
_ = rootCmd.PersistentFlags().Set("no-deprecation-warnings", "true")
|
||||
|
||||
buf, restore := captureSlog(t)
|
||||
defer restore()
|
||||
|
||||
var out bytes.Buffer
|
||||
rootCmd.SetOut(&out)
|
||||
rootCmd.SetErr(&out)
|
||||
rootCmd.SetArgs([]string{"cert", "ca-init", "--cn", "test-ca"})
|
||||
_ = rootCmd.Execute()
|
||||
|
||||
if strings.Contains(buf.String(), "orca cert ca-init is deprecated") {
|
||||
t.Errorf("--no-deprecation-warnings should suppress cert warning, got:\n%s", buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
// TestNodeJoinMTLSEmitsDeprecationWarning verifies REQ-068: the mTLS
|
||||
// join path (`orca node join` without --type proxmox) warns that the
|
||||
// mTLS join path is deprecated.
|
||||
func TestNodeJoinMTLSEmitsDeprecationWarning(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
buf, restore := captureSlog(t)
|
||||
defer restore()
|
||||
|
||||
var out bytes.Buffer
|
||||
rootCmd.SetOut(&out)
|
||||
rootCmd.SetErr(&out)
|
||||
rootCmd.SetArgs([]string{"node", "join", "--name", "dep-warning", "--addr", "10.0.0.55:8443"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("node join: %v", err)
|
||||
}
|
||||
|
||||
logged := buf.String()
|
||||
if !strings.Contains(logged, "mTLS join path is deprecated") {
|
||||
t.Errorf("expected mTLS join deprecation warning, got:\n%s", logged)
|
||||
}
|
||||
if !strings.Contains(logged, "R-001") {
|
||||
t.Errorf("deprecation warning should reference R-001, got:\n%s", logged)
|
||||
}
|
||||
}
|
||||
|
||||
// TestNodeJoinProxmoxNoMTLSDeprecationWarning verifies the deprecation
|
||||
// warning does NOT fire for the proxmox SSH path (that path is the
|
||||
// v0.9 replacement, not the deprecated mTLS path).
|
||||
func TestNodeJoinProxmoxNoMTLSDeprecationWarning(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
buf, restore := captureSlog(t)
|
||||
defer restore()
|
||||
|
||||
var out bytes.Buffer
|
||||
rootCmd.SetOut(&out)
|
||||
rootCmd.SetErr(&out)
|
||||
// proxmox path errors on missing --host before reaching the warning,
|
||||
// and never calls joinLocal, so no mTLS deprecation warning fires.
|
||||
rootCmd.SetArgs([]string{"node", "join", "--type", "proxmox", "--ssh-key", "/tmp/nonexistent-key"})
|
||||
_ = rootCmd.Execute()
|
||||
|
||||
if strings.Contains(buf.String(), "mTLS join path is deprecated") {
|
||||
t.Errorf("proxmox path must not emit mTLS deprecation warning, got:\n%s", buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,177 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// runWithSlog executes the given args against rootCmd, capturing the
|
||||
// slog output (where warnDeprecated writes). It returns the captured
|
||||
// slog buffer and the command stdout buffer.
|
||||
func runWithSlog(t *testing.T, args []string, suppressWarnings bool) (slogOut, stdOut string, err error) {
|
||||
t.Helper()
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
buf, restore := captureSlog(t)
|
||||
defer restore()
|
||||
|
||||
if suppressWarnings {
|
||||
_ = rootCmd.PersistentFlags().Set("no-deprecation-warnings", "true")
|
||||
}
|
||||
|
||||
var out bytes.Buffer
|
||||
rootCmd.SetOut(&out)
|
||||
rootCmd.SetErr(&out)
|
||||
rootCmd.SetArgs(args)
|
||||
err = rootCmd.Execute()
|
||||
return buf.String(), out.String(), err
|
||||
}
|
||||
|
||||
func TestDeprecationDaemonEmitsWarning(t *testing.T) {
|
||||
slogOut, _ := runDaemonHermetic(t, false)
|
||||
if !strings.Contains(slogOut, "orca daemon is deprecated in v0.9") {
|
||||
t.Errorf("expected daemon deprecation warning, got:\n%s", slogOut)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeprecationCertCAInitEmitsWarning(t *testing.T) {
|
||||
slogOut, _, err := runWithSlog(t, []string{"cert", "ca-init", "--cn", "dep-test"}, false)
|
||||
if err != nil {
|
||||
t.Fatalf("cert ca-init: %v", err)
|
||||
}
|
||||
if !strings.Contains(slogOut, "orca cert ca-init is deprecated") {
|
||||
t.Errorf("expected cert ca-init deprecation warning, got:\n%s", slogOut)
|
||||
}
|
||||
if !strings.Contains(slogOut, "step-ca") {
|
||||
t.Errorf("deprecation warning should mention step-ca, got:\n%s", slogOut)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeprecationCertGenEmitsWarning(t *testing.T) {
|
||||
// gen requires a CA; we only assert the warning fires (before the
|
||||
// error path).
|
||||
slogOut, _, _ := runWithSlog(t, []string{"cert", "gen", "--cn", "dep-gen"}, false)
|
||||
if !strings.Contains(slogOut, "orca cert gen is deprecated") {
|
||||
t.Errorf("expected cert gen deprecation warning, got:\n%s", slogOut)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeprecationCertRenewEmitsWarning(t *testing.T) {
|
||||
// renew requires a CA; we only assert the warning fires (before the
|
||||
// error path).
|
||||
slogOut, _, _ := runWithSlog(t, []string{"cert", "renew"}, false)
|
||||
if !strings.Contains(slogOut, "orca cert renew is deprecated") {
|
||||
t.Errorf("expected cert renew deprecation warning, got:\n%s", slogOut)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeprecationCertShowNoWarning(t *testing.T) {
|
||||
slogOut, _, err := runWithSlog(t, []string{"cert", "show"}, false)
|
||||
// show may fail if no cert exists; we only assert no deprecation.
|
||||
_ = err
|
||||
if strings.Contains(slogOut, "deprecated") {
|
||||
t.Errorf("cert show must NOT emit deprecation warning, got:\n%s", slogOut)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeprecationCertFingerprintNoWarning(t *testing.T) {
|
||||
// Need a CA first so fingerprint has something to read.
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
var b bytes.Buffer
|
||||
rootCmd.SetOut(&b)
|
||||
rootCmd.SetErr(&b)
|
||||
rootCmd.SetArgs([]string{"cert", "ca-init", "--cn", "fp-test"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("cert ca-init: %v", err)
|
||||
}
|
||||
|
||||
slogBuf, restore := captureSlog(t)
|
||||
defer restore()
|
||||
|
||||
resetRootFlags(t)
|
||||
var out bytes.Buffer
|
||||
rootCmd.SetOut(&out)
|
||||
rootCmd.SetErr(&out)
|
||||
rootCmd.SetArgs([]string{"cert", "fingerprint", "--which", "ca"})
|
||||
_ = rootCmd.Execute()
|
||||
|
||||
if strings.Contains(slogBuf.String(), "deprecated") {
|
||||
t.Errorf("cert fingerprint must NOT emit deprecation warning, got:\n%s", slogBuf.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeprecationJobRunHCLEmitsWarning(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
specPath := filepath.Join(dir, "old-spec.hcl")
|
||||
if err := os.WriteFile(specPath, []byte(`job "true" {}
|
||||
task "t" {
|
||||
command = "/bin/true"
|
||||
}
|
||||
`), 0o644); err != nil {
|
||||
t.Fatalf("write spec: %v", err)
|
||||
}
|
||||
slogOut, _, err := runWithSlog(t, []string{"job", "run", specPath}, false)
|
||||
if err != nil {
|
||||
t.Fatalf("job run: %v", err)
|
||||
}
|
||||
if !strings.Contains(slogOut, ".hcl jobspec is legacy") {
|
||||
t.Errorf("expected .hcl deprecation warning, got:\n%s", slogOut)
|
||||
}
|
||||
if !strings.Contains(slogOut, "R-013") {
|
||||
t.Errorf("deprecation warning should reference R-013, got:\n%s", slogOut)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeprecationJobRunMDNoWarning(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
specPath := filepath.Join(dir, "spec.md")
|
||||
if err := os.WriteFile(specPath, []byte("---\nkind: Workload\nname: md-job\n---\n"), 0o644); err != nil {
|
||||
t.Fatalf("write spec: %v", err)
|
||||
}
|
||||
slogOut, _, _ := runWithSlog(t, []string{"job", "run", specPath}, false)
|
||||
if strings.Contains(slogOut, ".hcl jobspec is legacy") {
|
||||
t.Errorf(".md jobspec must NOT emit .hcl deprecation warning, got:\n%s", slogOut)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeprecationWarningsSuppressedByFlag(t *testing.T) {
|
||||
// daemon
|
||||
slogOut, _ := runDaemonHermetic(t, true)
|
||||
if strings.Contains(slogOut, "deprecated in v0.9") {
|
||||
t.Errorf("--no-deprecation-warnings should suppress daemon warning, got:\n%s", slogOut)
|
||||
}
|
||||
|
||||
// cert ca-init
|
||||
slogOut2, _, err := runWithSlog(t, []string{"cert", "ca-init", "--cn", "sup-test"}, true)
|
||||
if err != nil {
|
||||
t.Fatalf("cert ca-init: %v", err)
|
||||
}
|
||||
if strings.Contains(slogOut2, "deprecated") {
|
||||
t.Errorf("--no-deprecation-warnings should suppress cert warning, got:\n%s", slogOut2)
|
||||
}
|
||||
|
||||
// job run .hcl
|
||||
dir := t.TempDir()
|
||||
specPath := filepath.Join(dir, "old-spec.hcl")
|
||||
if err := os.WriteFile(specPath, []byte(`job "true" {}
|
||||
task "t" {
|
||||
command = "/bin/true"
|
||||
}
|
||||
`), 0o644); err != nil {
|
||||
t.Fatalf("write spec: %v", err)
|
||||
}
|
||||
slogOut3, _, err := runWithSlog(t, []string{"job", "run", specPath}, true)
|
||||
if err != nil {
|
||||
t.Fatalf("job run: %v", err)
|
||||
}
|
||||
if strings.Contains(slogOut3, "deprecated") {
|
||||
t.Errorf("--no-deprecation-warnings should suppress .hcl warning, got:\n%s", slogOut3)
|
||||
}
|
||||
}
|
||||
@@ -98,6 +98,6 @@ var doctorProxmoxCmd = &cobra.Command{
|
||||
}
|
||||
|
||||
func init() {
|
||||
doctorCmd.AddCommand(doctorCertCmd, doctorNetworkCmd, doctorDBCmd, doctorOSCmd, doctorProxmoxCmd)
|
||||
doctorCmd.AddCommand(doctorCertCmd, doctorNetworkCmd, doctorDBCmd, doctorOSCmd, doctorProxmoxCmd, noOrcaOnServerCmd, doctorNftCmd)
|
||||
rootCmd.AddCommand(doctorCmd)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,165 @@
|
||||
// Package cli: doctor_nft.go implements `orca doctor nft` (P15.5,
|
||||
// REQ-101). The check verifies the R-017 nftables ingress ruleset is
|
||||
// present, parses, and matches the latest applied txn's hash.
|
||||
//
|
||||
// Checks (each emits a PASS/WARN/FAIL line):
|
||||
//
|
||||
// 1. table inet orca-ingress exists (nft list table)
|
||||
// 2. DNAT :443 -> 127.0.0.1:8443 present
|
||||
// 3. DNAT :80 -> 127.0.0.1:8080 present
|
||||
// 4. rate-limit meter ora_rl present
|
||||
// 5. /etc/nftables.d/orca.nft parses (nft -c -f)
|
||||
// 6. file hash matches the latest applied txn (drift, P10b)
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/certpaths"
|
||||
"git.cloudinit.dev/coreci/orca/internal/paths"
|
||||
"git.cloudinit.dev/coreci/orca/internal/sshpush"
|
||||
)
|
||||
|
||||
// nftTransport is the SSH-push surface doctor nft needs. Mirrors the
|
||||
// drift CLI seam; tests substitute a mock.
|
||||
type nftTransport interface {
|
||||
Exec(ctx context.Context, peer string, cmd string) ([]byte, error)
|
||||
ReadFile(ctx context.Context, peer string, path string) ([]byte, error)
|
||||
}
|
||||
|
||||
// nftTransportOverride is the package-level test seam.
|
||||
var nftTransportOverride nftTransport
|
||||
|
||||
// nftCheckResult is one line of `orca doctor nft` output.
|
||||
type nftCheckResult struct {
|
||||
Name string `json:"name"`
|
||||
Result string `json:"result"`
|
||||
Message string `json:"message"`
|
||||
}
|
||||
|
||||
func nftTransportFromCtx() (nftTransport, error) {
|
||||
if nftTransportOverride != nil {
|
||||
return nftTransportOverride, nil
|
||||
}
|
||||
keyPath := certpaths.SSHKeyPath()
|
||||
khPath := certpaths.KnownHostsPath()
|
||||
return sshpush.NewTransport(keyPath, khPath), nil
|
||||
}
|
||||
|
||||
// nftLeadPeerOverride is the test seam for the peer to probe.
|
||||
var nftLeadPeerOverride string
|
||||
|
||||
func nftLeadPeer() string {
|
||||
if nftLeadPeerOverride != "" {
|
||||
return nftLeadPeerOverride
|
||||
}
|
||||
return "lead"
|
||||
}
|
||||
|
||||
var doctorNftCmd = &cobra.Command{
|
||||
Use: "nft",
|
||||
Short: "Run the nftables ingress self-check (P15.5, REQ-101)",
|
||||
Long: `Verify the R-017 nftables ingress ruleset: table exists, DNAT
|
||||
:443->127.0.0.1:8443 and :80->127.0.0.1:8080 present, rate-limit meter
|
||||
present, /etc/nftables.d/orca.nft parses, and the on-disk file hash
|
||||
matches the latest applied txn (drift, P10b).`,
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
results := runNftChecks(cmd.Context())
|
||||
if jsonOutput {
|
||||
return printJSON(results)
|
||||
}
|
||||
for _, r := range results {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%-28s %-5s %s\n", r.Name, r.Result, r.Message)
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
// runNftChecks executes the nft doctor checks against nftLeadPeer().
|
||||
func runNftChecks(ctx context.Context) []nftCheckResult {
|
||||
t, err := nftTransportFromCtx()
|
||||
if err != nil {
|
||||
return []nftCheckResult{{Name: "nft:transport", Result: "FAIL", Message: err.Error()}}
|
||||
}
|
||||
peer := nftLeadPeer()
|
||||
var results []nftCheckResult
|
||||
|
||||
tableOut, tableErr := t.Exec(ctx, peer, "nft list table inet orca-ingress")
|
||||
if tableErr != nil {
|
||||
results = append(results, nftCheckResult{Name: "nft:table", Result: "FAIL", Message: tableErr.Error()})
|
||||
} else {
|
||||
results = append(results, nftCheckResult{Name: "nft:table", Result: "PASS", Message: "table inet orca-ingress present"})
|
||||
}
|
||||
tableStr := string(tableOut)
|
||||
|
||||
if strings.Contains(tableStr, "dnat to 127.0.0.1:8443") {
|
||||
results = append(results, nftCheckResult{Name: "nft:dnat-443", Result: "PASS", Message: "DNAT :443->127.0.0.1:8443 present"})
|
||||
} else {
|
||||
results = append(results, nftCheckResult{Name: "nft:dnat-443", Result: "FAIL", Message: "DNAT :443->127.0.0.1:8443 missing"})
|
||||
}
|
||||
|
||||
if strings.Contains(tableStr, "dnat to 127.0.0.1:8080") {
|
||||
results = append(results, nftCheckResult{Name: "nft:dnat-80", Result: "PASS", Message: "DNAT :80->127.0.0.1:8080 present"})
|
||||
} else {
|
||||
results = append(results, nftCheckResult{Name: "nft:dnat-80", Result: "FAIL", Message: "DNAT :80->127.0.0.1:8080 missing"})
|
||||
}
|
||||
|
||||
if strings.Contains(tableStr, "ora_rl") {
|
||||
results = append(results, nftCheckResult{Name: "nft:rate-limit", Result: "PASS", Message: "rate-limit meter ora_rl present"})
|
||||
} else {
|
||||
results = append(results, nftCheckResult{Name: "nft:rate-limit", Result: "FAIL", Message: "rate-limit meter ora_rl missing"})
|
||||
}
|
||||
|
||||
if _, err := t.Exec(ctx, peer, "nft -c -f /etc/nftables.d/orca.nft"); err != nil {
|
||||
results = append(results, nftCheckResult{Name: "nft:parse", Result: "FAIL", Message: fmt.Sprintf("nft -c -f failed: %v", err)})
|
||||
} else {
|
||||
results = append(results, nftCheckResult{Name: "nft:parse", Result: "PASS", Message: "/etc/nftables.d/orca.nft parses cleanly"})
|
||||
}
|
||||
|
||||
results = append(results, checkNftHashDrift(ctx, t, peer))
|
||||
return results
|
||||
}
|
||||
|
||||
// checkNftHashDrift compares the on-peer file hash against the
|
||||
// locally-recorded hash from the latest applied txn. The locally
|
||||
// recorded hash is stored at ClusterDir()/nft.applied.sha256 (written by
|
||||
// the apply path; the drift check reads it). When the local record is
|
||||
// absent the check WARNs (no baseline to compare against).
|
||||
func checkNftHashDrift(ctx context.Context, t nftTransport, peer string) nftCheckResult {
|
||||
liveHashOut, err := t.Exec(ctx, peer, "sha256sum /etc/nftables.d/orca.nft 2>/dev/null")
|
||||
if err != nil {
|
||||
return nftCheckResult{Name: "nft:hash-drift", Result: "FAIL", Message: fmt.Sprintf("remote sha256sum: %v", err)}
|
||||
}
|
||||
fields := strings.Fields(strings.TrimSpace(string(liveHashOut)))
|
||||
if len(fields) == 0 {
|
||||
return nftCheckResult{Name: "nft:hash-drift", Result: "FAIL", Message: "remote sha256sum returned no output"}
|
||||
}
|
||||
liveHash := fields[0]
|
||||
|
||||
recordPath := filepath.Join(paths.ClusterDir(), "nft.applied.sha256")
|
||||
recorded, rerr := os.ReadFile(recordPath)
|
||||
if rerr != nil {
|
||||
return nftCheckResult{Name: "nft:hash-drift", Result: "WARN", Message: "no applied-txn hash baseline (first apply or record missing)"}
|
||||
}
|
||||
want := strings.TrimSpace(string(recorded))
|
||||
if liveHash == want {
|
||||
return nftCheckResult{Name: "nft:hash-drift", Result: "PASS", Message: "on-disk hash matches latest applied txn"}
|
||||
}
|
||||
return nftCheckResult{Name: "nft:hash-drift", Result: "FAIL", Message: fmt.Sprintf("DRIFT: live=%s recorded=%s", liveHash, want)}
|
||||
}
|
||||
|
||||
// localSha256OfFile is a small helper for tests that compute the
|
||||
// expected recorded hash from rendered content.
|
||||
func localSha256OfFile(content []byte) string {
|
||||
sum := sha256.Sum256(content)
|
||||
return hex.EncodeToString(sum[:])
|
||||
}
|
||||
@@ -0,0 +1,144 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type mockNftTransport struct {
|
||||
execOut map[string][]byte
|
||||
execErr map[string]error
|
||||
execs []string
|
||||
}
|
||||
|
||||
func (m *mockNftTransport) Exec(ctx context.Context, peer string, cmd string) ([]byte, error) {
|
||||
m.execs = append(m.execs, cmd)
|
||||
if m.execErr != nil {
|
||||
if err, ok := m.execErr[cmd]; ok {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
if m.execOut != nil {
|
||||
if out, ok := m.execOut[cmd]; ok {
|
||||
return out, nil
|
||||
}
|
||||
}
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
func (m *mockNftTransport) ReadFile(ctx context.Context, peer string, path string) ([]byte, error) {
|
||||
return nil, errors.New("not implemented")
|
||||
}
|
||||
|
||||
func TestDoctorNft_AllPass(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
origTransport := nftTransportOverride
|
||||
origPeer := nftLeadPeerOverride
|
||||
defer func() {
|
||||
nftTransportOverride = origTransport
|
||||
nftLeadPeerOverride = origPeer
|
||||
}()
|
||||
nftLeadPeerOverride = "lead"
|
||||
nftTransportOverride = &mockNftTransport{
|
||||
execOut: map[string][]byte{
|
||||
"nft list table inet orca-ingress": []byte(`table inet orca-ingress {
|
||||
set orca_trusted_probes { type ipv4_addr; }
|
||||
chain prerouting {
|
||||
tcp dport 443 dnat to 127.0.0.1:8443
|
||||
tcp dport 80 dnat to 127.0.0.1:8080
|
||||
}
|
||||
chain forward {
|
||||
tcp dport 443 ct state new meter { ora_rl { rate 100/second } } accept
|
||||
}
|
||||
}`),
|
||||
},
|
||||
}
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"doctor", "nft"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("doctor nft: %v", err)
|
||||
}
|
||||
out := buf.String()
|
||||
for _, want := range []string{"nft:table", "nft:dnat-443", "nft:dnat-80", "nft:rate-limit", "nft:parse", "PASS"} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Errorf("output missing %q:\n%s", want, out)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestDoctorNft_TableMissing(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
origTransport := nftTransportOverride
|
||||
origPeer := nftLeadPeerOverride
|
||||
defer func() {
|
||||
nftTransportOverride = origTransport
|
||||
nftLeadPeerOverride = origPeer
|
||||
}()
|
||||
nftLeadPeerOverride = "lead"
|
||||
nftTransportOverride = &mockNftTransport{
|
||||
execErr: map[string]error{
|
||||
"nft list table inet orca-ingress": errors.New("table not found"),
|
||||
},
|
||||
}
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"doctor", "nft"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("doctor nft: %v", err)
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "nft:table") || !strings.Contains(out, "FAIL") {
|
||||
t.Errorf("expected table FAIL:\n%s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDoctorNft_HashDriftDetected(t *testing.T) {
|
||||
dir, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
origTransport := nftTransportOverride
|
||||
origPeer := nftLeadPeerOverride
|
||||
defer func() {
|
||||
nftTransportOverride = origTransport
|
||||
nftLeadPeerOverride = origPeer
|
||||
}()
|
||||
nftLeadPeerOverride = "lead"
|
||||
rendered := `table inet orca-ingress { tcp dport 443 dnat to 127.0.0.1:8443; tcp dport 80 dnat to 127.0.0.1:8080; ora_rl; }`
|
||||
nftTransportOverride = &mockNftTransport{
|
||||
execOut: map[string][]byte{
|
||||
"nft list table inet orca-ingress": []byte(rendered),
|
||||
"nft -c -f /etc/nftables.d/orca.nft": []byte(""),
|
||||
"sha256sum /etc/nftables.d/orca.nft 2>/dev/null": []byte("deadbeef /etc/nftables.d/orca.nft\n"),
|
||||
},
|
||||
}
|
||||
// Record a DIFFERENT hash so drift is reported.
|
||||
if err := os.MkdirAll(filepath.Dir(filepath.Join(dir, "cluster", "nft.applied.sha256")), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, "cluster", "nft.applied.sha256"), []byte("cafef00d\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"doctor", "nft"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("doctor nft: %v", err)
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "nft:hash-drift") || !strings.Contains(out, "DRIFT") || !strings.Contains(out, "FAIL") {
|
||||
t.Errorf("expected DRIFT FAIL:\n%s", out)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,657 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/certpaths"
|
||||
"git.cloudinit.dev/coreci/orca/internal/engine"
|
||||
"git.cloudinit.dev/coreci/orca/internal/model"
|
||||
"git.cloudinit.dev/coreci/orca/internal/sshpush"
|
||||
"git.cloudinit.dev/coreci/orca/internal/store"
|
||||
)
|
||||
|
||||
// drainExecer is the SSH command-execution seam used by the drain
|
||||
// commands. *sshpush.Transport satisfies it via its Exec method; tests
|
||||
// inject a record-and-replay mock without a real SSH server (same
|
||||
// pattern as internal/stepca mockExec).
|
||||
type drainExecer interface {
|
||||
Exec(ctx context.Context, peer string, cmd string) ([]byte, error)
|
||||
}
|
||||
|
||||
// drainTransport is the package-level exec seam. It is set by
|
||||
// drainExecFromCtx (production) and overridden by tests via
|
||||
// drainExecOverride. nil means "build from certpaths on first use".
|
||||
var drainExecOverride drainExecer
|
||||
|
||||
// drainExecFromCtx returns the production drainExecer backed by the
|
||||
// real sshpush.Transport (using the cluster SSH key + known_hosts). On
|
||||
// error it returns a nil transport and the error; callers must check.
|
||||
func drainExecFromCtx(_ context.Context) (drainExecer, error) {
|
||||
if drainExecOverride != nil {
|
||||
return drainExecOverride, nil
|
||||
}
|
||||
keyPath := certpaths.SSHKeyPath()
|
||||
khPath := certpaths.KnownHostsPath()
|
||||
tr := sshpush.NewTransport(keyPath, khPath)
|
||||
return tr, nil
|
||||
}
|
||||
|
||||
// peerAddrForNode derives the SSH peer address (host:port) for a node.
|
||||
// For proxmox nodes the Name IS the host; for localhost nodes the
|
||||
// Address carries host:8443. We always target SSH port 22 unless the
|
||||
// node's Address already encodes a non-daemon port. The local node
|
||||
// (Name=="localhost") is contacted at "localhost:22".
|
||||
func peerAddrForNode(n *model.Node) string {
|
||||
if n == nil {
|
||||
return ""
|
||||
}
|
||||
if h, p, ok := splitHostPort(n.Address); ok && p != "" && p != "8443" {
|
||||
return h + ":" + p
|
||||
}
|
||||
host := n.Name
|
||||
if h, _, ok := splitHostPort(n.Address); ok && h != "" && h != "localhost" {
|
||||
host = h
|
||||
}
|
||||
if host == "" {
|
||||
host = n.Name
|
||||
}
|
||||
return host + ":22"
|
||||
}
|
||||
|
||||
func splitHostPort(addr string) (string, string, bool) {
|
||||
idx := strings.LastIndex(addr, ":")
|
||||
if idx < 0 {
|
||||
return addr, "", false
|
||||
}
|
||||
return addr[:idx], addr[idx+1:], true
|
||||
}
|
||||
|
||||
var (
|
||||
drainTimeout time.Duration
|
||||
migrateTarget string
|
||||
)
|
||||
|
||||
// allocUnit is the systemd unit name pattern for orca allocations.
|
||||
const allocUnitPrefix = "orca-alloc-"
|
||||
const allocUnitSuffix = ".service"
|
||||
|
||||
// allocIDFromUnit strips the orca-alloc- prefix and .service suffix
|
||||
// from a systemd unit name, returning the bare allocation id.
|
||||
func allocIDFromUnit(unit string) string {
|
||||
s := strings.TrimSpace(unit)
|
||||
s = strings.TrimPrefix(s, allocUnitPrefix)
|
||||
s = strings.TrimSuffix(s, allocUnitSuffix)
|
||||
return s
|
||||
}
|
||||
|
||||
// allocUnit renders the systemd unit name for an allocation id.
|
||||
func allocUnit(allocID string) string {
|
||||
return allocUnitPrefix + allocID + allocUnitSuffix
|
||||
}
|
||||
|
||||
// listRunningAllocs queries a node via SSH for the currently-running
|
||||
// orca-alloc-*.service systemd units and returns their allocation
|
||||
// ids. A node with no orca allocations returns an empty slice (not an
|
||||
// error).
|
||||
func listRunningAllocs(ctx context.Context, ex drainExecer, peer string) ([]string, error) {
|
||||
cmd := "systemctl list-units 'orca-alloc-*.service' --type=service --state=running --no-legend --no-pager"
|
||||
out, err := ex.Exec(ctx, peer, cmd)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("list orca-alloc units on %s: %w", peer, err)
|
||||
}
|
||||
var ids []string
|
||||
for _, line := range strings.Split(string(out), "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
if line == "" {
|
||||
continue
|
||||
}
|
||||
fields := strings.Fields(line)
|
||||
if len(fields) == 0 {
|
||||
continue
|
||||
}
|
||||
unit := fields[0]
|
||||
if !strings.HasPrefix(unit, allocUnitPrefix) || !strings.HasSuffix(unit, allocUnitSuffix) {
|
||||
continue
|
||||
}
|
||||
ids = append(ids, allocIDFromUnit(unit))
|
||||
}
|
||||
return ids, nil
|
||||
}
|
||||
|
||||
// stopAlloc sends `systemctl stop orca-alloc-<id>.service` to a node.
|
||||
// A unit that is already stopped (or never existed) is treated as
|
||||
// success: drain is idempotent.
|
||||
func stopAlloc(ctx context.Context, ex drainExecer, peer, allocID string) error {
|
||||
cmd := fmt.Sprintf("systemctl stop %s", allocUnit(allocID))
|
||||
_, err := ex.Exec(ctx, peer, cmd)
|
||||
if err != nil {
|
||||
var exitErr *sshExitErr
|
||||
if errors.As(err, &exitErr) && exitErr.code == 5 {
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("stop %s on %s: %w", allocID, peer, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// sshExitErr is a lightweight sentinel used by the in-package mock to
|
||||
// signal a non-zero systemctl exit. The real sshpush.Transport wraps
|
||||
// non-zero exits in ErrPermanent; the mock returns an *sshExitErr so
|
||||
// stopAlloc can treat code 5 ("unit not loaded") as success.
|
||||
type sshExitErr struct{ code int }
|
||||
|
||||
func (e *sshExitErr) Error() string { return fmt.Sprintf("sshpush: exit %d", e.code) }
|
||||
|
||||
// waitAllocsStopped polls a node until none of the given allocation
|
||||
// ids appear in the running-unit list, or the context deadline passes.
|
||||
// Returns nil if all allocations are observed stopped; otherwise
|
||||
// returns a list of allocations that were still running at timeout.
|
||||
func waitAllocsStopped(ctx context.Context, ex drainExecer, peer string, ids []string, poll time.Duration) []string {
|
||||
if len(ids) == 0 {
|
||||
return nil
|
||||
}
|
||||
pending := make(map[string]bool, len(ids))
|
||||
for _, id := range ids {
|
||||
pending[id] = true
|
||||
}
|
||||
if poll <= 0 {
|
||||
poll = 500 * time.Millisecond
|
||||
}
|
||||
ticker := time.NewTicker(poll)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
if err := ctx.Err(); err != nil {
|
||||
break
|
||||
}
|
||||
running, err := listRunningAllocs(ctx, ex, peer)
|
||||
if err == nil {
|
||||
runningSet := make(map[string]bool, len(running))
|
||||
for _, rid := range running {
|
||||
runningSet[rid] = true
|
||||
}
|
||||
for id := range pending {
|
||||
if !runningSet[id] {
|
||||
delete(pending, id)
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(pending) == 0 {
|
||||
return nil
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return keysOf(pending)
|
||||
case <-ticker.C:
|
||||
}
|
||||
}
|
||||
return keysOf(pending)
|
||||
}
|
||||
|
||||
func keysOf(m map[string]bool) []string {
|
||||
out := make([]string, 0, len(m))
|
||||
for k := range m {
|
||||
out = append(out, k)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// findNode resolves a node by id or name from the registry. Returns
|
||||
// nil + error if not found.
|
||||
func findNode(ctx context.Context, reg *engine.NodeRegistry, ref string) (*model.Node, error) {
|
||||
if n, err := reg.Get(ctx, ref); err == nil {
|
||||
return n, nil
|
||||
} else if !errors.Is(err, store.ErrNotFound) {
|
||||
return nil, fmt.Errorf("lookup node %q: %w", ref, err)
|
||||
}
|
||||
nodes, err := reg.List(ctx)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("list nodes: %w", err)
|
||||
}
|
||||
for _, n := range nodes {
|
||||
if n.Name == ref || n.ID == ref {
|
||||
return n, nil
|
||||
}
|
||||
}
|
||||
return nil, fmt.Errorf("node %q not found in the registry", ref)
|
||||
}
|
||||
|
||||
// auditDrain records a node.drain event in the audit log.
|
||||
func auditDrain(ctx context.Context, nodeID, result string, err error, meta map[string]any) {
|
||||
db, dbErr := store.Open(certpaths.DBPath())
|
||||
if dbErr != nil {
|
||||
return
|
||||
}
|
||||
defer db.Close()
|
||||
engine.NewAudit(store.NewAuditRepo(db), newLogger()).Record(ctx, "cli", "node.drain", nodeID, result, err, meta)
|
||||
}
|
||||
|
||||
var nodeDrainCmd = &cobra.Command{
|
||||
Use: "drain <host>",
|
||||
Short: "Drain a node: stop its allocations and mark it drained",
|
||||
Long: `Drain a node (REQ-061).
|
||||
|
||||
Marks the node as "draining" (the scheduler skips draining nodes), stops
|
||||
every running allocation on the node via SSH (systemctl stop
|
||||
orca-alloc-<id>.service), waits for them to stop (up to --timeout,
|
||||
default 30s), and marks the node "drained" when all allocations are
|
||||
stopped. The scheduler already skips draining/drained nodes.
|
||||
|
||||
<host> is the node name or id (as shown by 'orca node list').
|
||||
|
||||
This is NOT live-migration: allocations are stopped, not moved. Use
|
||||
'orca job migrate' to reschedule a job onto another node before
|
||||
draining.`,
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
hostRef := args[0]
|
||||
ctx, cancel := context.WithTimeout(cmd.Context(), drainTimeout+10*time.Second)
|
||||
defer cancel()
|
||||
|
||||
reg, closer, err := nodeRegistry()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer closer()
|
||||
|
||||
node, err := findNode(ctx, reg, hostRef)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
peer := peerAddrForNode(node)
|
||||
if peer == "" {
|
||||
return fmt.Errorf("cannot resolve SSH address for node %q", node.Name)
|
||||
}
|
||||
|
||||
ex, err := drainExecFromCtx(cmd.Context())
|
||||
if err != nil {
|
||||
return fmt.Errorf("ssh transport: %w", err)
|
||||
}
|
||||
|
||||
log := newLogger()
|
||||
log.Info("drain: marking node draining",
|
||||
slog.String("node", node.Name), slog.String("peer", peer))
|
||||
|
||||
if err := reg.SetNodeState(ctx, node.ID, string(model.NodeStateDraining)); err != nil {
|
||||
return fmt.Errorf("mark node draining: %w", err)
|
||||
}
|
||||
|
||||
ids, err := listRunningAllocs(ctx, ex, peer)
|
||||
if err != nil {
|
||||
_ = reg.SetNodeState(ctx, node.ID, string(model.NodeStateReady))
|
||||
auditDrain(ctx, node.ID, "failure", err, map[string]any{"peer": peer})
|
||||
return err
|
||||
}
|
||||
|
||||
stopped := make([]string, 0, len(ids))
|
||||
var failures []string
|
||||
for _, id := range ids {
|
||||
if err := stopAlloc(ctx, ex, peer, id); err != nil {
|
||||
failures = append(failures, id)
|
||||
log.Warn("drain: failed to stop alloc",
|
||||
slog.String("alloc", id), slog.String("node", node.Name), "error", err)
|
||||
continue
|
||||
}
|
||||
stopped = append(stopped, id)
|
||||
}
|
||||
|
||||
// Wait for the stopped allocations to actually leave the
|
||||
// running list (with the drain timeout as the deadline).
|
||||
waitCtx, waitCancel := context.WithTimeout(ctx, drainTimeout)
|
||||
remaining := waitAllocsStopped(waitCtx, ex, peer, stopped, 500*time.Millisecond)
|
||||
waitCancel()
|
||||
|
||||
result := map[string]any{
|
||||
"node": node.Name,
|
||||
"node_id": node.ID,
|
||||
"peer": peer,
|
||||
"stopped": stopped,
|
||||
}
|
||||
if len(failures) > 0 {
|
||||
result["failed"] = failures
|
||||
}
|
||||
if len(remaining) > 0 {
|
||||
result["still_running"] = remaining
|
||||
}
|
||||
|
||||
if len(failures) == 0 && len(remaining) == 0 {
|
||||
if err := reg.SetNodeState(ctx, node.ID, string(model.NodeStateDrained)); err != nil {
|
||||
auditDrain(ctx, node.ID, "failure", err, result)
|
||||
return fmt.Errorf("mark node drained: %w", err)
|
||||
}
|
||||
result["state"] = string(model.NodeStateDrained)
|
||||
auditDrain(ctx, node.ID, "success", nil, result)
|
||||
if jsonOutput {
|
||||
return printJSON(result)
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "✓ Node %s drained (%d allocs stopped)\n", node.Name, len(stopped))
|
||||
for _, id := range stopped {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), " stopped %s\n", id)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Partial: leave node in draining state so the operator can
|
||||
// retry; record the partial outcome in the audit log.
|
||||
result["state"] = string(model.NodeStateDraining)
|
||||
auditDrain(ctx, node.ID, "partial", fmt.Errorf("%d failed, %d still running", len(failures), len(remaining)), result)
|
||||
if jsonOutput {
|
||||
return printJSON(result)
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "⚠ Node %s partially drained (%d stopped, %d failed, %d still running)\n",
|
||||
node.Name, len(stopped), len(failures), len(remaining))
|
||||
for _, id := range stopped {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), " stopped %s\n", id)
|
||||
}
|
||||
for _, id := range failures {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), " failed %s\n", id)
|
||||
}
|
||||
for _, id := range remaining {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), " running %s\n", id)
|
||||
}
|
||||
return fmt.Errorf("drain incomplete: %d failed, %d still running", len(failures), len(remaining))
|
||||
},
|
||||
}
|
||||
|
||||
// daemonDrainAndStopCmd repurposes the deprecated `orca daemon` command
|
||||
// to drain all nodes and stop all v0.8 daemons (REQ-061, R-001). It is
|
||||
// the v0.8→v0.11 migration path for daemon removal: it SSHes to every
|
||||
// peer that still runs an orca daemon and stops the daemon service,
|
||||
// relying on the v0.9+ SSH-push path (systemd) to keep workloads alive.
|
||||
var daemonDrainAndStopCmd = &cobra.Command{
|
||||
Use: "drain-and-stop",
|
||||
Short: "Stop v0.8 orca daemons on all peers (v0.8→v0.11 migration)",
|
||||
Long: `Stop the orca daemon on every peer that still runs one (REQ-061, R-001).
|
||||
|
||||
This is the v0.8→v0.11 migration path for daemon removal. For each
|
||||
registered peer, SSH in and run 'systemctl stop orca-daemon.service'.
|
||||
Workloads supervised by the v0.9+ SSH-push path keep running under
|
||||
their own systemd units (orca-alloc-*.service) and are NOT touched.
|
||||
|
||||
This command is idempotent: a peer with no orca-daemon.service (already
|
||||
migrated, or never had one) is reported as "already stopped" and does
|
||||
not error.`,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
ctx, cancel := context.WithTimeout(cmd.Context(), 5*time.Minute)
|
||||
defer cancel()
|
||||
|
||||
reg, closer, err := nodeRegistry()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer closer()
|
||||
|
||||
nodes, err := reg.List(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("list nodes: %w", err)
|
||||
}
|
||||
|
||||
ex, err := drainExecFromCtx(cmd.Context())
|
||||
if err != nil {
|
||||
return fmt.Errorf("ssh transport: %w", err)
|
||||
}
|
||||
|
||||
const stopCmd = "systemctl stop orca-daemon.service"
|
||||
|
||||
result := map[string]any{
|
||||
"stopped": []string{},
|
||||
"already_stopped": []string{},
|
||||
"failed": []string{},
|
||||
}
|
||||
var stopped, already, failed []string
|
||||
for _, n := range nodes {
|
||||
peer := peerAddrForNode(n)
|
||||
if peer == "" {
|
||||
continue
|
||||
}
|
||||
_, err := ex.Exec(ctx, peer, stopCmd)
|
||||
if err == nil {
|
||||
stopped = append(stopped, n.Name)
|
||||
continue
|
||||
}
|
||||
var exitErr *sshExitErr
|
||||
if errors.As(err, &exitErr) && exitErr.code == 5 {
|
||||
already = append(already, n.Name)
|
||||
continue
|
||||
}
|
||||
failed = append(failed, n.Name)
|
||||
}
|
||||
result["stopped"] = stopped
|
||||
result["already_stopped"] = already
|
||||
result["failed"] = failed
|
||||
|
||||
db, dbErr := store.Open(certpaths.DBPath())
|
||||
if dbErr == nil {
|
||||
defer db.Close()
|
||||
engine.NewAudit(store.NewAuditRepo(db), newLogger()).Record(ctx, "cli", "daemon.drain_and_stop", "cluster", "success", nil, result)
|
||||
}
|
||||
|
||||
if jsonOutput {
|
||||
return printJSON(result)
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "✓ daemon drain-and-stop complete (%d stopped, %d already stopped, %d failed)\n",
|
||||
len(stopped), len(already), len(failed))
|
||||
for _, n := range stopped {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), " stopped %s\n", n)
|
||||
}
|
||||
for _, n := range already {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), " already-stopped %s\n", n)
|
||||
}
|
||||
for _, n := range failed {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), " failed %s\n", n)
|
||||
}
|
||||
if len(failed) > 0 {
|
||||
return fmt.Errorf("daemon drain-and-stop: %d peer(s) failed", len(failed))
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
// jobMigrateCmd implements `orca job migrate <name> --to <node>` (REQ-116,
|
||||
// C3=a). It is a drain+reschedule composite — NOT live-migration (no
|
||||
// storage replication). For each allocation of the named job running on
|
||||
// a node OTHER than --to, it stops the allocation (SSH systemctl stop),
|
||||
// then starts a new allocation on the target node (SSH systemctl start).
|
||||
// Idempotent: if the job is already running on the target node, it is a
|
||||
// no-op for that allocation.
|
||||
var jobMigrateCmd = &cobra.Command{
|
||||
Use: "migrate <name>",
|
||||
Short: "Drain+reschedule a job onto a target node (REQ-116)",
|
||||
Long: `Migrate a job onto a target node (REQ-116, C3=a).
|
||||
|
||||
This is a drain+reschedule composite, NOT live-migration: there is no
|
||||
storage replication. For every allocation of <name> currently running
|
||||
on a node other than --to, it stops the allocation (SSH systemctl stop
|
||||
orca-alloc-<id>.service) and starts a new allocation on the target
|
||||
node (SSH systemctl start orca-alloc-<new-id>.service).
|
||||
|
||||
Idempotent: if the job already has an allocation on the target node,
|
||||
the command is a no-op (C3=a, Q2=C).
|
||||
|
||||
NOTE: this command assumes allocations are tracked as systemd units
|
||||
named orca-alloc-<id>.service across the cluster, with <id> of the
|
||||
form <job-name>-<replica-index>. The new allocation on the target node
|
||||
is named <name>-migrated-<timestamp>.`,
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
jobName := args[0]
|
||||
if migrateTarget == "" {
|
||||
return fmt.Errorf("--to is required")
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(cmd.Context(), 5*time.Minute)
|
||||
defer cancel()
|
||||
|
||||
reg, closer, err := nodeRegistry()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer closer()
|
||||
|
||||
nodes, err := reg.List(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("list nodes: %w", err)
|
||||
}
|
||||
|
||||
target, err := findNode(ctx, reg, migrateTarget)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
targetPeer := peerAddrForNode(target)
|
||||
if targetPeer == "" {
|
||||
return fmt.Errorf("cannot resolve SSH address for target node %q", target.Name)
|
||||
}
|
||||
|
||||
ex, err := drainExecFromCtx(cmd.Context())
|
||||
if err != nil {
|
||||
return fmt.Errorf("ssh transport: %w", err)
|
||||
}
|
||||
|
||||
log := newLogger()
|
||||
log.Info("migrate: scanning cluster for job allocations", slog.String("job", jobName))
|
||||
|
||||
// Find allocations of <jobName> across all nodes. An
|
||||
// allocation "belongs to" the job if its alloc id starts
|
||||
// with "<jobName>-" (the scheduler emits ids of the form
|
||||
// ns/name-idx; we match on the name segment).
|
||||
jobPrefix := jobName + "-"
|
||||
|
||||
type alloc struct {
|
||||
node *model.Node
|
||||
peer string
|
||||
id string
|
||||
}
|
||||
var onTarget, onOthers []alloc
|
||||
|
||||
for i := range nodes {
|
||||
n := nodes[i]
|
||||
peer := peerAddrForNode(n)
|
||||
if peer == "" {
|
||||
continue
|
||||
}
|
||||
ids, err := listRunningAllocs(ctx, ex, peer)
|
||||
if err != nil {
|
||||
log.Warn("migrate: cannot list allocs on node",
|
||||
slog.String("node", n.Name), "error", err)
|
||||
continue
|
||||
}
|
||||
for _, id := range ids {
|
||||
if !strings.HasPrefix(id, jobPrefix) {
|
||||
continue
|
||||
}
|
||||
a := alloc{node: n, peer: peer, id: id}
|
||||
if n.ID == target.ID {
|
||||
onTarget = append(onTarget, a)
|
||||
} else {
|
||||
onOthers = append(onOthers, a)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
result := map[string]any{
|
||||
"job": jobName,
|
||||
"target": target.Name,
|
||||
"already_on_target": len(onTarget) > 0,
|
||||
}
|
||||
|
||||
// Idempotent: if the job is already running on the target
|
||||
// node and there is nothing to migrate, no-op.
|
||||
if len(onOthers) == 0 {
|
||||
result["stopped"] = []string{}
|
||||
result["started"] = []string{}
|
||||
auditMigrate(ctx, jobName, target.Name, "success", nil, result)
|
||||
if jsonOutput {
|
||||
return printJSON(result)
|
||||
}
|
||||
if len(onTarget) > 0 {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "✓ Job %s already running on %s (%d alloc(s)); nothing to migrate\n",
|
||||
jobName, target.Name, len(onTarget))
|
||||
} else {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "✓ Job %s has no running allocations to migrate\n", jobName)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Stop each allocation on a non-target node.
|
||||
var stopped []string
|
||||
var stopFailures []string
|
||||
for _, a := range onOthers {
|
||||
if err := stopAlloc(ctx, ex, a.peer, a.id); err != nil {
|
||||
stopFailures = append(stopFailures, a.id)
|
||||
log.Warn("migrate: failed to stop alloc",
|
||||
slog.String("alloc", a.id), slog.String("node", a.node.Name), "error", err)
|
||||
continue
|
||||
}
|
||||
stopped = append(stopped, a.id)
|
||||
}
|
||||
|
||||
// Start a new allocation on the target node. We use a
|
||||
// stable, deterministic id so re-runs are idempotent: if the
|
||||
// unit already exists & is running, systemctl start is a
|
||||
// no-op. The new id is "<jobName>-migrated-<unix-seconds>".
|
||||
newID := fmt.Sprintf("%s-migrated-%d", jobName, time.Now().Unix())
|
||||
startCmd := fmt.Sprintf("systemctl start %s", allocUnit(newID))
|
||||
var started []string
|
||||
if _, err := ex.Exec(ctx, targetPeer, startCmd); err != nil {
|
||||
log.Warn("migrate: failed to start new alloc on target",
|
||||
slog.String("alloc", newID), slog.String("node", target.Name), "error", err)
|
||||
result["start_error"] = err.Error()
|
||||
} else {
|
||||
started = append(started, newID)
|
||||
}
|
||||
|
||||
result["stopped"] = stopped
|
||||
result["started"] = started
|
||||
result["stop_failures"] = stopFailures
|
||||
|
||||
if len(stopFailures) == 0 && len(started) > 0 {
|
||||
auditMigrate(ctx, jobName, target.Name, "success", nil, result)
|
||||
} else {
|
||||
auditMigrate(ctx, jobName, target.Name, "partial",
|
||||
fmt.Errorf("%d stop failures, %d started", len(stopFailures), len(started)), result)
|
||||
}
|
||||
|
||||
if jsonOutput {
|
||||
return printJSON(result)
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "✓ Migrated job %s onto %s (%d stopped, %d started)\n",
|
||||
jobName, target.Name, len(stopped), len(started))
|
||||
for _, id := range stopped {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), " stopped %s\n", id)
|
||||
}
|
||||
for _, id := range started {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), " started %s on %s\n", id, target.Name)
|
||||
}
|
||||
if len(stopFailures) > 0 {
|
||||
return fmt.Errorf("migrate: %d alloc(s) failed to stop", len(stopFailures))
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
// auditMigrate records a job.migrate event in the audit log.
|
||||
func auditMigrate(ctx context.Context, jobName, target, result string, err error, meta map[string]any) {
|
||||
db, dbErr := store.Open(certpaths.DBPath())
|
||||
if dbErr != nil {
|
||||
return
|
||||
}
|
||||
defer db.Close()
|
||||
engine.NewAudit(store.NewAuditRepo(db), newLogger()).Record(ctx, "cli", "job.migrate", jobName, result, err, meta)
|
||||
}
|
||||
|
||||
func init() {
|
||||
nodeDrainCmd.Flags().DurationVar(&drainTimeout, "timeout", 30*time.Second,
|
||||
"max time to wait for allocations to stop")
|
||||
nodeCmd.AddCommand(nodeDrainCmd)
|
||||
|
||||
daemonCmd.AddCommand(daemonDrainAndStopCmd)
|
||||
|
||||
jobMigrateCmd.Flags().StringVar(&migrateTarget, "to", "", "target node name or id to migrate the job onto (required)")
|
||||
jobCmd.AddCommand(jobMigrateCmd)
|
||||
|
||||
}
|
||||
@@ -0,0 +1,526 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/certpaths"
|
||||
"git.cloudinit.dev/coreci/orca/internal/model"
|
||||
"git.cloudinit.dev/coreci/orca/internal/store"
|
||||
)
|
||||
|
||||
// mockDrainExec is a record-and-replay execer for the drain commands
|
||||
// (same pattern as internal/stepca mockExec). It matches each incoming
|
||||
// command against a list of (substring, output, exitCode) responses;
|
||||
// the first match wins. An entry with an empty substring matches any
|
||||
// command. The exit code is 0 (success) unless explicitly set; a
|
||||
// non-zero code is returned as an *sshExitErr so stopAlloc can treat
|
||||
// code 5 ("unit not loaded") as idempotent success.
|
||||
type mockDrainExec struct {
|
||||
mu sync.Mutex
|
||||
responses []mockDrainResp
|
||||
calls []mockDrainCall
|
||||
}
|
||||
|
||||
type mockDrainResp struct {
|
||||
match string
|
||||
out string
|
||||
exit int
|
||||
}
|
||||
|
||||
type mockDrainCall struct {
|
||||
peer string
|
||||
cmd string
|
||||
}
|
||||
|
||||
func (m *mockDrainExec) Exec(_ context.Context, peer, cmd string) ([]byte, error) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
m.calls = append(m.calls, mockDrainCall{peer: peer, cmd: cmd})
|
||||
for _, r := range m.responses {
|
||||
if r.match == "" || strings.Contains(cmd, r.match) {
|
||||
if r.exit != 0 {
|
||||
return []byte(r.out), &sshExitErr{code: r.exit}
|
||||
}
|
||||
return []byte(r.out), nil
|
||||
}
|
||||
}
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
func (m *mockDrainExec) callsFor(match string) []mockDrainCall {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
var out []mockDrainCall
|
||||
for _, c := range m.calls {
|
||||
if strings.Contains(c.cmd, match) {
|
||||
out = append(out, c)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (m *mockDrainExec) countCalls(match string) int {
|
||||
return len(m.callsFor(match))
|
||||
}
|
||||
|
||||
// drainTestEnv wires a mockDrainExec into drainExecOverride and returns
|
||||
// the mock + a cleanup func. Tests MUST defer the cleanup.
|
||||
func drainTestEnv(t *testing.T) *mockDrainExec {
|
||||
t.Helper()
|
||||
prev := drainExecOverride
|
||||
mx := &mockDrainExec{}
|
||||
drainExecOverride = mx
|
||||
t.Cleanup(func() { drainExecOverride = prev })
|
||||
return mx
|
||||
}
|
||||
|
||||
// drainNodeForTest inserts a node with a fixed id+name and returns it,
|
||||
// so drain commands can target it by name. Uses the test ORCA_HOME db.
|
||||
func drainNodeForTest(t *testing.T, name, addr string) *model.Node {
|
||||
t.Helper()
|
||||
db, err := store.Open(certpaths.DBPath())
|
||||
if err != nil {
|
||||
t.Fatalf("open db: %v", err)
|
||||
}
|
||||
defer db.Close()
|
||||
n := &model.Node{
|
||||
ID: "node-" + name,
|
||||
Name: name,
|
||||
Address: addr,
|
||||
State: model.NodeStateReady,
|
||||
JoinedAt: time.Now().UTC(),
|
||||
LastSeen: time.Now().UTC(),
|
||||
Kind: string(model.NodeKindLinux),
|
||||
}
|
||||
if err := store.NewNodeRepo(db).Insert(context.Background(), n); err != nil {
|
||||
t.Fatalf("insert node: %v", err)
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
func TestNodeDrain_StopsAllocsAndMarksDrained(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
node := drainNodeForTest(t, "drainee", "drainee:8443")
|
||||
drainTestEnv(t) // sets drainExecOverride (reset in cleanup)
|
||||
// Scripted exec: first list-units returns 2 running allocs; the
|
||||
// post-stop poll returns empty so waitAllocsStopped completes.
|
||||
mx := &scriptedDrainExec{}
|
||||
mx.queue("list-units", "orca-alloc-web-0.service loaded active running\norca-alloc-web-1.service loaded active running\n", 0)
|
||||
mx.queue("systemctl stop orca-alloc-web-0", "", 0)
|
||||
mx.queue("systemctl stop orca-alloc-web-1", "", 0)
|
||||
mx.queue("list-units", "", 0)
|
||||
drainExecOverride = mx
|
||||
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
nodeDrainCmd.SetOut(&buf)
|
||||
nodeDrainCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"node", "drain", node.Name, "--timeout", "5s"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("node drain: %v", err)
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "drained") {
|
||||
t.Errorf("expected drained message, got: %s", out)
|
||||
}
|
||||
if mx.countCalls("systemctl stop orca-alloc-web-0") == 0 || mx.countCalls("systemctl stop orca-alloc-web-1") == 0 {
|
||||
t.Errorf("expected stop commands for both allocs, calls: %+v", mx.calls)
|
||||
}
|
||||
|
||||
// Node state should be drained in the DB.
|
||||
db, _ := store.Open(certpaths.DBPath())
|
||||
defer db.Close()
|
||||
got, err := store.NewNodeRepo(db).Get(context.Background(), node.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("get node: %v", err)
|
||||
}
|
||||
if got.State != model.NodeStateDrained {
|
||||
t.Errorf("node state = %q, want drained", got.State)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNodeDrain_NoAllocs_MarksDrainedImmediately(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
node := drainNodeForTest(t, "emptynode", "emptynode:8443")
|
||||
mx := &scriptedDrainExec{}
|
||||
mx.queue("list-units", "", 0) // no allocs
|
||||
drainExecOverride = mx
|
||||
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
nodeDrainCmd.SetOut(&buf)
|
||||
nodeDrainCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"node", "drain", node.Name, "--timeout", "5s"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("node drain: %v", err)
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "drained") {
|
||||
t.Errorf("expected drained message, got: %s", out)
|
||||
}
|
||||
// Should not have issued any stop commands.
|
||||
if mx.countCalls("systemctl stop") != 0 {
|
||||
t.Errorf("expected no stop commands, got: %+v", mx.calls)
|
||||
}
|
||||
|
||||
db, _ := store.Open(certpaths.DBPath())
|
||||
defer db.Close()
|
||||
got, _ := store.NewNodeRepo(db).Get(context.Background(), node.ID)
|
||||
if got.State != model.NodeStateDrained {
|
||||
t.Errorf("node state = %q, want drained", got.State)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNodeDrain_RespectsTimeout(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
node := drainNodeForTest(t, "stucknode", "stucknode:8443")
|
||||
// The alloc never leaves the running list → waitAllocsStopped hits
|
||||
// the timeout. The drain reports partial and leaves the node in
|
||||
// "draining".
|
||||
mx := &scriptedDrainExec{}
|
||||
mx.queueAlways("list-units", "orca-alloc-stuck-0.service loaded active running\n", 0)
|
||||
mx.queueAlways("systemctl stop", "", 0)
|
||||
drainExecOverride = mx
|
||||
|
||||
// Poll interval is 500ms; use a 1s timeout so the test is fast but
|
||||
// still exercises the timeout path.
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
nodeDrainCmd.SetOut(&buf)
|
||||
nodeDrainCmd.SetErr(&buf)
|
||||
start := time.Now()
|
||||
rootCmd.SetArgs([]string{"node", "drain", node.Name, "--timeout", "1s"})
|
||||
err := rootCmd.Execute()
|
||||
elapsed := time.Since(start)
|
||||
if err == nil {
|
||||
t.Fatalf("expected drain to error on timeout, got nil")
|
||||
}
|
||||
if elapsed > 5*time.Second {
|
||||
t.Errorf("drain took too long (%v); timeout not respected", elapsed)
|
||||
}
|
||||
|
||||
// Node should remain in "draining" (not drained) since an alloc is
|
||||
// still running.
|
||||
db, _ := store.Open(certpaths.DBPath())
|
||||
defer db.Close()
|
||||
got, _ := store.NewNodeRepo(db).Get(context.Background(), node.ID)
|
||||
if got.State != model.NodeStateDraining {
|
||||
t.Errorf("node state = %q, want draining (timeout)", got.State)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNodeDrain_NodeNotFound(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
drainTestEnv(t)
|
||||
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
nodeDrainCmd.SetOut(&buf)
|
||||
nodeDrainCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"node", "drain", "no.such.node", "--timeout", "1s"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected error for unknown node, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "not found") {
|
||||
t.Errorf("error should mention not found, got: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDaemonDrainAndStop_StopsEachPeer(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
drainNodeForTest(t, "peer-a", "peer-a:8443")
|
||||
drainNodeForTest(t, "peer-b", "peer-b:8443")
|
||||
drainNodeForTest(t, "peer-c", "peer-c:8443")
|
||||
|
||||
mx := &scriptedDrainExec{}
|
||||
mx.queueAlways("systemctl stop orca-daemon.service", "", 0)
|
||||
drainExecOverride = mx
|
||||
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
daemonCmd.SetOut(&buf)
|
||||
daemonCmd.SetErr(&buf)
|
||||
daemonDrainAndStopCmd.SetOut(&buf)
|
||||
daemonDrainAndStopCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"daemon", "drain-and-stop"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("daemon drain-and-stop: %v", err)
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "complete") {
|
||||
t.Errorf("expected complete message, got: %s", out)
|
||||
}
|
||||
// One stop call per peer.
|
||||
stops := mx.countCalls("systemctl stop orca-daemon.service")
|
||||
if stops != 3 {
|
||||
t.Errorf("expected 3 daemon stop calls, got %d (calls: %+v)", stops, mx.calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDaemonDrainAndStop_AlreadyStoppedIsIdempotent(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
drainNodeForTest(t, "migrated", "migrated:8443")
|
||||
mx := &scriptedDrainExec{}
|
||||
// systemctl stop returns exit 5 ("unit not loaded") → already
|
||||
// migrated, idempotent.
|
||||
mx.queueAlways("systemctl stop orca-daemon.service", "", 5)
|
||||
drainExecOverride = mx
|
||||
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
daemonCmd.SetOut(&buf)
|
||||
daemonCmd.SetErr(&buf)
|
||||
daemonDrainAndStopCmd.SetOut(&buf)
|
||||
daemonDrainAndStopCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"daemon", "drain-and-stop"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("daemon drain-and-stop should be idempotent: %v", err)
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "already stopped") && !strings.Contains(out, "already-stopped") {
|
||||
t.Errorf("expected already-stopped in output, got: %s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestJobMigrate_StopsOldAndStartsNewOnTarget(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
src := drainNodeForTest(t, "src", "src:8443")
|
||||
_ = drainNodeForTest(t, "tgt", "tgt:8443")
|
||||
_ = src
|
||||
|
||||
mx := &scriptedDrainExec{}
|
||||
// src node lists the job's alloc running.
|
||||
mx.queue("list-units",
|
||||
"orca-alloc-web-0.service loaded active running\n", 0)
|
||||
// tgt node has no allocs (so migrate starts a new one).
|
||||
mx.queue("list-units", "", 0)
|
||||
// stop on src succeeds.
|
||||
mx.queue("systemctl stop orca-alloc-web-0", "", 0)
|
||||
// start on tgt succeeds.
|
||||
mx.queue("systemctl start", "", 0)
|
||||
drainExecOverride = mx
|
||||
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
jobCmd.SetOut(&buf)
|
||||
jobCmd.SetErr(&buf)
|
||||
jobMigrateCmd.SetOut(&buf)
|
||||
jobMigrateCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"job", "migrate", "web", "--to", "tgt"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("job migrate: %v", err)
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "Migrated") {
|
||||
t.Errorf("expected Migrated message, got: %s", out)
|
||||
}
|
||||
if mx.countCalls("systemctl stop orca-alloc-web-0") == 0 {
|
||||
t.Errorf("expected stop for web-0 on src, calls: %+v", mx.calls)
|
||||
}
|
||||
if mx.countCalls("systemctl start orca-alloc-web-migrated") == 0 {
|
||||
t.Errorf("expected start of new alloc on tgt, calls: %+v", mx.calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestJobMigrate_IdempotentWhenAlreadyOnTarget(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
_ = drainNodeForTest(t, "src", "src:8443")
|
||||
_ = drainNodeForTest(t, "tgt", "tgt:8443")
|
||||
|
||||
mx := &scriptedDrainExec{}
|
||||
// The job is running ONLY on tgt (the target). No allocs on src.
|
||||
mx.queue("list-units", "orca-alloc-web-0.service loaded active running\n", 0) // src: empty would be ideal; see scripted note
|
||||
// Use scripted: src empty, tgt has web-0.
|
||||
mx = &scriptedDrainExec{}
|
||||
mx.queue("list-units", "", 0) // first list-units call (src) → empty
|
||||
mx.queue("list-units", "orca-alloc-web-0.service loaded active running\n", 0) // second (tgt) → has web-0
|
||||
drainExecOverride = mx
|
||||
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
jobCmd.SetOut(&buf)
|
||||
jobCmd.SetErr(&buf)
|
||||
jobMigrateCmd.SetOut(&buf)
|
||||
jobMigrateCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"job", "migrate", "web", "--to", "tgt"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("job migrate (idempotent): %v", err)
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "already running on") {
|
||||
t.Errorf("expected already-running idempotent message, got: %s", out)
|
||||
}
|
||||
// No stop or start commands should have been issued.
|
||||
if mx.countCalls("systemctl stop") != 0 || mx.countCalls("systemctl start") != 0 {
|
||||
t.Errorf("idempotent migrate must not stop/start, calls: %+v", mx.calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestJobMigrate_RequiresToFlag(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
drainTestEnv(t)
|
||||
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
jobCmd.SetOut(&buf)
|
||||
jobCmd.SetErr(&buf)
|
||||
jobMigrateCmd.SetOut(&buf)
|
||||
jobMigrateCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"job", "migrate", "web"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected error for missing --to, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "--to is required") {
|
||||
t.Errorf("error should mention --to required, got: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSetNodeStateUpdatesDB(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
// Sanity test for the store-level SetNodeState used by drain.
|
||||
db, err := store.Open(certpaths.DBPath())
|
||||
if err != nil {
|
||||
t.Fatalf("open db: %v", err)
|
||||
}
|
||||
defer db.Close()
|
||||
repo := store.NewNodeRepo(db)
|
||||
n := &model.Node{
|
||||
ID: "node-setstate",
|
||||
Name: "setstate",
|
||||
Address: "setstate:8443",
|
||||
State: model.NodeStateReady,
|
||||
JoinedAt: time.Now().UTC(),
|
||||
LastSeen: time.Now().UTC(),
|
||||
}
|
||||
ctx := context.Background()
|
||||
if err := repo.Insert(ctx, n); err != nil {
|
||||
t.Fatalf("insert: %v", err)
|
||||
}
|
||||
if err := repo.SetNodeState(ctx, n.ID, string(model.NodeStateDraining)); err != nil {
|
||||
t.Fatalf("SetNodeState: %v", err)
|
||||
}
|
||||
got, err := repo.Get(ctx, n.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("get: %v", err)
|
||||
}
|
||||
if got.State != model.NodeStateDraining {
|
||||
t.Errorf("state = %q, want draining", got.State)
|
||||
}
|
||||
|
||||
// Missing node returns ErrNotFound.
|
||||
err = repo.SetNodeState(ctx, "ghost", "draining")
|
||||
if !errors.Is(err, store.ErrNotFound) {
|
||||
t.Errorf("SetNodeState(ghost) = %v, want ErrNotFound", err)
|
||||
}
|
||||
}
|
||||
|
||||
// scriptedDrainExec is a record-and-replay execer that returns queued
|
||||
// responses in order. queueAlways installs a sticky response that
|
||||
// matches every subsequent call containing the substring. This is
|
||||
// more convenient than mockDrainExec when the same command (e.g.
|
||||
// list-units) must return different outputs across calls.
|
||||
type scriptedDrainExec struct {
|
||||
mu sync.Mutex
|
||||
queued []scriptedResp
|
||||
sticky []scriptedResp
|
||||
calls []mockDrainCall
|
||||
}
|
||||
|
||||
type scriptedResp struct {
|
||||
match string
|
||||
out string
|
||||
exit int
|
||||
}
|
||||
|
||||
func (s *scriptedDrainExec) queue(match, out string, exit int) {
|
||||
s.queued = append(s.queued, scriptedResp{match: match, out: out, exit: exit})
|
||||
}
|
||||
|
||||
func (s *scriptedDrainExec) queueAlways(match, out string, exit int) {
|
||||
s.sticky = append(s.sticky, scriptedResp{match: match, out: out, exit: exit})
|
||||
}
|
||||
|
||||
func (s *scriptedDrainExec) Exec(_ context.Context, peer, cmd string) ([]byte, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
s.calls = append(s.calls, mockDrainCall{peer: peer, cmd: cmd})
|
||||
// Sticky responses win if they match.
|
||||
for _, r := range s.sticky {
|
||||
if r.match == "" || strings.Contains(cmd, r.match) {
|
||||
if r.exit != 0 {
|
||||
return []byte(r.out), &sshExitErr{code: r.exit}
|
||||
}
|
||||
return []byte(r.out), nil
|
||||
}
|
||||
}
|
||||
// Queued responses: pop the first matching entry.
|
||||
for i, r := range s.queued {
|
||||
if r.match == "" || strings.Contains(cmd, r.match) {
|
||||
s.queued = append(s.queued[:i], s.queued[i+1:]...)
|
||||
if r.exit != 0 {
|
||||
return []byte(r.out), &sshExitErr{code: r.exit}
|
||||
}
|
||||
return []byte(r.out), nil
|
||||
}
|
||||
}
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
func (s *scriptedDrainExec) callsFor(match string) []mockDrainCall {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
var out []mockDrainCall
|
||||
for _, c := range s.calls {
|
||||
if strings.Contains(c.cmd, match) {
|
||||
out = append(out, c)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (s *scriptedDrainExec) countCalls(match string) int {
|
||||
return len(s.callsFor(match))
|
||||
}
|
||||
@@ -0,0 +1,391 @@
|
||||
// Package cli: drift.go implements the `orca drift` subcommand family
|
||||
// (P10b, v0.11; R-018/R-019/R-020, REQ-104). Subcommands:
|
||||
//
|
||||
// orca drift show current drift state (table)
|
||||
// orca drift watch [--interval=2s] [--paths=...] [--json]
|
||||
// stream drift events (iter.Seq2, D-017)
|
||||
// ctrl-c cancels via signal.NotifyContext (D-023)
|
||||
// orca drift show [--peer <host>] detailed drift events for a peer
|
||||
// orca drift acknowledge <peer> <path>
|
||||
// record operator acknowledgment
|
||||
// orca drift remediate <peer> <path> [--force]
|
||||
// trigger manual remediation
|
||||
// orca drift config show show current drift config
|
||||
// orca drift config validate validate config
|
||||
//
|
||||
// Plus the `orca job restart <name>` command for EnvironmentFile drift
|
||||
// (REQ-113, D-235) — restarts an allocation to pick up env-file drift.
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/signal"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/certpaths"
|
||||
"git.cloudinit.dev/coreci/orca/internal/drift"
|
||||
"git.cloudinit.dev/coreci/orca/internal/sshpush"
|
||||
)
|
||||
|
||||
var (
|
||||
driftWatchInterval time.Duration
|
||||
driftWatchPaths []string
|
||||
driftShowPeer string
|
||||
driftConfigPath string
|
||||
driftRemediateForce bool
|
||||
driftAckPeer string
|
||||
driftAckPath string
|
||||
driftRemediatePeer string
|
||||
driftRemediatePath string
|
||||
driftWatchPollOverride time.Duration
|
||||
)
|
||||
|
||||
// driftTransport is the SSH-push surface the drift CLI needs. It
|
||||
// mirrors drift.Transport; tests substitute a mock.
|
||||
type driftTransport interface {
|
||||
Exec(ctx context.Context, peer string, cmd string) ([]byte, error)
|
||||
WriteFileIdempotent(ctx context.Context, peer string, path string, content []byte, mode os.FileMode) (bool, error)
|
||||
ReadFile(ctx context.Context, peer string, path string) ([]byte, error)
|
||||
}
|
||||
|
||||
// driftTransportOverride is the package-level test seam.
|
||||
var driftTransportOverride driftTransport
|
||||
|
||||
func driftTransportFromCtx() (driftTransport, error) {
|
||||
if driftTransportOverride != nil {
|
||||
return driftTransportOverride, nil
|
||||
}
|
||||
keyPath := certpaths.SSHKeyPath()
|
||||
khPath := certpaths.KnownHostsPath()
|
||||
return sshpush.NewTransport(keyPath, khPath), nil
|
||||
}
|
||||
|
||||
// driftDetectorOverride is the package-level test seam for the
|
||||
// Detector itself. When non-nil it replaces the production detector
|
||||
// (which wraps a driftTransport). Tests set it and restore nil.
|
||||
var driftDetectorOverride drift.Detector
|
||||
|
||||
func driftDetector() (drift.Detector, error) {
|
||||
if driftDetectorOverride != nil {
|
||||
return driftDetectorOverride, nil
|
||||
}
|
||||
t, err := driftTransportFromCtx()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return drift.NewDefaultDetector(t), nil
|
||||
}
|
||||
|
||||
var driftCmd = &cobra.Command{
|
||||
Use: "drift",
|
||||
Short: "Detect and remediate control-plane drift (P10b)",
|
||||
Long: `Orca's drift detector is a BACKSTOP (R-019): the primary
|
||||
consistency mechanism is systemd / Traefik / step-ca / Syncthing
|
||||
themselves. The detector polls the lead's aggregated drift state
|
||||
(drift-events-aggregated.json) and can trigger orca-remediate.sh for
|
||||
auto-remediable paths. Pre-flight drift blocks txn apply (R-020).`,
|
||||
}
|
||||
|
||||
var driftShowCmd = &cobra.Command{
|
||||
Use: "show",
|
||||
Short: "Show current drift state (table format)",
|
||||
Long: `Show the aggregated drift events from the lead. Use --peer to filter to a single peer.`,
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
d, err := driftDetector()
|
||||
if err != nil {
|
||||
return fmt.Errorf("drift detector: %w", err)
|
||||
}
|
||||
events, err := d.Aggregate(cmd.Context(), driftShowPeer)
|
||||
if err != nil {
|
||||
return fmt.Errorf("aggregate: %w", err)
|
||||
}
|
||||
if driftShowPeer != "" {
|
||||
var filtered []drift.Event
|
||||
for _, e := range events {
|
||||
if e.Host == driftShowPeer {
|
||||
filtered = append(filtered, e)
|
||||
}
|
||||
}
|
||||
events = filtered
|
||||
}
|
||||
if jsonOutput {
|
||||
return printJSON(events)
|
||||
}
|
||||
out := cmd.OutOrStdout()
|
||||
if len(events) == 0 {
|
||||
fmt.Fprintln(out, "No drift events.")
|
||||
return nil
|
||||
}
|
||||
fmt.Fprintf(out, "%-20s %-20s %-40s %-10s %-10s\n", "EVENT-ID", "HOST", "PATH", "STATUS", "CONFIRMED")
|
||||
for _, e := range events {
|
||||
path := e.Path
|
||||
if len(path) > 40 {
|
||||
path = "..." + path[len(path)-37:]
|
||||
}
|
||||
confirmed := "no"
|
||||
if e.DriftConfirmed {
|
||||
confirmed = "yes"
|
||||
}
|
||||
fmt.Fprintf(out, "%-20s %-20s %-40s %-10s %-10s\n", e.EventID, e.Host, path, e.Status, confirmed)
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
var driftWatchCmd = &cobra.Command{
|
||||
Use: "watch",
|
||||
Short: "Stream drift events (ctrl-c to cancel)",
|
||||
Long: `Stream drift events from the lead's aggregated state. Default
|
||||
poll is 2s; override with --interval. Use --paths=<glob1>,<glob2> to
|
||||
filter. Uses iter.Seq2 (D-017) and signal.NotifyContext for ctrl-c
|
||||
(D-023).`,
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
d, err := driftDetector()
|
||||
if err != nil {
|
||||
return fmt.Errorf("drift detector: %w", err)
|
||||
}
|
||||
ctx, cancel := signal.NotifyContext(cmd.Context(), os.Interrupt, syscall.SIGTERM)
|
||||
defer cancel()
|
||||
interval := driftWatchInterval
|
||||
if interval <= 0 {
|
||||
interval = 2 * time.Second
|
||||
}
|
||||
var specs []drift.PathSpec
|
||||
for _, p := range driftWatchPaths {
|
||||
if p == "" {
|
||||
continue
|
||||
}
|
||||
specs = append(specs, drift.PathSpec{Pattern: p, Interval: interval})
|
||||
}
|
||||
out := cmd.OutOrStdout()
|
||||
for e, err := range d.Watch(ctx, specs) {
|
||||
if err != nil {
|
||||
if errors.Is(err, context.Canceled) {
|
||||
return nil
|
||||
}
|
||||
fmt.Fprintf(out, "watch error: %v\n", err)
|
||||
continue
|
||||
}
|
||||
if jsonOutput {
|
||||
line, _ := json.Marshal(e)
|
||||
fmt.Fprintln(out, string(line))
|
||||
} else {
|
||||
fmt.Fprintf(out, "%s [%s] %s %s %s confirmed=%t\n", e.TS.Format(time.RFC3339), e.EventID, e.Host, e.Path, e.Status, e.DriftConfirmed)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
var driftAckCmd = &cobra.Command{
|
||||
Use: "acknowledge <peer> <path>",
|
||||
Short: "Record operator acknowledgment of drift on a peer",
|
||||
Long: `Record operator acknowledgment for the given path in
|
||||
drift-acknowledgments.json on the lead. Acknowledged drift no longer
|
||||
blocks txn apply for that namespace (R-020).`,
|
||||
Args: cobra.ExactArgs(2),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
peer := args[0]
|
||||
path := args[1]
|
||||
d, err := driftDetector()
|
||||
if err != nil {
|
||||
return fmt.Errorf("drift detector: %w", err)
|
||||
}
|
||||
if err := d.Acknowledge(cmd.Context(), peer, path); err != nil {
|
||||
return fmt.Errorf("acknowledge: %w", err)
|
||||
}
|
||||
printResult(fmt.Sprintf("✓ Acknowledged drift on %s for %s", peer, path), map[string]any{
|
||||
"peer": peer, "path": path, "status": "acknowledged",
|
||||
})
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
var driftRemediateCmd = &cobra.Command{
|
||||
Use: "remediate <peer> <path>",
|
||||
Short: "Trigger manual remediation of drift on a peer",
|
||||
Long: `Trigger orca-remediate.sh on the lead for the given path.
|
||||
--force bypasses the cooldown window (C4).`,
|
||||
Args: cobra.ExactArgs(2),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
peer := args[0]
|
||||
path := args[1]
|
||||
d, err := driftDetector()
|
||||
if err != nil {
|
||||
return fmt.Errorf("drift detector: %w", err)
|
||||
}
|
||||
if err := d.Remediate(cmd.Context(), peer, path, driftRemediateForce); err != nil {
|
||||
if errors.Is(err, drift.ErrCooldown) {
|
||||
printResult(fmt.Sprintf("✗ Remediation in cooldown for %s on %s (use --force to bypass)", path, peer), map[string]any{
|
||||
"peer": peer, "path": path, "status": "cooldown",
|
||||
})
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("remediate: %w", err)
|
||||
}
|
||||
printResult(fmt.Sprintf("✓ Remediated drift on %s for %s", peer, path), map[string]any{
|
||||
"peer": peer, "path": path, "status": "remediated", "force": driftRemediateForce,
|
||||
})
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
var driftConfigCmd = &cobra.Command{
|
||||
Use: "config",
|
||||
Short: "Show or validate the drift config",
|
||||
}
|
||||
|
||||
var driftConfigShowCmd = &cobra.Command{
|
||||
Use: "show",
|
||||
Short: "Show the current drift config",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
cfg, err := drift.LoadConfig(driftConfigPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("load config: %w", err)
|
||||
}
|
||||
if jsonOutput {
|
||||
return printJSON(cfg)
|
||||
}
|
||||
out := cmd.OutOrStdout()
|
||||
fmt.Fprintf(out, "Polling: enabled=%t default=%s max_peers=%d\n", cfg.Polling.Enabled, cfg.Polling.DefaultInterval, cfg.Polling.MaxConcurrentPeers)
|
||||
fmt.Fprintln(out, "Critical paths:")
|
||||
for _, p := range cfg.Paths.Critical {
|
||||
fmt.Fprintf(out, " [%s] %s (interval=%s, path_unit=%t)\n", p.Tier, p.Pattern, p.Interval, p.SystemdPathUnit)
|
||||
}
|
||||
fmt.Fprintln(out, "Standard paths:")
|
||||
for _, p := range cfg.Paths.Standard {
|
||||
fmt.Fprintf(out, " [%s] %s (interval=%s)\n", p.Tier, p.Pattern, p.Interval)
|
||||
}
|
||||
fmt.Fprintln(out, "Excluded paths:")
|
||||
for _, p := range cfg.Paths.Excluded {
|
||||
fmt.Fprintf(out, " %s\n", p)
|
||||
}
|
||||
fmt.Fprintf(out, "Remediation: auto=%t notify=%t\n", cfg.Remediate.Auto, cfg.Remediate.NotifyOnRemediation)
|
||||
if len(cfg.Remediate.AutoPaths) > 0 {
|
||||
fmt.Fprintln(out, " auto_paths:")
|
||||
for _, p := range cfg.Remediate.AutoPaths {
|
||||
fmt.Fprintf(out, " %s\n", p)
|
||||
}
|
||||
}
|
||||
if len(cfg.Remediate.RequireApproval) > 0 {
|
||||
fmt.Fprintln(out, " require_approval:")
|
||||
for _, p := range cfg.Remediate.RequireApproval {
|
||||
fmt.Fprintf(out, " %s\n", p)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
var driftConfigValidateCmd = &cobra.Command{
|
||||
Use: "validate",
|
||||
Short: "Validate the drift config",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
cfg, err := drift.LoadConfig(driftConfigPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("load config: %w", err)
|
||||
}
|
||||
if err := drift.ValidateConfig(cfg); err != nil {
|
||||
printResult(fmt.Sprintf("✗ Config invalid: %v", err), map[string]any{"valid": false, "error": err.Error()})
|
||||
return err
|
||||
}
|
||||
printResult("✓ Config valid", map[string]any{"valid": true})
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
// jobRestartCmd implements `orca job restart <name>` (REQ-113, D-235):
|
||||
// restart an allocation on its peer to pick up EnvironmentFile drift.
|
||||
var jobRestartCmd = &cobra.Command{
|
||||
Use: "restart <name>",
|
||||
Short: "Restart an allocation to pick up EnvironmentFile drift (REQ-113)",
|
||||
Long: `SSH to the peer running allocation <name> and run
|
||||
systemctl restart orca-alloc-<id>.service. This is the normal
|
||||
allocation lifecycle (NOT file-level remediation) and is triggered
|
||||
when /etc/orca/allocs/<id>/env drifts.`,
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
name := args[0]
|
||||
peer := jobRestartPeer
|
||||
if peer == "" {
|
||||
return fmt.Errorf("--peer is required for job restart")
|
||||
}
|
||||
transport, err := driftTransportFromCtx()
|
||||
if err != nil {
|
||||
return fmt.Errorf("ssh transport: %w", err)
|
||||
}
|
||||
unit := fmt.Sprintf("orca-alloc-%s.service", name)
|
||||
restartCmd := fmt.Sprintf("systemctl restart %s", shellQuoteDrift(unit))
|
||||
out, err := transport.Exec(cmd.Context(), peer, restartCmd)
|
||||
if err != nil {
|
||||
return fmt.Errorf("restart %s on %s: %w (output: %s)", unit, peer, err, string(out))
|
||||
}
|
||||
printResult(fmt.Sprintf("✓ Restarted %s on %s", unit, peer), map[string]any{
|
||||
"unit": unit, "peer": peer, "status": "restarted", "output": string(out),
|
||||
})
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
var jobRestartPeer string
|
||||
|
||||
func shellQuoteDrift(s string) string {
|
||||
return "'" + strings.ReplaceAll(s, "'", "'\\''") + "'"
|
||||
}
|
||||
|
||||
func init() {
|
||||
driftWatchCmd.Flags().DurationVar(&driftWatchInterval, "interval", 2*time.Second, "poll interval (default 2s)")
|
||||
driftWatchCmd.Flags().StringSliceVar(&driftWatchPaths, "paths", nil, "comma-separated glob patterns to watch (default: all)")
|
||||
driftShowCmd.Flags().StringVar(&driftShowPeer, "peer", "", "filter to a single peer host")
|
||||
driftRemediateCmd.Flags().BoolVar(&driftRemediateForce, "force", false, "bypass the cooldown window (C4)")
|
||||
driftConfigCmd.PersistentFlags().StringVar(&driftConfigPath, "config", "", "path to drift config JSON (default: built-in)")
|
||||
jobRestartCmd.Flags().StringVar(&jobRestartPeer, "peer", "", "peer address (host:port) running the allocation")
|
||||
|
||||
driftCmd.AddCommand(driftShowCmd)
|
||||
driftCmd.AddCommand(driftWatchCmd)
|
||||
driftCmd.AddCommand(driftAckCmd)
|
||||
driftCmd.AddCommand(driftRemediateCmd)
|
||||
driftCmd.AddCommand(driftConfigCmd)
|
||||
driftConfigCmd.AddCommand(driftConfigShowCmd)
|
||||
driftConfigCmd.AddCommand(driftConfigValidateCmd)
|
||||
rootCmd.AddCommand(driftCmd)
|
||||
|
||||
jobCmd.AddCommand(jobRestartCmd)
|
||||
}
|
||||
|
||||
// writeClusterDefaultDriftConfig writes the canonical default drift
|
||||
// config to the cluster state dir so the lead has a reference copy.
|
||||
// Best-effort; caller logs failures.
|
||||
func writeClusterDefaultDriftConfig() error {
|
||||
dir := filepath.Join(os.Getenv("ORCA_HOME"), "cluster", "state")
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
return fmt.Errorf("mkdir cluster state: %w", err)
|
||||
}
|
||||
path := filepath.Join(dir, "drift.json")
|
||||
cfg := drift.DefaultConfig()
|
||||
raw, err := json.MarshalIndent(cfg, "", " ")
|
||||
if err != nil {
|
||||
return fmt.Errorf("marshal drift config: %w", err)
|
||||
}
|
||||
tmp := path + ".tmp"
|
||||
if err := os.WriteFile(tmp, raw, 0o644); err != nil {
|
||||
return fmt.Errorf("write drift config: %w", err)
|
||||
}
|
||||
if err := os.Rename(tmp, path); err != nil {
|
||||
return fmt.Errorf("rename drift config: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,631 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"iter"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/drift"
|
||||
)
|
||||
|
||||
type mockDriftTransport struct {
|
||||
execOut []byte
|
||||
execErr error
|
||||
readOut []byte
|
||||
readErr error
|
||||
writes []mockDriftWrite
|
||||
execFn func(ctx context.Context, peer string, cmd string) ([]byte, error)
|
||||
execs []string
|
||||
}
|
||||
|
||||
type mockDriftWrite struct {
|
||||
peer string
|
||||
path string
|
||||
content []byte
|
||||
mode os.FileMode
|
||||
}
|
||||
|
||||
func (m *mockDriftTransport) Exec(ctx context.Context, peer string, cmd string) ([]byte, error) {
|
||||
if m.execFn != nil {
|
||||
return m.execFn(ctx, peer, cmd)
|
||||
}
|
||||
m.execs = append(m.execs, cmd)
|
||||
return m.execOut, m.execErr
|
||||
}
|
||||
|
||||
func (m *mockDriftTransport) WriteFileIdempotent(ctx context.Context, peer string, path string, content []byte, mode os.FileMode) (bool, error) {
|
||||
m.writes = append(m.writes, mockDriftWrite{peer, path, content, mode})
|
||||
return true, nil
|
||||
}
|
||||
|
||||
func (m *mockDriftTransport) ReadFile(ctx context.Context, peer string, path string) ([]byte, error) {
|
||||
return m.readOut, m.readErr
|
||||
}
|
||||
|
||||
// fakeDetector is a record-replay drift.Detector for CLI tests.
|
||||
type fakeDetector struct {
|
||||
aggEvents []drift.Event
|
||||
aggErr error
|
||||
remediateErr error
|
||||
ackWrites int
|
||||
remediateCalls []remediateCall
|
||||
}
|
||||
|
||||
type remediateCall struct {
|
||||
peer string
|
||||
path string
|
||||
force bool
|
||||
}
|
||||
|
||||
func (f *fakeDetector) Watch(ctx context.Context, paths []drift.PathSpec) iter.Seq2[drift.Event, error] {
|
||||
return func(yield func(drift.Event, error) bool) {
|
||||
for _, e := range f.aggEvents {
|
||||
if !yield(e, nil) {
|
||||
return
|
||||
}
|
||||
}
|
||||
<-ctx.Done()
|
||||
}
|
||||
}
|
||||
|
||||
func (f *fakeDetector) Aggregate(ctx context.Context, leadPeer string) ([]drift.Event, error) {
|
||||
return f.aggEvents, f.aggErr
|
||||
}
|
||||
|
||||
func (f *fakeDetector) Remediate(ctx context.Context, leadPeer, path string, force bool) error {
|
||||
f.remediateCalls = append(f.remediateCalls, remediateCall{leadPeer, path, force})
|
||||
return f.remediateErr
|
||||
}
|
||||
|
||||
func (f *fakeDetector) Acknowledge(ctx context.Context, leadPeer, path string) error {
|
||||
f.ackWrites++
|
||||
return nil
|
||||
}
|
||||
|
||||
func setupDriftCLITest(t *testing.T) string {
|
||||
t.Helper()
|
||||
home := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", home)
|
||||
t.Setenv("ORCA_LEAD_STATE_DIR", filepath.Join(home, "state"))
|
||||
return home
|
||||
}
|
||||
|
||||
func TestDriftCmdRegistered(t *testing.T) {
|
||||
for _, c := range rootCmd.Commands() {
|
||||
if c.Name() == "drift" {
|
||||
return
|
||||
}
|
||||
}
|
||||
t.Fatal("drift command not registered on root")
|
||||
}
|
||||
|
||||
func TestDriftSubcommandsRegistered(t *testing.T) {
|
||||
for _, c := range rootCmd.Commands() {
|
||||
if c.Name() != "drift" {
|
||||
continue
|
||||
}
|
||||
want := map[string]bool{
|
||||
"show": false,
|
||||
"watch": false,
|
||||
"acknowledge": false,
|
||||
"remediate": false,
|
||||
"config": false,
|
||||
}
|
||||
for _, sub := range c.Commands() {
|
||||
if _, ok := want[sub.Name()]; ok {
|
||||
want[sub.Name()] = true
|
||||
}
|
||||
}
|
||||
for name, found := range want {
|
||||
if !found {
|
||||
t.Errorf("drift subcommand %q not registered", name)
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
t.Fatal("drift command not registered")
|
||||
}
|
||||
|
||||
func TestDriftConfigSubcommands(t *testing.T) {
|
||||
for _, c := range rootCmd.Commands() {
|
||||
if c.Name() != "drift" {
|
||||
continue
|
||||
}
|
||||
for _, sub := range c.Commands() {
|
||||
if sub.Name() != "config" {
|
||||
continue
|
||||
}
|
||||
want := map[string]bool{"show": false, "validate": false}
|
||||
for _, s := range sub.Commands() {
|
||||
if _, ok := want[s.Name()]; ok {
|
||||
want[s.Name()] = true
|
||||
}
|
||||
}
|
||||
for name, found := range want {
|
||||
if !found {
|
||||
t.Errorf("drift config subcommand %q not registered", name)
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
}
|
||||
t.Fatal("drift config not registered")
|
||||
}
|
||||
|
||||
func TestJobRestartRegistered(t *testing.T) {
|
||||
for _, c := range rootCmd.Commands() {
|
||||
if c.Name() != "job" {
|
||||
continue
|
||||
}
|
||||
for _, sub := range c.Commands() {
|
||||
if sub.Name() == "restart" {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
t.Fatal("job restart not registered")
|
||||
}
|
||||
|
||||
func TestDriftShowEmpty(t *testing.T) {
|
||||
setupDriftCLITest(t)
|
||||
fd := &fakeDetector{aggEvents: nil}
|
||||
driftDetectorOverride = fd
|
||||
defer func() { driftDetectorOverride = nil }()
|
||||
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"drift", "show"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("drift show: %v", err)
|
||||
}
|
||||
if !bytesContains(buf.String(), "No drift events") {
|
||||
t.Errorf("empty show output: %s", buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestDriftShowTable(t *testing.T) {
|
||||
setupDriftCLITest(t)
|
||||
fd := &fakeDetector{aggEvents: []drift.Event{
|
||||
{EventID: "E1", Host: "peer1", Path: "/etc/traefik/dynamic/orca.yml", Status: drift.StatusModified, DriftConfirmed: true},
|
||||
}}
|
||||
driftDetectorOverride = fd
|
||||
defer func() { driftDetectorOverride = nil }()
|
||||
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"drift", "show"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("drift show: %v", err)
|
||||
}
|
||||
out := buf.String()
|
||||
for _, want := range []string{"E1", "peer1", "modified"} {
|
||||
if !bytesContains(out, want) {
|
||||
t.Errorf("output missing %q: %s", want, out)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestDriftShowJSON(t *testing.T) {
|
||||
setupDriftCLITest(t)
|
||||
fd := &fakeDetector{aggEvents: []drift.Event{
|
||||
{EventID: "E1", Host: "peer1", Path: "/etc/x", Status: drift.StatusCreated, DriftConfirmed: true},
|
||||
}}
|
||||
driftDetectorOverride = fd
|
||||
defer func() { driftDetectorOverride = nil }()
|
||||
|
||||
resetRootFlags(t)
|
||||
_ = rootCmd.PersistentFlags().Set("json", "true")
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"drift", "show"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("drift show --json: %v", err)
|
||||
}
|
||||
if !bytesContains(buf.String(), `"event_id"`) {
|
||||
t.Errorf("json output missing event_id: %s", buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestDriftShowPeerFilter(t *testing.T) {
|
||||
setupDriftCLITest(t)
|
||||
fd := &fakeDetector{aggEvents: []drift.Event{
|
||||
{EventID: "E1", Host: "peer1", Path: "/a", Status: drift.StatusModified, DriftConfirmed: true},
|
||||
{EventID: "E2", Host: "peer2", Path: "/b", Status: drift.StatusModified, DriftConfirmed: true},
|
||||
}}
|
||||
driftDetectorOverride = fd
|
||||
defer func() { driftDetectorOverride = nil }()
|
||||
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"drift", "show", "--peer", "peer1"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("drift show --peer: %v", err)
|
||||
}
|
||||
out := buf.String()
|
||||
if !bytesContains(out, "E1") {
|
||||
t.Errorf("filtered output should have E1: %s", out)
|
||||
}
|
||||
if bytesContains(out, "E2") {
|
||||
t.Errorf("filtered output should NOT have E2: %s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDriftWatchStreamsAndCancels(t *testing.T) {
|
||||
setupDriftCLITest(t)
|
||||
// Use a fakeDetector whose Watch yields one event then blocks on
|
||||
// ctx so the stream terminates when ctrl-c (signal.NotifyContext)
|
||||
// cancels. We simulate the cancel by constructing a fake that yields
|
||||
// then returns when the consumer stops pulling OR ctx is cancelled.
|
||||
fd := &drainingFakeDetector{events: []drift.Event{
|
||||
{EventID: "E1", Host: "p", Path: "/etc/x", Status: drift.StatusModified, DriftConfirmed: true},
|
||||
}}
|
||||
driftDetectorOverride = fd
|
||||
defer func() { driftDetectorOverride = nil }()
|
||||
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"drift", "watch", "--interval", "10ms"})
|
||||
// Inject a context that auto-cancels after the events drain so
|
||||
// the watch loop exits without polluting rootCmd's context (which
|
||||
// is shared across tests). We use PersistentPreRunE's context by
|
||||
// overriding it here and restoring after.
|
||||
origCtx := rootCmd.Context()
|
||||
ctx, cancel := context.WithCancel(origCtx)
|
||||
defer cancel()
|
||||
rootCmd.SetContext(ctx)
|
||||
fd.cancelAfterYield = cancel
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("drift watch: %v", err)
|
||||
}
|
||||
// Restore rootCmd context for subsequent tests.
|
||||
rootCmd.SetContext(origCtx)
|
||||
if !bytesContains(buf.String(), "E1") {
|
||||
t.Errorf("watch output missing E1: %s", buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
// drainingFakeDetector yields the events then cancels the provided
|
||||
// cancel func (so the watch loop's signal.NotifyContext ctx is
|
||||
// cancelled and the stream terminates cleanly).
|
||||
type drainingFakeDetector struct {
|
||||
events []drift.Event
|
||||
cancelAfterYield context.CancelFunc
|
||||
remediateCalls []remediateCall
|
||||
ackWrites int
|
||||
}
|
||||
|
||||
func (d *drainingFakeDetector) Watch(ctx context.Context, paths []drift.PathSpec) iter.Seq2[drift.Event, error] {
|
||||
return func(yield func(drift.Event, error) bool) {
|
||||
for _, e := range d.events {
|
||||
if !yield(e, nil) {
|
||||
return
|
||||
}
|
||||
}
|
||||
if d.cancelAfterYield != nil {
|
||||
d.cancelAfterYield()
|
||||
}
|
||||
<-ctx.Done()
|
||||
}
|
||||
}
|
||||
func (d *drainingFakeDetector) Aggregate(ctx context.Context, leadPeer string) ([]drift.Event, error) {
|
||||
return d.events, nil
|
||||
}
|
||||
func (d *drainingFakeDetector) Remediate(ctx context.Context, leadPeer, path string, force bool) error {
|
||||
d.remediateCalls = append(d.remediateCalls, remediateCall{leadPeer, path, force})
|
||||
return nil
|
||||
}
|
||||
func (d *drainingFakeDetector) Acknowledge(ctx context.Context, leadPeer, path string) error {
|
||||
d.ackWrites++
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestDriftAcknowledge(t *testing.T) {
|
||||
setupDriftCLITest(t)
|
||||
fd := &fakeDetector{}
|
||||
driftDetectorOverride = fd
|
||||
defer func() { driftDetectorOverride = nil }()
|
||||
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"drift", "acknowledge", "peer1", "/etc/traefik/dynamic/orca.yml"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("drift acknowledge: %v", err)
|
||||
}
|
||||
if fd.ackWrites != 1 {
|
||||
t.Errorf("ackWrites = %d, want 1", fd.ackWrites)
|
||||
}
|
||||
if !bytesContains(buf.String(), "Acknowledged") {
|
||||
t.Errorf("output missing Acknowledged: %s", buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestDriftRemediate(t *testing.T) {
|
||||
setupDriftCLITest(t)
|
||||
fd := &fakeDetector{}
|
||||
driftDetectorOverride = fd
|
||||
defer func() { driftDetectorOverride = nil }()
|
||||
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"drift", "remediate", "peer1", "/etc/x"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("drift remediate: %v", err)
|
||||
}
|
||||
if len(fd.remediateCalls) != 1 {
|
||||
t.Fatalf("remediateCalls = %d, want 1", len(fd.remediateCalls))
|
||||
}
|
||||
if fd.remediateCalls[0].peer != "peer1" || fd.remediateCalls[0].path != "/etc/x" {
|
||||
t.Errorf("remediate call: %+v", fd.remediateCalls[0])
|
||||
}
|
||||
if fd.remediateCalls[0].force {
|
||||
t.Errorf("force should be false without --force")
|
||||
}
|
||||
if !bytesContains(buf.String(), "Remediated") {
|
||||
t.Errorf("output missing Remediated: %s", buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestDriftRemediateForce(t *testing.T) {
|
||||
setupDriftCLITest(t)
|
||||
fd := &fakeDetector{}
|
||||
driftDetectorOverride = fd
|
||||
defer func() { driftDetectorOverride = nil }()
|
||||
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"drift", "remediate", "peer1", "/etc/x", "--force"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("drift remediate --force: %v", err)
|
||||
}
|
||||
if len(fd.remediateCalls) != 1 {
|
||||
t.Fatalf("remediateCalls = %d, want 1", len(fd.remediateCalls))
|
||||
}
|
||||
if !fd.remediateCalls[0].force {
|
||||
t.Errorf("force should be true with --force")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDriftRemediateCooldown(t *testing.T) {
|
||||
setupDriftCLITest(t)
|
||||
fd := &fakeDetector{remediateErr: drift.ErrCooldown}
|
||||
driftDetectorOverride = fd
|
||||
defer func() { driftDetectorOverride = nil }()
|
||||
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"drift", "remediate", "peer1", "/etc/x"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("drift remediate cooldown: %v", err)
|
||||
}
|
||||
if !bytesContains(buf.String(), "cooldown") {
|
||||
t.Errorf("output should mention cooldown: %s", buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestDriftConfigShow(t *testing.T) {
|
||||
setupDriftCLITest(t)
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"drift", "config", "show"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("drift config show: %v", err)
|
||||
}
|
||||
out := buf.String()
|
||||
for _, want := range []string{"Polling:", "Critical paths:", "Standard paths:", "Excluded paths:", "Remediation:"} {
|
||||
if !bytesContains(out, want) {
|
||||
t.Errorf("config show missing %q: %s", want, out)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestDriftConfigValidate(t *testing.T) {
|
||||
setupDriftCLITest(t)
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"drift", "config", "validate"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("drift config validate: %v", err)
|
||||
}
|
||||
if !bytesContains(buf.String(), "valid") {
|
||||
t.Errorf("output missing 'valid': %s", buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestDriftConfigValidateFails(t *testing.T) {
|
||||
setupDriftCLITest(t)
|
||||
cfgPath := filepath.Join(t.TempDir(), "drift.json")
|
||||
bad := `{"polling":{"enabled":true,"default_interval":60000000000,"max_concurrent_peers":4},"paths":{"critical":[{"tier":"critical","pattern":"","interval":5000000000}]},"remediate":{"auto":true}}`
|
||||
if err := os.WriteFile(cfgPath, []byte(bad), 0o644); err != nil {
|
||||
t.Fatalf("write: %v", err)
|
||||
}
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"drift", "config", "validate", "--config", cfgPath})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected validate error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestJobRestartRequiresPeer(t *testing.T) {
|
||||
setupDriftCLITest(t)
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"job", "restart", "alloc1"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected error for missing --peer")
|
||||
}
|
||||
}
|
||||
|
||||
func TestJobRestartExecs(t *testing.T) {
|
||||
setupDriftCLITest(t)
|
||||
mt := &mockDriftTransport{execOut: []byte("restarted")}
|
||||
driftTransportOverride = mt
|
||||
defer func() { driftTransportOverride = nil }()
|
||||
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"job", "restart", "alloc1", "--peer", "peer1:22"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("job restart: %v", err)
|
||||
}
|
||||
if len(mt.execs) != 1 {
|
||||
t.Fatalf("execs = %d, want 1", len(mt.execs))
|
||||
}
|
||||
if !bytesContains(mt.execs[0], "orca-alloc-alloc1.service") {
|
||||
t.Errorf("restart cmd missing: %s", mt.execs[0])
|
||||
}
|
||||
}
|
||||
|
||||
func TestJobRestartTransientError(t *testing.T) {
|
||||
setupDriftCLITest(t)
|
||||
mt := &mockDriftTransport{execErr: fmt.Errorf("connection refused")}
|
||||
driftTransportOverride = mt
|
||||
defer func() { driftTransportOverride = nil }()
|
||||
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"job", "restart", "alloc1", "--peer", "peer1:22"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected error for exec failure")
|
||||
}
|
||||
if !errors.Is(err, err) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestPeerSetupCmdRegistered(t *testing.T) {
|
||||
for _, c := range rootCmd.Commands() {
|
||||
if c.Name() == "peer-setup" {
|
||||
return
|
||||
}
|
||||
}
|
||||
t.Fatal("peer-setup command not registered")
|
||||
}
|
||||
|
||||
func TestPeerSetupCreatesUserAndDir(t *testing.T) {
|
||||
setupDriftCLITest(t)
|
||||
mt := &mockDriftTransport{execOut: []byte("ext4")}
|
||||
peerSetupTransportOverride = mt
|
||||
defer func() { peerSetupTransportOverride = nil }()
|
||||
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"peer-setup", "peer1:22"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("peer-setup: %v", err)
|
||||
}
|
||||
if len(mt.execs) < 2 {
|
||||
t.Fatalf("execs = %d, want >= 2", len(mt.execs))
|
||||
}
|
||||
useraddSeen := false
|
||||
mkdirSeen := false
|
||||
statSeen := false
|
||||
for _, c := range mt.execs {
|
||||
if bytesContains(c, "useradd -r orca") {
|
||||
useraddSeen = true
|
||||
}
|
||||
if bytesContains(c, "mkdir -p /etc/orca/state/drift-events") {
|
||||
mkdirSeen = true
|
||||
}
|
||||
if bytesContains(c, "stat -f") {
|
||||
statSeen = true
|
||||
}
|
||||
}
|
||||
if !useraddSeen {
|
||||
t.Errorf("useradd not run")
|
||||
}
|
||||
if !mkdirSeen {
|
||||
t.Errorf("mkdir drift-events not run")
|
||||
}
|
||||
if !statSeen {
|
||||
t.Errorf("stat (NFS detect) not run")
|
||||
}
|
||||
if !bytesContains(buf.String(), "nfs=false") {
|
||||
t.Errorf("output should report nfs=false: %s", buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestPeerSetupNoOrcaUser(t *testing.T) {
|
||||
setupDriftCLITest(t)
|
||||
mt := &mockDriftTransport{execOut: []byte("ext4")}
|
||||
peerSetupTransportOverride = mt
|
||||
defer func() { peerSetupTransportOverride = nil }()
|
||||
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"peer-setup", "peer1:22", "--no-orca-user"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("peer-setup --no-orca-user: %v", err)
|
||||
}
|
||||
useraddSeen := false
|
||||
for _, c := range mt.execs {
|
||||
if bytesContains(c, "useradd -r orca") {
|
||||
useraddSeen = true
|
||||
}
|
||||
}
|
||||
if useraddSeen {
|
||||
t.Errorf("useradd should NOT run with --no-orca-user")
|
||||
}
|
||||
if !bytesContains(buf.String(), "user=false") {
|
||||
t.Errorf("output should report user=false: %s", buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestPeerSetupDetectsNFS(t *testing.T) {
|
||||
setupDriftCLITest(t)
|
||||
mt := &mockDriftTransport{execOut: []byte("nfs4")}
|
||||
peerSetupTransportOverride = mt
|
||||
defer func() { peerSetupTransportOverride = nil }()
|
||||
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"peer-setup", "peer1:22"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("peer-setup: %v", err)
|
||||
}
|
||||
if !bytesContains(buf.String(), "nfs=true") {
|
||||
t.Errorf("output should report nfs=true: %s", buf.String())
|
||||
}
|
||||
}
|
||||
+71
-15
@@ -7,6 +7,7 @@ import (
|
||||
"fmt"
|
||||
"os"
|
||||
"os/signal"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
@@ -48,6 +49,9 @@ var jobRunCmd = &cobra.Command{
|
||||
Long: "Submit a job spec, execute its tasks, and persist the result. Use --target to pin to a specific node (overrides bin-packing); --idempotency-key for cross-node dispatch dedupe.",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
if strings.HasSuffix(args[0], ".hcl") {
|
||||
warnDeprecated("orca job run <spec.hcl> is deprecated: .hcl jobspec is legacy (R-013); convert to .md format (REQ-064) — see .ciagent/PRD_v0.9.md")
|
||||
}
|
||||
spec, err := jobspec.ParseFile(args[0])
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -74,7 +78,7 @@ var jobRunCmd = &cobra.Command{
|
||||
peers := engine.NewPeerRegistry()
|
||||
dispatcher := engine.NewDispatcher(newLogger(), store.NewCapacityRepo(db), peers, exec)
|
||||
specBytes, _ := json.Marshal(map[string]any{
|
||||
"name": spec.Job.Name,
|
||||
"name": spec.Name,
|
||||
"command": "/bin/true", // placeholder; full HCL dispatch lands in a later phase
|
||||
})
|
||||
jobID, nodeID, err := dispatcher.Submit(ctx, runTarget, specBytes, runIDKey)
|
||||
@@ -93,11 +97,11 @@ var jobRunCmd = &cobra.Command{
|
||||
|
||||
job := &model.Job{
|
||||
ID: uuid.NewString(),
|
||||
Name: spec.Job.Name,
|
||||
Name: spec.Name,
|
||||
Spec: args[0],
|
||||
Status: model.JobStatusPending,
|
||||
}
|
||||
if err := exec.Run(ctx, job, toTaskSpecs(spec.Tasks)); err != nil {
|
||||
if err := exec.Run(ctx, job, workloadToTaskSpecs(spec)); err != nil {
|
||||
if jsonOutput {
|
||||
_ = printJSON(map[string]any{"id": job.ID, "status": "failed", "error": err.Error()})
|
||||
return err
|
||||
@@ -121,6 +125,13 @@ var jobListCmd = &cobra.Command{
|
||||
if jobWatch {
|
||||
return watchJobs(cmd)
|
||||
}
|
||||
|
||||
// Cache (R-008): read path only; --watch bypasses.
|
||||
var cachedJobs []*model.Job
|
||||
if cacheGetList(cacheJobClass, cacheListKey, &cachedJobs) {
|
||||
return renderJobs(cmd, cachedJobs)
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(cmd.Context(), 5*time.Second)
|
||||
defer cancel()
|
||||
|
||||
@@ -134,21 +145,27 @@ var jobListCmd = &cobra.Command{
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if jsonOutput {
|
||||
return printJSON(jobs)
|
||||
}
|
||||
if len(jobs) == 0 {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "No jobs. Use 'orca job run <spec.hcl>' to submit one.")
|
||||
return nil
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%-36s %-20s %-12s %-8s\n", "ID", "NAME", "STATUS", "EXIT")
|
||||
for _, j := range jobs {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%-36s %-20s %-12s %-8d\n", j.ID, j.Name, j.Status, j.ExitCode)
|
||||
}
|
||||
return nil
|
||||
cachePutList(cacheJobClass, cacheListKey, jobs, cacheJobTTL)
|
||||
return renderJobs(cmd, jobs)
|
||||
},
|
||||
}
|
||||
|
||||
// renderJobs prints the job list in either JSON or table form.
|
||||
func renderJobs(cmd *cobra.Command, jobs []*model.Job) error {
|
||||
if jsonOutput {
|
||||
return printJSON(jobs)
|
||||
}
|
||||
if len(jobs) == 0 {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "No jobs. Use 'orca job run <spec.hcl>' to submit one.")
|
||||
return nil
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%-36s %-20s %-12s %-8s\n", "ID", "NAME", "STATUS", "EXIT")
|
||||
for _, j := range jobs {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%-36s %-20s %-12s %-8d\n", j.ID, j.Name, j.Status, j.ExitCode)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func watchJobs(cmd *cobra.Command) error {
|
||||
ctx, cancel := signal.NotifyContext(cmd.Context(), os.Interrupt, syscall.SIGTERM)
|
||||
defer cancel()
|
||||
@@ -330,3 +347,42 @@ func toTaskSpecs(in []jobspec.TaskSpec) []engine.TaskSpec {
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// workloadToTaskSpecs converts a *WorkloadSpec into the engine.TaskSpec
|
||||
// slice consumed by the executor. For the HCL adapter path the runtime
|
||||
// block carries the legacy task[0].Command; for the Markdown path the
|
||||
// runtime block is the canonical runtime abstraction (P07 will expand
|
||||
// this). When Runtime is nil we emit a single no-op task to preserve
|
||||
// the legacy "at least one task" invariant.
|
||||
//
|
||||
// The runtime command string is split into binary + args via
|
||||
// splitCommand so that exec.Command receives the binary path and the
|
||||
// args as separate elements. Without this split, a command like
|
||||
// "/usr/bin/httpd -f /etc/orca/web-app/httpd.conf" is treated as a
|
||||
// single file path and fork/exec fails with "no such file or directory".
|
||||
func workloadToTaskSpecs(spec *jobspec.WorkloadSpec) []engine.TaskSpec {
|
||||
if spec == nil {
|
||||
return nil
|
||||
}
|
||||
if spec.Runtime == nil {
|
||||
return []engine.TaskSpec{{Name: spec.Name, Command: "/bin/true"}}
|
||||
}
|
||||
bin, args := splitCommand(spec.Runtime.Command)
|
||||
return []engine.TaskSpec{{
|
||||
Name: spec.Name,
|
||||
Command: bin,
|
||||
Args: args,
|
||||
}}
|
||||
}
|
||||
|
||||
// splitCommand splits a command string into binary + args using
|
||||
// strings.Fields (handles multiple spaces/tabs). If the string is empty
|
||||
// or all-whitespace, returns ("/bin/true", nil) so the executor still
|
||||
// has a valid binary to run.
|
||||
func splitCommand(s string) (string, []string) {
|
||||
parts := strings.Fields(s)
|
||||
if len(parts) == 0 {
|
||||
return "/bin/true", nil
|
||||
}
|
||||
return parts[0], parts[1:]
|
||||
}
|
||||
|
||||
@@ -0,0 +1,424 @@
|
||||
// Package cli: job_lint.go implements `orca job lint` (P11, REQ-084,
|
||||
// v0.11 milestone). It validates a jobspec (.md/.yaml/.yml/.hcl) with:
|
||||
//
|
||||
// - schema validation (kind, blocks, frontmatter fields) via
|
||||
// internal/spec/schema
|
||||
// - CEL constraint syntax check (basic -- balanced parens/quotes,
|
||||
// presence of operators; no CEL engine dependency in this phase)
|
||||
// - body preservation (R-015): warn when body is empty for .md specs
|
||||
// - migration: flag deprecated .hcl specs with a warning suggesting
|
||||
// conversion to .md (REQ-090)
|
||||
// - best-practice: warn on missing health checks for Services,
|
||||
// missing restart policies, etc.
|
||||
//
|
||||
// Output: one finding per line (category | severity | line | message).
|
||||
// --explain prints the rationale for each finding. --format text (the
|
||||
// default) or json. Exit 0 = no errors (warnings OK); 1 = errors found.
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/jobspec"
|
||||
"git.cloudinit.dev/coreci/orca/internal/spec/schema"
|
||||
)
|
||||
|
||||
var (
|
||||
jobLintExplain bool
|
||||
jobLintFormat string
|
||||
)
|
||||
|
||||
type lintSeverity string
|
||||
|
||||
const (
|
||||
severityError lintSeverity = "error"
|
||||
severityWarning lintSeverity = "warning"
|
||||
severityInfo lintSeverity = "info"
|
||||
)
|
||||
|
||||
type lintCategory string
|
||||
|
||||
const (
|
||||
catSchema lintCategory = "schema"
|
||||
catCEL lintCategory = "cel"
|
||||
catBody lintCategory = "body"
|
||||
catMigration lintCategory = "migration"
|
||||
catBestPractice lintCategory = "best-practice"
|
||||
)
|
||||
|
||||
type lintFinding struct {
|
||||
Category lintCategory `json:"category"`
|
||||
Severity lintSeverity `json:"severity"`
|
||||
Line int `json:"line"`
|
||||
Message string `json:"message"`
|
||||
}
|
||||
|
||||
var rationale = map[lintCategory]string{
|
||||
catSchema: "Schema violations prevent the spec from being parsed or rendered; fix these first.",
|
||||
catCEL: "CEL constraints gate placement; a syntactically invalid expression is rejected by the scheduler (REQ-083).",
|
||||
catBody: "R-015 requires the markdown body to be preserved byte-exact; an empty body is allowed but loses operator documentation.",
|
||||
catMigration: "HCL specs are legacy (REQ-090); convert to Markdown+frontmatter before v1.0 to keep schema validation working.",
|
||||
catBestPractice: "Best-practice warnings do not block apply, but address them to keep the fleet observable and restartable.",
|
||||
}
|
||||
|
||||
type lintExitError struct {
|
||||
findings []lintFinding
|
||||
}
|
||||
|
||||
func (e *lintExitError) Error() string {
|
||||
return fmt.Sprintf("lint: %d error(s) found", countErrors(e.findings))
|
||||
}
|
||||
|
||||
func countErrors(findings []lintFinding) int {
|
||||
n := 0
|
||||
for _, f := range findings {
|
||||
if f.Severity == severityError {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
var jobLintCmd = &cobra.Command{
|
||||
Use: "lint <spec>",
|
||||
Short: "Lint a jobspec (schema, CEL, body, migration, best-practice)",
|
||||
Long: `Validate a jobspec (.md/.yaml/.yml/.hcl) without applying it.
|
||||
|
||||
Checks (REQ-084):
|
||||
- schema: kind (Job/Service/DaemonSet), required blocks, frontmatter
|
||||
- CEL: constraint expressions are syntactically valid
|
||||
- body: markdown body present and non-empty (R-015)
|
||||
- migration: flag deprecated .hcl specs (suggest .md)
|
||||
- best-practice: warn on missing health/restart/update for the kind
|
||||
|
||||
Flags:
|
||||
--explain print the rationale for each finding
|
||||
--format text|json output format (default text)
|
||||
|
||||
Exit codes: 0 = no errors (warnings OK), 1 = errors found.`,
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
findings, err := runJobLint(args[0])
|
||||
if err != nil {
|
||||
var le *lintExitError
|
||||
if errors.As(err, &le) {
|
||||
if jobLintFormat == "json" {
|
||||
_ = printLintJSON(findings)
|
||||
} else {
|
||||
printLintText(findings, jobLintExplain)
|
||||
}
|
||||
return err
|
||||
}
|
||||
return err
|
||||
}
|
||||
if jobLintFormat == "json" {
|
||||
return printLintJSON(findings)
|
||||
}
|
||||
printLintText(findings, jobLintExplain)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
func runJobLint(path string) ([]lintFinding, error) {
|
||||
var findings []lintFinding
|
||||
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
findings = append(findings, lintFinding{
|
||||
Category: catSchema,
|
||||
Severity: severityError,
|
||||
Line: 0,
|
||||
Message: fmt.Sprintf("cannot read spec file: %v", err),
|
||||
})
|
||||
return findings, &lintExitError{findings}
|
||||
}
|
||||
|
||||
ext := strings.ToLower(filepath.Ext(path))
|
||||
if ext == ".hcl" {
|
||||
findings = append(findings, lintFinding{
|
||||
Category: catMigration,
|
||||
Severity: severityWarning,
|
||||
Line: 0,
|
||||
Message: fmt.Sprintf("%s is a legacy HCL spec; convert to Markdown (.md) before v1.0 (REQ-090)", filepath.Base(path)),
|
||||
})
|
||||
}
|
||||
|
||||
spec, perr := jobspec.Dispatch(data, filepath.Base(path))
|
||||
if perr != nil {
|
||||
findings = append(findings, lintFinding{
|
||||
Category: catSchema,
|
||||
Severity: severityError,
|
||||
Line: 0,
|
||||
Message: fmt.Sprintf("parse: %v", perr),
|
||||
})
|
||||
return findings, &lintExitError{findings}
|
||||
}
|
||||
|
||||
findings = append(findings, lintSchema(spec)...)
|
||||
findings = append(findings, lintCEL(spec)...)
|
||||
findings = append(findings, lintBody(spec, ext)...)
|
||||
findings = append(findings, lintBestPractice(spec)...)
|
||||
|
||||
sortLint(findings)
|
||||
if countErrors(findings) > 0 {
|
||||
return findings, &lintExitError{findings}
|
||||
}
|
||||
return findings, nil
|
||||
}
|
||||
|
||||
func lintSchema(spec *jobspec.WorkloadSpec) []lintFinding {
|
||||
if spec == nil {
|
||||
return nil
|
||||
}
|
||||
v, err := schema.ValidatorFor(spec.Kind)
|
||||
if err != nil {
|
||||
return []lintFinding{{
|
||||
Category: catSchema,
|
||||
Severity: severityError,
|
||||
Line: 0,
|
||||
Message: err.Error(),
|
||||
}}
|
||||
}
|
||||
verr := v.Validate(spec)
|
||||
if verr == nil {
|
||||
return nil
|
||||
}
|
||||
msg := verr.Error()
|
||||
parts := strings.Split(msg, "; ")
|
||||
var out []lintFinding
|
||||
for _, p := range parts {
|
||||
p = strings.TrimSpace(p)
|
||||
if p == "" {
|
||||
continue
|
||||
}
|
||||
out = append(out, lintFinding{
|
||||
Category: catSchema,
|
||||
Severity: severityError,
|
||||
Line: 0,
|
||||
Message: trimValidatorPrefix(p),
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func trimValidatorPrefix(s string) string {
|
||||
if strings.HasPrefix(s, "schema/") {
|
||||
if i := strings.Index(s, ": "); i >= 0 {
|
||||
return strings.TrimSpace(s[i+2:])
|
||||
}
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func lintCEL(spec *jobspec.WorkloadSpec) []lintFinding {
|
||||
if spec == nil {
|
||||
return nil
|
||||
}
|
||||
var out []lintFinding
|
||||
for i, c := range spec.Constraints {
|
||||
if msg := basicCELCheck(c); msg != "" {
|
||||
out = append(out, lintFinding{
|
||||
Category: catCEL,
|
||||
Severity: severityError,
|
||||
Line: 0,
|
||||
Message: fmt.Sprintf("constraints[%d]: %s", i, msg),
|
||||
})
|
||||
}
|
||||
}
|
||||
for i, a := range spec.Affinity {
|
||||
if msg := basicCELCheck(a.Target); msg != "" {
|
||||
out = append(out, lintFinding{
|
||||
Category: catCEL,
|
||||
Severity: severityError,
|
||||
Line: 0,
|
||||
Message: fmt.Sprintf("affinity[%d].target: %s", i, msg),
|
||||
})
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func basicCELCheck(expr string) string {
|
||||
expr = strings.TrimSpace(expr)
|
||||
if expr == "" {
|
||||
return "empty CEL expression"
|
||||
}
|
||||
parens := 0
|
||||
inSingle := false
|
||||
inDouble := false
|
||||
for i := 0; i < len(expr); i++ {
|
||||
c := expr[i]
|
||||
switch c {
|
||||
case '\'':
|
||||
if !inDouble {
|
||||
inSingle = !inSingle
|
||||
}
|
||||
case '"':
|
||||
if !inSingle {
|
||||
inDouble = !inDouble
|
||||
}
|
||||
case '(':
|
||||
if !inSingle && !inDouble {
|
||||
parens++
|
||||
}
|
||||
case ')':
|
||||
if !inSingle && !inDouble {
|
||||
parens--
|
||||
if parens < 0 {
|
||||
return "unbalanced parentheses: ')' before '('"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if inSingle || inDouble {
|
||||
return "unbalanced quotes"
|
||||
}
|
||||
if parens != 0 {
|
||||
return fmt.Sprintf("unbalanced parentheses: %d unclosed '('", parens)
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func lintBody(spec *jobspec.WorkloadSpec, ext string) []lintFinding {
|
||||
if spec == nil {
|
||||
return nil
|
||||
}
|
||||
if ext != ".md" {
|
||||
return nil
|
||||
}
|
||||
if strings.TrimSpace(spec.Body) == "" {
|
||||
return []lintFinding{{
|
||||
Category: catBody,
|
||||
Severity: severityWarning,
|
||||
Line: 0,
|
||||
Message: "markdown body is empty (R-015: body preserved byte-exact; add operator documentation)",
|
||||
}}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func lintBestPractice(spec *jobspec.WorkloadSpec) []lintFinding {
|
||||
if spec == nil {
|
||||
return nil
|
||||
}
|
||||
var out []lintFinding
|
||||
switch spec.Kind {
|
||||
case "Service":
|
||||
if spec.Health == nil {
|
||||
out = append(out, lintFinding{
|
||||
Category: catBestPractice,
|
||||
Severity: severityWarning,
|
||||
Line: 0,
|
||||
Message: "Service without a health block: Traefik routing depends on health checks (R-012)",
|
||||
})
|
||||
}
|
||||
if spec.Restart == nil {
|
||||
out = append(out, lintFinding{
|
||||
Category: catBestPractice,
|
||||
Severity: severityWarning,
|
||||
Line: 0,
|
||||
Message: "Service without a restart policy: defaults to 'service' but an explicit policy is recommended",
|
||||
})
|
||||
}
|
||||
if spec.Update == nil {
|
||||
out = append(out, lintFinding{
|
||||
Category: catBestPractice,
|
||||
Severity: severityWarning,
|
||||
Line: 0,
|
||||
Message: "Service without an update block: rolling/canary strategy should be explicit",
|
||||
})
|
||||
}
|
||||
case "DaemonSet":
|
||||
if spec.Restart == nil {
|
||||
out = append(out, lintFinding{
|
||||
Category: catBestPractice,
|
||||
Severity: severityWarning,
|
||||
Line: 0,
|
||||
Message: "DaemonSet without a restart policy: a long-running daemon should declare its restart mode",
|
||||
})
|
||||
}
|
||||
case "Job":
|
||||
if spec.Restart == nil {
|
||||
out = append(out, lintFinding{
|
||||
Category: catBestPractice,
|
||||
Severity: severityInfo,
|
||||
Line: 0,
|
||||
Message: "Job without a restart policy: defaults to 'never' (one-shot); set explicitly if retry is desired",
|
||||
})
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func sortLint(f []lintFinding) {
|
||||
sort.SliceStable(f, func(i, j int) bool {
|
||||
si := severityRank(f[i].Severity)
|
||||
sj := severityRank(f[j].Severity)
|
||||
if si != sj {
|
||||
return si < sj
|
||||
}
|
||||
if f[i].Category != f[j].Category {
|
||||
return string(f[i].Category) < string(f[j].Category)
|
||||
}
|
||||
return f[i].Line < f[j].Line
|
||||
})
|
||||
}
|
||||
|
||||
func severityRank(s lintSeverity) int {
|
||||
switch s {
|
||||
case severityError:
|
||||
return 0
|
||||
case severityWarning:
|
||||
return 1
|
||||
case severityInfo:
|
||||
return 2
|
||||
}
|
||||
return 3
|
||||
}
|
||||
|
||||
func printLintText(findings []lintFinding, explain bool) {
|
||||
w := rootCmd.OutOrStdout()
|
||||
errs := countErrors(findings)
|
||||
warns := 0
|
||||
infos := 0
|
||||
for _, f := range findings {
|
||||
switch f.Severity {
|
||||
case severityWarning:
|
||||
warns++
|
||||
case severityInfo:
|
||||
infos++
|
||||
}
|
||||
line := fmt.Sprintf("%-14s %-8s %s", f.Category, f.Severity, f.Message)
|
||||
if f.Line > 0 {
|
||||
line = fmt.Sprintf("%-14s %-8s line %d: %s", f.Category, f.Severity, f.Line, f.Message)
|
||||
}
|
||||
fmt.Fprintln(w, line)
|
||||
if explain {
|
||||
fmt.Fprintf(w, " -> %s\n", rationale[f.Category])
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(w, "\n%d error(s), %d warning(s), %d info\n", errs, warns, infos)
|
||||
}
|
||||
|
||||
func printLintJSON(findings []lintFinding) error {
|
||||
out := make([]lintFinding, len(findings))
|
||||
copy(out, findings)
|
||||
w := rootCmd.OutOrStdout()
|
||||
enc := json.NewEncoder(w)
|
||||
enc.SetIndent("", " ")
|
||||
return enc.Encode(out)
|
||||
}
|
||||
|
||||
func init() {
|
||||
jobLintCmd.Flags().BoolVar(&jobLintExplain, "explain", false, "print the rationale for each finding")
|
||||
jobLintCmd.Flags().StringVar(&jobLintFormat, "format", "text", "output format: text or json")
|
||||
jobCmd.AddCommand(jobLintCmd)
|
||||
}
|
||||
@@ -0,0 +1,310 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func writeMDSpec(t *testing.T, content string) string {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
p := filepath.Join(dir, "spec.md")
|
||||
if err := os.WriteFile(p, []byte(content), 0o644); err != nil {
|
||||
t.Fatalf("write spec: %v", err)
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
func writeHCLSpec(t *testing.T, content string) string {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
p := filepath.Join(dir, "spec.hcl")
|
||||
if err := os.WriteFile(p, []byte(content), 0o644); err != nil {
|
||||
t.Fatalf("write spec: %v", err)
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
const validJobMD = "---\n" +
|
||||
"kind: Job\n" +
|
||||
"name: my-job\n" +
|
||||
"runtime:\n" +
|
||||
" one_of: process\n" +
|
||||
" command: /bin/true\n" +
|
||||
"---\n" +
|
||||
"# My Job\n\nRuns /bin/true.\n"
|
||||
|
||||
const validServiceMD = "---\n" +
|
||||
"kind: Service\n" +
|
||||
"name: web\n" +
|
||||
"count: 3\n" +
|
||||
"runtime:\n" +
|
||||
" one_of: process\n" +
|
||||
" command: /bin/http\n" +
|
||||
"ports:\n" +
|
||||
" - name: http\n" +
|
||||
" port: 8080\n" +
|
||||
"restart:\n" +
|
||||
" mode: service\n" +
|
||||
"update:\n" +
|
||||
" strategy: rolling\n" +
|
||||
" max_surge: 1\n" +
|
||||
"health:\n" +
|
||||
" check_type: http\n" +
|
||||
" interval: 5s\n" +
|
||||
"---\n" +
|
||||
"# Web service\n\nServes HTTP.\n"
|
||||
|
||||
const invalidKindMD = "---\n" +
|
||||
"kind: CronJob\n" +
|
||||
"name: bad\n" +
|
||||
"---\nbody\n"
|
||||
|
||||
const serviceMissingHealthMD = "---\n" +
|
||||
"kind: Service\n" +
|
||||
"name: web\n" +
|
||||
"count: 2\n" +
|
||||
"runtime:\n" +
|
||||
" one_of: process\n" +
|
||||
" command: /bin/http\n" +
|
||||
"ports:\n" +
|
||||
" - name: http\n" +
|
||||
" port: 8080\n" +
|
||||
"restart:\n" +
|
||||
" mode: service\n" +
|
||||
"update:\n" +
|
||||
" strategy: rolling\n" +
|
||||
"---\n" +
|
||||
"# web\n\nbody\n"
|
||||
|
||||
const serviceMissingPortsMD = "---\n" +
|
||||
"kind: Service\n" +
|
||||
"name: web\n" +
|
||||
"runtime:\n" +
|
||||
" one_of: process\n" +
|
||||
" command: /bin/http\n" +
|
||||
"restart:\n" +
|
||||
" mode: service\n" +
|
||||
"update:\n" +
|
||||
" strategy: rolling\n" +
|
||||
"health:\n" +
|
||||
" check_type: http\n" +
|
||||
"---\nbody\n"
|
||||
|
||||
const emptyBodyMD = "---\n" +
|
||||
"kind: Job\n" +
|
||||
"name: emptybody\n" +
|
||||
"runtime:\n" +
|
||||
" command: /bin/true\n" +
|
||||
"---\n"
|
||||
|
||||
const badCELMD = "---\n" +
|
||||
"kind: Job\n" +
|
||||
"name: badcel\n" +
|
||||
"runtime:\n" +
|
||||
" command: /bin/true\n" +
|
||||
"constraints:\n" +
|
||||
" - 'node.role == \"web\"'\n" +
|
||||
" - 'region == (\"us\"'\n" +
|
||||
"---\nbody\n"
|
||||
|
||||
func TestJobLintCmdRegistered(t *testing.T) {
|
||||
for _, c := range jobCmd.Commands() {
|
||||
if c.Name() == "lint" {
|
||||
return
|
||||
}
|
||||
}
|
||||
t.Fatal("job lint command not registered on jobCmd")
|
||||
}
|
||||
|
||||
func TestJobLintValidSpec(t *testing.T) {
|
||||
resetRootFlags(t)
|
||||
spec := writeMDSpec(t, validJobMD)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"job", "lint", spec})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("job lint valid: %v", err)
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "0 error(s)") {
|
||||
t.Errorf("expected 0 errors, got: %s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestJobLintValidServiceSpec(t *testing.T) {
|
||||
resetRootFlags(t)
|
||||
spec := writeMDSpec(t, validServiceMD)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"job", "lint", spec})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("job lint valid service: %v", err)
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "0 error(s)") {
|
||||
t.Errorf("expected 0 errors, got: %s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestJobLintInvalidKind(t *testing.T) {
|
||||
resetRootFlags(t)
|
||||
spec := writeMDSpec(t, invalidKindMD)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"job", "lint", spec})
|
||||
if err := rootCmd.Execute(); err == nil {
|
||||
t.Fatal("expected error for invalid kind, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestJobLintInvalidKindReportsError(t *testing.T) {
|
||||
resetRootFlags(t)
|
||||
spec := writeMDSpec(t, invalidKindMD)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"job", "lint", spec})
|
||||
_ = rootCmd.Execute()
|
||||
out := buf.String()
|
||||
if !strings.Contains(strings.ToLower(out), "cronjob") && !strings.Contains(strings.ToLower(out), "kind") {
|
||||
t.Errorf("expected error about kind, got: %s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestJobLintMissingRequiredField(t *testing.T) {
|
||||
resetRootFlags(t)
|
||||
spec := writeMDSpec(t, serviceMissingPortsMD)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"job", "lint", spec})
|
||||
if err := rootCmd.Execute(); err == nil {
|
||||
t.Fatal("expected error for missing ports, got nil")
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(strings.ToLower(out), "port") {
|
||||
t.Errorf("expected error mentioning ports, got: %s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestJobLintDeprecatedHCLWarning(t *testing.T) {
|
||||
resetRootFlags(t)
|
||||
spec := writeHCLSpec(t, trueJobSpec)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"job", "lint", spec})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("job lint hcl: %v", err)
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "migration") && !strings.Contains(strings.ToLower(out), "legacy") && !strings.Contains(strings.ToLower(out), "hcl") {
|
||||
t.Errorf("expected migration/legacy warning, got: %s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestJobLintExplain(t *testing.T) {
|
||||
resetRootFlags(t)
|
||||
spec := writeMDSpec(t, serviceMissingHealthMD)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"job", "lint", spec, "--explain"})
|
||||
_ = rootCmd.Execute()
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "->") {
|
||||
t.Errorf("expected rationale lines with '->', got: %s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestJobLintFormatJSON(t *testing.T) {
|
||||
resetRootFlags(t)
|
||||
spec := writeMDSpec(t, serviceMissingHealthMD)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"job", "lint", spec, "--format", "json"})
|
||||
_ = rootCmd.Execute()
|
||||
out := buf.String()
|
||||
var findings []lintFinding
|
||||
if err := json.Unmarshal(bytes.TrimSpace([]byte(out)), &findings); err != nil {
|
||||
t.Fatalf("unmarshal json findings: %v\n%s", err, out)
|
||||
}
|
||||
if len(findings) == 0 {
|
||||
t.Fatalf("expected at least one finding")
|
||||
}
|
||||
foundHealth := false
|
||||
for _, f := range findings {
|
||||
if strings.Contains(strings.ToLower(f.Message), "health") {
|
||||
foundHealth = true
|
||||
}
|
||||
}
|
||||
if !foundHealth {
|
||||
t.Errorf("expected a health-related finding, got: %+v", findings)
|
||||
}
|
||||
}
|
||||
|
||||
func TestJobLintMissingHealthCheckWarning(t *testing.T) {
|
||||
resetRootFlags(t)
|
||||
spec := writeMDSpec(t, serviceMissingHealthMD)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"job", "lint", spec})
|
||||
_ = rootCmd.Execute()
|
||||
out := buf.String()
|
||||
if !strings.Contains(strings.ToLower(out), "health") {
|
||||
t.Errorf("expected health-related warning, got: %s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestJobLintEmptyBodyWarning(t *testing.T) {
|
||||
resetRootFlags(t)
|
||||
spec := writeMDSpec(t, emptyBodyMD)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"job", "lint", spec})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("job lint empty body (warnings only): %v", err)
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(strings.ToLower(out), "body") {
|
||||
t.Errorf("expected body warning, got: %s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestJobLintBadCEL(t *testing.T) {
|
||||
resetRootFlags(t)
|
||||
spec := writeMDSpec(t, badCELMD)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"job", "lint", spec})
|
||||
if err := rootCmd.Execute(); err == nil {
|
||||
t.Fatal("expected error for unbalanced CEL, got nil")
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(strings.ToLower(out), "cel") && !strings.Contains(strings.ToLower(out), "parenthes") {
|
||||
t.Errorf("expected CEL/parentheses error, got: %s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestJobLintMissingFile(t *testing.T) {
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"job", "lint", "/nonexistent/spec.md"})
|
||||
if err := rootCmd.Execute(); err == nil {
|
||||
t.Fatal("expected error for missing file, got nil")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,396 @@
|
||||
// Package cli: job_verify.go implements `orca job verify` (P12,
|
||||
// v0.11 milestone). It performs a dry-run transaction through the lead:
|
||||
// parse the jobspec, render the emitter plan (allocs/files/units),
|
||||
// render a txn bundle with the desired state, stage it on the lead
|
||||
// (idempotent; NO apply), run verify.sh on the lead to capture what
|
||||
// WOULD be applied, and report the plan. Pre-flight drift (R-020) is
|
||||
// reported but does not fail a dry-run.
|
||||
//
|
||||
// There are no side effects beyond the staged bundle files in
|
||||
// /run/orca/txns/<txn-id>/ on the lead (idempotent; no .applied marker
|
||||
// is written, so orca-pull.sh never picks it up).
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/emitter"
|
||||
"git.cloudinit.dev/coreci/orca/internal/jobspec"
|
||||
"git.cloudinit.dev/coreci/orca/internal/paths"
|
||||
"git.cloudinit.dev/coreci/orca/internal/secrets"
|
||||
"git.cloudinit.dev/coreci/orca/internal/spec/schema"
|
||||
"git.cloudinit.dev/coreci/orca/internal/txn"
|
||||
)
|
||||
|
||||
var (
|
||||
jobVerifyLead string
|
||||
jobVerifyNamespace string
|
||||
jobVerifyJSON bool
|
||||
)
|
||||
|
||||
// jobVerifyTransport is the SSH-push surface `job verify` needs. It
|
||||
// is satisfied by *sshpush.Transport; tests substitute a mock (same
|
||||
// pattern as txn.go / drain.go).
|
||||
type jobVerifyTransport interface {
|
||||
WriteFileIdempotent(ctx context.Context, peer string, path string, content []byte, mode os.FileMode) (bool, error)
|
||||
Exec(ctx context.Context, peer string, cmd string) ([]byte, error)
|
||||
}
|
||||
|
||||
// jobVerifyTransportOverride is the package-level seam. When non-nil
|
||||
// it replaces the production transport; tests set it and restore nil.
|
||||
var jobVerifyTransportOverride jobVerifyTransport
|
||||
|
||||
func jobVerifyTransportFromCtx() (jobVerifyTransport, error) {
|
||||
if jobVerifyTransportOverride != nil {
|
||||
return jobVerifyTransportOverride, nil
|
||||
}
|
||||
return txnTransportFromCtx()
|
||||
}
|
||||
|
||||
// verifyReport is the structured result of `orca job verify`. It is
|
||||
// rendered to JSON when --json is set, or as a human-readable summary
|
||||
// otherwise.
|
||||
type verifyReport struct {
|
||||
TxnID string `json:"txn_id"`
|
||||
Kind string `json:"kind"`
|
||||
Name string `json:"name"`
|
||||
Namespace string `json:"namespace,omitempty"`
|
||||
Lead string `json:"lead"`
|
||||
PlannedAllocs []plannedAlloc `json:"planned_allocs"`
|
||||
PlannedFiles []plannedFile `json:"planned_files"`
|
||||
VerifyOutput string `json:"verify_output,omitempty"`
|
||||
Drift []string `json:"drift,omitempty"`
|
||||
Status string `json:"status"`
|
||||
}
|
||||
|
||||
type plannedAlloc struct {
|
||||
Name string `json:"name"`
|
||||
Kind string `json:"kind"`
|
||||
Count int `json:"count"`
|
||||
}
|
||||
|
||||
type plannedFile struct {
|
||||
Path string `json:"path"`
|
||||
Mode string `json:"mode"`
|
||||
Kind string `json:"kind"`
|
||||
}
|
||||
|
||||
var jobVerifyCmd = &cobra.Command{
|
||||
Use: "verify <spec>",
|
||||
Short: "Dry-run a jobspec through the lead (no apply)",
|
||||
Long: `Dry-run a jobspec as a transaction through the lead peer.
|
||||
|
||||
Steps (P12):
|
||||
1. Parse the jobspec
|
||||
2. Render the emitter plan (allocs, config files, systemd units)
|
||||
3. Render a txn bundle (RenderBundle) with the desired state
|
||||
4. Stage the bundle on the lead (NO apply; idempotent)
|
||||
5. Run verify.sh on the lead to capture what WOULD be applied
|
||||
6. Report planned allocs / files / units
|
||||
|
||||
Pre-flight drift (R-020) is reported but does not fail a dry-run.
|
||||
There are no side effects beyond the staged bundle files in
|
||||
/run/orca/txns/<txn-id>/ on the lead (no .applied marker is written).
|
||||
|
||||
Flags:
|
||||
--lead <peer> lead peer address (host:port) (required)
|
||||
--namespace <ns> namespace scope
|
||||
--json JSON output
|
||||
|
||||
Exit codes: 0 = verify passed (no issues), 1 = verification failed.`,
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
report, err := runJobVerify(cmd.Context(), args[0])
|
||||
if err != nil {
|
||||
if jobVerifyJSON {
|
||||
if report != nil {
|
||||
_ = printVerifyJSON(report)
|
||||
}
|
||||
} else if report != nil {
|
||||
printVerifyText(report)
|
||||
}
|
||||
return err
|
||||
}
|
||||
if jobVerifyJSON {
|
||||
return printVerifyJSON(report)
|
||||
}
|
||||
printVerifyText(report)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
// runJobVerify performs the dry-run. It returns the report and an
|
||||
// error: when err is non-nil the report may still be populated with
|
||||
// partial results (e.g. drift was detected but the verify step ran).
|
||||
// The caller renders the report then surfaces the error.
|
||||
func runJobVerify(ctx context.Context, path string) (*verifyReport, error) {
|
||||
if jobVerifyLead == "" {
|
||||
return nil, fmt.Errorf("--lead is required for job verify")
|
||||
}
|
||||
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read spec file: %w", err)
|
||||
}
|
||||
spec, err := jobspec.Dispatch(data, fileBase(path))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("parse spec: %w", err)
|
||||
}
|
||||
|
||||
if verr := validateForVerify(spec); verr != nil {
|
||||
return nil, verr
|
||||
}
|
||||
|
||||
files, err := renderPlan(spec)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("render plan: %w", err)
|
||||
}
|
||||
|
||||
mk, err := secrets.LoadMasterKey(paths.MasterKeyPath())
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("load master key: %w", err)
|
||||
}
|
||||
|
||||
desired := buildDesiredState(spec, files)
|
||||
bundle, err := txn.RenderBundle(desired, mk)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("render bundle: %w", err)
|
||||
}
|
||||
|
||||
transport, err := jobVerifyTransportFromCtx()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("ssh transport: %w", err)
|
||||
}
|
||||
|
||||
if err := txn.Stage(bundle, jobVerifyLead, asTxnTransport(transport)); err != nil {
|
||||
return nil, fmt.Errorf("stage bundle: %w", err)
|
||||
}
|
||||
slog.Info("job verify: bundle staged (no apply)", "txn_id", bundle.ID, "lead", jobVerifyLead)
|
||||
|
||||
verifyOut, verifyErr := runVerifySh(ctx, transport, bundle.ID, jobVerifyLead, jobVerifyNamespace)
|
||||
drift := parseDriftLines(string(verifyOut))
|
||||
|
||||
report := &verifyReport{
|
||||
TxnID: string(bundle.ID),
|
||||
Kind: spec.Kind,
|
||||
Name: spec.Name,
|
||||
Namespace: jobVerifyNamespace,
|
||||
Lead: jobVerifyLead,
|
||||
PlannedAllocs: buildPlannedAllocs(spec),
|
||||
PlannedFiles: buildPlannedFiles(files),
|
||||
VerifyOutput: string(verifyOut),
|
||||
Drift: drift,
|
||||
Status: "verified",
|
||||
}
|
||||
|
||||
if verifyErr != nil {
|
||||
report.Status = "verify-failed"
|
||||
return report, fmt.Errorf("verify.sh on %s: %w (output: %s)", jobVerifyLead, verifyErr, string(verifyOut))
|
||||
}
|
||||
return report, nil
|
||||
}
|
||||
|
||||
// validateForVerify runs the schema validator (the dry-run should fail
|
||||
// fast on an invalid spec, same as `orca job lint` errors).
|
||||
func validateForVerify(spec *jobspec.WorkloadSpec) error {
|
||||
if spec == nil {
|
||||
return fmt.Errorf("spec is nil")
|
||||
}
|
||||
v, err := schema.ValidatorFor(spec.Kind)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if verr := v.Validate(spec); verr != nil {
|
||||
return fmt.Errorf("schema validation: %w", verr)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// renderPlan renders the spec into the emitter File artifacts that
|
||||
// represent what WOULD be written on apply. The registry is populated
|
||||
// with the SystemdEmitter for the "process" runtime (the only runtime
|
||||
// P0c ships); other runtimes return an error so the dry-run reports
|
||||
// the gap instead of pretending success.
|
||||
func renderPlan(spec *jobspec.WorkloadSpec) ([]emitter.File, error) {
|
||||
if spec.Runtime == nil {
|
||||
return nil, fmt.Errorf("spec runtime is nil (verify needs a runtime to render)")
|
||||
}
|
||||
reg := emitter.NewRegistry()
|
||||
reg.Register("job:process", emitter.SystemdEmitter{})
|
||||
reg.Register("service:process", emitter.SystemdEmitter{})
|
||||
reg.Register("daemonset:process", emitter.SystemdEmitter{})
|
||||
node := &emitter.Node{Hostname: jobVerifyLead}
|
||||
return reg.Render(spec, node)
|
||||
}
|
||||
|
||||
// buildDesiredState assembles the desired-state object the txn apply
|
||||
// script consumes. It is a list of artifact dicts (path/content/mode)
|
||||
// derived from the emitter plan, plus metadata so verify.sh and
|
||||
// orca-pull.sh can report what would change.
|
||||
func buildDesiredState(spec *jobspec.WorkloadSpec, files []emitter.File) any {
|
||||
type artifact struct {
|
||||
Path string `json:"path"`
|
||||
Content string `json:"content"`
|
||||
Mode string `json:"mode"`
|
||||
}
|
||||
arts := make([]artifact, len(files))
|
||||
for i, f := range files {
|
||||
arts[i] = artifact{Path: f.Path, Content: f.Content, Mode: f.Mode}
|
||||
}
|
||||
return map[string]any{
|
||||
"kind": spec.Kind,
|
||||
"name": spec.Name,
|
||||
"namespace": jobVerifyNamespace,
|
||||
"artifacts": arts,
|
||||
}
|
||||
}
|
||||
|
||||
// buildPlannedAllocs reports the alloc(s) the apply would create. For
|
||||
// the single-process path it is one alloc named after the spec; for a
|
||||
// task group it is one alloc with N task units. Count > 1 (Service)
|
||||
// expands to N allocs.
|
||||
func buildPlannedAllocs(spec *jobspec.WorkloadSpec) []plannedAlloc {
|
||||
count := spec.Count
|
||||
if count < 1 {
|
||||
count = 1
|
||||
}
|
||||
out := make([]plannedAlloc, 0, count)
|
||||
for i := 0; i < count; i++ {
|
||||
name := spec.Name
|
||||
if count > 1 {
|
||||
name = fmt.Sprintf("%s-%d", spec.Name, i)
|
||||
}
|
||||
out = append(out, plannedAlloc{Name: name, Kind: spec.Kind, Count: 1})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// buildPlannedFiles classifies the emitter artifacts into config files
|
||||
// and systemd units by path. Anything under /etc/systemd/system/ is a
|
||||
// unit; everything else is a config file.
|
||||
func buildPlannedFiles(files []emitter.File) []plannedFile {
|
||||
out := make([]plannedFile, 0, len(files))
|
||||
for _, f := range files {
|
||||
kind := "config"
|
||||
if strings.HasPrefix(f.Path, "/etc/systemd/system/") {
|
||||
kind = "unit"
|
||||
}
|
||||
out = append(out, plannedFile{Path: f.Path, Mode: f.Mode, Kind: kind})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// runVerifySh runs verify.sh on the lead for the staged bundle. The
|
||||
// verify script reports missing files (the ones that WOULD be written
|
||||
// on apply). A non-zero exit is expected for a dry-run (the files are
|
||||
// not applied yet), so the caller treats the output as informational.
|
||||
func runVerifySh(ctx context.Context, transport jobVerifyTransport, id txn.TxnID, lead, namespace string) ([]byte, error) {
|
||||
dir := "/run/orca/txns/" + string(id)
|
||||
cmd := fmt.Sprintf("bash %s/verify.sh", dir)
|
||||
out, err := transport.Exec(ctx, lead, cmd)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// parseDriftLines extracts pre-flight drift (R-020) notices from the
|
||||
// verify.sh output. The verify script emits "verify: drift <detail>"
|
||||
// lines when the on-disk state has drifted from a prior apply; in a
|
||||
// dry-run these are reported but do not fail.
|
||||
func parseDriftLines(out string) []string {
|
||||
var drift []string
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
if strings.HasPrefix(line, "verify: drift") {
|
||||
drift = append(drift, strings.TrimSpace(strings.TrimPrefix(line, "verify:")))
|
||||
}
|
||||
}
|
||||
return drift
|
||||
}
|
||||
|
||||
func printVerifyText(r *verifyReport) {
|
||||
w := rootCmd.OutOrStdout()
|
||||
fmt.Fprintf(w, "Txn: %s\n", r.TxnID)
|
||||
fmt.Fprintf(w, "Kind: %s Name: %s\n", r.Kind, r.Name)
|
||||
if r.Namespace != "" {
|
||||
fmt.Fprintf(w, "Namespace: %s\n", r.Namespace)
|
||||
}
|
||||
fmt.Fprintf(w, "Lead: %s\n", r.Lead)
|
||||
fmt.Fprintf(w, "Status: %s\n", r.Status)
|
||||
fmt.Fprintf(w, "\nPlanned allocations (%d):\n", len(r.PlannedAllocs))
|
||||
for _, a := range r.PlannedAllocs {
|
||||
fmt.Fprintf(w, " - %s (kind=%s)\n", a.Name, a.Kind)
|
||||
}
|
||||
units := 0
|
||||
configs := 0
|
||||
for _, f := range r.PlannedFiles {
|
||||
if f.Kind == "unit" {
|
||||
units++
|
||||
} else {
|
||||
configs++
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(w, "\nPlanned files: %d config, %d systemd units\n", configs, units)
|
||||
for _, f := range r.PlannedFiles {
|
||||
fmt.Fprintf(w, " - [%s] %s (mode %s)\n", f.Kind, f.Path, f.Mode)
|
||||
}
|
||||
if len(r.Drift) > 0 {
|
||||
fmt.Fprintf(w, "\nPre-flight drift (R-020, reported -- dry-run does not fail):\n")
|
||||
for _, d := range r.Drift {
|
||||
fmt.Fprintf(w, " ! %s\n", d)
|
||||
}
|
||||
}
|
||||
if r.VerifyOutput != "" {
|
||||
fmt.Fprintf(w, "\nverify.sh output:\n%s\n", r.VerifyOutput)
|
||||
}
|
||||
}
|
||||
|
||||
func printVerifyJSON(r *verifyReport) error {
|
||||
w := rootCmd.OutOrStdout()
|
||||
enc := json.NewEncoder(w)
|
||||
enc.SetIndent("", " ")
|
||||
return enc.Encode(r)
|
||||
}
|
||||
|
||||
// asTxnTransport adapts the jobVerifyTransport seam to the txn.Transport
|
||||
// interface (they have the same shape; this is a thin wrapper so the
|
||||
// two packages stay decoupled).
|
||||
type txnTransportAdapter struct {
|
||||
inner jobVerifyTransport
|
||||
}
|
||||
|
||||
func (a txnTransportAdapter) WriteFileIdempotent(ctx context.Context, peer string, path string, content []byte, mode os.FileMode) (bool, error) {
|
||||
return a.inner.WriteFileIdempotent(ctx, peer, path, content, mode)
|
||||
}
|
||||
|
||||
func (a txnTransportAdapter) Exec(ctx context.Context, peer string, cmd string) ([]byte, error) {
|
||||
return a.inner.Exec(ctx, peer, cmd)
|
||||
}
|
||||
|
||||
func asTxnTransport(t jobVerifyTransport) txn.Transport {
|
||||
return txnTransportAdapter{inner: t}
|
||||
}
|
||||
|
||||
// fileBase returns filepath.Base(path) without importing filepath in
|
||||
// the top of the file (kept here so the import block stays small).
|
||||
func fileBase(path string) string {
|
||||
if i := strings.LastIndexAny(path, "/\\"); i >= 0 {
|
||||
return path[i+1:]
|
||||
}
|
||||
return path
|
||||
}
|
||||
|
||||
func init() {
|
||||
jobVerifyCmd.Flags().StringVar(&jobVerifyLead, "lead", "", "lead peer address (host:port) (required)")
|
||||
jobVerifyCmd.Flags().StringVar(&jobVerifyNamespace, "namespace", "", "namespace scope")
|
||||
jobVerifyCmd.Flags().BoolVar(&jobVerifyJSON, "json", false, "JSON output")
|
||||
jobCmd.AddCommand(jobVerifyCmd)
|
||||
}
|
||||
@@ -0,0 +1,246 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/paths"
|
||||
"git.cloudinit.dev/coreci/orca/internal/secrets"
|
||||
)
|
||||
|
||||
// mockVerifyTransport is a record-and-replay mock of jobVerifyTransport.
|
||||
type mockVerifyTransport struct {
|
||||
writes []mockWriteCall
|
||||
execs []string
|
||||
execOut []byte
|
||||
execErr error
|
||||
}
|
||||
|
||||
type mockWriteCall struct {
|
||||
peer string
|
||||
path string
|
||||
content []byte
|
||||
mode os.FileMode
|
||||
}
|
||||
|
||||
func (m *mockVerifyTransport) WriteFileIdempotent(_ context.Context, peer string, path string, content []byte, mode os.FileMode) (bool, error) {
|
||||
m.writes = append(m.writes, mockWriteCall{peer, path, content, mode})
|
||||
return true, nil
|
||||
}
|
||||
|
||||
func (m *mockVerifyTransport) Exec(_ context.Context, _ string, cmd string) ([]byte, error) {
|
||||
m.execs = append(m.execs, cmd)
|
||||
return m.execOut, m.execErr
|
||||
}
|
||||
|
||||
// setupVerifyEnv sets ORCA_HOME to a temp dir and writes a master key
|
||||
// (txn.RenderBundle requires it).
|
||||
func setupVerifyEnv(t *testing.T) string {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", dir)
|
||||
mk, err := secrets.GenerateMasterKey()
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateMasterKey: %v", err)
|
||||
}
|
||||
if err := secrets.SaveMasterKey(paths.MasterKeyPath(), mk); err != nil {
|
||||
t.Fatalf("SaveMasterKey: %v", err)
|
||||
}
|
||||
return dir
|
||||
}
|
||||
|
||||
func TestJobVerifyCmdRegistered(t *testing.T) {
|
||||
for _, c := range jobCmd.Commands() {
|
||||
if c.Name() == "verify" {
|
||||
return
|
||||
}
|
||||
}
|
||||
t.Fatal("job verify command not registered on jobCmd")
|
||||
}
|
||||
|
||||
func TestJobVerifyValidSpec(t *testing.T) {
|
||||
setupVerifyEnv(t)
|
||||
mt := &mockVerifyTransport{execOut: []byte("verified\n")}
|
||||
jobVerifyTransportOverride = mt
|
||||
defer func() { jobVerifyTransportOverride = nil }()
|
||||
|
||||
resetRootFlags(t)
|
||||
spec := writeMDSpec(t, validJobMD)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"job", "verify", spec, "--lead", "lead:22"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("job verify valid: %v", err)
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "Planned allocations") {
|
||||
t.Errorf("expected planned allocs in output: %s", out)
|
||||
}
|
||||
if !strings.Contains(out, "systemd") && !strings.Contains(out, "unit") {
|
||||
t.Errorf("expected systemd unit in planned files: %s", out)
|
||||
}
|
||||
if len(mt.writes) == 0 {
|
||||
t.Errorf("expected bundle to be staged (writes), got 0")
|
||||
}
|
||||
if len(mt.execs) != 1 {
|
||||
t.Errorf("expected 1 exec (verify.sh), got %d", len(mt.execs))
|
||||
}
|
||||
}
|
||||
|
||||
func TestJobVerifyInvalidSpecFails(t *testing.T) {
|
||||
setupVerifyEnv(t)
|
||||
mt := &mockVerifyTransport{}
|
||||
jobVerifyTransportOverride = mt
|
||||
defer func() { jobVerifyTransportOverride = nil }()
|
||||
|
||||
resetRootFlags(t)
|
||||
spec := writeMDSpec(t, serviceMissingPortsMD)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"job", "verify", spec, "--lead", "lead:22"})
|
||||
if err := rootCmd.Execute(); err == nil {
|
||||
t.Fatal("expected error for invalid spec, got nil")
|
||||
}
|
||||
if len(mt.writes) != 0 {
|
||||
t.Errorf("should not stage bundle for invalid spec, got %d writes", len(mt.writes))
|
||||
}
|
||||
}
|
||||
|
||||
func TestJobVerifyJSONOutput(t *testing.T) {
|
||||
setupVerifyEnv(t)
|
||||
mt := &mockVerifyTransport{execOut: []byte("verified\n")}
|
||||
jobVerifyTransportOverride = mt
|
||||
defer func() { jobVerifyTransportOverride = nil }()
|
||||
|
||||
resetRootFlags(t)
|
||||
spec := writeMDSpec(t, validJobMD)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"job", "verify", spec, "--lead", "lead:22", "--json"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("job verify --json: %v", err)
|
||||
}
|
||||
var report verifyReport
|
||||
if err := json.Unmarshal(bytes.TrimSpace(buf.Bytes()), &report); err != nil {
|
||||
t.Fatalf("unmarshal verify json: %v\n%s", err, buf.String())
|
||||
}
|
||||
if report.Name != "my-job" {
|
||||
t.Errorf("report.Name = %q, want my-job", report.Name)
|
||||
}
|
||||
if len(report.PlannedFiles) == 0 {
|
||||
t.Errorf("expected planned files, got 0")
|
||||
}
|
||||
if report.TxnID == "" {
|
||||
t.Errorf("expected txn id, got empty")
|
||||
}
|
||||
}
|
||||
|
||||
func TestJobVerifyNamespaceScoping(t *testing.T) {
|
||||
setupVerifyEnv(t)
|
||||
mt := &mockVerifyTransport{execOut: []byte("verified\n")}
|
||||
jobVerifyTransportOverride = mt
|
||||
defer func() { jobVerifyTransportOverride = nil }()
|
||||
|
||||
resetRootFlags(t)
|
||||
spec := writeMDSpec(t, validJobMD)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"job", "verify", spec, "--lead", "lead:22", "--namespace", "prod"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("job verify --namespace: %v", err)
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "Namespace: prod") {
|
||||
t.Errorf("expected namespace in output: %s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestJobVerifyNoSideEffects(t *testing.T) {
|
||||
setupVerifyEnv(t)
|
||||
mt := &mockVerifyTransport{execOut: []byte("verified\n")}
|
||||
jobVerifyTransportOverride = mt
|
||||
defer func() { jobVerifyTransportOverride = nil }()
|
||||
|
||||
resetRootFlags(t)
|
||||
spec := writeMDSpec(t, validJobMD)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"job", "verify", spec, "--lead", "lead:22"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("job verify: %v", err)
|
||||
}
|
||||
// Staging writes the bundle files but apply is NOT run.
|
||||
for _, w := range mt.writes {
|
||||
if strings.Contains(w.path, ".applied") {
|
||||
t.Errorf("verify staged a .applied marker (side effect): %s", w.path)
|
||||
}
|
||||
}
|
||||
for _, c := range mt.execs {
|
||||
if strings.Contains(c, "apply") {
|
||||
t.Errorf("verify ran apply (side effect): %s", c)
|
||||
}
|
||||
if strings.Contains(c, "orca-pull.sh") {
|
||||
t.Errorf("verify ran orca-pull.sh (side effect): %s", c)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestJobVerifyPreflightDriftReported(t *testing.T) {
|
||||
setupVerifyEnv(t)
|
||||
mt := &mockVerifyTransport{execOut: []byte("verify: drift /etc/systemd/system/orca-v1-my-job.service has drifted from last apply\n")}
|
||||
jobVerifyTransportOverride = mt
|
||||
defer func() { jobVerifyTransportOverride = nil }()
|
||||
|
||||
resetRootFlags(t)
|
||||
spec := writeMDSpec(t, validJobMD)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"job", "verify", spec, "--lead", "lead:22"})
|
||||
// verify.sh exit 1 from drift is treated as a verify-failure by
|
||||
// the mock (execErr). But here execOut is set and execErr is nil,
|
||||
// so verify "passes" and drift is reported. We assert drift is
|
||||
// surfaced in the output and does NOT fail the dry-run.
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("job verify with drift should not fail dry-run: %v", err)
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(strings.ToLower(out), "drift") {
|
||||
t.Errorf("expected drift in output: %s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestJobVerifyMissingLeadFails(t *testing.T) {
|
||||
setupVerifyEnv(t)
|
||||
resetRootFlags(t)
|
||||
spec := writeMDSpec(t, validJobMD)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"job", "verify", spec})
|
||||
if err := rootCmd.Execute(); err == nil {
|
||||
t.Fatal("expected error for missing --lead, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestJobVerifyMissingMasterKeyFails(t *testing.T) {
|
||||
t.Setenv("ORCA_HOME", t.TempDir())
|
||||
resetRootFlags(t)
|
||||
spec := writeMDSpec(t, validJobMD)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"job", "verify", spec, "--lead", "lead:22"})
|
||||
if err := rootCmd.Execute(); err == nil {
|
||||
t.Fatal("expected error for missing master key, got nil")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,321 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"iter"
|
||||
"log/slog"
|
||||
"os"
|
||||
"os/signal"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/certpaths"
|
||||
"git.cloudinit.dev/coreci/orca/internal/model"
|
||||
"git.cloudinit.dev/coreci/orca/internal/sshpush"
|
||||
"git.cloudinit.dev/coreci/orca/internal/store"
|
||||
)
|
||||
|
||||
// logsExecer is the SSH command-execution seam used by `orca logs`.
|
||||
// *sshpush.Transport satisfies it via Exec; tests inject a mock
|
||||
// without a real SSH server (same pattern as drain.go / stepca mockExec).
|
||||
type logsExecer interface {
|
||||
Exec(ctx context.Context, peer string, cmd string) ([]byte, error)
|
||||
}
|
||||
|
||||
// logsExecOverride is the package-level exec seam. When non-nil it
|
||||
// replaces the production transport; tests set it and restore nil in
|
||||
// cleanup. nil means "build the real transport on first use".
|
||||
var logsExecOverride logsExecer
|
||||
|
||||
func logsExecFromCtx(_ context.Context) (logsExecer, error) {
|
||||
if logsExecOverride != nil {
|
||||
return logsExecOverride, nil
|
||||
}
|
||||
keyPath := certpaths.SSHKeyPath()
|
||||
khPath := certpaths.KnownHostsPath()
|
||||
return sshpush.NewTransport(keyPath, khPath), nil
|
||||
}
|
||||
|
||||
// LogLine is a single journald log entry parsed from journalctl --output
|
||||
// json. Host is the peer the line came from (set by the aggregator).
|
||||
type LogLine struct {
|
||||
Host string `json:"host"`
|
||||
Timestamp time.Time `json:"timestamp"`
|
||||
Unit string `json:"unit"`
|
||||
Message string `json:"message"`
|
||||
Priority string `json:"priority"`
|
||||
}
|
||||
|
||||
// journalRaw is the subset of journalctl --output json fields we
|
||||
// decode. Extra fields are ignored.
|
||||
type journalRaw struct {
|
||||
Realtime int64 `json:"__REALTIME_TIMESTAMP"`
|
||||
Unit string `json:"_SYSTEMD_UNIT"`
|
||||
Identifier string `json:"SYSLOG_IDENTIFIER"`
|
||||
Comm string `json:"_COMM"`
|
||||
Message string `json:"MESSAGE"`
|
||||
Priority any `json:"PRIORITY"`
|
||||
}
|
||||
|
||||
func (j journalRaw) unit() string {
|
||||
if j.Unit != "" {
|
||||
return strings.TrimSuffix(j.Unit, ".service")
|
||||
}
|
||||
if j.Identifier != "" {
|
||||
return j.Identifier
|
||||
}
|
||||
if j.Comm != "" {
|
||||
return j.Comm
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (j journalRaw) priority() string {
|
||||
switch p := j.Priority.(type) {
|
||||
case string:
|
||||
return p
|
||||
case float64:
|
||||
return fmt.Sprintf("%v", int(p))
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
|
||||
func (j journalRaw) timestamp() time.Time {
|
||||
if j.Realtime == 0 {
|
||||
return time.Time{}
|
||||
}
|
||||
return time.Unix(0, j.Realtime).UTC()
|
||||
}
|
||||
|
||||
var (
|
||||
logsAllNodes bool
|
||||
logsNode string
|
||||
logsJob string
|
||||
logsSince string
|
||||
logsJSON bool
|
||||
)
|
||||
|
||||
var logsCmd = &cobra.Command{
|
||||
Use: "logs",
|
||||
Short: "Aggregate journald logs across nodes (REQ-117)",
|
||||
Long: `Aggregate journald logs across registered nodes via SSH fanout.
|
||||
|
||||
orca logs --all-nodes --since 5m
|
||||
orca logs --node web-1 --since 1h
|
||||
orca logs --all-nodes --job web --since 30m --json
|
||||
|
||||
Runs 'journalctl -u 'orca-alloc-*' --since <dur> --output json' on each
|
||||
peer, parses the JSON-per-line output, and streams the entries with a
|
||||
[<hostname>] prefix (multi-node) or raw (single-node). --json outputs
|
||||
the raw journalctl JSON lines verbatim.
|
||||
|
||||
Use --all-nodes to fan out to every registered node, or --node <host>
|
||||
for a single node. --job <name> filters units to orca-alloc-<name>-*.
|
||||
|
||||
Ctrl-C cancels the fan-out via signal.NotifyContext.`,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
if !logsAllNodes && logsNode == "" {
|
||||
return fmt.Errorf("specify --all-nodes or --node <host>")
|
||||
}
|
||||
if logsAllNodes && logsNode != "" {
|
||||
return fmt.Errorf("--all-nodes and --node are mutually exclusive")
|
||||
}
|
||||
since, err := parseSince(logsSince)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
ctx, cancel := signal.NotifyContext(cmd.Context(), os.Interrupt, syscall.SIGTERM)
|
||||
defer cancel()
|
||||
|
||||
nodes, err := resolveLogNodes(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(nodes) == 0 {
|
||||
return fmt.Errorf("no nodes to query")
|
||||
}
|
||||
|
||||
ex, err := logsExecFromCtx(cmd.Context())
|
||||
if err != nil {
|
||||
return fmt.Errorf("ssh transport: %w", err)
|
||||
}
|
||||
|
||||
out := cmd.OutOrStdout()
|
||||
multi := len(nodes) > 1
|
||||
for line := range streamLogs(ctx, ex, nodes, since, logsJob) {
|
||||
if logsJSON {
|
||||
raw, _ := json.Marshal(line)
|
||||
fmt.Fprintln(out, string(raw))
|
||||
continue
|
||||
}
|
||||
if multi {
|
||||
fmt.Fprintf(out, "[%s] %s %s\n", line.Host, line.Timestamp.Format(time.RFC3339), line.Message)
|
||||
} else {
|
||||
fmt.Fprintln(out, line.Message)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
// parseSince parses a duration string like "5m", "1h30m", "500ms". The
|
||||
// returned time is time.Now().UTC().Add(-d). An empty string defaults
|
||||
// to 5 minutes.
|
||||
func parseSince(s string) (time.Time, error) {
|
||||
if s == "" {
|
||||
s = "5m"
|
||||
}
|
||||
d, err := time.ParseDuration(s)
|
||||
if err != nil {
|
||||
return time.Time{}, fmt.Errorf("--since %q: %w", s, err)
|
||||
}
|
||||
if d <= 0 {
|
||||
return time.Time{}, fmt.Errorf("--since must be positive, got %s", d)
|
||||
}
|
||||
return time.Now().UTC().Add(-d), nil
|
||||
}
|
||||
|
||||
// resolveLogNodes returns the set of nodes to query. For --all-nodes it
|
||||
// lists every registered node; for --node <host> it resolves a single
|
||||
// node by name or id.
|
||||
func resolveLogNodes(ctx context.Context) ([]*model.Node, error) {
|
||||
db, closer, err := openDB()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer closer()
|
||||
reg := store.NewNodeRepo(db)
|
||||
if logsAllNodes {
|
||||
return reg.List(ctx)
|
||||
}
|
||||
n, err := reg.GetByName(ctx, logsNode)
|
||||
if err == nil {
|
||||
return []*model.Node{n}, nil
|
||||
}
|
||||
if !errors.Is(err, store.ErrNotFound) {
|
||||
return nil, fmt.Errorf("lookup node %q: %w", logsNode, err)
|
||||
}
|
||||
n, err = reg.Get(ctx, logsNode)
|
||||
if err == nil {
|
||||
return []*model.Node{n}, nil
|
||||
}
|
||||
if errors.Is(err, store.ErrNotFound) {
|
||||
return nil, fmt.Errorf("node %q not found", logsNode)
|
||||
}
|
||||
return nil, fmt.Errorf("lookup node %q: %w", logsNode, err)
|
||||
}
|
||||
|
||||
// streamLogs fans out journalctl across nodes and yields parsed
|
||||
// LogLine values as they arrive. It runs each node's exec in its own
|
||||
// goroutine, scans the output line-by-line, and yields each parsed
|
||||
// JSON entry immediately. The stream ends when every node has
|
||||
// completed (or the context is cancelled). The caller drives the
|
||||
// iteration via range-over-func (D-017 iter.Seq pattern).
|
||||
func streamLogs(ctx context.Context, ex logsExecer, nodes []*model.Node, since time.Time, job string) iter.Seq[LogLine] {
|
||||
return func(yield func(LogLine) bool) {
|
||||
merged := make(chan LogLine)
|
||||
var wg sync.WaitGroup
|
||||
for _, n := range nodes {
|
||||
wg.Add(1)
|
||||
go func(n *model.Node) {
|
||||
defer wg.Done()
|
||||
streamNodeLines(ctx, ex, n, since, job, merged)
|
||||
}(n)
|
||||
}
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
wg.Wait()
|
||||
close(done)
|
||||
}()
|
||||
|
||||
// Pump merged lines to the yield function until either all
|
||||
// nodes finish or the consumer stops pulling (yield==false)
|
||||
// or the context is cancelled.
|
||||
for {
|
||||
select {
|
||||
case <-done:
|
||||
return
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case line := <-merged:
|
||||
if !yield(line) {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// streamNodeLines runs journalctl on a single node and pushes each
|
||||
// parsed line into out. It blocks until the exec completes (or the
|
||||
// context is cancelled); the caller is responsible for waiting on the
|
||||
// goroutine. Send is non-blocking via select on ctx.Done so a slow
|
||||
// consumer does not stall the fanout forever.
|
||||
func streamNodeLines(ctx context.Context, ex logsExecer, n *model.Node, since time.Time, job string, out chan<- LogLine) {
|
||||
peer := peerAddrForNode(n)
|
||||
if peer == "" {
|
||||
slog.Default().Warn("logs: cannot resolve SSH address for node", "node", n.Name)
|
||||
return
|
||||
}
|
||||
unitPattern := "orca-alloc-*"
|
||||
if job != "" {
|
||||
unitPattern = "orca-alloc-" + job + "-*"
|
||||
}
|
||||
sinceStr := since.Format("2006-01-02 15:04:05")
|
||||
cmd := fmt.Sprintf("journalctl -u %q --since %q --output json --no-pager", unitPattern, sinceStr)
|
||||
raw, err := ex.Exec(ctx, peer, cmd)
|
||||
if err != nil {
|
||||
slog.Default().Warn("logs: exec failed", "node", n.Name, "peer", peer, "error", err)
|
||||
return
|
||||
}
|
||||
host := n.Name
|
||||
for _, line := range strings.Split(string(raw), "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
if line == "" {
|
||||
continue
|
||||
}
|
||||
ll, perr := parseJournalLine(line)
|
||||
if perr != nil {
|
||||
continue
|
||||
}
|
||||
ll.Host = host
|
||||
select {
|
||||
case out <- ll:
|
||||
case <-ctx.Done():
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// parseJournalLine decodes a single journalctl --output json line into
|
||||
// a LogLine. Unknown fields are ignored.
|
||||
func parseJournalLine(s string) (LogLine, error) {
|
||||
var j journalRaw
|
||||
if err := json.Unmarshal([]byte(s), &j); err != nil {
|
||||
return LogLine{}, fmt.Errorf("parse journal line: %w", err)
|
||||
}
|
||||
return LogLine{
|
||||
Timestamp: j.timestamp(),
|
||||
Unit: j.unit(),
|
||||
Message: j.Message,
|
||||
Priority: j.priority(),
|
||||
}, nil
|
||||
}
|
||||
|
||||
func init() {
|
||||
logsCmd.Flags().BoolVar(&logsAllNodes, "all-nodes", false, "fan out to all registered nodes")
|
||||
logsCmd.Flags().StringVar(&logsNode, "node", "", "restrict to a single node (name or id)")
|
||||
logsCmd.Flags().StringVar(&logsJob, "job", "", "filter by job name (matches orca-alloc-<name>-* units)")
|
||||
logsCmd.Flags().StringVar(&logsSince, "since", "5m", "duration lookback (e.g. 5m, 1h, 30m); default 5m")
|
||||
logsCmd.Flags().BoolVar(&logsJSON, "json", false, "output raw JSON (one LogLine per line)")
|
||||
rootCmd.AddCommand(logsCmd)
|
||||
}
|
||||
@@ -0,0 +1,359 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/certpaths"
|
||||
"git.cloudinit.dev/coreci/orca/internal/model"
|
||||
"git.cloudinit.dev/coreci/orca/internal/store"
|
||||
)
|
||||
|
||||
// mockLogsExec is a record-and-replay execer for `orca logs` tests
|
||||
// (same pattern as internal/stepca mockExec / drain_test mockDrainExec).
|
||||
// It matches each incoming command against a list of (substring,
|
||||
// output) responses; the first match wins. An entry with an empty
|
||||
// substring matches any command.
|
||||
type mockLogsExec struct {
|
||||
mu sync.Mutex
|
||||
responses []logsMockResp
|
||||
calls []logsMockCall
|
||||
}
|
||||
|
||||
type logsMockResp struct {
|
||||
match string
|
||||
peer string // if set, must match the peer too
|
||||
out string
|
||||
}
|
||||
|
||||
type logsMockCall struct {
|
||||
peer string
|
||||
cmd string
|
||||
}
|
||||
|
||||
func (m *mockLogsExec) Exec(_ context.Context, peer, cmd string) ([]byte, error) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
m.calls = append(m.calls, logsMockCall{peer: peer, cmd: cmd})
|
||||
for _, r := range m.responses {
|
||||
if r.peer != "" && !strings.Contains(peer, r.peer) {
|
||||
continue
|
||||
}
|
||||
if r.match == "" || strings.Contains(cmd, r.match) {
|
||||
return []byte(r.out), nil
|
||||
}
|
||||
}
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
func (m *mockLogsExec) countCalls(match string) int {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
n := 0
|
||||
for _, c := range m.calls {
|
||||
if strings.Contains(c.cmd, match) {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// logsTestEnv wires a mockLogsExec into logsExecOverride and returns
|
||||
// the mock + a cleanup func. Tests MUST defer the cleanup.
|
||||
func logsTestEnv(t *testing.T) *mockLogsExec {
|
||||
t.Helper()
|
||||
prev := logsExecOverride
|
||||
mx := &mockLogsExec{}
|
||||
logsExecOverride = mx
|
||||
t.Cleanup(func() { logsExecOverride = prev })
|
||||
return mx
|
||||
}
|
||||
|
||||
// logsNodeForTest inserts a node with a fixed name and returns it so
|
||||
// logs commands can target it by name. Uses the test ORCA_HOME db.
|
||||
func logsNodeForTest(t *testing.T, name, addr string) *model.Node {
|
||||
t.Helper()
|
||||
db, err := store.Open(certpaths.DBPath())
|
||||
if err != nil {
|
||||
t.Fatalf("open db: %v", err)
|
||||
}
|
||||
defer db.Close()
|
||||
n := &model.Node{
|
||||
ID: "node-" + name,
|
||||
Name: name,
|
||||
Address: addr,
|
||||
State: model.NodeStateReady,
|
||||
JoinedAt: time.Now().UTC(),
|
||||
LastSeen: time.Now().UTC(),
|
||||
Kind: string(model.NodeKindLinux),
|
||||
}
|
||||
if err := store.NewNodeRepo(db).Insert(context.Background(), n); err != nil {
|
||||
t.Fatalf("insert node: %v", err)
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// journalJSONLine renders a single journalctl --output json line.
|
||||
func journalJSONLine(ts time.Time, unit, msg, prio string) string {
|
||||
return `{"__REALTIME_TIMESTAMP":` + strconv.FormatInt(ts.UnixNano(), 10) +
|
||||
`,"_SYSTEMD_UNIT":"` + unit + `.service","MESSAGE":"` + msg + `","PRIORITY":"` + prio + `"}`
|
||||
}
|
||||
|
||||
// runLogsCmd executes logsCmd with the given args against a buffered
|
||||
// stdout, returning the captured output and any error.
|
||||
func runLogsCmd(t *testing.T, args []string) (string, error) {
|
||||
t.Helper()
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs(args)
|
||||
err := rootCmd.Execute()
|
||||
return buf.String(), err
|
||||
}
|
||||
|
||||
func TestLogsCommandRegistered(t *testing.T) {
|
||||
found := false
|
||||
for _, cmd := range rootCmd.Commands() {
|
||||
if cmd.Name() == "logs" {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Fatal("logs command not registered on rootCmd")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLogsRequiresAllNodesOrNode(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
_, err := runLogsCmd(t, []string{"logs"})
|
||||
if err == nil {
|
||||
t.Fatal("expected error when neither --all-nodes nor --node is set")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "--all-nodes") && !strings.Contains(err.Error(), "--node") {
|
||||
t.Errorf("error = %q, want mention of --all-nodes/--node", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLogsAllNodesAndNodeMutuallyExclusive(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
_, err := runLogsCmd(t, []string{"logs", "--all-nodes", "--node", "x"})
|
||||
if err == nil {
|
||||
t.Fatal("expected error for --all-nodes + --node together")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLogsSinceParsing(t *testing.T) {
|
||||
cases := []struct {
|
||||
in string
|
||||
wantErr bool
|
||||
}{
|
||||
{"5m", false},
|
||||
{"1h30m", false},
|
||||
{"500ms", false},
|
||||
{"", false}, // defaults to 5m
|
||||
{"notaduration", true},
|
||||
{"-5m", true},
|
||||
{"0s", true},
|
||||
}
|
||||
for _, c := range cases {
|
||||
_, err := parseSince(c.in)
|
||||
if c.wantErr && err == nil {
|
||||
t.Errorf("parseSince(%q): expected error, got nil", c.in)
|
||||
}
|
||||
if !c.wantErr && err != nil {
|
||||
t.Errorf("parseSince(%q): unexpected error: %v", c.in, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestLogsSinceDefaultIs5m(t *testing.T) {
|
||||
before := time.Now().UTC()
|
||||
got, err := parseSince("")
|
||||
if err != nil {
|
||||
t.Fatalf("parseSince(empty): %v", err)
|
||||
}
|
||||
after := time.Now().UTC()
|
||||
// got should be ~5m ago. The call's "now" is in [before, after],
|
||||
// so got = now-5m is in [before-5m, after-5m].
|
||||
lo := before.Add(-5 * time.Minute)
|
||||
hi := after.Add(-5 * time.Minute)
|
||||
if got.Before(lo) || got.After(hi) {
|
||||
t.Errorf("parseSince(empty) = %v, want ~5m ago (between %v and %v)", got, lo, hi)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLogsAllNodes_FansOutAndStreams(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
|
||||
logsNodeForTest(t, "web-1", "web-1:8443")
|
||||
logsNodeForTest(t, "web-2", "web-2:8443")
|
||||
mx := logsTestEnv(t)
|
||||
|
||||
ts := time.Date(2026, 1, 1, 12, 0, 0, 0, time.UTC)
|
||||
// Two nodes, each returns one journal line.
|
||||
mx.responses = []logsMockResp{
|
||||
{peer: "web-1", match: "journalctl", out: journalJSONLine(ts, "orca-alloc-web-0", "hello from web-1", "6") + "\n"},
|
||||
{peer: "web-2", match: "journalctl", out: journalJSONLine(ts, "orca-alloc-web-0", "hello from web-2", "6") + "\n"},
|
||||
}
|
||||
|
||||
out, err := runLogsCmd(t, []string{"logs", "--all-nodes", "--since", "5m"})
|
||||
if err != nil {
|
||||
t.Fatalf("logs: %v", err)
|
||||
}
|
||||
if mx.countCalls("journalctl") != 2 {
|
||||
t.Errorf("expected 2 journalctl calls, got %d", mx.countCalls("journalctl"))
|
||||
}
|
||||
if !strings.Contains(out, "hello from web-1") {
|
||||
t.Errorf("output missing web-1 line:\n%s", out)
|
||||
}
|
||||
if !strings.Contains(out, "hello from web-2") {
|
||||
t.Errorf("output missing web-2 line:\n%s", out)
|
||||
}
|
||||
if !strings.Contains(out, "[web-1]") || !strings.Contains(out, "[web-2]") {
|
||||
t.Errorf("output missing [host] prefix for multi-node:\n%s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLogsSingleNode_NoHostPrefix(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
|
||||
logsNodeForTest(t, "solo", "solo:8443")
|
||||
mx := logsTestEnv(t)
|
||||
|
||||
ts := time.Date(2026, 1, 1, 12, 0, 0, 0, time.UTC)
|
||||
mx.responses = []logsMockResp{
|
||||
{match: "journalctl", out: journalJSONLine(ts, "orca-alloc-web-0", "single node msg", "6") + "\n"},
|
||||
}
|
||||
|
||||
out, err := runLogsCmd(t, []string{"logs", "--node", "solo", "--since", "1h"})
|
||||
if err != nil {
|
||||
t.Fatalf("logs: %v", err)
|
||||
}
|
||||
if !strings.Contains(out, "single node msg") {
|
||||
t.Errorf("output missing message:\n%s", out)
|
||||
}
|
||||
if strings.Contains(out, "[solo]") {
|
||||
t.Errorf("single-node output should NOT have [host] prefix:\n%s", out)
|
||||
}
|
||||
if mx.countCalls("journalctl") != 1 {
|
||||
t.Errorf("expected 1 journalctl call, got %d", mx.countCalls("journalctl"))
|
||||
}
|
||||
}
|
||||
|
||||
func TestLogsJSONOutput(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
|
||||
logsNodeForTest(t, "jsonnode", "jsonnode:8443")
|
||||
mx := logsTestEnv(t)
|
||||
|
||||
ts := time.Date(2026, 1, 1, 12, 0, 0, 0, time.UTC)
|
||||
mx.responses = []logsMockResp{
|
||||
{match: "journalctl", out: journalJSONLine(ts, "orca-alloc-web-0", "json line", "6") + "\n"},
|
||||
}
|
||||
|
||||
out, err := runLogsCmd(t, []string{"logs", "--node", "jsonnode", "--since", "1m", "--json"})
|
||||
if err != nil {
|
||||
t.Fatalf("logs: %v", err)
|
||||
}
|
||||
if !strings.Contains(out, `"message":"json line"`) {
|
||||
t.Errorf("json output missing message field:\n%s", out)
|
||||
}
|
||||
if !strings.Contains(out, `"host":"jsonnode"`) {
|
||||
t.Errorf("json output missing host field:\n%s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLogsJobFilterAffectsUnitPattern(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
|
||||
logsNodeForTest(t, "jobnode", "jobnode:8443")
|
||||
mx := logsTestEnv(t)
|
||||
|
||||
mx.responses = []logsMockResp{
|
||||
{match: "orca-alloc-web-*", out: ""},
|
||||
}
|
||||
|
||||
_, err := runLogsCmd(t, []string{"logs", "--node", "jobnode", "--job", "web", "--since", "1m"})
|
||||
if err != nil {
|
||||
t.Fatalf("logs: %v", err)
|
||||
}
|
||||
if mx.countCalls("orca-alloc-web-*") != 1 {
|
||||
t.Errorf("expected unit pattern orca-alloc-web-* in cmd, calls=%v", mx.calls)
|
||||
}
|
||||
if mx.countCalls("orca-alloc-*") != 1 {
|
||||
// The job-specific pattern is a subset of the bare pattern;
|
||||
// substring match counts both. Ensure the job pattern was used.
|
||||
}
|
||||
}
|
||||
|
||||
func TestLogsCancelStopsStream(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
|
||||
logsNodeForTest(t, "cancelnode", "cancelnode:8443")
|
||||
mx := logsTestEnv(t)
|
||||
|
||||
mx.responses = []logsMockResp{
|
||||
{match: "journalctl", out: journalJSONLine(time.Now().UTC(), "orca-alloc-x", "msg", "6") + "\n"},
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel() // already-cancelled context: streamLogs should return immediately
|
||||
|
||||
ex, err := logsExecFromCtx(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("exec: %v", err)
|
||||
}
|
||||
nodes := []*model.Node{
|
||||
{ID: "n1", Name: "cancelnode", Address: "cancelnode:8443"},
|
||||
}
|
||||
consumed := 0
|
||||
for range streamLogs(ctx, ex, nodes, time.Now().UTC().Add(-1*time.Minute), "") {
|
||||
consumed++
|
||||
}
|
||||
if consumed > 1 {
|
||||
t.Errorf("cancelled stream yielded %d lines, want <= 1", consumed)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLogsParseJournalLine(t *testing.T) {
|
||||
ts := time.Date(2026, 1, 1, 12, 0, 0, 123456789, time.UTC)
|
||||
raw := journalJSONLine(ts, "orca-alloc-web-0", "hello world", "6")
|
||||
ll, err := parseJournalLine(raw)
|
||||
if err != nil {
|
||||
t.Fatalf("parseJournalLine: %v", err)
|
||||
}
|
||||
if ll.Message != "hello world" {
|
||||
t.Errorf("message = %q, want hello world", ll.Message)
|
||||
}
|
||||
if ll.Unit != "orca-alloc-web-0" {
|
||||
t.Errorf("unit = %q, want orca-alloc-web-0", ll.Unit)
|
||||
}
|
||||
if ll.Priority != "6" {
|
||||
t.Errorf("priority = %q, want 6", ll.Priority)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLogsParseJournalLine_InvalidJSON(t *testing.T) {
|
||||
_, err := parseJournalLine("not json")
|
||||
if err == nil {
|
||||
t.Error("expected error for invalid json, got nil")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,133 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/store"
|
||||
"git.cloudinit.dev/coreci/orca/internal/transport"
|
||||
)
|
||||
|
||||
var metricsAddr string
|
||||
|
||||
var metricsCmd = &cobra.Command{
|
||||
Use: "metrics",
|
||||
Short: "Run the orca metrics endpoint (Prometheus text exposition)",
|
||||
Long: `Start a standalone HTTP server exposing Prometheus text-exposition
|
||||
metrics at /metrics and a liveness probe at /healthz. Designed to run on the
|
||||
CLI host or a designated metrics host; polls cluster state periodically and
|
||||
updates gauges. No orca daemon required (R-001).`,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
log := newLogger()
|
||||
m := transport.NewMetrics()
|
||||
|
||||
db, closer, err := openDB()
|
||||
if err != nil {
|
||||
log.Warn("metrics: open db failed, gauges will stay 0", "err", err)
|
||||
db = nil
|
||||
closer = func() error { return nil }
|
||||
}
|
||||
defer closer()
|
||||
|
||||
var pollWG sync.WaitGroup
|
||||
pollCtx, pollCancel := context.WithCancel(cmd.Context())
|
||||
defer func() {
|
||||
pollCancel()
|
||||
pollWG.Wait()
|
||||
}()
|
||||
|
||||
pollWG.Add(1)
|
||||
go func() {
|
||||
defer pollWG.Done()
|
||||
pollLoop(pollCtx, m, db, log)
|
||||
}()
|
||||
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("/metrics", func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "text/plain; version=0.0.4; charset=utf-8")
|
||||
if err := m.WritePrometheus(w); err != nil {
|
||||
log.Warn("metrics: write exposition failed", "err", err)
|
||||
}
|
||||
})
|
||||
mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusOK)
|
||||
_, _ = w.Write([]byte("ok\n"))
|
||||
})
|
||||
|
||||
srv := &http.Server{
|
||||
Addr: metricsAddr,
|
||||
Handler: mux,
|
||||
}
|
||||
|
||||
errCh := make(chan error, 1)
|
||||
go func() {
|
||||
err := srv.ListenAndServe()
|
||||
if err != nil && !errors.Is(err, http.ErrServerClosed) {
|
||||
errCh <- err
|
||||
}
|
||||
}()
|
||||
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "✓ orca metrics listening on %s\n", metricsAddr)
|
||||
fmt.Fprintln(cmd.OutOrStdout(), " /metrics - Prometheus text exposition")
|
||||
fmt.Fprintln(cmd.OutOrStdout(), " /healthz - liveness probe")
|
||||
fmt.Fprintln(cmd.OutOrStdout(), " press Ctrl+C to stop")
|
||||
|
||||
ctx, stop := signal.NotifyContext(cmd.Context(), os.Interrupt, syscall.SIGTERM)
|
||||
defer stop()
|
||||
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "\nshutting down...")
|
||||
shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
return srv.Shutdown(shutdownCtx)
|
||||
case err := <-errCh:
|
||||
return fmt.Errorf("metrics server: %w", err)
|
||||
}
|
||||
},
|
||||
}
|
||||
|
||||
func pollLoop(ctx context.Context, m *transport.Metrics, db *sql.DB, log interface{ Warn(string, ...any) }) {
|
||||
if db == nil {
|
||||
return
|
||||
}
|
||||
ticker := time.NewTicker(15 * time.Second)
|
||||
defer ticker.Stop()
|
||||
refresh(ctx, m, db, log)
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
refresh(ctx, m, db, log)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func refresh(ctx context.Context, m *transport.Metrics, db *sql.DB, log interface{ Warn(string, ...any) }) {
|
||||
if nodes, err := store.NewNodeRepo(db).List(ctx); err != nil {
|
||||
log.Warn("metrics: node list failed", "err", err)
|
||||
} else {
|
||||
m.SetGauge("nodes_total", float64(len(nodes)))
|
||||
}
|
||||
if jobs, err := store.NewJobRepo(db).List(ctx); err != nil {
|
||||
log.Warn("metrics: job list failed", "err", err)
|
||||
} else {
|
||||
m.SetGauge("allocs_total", float64(len(jobs)))
|
||||
}
|
||||
}
|
||||
|
||||
func init() {
|
||||
metricsCmd.Flags().StringVar(&metricsAddr, "addr", ":9100", "listen address for the metrics HTTP server")
|
||||
rootCmd.AddCommand(metricsCmd)
|
||||
}
|
||||
@@ -0,0 +1,116 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestMetricsCmdRegistered(t *testing.T) {
|
||||
found := false
|
||||
for _, c := range rootCmd.Commands() {
|
||||
if c.Name() == "metrics" {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Fatal("metricsCmd not registered on root")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetricsAddrFlagDefault(t *testing.T) {
|
||||
f := metricsCmd.Flags().Lookup("addr")
|
||||
if f == nil {
|
||||
t.Fatal("--addr flag not registered on metricsCmd")
|
||||
}
|
||||
if f.DefValue != ":9100" {
|
||||
t.Errorf("--addr default = %q, want %q", f.DefValue, ":9100")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetricsEndpoints(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
|
||||
// Pick a free port by briefly listening then closing.
|
||||
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatalf("listen probe: %v", err)
|
||||
}
|
||||
addr := ln.Addr().String()
|
||||
_ = ln.Close()
|
||||
|
||||
metricsAddr = addr
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
|
||||
cmd := metricsCmd
|
||||
var out bytes.Buffer
|
||||
cmd.SetOut(&out)
|
||||
cmd.SetErr(&out)
|
||||
cmd.SetContext(ctx)
|
||||
|
||||
errCh := make(chan error, 1)
|
||||
go func() {
|
||||
errCh <- cmd.RunE(cmd, nil)
|
||||
}()
|
||||
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
var resp *http.Response
|
||||
for time.Now().Before(deadline) {
|
||||
resp, err = http.Get("http://" + addr + "/healthz")
|
||||
if err == nil {
|
||||
break
|
||||
}
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("GET /healthz: %v", err)
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Errorf("/healthz status = %d, want 200", resp.StatusCode)
|
||||
}
|
||||
body, _ := io.ReadAll(resp.Body)
|
||||
resp.Body.Close()
|
||||
if !strings.HasPrefix(string(body), "ok") {
|
||||
t.Errorf("/healthz body = %q, want \"ok\"", string(body))
|
||||
}
|
||||
|
||||
resp2, err := http.Get("http://" + addr + "/metrics")
|
||||
if err != nil {
|
||||
t.Fatalf("GET /metrics: %v", err)
|
||||
}
|
||||
defer resp2.Body.Close()
|
||||
if resp2.StatusCode != http.StatusOK {
|
||||
t.Errorf("/metrics status = %d, want 200", resp2.StatusCode)
|
||||
}
|
||||
mbody, _ := io.ReadAll(resp2.Body)
|
||||
ms := string(mbody)
|
||||
for _, name := range []string{
|
||||
"txns_applied_total",
|
||||
"txns_drifted_total",
|
||||
"drifts_remediated_total",
|
||||
"peers_total",
|
||||
"nodes_total",
|
||||
"allocs_total",
|
||||
} {
|
||||
if !strings.Contains(ms, name) {
|
||||
t.Errorf("/metrics missing %q\n---\n%s", name, ms)
|
||||
}
|
||||
}
|
||||
|
||||
cancel()
|
||||
select {
|
||||
case <-errCh:
|
||||
case <-time.After(3 * time.Second):
|
||||
t.Fatal("metrics command did not stop after cancel")
|
||||
}
|
||||
}
|
||||
@@ -6,8 +6,13 @@ import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/certpaths"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/migration"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
func resetRootFlags(t *testing.T) {
|
||||
@@ -18,6 +23,7 @@ func resetRootFlags(t *testing.T) {
|
||||
rootCmd.SetErr(&buf)
|
||||
_ = rootCmd.PersistentFlags().Set("system", "false")
|
||||
_ = rootCmd.PersistentFlags().Set("json", "false")
|
||||
_ = rootCmd.PersistentFlags().Set("no-deprecation-warnings", "false")
|
||||
resetCommandFlags()
|
||||
}
|
||||
|
||||
@@ -28,11 +34,62 @@ func resetRootFlags(t *testing.T) {
|
||||
// command without resetRootFlags may call it directly.
|
||||
func resetCommandFlags() {
|
||||
joinName, joinAddr, joinCAFinger, joinType = "", "", "", "localhost"
|
||||
joinHost, joinSSHUser, joinPassword, proxmoxUser, proxmoxRole = "", "root", "", "orca", "OrcaOperator"
|
||||
joinHost, joinSSHUser, joinSSHKey, proxmoxUser, proxmoxRole = "", "root", "", "orca", "OrcaOperator"
|
||||
joinSSHPort, leaveID, nodeWatch = 22, "", false
|
||||
stopID, runTarget, runIDKey, jobWatch = "", "", "", false
|
||||
migrateTarget = ""
|
||||
drainTimeout = 30 * time.Second
|
||||
logsAllNodes, logsNode, logsJob, logsSince, logsJSON = false, "", "", "5m", false
|
||||
capSetCPU, capSetMem, capSetDisk, capNodeID = 0, 0, 0, ""
|
||||
auditLimit = 50
|
||||
backupOutPath, restoreInPath, restoreTargetDir = "", "", ""
|
||||
restoreForce = false
|
||||
restoreDryRun = false
|
||||
collectorRoot = "/"
|
||||
collectorDryRun = false
|
||||
txnApplyForce, txnApplyAckRisk, txnApplyYes = false, false, false
|
||||
txnApplyNamespace = ""
|
||||
txnApplyTimeout = 5 * time.Minute
|
||||
txnApplyLead, txnRollbackLead = "", ""
|
||||
driftWatchInterval = 2 * time.Second
|
||||
driftWatchPaths = nil
|
||||
driftShowPeer = ""
|
||||
driftConfigPath = ""
|
||||
driftRemediateForce = false
|
||||
jobRestartPeer = ""
|
||||
jobLintExplain = false
|
||||
jobLintFormat = "text"
|
||||
jobVerifyLead = ""
|
||||
jobVerifyNamespace = ""
|
||||
jobVerifyJSON = false
|
||||
peerSetupNoOrcaUser = false
|
||||
upgradeTo = ""
|
||||
upgradeImportCA = false
|
||||
upgradeForce = false
|
||||
upgradeDryRun = false
|
||||
upgradeRunnerOverride = nil
|
||||
httpClientOverride = nil
|
||||
upgradeTransportOverride = nil
|
||||
peersListerOverride = nil
|
||||
migration.SetCAImporter(nil)
|
||||
cutoverTimeout = 5 * time.Minute
|
||||
rotateLeadTo = ""
|
||||
rotateLeadForce = false
|
||||
resetNSFlags()
|
||||
// Reset per-command output writers so tests that polluted them
|
||||
// (e.g. daemon tests calling cmd.SetOut(&buf)) don't leak into
|
||||
// other tests. nil → cobra walks to rootCmd's writer.
|
||||
for _, c := range []*cobra.Command{
|
||||
nodeDrainCmd, daemonCmd, daemonDrainAndStopCmd,
|
||||
jobCmd, jobMigrateCmd, jobRunCmd, jobListCmd, jobStopCmd, jobLogsCmd, jobLintCmd, jobVerifyCmd,
|
||||
logsCmd,
|
||||
clusterCmd, clusterCutoverCmd, clusterRotateLeadCmd, compatCheckCmd, noOrcaOnServerCmd,
|
||||
} {
|
||||
if c != nil {
|
||||
c.SetOut(nil)
|
||||
c.SetErr(nil)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestNamespaceDefaultsToUserHome(t *testing.T) {
|
||||
|
||||
@@ -0,0 +1,266 @@
|
||||
// Package cli: nft.go implements the `orca nft` command family (P15.5,
|
||||
// REQ-102). Subcommands:
|
||||
//
|
||||
// orca nft show [--peer <host>] show current nft ruleset
|
||||
// orca nft diff --against <txn-id> compare live vs expected
|
||||
// orca nft doctor alias for `orca doctor nft`
|
||||
// orca nft country block add <cc-list> opt-in GeoIP blocking
|
||||
// orca nft rate limit set --rate <N>/s adjust rate-limit meter
|
||||
package cli
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/emitter"
|
||||
"git.cloudinit.dev/coreci/orca/internal/paths"
|
||||
)
|
||||
|
||||
var (
|
||||
nftShowPeer string
|
||||
nftDiffAgainst string
|
||||
nftRateLimitRate int
|
||||
nftCountryBlockCC string
|
||||
)
|
||||
|
||||
var nftCmd = &cobra.Command{
|
||||
Use: "nft",
|
||||
Short: "Inspect and manage the nftables ingress ruleset (P15.5, R-017)",
|
||||
Long: `Orca's ingress is hybrid (R-017): Traefik binds 127.0.0.1:8443
|
||||
and nftables DNATs the public :443 to it. The nft family inspects and
|
||||
adjusts the kernel-side ruleset.`,
|
||||
}
|
||||
|
||||
var nftShowCmd = &cobra.Command{
|
||||
Use: "show",
|
||||
Short: "Show the current nft ruleset (SSHes to peer, nft list table)",
|
||||
Long: `Show the live inet orca-ingress table on the peer (default: lead).`,
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
t, err := nftTransportFromCtx()
|
||||
if err != nil {
|
||||
return fmt.Errorf("nft transport: %w", err)
|
||||
}
|
||||
peer := nftShowPeer
|
||||
if peer == "" {
|
||||
peer = nftLeadPeer()
|
||||
}
|
||||
out, err := t.Exec(cmd.Context(), peer, "nft list table inet orca-ingress")
|
||||
if err != nil {
|
||||
return fmt.Errorf("nft list: %w", err)
|
||||
}
|
||||
fmt.Fprint(cmd.OutOrStdout(), string(out))
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
var nftDiffCmd = &cobra.Command{
|
||||
Use: "diff",
|
||||
Short: "Compare the live nft ruleset against the expected from a txn",
|
||||
Long: `Compare the live inet orca-ingress table against the ruleset
|
||||
expected from the given txn-id (the rendered /etc/nftables.d/orca.nft
|
||||
recorded at apply time). Reports per-rule diffs.`,
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
if nftDiffAgainst == "" {
|
||||
return errors.New("nft diff: --against <txn-id> is required")
|
||||
}
|
||||
t, err := nftTransportFromCtx()
|
||||
if err != nil {
|
||||
return fmt.Errorf("nft transport: %w", err)
|
||||
}
|
||||
peer := nftLeadPeer()
|
||||
live, err := t.Exec(cmd.Context(), peer, "nft list table inet orca-ingress")
|
||||
if err != nil {
|
||||
return fmt.Errorf("nft list: %w", err)
|
||||
}
|
||||
expected, err := loadExpectedNftFromTxn(nftDiffAgainst)
|
||||
if err != nil {
|
||||
return fmt.Errorf("load txn %s: %w", nftDiffAgainst, err)
|
||||
}
|
||||
liveLines := strings.Split(string(live), "\n")
|
||||
expLines := strings.Split(expected, "\n")
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "live=%d lines expected=%d lines (txn %s)\n", len(liveLines), len(expLines), nftDiffAgainst)
|
||||
diff := diffLineSets(expLines, liveLines)
|
||||
if len(diff) == 0 {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "no drift: live ruleset matches txn")
|
||||
return nil
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "drift detected (%d differing lines):\n", len(diff))
|
||||
for _, d := range diff {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), d)
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
var nftDoctorAliasCmd = &cobra.Command{
|
||||
Use: "doctor",
|
||||
Short: "Alias for `orca doctor nft`",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: doctorNftCmd.RunE,
|
||||
}
|
||||
|
||||
var nftCountryCmd = &cobra.Command{
|
||||
Use: "country",
|
||||
Short: "GeoIP country-block management (opt-in)",
|
||||
Long: `Manage the orca_geoip_block nft set (opt-in GeoIP blocking).
|
||||
Adds ISO-3166 alpha-2 country codes to a blacklist set.`,
|
||||
}
|
||||
|
||||
var nftCountryBlockCmd = &cobra.Command{
|
||||
Use: "block",
|
||||
Short: "Block management for the GeoIP country set",
|
||||
}
|
||||
|
||||
var nftCountryBlockAddCmd = &cobra.Command{
|
||||
Use: "add <cc-list>",
|
||||
Short: "Add country codes to the GeoIP block set",
|
||||
Long: `Add one or more ISO-3166 alpha-2 country codes (comma-separated)
|
||||
to the orca_geoip_block nft set on the lead. Example:
|
||||
orca nft country block add RU,CN`,
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
ccList := strings.ToUpper(strings.TrimSpace(args[0]))
|
||||
if ccList == "" {
|
||||
return errors.New("nft country block add: empty cc-list")
|
||||
}
|
||||
codes := strings.Split(ccList, ",")
|
||||
for _, c := range codes {
|
||||
if len(c) != 2 {
|
||||
return fmt.Errorf("nft country block add: %q is not a 2-letter country code", c)
|
||||
}
|
||||
}
|
||||
t, err := nftTransportFromCtx()
|
||||
if err != nil {
|
||||
return fmt.Errorf("nft transport: %w", err)
|
||||
}
|
||||
peer := nftLeadPeer()
|
||||
cmdStr := fmt.Sprintf("nft add element inet orca-ingress orca_geoip_block { %s }", strings.Join(quoteAll(codes), ", "))
|
||||
if _, err := t.Exec(cmd.Context(), peer, cmdStr); err != nil {
|
||||
return fmt.Errorf("nft add element: %w", err)
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "added %d country code(s) to orca_geoip_block on %s\n", len(codes), peer)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
var nftRateCmd = &cobra.Command{
|
||||
Use: "rate",
|
||||
Short: "Rate-limit meter management",
|
||||
}
|
||||
|
||||
var nftRateLimitCmd = &cobra.Command{
|
||||
Use: "limit",
|
||||
Short: "Rate-limit meter management",
|
||||
}
|
||||
|
||||
var nftRateLimitSetCmd = &cobra.Command{
|
||||
Use: "set",
|
||||
Short: "Adjust the forward-chain rate-limit meter (--rate <N>/s)",
|
||||
Long: `Re-render /etc/nftables.d/orca.nft with the new rate and apply it
|
||||
on the lead. The burst is set to 2x the rate when not specified.`,
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
if nftRateLimitRate <= 0 {
|
||||
return errors.New("nft rate limit set: --rate <N>/s is required and must be > 0")
|
||||
}
|
||||
t, err := nftTransportFromCtx()
|
||||
if err != nil {
|
||||
return fmt.Errorf("nft transport: %w", err)
|
||||
}
|
||||
cfg := emitter.NftClusterConfig{RateLimit: nftRateLimitRate, RateBurst: nftRateLimitRate * 2}
|
||||
files, err := (emitter.NftEmitter{}).RenderNftConfig(cfg)
|
||||
if err != nil {
|
||||
return fmt.Errorf("render nft: %w", err)
|
||||
}
|
||||
peer := nftLeadPeer()
|
||||
for _, f := range files {
|
||||
out, err := t.Exec(cmd.Context(), peer, fmt.Sprintf("nft -f - <<'ORCA-NFT'\n%s\nORCA-NFT", f.Content))
|
||||
if err != nil {
|
||||
return fmt.Errorf("nft -f on %s: %w (out=%s)", peer, err, string(out))
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "rate-limit set to %d/s burst %d on %s\n", cfg.RateLimit, cfg.RateBurst, peer)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
func init() {
|
||||
nftShowCmd.Flags().StringVar(&nftShowPeer, "peer", "", "peer host to query (default: lead)")
|
||||
nftDiffCmd.Flags().StringVar(&nftDiffAgainst, "against", "", "txn-id to diff against (required)")
|
||||
nftRateLimitSetCmd.Flags().IntVar(&nftRateLimitRate, "rate", 0, "rate limit in packets/second (required, >0)")
|
||||
|
||||
nftCountryBlockCmd.AddCommand(nftCountryBlockAddCmd)
|
||||
nftCountryCmd.AddCommand(nftCountryBlockCmd)
|
||||
nftRateLimitCmd.AddCommand(nftRateLimitSetCmd)
|
||||
nftRateCmd.AddCommand(nftRateLimitCmd)
|
||||
|
||||
nftCmd.AddCommand(nftShowCmd, nftDiffCmd, nftDoctorAliasCmd, nftCountryCmd, nftRateCmd)
|
||||
rootCmd.AddCommand(nftCmd)
|
||||
}
|
||||
|
||||
// loadExpectedNftFromTxn returns the rendered nft ruleset recorded for
|
||||
// the given txn-id. The txn record is stored at TxnDir()/txn-id/ with
|
||||
// the rendered file contents; this helper reads the nft file artifact.
|
||||
// When the txn record is absent it returns an error.
|
||||
func loadExpectedNftFromTxn(txnID string) (string, error) {
|
||||
artifactPath := filepath.Join(paths.TxnDir(), txnID, "orca.nft")
|
||||
if data, err := os.ReadFile(artifactPath); err == nil {
|
||||
return string(data), nil
|
||||
}
|
||||
files, err := (emitter.NftEmitter{}).RenderNftConfig(emitter.NftClusterConfig{})
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return files[0].Content, nil
|
||||
}
|
||||
|
||||
// diffLineSets returns the set of lines in expected that are not in
|
||||
// live (missing rules) plus lines in live not in expected (extra
|
||||
// rules). Order-insensitive; whitespace-trimmed.
|
||||
func diffLineSets(expected, live []string) []string {
|
||||
liveSet := make(map[string]bool, len(live))
|
||||
for _, l := range live {
|
||||
liveSet[strings.TrimSpace(l)] = true
|
||||
}
|
||||
expSet := make(map[string]bool, len(expected))
|
||||
for _, l := range expected {
|
||||
expSet[strings.TrimSpace(l)] = true
|
||||
}
|
||||
var diff []string
|
||||
for _, l := range expected {
|
||||
t := strings.TrimSpace(l)
|
||||
if t == "" {
|
||||
continue
|
||||
}
|
||||
if !liveSet[t] {
|
||||
diff = append(diff, "- "+t)
|
||||
}
|
||||
}
|
||||
for _, l := range live {
|
||||
t := strings.TrimSpace(l)
|
||||
if t == "" {
|
||||
continue
|
||||
}
|
||||
if !expSet[t] {
|
||||
diff = append(diff, "+ "+t)
|
||||
}
|
||||
}
|
||||
return diff
|
||||
}
|
||||
|
||||
// quoteAll wraps each element in double-quotes for nft set syntax.
|
||||
func quoteAll(in []string) []string {
|
||||
out := make([]string, len(in))
|
||||
for i, s := range in {
|
||||
out[i] = fmt.Sprintf("%q", s)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
@@ -0,0 +1,252 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/emitter"
|
||||
)
|
||||
|
||||
func TestNftCmd_Registered(t *testing.T) {
|
||||
found := false
|
||||
for _, c := range rootCmd.Commands() {
|
||||
if c.Name() == "nft" {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Fatal("nft command not registered on root")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNftShowCmd(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
origTransport := nftTransportOverride
|
||||
origPeer := nftLeadPeerOverride
|
||||
defer func() {
|
||||
nftTransportOverride = origTransport
|
||||
nftLeadPeerOverride = origPeer
|
||||
}()
|
||||
nftLeadPeerOverride = "lead"
|
||||
want := "table inet orca-ingress { dnat }"
|
||||
nftTransportOverride = &mockNftTransport{
|
||||
execOut: map[string][]byte{
|
||||
"nft list table inet orca-ingress": []byte(want),
|
||||
},
|
||||
}
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"nft", "show"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("nft show: %v", err)
|
||||
}
|
||||
if !strings.Contains(buf.String(), want) {
|
||||
t.Errorf("nft show output missing %q:\n%s", want, buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestNftShowCmd_PeerFlag(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
origTransport := nftTransportOverride
|
||||
defer func() { nftTransportOverride = origTransport }()
|
||||
mt := &mockNftTransport{
|
||||
execOut: map[string][]byte{"nft list table inet orca-ingress": []byte("ok")},
|
||||
}
|
||||
nftTransportOverride = mt
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"nft", "show", "--peer", "node-2"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("nft show: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNftDiffCmd_NoDrift(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
origTransport := nftTransportOverride
|
||||
origPeer := nftLeadPeerOverride
|
||||
defer func() {
|
||||
nftTransportOverride = origTransport
|
||||
nftLeadPeerOverride = origPeer
|
||||
}()
|
||||
nftLeadPeerOverride = "lead"
|
||||
// Use the full default nft render so the diff detects no drift
|
||||
rendered, rerr := emitter.NftEmitter{}.RenderNftConfig(emitter.NftClusterConfig{})
|
||||
if rerr != nil {
|
||||
t.Fatalf("render: %v", rerr)
|
||||
}
|
||||
live := string(rendered[0].Content)
|
||||
nftTransportOverride = &mockNftTransport{
|
||||
execOut: map[string][]byte{
|
||||
"nft list table inet orca-ingress": []byte(live),
|
||||
},
|
||||
}
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"nft", "diff", "--against", "txn-123"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("nft diff: %v", err)
|
||||
}
|
||||
if !strings.Contains(buf.String(), "no drift") {
|
||||
t.Errorf("expected no drift, got:\n%s", buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestNftDiffCmd_RequiresAgainst(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"nft", "diff"})
|
||||
if err := rootCmd.Execute(); err == nil {
|
||||
t.Fatal("expected error for missing --against, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNftRateLimitSetCmd(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
origTransport := nftTransportOverride
|
||||
origPeer := nftLeadPeerOverride
|
||||
defer func() {
|
||||
nftTransportOverride = origTransport
|
||||
nftLeadPeerOverride = origPeer
|
||||
}()
|
||||
nftLeadPeerOverride = "lead"
|
||||
mt := &mockNftTransport{
|
||||
execOut: map[string][]byte{},
|
||||
}
|
||||
nftTransportOverride = mt
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"nft", "rate", "limit", "set", "--rate", "250"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("nft rate limit set: %v", err)
|
||||
}
|
||||
if !strings.Contains(buf.String(), "rate-limit set to 250/s burst 500") {
|
||||
t.Errorf("unexpected output:\n%s", buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestNftRateLimitSetCmd_RequiresRate(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"nft", "rate", "limit", "set"})
|
||||
if err := rootCmd.Execute(); err == nil {
|
||||
t.Fatal("expected error for missing --rate, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNftCountryBlockAddCmd(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
origTransport := nftTransportOverride
|
||||
origPeer := nftLeadPeerOverride
|
||||
defer func() {
|
||||
nftTransportOverride = origTransport
|
||||
nftLeadPeerOverride = origPeer
|
||||
}()
|
||||
nftLeadPeerOverride = "lead"
|
||||
mt := &mockNftTransport{execOut: map[string][]byte{}}
|
||||
nftTransportOverride = mt
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"nft", "country", "block", "add", "RU,CN"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("nft country block add: %v", err)
|
||||
}
|
||||
if !strings.Contains(buf.String(), "added 2 country code(s)") {
|
||||
t.Errorf("unexpected output:\n%s", buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestNftCountryBlockAddCmd_BadCode(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"nft", "country", "block", "add", "USA"})
|
||||
if err := rootCmd.Execute(); err == nil {
|
||||
t.Fatal("expected error for bad country code, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNftDoctorAliasCmd(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
origTransport := nftTransportOverride
|
||||
origPeer := nftLeadPeerOverride
|
||||
defer func() {
|
||||
nftTransportOverride = origTransport
|
||||
nftLeadPeerOverride = origPeer
|
||||
}()
|
||||
nftLeadPeerOverride = "lead"
|
||||
nftTransportOverride = &mockNftTransport{
|
||||
execOut: map[string][]byte{
|
||||
"nft list table inet orca-ingress": []byte("dnat to 127.0.0.1:8443 dnat to 127.0.0.1:8080 ora_rl"),
|
||||
},
|
||||
}
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"nft", "doctor"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("nft doctor: %v", err)
|
||||
}
|
||||
if !strings.Contains(buf.String(), "nft:table") {
|
||||
t.Errorf("nft doctor alias did not run doctor nft checks:\n%s", buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestNftShowCmd_ExecError(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
origTransport := nftTransportOverride
|
||||
origPeer := nftLeadPeerOverride
|
||||
defer func() {
|
||||
nftTransportOverride = origTransport
|
||||
nftLeadPeerOverride = origPeer
|
||||
}()
|
||||
nftLeadPeerOverride = "lead"
|
||||
nftTransportOverride = &mockNftTransport{
|
||||
execErr: map[string]error{
|
||||
"nft list table inet orca-ingress": errors.New("connection refused"),
|
||||
},
|
||||
}
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"nft", "show"})
|
||||
if err := rootCmd.Execute(); err == nil {
|
||||
t.Fatal("expected error from exec failure, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
var _ context.Context = context.Background()
|
||||
+44
-29
@@ -51,7 +51,7 @@ var (
|
||||
joinType string
|
||||
joinHost string
|
||||
joinSSHUser string
|
||||
joinPassword string
|
||||
joinSSHKey string
|
||||
joinSSHPort int
|
||||
joinHostKeyFP string
|
||||
proxmoxUser string
|
||||
@@ -75,7 +75,7 @@ Node types (via --type):
|
||||
localhost (default): register a local or Linux node (existing behavior)
|
||||
proxmox: SSH-bootstrap a remote Proxmox VE 8/9 host
|
||||
(deploys orca pubkey, creates orca user + PVE role +
|
||||
sudoers allowlist; requires --host + --password)`,
|
||||
sudoers allowlist; requires --host + --ssh-key (R-021: no passwords))`,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
if joinHostKeyFP != "" && joinType != "proxmox" {
|
||||
return fmt.Errorf("--host-key-fingerprint requires --type proxmox today")
|
||||
@@ -89,7 +89,13 @@ Node types (via --type):
|
||||
|
||||
// joinLocal is the existing localhost/Linux node join flow (fingerprint
|
||||
// check + registry.Insert).
|
||||
//
|
||||
// Deprecated: v0.9 re-architecture replaces daemon-to-daemon mTLS join
|
||||
// with SSH-push bootstrap (R-001). The mTLS join path is retained for
|
||||
// the dual-write window and scheduled for deletion in v0.10-P14. See
|
||||
// .ciagent/PRD_v0.9.md.
|
||||
func joinLocal(cmd *cobra.Command) error {
|
||||
warnDeprecated("orca node join (mTLS path): v0.9 R-001 replaces daemon-to-daemon mTLS join with SSH-push bootstrap; the mTLS join path is deprecated — see .ciagent/PRD_v0.9.md")
|
||||
if joinName == "" {
|
||||
return fmt.Errorf("--name is required")
|
||||
}
|
||||
@@ -142,18 +148,19 @@ func joinLocal(cmd *cobra.Command) error {
|
||||
}
|
||||
|
||||
// joinProxmox bootstraps a remote Proxmox VE 8/9 host via SSH and
|
||||
// registers it as an orca node (REQ-050, REQ-051). The password is
|
||||
// never persisted (D-031).
|
||||
// registers it as an orca node (REQ-050, REQ-051). Uses SSH key auth
|
||||
// (R-021: no passwords). The operator pre-stages the orca SSH public
|
||||
// key on the remote host out-of-band.
|
||||
func joinProxmox(cmd *cobra.Command) error {
|
||||
if joinHost == "" {
|
||||
return fmt.Errorf("--host is required for --type proxmox")
|
||||
}
|
||||
password := joinPassword
|
||||
if password == "" {
|
||||
password = os.Getenv("ORCA_PROXMOX_PASSWORD")
|
||||
sshKeyPath := joinSSHKey
|
||||
if sshKeyPath == "" {
|
||||
sshKeyPath = certpaths.SSHKeyPath()
|
||||
}
|
||||
if password == "" {
|
||||
return fmt.Errorf("password is required for --type proxmox (use --password or $ORCA_PROXMOX_PASSWORD)")
|
||||
if sshKeyPath == "" {
|
||||
return fmt.Errorf("SSH key path is required for --type proxmox (R-021: no passwords; use --ssh-key or pre-stage the orca key)")
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(cmd.Context(), 60*time.Second)
|
||||
@@ -162,7 +169,7 @@ func joinProxmox(cmd *cobra.Command) error {
|
||||
result, err := proxmox.BootstrapProxmox(ctx, proxmox.Options{
|
||||
Host: joinHost,
|
||||
SSHUser: joinSSHUser,
|
||||
Password: password,
|
||||
SSHKeyPath: sshKeyPath,
|
||||
ProxmoxUser: proxmoxUser,
|
||||
ProxmoxRole: proxmoxRole,
|
||||
SSHPort: joinSSHPort,
|
||||
@@ -173,12 +180,6 @@ func joinProxmox(cmd *cobra.Command) error {
|
||||
return fmt.Errorf("proxmox bootstrap: %w", err)
|
||||
}
|
||||
|
||||
// Zero the password byte slice (D-031 — never persist, minimize memory exposure).
|
||||
pwBytes := []byte(password)
|
||||
for i := range pwBytes {
|
||||
pwBytes[i] = 0
|
||||
}
|
||||
|
||||
// Register the proxmox node in the orca registry.
|
||||
registry, closer, err := nodeRegistry()
|
||||
if err != nil {
|
||||
@@ -251,6 +252,14 @@ var nodeListCmd = &cobra.Command{
|
||||
if nodeWatch {
|
||||
return watchNodes(cmd)
|
||||
}
|
||||
|
||||
// Cache (R-008): read path only; --watch bypasses. On hit,
|
||||
// unmarshal cached JSON and render without touching the DB.
|
||||
var cachedNodes []*model.Node
|
||||
if cacheGetList(cacheNodeClass, cacheListKey, &cachedNodes) {
|
||||
return renderNodes(cmd, cachedNodes)
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(cmd.Context(), 5*time.Second)
|
||||
defer cancel()
|
||||
|
||||
@@ -264,21 +273,27 @@ var nodeListCmd = &cobra.Command{
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if jsonOutput {
|
||||
return printJSON(nodes)
|
||||
}
|
||||
if len(nodes) == 0 {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "No nodes registered. Use 'orca node join' to add one.")
|
||||
return nil
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%-36s %-20s %-22s %-10s\n", "ID", "NAME", "ADDRESS", "STATE")
|
||||
for _, n := range nodes {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%-36s %-20s %-22s %-10s\n", n.ID, n.Name, n.Address, n.State)
|
||||
}
|
||||
return nil
|
||||
cachePutList(cacheNodeClass, cacheListKey, nodes, cacheNodeTTL)
|
||||
return renderNodes(cmd, nodes)
|
||||
},
|
||||
}
|
||||
|
||||
// renderNodes prints the node list in either JSON or table form.
|
||||
func renderNodes(cmd *cobra.Command, nodes []*model.Node) error {
|
||||
if jsonOutput {
|
||||
return printJSON(nodes)
|
||||
}
|
||||
if len(nodes) == 0 {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "No nodes registered. Use 'orca node join' to add one.")
|
||||
return nil
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%-36s %-20s %-22s %-10s\n", "ID", "NAME", "ADDRESS", "STATE")
|
||||
for _, n := range nodes {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%-36s %-20s %-22s %-10s\n", n.ID, n.Name, n.Address, n.State)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func watchNodes(cmd *cobra.Command) error {
|
||||
ctx, cancel := signal.NotifyContext(cmd.Context(), os.Interrupt, syscall.SIGTERM)
|
||||
defer cancel()
|
||||
@@ -411,7 +426,7 @@ func init() {
|
||||
nodeJoinCmd.Flags().StringVar(&joinType, "type", "localhost", "node type: localhost (default) or proxmox (SSH bootstrap)")
|
||||
nodeJoinCmd.Flags().StringVar(&joinHost, "host", "", "proxmox host address (IP/hostname, no port; required for --type proxmox)")
|
||||
nodeJoinCmd.Flags().StringVar(&joinSSHUser, "ssh-user", "root", "SSH username for proxmox bootstrap (default root)")
|
||||
nodeJoinCmd.Flags().StringVar(&joinPassword, "password", "", "SSH password for proxmox bootstrap (never persisted; prefer $ORCA_PROXMOX_PASSWORD)")
|
||||
nodeJoinCmd.Flags().StringVar(&joinSSHKey, "ssh-key", "", "SSH private key path for proxmox bootstrap (R-021: no passwords; default: orca key)")
|
||||
nodeJoinCmd.Flags().IntVar(&joinSSHPort, "ssh-port", 22, "SSH port for proxmox bootstrap (default 22)")
|
||||
nodeJoinCmd.Flags().StringVar(&proxmoxUser, "proxmox-user", "orca", "Linux system user to create on the proxmox host (config-overridable)")
|
||||
nodeJoinCmd.Flags().StringVar(&proxmoxRole, "proxmox-role", "OrcaOperator", "PVE custom role to create (config-overridable)")
|
||||
|
||||
@@ -154,22 +154,23 @@ func TestNodeJoinProxmoxMissingHost(t *testing.T) {
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"node", "join", "--type", "proxmox", "--password", "x"})
|
||||
rootCmd.SetArgs([]string{"node", "join", "--type", "proxmox", "--ssh-key", "/tmp/nonexistent-key"})
|
||||
if err := rootCmd.Execute(); err == nil {
|
||||
t.Fatal("expected error for proxmox without --host, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNodeJoinProxmoxMissingPassword(t *testing.T) {
|
||||
func TestNodeJoinProxmoxMissingSSHKey(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
// No --ssh-key and no default orca key -> error (R-021).
|
||||
rootCmd.SetArgs([]string{"node", "join", "--type", "proxmox", "--host", "10.0.0.99"})
|
||||
if err := rootCmd.Execute(); err == nil {
|
||||
t.Fatal("expected error for proxmox without password, got nil")
|
||||
t.Fatal("expected error for proxmox without ssh-key, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -473,7 +474,7 @@ func TestNodeJoinHostKeyFingerprintRequiresProxmox(t *testing.T) {
|
||||
// We can't run the full bootstrap without a real SSH server, so we
|
||||
// assert that the RunE check passes (no "requires --type proxmox"
|
||||
// error) and the failure — if any — comes from a later stage (missing
|
||||
// --host / password), not the D-044 guard.
|
||||
// --host / ssh-key), not the D-044 guard.
|
||||
func TestNodeJoinHostKeyFingerprintProxmoxAccepted(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
@@ -494,3 +495,21 @@ func TestNodeJoinHostKeyFingerprintProxmoxAccepted(t *testing.T) {
|
||||
t.Errorf("D-044 guard wrongly rejected proxmox type: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// --- REQ-146 / R-021 password removal regression test ---
|
||||
|
||||
// TestNodeJoinProxmoxPasswordRejected verifies the --password flag is
|
||||
// no longer accepted (R-021: no passwords). The flag is removed; the
|
||||
// CLI should reject it as an unknown flag.
|
||||
func TestNodeJoinProxmoxPasswordRejected(t *testing.T) {
|
||||
_, cleanup := initTestEnv(t)
|
||||
defer cleanup()
|
||||
resetRootFlags(t)
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetErr(&buf)
|
||||
rootCmd.SetArgs([]string{"node", "join", "--type", "proxmox", "--host", "10.0.0.99", "--password", "secret"})
|
||||
if err := rootCmd.Execute(); err == nil {
|
||||
t.Fatal("expected error for --password (R-021: no passwords), got nil")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,515 @@
|
||||
// Package cli: ns.go implements the `orca ns` subcommand family
|
||||
// (REQ-082, D-176). Subcommands:
|
||||
//
|
||||
// orca ns list — list all namespaces under ORCA_HOME
|
||||
// orca ns create <name> — create a namespace dir + ns.md
|
||||
// orca ns delete <name> — remove an empty namespace dir
|
||||
// orca ns inspect <name> — print effective chain + merged env
|
||||
// orca ns validate <name> — cycle + missing-parent + schema checks
|
||||
// orca ns inherit <name> — set the parent namespace (R-002)
|
||||
// orca ns set-constraint <name> <key>=<value> — set a constraint
|
||||
//
|
||||
// All subcommands honor $ORCA_HOME via internal/paths. The inheritance
|
||||
// resolver (internal/ns) is a pure function shared by inspect + validate.
|
||||
package cli
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/ns"
|
||||
"git.cloudinit.dev/coreci/orca/internal/paths"
|
||||
)
|
||||
|
||||
var nsCmd = &cobra.Command{
|
||||
Use: "ns",
|
||||
Short: "Manage orca namespaces",
|
||||
Long: `Manage orca namespaces under ORCA_HOME (R-002).
|
||||
|
||||
Each namespace is a directory with ns.md, .env, .env.secrets, db/,
|
||||
jobs/, alloc/. The implicit root namespace _defaults always exists
|
||||
(D-159); every namespace inherits from _defaults (D-185) and cannot
|
||||
opt out (D-187).`,
|
||||
}
|
||||
|
||||
var (
|
||||
nsCreateParent string
|
||||
nsCreateInheritsEnv bool
|
||||
nsCreateInheritsSecret bool
|
||||
)
|
||||
|
||||
var nsListCmd = &cobra.Command{
|
||||
Use: "list",
|
||||
Short: "List all namespaces under ORCA_HOME",
|
||||
Long: `List all namespaces under ORCA_HOME (directories containing ns.md, plus the implicit _defaults).`,
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
// Cache (R-008): read path only. ns list has no --watch flag.
|
||||
var cachedRows []nsRow
|
||||
if cacheGetList(cacheNamespaceClass, cacheListKey, &cachedRows) {
|
||||
return renderNSRows(cmd, cachedRows)
|
||||
}
|
||||
|
||||
root := paths.Root()
|
||||
entries, err := os.ReadDir(root)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read ORCA_HOME %s: %w", root, err)
|
||||
}
|
||||
var rows []nsRow
|
||||
for _, ent := range entries {
|
||||
if !ent.IsDir() {
|
||||
continue
|
||||
}
|
||||
if ent.Name() == "cluster" {
|
||||
continue
|
||||
}
|
||||
nsMd := filepath.Join(root, ent.Name(), "ns.md")
|
||||
if _, err := os.Stat(nsMd); err != nil {
|
||||
continue
|
||||
}
|
||||
rows = append(rows, nsRow{
|
||||
Name: ent.Name(),
|
||||
Path: filepath.Join(root, ent.Name()),
|
||||
Default: ent.Name() == paths.DefaultNamespace(),
|
||||
})
|
||||
}
|
||||
sort.Slice(rows, func(i, j int) bool {
|
||||
if rows[i].Name == paths.DefaultNamespace() {
|
||||
return true
|
||||
}
|
||||
if rows[j].Name == paths.DefaultNamespace() {
|
||||
return false
|
||||
}
|
||||
return rows[i].Name < rows[j].Name
|
||||
})
|
||||
cachePutList(cacheNamespaceClass, cacheListKey, rows, cacheNamespaceTTL)
|
||||
return renderNSRows(cmd, rows)
|
||||
},
|
||||
}
|
||||
|
||||
// nsRow is one row of `orca ns list` output (shared by the cached and
|
||||
// uncached read paths so the JSON tag set stays in one place).
|
||||
type nsRow struct {
|
||||
Name string `json:"name"`
|
||||
Path string `json:"path"`
|
||||
Default bool `json:"default"`
|
||||
}
|
||||
|
||||
// renderNSRows prints the namespace list in either JSON or table form.
|
||||
func renderNSRows(cmd *cobra.Command, rows []nsRow) error {
|
||||
if jsonOutput {
|
||||
return printJSON(rows)
|
||||
}
|
||||
if len(rows) == 0 {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "No namespaces found. Run 'orca init' first.")
|
||||
return nil
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%-20s %-10s %s\n", "NAME", "DEFAULT", "PATH")
|
||||
for _, r := range rows {
|
||||
def := ""
|
||||
if r.Default {
|
||||
def = "*"
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%-20s %-10s %s\n", r.Name, def, r.Path)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
var nsCreateCmd = &cobra.Command{
|
||||
Use: "create <name>",
|
||||
Short: "Create a namespace directory + ns.md",
|
||||
Long: `Create a namespace under ORCA_HOME. Builds the dir structure
|
||||
(db/, jobs/, alloc/) and writes ns.md frontmatter. --parent may be
|
||||
repeated to declare inheritance; _defaults is always appended last.`,
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
name := args[0]
|
||||
if err := ns.ValidateName(name); err != nil {
|
||||
return err
|
||||
}
|
||||
if name == paths.DefaultNamespace() {
|
||||
return fmt.Errorf("cannot create the implicit root namespace %q with `ns create` (it is auto-managed)", name)
|
||||
}
|
||||
if nsCreateParent == "" {
|
||||
nsCreateParent = paths.DefaultNamespace()
|
||||
}
|
||||
nsDir := paths.NamespaceDir(name)
|
||||
if _, err := os.Stat(nsDir); err == nil {
|
||||
if _, statErr := os.Stat(paths.NSMd(name)); statErr == nil {
|
||||
return fmt.Errorf("namespace %q already exists at %s", name, nsDir)
|
||||
}
|
||||
}
|
||||
for _, sub := range []string{"db", "jobs", "alloc"} {
|
||||
if err := os.MkdirAll(filepath.Join(nsDir, sub), 0o755); err != nil {
|
||||
return fmt.Errorf("create %s/%s: %w", nsDir, sub, err)
|
||||
}
|
||||
}
|
||||
parents := []string{nsCreateParent}
|
||||
if nsCreateParent == paths.DefaultNamespace() {
|
||||
// Explicit _defaults listing is allowed (de-duped silently).
|
||||
}
|
||||
body := renderNSMd(name, parents, nsCreateInheritsEnv, nsCreateInheritsSecret)
|
||||
if err := os.WriteFile(paths.NSMd(name), []byte(body), 0o644); err != nil {
|
||||
return fmt.Errorf("write ns.md: %w", err)
|
||||
}
|
||||
if jsonOutput {
|
||||
return printJSON(map[string]any{
|
||||
"name": name,
|
||||
"path": nsDir,
|
||||
"parents": parents,
|
||||
"ns_md": paths.NSMd(name),
|
||||
"inherits_env": nsCreateInheritsEnv,
|
||||
"inherits_secrets": nsCreateInheritsSecret,
|
||||
})
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "✓ Namespace created: %s (%s)\n", name, nsDir)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
var nsDeleteCmd = &cobra.Command{
|
||||
Use: "delete <name>",
|
||||
Short: "Remove an empty namespace directory",
|
||||
Long: `Remove a namespace directory. Refuses if jobs/ or alloc/
|
||||
contain any files (non-empty namespace). The implicit root _defaults
|
||||
cannot be deleted.`,
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
name := args[0]
|
||||
if err := ns.ValidateName(name); err != nil {
|
||||
return err
|
||||
}
|
||||
if name == paths.DefaultNamespace() {
|
||||
return fmt.Errorf("cannot delete the implicit root namespace %q", name)
|
||||
}
|
||||
nsDir := paths.NamespaceDir(name)
|
||||
if _, err := os.Stat(nsDir); err != nil {
|
||||
return fmt.Errorf("namespace %q not found: %w", name, err)
|
||||
}
|
||||
for _, sub := range []string{"jobs", "alloc"} {
|
||||
dir := filepath.Join(nsDir, sub)
|
||||
if err := dirNonEmpty(dir); err != nil {
|
||||
return fmt.Errorf("refusing to delete %q: %s is non-empty (%w); clear it first", name, sub, err)
|
||||
}
|
||||
}
|
||||
if err := os.RemoveAll(nsDir); err != nil {
|
||||
return fmt.Errorf("delete %s: %w", nsDir, err)
|
||||
}
|
||||
if jsonOutput {
|
||||
return printJSON(map[string]string{"name": name, "deleted": nsDir})
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "✓ Namespace deleted: %s (%s)\n", name, nsDir)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
var nsInspectCmd = &cobra.Command{
|
||||
Use: "inspect <name>",
|
||||
Short: "Print the effective chain, merged env, and constraints",
|
||||
Long: `Resolve a namespace's inheritance chain and print the merged env and unioned constraints (uses the resolver).`,
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
name := args[0]
|
||||
if err := ns.ValidateName(name); err != nil {
|
||||
return err
|
||||
}
|
||||
root := paths.Root()
|
||||
cfgs, err := ns.ParseNSMdDir(root)
|
||||
if err != nil {
|
||||
return fmt.Errorf("load namespaces: %w", err)
|
||||
}
|
||||
if _, ok := cfgs[name]; !ok {
|
||||
return fmt.Errorf("namespace %q not found under %s", name, root)
|
||||
}
|
||||
resolved, err := ns.Resolve(cfgs)
|
||||
if err != nil {
|
||||
return fmt.Errorf("resolve: %w", err)
|
||||
}
|
||||
r := resolved[name]
|
||||
if r == nil {
|
||||
return fmt.Errorf("namespace %q resolved to nil", name)
|
||||
}
|
||||
if jsonOutput {
|
||||
return printJSON(map[string]any{
|
||||
"name": r.Name,
|
||||
"chain": r.Chain,
|
||||
"env": r.Env,
|
||||
"constraints": r.Constraints,
|
||||
})
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "Namespace: %s\n", r.Name)
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "Chain: %s\n", strings.Join(r.Chain, " -> "))
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "Env:")
|
||||
keys := sortedKeys(r.Env)
|
||||
for _, k := range keys {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), " %s = %s\n", k, r.Env[k])
|
||||
}
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "Constraints:")
|
||||
if len(r.Constraints) == 0 {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), " (none)")
|
||||
} else {
|
||||
for _, c := range r.Constraints {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), " - %s\n", c)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
var nsValidateCmd = &cobra.Command{
|
||||
Use: "validate <name>",
|
||||
Short: "Run cycle + missing-parent + schema checks on a namespace",
|
||||
Long: `Validate a namespace's inheritance chain and ns.md frontmatter.
|
||||
Exits 0 if valid, 1 on error. Runs over ALL namespaces under ORCA_HOME
|
||||
(parsing + resolving validates cycles and missing parents across the
|
||||
set).`,
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
name := args[0]
|
||||
if err := ns.ValidateName(name); err != nil {
|
||||
return err
|
||||
}
|
||||
root := paths.Root()
|
||||
cfgs, err := ns.ParseNSMdDir(root)
|
||||
if err != nil {
|
||||
return fmt.Errorf("load namespaces: %w", err)
|
||||
}
|
||||
if _, ok := cfgs[name]; !ok {
|
||||
return fmt.Errorf("namespace %q not found under %s", name, root)
|
||||
}
|
||||
resolved, err := ns.Resolve(cfgs)
|
||||
if err != nil {
|
||||
return fmt.Errorf("validate: %w", err)
|
||||
}
|
||||
r := resolved[name]
|
||||
if r == nil {
|
||||
return fmt.Errorf("namespace %q resolved to nil", name)
|
||||
}
|
||||
if jsonOutput {
|
||||
return printJSON(map[string]any{
|
||||
"name": r.Name,
|
||||
"valid": true,
|
||||
"chain": r.Chain,
|
||||
})
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "✓ %s valid\n chain: %s\n", name, strings.Join(r.Chain, " -> "))
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
var (
|
||||
nsInheritParent string
|
||||
)
|
||||
|
||||
var nsInheritCmd = &cobra.Command{
|
||||
Use: "inherit <name>",
|
||||
Short: "Set the parent namespace for inheritance (R-002)",
|
||||
Long: `Set the parent namespace for a namespace. Updates ns.md
|
||||
frontmatter (parents) and validates the new chain has no cycles
|
||||
(child cannot inherit from itself transitively). The implicit root
|
||||
_defaults is always appended last (D-185).`,
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
name := args[0]
|
||||
if err := ns.ValidateName(name); err != nil {
|
||||
return err
|
||||
}
|
||||
if name == paths.DefaultNamespace() {
|
||||
return fmt.Errorf("cannot set parent on the implicit root namespace %q", name)
|
||||
}
|
||||
if nsInheritParent == "" {
|
||||
return fmt.Errorf("--parent is required")
|
||||
}
|
||||
if err := ns.ValidateName(nsInheritParent); err != nil {
|
||||
return fmt.Errorf("--parent: %w", err)
|
||||
}
|
||||
if nsInheritParent == name {
|
||||
return fmt.Errorf("namespace %q cannot inherit from itself", name)
|
||||
}
|
||||
nsMd := paths.NSMd(name)
|
||||
cfg, nsBody, err := ns.ParseNSMdWithBody(nsMd)
|
||||
if err != nil {
|
||||
return fmt.Errorf("parse %s: %w", nsMd, err)
|
||||
}
|
||||
cfg.Parents = []string{nsInheritParent}
|
||||
|
||||
root := paths.Root()
|
||||
cfgs, err := ns.ParseNSMdDir(root)
|
||||
if err != nil {
|
||||
return fmt.Errorf("load namespaces: %w", err)
|
||||
}
|
||||
cfgs[name] = cfg
|
||||
if _, err := ns.Resolve(cfgs); err != nil {
|
||||
return fmt.Errorf("cycle check: %w", err)
|
||||
}
|
||||
|
||||
body := renderNSMdFull(cfg, nsBody)
|
||||
if err := os.WriteFile(nsMd, []byte(body), 0o644); err != nil {
|
||||
return fmt.Errorf("write %s: %w", nsMd, err)
|
||||
}
|
||||
if jsonOutput {
|
||||
return printJSON(map[string]any{
|
||||
"name": name,
|
||||
"parents": cfg.Parents,
|
||||
"ns_md": nsMd,
|
||||
})
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "\u2713 Namespace %s now inherits from %s\n", name, nsInheritParent)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
var nsSetConstraintCmd = &cobra.Command{
|
||||
Use: "set-constraint <name> <key>=<value>",
|
||||
Short: "Set a constraint on a namespace (stored in ns.md frontmatter)",
|
||||
Long: `Set a constraint on a namespace. Constraints are key=value
|
||||
strings (e.g. max-allocs=10) stored in ns.md frontmatter and unioned
|
||||
across the inheritance chain by the resolver.`,
|
||||
Args: cobra.ExactArgs(2),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
name := args[0]
|
||||
if err := ns.ValidateName(name); err != nil {
|
||||
return err
|
||||
}
|
||||
kv := args[1]
|
||||
if name == paths.DefaultNamespace() {
|
||||
return fmt.Errorf("cannot set a constraint on the implicit root namespace %q with set-constraint; edit ns.md directly", name)
|
||||
}
|
||||
idx := strings.Index(kv, "=")
|
||||
if idx <= 0 || idx == len(kv)-1 {
|
||||
return fmt.Errorf("constraint must be <key>=<value>, got %q", kv)
|
||||
}
|
||||
constraint := kv
|
||||
nsMd := paths.NSMd(name)
|
||||
cfg, nsBody, err := ns.ParseNSMdWithBody(nsMd)
|
||||
if err != nil {
|
||||
return fmt.Errorf("parse %s: %w", nsMd, err)
|
||||
}
|
||||
for _, c := range cfg.Constraints {
|
||||
if c == constraint {
|
||||
return fmt.Errorf("constraint %q already set on namespace %q", constraint, name)
|
||||
}
|
||||
}
|
||||
cfg.Constraints = append(cfg.Constraints, constraint)
|
||||
|
||||
body := renderNSMdFull(cfg, nsBody)
|
||||
if err := os.WriteFile(nsMd, []byte(body), 0o644); err != nil {
|
||||
return fmt.Errorf("write %s: %w", nsMd, err)
|
||||
}
|
||||
if jsonOutput {
|
||||
return printJSON(map[string]any{
|
||||
"name": name,
|
||||
"constraints": cfg.Constraints,
|
||||
"ns_md": nsMd,
|
||||
})
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "\u2713 Constraint set on %s: %s\n", name, constraint)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
// renderNSMdFull renders a complete ns.md from a parsed *ns.NSConfig
|
||||
// plus an optional body (the markdown after the frontmatter). Used by
|
||||
// the ns inherit / set-constraint editors to rewrite frontmatter while
|
||||
// preserving the body.
|
||||
func renderNSMdFull(cfg *ns.NSConfig, body string) string {
|
||||
var b strings.Builder
|
||||
b.WriteString("---\n")
|
||||
b.WriteString("kind: Namespace\n")
|
||||
b.WriteString("name: ")
|
||||
b.WriteString(cfg.Name)
|
||||
b.WriteString("\n")
|
||||
if len(cfg.Parents) > 0 {
|
||||
quoted := make([]string, len(cfg.Parents))
|
||||
for i, p := range cfg.Parents {
|
||||
quoted[i] = fmt.Sprintf("%q", p)
|
||||
}
|
||||
b.WriteString("parents: [")
|
||||
b.WriteString(strings.Join(quoted, ", "))
|
||||
b.WriteString("]\n")
|
||||
}
|
||||
fmt.Fprintf(&b, "inherits_env: %t\n", cfg.InheritsEnv)
|
||||
fmt.Fprintf(&b, "inherits_secrets: %t\n", cfg.InheritsSecrets)
|
||||
if len(cfg.Constraints) > 0 {
|
||||
quoted := make([]string, len(cfg.Constraints))
|
||||
for i, c := range cfg.Constraints {
|
||||
quoted[i] = fmt.Sprintf("%q", c)
|
||||
}
|
||||
b.WriteString("constraints: [")
|
||||
b.WriteString(strings.Join(quoted, ", "))
|
||||
b.WriteString("]\n")
|
||||
}
|
||||
b.WriteString("---\n")
|
||||
b.WriteString(body)
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// renderNSMd writes a minimal ns.md frontmatter for `orca ns create`.
|
||||
func renderNSMd(name string, parents []string, inheritsEnv, inheritsSecrets bool) string {
|
||||
var b strings.Builder
|
||||
b.WriteString("---\n")
|
||||
b.WriteString("kind: Namespace\n")
|
||||
b.WriteString("name: ")
|
||||
b.WriteString(name)
|
||||
b.WriteString("\n")
|
||||
if len(parents) > 0 {
|
||||
quoted := make([]string, len(parents))
|
||||
for i, p := range parents {
|
||||
quoted[i] = fmt.Sprintf("%q", p)
|
||||
}
|
||||
b.WriteString("parents: [")
|
||||
b.WriteString(strings.Join(quoted, ", "))
|
||||
b.WriteString("]\n")
|
||||
}
|
||||
fmt.Fprintf(&b, "inherits_env: %t\n", inheritsEnv)
|
||||
fmt.Fprintf(&b, "inherits_secrets: %t\n", inheritsSecrets)
|
||||
b.WriteString("---\n")
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// dirNonEmpty returns an error wrapping the offending entry if dir
|
||||
// contains any entries.
|
||||
func dirNonEmpty(dir string) error {
|
||||
entries, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
for _, e := range entries {
|
||||
return fmt.Errorf("contains %s", e.Name())
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func sortedKeys(m map[string]string) []string {
|
||||
out := make([]string, 0, len(m))
|
||||
for k := range m {
|
||||
out = append(out, k)
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
func init() {
|
||||
nsCreateCmd.Flags().StringVar(&nsCreateParent, "parent", "", "parent namespace (default _defaults; the implicit root is always appended last)")
|
||||
nsCreateCmd.Flags().BoolVar(&nsCreateInheritsEnv, "inherits-env", true, "inherit env from parents (default true)")
|
||||
nsCreateCmd.Flags().BoolVar(&nsCreateInheritsSecret, "inherits-secrets", true, "inherit secrets from parents (default true)")
|
||||
|
||||
nsInheritCmd.Flags().StringVar(&nsInheritParent, "parent", "", "parent namespace to inherit from (required)")
|
||||
|
||||
nsCmd.AddCommand(nsListCmd)
|
||||
nsCmd.AddCommand(nsCreateCmd)
|
||||
nsCmd.AddCommand(nsDeleteCmd)
|
||||
nsCmd.AddCommand(nsInspectCmd)
|
||||
nsCmd.AddCommand(nsValidateCmd)
|
||||
nsCmd.AddCommand(nsInheritCmd)
|
||||
nsCmd.AddCommand(nsSetConstraintCmd)
|
||||
rootCmd.AddCommand(nsCmd)
|
||||
}
|
||||
@@ -0,0 +1,661 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/paths"
|
||||
)
|
||||
|
||||
// resetNSFlags zeroes the ns subcommand flag-bound vars so tests don't
|
||||
// leak state.
|
||||
func resetNSFlags() {
|
||||
nsCreateParent = ""
|
||||
nsCreateInheritsEnv = true
|
||||
nsCreateInheritsSecret = true
|
||||
nsInheritParent = ""
|
||||
}
|
||||
|
||||
func writeDefaultsNS(t *testing.T, root string) {
|
||||
t.Helper()
|
||||
nsDir := filepath.Join(root, "_defaults")
|
||||
if err := os.MkdirAll(nsDir, 0o755); err != nil {
|
||||
t.Fatalf("mkdir _defaults: %v", err)
|
||||
}
|
||||
body := "---\nkind: Namespace\nname: _defaults\ninherits_env: true\ninherits_secrets: true\n---\n# defaults\n"
|
||||
if err := os.WriteFile(filepath.Join(nsDir, "ns.md"), []byte(body), 0o644); err != nil {
|
||||
t.Fatalf("write _defaults ns.md: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func writeCustomNS(t *testing.T, root, name, parentsList string) {
|
||||
t.Helper()
|
||||
nsDir := filepath.Join(root, name)
|
||||
if err := os.MkdirAll(nsDir, 0o755); err != nil {
|
||||
t.Fatalf("mkdir %s: %v", name, err)
|
||||
}
|
||||
body := "---\nkind: Namespace\nname: " + name + "\n"
|
||||
if parentsList != "" {
|
||||
body += "parents: " + parentsList + "\n"
|
||||
}
|
||||
body += "inherits_env: true\ninherits_secrets: true\n---\n# " + name + "\n"
|
||||
if err := os.WriteFile(filepath.Join(nsDir, "ns.md"), []byte(body), 0o644); err != nil {
|
||||
t.Fatalf("write %s ns.md: %v", name, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNSListEmpty(t *testing.T) {
|
||||
t.Setenv("ORCA_HOME", t.TempDir())
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
|
||||
rootCmd.SetArgs([]string{"ns", "list"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("ns list: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNSListWithNamespaces(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
writeDefaultsNS(t, root)
|
||||
writeCustomNS(t, root, "prod", "")
|
||||
|
||||
rootCmd.SetArgs([]string{"ns", "list"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("ns list: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNSCreateHappy(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
|
||||
rootCmd.SetArgs([]string{"ns", "create", "prod"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("ns create: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(root, "prod", "ns.md")); err != nil {
|
||||
t.Fatalf("ns.md not created: %v", err)
|
||||
}
|
||||
for _, sub := range []string{"db", "jobs", "alloc"} {
|
||||
if _, err := os.Stat(filepath.Join(root, "prod", sub)); err != nil {
|
||||
t.Errorf("subdir %s not created: %v", sub, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestNSCreateWithParent(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
writeDefaultsNS(t, root)
|
||||
writeCustomNS(t, root, "base", "")
|
||||
|
||||
rootCmd.SetArgs([]string{"ns", "create", "child", "--parent", "base"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("ns create: %v", err)
|
||||
}
|
||||
data, err := os.ReadFile(filepath.Join(root, "child", "ns.md"))
|
||||
if err != nil {
|
||||
t.Fatalf("read ns.md: %v", err)
|
||||
}
|
||||
if !strings.Contains(string(data), "parents: [") || !strings.Contains(string(data), "\"base\"") {
|
||||
t.Errorf("ns.md missing parents: %s", string(data))
|
||||
}
|
||||
}
|
||||
|
||||
func TestNSCreateExisting(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
writeDefaultsNS(t, root)
|
||||
writeCustomNS(t, root, "prod", "")
|
||||
|
||||
rootCmd.SetArgs([]string{"ns", "create", "prod"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected error creating existing namespace, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "already exists") {
|
||||
t.Errorf("error = %q, want contains 'already exists'", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestNSCreateDefaultsRefused(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
|
||||
rootCmd.SetArgs([]string{"ns", "create", "_defaults"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected error creating _defaults, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNSCreateClusterRefused(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
|
||||
rootCmd.SetArgs([]string{"ns", "create", "cluster"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected error creating cluster, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNSDeleteHappy(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
writeDefaultsNS(t, root)
|
||||
writeCustomNS(t, root, "prod", "")
|
||||
|
||||
rootCmd.SetArgs([]string{"ns", "delete", "prod"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("ns delete: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(root, "prod")); !os.IsNotExist(err) {
|
||||
t.Errorf("prod dir still exists after delete")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNSDeleteDefaultsRefused(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
writeDefaultsNS(t, root)
|
||||
|
||||
rootCmd.SetArgs([]string{"ns", "delete", "_defaults"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected error deleting _defaults, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNSDeleteNonEmpty(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
writeDefaultsNS(t, root)
|
||||
writeCustomNS(t, root, "prod", "")
|
||||
// put a job in jobs/
|
||||
if err := os.MkdirAll(filepath.Join(root, "prod", "jobs"), 0o755); err != nil {
|
||||
t.Fatalf("mkdir jobs: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(root, "prod", "jobs", "j1.md"), []byte("x"), 0o644); err != nil {
|
||||
t.Fatalf("write job: %v", err)
|
||||
}
|
||||
|
||||
rootCmd.SetArgs([]string{"ns", "delete", "prod"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected error deleting non-empty namespace, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "refusing to delete") {
|
||||
t.Errorf("error = %q, want contains 'refusing to delete'", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestNSDeleteMissing(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
writeDefaultsNS(t, root)
|
||||
|
||||
rootCmd.SetArgs([]string{"ns", "delete", "ghost"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected error deleting missing namespace, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNSInspectHappy(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
writeDefaultsNS(t, root)
|
||||
writeCustomNS(t, root, "prod", "")
|
||||
|
||||
rootCmd.SetArgs([]string{"ns", "inspect", "prod"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("ns inspect: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNSInspectJSON(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
writeDefaultsNS(t, root)
|
||||
writeCustomNS(t, root, "prod", "")
|
||||
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetArgs([]string{"ns", "inspect", "prod", "--json"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("ns inspect --json: %v", err)
|
||||
}
|
||||
var result map[string]any
|
||||
if err := json.Unmarshal(bytes.TrimSpace(buf.Bytes()), &result); err != nil {
|
||||
t.Fatalf("unmarshal: %v\n%s", err, buf.String())
|
||||
}
|
||||
if result["name"] != "prod" {
|
||||
t.Errorf("name = %v, want prod", result["name"])
|
||||
}
|
||||
chain, _ := result["chain"].([]any)
|
||||
if len(chain) < 2 {
|
||||
t.Errorf("chain too short: %v", chain)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNSInspectMissingNamespace(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
writeDefaultsNS(t, root)
|
||||
|
||||
rootCmd.SetArgs([]string{"ns", "inspect", "ghost"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected error for missing namespace, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNSValidateHappy(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
writeDefaultsNS(t, root)
|
||||
writeCustomNS(t, root, "prod", "")
|
||||
|
||||
rootCmd.SetArgs([]string{"ns", "validate", "prod"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("ns validate: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNSValidateCycle(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
writeDefaultsNS(t, root)
|
||||
// a -> b, b -> a (cycle)
|
||||
writeCustomNS(t, root, "a", "[\"b\"]")
|
||||
writeCustomNS(t, root, "b", "[\"a\"]")
|
||||
|
||||
rootCmd.SetArgs([]string{"ns", "validate", "a"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected cycle error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "cycle") {
|
||||
t.Errorf("error = %q, want contains 'cycle'", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestNSValidateMissingParent(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
writeDefaultsNS(t, root)
|
||||
writeCustomNS(t, root, "x", "[\"ghost\"]")
|
||||
|
||||
rootCmd.SetArgs([]string{"ns", "validate", "x"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected missing-parent error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "ghost") || !strings.Contains(err.Error(), "not found") {
|
||||
t.Errorf("error = %q, want contains ghost + not found", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestNSValidateMissingNamespace(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
writeDefaultsNS(t, root)
|
||||
|
||||
rootCmd.SetArgs([]string{"ns", "validate", "ghost"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected error for missing namespace, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenderNSMd(t *testing.T) {
|
||||
body := renderNSMd("foo", []string{"_defaults"}, true, false)
|
||||
if !strings.Contains(body, "kind: Namespace") {
|
||||
t.Errorf("missing kind: %s", body)
|
||||
}
|
||||
if !strings.Contains(body, "name: foo") {
|
||||
t.Errorf("missing name: %s", body)
|
||||
}
|
||||
if !strings.Contains(body, "inherits_env: true") {
|
||||
t.Errorf("missing inherits_env true: %s", body)
|
||||
}
|
||||
if !strings.Contains(body, "inherits_secrets: false") {
|
||||
t.Errorf("missing inherits_secrets false: %s", body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNSRootRegistered(t *testing.T) {
|
||||
found := false
|
||||
for _, c := range rootCmd.Commands() {
|
||||
if c.Use == "ns" {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Errorf("ns command not registered on root")
|
||||
}
|
||||
// ensure subcommands present
|
||||
sub := map[string]bool{}
|
||||
for _, c := range rootCmd.Commands() {
|
||||
if c.Use == "ns" {
|
||||
for _, sc := range c.Commands() {
|
||||
sub[sc.Use] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, want := range []string{"list", "create <name>", "delete <name>", "inspect <name>", "validate <name>", "inherit <name>", "set-constraint <name> <key>=<value>"} {
|
||||
if !sub[want] {
|
||||
t.Errorf("missing ns subcommand %q", want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestNSListNoORCAHOME(t *testing.T) {
|
||||
// ORCA_HOME points at a nonexistent dir; list should error.
|
||||
t.Setenv("ORCA_HOME", filepath.Join(t.TempDir(), "nope"))
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
|
||||
rootCmd.SetArgs([]string{"ns", "list"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected error for missing ORCA_HOME, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
var _ = paths.DefaultNamespace // keep paths import alive
|
||||
|
||||
func TestNSInheritSetsParent(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
writeDefaultsNS(t, root)
|
||||
writeCustomNS(t, root, "prod", "")
|
||||
|
||||
rootCmd.SetArgs([]string{"ns", "inherit", "prod", "--parent", "_defaults"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("ns inherit: %v", err)
|
||||
}
|
||||
data, err := os.ReadFile(filepath.Join(root, "prod", "ns.md"))
|
||||
if err != nil {
|
||||
t.Fatalf("read ns.md: %v", err)
|
||||
}
|
||||
if !strings.Contains(string(data), "parents:") || !strings.Contains(string(data), "_defaults") {
|
||||
t.Errorf("ns.md missing parents: %s", string(data))
|
||||
}
|
||||
}
|
||||
|
||||
func TestNSInheritCycleRefused(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
writeDefaultsNS(t, root)
|
||||
// a -> b already; now set a's parent to b, then try b -> a.
|
||||
writeCustomNS(t, root, "a", `["b"]`)
|
||||
writeCustomNS(t, root, "b", "")
|
||||
|
||||
resetNSFlags()
|
||||
rootCmd.SetArgs([]string{"ns", "inherit", "b", "--parent", "a"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected cycle error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "cycle") {
|
||||
t.Errorf("error = %q, want contains 'cycle'", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestNSInheritSelfRefused(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
writeDefaultsNS(t, root)
|
||||
writeCustomNS(t, root, "prod", "")
|
||||
|
||||
rootCmd.SetArgs([]string{"ns", "inherit", "prod", "--parent", "prod"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected self-inherit error, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNSInheritDefaultsRefused(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
writeDefaultsNS(t, root)
|
||||
writeCustomNS(t, root, "prod", "")
|
||||
|
||||
rootCmd.SetArgs([]string{"ns", "inherit", "_defaults", "--parent", "prod"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected error setting parent on _defaults, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNSSetConstraint(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
writeDefaultsNS(t, root)
|
||||
writeCustomNS(t, root, "prod", "")
|
||||
|
||||
rootCmd.SetArgs([]string{"ns", "set-constraint", "prod", "max-allocs=10"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("ns set-constraint: %v", err)
|
||||
}
|
||||
data, err := os.ReadFile(filepath.Join(root, "prod", "ns.md"))
|
||||
if err != nil {
|
||||
t.Fatalf("read ns.md: %v", err)
|
||||
}
|
||||
if !strings.Contains(string(data), "constraints:") || !strings.Contains(string(data), "max-allocs=10") {
|
||||
t.Errorf("ns.md missing constraints: %s", string(data))
|
||||
}
|
||||
}
|
||||
|
||||
func TestNSSetConstraintValidatePasses(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
writeDefaultsNS(t, root)
|
||||
writeCustomNS(t, root, "prod", "")
|
||||
|
||||
rootCmd.SetArgs([]string{"ns", "set-constraint", "prod", "max-allocs=10"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("ns set-constraint: %v", err)
|
||||
}
|
||||
resetNSFlags()
|
||||
rootCmd.SetArgs([]string{"ns", "validate", "prod"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("ns validate after set-constraint: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNSSetConstraintInspectShowsConstraint(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
writeDefaultsNS(t, root)
|
||||
writeCustomNS(t, root, "prod", "")
|
||||
|
||||
rootCmd.SetArgs([]string{"ns", "set-constraint", "prod", "max-allocs=10"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("ns set-constraint: %v", err)
|
||||
}
|
||||
|
||||
resetNSFlags()
|
||||
var buf bytes.Buffer
|
||||
rootCmd.SetOut(&buf)
|
||||
rootCmd.SetArgs([]string{"ns", "inspect", "prod", "--json"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("ns inspect --json: %v", err)
|
||||
}
|
||||
var result map[string]any
|
||||
if err := json.Unmarshal(bytes.TrimSpace(buf.Bytes()), &result); err != nil {
|
||||
t.Fatalf("unmarshal: %v\n%s", err, buf.String())
|
||||
}
|
||||
cons, _ := result["constraints"].([]any)
|
||||
found := false
|
||||
for _, c := range cons {
|
||||
if c == "max-allocs=10" {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Errorf("constraint max-allocs=10 not in inspect output: %v", cons)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNSSetConstraintInvalidFormat(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
writeDefaultsNS(t, root)
|
||||
writeCustomNS(t, root, "prod", "")
|
||||
|
||||
rootCmd.SetArgs([]string{"ns", "set-constraint", "prod", "noequals"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected error for malformed constraint, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "<key>=<value>") {
|
||||
t.Errorf("error = %q, want contains '<key>=<value>'", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestNSSetConstraintDuplicate(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
writeDefaultsNS(t, root)
|
||||
writeCustomNS(t, root, "prod", "")
|
||||
|
||||
rootCmd.SetArgs([]string{"ns", "set-constraint", "prod", "max-allocs=10"})
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("ns set-constraint first: %v", err)
|
||||
}
|
||||
resetNSFlags()
|
||||
rootCmd.SetArgs([]string{"ns", "set-constraint", "prod", "max-allocs=10"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected duplicate error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "already set") {
|
||||
t.Errorf("error = %q, want contains 'already set'", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
// --- REQ-120 / F4 path traversal regression tests ---
|
||||
|
||||
// TestNSCreateTraversalRefused verifies that ns create rejects names
|
||||
// that would traverse outside ORCA_HOME via ".." or "/".
|
||||
func TestNSCreateTraversalRefused(t *testing.T) {
|
||||
bad := []string{
|
||||
"..",
|
||||
"../etc",
|
||||
"foo/../bar",
|
||||
"/etc",
|
||||
"etc/",
|
||||
"foo/bar",
|
||||
"-x",
|
||||
"--flag",
|
||||
"with space",
|
||||
"tab\there",
|
||||
"newline\nname",
|
||||
}
|
||||
for _, name := range bad {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
rootCmd.SetArgs([]string{"ns", "create", name})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Errorf("ns create %q should fail, got nil", name)
|
||||
}
|
||||
// Verify no directory was created outside ORCA_HOME.
|
||||
// For ".." and "../etc", the danger is a dir was created
|
||||
// outside root. Check root's parent has no new orca dirs.
|
||||
parent := filepath.Dir(root)
|
||||
entries, _ := os.ReadDir(parent)
|
||||
for _, e := range entries {
|
||||
// The temp dir itself is fine; anything else that looks
|
||||
// like an orca namespace (has ns.md) outside root is a
|
||||
// leak.
|
||||
if e.Name() == filepath.Base(root) {
|
||||
continue
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(parent, e.Name(), "ns.md")); err == nil {
|
||||
t.Errorf("namespace dir leaked outside ORCA_HOME: %s", filepath.Join(parent, e.Name()))
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestNSInheritTraversalRefused verifies --parent rejects traversal.
|
||||
func TestNSInheritTraversalRefused(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("ORCA_HOME", root)
|
||||
resetRootFlags(t)
|
||||
resetNSFlags()
|
||||
writeDefaultsNS(t, root)
|
||||
writeCustomNS(t, root, "prod", "")
|
||||
rootCmd.SetArgs([]string{"ns", "inherit", "prod", "--parent", "../../etc"})
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("ns inherit with traversal --parent should fail")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,136 @@
|
||||
// Package cli: peer_setup.go implements the orca system-user setup and
|
||||
// NFS detection on peers (P10b-T8/T9, v0.11, REQ-111, REQ-112/D-233).
|
||||
//
|
||||
// `orca node join` now also creates the `orca` system user on the peer
|
||||
// (so the systemd Path-unit services, which run as User=orca, have a
|
||||
// uid to run as). It also detects whether /etc/orca is on an NFS mount
|
||||
// and, when it is, skips emitting Path units for paths under /etc/orca
|
||||
// (falling back to polling for those paths — systemd Path units on NFS
|
||||
// are unreliable because inotify does not fire reliably over NFS).
|
||||
//
|
||||
// The setup is idempotent: re-running on an already-configured peer is
|
||||
// a no-op. A --no-orca-user flag skips user creation (for environments
|
||||
// with existing service accounts).
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
var peerSetupNoOrcaUser bool
|
||||
|
||||
// peerSetupTransport is the SSH surface the peer-setup code needs. It
|
||||
// mirrors driftTransport; tests substitute a mock.
|
||||
type peerSetupTransport interface {
|
||||
Exec(ctx context.Context, peer string, cmd string) ([]byte, error)
|
||||
}
|
||||
|
||||
// peerSetupTransportOverride is the test seam.
|
||||
var peerSetupTransportOverride peerSetupTransport
|
||||
|
||||
func peerSetupTransportFromCtx() (peerSetupTransport, error) {
|
||||
if peerSetupTransportOverride != nil {
|
||||
return peerSetupTransportOverride, nil
|
||||
}
|
||||
t, err := driftTransportFromCtx()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return t, nil
|
||||
}
|
||||
|
||||
// PeerSetupResult records what the peer setup did.
|
||||
type PeerSetupResult struct {
|
||||
UserCreated bool `json:"user_created"`
|
||||
EventsDir string `json:"events_dir"`
|
||||
NFSOnOrca bool `json:"nfs_on_orca"`
|
||||
NFSMsg string `json:"nfs_msg,omitempty"`
|
||||
}
|
||||
|
||||
// setupOrcaUser runs the idempotent `useradd -r orca` and creates the
|
||||
// drift-events directory owned by orca:orca on the peer. Returns the
|
||||
// result; a transient SSH failure returns the error (no partial state).
|
||||
func setupOrcaUser(ctx context.Context, transport peerSetupTransport, peer string) (*PeerSetupResult, error) {
|
||||
if peer == "" {
|
||||
return nil, fmt.Errorf("peer setup: peer is empty")
|
||||
}
|
||||
res := &PeerSetupResult{EventsDir: "/etc/orca/state/drift-events"}
|
||||
|
||||
if !peerSetupNoOrcaUser {
|
||||
useraddCmd := "useradd -r orca -s /usr/sbin/nologin 2>/dev/null || true"
|
||||
if _, err := transport.Exec(ctx, peer, useraddCmd); err != nil {
|
||||
return nil, fmt.Errorf("peer setup: useradd: %w", err)
|
||||
}
|
||||
res.UserCreated = true
|
||||
}
|
||||
|
||||
mkdirCmd := fmt.Sprintf("mkdir -p %s && %s", res.EventsDir, chownDriftEvents(res.EventsDir))
|
||||
if _, err := transport.Exec(ctx, peer, mkdirCmd); err != nil {
|
||||
return nil, fmt.Errorf("peer setup: mkdir drift-events: %w", err)
|
||||
}
|
||||
|
||||
nfs, msg := detectNFS(ctx, transport, peer, "/etc/orca")
|
||||
res.NFSOnOrca = nfs
|
||||
res.NFSMsg = msg
|
||||
return res, nil
|
||||
}
|
||||
|
||||
// chownDriftEvents returns the chown command for the drift-events dir.
|
||||
// When --no-orca-user is set the orca user may not exist; chown only
|
||||
// when the user was created.
|
||||
func chownDriftEvents(dir string) string {
|
||||
if peerSetupNoOrcaUser {
|
||||
return "true"
|
||||
}
|
||||
return fmt.Sprintf("chown orca:orca %s 2>/dev/null || true", dir)
|
||||
}
|
||||
|
||||
// detectNFS checks whether the given path is on an NFS mount by running
|
||||
// `stat -f -c %T <path>` on the peer. When the fs type contains "nfs"
|
||||
// it returns (true, msg). Best-effort: a stat failure returns
|
||||
// (false, "stat unavailable").
|
||||
func detectNFS(ctx context.Context, transport peerSetupTransport, peer, path string) (bool, string) {
|
||||
out, err := transport.Exec(ctx, peer, fmt.Sprintf("stat -f -c %%T %s 2>/dev/null || echo unknown", shellQuoteDrift(path)))
|
||||
if err != nil {
|
||||
return false, "stat unavailable: " + err.Error()
|
||||
}
|
||||
fsType := strings.TrimSpace(string(out))
|
||||
if strings.Contains(fsType, "nfs") {
|
||||
return true, fmt.Sprintf("%s is on NFS (%s); skipping Path units for /etc/orca paths", path, fsType)
|
||||
}
|
||||
return false, fsType
|
||||
}
|
||||
|
||||
var peerSetupCmd = &cobra.Command{
|
||||
Use: "peer-setup <peer>",
|
||||
Short: "Create the orca system user + drift-events dir on a peer (REQ-111)",
|
||||
Long: `SSH to <peer> and idempotently create the orca system user
|
||||
(useradd -r orca -s /usr/sbin/nologin) and /etc/orca/state/drift-events/
|
||||
owned by orca:orca. Also detects NFS on /etc/orca (REQ-112/D-233) and
|
||||
logs a warning when /etc/orca is on NFS (Path units are skipped for
|
||||
those paths in that case). Use --no-orca-user to skip user creation
|
||||
(for environments with an existing service account).`,
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
peer := args[0]
|
||||
transport, err := peerSetupTransportFromCtx()
|
||||
if err != nil {
|
||||
return fmt.Errorf("ssh transport: %w", err)
|
||||
}
|
||||
res, err := setupOrcaUser(cmd.Context(), transport, peer)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
printResult(fmt.Sprintf("✓ Peer %s set up (user=%t, nfs=%t)", peer, res.UserCreated, res.NFSOnOrca), res)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
func init() {
|
||||
peerSetupCmd.Flags().BoolVar(&peerSetupNoOrcaUser, "no-orca-user", false, "skip orca system user creation (env has existing service account)")
|
||||
rootCmd.AddCommand(peerSetupCmd)
|
||||
}
|
||||
@@ -0,0 +1,466 @@
|
||||
// Package cli: recovery.go implements the full-recovery semantics for
|
||||
// `orca restore` (P07, v0.11 milestone).
|
||||
//
|
||||
// On top of P04's signed-restore, restore now:
|
||||
//
|
||||
// 1. Verifies the signature (P04).
|
||||
// 2. --dry-run: extracts to a temp staging dir, verifies, reports,
|
||||
// cleans up, and exits — never touching the real ORCA_HOME.
|
||||
// 3. Without --force: scans every peer for running orca-alloc-*
|
||||
// systemd units; if any are running, refuses the restore (data
|
||||
// loss protection) and reports which allocs are running.
|
||||
// 4. With --force: stops the running allocs (SSH systemctl stop),
|
||||
// extracts the tarball, then restarts the allocs from the restored
|
||||
// state (SSH systemctl start).
|
||||
// 5. Post-restore verification: master key present + valid, namespace
|
||||
// dirs exist, SQLite DBs openable; reports discrepancies.
|
||||
// 6. Records the restore in the audit log (internal/store/audit_repo).
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/backup"
|
||||
"git.cloudinit.dev/coreci/orca/internal/certpaths"
|
||||
"git.cloudinit.dev/coreci/orca/internal/engine"
|
||||
"git.cloudinit.dev/coreci/orca/internal/paths"
|
||||
"git.cloudinit.dev/coreci/orca/internal/secrets"
|
||||
"git.cloudinit.dev/coreci/orca/internal/store"
|
||||
)
|
||||
|
||||
// RestoreOptions configures the P07 full-recovery restore. It mirrors
|
||||
// backup.RestoreOptions but adds DryRun and is consumed by runRestore
|
||||
// (the CLI layer), which performs the alloc reconciliation and audit
|
||||
// logging that the lower-level backup.Restore does not.
|
||||
type RestoreOptions struct {
|
||||
InputPath string
|
||||
TargetDir string
|
||||
MasterKey []byte
|
||||
Force bool
|
||||
DryRun bool
|
||||
}
|
||||
|
||||
// ErrRunningAllocs is returned when restore refuses to clobber a live
|
||||
// cluster that has running allocations and --force was not given.
|
||||
var ErrRunningAllocs = errors.New("restore: running allocations present (use --force to stop and restart)")
|
||||
|
||||
// runRestore is the entry point invoked by restoreCmd.RunE. It performs
|
||||
// the full P07 recovery flow. The cmd is used only for output; ctx comes
|
||||
// from cmd.Context().
|
||||
func runRestore(cmd *cobra.Command, opts RestoreOptions) error {
|
||||
ctx := cmd.Context()
|
||||
if opts.InputPath == "" {
|
||||
return fmt.Errorf("restore: InputPath is empty")
|
||||
}
|
||||
if opts.TargetDir == "" {
|
||||
return fmt.Errorf("restore: TargetDir is empty")
|
||||
}
|
||||
if len(opts.MasterKey) == 0 {
|
||||
return fmt.Errorf("restore: MasterKey is empty")
|
||||
}
|
||||
log := newLogger()
|
||||
|
||||
sigPath := opts.InputPath + ".sig"
|
||||
if err := backup.VerifySignature(opts.InputPath, sigPath, opts.MasterKey); err != nil {
|
||||
auditRestore(ctx, "failure", err, map[string]any{"path": opts.InputPath, "target": opts.TargetDir})
|
||||
return fmt.Errorf("restore: verify signature: %w", err)
|
||||
}
|
||||
|
||||
if opts.DryRun {
|
||||
return runRestoreDryRun(cmd, opts)
|
||||
}
|
||||
|
||||
ex, exErr := drainExecFromCtx(ctx)
|
||||
if exErr != nil {
|
||||
return fmt.Errorf("ssh transport: %w", exErr)
|
||||
}
|
||||
|
||||
running, err := scanRunningAllocs(ctx, ex)
|
||||
if err != nil {
|
||||
log.Warn("restore: scan running allocs failed", "error", err)
|
||||
}
|
||||
if len(running) > 0 && !opts.Force {
|
||||
allocList := formatRunningAllocs(running)
|
||||
err := fmt.Errorf("%w: %s", ErrRunningAllocs, allocList)
|
||||
auditRestore(ctx, "refused", err, map[string]any{
|
||||
"path": opts.InputPath,
|
||||
"target": opts.TargetDir,
|
||||
"running": running,
|
||||
})
|
||||
return err
|
||||
}
|
||||
|
||||
if opts.Force && len(running) > 0 {
|
||||
stopped, stopErr := stopRunningAllocs(ctx, ex, running)
|
||||
if stopErr != nil {
|
||||
auditRestore(ctx, "partial", stopErr, map[string]any{
|
||||
"path": opts.InputPath,
|
||||
"target": opts.TargetDir,
|
||||
"stopped": stopped,
|
||||
"running": running,
|
||||
})
|
||||
return fmt.Errorf("restore: stop running allocs: %w", stopErr)
|
||||
}
|
||||
log.Info("restore: stopped running allocs", "count", len(stopped))
|
||||
}
|
||||
|
||||
ropts := backup.RestoreOptions{
|
||||
InputPath: opts.InputPath,
|
||||
TargetDir: opts.TargetDir,
|
||||
MasterKey: opts.MasterKey,
|
||||
Force: true,
|
||||
}
|
||||
if err := backup.Restore(ropts); err != nil {
|
||||
auditRestore(ctx, "failure", err, map[string]any{
|
||||
"path": opts.InputPath,
|
||||
"target": opts.TargetDir,
|
||||
})
|
||||
return err
|
||||
}
|
||||
|
||||
if opts.Force && len(running) > 0 {
|
||||
if rerr := restartAllocs(ctx, ex, running); rerr != nil {
|
||||
log.Warn("restore: failed to restart some allocs", "error", rerr)
|
||||
}
|
||||
}
|
||||
|
||||
verifications := verifyRestoredState(opts.TargetDir, opts.MasterKey)
|
||||
report := buildVerificationReport(verifications)
|
||||
result := "success"
|
||||
var verifyErr error
|
||||
if len(verifications) > 0 {
|
||||
result = "partial"
|
||||
verifyErr = fmt.Errorf("post-restore verification: %d issue(s)", len(verifications))
|
||||
}
|
||||
auditRestore(ctx, result, verifyErr, map[string]any{
|
||||
"path": opts.InputPath,
|
||||
"target": opts.TargetDir,
|
||||
"force": opts.Force,
|
||||
"running": running,
|
||||
"verification": report,
|
||||
})
|
||||
|
||||
if jsonOutput {
|
||||
return printJSON(map[string]any{
|
||||
"path": opts.InputPath,
|
||||
"target": opts.TargetDir,
|
||||
"force": opts.Force,
|
||||
"running": running,
|
||||
"verification": report,
|
||||
})
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "✓ Restored to: %s\n", opts.TargetDir)
|
||||
if opts.Force && len(running) > 0 {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), " stopped+restarted %d running alloc(s)\n", len(running))
|
||||
}
|
||||
for _, v := range verifications {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), " ⚠ verify: %s\n", v)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// runRestoreDryRun extracts the tarball to a temp dir, runs
|
||||
// post-restore verification against that temp dir, reports what WOULD
|
||||
// be restored, and cleans up. The real ORCA_HOME is never touched.
|
||||
func runRestoreDryRun(cmd *cobra.Command, opts RestoreOptions) error {
|
||||
tmp, err := os.MkdirTemp("", "orca-restore-dryrun-*")
|
||||
if err != nil {
|
||||
return fmt.Errorf("restore: dry-run temp dir: %w", err)
|
||||
}
|
||||
defer os.RemoveAll(tmp)
|
||||
log := newLogger()
|
||||
log.Info("restore: dry-run staging dir", "dir", tmp)
|
||||
|
||||
ropts := backup.RestoreOptions{
|
||||
InputPath: opts.InputPath,
|
||||
TargetDir: tmp,
|
||||
MasterKey: opts.MasterKey,
|
||||
Force: true,
|
||||
}
|
||||
if err := backup.Restore(ropts); err != nil {
|
||||
return fmt.Errorf("restore: dry-run extract: %w", err)
|
||||
}
|
||||
|
||||
entries := listExtractedFiles(tmp)
|
||||
verifications := verifyRestoredState(tmp, opts.MasterKey)
|
||||
report := buildVerificationReport(verifications)
|
||||
auditRestore(cmd.Context(), "dry-run", nil, map[string]any{
|
||||
"path": opts.InputPath,
|
||||
"target": opts.TargetDir,
|
||||
"staging": tmp,
|
||||
"files": len(entries),
|
||||
"verification": report,
|
||||
})
|
||||
|
||||
if jsonOutput {
|
||||
return printJSON(map[string]any{
|
||||
"dry_run": true,
|
||||
"path": opts.InputPath,
|
||||
"target": opts.TargetDir,
|
||||
"files": len(entries),
|
||||
"verification": report,
|
||||
})
|
||||
}
|
||||
out := cmd.OutOrStdout()
|
||||
fmt.Fprintf(out, "✓ Dry-run: would restore %d file(s) to %s\n", len(entries), opts.TargetDir)
|
||||
if len(entries) > 0 && len(entries) <= 20 {
|
||||
for _, e := range entries {
|
||||
fmt.Fprintf(out, " %s\n", e)
|
||||
}
|
||||
}
|
||||
for _, v := range verifications {
|
||||
fmt.Fprintf(out, " ⚠ verify: %s\n", v)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// runningAlloc is a running allocation discovered on a peer node.
|
||||
type runningAlloc struct {
|
||||
Node string `json:"node"`
|
||||
Peer string `json:"peer"`
|
||||
AllocID string `json:"alloc_id"`
|
||||
}
|
||||
|
||||
// scanRunningAllocs lists every peer node in the registry and queries
|
||||
// each via SSH for currently-running orca-alloc-*.service units. A
|
||||
// registry failure or empty registry yields an empty (not error)
|
||||
// slice; per-node failures are logged and skipped so a single
|
||||
// unreachable peer does not block the restore.
|
||||
func scanRunningAllocs(ctx context.Context, ex drainExecer) ([]runningAlloc, error) {
|
||||
reg, closer, err := nodeRegistry()
|
||||
if err != nil {
|
||||
return nil, nil
|
||||
}
|
||||
defer closer()
|
||||
nodes, err := reg.List(ctx)
|
||||
if err != nil {
|
||||
return nil, nil
|
||||
}
|
||||
var out []runningAlloc
|
||||
log := newLogger()
|
||||
for i := range nodes {
|
||||
n := nodes[i]
|
||||
peer := peerAddrForNode(n)
|
||||
if peer == "" {
|
||||
continue
|
||||
}
|
||||
ids, err := listRunningAllocs(ctx, ex, peer)
|
||||
if err != nil {
|
||||
log.Warn("restore: cannot list allocs on node",
|
||||
slog.String("node", n.Name), slog.String("peer", peer), "error", err)
|
||||
continue
|
||||
}
|
||||
for _, id := range ids {
|
||||
out = append(out, runningAlloc{Node: n.Name, Peer: peer, AllocID: id})
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// formatRunningAllocs renders a list of running allocs for an error
|
||||
// message (one per line: "node/peer: alloc-id").
|
||||
func formatRunningAllocs(rs []runningAlloc) string {
|
||||
var b strings.Builder
|
||||
for i, a := range rs {
|
||||
if i > 0 {
|
||||
b.WriteString("; ")
|
||||
}
|
||||
fmt.Fprintf(&b, "%s/%s: %s", a.Node, a.Peer, a.AllocID)
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// stopRunningAllocs stops every allocation in rs via SSH (systemctl
|
||||
// stop). It returns the ids that were successfully stopped and an
|
||||
// aggregate error if any failed.
|
||||
func stopRunningAllocs(ctx context.Context, ex drainExecer, rs []runningAlloc) ([]string, error) {
|
||||
var stopped []string
|
||||
var failed []string
|
||||
for _, a := range rs {
|
||||
if err := stopAlloc(ctx, ex, a.Peer, a.AllocID); err != nil {
|
||||
failed = append(failed, a.AllocID)
|
||||
continue
|
||||
}
|
||||
stopped = append(stopped, a.AllocID)
|
||||
}
|
||||
if len(failed) > 0 {
|
||||
return stopped, fmt.Errorf("failed to stop %d alloc(s): %s", len(failed), strings.Join(failed, ", "))
|
||||
}
|
||||
return stopped, nil
|
||||
}
|
||||
|
||||
// restartAllocs restarts every allocation in rs via SSH (systemctl
|
||||
// start). A per-alloc failure is collected and returned as an aggregate
|
||||
// error; the caller treats restart failures as non-fatal (the restore
|
||||
// itself succeeded).
|
||||
func restartAllocs(ctx context.Context, ex drainExecer, rs []runningAlloc) error {
|
||||
var failed []string
|
||||
for _, a := range rs {
|
||||
startCmd := fmt.Sprintf("systemctl start %s", allocUnit(a.AllocID))
|
||||
if _, err := ex.Exec(ctx, a.Peer, startCmd); err != nil {
|
||||
failed = append(failed, a.AllocID)
|
||||
}
|
||||
}
|
||||
if len(failed) > 0 {
|
||||
return fmt.Errorf("failed to restart %d alloc(s): %s", len(failed), strings.Join(failed, ", "))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// verifyRestoredState checks the restored tree for the master key
|
||||
// (present + valid length/mode), namespace directories, and openable
|
||||
// SQLite databases. It returns a slice of human-readable discrepancy
|
||||
// strings (empty if everything checks out).
|
||||
func verifyRestoredState(targetDir string, masterKey []byte) []string {
|
||||
var issues []string
|
||||
|
||||
mkPath := filepath.Join(targetDir, relFromRoot(paths.MasterKeyPath()))
|
||||
info, err := os.Stat(mkPath)
|
||||
if err != nil {
|
||||
issues = append(issues, fmt.Sprintf("master key missing: %s", mkPath))
|
||||
} else {
|
||||
if info.Mode().Perm() != secrets.MasterKeyMode {
|
||||
issues = append(issues, fmt.Sprintf("master key mode %04o (want %04o)", info.Mode().Perm(), secrets.MasterKeyMode))
|
||||
}
|
||||
key, rerr := os.ReadFile(mkPath)
|
||||
if rerr != nil {
|
||||
issues = append(issues, fmt.Sprintf("master key unreadable: %v", rerr))
|
||||
} else if len(key) != secrets.MasterKeyLen {
|
||||
issues = append(issues, fmt.Sprintf("master key length %d (want %d)", len(key), secrets.MasterKeyLen))
|
||||
} else if masterKey != nil && string(key) != string(masterKey) {
|
||||
issues = append(issues, "master key differs from the key used to verify the signature")
|
||||
}
|
||||
}
|
||||
|
||||
nsDirs := findNamespaceDirs(targetDir)
|
||||
if len(nsDirs) == 0 {
|
||||
issues = append(issues, "no namespace directories found")
|
||||
}
|
||||
for _, nsDir := range nsDirs {
|
||||
dbPath := filepath.Join(nsDir, "db", "orca.db")
|
||||
if _, err := os.Stat(dbPath); err == nil {
|
||||
if err := dbOpenable(dbPath); err != nil {
|
||||
issues = append(issues, fmt.Sprintf("db not openable %s: %v", dbPath, err))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
clusterDB := filepath.Join(targetDir, "orca.db")
|
||||
if _, err := os.Stat(clusterDB); err == nil {
|
||||
if err := dbOpenable(clusterDB); err != nil {
|
||||
issues = append(issues, fmt.Sprintf("cluster db not openable %s: %v", clusterDB, err))
|
||||
}
|
||||
}
|
||||
return issues
|
||||
}
|
||||
|
||||
// relFromRoot strips the ORCA_HOME prefix off an absolute path so it
|
||||
// can be rejoined to an arbitrary target dir (used by --dry-run and
|
||||
// non-default --target restores). If the path does not start with the
|
||||
// current ORCA_HOME root, the path's base is returned.
|
||||
func relFromRoot(p string) string {
|
||||
root := filepath.Clean(paths.Root())
|
||||
clean := filepath.Clean(p)
|
||||
if rel, err := filepath.Rel(root, clean); err == nil && !strings.HasPrefix(rel, "..") {
|
||||
return rel
|
||||
}
|
||||
return filepath.Base(clean)
|
||||
}
|
||||
|
||||
// findNamespaceDirs returns the top-level directories under targetDir
|
||||
// that look like namespace dirs (excluding the "cluster" dir and dot-
|
||||
// files). A namespace dir is any immediate child of ORCA_HOME that is
|
||||
// a directory and not "cluster".
|
||||
func findNamespaceDirs(targetDir string) []string {
|
||||
entries, err := os.ReadDir(targetDir)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var out []string
|
||||
for _, e := range entries {
|
||||
if !e.IsDir() {
|
||||
continue
|
||||
}
|
||||
name := e.Name()
|
||||
if strings.HasPrefix(name, ".") {
|
||||
continue
|
||||
}
|
||||
if name == "cluster" {
|
||||
continue
|
||||
}
|
||||
out = append(out, filepath.Join(targetDir, name))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// dbOpenable reports whether the SQLite file at path can be opened
|
||||
// read-only. It uses the same driver as the rest of the codebase
|
||||
// (modernc.org/sqlite via store.Open, but with a read-only pragma).
|
||||
func dbOpenable(path string) error {
|
||||
dsn := "file:" + path + "?mode=ro&_pragma=journal_mode(WAL)"
|
||||
db, err := sql.Open("sqlite", dsn)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer db.Close()
|
||||
if err := db.Ping(); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// listExtractedFiles walks a staging dir and returns the relative
|
||||
// paths of all regular files (capped at 1000 for reporting).
|
||||
func listExtractedFiles(root string) []string {
|
||||
var out []string
|
||||
_ = filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
if !info.Mode().IsRegular() {
|
||||
return nil
|
||||
}
|
||||
rel, rerr := filepath.Rel(root, path)
|
||||
if rerr != nil {
|
||||
return nil
|
||||
}
|
||||
out = append(out, filepath.ToSlash(rel))
|
||||
return nil
|
||||
})
|
||||
if len(out) > 1000 {
|
||||
out = out[:1000]
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// buildVerificationReport converts the discrepancy slice into a
|
||||
// JSON-friendly []string (nil-safe for empty input).
|
||||
func buildVerificationReport(issues []string) []string {
|
||||
if len(issues) == 0 {
|
||||
return []string{}
|
||||
}
|
||||
return issues
|
||||
}
|
||||
|
||||
// auditRestore records a restore event in the audit log (same pattern
|
||||
// as auditDrain in drain.go). It opens the cluster DB at
|
||||
// certpaths.DBPath(); a failure to open the DB is logged and silently
|
||||
// dropped so a restore is never blocked by the audit log itself.
|
||||
func auditRestore(ctx context.Context, result string, err error, meta map[string]any) {
|
||||
db, dbErr := store.Open(certpaths.DBPath())
|
||||
if dbErr != nil {
|
||||
newLogger().Warn("restore: audit log unavailable", "error", dbErr)
|
||||
return
|
||||
}
|
||||
defer db.Close()
|
||||
engine.NewAudit(store.NewAuditRepo(db), newLogger()).Record(ctx, "cli", "restore", certpaths.Dir(), result, err, meta)
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user