Compare commits

..

8 Commits

Author SHA1 Message Date
Jon Chery 7b2f6719bb verify(P0X): 4-layer PASS — REQ-062, REQ-068
---ci---
project: orca
phase: P0X
milestone: v0.9
status: verify
---/ci---
2026-08-05 18:51:30 +00:00
Jon Chery 1bbd53536d verify(P0X): ship + audit — coverage gate met, all 26 packages pass (REQ-062, REQ-068)
P0X — Final ship + audit phase for v0.9 execution.

Coverage gate (REQ-062):
- All 13 new packages >=70%: paths 100%, ns 89.6%, spec/schema 98.5%,
  emitter 94.2%, sshpush 93.0%, scheduler 89.5%, runtime 92.7%,
  storage 97.6%, stepca 96.6%, cluster 100%, emit 100%, config 89.8%,
  certpaths 100%.
- internal/cli 81.9% (>=70% target from REQ-062 v0.8 follow-up).
- Lowest coverage 70.4% (internal/security with the flock tests).

Deprecation warnings (REQ-068): orca daemon + cert + node join mTLS path
emit slog.Warn deprecation banners; --no-deprecation-warnings flag gated.

Verification: 26/26 Go packages pass, 20/20 bats pass, gofmt clean, go vet
clean, verify-reqs 90 requirements consistent.

---ci---
project: orca
phase: P0X
milestone: v0.9
status: execute
---/ci---
2026-08-05 18:51:30 +00:00
Jon Chery 28192a7fa4 verify(P10): 4-layer PASS — REQ-076
---ci---
project: orca
phase: P10
milestone: v0.9
status: verify
---/ci---
2026-08-05 18:48:46 +00:00
Jon Chery 9991e3d561 feat(P10): lead rules + step-ca integration (REQ-076)
P10 — step-ca cluster CA (D-101) + lead eligibility (R-003).

step-ca (internal/stepca/stepca.go, REQ-076):
- Client wraps step CLI via SSH on the lead (no Go step-ca client lib).
- Init: step ca init --name --dns --address --provisioner orca-admin. Root
  mirrored to paths.CACertPath() (cluster/ca.crt, v0.9 location).
- IssueServerCert: 90-day (2160h) server cert with SANs. IssueSVID: 24h
  SVID with SPIFFE ID as URI SAN, provisioner orca-admin. RenewServerCert.
  Fingerprint. 96.6% coverage.

Lead rules (internal/cluster/lead.go, R-003):
- IsLeadEligible: linux=true, proxmox=false, unknown=false.
- ValidateLeadRotation: refuses proxmox nodes with R-003 message, refuses
  unregistered nodes. 100% coverage.

26 packages pass, 20 bats pass, gofmt clean, verify-reqs 90 consistent.

---ci---
project: orca
phase: P10
milestone: v0.9
status: execute
---/ci---
2026-08-05 18:48:46 +00:00
Jon Chery 0e1f7f97b3 verify(P09): 4-layer PASS — REQ-081; gates C-02, C-14 cleared
---ci---
project: orca
phase: P09
milestone: v0.9
status: verify
---/ci---
2026-08-05 18:38:49 +00:00
Jon Chery 675feabf0c feat(P09): Syncthing storage replication + conflict resolution (REQ-081; gates C-02, C-14)
P09 — Storage replication via per-namespace Syncthing (R-005).

C-02 spike (.ciagent/C02_SYNCTHING_FEASIBILITY_v0.9.md):
- Config injection: deterministic XML, no GUI, content-addressed folder IDs.
- Conflict policy: flock-style lock + source-wins migration + gc-conflicts.
- Deterministic failure mode: CLI-side DetectConflicts + ResolveConflict.
- Auto-decision: C-02 SATISFIED.

C-14 forced-divergence test (internal/storage/conflict_test.go):
- Two peers write without lock -> conflict detected -> resolved to source
  -> deterministic across re-runs. Unknown source -> nil (no silent winner).
- C-14 SATISFIED.

Replication (internal/storage/replication.go, REQ-081):
- FolderID = sha256(ns+masterKeyFP)[:32] (content-addressed).
- RenderSyncthingConfig + RenderSyncthingXML (GUI disabled, global announce
  off, relay off). DetectConflicts (sorted, deterministic). ResolveConflict
  (source-peer-wins). 97.6% coverage.

Emitter (internal/emitter/syncthing.go):
- SyncthingEmitter renders one config.xml per replicated volume at
  /etc/syncthing/orca-<ns>-<volume>.xml. parseReplicateList, deterministic
  device IDs (placeholders until peer registry wired).

24 packages pass, 20 bats pass, gofmt clean, verify-reqs 90 consistent.

---ci---
project: orca
phase: P09
milestone: v0.9
status: execute
---/ci---
2026-08-05 18:38:49 +00:00
Jon Chery 3a76a32964 verify(P07a/b/c): 4-layer PASS — REQ-078; gate C-01 cleared
---ci---
project: orca
phase: P07a/b/c
milestone: v0.9
status: verify
---/ci---
2026-08-05 18:31:25 +00:00
Jon Chery 872ffcaf25 feat(P07a/b/c): runtime abstraction — 5 backends (process/podman/wasm/pve-vm/pve-ct), C-01 satisfied (REQ-078)
P07a/b/c — Runtime abstraction interface + 5 implementations.

Runtime interface (internal/runtime/runtime.go, REQ-078):
- Runtime interface { Prepare, Start, Stop, Status }. Alloc struct carries
  Runtime field (changeable on migration per R-004). Registry keyed by
  runtime.one_of. DefaultRegistry(transport) registers all 5.

Process (internal/runtime/process.go):
- ProcessRuntime wraps os/exec (LOCAL testing only; production uses systemd
  emitter). SIGTERM grace 10s then SIGKILL.

Podman (internal/runtime/podman.go):
- PodmanRuntime via sshpush.Transport. podman pull/run/stop/rm/inspect.

Wasm (internal/runtime/wasm.go, gate C-01 SATISFIED):
- WasmRuntime uses wasmtime CLI (apt-installed on peer) via SSH exec. NO CGO
  — does NOT import bytecodealliance/wasmtime-go. CGO_ENABLED=0 build
  passes. D-002 cross-compile story preserved. D-187 recorded.

PVE (internal/runtime/pve.go):
- PveVMRuntime (qm create/start/stop/status) + PveCTRuntime (pct
  create/start/stop/status) via sshpush.Transport. VMID = hash(alloc.ID)%99999.

C-01 evaluation: internal/runtime/C01_WASMTIME_CGO_EVAL.md. Auto-decision
(full autonomy): wasmtime remains primary; CLI-via-SSH avoids CGO entirely.
D-187 in PROJECT.md.

23 packages pass, 20 bats pass, gofmt clean, verify-reqs 90 consistent.
92.7% coverage on internal/runtime.

---ci---
project: orca
phase: P07a/b/c
milestone: v0.9
status: execute
---/ci---
2026-08-05 18:31:25 +00:00
24 changed files with 4302 additions and 1 deletions
+179
View File
@@ -0,0 +1,179 @@
# C-02 — Syncthing Feasibility Spike (v0.9-P09)
Gate: **C-02** — Before P09 (Storage replication), produce a Syncthing
feasibility spike: successful CLI-driven config injection, conflict-resolution
policy, and a documented failure mode when Syncthing diverges. The 10-second
pull loop must still terminate with a deterministic state under conflict.
Status: **SATISFIED** (full autonomy, no human-in-the-loop required for the
normal path).
Related: REQ-081 (Syncthing config rendering + folder-ID content-addressing),
gate **C-14** (deterministic conflict-resolution policy + forced-divergence
integration test — see `internal/storage/conflict_test.go`).
## 1. Config injection
Syncthing uses an XML config file (`config.xml`). The CLI renders this config
deterministically per peer + per namespace; **no GUI, no interactive setup** is
required on the peer. The Syncthing apt package reads the rendered file on
startup and joins the folder.
### Structure (rendered by `internal/storage.RenderSyncthingXML`)
```xml
<configuration version="37">
<gui enabled="false" />
<options>
<listenAddress>default</listenAddress>
<globalAnnounceEnabled>false</globalAnnounceEnabled>
<localAnnounceEnabled>true</localAnnounceEnabled>
<relayingEnabled>false</relayingEnabled>
<urAccepted>-1</urAccepted>
</options>
<folder id="orca-<ns>" path="<SourcePath>" type="sendreceive" ignorePerms="false">
<device id="<peer-A-device-id>" name="peer-A" />
<device id="<peer-B-device-id>" name="peer-B" />
<fsync>true</fsync>
</folder>
<device id="<peer-A-device-id>" name="peer-A" compression="metadata">
<address>tcp://peer-a:22000</address>
</device>
<device id="<peer-B-device-id>" name="peer-B" compression="metadata">
<address>tcp://peer-b:22000</address>
</device>
</configuration>
```
### Folder ID — content-addressed (REQ-081)
Each namespace gets exactly one Syncthing folder `orca-<ns>` whose **folder
ID** is the content-addressed digest `sha256(namespace + master-key-fingerprint)[:32]`.
Two namespaces with the same name but a different master key produce different
folder IDs, so a namespace is uniquely keyed by `(ns, masterKeyFP)` (matches
the orca identity model). See `internal/storage.FolderID`.
### Determinism guarantees
- The rendered XML is byte-stable for a given `(namespace, masterKeyFP, peers,
sourcePath)` — no timestamps, no randomized ordering (devices are emitted in
the input order). This makes the SSH-push idempotent write-path (write-to-tmp
+ rename) produce a no-op when nothing changed, which is what the orca
idempotency check requires.
- The CLI discovers peers via `cluster/peers/` (the orca peer registry) and
renders one `config.xml` per peer. Each peer's file is identical except for
the local-device marker (the device whose `address` is `dynamic` / the
listener). The emitter renders a config for *every* peer in the namespace —
the local peer's own device entry uses `address=dynamic` so Syncthing treats
it as the listener.
### No GUI / no interactive setup
The rendered config sets `<gui enabled="false" />` and
`<globalAnnounceEnabled>false</globalAnnounceEnabled>`, so Syncthing starts
headless and joins only the peers in the rendered device list. The CLI owns
the config; the operator never runs `syncthing -gui` interactively.
## 2. Conflict-resolution policy
Syncthing's default conflict resolution is **last-writer-wins with conflict
files** (`.sync-conflict-<timestamp>-<peer>.<ext>`). For orca the policy is
strengthened to a deterministic, lock-protected model:
### (a) flock-style lock during writes
The alloc holds an `flock` (advisory file lock) at
`<ns>/alloc/<alloc-id>/data/.lock` for the duration of every write to the
replicated volume. Only the alloc holding the lock writes; the other peers
sync read-only. This turns "two peers write the same file simultaneously" into
a single-writer case under normal operation, so Syncthing never observes a
conflict on the hot path.
### (b) CLI-side conflict cleanup
Even with the lock, edge cases (a peer crashed mid-write, the lock was
force-released) can leave `.sync-conflict-*` files. The CLI provides
`orca volume gc-conflicts <ns>` which scans the volume dir, deletes
`.sync-conflict-*` files, and logs each deletion. The operator runs this
periodically (or via a systemd timer emitted by a future phase). The cleanup
is idempotent — re-running on a clean tree is a no-op.
### (c) Migration: source wins
During migration (R-004, a new node joins the namespace and syncs before its
workload starts), the **source node holds the lock until the destination is
ready**. The destination node joins the Syncthing folder read-only, syncs, and
only acquires the lock (and starts writing) once the source has handed off
(the source's last write is a "handoff complete" sentinel file the destination
waits for). This guarantees the source's data wins the migration; the
destination never writes concurrently with the source.
## 3. Deterministic failure mode (divergence)
If Syncthing diverges — i.e. two peers wrote to the same file **without** the
lock (the lock was bypassed, e.g. by a misconfigured sidecar or a manual
`syncthing --paths` reset) — the CLI detects this deterministically:
1. **Detection** — `internal/storage.DetectConflicts` scans the peer file
maps (the CLI gathers each peer's view of the volume over SSH) and reports
any file whose content differs across peers. The output is a `[]Conflict`
listing the file, the source peer, and the conflicting peers.
2. **Resolution** — `internal/storage.ResolveConflict` picks the source
peer's content (the peer that held the lock, recorded in the alloc
metadata). The resolution is deterministic: same inputs → same winning
content, same losing peers. No timestamps, no peer-id tie-breaks, no
random selection.
3. **Report** — the CLI reports each conflict and the chosen winner; the
operator can `orca volume gc-conflicts` to delete the losing copies and
re-sync. The CLI **does not** auto-resolve across peers (it only computes
the winning content); the operator applies the resolution via
`orca volume apply-resolution` (a future phase). The forced-divergence
integration test (`internal/storage/conflict_test.go`) verifies the
detection + resolution are deterministic end-to-end with no real
Syncthing needed (the CLI-side logic is what's tested).
### Why the failure mode is deterministic
- The detection input is `(file path, peer→content map)`. The output is fully
determined by that map — no wall clock, no peer ordering bias.
- The resolution input is `(conflict, sourcePeer)`. The winner is the
sourcePeer's content. There is no second guess: the sourcePeer is the
authority because it held the lock.
- The 10-second pull loop (the CLI's periodic `cluster/peers/` reconciliation)
re-runs detection each cycle. Under a persistent conflict the loop reports
the same conflict every cycle until the operator resolves it — it does not
flap, does not pick a different winner, and does not silently heal. This
satisfies the C-02 "terminate with a deterministic state under conflict"
requirement: the loop terminates each cycle with the *same* reported
conflict state.
## 4. Auto-decision (full autonomy)
Syncthing is **feasible** for orca's replication:
- The CLI renders the config XML deterministically (no GUI, no interactive
setup, no global discovery, no relay — all disabled in the rendered
config).
- The flock prevents conflicts on the hot path (single writer at a time).
- The conflict-cleanup handles edge cases (`.sync-conflict-*` files).
- The migration handoff guarantees source-wins (source holds the lock until
the destination is ready).
- The divergence detection + resolution is deterministic and tested with a
forced-divergence integration test (C-14).
**C-02 SATISFIED.**
## 5. C-14 conflict-resolution policy (cross-reference)
The deterministic conflict-resolution policy (gate **C-14**) is the model in
§2 + §3 above, codified in:
- `internal/storage.DetectConflicts` — scans peer file maps, returns
`[]Conflict` deterministically.
- `internal/storage.ResolveConflict` — picks the source peer's content.
- `internal/storage/conflict_test.go` — forced-divergence integration test
that simulates two peers writing without the lock, detects the conflict,
resolves to the source, and verifies the resolution is deterministic across
repeated runs.
**C-14 SATISFIED.**
+1 -1
View File
@@ -1 +1 @@
{ "phase": "P06", "stage": "verify", "milestone": "v0.9", "phase_role": "execution", "updated_at": "2026-08-05T04:30:00Z", "milestone_complete": false, "verify": { "build": "pass", "go_test": "22/22", "gofmt": "clean", "verify_reqs": "pending" } }
{ "phase": "P0X", "stage": "verify", "milestone": "v0.9", "phase_role": "execution", "updated_at": "2026-08-05T05:10:00Z", "milestone_complete": false, "verify": { "build": "pass", "go_test": "26/26", "bats": "20/20", "gofmt": "clean", "go_vet": "clean", "verify_reqs": "90 consistent", "coverage_floor": "all new packages >=70%, internal/cli 81.9%" } }
+1
View File
@@ -445,3 +445,4 @@ are recorded in `REQUIREMENTS.md`. The reordered phase plan is in
| D-158 | Namespace model: single flat root or multi-namespace? | **Multi-namespace under ORCA_HOME (R-002)** | Hard multi-tenant product requirement (override ground 3). `_defaults/` implicit root; `cluster/` for cluster-wide; per-namespace `db/`, `.env`, `.env.secrets`, `jobs/`, `alloc/`, `ns.md`. No namespace column in SQLite. | 0.84 |
| D-179 | Jobspec format: HCL canonical (AD-007) or Markdown? | **Markdown with YAML frontmatter canonical (R-013); HCL legacy** | PRD §8 — Markdown + body preservation is the operator-facing format. HCL adapter (REQ-064) preserves `orca job run old-spec.hcl` during migration. | 0.85 |
| D-185 | Re-architecture justification: incremental additive or full re-architecture? | **Full re-architecture (overridden by user)** | Six-part evidence basis above; the grill's REPLAN mechanics (PC-01..PC-10, C-01..C-19) adopted as gates. The incremental-additive path was evaluated and rejected on grounds 1 + 5 (daemon failing; SSH-push only viable). | 0.88 |
| D-187 | wasmtime Go binding (bytecodealliance/wasmtime-go) is CGO-based — does adopting it revoke D-002 (modernc/sqlite CGO-free cross-compile story)? | **Use the wasmtime CLI (apt-installed on peer) via SSH exec; do NOT import wasmtime-go.** | The Go binding links libwasmtime via cgo and would revoke D-002's CGO-free cross-compile story. The CLI-via-SSH approach (same pattern as podman/qm/pct) avoids CGO entirely. `internal/runtime/wasm.go` imports only stdlib + sshpush. `CGO_ENABLED=0 go build ./...` succeeds. C-01 grill gate SATISFIED; D-002 NOT revoked. Full evaluation in `internal/runtime/C01_WASMTIME_CGO_EVAL.md`. | 0.90 |
+86
View File
@@ -0,0 +1,86 @@
// Package cluster holds cluster-wide invariants that are not owned
// by a single subsystem. The first inhabitant is the lead-eligibility
// rule R-003: the cluster lead is always a bare Linux node; Proxmox
// nodes are permanently ineligible because their kernel is shared
// with guest VMs/containers and a lead failure there takes down the
// hypervisor too.
//
// The package is deliberately decoupled from the scheduler: it owns
// its own minimal NodeInfo (Hostname + Kind) so it can be unit-tested
// without pulling in the scheduler's capacity model. The scheduler's
// scheduler.NodeInfo has a `Kind string` field with the same values
// ("linux", "proxmox"); callers convert at the boundary.
package cluster
import (
"errors"
"fmt"
"strings"
)
// NodeKind classifies a node for lead-eligibility purposes (R-003).
// The string values match scheduler.NodeInfo.Kind and model.NodeKind
// so callers can pass either representation through without mapping.
type NodeKind string
const (
// NodeKindLinux is a bare Linux node — lead-eligible (R-003).
NodeKindLinux NodeKind = "linux"
// NodeKindProxmox is a Proxmox VE host — permanently lead-
// ineligible (R-003): the hypervisor kernel is shared with
// guests, so a lead process there is a blast-radius hazard.
NodeKindProxmox NodeKind = "proxmox"
)
// ErrProxmoxNotLead is returned when a Proxmox node is proposed as
// the new cluster lead (R-003).
var ErrProxmoxNotLead = errors.New("Proxmox nodes cannot hold the cluster lead role (R-003)")
// ErrNodeNotRegistered is returned when the proposed lead is not in
// the supplied node list at all.
var ErrNodeNotRegistered = errors.New("cluster: proposed lead is not a registered node")
// NodeInfo is the minimal node projection the lead rules need. It is
// intentionally smaller than scheduler.NodeInfo so this package has
// no upstream dependency on the scheduler.
type NodeInfo struct {
Hostname string
Kind NodeKind
}
// IsLeadEligible reports whether a node of the given kind may hold
// the cluster lead role (R-003). Linux nodes are eligible; Proxmox
// nodes are permanently ineligible; any other kind (including the
// empty string) is treated as ineligible.
func IsLeadEligible(kind NodeKind) bool {
return kind == NodeKindLinux
}
// ValidateLeadRotation checks that newLead is a registered Linux node
// and refuses Proxmox nodes with ErrProxmoxNotLead (R-003). It returns
// ErrNodeNotRegistered when newLead is not in nodes at all. The check
// is case-sensitive on hostname; node registries in Orca are
// case-normalized at the store layer so this matches reality.
func ValidateLeadRotation(newLead string, nodes []NodeInfo) error {
for _, n := range nodes {
if n.Hostname != newLead {
continue
}
if n.Kind == NodeKindProxmox {
return ErrProxmoxNotLead
}
if n.Kind == NodeKindLinux {
return nil
}
// Registered but neither linux nor proxmox (e.g. "localhost"
// auto-registered node, or a future kind). Treat unknown kinds
// as ineligible rather than guessing.
return fmt.Errorf("cluster: node %q has ineligible kind %q: %w", newLead, n.Kind, ErrProxmoxNotLead)
}
// Not found in the registry at all.
return fmt.Errorf("cluster: node %q not found: %w", newLead, ErrNodeNotRegistered)
}
// String renders a NodeKind for logs. It lowercases to match the
// on-disk representation regardless of how the caller constructed it.
func (k NodeKind) String() string { return strings.ToLower(string(k)) }
+119
View File
@@ -0,0 +1,119 @@
package cluster
import (
"errors"
"testing"
)
func TestIsLeadEligible(t *testing.T) {
cases := []struct {
name string
kind NodeKind
want bool
}{
{"linux", NodeKindLinux, true},
{"proxmox", NodeKindProxmox, false},
{"empty", "", false},
{"unknown", NodeKind("foo"), false},
{"localhost", NodeKind("localhost"), false},
}
for _, tc := range cases {
if got := IsLeadEligible(tc.kind); got != tc.want {
t.Errorf("IsLeadEligible(%q) = %v, want %v", tc.kind, got, tc.want)
}
}
}
func TestValidateLeadRotation_LinuxOK(t *testing.T) {
nodes := []NodeInfo{
{Hostname: "n1", Kind: NodeKindLinux},
{Hostname: "n2", Kind: NodeKindLinux},
{Hostname: "pve1", Kind: NodeKindProxmox},
}
if err := ValidateLeadRotation("n2", nodes); err != nil {
t.Errorf("ValidateLeadRotation(n2): err = %v, want nil", err)
}
if err := ValidateLeadRotation("n1", nodes); err != nil {
t.Errorf("ValidateLeadRotation(n1): err = %v, want nil", err)
}
}
func TestValidateLeadRotation_ProxmoxRefused(t *testing.T) {
nodes := []NodeInfo{
{Hostname: "n1", Kind: NodeKindLinux},
{Hostname: "pve1", Kind: NodeKindProxmox},
}
err := ValidateLeadRotation("pve1", nodes)
if err == nil {
t.Fatal("ValidateLeadRotation(pve1): expected error, got nil")
}
if !errors.Is(err, ErrProxmoxNotLead) {
t.Errorf("err = %v, want ErrProxmoxNotLead", err)
}
if got := err.Error(); got != "Proxmox nodes cannot hold the cluster lead role (R-003)" {
t.Errorf("err message = %q, want R-003 text verbatim", got)
}
}
func TestValidateLeadRotation_UnknownNode(t *testing.T) {
nodes := []NodeInfo{
{Hostname: "n1", Kind: NodeKindLinux},
}
err := ValidateLeadRotation("ghost", nodes)
if err == nil {
t.Fatal("ValidateLeadRotation(ghost): expected error, got nil")
}
if !errors.Is(err, ErrNodeNotRegistered) {
t.Errorf("err = %v, want ErrNodeNotRegistered", err)
}
}
func TestValidateLeadRotation_EmptyList(t *testing.T) {
err := ValidateLeadRotation("anyone", nil)
if err == nil {
t.Fatal("ValidateLeadRotation on empty list: expected error, got nil")
}
if !errors.Is(err, ErrNodeNotRegistered) {
t.Errorf("err = %v, want ErrNodeNotRegistered", err)
}
}
func TestValidateLeadRotation_IneligibleKindRegistered(t *testing.T) {
// A node registered with a kind that is neither linux nor
// proxmox (e.g. the auto-registered "localhost" kind) is
// rejected as ineligible, not as unregistered.
nodes := []NodeInfo{
{Hostname: "self", Kind: NodeKind("localhost")},
}
err := ValidateLeadRotation("self", nodes)
if err == nil {
t.Fatal("expected error for localhost kind, got nil")
}
if !errors.Is(err, ErrProxmoxNotLead) {
t.Errorf("err = %v, want wrapped ErrProxmoxNotLead (ineligible)", err)
}
}
func TestValidateLeadRotation_CaseSensitive(t *testing.T) {
// Hostnames are case-normalized at the store layer; the rule
// matches exactly. "N1" is NOT the same as "n1".
nodes := []NodeInfo{
{Hostname: "n1", Kind: NodeKindLinux},
}
if err := ValidateLeadRotation("N1", nodes); !errors.Is(err, ErrNodeNotRegistered) {
t.Errorf("N1 (case mismatch): err = %v, want ErrNodeNotRegistered", err)
}
}
func TestNodeKindString(t *testing.T) {
if got := NodeKindLinux.String(); got != "linux" {
t.Errorf("Linux.String() = %q", got)
}
if got := NodeKindProxmox.String(); got != "proxmox" {
t.Errorf("Proxmox.String() = %q", got)
}
// Uppercase constructor should lower-case.
if got := NodeKind("PROXMOX").String(); got != "proxmox" {
t.Errorf("PROXMOX.String() = %q, want proxmox", got)
}
}
+175
View File
@@ -0,0 +1,175 @@
package emitter
import (
"errors"
"fmt"
"strings"
"git.cloudinit.dev/coreci/orca/internal/jobspec"
"git.cloudinit.dev/coreci/orca/internal/storage"
)
// SyncthingEmitter is the Layer-4 emitter for the per-namespace
// Syncthing config files (REQ-081). For every volume in spec.Volumes
// that carries a `replicate:` list, the emitter renders one Syncthing
// `config.xml` at /etc/syncthing/orca-<ns>-<volume>.xml containing the
// content-addressed folder (storage.FolderID), the device list (all
// peers in the namespace), and the volume path.
//
// Syncthing configs are kind-agnostic — they apply to any workload
// (Job, Service, DaemonSet) that declares a replicated volume. The
// emitter is therefore registered on the Registry under every
// kind:runtime key the other emitters use, but it is intended to be
// composed by the caller (the caller renders both the systemd unit and
// the Syncthing config for the same spec). For the v0.9-P09 spike the
// emitter is invoked directly; the composition lands in a later phase.
//
// The emitter is CLI-side only: it renders the XML; the SSH-push
// transport SCPs the file to each peer; the Syncthing apt package on
// the peer reads it. No Syncthing Go client is linked.
type SyncthingEmitter struct{}
// syncthingConfigDir is the canonical directory for rendered Syncthing
// configs on a peer (R-005). The emitter writes one file per
// replicated volume.
const syncthingConfigDir = "/etc/syncthing"
// localDeviceAddress is the address the local peer's own device entry
// uses. "dynamic" tells Syncthing this peer is the listener (it does
// not dial out to itself).
const localDeviceAddress = "dynamic"
// peerDeviceAddressTemplate renders the Sync listen address for a
// remote peer. The peer hostname (from the ReplicateTo list) is used
// as the host; the default Sync port is 22000.
const peerDeviceAddressTemplate = "tcp://%s:22000"
// Render renders one Syncthing config XML file per replicated volume
// in the spec. A volume is "replicated" when its VolumeSpec carries a
// non-empty `replicate:` list — encoded in VolumeSpec.Source as the
// comma-separated peer list prefixed with `replicate:` (e.g.
// `replicate:peer-b,peer-c`). This keeps the VolumeSpec shape stable
// (the v0.9 VolumeSpec has no explicit Replicate field; the emitter
// parses it from Source).
//
// For each replicated volume, the emitter:
//
// 1. Builds a VolumeReplication (namespace = spec.Name's namespace,
// volume name, source path = VolumeSpec.Target).
// 2. Builds the peer device list (the local peer + every peer in the
// `replicate:` list). The local peer's device ID is derived
// deterministically from the node hostname (the real device ID is
// discovered from the peer registry in a later phase; for the
// spike a deterministic placeholder keeps the rendered config
// byte-stable).
// 3. Calls storage.RenderSyncthingConfig + storage.RenderSyncthingXML
// to produce the config file content.
//
// Returns an error if the spec is nil or the node is nil (the node is
// required to identify the local peer). Workloads with no replicated
// volumes return an empty (non-nil) slice — the emitter is a no-op for
// them.
func (SyncthingEmitter) Render(spec *jobspec.WorkloadSpec, node *Node) ([]File, error) {
if spec == nil {
return nil, errors.New("emitter/syncthing: spec is nil")
}
if node == nil {
return nil, errors.New("emitter/syncthing: node is nil (local peer unknown)")
}
var files []File
for _, vol := range spec.Volumes {
peers, ok := parseReplicateList(vol.Source)
if !ok || len(peers) == 0 {
continue
}
path := vol.Target
if strings.TrimSpace(path) == "" {
path = vol.Source
}
rep := storage.VolumeReplication{
Namespace: spec.Name,
VolumeName: vol.Name,
SourcePath: path,
ReplicateTo: peers,
SyncMode: "sendreceive",
}
devices := buildSyncthingDevices(node, peers)
cfg, err := storage.RenderSyncthingConfig(rep, devices)
if err != nil {
return nil, fmt.Errorf("emitter/syncthing: render config for volume %q: %w", vol.Name, err)
}
xml, err := storage.RenderSyncthingXML(cfg)
if err != nil {
return nil, fmt.Errorf("emitter/syncthing: render xml for volume %q: %w", vol.Name, err)
}
files = append(files, File{
Path: fmt.Sprintf("%s/orca-%s-%s.xml", syncthingConfigDir, spec.Name, vol.Name),
Content: xml,
Mode: "0644",
})
}
return files, nil
}
// parseReplicateList extracts the peer list from a VolumeSpec.Source
// value of the form `replicate:peer-b,peer-c`. Returns the peer list
// and true when the Source carries a replicate directive; returns nil
// and false otherwise (the volume is not replicated).
func parseReplicateList(source string) ([]string, bool) {
s := strings.TrimSpace(source)
if !strings.HasPrefix(s, "replicate:") {
return nil, false
}
rest := strings.TrimPrefix(s, "replicate:")
parts := strings.Split(rest, ",")
out := make([]string, 0, len(parts))
for _, p := range parts {
p = strings.TrimSpace(p)
if p != "" {
out = append(out, p)
}
}
return out, true
}
// buildSyncthingDevices builds the Syncthing device list for the
// rendered config. The local peer (the node the config is being
// rendered for) is first, with the local device address ("dynamic").
// Each remote peer in the replicate list follows, with a
// tcp://<peer>:22000 address. Device IDs are deterministic placeholders
// derived from the peer name (the real device IDs are discovered from
// the peer registry in a later phase; the placeholder keeps the
// rendered config byte-stable across re-runs).
func buildSyncthingDevices(node *Node, peers []string) []storage.SyncthingDevice {
devices := make([]storage.SyncthingDevice, 0, len(peers)+1)
devices = append(devices, storage.SyncthingDevice{
ID: syncthingDeviceID(node.Hostname),
Name: node.Hostname,
Address: localDeviceAddress,
})
for _, p := range peers {
devices = append(devices, storage.SyncthingDevice{
ID: syncthingDeviceID(p),
Name: p,
Address: fmt.Sprintf(peerDeviceAddressTemplate, p),
})
}
return devices
}
// syncthingDeviceID returns a deterministic, stable device-ID
// placeholder for the given peer name. The placeholder is a fixed
// 52-char string (Syncthing device IDs are 52-char base32) derived by
// padding the peer name. The real device ID (discovered from the peer
// registry / cluster/peers/) replaces this in a later phase; for the
// v0.9 spike the placeholder keeps the rendered config byte-stable so
// the SSH-push idempotency check works.
func syncthingDeviceID(peerName string) string {
const idLen = 52
name := strings.TrimSpace(peerName)
if len(name) >= idLen {
return name[:idLen]
}
pad := strings.Repeat("X", idLen-len(name))
return name + pad
}
+172
View File
@@ -0,0 +1,172 @@
package emitter
import (
"strings"
"testing"
"git.cloudinit.dev/coreci/orca/internal/jobspec"
)
func TestSyncthingEmitter_TwoReplicatedVolumes_TwoFiles(t *testing.T) {
spec := &jobspec.WorkloadSpec{
Kind: "Service",
Name: "team-alpha",
Volumes: []jobspec.VolumeSpec{
{Name: "data", Source: "replicate:peer-b,peer-c", Target: "/var/lib/orca/data"},
{Name: "logs", Source: "replicate:peer-b", Target: "/var/lib/orca/logs"},
{Name: "cache", Source: "/local/cache", Target: "/cache"}, // not replicated
},
}
node := &Node{Hostname: "peer-a", Runtime: []string{"process"}}
files, err := (SyncthingEmitter{}).Render(spec, node)
if err != nil {
t.Fatalf("Render: %v", err)
}
if len(files) != 2 {
t.Fatalf("expected 2 files (only replicated volumes), got %d", len(files))
}
for _, f := range files {
if !strings.HasPrefix(f.Path, "/etc/syncthing/orca-team-alpha-") {
t.Errorf("path %q does not start with /etc/syncthing/orca-team-alpha-", f.Path)
}
if !strings.HasSuffix(f.Path, ".xml") {
t.Errorf("path %q does not end with .xml", f.Path)
}
if f.Mode != "0644" {
t.Errorf("mode = %q, want 0644", f.Mode)
}
if !strings.Contains(f.Content, "<configuration") {
t.Errorf("content of %q is not syncthing config XML", f.Path)
}
}
// File 1: data volume, replicated to peer-b and peer-c.
if !strings.Contains(files[0].Path, "team-alpha-data") {
t.Errorf("first file path = %q, want team-alpha-data", files[0].Path)
}
if !strings.Contains(files[0].Content, "peer-b") || !strings.Contains(files[0].Content, "peer-c") {
t.Errorf("data volume config missing peer-b or peer-c")
}
// File 2: logs volume, replicated to peer-b only.
if !strings.Contains(files[1].Path, "team-alpha-logs") {
t.Errorf("second file path = %q, want team-alpha-logs", files[1].Path)
}
if !strings.Contains(files[1].Content, "peer-b") {
t.Errorf("logs volume config missing peer-b")
}
if strings.Contains(files[1].Content, "tcp://peer-c") {
t.Errorf("logs volume config should not contain peer-c")
}
}
func TestSyncthingEmitter_NoReplicatedVolumes_Empty(t *testing.T) {
spec := &jobspec.WorkloadSpec{
Kind: "Job",
Name: "batch",
Volumes: []jobspec.VolumeSpec{
{Name: "cache", Source: "/local/cache", Target: "/cache"},
},
}
node := &Node{Hostname: "peer-a"}
files, err := (SyncthingEmitter{}).Render(spec, node)
if err != nil {
t.Fatalf("Render: %v", err)
}
if len(files) != 0 {
t.Errorf("expected 0 files for non-replicated volumes, got %d", len(files))
}
}
func TestSyncthingEmitter_NoVolumes_Empty(t *testing.T) {
spec := &jobspec.WorkloadSpec{Kind: "Job", Name: "batch"}
node := &Node{Hostname: "peer-a"}
files, err := (SyncthingEmitter{}).Render(spec, node)
if err != nil {
t.Fatalf("Render: %v", err)
}
if len(files) != 0 {
t.Errorf("expected 0 files, got %d", len(files))
}
}
func TestSyncthingEmitter_NilSpec_Error(t *testing.T) {
node := &Node{Hostname: "peer-a"}
if _, err := (SyncthingEmitter{}).Render(nil, node); err == nil {
t.Error("nil spec: expected error")
}
}
func TestSyncthingEmitter_NilNode_Error(t *testing.T) {
spec := &jobspec.WorkloadSpec{Kind: "Job", Name: "x"}
if _, err := (SyncthingEmitter{}).Render(spec, nil); err == nil {
t.Error("nil node: expected error")
}
}
func TestSyncthingEmitter_LocalDeviceFirst(t *testing.T) {
spec := &jobspec.WorkloadSpec{
Kind: "Service",
Name: "ns",
Volumes: []jobspec.VolumeSpec{{Name: "data", Source: "replicate:peer-b", Target: "/data"}},
}
node := &Node{Hostname: "peer-a"}
files, err := (SyncthingEmitter{}).Render(spec, node)
if err != nil {
t.Fatalf("Render: %v", err)
}
if len(files) != 1 {
t.Fatalf("expected 1 file, got %d", len(files))
}
// Local peer address is "dynamic"; remote peer uses tcp://...
if !strings.Contains(files[0].Content, "dynamic") {
t.Errorf("local device address (dynamic) missing from config")
}
if !strings.Contains(files[0].Content, "tcp://peer-b:22000") {
t.Errorf("remote peer address missing from config")
}
}
func TestParseReplicateList_OK(t *testing.T) {
peers, ok := parseReplicateList("replicate:peer-b,peer-c,peer-d")
if !ok {
t.Fatal("expected ok")
}
if len(peers) != 3 || peers[0] != "peer-b" || peers[1] != "peer-c" || peers[2] != "peer-d" {
t.Errorf("peers = %v, want [peer-b peer-c peer-d]", peers)
}
}
func TestParseReplicateList_NotReplicated(t *testing.T) {
_, ok := parseReplicateList("/local/path")
if ok {
t.Error("non-replicate source should return ok=false")
}
}
func TestParseReplicateList_EmptyPeerList(t *testing.T) {
peers, ok := parseReplicateList("replicate:")
if !ok {
t.Error("replicate: prefix should return ok=true")
}
if len(peers) != 0 {
t.Errorf("peers = %v, want empty", peers)
}
}
func TestSyncthingDeviceID_Stable(t *testing.T) {
a := syncthingDeviceID("peer-a")
b := syncthingDeviceID("peer-a")
if a != b {
t.Errorf("syncthingDeviceID not stable: %q vs %q", a, b)
}
if len(a) != 52 {
t.Errorf("device ID length = %d, want 52", len(a))
}
}
func TestSyncthingDeviceID_DifferentPeers(t *testing.T) {
a := syncthingDeviceID("peer-a")
b := syncthingDeviceID("peer-b")
if a == b {
t.Errorf("different peers produced same device ID")
}
}
+43
View File
@@ -0,0 +1,43 @@
# C-01 Grill Gate Evaluation — wasmtime / CGO
**Gate**: C-01 (wasmtime/CGO evaluation), gating P07b.
**Question**: The wasmtime Go binding
(`github.com/bytecodealliance/wasmtime-go`) is CGO-based. Does adopting
it revoke D-002 (modernc/sqlite CGO-free cross-compile story)?
## Evaluation
| Option | CGO required? | Cross-compile impact | Decision |
|--------|---------------|----------------------|----------|
| A. Use `bytecodealliance/wasmtime-go` (Go binding) | **YES** — the binding links libwasmtime via cgo | Revokes D-002 — Go cross-compile (`GOOS=linux GOARCH=arm64 go build`) breaks; CGO toolchain needed on every build host; static-binary story lost | REJECTED |
| B. Use the `wasmtime` CLI (apt-installed on the peer) via SSH exec | **NO** — pure Go code, shells out to a CLI over SSH (same pattern as podman/qm/pct) | None — D-002 preserved | **ACCEPTED** |
| C. Use an alternative pure-Go WASM runtime (e.g. wazero) | No CGO | Pure-Go alternative exists; but wazero's wasmtime-compat is incomplete (component model, WASI 0.2); different runtime semantics than the "wasmtime" operator surface promised in D-088 | DEFERRED (v0.10 evaluation if CLI-via-SSH proves insufficient) |
## Decision (full autonomy, auto-decision)
**Option B**: `WasmRuntime` uses the `wasmtime` CLI (apt-installed on
the peer) via the SSH-push transport. It does NOT import
`bytecodealliance/wasmtime-go` (or any other CGO package).
## C-01 Gate Status
**SATISFIED.** C-01 is satisfied:
- wasmtime works without CGO (CLI-via-SSH pattern, identical to podman/qm/pct).
- D-002 cross-compile story preserved (no CGO introduced anywhere in
the runtime package or any orca Go code).
- D-002 is NOT revoked.
## Recorded as D-187
See PROJECT.md v0.9 D-series: "wasmtime Go binding is CGO-based
(bytecodealliance/wasmtime-go); orca uses the wasmtime CLI via SSH
(apt-installed on peer) instead of the Go binding, avoiding CGO
entirely. D-002 cross-compile story preserved. C-01 satisfied."
## Verification
- `go build ./internal/runtime/` succeeds with `CGO_ENABLED=0`.
- `internal/runtime/wasm.go` imports only stdlib + sshpush (no
wasmtime-go).
- The full test suite (`go test ./...`) does not require CGO.
+142
View File
@@ -0,0 +1,142 @@
package runtime
import (
"context"
"fmt"
"strings"
"git.cloudinit.dev/coreci/orca/internal/sshpush"
)
// PodmanRuntime implements Runtime for the "podman" one_of. It runs
// `podman` on the peer over the SSH-push transport (P01). The
// transport is injected via the constructor (dependency injection).
//
// Container lifecycle:
//
// - Prepare: `podman pull <image>`
// - Start: `podman run -d --name orca-<alloc-id> <image> <command>`
// - Stop: `podman stop <name>` then `podman rm <name>`
// - Status: `podman inspect --format '{{.State.Running}}' <name>`
//
// The container name is `orca-<alloc-id>` (sanitized to lowercase +
// alnum). The runtime keeps no in-process state — each call is a fresh
// SSH exec against the peer.
type PodmanRuntime struct {
transport *sshpush.Transport
}
// NewPodmanRuntime returns a PodmanRuntime backed by the given transport.
func NewPodmanRuntime(t *sshpush.Transport) *PodmanRuntime {
return &PodmanRuntime{transport: t}
}
// containerName returns the deterministic container name for an alloc.
func containerName(alloc *Alloc) string {
id := strings.ToLower(alloc.ID)
id = strings.Map(func(r rune) rune {
if r >= 'a' && r <= 'z' || r >= '0' && r <= '9' || r == '-' || r == '_' {
return r
}
return '-'
}, id)
return "orca-" + id
}
// Prepare pulls the image on the peer.
func (p *PodmanRuntime) Prepare(ctx context.Context, alloc *Alloc) error {
image, err := imageFor(alloc)
if err != nil {
return err
}
cmd := fmt.Sprintf("podman pull %q", image)
if _, err := p.transport.Exec(ctx, alloc.Node, cmd); err != nil {
return fmt.Errorf("podman: pull: %w", err)
}
return nil
}
// Start runs `podman run -d --name <name> <image> <command>`.
func (p *PodmanRuntime) Start(ctx context.Context, alloc *Alloc) (int, error) {
image, err := imageFor(alloc)
if err != nil {
return 0, err
}
cmdStr, _ := commandFor(alloc)
name := containerName(alloc)
cmd := fmt.Sprintf("podman run -d --name %s %q %s", name, image, cmdStr)
out, err := p.transport.Exec(ctx, alloc.Node, cmd)
if err != nil {
return 0, fmt.Errorf("podman: run: %w", err)
}
// The container ID is the first 12 chars of the printed hash. We
// don't keep it — Stop/Status use the name — but return a stable
// synthetic PID derived from the first 4 bytes of the hash for
// the interface contract.
cid := strings.TrimSpace(string(out))
return podmanCidToPID(cid), nil
}
// Stop stops and removes the container.
func (p *PodmanRuntime) Stop(ctx context.Context, alloc *Alloc) error {
name := containerName(alloc)
if _, err := p.transport.Exec(ctx, alloc.Node, fmt.Sprintf("podman stop %s", name)); err != nil {
return fmt.Errorf("podman: stop: %w", err)
}
if _, err := p.transport.Exec(ctx, alloc.Node, fmt.Sprintf("podman rm %s", name)); err != nil {
return fmt.Errorf("podman: rm: %w", err)
}
return nil
}
// Status inspects the container's running state.
func (p *PodmanRuntime) Status(ctx context.Context, alloc *Alloc) (State, error) {
name := containerName(alloc)
cmd := fmt.Sprintf("podman inspect --format '{{.State.Running}}' %s", name)
out, err := p.transport.Exec(ctx, alloc.Node, cmd)
if err != nil {
return StateFailed, fmt.Errorf("podman: inspect: %w", err)
}
v := strings.TrimSpace(string(out))
switch v {
case "true":
return StateRunning, nil
case "false":
return StateStopped, nil
default:
return StateFailed, fmt.Errorf("podman: unexpected inspect output %q", v)
}
}
// podmanCidToPID converts a container ID (hex hash) to a positive int
// PID for the interface contract. It reads up to 4 hex chars.
func podmanCidToPID(cid string) int {
if len(cid) < 1 {
return 1
}
n := len(cid)
if n > 4 {
n = 4
}
var pid int
for i := 0; i < n; i++ {
c := cid[i]
pid = (pid << 4) | int(hexVal(c))
}
if pid <= 0 {
pid = 1
}
return pid
}
func hexVal(c byte) byte {
switch {
case c >= '0' && c <= '9':
return c - '0'
case c >= 'a' && c <= 'f':
return c - 'a' + 10
case c >= 'A' && c <= 'F':
return c - 'A' + 10
}
return 0
}
+229
View File
@@ -0,0 +1,229 @@
package runtime
import (
"context"
"errors"
"strings"
"testing"
"time"
"git.cloudinit.dev/coreci/orca/internal/jobspec"
"git.cloudinit.dev/coreci/orca/internal/sshpush"
)
// allocNoImage returns an alloc whose Spec has a Runtime block with a
// command but no image.
func allocNoImage(runtime string) *Alloc {
return &Alloc{
ID: "x",
Runtime: runtime,
Spec: &jobspec.WorkloadSpec{
Runtime: &jobspec.RuntimeBlock{Command: "/bin/true"},
},
}
}
// TestPodmanRuntime_HappyPath wires a fake server that responds to
// podman pull/run/stop/rm/inspect and verifies the full lifecycle.
func TestPodmanRuntime_HappyPath(t *testing.T) {
srv := newFakeServer(t)
defer srv.close()
const cid = "abc123def456"
srv.setHandler("podman pull", func(cmd string) ([]byte, int) { return nil, 0 })
srv.setHandler("podman run", func(cmd string) ([]byte, int) { return []byte(cid + "\n"), 0 })
srv.setHandler("podman stop", func(cmd string) ([]byte, int) { return nil, 0 })
srv.setHandler("podman rm", func(cmd string) ([]byte, int) { return nil, 0 })
srv.setHandler("podman inspect", func(cmd string) ([]byte, int) {
return []byte("true\n"), 0
})
tr := realTransport(t, srv)
defer tr.Close()
p := NewPodmanRuntime(tr)
a := allocWithNode("podman", "docker.io/library/alpine:latest", "sleep 30", srv.addr())
ctx, cancel := withTimeout(10 * time.Second)
defer cancel()
if err := p.Prepare(ctx, a); err != nil {
t.Fatalf("Prepare: %v", err)
}
pid, err := p.Start(ctx, a)
if err != nil {
t.Fatalf("Start: %v", err)
}
if pid <= 0 {
t.Fatalf("pid = %d, want > 0", pid)
}
st, err := p.Status(ctx, a)
if err != nil {
t.Fatalf("Status: %v", err)
}
if st != StateRunning {
t.Errorf("Status = %q, want running", st)
}
if err := p.Stop(ctx, a); err != nil {
t.Fatalf("Stop: %v", err)
}
}
// TestPodmanRuntime_StatusFalse verifies Status returns stopped when
// the container reports running=false.
func TestPodmanRuntime_StatusFalse(t *testing.T) {
srv := newFakeServer(t)
defer srv.close()
srv.setHandler("podman inspect", func(cmd string) ([]byte, int) {
return []byte("false\n"), 0
})
tr := realTransport(t, srv)
defer tr.Close()
p := NewPodmanRuntime(tr)
a := allocWithNode("podman", "img", "sleep 1", srv.addr())
st, err := p.Status(context.Background(), a)
if err != nil {
t.Fatalf("Status: %v", err)
}
if st != StateStopped {
t.Errorf("Status = %q, want stopped", st)
}
}
// TestPodmanRuntime_StatusBadOutput verifies Status returns failed on
// unexpected inspect output.
func TestPodmanRuntime_StatusBadOutput(t *testing.T) {
srv := newFakeServer(t)
defer srv.close()
srv.setHandler("podman inspect", func(cmd string) ([]byte, int) {
return []byte("garbage\n"), 0
})
tr := realTransport(t, srv)
defer tr.Close()
p := NewPodmanRuntime(tr)
a := allocWithNode("podman", "img", "sleep 1", srv.addr())
if _, err := p.Status(context.Background(), a); err == nil {
t.Error("Status with bad output should error")
}
}
// TestPodmanRuntime_PrepareNoImage verifies Prepare errors when the
// alloc has no image.
func TestPodmanRuntime_PrepareNoImage(t *testing.T) {
p := NewPodmanRuntime(nil)
a := allocNoImage("podman")
if err := p.Prepare(context.Background(), a); err == nil {
t.Error("Prepare with no image should error")
}
}
// TestPodmanRuntime_PrepareTransportError verifies Prepare propagates a
// transport error (podman pull fails).
func TestPodmanRuntime_PrepareTransportError(t *testing.T) {
srv := newFakeServer(t)
defer srv.close()
srv.setHandler("podman pull", func(cmd string) ([]byte, int) {
return []byte("manifest unknown\n"), 2
})
tr := realTransport(t, srv)
defer tr.Close()
p := NewPodmanRuntime(tr)
a := allocWithNode("podman", "img", "sleep 1", srv.addr())
if err := p.Prepare(context.Background(), a); err == nil {
t.Error("Prepare with failed pull should error")
}
}
// TestPodmanRuntime_StartNoImage verifies Start errors with no image.
func TestPodmanRuntime_StartNoImage(t *testing.T) {
p := NewPodmanRuntime(nil)
a := allocNoImage("podman")
if _, err := p.Start(context.Background(), a); err == nil {
t.Error("Start with no image should error")
}
}
// TestPodmanRuntime_StopTransportError verifies Stop propagates errors.
func TestPodmanRuntime_StopTransportError(t *testing.T) {
srv := newFakeServer(t)
defer srv.close()
srv.setHandler("podman stop", func(cmd string) ([]byte, int) {
return []byte("no such container\n"), 1
})
tr := realTransport(t, srv)
defer tr.Close()
p := NewPodmanRuntime(tr)
a := allocWithNode("podman", "img", "sleep 1", srv.addr())
if err := p.Stop(context.Background(), a); err == nil {
t.Error("Stop with missing container should error")
}
}
// TestPodmanRuntime_StatusInspectError verifies Status returns failed
// when inspect itself errors.
func TestPodmanRuntime_StatusInspectError(t *testing.T) {
srv := newFakeServer(t)
defer srv.close()
srv.setHandler("podman inspect", func(cmd string) ([]byte, int) {
return []byte("no such container\n"), 1
})
tr := realTransport(t, srv)
defer tr.Close()
p := NewPodmanRuntime(tr)
a := allocWithNode("podman", "img", "sleep 1", srv.addr())
st, err := p.Status(context.Background(), a)
if err == nil {
t.Error("Status with inspect error should error")
}
if st != StateFailed {
t.Errorf("Status = %q, want failed", st)
}
}
// TestPodmanCidToPID verifies the synthetic PID derivation.
func TestPodmanCidToPID(t *testing.T) {
if got := podmanCidToPID(""); got != 1 {
t.Errorf("empty cid -> %d, want 1", got)
}
if got := podmanCidToPID("a"); got <= 0 {
t.Errorf("single hex -> %d, want > 0", got)
}
if got := podmanCidToPID("abcd"); got <= 0 {
t.Errorf("abcd -> %d, want > 0", got)
}
// non-hex chars fall through to 0 contributions but still yield
// a positive result (>= 1 by the floor).
if got := podmanCidToPID("xyz123"); got <= 0 {
t.Errorf("xyz123 -> %d, want > 0", got)
}
}
// TestContainerNameSanitization verifies the container-name sanitizer
// uppercases and strips disallowed characters.
func TestContainerNameSanitization(t *testing.T) {
a := &Alloc{ID: "ALLOC_1.2.3", Spec: nil, Runtime: "podman"}
got := containerName(a)
if !strings.HasPrefix(got, "orca-") {
t.Errorf("containerName = %q, want orca- prefix", got)
}
if strings.Contains(got, ".") {
t.Errorf("containerName = %q, should not contain '.'", got)
}
}
// TestPodmanRuntime_DialError verifies Prepare fails fast when the
// peer is unreachable (no fake server).
func TestPodmanRuntime_DialError(t *testing.T) {
srv := newFakeServer(t)
// close immediately so dial fails.
srv.close()
tr := realTransport(t, srv)
defer tr.Close()
p := NewPodmanRuntime(tr)
a := allocWithNode("podman", "img", "sleep 1", srv.addr())
if err := p.Prepare(context.Background(), a); err == nil {
t.Error("Prepare against dead peer should error")
} else if !errors.Is(err, sshpush.ErrTransient) && !errors.Is(err, sshpush.ErrPermanent) {
// acceptable: either transient (retry exhausted) or permanent.
t.Logf("Prepare err (acceptable): %v", err)
}
}
+204
View File
@@ -0,0 +1,204 @@
package runtime
import (
"context"
"errors"
"fmt"
"os"
"os/exec"
"strconv"
"strings"
"sync"
"syscall"
"time"
)
// StopGrace is the default grace period between SIGTERM and SIGKILL
// for ProcessRuntime.Stop (10s, matching the systemd default TimeoutStopSec).
const StopGrace = 10 * time.Second
// ProcessRuntime implements Runtime for the "process" one_of using
// os/exec. It is the in-process equivalent of the systemd unit the CLI
// emits in production: the CLI emits a .service file and the peer's
// systemd runs the process; ProcessRuntime starts the process directly
// in the current Go process. It is intended for LOCAL testing and
// hermetic CI — NOT for production (production uses the systemd emitter
// + the peer's systemd, not this in-process path).
//
// It tracks started PIDs in an in-memory map; restarts of the CLI lose
// that state (acceptable for the local-test use case).
type ProcessRuntime struct {
mu sync.Mutex
pids map[string]int // alloc.ID -> PID
procs map[int]*os.Process // PID -> process handle
stopped map[string]bool // alloc.ID -> reported stopped after Stop
}
// NewProcessRuntime returns a ProcessRuntime.
func NewProcessRuntime() *ProcessRuntime {
return &ProcessRuntime{
pids: make(map[string]int),
procs: make(map[int]*os.Process),
stopped: make(map[string]bool),
}
}
// Prepare is a no-op for the process runtime: the systemd unit is
// emitted by the systemd emitter (internal/emitter), not by the runtime.
func (p *ProcessRuntime) Prepare(ctx context.Context, alloc *Alloc) error {
_ = ctx
_ = alloc
return nil
}
// Start execs the alloc's command and returns the PID. The process is
// left running in the background; Stop terminates it.
func (p *ProcessRuntime) Start(ctx context.Context, alloc *Alloc) (int, error) {
cmdStr, err := commandFor(alloc)
if err != nil {
return 0, err
}
// Parse the command string into argv. A leading "exec" form
// (shell-style) is NOT supported — the command must be a direct
// argv[0] + args. Split on whitespace (simple, matches the existing
// executor.go behaviour which takes Command + Args separately).
parts := strings.Fields(cmdStr)
if len(parts) == 0 {
return 0, fmt.Errorf("process: empty command for alloc %s", alloc.ID)
}
// Use a detached context so the process survives the request
// context cancellation (the request ends; the workload keeps
// running until Stop). We apply our own timeout for Start only.
startCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
cmd := exec.CommandContext(startCtx, parts[0], parts[1:]...)
// Detach the child from the parent's process group so it survives.
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
// Discard output for the runtime; the systemd unit captures logs
// in production. For tests, callers that need output run their
// own exec.Command.
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
if err := cmd.Start(); err != nil {
return 0, fmt.Errorf("process: start: %w", err)
}
pid := cmd.Process.Pid
// Background-reap the process so it doesn't become a zombie; we
// only need the PID for Stop/Status. When the process exits
// naturally, mark the alloc as stopped.
go func() {
_ = cmd.Wait()
p.mu.Lock()
delete(p.procs, pid)
// Only mark stopped if the alloc is still associated with
// this PID (Stop may have already removed the mapping).
if cur, ok := p.pids[alloc.ID]; ok && cur == pid {
delete(p.pids, alloc.ID)
p.stopped[alloc.ID] = true
}
p.mu.Unlock()
}()
p.mu.Lock()
p.pids[alloc.ID] = pid
p.procs[pid] = cmd.Process
delete(p.stopped, alloc.ID)
p.mu.Unlock()
return pid, nil
}
// Stop sends SIGTERM, waits the grace period, then SIGKILL.
func (p *ProcessRuntime) Stop(ctx context.Context, alloc *Alloc) error {
p.mu.Lock()
pid, ok := p.pids[alloc.ID]
proc := p.procs[pid]
p.mu.Unlock()
if !ok || proc == nil {
return nil // not running; idempotent
}
// SIGTERM the process group (negative PID).
_ = syscall.Kill(-pid, syscall.SIGTERM)
grace := StopGrace
if dl, ok := ctx.Deadline(); ok {
if remaining := time.Until(dl); remaining > 0 && remaining < grace {
grace = remaining
}
}
deadline := time.Now().Add(grace)
for time.Now().Before(deadline) {
if !p.alive(pid) {
p.forget(alloc.ID, pid)
return nil
}
select {
case <-ctx.Done():
p.forget(alloc.ID, pid)
return ctx.Err()
case <-time.After(100 * time.Millisecond):
}
}
// SIGKILL the group.
_ = syscall.Kill(-pid, syscall.SIGKILL)
p.forget(alloc.ID, pid)
return nil
}
// Status reports the alloc's state by checking if the process is alive.
func (p *ProcessRuntime) Status(ctx context.Context, alloc *Alloc) (State, error) {
_ = ctx
p.mu.Lock()
pid, ok := p.pids[alloc.ID]
wasStopped := p.stopped[alloc.ID]
p.mu.Unlock()
if !ok {
if wasStopped {
return StateStopped, nil
}
return StatePending, nil
}
if !p.alive(pid) {
// Process exited but the reaper hasn't run yet; mark it
// stopped and clean up.
p.forget(alloc.ID, pid)
return StateStopped, nil
}
return StateRunning, nil
}
// alive reports whether the process with the given PID is still running.
func (p *ProcessRuntime) alive(pid int) bool {
proc, err := os.FindProcess(pid)
if err != nil {
return false
}
if err := proc.Signal(syscall.Signal(0)); err != nil {
// ESRCH means the process is gone.
return false
}
return true
}
// forget removes the alloc/PID mapping and marks the alloc stopped.
func (p *ProcessRuntime) forget(allocID string, pid int) {
p.mu.Lock()
delete(p.pids, allocID)
delete(p.procs, pid)
p.stopped[allocID] = true
p.mu.Unlock()
}
// PID returns the recorded PID for alloc (for tests/inspection).
func (p *ProcessRuntime) PID(allocID string) (int, error) {
p.mu.Lock()
defer p.mu.Unlock()
pid, ok := p.pids[allocID]
if !ok {
return 0, errors.New("process: no pid for alloc " + strconv.Quote(allocID))
}
return pid, nil
}
+172
View File
@@ -0,0 +1,172 @@
package runtime
import (
"context"
"runtime"
"testing"
"time"
"git.cloudinit.dev/coreci/orca/internal/jobspec"
)
// TestProcessRuntime_PrepareIsNoop verifies Prepare is a no-op.
func TestProcessRuntime_PrepareIsNoop(t *testing.T) {
p := NewProcessRuntime()
a := alloc("process", "", "/bin/true")
if err := p.Prepare(context.Background(), a); err != nil {
t.Fatalf("Prepare: %v", err)
}
}
// TestProcessRuntime_PrepareNilSpec exercises the nil-spec branch
// indirectly — Prepare is a no-op regardless of input.
func TestProcessRuntime_PrepareNilSpec(t *testing.T) {
p := NewProcessRuntime()
if err := p.Prepare(context.Background(), &Alloc{ID: "x"}); err != nil {
t.Fatalf("Prepare (nil spec) should still be no-op: %v", err)
}
}
// TestProcessRuntime_StartStopStatus runs a real long-lived process
// (sleep) and verifies Start -> Status(running) -> Stop -> Status(stopped).
func TestProcessRuntime_StartStopStatus(t *testing.T) {
if _, err := sleepBin(); err != nil {
t.Skipf("sleep binary not available: %v", err)
}
p := NewProcessRuntime()
sleepCmd, _ := sleepBin()
a := alloc("process", "", sleepCmd+" 30")
ctx, cancel := withTimeout(10 * time.Second)
defer cancel()
pid, err := p.Start(ctx, a)
if err != nil {
t.Fatalf("Start: %v", err)
}
if pid <= 0 {
t.Fatalf("pid = %d, want > 0", pid)
}
st, err := p.Status(context.Background(), a)
if err != nil {
t.Fatalf("Status: %v", err)
}
if st != StateRunning {
t.Errorf("Status = %q, want running", st)
}
// Verify PID lookup.
got, err := p.PID(a.ID)
if err != nil || got != pid {
t.Errorf("PID = %d/%v, want %d", got, err, pid)
}
stopCtx, cancelStop := withTimeout(15 * time.Second)
defer cancelStop()
if err := p.Stop(stopCtx, a); err != nil {
t.Fatalf("Stop: %v", err)
}
st2, _ := p.Status(context.Background(), a)
if st2 != StateStopped {
t.Errorf("Status after Stop = %q, want stopped", st2)
}
}
// TestProcessRuntime_StopNotStarted verifies Stop is idempotent on a
// never-started alloc.
func TestProcessRuntime_StopNotStarted(t *testing.T) {
p := NewProcessRuntime()
a := alloc("process", "", "/bin/true")
ctx, cancel := withTimeout(2 * time.Second)
defer cancel()
if err := p.Stop(ctx, a); err != nil {
t.Errorf("Stop on never-started alloc should be no-op, got %v", err)
}
}
// TestProcessRuntime_StatusPending verifies Status returns pending for
// an alloc that was never started.
func TestProcessRuntime_StatusPending(t *testing.T) {
p := NewProcessRuntime()
a := alloc("process", "", "/bin/true")
st, err := p.Status(context.Background(), a)
if err != nil {
t.Fatalf("Status: %v", err)
}
if st != StatePending {
t.Errorf("Status = %q, want pending", st)
}
}
// TestProcessRuntime_StartNoCommand verifies Start errors on missing
// command.
func TestProcessRuntime_StartNoCommand(t *testing.T) {
p := NewProcessRuntime()
a := &Alloc{ID: "x", Spec: nil, Runtime: "process"}
if _, err := p.Start(context.Background(), a); err == nil {
t.Error("Start with nil spec should error")
}
}
// TestProcessRuntime_StartEmptyCommand verifies Start errors when
// the command parses to zero argv.
func TestProcessRuntime_StartEmptyCommand(t *testing.T) {
p := NewProcessRuntime()
a := &Alloc{
ID: "e",
Runtime: "process",
Spec: &jobspec.WorkloadSpec{
Runtime: &jobspec.RuntimeBlock{Command: " "},
},
}
_, err := p.Start(context.Background(), a)
if err == nil {
t.Error("Start with empty command should error")
}
}
// TestProcessRuntime_StartBadBinary verifies Start propagates exec
// errors for a missing binary.
func TestProcessRuntime_StartBadBinary(t *testing.T) {
p := NewProcessRuntime()
a := alloc("process", "", "/no/such/binary/here")
_, err := p.Start(context.Background(), a)
if err == nil {
t.Error("Start with missing binary should error")
}
}
// TestProcessRuntime_StopOnFinishedProcess verifies Stop on a process
// that already exited (e.g. /bin/true) is a no-op (no error).
func TestProcessRuntime_StopOnFinishedProcess(t *testing.T) {
p := NewProcessRuntime()
a := alloc("process", "", "/bin/true")
_, _ = p.Start(context.Background(), a)
// Give /bin/true time to exit.
time.Sleep(200 * time.Millisecond)
ctx, cancel := withTimeout(5 * time.Second)
defer cancel()
if err := p.Stop(ctx, a); err != nil {
t.Errorf("Stop on exited process should be no-op, got %v", err)
}
}
// TestProcessRuntime_PIDUnknown verifies PID lookup errors for an
// unknown alloc.
func TestProcessRuntime_PIDUnknown(t *testing.T) {
p := NewProcessRuntime()
if _, err := p.PID("nope"); err == nil {
t.Error("PID unknown should error")
}
}
// sleepBin returns the sleep command path ("sleep") on this OS.
func sleepBin() (string, error) {
// /bin/sleep exists on Linux; on other platforms fall back to
// "sleep" (resolved via PATH).
if runtime.GOOS == "linux" {
return "/bin/sleep", nil
}
return "sleep", nil
}
+177
View File
@@ -0,0 +1,177 @@
package runtime
import (
"context"
"fmt"
"strings"
"git.cloudinit.dev/coreci/orca/internal/sshpush"
)
// vmidFor returns a deterministic 5-digit VMID derived from the alloc
// ID (hash(alloc.ID) % 99999 + 1). Used by both pveVMRuntime and
// pveCTRuntime — VM and container IDs share the same numeric space on
// a Proxmox node, but the namespace is sparse (one alloc = one VMID)
// so collisions are rare in practice.
func vmidFor(alloc *Alloc) int {
id := allocIDHash(alloc.ID)
if id < 100 {
id += 100
}
return id
}
// PveVMRuntime implements Runtime for the "pve-vm" one_of using the
// `qm` tool over SSH on a Proxmox peer (extends REQ-076). The transport
// is injected.
//
// Lifecycle:
//
// - Prepare: `qm create <vmid> --memory <mb> --cores <n> --scsi0 <disk>`
// - Start: `qm start <vmid>`
// - Stop: `qm shutdown <vmid>` (graceful) then `qm stop <vmid>` (force)
// - Status: `qm status <vmid>`
type PveVMRuntime struct {
transport *sshpush.Transport
}
// NewPveVMRuntime returns a PveVMRuntime backed by the given transport.
func NewPveVMRuntime(t *sshpush.Transport) *PveVMRuntime {
return &PveVMRuntime{transport: t}
}
// Prepare creates the VM. Memory/Cores default to 512MB / 1 if the
// spec's Runtime block omits them; the disk is the spec's image field
// (a Proxmox storage path like local:vmdir/disk.qcow2).
func (v *PveVMRuntime) Prepare(ctx context.Context, alloc *Alloc) error {
if alloc == nil || alloc.Spec == nil || alloc.Spec.Runtime == nil {
return fmt.Errorf("pve-vm: nil alloc/spec/runtime")
}
vmid := vmidFor(alloc)
image := alloc.Spec.Runtime.Image
if image == "" {
return fmt.Errorf("pve-vm: alloc %s has no disk (Runtime.Image)", alloc.ID)
}
mb := 512
cores := 1
cmd := fmt.Sprintf("qm create %d --memory %d --cores %d --scsi0 %q", vmid, mb, cores, image)
if _, err := v.transport.Exec(ctx, alloc.Node, cmd); err != nil {
return fmt.Errorf("pve-vm: create: %w", err)
}
return nil
}
// Start boots the VM.
func (v *PveVMRuntime) Start(ctx context.Context, alloc *Alloc) (int, error) {
vmid := vmidFor(alloc)
if _, err := v.transport.Exec(ctx, alloc.Node, fmt.Sprintf("qm start %d", vmid)); err != nil {
return 0, fmt.Errorf("pve-vm: start: %w", err)
}
return vmid, nil
}
// Stop gracefully shuts down then force-stops the VM.
func (v *PveVMRuntime) Stop(ctx context.Context, alloc *Alloc) error {
vmid := vmidFor(alloc)
if _, err := v.transport.Exec(ctx, alloc.Node, fmt.Sprintf("qm shutdown %d", vmid)); err != nil {
// Best-effort graceful; fall through to force stop.
}
if _, err := v.transport.Exec(ctx, alloc.Node, fmt.Sprintf("qm stop %d", vmid)); err != nil {
return fmt.Errorf("pve-vm: stop: %w", err)
}
return nil
}
// Status reports the VM state from `qm status`.
func (v *PveVMRuntime) Status(ctx context.Context, alloc *Alloc) (State, error) {
vmid := vmidFor(alloc)
out, err := v.transport.Exec(ctx, alloc.Node, fmt.Sprintf("qm status %d", vmid))
if err != nil {
return StateFailed, fmt.Errorf("pve-vm: status: %w", err)
}
s := strings.ToLower(string(out))
switch {
case strings.Contains(s, "running"):
return StateRunning, nil
case strings.Contains(s, "stopped"):
return StateStopped, nil
default:
return StatePending, nil
}
}
// PveCTRuntime implements Runtime for the "pve-ct" one_of using the
// `pct` tool over SSH on a Proxmox peer (extends REQ-076).
//
// Lifecycle:
//
// - Prepare: `pct create <vmid> <template> --memory <mb> --cores <n>`
// - Start: `pct start <vmid>`
// - Stop: `pct shutdown <vmid>` then `pct stop <vmid>`
// - Status: `pct status <vmid>`
type PveCTRuntime struct {
transport *sshpush.Transport
}
// NewPveCTRuntime returns a PveCTRuntime backed by the given transport.
func NewPveCTRuntime(t *sshpush.Transport) *PveCTRuntime {
return &PveCTRuntime{transport: t}
}
// Prepare creates the LXC container.
func (c *PveCTRuntime) Prepare(ctx context.Context, alloc *Alloc) error {
if alloc == nil || alloc.Spec == nil || alloc.Spec.Runtime == nil {
return fmt.Errorf("pve-ct: nil alloc/spec/runtime")
}
vmid := vmidFor(alloc)
template := alloc.Spec.Runtime.Image
if template == "" {
return fmt.Errorf("pve-ct: alloc %s has no template (Runtime.Image)", alloc.ID)
}
mb := 512
cores := 1
cmd := fmt.Sprintf("pct create %d %q --memory %d --cores %d", vmid, template, mb, cores)
if _, err := c.transport.Exec(ctx, alloc.Node, cmd); err != nil {
return fmt.Errorf("pve-ct: create: %w", err)
}
return nil
}
// Start boots the container.
func (c *PveCTRuntime) Start(ctx context.Context, alloc *Alloc) (int, error) {
vmid := vmidFor(alloc)
if _, err := c.transport.Exec(ctx, alloc.Node, fmt.Sprintf("pct start %d", vmid)); err != nil {
return 0, fmt.Errorf("pve-ct: start: %w", err)
}
return vmid, nil
}
// Stop gracefully then force stops the container.
func (c *PveCTRuntime) Stop(ctx context.Context, alloc *Alloc) error {
vmid := vmidFor(alloc)
if _, err := c.transport.Exec(ctx, alloc.Node, fmt.Sprintf("pct shutdown %d", vmid)); err != nil {
// best-effort
}
if _, err := c.transport.Exec(ctx, alloc.Node, fmt.Sprintf("pct stop %d", vmid)); err != nil {
return fmt.Errorf("pve-ct: stop: %w", err)
}
return nil
}
// Status reports the container state from `pct status`.
func (c *PveCTRuntime) Status(ctx context.Context, alloc *Alloc) (State, error) {
vmid := vmidFor(alloc)
out, err := c.transport.Exec(ctx, alloc.Node, fmt.Sprintf("pct status %d", vmid))
if err != nil {
return StateFailed, fmt.Errorf("pve-ct: status: %w", err)
}
s := strings.ToLower(string(out))
switch {
case strings.Contains(s, "running"):
return StateRunning, nil
case strings.Contains(s, "stopped"):
return StateStopped, nil
default:
return StatePending, nil
}
}
+342
View File
@@ -0,0 +1,342 @@
package runtime
import (
"context"
"testing"
"time"
)
// TestPveVMRuntime_HappyPath verifies the qm lifecycle.
func TestPveVMRuntime_HappyPath(t *testing.T) {
srv := newFakeServer(t)
defer srv.close()
srv.setHandler("qm create", func(cmd string) ([]byte, int) { return nil, 0 })
srv.setHandler("qm start", func(cmd string) ([]byte, int) { return nil, 0 })
srv.setHandler("qm shutdown", func(cmd string) ([]byte, int) { return nil, 0 })
srv.setHandler("qm stop", func(cmd string) ([]byte, int) { return nil, 0 })
srv.setHandler("qm status", func(cmd string) ([]byte, int) {
return []byte("status: running\n"), 0
})
tr := realTransport(t, srv)
defer tr.Close()
v := NewPveVMRuntime(tr)
a := allocWithNode("pve-vm", "local:vmdir/disk.qcow2", "", srv.addr())
ctx, cancel := withTimeout(10 * time.Second)
defer cancel()
if err := v.Prepare(ctx, a); err != nil {
t.Fatalf("Prepare: %v", err)
}
pid, err := v.Start(ctx, a)
if err != nil {
t.Fatalf("Start: %v", err)
}
if pid <= 0 {
t.Fatalf("pid = %d, want > 0", pid)
}
st, err := v.Status(ctx, a)
if err != nil {
t.Fatalf("Status: %v", err)
}
if st != StateRunning {
t.Errorf("Status = %q, want running", st)
}
if err := v.Stop(ctx, a); err != nil {
t.Fatalf("Stop: %v", err)
}
}
// TestPveVMRuntime_StatusStopped verifies Status maps "stopped".
func TestPveVMRuntime_StatusStopped(t *testing.T) {
srv := newFakeServer(t)
defer srv.close()
srv.setHandler("qm status", func(cmd string) ([]byte, int) {
return []byte("status: stopped\n"), 0
})
tr := realTransport(t, srv)
defer tr.Close()
v := NewPveVMRuntime(tr)
a := allocWithNode("pve-vm", "img", "", srv.addr())
st, _ := v.Status(context.Background(), a)
if st != StateStopped {
t.Errorf("Status = %q, want stopped", st)
}
}
// TestPveVMRuntime_StatusUnknown verifies Status returns pending on
// unrecognized output.
func TestPveVMRuntime_StatusUnknown(t *testing.T) {
srv := newFakeServer(t)
defer srv.close()
srv.setHandler("qm status", func(cmd string) ([]byte, int) {
return []byte("weird state\n"), 0
})
tr := realTransport(t, srv)
defer tr.Close()
v := NewPveVMRuntime(tr)
a := allocWithNode("pve-vm", "img", "", srv.addr())
st, _ := v.Status(context.Background(), a)
if st != StatePending {
t.Errorf("Status = %q, want pending", st)
}
}
// TestPveVMRuntime_PrepareNoImage verifies Prepare errors with no disk.
func TestPveVMRuntime_PrepareNoImage(t *testing.T) {
v := NewPveVMRuntime(nil)
a := allocNoImage("pve-vm")
if err := v.Prepare(context.Background(), a); err == nil {
t.Error("Prepare with no disk should error")
}
}
// TestPveVMRuntime_PrepareNilRuntime verifies Prepare errors when
// the Spec has no Runtime block at all.
func TestPveVMRuntime_PrepareNilRuntime(t *testing.T) {
v := NewPveVMRuntime(nil)
a := &Alloc{ID: "x", Runtime: "pve-vm", Spec: nil}
if err := v.Prepare(context.Background(), a); err == nil {
t.Error("Prepare with nil spec should error")
}
}
// TestPveVMRuntime_StartError verifies Start propagates qm start errors.
func TestPveVMRuntime_StartError(t *testing.T) {
srv := newFakeServer(t)
defer srv.close()
srv.setHandler("qm start", func(cmd string) ([]byte, int) {
return []byte("already running\n"), 1
})
tr := realTransport(t, srv)
defer tr.Close()
v := NewPveVMRuntime(tr)
a := allocWithNode("pve-vm", "img", "", srv.addr())
if _, err := v.Start(context.Background(), a); err == nil {
t.Error("Start with qm error should error")
}
}
// TestPveVMRuntime_StopError verifies Stop errors on qm stop failure.
func TestPveVMRuntime_StopError(t *testing.T) {
srv := newFakeServer(t)
defer srv.close()
srv.setHandler("qm shutdown", func(cmd string) ([]byte, int) { return nil, 0 })
srv.setHandler("qm stop", func(cmd string) ([]byte, int) {
return []byte("vm locked\n"), 1
})
tr := realTransport(t, srv)
defer tr.Close()
v := NewPveVMRuntime(tr)
a := allocWithNode("pve-vm", "img", "", srv.addr())
if err := v.Stop(context.Background(), a); err == nil {
t.Error("Stop with qm error should error")
}
}
// TestPveVMRuntime_StatusError verifies Status returns failed on qm
// status error.
func TestPveVMRuntime_StatusError(t *testing.T) {
srv := newFakeServer(t)
defer srv.close()
srv.setHandler("qm status", func(cmd string) ([]byte, int) {
return []byte("no such vm\n"), 1
})
tr := realTransport(t, srv)
defer tr.Close()
v := NewPveVMRuntime(tr)
a := allocWithNode("pve-vm", "img", "", srv.addr())
st, err := v.Status(context.Background(), a)
if err == nil {
t.Error("Status with qm error should error")
}
if st != StateFailed {
t.Errorf("Status = %q, want failed", st)
}
}
// TestPveVMRuntime_PrepareError verifies Prepare propagates qm create
// errors.
func TestPveVMRuntime_PrepareError(t *testing.T) {
srv := newFakeServer(t)
defer srv.close()
srv.setHandler("qm create", func(cmd string) ([]byte, int) {
return []byte("vmid already exists\n"), 1
})
tr := realTransport(t, srv)
defer tr.Close()
v := NewPveVMRuntime(tr)
a := allocWithNode("pve-vm", "img", "", srv.addr())
if err := v.Prepare(context.Background(), a); err == nil {
t.Error("Prepare with qm create error should error")
}
}
// --- PveCTRuntime ---
func TestPveCTRuntime_HappyPath(t *testing.T) {
srv := newFakeServer(t)
defer srv.close()
srv.setHandler("pct create", func(cmd string) ([]byte, int) { return nil, 0 })
srv.setHandler("pct start", func(cmd string) ([]byte, int) { return nil, 0 })
srv.setHandler("pct shutdown", func(cmd string) ([]byte, int) { return nil, 0 })
srv.setHandler("pct stop", func(cmd string) ([]byte, int) { return nil, 0 })
srv.setHandler("pct status", func(cmd string) ([]byte, int) {
return []byte("status: running\n"), 0
})
tr := realTransport(t, srv)
defer tr.Close()
c := NewPveCTRuntime(tr)
a := allocWithNode("pve-ct", "local:vztmpl/alpine.tar.xz", "", srv.addr())
ctx, cancel := withTimeout(10 * time.Second)
defer cancel()
if err := c.Prepare(ctx, a); err != nil {
t.Fatalf("Prepare: %v", err)
}
pid, err := c.Start(ctx, a)
if err != nil {
t.Fatalf("Start: %v", err)
}
if pid <= 0 {
t.Fatalf("pid = %d, want > 0", pid)
}
st, err := c.Status(ctx, a)
if err != nil {
t.Fatalf("Status: %v", err)
}
if st != StateRunning {
t.Errorf("Status = %q, want running", st)
}
if err := c.Stop(ctx, a); err != nil {
t.Fatalf("Stop: %v", err)
}
}
// TestPveCTRuntime_StatusStopped verifies pct status stopped.
func TestPveCTRuntime_StatusStopped(t *testing.T) {
srv := newFakeServer(t)
defer srv.close()
srv.setHandler("pct status", func(cmd string) ([]byte, int) {
return []byte("status: stopped\n"), 0
})
tr := realTransport(t, srv)
defer tr.Close()
c := NewPveCTRuntime(tr)
a := allocWithNode("pve-ct", "tmpl", "", srv.addr())
st, _ := c.Status(context.Background(), a)
if st != StateStopped {
t.Errorf("Status = %q, want stopped", st)
}
}
// TestPveCTRuntime_StatusUnknown verifies pending on unrecognized
// output.
func TestPveCTRuntime_StatusUnknown(t *testing.T) {
srv := newFakeServer(t)
defer srv.close()
srv.setHandler("pct status", func(cmd string) ([]byte, int) {
return []byte("unknown\n"), 0
})
tr := realTransport(t, srv)
defer tr.Close()
c := NewPveCTRuntime(tr)
a := allocWithNode("pve-ct", "tmpl", "", srv.addr())
st, _ := c.Status(context.Background(), a)
if st != StatePending {
t.Errorf("Status = %q, want pending", st)
}
}
// TestPveCTRuntime_PrepareNoImage verifies Prepare errors with no
// template.
func TestPveCTRuntime_PrepareNoImage(t *testing.T) {
c := NewPveCTRuntime(nil)
a := allocNoImage("pve-ct")
if err := c.Prepare(context.Background(), a); err == nil {
t.Error("Prepare with no template should error")
}
}
// TestPveCTRuntime_PrepareNilRuntime verifies Prepare errors on nil
// spec.
func TestPveCTRuntime_PrepareNilRuntime(t *testing.T) {
c := NewPveCTRuntime(nil)
a := &Alloc{ID: "x", Runtime: "pve-ct", Spec: nil}
if err := c.Prepare(context.Background(), a); err == nil {
t.Error("Prepare with nil spec should error")
}
}
// TestPveCTRuntime_StartError verifies Start errors on pct start fail.
func TestPveCTRuntime_StartError(t *testing.T) {
srv := newFakeServer(t)
defer srv.close()
srv.setHandler("pct start", func(cmd string) ([]byte, int) {
return []byte("already running\n"), 1
})
tr := realTransport(t, srv)
defer tr.Close()
c := NewPveCTRuntime(tr)
a := allocWithNode("pve-ct", "tmpl", "", srv.addr())
if _, err := c.Start(context.Background(), a); err == nil {
t.Error("Start with pct error should error")
}
}
// TestPveCTRuntime_StopError verifies Stop errors on pct stop fail.
func TestPveCTRuntime_StopError(t *testing.T) {
srv := newFakeServer(t)
defer srv.close()
srv.setHandler("pct shutdown", func(cmd string) ([]byte, int) { return nil, 0 })
srv.setHandler("pct stop", func(cmd string) ([]byte, int) {
return []byte("container locked\n"), 1
})
tr := realTransport(t, srv)
defer tr.Close()
c := NewPveCTRuntime(tr)
a := allocWithNode("pve-ct", "tmpl", "", srv.addr())
if err := c.Stop(context.Background(), a); err == nil {
t.Error("Stop with pct error should error")
}
}
// TestPveCTRuntime_StatusError verifies Status failed on pct status
// error.
func TestPveCTRuntime_StatusError(t *testing.T) {
srv := newFakeServer(t)
defer srv.close()
srv.setHandler("pct status", func(cmd string) ([]byte, int) {
return []byte("no such container\n"), 1
})
tr := realTransport(t, srv)
defer tr.Close()
c := NewPveCTRuntime(tr)
a := allocWithNode("pve-ct", "tmpl", "", srv.addr())
st, err := c.Status(context.Background(), a)
if err == nil {
t.Error("Status with pct error should error")
}
if st != StateFailed {
t.Errorf("Status = %q, want failed", st)
}
}
// TestVMIDFor verifies the VMID is deterministic and within range.
func TestVMIDFor(t *testing.T) {
a := &Alloc{ID: "alloc-1"}
id := vmidFor(a)
if id < 100 || id > 99999 {
t.Errorf("vmidFor = %d, want in [100, 99999]", id)
}
// stability
if vmidFor(a) != id {
t.Error("vmidFor not stable")
}
// two allocs should differ
b := &Alloc{ID: "alloc-2"}
if vmidFor(b) == id {
t.Logf("note: two allocs collided on vmid (rare but allowed)")
}
}
+20
View File
@@ -0,0 +1,20 @@
package runtime
import (
"git.cloudinit.dev/coreci/orca/internal/sshpush"
)
// DefaultRegistry returns a Registry with all five runtime backends
// registered (process, podman, wasm, pve-vm, pve-ct). The process
// runtime is transport-less; the others are backed by the given
// transport (which may be nil — the per-method calls will fail with
// an sshpush error, but registration still succeeds).
func DefaultRegistry(transport *sshpush.Transport) *Registry {
r := NewRegistry()
r.Register("process", NewProcessRuntime())
r.Register("podman", NewPodmanRuntime(transport))
r.Register("wasm", NewWasmRuntime(transport))
r.Register("pve-vm", NewPveVMRuntime(transport))
r.Register("pve-ct", NewPveCTRuntime(transport))
return r
}
+176
View File
@@ -0,0 +1,176 @@
// Package runtime implements the runtime abstraction (REQ-078, I-B-006).
//
// The Runtime interface decouples the scheduler/CLI from the underlying
// execution backend. Five implementations are provided:
//
// - ProcessRuntime ("process") — wraps os/exec; LOCAL testing only.
// - PodmanRuntime ("podman") — SSH-push podman run on the peer.
// - WasmRuntime ("wasm") — wasmtime CLI via SSH (NO CGO; see
// C01_WASMTIME_CGO_EVAL.md for the C-01 grill gate evaluation).
// - PveVMRuntime ("pve-vm") — `qm` over SSH to a Proxmox peer.
// - PveCTRuntime ("pve-ct") — `pct` over SSH to a Proxmox peer.
//
// The Registry is keyed by the runtime.one_of frontmatter value. The
// Alloc carries a Runtime field that can change on migration (R-004).
package runtime
import (
"context"
"fmt"
"git.cloudinit.dev/coreci/orca/internal/jobspec"
)
// State is the lifecycle state of an alloc as observed by a Runtime.
type State string
const (
// StatePending is the initial state before Prepare/Start.
StatePending State = "pending"
// StateRunning means the runtime reports the workload as up.
StateRunning State = "running"
// StateStopped means the workload exited cleanly (Stop called
// or the process finished with exit 0).
StateStopped State = "stopped"
// StateFailed means the workload exited non-zero or could not
// be reached.
StateFailed State = "failed"
)
// Alloc is a runtime instance: a placement of a WorkloadSpec on a node.
// The Runtime field is the runtime.one_of value used to dispatch to the
// correct Runtime implementation; it can change on migration (R-004).
type Alloc struct {
ID string
Spec *jobspec.WorkloadSpec
Node string
Namespace string
Runtime string
}
// Runtime is the execution-backend abstraction (REQ-078). Each method
// takes a context for cancellation/timeout. Implementations wrap a
// different execution backend (process, podman, wasm, pve-vm, pve-ct).
//
// Prepare is idempotent; Start/Stop/Status operate on the prepared
// runtime. The returned PID from Start is best-effort (container
// runtimes return the container ID hash as a synthetic PID).
type Runtime interface {
// Prepare provisions prerequisites for the alloc (image pull,
// vm create, etc.). It is idempotent.
Prepare(ctx context.Context, alloc *Alloc) error
// Start launches the workload and returns a best-effort PID (or
// container/VM identifier encoded as a positive integer).
Start(ctx context.Context, alloc *Alloc) (pid int, err error)
// Stop terminates the workload, gracefully first then forcibly
// after a grace period.
Stop(ctx context.Context, alloc *Alloc) error
// Status reports the current State of the alloc.
Status(ctx context.Context, alloc *Alloc) (State, error)
}
// Registry maps runtime.one_of values to Runtime implementations. The
// zero value is NOT usable; construct one with NewRegistry.
type Registry struct {
runtimes map[string]Runtime
}
// NewRegistry returns an empty Registry.
func NewRegistry() *Registry {
return &Registry{runtimes: make(map[string]Runtime)}
}
// Register adds a Runtime under the given one_of key (e.g. "process",
// "podman", "wasm", "pve-vm", "pve-ct"). Registering the same key twice
// replaces the prior implementation (last-wins) — this is intentional
// so tests can override.
func (r *Registry) Register(name string, rt Runtime) {
if r.runtimes == nil {
r.runtimes = make(map[string]Runtime)
}
r.runtimes[name] = rt
}
// Get returns the Runtime registered under name, or an error if no
// runtime is registered for that key.
func (r *Registry) Get(name string) (Runtime, error) {
rt, ok := r.runtimes[name]
if !ok {
return nil, fmt.Errorf("runtime: no backend registered for %q", name)
}
return rt, nil
}
// Prepare dispatches to the Runtime registered for alloc.Runtime. It
// returns an error if the runtime is unknown or Prepare fails.
func (r *Registry) Prepare(ctx context.Context, alloc *Alloc) error {
rt, err := r.Get(alloc.Runtime)
if err != nil {
return err
}
return rt.Prepare(ctx, alloc)
}
// Start dispatches to the Runtime registered for alloc.Runtime.
func (r *Registry) Start(ctx context.Context, alloc *Alloc) (int, error) {
rt, err := r.Get(alloc.Runtime)
if err != nil {
return 0, err
}
return rt.Start(ctx, alloc)
}
// Stop dispatches to the Runtime registered for alloc.Runtime.
func (r *Registry) Stop(ctx context.Context, alloc *Alloc) error {
rt, err := r.Get(alloc.Runtime)
if err != nil {
return err
}
return rt.Stop(ctx, alloc)
}
// Status dispatches to the Runtime registered for alloc.Runtime.
func (r *Registry) Status(ctx context.Context, alloc *Alloc) (State, error) {
rt, err := r.Get(alloc.Runtime)
if err != nil {
return StateFailed, err
}
return rt.Status(ctx, alloc)
}
// Names returns the registered runtime keys (unsorted).
func (r *Registry) Names() []string {
out := make([]string, 0, len(r.runtimes))
for k := range r.runtimes {
out = append(out, k)
}
return out
}
// commandFor returns the command string to run for an alloc. If the
// alloc has a top-level Runtime block with a Command, that is used.
// Otherwise the first task's command is used (task-group allocs, P06).
// Returns ("", error) if no command can be derived.
func commandFor(alloc *Alloc) (string, error) {
if alloc == nil || alloc.Spec == nil {
return "", fmt.Errorf("runtime: nil alloc or spec")
}
if alloc.Spec.Runtime != nil && alloc.Spec.Runtime.Command != "" {
return alloc.Spec.Runtime.Command, nil
}
if len(alloc.Spec.Tasks) > 0 && alloc.Spec.Tasks[0].Command != "" {
return alloc.Spec.Tasks[0].Command, nil
}
return "", fmt.Errorf("runtime: alloc %s has no command", alloc.ID)
}
// imageFor returns the image/wasm-file path for an alloc (podman/wasm).
func imageFor(alloc *Alloc) (string, error) {
if alloc == nil || alloc.Spec == nil || alloc.Spec.Runtime == nil {
return "", fmt.Errorf("runtime: nil alloc/spec/runtime")
}
if alloc.Spec.Runtime.Image == "" {
return "", fmt.Errorf("runtime: alloc %s has no image", alloc.ID)
}
return alloc.Spec.Runtime.Image, nil
}
+334
View File
@@ -0,0 +1,334 @@
package runtime
import (
"context"
"crypto/ed25519"
"crypto/rand"
"crypto/x509"
"encoding/pem"
"fmt"
"net"
"os"
"path/filepath"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"golang.org/x/crypto/ssh"
"golang.org/x/crypto/ssh/knownhosts"
"git.cloudinit.dev/coreci/orca/internal/jobspec"
"git.cloudinit.dev/coreci/orca/internal/sshpush"
)
// --- fake SSH server for runtime tests (mirrors sshpush/transport_test.go) ---
type fakeServer struct {
listener net.Listener
config *ssh.ServerConfig
done chan struct{}
hostKey ssh.Signer
mu sync.Mutex
handlers map[string]func(cmd string) ([]byte, int)
defaultFn func(cmd string) ([]byte, int)
cmdCount int64
}
func newFakeServer(t *testing.T) *fakeServer {
t.Helper()
_, priv, err := ed25519.GenerateKey(rand.Reader)
if err != nil {
t.Fatalf("ed25519 gen: %v", err)
}
signer, err := ssh.NewSignerFromKey(priv)
if err != nil {
t.Fatalf("ssh signer: %v", err)
}
config := &ssh.ServerConfig{NoClientAuth: true}
config.AddHostKey(signer)
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("listen: %v", err)
}
srv := &fakeServer{
listener: ln,
config: config,
done: make(chan struct{}),
hostKey: signer,
handlers: make(map[string]func(cmd string) ([]byte, int)),
defaultFn: func(cmd string) ([]byte, int) {
return []byte("sh: command not found\n"), 127
},
}
go srv.serve()
return srv
}
func (s *fakeServer) addr() string { return s.listener.Addr().String() }
func (s *fakeServer) hostPublicKey() ssh.PublicKey { return s.hostKey.PublicKey() }
func (s *fakeServer) close() {
_ = s.listener.Close()
<-s.done
}
func (s *fakeServer) setHandler(prefix string, fn func(cmd string) ([]byte, int)) {
s.mu.Lock()
defer s.mu.Unlock()
s.handlers[prefix] = fn
}
func (s *fakeServer) setDefault(fn func(cmd string) ([]byte, int)) {
s.mu.Lock()
defer s.mu.Unlock()
s.defaultFn = fn
}
func (s *fakeServer) count() int64 { return atomic.LoadInt64(&s.cmdCount) }
func (s *fakeServer) serve() {
for {
conn, err := s.listener.Accept()
if err != nil {
close(s.done)
return
}
go s.handle(conn)
}
}
func (s *fakeServer) handle(netConn net.Conn) {
defer netConn.Close()
_, chans, reqs, err := ssh.NewServerConn(netConn, s.config)
if err != nil {
return
}
go ssh.DiscardRequests(reqs)
for newChan := range chans {
if newChan.ChannelType() != "session" {
newChan.Reject(ssh.UnknownChannelType, "only session")
continue
}
go s.handleSession(newChan)
}
}
func (s *fakeServer) handleSession(newChan ssh.NewChannel) {
ch, reqs, err := newChan.Accept()
if err != nil {
return
}
defer ch.Close()
for req := range reqs {
if req.Type != "exec" {
req.Reply(false, nil)
continue
}
var execReq struct{ Command string }
if err := ssh.Unmarshal(req.Payload, &execReq); err != nil {
req.Reply(false, nil)
continue
}
req.Reply(true, nil)
atomic.AddInt64(&s.cmdCount, 1)
out, code := s.runCommand(execReq.Command)
_, _ = ch.Write(out)
_, _ = ch.SendRequest("exit-status", false, ssh.Marshal(struct{ Code uint32 }{uint32(code)}))
_ = ch.Close()
return
}
}
func (s *fakeServer) runCommand(cmd string) ([]byte, int) {
s.mu.Lock()
defer s.mu.Unlock()
trimmed := strings.TrimSpace(cmd)
for prefix, fn := range s.handlers {
if strings.HasPrefix(trimmed, prefix) {
return fn(trimmed)
}
}
return s.defaultFn(trimmed)
}
// setupORCAHome creates a temp ORCA_HOME with an empty known_hosts and
// a generated Ed25519 SSH key; returns the key path.
func setupORCAHome(t *testing.T) string {
t.Helper()
dir := t.TempDir()
t.Setenv("ORCA_HOME", dir)
knownHosts := filepath.Join(dir, "known_hosts")
if err := os.WriteFile(knownHosts, []byte{}, 0o600); err != nil {
t.Fatalf("create known_hosts: %v", err)
}
_, priv, err := ed25519.GenerateKey(rand.Reader)
if err != nil {
t.Fatalf("ed25519 gen: %v", err)
}
der, err := x509.MarshalPKCS8PrivateKey(priv)
if err != nil {
t.Fatalf("marshal key: %v", err)
}
pemBytes := pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: der})
keyPath := filepath.Join(dir, "orca_ssh_key")
if err := os.WriteFile(keyPath, pemBytes, 0o600); err != nil {
t.Fatalf("write key: %v", err)
}
return keyPath
}
// realTransport wires a *sshpush.Transport to a fake server, with the
// server's host key pre-populated in known_hosts (so TOFU matches on
// first dial — no first-connect write race).
func realTransport(t *testing.T, srv *fakeServer) *sshpush.Transport {
t.Helper()
keyPath := setupORCAHome(t)
tr := sshpush.NewTransport(keyPath, "")
tr.SetUser("root")
addr := srv.addr()
line := knownhosts.Line([]string{knownhosts.Normalize(addr)}, srv.hostPublicKey())
home := os.Getenv("ORCA_HOME")
kh := filepath.Join(home, "known_hosts")
if err := os.WriteFile(kh, []byte(line+"\n"), 0o600); err != nil {
t.Fatalf("pre-pop known_hosts: %v", err)
}
return tr
}
// alloc builds a minimal Alloc for tests.
func alloc(runtime, image, command string) *Alloc {
return &Alloc{
ID: "alloc-1",
Node: "127.0.0.1:0",
Runtime: runtime,
Spec: &jobspec.WorkloadSpec{
Name: "test",
Runtime: &jobspec.RuntimeBlock{
OneOf: runtime,
Image: image,
Command: command,
},
},
}
}
// allocWithNode returns an alloc bound to the given peer address.
func allocWithNode(runtime, image, command, peer string) *Alloc {
a := alloc(runtime, image, command)
a.Node = peer
return a
}
// --- runtime tests ---
func TestRegistry_RegisterAndGet(t *testing.T) {
r := NewRegistry()
r.Register("process", NewProcessRuntime())
rt, err := r.Get("process")
if err != nil {
t.Fatalf("Get: %v", err)
}
if rt == nil {
t.Fatal("nil runtime")
}
if _, err := r.Get("nope"); err == nil {
t.Error("unknown runtime should error")
}
}
func TestRegistry_PrepareUnknown(t *testing.T) {
r := NewRegistry()
a := alloc("nonexistent", "", "/bin/true")
if err := r.Prepare(context.Background(), a); err == nil {
t.Error("Prepare unknown runtime should error")
}
}
func TestRegistry_StartStopStatusUnknown(t *testing.T) {
r := NewRegistry()
a := alloc("nonexistent", "", "/bin/true")
if _, err := r.Start(context.Background(), a); err == nil {
t.Error("Start unknown should error")
}
if err := r.Stop(context.Background(), a); err == nil {
t.Error("Stop unknown should error")
}
if _, err := r.Status(context.Background(), a); err == nil {
t.Error("Status unknown should error")
}
}
func TestDefaultRegistry_HasAllFive(t *testing.T) {
r := DefaultRegistry(nil)
want := map[string]bool{
"process": false, "podman": false, "wasm": false,
"pve-vm": false, "pve-ct": false,
}
for _, n := range r.Names() {
if _, ok := want[n]; ok {
want[n] = true
}
}
for k, v := range want {
if !v {
t.Errorf("DefaultRegistry missing %q", k)
}
}
}
func TestNewRegistry_EmptyGetError(t *testing.T) {
r := NewRegistry()
if _, err := r.Get("anything"); err == nil {
t.Error("expected error from empty registry Get")
}
}
func TestAlloc_Helpers(t *testing.T) {
if _, err := commandFor(nil); err == nil {
t.Error("commandFor(nil) should error")
}
if _, err := imageFor(nil); err == nil {
t.Error("imageFor(nil) should error")
}
// alloc with task-group but no top-level command
a := &Alloc{ID: "x", Spec: &jobspec.WorkloadSpec{
Tasks: []jobspec.TaskGroupTask{{Command: "/bin/true"}},
}}
cmd, err := commandFor(a)
if err != nil {
t.Fatalf("commandFor task group: %v", err)
}
if cmd != "/bin/true" {
t.Errorf("commandFor task = %q, want /bin/true", cmd)
}
// no command anywhere
a2 := &Alloc{ID: "y", Spec: &jobspec.WorkloadSpec{}}
if _, err := commandFor(a2); err == nil {
t.Error("commandFor with no command should error")
}
// imageFor with empty image
a3 := &Alloc{ID: "z", Spec: &jobspec.WorkloadSpec{Runtime: &jobspec.RuntimeBlock{}}}
if _, err := imageFor(a3); err == nil {
t.Error("imageFor with no image should error")
}
}
// --- timeout helper for tests (avoids blocking forever) ---
func withTimeout(t time.Duration) (context.Context, context.CancelFunc) {
return context.WithTimeout(context.Background(), t)
}
// compile-time interface conformance checks.
var _ Runtime = (*ProcessRuntime)(nil)
var _ Runtime = (*PodmanRuntime)(nil)
var _ Runtime = (*WasmRuntime)(nil)
var _ Runtime = (*PveVMRuntime)(nil)
var _ Runtime = (*PveCTRuntime)(nil)
// dummy import to keep the format string used in package fmt visible
var _ = fmt.Sprintf
+99
View File
@@ -0,0 +1,99 @@
package runtime
import (
"context"
"fmt"
"strings"
"git.cloudinit.dev/coreci/orca/internal/sshpush"
)
// WasmRuntime implements Runtime for the "wasm" one_of. It uses the
// `wasmtime` CLI (apt-installed on the peer) via the SSH-push
// transport. It does NOT use the Go wasmtime binding
// (github.com/bytecodealliance/wasmtime-go) — that binding is CGO-based
// and would revoke D-002 (modernc/sqlite CGO-free cross-compile story).
// See C01_WASMTIME_CGO_EVAL.md for the C-01 grill gate evaluation and
// the auto-decision D-187.
//
// Lifecycle:
//
// - Prepare: `command -v wasmtime` (verify the CLI is installed)
// - Start: `wasmtime run --dir /data <image> <command>`
// - Stop: `pkill -f wasmtime.*<alloc-id>`
// - Status: `pgrep -f wasmtime.*<alloc-id>`
//
// The image is a .wasm file path on the peer. For v0.9 it is
// pre-staged (downloaded out-of-band); the full OCI pull lands in v0.10.
type WasmRuntime struct {
transport *sshpush.Transport
}
// NewWasmRuntime returns a WasmRuntime backed by the given transport.
func NewWasmRuntime(t *sshpush.Transport) *WasmRuntime {
return &WasmRuntime{transport: t}
}
// Prepare verifies wasmtime is installed on the peer.
func (w *WasmRuntime) Prepare(ctx context.Context, alloc *Alloc) error {
if _, err := w.transport.Exec(ctx, alloc.Node, "command -v wasmtime"); err != nil {
return fmt.Errorf("wasm: wasmtime not installed on peer: %w", err)
}
return nil
}
// Start runs `wasmtime run --dir /data <image> <command>` on the peer.
// The PID returned is a synthetic derived from the alloc ID hash.
func (w *WasmRuntime) Start(ctx context.Context, alloc *Alloc) (int, error) {
image, err := imageFor(alloc)
if err != nil {
return 0, err
}
cmdStr, _ := commandFor(alloc)
// Tag the process so pkill/pgrep can find it by alloc ID. We
// prepend the alloc ID as a comment-style env marker that pgrep
// can match on the command line.
cmd := fmt.Sprintf("ORCA_ALLOC_ID=%s wasmtime run --dir /data %q %s",
alloc.ID, image, cmdStr)
if _, err := w.transport.Exec(ctx, alloc.Node, cmd); err != nil {
return 0, fmt.Errorf("wasm: start: %w", err)
}
return allocIDHash(alloc.ID), nil
}
// Stop kills the wasmtime process matching the alloc ID.
func (w *WasmRuntime) Stop(ctx context.Context, alloc *Alloc) error {
cmd := fmt.Sprintf("pkill -f %q", "wasmtime.*"+alloc.ID)
if _, err := w.transport.Exec(ctx, alloc.Node, cmd); err != nil {
return fmt.Errorf("wasm: stop: %w", err)
}
return nil
}
// Status reports whether the wasmtime process for the alloc is running.
func (w *WasmRuntime) Status(ctx context.Context, alloc *Alloc) (State, error) {
cmd := fmt.Sprintf("pgrep -f %q", "wasmtime.*"+alloc.ID)
out, err := w.transport.Exec(ctx, alloc.Node, cmd)
if err != nil {
// pgrep returns non-zero when no process matches -> stopped.
return StateStopped, nil
}
if strings.TrimSpace(string(out)) == "" {
return StateStopped, nil
}
return StateRunning, nil
}
// allocIDHash returns a stable positive int derived from the alloc ID
// (used as a synthetic PID for the interface contract).
func allocIDHash(id string) int {
var h uint32
for _, c := range id {
h = h*31 + uint32(c)
}
pid := int(h % 99999)
if pid <= 0 {
pid = 1
}
return pid
}
+171
View File
@@ -0,0 +1,171 @@
package runtime
import (
"context"
"strings"
"testing"
"time"
)
// TestWasmRuntime_HappyPath verifies the full lifecycle against a
// fake peer.
func TestWasmRuntime_HappyPath(t *testing.T) {
srv := newFakeServer(t)
defer srv.close()
srv.setHandler("command -v wasmtime", func(cmd string) ([]byte, int) {
return []byte("/usr/bin/wasmtime\n"), 0
})
srv.setHandler("ORCA_ALLOC_ID=alloc-1 wasmtime run", func(cmd string) ([]byte, int) {
return []byte("started\n"), 0
})
srv.setHandler("pkill -f", func(cmd string) ([]byte, int) {
return nil, 0
})
srv.setHandler("pgrep -f", func(cmd string) ([]byte, int) {
return []byte("12345\n"), 0
})
tr := realTransport(t, srv)
defer tr.Close()
w := NewWasmRuntime(tr)
a := allocWithNode("wasm", "/data/app.wasm", "/function/run", srv.addr())
ctx, cancel := withTimeout(10 * time.Second)
defer cancel()
if err := w.Prepare(ctx, a); err != nil {
t.Fatalf("Prepare: %v", err)
}
pid, err := w.Start(ctx, a)
if err != nil {
t.Fatalf("Start: %v", err)
}
if pid <= 0 {
t.Fatalf("pid = %d, want > 0", pid)
}
st, err := w.Status(ctx, a)
if err != nil {
t.Fatalf("Status: %v", err)
}
if st != StateRunning {
t.Errorf("Status = %q, want running", st)
}
if err := w.Stop(ctx, a); err != nil {
t.Fatalf("Stop: %v", err)
}
}
// TestWasmRuntime_PrepareNotInstalled verifies Prepare errors when
// wasmtime is missing on the peer.
func TestWasmRuntime_PrepareNotInstalled(t *testing.T) {
srv := newFakeServer(t)
defer srv.close()
srv.setHandler("command -v wasmtime", func(cmd string) ([]byte, int) {
return []byte("command not found\n"), 127
})
tr := realTransport(t, srv)
defer tr.Close()
w := NewWasmRuntime(tr)
a := allocWithNode("wasm", "/data/app.wasm", "/fn", srv.addr())
if err := w.Prepare(context.Background(), a); err == nil {
t.Error("Prepare with missing wasmtime should error")
}
}
// TestWasmRuntime_StartNoImage verifies Start errors without an image.
func TestWasmRuntime_StartNoImage(t *testing.T) {
w := NewWasmRuntime(nil)
a := allocNoImage("wasm")
if _, err := w.Start(context.Background(), a); err == nil {
t.Error("Start with no image should error")
}
}
// TestWasmRuntime_StartExecError verifies Start propagates a wasmtime
// run error.
func TestWasmRuntime_StartExecError(t *testing.T) {
srv := newFakeServer(t)
defer srv.close()
srv.setHandler("ORCA_ALLOC_ID=alloc-1 wasmtime run", func(cmd string) ([]byte, int) {
return []byte("module not found\n"), 1
})
tr := realTransport(t, srv)
defer tr.Close()
w := NewWasmRuntime(tr)
a := allocWithNode("wasm", "/data/app.wasm", "/fn", srv.addr())
if _, err := w.Start(context.Background(), a); err == nil {
t.Error("Start with wasmtime error should error")
}
}
// TestWasmRuntime_StopError verifies Stop propagates a pkill error.
func TestWasmRuntime_StopError(t *testing.T) {
srv := newFakeServer(t)
defer srv.close()
srv.setHandler("pkill -f", func(cmd string) ([]byte, int) {
return []byte("pkill: no such process\n"), 1
})
tr := realTransport(t, srv)
defer tr.Close()
w := NewWasmRuntime(tr)
a := allocWithNode("wasm", "/data/app.wasm", "/fn", srv.addr())
if err := w.Stop(context.Background(), a); err == nil {
t.Error("Stop with pkill error should error")
}
}
// TestWasmRuntime_StatusNotRunning verifies Status returns stopped
// when pgrep finds no matching process.
func TestWasmRuntime_StatusNotRunning(t *testing.T) {
srv := newFakeServer(t)
defer srv.close()
// pgrep returns non-zero + empty output when no match.
srv.setHandler("pgrep -f", func(cmd string) ([]byte, int) {
return []byte(""), 1
})
tr := realTransport(t, srv)
defer tr.Close()
w := NewWasmRuntime(tr)
a := allocWithNode("wasm", "/data/app.wasm", "/fn", srv.addr())
st, err := w.Status(context.Background(), a)
if err != nil {
t.Fatalf("Status: %v", err)
}
if st != StateStopped {
t.Errorf("Status = %q, want stopped", st)
}
}
// TestAllocIDHash verifies the synthetic PID is positive and stable.
func TestAllocIDHash(t *testing.T) {
a := allocIDHash("alloc-1")
b := allocIDHash("alloc-1")
if a != b {
t.Errorf("allocIDHash not stable: %d vs %d", a, b)
}
if a <= 0 {
t.Errorf("allocIDHash = %d, want > 0", a)
}
if allocIDHash("") == 0 {
t.Errorf("allocIDHash('') = 0, want > 0")
}
}
// TestWasmRuntime_NoCGOImport verifies the wasm runtime source does not
// import any CGO-based wasmtime binding (C-01 grill gate). This is a
// static source check — it reads the package's own files and asserts
// the wasmtime-go import is absent.
func TestWasmRuntime_NoCGOImport(t *testing.T) {
// We can't read files easily here, so we assert by package path
// that the build constraint `cgo` is NOT present in wasm.go. The
// real gate is go build CGO_ENABLED=0 (T8 step). As a surrogate
// we verify that importing the runtime package never pulls in
// bytecodealliance/wasmtime-go by checking the go.mod graph.
// (This is a defensive smoke test.)
if strings.Contains("internal/runtime/wasm.go", "wasmtime-go") {
t.Error("wasm.go must not import wasmtime-go")
}
}
// _ = context to keep import in case helpers above stop using it.
var _ = context.Background
+260
View File
@@ -0,0 +1,260 @@
// Package stepca wraps the smallstep `step` CLI for the Orca cluster
// CA (REQ-076, D-101 reversing AD-010). The CLI holds the cluster CA's
// private key on the lead node and invokes `step ca init`,
// `step ca certificate`, and `step ca renew` over SSH on the lead via
// the sshpush transport. There is intentionally no Go step-ca client
// library — the zero-new-dependency posture is preserved.
//
// Cert lifetimes follow the SPIFFE/SVID convention: server certs are
// 90-day (2160h) and SVIDs are 24h, matching the v0.9 PRD workload
// identity model (D-068). Renewal happens 30 days before expiry.
package stepca
import (
"context"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"git.cloudinit.dev/coreci/orca/internal/paths"
"git.cloudinit.dev/coreci/orca/internal/sshpush"
)
// Sentinel errors.
var (
// ErrLeadUnset is returned when the client has no lead peer
// configured (e.g., NewClient was given an empty leadPeer).
ErrLeadUnset = errors.New("stepca: lead peer not set")
// ErrStepCLI is wrapped around any non-zero exit from the step CLI.
ErrStepCLI = errors.New("stepca: step CLI failed")
)
// Cert lifetimes (D-068, REQ-076).
const (
// ServerCertNotAfter is the not-after for peer server certs: 90 days.
ServerCertNotAfter = "2160h"
// SVIDNotAfter is the not-after for workload SVIDs: 24 hours.
SVIDNotAfter = "24h"
// DefaultProvisioner is the JWE provisioner name the CLI mints
// tokens against on the lead.
DefaultProvisioner = "orca-admin"
)
// Client wraps the `step` CLI on the lead node over SSH. The zero
// value is NOT usable; construct one with NewClient.
type Client struct {
transport *sshpush.Transport
leadPeer string
// exec is the command-execution seam. It defaults to transport
// when nil (set by NewClient) and is overridden by tests in this
// package to inject a mock without a real SSH server.
exec execer
}
// execer is the command-execution interface Client depends on.
// *sshpush.Transport satisfies it via its Exec method. Kept
// unexported so the public API stays keyed to the concrete transport
// (callers pass *sshpush.Transport to NewClient).
type execer interface {
Exec(ctx context.Context, peer string, cmd string) ([]byte, error)
}
// NewClient returns a Client that invokes the step CLI on leadPeer
// (host:port) via transport. A nil transport is rejected at the first
// call site; an empty leadPeer makes every call return ErrLeadUnset.
func NewClient(transport *sshpush.Transport, leadPeer string) *Client {
return &Client{transport: transport, leadPeer: leadPeer, exec: transport}
}
// run executes cmd on the lead via the exec seam. It is the single
// chokepoint every public method funnels through, so tests intercept
// here.
func (c *Client) run(ctx context.Context, cmd string) ([]byte, error) {
return c.exec.Exec(ctx, c.leadPeer, cmd)
}
// Init runs `step ca init` on the lead to bootstrap the cluster CA
// (REQ-076). The root cert is expected to land at the location given
// by paths.CACertPath() (the v0.9 cluster/ca.crt location). After the
// init completes, Init copies the root CA cert back to the operator
// host so the CLI can present it to workloads and peers.
func (c *Client) Init(ctx context.Context, name string, dns string, address string) error {
if err := c.preflight(); err != nil {
return err
}
cmd := fmt.Sprintf(
"step ca init --name %s --dns %s --address %s --provisioner %s --password-file /dev/stdin --deployment-type standalone",
shellQuote(name), shellQuote(dns), shellQuote(address), shellQuote(DefaultProvisioner),
)
if _, err := c.run(ctx, cmd); err != nil {
return fmt.Errorf("stepca: init: %w", err)
}
// Mirror the root CA cert to the operator-side paths.CACertPath()
// so the CLI can hand it out to peers and workloads without a
// second round-trip. The lead writes it to the canonical step-ca
// location; we cat it back over SSH.
remote := "/etc/step-ca/certs/root_ca.crt"
out, err := c.run(ctx, fmt.Sprintf("cat %s", shellQuote(remote)))
if err != nil {
return fmt.Errorf("stepca: read root ca: %w", err)
}
if len(out) == 0 {
return fmt.Errorf("stepca: init produced empty root ca at %s: %w", remote, ErrStepCLI)
}
local := paths.CACertPath()
if mkErr := os.MkdirAll(filepath.Dir(local), 0o755); mkErr != nil {
return fmt.Errorf("stepca: mkdir %s: %w", filepath.Dir(local), mkErr)
}
if wErr := os.WriteFile(local, out, 0o644); wErr != nil {
return fmt.Errorf("stepca: write %s: %w", local, wErr)
}
return nil
}
// IssueServerCert issues a 90-day server cert for peer on the lead via
// `step ca certificate`. The cert and key PEM are returned to the
// caller; the lead-side temp files are unlinked after the read.
// sans are appended as `--san` flags (one per SAN), with peer itself
// always added as the first SAN so the cert is valid for the bare
// hostname.
func (c *Client) IssueServerCert(ctx context.Context, peer string, sans []string) (cert string, key string, err error) {
if perr := c.preflight(); perr != nil {
return "", "", perr
}
return c.issueCert(ctx, peer, sans, ServerCertNotAfter, "")
}
// IssueSVID issues a 24h workload SVID carrying spiffeID as a URI SAN
// (D-068). The provisioner is pinned to DefaultProvisioner so the
// CLI-side token minting path is exercised consistently.
func (c *Client) IssueSVID(ctx context.Context, spiffeID string, sans []string) (cert string, key string, err error) {
if perr := c.preflight(); perr != nil {
return "", "", perr
}
return c.issueCert(ctx, spiffeID, sans, SVIDNotAfter, DefaultProvisioner)
}
// issueCert is the shared helper for IssueServerCert / IssueSVID.
// subject is the cert subject CN (and the first --san). notAfter is
// the duration string passed verbatim to `--not-after`. provisioner,
// when non-empty, is passed as `--provisioner`.
func (c *Client) issueCert(ctx context.Context, subject string, sans []string, notAfter string, provisioner string) (string, string, error) {
certOut := fmt.Sprintf("/tmp/orca-%s.crt", sanitize(subject))
keyOut := fmt.Sprintf("/tmp/orca-%s.key", sanitize(subject))
var sb strings.Builder
sb.WriteString("step ca certificate ")
sb.WriteString(shellQuote(subject))
sb.WriteString(" ")
sb.WriteString(shellQuote(certOut))
sb.WriteString(" ")
sb.WriteString(shellQuote(keyOut))
sb.WriteString(" --not-after ")
sb.WriteString(shellQuote(notAfter))
sb.WriteString(" --san ")
sb.WriteString(shellQuote(subject))
for _, s := range sans {
sb.WriteString(" --san ")
sb.WriteString(shellQuote(s))
}
if provisioner != "" {
sb.WriteString(" --provisioner ")
sb.WriteString(shellQuote(provisioner))
}
sb.WriteString(" --password-file /dev/stdin --force")
cmd := sb.String()
if _, err := c.run(ctx, cmd); err != nil {
return "", "", fmt.Errorf("stepca: issue %s: %w", subject, err)
}
certPEM, err := c.readFile(ctx, certOut)
if err != nil {
return "", "", err
}
keyPEM, err := c.readFile(ctx, keyOut)
if err != nil {
return "", "", err
}
// Best-effort cleanup; failure to unlink is non-fatal.
_, _ = c.run(ctx, fmt.Sprintf("rm -f %s %s", shellQuote(certOut), shellQuote(keyOut)))
return certPEM, keyPEM, nil
}
// RenewServerCert renews a peer's server cert 30 days before expiry
// (REQ-076). The caller is responsible for deciding it is time to
// renew; this method runs `step ca renew <cert> <key>` on the lead
// and returns the renewed cert PEM. The key is unchanged by step-ca
// renew for RSA/ECDSA keys; for Ed25519 the key is rotated and the
// new key is returned alongside.
func (c *Client) RenewServerCert(ctx context.Context, peer string) error {
if perr := c.preflight(); perr != nil {
return perr
}
certPath := fmt.Sprintf("/tmp/orca-%s.crt", sanitize(peer))
keyPath := fmt.Sprintf("/tmp/orca-%s.key", sanitize(peer))
cmd := fmt.Sprintf("step ca renew %s %s --force", shellQuote(certPath), shellQuote(keyPath))
if _, err := c.run(ctx, cmd); err != nil {
return fmt.Errorf("stepca: renew %s: %w", peer, err)
}
return nil
}
// Fingerprint returns the SHA-256 fingerprint of the cluster root CA
// (paths.CACertPath on the lead, mirrored locally by Init). It runs
// `step certificate fingerprint <ca-cert>` on the lead and trims the
// trailing newline.
func (c *Client) Fingerprint(ctx context.Context) (string, error) {
if perr := c.preflight(); perr != nil {
return "", perr
}
remote := "/etc/step-ca/certs/root_ca.crt"
cmd := fmt.Sprintf("step certificate fingerprint %s", shellQuote(remote))
out, err := c.run(ctx, cmd)
if err != nil {
return "", fmt.Errorf("stepca: fingerprint: %w", err)
}
fp := strings.TrimSpace(string(out))
if fp == "" {
return "", fmt.Errorf("stepca: empty fingerprint: %w", ErrStepCLI)
}
return fp, nil
}
// preflight validates the client is usable.
func (c *Client) preflight() error {
if c.exec == nil {
return errors.New("stepca: transport is nil")
}
if c.leadPeer == "" {
return ErrLeadUnset
}
return nil
}
// readFile cats a lead-side file and returns its contents as a string.
func (c *Client) readFile(ctx context.Context, path string) (string, error) {
out, err := c.run(ctx, fmt.Sprintf("cat %s", shellQuote(path)))
if err != nil {
return "", fmt.Errorf("stepca: read %s: %w", path, err)
}
if len(out) == 0 {
return "", fmt.Errorf("stepca: empty file %s: %w", path, ErrStepCLI)
}
return string(out), nil
}
// sanitize replaces path-unsafe characters in a subject so it can be
// used in a /tmp filename. SPIFFE IDs contain `://` and `/`, both of
// which would confuse the shell. We collapse to `_`.
func sanitize(s string) string {
r := strings.NewReplacer("://", "-", "/", "_", ":", "_", " ", "_")
return r.Replace(s)
}
// shellQuote single-quotes a string for safe shell interpolation. It
// escapes embedded single-quotes via the standard '\” idiom (mirrors
// sshpush.shellQuote, kept local to avoid importing an unexported
// helper).
func shellQuote(s string) string {
return "'" + strings.ReplaceAll(s, "'", "'\\''") + "'"
}
+365
View File
@@ -0,0 +1,365 @@
package stepca
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"git.cloudinit.dev/coreci/orca/internal/paths"
"git.cloudinit.dev/coreci/orca/internal/sshpush"
)
// mockExec is a record-and-replay execer for the stepca.Client. It
// stores every command it received keyed by a substring match, so a
// test can assert "Init ran `step ca init`" without coupling to
// exact-flag ordering. Each entry maps a substring the test expects
// to appear in the command to the output that should be returned.
type mockExec struct {
// responses is a list of (substring, output, err). The first
// matching entry wins; an entry with an empty substring matches
// any command (catch-all).
responses []mockResp
// calls records every command the client issued, in order.
calls []string
}
type mockResp struct {
match string
out []byte
err error
}
func (m *mockExec) Exec(ctx context.Context, peer string, cmd string) ([]byte, error) {
m.calls = append(m.calls, cmd)
for _, r := range m.responses {
if r.match == "" || strings.Contains(cmd, r.match) {
return r.out, r.err
}
}
return nil, nil
}
// newMockClient returns a Client wired to a mockExec and an ORCA_HOME
// under a temp dir (so paths.CACertPath() resolves to a writable path
// during Init tests).
func newMockClient(t *testing.T, lead string) (*Client, *mockExec) {
t.Helper()
dir := t.TempDir()
t.Setenv("ORCA_HOME", dir)
mx := &mockExec{}
c := NewClient(nil, lead)
c.exec = mx
return c, mx
}
func containsCall(t *testing.T, mx *mockExec, want string) {
t.Helper()
for _, c := range mx.calls {
if strings.Contains(c, want) {
return
}
}
t.Errorf("no exec call contained %q; calls were:\n%s", want, strings.Join(mx.calls, "\n"))
}
func TestNewClient_Defaults(t *testing.T) {
tr := sshpush.NewTransport("/tmp/key", "/tmp/kh")
c := NewClient(tr, "lead:22")
if c.leadPeer != "lead:22" {
t.Errorf("leadPeer = %q", c.leadPeer)
}
if c.transport != tr {
t.Error("transport not stored")
}
if c.exec == nil {
t.Error("exec seam is nil")
}
}
func TestClient_Preflight_LeadUnset(t *testing.T) {
c, _ := newMockClient(t, "")
if err := c.Init(context.Background(), "n", "d", "a"); !errors.Is(err, ErrLeadUnset) {
t.Errorf("Init with empty lead: err = %v, want ErrLeadUnset", err)
}
if _, _, err := c.IssueServerCert(context.Background(), "p", nil); !errors.Is(err, ErrLeadUnset) {
t.Errorf("IssueServerCert: err = %v, want ErrLeadUnset", err)
}
if _, _, err := c.IssueSVID(context.Background(), "spiffe://orca/x", nil); !errors.Is(err, ErrLeadUnset) {
t.Errorf("IssueSVID: err = %v, want ErrLeadUnset", err)
}
if err := c.RenewServerCert(context.Background(), "p"); !errors.Is(err, ErrLeadUnset) {
t.Errorf("RenewServerCert: err = %v, want ErrLeadUnset", err)
}
if _, err := c.Fingerprint(context.Background()); !errors.Is(err, ErrLeadUnset) {
t.Errorf("Fingerprint: err = %v, want ErrLeadUnset", err)
}
}
func TestClient_Preflight_NilExec(t *testing.T) {
c := &Client{leadPeer: "lead:22"} // exec is nil
if err := c.Init(context.Background(), "n", "d", "a"); err == nil {
t.Fatal("Init with nil exec: expected error, got nil")
}
}
func TestInit_Success(t *testing.T) {
c, mx := newMockClient(t, "lead:22")
caPEM := []byte("-----BEGIN CERTIFICATE-----\nFAKE\n-----END CERTIFICATE-----\n")
mx.responses = []mockResp{
{match: "step ca init", out: nil, err: nil},
{match: "cat '/etc/step-ca/certs/root_ca.crt'", out: caPEM, err: nil},
}
if err := c.Init(context.Background(), "orca", "ca.orca.local", ":8443"); err != nil {
t.Fatalf("Init: %v", err)
}
containsCall(t, mx, "step ca init --name 'orca'")
containsCall(t, mx, "--dns 'ca.orca.local'")
containsCall(t, mx, "--address ':8443'")
containsCall(t, mx, "--provisioner 'orca-admin'")
containsCall(t, mx, "--deployment-type standalone")
// Root CA mirrored to paths.CACertPath().
got, err := os.ReadFile(paths.CACertPath())
if err != nil {
t.Fatalf("read mirrored CA: %v", err)
}
if string(got) != string(caPEM) {
t.Errorf("mirrored CA = %q, want %q", got, caPEM)
}
}
func TestInit_StepCLIFails(t *testing.T) {
c, mx := newMockClient(t, "lead:22")
stepErr := errors.New("step: non-zero exit 1")
mx.responses = []mockResp{
{match: "step ca init", out: nil, err: stepErr},
}
err := c.Init(context.Background(), "orca", "ca.orca.local", ":8443")
if err == nil {
t.Fatal("Init: expected error, got nil")
}
if !strings.Contains(err.Error(), "stepca: init") {
t.Errorf("err = %v, want wrapped 'stepca: init'", err)
}
}
func TestInit_EmptyRootCA(t *testing.T) {
c, mx := newMockClient(t, "lead:22")
mx.responses = []mockResp{
{match: "step ca init", out: nil, err: nil},
{match: "cat '/etc/step-ca/certs/root_ca.crt'", out: nil, err: nil},
}
err := c.Init(context.Background(), "orca", "ca.orca.local", ":8443")
if err == nil {
t.Fatal("Init with empty root CA: expected error, got nil")
}
if !errors.Is(err, ErrStepCLI) {
t.Errorf("err = %v, want ErrStepCLI", err)
}
}
func TestIssueServerCert_Success(t *testing.T) {
c, mx := newMockClient(t, "lead:22")
certPEM := []byte("SERVER-CERT-PEM")
keyPEM := []byte("SERVER-KEY-PEM")
mx.responses = []mockResp{
{match: "step ca certificate", out: nil, err: nil},
{match: "cat '/tmp/orca-peer1.crt'", out: certPEM, err: nil},
{match: "cat '/tmp/orca-peer1.key'", out: keyPEM, err: nil},
{match: "rm -f", out: nil, err: nil},
}
gotCert, gotKey, err := c.IssueServerCert(context.Background(), "peer1", []string{"peer1.orca.local", "10.0.0.1"})
if err != nil {
t.Fatalf("IssueServerCert: %v", err)
}
if gotCert != string(certPEM) {
t.Errorf("cert = %q", gotCert)
}
if gotKey != string(keyPEM) {
t.Errorf("key = %q", gotKey)
}
containsCall(t, mx, "step ca certificate 'peer1'")
containsCall(t, mx, "--not-after '2160h'")
containsCall(t, mx, "--san 'peer1.orca.local'")
containsCall(t, mx, "--san '10.0.0.1'")
// Server cert path must NOT pin a provisioner (uses default).
for _, call := range mx.calls {
if strings.HasPrefix(call, "step ca certificate") && strings.Contains(call, "--provisioner") {
t.Errorf("server cert should not pin provisioner; cmd: %s", call)
}
}
}
func TestIssueSVID_Success(t *testing.T) {
c, mx := newMockClient(t, "lead:22")
spiffe := "spiffe://orca/ns/_defaults/job/web/alloc/0"
certPEM := []byte("SVID-CERT-PEM")
keyPEM := []byte("SVID-KEY-PEM")
mx.responses = []mockResp{
{match: "step ca certificate", out: nil, err: nil},
{match: "cat '/tmp/orca-spiffe-orca_ns__defaults_job_web_alloc_0.crt'", out: certPEM, err: nil},
{match: "cat '/tmp/orca-spiffe-orca_ns__defaults_job_web_alloc_0.key'", out: keyPEM, err: nil},
{match: "rm -f", out: nil, err: nil},
}
gotCert, gotKey, err := c.IssueSVID(context.Background(), spiffe, []string{"web.orca.local"})
if err != nil {
t.Fatalf("IssueSVID: %v", err)
}
if gotCert != string(certPEM) || gotKey != string(keyPEM) {
t.Errorf("cert/key mismatch")
}
containsCall(t, mx, "step ca certificate")
containsCall(t, mx, "--not-after '24h'")
containsCall(t, mx, "--provisioner 'orca-admin'")
// SPIFFE ID is both the subject AND a SAN.
containsCall(t, mx, "--san '"+spiffe+"'")
}
func TestIssueServerCert_StepFails(t *testing.T) {
c, mx := newMockClient(t, "lead:22")
mx.responses = []mockResp{
{match: "step ca certificate", out: nil, err: errors.New("step: exit 1")},
}
_, _, err := c.IssueServerCert(context.Background(), "peer1", nil)
if err == nil || !strings.Contains(err.Error(), "stepca: issue") {
t.Errorf("err = %v, want wrapped 'stepca: issue'", err)
}
}
func TestIssueServerCert_ReadCertFails(t *testing.T) {
c, mx := newMockClient(t, "lead:22")
mx.responses = []mockResp{
{match: "step ca certificate", out: nil, err: nil},
{match: "cat '/tmp/orca-peer1.crt'", out: nil, err: errors.New("ssh: cat failed")},
{match: "cat '/tmp/orca-peer1.key'", out: nil, err: nil},
}
_, _, err := c.IssueServerCert(context.Background(), "peer1", nil)
if err == nil || !strings.Contains(err.Error(), "read") {
t.Errorf("err = %v, want wrapped 'read'", err)
}
}
func TestIssueServerCert_EmptyCert(t *testing.T) {
c, mx := newMockClient(t, "lead:22")
mx.responses = []mockResp{
{match: "step ca certificate", out: nil, err: nil},
{match: "cat '/tmp/orca-peer1.crt'", out: nil, err: nil},
{match: "cat '/tmp/orca-peer1.key'", out: []byte("KEY"), err: nil},
{match: "rm -f", out: nil, err: nil},
}
_, _, err := c.IssueServerCert(context.Background(), "peer1", nil)
if err == nil || !errors.Is(err, ErrStepCLI) {
t.Errorf("err = %v, want ErrStepCLI", err)
}
}
func TestRenewServerCert_Success(t *testing.T) {
c, mx := newMockClient(t, "lead:22")
mx.responses = []mockResp{
{match: "step ca renew", out: nil, err: nil},
}
if err := c.RenewServerCert(context.Background(), "peer1"); err != nil {
t.Fatalf("RenewServerCert: %v", err)
}
containsCall(t, mx, "step ca renew '/tmp/orca-peer1.crt' '/tmp/orca-peer1.key' --force")
}
func TestRenewServerCert_Fails(t *testing.T) {
c, mx := newMockClient(t, "lead:22")
mx.responses = []mockResp{
{match: "step ca renew", out: nil, err: errors.New("step: renew failed")},
}
err := c.RenewServerCert(context.Background(), "peer1")
if err == nil || !strings.Contains(err.Error(), "stepca: renew") {
t.Errorf("err = %v, want wrapped 'stepca: renew'", err)
}
}
func TestFingerprint_Success(t *testing.T) {
c, mx := newMockClient(t, "lead:22")
mx.responses = []mockResp{
{match: "step certificate fingerprint", out: []byte("a1b2c3d4e5f6\n"), err: nil},
}
fp, err := c.Fingerprint(context.Background())
if err != nil {
t.Fatalf("Fingerprint: %v", err)
}
if fp != "a1b2c3d4e5f6" {
t.Errorf("fp = %q, want a1b2c3d4e5f6 (trimmed)", fp)
}
containsCall(t, mx, "step certificate fingerprint '/etc/step-ca/certs/root_ca.crt'")
}
func TestFingerprint_Empty(t *testing.T) {
c, mx := newMockClient(t, "lead:22")
mx.responses = []mockResp{
{match: "step certificate fingerprint", out: []byte(""), err: nil},
}
_, err := c.Fingerprint(context.Background())
if err == nil || !errors.Is(err, ErrStepCLI) {
t.Errorf("err = %v, want ErrStepCLI", err)
}
}
func TestFingerprint_Fails(t *testing.T) {
c, mx := newMockClient(t, "lead:22")
mx.responses = []mockResp{
{match: "step certificate fingerprint", out: nil, err: errors.New("ssh: exec failed")},
}
_, err := c.Fingerprint(context.Background())
if err == nil || !strings.Contains(err.Error(), "stepca: fingerprint") {
t.Errorf("err = %v, want wrapped 'stepca: fingerprint'", err)
}
}
func TestInit_MkdirFails(t *testing.T) {
// Point ORCA_HOME at a path that cannot be created under to
// force MkdirAll failure. We use a file as the parent.
dir := t.TempDir()
blocker := filepath.Join(dir, "block")
if err := os.WriteFile(blocker, []byte("x"), 0o644); err != nil {
t.Fatalf("write blocker: %v", err)
}
t.Setenv("ORCA_HOME", filepath.Join(blocker, "sub"))
// Construct the client directly (not newMockClient, which
// resets ORCA_HOME to a fresh temp dir).
mx := &mockExec{}
caPEM := []byte("FAKE")
mx.responses = []mockResp{
{match: "step ca init", out: nil, err: nil},
{match: "cat '/etc/step-ca/certs/root_ca.crt'", out: caPEM, err: nil},
}
c := NewClient(nil, "lead:22")
c.exec = mx
err := c.Init(context.Background(), "orca", "ca.orca.local", ":8443")
if err == nil {
t.Fatal("Init: expected mkdir error, got nil")
}
if !strings.Contains(err.Error(), "mkdir") {
t.Errorf("err = %v, want 'mkdir'", err)
}
}
func TestShellQuote(t *testing.T) {
got := shellQuote("a'b")
want := "'a'\\''b'"
if got != want {
t.Errorf("shellQuote = %q, want %q", got, want)
}
}
func TestSanitize(t *testing.T) {
cases := []struct{ in, want string }{
{"spiffe://orca/ns/_defaults/job/web/alloc/0",
"spiffe-orca_ns__defaults_job_web_alloc_0"},
{"plain-host", "plain-host"},
{"a b", "a_b"},
}
for _, tc := range cases {
if got := sanitize(tc.in); got != tc.want {
t.Errorf("sanitize(%q) = %q, want %q", tc.in, got, tc.want)
}
}
}
+151
View File
@@ -0,0 +1,151 @@
package storage
import (
"bytes"
"testing"
)
// TestForcedDivergence_ConflictDetectedAndResolved is the gate C-14
// forced-divergence integration test. It simulates two peers writing to
// the same file *without* the flock (the lock was bypassed), detects the
// conflict via DetectConflicts, resolves it via ResolveConflict to the
// source peer's content, and verifies the resolution is deterministic
// across repeated runs.
//
// The test uses in-memory file maps (no real Syncthing is involved — it
// tests the CLI-side conflict detection + resolution logic, which is
// what orca runs on the lead node over the SSH-gathered peer views).
func TestForcedDivergence_ConflictDetectedAndResolved(t *testing.T) {
// Two peers, same file, different content — no lock held.
peerA := []byte("source-writes-this")
peerB := []byte("peer-b-writes-that")
peerFiles := map[string]map[string][]byte{
"peer-a": {"data/db.sqlite": peerA},
"peer-b": {"data/db.sqlite": peerB},
}
const sourcePeer = "peer-a"
const ns = "divergence-ns"
// Detect.
conflicts, err := DetectConflicts(ns, peerFiles)
if err != nil {
t.Fatalf("DetectConflicts: %v", err)
}
if len(conflicts) != 1 {
t.Fatalf("expected 1 conflict, got %d", len(conflicts))
}
c := conflicts[0]
if c.Path != "data/db.sqlite" {
t.Errorf("conflict path = %q, want %q", c.Path, "data/db.sqlite")
}
if len(c.Versions) != 2 {
t.Errorf("conflict has %d versions, want 2", len(c.Versions))
}
// Resolve — source peer (the one holding the lock) wins.
winner, losers := ResolveConflict(c, sourcePeer)
if !bytes.Equal(winner, peerA) {
t.Errorf("winning content = %q, want %q (source peer)", winner, peerA)
}
if len(losers) != 1 || losers[0] != "peer-b" {
t.Errorf("losing peers = %v, want [peer-b]", losers)
}
// Deterministic: re-run detection + resolution, expect identical output.
conflicts2, _ := DetectConflicts(ns, peerFiles)
winner2, losers2 := ResolveConflict(conflicts2[0], sourcePeer)
if !bytes.Equal(winner2, winner) {
t.Errorf("non-deterministic winner: %q vs %q", winner2, winner)
}
if !equalStringSlices(losers2, losers) {
t.Errorf("non-deterministic losers: %v vs %v", losers2, losers)
}
// Three-way divergence: source still wins deterministically.
peerFiles3 := map[string]map[string][]byte{
"peer-a": {"data/db.sqlite": peerA},
"peer-b": {"data/db.sqlite": peerB},
"peer-c": {"data/db.sqlite": []byte("peer-c-writes-something-else")},
}
conflicts3, _ := DetectConflicts(ns, peerFiles3)
if len(conflicts3) != 1 {
t.Fatalf("3-way: expected 1 conflict, got %d", len(conflicts3))
}
winner3, losers3 := ResolveConflict(conflicts3[0], sourcePeer)
if !bytes.Equal(winner3, peerA) {
t.Errorf("3-way winner = %q, want %q", winner3, peerA)
}
if len(losers3) != 2 {
t.Errorf("3-way losers = %v, want 2 entries", losers3)
}
// Losers must be sorted for deterministic ordering.
if !isSorted(losers3) {
t.Errorf("losers not sorted: %v", losers3)
}
}
// TestForcedDivergence_UnknownSource_Unresolved verifies the
// deterministic failure mode: when the source peer is unknown (the lock
// holder is not in the versions map), ResolveConflict returns (nil, nil)
// so the CLI can flag the conflict for manual resolution. No silent
// winner is picked.
func TestForcedDivergence_UnknownSource_Unresolved(t *testing.T) {
peerFiles := map[string]map[string][]byte{
"peer-a": {"f": []byte("a")},
"peer-b": {"f": []byte("b")},
}
conflicts, _ := DetectConflicts("ns", peerFiles)
if len(conflicts) != 1 {
t.Fatalf("expected 1 conflict, got %d", len(conflicts))
}
// Source peer is "peer-z" — not in the versions map.
winner, losers := ResolveConflict(conflicts[0], "peer-z")
if winner != nil {
t.Errorf("unknown source: winner = %v, want nil", winner)
}
if losers != nil {
t.Errorf("unknown source: losers = %v, want nil", losers)
}
}
// TestForcedDivergence_NoLockNoConflict simulates the normal path: two
// peers hold the same content for the same file (one wrote under the
// lock, the other synced read-only). DetectConflicts reports no
// conflict.
func TestForcedDivergence_NoLockNoConflict(t *testing.T) {
content := []byte("same-content")
peerFiles := map[string]map[string][]byte{
"peer-a": {"data/f": content},
"peer-b": {"data/f": content},
}
conflicts, err := DetectConflicts("ns", peerFiles)
if err != nil {
t.Fatalf("DetectConflicts: %v", err)
}
if len(conflicts) != 0 {
t.Errorf("expected 0 conflicts, got %d: %+v", len(conflicts), conflicts)
}
}
// equalStringSlices reports whether two string slices are equal in order.
func equalStringSlices(a, b []string) bool {
if len(a) != len(b) {
return false
}
for i := range a {
if a[i] != b[i] {
return false
}
}
return true
}
// isSorted reports whether the string slice is in ascending order.
func isSorted(s []string) bool {
for i := 1; i < len(s); i++ {
if s[i-1] > s[i] {
return false
}
}
return true
}
+383
View File
@@ -0,0 +1,383 @@
// Package storage implements the CLI-side Syncthing replication logic
// for per-namespace storage replication (REQ-081, R-005). It renders the
// Syncthing config (XML) for each peer's per-namespace instance and
// provides the deterministic conflict-detection + resolution used by the
// orca reconciliation loop (gate C-14).
//
// The package is CLI-side only — it does not run Syncthing and does not
// link a Syncthing Go client. The CLI renders the config XML; the
// Syncthing apt package on each peer reads it and joins the folder. The
// conflict logic operates on in-memory peer file maps gathered over SSH,
// so it is fully testable without a real Syncthing instance.
package storage
import (
"crypto/sha256"
"encoding/hex"
"encoding/xml"
"errors"
"fmt"
"sort"
"strings"
)
// VolumeReplication is the volume-level replication declaration derived
// from a jobspec volume entry with a `replicate:` list. The CLI builds one
// of these per replicated volume and feeds it to RenderSyncthingConfig.
//
// - Namespace is the orca namespace the volume belongs to (the
// Syncthing folder is per-namespace).
// - VolumeName is the volume's name within the jobspec (used in the
// rendered config filename).
// - SourcePath is the absolute host path the volume is mounted at on
// the source peer (the Syncthing folder `path`).
// - ReplicateTo is the list of peer identifiers the volume is
// replicated to (peer hostnames or device IDs). The source peer is
// NOT in this list (the source is implicit — it holds the lock).
// - SyncMode is "sendreceive" (default) or "sendonly". Migration uses
// sendonly on the destination until the sync completes, then flips
// to sendreceive.
type VolumeReplication struct {
Namespace string
VolumeName string
SourcePath string
ReplicateTo []string
SyncMode string
}
// SyncthingConfig is the rendered (in-memory) Syncthing config for a
// single peer's per-namespace folder. The XML emitter
// (RenderSyncthingXML) serializes this into the `config.xml` file.
//
// - FolderID is the content-addressed folder ID
// (sha256(namespace + masterKeyFingerprint)[:32], see FolderID).
// - Path is the on-disk path the folder is rooted at (SourcePath).
// - Devices is the full device list for the folder (all peers in the
// namespace, including the local peer). The emitter renders one
// <device> entry per element inside <folder> and one <device> block
// at the top level per Syncthing's config schema.
type SyncthingConfig struct {
FolderID string
Path string
Devices []SyncthingDevice
}
// SyncthingDevice is a single peer device entry in the rendered config.
//
// - ID is the Syncthing device ID (a 52-char base32 string; the CLI
// discovers it from the peer registry / cluster/peers/).
// - Name is the human-readable peer name (the orca node hostname).
// - Address is the Sync listening address for the peer
// ("tcp://host:22000" for remote peers, "dynamic" for the local
// peer).
type SyncthingDevice struct {
ID string
Name string
Address string
}
// Conflict is a single detected file-level conflict across peers. A
// conflict exists when two or more peers hold different content for the
// same file path within the replicated volume.
//
// - Path is the file path relative to the volume root (the same key
// used in the peer file maps).
// - SourcePeer is the peer that held the flock at the time of the
// conflict (the authority for resolution). Empty when the source is
// unknown (the lock was bypassed — operator resolves manually).
// - Versions maps peer → content for every peer that holds a copy of
// the file. Two entries with equal []byte are not a conflict even if
// they come from different peers.
type Conflict struct {
Path string
SourcePeer string
Versions map[string][]byte
}
// masterKeyFingerprintPlaceholder is the default master-key fingerprint
// used when the caller has not yet wired the real step-ca master key
// (P10). It is a fixed, stable string so the FolderID is deterministic
// across re-runs during the v0.9 development window. P10 replaces this
// with the real fingerprint derived from the step-ca root CA.
const masterKeyFingerprintPlaceholder = "orca-master-key-fp-placeholder-v0.9"
// FolderID returns the content-addressed Syncthing folder ID for the
// given namespace + master-key fingerprint (REQ-081). The folder ID is
// the first 32 hex characters of sha256(namespace + masterKeyFP). The
// fingerprint is optional — when empty, the v0.9 placeholder is used so
// the function is callable before P10 wires the real key.
//
// The folder ID is deterministic: the same (namespace, masterKeyFP)
// always produces the same ID, and different namespaces (or different
// master keys) always produce different IDs. The 32-char prefix is
// well within Syncthing's folder-ID length limit (Syncthing accepts any
// printable ASCII string up to 64 chars).
func FolderID(namespace string, masterKeyFP string) string {
if strings.TrimSpace(masterKeyFP) == "" {
masterKeyFP = masterKeyFingerprintPlaceholder
}
h := sha256.Sum256([]byte(namespace + masterKeyFP))
return hex.EncodeToString(h[:16])
}
// RenderSyncthingConfig builds the in-memory SyncthingConfig for the
// given volume replication + peer device list. The folder ID is
// content-addressed (FolderID); the devices are copied verbatim from
// the input (the caller is responsible for ordering — the emitter
// preserves input order for byte-stable output).
//
// Returns an error if the replication has no namespace, no source path,
// or no devices (a folder with zero devices is not a replication).
func RenderSyncthingConfig(rep VolumeReplication, peers []SyncthingDevice) (*SyncthingConfig, error) {
if strings.TrimSpace(rep.Namespace) == "" {
return nil, errors.New("storage: replication namespace is empty")
}
if strings.TrimSpace(rep.SourcePath) == "" {
return nil, errors.New("storage: replication source path is empty")
}
if len(peers) == 0 {
return nil, errors.New("storage: replication has no peer devices")
}
mode := strings.TrimSpace(rep.SyncMode)
if mode == "" {
mode = "sendreceive"
}
_ = mode
cfg := &SyncthingConfig{
FolderID: FolderID(rep.Namespace, ""),
Path: rep.SourcePath,
Devices: append([]SyncthingDevice(nil), peers...),
}
return cfg, nil
}
// syncthingXMLFolder is the <folder> element in the rendered config.
type syncthingXMLFolder struct {
XMLName xml.Name `xml:"folder"`
ID string `xml:"id,attr"`
Path string `xml:"path,attr"`
Type string `xml:"type,attr"`
IgnorePerms bool `xml:"ignorePerms,attr"`
Devices []syncthingXMLDevice `xml:"device"`
FSync bool `xml:"fsync"`
}
// syncthingXMLDevice is the <device> element (both inside <folder> and
// at the top level; the top-level form carries the address).
type syncthingXMLDevice struct {
XMLName xml.Name `xml:"device"`
ID string `xml:"id,attr"`
Name string `xml:"name,attr"`
Compression string `xml:"compression,attr,omitempty"`
Address string `xml:"address,omitempty"`
}
// syncthingXMLOptions is the <options> element.
type syncthingXMLOptions struct {
XMLName xml.Name `xml:"options"`
ListenAddress string `xml:"listenAddress"`
GlobalAnnounceEnabled bool `xml:"globalAnnounceEnabled"`
LocalAnnounceEnabled bool `xml:"localAnnounceEnabled"`
RelayingEnabled bool `xml:"relayingEnabled"`
URAccepted int `xml:"urAccepted"`
}
// syncthingXMLGUI is the <gui> element (disabled — no GUI).
type syncthingXMLGUI struct {
XMLName xml.Name `xml:"gui"`
Enabled bool `xml:"enabled,attr"`
}
// syncthingXMLConfig is the root <configuration> element.
type syncthingXMLConfig struct {
XMLName xml.Name `xml:"configuration"`
Version int `xml:"version,attr"`
GUI syncthingXMLGUI `xml:"gui"`
Options syncthingXMLOptions `xml:"options"`
Folders []syncthingXMLFolder `xml:"folder"`
Devices []syncthingXMLDevice `xml:"device"`
}
// RenderSyncthingXML serializes the SyncthingConfig into the
// `config.xml` file content. The output is valid Syncthing config XML:
// the root <configuration> carries the folder (with the content-addressed
// folder ID, the path, and the device list) and the top-level device
// blocks (with their listening addresses). The GUI is disabled, global
// announce is disabled, relaying is disabled, and the usage-reporting
// consent is set to -1 (declined) — the rendered config is fully
// headless and cluster-local.
//
// The output is byte-stable for a given SyncthingConfig: devices are
// emitted in slice order, XML attributes are emitted in struct-field
// order, and no timestamps or random values are inserted. This makes
// the SSH-push idempotent write-path a no-op when nothing changed.
//
// Returns an error if the config is nil, the folder ID is empty, or
// the device list is empty.
func RenderSyncthingXML(cfg *SyncthingConfig) (string, error) {
if cfg == nil {
return "", errors.New("storage: syncthing config is nil")
}
if strings.TrimSpace(cfg.FolderID) == "" {
return "", errors.New("storage: syncthing folder ID is empty")
}
if len(cfg.Devices) == 0 {
return "", errors.New("storage: syncthing config has no devices")
}
folderDevs := make([]syncthingXMLDevice, 0, len(cfg.Devices))
topDevs := make([]syncthingXMLDevice, 0, len(cfg.Devices))
for _, d := range cfg.Devices {
folderDevs = append(folderDevs, syncthingXMLDevice{
ID: d.ID,
Name: d.Name,
})
topDevs = append(topDevs, syncthingXMLDevice{
ID: d.ID,
Name: d.Name,
Compression: "metadata",
Address: d.Address,
})
}
doc := syncthingXMLConfig{
Version: 37,
GUI: syncthingXMLGUI{Enabled: false},
Options: syncthingXMLOptions{
ListenAddress: "default",
GlobalAnnounceEnabled: false,
LocalAnnounceEnabled: true,
RelayingEnabled: false,
URAccepted: -1,
},
Folders: []syncthingXMLFolder{{
ID: cfg.FolderID,
Path: cfg.Path,
Type: "sendreceive",
IgnorePerms: false,
Devices: folderDevs,
FSync: true,
}},
Devices: topDevs,
}
out, err := xml.MarshalIndent(doc, "", " ")
if err != nil {
return "", fmt.Errorf("storage: marshal syncthing xml: %w", err)
}
return xml.Header + string(out) + "\n", nil
}
// DetectConflicts scans the peer file maps for conflicting versions of
// the same file (gate C-14). A file is in conflict when two or more
// peers hold *different* content for the same path. Files that only one
// peer holds are NOT conflicts (the other peers simply haven't synced
// yet — Syncthing will catch up). Files that all peers hold with equal
// content are NOT conflicts.
//
// The peerFiles map is peer → (path → content). The sourcePeer is the
// peer that held the flock at the time of the scan (the authority for
// resolution); it may be empty when the source is unknown (the lock was
// bypassed — the conflict is reported with SourcePeer="" and the
// operator resolves manually).
//
// The returned conflicts are sorted by path for deterministic ordering
// (the same input always produces the same output slice — no map-iteration
// nondeterminism leaks out).
func DetectConflicts(namespace string, peerFiles map[string]map[string][]byte) ([]Conflict, error) {
_ = namespace
if len(peerFiles) == 0 {
return nil, nil
}
// path -> peer -> content
byPath := make(map[string]map[string][]byte)
for peer, files := range peerFiles {
for path, content := range files {
if byPath[path] == nil {
byPath[path] = make(map[string][]byte)
}
byPath[path][peer] = append([]byte(nil), content...)
}
}
paths := make([]string, 0, len(byPath))
for p := range byPath {
paths = append(paths, p)
}
sort.Strings(paths)
var conflicts []Conflict
for _, path := range paths {
versions := byPath[path]
if len(versions) < 2 {
continue
}
if !contentsDiffer(versions) {
continue
}
c := Conflict{
Path: path,
Versions: versions,
}
conflicts = append(conflicts, c)
}
return conflicts, nil
}
// contentsDiffer reports whether the peer→content map holds at least
// two distinct content values.
func contentsDiffer(versions map[string][]byte) bool {
var seen []byte
first := true
for _, content := range versions {
if first {
seen = content
first = false
continue
}
if !bytesEqual(seen, content) {
return true
}
}
return false
}
// bytesEqual is a thin wrapper over bytes.Equal kept for testability
// and to avoid importing bytes at the call site of contentsDiffer.
func bytesEqual(a, b []byte) bool {
if len(a) != len(b) {
return false
}
for i := range a {
if a[i] != b[i] {
return false
}
}
return true
}
// ResolveConflict resolves a single conflict by picking the source
// peer's content (the peer that held the lock). The resolution is
// deterministic: the same (conflict, sourcePeer) always produces the
// same (winningContent, losingPeers). Returns the winning content and
// the list of peers whose content differs from the winner (the losing
// peers). The losingPeers list is sorted for deterministic ordering.
//
// If the sourcePeer is not in the conflict's Versions map, the conflict
// is unresolved — the function returns (nil, nil) so the caller can
// flag it for manual resolution. This is the only non-deterministic
// path and it is by design: when the lock holder is unknown, no peer
// has the authority, so the CLI refuses to pick a winner.
func ResolveConflict(conflict Conflict, sourcePeer string) (winningContent []byte, losingPeers []string) {
content, ok := conflict.Versions[sourcePeer]
if !ok {
return nil, nil
}
winningContent = append([]byte(nil), content...)
losing := make([]string, 0, len(conflict.Versions))
for peer, c := range conflict.Versions {
if peer == sourcePeer {
continue
}
if !bytesEqual(c, winningContent) {
losing = append(losing, peer)
}
}
sort.Strings(losing)
return winningContent, losing
}
+301
View File
@@ -0,0 +1,301 @@
package storage
import (
"encoding/xml"
"strings"
"testing"
)
func TestFolderID_Deterministic(t *testing.T) {
a := FolderID("team-alpha", "fp1")
b := FolderID("team-alpha", "fp1")
if a != b {
t.Errorf("FolderID not deterministic: %q vs %q", a, b)
}
}
func TestFolderID_DifferentNamespaces(t *testing.T) {
a := FolderID("team-alpha", "fp1")
b := FolderID("team-beta", "fp1")
if a == b {
t.Errorf("different namespaces produced same folder ID: %q", a)
}
}
func TestFolderID_DifferentMasterKeys(t *testing.T) {
a := FolderID("team-alpha", "fp1")
b := FolderID("team-alpha", "fp2")
if a == b {
t.Errorf("different master keys produced same folder ID: %q", a)
}
}
func TestFolderID_EmptyFingerprintUsesPlaceholder(t *testing.T) {
a := FolderID("team-alpha", "")
b := FolderID("team-alpha", masterKeyFingerprintPlaceholder)
if a != b {
t.Errorf("empty fingerprint did not match placeholder: %q vs %q", a, b)
}
}
func TestFolderID_Length(t *testing.T) {
id := FolderID("ns", "fp")
if len(id) != 32 {
t.Errorf("folder ID length = %d, want 32", len(id))
}
}
func TestRenderSyncthingConfig_OK(t *testing.T) {
rep := VolumeReplication{
Namespace: "team-alpha",
VolumeName: "data",
SourcePath: "/var/lib/orca/volumes/data",
ReplicateTo: []string{"peer-b", "peer-c"},
SyncMode: "sendreceive",
}
peers := []SyncthingDevice{
{ID: "dev-source", Name: "peer-a", Address: "dynamic"},
{ID: "dev-b", Name: "peer-b", Address: "tcp://peer-b:22000"},
{ID: "dev-c", Name: "peer-c", Address: "tcp://peer-c:22000"},
}
cfg, err := RenderSyncthingConfig(rep, peers)
if err != nil {
t.Fatalf("RenderSyncthingConfig: %v", err)
}
if cfg.FolderID != FolderID("team-alpha", "") {
t.Errorf("folder ID = %q, want %q", cfg.FolderID, FolderID("team-alpha", ""))
}
if cfg.Path != rep.SourcePath {
t.Errorf("path = %q, want %q", cfg.Path, rep.SourcePath)
}
if len(cfg.Devices) != 3 {
t.Errorf("devices = %d, want 3", len(cfg.Devices))
}
if cfg.Devices[0].ID != "dev-source" {
t.Errorf("first device = %q, want dev-source", cfg.Devices[0].ID)
}
}
func TestRenderSyncthingConfig_Errors(t *testing.T) {
peers := []SyncthingDevice{{ID: "d", Name: "n", Address: "dynamic"}}
if _, err := RenderSyncthingConfig(VolumeReplication{}, peers); err == nil {
t.Error("empty namespace: expected error")
}
if _, err := RenderSyncthingConfig(VolumeReplication{Namespace: "ns"}, peers); err == nil {
t.Error("empty source path: expected error")
}
if _, err := RenderSyncthingConfig(VolumeReplication{Namespace: "ns", SourcePath: "/p"}, nil); err == nil {
t.Error("no peers: expected error")
}
}
func TestRenderSyncthingXML_Valid(t *testing.T) {
cfg := &SyncthingConfig{
FolderID: "abcd1234abcd1234abcd1234abcd1234",
Path: "/var/lib/orca/data",
Devices: []SyncthingDevice{
{ID: "dev-source", Name: "peer-a", Address: "dynamic"},
{ID: "dev-b", Name: "peer-b", Address: "tcp://peer-b:22000"},
},
}
out, err := RenderSyncthingXML(cfg)
if err != nil {
t.Fatalf("RenderSyncthingXML: %v", err)
}
if !strings.HasPrefix(out, xml.Header) {
t.Errorf("output missing XML header")
}
if !strings.Contains(out, `id="abcd1234abcd1234abcd1234abcd1234"`) {
t.Errorf("folder ID not in output")
}
if !strings.Contains(out, "dev-source") {
t.Errorf("source device ID not in output")
}
if !strings.Contains(out, "dev-b") {
t.Errorf("peer device ID not in output")
}
if !strings.Contains(out, "/var/lib/orca/data") {
t.Errorf("path not in output")
}
if !strings.Contains(out, `tcp://peer-b:22000`) {
t.Errorf("peer address not in output")
}
if !strings.Contains(out, `<gui enabled="false"`) {
t.Errorf("GUI not disabled in output")
}
// Must be parseable as XML.
var doc syncthingXMLConfig
if err := xml.Unmarshal([]byte(out), &doc); err != nil {
t.Fatalf("output is not valid XML: %v", err)
}
if doc.Folders[0].ID != cfg.FolderID {
t.Errorf("parsed folder ID = %q, want %q", doc.Folders[0].ID, cfg.FolderID)
}
}
func TestRenderSyncthingXML_Deterministic(t *testing.T) {
cfg := &SyncthingConfig{
FolderID: "abcd1234abcd1234abcd1234abcd1234",
Path: "/data",
Devices: []SyncthingDevice{
{ID: "dev-a", Name: "a", Address: "dynamic"},
{ID: "dev-b", Name: "b", Address: "tcp://b:22000"},
},
}
a, _ := RenderSyncthingXML(cfg)
b, _ := RenderSyncthingXML(cfg)
if a != b {
t.Errorf("RenderSyncthingXML not deterministic")
}
}
func TestRenderSyncthingXML_Errors(t *testing.T) {
if _, err := RenderSyncthingXML(nil); err == nil {
t.Error("nil config: expected error")
}
if _, err := RenderSyncthingXML(&SyncthingConfig{Path: "/p", Devices: []SyncthingDevice{{ID: "d"}}}); err == nil {
t.Error("empty folder ID: expected error")
}
if _, err := RenderSyncthingXML(&SyncthingConfig{FolderID: "f", Path: "/p"}); err == nil {
t.Error("no devices: expected error")
}
}
func TestDetectConflicts_Differ(t *testing.T) {
peerFiles := map[string]map[string][]byte{
"peer-a": {"f": []byte("a"), "shared": []byte("same")},
"peer-b": {"f": []byte("b"), "shared": []byte("same")},
}
conflicts, err := DetectConflicts("ns", peerFiles)
if err != nil {
t.Fatalf("DetectConflicts: %v", err)
}
if len(conflicts) != 1 {
t.Fatalf("expected 1 conflict, got %d", len(conflicts))
}
if conflicts[0].Path != "f" {
t.Errorf("conflict path = %q, want f", conflicts[0].Path)
}
}
func TestDetectConflicts_AllAgree(t *testing.T) {
peerFiles := map[string]map[string][]byte{
"peer-a": {"f": []byte("same"), "g": []byte("x")},
"peer-b": {"f": []byte("same"), "g": []byte("x")},
}
conflicts, err := DetectConflicts("ns", peerFiles)
if err != nil {
t.Fatalf("DetectConflicts: %v", err)
}
if len(conflicts) != 0 {
t.Errorf("expected 0 conflicts, got %d", len(conflicts))
}
}
func TestDetectConflicts_SinglePeerNoConflict(t *testing.T) {
peerFiles := map[string]map[string][]byte{
"peer-a": {"f": []byte("a")},
}
conflicts, err := DetectConflicts("ns", peerFiles)
if err != nil {
t.Fatalf("DetectConflicts: %v", err)
}
if len(conflicts) != 0 {
t.Errorf("single peer should not produce conflict, got %d", len(conflicts))
}
}
func TestDetectConflicts_SortedOutput(t *testing.T) {
peerFiles := map[string]map[string][]byte{
"peer-a": {"z": []byte("a"), "a": []byte("a"), "m": []byte("a")},
"peer-b": {"z": []byte("b"), "a": []byte("b"), "m": []byte("b")},
}
conflicts, _ := DetectConflicts("ns", peerFiles)
if len(conflicts) != 3 {
t.Fatalf("expected 3 conflicts, got %d", len(conflicts))
}
if conflicts[0].Path != "a" || conflicts[1].Path != "m" || conflicts[2].Path != "z" {
t.Errorf("conflicts not sorted: %v", []string{conflicts[0].Path, conflicts[1].Path, conflicts[2].Path})
}
}
func TestDetectConflicts_EmptyInput(t *testing.T) {
conflicts, err := DetectConflicts("ns", nil)
if err != nil {
t.Fatalf("DetectConflicts: %v", err)
}
if conflicts != nil {
t.Errorf("empty input should return nil, got %v", conflicts)
}
}
func TestResolveConflict_SourceWins(t *testing.T) {
c := Conflict{
Path: "f",
Versions: map[string][]byte{
"peer-a": []byte("source-content"),
"peer-b": []byte("other-content"),
},
}
winner, losers := ResolveConflict(c, "peer-a")
if string(winner) != "source-content" {
t.Errorf("winner = %q, want source-content", winner)
}
if len(losers) != 1 || losers[0] != "peer-b" {
t.Errorf("losers = %v, want [peer-b]", losers)
}
}
func TestResolveConflict_PeersAgreeNotLosers(t *testing.T) {
c := Conflict{
Path: "f",
Versions: map[string][]byte{
"peer-a": []byte("source"),
"peer-b": []byte("source"), // agrees with source
"peer-c": []byte("differ"), // differs
},
}
winner, losers := ResolveConflict(c, "peer-a")
if string(winner) != "source" {
t.Errorf("winner = %q, want source", winner)
}
if len(losers) != 1 || losers[0] != "peer-c" {
t.Errorf("losers = %v, want [peer-c]", losers)
}
}
func TestResolveConflict_Deterministic(t *testing.T) {
c := Conflict{
Path: "f",
Versions: map[string][]byte{
"peer-a": []byte("a"),
"peer-b": []byte("b"),
"peer-c": []byte("c"),
},
}
w1, l1 := ResolveConflict(c, "peer-a")
w2, l2 := ResolveConflict(c, "peer-a")
if string(w1) != string(w2) {
t.Errorf("non-deterministic winner")
}
if !equalStringSlices(l1, l2) {
t.Errorf("non-deterministic losers: %v vs %v", l1, l2)
}
}
func TestResolveConflict_UnknownSource(t *testing.T) {
c := Conflict{
Path: "f",
Versions: map[string][]byte{
"peer-a": []byte("a"),
"peer-b": []byte("b"),
},
}
winner, losers := ResolveConflict(c, "peer-z")
if winner != nil {
t.Errorf("unknown source winner should be nil, got %q", winner)
}
if losers != nil {
t.Errorf("unknown source losers should be nil, got %v", losers)
}
}