Files
orca/.ciagent/RESEARCH_v0.8.md
T
Jon Chery 285ffee863 docs(P00): v0.8 research findings + persona assessment
RESEARCH_v0.8.md (35KB): per-package coverage strategy for 9 pkgs,
SSH trust research (uncovered latent TOFU capture bug + unpopulated
HostKeyFingerprint field), verify-reqs Go program approach, 4 ADs,
10 pitfalls. PERSONAS.md updated for v0.8 (3-persona roster retained,
connectrpc removed from backend frameworks per AD-014, territory globs
aligned to actual file structure).

Key findings flagged for PLAN:
- proxmox needs sessionRunner seam (~10 LOC) or stalls at ~55%
- cert_repo_test.go missing (v0.7 P01 leftover) blocks store 70%
- TOFU known_hosts capture broken + HostKeyFingerprint never populated
  (bootstrap.go:195-198) — P02 fixes both
- verify-reqs = Go program at cmd/verify-reqs (~80 LOC, stdlib only)

---ci---
project: orca
phase: 0
milestone: v0.8
status: research
---/ci---
2026-08-04 00:49:07 +00:00

34 KiB

Research: Orca v0.8 — Coverage & Trust Hardening

Findings grounded in codebase analysis (44 source/test files read, coverage re-measured for all 9 target packages) + golang.org/x/crypto v0.54.0 API verification (ssh.FingerprintSHA256, knownhosts.Line/Normalize/KeyError).

1. Coverage analysis (P01 — REQ-057)

1.1 Re-measured coverage (confirmed via go test ./<pkg>/... -cover)

Package Coverage Tier (D-047) Notes
internal/engine 8.3% ≥ 70% floor Only scheduler_test.go (4 tests, 66 LOC); executor/dispatcher/peer/registry/audit untested
internal/proxmox 5.1% ≥ 70% floor Only bootstrap_test.go (4 tests, validation + sudoersContent string asserts); SSH dial path untested
internal/cli 27.6% ≥ 70% floor 5 test files (root, init, namespace, osdetect, watch); node/job/cert/doctor/audit/cmds untested
internal/transport 26.3% ≥ 70% floor Only idempotency_test.go (7 tests); mtls/dispatch/retry/handshake_log untested
internal/store 47.2% ≥ 70% floor node_repo + job_task + capacity + audit + migrate tested; cert_repo has NO test (REQ-053 leftover — v0.7 P01 was supposed to add it but it's missing)
internal/jobspec 47.6% ≥ 70% floor Only spec_test.go (4 tests); Validate(), ParseFile (file I/O), edge cases untested
internal/audit 0% (no test files) ≥ 50% toe-hold go: no such tool "covdata" is a known tooling gap, NOT a real number — the package simply has no _test.go
internal/certpaths 0% (no test files) ≥ 50% toe-hold Same covdata tooling gap; no _test.go exists
cmd/orca 0% (no test files) ≥ 50% toe-hold Same; main.go is 15 LOC of glue (cli.Execute() + error print)

Coverage-floor achievability assessment (per package):

  • engine → 70% REALISTIC. The package has clean seams: LocalExecutor interface (dispatcher.go:39), PeerRegistry is in-memory with Add/Remove/All/Get (peer.go), Executor.Submit/Status take a *store.JobRepo+*store.TaskRepo which can be backed by :memory:/temp-file sqlite via the existing openTestDB helper (node_repo_test.go:12). The sshDialer seam pattern (proxmox) has an analogue here: transport.NewDispatchClient is called inside dispatchToPeer (dispatcher.go:158) — to test dispatch-to-peer without a real mTLS server, either (a) inject a fake DispatchClient via a new interface seam, or (b) use httptest.NewTLSServer with a self-signed CA. Option (a) is lower-effort and aligns with the LocalExecutor pattern. Recommendation: extract a peerDispatcher interface (Submit(ctx, spec, key) (*SubmitResponse, error)) and inject it, OR test via LocalSubmit/LocalStatus paths (which only need a stubbed LocalExecutor) — the latter covers ~60% of dispatcher.go without a new seam. Flag: 70% may require a small refactor to inject the dispatch client; 60-65% is achievable without one. Plan should decide whether to add the seam or accept 65%.
  • proxmox → 70% REALISTIC. The sshDialer seam already exists (bootstrap.go:201-213, sshDialerType interface + defaultSSHDialer struct, overridable package-level var). A fake SSH dialer returning a mock *ssh.Client is the path. However: *ssh.Client.NewSession() + session.CombinedOutput() are concrete methods on the real *ssh.Client — there's no sshSession interface seam. To test runRemote/deployPubKey/createLinuxUser/createPVERole/etc. without a real SSH server, EITHER (a) introduce a sessionRunner interface seam (small refactor), OR (b) use httptest.NewTLSServer is wrong (it's SSH not HTTP) — instead use a real in-process SSH server via golang.org/x/crypto/ssh NewServerConn (more code but no new dep). Flag: 70% likely requires either a sessionRunner interface refactor OR an in-process SSH server fixture. 50-55% is achievable with just the existing sshDialer seam + testing validation paths + sudoersContent string asserts (already done). Plan should add the sessionRunner seam — it's a 1-interface, ~10-LOC change that unlocks the bulk of the package.
  • cli → 70% AMBITIOUS but realistic. The package is the largest (17 source files, ~2000 LOC). The existing tests use rootCmd.SetArgs() + rootCmd.Execute() + t.TempDir() + ORCA_HOME env (namespace_test.go:46-53 — TestInitHonorsORCAHOME is the template). The untested commands are node join/leave/list, job run/list/stop/logs, cert *, doctor *, audit list, status, version, daemon. Many touch the DB + certpaths + (for node join --type proxmox) the SSH dialer. Strategy: table-driven rootCmd.Execute() against a temp ORCA_HOME for each subcommand; mock the proxmox path via the existing sshDialer seam; capture stdout via rootCmd.SetOut(&buf). Flag: 70% across the whole package is a lot of test code; 55-65% is more realistic for one phase. The daemon command (background server) is hard to test without a lifecycle harness — recommend excluding it from the 70% target and documenting why.
  • transport → 70% REALISTIC. httptest.NewTLSServer is the standard seam (already used in internal/daemon/dispatch_test.go:59 and server_test.go). The Dispatcher interface (dispatch.go:49) is already mockable (stubDispatcher in dispatch_test.go:24 is the template). MTLSClient.Do wraps http.Client.Do — testable via httptest.NewTLSServer with a CA + client cert. retry.go Do[T] is generic + already partly tested via idempotency_test.go (TestRetrySucceedsAfterTransient etc.) — extend with backoff-timing asserts. handshake_log.go is pure slog calls — trivial to test by capturing into a slog.Handler. No new seams needed; 70% achievable.
  • store → 70% REALISTIC. The existing openTestDB helper (node_repo_test.go:12) + withFastWatch (job_task_repo_test.go:36) are reusable. Critical gap: cert_repo.go has NO test file despite v0.7 P01 REQ-053 claiming it was added — this is a v0.7 leftover bug. Adding cert_repo_test.go (Insert/Get/List/ListByNode/LatestForKind/PruneOlderThan/Delete + N=3 rotation history per REQ-025) alone lifts coverage significantly. Job/Task repo Watch is tested; ListRecent, error paths, scan-edge cases need coverage. No new seams; 70% achievable.
  • jobspec → 70% REALISTIC. Parse + Validate + ParseFile are pure functions over HCL bytes. Add golden-file HCL fixtures (multi-task, env vars, args) + error-path table (missing job, no tasks, missing command, malformed HCL, empty file, nonexistent file for ParseFile). testdata/ dir doesn't exist yet — create it. No new seams; 70% achievable, likely the easiest of the six.
  • audit → 50% toe-hold REALISTIC. Package is 125 LOC, 4 exported funcs (New, Emit, EmitWithErr, LogHandshakeOK, LogHandshakeFailed, FormatAction, Action.String, Result.String). Strategy: construct Audit with a real engine.Audit backed by :memory: sqlite (via store.NewAuditRepo + engine.NewAudit) + assert rows in audit_log table; capture slog output via a test slog.Handler. No new seams; 50% easily achievable, 70% achievable if desired.
  • certpaths → 50% toe-hold TRIVIAL. Package is 64 LOC, pure path-join functions honoring ORCA_HOME/ORCA_DB env. Strategy: temp dir + t.Setenv("ORCA_HOME", dir) + assert each *Path() returns filepath.Join(dir, <file>); test ORCA_DB override; test default-to-~/.orca fallback. Model the test on namespace_test.go (cli). No new seams; 50%+ trivially achievable.
  • cmd/orca → 50% toe-hold REALISTIC but LOW VALUE. main.go is 15 LOC: cli.Execute() + fmt.Fprintf(os.Stderr, "error: %v") + os.Exit(1). The only testable behavior is "main() calls Execute and exits non-zero on error." A smoke test that calls main() in a subprocess (or refactors main into a run() int for testability) is the path. Flag: 50% on a 15-LOC glue file is ~7 lines of covered code — the effort:coverage ratio is poor. D-047 explicitly called this out ("0→70% risks a coverage rathole on cmd/orca which is glue code"). Recommend the plan keep this at the 50% toe-hold and not over-invest.

1.2 Existing test-helper utilities (reuse, do NOT re-create)

Helper Location Reuse for
openTestDB(t) internal/store/node_repo_test.go:12 engine, audit, store tests — returns (*NodeRepo, func()) backed by temp-file sqlite; adapt to return *sql.DB for JobRepo/TaskRepo/AuditRepo/CapacityRepo/CertRepo
withFastWatch(t, d) internal/store/job_task_repo_test.go:36 store Watch tests — overrides watchInterval for deterministic ticks
initTestEnv(t) internal/cli/init_test.go:17 cli tests — sets ORCA_HOME to temp dir + returns cleanup
resetRootFlags(t) internal/cli/namespace_test.go:13 cli tests — resets rootCmd args/out/json/system flags between subtests
discardWriter internal/cli/init_test.go:33 cli tests — io.Writer that discards stdout
stubDispatcher internal/daemon/dispatch_test.go:24 transport/engine tests — implements transport.Dispatcher (LocalSubmit/LocalStatus); reusable as a LocalExecutor too since the signatures match
insertNode(t, repo, ctx, id, name) internal/store/node_repo_test.go:217 store/doctor tests — inserts a minimal node
security.CAInit/LoadCA/GenerateCSR/SignCSR/WriteCert/WriteKey internal/security/ca.go transport mTLS tests — bootstrap a real CA + server cert into a temp dir (pattern in doctor_test.go:69-94)
t.Setenv("ORCA_HOME", dir) + t.Setenv("ORCA_DB", ...) internal/doctor/doctor_test.go:23-24 any test needing the orca namespace — preferred over manual os.Setenv (auto-cleanup)

1.3 Injected seams already present in the codebase (confirm by reading)

  1. sshDialer (proxmox)internal/proxmox/bootstrap.go:201-213: package-level var sshDialer sshDialerType = defaultSSHDialer{}; interface sshDialerType{ DialContext(ctx, network, addr, *ssh.ClientConfig) (*ssh.Client, error) }. Tests can swap sshDialer for a fake. GAP: no sessionRunner seam — runRemote (line 217) calls conn.NewSession() + session.CombinedOutput(cmd) directly on the concrete *ssh.Client. Recommend P01 plan add a sessionRunner interface (CombinedOutput(cmd) ([]byte, error)) so deployPubKey/createLinuxUser/createPVERole/createPVEUser/assignPVEACL/writeSudoers/validateSudoers become testable without a real SSH endpoint.
  2. LocalExecutor (engine dispatcher)internal/engine/dispatcher.go:39: interface Submit(ctx, []byte) (string, error) + Status(ctx, string) (string, error). Dispatcher depends on it; tests inject a stub. GAP: dispatchToPeer (line 154) calls transport.NewDispatchClient directly (no seam) — to test the remote-dispatch branch, either add a peerDispatcher interface or test via httptest.NewTLSServer.
  3. PeerPersister (engine peer)internal/engine/peer.go:39: optional persist callback; unused in production but available as a seam.
  4. Dispatcher (transport)internal/transport/dispatch.go:49: LocalSubmit/LocalStatus interface; stubDispatcher in daemon/dispatch_test.go:24 is the template stub.
  5. watchInterval (store)internal/store/job_task_repo.go:20: unexported var watchInterval = 1 * time.Second; tests override via withFastWatch.

1.4 Packages where 70% is unrealistic in a single phase (with evidence)

  • internal/cli — 70% is ambitious. 17 source files, ~2000 LOC. The daemon command (internal/cli/daemon.go) starts a long-running mTLS server — testing it requires a lifecycle harness (start, probe, shutdown) and is better covered by internal/daemon/server_test.go (already exists, 150 LOC). Recommend the P01 plan exclude daemon.go from the cli 70% target (document it as covered by the daemon package's own tests) and aim for 70% of the remaining cli files. Even so, 55-65% is the realistic single-phase outcome for the rest.
  • cmd/orca — 70% is explicitly out of scope per D-047. 15 LOC of glue; 50% toe-hold is the right call.
  • internal/proxmox — 70% likely requires the sessionRunner seam refactor. Without it, only the validation paths + sudoersContent string asserts are testable (~50-55%). The plan should add the seam; with it, 70% is achievable.

2. SSH trust hardening research (P02 — REQ-058, REQ-059)

2.1 Current TOFU knownhosts.New() callback — how it works

Location: internal/proxmox/bootstrap.go:125-128 (bootstrap) + internal/doctor/doctor.go:412-415 (doctor proxmox probe).

hostKeyCallback, err := knownhosts.New(certpaths.KnownHostsPath())
// ...
sshConfig := &ssh.ClientConfig{
    HostKeyCallback: hostKeyCallback,
    // ...
}

Mechanism (golang.org/x/crypto/ssh/knownhosts):

  • knownhosts.New(files ...string) returns an ssh.HostKeyCallback that reads the OpenSSH-format known_hosts file at certpaths.KnownHostsPath() (= $ORCA_HOME/known_hosts, see internal/certpaths/certpaths.go:62).
  • First connect (host absent from file): the callback returns a *knownhosts.KeyError with Want: [] (empty). This is a "host unknown" signal. IMPORTANT: knownhosts.New does NOT auto-write the key on first connect — it returns an error. The current orca code at bootstrap.go:140 treats ANY dial error as a failure (return nil, fmt.Errorf("ssh dial %s: %w", sshAddr, err)). This means the current TOFU flow is INCOMPLETE: on a truly first connect, knownhosts.New returns KeyError{Want:[]} and the dial fails — there is no capture-and-persist step. The v0.6 RESEARCH_v0.6.md §A.5 claimed knownhosts.New "handles both capture and verify in one callback" but the actual golang.org/x/crypto API does NOT auto-capture; it only verifies. This is a latent bug OR the operator is expected to pre-populate known_hosts manually (which contradicts the TOFU UX). P02 must address this: either (a) wrap knownhosts.New with a custom callback that captures on KeyError{Want:[]} and writes via knownhosts.Line, or (b) accept that --host-key-fingerprint (REQ-058) becomes the required path for first connect and TOFU capture is a separate enhancement. Flag for plan: the current TOFU capture is broken; P02 should fix it as part of the trust-hardening work (the --host-key-fingerprint path is actually simpler than TOFU because it doesn't need capture).
  • Subsequent connects (host present, key matches): callback returns nil → dial proceeds.
  • Subsequent connects (host present, key MISMATCH): callback returns *knownhosts.KeyError{Want: [knownKey]} → dial fails with a clear error. This is the MITM-detection path.

File format: OpenSSH known_hosts — one line per host: [host]:port ssh-key-type base64-key (or hashed-host form via knownhosts.HashHostname). knownhosts.Line(addresses []string, key ssh.PublicKey) string produces the line; knownhosts.Normalize(address) normalizes the host:port.

2.2 Result.HostKeyFingerprint — current computation (CRITICAL FINDING)

Location: internal/proxmox/bootstrap.go:83-85 (field declaration) + bootstrap.go:195-198 (return statement).

type Result struct {
    NodeName           string
    NodeAddress        string
    HostKeyFingerprint string  // field EXISTS
}
// ...
return &Result{
    NodeName:    opts.Host,
    NodeAddress: opts.Host + ":8443",
    // HostKeyFingerprint is NOT SET — always empty string
}, nil

Finding: Result.HostKeyFingerprint is declared but never populated. The current BootstrapProxmox returns it as "". There is no fingerprint computation today — no ssh.FingerprintSHA256 call, no hex digest, nothing. D-045's rationale ("matches the fingerprint format operators already see from orca node join's own Result.HostKeyFingerprint output") is based on a field that is currently always empty.

Implication for P02: The plan must ADD the fingerprint computation. The correct function is ssh.FingerprintSHA256(pubKey ssh.PublicKey) string (verified via go doc), which returns the OpenSSH SHA256:base64 format (unpadded base64, exactly what ssh-keyscan -E sha256 emits and what D-045 specifies). So D-045's format choice is correct by intent but the code doesn't produce it yet — P02 populates Result.HostKeyFingerprint = ssh.FingerprintSHA256(hostKey) during the capture path, and --host-key-fingerprint compares against ssh.FingerprintSHA256 of the server-presented key.

No existing fingerprint-comparison utility in internal/security/. security.Fingerprint (fingerprint.go:17) computes SHA-256 hex of an X.509 cert's DER — a DIFFERENT format (hex, not base64; X.509, not SSH). security.FingerprintOf (fingerprint.go:34) is the same. Do NOT reuse these for SSH host-key comparison — they're for the mTLS CA pin (--ca-fingerprint). P02 needs a new SSH-specific helper, e.g. security.SSHFingerprintSHA256(pubKey ssh.PublicKey) string (thin wrapper over ssh.FingerprintSHA256) or inline in proxmox/bootstrap.go.

2.3 Where --host-key-fingerprint plugs in (REQ-058)

CLI seam: internal/cli/node.go:344-354 — the init() registers flags on nodeJoinCmd. Add:

nodeJoinCmd.Flags().StringVar(&joinHostKeyFP, "host-key-fingerprint", "", "SSH host key SHA256:base64 fingerprint (pre-pin; supersedes TOFU for --type proxmox)")

Per D-044, the flag lives on orca node join (not just --type proxmox); validation in RunE (node.go:78-83) emits a clear error if the flag is set for a non-proxmox type.

Transport seam: internal/proxmox/bootstrap.go:131-136ssh.ClientConfig.HostKeyCallback. Currently knownhosts.New(...). When --host-key-fingerprint is supplied, replace the callback with a ssh.FixedHostKey-style verifier that:

  1. Parses the operator-supplied SHA256:base64 string (strip SHA256: prefix, base64-decode → 32 bytes).
  2. In the callback, receives the server's ssh.PublicKey, computes ssh.FingerprintSHA256(key), compares to the operator string.
  3. Returns nil on match, error on mismatch (fail closed).

Recommended callback shape (concrete):

func pinnedHostKeyCallback(expectedSHA256Base64 string) (ssh.HostKeyCallback, error) {
    // Validate format: must start with "SHA256:".
    if !strings.HasPrefix(expectedSHA256Base64, "SHA256:") {
        return nil, fmt.Errorf("host-key-fingerprint: must be OpenSSH SHA256:base64 format (got %q)", expectedSHA256Base64)
    }
    expected := expectedSHA256Base64 // store full string for direct compare
    return func(_ string, _ net.Addr, key ssh.PublicKey) error {
        got := ssh.FingerprintSHA256(key)
        if got != expected {
            return fmt.Errorf("host key fingerprint mismatch: got %s, want %s — refusing to connect (REQ-058)", got, expected)
        }
        return nil
    }, nil
}

Why compare full strings (not base64-decoded bytes): ssh.FingerprintSHA256 returns the canonical SHA256:base64 string; comparing it directly to the operator-supplied string is simplest and avoids a base64-decode step. Reject non-SHA256:-prefixed input up front with a clear error (D-045: "Accept only SHA256:-prefixed base64; reject raw hex with a clear error").

Pass-through to proxmox: internal/cli/node.go:158-166 — add HostKeyFingerprint string to proxmox.Options (bootstrap.go:55) and pass joinHostKeyFP through. BootstrapProxmox selects the callback: if opts.HostKeyFingerprint != "" use pinnedHostKeyCallback, else fall back to the TOFU knownhosts.New (with the capture-fix from §2.1).

2.4 orca node key-reset <node> (REQ-059, D-046 — local known_hosts only)

Scope (D-046): clear the local ~/.orca/known_hosts entry for the node ONLY; do NOT revoke the remote authorized_keys entry (would orphan a working node). Audit-log event=node.key_reset with actor + node.

known_hosts line format written by golang.org/x/crypto/ssh/knownhosts:

  • knownhosts.Line(addresses []string, key ssh.PublicKey) string"[host]:port ssh-ed25519 AAAA...\n" (or host ssh-ed25519 AAAA... if port 22 — knownhosts.Normalize handles the :22 vs bare-host normalization).
  • The file is plain text, one entry per line, #-prefixed comments allowed.

No library function to remove a host's entries. knownhosts.New only reads. The reset must be implemented manually:

  1. Read certpaths.KnownHostsPath() (internal/certpaths/certpaths.go:62).
  2. Filter lines: keep lines whose host field (before the first whitespace) does NOT match knownhosts.Normalize(nodeName) (or the node's address). Edge: a host may have multiple entries (one per key type); remove all matching lines.
  3. Write the filtered content back via atomic rewrite (temp file in same dir + os.Rename) — reuse security.writeAtomic (ca.go:305) OR implement inline (it's unexported in security; either export it or copy the ~20-LOC pattern). Recommend atomic rewrite, NOT in-place truncation — in-place rewrite via os.OpenFile(O_TRUNC|O_WRONLY) risks data loss on crash mid-write.

CLI registration seam: internal/cli/node.go:358-360 — the init() does nodeCmd.AddCommand(nodeJoinCmd), nodeLeaveCmd, nodeListCmd. Add:

nodeCmd.AddCommand(nodeKeyResetCmd)

where nodeKeyResetCmd is a new &cobra.Command{Use: "key-reset <node>", Args: cobra.ExactArgs(1), RunE: ...}. The RunE:

  1. Resolve <node> arg → look up the node in the registry (nodeRegistry() at node.go:37) to get its address (for matching known_hosts lines) — OR accept the raw host string directly. Recommend: accept the node NAME (consistent with doctor proxmox which iterates node.Name), look up the node row, use node.Name (which is the host address for proxmox nodes per bootstrap.go:196) as the known_hosts match key.
  2. Call a new proxmox.ResetHostKey(host string) error (or inline in cli) that does the atomic rewrite.
  3. Audit-log via engine.Audit.Record(ctx, "cli", "node.key_reset", nodeID, "success", nil, map[string]any{"host": host}).
  4. Print ✓ Host key reset for <node> (next connect will re-pin via TOFU or --host-key-fingerprint).

Reusability: the nodeRegistry() helper (node.go:37) + openDB() (node.go:25) + newLogger() (node.go:33) are all available for the key-reset command.

2.5 CLI registration seam summary (P02)

Addition File:line Change
--host-key-fingerprint flag internal/cli/node.go:344-354 (init) nodeJoinCmd.Flags().StringVar(&joinHostKeyFP, "host-key-fingerprint", "", "...")
joinHostKeyFP var internal/cli/node.go:47-60 (var block) add joinHostKeyFP string
Pass-through to proxmox internal/cli/node.go:158-166 (joinProxmox) add HostKeyFingerprint: joinHostKeyFP to proxmox.Options
HostKeyFingerprint field internal/proxmox/bootstrap.go:55 (Options) add field
Pinned callback internal/proxmox/bootstrap.go:131-136 branch: if opts.HostKeyFingerprint != "" use pinned callback else TOFU
Populate Result.HostKeyFingerprint internal/proxmox/bootstrap.go:195-198 set HostKeyFingerprint: ssh.FingerprintSHA256(hostKey) during capture
key-reset subcommand internal/cli/node.go:358-360 (init) nodeCmd.AddCommand(nodeKeyResetCmd) + new cmd var
ResetHostKey helper internal/proxmox/bootstrap.go (new) OR internal/security/sshkey.go atomic known_hosts rewrite

3. Requirements-hygiene gate research (P03 — REQ-060)

3.1 Current Makefile targets

Makefile has 11 targets: build, test, test-race, lint, fmt, clean, run, version, changelog, release, security-scan (Makefile:1-100). No verify-reqs target exists. The .PHONY list at line 1 must be extended.

3.2 Current .coreci.yml pipeline structure

4 pipelines (.coreci.yml:19-134):

  • validate (line 20): 4 steps — go-version (gofmt+vet), gosec, govulncheck, gitleaks.
  • build (line 53): 1 step — version-injected go build.
  • test (line 72): 1 step — go test -race -coverprofile=coverage.out ./... + go tool cover -func | tail -1.
  • release (line 81): gated on refs/tags/v*; 3 steps — build-artifact, gitea-release, container-publish.

Hook for verify-reqs: add a 5th step to the validate pipeline (after go-version, before/after gosec) OR add it to the test pipeline. Recommend validate pipeline — requirements hygiene is a static check (no test run needed), belongs alongside gofmt/vet/lint. Step shape:

- name: verify-reqs
  image: golang:1.25
  commands:
    - make verify-reqs

3.3 verify-reqs implementation recommendation

Assertion (REQ-060): every REQ row in ROADMAP.md marked [x]/Complete must have a matching REQ-ID row in REQUIREMENTS.md with Complete status. (Reverse direction — every REQUIREMENTS Complete has a ROADMAP [x] — is also worth checking but the drift that motivated this was ROADMAP-shipped-but-REQUIREMENTS-Pending, so the forward direction is the priority.)

Approach: small Go program in cmd/verify-reqs OR a shell+awk script?

  • Go program (~80 LOC): parse both markdown tables with regexp, build two map[string]string (REQ-ID → status), diff. Pros: type-safe, testable, consistent with the Go toolchain; can be a cmd/verify-reqs/main.go with its own _test.go. Cons: adds a binary target.
  • Shell+awk (~30 LOC): awk over the markdown tables. Pros: no new Go package; minimal. Cons: fragile parsing, hard to test, shell-quoting issues.

Recommendation: Go program at cmd/verify-reqs/main.go. Reasons: (1) testable with golden-file fixtures (parse a sample ROADMAP+REQUIREMENTS pair, assert diff); (2) consistent with the project's Go-only tooling ethos (no shell-awk fragility); (3) the make verify-reqs target just calls go run ./cmd/verify-reqs; (4) CoreCI's golang:1.25 image has go available — no extra dep.

Parsing approach (concrete):

  1. ROADMAP.md: regex ^\s*-\s*\[(x|X| )\]\s*Phase.*—.*tag is NOT the right pattern (that's phase lines, not REQ rows). The REQ coverage is in per-phase bullet lists under "### Per-phase REQ coverage" (ROADMAP.md:161-180) AND in the milestone section bodies. Simpler: the ROADMAP uses - [x] Phase N: ... for completed phases. The authoritative REQ↔status mapping lives in REQUIREMENTS.md (the single table at lines 9-56 + per-milestone tables at 103-142). Re-interpret REQ-060: the assertion is really "ROADMAP milestone sections marked COMPLETE ↔ REQUIREMENTS rows for that milestone marked Complete." The drift was: v0.7 ROADMAP said "COMPLETE" (line 116) but REQUIREMENTS v0.7 rows (REQ-053..056) were "Pending" (now corrected to "Complete" in SPECIFY).
  2. Refined assertion: parse REQUIREMENTS.md table rows (| REQ-XXX | ... | ... | ... | **Complete** | or | Pending |); for each REQ-ID, record status. Then parse ROADMAP.md for milestone-level "COMPLETE" markers (## Milestone v0.X: ... — **COMPLETE**) AND phase-level - [x] markers. For each milestone marked COMPLETE in ROADMAP, assert every REQ-ID belonging to that milestone (per the REQUIREMENTS milestone column) is Complete in REQUIREMENTS. OR (simpler, matches the SPECIFY wording): for every REQ-ID in REQUIREMENTS.md whose Phase column references a milestone that ROADMAP marks COMPLETE, the Status must be Complete. This catches the exact drift (ROADMAP-shipped, REQUIREMENTS-stale).

Concrete regex:

  • REQUIREMENTS row: ^\|\s*(REQ-\d+)\s*\|.*?\|\s*\*\*(Complete|Pending)\*\*\s*\| (capture ID + status).
  • ROADMAP milestone-complete: ^##\s*Milestone\s+v0\.\d+:.*—\s*\*\*COMPLETE\*\* (capture milestone label).
  • Map milestone → REQs via the REQUIREMENTS Phase column (e.g. v0.7 P1 → milestone v0.7).

Where it hooks in: make verify-reqs runs go run ./cmd/verify-reqs .ciagent/ROADMAP.md .ciagent/REQUIREMENTS.md; .coreci.yml validate pipeline adds the step. Exit 0 on consistency, exit 1 with a diff listing on drift.

3.4 The drift that motivated REQ-060

After v0.7 ship, REQUIREMENTS.md rows REQ-053..056 were "Pending" despite ROADMAP.md marking milestone v0.7 COMPLETE and all phases [x]. This was corrected during v0.8 SPECIFY (the rows now read **Complete**). REQ-060 ensures the drift cannot recur: the CI validate pipeline fails if ROADMAP says COMPLETE but REQUIREMENTS says Pending.


4. Architectural decisions surfaced (AD-027..AD-030)

ID Decision Rationale
AD-027 ssh.FingerprintSHA256 (OpenSSH SHA256:base64) as the SSH host-key fingerprint format Matches D-045 + ssh-keyscan -E sha256 output. The existing security.Fingerprint (hex, X.509) is NOT reused — different domain. P02 adds a thin SSH-specific helper.
AD-028 --host-key-fingerprint callback compares full SHA256:base64 strings, not decoded bytes ssh.FingerprintSHA256 returns the canonical string; direct string compare avoids a base64-decode step and is less error-prone. Validate SHA256: prefix up front.
AD-029 orca node key-reset rewrites known_hosts via atomic temp-file + rename Prevents data loss on crash mid-write. Reuse the writeAtomic pattern from security/ca.go:305 (export it or copy the ~20 LOC).
AD-030 verify-reqs implemented as cmd/verify-reqs/main.go (Go program), not shell+awk Testable, type-safe, consistent with Go-only tooling. make verify-reqs runs go run ./cmd/verify-reqs. Hooked into .coreci.yml validate pipeline.

5. Pitfalls, gaps, and flags for the plan

  1. TOFU capture is currently BROKEN (§2.1). knownhosts.New returns KeyError{Want:[]} on first connect and does NOT auto-write the key. The current BootstrapProxmox treats this as a dial failure. P02 must either (a) wrap the callback to capture-and-persist on KeyError{Want:[]} via knownhosts.Line + atomic write, or (b) make --host-key-fingerprint the required first-connect path. Recommend (a) — fix TOFU + add pre-pin as superset. This is a v0.6 latent bug that P02 closes.
  2. Result.HostKeyFingerprint is never populated (§2.2). D-045's rationale references "existing output" that doesn't exist. P02 must ADD the computation (ssh.FingerprintSHA256). Low risk — it's a 1-line addition once the host key is available.
  3. No sessionRunner seam in proxmox (§1.3). Testing the SSH command sequence (deployPubKey, createLinuxUser, pveum, sudoers, visudo) without a real SSH server requires a new interface seam. Recommend P01 plan add it — 1 interface, ~10 LOC, unlocks ~40% of proxmox coverage.
  4. internal/store/cert_repo.go has NO test (§1.1). v0.7 P01 REQ-053 was supposed to add cert_repo_test.go but it's missing — internal/store/ glob shows no cert_repo_test.go. This is a v0.7 leftover. P01 should add it (it directly lifts store coverage toward 70%).
  5. internal/cli/daemon.go excluded from cli 70% target (§1.4). The daemon command starts a long-running server; it's covered by internal/daemon/server_test.go (150 LOC). Don't double-test in cli.
  6. cmd/orca 50% toe-hold is low-value (§1.1). 15 LOC of glue; the test effort:coverage ratio is poor. D-047 already called this out. Don't over-invest.
  7. go: no such tool "covdata" for zero-test packages (§1.1). This is a Go toolchain quirk when a package has no test files — go test -cover can't compute coverage without a test binary. It's NOT a real 0% number (it's "undefined"). Adding any _test.go file makes the number computable. Don't treat the error as a coverage measurement.
  8. transport.dispatchToPeer has no seam (§1.3). Testing the remote-dispatch branch of Dispatcher.Submit requires either a new peerDispatcher interface OR httptest.NewTLSServer. The latter is already used in daemon/dispatch_test.go; recommend the plan use httptest.NewTLSServer (no refactor needed) for transport coverage.
  9. knownhosts.Line + knownhosts.Normalize are the helpers for the TOFU-capture fix and for key-reset matching (§2.1, §2.4). Use Normalize to match host strings consistently (handles host:22 vs host).
  10. security.writeAtomic is unexported (ca.go:305). key-reset's atomic known_hosts rewrite needs it. Either export WriteAtomic from security, or copy the ~20-LOC pattern into proxmox/cli. Recommend export — it's already used across ca.go + sshkey.go and is generally useful.

6. Dependencies

v0.8 adds zero new direct dependencies:

  • SSH host-key fingerprint: ssh.FingerprintSHA256 (already in golang.org/x/crypto/ssh v0.54.0, direct dep since v0.6).
  • knownhosts.Line/Normalize/KeyError: same golang.org/x/crypto module.
  • verify-reqs: stdlib only (regexp, os, fmt).
  • Tests: net/http/httptest (stdlib), existing interfaces.

go.mod is unchanged by v0.8.


7. PERSONAS assessment (v0.8)

v0.8 is an NFR milestone touching tests (9 packages), SSH trust surface (proxmox + cli/node + security), and a requirements-hygiene Go program. The 3-persona roster from config.json (lead-developer, backend-engineer, data-engineer) is sufficient — no phase-specific personas needed.

Roster confirmation:

  • lead-developer — owns coordination + cmd/orca smoke test + internal/cli coverage (cert/doctor/audit/status/version subcommands) + the verify-reqs Go program (coordination territory).
  • backend-engineer — owns internal/transport tests (httptest.NewTLSServer) + internal/engine tests (LocalExecutor stubs, PeerRegistry) + SSH trust-surface in internal/proxmox/bootstrap.go (pinned callback, TOFU capture fix, sessionRunner seam) + internal/cli/node.go (--host-key-fingerprint flag, key-reset subcommand).
  • data-engineer — owns internal/store tests (cert_repo_test.go gap + coverage uplift) + internal/audit tests (sqlite-backed audit_log asserts) + internal/certpaths tests (path-join asserts) + internal/jobspec tests (golden HCL fixtures).

No frontend persona (no UI). No devops persona (no packaging/distribution — verify-reqs is a Go program, not a CI config change; the .coreci.yml edit is a 3-line hook, lead-developer territory). No security-engineer persona (the SSH trust work is backend-engineer territory — the security-engineer was deactivated in v0.7 and v0.8 doesn't re-add it; the trust-surface hardening is a refinement of the existing proxmox package, not new security architecture).

See .ciagent/PERSONAS.md (updated with v0.8 YAML frontmatter + territory globs matching the actual file structure).