orca/.ciagent/RESEARCH_v0.3.md

Research: Orca v0.3 — scheduling-streaming

Phase: 0 (research) for milestone v0.3 (scheduling-streaming). Branch: phase/00-pre-execution (cut from milestone/v0.3-scheduling-streaming). Go toolchain: go1.25.0 (confirmed via go version; go.mod declares go 1.25.0).

This document provides concrete, file-level implementation guidance for the two v0.3 execution phases:

• P01 — iter.Seq streaming for --watch flags (REQ-022, REQ-030)
• P02 — orca doctor network + db full implementation (REQ-032 completion)

All assumptions are logged as decisions (D-025..D-038) with confidence scores. Full autonomy mode — no items flagged for human validation.

---

Codebase Audit Summary

Current state (commit ba5ffd7 + phase docs)

Area	File	Key finding
CLI job list	internal/cli/job.go:112-143	jobListCmd.RunE calls store.NewJobRepo(db).List(ctx), prints a fixed-width table; --json via printJSON(jobs). No --watch flag exists.
CLI node list	internal/cli/node.go:155-186	nodeListCmd.RunE calls registry.List(ctx) → repo.List(ctx). Table + --json. No --watch flag.
CLI root	internal/cli/root.go	jsonOutput is a package-level bool set by --json persistent flag. printJSON uses json.NewEncoder with 2-space indent.
Job repo	internal/store/job_task_repo.go	JobRepo.List(ctx) ([]*model.Job, error) — single-shot query, closes rows. scanJob helper is reusable.
Node repo	internal/store/node_repo.go	NodeRepo.List(ctx) ([]*model.Node, error). scanNode helper is reusable. scanner interface defined here (Scan(dest ...any) error) — shared by *sql.Row and *sql.Rows.
Store open	internal/store/store.go	store.Open(path) opens with ?_pragma=journal_mode(WAL)&_pragma=foreign_keys(ON) and runs migrate(db).
Migrations	internal/store/migrate.go	migrate is unexported, runs at Open time. schema_migrations table tracks applied migrations by filename. Migrations are embedded via //go:embed migrations/*.sql. No public API to query migration version.
Migrations on disk	internal/store/migrations/	0001_nodes.sql, 0002_jobs_tasks.sql, 0003_audit_log.sql, 0004_certs.sql, 0005_node_capacity.sql. Highest = 0005.
Doctor	internal/doctor/doctor.go	NetworkStub() and DBStub() return WARN stubs. All() aggregates 6 checks. Run(ctx) iterates checks. Check.Run signature: func(ctx context.Context) (Result, string). Result is `PASS
Doctor CLI	internal/cli/doctor.go	doctorNetworkCmd/doctorDBCmd call doctor.NetworkStub()/doctor.DBStub() directly.
Doctor tests	internal/doctor/doctor_test.go	Two tests: TestRunAllChecksWithNoCA (expects FAIL + WARN for stubs), TestRunWithCAAndServerCert (cert checks PASS). Uses t.Setenv("ORCA_HOME", dir). The "expects WARN" assertion will break when stubs become real checks — must be updated in P02.
Transport mTLS client	internal/transport/mtls.go	NewMTLSClient(caPath, serverName, certPath, keyPath) builds an http.Client with a TLS-1.3-only config from security.ClientTLSConfig. MTLSClient.Do(req). DialContext for low-level TLS dial.
Transport dispatch client	internal/transport/dispatch.go	NewDispatchClient(caPath, serverName, peerAddr) wraps MTLSClient. PeerAddr is http:// or https://. Submit/Status POST to /orca.v1.Dispatch/*. No /healthz GET helper.
Daemon health	internal/daemon/health.go	handleHealthz → 200 {"status":"alive"}. handleReadyz → 200/503 with db ping. Mounted at mux.HandleFunc("/healthz", ...) in server.go:104.
Daemon TLS	internal/daemon/tls.go	StartMTLS(state) sets httpServer.TLSConfig with ClientAuth = RequireAndVerifyClientCert. The daemon requires client certs in mTLS mode.
Peer registry	internal/engine/peer.go	PeerRegistry is in-memory only (map[string]*Peer under sync.RWMutex). NewPeerRegistry() returns empty. Peer has NodeID, Address, ServerName, CAPath, LastSeen, Capacity. Not persisted to SQLite.
Peer registry usage	internal/cli/job.go:70, internal/cli/daemon.go:47	Both create a fresh empty NewPeerRegistry() per process. No code ever calls peers.Add(...). The registry is currently a structural placeholder.
Node registry	internal/engine/registry.go	NodeRegistry wraps store.NodeRepo + Audit. List(ctx) → repo.List(ctx). Persisted to nodes table.
Node model	internal/model/node.go	Node{ID, Name, Address, State, JoinedAt, LastSeen, Metadata}. No ServerName or CAPath field — model.Node differs from engine.Peer.
Cert paths	internal/certpaths/certpaths.go	Dir() honors ORCA_HOME; CACertPath(), ServerCertPath(), ServerKeyPath().
Security client TLS	internal/security/tls_config.go:106	ClientTLSConfig(caPath, serverName, certPath, keyPath) — TLS 1.3 only, AEAD allowlist, RootCAs = single CA. Both-or-neither for cert/key.
Go version	go.mod + go version	go 1.25.0 — iter package and range-over-func are stable stdlib.
Deps	go.mod	cobra, hcl/v2, modernc/sqlite, uuid. No new deps needed for v0.3. iter is stdlib.

Critical gap analysis

1. PeerRegistry is non-persistent and always empty at CLI time. The doctor network check cannot rely on it — there is no code path that populates it. The nodes table IS persisted and has Address, but lacks the ServerName/CAPath needed for an mTLS probe. Resolution: doctor network reads the nodes table via NodeRepo.List, and derives ServerName + CAPath from local config (certpaths.CACertPath() + node name/addr). See D-029.

2. doctor.Run / Check.Run do not plumb a *sql.DB or transport client. The cert checks are filesystem-only. P02 must extend the check constructors to accept a DB handle and (for network) a transport client factory. The Check.Run signature (func(ctx) (Result, string)) is preserved by closure-capturing the handles in the constructor. See D-027, D-031.

3. No public migration-version query. migrate() is unexported and writes to schema_migrations(name, applied_at). P02 adds a public store.MigrationVersion(ctx, db) (or method on a repo) that selects the max applied migration name. See D-033.

4. Doctor test TestRunAllChecksWithNoCA asserts a WARN from stubs. This will break when stubs become real (the db check will PASS with a fresh test DB, and the network check will WARN/FAIL on zero peers). Must be updated. See D-036.

---

P01: iter.Seq Streaming for --watch Flags

Covers REQ-022 (iter.Seq for streaming job lists), REQ-030 (--watch output format: table default vs streaming one-line JSON per event).

Decisions

ID	Decision	Confidence
D-025	iter.Seq lives on the store repos, not the engine registry. JobRepo.Watch(ctx) iter.Seq[*model.Job] and NodeRepo.Watch(ctx) iter.Seq[*model.Node]. Rationale: repos already own the *sql.DB and the scanJob/scanNode helpers; engine.Registry.List just delegates to repo. Keeping Watch in the store layer matches the data-engineer territory and avoids a new engine→store iter dependency.	0.90
D-026	Element type is *model.Job / *model.Node (pointer), matching the existing []*model.Job return of List. This keeps printJSON and table rendering identical between one-shot and watch paths.	0.88
D-027	The Check.Run signature in doctor is unchanged; P02 captures DB/transport handles in closure at constructor time (Network(db), DB(db)). This is the established pattern (cert checks already closure-capture certpaths).	0.92
D-028	Watch refresh = poll-based 1s ticker (per D-019). No event channel, no daemon coupling. Each tick re-runs the existing List query and yields the full current snapshot (one element per row). The CLI dedupes by detecting snapshot equality before re-rendering (see D-030). Rationale: simpler than NOTIFY/LISTEN, no daemon dependency, matches offline-first.	0.90
D-029	--watch output: default = re-print the table on every changed snapshot (clear screen via ANSI \033[2J\033[H then table); --watch --json = one compact JSON line per snapshot (an array on each line, OR one line per element — see D-030). The CLI tracks the previous snapshot's hash to avoid spamming identical frames.	0.85
D-030	--watch --json emits one JSON object per element per tick where the element changed, i.e. streaming one-line JSON per event (per REQ-030 wording). Implementation: on each tick, for each element, if its JSON bytes differ from the previous snapshot's bytes for that ID, print {"event":"update","job":{...}}\n. On first tick, print all as {"event":"init",...}. This is the most useful for AI agents tailing the stream. Confidence lower because REQ-030 is ambiguous between "array per tick" and "object per event"; the per-event interpretation matches "streaming one-line JSON per event" literally.	0.72
D-031	Cancellation: signal.NotifyContext(ctx, os.Interrupt, syscall.SIGTERM) at the CLI command layer, replacing the current context.WithTimeout(cmd.Context(), 5*time.Second) for the watch path only. The non-watch list path keeps its 5s timeout. The iter.Seq receives this ctx and stops yielding on ctx.Done().	0.93
D-032	The iter.Seq implementation must not leak goroutines: the polling loop runs inline in the yield callback's caller goroutine (the range loop), not a separate goroutine. for range seq { ... } drives the pull; inside Watch, we loop for { select { <-ticker.C: query+yield each; <-ctx.Done(): return } } and call yield(item) directly. When yield returns false (consumer broke the loop), we stop and return. No goroutine is spawned by Watch. This is the cleanest Go 1.25 iter pattern and avoids leak surface entirely.	0.95

Implementation approach — store layer

File: internal/store/job_task_repo.go — add method:

// Watch yields the current snapshot of jobs on a 1-second ticker until
// ctx is cancelled or the consumer stops pulling (yield returns false).
// It does not spawn a goroutine; the polling loop runs in the caller's
// goroutine via the range-over-func pull protocol.
//
// Each tick re-runs the List query and yields one *model.Job per row.
// The caller is responsible for deduping across ticks if desired.
func (r *JobRepo) Watch(ctx context.Context) iter.Seq[*model.Job] {
    return func(yield func(*model.Job) bool) {
        ticker := time.NewTicker(1 * time.Second)
        defer ticker.Stop()
        for {
            select {
            case <-ctx.Done():
                return
            case <-ticker.C:
            }
            // Reuse the existing List query + scanJob helper.
            rows, err := r.db.QueryContext(ctx,
                `SELECT id, name, spec, status, exit_code, created_at, started_at, ended_at
                 FROM jobs ORDER BY created_at DESC`)
            if err != nil {
                // Surfacing errors from inside iter.Seq is awkward; the
                // CLI layer cannot receive a returned error. Log via slog
                // (the repo doesn't hold a logger today — see D-034) and
                // continue to next tick rather than terminating the
                // stream. A transient DB blip should not kill the watch.
                continue
            }
            for rows.Next() {
                j, err := scanJob(rows)
                if err != nil {
                    rows.Close()
                    return
                }
                if !yield(j) {
                    rows.Close()
                    return // consumer stopped
                }
            }
            rows.Close()
        }
    }
}

File: internal/store/node_repo.go — add analogous Watch:

func (r *NodeRepo) Watch(ctx context.Context) iter.Seq[*model.Node] {
    return func(yield func(*model.Node) bool) {
        ticker := time.NewTicker(1 * time.Second)
        defer ticker.Stop()
        for {
            select {
            case <-ctx.Done():
                return
            case <-ticker.C:
            }
            rows, err := r.db.QueryContext(ctx,
                `SELECT id, name, address, state, joined_at, last_seen, metadata
                 FROM nodes ORDER BY joined_at ASC`)
            if err != nil {
                continue
            }
            for rows.Next() {
                n, err := scanNode(rows)
                if err != nil {
                    rows.Close()
                    return
                }
                if !yield(n) {
                    rows.Close()
                    return
                }
            }
            rows.Close()
        }
    }
}

Imports: add "iter" and "time" (time already present in both files). The iter package is imported only for the return type; yield is the callback.

Note on the existing scanner interface: scanJob/scanNode accept the scanner interface (Scan(dest ...any) error) satisfied by both *sql.Row and *sql.Rows, so they are directly reusable in Watch — no refactor needed.

Implementation approach — CLI layer

File: internal/cli/job.go — modify jobListCmd:

1. Add a package var jobWatch bool and register jobListCmd.Flags().BoolVar(&jobWatch, "watch", false, "stream jobs until Ctrl-C").

2. In RunE, branch on jobWatch:

   • If !jobWatch: keep the existing 5s-timeout List path.
   • If jobWatch:
     • ctx, cancel := signal.NotifyContext(cmd.Context(), os.Interrupt, syscall.SIGTERM); defer cancel() (drop the 5s timeout).
     • seq := store.NewJobRepo(db).Watch(ctx)
     • If jsonOutput: stream one-line JSON per event (D-030). Maintain a map[string][]byte of last-seen JSON per job ID. On each yielded job, marshal compact JSON; if it differs from the stored bytes (or ID unseen), print {"event":"update","job":{...}}\n and update the map.
     • Else (table): on each tick, after collecting the full snapshot, compare against the previous snapshot (by hashing the rendered table string or by comparing the slice of []*model.Job via reflect/cmp). If changed, emit "\033[2J\033[H" (clear) then the table header + rows. This gives a "top-like" refresh.

   Because iter.Seq does not return an error, the watch path swallows per-tick query errors inside Watch (D-034). The CLI relies on ctx.Done() for termination.

File: internal/cli/node.go — analogous change to nodeListCmd:

• Add nodeWatch bool, register --watch flag.
• Branch in RunE; seq := store.NewNodeRepo(db).Watch(ctx) (open a fresh openDB() for the watch path; registry.List is not used for watch — go straight to the repo to get the iter).

Note: nodeListCmd currently goes through nodeRegistry() which wraps NodeRepo in engine.NodeRegistry. For watch, bypass the registry and use store.NewNodeRepo(db).Watch(ctx) directly — the registry adds no value for a read-only stream and would require a Watch passthrough method. This keeps the iter boundary clean in the store layer (D-025).

Pitfalls & mitigations (P01)

Pitfall	Mitigation
Goroutine leak if Watch spawned a goroutine.	It doesn't — the polling loop is inline in the pull callback (D-032). defer ticker.Stop() + rows.Close() on every exit path.
Rows cursor held open across yield — if yield blocks (e.g. slow consumer), the *sql.Rows stays open and holds a SQLite read lock.	Yield is called per-row inside the rows.Next() loop; the consumer (range) is fast (prints to stdout). For safety, close rows immediately after the loop or on yield==false. The 1s tick cadence bounds how long a cursor is held. WAL mode (set in store.Open) allows concurrent reads, so this does not block writers.
No error channel from iter.Seq — a transient DB error is invisible to the CLI.	Log inside Watch via a package-level slog default (slog.Default().Warn(...)) since the repo has no logger field today (D-034: add an optional logger *slog.Logger to JobRepo/NodeRepo, defaulting to slog.Default() in the constructors — minimal change). Continue to next tick rather than terminating.
ctx cancellation mid-query — QueryContext returns an error; rows.Next() returns false.	Handled: the select on ctx.Done() returns before the next tick; an in-flight query is cancelled by the ctx.
Rapid re-render flicker in table mode.	Clear-screen + full re-render on changed snapshot only (hash compare). Unchanged snapshots produce no output.
--watch + --json interleaving with slog stderr.	slog writes to stderr; CLI output to stdout — no interleaving on stdout. Safe.
Test determinism — ticker is 1s, tests would be slow/flaky.	Provide a test-only constructor WatchWithInterval(ctx, d time.Duration) OR make the interval a field on the repo set via an unexported option. Preferred: an unexported watchInterval package var defaulting to 1s, overridable from internal/store tests. See D-035.
Signal handling clobbers root signal handler.	signal.NotifyContext with os.Interrupt returns a fresh ctx; the root cobra.Command does not install its own SIGINT handler, so no conflict. defer cancel() restores default behavior on exit.

Test strategy (P01)

internal/store/job_task_repo_test.go (new file or appended):

• TestJobRepoWatch_YieldsSnapshots: insert 1 job, call Watch with a 10ms interval (via test hook), range over seq collecting into a slice, insert a 2nd job from a goroutine after 30ms, cancel ctx after 80ms, assert the 2nd job appeared in the collected slice. Use context.WithTimeout for cancellation.
• TestJobRepoWatch_StopsOnConsumerBreak: range over seq and break after the first yield; assert the function returns (no hang) within a short deadline. This validates the yield==false path.
• TestJobRepoWatch_StopsOnCtxCancel: cancel ctx; assert the range loop exits within 50ms.
• TestNodeRepoWatch_*: mirror the above for nodes.

internal/cli/job_test.go (new) / node_test.go (new) — if CLI tests exist; otherwise add:

• TestJobListWatch_JSONStreaming: spin a temp DB, insert a job, invoke the jobListCmd.RunE with --watch --json in a goroutine, insert a 2nd job, capture stdout for ~200ms, assert two JSON lines appear. Cancel via ctx.
• TestJobListWatch_TableRefresh: assert clear-screen escape + table re-render on change.
• These CLI tests are harder to make deterministic; prefer testing the store-layer Watch thoroughly and keep CLI watch tests to a smoke-level "produces output, exits on ctx.Done".

Dependency check (P01)

• iter — Go 1.25 stdlib (go.mod declares go 1.25.0). ✅ no new dep.
• time, context, os/signal, syscall, encoding/json — stdlib. ✅
• No new go.mod dependencies required.

---

P02: orca doctor network + db full implementation

Covers REQ-032 completion (network + db checks replacing NetworkStub/DBStub).

Decisions

ID	Decision	Confidence
D-033	Add a public store.MigrationVersion(ctx, db) (string, error) function (in migrate.go or a new internal/store/migrate_query.go) that returns the highest applied migration filename from schema_migrations. SQL: SELECT name FROM schema_migrations ORDER BY name DESC LIMIT 1. This is the source of truth for "migration version" — it reflects what migrate() actually applied. Returns ("", nil) if no migrations applied (fresh empty table) and ("", sql.ErrNoRows) is treated as empty.	0.90
D-034	The doctor DB check opens its own *sql.DB via store.Open(dbPath()) (reusing the CLI's dbPath) inside the check constructor closure, rather than receiving a shared handle. Rationale: doctor should be runnable whether the daemon is up or down; store.Open uses WAL so a concurrent daemon is fine. The check defer db.Close(). This avoids threading a *sql.DB through doctor.Run/All() and keeps the Check.Run signature stable.	0.86
D-035	The doctor DB check runs PRAGMA integrity_check via db.QueryRow("PRAGMA integrity_check"). SQLite returns a single row with a TEXT value: "ok" on success, or a multi-line error description on failure. PASS if the value is "ok"; FAIL otherwise (with the first line of the message). Plus query the migration version (D-033); WARN if schema_migrations is empty (fresh/never-migrated db) — this is suspicious but not corrupt.	0.92
D-036	Doctor network check sources peer addresses from the nodes table (NodeRepo.List), NOT from engine.PeerRegistry (which is in-memory and always empty at CLI time — see gap #1). For each node with state != 'left', probe https://<address>/healthz over mTLS.	0.88
D-037	For each peer, the network check builds an mTLS client via transport.NewMTLSClient(certpaths.CACertPath(), serverName, certpaths.ServerCertPath(), certpaths.ServerKeyPath()). serverName is derived as the node's Name (the SAN on a peer's server cert is its node name, per security.GenerateCSR(nodeName, sans) — confirmed in integration_test.go:41 GenerateCSR("test-server", ...). If the SAN uses a different value the probe will fail handshake, which is itself a useful diagnostic). CAPath is the local ca.crt (all peers share one CA per D-011). This presents the local node's client cert, satisfying the daemon's RequireAndVerifyClientCert.	0.80
D-038	Network check result semantics: zero peers registered → WARN ("no peers registered; network check skipped") — not FAIL, because a single-node install legitimately has no peers. A peer unreachable / handshake failed → FAIL for that peer, aggregated to a single network check result that is FAIL if any peer failed, PASS if all peers probed OK, WARN if zero peers. Each peer's per-line outcome is folded into the message string (e.g. PASS — 2/2 peers reachable; FAIL — peer node-b (host:port): tls handshake error).	0.85

Implementation approach — db check

File: internal/store/migrate.go — add:

// MigrationVersion returns the filename of the most recently applied
// migration, or "" if no migrations have been applied (empty db or
// schema_migrations table missing). Used by `orca doctor db`.
func MigrationVersion(ctx context.Context, db *sql.DB) (string, error) {
    var name string
    err := db.QueryRowContext(ctx,
        `SELECT name FROM schema_migrations ORDER BY name DESC LIMIT 1`).Scan(&name)
    if err == sql.ErrNoRows {
        return "", nil
    }
    if err != nil {
        return "", fmt.Errorf("query migration version: %w", err)
    }
    return name, nil
}

(Add "context" import — already imported in migrate.go.)

File: internal/doctor/doctor.go — replace DBStub() with DB():

// DB checks SQLite integrity and migration version (REQ-032).
// It opens its own *sql.DB so it can run whether or not the daemon is up.
func DB() Check {
    return Check{
        Name:        "db",
        Description: "SQLite PRAGMA integrity_check + migration version",
        Run: func(ctx context.Context) (Result, string) {
            path := dbPath() // dbPath currently lives in internal/cli; see D-039
            db, err := store.Open(path)
            if err != nil {
                return ResultFail, fmt.Sprintf("open %s: %v", path, err)
            }
            defer db.Close()

            // 1. integrity_check
            var integrity string
            if err := db.QueryRowContext(ctx, "PRAGMA integrity_check").Scan(&integrity); err != nil {
                return ResultFail, fmt.Sprintf("integrity_check query: %v", err)
            }
            if integrity != "ok" {
                first := strings.SplitN(integrity, "\n", 2)[0]
                return ResultFail, fmt.Sprintf("integrity_check: %s", first)
            }

            // 2. migration version
            ver, err := store.MigrationVersion(ctx, db)
            if err != nil {
                return ResultFail, fmt.Sprintf("migration version: %v", err)
            }
            if ver == "" {
                return ResultWarn, "integrity ok; no migrations applied (fresh db?)"
            }
            return ResultPass, fmt.Sprintf("integrity ok; migrations up to %s", ver)
        },
    }
}

D-039 (assumption, confidence 0.78): dbPath() currently lives in internal/cli/node.go and is unexported. The doctor package cannot import internal/cli (would create a cycle: cli imports doctor). Resolution: move dbPath() (and the ORCA_DB env logic) into certpaths (rename the package conceptually, or add a sibling internal/paths package) OR duplicate the ~5-line dbPath function inside internal/doctor. The cleanest is to add func DBPath() string to internal/certpaths/certpaths.go (it already owns Dir() honoring ORCA_HOME) and have both cli and doctor call it. cli.dbPath becomes a thin wrapper or is replaced. This is a small refactor within P02's scope. Logged as D-039, confidence 0.78 (territory overlap between cli-engineer and the doctor package; lead-developer adjudicates).

Implementation approach — network check

File: internal/doctor/doctor.go — replace NetworkStub() with Network():

// Network probes each registered peer's /healthz over mTLS (REQ-032).
// Peers are sourced from the nodes table. Zero peers => WARN (single-node
// install is legitimate). Any peer unreachable => FAIL.
func Network() Check {
    return Check{
        Name:        "network",
        Description: "peer reachability via mTLS /healthz probe",
        Run: func(ctx context.Context) (Result, string) {
            // 1. Load registered nodes (skip 'left').
            path := certpaths.DBPath() // same resolution as D-039
            db, err := store.Open(path)
            if err != nil {
                return ResultFail, fmt.Sprintf("open db for node list: %v", err)
            }
            defer db.Close()
            nodes, err := store.NewNodeRepo(db).List(ctx)
            if err != nil {
                return ResultFail, fmt.Sprintf("list nodes: %v", err)
            }
            // filter out left nodes
            var live []*model.Node
            for _, n := range nodes {
                if n.State != model.NodeStateLeft {
                    live = append(live, n)
                }
            }
            if len(live) == 0 {
                return ResultWarn, "no peers registered; network check skipped (single-node?)"
            }

            caPath := certpaths.CACertPath()
            certPath := certpaths.ServerCertPath()
            keyPath := certpaths.ServerKeyPath()
            // Short per-probe timeout so one slow peer doesn't stall doctor.
            var lines []string
            overall := ResultPass
            for _, n := range live {
                probeCtx, cancel := context.WithTimeout(ctx, 3*time.Second)
                err := probeHealthz(probeCtx, caPath, certPath, keyPath, n.Name, n.Address)
                cancel()
                if err != nil {
                    overall = ResultFail
                    lines = append(lines, fmt.Sprintf("FAIL %s (%s): %v", n.Name, n.Address, err))
                } else {
                    lines = append(lines, fmt.Sprintf("PASS %s (%s)", n.Name, n.Address))
                }
            }
            if overall == ResultPass {
                return ResultPass, fmt.Sprintf("%d/%d peers reachable: %s", len(live), len(live), strings.Join(lines, "; "))
            }
            return ResultFail, strings.Join(lines, "; ")
        },
    }
}

// probeHealthz does a GET https://addr/healthz over mTLS.
func probeHealthz(ctx context.Context, caPath, certPath, keyPath, serverName, addr string) error {
    client, err := transport.NewMTLSClient(caPath, serverName, certPath, keyPath)
    if err != nil {
        return fmt.Errorf("build mTLS client: %w", err)
    }
    req, err := http.NewRequestWithContext(ctx, http.MethodGet, "https://"+addr+"/healthz", nil)
    if err != nil {
        return fmt.Errorf("build request: %w", err)
    }
    resp, err := client.Do(req)
    if err != nil {
        return fmt.Errorf("probe: %w", err)
    }
    defer resp.Body.Close()
    if resp.StatusCode != http.StatusOK {
        return fmt.Errorf("healthz status %d", resp.StatusCode)
    }
    return nil
}

New imports in doctor.go: net/http, strings, time, git.cloudinit.dev/coreci/orca/internal/store, git.cloudinit.dev/coreci/orca/internal/transport, git.cloudinit.dev/coreci/orca/internal/model.

File: internal/doctor/doctor.go — update All():

func All() []Check {
    return []Check{
        CertCA(), CertServer(), CertExpiry(), CertFingerprint(),
        Network(),  // was NetworkStub()
        DB(),       // was DBStub()
    }
}

Keep NetworkStub/DBStub exported functions for one release as thin wrappers that call the new ones? No — delete them; the CLI doctor.go references them and must be updated in lockstep (they are internal). See D-040.

File: internal/cli/doctor.go — update doctorNetworkCmd and doctorDBCmd:

doctorNetworkCmd.RunE: c := doctor.Network()  // was doctor.NetworkStub()
doctorDBCmd.RunE:      c := doctor.DB()       // was doctor.DBStub()

Also: the per-subcommand render should honor jsonOutput (currently it only prints text). Minor enhancement, in scope.

Pitfalls & mitigations (P02)

Pitfall	Mitigation
model.Node has no ServerName/CAPath — mTLS needs ServerName to match the cert SAN.	Derive ServerName = node.Name (D-037). This assumes peer server certs are issued with SAN = node name, which matches GenerateCSR(nodeName, sans). If a deployment uses DNS SANs instead, the probe fails — which is itself a diagnostic. Document this assumption in the check message.
No local client cert/key — doctor can't present a client cert if server.crt/server.key are missing.	The check should FAIL with a clear message if certpaths.ServerCertPath() doesn't exist, BEFORE attempting probes. Reuse os.Stat. This also covers the single-node-never-joined case.
Daemon down — peer's /healthz unreachable.	Per-probe 3s timeout (D-038). Surfaces as FAIL per peer with the dial/handshake error in the message. Doctor is designed to run with daemon up or down, so this is expected behavior, not a crash.
Self-probe — the local node is likely in the nodes table too. Doctor will probe itself over mTLS. This is fine (validates the local daemon's mTLS stack) but requires the local daemon to be running. If the daemon is down, the self-probe fails → FAIL, which is the correct signal.	Document; no special-casing.
DB file doesn't exist — store.Open creates the dir + file + runs migrations (so a missing db becomes a fresh empty db). The db check would then PASS with "no migrations applied" WARN.	This is acceptable: store.Open is idempotent. If the operator expected an existing db, the WARN surfaces the surprise. Could additionally os.Stat the path before Open and WARN if it didn't exist pre-open — optional refinement.
PRAGMA integrity_check can return multiple rows in rare cases (when there are multiple errors). QueryRow only reads the first.	For integrity_check, a single row containing "ok" or the first error is the documented SQLite behavior for the common case. Use QueryRow + Scan; if it's not "ok", that's already a FAIL. Acceptable.
Doctor test TestRunAllChecksWithNoCA asserts WARN from stubs.	Update the test: with real checks, a no-CA scenario yields FAIL on cert.ca (unchanged) AND FAIL on db (open succeeds, integrity ok, but no migrations if fresh — actually WARN) AND WARN on network (no peers). Rewrite assertions to check each check by name rather than "hasWarn globally". See test strategy.
Import cycle: doctor → cli (for dbPath).	Resolved by D-039: move dbPath to certpaths (or a new internal/paths); both cli and doctor import it. No cycle.
store.Open runs migrations on every open — doctor opening the db to run integrity_check would also (re)migrate.	migrate() is idempotent (checks schema_migrations per name). Re-opening is safe; no-op if already migrated. Acceptable.

Test strategy (P02)

internal/store/migrate_test.go (new or appended):

• TestMigrationVersion: open a fresh test db (which runs migrate), call MigrationVersion, assert it returns 0005_node_capacity.sql (the highest current migration). Then manually delete all rows from schema_migrations, assert returns "" with nil error.

internal/doctor/doctor_test.go (update):

• Update TestRunAllChecksWithNoCA: set ORCA_HOME to temp dir (no CA). Expect: cert.ca FAIL, cert.server FAIL, cert.expiry FAIL, cert.fingerprint FAIL, db WARN (fresh db, no migrations — actually store.Open runs migrations, so db will PASS with version 0005; adjust: db PASS), network WARN (no peers). Rewrite to assert per-check rather than "hasWarn/hasFail globally". Remove the stale "expected WARN (stubs)" comment.
• New TestDBCheck_IntegrityOK: open a fresh db via store.Open in temp, run doctor.DB().Run(ctx), expect PASS and message contains "0005".
• New TestDBCheck_Corrupt: open db, manually db.Exec("DROP TABLE jobs") to introduce inconsistency, run integrity_check — but integrity_check mostly detects corruption, not missing tables. More reliable: write garbage to the db file via raw file write, then open — store.Open may fail at Ping. Assert FAIL. (This test is brittle; prefer a unit test on the integrity string-parsing logic with a stub.)
• New TestNetworkCheck_NoPeers: fresh db, no nodes, run doctor.Network().Run(ctx), expect WARN.
• New TestNetworkCheck_PeerReachable: this is an integration test — bootstrap a CA (security.CAInit), generate+sign a server cert with SAN localhost, start an httptest.NewUnstartedServer with ts.TLS = serverTLS and ClientAuth = RequireAndVerifyClientCert (mirror security/integration_test.go:88-108), generate+sign a client cert, insert a node row with Address = ts.Listener.Addr().String() and Name = "localhost", set ORCA_HOME to the temp dir holding the CA + client cert, run doctor.Network().Run(ctx), expect PASS. This reuses the proven pattern from TestEndToEndMTLS.
• New TestNetworkCheck_PeerUnreachable: insert a node with Address = "127.0.0.1:1" (nothing listening), run, expect FAIL with the peer name in the message.

Dependency check (P02)

• net/http, crypto/tls (via transport), strings, time, context — stdlib. ✅
• internal/transport, internal/store, internal/model, internal/certpaths — existing internal packages. ✅
• No new go.mod dependencies required.

---

Cross-cutting decisions

ID	Decision	Confidence
D-039	Move dbPath() (the ORCA_DB-honoring path resolver) from internal/cli to internal/certpaths as DBPath(), to break the would-be doctor→cli import cycle. Both cli and doctor then import certpaths. certpaths already owns the ORCA_HOME-honoring Dir(). Territory: this is a shared infra concern; lead-developer adjudicates.	0.78
D-040	Delete doctor.NetworkStub and doctor.DBStub (no backward-compat shims). They are internal, referenced only by internal/cli/doctor.go which is updated in the same phase. Keeping dead stub code violates no-redundant-implementations.	0.95
D-041	No new go.mod dependencies for v0.3. iter (P01) and mTLS health probe (P02) use stdlib + existing internal packages only. The 4 existing direct deps (cobra, hcl, modernc/sqlite, uuid) are unchanged.	0.97
D-042	Phase ordering: P01 (iter.Seq) and P02 (doctor) are independent — no file is modified by both (P01 touches cli/job.go, cli/node.go, store repos; P02 touches doctor.go, cli/doctor.go, store/migrate.go, certpaths). They can be developed in either order or in parallel. Recommend P01 first only because it's the lower-risk change.	0.85

---

Summary of assumptions logged

All assumptions below are logged as decisions with confidence scores; none are flagged for human validation (full autonomy). Low-confidence (<0.80) items that warrant normal decision-flow attention:

• D-030 (0.72): --watch --json emits one JSON object per changed element per tick (vs. one array per tick). REQ-030 wording is ambiguous; this interpretation matches "streaming one-line JSON per event" literally.
• D-037 (0.80): peer ServerName = node Name (SAN convention).
• D-039 (0.78): dbPath relocation to certpaths — territory overlap.

These three are escalated through the normal decision flow (DecisionEngine) per the researcher protocol, NOT flagged for human validation.