# Research: Orca v0.3 — scheduling-streaming

Phase: 0 (research) for milestone v0.3 (scheduling-streaming).
Branch: `phase/00-pre-execution` (cut from `milestone/v0.3-scheduling-streaming`).
Go toolchain: `go1.25.0` (confirmed via `go version`; `go.mod` declares `go 1.25.0`).

This document provides concrete, file-level implementation guidance for the
two v0.3 execution phases:

- **P01** — `iter.Seq` streaming for `--watch` flags (REQ-022, REQ-030)
- **P02** — `orca doctor` network + db full implementation (REQ-032 completion)

All assumptions are logged as decisions (D-025..D-038) with confidence scores.
Full autonomy mode — no items flagged for human validation.

---

## Codebase Audit Summary

### Current state (commit ba5ffd7 + phase docs)

| Area | File | Key finding |
|------|------|-------------|
| CLI `job list` | `internal/cli/job.go:112-143` | `jobListCmd.RunE` calls `store.NewJobRepo(db).List(ctx)`, prints a fixed-width table; `--json` via `printJSON(jobs)`. No `--watch` flag exists. |
| CLI `node list` | `internal/cli/node.go:155-186` | `nodeListCmd.RunE` calls `registry.List(ctx)` → `repo.List(ctx)`. Table + `--json`. No `--watch` flag. |
| CLI root | `internal/cli/root.go` | `jsonOutput` is a package-level `bool` set by `--json` persistent flag. `printJSON` uses `json.NewEncoder` with 2-space indent. |
| Job repo | `internal/store/job_task_repo.go` | `JobRepo.List(ctx) ([]*model.Job, error)` — single-shot query, closes rows. `scanJob` helper is reusable. |
| Node repo | `internal/store/node_repo.go` | `NodeRepo.List(ctx) ([]*model.Node, error)`. `scanNode` helper is reusable. `scanner` interface defined here (`Scan(dest ...any) error`) — shared by `*sql.Row` and `*sql.Rows`. |
| Store open | `internal/store/store.go` | `store.Open(path)` opens with `?_pragma=journal_mode(WAL)&_pragma=foreign_keys(ON)` and runs `migrate(db)`. |
| Migrations | `internal/store/migrate.go` | `migrate` is unexported, runs at `Open` time. `schema_migrations` table tracks applied migrations by filename. Migrations are embedded via `//go:embed migrations/*.sql`. No public API to query migration version. |
| Migrations on disk | `internal/store/migrations/` | `0001_nodes.sql`, `0002_jobs_tasks.sql`, `0003_audit_log.sql`, `0004_certs.sql`, `0005_node_capacity.sql`. Highest = 0005. |
| Doctor | `internal/doctor/doctor.go` | `NetworkStub()` and `DBStub()` return WARN stubs. `All()` aggregates 6 checks. `Run(ctx)` iterates checks. `Check.Run` signature: `func(ctx context.Context) (Result, string)`. `Result` is `PASS|WARN|FAIL`. No DB or transport imports — cert-only. |
| Doctor CLI | `internal/cli/doctor.go` | `doctorNetworkCmd`/`doctorDBCmd` call `doctor.NetworkStub()`/`doctor.DBStub()` directly. |
| Doctor tests | `internal/doctor/doctor_test.go` | Two tests: `TestRunAllChecksWithNoCA` (expects FAIL + WARN for stubs), `TestRunWithCAAndServerCert` (cert checks PASS). Uses `t.Setenv("ORCA_HOME", dir)`. **The "expects WARN" assertion will break when stubs become real checks** — must be updated in P02. |
| Transport mTLS client | `internal/transport/mtls.go` | `NewMTLSClient(caPath, serverName, certPath, keyPath)` builds an `http.Client` with a TLS-1.3-only config from `security.ClientTLSConfig`. `MTLSClient.Do(req)`. `DialContext` for low-level TLS dial. |
| Transport dispatch client | `internal/transport/dispatch.go` | `NewDispatchClient(caPath, serverName, peerAddr)` wraps `MTLSClient`. `PeerAddr` is `http://` or `https://`. `Submit`/`Status` POST to `/orca.v1.Dispatch/*`. No `/healthz` GET helper. |
| Daemon health | `internal/daemon/health.go` | `handleHealthz` → 200 `{"status":"alive"}`. `handleReadyz` → 200/503 with db ping. Mounted at `mux.HandleFunc("/healthz", ...)` in `server.go:104`. |
| Daemon TLS | `internal/daemon/tls.go` | `StartMTLS(state)` sets `httpServer.TLSConfig` with `ClientAuth = RequireAndVerifyClientCert`. The daemon **requires client certs** in mTLS mode. |
| Peer registry | `internal/engine/peer.go` | `PeerRegistry` is **in-memory only** (`map[string]*Peer` under `sync.RWMutex`). `NewPeerRegistry()` returns empty. `Peer` has `NodeID, Address, ServerName, CAPath, LastSeen, Capacity`. **Not persisted to SQLite.** |
| Peer registry usage | `internal/cli/job.go:70`, `internal/cli/daemon.go:47` | Both create a **fresh empty** `NewPeerRegistry()` per process. No code ever calls `peers.Add(...)`. The registry is currently a structural placeholder. |
| Node registry | `internal/engine/registry.go` | `NodeRegistry` wraps `store.NodeRepo` + `Audit`. `List(ctx)` → `repo.List(ctx)`. Persisted to `nodes` table. |
| Node model | `internal/model/node.go` | `Node{ID, Name, Address, State, JoinedAt, LastSeen, Metadata}`. **No `ServerName` or `CAPath` field** — `model.Node` differs from `engine.Peer`. |
| Cert paths | `internal/certpaths/certpaths.go` | `Dir()` honors `ORCA_HOME`; `CACertPath()`, `ServerCertPath()`, `ServerKeyPath()`. |
| Security client TLS | `internal/security/tls_config.go:106` | `ClientTLSConfig(caPath, serverName, certPath, keyPath)` — TLS 1.3 only, AEAD allowlist, `RootCAs` = single CA. Both-or-neither for cert/key. |
| Go version | `go.mod` + `go version` | `go 1.25.0` — `iter` package and range-over-func are stable stdlib. |
| Deps | `go.mod` | cobra, hcl/v2, modernc/sqlite, uuid. **No new deps needed for v0.3.** `iter` is stdlib. |

### Critical gap analysis

1. **`PeerRegistry` is non-persistent and always empty at CLI time.** The
   doctor network check cannot rely on it — there is no code path that populates
   it. The `nodes` table IS persisted and has `Address`, but lacks the
   `ServerName`/`CAPath` needed for an mTLS probe. **Resolution: doctor network
   reads the `nodes` table via `NodeRepo.List`, and derives `ServerName` +
   `CAPath` from local config (`certpaths.CACertPath()` + node name/addr).**
   See D-029.

2. **`doctor.Run` / `Check.Run` do not plumb a `*sql.DB` or transport client.**
   The cert checks are filesystem-only. P02 must extend the check constructors
   to accept a DB handle and (for network) a transport client factory. The
   `Check.Run` signature (`func(ctx) (Result, string)`) is preserved by
   closure-capturing the handles in the constructor. See D-027, D-031.

3. **No public migration-version query.** `migrate()` is unexported and writes
   to `schema_migrations(name, applied_at)`. P02 adds a public
   `store.MigrationVersion(ctx, db)` (or method on a repo) that selects the max
   applied migration name. See D-033.

4. **Doctor test `TestRunAllChecksWithNoCA` asserts a WARN from stubs.** This
   will break when stubs become real (the db check will PASS with a fresh test
   DB, and the network check will WARN/FAIL on zero peers). Must be updated.
   See D-036.

---

## P01: iter.Seq Streaming for `--watch` Flags

Covers REQ-022 (`iter.Seq` for streaming job lists), REQ-030 (`--watch` output
format: table default vs streaming one-line JSON per event).

### Decisions

| ID | Decision | Confidence |
|----|----------|------------|
| **D-025** | `iter.Seq` lives on the store repos, not the engine registry. `JobRepo.Watch(ctx) iter.Seq[*model.Job]` and `NodeRepo.Watch(ctx) iter.Seq[*model.Node]`. Rationale: repos already own the `*sql.DB` and the `scanJob`/`scanNode` helpers; engine.Registry.List just delegates to repo. Keeping Watch in the store layer matches the data-engineer territory and avoids a new engine→store iter dependency. | 0.90 |
| **D-026** | Element type is `*model.Job` / `*model.Node` (pointer), matching the existing `[]*model.Job` return of `List`. This keeps `printJSON` and table rendering identical between one-shot and watch paths. | 0.88 |
| **D-027** | The `Check.Run` signature in `doctor` is unchanged; P02 captures DB/transport handles in closure at constructor time (`Network(db)`, `DB(db)`). This is the established pattern (cert checks already closure-capture `certpaths`). | 0.92 |
| **D-028** | Watch refresh = poll-based 1s ticker (per D-019). No event channel, no daemon coupling. Each tick re-runs the existing `List` query and yields the **full current snapshot** (one element per row). The CLI dedupes by detecting snapshot equality before re-rendering (see D-030). Rationale: simpler than NOTIFY/LISTEN, no daemon dependency, matches offline-first. | 0.90 |
| **D-029** | `--watch` output: default = re-print the table on every changed snapshot (clear screen via ANSI `\033[2J\033[H` then table); `--watch --json` = one compact JSON line **per snapshot** (an array on each line, OR one line per element — see D-030). The CLI tracks the previous snapshot's hash to avoid spamming identical frames. | 0.85 |
| **D-030** | `--watch --json` emits **one JSON object per element per tick where the element changed**, i.e. streaming one-line JSON per event (per REQ-030 wording). Implementation: on each tick, for each element, if its JSON bytes differ from the previous snapshot's bytes for that ID, print `{"event":"update","job":{...}}\n`. On first tick, print all as `{"event":"init",...}`. This is the most useful for AI agents tailing the stream. Confidence lower because REQ-030 is ambiguous between "array per tick" and "object per event"; the per-event interpretation matches "streaming one-line JSON per event" literally. | 0.72 |
| **D-031** | Cancellation: `signal.NotifyContext(ctx, os.Interrupt, syscall.SIGTERM)` at the CLI command layer, **replacing** the current `context.WithTimeout(cmd.Context(), 5*time.Second)` for the watch path only. The non-watch `list` path keeps its 5s timeout. The `iter.Seq` receives this ctx and stops yielding on `ctx.Done()`. | 0.93 |
| **D-032** | The `iter.Seq` implementation **must not leak goroutines**: the polling loop runs **inline in the yield callback's caller goroutine** (the `range` loop), not a separate goroutine. `for range seq { ... }` drives the pull; inside `Watch`, we loop `for { select { <-ticker.C: query+yield each; <-ctx.Done(): return } }` and call `yield(item)` directly. When `yield` returns false (consumer broke the loop), we stop and return. **No goroutine is spawned by Watch.** This is the cleanest Go 1.25 iter pattern and avoids leak surface entirely. | 0.95 |

### Implementation approach — store layer

**File: `internal/store/job_task_repo.go`** — add method:

```go
// Watch yields the current snapshot of jobs on a 1-second ticker until
// ctx is cancelled or the consumer stops pulling (yield returns false).
// It does not spawn a goroutine; the polling loop runs in the caller's
// goroutine via the range-over-func pull protocol.
//
// Each tick re-runs the List query and yields one *model.Job per row.
// The caller is responsible for deduping across ticks if desired.
func (r *JobRepo) Watch(ctx context.Context) iter.Seq[*model.Job] {
    return func(yield func(*model.Job) bool) {
        ticker := time.NewTicker(1 * time.Second)
        defer ticker.Stop()
        for {
            select {
            case <-ctx.Done():
                return
            case <-ticker.C:
            }
            // Reuse the existing List query + scanJob helper.
            rows, err := r.db.QueryContext(ctx,
                `SELECT id, name, spec, status, exit_code, created_at, started_at, ended_at
                 FROM jobs ORDER BY created_at DESC`)
            if err != nil {
                // Surfacing errors from inside iter.Seq is awkward; the
                // CLI layer cannot receive a returned error. Log via slog
                // (the repo doesn't hold a logger today — see D-034) and
                // continue to next tick rather than terminating the
                // stream. A transient DB blip should not kill the watch.
                continue
            }
            for rows.Next() {
                j, err := scanJob(rows)
                if err != nil {
                    rows.Close()
                    return
                }
                if !yield(j) {
                    rows.Close()
                    return // consumer stopped
                }
            }
            rows.Close()
        }
    }
}
```

**File: `internal/store/node_repo.go`** — add analogous `Watch`:

```go
func (r *NodeRepo) Watch(ctx context.Context) iter.Seq[*model.Node] {
    return func(yield func(*model.Node) bool) {
        ticker := time.NewTicker(1 * time.Second)
        defer ticker.Stop()
        for {
            select {
            case <-ctx.Done():
                return
            case <-ticker.C:
            }
            rows, err := r.db.QueryContext(ctx,
                `SELECT id, name, address, state, joined_at, last_seen, metadata
                 FROM nodes ORDER BY joined_at ASC`)
            if err != nil {
                continue
            }
            for rows.Next() {
                n, err := scanNode(rows)
                if err != nil {
                    rows.Close()
                    return
                }
                if !yield(n) {
                    rows.Close()
                    return
                }
            }
            rows.Close()
        }
    }
}
```

Imports: add `"iter"` and `"time"` (time already present in both files). The
`iter` package is imported only for the return type; `yield` is the callback.

**Note on the existing `scanner` interface:** `scanJob`/`scanNode` accept the
`scanner` interface (`Scan(dest ...any) error`) satisfied by both `*sql.Row`
and `*sql.Rows`, so they are directly reusable in `Watch` — no refactor needed.

### Implementation approach — CLI layer

**File: `internal/cli/job.go`** — modify `jobListCmd`:

1. Add a package var `jobWatch bool` and register `jobListCmd.Flags().BoolVar(&jobWatch, "watch", false, "stream jobs until Ctrl-C")`.
2. In `RunE`, branch on `jobWatch`:
   - If `!jobWatch`: keep the existing 5s-timeout `List` path.
   - If `jobWatch`:
     - `ctx, cancel := signal.NotifyContext(cmd.Context(), os.Interrupt, syscall.SIGTERM); defer cancel()` (drop the 5s timeout).
     - `seq := store.NewJobRepo(db).Watch(ctx)`
     - If `jsonOutput`: stream one-line JSON per event (D-030). Maintain a `map[string][]byte` of last-seen JSON per job ID. On each yielded job, marshal compact JSON; if it differs from the stored bytes (or ID unseen), print `{"event":"update","job":{...}}\n` and update the map.
     - Else (table): on each tick, after collecting the full snapshot, compare against the previous snapshot (by hashing the rendered table string or by comparing the slice of `[]*model.Job` via reflect/cmp). If changed, emit `"\033[2J\033[H"` (clear) then the table header + rows. This gives a "top-like" refresh.

   Because `iter.Seq` does not return an error, the watch path swallows
   per-tick query errors inside `Watch` (D-034). The CLI relies on `ctx.Done()`
   for termination.

**File: `internal/cli/node.go`** — analogous change to `nodeListCmd`:
- Add `nodeWatch bool`, register `--watch` flag.
- Branch in `RunE`; `seq := store.NewNodeRepo(db).Watch(ctx)` (open a fresh `openDB()` for the watch path; `registry.List` is not used for watch — go straight to the repo to get the iter).

**Note:** `nodeListCmd` currently goes through `nodeRegistry()` which wraps
`NodeRepo` in `engine.NodeRegistry`. For watch, bypass the registry and use
`store.NewNodeRepo(db).Watch(ctx)` directly — the registry adds no value for a
read-only stream and would require a `Watch` passthrough method. This keeps the
iter boundary clean in the store layer (D-025).

### Pitfalls & mitigations (P01)

| Pitfall | Mitigation |
|---------|------------|
| **Goroutine leak** if Watch spawned a goroutine. | It doesn't — the polling loop is inline in the pull callback (D-032). `defer ticker.Stop()` + `rows.Close()` on every exit path. |
| **Rows cursor held open across yield** — if `yield` blocks (e.g. slow consumer), the `*sql.Rows` stays open and holds a SQLite read lock. | Yield is called per-row inside the `rows.Next()` loop; the consumer (`range`) is fast (prints to stdout). For safety, close rows immediately after the loop or on `yield==false`. The 1s tick cadence bounds how long a cursor is held. WAL mode (set in `store.Open`) allows concurrent reads, so this does not block writers. |
| **No error channel from iter.Seq** — a transient DB error is invisible to the CLI. | Log inside Watch via a package-level slog default (`slog.Default().Warn(...)`) since the repo has no logger field today (D-034: add an optional `logger *slog.Logger` to JobRepo/NodeRepo, defaulting to `slog.Default()` in the constructors — minimal change). Continue to next tick rather than terminating. |
| **ctx cancellation mid-query** — `QueryContext` returns an error; `rows.Next()` returns false. | Handled: the `select` on `ctx.Done()` returns before the next tick; an in-flight query is cancelled by the ctx. |
| **Rapid re-render flicker** in table mode. | Clear-screen + full re-render on changed snapshot only (hash compare). Unchanged snapshots produce no output. |
| **`--watch` + `--json` interleaving with slog stderr.** | slog writes to stderr; CLI output to stdout — no interleaving on stdout. Safe. |
| **Test determinism** — ticker is 1s, tests would be slow/flaky. | Provide a test-only constructor `WatchWithInterval(ctx, d time.Duration)` OR make the interval a field on the repo set via an unexported option. Preferred: an unexported `watchInterval` package var defaulting to 1s, overridable from `internal/store` tests. See D-035. |
| **Signal handling clobbers root signal handler.** | `signal.NotifyContext` with `os.Interrupt` returns a fresh ctx; the root `cobra.Command` does not install its own SIGINT handler, so no conflict. `defer cancel()` restores default behavior on exit. |

### Test strategy (P01)

**`internal/store/job_task_repo_test.go` (new file or appended):**
- `TestJobRepoWatch_YieldsSnapshots`: insert 1 job, call `Watch` with a 10ms interval (via test hook), range over `seq` collecting into a slice, insert a 2nd job from a goroutine after 30ms, cancel ctx after 80ms, assert the 2nd job appeared in the collected slice. Use `context.WithTimeout` for cancellation.
- `TestJobRepoWatch_StopsOnConsumerBreak`: range over `seq` and `break` after the first yield; assert the function returns (no hang) within a short deadline. This validates the `yield==false` path.
- `TestJobRepoWatch_StopsOnCtxCancel`: cancel ctx; assert the range loop exits within 50ms.
- `TestNodeRepoWatch_*`: mirror the above for nodes.

**`internal/cli/job_test.go` (new) / `node_test.go` (new) — if CLI tests exist; otherwise add:**
- `TestJobListWatch_JSONStreaming`: spin a temp DB, insert a job, invoke the `jobListCmd.RunE` with `--watch --json` in a goroutine, insert a 2nd job, capture stdout for ~200ms, assert two JSON lines appear. Cancel via ctx.
- `TestJobListWatch_TableRefresh`: assert clear-screen escape + table re-render on change.
- These CLI tests are harder to make deterministic; prefer testing the store-layer Watch thoroughly and keep CLI watch tests to a smoke-level "produces output, exits on ctx.Done".

### Dependency check (P01)

- `iter` — Go 1.25 stdlib (`go.mod` declares `go 1.25.0`). ✅ no new dep.
- `time`, `context`, `os/signal`, `syscall`, `encoding/json` — stdlib. ✅
- No new go.mod dependencies required.

---

## P02: `orca doctor` network + db full implementation

Covers REQ-032 completion (network + db checks replacing `NetworkStub`/`DBStub`).

### Decisions

| ID | Decision | Confidence |
|----|----------|------------|
| **D-033** | Add a public `store.MigrationVersion(ctx, db) (string, error)` function (in `migrate.go` or a new `internal/store/migrate_query.go`) that returns the **highest applied migration filename** from `schema_migrations`. SQL: `SELECT name FROM schema_migrations ORDER BY name DESC LIMIT 1`. This is the source of truth for "migration version" — it reflects what `migrate()` actually applied. Returns `("", nil)` if no migrations applied (fresh empty table) and `("", sql.ErrNoRows)` is treated as empty. | 0.90 |
| **D-034** | The doctor DB check opens its own `*sql.DB` via `store.Open(dbPath())` (reusing the CLI's `dbPath`) inside the check constructor closure, rather than receiving a shared handle. Rationale: doctor should be runnable whether the daemon is up or down; `store.Open` uses WAL so a concurrent daemon is fine. The check `defer db.Close()`. This avoids threading a `*sql.DB` through `doctor.Run`/`All()` and keeps the `Check.Run` signature stable. | 0.86 |
| **D-035** | The doctor DB check runs `PRAGMA integrity_check` via `db.QueryRow("PRAGMA integrity_check")`. SQLite returns a single row with a TEXT value: `"ok"` on success, or a multi-line error description on failure. PASS if the value is `"ok"`; FAIL otherwise (with the first line of the message). Plus query the migration version (D-033); WARN if `schema_migrations` is empty (fresh/never-migrated db) — this is suspicious but not corrupt. | 0.92 |
| **D-036** | Doctor network check sources peer addresses from the **`nodes` table** (`NodeRepo.List`), NOT from `engine.PeerRegistry` (which is in-memory and always empty at CLI time — see gap #1). For each node with `state != 'left'`, probe `https://<address>/healthz` over mTLS. | 0.88 |
| **D-037** | For each peer, the network check builds an mTLS client via `transport.NewMTLSClient(certpaths.CACertPath(), serverName, certpaths.ServerCertPath(), certpaths.ServerKeyPath())`. `serverName` is derived as the node's `Name` (the SAN on a peer's server cert is its node name, per `security.GenerateCSR(nodeName, sans)` — confirmed in `integration_test.go:41` `GenerateCSR("test-server", ...)`. If the SAN uses a different value the probe will fail handshake, which is itself a useful diagnostic). `CAPath` is the local `ca.crt` (all peers share one CA per D-011). This presents the local node's client cert, satisfying the daemon's `RequireAndVerifyClientCert`. | 0.80 |
| **D-038** | Network check result semantics: **zero peers registered** → `WARN` ("no peers registered; network check skipped") — not FAIL, because a single-node install legitimately has no peers. **A peer unreachable / handshake failed** → `FAIL` for that peer, aggregated to a single `network` check result that is FAIL if any peer failed, PASS if all peers probed OK, WARN if zero peers. Each peer's per-line outcome is folded into the message string (e.g. `PASS — 2/2 peers reachable; FAIL — peer node-b (host:port): tls handshake error`). | 0.85 |

### Implementation approach — db check

**File: `internal/store/migrate.go`** — add:

```go
// MigrationVersion returns the filename of the most recently applied
// migration, or "" if no migrations have been applied (empty db or
// schema_migrations table missing). Used by `orca doctor db`.
func MigrationVersion(ctx context.Context, db *sql.DB) (string, error) {
    var name string
    err := db.QueryRowContext(ctx,
        `SELECT name FROM schema_migrations ORDER BY name DESC LIMIT 1`).Scan(&name)
    if err == sql.ErrNoRows {
        return "", nil
    }
    if err != nil {
        return "", fmt.Errorf("query migration version: %w", err)
    }
    return name, nil
}
```

(Add `"context"` import — already imported in migrate.go.)

**File: `internal/doctor/doctor.go`** — replace `DBStub()` with `DB()`:

```go
// DB checks SQLite integrity and migration version (REQ-032).
// It opens its own *sql.DB so it can run whether or not the daemon is up.
func DB() Check {
    return Check{
        Name:        "db",
        Description: "SQLite PRAGMA integrity_check + migration version",
        Run: func(ctx context.Context) (Result, string) {
            path := dbPath() // dbPath currently lives in internal/cli; see D-039
            db, err := store.Open(path)
            if err != nil {
                return ResultFail, fmt.Sprintf("open %s: %v", path, err)
            }
            defer db.Close()

            // 1. integrity_check
            var integrity string
            if err := db.QueryRowContext(ctx, "PRAGMA integrity_check").Scan(&integrity); err != nil {
                return ResultFail, fmt.Sprintf("integrity_check query: %v", err)
            }
            if integrity != "ok" {
                first := strings.SplitN(integrity, "\n", 2)[0]
                return ResultFail, fmt.Sprintf("integrity_check: %s", first)
            }

            // 2. migration version
            ver, err := store.MigrationVersion(ctx, db)
            if err != nil {
                return ResultFail, fmt.Sprintf("migration version: %v", err)
            }
            if ver == "" {
                return ResultWarn, "integrity ok; no migrations applied (fresh db?)"
            }
            return ResultPass, fmt.Sprintf("integrity ok; migrations up to %s", ver)
        },
    }
}
```

**D-039 (assumption, confidence 0.78):** `dbPath()` currently lives in
`internal/cli/node.go` and is unexported. The `doctor` package cannot import
`internal/cli` (would create a cycle: `cli` imports `doctor`). **Resolution:**
move `dbPath()` (and the `ORCA_DB` env logic) into `certpaths` (rename the
package conceptually, or add a sibling `internal/paths` package) OR duplicate
the ~5-line `dbPath` function inside `internal/doctor`. The cleanest is to add
`func DBPath() string` to `internal/certpaths/certpaths.go` (it already owns
`Dir()` honoring `ORCA_HOME`) and have both `cli` and `doctor` call it.
`cli.dbPath` becomes a thin wrapper or is replaced. This is a small refactor
within P02's scope. Logged as D-039, confidence 0.78 (territory overlap between
cli-engineer and the doctor package; lead-developer adjudicates).

### Implementation approach — network check

**File: `internal/doctor/doctor.go`** — replace `NetworkStub()` with `Network()`:

```go
// Network probes each registered peer's /healthz over mTLS (REQ-032).
// Peers are sourced from the nodes table. Zero peers => WARN (single-node
// install is legitimate). Any peer unreachable => FAIL.
func Network() Check {
    return Check{
        Name:        "network",
        Description: "peer reachability via mTLS /healthz probe",
        Run: func(ctx context.Context) (Result, string) {
            // 1. Load registered nodes (skip 'left').
            path := certpaths.DBPath() // same resolution as D-039
            db, err := store.Open(path)
            if err != nil {
                return ResultFail, fmt.Sprintf("open db for node list: %v", err)
            }
            defer db.Close()
            nodes, err := store.NewNodeRepo(db).List(ctx)
            if err != nil {
                return ResultFail, fmt.Sprintf("list nodes: %v", err)
            }
            // filter out left nodes
            var live []*model.Node
            for _, n := range nodes {
                if n.State != model.NodeStateLeft {
                    live = append(live, n)
                }
            }
            if len(live) == 0 {
                return ResultWarn, "no peers registered; network check skipped (single-node?)"
            }

            caPath := certpaths.CACertPath()
            certPath := certpaths.ServerCertPath()
            keyPath := certpaths.ServerKeyPath()
            // Short per-probe timeout so one slow peer doesn't stall doctor.
            var lines []string
            overall := ResultPass
            for _, n := range live {
                probeCtx, cancel := context.WithTimeout(ctx, 3*time.Second)
                err := probeHealthz(probeCtx, caPath, certPath, keyPath, n.Name, n.Address)
                cancel()
                if err != nil {
                    overall = ResultFail
                    lines = append(lines, fmt.Sprintf("FAIL %s (%s): %v", n.Name, n.Address, err))
                } else {
                    lines = append(lines, fmt.Sprintf("PASS %s (%s)", n.Name, n.Address))
                }
            }
            if overall == ResultPass {
                return ResultPass, fmt.Sprintf("%d/%d peers reachable: %s", len(live), len(live), strings.Join(lines, "; "))
            }
            return ResultFail, strings.Join(lines, "; ")
        },
    }
}

// probeHealthz does a GET https://addr/healthz over mTLS.
func probeHealthz(ctx context.Context, caPath, certPath, keyPath, serverName, addr string) error {
    client, err := transport.NewMTLSClient(caPath, serverName, certPath, keyPath)
    if err != nil {
        return fmt.Errorf("build mTLS client: %w", err)
    }
    req, err := http.NewRequestWithContext(ctx, http.MethodGet, "https://"+addr+"/healthz", nil)
    if err != nil {
        return fmt.Errorf("build request: %w", err)
    }
    resp, err := client.Do(req)
    if err != nil {
        return fmt.Errorf("probe: %w", err)
    }
    defer resp.Body.Close()
    if resp.StatusCode != http.StatusOK {
        return fmt.Errorf("healthz status %d", resp.StatusCode)
    }
    return nil
}
```

New imports in `doctor.go`: `net/http`, `strings`, `time`, `git.cloudinit.dev/coreci/orca/internal/store`, `git.cloudinit.dev/coreci/orca/internal/transport`, `git.cloudinit.dev/coreci/orca/internal/model`.

**File: `internal/doctor/doctor.go`** — update `All()`:
```go
func All() []Check {
    return []Check{
        CertCA(), CertServer(), CertExpiry(), CertFingerprint(),
        Network(),  // was NetworkStub()
        DB(),       // was DBStub()
    }
}
```
Keep `NetworkStub`/`DBStub` exported functions for one release as thin
wrappers that call the new ones? **No** — delete them; the CLI doctor.go
references them and must be updated in lockstep (they are internal). See D-040.

**File: `internal/cli/doctor.go`** — update `doctorNetworkCmd` and `doctorDBCmd`:
```go
doctorNetworkCmd.RunE: c := doctor.Network()  // was doctor.NetworkStub()
doctorDBCmd.RunE:      c := doctor.DB()       // was doctor.DBStub()
```
Also: the per-subcommand render should honor `jsonOutput` (currently it only
prints text). Minor enhancement, in scope.

### Pitfalls & mitigations (P02)

| Pitfall | Mitigation |
|---------|------------|
| **`model.Node` has no `ServerName`/`CAPath`** — mTLS needs `ServerName` to match the cert SAN. | Derive `ServerName = node.Name` (D-037). This assumes peer server certs are issued with SAN = node name, which matches `GenerateCSR(nodeName, sans)`. If a deployment uses DNS SANs instead, the probe fails — which is itself a diagnostic. Document this assumption in the check message. |
| **No local client cert/key** — doctor can't present a client cert if `server.crt`/`server.key` are missing. | The check should FAIL with a clear message if `certpaths.ServerCertPath()` doesn't exist, BEFORE attempting probes. Reuse `os.Stat`. This also covers the single-node-never-joined case. |
| **Daemon down** — peer's `/healthz` unreachable. | Per-probe 3s timeout (D-038). Surfaces as FAIL per peer with the dial/handshake error in the message. Doctor is designed to run with daemon up or down, so this is expected behavior, not a crash. |
| **Self-probe** — the local node is likely in the `nodes` table too. Doctor will probe itself over mTLS. This is fine (validates the local daemon's mTLS stack) but requires the local daemon to be running. If the daemon is down, the self-probe fails → FAIL, which is the correct signal. | Document; no special-casing. |
| **DB file doesn't exist** — `store.Open` creates the dir + file + runs migrations (so a missing db becomes a fresh empty db). The db check would then PASS with "no migrations applied" WARN. | This is acceptable: `store.Open` is idempotent. If the operator expected an existing db, the WARN surfaces the surprise. Could additionally `os.Stat` the path before `Open` and WARN if it didn't exist pre-open — optional refinement. |
| **`PRAGMA integrity_check` can return multiple rows** in rare cases (when there are multiple errors). `QueryRow` only reads the first. | For `integrity_check`, a single row containing `"ok"` or the first error is the documented SQLite behavior for the common case. Use `QueryRow` + `Scan`; if it's not `"ok"`, that's already a FAIL. Acceptable. |
| **Doctor test `TestRunAllChecksWithNoCA`** asserts WARN from stubs. | Update the test: with real checks, a no-CA scenario yields FAIL on cert.ca (unchanged) AND FAIL on db (open succeeds, integrity ok, but no migrations if fresh — actually WARN) AND WARN on network (no peers). Rewrite assertions to check each check by name rather than "hasWarn globally". See test strategy. |
| **Import cycle:** `doctor` → `cli` (for `dbPath`). | Resolved by D-039: move `dbPath` to `certpaths` (or a new `internal/paths`); both `cli` and `doctor` import it. No cycle. |
| **`store.Open` runs migrations on every open** — doctor opening the db to run integrity_check would also (re)migrate. | `migrate()` is idempotent (checks `schema_migrations` per name). Re-opening is safe; no-op if already migrated. Acceptable. |

### Test strategy (P02)

**`internal/store/migrate_test.go` (new or appended):**
- `TestMigrationVersion`: open a fresh test db (which runs migrate), call `MigrationVersion`, assert it returns `0005_node_capacity.sql` (the highest current migration). Then manually delete all rows from `schema_migrations`, assert returns `""` with nil error.

**`internal/doctor/doctor_test.go` (update):**
- Update `TestRunAllChecksWithNoCA`: set `ORCA_HOME` to temp dir (no CA). Expect: cert.ca FAIL, cert.server FAIL, cert.expiry FAIL, cert.fingerprint FAIL, **db WARN** (fresh db, no migrations — actually `store.Open` runs migrations, so db will PASS with version 0005; adjust: db PASS), **network WARN** (no peers). Rewrite to assert per-check rather than "hasWarn/hasFail globally". Remove the stale "expected WARN (stubs)" comment.
- New `TestDBCheck_IntegrityOK`: open a fresh db via `store.Open` in temp, run `doctor.DB().Run(ctx)`, expect PASS and message contains "0005".
- New `TestDBCheck_Corrupt`: open db, manually `db.Exec("DROP TABLE jobs")` to introduce inconsistency, run integrity_check — but `integrity_check` mostly detects corruption, not missing tables. More reliable: write garbage to the db file via raw file write, then open — `store.Open` may fail at Ping. Assert FAIL. (This test is brittle; prefer a unit test on the integrity string-parsing logic with a stub.)
- New `TestNetworkCheck_NoPeers`: fresh db, no nodes, run `doctor.Network().Run(ctx)`, expect WARN.
- New `TestNetworkCheck_PeerReachable`: this is an integration test — bootstrap a CA (`security.CAInit`), generate+sign a server cert with SAN `localhost`, start an `httptest.NewUnstartedServer` with `ts.TLS = serverTLS` and `ClientAuth = RequireAndVerifyClientCert` (mirror `security/integration_test.go:88-108`), generate+sign a client cert, insert a node row with `Address = ts.Listener.Addr().String()` and `Name = "localhost"`, set `ORCA_HOME` to the temp dir holding the CA + client cert, run `doctor.Network().Run(ctx)`, expect PASS. This reuses the proven pattern from `TestEndToEndMTLS`.
- New `TestNetworkCheck_PeerUnreachable`: insert a node with `Address = "127.0.0.1:1"` (nothing listening), run, expect FAIL with the peer name in the message.

### Dependency check (P02)

- `net/http`, `crypto/tls` (via transport), `strings`, `time`, `context` — stdlib. ✅
- `internal/transport`, `internal/store`, `internal/model`, `internal/certpaths` — existing internal packages. ✅
- No new go.mod dependencies required.

---

## Cross-cutting decisions

| ID | Decision | Confidence |
|----|----------|------------|
| **D-039** | Move `dbPath()` (the `ORCA_DB`-honoring path resolver) from `internal/cli` to `internal/certpaths` as `DBPath()`, to break the would-be `doctor→cli` import cycle. Both `cli` and `doctor` then import `certpaths`. `certpaths` already owns the `ORCA_HOME`-honoring `Dir()`. Territory: this is a shared infra concern; `lead-developer` adjudicates. | 0.78 |
| **D-040** | Delete `doctor.NetworkStub` and `doctor.DBStub` (no backward-compat shims). They are internal, referenced only by `internal/cli/doctor.go` which is updated in the same phase. Keeping dead stub code violates `no-redundant-implementations`. | 0.95 |
| **D-041** | No new go.mod dependencies for v0.3. `iter` (P01) and mTLS health probe (P02) use stdlib + existing internal packages only. The 4 existing direct deps (cobra, hcl, modernc/sqlite, uuid) are unchanged. | 0.97 |
| **D-042** | Phase ordering: P01 (iter.Seq) and P02 (doctor) are **independent** — no file is modified by both (P01 touches cli/job.go, cli/node.go, store repos; P02 touches doctor.go, cli/doctor.go, store/migrate.go, certpaths). They can be developed in either order or in parallel. Recommend P01 first only because it's the lower-risk change. | 0.85 |

---

## Summary of assumptions logged

All assumptions below are logged as decisions with confidence scores; none are
flagged for human validation (full autonomy). Low-confidence (<0.80) items that
warrant normal decision-flow attention:

- **D-030** (0.72): `--watch --json` emits one JSON object per changed element
  per tick (vs. one array per tick). REQ-030 wording is ambiguous; this
  interpretation matches "streaming one-line JSON per event" literally.
- **D-037** (0.80): peer `ServerName` = node `Name` (SAN convention).
- **D-039** (0.78): `dbPath` relocation to `certpaths` — territory overlap.

These three are escalated through the normal decision flow (DecisionEngine) per
the researcher protocol, NOT flagged for human validation.