From bed5a2e8e520f0bc6d9998426064a5db91aa41ff Mon Sep 17 00:00:00 2001 From: Jon Chery Date: Wed, 3 Jun 2026 21:09:15 +0000 Subject: [PATCH] docs(P00): create 4 v0.2 phase plans MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit v0.2 PLAN stage. Translates the IDEATE output (commit 1ee82fc) into 4 executable, vertical-slice phase plans with wave ordering, REQ coverage, must-haves, and per-layer verification. The v0.1 section above is preserved unchanged. Plan structure mirrors v0.1 (per the v0.1 spec at lines 7-166): - Phase heading with **Branch** + **REQ Coverage** lines - ### Must-Haves checkbox list (atomic, file-anchored) - ### Verification block (build / behavior / smoke) - End-of-section ## Wave Ordering + ## Versioning Phase numbering: v0.2 phases are numbered 8-11 (not 1-4) to avoid colliding with v0.1's phase/01..07 branches. v0.1 already shipped 6 phases plus a P07 backfill; the v0.2 work is therefore the 8th project phase overall. Phase 8: mTLS handshake + internal CA with CSR join (Wave 1, branch phase/08-mtls, REQs: 011, 023, 025, 026, 032, 033, 034, 035, 036, 038). 19 must-haves spanning internal/security (CA, CSR, fingerprint, TLS config, rotation, redaction), internal/store (certs migration 0004 + cert_repo), internal/daemon (mTLS server bootstrap with GetCertificate hot-swap), internal/transport (mTLS client + SAN validation + handshake failure logging), internal/audit, internal/cli (cert, node_join, doctor), config surface for trusted CA fingerprint pinning. 11 verification points including a two-node mTLS handshake integration test. Phase 9: Multi-node scheduling & job dispatch (Wave 1, branch phase/09-scheduling, REQs: 004 expansion, 017, 021, 028, 037). 15 must-haves covering internal/transport (dispatch h2c service, idempotency, retry), internal/engine (dispatcher, scheduler bin-pack extension, peer registry), internal/store (migration 0005 node_capacity + capacity_repo), HCL schema for NodeCapacity, internal/cli (node capacity, job run --target), and the daemon dispatch handler. 9 verification points including a two-node dispatch integration test, cancellation with goleak, idempotency dedupe, and deterministic bin-pack scoring. Phase 10: gosec + govulncheck + gitleaks in CI (Wave 2, branch phase/10-security-scan, REQs: 014, 027, 029, 031, 039, 040). 12 must-haves centered on .coreci.yml pipeline additions, scripts/security_scan.sh wrapper, gosec.json baseline, offline govulncheck (GOFLAGS=-mod=mod + GOVULNDB=offline per ROADMAP.md P03 scope change), .gitleaks.toml stopwords (REQ-039), gitleaks baseline (REQ-029), .golangci.yml (REQ-040), pre-commit gitleaks hook, Makefile test-race and security-scan targets, and docs/security-scanning.md. 7 verification points including a network-namespace test that proves govulncheck runs offline. Phase 11: iter.Seq streaming job/node lists (Wave 2, branch phase/11-iter-seq, REQs: 022, 030, 032 expansion). 8 must-haves for internal/store/iter.go (Watch(ctx, query) iter.Seq[T] with 500ms poll + notify hook), internal/cli job_list/node_list --watch and --watch --json modes, signal.NotifyContext cancellation wiring, internal/doctor expansion for jobs/nodes/certs streaming output, plus unit and integration tests. 8 verification points including NDJSON validation via jq -c and goroutine leak assertion via goleak. Wave ordering: P01 and P02 are Wave 1 (sequential because parallelization.enabled=false; P01 is a hard prerequisite for P02 since dispatch endpoints are mTLS-protected). P03 and P04 are Wave 2 (sequential; either order is viable but P03 first keeps the security baseline in place while P04 lands the new CLI surface). Versioning: per-phase tags v0.2.1 (P01), v0.2.2 (P02), v0.2.3 (P03), v0.2.4 (P04); milestone tag v0.3.0 (next minor per run.md feature-milestone promotion). Per RELEASE_POLICY.md, every per- phase tag also produces a Gitea release with tarball asset. No changes to PROJECT.md, REQUIREMENTS.md, ROADMAP.md, ARCHITECTURE.md, PERSONAS.md, IDEATION.md, or config.json — this is a docs-only commit per the PLAN stage contract. No Go code changes. ---ci--- project: orca phase: 0 milestone: v0.2 status: plan ---/ci--- --- .ciagent/PLANS.md | 176 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 176 insertions(+) diff --git a/.ciagent/PLANS.md b/.ciagent/PLANS.md index a103df7..c641012 100644 --- a/.ciagent/PLANS.md +++ b/.ciagent/PLANS.md @@ -163,3 +163,179 @@ For v0.1, `parallelization.enabled=false` — phases run sequentially. - **Milestone type**: `feature` (Phases 1-6 all produce features) - **Patch per phase**: `v0.1.1`, `v0.1.2`, ..., `v0.1.6` - **Final tag on COMPLETE**: `v0.2.0` (next minor per `run.md` versioning logic) + +--- + +# Phase Plans: Orca v0.2 + +All 4 phases with vertical-slice structure, wave ordering, and REQ-ID mapping. +v0.2 scope: **Networking, Observability, Security Hardening** — extends v0.1 +with secure cross-node transport, multi-node scheduling, richer CI security +scanning, and streaming I/O. + +Branching convention: branches are numbered after v0.2's milestone branch +`milestone/v0.2-networking-observability-security`. v0.2's P01 uses phase +number `08`, P02 uses `09`, etc., to avoid colliding with v0.1's +`phase/01..07` branches (see `RELEASE_POLICY.md` and `run.md` for tag +hygiene). Milestone branch name in heading reflects the v0.1 retcon where +P00-P07 are the v0.1 work; v0.2's first phase is the eighth phase of the +project overall. + +--- + +## Phase 8: mTLS Handshake + Internal CA with CSR Join (Wave 1) + +**Branch**: `phase/08-mtls` +**REQ Coverage**: REQ-011, REQ-023, REQ-025, REQ-026, REQ-032, REQ-033, REQ-034, REQ-035, REQ-036, REQ-038 + +### Must-Haves + +- [ ] `internal/security/ca.go` — CA init, sign CSR, CA cert persistence to `~/.orca/ca.crt` (0644) and `~/.orca/ca.key` (0600) per REQ-033 +- [ ] `internal/security/csr.go` — CSR generation from a private key with SANs populated (REQ-036) +- [ ] `internal/security/certgen_test.go` — round-trip test: CA-init → build CSR → sign → verify the chain programmatically +- [ ] `internal/security/fingerprint.go` — `Fingerprint(certPath) (sha256hex, error)` (used by `orca cert join --ca-fingerprint`) +- [ ] `internal/security/tls_config.go` — `ServerTLSConfig()` and `ClientTLSConfig(caPath)` builders, with `MinVersion = tls.VersionTLS13` and AEAD cipher allowlist +- [ ] `internal/security/rotation.go` — proactive rotation alarm: returns WARN 30d before `not_after` (REQ-034); history table bounded at 10 generations per cert kind (REQ-025) +- [ ] `internal/security/redact.go` — `orca cert show` redaction: strips private key material from default and `--json` output (REQ-035) +- [ ] `internal/store/migrations/0004_certs.sql` — `certs` table (`id`, `kind`, `node_id`, `serial_hex`, `subject_cn`, `issuer_cn`, `not_before`, `not_after`, `fingerprint`, `source_path`, `created_at`) plus indexes +- [ ] `internal/store/cert_repo.go` — CRUD for the `certs` table; rotation history pruning helper (REQ-025) +- [ ] `internal/daemon/tls.go` — mTLS server bootstrap; `GetCertificate` hot-swap callback so `orca cert renew` takes effect without daemon restart +- [ ] `internal/transport/mtls.go` — mTLS client with cipher allowlist; SAN validation against the pinned peer identity (REQ-036) +- [ ] `internal/transport/handshake_log.go` — structured slog fields on mTLS failure: `event=mtls.handshake`, `peer`, `cert_fp`, `err` (REQ-038) +- [ ] `internal/audit/audit.go` — emit `cert.issued`, `cert.renewed`, `cert.joined`, `node.handshake_ok`, `node.handshake_failed` entries +- [ ] `internal/cli/cert.go` — `orca cert {gen,ca-init,csr,show,renew}` subcommands +- [ ] `internal/cli/node_join.go` (extend v0.1 stub) — `orca node join --ca-fingerprint ` verifies on-disk CA matches the pinned value (REQ-026); refuses to start the daemon on mismatch +- [ ] `internal/cli/doctor.go` (NEW package `internal/doctor`) — `orca doctor`, `orca doctor cert`, `orca doctor network`, `orca doctor db` subcommands (REQ-032; `network` and `db` checks may stub in P01, full impl in later phases) +- [ ] Config surface: `~/.orca/orca.hcl` gains a `trusted_ca_fingerprint` field consumed at daemon start (REQ-026) +- [ ] Unit tests for: file mode enforcement (REFUSE on wrong mode, REQ-033), rotation alarm firing at 30d, redaction in `cert show`, fingerprint mismatch at `node join` +- [ ] Integration test: two-node mTLS handshake — node A signs node B's CSR; node B dials node A and `/healthz` returns 200; cross-signed with a non-matching CA returns a structured `mtls.handshake` failure log line + +### Verification + +- `go build ./...` PASS +- `go test ./internal/security/... ./internal/store/... ./internal/transport/...` PASS +- `go test -race ./...` PASS (REQ-031 cross-cutting) +- `orca cert ca-init` produces a valid CA; `ca.crt` is 0644, `ca.key` is 0600; daemon refuses to start if either is wrong (REQ-033) +- `orca cert gen` produces a server cert signed by the CA, with DNS and IP SANs present (REQ-036); CSR without SANs is rejected at sign-time +- `orca node join --ca-fingerprint ` dials over mTLS; handshake succeeds when CA matches, fails (and logs `event=mtls.handshake peer=... cert_fp=... err=...`) when it does not (REQ-026, REQ-038) +- `orca cert renew` rotates the cert without daemon restart (hot-swap via `GetCertificate`); new connections use the new cert +- `orca cert show` (default and `--json`) never prints private key material (REQ-035) +- Cert rotation history is bounded: inserting an 11th cert per `(node_id, kind)` prunes the oldest (REQ-025) +- Proactive rotation alarm: a cert with `not_after` 30d from now triggers a structured WARN at daemon start (REQ-034) +- `orca doctor cert` reports PASS/WARN/FAIL for CA, server cert, expiry window, and fingerprint pin match (REQ-032) +- Every cert issuance produces an `audit_log` row with `event` and `cert_fp` + +--- + +## Phase 9: Multi-Node Scheduling & Job Dispatch (Wave 1) + +**Branch**: `phase/09-scheduling` +**REQ Coverage**: REQ-004 (expansion), REQ-017, REQ-021, REQ-028, REQ-037 + +### Must-Haves + +- [ ] `internal/transport/dispatch.go` — JSON-over-HTTP `orca.v1.Dispatch` service via stdlib h2c (no ConnectRPC — not in go.mod per research); routes `POST /orca.v1.Dispatch/Submit` and `POST /orca.v1.Dispatch/Status` +- [ ] `internal/transport/idempotency.go` — `X-Orca-Idempotency-Key` header parsing; server-side dedupe store (REQ-037); client-side retry only when header is present +- [ ] `internal/transport/retry.go` — exponential backoff with jitter (100ms, x2, cap 5s, max 5 attempts); only idempotent verbs auto-retry without the key +- [ ] `internal/engine/dispatcher.go` — `Submit(peerID, spec) (jobID, error)` blocking call; bin-pack selector falls through to remote peer when local node cannot fit +- [ ] `internal/engine/scheduler.go` (extend v0.1) — best-fit bin-packing by `available_cpu` and `available_memory`; within-node FIFO queue +- [ ] `internal/engine/peer.go` — peer registry: in-memory map plus SQLite-persisted; records `last_seen`, address, capacity snapshot +- [ ] `internal/store/migrations/0005_node_capacity.sql` — `node_capacity` table (`node_id`, `cpu_millicores`, `memory_mib`, `disk_mib`, `updated_at`) +- [ ] `internal/store/capacity_repo.go` — capacity CRUD +- [ ] HCL schema for `NodeCapacity` (REQ-028): `cpu_millicores`, `memory_mib`, `disk_mib`; loaded from `~/.orca/node.hcl` at `orca node join` and CLI flags +- [ ] `internal/cli/node_capacity.go` — `orca node capacity --set` and `orca node capacity` subcommands +- [ ] `internal/cli/job_run.go` (extend v0.1) — `orca job run --target ` explicit target (overrides bin-pack); `orca job run` (no target) lets the dispatcher pick best-fit +- [ ] `internal/daemon/dispatch_handler.go` — mTLS-protected endpoints for `Submit` and `Status`; honors `X-Orca-Idempotency-Key` for dedupe +- [ ] Cancellation propagation: `context.Context` flows from CLI → daemon → executor → transport → peer; ctrl-c aborts the local task AND the in-flight dispatch call (REQ-017) +- [ ] Unit tests: bin-pack scoring (3 jobs across 2 nodes picks the node with the most free capacity each time); idempotency dedupe; retry only on transient errors; cancellation teardown +- [ ] Integration test: two-node dispatch — job submitted to node A with no local capacity, dispatched to node B over mTLS, returns the job ID issued by node B; ctrl-c mid-run aborts both sides cleanly + +### Verification + +- `go build ./...` PASS +- `go test ./internal/engine/... ./internal/transport/... ./internal/store/...` PASS +- `go test -race ./...` PASS (REQ-031 cross-cutting) +- Two-node integration test: `orca job run spec.hcl` on node A with insufficient local capacity dispatches to node B and returns node B's job ID +- Cancellation: `Ctrl-C` during a dispatched job aborts the local call AND the in-flight `POST /orca.v1.Dispatch/Submit`; no orphan goroutines (assert with `goleak`) +- Idempotency: same `X-Orca-Idempotency-Key` submitted twice within the dedupe window returns the same job ID and does NOT create a duplicate row +- Bin-packing: 3 jobs across 2 nodes, each picks the node with the most free capacity (deterministic test) +- `orca node capacity --set` updates the persisted `node_capacity` row; subsequent dispatches see the new value +- `X-Orca-Idempotency-Key` header is REQUIRED for `POST /orca.v1.Dispatch/Submit` retries; absent header + transient error → no retry + +--- + +## Phase 10: `gosec` + `govulncheck` + `gitleaks` in CI (Wave 2) + +**Branch**: `phase/10-security-scan` +**REQ Coverage**: REQ-014, REQ-027, REQ-029, REQ-031, REQ-039, REQ-040 + +### Must-Haves + +- [ ] `.coreci.yml` `validate` pipeline: add `gosec`, `govulncheck`, `gitleaks` stages in this order; `make security-scan` is the local equivalent +- [ ] `scripts/security_scan.sh` — wrapper that runs all three tools, exits non-zero on any unsuppressed finding +- [ ] `gosec.json` baseline: initial run via `gosec -fmt json -no-fail > gosec.json`; committed to the repo; empty baseline (clean repo) so any new G101 (hardcoded credentials) finding fails the build +- [ ] `govulncheck` invocation: runs in **offline mode** per REQ-027 — use `GOFLAGS=-mod=mod` and `GOVULNDB=offline` (or pre-mirrored DB via `GOVULNCHECK_DB`); the chosen mechanism is documented in `docs/security-scanning.md` +- [ ] `govulncheck` output gate: `govulncheck -format json ./...` piped through a small Go program (or `jq`) that exits non-zero on any unsuppressed finding +- [ ] `.gitleaks.toml` (REQ-039) — allowlist `-----BEGIN CERTIFICATE-----` PEM blocks; flag `-----BEGIN RSA PRIVATE KEY-----`; stopwords for `internal/security/testdata/` paths +- [ ] `.gitleaks-baseline.json` (REQ-029) — baseline file committed to suppress the pre-existing `.env` SHA-1 leak from v0.1 history (rotated forward; baseline gates future re-leaks) +- [ ] `.golangci.yml` (REQ-040) — unified lint config: `gosec`, `govet`, `gofmt`, `ineffassign`, `misspell` linters; supersedes any per-tool invocations +- [ ] `.githooks/pre-commit` — gitleaks protect; commits remain allowed when gitleaks is not installed (gate, not block) +- [ ] `Makefile` — add `make test-race` target that runs `go test -race ./...` (REQ-031); wire into `.coreci.yml` `validate` pipeline +- [ ] `Makefile` — add `make security-scan` target that invokes `scripts/security_scan.sh` +- [ ] `docs/security-scanning.md` — operator-facing doc: what each tool checks, how the offline mode is achieved, how to add a baseline entry + +### Verification + +- `.coreci.yml` parses (yaml validation) and `make validate` is green locally +- `make security-scan` runs all three tools and returns 0 on a clean working tree +- `gosec`: introducing a new `G101` (hardcoded credential) finding in a Go file causes `make security-scan` to fail +- `govulncheck`: with `GOFLAGS=-mod=mod`, the run completes without network access (offline mode) — verified by running the CI step under a network namespace that blocks outbound HTTPS to `vuln.go.dev`; unsuppressed CVE in a dep still fails the build +- `gitleaks`: a sample secret injected into a test file is detected; a `-----BEGIN CERTIFICATE-----` PEM block in `internal/security/testdata/` is allowed (not flagged) +- `make test-race` passes against the current test suite (REQ-031 cross-cutting) +- `.gitleaks-baseline.json` round-trips: re-running the gitleaks pre-commit hook does not re-flag the historical `.env` SHA-1 +- `.golangci.yml` `make lint` is green against the current code + +--- + +## Phase 11: `iter.Seq` Streaming Job/Node Lists (Wave 2) + +**Branch**: `phase/11-iter-seq` +**REQ Coverage**: REQ-022, REQ-030, REQ-032 (expansion) + +### Must-Haves + +- [ ] `internal/store/iter.go` — `Watch(ctx, query) iter.Seq[T]` for jobs and nodes; poll-based at 500ms initially, with an internal notify channel hook so a future event-driven source can replace the poll without API churn +- [ ] `internal/store/iter_test.go` — round-trip: insert N rows, range over `Watch`, assert all N are yielded; cancel mid-stream and assert the seq stops cleanly with no goroutine leak (`goleak` or `runtime.NumGoroutine` snapshot) +- [ ] `internal/cli/job_list.go` (extend v0.1) — `orca job list --watch` returns `iter.Seq[Job]`; default output is a human-readable table that updates; `orca job list --watch --json` outputs one JSON object per line for piping (REQ-030) +- [ ] `internal/cli/node_list.go` (extend v0.1) — `orca node list --watch` returns `iter.Seq[Node]`; same table/JSON split as jobs +- [ ] Cancellation wiring: `signal.NotifyContext(parent, os.Interrupt)` — ctrl-c stops the stream cleanly without orphan goroutines +- [ ] `internal/doctor/` (extend P01 stub) — `orca doctor jobs`, `orca doctor nodes`, `orca doctor certs` stream results as `iter.Seq[DoctorResult]`; each row carries a status (`PASS|WARN|FAIL`) and a human-readable message (REQ-032 expansion) +- [ ] Unit tests: `--watch` mode yields on insert; `--watch --json` produces one JSON object per line (line-by-line parse); `orca doctor certs` lists all certs with expiry and rotation status +- [ ] Integration test: start `orca job list --watch` as a subprocess, insert a new job, assert the subprocess output contains the new job's ID + +### Verification + +- `go build ./...` PASS +- `go test ./internal/store/... ./internal/cli/... ./internal/doctor/...` PASS +- `go test -race ./...` PASS (REQ-031 cross-cutting) +- `orca job list --watch` streams and updates on new job insertion (integration test, two-process or two-goroutine) +- `orca job list --watch --json` produces one JSON object per line (NDJSON); validatable by piping through `jq -c .` +- `orca doctor certs` lists every cert with its `not_after`, days-until-expiry, and rotation status (REQ-032 expansion; ties into P01's cert health checks) +- `Ctrl-C` during a watch cleanly cancels the seq; `runtime.NumGoroutine()` returns to the pre-watch baseline (asserted in tests via `goleak.VerifyNone` or a manual snapshot diff) +- `orca node list --watch --json` behaves identically to the jobs variant + +--- + +## Wave Ordering + +- **Wave 1** (Phases 8-9): Networking & scheduling — mTLS handshake and internal CA (P01) is a hard prerequisite for cross-node dispatch (P02), since the dispatch endpoints are mTLS-protected. Both phases run sequentially because `parallelization.enabled=false`. +- **Wave 2** (Phases 10-11): Security scan & streaming I/O — security scanning (P03) and `iter.Seq` streaming (P04) are independent; `parallelization.enabled=false` so they run sequentially, but either order is technically viable. P03 first keeps the security baseline in place while P04 lands the new CLI surface. + +Phases within a wave can be parallelized if `parallelization.enabled=true`. +For v0.2, `parallelization.enabled=false` — phases run sequentially. + +## Versioning + +- **Milestone type**: `feature` (all 4 phases ship features) +- **Patch per phase**: `v0.2.1` (P01 mTLS), `v0.2.2` (P02 scheduling), `v0.2.3` (P03 security scan), `v0.2.4` (P04 iter.Seq) +- **Final tag on COMPLETE**: `v0.3.0` (next minor per `run.md` versioning logic; per `RELEASE_POLICY.md`, every per-phase tag also produces a Gitea release)