Operator decision (resolves grill C-04 + escalation E-03): keep 2 milestones (v0.9 + v0.10), keep all phases (40 total, exceeds 35 soft limit), v1.0 is UAT-gated and cut as a separate tag (v1.0.0) after v0.10 completion per operator sign-off — not a separate milestone. Relabels all v1.0 milestone references to v0.10 across ROADMAP, REQUIREMENTS, GRILL_v0.9, IDEATION_v0.9, PRD_v0.9, PROJECT. Phase content unchanged; only the milestone label moves. Historical grill narrative (the original PRD §23 counts and the E-03 auto-split reasoning) preserved verbatim for audit integrity. C-04 and E-03 marked RESOLVED in GRILL_v0.9.md. Milestone structure: - v0.9: Re-architecture Foundation & Workloads (13 phases P00..P0X) - v0.10: Production Hardening (19 phases P00..P16, milestone tag v0.10.0) - v1.0: UAT-gated production-ready cut (separate v1.0.0 tag, not a milestone) verify-reqs: 90 requirements consistent. ---ci--- project: orca phase: 0 milestone: v0.9 status: complete gate: C-04 resolved ---/ci---
46 KiB
Ideation v0.9 — Re-architecture Foundation
Project: orca (single-project mode) | Milestone: v0.9/v0.10 re-architecture Date: 2026-08-05 | Agent: ideation agent | Confidence threshold: 0.60 Next REQ ID prior to this run: REQ-060 (v0.8 complete)
Context
The v0.8 codebase (REQ-001..060, all Complete) is a daemon-based, mTLS, HCL, single-namespace orchestration engine. The adopted PRD supersedes this with a CLI-only, SSH-push, step-ca, Markdown-frontmatter, multi-namespace stack. 9 packages are deprecation targets (~2,400 LOC of v0.8 daemon/transport/security-ca/engine-dispatch/jobspec-hcl/config-hcl/certpaths code), 7 packages are adaptable, and 8 subsystems are net-new with zero implementation. The §23 milestone plan has 11 v0.9 phases + 17 v0.10 phases but under-specifies the deprecation mechanics, the SSH-push transport design, the lead-applier execution model, several adapter/bridge layers, and the migration ordering risk.
This ideation produced 30 ideas across three tiers, all accepted at ≥0.60 confidence, mapped to REQ-061..REQ-090. Seven phase-reordering flags against the PRD §23 plan are listed at the end.
Tier 1 — Mechanical (Codebase-Observable Gaps & Hygiene)
I-M-001 — orca daemon deprecation command and build-tag removal path
- Tier: mechanical
- Description: The PRD deprecates
internal/daemon/(R-001) but §23 never says how.internal/cli/daemon.go(100 LOC) registers thedaemoncobra command and wiresdaemon.NewServer+engine.Dispatcher. Big-bang removal would break the v0.8→v1.0 migration path (v0.10-P14) becauseorca upgrade --to-v1.0must run against a live v0.8 cluster that still has daemons. Proposal: (1) in v0.9,orca daemonemits a deprecation warning and still runs (dual-write window); (2) in v1.0,orca daemonis repurposed toorca daemon drain-and-stop(stops v0.8 daemons on peers via SSH, confirms workloads survive via systemd); (3) post-v1.0, the command andinternal/daemon/are deleted. Add// DeprecatedGo doc comments +slog.Warnon every run. - Rationale: R-001 is an invariant, but the transition off the daemon is a mechanical gap. The v0.8
daemon.gois wired inroot.goinit; removing it without a transition plan breaks the §24 migration. - Proposed REQ ID: REQ-061
- Proposed phase placement: v0.10-P14 (migration) — deprecation warning lands in v0.9-P0X
- Confidence: 0.82
- Accept/Defer: accept
I-M-002 — Coverage follow-ups: 3 zero-test packages + internal/cli to 70%
- Tier: mechanical
- Description: v0.8 P01 (REQ-057) raised 6 packages to ≥70% and added first tests for
internal/audit,internal/certpaths,cmd/orcaat a 50% toe-hold. The v0.9 re-architecture will replace several of these packages, but the adaptable ones (internal/store,internal/doctor,internal/cli) must keep their 70% floor through the refactor. Oncedaemon.gois deprecated/removed (I-M-001), the exclusion reason disappears and the floor applies to the whole package. Add a coverage-gate assertion in the v0.9 P0X ship phase thatinternal/cli≥ 70% including all new subcommand files (ns, txn, pve, secrets, volume, cache, backup). - Rationale: The PRD §23 does not mention coverage. The config.json policy says 70% floor for new packages, 50% minimum. The 8 net-new subsystems will need 70% floors from their first phase. Without an explicit gate, the v0.7/v0.8 "toe-hold at 50% then defer" pattern will repeat.
- Proposed REQ ID: REQ-062
- Proposed phase placement: v0.9-P0X (ship+audit) + each net-new package's first phase
- Confidence: 0.88
- Accept/Defer: accept
I-M-003 — known_hosts flock concurrency gap (deferred P1 from REVIEW_v0.8 A2)
- Tier: mechanical
- Description: REVIEW_v0.8 flagged A2 (P1):
TOFUHostKeyCallbackcapture path (bootstrap.go:290-302) andResetHostKey(bootstrap.go:479-523) both do read-modify-write onknown_hostswith no lock. The review said "Defer to v0.9." This is now load-bearing because the SSH-push transport (R-001) will do many more concurrent SSH operations than v0.8 did. Add aflock-style advisory lock (stdlibsyscall.Flockwrapper) around the RMW in both paths. Lock file iscluster/known_hosts.lock(multi-namespace layout, R-002). - Rationale: The v0.8 single-operator mitigation is weaker under v0.9's parallel SSH fan-out. The PRD doesn't mention this, but the SSH-push transport makes the race more likely.
- Proposed REQ ID: REQ-063
- Proposed phase placement: v0.9-P0a1 (path resolver, since it establishes
cluster/layout) - Confidence: 0.74
- Accept/Defer: accept
I-M-004 — HCL→Markdown jobspec adapter/bridge layer
- Tier: mechanical
- Description: R-013 says the Markdown parser is canonical;
.yamland.hclare "accepted by parser dispatcher." Butinternal/jobspec/spec.go(69 LOC) is an HCL-only parser with a flatSpec{Job, Tasks}schema — nokind:(R-012), runtime blocks, or body preservation (R-014/R-015). The "dispatcher" implies the new parser detects file extension and dispatches. Proposal: keepinternal/jobspec/spec.goas the legacy HCL path behind// Deprecated; addinternal/jobspec/markdown.go(canonical) +internal/jobspec/dispatch.go(extension-based dispatcher:.md→Markdown,.hcl→legacy,.yaml→Markdown-with-empty-body). The dispatcher returns a unified*WorkloadSpecthat the legacy parser populates via an adapter. Preservesorca job run old-spec.hclduring the migration window. - Rationale: R-013 explicitly accepts
.hcl, so a dispatcher is required. §23 v0.9-P0b says "parser dispatcher" but doesn't specify the adapter. - Proposed REQ ID: REQ-064
- Proposed phase placement: v0.9-P0b (Markdown jobspec parser)
- Confidence: 0.85
- Accept/Defer: accept
I-M-005 — orca doctor --legacy-paths detection for v0.8 residue
- Tier: mechanical
- Description: The v0.8 layout is
~/.orca/{orca.db, ca.crt, ca.key, server.crt, server.key, orca_ssh_key, known_hosts, config.hcl}. The v1.0 layout isORCA_HOME/{_defaults/, cluster/{ca,master.key,peers,pve,txns}, <ns>/{db,.env,.env.secrets,jobs,alloc,ns.md}, orca_cache.db}.orca doctor(internal/doctor/doctor.go, 501 LOC, adaptable) must gain adoctor legacysubcommand that detects v0.8 residue: presence oforca.dbat ORCA_HOME root,ca.crt/ca.key(internal CA, superseded by step-ca),config.hcl(HCL, demoted), flatserver.crt(single-namespace), and anamespacecolumn in any*.db(R-002 says no namespace column). Output: list of detected legacy artifacts with migration recommendations. This is the detection half of v0.10-P14; the migration half is I-C-001. - Rationale: §23 v0.10-P14 says "orca upgrade --to-v1.0, post-invariant checks" but doesn't specify the detection surface.
doctoris the diagnostics framework and is explicitly adaptable. - Proposed REQ ID: REQ-065
- Proposed phase placement: v0.10-P14c (mixed-version tolerance + no-orca enforcement)
- Confidence: 0.80
- Accept/Defer: accept
I-M-006 — Legacy CA state migration to step-ca (cert import)
- Tier: mechanical
- Description:
internal/security/ca.go(338 LOC) holds an internal Go CA withca.crt/ca.key(RSA 3072, 10-year). The PRD replaces this with step-ca (R-006, D-101 reverses AD-010). The v0.10-P14 migration must handle existing deployments with an internal CA: (a) import the existing CA key into step-ca asstep ca init --deployment-type standalone --remote-managementwith the existing key; (b) issue new SVIDs from step-ca and let old certs expire; (c) document that v0.8 certs are invalidated and re-bootstrap is required. The codebase audit saysca.go+csr.goare replaced — but the state (the CA key + issued server certs incert_repoSQLite) may need to be preserved for audit history even if the live trust root changes. Proposal:orca upgrade --to-v1.0 --import-careads~/.orca/ca.key, initializes step-ca with it, and re-issues workload SVIDs. Without this, existing deployments lose their trust root with no path back. - Rationale: AD-010 is explicitly reversed by D-101, but the reversal doesn't address what happens to the existing CA material. §24 covers data migration but not CA migration.
- Proposed REQ ID: REQ-066
- Proposed phase placement: v0.10-P14a (data migration)
- Confidence: 0.70
- Accept/Defer: accept (design in v0.9-P00 so step-ca integration knows the import contract)
I-M-007 — Fuzz test harness for the Markdown frontmatter parser
- Tier: mechanical
- Description: R-014/R-015 require byte-exact body preservation — "body of every .md config file preserved verbatim." This is a class of bug that's easy to get wrong (off-by-one on the
---delimiter, trailing newline handling, BOM, CRLF, nested code fences containing---). v0.8 has no fuzz tests at all. Proposal: add atesting.Ffuzz target ininternal/jobspec/markdown_test.gothat round-trips random frontmatter+body throughParseMarkdownand assertsbody == roundtripped.bodybyte-exact. Also add a corpus of adversarial fixtures (CRLF, BOM, no-frontmatter, empty-frontmatter, frontmatter-with-only-separator). §23 v0.10-P08 mentions integration tests but not fuzzing. - Rationale: R-015 is a load-bearing invariant (body appears in inspect/history). Byte-exactness is exactly what fuzz tests are for. The v0.8 jobspec tests are golden-file only (no fuzz).
- Proposed REQ ID: REQ-067
- Proposed phase placement: v0.9-P0b (Markdown parser) — fuzz from day one
- Confidence: 0.78
- Accept/Defer: accept
I-M-008 — Deprecation warnings on removed/repurposed CLI subcommands
- Tier: mechanical
- Description: The v0.8 CLI has
orca cert {ca-init,gen,show,renew,fingerprint}(internal/cli/cert.go),orca node joinwith mTLS handshake semantics (internal/cli/node.go),orca job run <spec.hcl>. The PRD repurposesnode jointo SSH-bootstrap (no mTLS), deprecatescert(step-ca handles it), and changesjob runto accept.mdspecs. Each removed/changed command should emit aslog.Warndeprecation banner with the v1.0 replacement, except when run underorca upgrade. The existingroot.goPersistentPreRunEis the natural hook for a global--no-deprecation-warningsflag. - Rationale: Operators running v0.8 commands against v0.9/v0.10 need to know what changed. The PRD doesn't mention deprecation UX.
- Proposed REQ ID: REQ-068
- Proposed phase placement: v0.9-P0X (ship) + v0.10-P13 (ns subcommands, when CLI surface is finalized)
- Confidence: 0.72
- Accept/Defer: accept
I-M-009 — internal/config/config.go HCL config demotion via adapter
- Tier: mechanical
- Description:
internal/config/config.go(127 LOC) parses HCL config with keysdb_path, listen_addr, ca_path, server_cert_path, server_key_path, node_capacity. The PRD replaces this with Markdown-frontmatter config (R-014) + per-namespace.env/.env.secrets(R-011). Thelisten_addrandserver_*_pathkeys are daemon-specific (deprecated by R-001). Theroot.goPersistentPreRunEcallsconfig.Load(configPath)on every command — must be repointed to the new Markdown config loader. Proposal: keepinternal/config/aslegacy_config.gowith// Deprecated; addinternal/config/markdown.gofor the new loader;root.godispatches on file extension (.hcl→legacy,.md→new). The--configflag semantics change:.hclis read-only legacy,.mdis canonical. - Rationale: R-014 makes Markdown canonical but
.hclmust still parse during migration. The existingconfig.Loadis called unconditionally inroot.go:42-47. - Proposed REQ ID: REQ-069
- Proposed phase placement: v0.9-P0a1 (path resolver + config demotion)
- Confidence: 0.76
- Accept/Defer: accept
I-M-010 — internal/certpaths/ replacement with multi-namespace path resolver
- Tier: mechanical
- Description:
internal/certpaths/certpaths.go(64 LOC) returns flat paths:Dir() = $ORCA_HOME,CACertPath() = Dir/ca.crt,DBPath() = Dir/orca.db. R-002 requires multi-namespace layout:ORCA_HOME/<ns>/db/,ORCA_HOME/cluster/{ca,master.key,peers,pve,txns},ORCA_HOME/_defaults/. The package is imported bydoctor,proxmox,store,cli— changing it is cross-cutting. Proposal: replacecertpathswith a newinternal/pathspackage:paths.NamespaceDir(ns),paths.ClusterDir(),paths.CacheDB(),paths.MasterKey(),paths.NSDb(ns),paths.NSEnv(ns),paths.NSSecrets(ns). Keepcertpathsas a thin shim that callspathswith the default namespace for v0.8 compat, then remove the shim post-v1.0. - Rationale: R-002 is foundational and
certpathsis the single source of path truth. Every adaptable package (store.Open,doctor,proxmox) imports it. Highest-blast-radius mechanical change. - Proposed REQ ID: REQ-070
- Proposed phase placement: v0.9-P0a1 (must come first)
- Confidence: 0.84
- Accept/Defer: accept
I-M-011 — internal/store/ schema: per-namespace DBs, drop ns column
- Tier: mechanical
- Description: R-002 says "No
namespacecolumn in SQLite." The v0.8 schema has 7 migrations (0001..0007) with a singleorca.db. The v0.10 model has one DB per namespace (<ns>/db/orca.db) plus a CLI-side cache DB (orca_cache.db, R-008). The existingstore.Open(path)takes a path arg — adaptable. But the migrations are global; they need to apply per namespace DB. Proposal:store.Opengains a namespace parameter (or caller passespaths.NSDb(ns));migrate.goruns0001..0007(minus0006_node_kind_oswhich is v0.8-specific) plus new0008_namespace_layout.sql. Thecert_repo(0004_certs.sql) is removed (step-ca handles certs). The audit_log table moves to the CLI-side cache DB (R-008). Existing v0.8orca.dbis migrated by splitting tables into per-namespace DBs during v0.10-P14. - Rationale: R-002 is explicit ("No namespace column in SQLite") but the existing schema has a single DB. §23 doesn't specify the schema split mechanics.
- Proposed REQ ID: REQ-071
- Proposed phase placement: v0.9-P0a1 + v0.10-P06 (alloc history, which uses cache DB)
- Confidence: 0.80
- Accept/Defer: accept
I-M-012 — internal/transport/ deletion + SSH-push package introduction
- Tier: mechanical
- Description:
internal/transport/(7 files, ~1300 LOC incl tests) implements mTLS client/server, dispatch, idempotency, retry, handshake logging. R-001 + R-006 replace this with SSH-push. The idempotency and retry logic (idempotency.go123 LOC,retry.go151 LOC) is conceptually reusable for SSH-push (retry on SSH failure, idempotency keys for SCP'd configs). Proposal: deletemtls.go,dispatch.go,handshake_log.go; extract retry/idempotency patterns into a newinternal/sshpush/package. The existingtransport.IdempotencyStore(in-memorysync.Mapof keys) is directly reusable. This avoids re-implementing retry semantics from scratch. - Rationale: The codebase audit marks
internal/transport/as fully replaced, but the retry/idempotency patterns are transport-agnostic. §23 doesn't call this out. - Proposed REQ ID: REQ-072
- Proposed phase placement: v0.9-P00 (deprecation sweep) — delete in v0.10-P14
- Confidence: 0.68
- Accept/Defer: accept (defer deletion to v0.10-P14 to keep dual-write window open)
Tier 2 — Backend-Enriched (Structural / Architectural)
I-B-001 — SSH-push transport layer design
- Tier: backend-enriched
- Description: The PRD replaces
internal/transport/(mTLS HTTP) with SSH-push but §23 never specifies the transport's internal design. Key decisions: (1) Connection pooling: reuse*ssh.Clientper peer across multiple SCP/exec operations within a single CLI invocation. (2) Idempotency: SCP of a config file is idempotent if content hash matches — use content-addressed filename (/run/orca/<hash>.unit) and skip if present. (3) Retry: reuse v0.8's exponential backoff (100ms start, ×2, cap 5s, max 5 attempts) applied to SSH dial/exec failures. (4) Timeout: per-operationcontext.WithTimeout(default 30s SCP, 10s exec). (5) Fan-out:errgroup.Groupwith bounded concurrency for N-peer ops (default 8). (6) known_hosts: reuseproxmox.TOFUHostKeyCallbackfor all peers, not just Proxmox. - Rationale: Load-bearing replacement for the entire v0.8 transport layer. §23 assumes it but never designs it. Without connection pooling, every CLI operation re-dials SSH.
- Proposed REQ ID: REQ-073
- Proposed phase placement: v0.9-P01 (first phase needing SSH-push) — design in v0.9-P0a1
- Confidence: 0.86
- Accept/Defer: accept
I-B-002 — Emitter template system (Layer 4)
- Tier: backend-enriched
- Description: The PRD §5 describes a 4-layer architecture where Layer 4 is "emitters" that render systemd units, Traefik dynamic config, Syncthing config, etc. from the workload spec. §23 never specifies the emitter interface. Proposal: an
internal/emitter/package withEmitterinterface:Render(spec *WorkloadSpec, node *Node) ([]File, error)whereFile{Path, Content, Mode}. Implementations:systemdEmitter,traefikEmitter,syncthingEmitter,socketEmitter. The SSH-push transport SCPs the[]Fileatomically (write-to-tmp + rename). Emitters registered per workload kind + runtime. - Rationale: The emitter layer is the bridge between the declarative spec and the server-side files. Without a defined interface, each phase (P02 service, P04 hooks, P08 sockets, P09 storage) will invent its own rendering.
- Proposed REQ ID: REQ-074
- Proposed phase placement: v0.9-P0c (schemas + emitter interface)
- Confidence: 0.82
- Accept/Defer: accept
I-B-003 — Lead applier execution model: pure bash + systemd timer vs CLI-invoked
- Tier: backend-enriched
- Description: R-001 says "no orca binary on servers." R-010 says the lead applies desired-state transactionally. Unresolved: does the lead run
orca-pull.sh(pure bash that SCPs a desired-state bundle and applies it viasystemctl daemon-reload+systemctl restart) or does the operator's CLI SSH into the lead and runsorca applyremotely (which would put an orca binary on the lead, violating R-001)? The PRD's intent is the former: the lead is bare Linux with systemd timers + bash. Proposal: (1) the CLI renders a transaction bundle (tarball of desired-state files +apply.sh+verify.sh) on the operator host; (2) SCPs it to the lead's/run/orca/txns/<txn-id>/; (3) the lead's systemd timer runs/run/orca/txns/<txn-id>/apply.shwhich idempotently applies and runs verify; (4) the CLI polls the lead for txn status via SSH (cat /run/orca/txns/<txn-id>/status.json). The bash scripts are generated by the CLI's emitter (I-B-002), not hand-written per cluster. - Rationale: The most ambiguous load-bearing design decision in the PRD. R-001 + R-010 together imply the lead runs no orca binary, but the lead must apply transactions. §23 doesn't resolve this. Getting it wrong means either violating R-001 or having no transactional apply.
- Proposed REQ ID: REQ-075
- Proposed phase placement: v0.10-P10 (transactional plane) — bundle format designed in v0.9-P00
- Confidence: 0.78
- Accept/Defer: accept
I-B-004 — step-ca integration: provisioning, CA bootstrap, cert signing API, SVID minting
- Tier: backend-enriched
- Description: D-101 reverses AD-010 (which rejected step-ca as "too heavyweight"). §23 mentions step-ca in R-006 but never specifies the integration. Key surfaces: (1) Provisioning:
orca init(adapted from v0.8'sinternal/cli/init.go) runsstep ca initon the lead, stores root + intermediate incluster/ca/. (2) CA bootstrap: CLI SSHs to the lead, installs step-ca via apt, runsstep ca init, storesstep-ca.jsonconfig. (3) Cert signing API: workloads request SVIDs viastep ca token(JWE provisioner token minted by CLI) →step ca certificate. The CLI mints the token because it holds the provisioner password (incluster/master.key-derived form). (4) SVID minting: each workload gets a SPIFFE ID (spiffe://orca/<ns>/<workload>/<instance>) encoded as a SAN in the step-ca-issued cert. The v0.8internal/security/ca.gois deleted; a newinternal/stepca/package wraps thestepCLI via SSH (no Go step-ca client library — keep zero-new-dep posture if possible, or addgithub.com/smallstep/clias a dep). - Rationale: step-ca is a new external dependency with its own config format, provisioner model, and CLI. §23 assumes it but never designs the integration. security-engineer persona must be reactivated.
- Proposed REQ ID: REQ-076
- Proposed phase placement: v0.9-P07 (runtime block — runtimes need SVIDs) + v0.10-P02 (ACL — SPIFFE identities)
- Confidence: 0.74
- Accept/Defer: accept
I-B-005 — Traefik dynamic config generation and atomic reload
- Tier: backend-enriched
- Description: R-006 makes Traefik load-bearing (mTLS termination + health checks). §23 puts service blocks + Traefik health checks in v0.9-P02. Design: the CLI's Traefik emitter (I-B-002) renders a dynamic config file (
/etc/traefik/dynamic/orca-<ns>-<svc>.yaml) with backends (the socket paths from R-007), health checks, and mTLS config pointing at step-ca's root. Atomic reload: Traefik watches the dynamic dir withfsnotify— writing the file atomically (tmp+rename) triggers a reload. Drain (v0.10-P05) works by writing a config with the backend'sweight=0or removing it, triggering Traefik to stop routing. The v0.8 codebase has no Traefik integration at all. Gated by grill C-10 (Traefik config atomicity protocol: tmpfile+fsync+rename + malformed-config hold-last-good verified). - Rationale: Traefik is net-new and load-bearing. §23 mentions it in R-006/P02/P05 but never specifies config generation or reload mechanism.
- Proposed REQ ID: REQ-077
- Proposed phase placement: v0.9-P02 (Service block + checks)
- Confidence: 0.80
- Accept/Defer: accept
I-B-006 — Runtime abstraction interface (5 backends: wasm/podman/process/pve-vm/pve-ct)
- Tier: backend-enriched
- Description: v0.8's
internal/engine/executor.go(211 LOC) isos/execonly. R-004/R-007 require 5 runtime backends. §23 puts this in v0.9-P07. Proposal: aRuntimeinterface ininternal/runtime/:Prepare(ctx, spec, node) (*Alloc, error),Start(ctx, alloc) (pid/unit, error),Stop(ctx, alloc) error,Status(ctx, alloc) (State, error). Implementations:processRuntime(wraps existingexecutor.go— directly reusable),wasmRuntime(wasmtime via CLI SSH exec),podmanRuntime(podman runvia SSH),pveVMRuntime(qm create/qm startvia v0.8proxmoxSSH session),pveCTRuntime(pct create/pct start). Each registered in aruntimeRegistrykeyed by theruntime:frontmatter value. R-004 (migration with runtime change) means theAlloccarries aruntimefield that can change on migration —Preparere-runs with the new runtime. - Rationale: The 5 backends are the largest net-new implementation surface. §23 lists them as one phase (P07) but under-specifies the interface contract. The existing
executor.gois a good starting point for theprocessRuntimeadapter. - Proposed REQ ID: REQ-078
- Proposed phase placement: v0.9-P07a/P07b/P07c (split per grill PC-10)
- Confidence: 0.82
- Accept/Defer: accept
I-B-007 — Transaction bundle format and atomicity across N peers
- Tier: backend-enriched
- Description: R-010 requires transactional control-plane updates. §23 puts this in v0.10-P10. Design: a transaction bundle is a tarball containing: (1)
desired-state.json(full desired state for affected namespaces), (2)apply.sh(idempotent apply script), (3)verify.sh(post-apply invariants), (4)rollback.sh(revert to previous state), (5)manifest.sig(signature withcluster/master.key). Atomicity across N peers: the CLI uploads the bundle to the lead; the lead applies to itself first, then fans out to peers via SSH. If any peer fails verify, the lead runsrollback.shon all peers that applied. The bundle is content-addressed (<txn-id> = sha256(desired-state.json)) and stored incluster/txns/<txn-id>/. Drift detection (R-010) compares the last applied bundle's desired-state against the live state (polled via SSHsystemctl show+ file checksums). Gated by grill C-09 (orca-pull.sh failure contract: idempotent re-run, bounded retry, deterministic state, structured syslog). - Rationale: Multi-peer atomicity is the hardest part of R-010. §23 says "ArgoCD-style" but ArgoCD is Kubernetes-native; the SSH-push model needs a custom bundle format.
- Proposed REQ ID: REQ-079
- Proposed phase placement: v0.10-P10 (transactional plane) — designed in v0.9-P00
- Confidence: 0.76
- Accept/Defer: accept
I-B-008 — Master key management and HKDF-SHA256 per-line .env.secrets encryption
- Tier: backend-enriched
- Description: R-011 specifies
.env.secretswith AES-256-GCM, per-line nonce, master key atcluster/master.key. §23 puts this in v0.10-P03. Design: (1)cluster/master.keyis a 32-byte random key generated byorca init(extend v0.8internal/security/ca.go'sWriteAtomicpattern for the file write). (2) Each line of.env.secretsisbase64(nonce || ciphertext || tag)wherenonce = random(12 bytes)andciphertext = AES-256-GCM(plaintext, key=master.key, nonce, aad=line-number). (3) The AAD is the 1-indexed line number to prevent line-swap attacks. (4) Decryption reads the master key, iterates lines, decrypts with AAD. (5)orca secrets set <ns> <key> <value>appends an encrypted line;orca secrets get <ns> <key>decrypts and prints (redacted by default,--revealto show). (6) The v0.8internal/security/redact.go(103 LOC) is directly reusable for redaction. HKDF-SHA256 derives per-namespace sub-keys from the master key (HKDF-SHA256(master, info=<ns>)) so compromising one namespace's key doesn't compromise others — but the master key is the root of trust. Gated by grill C-19 (threat model for master.key passphrase-less posture). - Rationale: R-011 is precise about the crypto but §23 doesn't specify key derivation, AAD, or CLI surface. The existing
redact.goandWriteAtomicare reusable. - Proposed REQ ID: REQ-080
- Proposed phase placement: v0.10-P03 (secrets subsystem)
- Confidence: 0.84
- Accept/Defer: accept
I-B-009 — Syncthing config rendering and folder-ID content-addressing
- Tier: backend-enriched
- Description: R-005 requires storage replication via per-namespace Syncthing. §23 puts this in v0.9-P09. Design: (1) each namespace gets a Syncthing folder
orca-<ns>with a content-addressed folder ID (sha256(ns + master-key-fingerprint)). (2) The CLI rendersconfig.xmlfor each peer's Syncthing instance, including the folder, devices (all peers in the namespace), and the path (<ns>/alloc/<alloc-id>/). (3) Syncthing runs as a systemd unit (emitted by the systemd emitter, I-B-002). (4) The CLI discovers peers viacluster/peers/and adds their Syncthing device IDs (each peer's Syncthing generates its own device key on first run, reported back via SSH). (5) R-005 says "a Service's count replicas share one runtime block" — the Syncthing folder is shared across the Service's alloc instances so all replicas see the same data. Migration (R-004) works because the new node joins the Syncthing folder and syncs before the workload starts. Gated by grill C-02 (Syncthing feasibility spike) and C-14 (deterministic conflict-resolution policy + forced-divergence integration test). - Rationale: Syncthing is net-new. §23 lists it in P09 but doesn't specify config rendering, folder-ID scheme, or device discovery.
- Proposed REQ ID: REQ-081
- Proposed phase placement: v0.9-P09 (storage replication) — spike in v0.9-P00
- Confidence: 0.72
- Accept/Defer: accept
I-B-010 — Namespace inheritance resolver algorithm
- Tier: backend-enriched
- Description: v0.9-P0a2 requires a "parent walker, cycle detection" for namespace inheritance. Each
ns.mdhas aparent:field in frontmatter. The resolver walks up the parent chain, merging inherited values (constraints, env, runtime defaults). Cycle detection: DFS with a visited set; if a namespace is revisited, return a cycle error. The resolver returns a flattenedResolvedNamespacestruct. The_defaults/namespace is the implicit root (always exists, has no parent). Inheritance semantics: child overrides parent for scalar fields; arrays (e.g., constraints) are unioned (child adds to parent, not replaces). The resolver is pure (no I/O) — it takes a map ofnsName → *NSConfigand returnsnsName → *ResolvedNS. This makes it trivially testable. - Rationale: §23 mentions "parent walker, cycle detection" but not the merge semantics (override vs union) or the resolver's purity for testing. Getting merge semantics wrong breaks constraint inheritance (P05).
- Proposed REQ ID: REQ-082
- Proposed phase placement: v0.9-P0a2 (namespace CRUD + inheritance)
- Confidence: 0.86
- Accept/Defer: accept
I-B-011 — Bin-packing scheduler redesign (CLI-side, runtime-compatibility scoring)
- Tier: backend-enriched
- Description: v0.8's
internal/engine/scheduler.go(117 LOC) does best-fit bin-packing by CPU+memory. The v0.9 scheduler must: (1) run CLI-side (not on a daemon), (2) score nodes by runtime compatibility (a wasm workload can only go to a node with wasmtime installed; a pve-vm workload can only go to Proxmox nodes), (3) respect constraints/affinity (CEL over node attributes, P05), (4) handle the 3 kinds differently (Job = one-shot, Service = count replicas spread across nodes, DaemonSet = one per node). The existingscheduler.gois a good skeleton but the scoring function changes entirely. Proposal:Score(node, workload) (score int, fits bool)wherefitschecks runtime compatibility + constraints, andscoreis the bin-packing score (most free capacity = highest score). For Services, the scheduler pickscountdistinct nodes (anti-affinity by default). For DaemonSets, it picks all matching nodes. - Rationale: The scheduler moves from daemon-side to CLI-side (R-001) and gains runtime-awareness. §23 scatters this across P05 (constraints), P06 (task groups), P07 (runtime), P10 (migration) but never designs the scheduler itself.
- Proposed REQ ID: REQ-083
- Proposed phase placement: v0.9-P05 (constraints & affinity — scheduler needs constraints to be meaningful) — skeleton in P0c
- Confidence: 0.80
- Accept/Defer: accept
I-B-012 — orca job lint category-driven lint engine design
- Tier: backend-enriched
- Description: v0.10-P11 requires
orca job lintwith--explain. Design: aLinterthat takes a*WorkloadSpecand runs a series ofRulechecks, each returning aFinding{Category, Severity, Message, Explanation}. Categories:schema(missing required fields),runtime(incompatible runtime+constraint),security(missing SVID, plaintext secret in env),migration(missing storage replication for a migratable service),best-practice(no health check on a Service).--explainprints the rationale for each finding. Rules are registered in aruleRegistryand individually testable. The linter is pure (no I/O) — it checks the spec against static rules, not live cluster state (that'sorca job verify, P12). - Rationale: §23 puts this in P11 but only says "category-driven." The rule interface and category taxonomy are unspecified.
- Proposed REQ ID: REQ-084
- Proposed phase placement: v0.10-P11 (orca job lint)
- Confidence: 0.78
- Accept/Defer: accept
Tier 3 — Cross-Cutting (Risk & Multi-Phase)
I-C-001 — v0.8→v1.0 migration ordering: daemon deprecation vs. new model rollout
- Tier: cross-cutting
- Description: The PRD §24 covers data migration but not binary/daemon deprecation ordering. The risk: v0.9 builds the new Markdown+kinds+runtime+SSH-push model, but v0.8 daemons are still running on peers. If v0.9 ships the new
orca job run(Markdown) while the old daemon is still the execution engine, there's a split-brain: new specs can't run on the old daemon. Ordering proposal: (1) v0.9 ships the new parser + kinds + runtime + SSH-push alongside the old daemon (dual-write window); (2)orca job runin v0.9 uses the new SSH-push path if the spec is.mdand the old daemon path if.hcl; (3) v0.10-P05 (drain) stops the old daemons; (4) v0.10-P14 (migration) converts remaining.hclspecs to.mdand removes the daemon. The dual-write window means v0.9 is not a clean break — it's a compatibility milestone. This must be explicit in the plan or the v0.9 phases will assume the daemon is gone. - Rationale: Single largest risk in the re-architecture. §23 implicitly assumes v0.9 builds the new model in isolation, but existing deployments have running daemons. Getting the ordering wrong means either (a) v0.9 can't be tested against real deployments, or (b) workloads are orphaned when the daemon is removed.
- Proposed REQ ID: REQ-085
- Proposed phase placement: spans v0.9-P00 through v0.10-P14 — the ordering decision must be made in v0.9-P00
- Confidence: 0.88
- Accept/Defer: accept (most important idea in this report)
I-C-002 — "No orca on server" enforcement (doctor post-migration invariant check)
- Tier: cross-cutting
- Description: R-001 is an invariant: "no orca Go binary on any server." §23 v0.10-P14 says "post-invariant checks" but doesn't specify them.
orca doctormust gain adoctor no-orca-on-servercheck that SSHs to each peer and verifies: (1) noorcabinary in PATH (ssh peer which orcareturns nothing), (2) noorcasystemd service (ssh peer systemctl list-units 'orca*'returns empty), (3) noorcaprocess (ssh peer pgrep -x orcareturns empty), (4) no/etc/orca/directory. This check must run after v0.10-P05 (drain) and before v0.10-P16 (ship). The v0.8internal/proxmox/bootstrap.goalready has the SSH session infrastructure (sessionRunnerseam) — directly reusable for the doctor check. - Rationale: R-001 is a hard invariant but §23 doesn't enforce it post-migration. Without this check, a failed migration could leave orphaned daemons that cause split-brain.
- Proposed REQ ID: REQ-086
- Proposed phase placement: v0.10-P14c (mixed-version tolerance)
- Confidence: 0.82
- Accept/Defer: accept
I-C-003 — Test infrastructure: hermetic 3-linux + 1-proxmox cluster pipeline
- Tier: cross-cutting
- Description: §23 v0.10-P08 requires "hermetic CoreCI integration pipeline." The PRD §26.E mentions 3 linux + 1 proxmox. This is net-new test infra with zero current implementation. Design: (1) a
test/integration/directory with adocker-compose.ymlorvagrantsetup that creates 4 containers/VMs (3 linux + 1 proxmox-simulated); (2) a Go test harness that SSHes to each, runs the CLI, and asserts end-to-end workflows (namespace create → workload submit → migrate → drain); (3) the proxmox node is simulated via a mockpct/qmscript (the v0.8proxmoxpackage already has asessionRunnerseam for testability — extend it). The integration tests run in CoreCI on every milestone merge. The v0.8 e2e tests (bootstrapE2ESetupinbootstrap_test.go) use an in-process SSH server — this is the foundation but needs to scale to 4 nodes. - Rationale: §23 assumes the infra exists but doesn't design it. devops-engineer persona should be reactivated. Without hermetic infra, the integration tests can't run in CI.
- Proposed REQ ID: REQ-087
- Proposed phase placement: v0.10-P08 (integration tests) — harness bootstrapped in v0.9-P00
- Confidence: 0.80
- Accept/Defer: accept
I-C-004 — Security-engineer + network-engineer persona reactivation for new attack surfaces
- Tier: cross-cutting
- Description: The config.json has
security-engineerandnetwork-engineerdormant. The re-architecture introduces step-ca (PKI), Traefik (edge proxy), Syncthing (P2P file sync), wasmtime (sandbox), podman (container runtime) — all new attack surfaces. AD-010 (step-ca rejection) is reversed. The v0.8 security posture (internal CA, mTLS daemon-to-daemon) is replaced by (step-ca, SSH-push, Traefik mTLS). The security-engineer persona must be reactivated to review: (1) step-ca provisioner model (the CLI holds the provisioner password — is that incluster/master.keyor a separate secret?), (2) SSH-push blast radius (compromised CLI key = full cluster), (3) Traefik as the new edge (DoS, config injection), (4).env.secretscrypto (I-B-008). The network-engineer persona must review: (1) socket-based service exposure (R-007), (2) Syncthing P2P ports, (3) Traefik routing. §23 doesn't mention persona reactivation. - Rationale: config.json explicitly notes the re-architecture "should reactivate security-engineer and network-engineer." Cross-cutting review concern, not a single phase.
- Proposed REQ ID: REQ-088
- Proposed phase placement: spans v0.9 through v0.10 — reactivation in v0.9-P00, review at v0.10-P15.5 (threat model) and v0.10-P16 (final audit)
- Confidence: 0.84
- Accept/Defer: accept
I-C-005 — Documentation rewrite: ARCHITECTURE.md, PROJECT.md, README, AD-010 supersession
- Tier: cross-cutting
- Description: All three docs describe the OLD architecture.
ARCHITECTURE.md(640 lines) describes the daemon layer, mTLS transport, internal CA, HCL jobspec — all deprecated.PROJECT.md(30k chars) has D-001..D-010 decisions, several now superseded.README.mdhas the v0.8 quickstart. AD-010 (step-ca rejection) must be explicitly superseded by D-101 with a dated rationale reversal. The anti-patterns section inARCHITECTURE.md:471-484lists "No external PKI" — now reversed. Proposal: (1) in v0.9-P00, add a "v0.9 Architecture (Supersedes v0.8)" section to ARCHITECTURE.md with the new 4-layer model; (2) mark the old sections as "v0.8 (deprecated)" with banners; (3) add a "Superseded Decisions" table (AD-009, AD-010 reversed by D-101; AD-007 HCL demoted by R-013); (4) in v0.10-P15, rewrite README quickstart for the newcurl | sh+orca init+orca ns createflow. - Rationale: The docs are the first thing new contributors read. Leaving v0.8 docs as canonical during v0.9 development causes confusion. §23 mentions README in P15 but not ARCHITECTURE.md/PROJECT.md.
- Proposed REQ ID: REQ-089
- Proposed phase placement: v0.9-P00 (banners + supersession table) + v0.10-P15 (README quickstart) + v0.10-P16 (final review)
- Confidence: 0.82
- Accept/Defer: accept
I-C-006 — Dual-write window: can v0.9 ship new parser while old daemon runs?
- Tier: cross-cutting
- Description: Focused version of I-C-001. The specific question: in v0.9, when the new Markdown parser + kinds + SSH-push are shipped, can they coexist with v0.8 daemons still running on peers? The answer depends on whether
orca job run <spec.md>uses the new SSH-push path (bypassing the daemon entirely) or routes through the old daemon. If it bypasses, the daemon is irrelevant for new specs but still serves old.hclspecs. If it routes through, the daemon can't handle.mdspecs. Proposal: v0.9orca job rundispatches on extension (.md→SSH-push new path,.hcl→old daemon path) via the parser dispatcher (I-M-004). This is a dual-write window where both paths coexist. The daemon is not removed until v0.10-P05 (drain). The risk: if a.mdworkload and a.hclworkload target the same node, the SSH-push path writes systemd units directly while the daemon also manages units — they can conflict. Mitigation: the SSH-push path writes to a separate systemd unit namespace (orca-v1-<alloc>.service) while the daemon usesorca-<job>.service. No unit name overlap = no conflict. - Rationale: Operational feasibility question for v0.9. §23 doesn't address it. If the answer is "no dual-write, daemon must be removed first," then v0.9 can't be tested incrementally and must ship as a big-bang — much higher risk.
- Proposed REQ ID: REQ-090
- Proposed phase placement: v0.9-P00 (decision before any v0.9 execution phase)
- Confidence: 0.86
- Accept/Defer: accept
Summary Table
| ID | Tier | Title | REQ | Phase | Conf | Accept |
|---|---|---|---|---|---|---|
| I-M-001 | M | orca daemon deprecation path |
REQ-061 | v0.10-P14 (warn v0.9-P0X) | 0.82 | accept |
| I-M-002 | M | Coverage follow-ups to 70% | REQ-062 | v0.9-P0X + each new pkg | 0.88 | accept |
| I-M-003 | M | known_hosts flock concurrency | REQ-063 | v0.9-P0a1 | 0.74 | accept |
| I-M-004 | M | HCL→Markdown jobspec adapter | REQ-064 | v0.9-P0b | 0.85 | accept |
| I-M-005 | M | doctor --legacy-paths detection |
REQ-065 | v0.10-P14c | 0.80 | accept |
| I-M-006 | M | Legacy CA state migration to step-ca | REQ-066 | v0.10-P14a | 0.70 | accept |
| I-M-007 | M | Fuzz harness for Markdown parser | REQ-067 | v0.9-P0b | 0.78 | accept |
| I-M-008 | M | Deprecation warnings on CLI subcommands | REQ-068 | v0.9-P0X + v0.10-P13 | 0.72 | accept |
| I-M-009 | M | HCL config demotion via adapter | REQ-069 | v0.9-P0a1 | 0.76 | accept |
| I-M-010 | M | certpaths → multi-namespace path resolver | REQ-070 | v0.9-P0a1 | 0.84 | accept |
| I-M-011 | M | store schema: per-namespace DBs | REQ-071 | v0.9-P0a1 + v0.10-P06 | 0.80 | accept |
| I-M-012 | M | transport deletion + SSH-push package | REQ-072 | v0.9-P00 (delete v0.10-P14) | 0.68 | accept |
| I-B-001 | B | SSH-push transport layer design | REQ-073 | v0.9-P01 | 0.86 | accept |
| I-B-002 | B | Emitter template system (Layer 4) | REQ-074 | v0.9-P0c | 0.82 | accept |
| I-B-003 | B | Lead applier execution model | REQ-075 | v0.10-P10 (design v0.9-P00) | 0.78 | accept |
| I-B-004 | B | step-ca integration | REQ-076 | v0.9-P07 + v0.10-P02 | 0.74 | accept |
| I-B-005 | B | Traefik dynamic config + atomic reload | REQ-077 | v0.9-P02 | 0.80 | accept |
| I-B-006 | B | Runtime abstraction (5 backends) | REQ-078 | v0.9-P07a/b/c | 0.82 | accept |
| I-B-007 | B | Transaction bundle + N-peer atomicity | REQ-079 | v0.10-P10 (design v0.9-P00) | 0.76 | accept |
| I-B-008 | B | Master key + HKDF per-line encryption | REQ-080 | v0.10-P03 | 0.84 | accept |
| I-B-009 | B | Syncthing config + folder-ID | REQ-081 | v0.9-P09 | 0.72 | accept |
| I-B-010 | B | Namespace inheritance resolver | REQ-082 | v0.9-P0a2 | 0.86 | accept |
| I-B-011 | B | CLI-side scheduler redesign | REQ-083 | v0.9-P05 (skeleton P0c) | 0.80 | accept |
| I-B-012 | B | orca job lint category-driven engine |
REQ-084 | v0.10-P11 | 0.78 | accept |
| I-C-001 | C | v0.8→v1.0 migration ordering | REQ-085 | spans v0.9-P00→v0.10-P14 | 0.88 | accept |
| I-C-002 | C | "No orca on server" enforcement | REQ-086 | v0.10-P14c | 0.82 | accept |
| I-C-003 | C | Hermetic test infra (3 linux + 1 pve) | REQ-087 | v0.10-P08 (bootstrap v0.9-P00) | 0.80 | accept |
| I-C-004 | C | security/network persona reactivation | REQ-088 | spans v0.9→v0.10-P16 | 0.84 | accept |
| I-C-005 | C | Docs rewrite + AD-010 supersession | REQ-089 | v0.9-P00 + v0.10-P15/P16 | 0.82 | accept |
| I-C-006 | C | Dual-write window decision | REQ-090 | v0.9-P00 | 0.86 | accept |
Phase Reordering / Addition Flags (against PRD §23)
- I-C-001 / I-C-006 (dual-write + migration ordering) — require a decision in v0.9-P00 (before any execution phase). Recommendation: add v0.9-P00 deprecation/migration-ordering pre-phase. Most important structural addition.
- I-M-010 / I-M-011 / I-M-009 / I-M-003 — all land in v0.9-P0a. P0a may be overloaded. Recommendation: split P0a into P0a1 (path/layout resolver + config demotion) and P0a2 (namespace CRUD + inheritance). Path resolver is prerequisite for everything; highest blast radius.
- I-B-001 (SSH-push transport) — §23 v0.9-P01 needs SSH-push. The design is a prerequisite. Recommendation: SSH-push design in P0a1, not deferred to P01.
- I-B-002 (emitter template system) — should be designed with the schemas (P0c). Recommendation: expand P0c to "schemas + emitter interface."
- I-B-003 (lead applier model) — bundle format + lead applier model must be designed in v0.9 so the emitter can produce bundle-compatible output. Recommendation: design spike in v0.9-P00.
- I-C-003 (test infra) — hermetic cluster harness should be bootstrapped in v0.9-P00 so every v0.9 phase can run integration tests. Recommendation: bootstrap in v0.9-P00, expand in v0.10-P08.
- I-C-004 / I-C-005 (persona reactivation + docs) — span the whole milestone. Recommendation: fold persona reviews into v0.9-P00 and v0.10-P16; fold doc banners into v0.9-P00.
Cross-Reference Against Existing Decisions
- AD-009 (Internal CA, no external PKI) — Superseded by D-101 (step-ca). I-B-004, I-M-006 implement the reversal.
- AD-010 (Roll-our-own CA) — Superseded by D-101. I-C-005 documents the supersession. No re-litigation — the PRD has decided; the override justification records the evidence basis.
- AD-007 (HCL for job specs) — Demoted by R-013 (Markdown canonical, HCL accepted). I-M-004 implements the adapter. Not a full reversal — HCL still parses.
- AD-001 (Single binary with subcommands) — Still holds. The CLI is the single binary; no orca on servers (R-001) refines this.
- AD-015 (Best-fit bin-packing) — Extended, not reversed. I-B-011 adds runtime-compatibility scoring.
- D-035 (TOFU host-key) — Still holds for non-Proxmox peers. I-M-003 hardens the concurrency. I-B-001 reuses
TOFUHostKeyCallback. - D-046 (key-reset is local-only) — Still holds. I-M-003 adds the lock.
- D-047 (tiered coverage floor) — Extended by I-M-002 to cover new packages.
No accepted idea re-litigates a settled decision. All reversals (AD-009, AD-010, SPIFFE, no-container, no-multi-tenancy, HCL-canonical, daemon-on-every-node) are explicitly mandated by the PRD and justified by the recorded override justification.
Final Notes
- Total ideas: 30 (12 mechanical, 12 backend-enriched, 6 cross-cutting).
- Highest-confidence, highest-impact: I-C-001 (migration ordering, 0.88) and I-C-006 (dual-write window, 0.86) — these shape the entire v0.9 execution strategy.
- Highest-blast-radius mechanical: I-M-010 (path resolver, 0.84) — touches every adaptable package.
- Most under-specified by PRD: I-B-003 (lead applier execution model, 0.78) — R-001 + R-010 create a tension the PRD doesn't resolve.