orca/.ciagent/GRILL_v0.11.md

Grill: v0.11 Production Hardening — Phase 0 Adversarial Review

Status: PROCEED-WITH-CONDITIONS. The v0.11 plan is sound; 6 binding conditions (C-23…C-28) gate specific phases. The plan adopts R-017…R-020 and D-215…D-237 from 5 research docs with operator decisions Q1=A, Q2=C, Q3=A, Q4=A, Q5=A. The grill reviewed the plan adversarially across the same 9 axes as GRILL_v0.9 (vision, feasibility, scope, risk, security, operational, cost, competitive, exit).

Forcing questions + verdicts

FQ1 — R-020 deadlock with --force + per-ns scoping

Question: With --force + per-namespace scoping (Q4=A), can a single drifted peer still block a cluster-wide txn (e.g., namespace creation)? If yes, is the --force escape hatch documented in C-09's failure contract?

Verdict: PARTIAL-BLOCK remains for cluster-wide txns. A namespace creation txn touches all peers (the new namespace dir is created on every peer). If one peer is drifted, the pre-flight gate refuses the txn cluster-wide. --force overrides this, but --force on a namespace-creation txn is risky (it forces the new namespace onto a drifted peer without reconciling the drift first).

Binding condition C-23: orca-pull.sh (C-09) must distinguish cluster-wide txns from namespace-scoped txns. Cluster-wide txns require --force with an explicit --i-understand-the-risk confirmation (or --yes for non-interactive). Namespace-scoped txns use per-ns scoping (drifted peer in ns-A doesn't block ns-B). Gate: P10.

Confidence: 0.88

FQ2 — P10 sizing (txn plane + drift detection in one phase)

Question: P10 now absorbs drift detection (~500 LoC Go + 150 LoC bash + systemd units), the largest single phase. Is this a vertical slice that can ship atomically, or does it need splitting (P10a txn plane, P10b drift)?

Verdict: SPLIT RECOMMENDED. P10 has 13 tasks spanning two distinct subsystems: (1) the transactional plane (T1-T2: txn bundle render, SCP, apply, C-09 failure contract) and (2) drift detection (T3-T13: internal/drift/, Path unit emitter, notify/remediate scripts, cadence config, pre-flight gate, orca user, NFS detection, job restart). The txn plane is a prerequisite for drift detection (T3's Aggregate reads applied txn manifests), so the split is clean: P10a (txn plane, T1-T2) ships first, P10b (drift detection, T3-T13) ships after P10a.

Binding condition C-24: Split P10 into P10a (transactional plane, REQ-075/079, C-09) and P10b (drift detection, R-018/R-019/R-020, REQ-103..113). P10a ships first; P10b depends on P10a. Tags: P10a v0.10.12, P10b v0.10.13. All subsequent phase tags shift by 1 (P11→v0.10.14, …, P16→v0.10.22). Phase count: 23 → 24.

Confidence: 0.92

FQ3 — Ingress default migration path (C1)

Question: Existing v0.9/v0.10 clusters run Traefik on :443. R-017 makes 127.0.0.1:8443 + nft the default. What's the upgrade path? Does orca upgrade (Q2=C) handle the binding cutover, or is it a manual operator step?

Verdict: UPGRADE HANDLES IT, but with a safety check. orca upgrade (REQ-115, P14a) is the thin wrapper (C2=a) that handles the Traefik binding cutover. The cutover is: (1) emit new Traefik static config with 127.0.0.1:8443, (2) emit /etc/nftables.d/orca.nft with DNAT, (3) systemctl reload traefik + nft -f, (4) verify curl :443 still routes. If step 4 fails, rollback to :443 + remove nft rules.

Binding condition C-25: orca upgrade (REQ-115) must include a post-cutover verification step (curl -k https://localhost:443/ returns 200 from Traefik) with automatic rollback on failure. Document the rollback procedure in docs/ingress.md. Gate: P14a.

Confidence: 0.90

FQ4 — Scope ceiling (LoC vs phase count)

Question: v0.11 stays at 23 phases (now 24 with C-24), but P09/P10 (now P10a/P10b)/P15.5 grow substantially. Is the phase count the right ceiling, or should there be a LoC/effort ceiling per phase?

Verdict: LOOSE LoC CEILING. Phase count is a proxy for effort, but P10b (drift detection) is ~650 LoC across Go + bash + systemd — at the upper end of what a single-phase vertical slice can handle. The grill recommends a soft LoC ceiling of ~800 LoC per phase (Go + bash + config), with splitting required above ~1200 LoC.

Binding condition C-26: Per-phase LoC soft ceiling: ~800 LoC (Go + bash + config). Split required above ~1200 LoC. P10b (~650 LoC) is within the soft ceiling; P15.5 (~400 LoC: nft emitter 200 + doctor mTLS 100 + threat model doc) is within. No action required for v0.11; recorded for future milestones. No gate.

Confidence: 0.85

FQ5 — orca system user on peers (operational impact)

Question: Creating a system user on every peer is a new operational requirement. Does this break any existing v0.9/v0.10 deployment that runs as root or as an existing service account?

Verdict: NO BREAK for existing deployments; NEW requirement for drift detection. The orca system user (REQ-111) is created at peer setup (orca node join / peer-setup script). Existing v0.9/v0.10 peers don't have the orca user, so drift detection's systemd Path units (which run as User=orca) won't start until the user is created. orca upgrade (REQ-115) must create the orca user on existing peers as part of the v0.11 migration.

Binding condition C-27: orca upgrade (REQ-115, P14a) must create the orca system user on existing peers (useradd -r orca idempotent) before P10b's drift detection can function. Document this as a migration prerequisite. Gate: P14a.

Confidence: 0.91

FQ6 — P15.5 is now a mega-phase (threat model + ingress + doctor mTLS)

Question: P15.5 was originally "threat model + security review" (C-19). It now absorbs ingress hybrid (R-017; REQ-099..102, ~400 LoC) + orca doctor mTLS (REQ-118). Is this too much for one phase?

Verdict: MANAGEABLE but at the ceiling. P15.5 is now ~500 LoC (nft emitter 200 + doctor mTLS 100 + threat model doc + tests). The ingress hybrid and threat model are related (both are security-hardening), so keeping them together is defensible. The orca doctor mTLS (REQ-118) is small and reuses P01/P01.5 infrastructure. The grill recommends keeping P15.5 as one phase but splitting the work into two sub-waves within the phase: (1) ingress hybrid + doctor nft, (2) threat model + doctor mTLS.

Binding condition C-28: P15.5 commits in two sub-waves: (1) ingress hybrid (REQ-099..102) + orca doctor nft (REQ-101), (2) threat model (C-19) + orca doctor mTLS (REQ-118). Both ship under the same phase tag (v0.10.20). No new phase; internal ordering only.

Confidence: 0.89

Binding conditions summary

ID	Condition	Gate	Verification
C-23	orca-pull.sh distinguishes cluster-wide vs namespace-scoped txns; cluster-wide requires --force + --i-understand-the-risk (or --yes)	P10a	Test: cluster-wide txn refused without --force; ns-scoped txn blocks only the drifted ns
C-24	Split P10 into P10a (txn plane, REQ-075/079, C-09) + P10b (drift detection, R-018/R-019/R-020, REQ-103..113); P10b depends on P10a; tags shift by 1	P10a→P10b	Plan shows P10a + P10b as separate phases; P10b tasks reference P10a txn manifests
C-25	orca upgrade (REQ-115) includes post-cutover verification (curl -k https://localhost:443/ returns 200) with automatic rollback on failure; rollback documented in docs/ingress.md	P14a	Test: cutover succeeds → 200; cutover fails → rollback to :443
C-26	Per-phase LoC soft ceiling: ~800 LoC (Go + bash + config); split required above ~1200 LoC	(no gate)	Recorded for future milestones
C-27	orca upgrade (REQ-115) creates orca system user on existing peers before P10b drift detection can function	P14a	Test: existing peer without orca user → orca upgrade creates it → drift detection starts
C-28	P15.5 commits in two sub-waves: (1) ingress hybrid + doctor nft, (2) threat model + doctor mTLS; same phase tag	P15.5	Commits show two sub-waves; both under v0.10.20

Phase challenge summary

PC	Phase	Challenge	Resolution
PC-11	P10a/P10b	Txn plane + drift detection too large for one phase	Split per C-24; P10a ships first, P10b depends on it
PC-12	P15.5	Mega-phase (threat model + ingress + doctor mTLS)	Keep as one phase; two sub-waves per C-28
PC-13	P14a	orca upgrade handles 3 migrations (data + binding + orca user)	All three land in P14a per C-25, C-27; thin wrapper (C2=a)
PC-14	P09	Aggregator extension depends on P10b drift detection	P09 in Wave 6 (after Wave 5 P10b); aggregator extension (REQ-107) only works once drift events exist

Overall verdict

PROCEED-WITH-CONDITIONS. The v0.11 plan is sound. 6 binding conditions (C-23…C-28) gate specific phases. The plan grows from 23 → 24 phases (C-24 splits P10 into P10a/P10b). All other phases are unchanged in count; their scope expands per the research folding (Q2=C, Q3=A).

The grill's confidence in the v0.11 plan is high (avg 0.89 across FQs). The primary risks (P10 sizing, R-020 deadlock, ingress migration) are all gated with verifiable conditions.