docs(P00): grill v0.14 — red-team review, RETHINK verdict
Adversarial review of PLAN_v0.14.md across 9 axes. Verdict: RETHINK (confidence 0.45). The research foundation is strong but the plan diverges from it and from the codebase in load-bearing ways. 4 binding decisions (G-001..G-004): - G-001: migration number 0009 (not 0007 — already taken by certs) - G-002: omit :Z flag (contradicts CLARIFY D-258 + REQ-172) - G-003: wire real mTLS now (scope expansion — plan has no such phase) - G-004: P2 T6 must remove legacy systemd unit + binary on upgrade 12 binding conditions (C-50..C-61) + 14 phase challenges (PC-01..14). 1 escalation (E-001): G-003 mTLS direction undetermined in plan. Key findings: - F1.1: migration 0007 collision (BLOCKER) - F1.2: certpaths.CAPath() does not exist (compile BLOCKER) - F2.2: REQ-100 traefik-on-public-ip opt-out regressed by baked image - F5.1: no v0.13 -> v0.14 upgrade path (BLOCKER) - F7.2: podman pull violates R-001 offline-first ---ci--- status: grill milestone: v0.14 binding-decisions: - G-001: migration 0009_ingress_mode.sql (not 0007) - G-002: omit :Z, use :ro on both mounts - G-003: wire real mTLS now (scope expansion, plan must add phase) - G-004: P2 T6 must remove legacy systemd unit + binary on upgrade escalations: - E-001: G-003 mTLS direction chosen but plan has no phase for it (conf 0.55) verdict: rethink confidence: 0.45
This commit is contained in:
Jon Chery
@@ -0,0 +1,225 @@
|
||||
# GRILL v0.14: Ingress Bootstrap Completeness — Red-Team Review
|
||||
|
||||
**Date**: 2026-08-10
|
||||
**Reviewer**: ci-griller (adversarial)
|
||||
**Subject**: PLAN_v0.14.md (9 phases, P0–P8)
|
||||
**Confidence in plan as-written**: **0.45 — RETHINK** (was Proceed-eligible until the mTLS scope expansion was chosen)
|
||||
**Verdict**: **RETHINK** — the plan is technically grounded in strong research but contains (a) one outright correctness defect that will break execution, (b) two requirements/plan contradictions that ship broken config, (c) one scope expansion chosen during this grill that adds a 10th phase the plan does not contain, and (d) one regression of a Completed requirement. The research is the strongest artifact; the plan diverges from it in load-bearing ways.
|
||||
|
||||
---
|
||||
|
||||
## How this grill was conducted
|
||||
|
||||
Every plan claim was checked against the actual codebase (`internal/traefik/install.go`, `internal/emitter/nft.go`, `internal/emitter/traefik.go`, `internal/cli/init.go`, `internal/cli/upgrade.go`, `internal/cli/doctor_nft.go`, `internal/proxmox/bootstrap.go`, `internal/linux/bootstrap.go`, `internal/store/migrations/`, `internal/certpaths/`, `internal/sshpush/`, `scripts/release.sh`, `.coreci.yml`, `Dockerfile`, git tags, git log). Findings cite file:line. Four binding questions were asked interactively; answers recorded as G-001..G-004 below.
|
||||
|
||||
---
|
||||
|
||||
## Per-Axis Findings
|
||||
|
||||
### Axis 1 — Feasibility
|
||||
|
||||
**Finding F1.1 (BLOCKER — migration number collision).** PLAN P5 T3 (line 194) specifies "Schema migration 0007: `ALTER TABLE nodes ADD COLUMN ingress_mode TEXT DEFAULT ''`". Migration 0007 **already exists** — `internal/store/migrations/0007_certs_serial_unique.sql` (added v0.7, P1-001). The current head is `0008_audit_tamper_evidence.sql`. The migrator runs files in lexical order and records applied versions in `schema_migrations`. Reusing 0007 will either (a) silently no-op on DBs that already recorded 0007, leaving `ingress_mode` un-added, or (b) break the migration ledger. This is a guaranteed execution-time defect.
|
||||
**Evidence**: `internal/store/migrations/0007_certs_serial_unique.sql`, `internal/store/migrations/0008_audit_tamper_evidence.sql`, `internal/cli/init_test.go:84` (test asserts head = 0008).
|
||||
**Resolved by**: G-001 (use `0009_ingress_mode.sql`).
|
||||
|
||||
**Finding F1.2 (BLOCKER — `certpaths.CAPath()` does not exist).** PLAN P3 T5 step 2 (line 142) says "Push cluster root CA … from `certpaths.CAPath()` if exists". The function `certpaths.CAPath()` does not exist. The real API is `certpaths.CACertPath()` (`internal/certpaths/certpaths.go:35`) and `certpaths.CAKeyPath()` (`:39`). P4 T1 step 2 repeats the same phantom reference. This will not compile.
|
||||
**Evidence**: `internal/certpaths/certpaths.go:35` (`func CACertPath() string`); grep for `CAPath` returns zero matches in `internal/certpaths/`.
|
||||
|
||||
**Finding F1.3 (no `install_test.go` to rewrite).** PLAN P2 T10 (line 112) says "rewrite `internal/traefik/install_test.go`". That file does not exist — `internal/traefik/` contains only `install.go` (glob confirms). The task is "create", not "rewrite". Minor, but signals the plan was written against an imagined codebase shape, not the real one.
|
||||
**Evidence**: `glob internal/traefik/*.go` → only `install.go`.
|
||||
|
||||
**Finding F1.4 (good — fake-SSH harness exists and is adequate).** PLAN P7 T8 claims a "hermetic fake-SSH harness" can assert `pct create` + `podman run` inside LXC. Verified: `internal/proxmox/ssh_session_test.go:23` (`fakeSSHServer`) and `internal/sshpush/transport_test.go:27` implement an in-process SSH server with `runCommand(cmd)` pattern-matching. It can assert the right commands are *sent* (e.g. `pct create … --features nesting=1,keyctl=1,fuse=1`). It cannot actually create an LXC or run podman — but the plan only claims command assertion, which is achievable. The harness is real and reusable.
|
||||
|
||||
**Finding F1.5 (good — callsites verified).** PLAN P2 T3/T4/T5 reference `init.go:254-266`, `linux/bootstrap.go:160-172`, `proxmox/bootstrap.go:250-255`. Verified: `init.go:256` calls `installTraefikLocal()`; `linux/bootstrap.go:170` calls `traefik.InstallRemote("", sshExecFn)`; `proxmox/bootstrap.go:253` calls `traefik.InstallRemote("", runRemote)`. The line numbers are accurate within a few lines. The plan was written against the real callsites.
|
||||
|
||||
### Axis 2 — Scope
|
||||
|
||||
**Finding F2.1 (BLOCKER — mTLS scope expansion chosen, no phase exists).** During this grill (G-003) the operator chose "Wire real mTLS now" over the plan's `tls: {}` default-cert approach. This requires step-ca to mint server certs into `/etc/traefik/dynamic` + dynamic `tls.certificates` + `tls.options.default.clientAuth.caFiles`. **Step-ca server-cert minting was deferred since v0.11 and is not implemented.** The 9-phase plan contains no such phase. The operator then chose (G-004-adjacent) to add a step-ca cert minting phase, growing v0.14 from 9 to 10 phases. The plan as written does not reflect this. Until a P-step for cert minting is added, the plan is incomplete relative to the chosen direction.
|
||||
**Impact**: v0.14 cannot ship real mTLS on its current 9 phases. Either add the phase (10 phases, more risk) or revert to `tls: {}` and defer mTLS to v0.15.
|
||||
|
||||
**Finding F2.2 (regression — `traefik-on-public-ip` opt-out lost).** REQ-100 (Complete, v0.11) established the `--public-binding=traefik-on-public-ip` opt-out: traefik binds `:443`/`:80` directly instead of `127.0.0.1:8443`/`8080` + nft DNAT. The existing `RenderTraefikStaticConfig` (`internal/emitter/traefik.go:273`) implements this via `TraefikStaticOpts.PublicBinding`. The v0.14 plan's baked image (`docker/orca-traefik/traefik.yml`, PLAN lines 33-37) hard-codes `127.0.0.1:8443`/`8080` — there is no opt-out path in the image. Baking the static config into the image freezes out the opt-out mode that v0.11 shipped. This is a regression of a Completed requirement.
|
||||
**Evidence**: `internal/emitter/traefik.go:255-266` (`publicWebSecure`/`publicWeb` switch on `PublicBinding`); `internal/emitter/traefik.go:282-300` (rendered static config); REQUIREMENTS REQ-100 line 218.
|
||||
|
||||
**Finding F2.3 (9 phases is borderline; 10 is too many).** Even before the mTLS expansion, 9 phases for an "ingress bootstrap completeness" milestone is heavy. P5 (proxmox native) and P6 (floating-IP) are the two most complex (LXC creation + podman-in-LXC + apt-get + nft-inside-LXC). They could potentially be merged into one phase with two code paths, since they share `ProvisionIngressLXC` plumbing. With the mTLS expansion, 10 phases is too many for a single milestone — split v0.14 into v0.14a (linux ingress + mTLS) and v0.14b (proxmox ingress).
|
||||
|
||||
**Finding F2.4 (good — research-validated decisions are sound).** The 7 research topics (nft postrouting, pve-firewall priority, LXC features, traefik Dockerfile, pct create syntax, podman restart, SELinux) are well-sourced and the recommended approaches are technically correct. This is the strongest artifact in the v0.14 dossier.
|
||||
|
||||
### Axis 3 — Dependencies
|
||||
|
||||
**Finding F3.1 (good — ordering is correct).** P1 (image) → P2 (reconciler) → P3 (nft+init) → P4 (remote linux) → P5 (proxmox native) → P6 (floating-IP) → P7 (doctor+tests) → P8 (ship). Each phase references the prior phase's output (P3 T6 calls `EnsureTraefikContainerLocal` from P2; P5 T4 step 10 calls `EnsureTraefikContainerRemote` from P2). No phase can ship before its prerequisite.
|
||||
|
||||
**Finding F3.2 (hidden dependency — P2 T8 depends on P1 T1).** P2 T8 (drop `certResolver: orca`) edits `internal/emitter/traefik.go`. P1 T1 bakes the static config. The dynamic config (`traefik.go`) and static config (`docker/orca-traefik/traefik.yml`) must be consistent: if the static config has no `certificatesResolvers.orca` (P1 T1 correctly omits it) but the dynamic config still references `certResolver: orca` (until P2 T8), traefik logs a warning on every reload. The plan orders P1 before P2, so there is a window (P1 shipped, P2 not yet) where the published image + the live dynamic config are inconsistent. This is acceptable only if P1 and P2 ship in quick succession; if P1 stalls, the image is published with a known TLS-config mismatch.
|
||||
|
||||
### Axis 4 — Security
|
||||
|
||||
**Finding F4.1 (good — nft injection guard already present).** `internal/emitter/nft.go:104-130` (`partitionTrustedProbes`) validates every `TrustedProbes` entry as IP/CIDR before rendering (F9 guard). The v0.14 plan adds `DNATTarget` (P3 T1) — this is a NEW string field rendered directly into `dnat to <DNATTarget>:8443`. **The plan does not specify validation of `DNATTarget`.** If `DNATTarget` is user-controllable (via `--floating-ip` or cluster config), an unvalidated value is an nft-syntax injection vector. The existing F9 guard covers `TrustedProbes`; `DNATTarget` needs the same treatment.
|
||||
**Binding**: C-51 (see below).
|
||||
|
||||
**Finding F4.2 (container escape surface — LXC nesting).** `--features nesting=1,keyctl=1,fuse=1` (P5/P6) is the documented requirement for podman-in-LXC, but nesting exposes host procfs/sysfs to the guest (Proxmox docs, RESEARCH Topic 3). This is an accepted tradeoff for container-in-container, but the plan does not document the threat-model acceptance. For a "production hardening" lineage, the ingress LXC is now a privileged-ish surface (nesting+keyctl) running a podman container pulling an image from a registry. The supply chain is: `git.cloudinit.dev/coreci/orca-traefik:<version>` (P1). If the registry is compromised or the tag is re-pushed, the ingress LXC runs attacker code at the host-LXC boundary. The plan has no image-signing/verification step (`podman pull --cert-dir` or cosign).
|
||||
**Binding**: C-52 (see below).
|
||||
|
||||
**Finding F4.3 (nft rules injection via re-apply).** P3 T4 changes `flush table` to `delete table`. RESEARCH Topic 1 establishes that `delete table` on a missing table errors, and the fix is to pre-create the table (`nft add table inet orca-ingress 2>/dev/null || true`) before `nft -f`. P3 T5 step 4 does this. Good. But the re-apply path (P3 T5 step 5, P4 T1 step 5) runs `nft -f` which replaces the whole table — if a concurrent process (pve-firewall, operator) adds rules to `orca-ingress` between the pre-create and the `nft -f`, they are wiped. This is by design (orca owns the table) but should be documented as "orca is the sole owner of `table inet orca-ingress`".
|
||||
|
||||
### Axis 5 — Operational
|
||||
|
||||
**Finding F5.1 (BLOCKER — no upgrade path from v0.13 binary+systemd to v0.14 podman).** P2 T6 (line 104) says "update legacy cutover to pull new image + recreate container instead of sed-ing traefik.yml". But `internal/cli/upgrade.go:244-407` implements a Traefik `:443` → `127.0.0.1:8443` cutover that does `systemctl restart traefik` (line 382) — it restarts the **systemd service**, not a container. P2 removes systemd unit generation. There is **no phase** that: (a) detects the legacy `orca-traefik.service`, (b) stops+disables it, (c) removes `/usr/local/bin/traefik` + `/etc/systemd/system/orca-traefik.service`, (d) then runs `EnsureTraefikContainerLocal`. Without this, upgrading a live v0.13 cluster leaves a dead systemd unit AND a new podman container both trying to bind `127.0.0.1:8443` → port conflict, traefik down.
|
||||
**Resolved by**: G-004 (P2 T6 must remove legacy unit + binary, idempotent, tested).
|
||||
|
||||
**Finding F5.2 (podman/podman-restart.service unavailable).** P2 T2 enables `podman-restart.service`. RESEARCH Topic 6 establishes this is not enabled by default on Ubuntu 24.04 and must be enabled. But the plan does not specify what happens if `podman` is not installed on the target host (a fresh linux node join, P4). `EnsureTraefikContainerLocal`/`Remote` calls `podman pull`/`podman run` — if `podman` is absent, this fails. The plan has no "install podman first" step for the linux topology (P3/P4). For proxmox (P5/P6) the plan installs podman via `apt-get install -y podman conmon crun fuse-overlayfs` inside the LXC (T4 step 4 / T1 step 6). But for linux nodes (P3/P4), there is no podman-install step. This is an under-specification.
|
||||
**Binding**: C-50 (see below).
|
||||
|
||||
**Finding F5.3 (good — `--network host` is correct).** RESEARCH Topic 3 confirms `--network host` inside an LXC binds the LXC's netns, so traefik binds `127.0.0.1:8080/8443` on the LXC loopback and nft on the PVE host DNATs to the LXC IP. This is sound. The `DNATTarget` parameterization (D-262) correctly distinguishes `127.0.0.1` (linux/localhost) from `<lxc-ip>` (proxmox native).
|
||||
|
||||
**Finding F5.4 (reboot persistence chain has a gap).** RESEARCH Topic 6 establishes the chain: Proxmox boot → `--onboot 1` starts LXC → LXC systemd starts → `podman-restart.service` restarts container. P5 T4 step 6 and P6 T1 step 8 enable `podman-restart.service` inside the LXC. But `--onboot 1` starts the LXC **after** the Proxmox host's network is up — if the floating IP is on a bridge that depends on a physical link that's slow to come up, the LXC may start before the bridge is ready, and the floating-IP `eth0` config may fail. The plan does not address LXC-start ordering relative to bridge readiness.
|
||||
|
||||
### Axis 6 — Testing
|
||||
|
||||
**Finding F6.1 (good — harness is real).** `fakeSSHServer` (Axis 1 F1.4) can assert `pct create` with the right `--features` and `net0` args, `apt-get install podman`, `podman run` with `--network host`. P7 T8's claims are achievable.
|
||||
|
||||
**Finding F6.2 (gap — no test for the v0.13→v0.14 upgrade).** G-004 requires the upgrade path to be tested. The plan's P2 Wave 4 (T10) tests the reconciler but not the legacy-removal path. There is no test that: (a) simulates a host with `orca-traefik.service` present, (b) runs the upgrade, (c) asserts the unit is stopped+disabled+removed, (d) asserts the podman container is running. This must be added to P2 T10.
|
||||
|
||||
**Finding F6.3 (gap — no test for DNATTarget validation).** F4.1 identifies `DNATTarget` as an injection vector. P3 T8 (nft_test) asserts `DNATTarget` substitution but does not specify a test for invalid `DNATTarget` values (e.g. `1.2.3.4:8443; flush ruleset`). The existing F9 guard test pattern should be extended.
|
||||
|
||||
**Finding F6.4 (good — doctor_nft extension is incremental).** P3 T7 extends `doctor_nft.go`. The existing file (`internal/cli/doctor_nft.go`) is well-structured with `nftCheckResult` lines; adding postrouting/masquerade/DNATTarget assertions is straightforward.
|
||||
|
||||
### Axis 7 — Performance
|
||||
|
||||
**Finding F7.1 (apt-get install podman on every proxmox join — 30-60s).** CLARIFY D-263 acknowledges "~30-60s to the join time". P5 T4 step 4 and P6 T1 step 6 run `apt-get update && apt-get install -y podman conmon crun fuse-overlayfs nftables` inside the LXC on every join. This is acceptable for a one-time bootstrap but painful if re-run. The reconciler must be idempotent (skip if podman already installed). The plan does not specify an idempotency check for the apt-get step.
|
||||
**Binding**: C-53 (see below).
|
||||
|
||||
**Finding F7.2 (image pull latency).** `podman pull orca-traefik:<tag>` (P2 T1 step 3, P5/P6) pulls from `git.cloudinit.dev/coreci/`. On a proxmox host behind a slow link, this can take 10-30s for a ~150MB traefik image. The plan has no pull-timeout. If the registry is unreachable (offline-first is R-001!), the pull fails and traefik never starts. **R-001 (offline-first) is violated**: the plan depends on a registry pull at bootstrap time. There is no "pre-pull" or "bundle image into the LXC template" fallback.
|
||||
**Binding**: C-54 (see below) — this is a tension with R-001 that the plan does not acknowledge.
|
||||
|
||||
**Finding F7.3 (nft re-apply disruption).** `nft -f` replaces the table atomically (single transaction). Existing connections are NOT disrupted (conntrack holds them). New connections during the apply window (<1ms) may be dropped. This is acceptable. No finding.
|
||||
|
||||
### Axis 8 — Cost
|
||||
|
||||
**Finding F8.1 (2 images per release — sustainable).** P1 adds `orca-traefik` image alongside `orca`. `.coreci.yml` gets a `container-publish-traefik` step (P1 T5). `scripts/release.sh` gets a second docker block (P1 T4). The traefik image is small (~150MB, Alpine-based). Registry storage: 2 images × N releases. At v0.13.x cadence (8 tags), that's 16 image-tags per milestone. Sustainable for a private Gitea registry. No finding.
|
||||
|
||||
**Finding F8.2 (good — release.sh extension is minimal).** P1 T4 adds ~8 lines to `scripts/release.sh` after line 212. Verified the insertion point (line 213 is end of existing docker block). Clean.
|
||||
|
||||
### Axis 9 — Completeness (3 topologies)
|
||||
|
||||
**Finding F9.1 (linux topology — covered by P3+P4).** `orca init` (P3) bootstraps nft+podman on the lead; `orca node join --type linux` (P4) does it remotely. Complete.
|
||||
|
||||
**Finding F9.2 (proxmox-native — covered by P5, but LXC IP discovery is hand-wavy).** P5 T4 step 9 says "Discover LXC IP via `pct config <vmid>` (parse `net0` line) or `pct exec <vmid> -- hostname -I`". But in native mode the LXC is created **without** a static IP (P5 T4 step 2 has no `ip=` in the `pct create` — unlike P6 which has `ip=<floating-ip>/<prefix>`). So the LXC gets a DHCP/bridge IP that is not known at create time. P5 step 9 discovers it after `pct start`. But step 10 then re-applies nft with the discovered IP. This is a two-phase apply: first apply with default `127.0.0.1` (wrong for native), then re-apply with LXC IP. The plan does not specify what happens to traffic between the first and second apply (it DNATs to 127.0.0.1:8443 on the PVE host where nothing listens → connections refused). There is a window of ingress downtime during native-mode bootstrap.
|
||||
**Binding**: C-55 (see below).
|
||||
|
||||
**Finding F9.3 (floating-IP — covered by P6, but MAC uniqueness is untested).** P6 T4 generates a random `02:XX:XX:XX:XX:XX` MAC in interactive mode. The plan does not check for MAC collision on the bridge. RESEARCH Topic 5 pitfall 2: "hwaddr must be unique on the bridge". A random 02: prefix has 46 bits of entropy — collision is unlikely on a single bridge but not impossible across a multi-node cluster.
|
||||
**Binding**: C-56 (see below).
|
||||
|
||||
**Finding F9.4 (no `localhost`/lead topology with podman).** P3 bootstraps the lead via `EnsureTraefikContainerLocal`. But `orca init` runs on the lead — if the lead has no podman installed (fresh host), `EnsureTraefikContainerLocal` fails. Same as F5.2 but for the lead. The plan assumes podman is present on the lead. No install step.
|
||||
|
||||
---
|
||||
|
||||
## Binding Conditions (gates that MUST be met before a phase ships)
|
||||
|
||||
| ID | Gate | Phase | Severity |
|
||||
|----|------|-------|----------|
|
||||
| **C-50** | P2 must not break existing `orca init` on a host without podman installed — either install podman as part of `BootstrapLocalIngress` (P3) or emit a clear error with install instructions. Same for `orca node join --type linux` (P4). | P2/P3/P4 | BLOCKER |
|
||||
| **C-51** | `NftClusterConfig.DNATTarget` must be validated as `net.ParseIP` or `ip:port` before rendering. Unvalidated values are an nft-syntax injection vector (same F9 guard as TrustedProbes). Test required. | P3 | BLOCKER |
|
||||
| **C-52** | The `orca-traefik` image supply chain must be documented: registry is public (anonymous pull per REQ-045), no image signing in v0.14. Document the threat-model acceptance: a compromised registry = attacker code in the ingress LXC. Add `podman image trust` or cosign verification as a v0.15 hardening item. | P1/P8 | High |
|
||||
| **C-53** | The `apt-get install podman` step inside the LXC (P5 T4 step 4, P6 T1 step 6) must be idempotent: check `command -v podman` first, skip if present. Re-running join on an existing LXC must not re-run apt-get. | P5/P6 | High |
|
||||
| **C-54** | R-001 (offline-first) tension: `podman pull` at bootstrap requires registry reachability. Either (a) document that ingress bootstrap requires online access (exception to R-001), or (b) pre-bundle the `orca-traefik` image into the LXC template / load from a local archive. The plan must acknowledge this tension explicitly. | P2/P5/P6 | BLOCKER |
|
||||
| **C-55** | P5 native-mode bootstrap must not create a window of ingress downtime. The first nft apply must use the LXC IP (discovered after `pct start` but before the first nft apply), OR the plan must accept and document the downtime window. Two-phase apply (default → LXC IP) is a transient outage. | P5 | High |
|
||||
| **C-56** | P6 MAC generation must check for collision against existing nodes' MACs in the cluster registry. Reject or regenerate on collision. | P6 | Medium |
|
||||
| **C-57** | P2 T6 must implement the v0.13→v0.14 upgrade: detect `orca-traefik.service`, stop+disable, remove `/usr/local/bin/traefik` + unit file, then `EnsureTraefikContainerLocal`. Idempotent. Tested with a simulated-legacy-host test (F6.2). | P2 | BLOCKER |
|
||||
| **C-58** | The baked `docker/orca-traefik/traefik.yml` must not regress REQ-100's `traefik-on-public-ip` opt-out. Either (a) bake both configs and select via env/flag, or (b) document that the opt-out is dropped in v0.14 and update REQ-100, or (c) mount the static config from host (not baked) so `RenderTraefikStaticConfig` still works. | P1 | High |
|
||||
| **C-59** | Migration for `ingress_mode` must be `0009_ingress_mode.sql`, NOT 0007 (already taken by certs_serial_unique). | P5 | BLOCKER |
|
||||
| **C-60** | `certpaths.CAPath()` references in P3 T5 and P4 T1 must be corrected to `certpaths.CACertPath()`. | P3/P4 | BLOCKER (compile) |
|
||||
| **C-61** | REQ-172 must be amended: `--restart=always` → `--restart=unless-stopped` (per RESEARCH Topic 6 + PLAN), and `:Z` → omitted (per G-002). The requirement text contradicts the plan and research. | P2 | High |
|
||||
| **C-62** | If mTLS is in-scope for v0.14 (per G-003), a new phase must be added implementing step-ca server-cert minting into `/etc/traefik/dynamic` + dynamic `tls.certificates` + `tls.options.default.clientAuth.caFiles`. The plan currently has no such phase (P2 T8 emits `tls: {}`). | NEW PHASE | BLOCKER |
|
||||
|
||||
---
|
||||
|
||||
## Phase Challenges (specific challenges a phase must overcome)
|
||||
|
||||
| ID | Challenge | Phase |
|
||||
|----|-----------|-------|
|
||||
| **PC-01** | P1 must bake a static config that doesn't regress the `traefik-on-public-ip` opt-out (REQ-100). Baking freezes the config; the opt-out needs a runtime switch. | P1 |
|
||||
| **PC-02** | P2 must handle the 3-way TLS contradiction (CLARIFY D-257 vs RESEARCH Topic 4 vs PLAN T8). Per G-003, real mTLS is chosen — P2 alone cannot deliver it; a new phase is needed. | P2 + new |
|
||||
| **PC-03** | P2 T6 must remove the legacy systemd unit + binary without breaking a running v0.13 cluster. The existing `upgrade.go` cutover logic (lines 244-407) must be rewritten to stop+disable+remove the unit, not `systemctl restart traefik`. | P2 |
|
||||
| **PC-04** | P3 must install podman on the lead if absent (C-50). The plan assumes podman is present. `BootstrapLocalIngress` must either install it or fail with a clear message. | P3 |
|
||||
| **PC-05** | P3 T5 must use `certpaths.CACertPath()` not the phantom `certpaths.CAPath()`. | P3 |
|
||||
| **PC-06** | P3 T1 (`DNATTarget`) must validate input (C-51). New string field rendered into nft ruleset — injection risk. | P3 |
|
||||
| **PC-07** | P3 T4 (first-apply flush-table fix) must work across nft versions. RESEARCH establishes `delete table` on missing table is version-dependent. The pre-create approach (`nft add table … 2>/dev/null \|\| true` before `nft -f`) is robust; the plan uses it (T5 step 4). Verify on Proxmox kernel. | P3 |
|
||||
| **PC-08** | P5 native-mode has a two-phase nft apply (default 127.0.0.1 → LXC IP) that creates a transient outage window. Must be eliminated or documented (C-55). | P5 |
|
||||
| **PC-09** | P5/P6 `apt-get install podman` inside LXC takes 30-60s and must be idempotent (C-53). Re-join must not re-install. | P5/P6 |
|
||||
| **PC-10** | P5/P6 `podman pull` requires registry reachability, violating R-001 (offline-first). Must be acknowledged or mitigated (C-54). | P5/P6 |
|
||||
| **PC-11** | P5/P6 LXC reboot chain (Proxmox boot → `--onboot 1` → LXC systemd → `podman-restart.service`) has a gap: LXC may start before bridge is ready. Floating-IP `eth0` config may fail. | P5/P6 |
|
||||
| **PC-12** | P6 MAC generation must check for collision (C-56). | P6 |
|
||||
| **PC-13** | P7 T8 integration test must include the v0.13→v0.14 upgrade path (F6.2): simulated legacy host → upgrade → assert unit removed + podman running. | P7 |
|
||||
| **PC-14** | If the new mTLS phase is added (G-003), it must mint server certs into `/etc/traefik/dynamic` atomically (C-10 protocol) and traefik must reload them via the file provider watch. Step-ca minting was deferred since v0.11 — this is net-new work, not a completion. | new phase |
|
||||
|
||||
---
|
||||
|
||||
## Binding Decisions (from interactive grill)
|
||||
|
||||
| ID | Decision | Rationale | Confidence | Alternatives rejected |
|
||||
|----|----------|-----------|------------|---------------------|
|
||||
| **G-001** | Migration for `ingress_mode` is `0009_ingress_mode.sql` | 0007 is already `certs_serial_unique`; 0008 is `audit_tamper_evidence`. Reusing 0007 breaks the migrator. | 0.95 | Renumber existing (breaks deployed DBs); accept collision (guaranteed defect) |
|
||||
| **G-002** | Omit `:Z` flag on volume mounts; use `:ro` on both | RESEARCH Topic 7: `:Z` relabels host dirs to private container label, blocks host-side orca writes on SELinux. No-op on Ubuntu/Proxmox but a latent footgun. PLAN P2 T1 + RESEARCH agree; CLARIFY D-258 + REQ-172 are wrong. | 0.90 | Keep `:Z` (wrong on future SELinux); `:z` shared (unnecessary) |
|
||||
| **G-003** | Wire real mTLS now (dynamic `tls.certificates` + `clientAuth.caFiles`) | Operator chose this over `tls: {}` default. Requires step-ca server-cert minting which is not implemented (deferred since v0.11). | 0.55 | `tls: {}` now (plan's approach, defers mTLS); keep `certResolver: orca` (broken — key doesn't exist in traefik v3.3) |
|
||||
| **G-004** | P2 T6 must remove legacy systemd unit + binary on upgrade from v0.13 | Without it, upgrading a live v0.13 cluster leaves a dead systemd unit + a new podman container both binding 127.0.0.1:8443 → port conflict. Idempotent + tested. | 0.90 | Leave legacy unit (port conflict); fresh-installs only (unacceptable) |
|
||||
|
||||
---
|
||||
|
||||
## Escalations (unresolved, confidence < 0.60)
|
||||
|
||||
| ID | Escalation | Confidence | Reason |
|
||||
|----|------------|------------|--------|
|
||||
| **E-001** | G-003 (real mTLS now) creates a scope expansion that the 9-phase plan does not contain. The operator chose to add a step-ca cert minting phase (growing v0.14 to 10 phases) but the plan has not been updated to reflect this. Until the new phase is specified (scope, tasks, tests), v0.14's mTLS direction is **undetermined**. The plan as written ships `tls: {}` (no mTLS), which contradicts G-003. | 0.55 | The choice is made but the plan does not reflect it. This is a plan-spec gap, not a technical unknown. |
|
||||
|
||||
---
|
||||
|
||||
## Meta — Closing Review
|
||||
|
||||
**What the auditor would flag**:
|
||||
1. The plan ships a known-broken TLS config. RESEARCH Topic 4 calls `certResolver: orca` "the biggest v0.14 finding" and says it does not exist in traefik v3.3. CLARIFY D-257 and REQ-171 still specify it. The plan (P2 T8) drops it but emits `tls: {}` (no real TLS). G-003 chose real mTLS, which the plan doesn't contain. **Three artifacts disagree on TLS.**
|
||||
2. The plan regresses REQ-100 (`traefik-on-public-ip` opt-out) by baking the static config.
|
||||
3. The plan has no upgrade path from v0.13 (binary+systemd) to v0.14 (podman). G-004 binds the fix but the plan must be updated.
|
||||
4. The plan references a phantom function (`certpaths.CAPath()`) and a phantom migration number (0007). Both will fail at compile/execution time.
|
||||
5. The plan's offline-first claim (R-001) is violated by `podman pull` at bootstrap (C-54).
|
||||
|
||||
**What the project is NOT doing that it should**:
|
||||
- Image supply-chain verification (no cosign, no `podman image trust`).
|
||||
- LXC-start ordering relative to bridge readiness (reboot persistence gap).
|
||||
- Idempotency check for `apt-get install podman` inside LXC.
|
||||
- MAC collision check on the bridge.
|
||||
- A test for the v0.13→v0.14 upgrade path.
|
||||
|
||||
**Simplest 80%-value version**: Ship P1 (image) + P2 (reconciler, with legacy-removal) + P3 (nft+init) + P4 (linux remote) + P7 (doctor+tests). Defer P5 (proxmox native) and P6 (floating-IP) to v0.15. This delivers the linux topology (the most common) + the podman migration + nft completeness, and avoids the two most complex phases (LXC+podman-in-LXC). If G-003 (real mTLS) holds, add the mTLS phase to v0.14a. Proxmox ingress becomes v0.14b/v0.15.
|
||||
|
||||
**What must be true for v0.14 to succeed in 90 days**:
|
||||
1. The 3-way TLS contradiction is resolved in the plan (not just in this grill). Today: unresolved.
|
||||
2. The migration number is 0009. Today: plan says 0007 (wrong).
|
||||
3. The upgrade path from v0.13 is specified and tested. Today: not specified.
|
||||
4. The `certpaths.CAPath()` phantom is fixed. Today: not fixed.
|
||||
5. The offline-first tension (podman pull) is acknowledged. Today: not acknowledged.
|
||||
6. If mTLS is in-scope, the new phase is written. Today: no such phase.
|
||||
|
||||
**Confidence**: 0.45 that the plan as-written can ship v0.14 without rework. The research is strong; the plan diverges from it and from the codebase in load-bearing ways. The mTLS scope expansion (G-003) makes it worse unless the plan is updated.
|
||||
|
||||
---
|
||||
|
||||
## Verdict
|
||||
|
||||
**RETHINK** (confidence 0.45).
|
||||
|
||||
The plan must be revised to:
|
||||
1. Fix the migration number → 0009 (C-59, G-001).
|
||||
2. Fix `certpaths.CAPath()` → `certpaths.CACertPath()` (C-60).
|
||||
3. Add the v0.13→v0.14 upgrade path to P2 T6 (C-57, G-004).
|
||||
4. Resolve the TLS model: either add a new mTLS phase (G-003) or revert to `tls: {}` and defer mTLS to v0.15. The plan cannot ship `certResolver: orca` (broken) and cannot ship `tls: {}` if G-003 holds.
|
||||
5. Amend REQ-172: `--restart=unless-stopped` (not `always`), omit `:Z` (C-61, G-002).
|
||||
6. Address the offline-first tension (C-54) or document the exception.
|
||||
7. Address the REQ-100 regression (C-58): bake-vs-mount the static config.
|
||||
8. Add `DNATTarget` validation (C-51).
|
||||
9. Add podman-install step for linux topology (C-50).
|
||||
10. If mTLS is in-scope, write the new phase (C-62, E-001).
|
||||
|
||||
Once these are addressed, the plan is feasible. The research foundation is solid; the plan just needs to actually follow it.
|
||||
|
||||
---
|
||||
|
||||
*This grill is recorded in `.ciagent/GRILL_v0.14.md`. Escalations are visible via `ciagent audit`. Binding decisions (G-001..G-004) should be promoted to PROJECT.md via a follow-up clarify or explicitly by the operator. The grill surfaces; it does not rewrite.*
|
||||
Reference in New Issue
Block a user