orca/.ciagent/PLAN_v0.14.md

PLAN v0.14: Ingress Bootstrap Completeness

Status: active. 9 phases (P0 + P1..P7 + P8 final). Each phase ships a patch tag on the v0.13.x line. This plan references requirement IDs from REQUIREMENTS.md and follows the vertical-slice integrity rule (each phase is independently shippable).

Research-validated decisions (from RESEARCH_v0.14.md + GRILL_v0.14.md):

• nft postrouting: ip saddr 127.0.0.0/8 oifname != "lo" masquerade
• nft first-apply: pre-create table (nft add table inet orca-ingress 2>/dev/null || true) before nft -f
• pve-firewall: shift orca input/forward chains to priority -10 (before pve-firewall's 0)
• LXC features: nesting=1,keyctl=1,fuse=1 (fuse=1 for fuse-overlayfs)
• traefik TLS: drop certResolver: orca — does not exist in v3.3; emit tls: {} for v0.14 (real mTLS via dynamic tls.certificates + clientAuth.caFiles deferred to v0.15 — grill G-003 confidence 0.55 < 0.60 threshold, auto-resolved to defer)
• podman restart: --restart=unless-stopped + enable podman-restart.service
• volumes: omit :Z flag, use :ro on both mounts
• traefik image: FROM traefik:v3.3.0, ENTRYPOINT ["/traefik"] inherited, CMD ["--configFile=/etc/traefik/traefik.yml"]
• NftClusterConfig.DNATTarget: default 127.0.0.1:8443/:8080; proxmox-native = <lxc-ip>:8443/:8080
• Migration: 0009_ingress_mode.sql (NOT 0007 — already taken by certs_serial_unique)
• CA path: certpaths.CACertPath() (NOT certpaths.CAPath() — does not exist)
• Upgrade path: P2 must detect+remove legacy orca-traefik.service + /usr/local/bin/traefik before starting podman container (C-57)
• Podman install: BootstrapLocalIngress and BootstrapRemoteIngress must install podman if absent (C-50)
• Offline-first: podman pull requires registry reachability — documented exception to R-001 for ingress bootstrap (C-54)
• Static config: mount from host (not baked) to preserve traefik-on-public-ip opt-out (C-58)
• DNATTarget validation: net.ParseIP or ip:port parse before render (C-51)
• apt-get idempotency: command -v podman check before install (C-53)
• MAC collision: check against existing nodes' MACs (C-56)
• Native-mode nft: first apply uses LXC IP (not default 127.0.0.1) — discover LXC IP before first nft apply (C-55)

Phase 0: Pre-execution (this phase)

Status: complete. SPECIFY → CLARIFY → RESEARCH → PLAN → GRILL → SHIP. Ships as v0.13.0.

Phase 1: orca-traefik container image + release pipeline (REQ-171)

Tag: v0.13.1 | Type: feat | Persona: release-engineer (phase-specific) + backend-engineer

Wave 1 (image)

• T1: Create docker/orca-traefik/traefik.yml — the default static config baked into the image (used when no host-side override is mounted):

  entryPoints:
    websecure:
      address: "127.0.0.1:8443"
    web:
      address: "127.0.0.1:8080"
    traefik:
      address: "127.0.0.1:8081"
  providers:
    file:
      directory: "/etc/traefik/dynamic"
      watch: true
  log:
    level: INFO
    format: json
  accessLog:
    format: json
  

  No certificatesResolvers (research finding: does not exist for CA-based; TLS is via dynamic config). C-58: The baked config is a default. The podman run command also mounts a host-side /etc/traefik/traefik.yml if it exists (overriding the baked one), preserving the traefik-on-public-ip opt-out (REQ-100). The reconciler renders the static config via emitter.RenderTraefikStaticConfig to /etc/traefik/traefik.yml on the host, then mounts it -v /etc/traefik/traefik.yml:/etc/traefik/traefik.yml:ro. This way PublicBinding opt-out still works.
• T2: Create Dockerfile.traefik at repo root:

  FROM traefik:v3.3.0
  LABEL org.opencontainers.image.title="orca-traefik"
  LABEL org.opencontainers.image.source="https://git.cloudinit.dev/coreci/orca"
  COPY docker/orca-traefik/traefik.yml /etc/traefik/traefik.yml
  CMD ["--configFile=/etc/traefik/traefik.yml"]
  

  (ENTRYPOINT inherited as ["/traefik"] from base image.)
• T3: Create placeholder docker/orca-traefik/step-ca-root.crt (empty file) — real CA is volume-mounted at runtime. If absent, traefik starts without TLS termination (graceful).

Wave 2 (release pipeline)

• T4: scripts/release.sh — add a second docker block after the existing one (~line 212):

  # Build + push orca-traefik image
  TRAEFIK_IMAGE="${CONTAINER_REGISTRY}/${CONTAINER_OWNER}/orca-traefik"
  if command -v docker >/dev/null 2>&1; then
    docker build -f Dockerfile.traefik -t "${TRAEFIK_IMAGE}:${VERSION}" -t "${TRAEFIK_IMAGE}:latest" .
    docker push "${TRAEFIK_IMAGE}:${VERSION}"
    docker push "${TRAEFIK_IMAGE}:latest"
  fi
  

• T5: .coreci.yml — add container-publish-traefik step mirroring container-publish with CONTAINER_IMAGE=orca-traefik + DOCKERFILE=Dockerfile.traefik.

Wave 3 (tests)

• T6: Verify docker build -f Dockerfile.traefik . succeeds and the resulting image starts traefik with --configFile=/etc/traefik/traefik.yml (can test with docker run --rm orca-traefik --version).

Must-haves

• Dockerfile.traefik builds successfully
• Image starts traefik with the baked static config
• release.sh publishes orca-traefik:<version> + :latest
• .coreci.yml has container-publish-traefik step

Phase 2: Podman traefik reconciler (REQ-172)

Tag: v0.13.2 | Type: feat | Persona: lead-developer

Wave 1 (reconciler)

• T1: Rewrite internal/traefik/install.go — replace binary+systemd install with podman container reconciler:
  • EnsureTraefikContainer(ctx, execFn, image, tag) — idempotent:
    1. podman inspect orca-traefik → if running, no-op; if stopped, podman start orca-traefik; if absent, go to step 2
    2. mkdir -p /etc/traefik/dynamic /etc/orca
    3. podman pull <image>:<tag>
    4. podman run -d --name orca-traefik --restart=unless-stopped --network host -v /etc/traefik/dynamic:/etc/traefik/dynamic:ro -v /etc/orca/step-ca-root.crt:/etc/orca/step-ca-root.crt:ro <image>:<tag>
  • EnsureTraefikContainerLocal(ctx, image, tag) — uses exec.CommandContext("podman", ...) locally
  • EnsureTraefikContainerRemote(ctx, execFn, image, tag) — uses SSH exec function
  • Image/tag resolution: git.cloudinit.dev/coreci/orca-traefik:<version> where version = internal/cli.version (or latest if dev)
  • Remove systemd unit generation + systemctl enable
• T2: Add podman-restart.service enable step: systemctl enable --now podman-restart.service (research finding: needed for reboot persistence)
• T2a: C-50: EnsureTraefikContainerLocal/Remote must check command -v podman first. If absent: on localhost, attempt apt-get install -y podman (or fail with clear install instructions if no apt). On remote, apt-get install -y podman conmon crun fuse-overlayfs via SSH. Non-fatal warn if podman unavailable (offline host) — traefik won't start but orca init succeeds (same tolerance as v0.13).

Wave 2 (callsite updates + v0.13 upgrade path)

• T3: internal/cli/init.go:254-266 — replace installTraefikLocal() with EnsureTraefikContainerLocal
• T4: internal/linux/bootstrap.go:160-172 — replace traefik.InstallRemote with EnsureTraefikContainerRemote
• T5: internal/proxmox/bootstrap.go:250-255 — replace traefik.InstallRemote with EnsureTraefikContainerRemote (for native mode; floating-IP calls it inside the LXC in P6)
• T6: C-57 (v0.13→v0.14 upgrade path): internal/cli/upgrade.go — rewrite the Traefik cutover to:
  1. Detect legacy orca-traefik.service: systemctl is-active orca-traefik.service
  2. If active: systemctl stop orca-traefik.service && systemctl disable orca-traefik.service
  3. Remove /etc/systemd/system/orca-traefik.service + /usr/local/bin/traefik (if exists)
  4. systemctl daemon-reload
  5. Render static config via emitter.RenderTraefikStaticConfig to /etc/traefik/traefik.yml
  6. EnsureTraefikContainerLocal (pull + run podman container)
  7. Idempotent: if no legacy unit, skip steps 1-4
• T7: internal/cli/traefik_install.go — update CLI wrapper

Wave 3 (TLS model fix — research finding)

• T8: internal/emitter/traefik.go — drop certResolver: orca from the dynamic config router TLS stanza (line ~185-188). Replace with tls: {} (empty TLS stanza — traefik uses its default cert). Document that real mTLS via tls.certificates + tls.options.default.clientAuth.caFiles will be wired when step-ca mints certs into the dynamic dir (post-v0.14 or v1.x).
• T9: Update internal/emitter/traefik_test.go — remove assertion for certResolver: orca, add assertion for tls: {} presence.

Wave 4 (tests)

• T10: Create internal/traefik/install_test.go (new file — F1.3: does not exist today) — assert podman run is invoked (not curl|tar), --restart=unless-stopped --network host present, volume mounts present, podman-restart.service enabled.
• T10a: C-57/F6.2: Add v0.13→v0.14 upgrade test: simulate a host with orca-traefik.service present (fake), run upgrade, assert unit stopped+disabled+removed, podman container running.

Must-haves

• orca init → podman inspect orca-traefik shows running
• podman logs orca-traefik shows traefik started with baked config
• No systemd orca-traefik.service generated
• --restart=unless-stopped + podman-restart.service enabled
• certResolver: orca removed from dynamic config

Phase 3: nft SNAT+DNAT + orca init ingress bootstrap (REQ-173)

Tag: v0.13.3 | Type: feat | Persona: lead-developer + security-engineer

Wave 1 (nft emitter extension)

• T1: internal/emitter/nft.go — add DNATTarget field to NftClusterConfig (default 127.0.0.1). Render DNAT rules as dnat to <DNATTarget>:8443 / dnat to <DNATTarget>:8080. C-51: Validate DNATTarget with net.ParseIP before rendering. Reject invalid values with error (same F9 injection guard pattern as partitionTrustedProbes).
• T2: internal/emitter/nft.go — add EnableSNAT bool (default true) + postrouting chain:

  chain postrouting {
      type nat hook postrouting priority 100; policy accept;
      ip saddr 127.0.0.0/8 oifname != "lo" masquerade
  }
  

  Only when EnableSNAT == true.
• T3: internal/emitter/nft.go — shift input and forward chain priorities from filter (=0) to -10 (research finding: avoids pve-firewall same-priority undefined order).
• T4: internal/emitter/nft.go — fix first-apply flush-table bug: change flush table inet orca-ingress to delete table inet orca-ingress (nft ≥1.0 treats delete-of-missing as warning in -f mode). If that's version-unsafe, the apply step (T7) pre-creates the table.

Wave 2 (ingress bootstrap)

• T5: New internal/ingress/bootstrap.go:
  • BootstrapLocalIngress(ctx):
    1. mkdir -p /etc/traefik/dynamic /etc/orca
    2. C-60: Push cluster root CA to /etc/orca/step-ca-root.crt from certpaths.CACertPath() (if exists, else empty placeholder)
    3. Render static config via emitter.RenderTraefikStaticConfig to /etc/traefik/traefik.yml (preserves traefik-on-public-ip opt-out — C-58)
    4. Render orca.nft via NftEmitter.RenderNftConfig + write to /etc/nftables.d/orca.nft
    5. Pre-create table: nft add table inet orca-ingress 2>/dev/null || true
    6. Apply: nft -f /etc/nftables.d/orca.nft
    7. C-50: Ensure podman installed (check command -v podman, install if absent)
    8. EnsureTraefikContainerLocal (from P2) — mounts /etc/traefik/traefik.yml:ro + /etc/traefik/dynamic:ro + /etc/orca/step-ca-root.crt:ro
  • Each step non-fatal warn (offline host tolerance)
• T6: Wire into internal/cli/init.go after EnsureTraefikContainerLocal (Step 4e, replacing the old traefik install step).

Wave 3 (doctor nft update)

• T7: internal/cli/doctor_nft.go — extend assertions: postrouting masquerade present, DNAT target matches NftClusterConfig.DNATTarget.

Wave 4 (tests)

• T8: internal/emitter/nft_test.go — assert postrouting chain present when EnableSNAT=true, absent when false. Assert DNATTarget substitution. Assert priority -10 on input/forward.
• T9: Integration test: orca init → nft list table inet orca-ingress shows DNAT + postrouting; podman inspect orca-traefik running.

Must-haves

• orca init → nft table has DNAT + postrouting masquerade
• nft input/forward chains at priority -10
• First-apply doesn't error (table pre-created or delete-table idiom)
• /etc/orca/step-ca-root.crt exists (real CA or placeholder)
• podman inspect orca-traefik running

Phase 4: orca node join --type linux remote ingress bootstrap (REQ-174)

Tag: v0.13.4 | Type: feat | Persona: lead-developer

Wave 1 (remote ingress)

• T1: internal/ingress/bootstrap.go — add BootstrapRemoteIngress(ctx, execFn):
  1. mkdir -p /etc/traefik/dynamic /etc/orca (remote)
  2. C-60: Push step-ca root CA to remote /etc/orca/step-ca-root.crt from certpaths.CACertPath() via WriteFile
  3. Render orca.nft + write to remote /etc/nftables.d/orca.nft via WriteFile
  4. nft add table inet orca-ingress 2>/dev/null || true (remote)
  5. nft -f /etc/nftables.d/orca.nft (remote)
  6. systemctl enable --now podman-restart.service (remote)
  7. EnsureTraefikContainerRemote (from P2)
• T2: Wire into internal/linux/bootstrap.go after the traefik container reconciler step.
• T3: Extend linux.Result with IngressOK bool for reporting.

Wave 2 (tests)

• T4: Fake-SSH test: assert remote nft -f + podman run + WriteFile for step-ca CA invoked.

Must-haves

• orca node join --type linux --host <ip> → remote has podman traefik running + nft applied + step-ca CA mounted
• doctor ingress --peer <linux-node> passes

Phase 5: Proxmox native ingress mode (REQ-175)

Tag: v0.13.5 | Type: feat | Persona: backend-engineer + data-engineer

Wave 1 (flags + schema)

• T1: Add flags to node join: --ingress-mode (values: native default, floating-ip), --floating-ip, --gateway, --mac, --net-prefix (default 24).
• T2: Add IngressMode field to model.Node (string: "", "native", "floating-ip").
• T3: C-59: Schema migration 0009_ingress_mode.sql (NOT 0007 — already taken): ALTER TABLE nodes ADD COLUMN ingress_mode TEXT DEFAULT '';

Wave 2 (native mode bootstrap)

• T4: In proxmox.BootstrapProxmox, when IngressMode == "native":
  1. On the PVE host: render+apply nft with DNATTarget = <lxc-bridge-ip> (the traefik LXC's IP, discovered after pct start)
  2. Create unprivileged LXC with --features nesting=1,keyctl=1,fuse=1 (research finding: fuse=1 for fuse-overlayfs). pct create <vmid> local:vztmpl/<template> --hostname orca-traefik --unprivileged 1 --features nesting=1,keyctl=1,fuse=1 --onboot 1 --memory 2048 --swap 0 --rootfs local:8
  3. pct start <vmid> 3a. C-55: Discover LXC IP via pct config <vmid> (parse net0 line for ip=) or pct exec <vmid> -- hostname -I BEFORE the first nft apply. The nft DNAT target is set to the LXC IP from the start — no two-phase apply, no downtime window.
  4. Inside the LXC: C-53: command -v podman >/dev/null 2>&1 || (apt-get update && apt-get install -y podman conmon crun fuse-overlayfs nftables) — idempotent, skip if podman already installed
  5. Configure podman storage (/etc/containers/storage.conf): mount_program = "/usr/bin/fuse-overlayfs" (fallback: driver = "vfs")
  6. systemctl enable --now podman-restart.service (inside LXC)
  7. Push step-ca root CA into LXC
  8. EnsureTraefikContainerRemote (podman pull + run inside LXC with --network host)
  9. C-55: Render+apply nft on PVE host with DNATTarget = <lxc-ip> (discovered in step 3a) — single apply, no downtime window

Wave 3 (registration)

• T5: Register PVE host as proxmox node with IngressMode: "native".

Wave 4 (tests)

• T6: Fake-SSH test: assert pct create with --features nesting=1,keyctl=1,fuse=1, apt-get install podman, podman run inside LXC, nft DNAT target = LXC IP.

Must-haves

• orca node join --type proxmox --host <ip> (native) → LXC created with nesting+keyctl+fuse
• Podman + orca-traefik running inside LXC
• PVE host nft DNATs to LXC IP
• IngressMode: "native" on node record
• Schema migration 0007 applied

Phase 6: Proxmox floating-IP LXC "ingress" (REQ-176)

Tag: v0.13.6 | Type: feat | Persona: backend-engineer

Wave 1 (LXC provisioning)

• T1: New internal/proxmox/ingress_lxc.go — ProvisionIngressLXC(ctx, opts):
  1. pveam download local <template> (idempotent, already in bootstrap)
  2. Deterministic VMID for "ingress" (hash of host+"ingress")
  3. pct create <vmid> local:vztmpl/<template> --hostname ingress --unprivileged 1 --features nesting=1,keyctl=1,fuse=1 --net0 name=eth0,bridge=vmbr0,hwaddr=<mac>,ip=<floating-ip>/<prefix>,gw=<gateway> --onboot 1 --memory 2048 --swap 0 --rootfs local:8
  4. pct start <vmid>
  5. Wait for LXC network (retry SSH to <floating-ip> with backoff, 60s timeout)
  6. Inside the LXC: C-53: command -v podman >/dev/null 2>&1 || (apt-get update && apt-get install -y podman conmon crun fuse-overlayfs nftables) — idempotent
  7. Configure podman storage (fuse-overlayfs / vfs fallback)
  8. systemctl enable --now podman-restart.service
  9. Push step-ca root CA into LXC
  10. Render+apply nft INSIDE the LXC (DNAT :443→127.0.0.1:8443, :80→127.0.0.1:8080 + postrouting masquerade) — DNATTarget = 127.0.0.1 here because traefik runs with --network host inside the LXC
  11. EnsureTraefikContainerRemote (podman pull + run inside LXC with --network host)
  12. Push orca SSH pubkey into LXC for future job run traefik dynamic-config pushes

Wave 2 (registration)

• T2: Register LXC as managed node: registry.Join with Kind: "linux", Name: "ingress", Address: "<floating-ip>:8443", OS: "linux", IngressMode: "floating-ip".
• T3: Also register PVE host as proxmox node (for workload dispatch).

Wave 3 (interactive prompting)

• T4: Interactive prompting in joinProxmox (node.go): when --ingress-mode empty and !jsonOutput:
  • Prompt "Ingress mode [native/floating-ip] (default native): "
  • If floating-ip: prompt for floating IP (validate net.ParseIP), gateway (validate net.ParseIP), MAC (validate net.ParseMAC; generate 02:XX:XX:XX:XX:XX random if empty + confirm; C-56: check MAC against existing nodes' MACs in cluster registry, regenerate on collision), net-prefix (default 24, validate 8-32)
  • In --json mode: require --mac explicitly if --ingress-mode floating-ip (no silent generation)

Wave 4 (routing)

• T5: joinProxmox in node.go routes: native → P5 path; floating-ip → ProvisionIngressLXC + register LXC + register PVE host.

Wave 5 (tests)

• T6: Fake-SSH test: assert pct create with net0 name=eth0,bridge=vmbr0,hwaddr=<mac>,ip=<floating-ip>/<prefix>,gw=<gateway>, --features nesting=1,keyctl=1,fuse=1; LXC registered as linux node named ingress at <floating-ip>:8443; PVE host registered as proxmox.
• T7: Interactive prompt test: stdin simulation → mode selection + param entry + validation.

Must-haves

• orca node join --type proxmox --host <ip> --ingress-mode floating-ip --floating-ip 203.0.113.10 --gateway 203.0.113.1 --mac 02:01:02:03:04:05 → LXC ingress created
• LXC has podman traefik running + nft applied inside LXC
• Node ingress registered as linux at 203.0.113.10:8443
• PVE host registered as proxmox
• Interactive prompt works when flags absent + not --json
• IP/MAC/gateway validation rejects invalid input

Phase 7: doctor ingress + docs + integration tests (REQ-177, REQ-178, REQ-179)

Tag: v0.13.7 | Type: feat+docs+test | Persona: backend-engineer + lead-developer

Wave 1 (doctor ingress)

• T1: internal/cli/doctor_ingress.go — orca doctor ingress [--peer <name>]:
  1. podman inspect orca-traefik → running?
  2. nft DNAT+SNAT applied (reuse doctor_nft logic)
  3. /etc/traefik/dynamic exists
  4. step-ca root CA mounted (podman inspect volume check or file-exists check)
  5. For proxmox-native: checks the LXC (via pct exec)
  6. For floating-ip: checks the ingress LXC over SSH
  7. Uses SSH-push for remote peers

Wave 2 (UAT assertions)

• T2: scripts/uat-signoff.sh — add assertions: 40 ingress_podman_traefik, 41 ingress_nft_dnat_snat, 42 ingress_linux_worker, 43 ingress_proxmox_native_lxc or 43 ingress_floating_ip_lxc (depending on topology).

Wave 3 (docs)

• T3: docs/cli.md — document --ingress-mode, --floating-ip, --gateway, --mac, --net-prefix flags + orca doctor ingress.
• T4: docs/uat.md — add floating-IP topology variant; update native topology to assert ingress bootstrap.
• T5: docs/ingress.md — podman-traefik image section: Dockerfile.traefik, volume mounts, TLS model (dynamic tls.certificates, not certResolver), --network host rationale.
• T6: docs/docker.md — orca-traefik image: build, publish, pull.
• T7: .ciagent/ARCHITECTURE.md — R-024 + ingress bootstrap section (3 topologies, nft+podman stack on each).

Wave 4 (integration tests)

• T8: tests/ingress_bootstrap_test.go — hermetic fake-SSH harness:
  • init → podman traefik running + nft applied
  • linux join → remote podman + nft + step-ca CA push
  • proxmox native → LXC created with nesting=1,keyctl=1,fuse=1 + podman traefik + nft DNAT to LXC IP
  • floating-ip → pct create with correct net0 args + LXC registered as linux node
  • release.sh builds orca-traefik image (Dockerfile.traefik parses)

Wave 5 (verify)

• T9: make build && make test && make lint && make verify-docs all pass.

Must-haves

• orca doctor ingress exits 0 on a properly bootstrapped node
• UAT signoff script includes new assertions
• make verify-docs passes
• Integration tests pass in CI validate
• ARCHITECTURE.md ingress section matches shipped code

Phase 8: Final review + ship + audit (milestone release)

Tag: v0.13.8 = v0.14 milestone release | Type: docs+review

• Multi-persona code review across all phases
• Audit: verify-reqs, git-log ↔ .ciagent/ reconstruction
• Merge phase/08 → milestone/v0.14 → main
• Tag v0.13.8 + release with milestone summary
• Build + publish both container images (orca + orca-traefik)
• Delete all milestone branches (tags preserve history)
• Mark all REQ-171..179 as complete in REQUIREMENTS.md + ROADMAP.md