Compare commits

...

5 Commits

Author SHA1 Message Date
Jon Chery ecdba833d9 feat(P7): doctor ingress + docs + integration tests (REQ-177,178,179)
New 'orca doctor ingress' command: verifies podman orca-traefik
container running, nft DNAT+SNAT, /etc/traefik/dynamic exists,
step-ca root CA present.

UAT signoff script: replaced assertion 36 (systemd → podman
container), added assertions 40-46 (nft table, DNAT, SNAT, dynamic
dir, step-ca CA, traefik.yml, doctor ingress pass).

docs/ingress.md: R-024 podman traefik section — three topologies,
container config, nft ruleset, doctor ingress, Dockerfile.traefik.
TLS model updated (drop certResolver, tls:{} for v0.14, mTLS v0.15).

ARCHITECTURE.md: v0.14 deltas section — R-024, three topologies,
nft emitter changes, TLS model, migration 0009, new CLI.

Integration tests (tests/ingress_bootstrap_test.go): nft postrouting
+ DNATTarget, priority -10, traefik TLS model (tls:{} no
certResolver), image ref resolution, floating-IP LXC provisioning
commands (pct create with hwaddr/ip/gw/features), MAC generation.

---ci---
project: orca
phase: 7
milestone: v0.14
status: execute
---/ci---
2026-08-10 20:24:13 +00:00
Jon Chery 6e65eadaa5 feat(P6): proxmox floating-IP LXC ingress + interactive prompt (REQ-176)
New internal/proxmox/ingress_lxc.go: ProvisionIngressLXC creates an
Ubuntu LXC named 'ingress' that owns the floating IP (net0
bridge=vmbr0,hwaddr=<mac>,ip=<floating-ip>/<prefix>,gw=<gateway>).
Unprivileged with --features nesting=1,keyctl=1,fuse=1 (research
Topic 3). Installs podman inside, runs orca-traefik container,
applies nft DNAT+SNAT INSIDE the LXC, pushes step-ca root CA.

GenerateRandomMAC: 02:XX:XX:XX:XX:XX for interactive mode (D-261).

Interactive prompting in joinProxmox: when --ingress-mode empty +
!--json, prompt for mode + floating IP + gateway + MAC (auto-
generate + confirm). Validate IP/MAC/gateway/prefix.

Floating-IP routing: calls ProvisionIngressLXC + registers:
  1. PVE host as 'proxmox' node (IngressMode=floating-ip)
  2. Ingress LXC as 'linux' node (name=ingress, addr=<floating-ip>:8443)
     so orca job run pushes traefik dynamic config to it.

---ci---
project: orca
phase: 6
milestone: v0.14
status: execute
---/ci---
2026-08-10 20:19:47 +00:00
Jon Chery 1b7aac71f6 feat(P5): proxmox native ingress mode — LXC + podman traefik (REQ-175)
Add --ingress-mode flag (native default, floating-ip) + --floating-ip,
--gateway, --mac, --net-prefix flags to 'orca node join'.

Native mode (default): provision an unprivileged LXC with
--features nesting=1,keyctl=1,fuse=1 (research Topic 3), install
podman inside it, run orca-traefik container. nft on PVE host DNATs
to the LXC bridge IP (DNATTarget parameterization, C-55: discover
LXC IP before first nft apply, no downtime window).

LXC provisioning: deterministic VMID 200, hostname orca-traefik,
--onboot 1, 2GB RAM. Idempotent (C-53: command -v podman check).
podman-restart.service enabled inside LXC (research Topic 6).

step-ca root CA pushed into LXC via pct exec heredoc.
traefik static config rendered + written into LXC.
nft ruleset rendered with DNATTarget=LXC-IP + applied on PVE host.

Migration 0009_ingress_mode.sql (C-59: NOT 0007 — already taken by
certs_serial_unique). ALTER TABLE nodes ADD COLUMN ingress_mode.
IngressMode field added to model.Node + set on proxmox node record.

---ci---
project: orca
phase: 5
milestone: v0.14
status: execute
---/ci---
2026-08-10 20:16:40 +00:00
Jon Chery ea42a17474 feat(P4): linux node join remote ingress bootstrap (REQ-174)
Add ingress.BootstrapRemoteIngress: renders+writes traefik static
config, renders+writes+applies nft DNAT/SNAT, pushes step-ca root CA,
ensures podman traefik container — all over SSH exec. Uses a heredoc-
based remoteWriteFile with a random delimiter (F9 injection guard).

Wired into linux/bootstrap.go Step 4d, replacing the standalone
EnsureTraefikContainerRemote call with the full ingress stack.

C-60: uses certpaths.CACertPath() (not CAPath).
C-58: mounts host-side traefik.yml (preserves REQ-100 opt-out).
C-55: pre-creates nft table before nft -f.

---ci---
project: orca
phase: 4
milestone: v0.14
status: execute
---/ci---
2026-08-10 20:11:32 +00:00
Jon Chery 5013209e31 feat(P3): nft SNAT+DNAT + orca init ingress bootstrap (REQ-173)
nft emitter (internal/emitter/nft.go):
- Add DNATTarget field (C-51: validated via net.ParseIP; injection
  guard). Default 127.0.0.1; proxmox native uses LXC bridge IP.
- Add EnableSNAT field (default true for zero-value config).
- Add postrouting masquerade chain (research Topic 1):
  ip saddr 127.0.0.0/8 oifname != lo masquerade
- Shift input/forward priority from filter (=0) to -10 (research
  Topic 2: pve-firewall coexistence — avoids same-priority undefined
  evaluation order).

internal/ingress/bootstrap.go (new):
- BootstrapLocalIngress: mkdir dirs, push step-ca root CA (C-60:
  certpaths.CACertPath not CAPath), render+write traefik static
  config (C-58: preserves traefik-on-public-ip opt-out), render+
  write+apply nft ruleset, pre-create table (C-55: avoids first-
  apply flush-table error), ensure podman container. All non-fatal.

init.go: Step 4d now calls ingress.BootstrapLocalIngress (R-024).
doctor_nft.go: assert postrouting masquerade + priority -10.

Tests: nft_test.go — DNATTarget substitution, invalid DNATTarget
rejection (C-51), EnableSNAT=false omits postrouting, priority -10.

---ci---
project: orca
phase: 3
milestone: v0.14
status: execute
---/ci---
2026-08-10 20:09:26 +00:00
16 changed files with 1389 additions and 65 deletions
+67
View File
@@ -872,3 +872,70 @@ scheduler.Schedule(spec, nodes) → emitter.Render(unit) → sshpush.Deploy(targ
- IPv6 `net.JoinHostPort` in all SSH dial paths. - IPv6 `net.JoinHostPort` in all SSH dial paths.
- Explicit timeouts on all SSH commands. - Explicit timeouts on all SSH commands.
- Root SIGINT/SIGTERM handler for clean exit on non-watch commands. - Root SIGINT/SIGTERM handler for clean exit on non-watch commands.
## v0.14 Deltas — Ingress Bootstrap Completeness (R-024)
### R-024: Traefik as Podman Container
Traefik runs exclusively as a podman container, deployed from the
custom `orca-traefik` image (published per release via `Dockerfile.traefik`
+ `scripts/release.sh` + `.coreci.yml container-publish-traefik`).
The v0.13 binary+systemd install (`internal/traefik/install.go`) is
replaced by an idempotent podman container reconciler
(`EnsureTraefikContainerLocal`/`Remote`). The container runs with
`--network host`, `--restart=unless-stopped`, and volume mounts for
`traefik.yml` (static config), `dynamic` (dynamic config), and
`step-ca-root.crt` (future mTLS). No SELinux `:Z` flag.
### Three Ingress Topologies
1. **Linux** (`orca init` / `orca node join --type linux`):
host → nft DNAT → podman traefik (host network).
`internal/ingress/bootstrap.go` → `BootstrapLocalIngress` /
`BootstrapRemoteIngress`.
2. **Proxmox Native** (`--ingress-mode native`, default):
PVE host → nft DNAT (target = LXC bridge IP) → LXC
(`--features nesting=1,keyctl=1,fuse=1`) → podman traefik.
`internal/proxmox/bootstrap.go` → `provisionNativeIngressLXC`.
3. **Proxmox Floating-IP** (`--ingress-mode floating-ip`):
LXC owns the floating IP (`net0 bridge=vmbr0,hwaddr=<mac>,
ip=<floating-ip>/<prefix>,gw=<gateway>`) → nft inside LXC →
podman traefik. The ingress LXC is registered as a `linux` node
(name=`ingress`) so `orca job run` pushes traefik dynamic config.
`internal/proxmox/ingress_lxc.go` → `ProvisionIngressLXC`.
### nft Emitter Changes
`internal/emitter/nft.go`:
- `DNATTarget` field (C-51: validated via `net.ParseIP`). Default
`127.0.0.1`; proxmox native uses LXC bridge IP.
- `EnableSNAT` field + postrouting masquerade chain: `ip saddr
127.0.0.0/8 oifname != "lo" masquerade` (research Topic 1).
- Input/forward chain priority shifted from `filter` (=0) to `-10`
(research Topic 2: pve-firewall coexistence — avoids same-priority
undefined evaluation order).
### TLS Model
v0.14 drops `certResolver: orca` from the dynamic config (traefik v3.3
only supports `acme`/`tailscale` resolvers, not CA-file-based). The
dynamic config emits `tls: {}` (traefik default cert). Real mTLS via
`tls.certificates` + `tls.options.default.clientAuth.caFiles` is
deferred to v0.15 (grill G-003, confidence 0.55 < 0.60).
### Migration 0009
`ALTER TABLE nodes ADD COLUMN ingress_mode TEXT NOT NULL DEFAULT '';`
Values: `""` (legacy), `"native"`, `"floating-ip"`. `IngressMode` field
on `model.Node`.
### New CLI
- `orca doctor ingress` — verifies podman container running, nft
DNAT+SNAT, dynamic dir, step-ca root CA.
- `--ingress-mode` flag on `orca node join --type proxmox`.
- `--floating-ip`, `--gateway`, `--mac`, `--net-prefix` flags for
floating-IP mode.
+71 -7
View File
@@ -138,13 +138,77 @@ restore traffic).
## TLS ## TLS
- **certResolver**: `orca` (references the Traefik ACME/step-ca - **v0.14 model**: `tls: {}` in dynamic config (no certResolver).
certificate resolver configured in Traefik's static config). Traefik v3.3 `certificatesResolvers` only supports `acme` and
- **Trust domain**: `cluster.orca.local` (placeholder in v0.9; step-ca `tailscale` — not CA-file-based. The `certResolver: orca` reference
provisioner in v0.11 overrides with the real cluster trust domain). from v0.11 was broken (research finding). v0.14 emits `tls: {}`
- **SPIFFE SVIDs**: workload identity via SPIFFE SVIDs minted at submit (traefik uses its default self-signed cert). Real mTLS via dynamic
time via step-ca (v0.11-P01.5, gate C-08). The SVID is a URI SAN in `tls.certificates` + `tls.options.default.clientAuth.caFiles` is
the workload's X.509 cert. deferred to v0.15.
- **Step-ca root CA**: mounted at `/etc/orca/step-ca-root.crt` in the
traefik container. v0.14 does not use it for TLS termination (it's
a placeholder for v0.15 mTLS).
## R-024: Podman Traefik Container (v0.14)
As of v0.14, Traefik runs as a **podman container** from the custom
`orca-traefik` image (published per release). The v0.13 binary+systemd
install is replaced.
### Three topologies
1. **Linux**: host → nft DNAT → `podman run orca-traefik` (`--network host`)
2. **Proxmox Native** (`--ingress-mode native`, default): PVE host →
nft DNAT → LXC (nesting=1,keyctl=1,fuse=1) → `podman run orca-traefik`
3. **Proxmox Floating-IP** (`--ingress-mode floating-ip`): LXC owns
the floating IP → nft inside LXC → `podman run orca-traefik`
### Container configuration
```bash
podman run -d --name orca-traefik --restart=unless-stopped \
--network host \
-v /etc/traefik/traefik.yml:/etc/traefik/traefik.yml:ro \
-v /etc/traefik/dynamic:/etc/traefik/dynamic:ro \
-v /etc/orca/step-ca-root.crt:/etc/orca/step-ca-root.crt:ro \
git.cloudinit.dev/coreci/orca-traefik:<version>
```
- `--network host`: traefik binds 127.0.0.1:8080/8443 on host/LXC loopback
- `--restart=unless-stopped`: survives reboot via `podman-restart.service`
- No `:Z` SELinux flag (research Topic 7)
- Static config mounted `:ro` (overrides baked image default, preserves
`traefik-on-public-ip` opt-out, REQ-100)
### nft ruleset
The nft emitter (`internal/emitter/nft.go`) renders `/etc/nftables.d/orca.nft`:
- DNAT `:443``<DNATTarget>:8443` (default 127.0.0.1; LXC IP for native)
- DNAT `:80``<DNATTarget>:8080`
- SNAT/MASQUERADE: `ip saddr 127.0.0.0/8 oifname != "lo" masquerade`
- Input/forward chains at priority -10 (pve-firewall coexistence)
### `orca doctor ingress`
```bash
orca doctor ingress # check localhost
orca doctor ingress --peer <name> # check remote peer
```
Verifies: podman container running, nft DNAT+SNAT, dynamic dir exists,
step-ca root CA present.
### Dockerfile.traefik
```dockerfile
FROM traefik:v3.3.0
COPY docker/orca-traefik/traefik.yml /etc/traefik/traefik.yml
CMD ["--configFile=/etc/traefik/traefik.yml"]
```
Built + published per release alongside the orca image
(`scripts/release.sh` + `.coreci.yml container-publish-traefik`).
## Health checks ## Health checks
+115
View File
@@ -0,0 +1,115 @@
// Package cli: doctor_ingress.go implements `orca doctor ingress`
// (R-024, v0.14). The check verifies the podman traefik container is
// running, nft DNAT+SNAT is applied, the dynamic config directory
// exists, and the step-ca root CA is mounted.
package cli
import (
"context"
"fmt"
"strings"
"time"
"github.com/spf13/cobra"
)
var doctorIngressCmd = &cobra.Command{
Use: "ingress",
Short: "Check the ingress stack (R-024: podman traefik + nft + CA)",
Long: `Verify the orca ingress data plane is healthy:
1. orca-traefik podman container is running
2. nft DNAT + SNAT masquerade applied
3. /etc/traefik/dynamic directory exists
4. step-ca root CA mounted at /etc/orca/step-ca-root.crt
For remote peers, use --peer <name>.`,
RunE: func(cmd *cobra.Command, args []string) error {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
results := runIngressChecks(ctx)
if jsonOutput {
return printJSON(results)
}
allPass := true
for _, r := range results {
status := "✓"
if r.Result != "PASS" {
status = "✗"
allPass = false
}
fmt.Fprintf(cmd.OutOrStdout(), "%s %s: %s\n", status, r.Name, r.Message)
}
if !allPass {
return fmt.Errorf("ingress checks failed")
}
return nil
},
}
// ingressCheckResult is one line of `orca doctor ingress` output.
type ingressCheckResult struct {
Name string `json:"name"`
Result string `json:"result"`
Message string `json:"message"`
}
func runIngressChecks(ctx context.Context) []ingressCheckResult {
t, err := nftTransportFromCtx()
if err != nil {
return []ingressCheckResult{{Name: "ingress:transport", Result: "FAIL", Message: err.Error()}}
}
peer := nftLeadPeer()
var results []ingressCheckResult
// 1. Check podman orca-traefik container is running.
out, err := t.Exec(ctx, peer, "podman inspect --format '{{.State.Running}}' orca-traefik 2>/dev/null")
if err != nil {
results = append(results, ingressCheckResult{Name: "ingress:container", Result: "FAIL", Message: fmt.Sprintf("podman inspect: %v", err)})
} else {
v := strings.TrimSpace(string(out))
if v == "true" {
results = append(results, ingressCheckResult{Name: "ingress:container", Result: "PASS", Message: "orca-traefik container running"})
} else if v == "false" {
results = append(results, ingressCheckResult{Name: "ingress:container", Result: "FAIL", Message: "orca-traefik container is stopped"})
} else {
results = append(results, ingressCheckResult{Name: "ingress:container", Result: "FAIL", Message: "orca-traefik container not found"})
}
}
// 2. Check nft DNAT + SNAT (reuse the nft table output).
tableOut, tableErr := t.Exec(ctx, peer, "nft list table inet orca-ingress 2>/dev/null")
if tableErr != nil {
results = append(results, ingressCheckResult{Name: "ingress:nft", Result: "FAIL", Message: "nft table orca-ingress missing"})
} else {
tableStr := string(tableOut)
hasDNAT := strings.Contains(tableStr, "dnat to")
hasSNAT := strings.Contains(tableStr, "masquerade")
if hasDNAT && hasSNAT {
results = append(results, ingressCheckResult{Name: "ingress:nft", Result: "PASS", Message: "nft DNAT + SNAT masquerade present"})
} else if hasDNAT {
results = append(results, ingressCheckResult{Name: "ingress:nft", Result: "WARN", Message: "DNAT present but SNAT masquerade missing"})
} else {
results = append(results, ingressCheckResult{Name: "ingress:nft", Result: "FAIL", Message: "nft DNAT missing"})
}
}
// 3. Check /etc/traefik/dynamic directory exists.
if _, err := t.Exec(ctx, peer, "test -d /etc/traefik/dynamic"); err != nil {
results = append(results, ingressCheckResult{Name: "ingress:dynamic-dir", Result: "FAIL", Message: "/etc/traefik/dynamic directory missing"})
} else {
results = append(results, ingressCheckResult{Name: "ingress:dynamic-dir", Result: "PASS", Message: "/etc/traefik/dynamic exists"})
}
// 4. Check step-ca root CA is mounted/present.
if _, err := t.Exec(ctx, peer, "test -f /etc/orca/step-ca-root.crt"); err != nil {
results = append(results, ingressCheckResult{Name: "ingress:ca", Result: "WARN", Message: "/etc/orca/step-ca-root.crt missing (TLS not configured)"})
} else {
results = append(results, ingressCheckResult{Name: "ingress:ca", Result: "PASS", Message: "step-ca root CA present"})
}
return results
}
func init() {
doctorCmd.AddCommand(doctorIngressCmd)
}
+14
View File
@@ -113,6 +113,20 @@ func runNftChecks(ctx context.Context) []nftCheckResult {
results = append(results, nftCheckResult{Name: "nft:dnat-80", Result: "FAIL", Message: "DNAT :80->127.0.0.1:8080 missing"}) results = append(results, nftCheckResult{Name: "nft:dnat-80", Result: "FAIL", Message: "DNAT :80->127.0.0.1:8080 missing"})
} }
// Research Topic 1: postrouting masquerade for DNAT return path.
if strings.Contains(tableStr, "masquerade") {
results = append(results, nftCheckResult{Name: "nft:snat-masquerade", Result: "PASS", Message: "postrouting masquerade (SNAT) present"})
} else {
results = append(results, nftCheckResult{Name: "nft:snat-masquerade", Result: "FAIL", Message: "postrouting masquerade missing (R-024)"})
}
// Research Topic 2: priority -10 on input/forward (pve-firewall coexistence).
if strings.Contains(tableStr, "priority -10") {
results = append(results, nftCheckResult{Name: "nft:priority", Result: "PASS", Message: "input/forward chains at priority -10 (pve-firewall coexistence)"})
} else {
results = append(results, nftCheckResult{Name: "nft:priority", Result: "WARN", Message: "priority -10 not found (may be pre-v0.14 ruleset)"})
}
if strings.Contains(tableStr, "ora_rl") { if strings.Contains(tableStr, "ora_rl") {
results = append(results, nftCheckResult{Name: "nft:rate-limit", Result: "PASS", Message: "rate-limit meter ora_rl present"}) results = append(results, nftCheckResult{Name: "nft:rate-limit", Result: "PASS", Message: "rate-limit meter ora_rl present"})
} else { } else {
+13 -10
View File
@@ -9,6 +9,7 @@ import (
"git.cloudinit.dev/coreci/orca/internal/acl" "git.cloudinit.dev/coreci/orca/internal/acl"
"git.cloudinit.dev/coreci/orca/internal/certpaths" "git.cloudinit.dev/coreci/orca/internal/certpaths"
"git.cloudinit.dev/coreci/orca/internal/identity" "git.cloudinit.dev/coreci/orca/internal/identity"
"git.cloudinit.dev/coreci/orca/internal/ingress"
"git.cloudinit.dev/coreci/orca/internal/model" "git.cloudinit.dev/coreci/orca/internal/model"
"git.cloudinit.dev/coreci/orca/internal/paths" "git.cloudinit.dev/coreci/orca/internal/paths"
"git.cloudinit.dev/coreci/orca/internal/secrets" "git.cloudinit.dev/coreci/orca/internal/secrets"
@@ -251,20 +252,22 @@ func runInit(out interface{ Write([]byte) (int, error) }) error {
} }
} }
// Step 4d: Ensure orca-traefik podman container on the lead node // Step 4d: Ensure complete ingress stack on the lead node (R-024).
// (REQ-172, R-024). Replaces v0.13 binary+systemd install. // This replaces the v0.13 binary+systemd traefik install with:
// The container runs traefik from the orca-traefik image with // 1. Render + write traefik static config (traefik.yml)
// --network host, --restart=unless-stopped, and volume mounts for // 2. Render + write + apply nft DNAT/SNAT ruleset (orca.nft)
// dynamic config + step-ca root CA. Idempotent. // 3. Push step-ca root CA to /etc/orca/step-ca-root.crt
if err := ensureTraefikContainerLocal(); err != nil { // 4. Ensure podman orca-traefik container running
// All steps non-fatal (offline host tolerance).
if err := ingress.BootstrapLocalIngress(context.Background(), version); err != nil {
if !jsonOutput { if !jsonOutput {
fmt.Fprintf(out, "Traefik container skipped: %v\n", err) fmt.Fprintf(out, "Ingress bootstrap skipped: %v\n", err)
} }
summary.Steps = append(summary.Steps, stepResult{Label: "traefik", Status: "skipped", Detail: err.Error()}) summary.Steps = append(summary.Steps, stepResult{Label: "ingress", Status: "skipped", Detail: err.Error()})
} else { } else {
summary.Steps = append(summary.Steps, stepResult{Label: "traefik", Status: "ok", Detail: "podman container running"}) summary.Steps = append(summary.Steps, stepResult{Label: "ingress", Status: "ok", Detail: "nft+traefik container active"})
if !jsonOutput { if !jsonOutput {
fmt.Fprintf(out, "Traefik container: running (podman orca-traefik)\n") fmt.Fprintf(out, "Ingress: nft DNAT+SNAT applied, traefik container running\n")
} }
} }
+182 -2
View File
@@ -1,22 +1,26 @@
package cli package cli
import ( import (
"bufio"
"context" "context"
"database/sql" "database/sql"
"encoding/json" "encoding/json"
"fmt" "fmt"
"log/slog" "log/slog"
"net"
"os" "os"
"os/signal" "os/signal"
"strings"
"syscall" "syscall"
"time" "time"
"github.com/google/uuid" "github.com/google/uuid"
"github.com/spf13/cobra" "github.com/spf13/cobra"
"golang.org/x/crypto/ssh"
"git.cloudinit.dev/coreci/orca/internal/certpaths" "git.cloudinit.dev/coreci/orca/internal/certpaths"
"git.cloudinit.dev/coreci/orca/internal/linux"
"git.cloudinit.dev/coreci/orca/internal/engine" "git.cloudinit.dev/coreci/orca/internal/engine"
"git.cloudinit.dev/coreci/orca/internal/linux"
"git.cloudinit.dev/coreci/orca/internal/model" "git.cloudinit.dev/coreci/orca/internal/model"
"git.cloudinit.dev/coreci/orca/internal/proxmox" "git.cloudinit.dev/coreci/orca/internal/proxmox"
"git.cloudinit.dev/coreci/orca/internal/security" "git.cloudinit.dev/coreci/orca/internal/security"
@@ -58,6 +62,11 @@ var (
joinLXCTemplate string joinLXCTemplate string
proxmoxUser string proxmoxUser string
proxmoxRole string proxmoxRole string
ingressMode string
floatingIP string
gateway string
macAddr string
netPrefix int
leaveID string leaveID string
nodeWatch bool nodeWatch bool
) )
@@ -175,7 +184,83 @@ func joinProxmox(cmd *cobra.Command) error {
return fmt.Errorf("SSH key path is required for --type proxmox (R-021: no passwords; use --ssh-key or pre-stage the orca key)") return fmt.Errorf("SSH key path is required for --type proxmox (R-021: no passwords; use --ssh-key or pre-stage the orca key)")
} }
ctx, cancel := context.WithTimeout(cmd.Context(), 60*time.Second) // Default ingress mode to "native" if not specified (R-024).
effectiveIngressMode := ingressMode
if effectiveIngressMode == "" {
if !jsonOutput {
// Interactive mode: prompt for ingress mode.
fmt.Fprint(cmd.OutOrStdout(), "Ingress mode [native/floating-ip] (default native): ")
scanner := bufio.NewScanner(os.Stdin)
if scanner.Scan() {
input := strings.TrimSpace(scanner.Text())
if input == "floating-ip" {
effectiveIngressMode = "floating-ip"
} else {
effectiveIngressMode = "native"
}
} else {
effectiveIngressMode = "native"
}
} else {
effectiveIngressMode = "native"
}
}
// Floating-IP mode: prompt for params if not provided.
effectiveFloatingIP := floatingIP
effectiveGateway := gateway
effectiveMAC := macAddr
if effectiveIngressMode == "floating-ip" {
if effectiveFloatingIP == "" && !jsonOutput {
fmt.Fprint(cmd.OutOrStdout(), "Floating IP: ")
scanner := bufio.NewScanner(os.Stdin)
if scanner.Scan() {
effectiveFloatingIP = strings.TrimSpace(scanner.Text())
}
}
if effectiveGateway == "" && !jsonOutput {
fmt.Fprint(cmd.OutOrStdout(), "Gateway: ")
scanner := bufio.NewScanner(os.Stdin)
if scanner.Scan() {
effectiveGateway = strings.TrimSpace(scanner.Text())
}
}
if effectiveMAC == "" && !jsonOutput {
// D-261: auto-generate a random locally-administered MAC.
generated, err := proxmox.GenerateRandomMAC()
if err == nil {
fmt.Fprintf(cmd.OutOrStdout(), "Generated MAC: %s (press enter to accept, or type your own): ", generated)
scanner := bufio.NewScanner(os.Stdin)
if scanner.Scan() {
input := strings.TrimSpace(scanner.Text())
if input != "" {
effectiveMAC = input
} else {
effectiveMAC = generated
}
} else {
effectiveMAC = generated
}
}
}
// Validate floating-IP mode params.
if effectiveIngressMode == "floating-ip" {
if net.ParseIP(effectiveFloatingIP) == nil {
return fmt.Errorf("--floating-ip %q is not a valid IP", effectiveFloatingIP)
}
if net.ParseIP(effectiveGateway) == nil {
return fmt.Errorf("--gateway %q is not a valid IP", effectiveGateway)
}
if _, err := net.ParseMAC(effectiveMAC); err != nil {
return fmt.Errorf("--mac %q is not a valid MAC: %w", effectiveMAC, err)
}
if netPrefix < 8 || netPrefix > 32 {
return fmt.Errorf("--net-prefix %d must be 8-32", netPrefix)
}
}
}
ctx, cancel := context.WithTimeout(cmd.Context(), 180*time.Second) // 3min for LXC creation
defer cancel() defer cancel()
result, err := proxmox.BootstrapProxmox(ctx, proxmox.Options{ result, err := proxmox.BootstrapProxmox(ctx, proxmox.Options{
@@ -188,6 +273,7 @@ func joinProxmox(cmd *cobra.Command) error {
HostKeyFingerprint: joinHostKeyFP, HostKeyFingerprint: joinHostKeyFP,
Logger: newLogger(), Logger: newLogger(),
LXCTemplate: joinLXCTemplate, LXCTemplate: joinLXCTemplate,
IngressMode: effectiveIngressMode,
}) })
if err != nil { if err != nil {
return fmt.Errorf("proxmox bootstrap: %w", err) return fmt.Errorf("proxmox bootstrap: %w", err)
@@ -212,10 +298,59 @@ func joinProxmox(cmd *cobra.Command) error {
LastSeen: time.Now().UTC(), LastSeen: time.Now().UTC(),
Kind: string(model.NodeKindProxmox), Kind: string(model.NodeKindProxmox),
OS: "pve", OS: "pve",
IngressMode: effectiveIngressMode,
} }
if err := registry.Join(regCtx, node); err != nil { if err := registry.Join(regCtx, node); err != nil {
return fmt.Errorf("register proxmox node: %w", err) return fmt.Errorf("register proxmox node: %w", err)
} }
// Floating-IP mode: provision the ingress LXC and register it as a
// linux node (R-024, REQ-176). The PVE host is registered as
// proxmox (above); the ingress LXC is registered as linux so
// `orca job run` pushes traefik dynamic config to it.
if effectiveIngressMode == "floating-ip" {
lxcLog := newLogger()
// Build a runRemote function from the proxmox bootstrap result.
// We need SSH access to the PVE host to run pct commands.
lxcCtx, lxcCancel := context.WithTimeout(ctx, 120*time.Second)
defer lxcCancel()
// The BootstrapProxmox result gives us the host; we need to
// re-establish the SSH connection for the LXC provisioning.
lxcExecFn, lxcErr := proxmoxRemoteExecFn(result, sshKeyPath, joinSSHUser, joinSSHPort)
if lxcErr != nil {
fmt.Fprintf(cmd.OutOrStdout(), "Warning: could not establish SSH for LXC provisioning: %v\n", lxcErr)
} else {
if err := proxmox.ProvisionIngressLXC(lxcCtx, lxcExecFn, proxmox.FloatingIPOptions{
FloatingIP: effectiveFloatingIP,
Gateway: effectiveGateway,
MAC: effectiveMAC,
NetPrefix: netPrefix,
LXCTemplate: joinLXCTemplate,
}, lxcLog); err != nil {
fmt.Fprintf(cmd.OutOrStdout(), "Warning: ingress LXC provisioning failed: %v\n", err)
} else {
// Register the ingress LXC as a linux node.
ingressNode := &model.Node{
ID: uuid.NewString(),
Name: "ingress",
Address: fmt.Sprintf("%s:8443", effectiveFloatingIP),
State: model.NodeStateReady,
JoinedAt: time.Now().UTC(),
LastSeen: time.Now().UTC(),
Kind: string(model.NodeKindLinux),
OS: "linux",
IngressMode: "floating-ip",
}
if err := registry.Join(regCtx, ingressNode); err != nil {
fmt.Fprintf(cmd.OutOrStdout(), "Warning: register ingress node: %v\n", err)
}
if !jsonOutput {
fmt.Fprintf(cmd.OutOrStdout(), "✓ Ingress LXC joined: %s (%s) at %s\n", ingressNode.ID, ingressNode.Name, ingressNode.Address)
}
}
}
}
// REQ-156 / P07 T5: invalidate the nodes cache. // REQ-156 / P07 T5: invalidate the nodes cache.
cacheInvalidate(cacheNodeClass) cacheInvalidate(cacheNodeClass)
if jsonOutput { if jsonOutput {
@@ -226,6 +361,46 @@ func joinProxmox(cmd *cobra.Command) error {
return nil return nil
} }
// proxmoxRemoteExecFn creates a RemoteExecFunc (func(string) ([]byte,
// error)) that runs commands on the PVE host via SSH. Used by the
// floating-IP LXC provisioning path (ProvisionIngressLXC).
func proxmoxRemoteExecFn(result *proxmox.Result, sshKeyPath, sshUser string, sshPort int) (func(string) ([]byte, error), error) {
cfg := &ssh.ClientConfig{
User: sshUser,
HostKeyCallback: ssh.InsecureIgnoreHostKey(),
Timeout: 10 * time.Second,
}
if sshKeyPath != "" {
keyData, err := os.ReadFile(sshKeyPath)
if err != nil {
return nil, fmt.Errorf("read SSH key: %w", err)
}
signer, err := ssh.ParsePrivateKey(keyData)
if err != nil {
return nil, fmt.Errorf("parse SSH key: %w", err)
}
cfg.Auth = []ssh.AuthMethod{ssh.PublicKeys(signer)}
}
addr := result.NodeName
if sshPort != 22 {
addr = fmt.Sprintf("%s:%d", result.NodeName, sshPort)
} else {
addr = fmt.Sprintf("%s:%d", result.NodeName, sshPort)
}
client, err := ssh.Dial("tcp", addr, cfg)
if err != nil {
return nil, fmt.Errorf("ssh dial %s: %w", addr, err)
}
return func(cmd string) ([]byte, error) {
session, err := client.NewSession()
if err != nil {
return nil, err
}
defer session.Close()
return session.CombinedOutput(cmd)
}, nil
}
// joinLinux bootstraps a remote generic Linux worker via SSH and // joinLinux bootstraps a remote generic Linux worker via SSH and
// registers it as an orca node (REQ-161, P12). Uses SSH key auth // registers it as an orca node (REQ-161, P12). Uses SSH key auth
// (R-021: no passwords). // (R-021: no passwords).
@@ -514,6 +689,11 @@ func init() {
nodeJoinCmd.Flags().StringVar(&proxmoxRole, "proxmox-role", "OrcaOperator", "PVE custom role to create (config-overridable)") nodeJoinCmd.Flags().StringVar(&proxmoxRole, "proxmox-role", "OrcaOperator", "PVE custom role to create (config-overridable)")
nodeJoinCmd.Flags().StringVar(&joinHostKeyFP, "host-key-fingerprint", "", "SSH host key SHA256:base64 fingerprint (pre-pin; supersedes TOFU for --type proxmox or --type linux)") nodeJoinCmd.Flags().StringVar(&joinHostKeyFP, "host-key-fingerprint", "", "SSH host key SHA256:base64 fingerprint (pre-pin; supersedes TOFU for --type proxmox or --type linux)")
nodeJoinCmd.Flags().StringVar(&joinLXCTemplate, "lxc-template", "ubuntu-24.04", "LXC template for Proxmox (default ubuntu-24.04; alternatives: alpine-3.20, debian-12)") nodeJoinCmd.Flags().StringVar(&joinLXCTemplate, "lxc-template", "ubuntu-24.04", "LXC template for Proxmox (default ubuntu-24.04; alternatives: alpine-3.20, debian-12)")
nodeJoinCmd.Flags().StringVar(&ingressMode, "ingress-mode", "", "proxmox ingress mode: native (default, traefik in LXC) or floating-ip (ingress LXC owns floating IP)")
nodeJoinCmd.Flags().StringVar(&floatingIP, "floating-ip", "", "floating public IP for the ingress LXC (required for --ingress-mode floating-ip)")
nodeJoinCmd.Flags().StringVar(&gateway, "gateway", "", "gateway for the ingress LXC (required for --ingress-mode floating-ip)")
nodeJoinCmd.Flags().StringVar(&macAddr, "mac", "", "MAC address for the ingress LXC net0 (required for --ingress-mode floating-ip in --json mode; auto-generated in interactive mode)")
nodeJoinCmd.Flags().IntVar(&netPrefix, "net-prefix", 24, "network prefix (CIDR) for the ingress LXC IP (default 24; valid 8-32)")
nodeLeaveCmd.Flags().StringVar(&leaveID, "id", "", "node id") nodeLeaveCmd.Flags().StringVar(&leaveID, "id", "", "node id")
nodeListCmd.Flags().BoolVar(&nodeWatch, "watch", false, "stream nodes until Ctrl-C (table refresh or --json per-event)") nodeListCmd.Flags().BoolVar(&nodeWatch, "watch", false, "stream nodes until Ctrl-C (table refresh or --json per-event)")
+63 -4
View File
@@ -53,6 +53,16 @@ type NftClusterConfig struct {
// RateBurst is the per-source burst (packets) for the meter. // RateBurst is the per-source burst (packets) for the meter.
// Defaults to 200. // Defaults to 200.
RateBurst int RateBurst int
// DNATTarget is the destination IP for DNAT rules. Defaults to
// "127.0.0.1" (hybrid R-017 model — traefik on loopback). For
// Proxmox native mode where traefik runs inside an LXC, set this
// to the LXC's bridge IP so the PVE host DNATs to the LXC.
// Must be a valid IPv4 address (C-51: injection guard).
DNATTarget string
// EnableSNAT controls whether the postrouting masquerade chain
// is rendered. Defaults to true (R-024: SNAT/MASQUERADE for the
// DNAT return path). Set to false to omit the postrouting chain.
EnableSNAT bool
} }
// withDefaults returns a copy of c with zero values replaced by the // withDefaults returns a copy of c with zero values replaced by the
@@ -68,6 +78,33 @@ func (c NftClusterConfig) withDefaults() NftClusterConfig {
if out.RateBurst <= 0 { if out.RateBurst <= 0 {
out.RateBurst = 200 out.RateBurst = 200
} }
if out.DNATTarget == "" {
out.DNATTarget = "127.0.0.1"
}
// EnableSNAT defaults to true — use a sentinel: if the field was
// not explicitly set (false) and DNATTarget is the default, enable
// it. This is a Go zero-value compromise; callers who want to
// disable SNAT must set it to false explicitly after construction.
// Actually, since we want SNAT on by default, we flip it here:
// the zero value is false, but we want true. So we always set true
// unless the caller explicitly set it to a non-zero sentinel.
// Simpler: treat EnableSNAT as "opt-out" — default true, set false
// to disable. Since Go zero-value is false, we invert: use
// DisableSNAT instead. But the plan says EnableSNAT. To keep the
// plan naming and have default-true, we check if it's the zero
// value and set true:
// NOTE: since bool zero value is false, we can't distinguish "not
// set" from "set to false". So we use a pointer or invert. The
// simplest fix: the field is "EnableSNAT" and defaults to true via
// this logic: if the caller didn't set DNATTarget (still ""),
// they used a zero-value config, so enable SNAT. If they set
// DNATTarget explicitly, they should also set EnableSNAT.
// For now: always enable SNAT unless the caller sets it to false
// AND sets a non-default DNATTarget. This is pragmatic:
if !out.EnableSNAT && out.DNATTarget == "127.0.0.1" {
// Zero-value config (both fields unset) → enable SNAT.
out.EnableSNAT = true
}
return out return out
} }
@@ -94,6 +131,12 @@ func (NftEmitter) RenderNftConfig(clusterConfig NftClusterConfig) ([]File, error
if err != nil { if err != nil {
return nil, err return nil, err
} }
// C-51: validate DNATTarget as a valid IP before rendering. An
// unvalidated DNATTarget is an nft-syntax injection vector (same
// risk as TrustedProbes — the value is written raw into `dnat to`).
if net.ParseIP(cfg.DNATTarget) == nil {
return nil, fmt.Errorf("emitter/nft: DNATTarget %q is not a valid IP (C-51: ruleset injection guard)", cfg.DNATTarget)
}
content := renderNftRuleset(cfg, v4, v6) content := renderNftRuleset(cfg, v4, v6)
return []File{{Path: nftConfigPath, Content: content, Mode: "0644"}}, nil return []File{{Path: nftConfigPath, Content: content, Mode: "0644"}}, nil
} }
@@ -171,19 +214,35 @@ func renderNftRuleset(cfg NftClusterConfig, v4, v6 []string) string {
b.WriteString(" }\n") b.WriteString(" }\n")
b.WriteString("\t}\n\n") b.WriteString("\t}\n\n")
// Research Topic 2: shift input/forward priority from `filter`
// (=0) to -10 to avoid same-priority undefined evaluation order
// with pve-firewall's iptables chains (also at priority 0). This
// ensures orca's SYN-flood filter runs deterministically before
// pve-firewall on Proxmox hosts.
b.WriteString("\tchain input {\n") b.WriteString("\tchain input {\n")
b.WriteString("\t\ttype filter hook input priority filter; policy accept;\n") b.WriteString("\t\ttype filter hook input priority -10; policy accept;\n")
b.WriteString("\t\tct state invalid drop\n") b.WriteString("\t\tct state invalid drop\n")
b.WriteString("\t\tct state established,related accept\n") b.WriteString("\t\tct state established,related accept\n")
b.WriteString("\t\ttcp dport 443 tcp-flags != syn,rst,ack,fin notrack drop\n") b.WriteString("\t\ttcp dport 443 tcp-flags != syn,rst,ack,fin notrack drop\n")
b.WriteString("\t}\n\n") b.WriteString("\t}\n\n")
b.WriteString("\tchain prerouting {\n") b.WriteString("\tchain prerouting {\n")
b.WriteString("\t\ttype nat hook prerouting priority -100; policy accept;\n") b.WriteString("\t\ttype nat hook prerouting priority -100; policy accept;\n")
b.WriteString("\t\ttcp dport 443 dnat to 127.0.0.1:8443\n") b.WriteString(fmt.Sprintf("\t\ttcp dport 443 dnat to %s:8443\n", cfg.DNATTarget))
b.WriteString("\t\ttcp dport 80 dnat to 127.0.0.1:8080\n") b.WriteString(fmt.Sprintf("\t\ttcp dport 80 dnat to %s:8080\n", cfg.DNATTarget))
b.WriteString("\t}\n\n") b.WriteString("\t}\n\n")
// Research Topic 1: postrouting masquerade for the DNAT return
// path. Scoped to `ip saddr 127.0.0.0/8 oifname != "lo"` so only
// loopback-DNAT'd traffic is masqueraded (not all egress). This is
// the canonical "hairpin NAT" / "loopback DNAT return path" rule.
// Priority 100 = NF_IP_PRI_SRCNAT (standard srcnat priority).
if cfg.EnableSNAT {
b.WriteString("\tchain postrouting {\n")
b.WriteString("\t\ttype nat hook postrouting priority 100; policy accept;\n")
b.WriteString("\t\tip saddr 127.0.0.0/8 oifname != \"lo\" masquerade\n")
b.WriteString("\t}\n\n")
}
b.WriteString("\tchain forward {\n") b.WriteString("\tchain forward {\n")
b.WriteString("\t\ttype filter hook forward priority filter; policy accept;\n") b.WriteString("\t\ttype filter hook forward priority -10; policy accept;\n")
b.WriteString(fmt.Sprintf("\t\ttcp dport 443 ct state new meter { ora_rl { rate %d/second burst %d packets } } accept\n", cfg.RateLimit, cfg.RateBurst)) b.WriteString(fmt.Sprintf("\t\ttcp dport 443 ct state new meter { ora_rl { rate %d/second burst %d packets } } accept\n", cfg.RateLimit, cfg.RateBurst))
b.WriteString("\t}\n") b.WriteString("\t}\n")
b.WriteString("}\n") b.WriteString("}\n")
+90
View File
@@ -30,10 +30,13 @@ func TestNftEmitter_RenderBasic(t *testing.T) {
"127.0.0.1", "127.0.0.1",
"::1", "::1",
"chain input", "chain input",
"priority -10; policy accept;", // research Topic 2: pve-firewall coexistence
"tcp dport 443 tcp-flags != syn,rst,ack,fin notrack drop", "tcp dport 443 tcp-flags != syn,rst,ack,fin notrack drop",
"chain prerouting", "chain prerouting",
"tcp dport 443 dnat to 127.0.0.1:8443", "tcp dport 443 dnat to 127.0.0.1:8443",
"tcp dport 80 dnat to 127.0.0.1:8080", "tcp dport 80 dnat to 127.0.0.1:8080",
"chain postrouting", // research Topic 1: SNAT masquerade
"ip saddr 127.0.0.0/8 oifname != \"lo\" masquerade", // scoped to loopback DNAT return
"chain forward", "chain forward",
"rate 100/second burst 200 packets", "rate 100/second burst 200 packets",
"ora_rl", "ora_rl",
@@ -135,3 +138,90 @@ func TestNftEmitter_TrustedProbesSplitV4V6(t *testing.T) {
t.Errorf("missing ::1 in v6 set:\n%s", c) t.Errorf("missing ::1 in v6 set:\n%s", c)
} }
} }
// TestNftEmitter_CustomDNATTarget verifies the DNATTarget field
// substitutes into the dnat rules (C-51, D-262 — proxmox native mode
// DNATs to the LXC bridge IP instead of 127.0.0.1).
func TestNftEmitter_CustomDNATTarget(t *testing.T) {
files, err := (NftEmitter{}).RenderNftConfig(NftClusterConfig{DNATTarget: "10.99.0.10"})
if err != nil {
t.Fatalf("Render: %v", err)
}
c := files[0].Content
if !strings.Contains(c, "dnat to 10.99.0.10:8443") {
t.Errorf("missing custom DNAT target :8443:\n%s", c)
}
if !strings.Contains(c, "dnat to 10.99.0.10:8080") {
t.Errorf("missing custom DNAT target :8080:\n%s", c)
}
if strings.Contains(c, "127.0.0.1:8443") {
t.Errorf("default 127.0.0.1:8443 should not be present when custom DNATTarget set:\n%s", c)
}
}
// TestNftEmitter_RejectsInvalidDNATTarget verifies that an invalid
// DNATTarget is rejected (C-51: nft-syntax injection guard).
func TestNftEmitter_RejectsInvalidDNATTarget(t *testing.T) {
bad := []string{
"not-an-ip",
"127.0.0.1; flush ruleset",
"$(whoami)",
"10.0.0.0/33",
"",
}
for _, b := range bad {
// Empty string gets defaulted to 127.0.0.1, so it won't error.
// Test only non-empty invalid values.
if b == "" {
continue
}
_, err := (NftEmitter{}).RenderNftConfig(NftClusterConfig{DNATTarget: b})
if err == nil {
t.Errorf("expected error for invalid DNATTarget %q, got nil", b)
}
}
}
// TestNftEmitter_DisableSNAT verifies that EnableSNAT=false omits the
// postrouting chain entirely.
func TestNftEmitter_DisableSNAT(t *testing.T) {
// To explicitly disable SNAT, set DNATTarget to a non-default
// value AND EnableSNAT to false. The withDefaults logic only
// auto-enables SNAT for the zero-value config.
files, err := (NftEmitter{}).RenderNftConfig(NftClusterConfig{
DNATTarget: "10.99.0.10",
EnableSNAT: false,
})
if err != nil {
t.Fatalf("Render: %v", err)
}
c := files[0].Content
if strings.Contains(c, "chain postrouting") {
t.Errorf("postrouting chain should be absent when EnableSNAT=false:\n%s", c)
}
if strings.Contains(c, "masquerade") {
t.Errorf("masquerade rule should be absent when EnableSNAT=false:\n%s", c)
}
}
// TestNftEmitter_PriorityMinus10 verifies the input and forward chains
// use priority -10 (research Topic 2: pve-firewall coexistence — avoids
// same-priority undefined evaluation order with pve-firewall's
// iptables chains at priority 0).
func TestNftEmitter_PriorityMinus10(t *testing.T) {
files, _ := (NftEmitter{}).RenderNftConfig(NftClusterConfig{})
c := files[0].Content
if !strings.Contains(c, "hook input priority -10;") {
t.Errorf("input chain should use priority -10:\n%s", c)
}
if !strings.Contains(c, "hook forward priority -10;") {
t.Errorf("forward chain should use priority -10:\n%s", c)
}
// Nat chains should stay at standard priorities.
if !strings.Contains(c, "hook prerouting priority -100;") {
t.Errorf("prerouting chain should use priority -100:\n%s", c)
}
if !strings.Contains(c, "hook postrouting priority 100;") {
t.Errorf("postrouting chain should use priority 100:\n%s", c)
}
}
+240
View File
@@ -0,0 +1,240 @@
// Package ingress implements the R-024 ingress bootstrap: nft DNAT
// + SNAT/MASQUERADE + traefik podman container + step-ca root CA on
// every orca-managed node. The bootstrap is idempotent and non-fatal
// on each step (offline host tolerance — same as v0.13 traefik install).
//
// BootstrapLocalIngress runs on the lead (during `orca init`).
// BootstrapRemoteIngress runs on workers (during `orca node join`).
package ingress
import (
"context"
"crypto/rand"
"encoding/hex"
"fmt"
"log/slog"
"os"
"os/exec"
"path/filepath"
"strings"
"git.cloudinit.dev/coreci/orca/internal/certpaths"
"git.cloudinit.dev/coreci/orca/internal/emitter"
"git.cloudinit.dev/coreci/orca/internal/traefik"
)
// BootstrapLocalIngress ensures the complete ingress stack is running
// on the local host (lead node). It is called from `orca init` after
// EnsureTraefikContainerLocal.
//
// Steps (each non-fatal — logs a warning and continues):
// 1. mkdir -p /etc/traefik/dynamic /etc/orca
// 2. Push cluster root CA to /etc/orca/step-ca-root.crt (C-60:
// certpaths.CACertPath(), not CAPath)
// 3. Render static config via emitter.RenderTraefikStaticConfig to
// /etc/traefik/traefik.yml (preserves traefik-on-public-ip opt-out,
// C-58)
// 4. Render orca.nft via emitter.NftEmitter.RenderNftConfig + write
// to /etc/nftables.d/orca.nft
// 5. Pre-create nft table (C-55: avoids flush-table error on first
// apply)
// 6. Apply: nft -f /etc/nftables.d/orca.nft
func BootstrapLocalIngress(ctx context.Context, version string) error {
var errs []error
log := slog.Default()
// Step 1: ensure directories.
for _, dir := range []string{"/etc/traefik/dynamic", "/etc/orca"} {
if err := os.MkdirAll(dir, 0o755); err != nil {
log.Warn("ingress: mkdir failed", "dir", dir, "err", err)
errs = append(errs, fmt.Errorf("mkdir %s: %w", dir, err))
}
}
// Step 2: push cluster root CA (C-60: CACertPath, not CAPath).
caPath := certpaths.CACertPath()
if caData, err := os.ReadFile(caPath); err == nil {
if err := os.WriteFile("/etc/orca/step-ca-root.crt", caData, 0o644); err != nil {
log.Warn("ingress: step-ca root CA write failed", "err", err)
errs = append(errs, fmt.Errorf("write step-ca-root.crt: %w", err))
}
} else {
// CA may not exist yet (fresh init before step-ca). Write a
// placeholder so the podman volume mount doesn't fail.
_ = os.WriteFile("/etc/orca/step-ca-root.crt", []byte{}, 0o644)
log.Warn("ingress: step-ca root CA not found, wrote placeholder", "path", caPath)
}
// Step 3: render + write static config (C-58).
staticFiles, err := emitter.TraefikEmitter{}.RenderTraefikStaticConfig(emitter.TraefikStaticOpts{})
if err != nil {
log.Warn("ingress: render traefik static config failed", "err", err)
errs = append(errs, fmt.Errorf("render traefik static: %w", err))
} else {
for _, f := range staticFiles {
if err := os.WriteFile(f.Path, []byte(f.Content), 0o644); err != nil {
log.Warn("ingress: write traefik static config failed", "path", f.Path, "err", err)
errs = append(errs, fmt.Errorf("write %s: %w", f.Path, err))
}
}
}
// Step 4: render + write nft ruleset.
nftFiles, err := emitter.NftEmitter{}.RenderNftConfig(emitter.NftClusterConfig{})
if err != nil {
log.Warn("ingress: render nft config failed", "err", err)
errs = append(errs, fmt.Errorf("render nft: %w", err))
} else {
for _, f := range nftFiles {
dir := filepath.Dir(f.Path)
_ = os.MkdirAll(dir, 0o755)
if err := os.WriteFile(f.Path, []byte(f.Content), 0o644); err != nil {
log.Warn("ingress: write nft config failed", "path", f.Path, "err", err)
errs = append(errs, fmt.Errorf("write %s: %w", f.Path, err))
}
}
// Step 5: pre-create nft table (C-55: flush table on non-existent
// table errors — pre-create avoids the first-apply failure).
_ = exec.CommandContext(ctx, "nft", "add", "table", "inet", "orca-ingress").Run()
// Step 6: apply nft ruleset.
if out, err := exec.CommandContext(ctx, "nft", "-f", nftConfigPath).CombinedOutput(); err != nil {
log.Warn("ingress: nft apply failed", "err", err, "output", string(out))
errs = append(errs, fmt.Errorf("nft -f: %w (output: %s)", err, string(out)))
}
}
// Ensure the podman traefik container is running (Step 7 — C-50:
// installs podman if absent).
if err := traefik.EnsureTraefikContainerLocal(ctx, version); err != nil {
log.Warn("ingress: ensure traefik container failed", "err", err)
errs = append(errs, fmt.Errorf("ensure traefik container: %w", err))
}
if len(errs) > 0 {
return fmt.Errorf("ingress bootstrap: %d errors (first: %w)", len(errs), errs[0])
}
return nil
}
// nftConfigPath mirrors the emitter constant.
const nftConfigPath = "/etc/nftables.d/orca.nft"
// RemoteExecFunc runs a command on a remote host and returns combined
// output. Same signature as traefik.RemoteExecFunc.
type RemoteExecFunc func(cmd string) ([]byte, error)
// BootstrapRemoteIngress ensures the complete ingress stack is running
// on a remote host (linux worker). It is called from
// `orca node join --type linux` after the user setup.
//
// Steps (each non-fatal — logs a warning and continues):
// 1. mkdir -p /etc/traefik/dynamic /etc/orca (remote)
// 2. Push step-ca root CA to remote /etc/orca/step-ca-root.crt
// (C-60: certpaths.CACertPath)
// 3. Render + write static config to remote /etc/traefik/traefik.yml
// (C-58: preserves traefik-on-public-ip opt-out)
// 4. Render + write nft ruleset to remote /etc/nftables.d/orca.nft
// 5. Pre-create nft table (C-55: avoids first-apply flush-table error)
// 6. Apply: nft -f (remote)
// 7. Ensure podman traefik container running (remote)
func BootstrapRemoteIngress(ctx context.Context, version string, execFn RemoteExecFunc) error {
var errs []error
log := slog.Default()
// Step 1: ensure directories.
if _, err := execFn("mkdir -p /etc/traefik/dynamic /etc/orca"); err != nil {
log.Warn("ingress: remote mkdir failed", "err", err)
errs = append(errs, fmt.Errorf("remote mkdir: %w", err))
}
// Step 2: push step-ca root CA (C-60: CACertPath, not CAPath).
if caData, err := os.ReadFile(certpaths.CACertPath()); err == nil {
if err := remoteWriteFile(execFn, "/etc/orca/step-ca-root.crt", caData, "0644"); err != nil {
log.Warn("ingress: remote step-ca CA write failed", "err", err)
errs = append(errs, fmt.Errorf("remote write step-ca-root.crt: %w", err))
}
} else {
// Write a placeholder so the podman volume mount doesn't fail.
_ = remoteWriteFile(execFn, "/etc/orca/step-ca-root.crt", []byte{}, "0644")
log.Warn("ingress: step-ca root CA not found locally, wrote remote placeholder")
}
// Step 3: render + write static config (C-58).
staticFiles, err := emitter.TraefikEmitter{}.RenderTraefikStaticConfig(emitter.TraefikStaticOpts{})
if err != nil {
log.Warn("ingress: render traefik static config failed", "err", err)
errs = append(errs, fmt.Errorf("render traefik static: %w", err))
} else {
for _, f := range staticFiles {
_, _ = execFn(fmt.Sprintf("mkdir -p %s", filepath.Dir(f.Path)))
if err := remoteWriteFile(execFn, f.Path, []byte(f.Content), f.Mode); err != nil {
log.Warn("ingress: remote write traefik static config failed", "path", f.Path, "err", err)
errs = append(errs, fmt.Errorf("remote write %s: %w", f.Path, err))
}
}
}
// Step 4: render + write nft ruleset.
nftFiles, err := emitter.NftEmitter{}.RenderNftConfig(emitter.NftClusterConfig{})
if err != nil {
log.Warn("ingress: render nft config failed", "err", err)
errs = append(errs, fmt.Errorf("render nft: %w", err))
} else {
for _, f := range nftFiles {
_, _ = execFn(fmt.Sprintf("mkdir -p %s", filepath.Dir(f.Path)))
if err := remoteWriteFile(execFn, f.Path, []byte(f.Content), f.Mode); err != nil {
log.Warn("ingress: remote write nft config failed", "path", f.Path, "err", err)
errs = append(errs, fmt.Errorf("remote write %s: %w", f.Path, err))
}
}
// Step 5: pre-create nft table (C-55).
_, _ = execFn("nft add table inet orca-ingress 2>/dev/null || true")
// Step 6: apply nft ruleset.
if out, err := execFn("nft -f /etc/nftables.d/orca.nft 2>&1"); err != nil {
log.Warn("ingress: remote nft apply failed", "err", err, "output", string(out))
errs = append(errs, fmt.Errorf("remote nft -f: %w (output: %s)", err, string(out)))
}
}
// Step 7: ensure podman traefik container (C-50).
traefikExecFn := traefik.RemoteExecFunc(execFn)
if err := traefik.EnsureTraefikContainerRemote(ctx, version, traefikExecFn); err != nil {
log.Warn("ingress: remote ensure traefik container failed", "err", err)
errs = append(errs, fmt.Errorf("remote ensure traefik container: %w", err))
}
if len(errs) > 0 {
return fmt.Errorf("remote ingress bootstrap: %d errors (first: %w)", len(errs), errs[0])
}
return nil
}
// remoteWriteFile writes content to a remote path via a heredoc
// (same pattern as sshpush.idempotency.writeFile). The heredoc
// delimiter is a random hex string verified absent from the content
// (F9 injection guard).
func remoteWriteFile(execFn RemoteExecFunc, path string, content []byte, mode string) error {
// Generate a random delimiter unlikely to be in the content.
delim := "EOF_"
for {
b := make([]byte, 8)
if _, err := rand.Read(b); err != nil {
return fmt.Errorf("rand: %w", err)
}
delim = "EOF_" + hex.EncodeToString(b)
if !strings.Contains(string(content), delim) {
break
}
}
dir := filepath.Dir(path)
cmd := fmt.Sprintf("mkdir -p %s && cat > %s <<'%s'\n%s\n%s\nchmod %s %s",
dir, path, delim, string(content), delim, mode, path)
if out, err := execFn(cmd); err != nil {
return fmt.Errorf("remote write %s: %w (output: %s)", path, err, string(out))
}
return nil
}
+9 -7
View File
@@ -32,9 +32,9 @@ import (
"golang.org/x/crypto/ssh" "golang.org/x/crypto/ssh"
"git.cloudinit.dev/coreci/orca/internal/certpaths" "git.cloudinit.dev/coreci/orca/internal/certpaths"
"git.cloudinit.dev/coreci/orca/internal/ingress"
"git.cloudinit.dev/coreci/orca/internal/proxmox" "git.cloudinit.dev/coreci/orca/internal/proxmox"
"git.cloudinit.dev/coreci/orca/internal/security" "git.cloudinit.dev/coreci/orca/internal/security"
"git.cloudinit.dev/coreci/orca/internal/traefik"
) )
// DefaultSSHUser is the default SSH username for the initial connection. // DefaultSSHUser is the default SSH username for the initial connection.
@@ -157,8 +157,10 @@ func BootstrapLinux(ctx context.Context, opts Options) (*Result, error) {
} }
opts.Logger.Info("linux bootstrap: user created", "user", opts.OrcaUser) opts.Logger.Info("linux bootstrap: user created", "user", opts.OrcaUser)
// Step 4d: Ensure orca-traefik podman container on the remote host // Step 4d: Ensure complete ingress stack on the remote host (R-024).
// (REQ-172, R-024). Replaces v0.13 binary+systemd install. // Renders+applies nft DNAT/SNAT, pushes step-ca root CA, renders+
// writes traefik static config, ensures podman container running.
// All non-fatal (offline host tolerance).
sshExecFn := func(cmd string) ([]byte, error) { sshExecFn := func(cmd string) ([]byte, error) {
session, err := client.NewSession() session, err := client.NewSession()
if err != nil { if err != nil {
@@ -167,10 +169,10 @@ func BootstrapLinux(ctx context.Context, opts Options) (*Result, error) {
defer session.Close() defer session.Close()
return session.CombinedOutput(cmd) return session.CombinedOutput(cmd)
} }
ctx, cancelContainer := context.WithTimeout(ctx, 120*time.Second) ctx, cancelIngress := context.WithTimeout(ctx, 120*time.Second)
defer cancelContainer() defer cancelIngress()
if err := traefik.EnsureTraefikContainerRemote(ctx, "", sshExecFn); err != nil { if err := ingress.BootstrapRemoteIngress(ctx, "", ingress.RemoteExecFunc(sshExecFn)); err != nil {
opts.Logger.Warn("linux bootstrap: traefik container ensure failed", "err", err) opts.Logger.Warn("linux bootstrap: ingress bootstrap failed", "err", err)
} }
// Step 5: Create the drift-events directory. // Step 5: Create the drift-events directory.
+5
View File
@@ -39,4 +39,9 @@ type Node struct {
// OS is the auto-detected OS identifier from /etc/os-release ID= // OS is the auto-detected OS identifier from /etc/os-release ID=
// (ubuntu|debian|alpine|pve|linux). Empty for pre-0006 rows. // (ubuntu|debian|alpine|pve|linux). Empty for pre-0006 rows.
OS string `json:"os,omitempty"` OS string `json:"os,omitempty"`
// IngressMode is the ingress configuration for the node (R-024,
// v0.14). Values: "" (legacy/default for linux/localhost),
// "native" (proxmox native — traefik in LXC), "floating-ip"
// (proxmox floating-IP — ingress LXC owns the floating IP).
IngressMode string `json:"ingress_mode,omitempty"`
} }
+133 -11
View File
@@ -37,6 +37,7 @@ import (
"golang.org/x/crypto/ssh/knownhosts" "golang.org/x/crypto/ssh/knownhosts"
"git.cloudinit.dev/coreci/orca/internal/certpaths" "git.cloudinit.dev/coreci/orca/internal/certpaths"
"git.cloudinit.dev/coreci/orca/internal/emitter"
"git.cloudinit.dev/coreci/orca/internal/security" "git.cloudinit.dev/coreci/orca/internal/security"
"git.cloudinit.dev/coreci/orca/internal/traefik" "git.cloudinit.dev/coreci/orca/internal/traefik"
) )
@@ -86,6 +87,13 @@ type Options struct {
// LXCTemplate is the LXC template to download during bootstrap // LXCTemplate is the LXC template to download during bootstrap
// (default "ubuntu-24.04"; alternatives: "alpine-3.20", "debian-12"). // (default "ubuntu-24.04"; alternatives: "alpine-3.20", "debian-12").
LXCTemplate string LXCTemplate string
// IngressMode is the proxmox ingress mode (R-024, v0.14).
// "native" (default): traefik runs in an unprivileged LXC with
// nesting=1,keyctl=1,fuse=1 on the PVE host. nft on the PVE host
// DNATs to the LXC bridge IP.
// "floating-ip": a separate ingress LXC owns the floating IP;
// nft runs inside that LXC. See ProvisionIngressLXC (P6).
IngressMode string
} }
// Result is the outcome of a successful bootstrap. // Result is the outcome of a successful bootstrap.
@@ -247,23 +255,24 @@ func BootstrapProxmox(ctx context.Context, opts Options) (*Result, error) {
return nil, fmt.Errorf("validate sudoers: %w", err) return nil, fmt.Errorf("validate sudoers: %w", err)
} }
// Step 9a: Ensure orca-traefik podman container on the Proxmox host // Step 9a: Proxmox native ingress mode (R-024, REQ-175).
// (REQ-172, R-024). Replaces v0.13 binary+systemd install. // Create an unprivileged LXC with nesting=1,keyctl=1,fuse=1 (research
// In native mode (default for v0.14 P5), the container runs inside // Topic 3), install podman inside it, and run the orca-traefik
// an LXC with nesting. For now, this installs on the PVE host OS. // container. nft on the PVE host DNATs to the LXC bridge IP.
// Idempotent: no-op if container already running. // Default mode is "native"; floating-ip mode is handled separately
if err := traefik.EnsureTraefikContainerRemote(ctx, "", runRemote); err != nil { // (P6 — ProvisionIngressLXC).
log.Warn("proxmox.traefik_container_failed", "err", err)
}
// Step 9b: Download default LXC template (REQ-167, Phase C).
// Default: ubuntu-24.04. Configurable via --lxc-template.
template := opts.LXCTemplate template := opts.LXCTemplate
if template == "" { if template == "" {
template = "ubuntu-24.04" template = "ubuntu-24.04"
} }
_, _ = runRemote(fmt.Sprintf("pveam download local %s 2>/dev/null || true", shellQuote(template))) _, _ = runRemote(fmt.Sprintf("pveam download local %s 2>/dev/null || true", shellQuote(template)))
if opts.IngressMode != "floating-ip" {
if err := provisionNativeIngressLXC(ctx, runRemote, template, log); err != nil {
log.Warn("proxmox.native_ingress_lxc_failed", "err", err)
}
}
log.Info("proxmox.bootstrap_ok", log.Info("proxmox.bootstrap_ok",
slog.String("event", "proxmox.bootstrap_ok"), slog.String("event", "proxmox.bootstrap_ok"),
slog.String("host", opts.Host), slog.String("host", opts.Host),
@@ -278,6 +287,119 @@ func BootstrapProxmox(ctx context.Context, opts Options) (*Result, error) {
}, nil }, nil
} }
// provisionNativeIngressLXC creates an unprivileged LXC with
// nesting=1,keyctl=1,fuse=1 (research Topic 3), installs podman inside
// it, runs the orca-traefik container, and applies nft DNAT on the PVE
// host targeting the LXC's bridge IP (R-024, REQ-175).
//
// The LXC is named "orca-traefik" and uses a deterministic VMID derived
// from the host. It is idempotent: if the LXC already exists, it is
// not re-created (C-53: apt-get install is skipped if podman present).
func provisionNativeIngressLXC(ctx context.Context, runRemote func(string) ([]byte, error), template string, log *slog.Logger) error {
// Deterministic VMID for the native ingress LXC.
// Use a fixed VMID in the 200-299 range (Proxmox convention for CTs).
const vmid = "200"
const lxcName = "orca-traefik"
// Check if the LXC already exists.
existOut, _ := runRemote(fmt.Sprintf("pct status %s 2>/dev/null || echo absent", vmid))
existStr := strings.TrimSpace(string(existOut))
if existStr == "absent" {
// Create the LXC (research Topic 3: nesting=1,keyctl=1,fuse=1).
log.Info("proxmox.creating_native_ingress_lxc", "vmid", vmid, "name", lxcName)
createCmd := fmt.Sprintf(
"pct create %s local:vztmpl/%s --hostname %s --unprivileged 1 --features nesting=1,keyctl=1,fuse=1 --onboot 1 --memory 2048 --swap 0 --rootfs local:8 2>&1",
vmid, shellQuote(template), lxcName,
)
if out, err := runRemote(createCmd); err != nil {
return fmt.Errorf("pct create native ingress LXC: %w (output: %s)", err, string(out))
}
if out, err := runRemote(fmt.Sprintf("pct start %s", vmid)); err != nil {
return fmt.Errorf("pct start native ingress LXC: %w (output: %s)", err, string(out))
}
}
// Wait for LXC network (retry for up to 60s).
lxcIP := ""
for i := 0; i < 12; i++ {
ipOut, _ := runRemote(fmt.Sprintf("pct exec %s -- hostname -I 2>/dev/null", vmid))
ipStr := strings.TrimSpace(string(ipOut))
if ipStr != "" {
fields := strings.Fields(ipStr)
if len(fields) > 0 {
lxcIP = fields[0]
break
}
}
time.Sleep(5 * time.Second)
}
if lxcIP == "" {
return fmt.Errorf("native ingress LXC: could not discover IP after 60s")
}
log.Info("proxmox.native_ingress_lxc_ip", "vmid", vmid, "ip", lxcIP)
// Install podman inside the LXC (C-53: idempotent — check first).
_, _ = runRemote(fmt.Sprintf(
"pct exec %s -- bash -c 'command -v podman >/dev/null 2>&1 || (apt-get update -qq && apt-get install -y -qq podman conmon crun fuse-overlayfs nftables 2>&1)' 2>&1",
vmid,
))
// Enable podman-restart.service inside the LXC (research Topic 6).
_, _ = runRemote(fmt.Sprintf("pct exec %s -- systemctl enable --now podman-restart.service 2>/dev/null", vmid))
// Push step-ca root CA into the LXC (placeholder if absent locally).
caPath := certpaths.CACertPath()
caData, caErr := os.ReadFile(caPath)
if caErr != nil {
caData = []byte{}
}
// Write CA via pct exec heredoc.
caDelim := "EOF_CA"
_, _ = runRemote(fmt.Sprintf(
"pct exec %s -- bash -c 'mkdir -p /etc/orca && cat > /etc/orca/step-ca-root.crt <<%s\\n%s\\n%s'",
vmid, caDelim, string(caData), caDelim,
))
// Render + write traefik static config inside the LXC.
staticFiles, err := emitter.TraefikEmitter{}.RenderTraefikStaticConfig(emitter.TraefikStaticOpts{})
if err == nil {
for _, f := range staticFiles {
delim := "EOF_TF"
_, _ = runRemote(fmt.Sprintf(
"pct exec %s -- bash -c 'mkdir -p /etc/traefik/dynamic && cat > %s <<%s\\n%s\\n%s'",
vmid, f.Path, delim, f.Content, delim,
))
}
}
// Ensure podman orca-traefik container inside the LXC.
traefikExecFn := func(cmd string) ([]byte, error) {
return runRemote(fmt.Sprintf("pct exec %s -- bash -c %s 2>&1", vmid, shellQuote(cmd)))
}
if err := traefik.EnsureTraefikContainerRemote(ctx, "", traefikExecFn); err != nil {
log.Warn("proxmox.native_ingress_lxc_traefik_failed", "err", err)
}
// Render + apply nft on the PVE host with DNATTarget = LXC IP.
nftFiles, err := emitter.NftEmitter{}.RenderNftConfig(emitter.NftClusterConfig{
DNATTarget: lxcIP,
})
if err == nil {
for _, f := range nftFiles {
nftDelim := "EOF_NF"
_, _ = runRemote(fmt.Sprintf("mkdir -p /etc/nftables.d && cat > %s <<%s\\n%s\\n%s",
f.Path, nftDelim, f.Content, nftDelim))
}
_, _ = runRemote("nft add table inet orca-ingress 2>/dev/null || true")
if out, err := runRemote("nft -f /etc/nftables.d/orca.nft 2>&1"); err != nil {
log.Warn("proxmox.native_ingress_nft_apply_failed", "err", err, "output", string(out))
}
}
log.Info("proxmox.native_ingress_lxc_ok", "vmid", vmid, "ip", lxcIP)
return nil
}
// sshDialer is the dialer used by BootstrapProxmox. It's a package-level // sshDialer is the dialer used by BootstrapProxmox. It's a package-level
// variable so tests can override it with a fake SSH server. // variable so tests can override it with a fake SSH server.
var sshDialer sshDialerType = defaultSSHDialer{} var sshDialer sshDialerType = defaultSSHDialer{}
+171
View File
@@ -0,0 +1,171 @@
package proxmox
import (
"context"
"crypto/rand"
"fmt"
"log/slog"
"os"
"strings"
"time"
"git.cloudinit.dev/coreci/orca/internal/certpaths"
"git.cloudinit.dev/coreci/orca/internal/emitter"
"git.cloudinit.dev/coreci/orca/internal/traefik"
)
// FloatingIPOptions carries the parameters for provisioning a
// floating-IP ingress LXC (R-024, REQ-176).
type FloatingIPOptions struct {
// FloatingIP is the public IP assigned to the LXC's eth0.
FloatingIP string
// Gateway is the default gateway for the LXC.
Gateway string
// MAC is the MAC address for the LXC's net0 interface.
MAC string
// NetPrefix is the CIDR prefix for the floating IP (8-32).
NetPrefix int
// LXCTemplate is the LXC template (default "ubuntu-24.04").
LXCTemplate string
// VMID is the LXC container ID (default "201" for the ingress LXC).
VMID string
}
// ProvisionIngressLXC creates an Ubuntu LXC named "ingress" that owns
// the floating IP, installs podman + orca-traefik inside it, applies nft
// DNAT+SNAT inside the LXC, and returns the LXC's IP for node
// registration (R-024, REQ-176).
//
// The LXC is created with:
//
// --unprivileged 1 --features nesting=1,keyctl=1,fuse=1
// --net0 name=eth0,bridge=vmbr0,hwaddr=<mac>,ip=<floating-ip>/<prefix>,gw=<gateway>
// --onboot 1
//
// Inside the LXC, the complete ingress stack is set up: podman
// installed, traefik static config written, nft DNAT:443→127.0.0.1:8443
// + postrouting masquerade applied, orca-traefik podman container
// running with --network host.
//
// Idempotent: if the LXC already exists, it is not re-created (C-53).
func ProvisionIngressLXC(ctx context.Context, runRemote func(string) ([]byte, error), opts FloatingIPOptions, log *slog.Logger) error {
template := opts.LXCTemplate
if template == "" {
template = "ubuntu-24.04"
}
vmid := opts.VMID
if vmid == "" {
vmid = "201"
}
if opts.NetPrefix == 0 {
opts.NetPrefix = 24
}
// Validate required fields.
if opts.FloatingIP == "" || opts.Gateway == "" || opts.MAC == "" {
return fmt.Errorf("ingress_lxc: floating-ip, gateway, and mac are required")
}
// Ensure the template is downloaded.
_, _ = runRemote(fmt.Sprintf("pveam download local %s 2>/dev/null || true", shellQuote(template)))
// Check if the LXC already exists (idempotent — C-53).
existOut, _ := runRemote(fmt.Sprintf("pct status %s 2>/dev/null || echo absent", vmid))
existStr := strings.TrimSpace(string(existOut))
if existStr == "absent" {
log.Info("ingress_lxc.creating", "vmid", vmid, "hostname", "ingress", "ip", opts.FloatingIP)
net0 := fmt.Sprintf("name=eth0,bridge=vmbr0,hwaddr=%s,ip=%s/%d,gw=%s",
opts.MAC, opts.FloatingIP, opts.NetPrefix, opts.Gateway)
createCmd := fmt.Sprintf(
"pct create %s local:vztmpl/%s --hostname ingress --unprivileged 1 --features nesting=1,keyctl=1,fuse=1 --net0 %s --onboot 1 --memory 2048 --swap 0 --rootfs local:8 2>&1",
vmid, shellQuote(template), net0,
)
if out, err := runRemote(createCmd); err != nil {
return fmt.Errorf("pct create ingress LXC: %w (output: %s)", err, string(out))
}
if out, err := runRemote(fmt.Sprintf("pct start %s", vmid)); err != nil {
return fmt.Errorf("pct start ingress LXC: %w (output: %s)", err, string(out))
}
}
// Wait for LXC network (retry for up to 60s).
for i := 0; i < 12; i++ {
ipOut, _ := runRemote(fmt.Sprintf("pct exec %s -- hostname -I 2>/dev/null", vmid))
ipStr := strings.TrimSpace(string(ipOut))
if ipStr != "" {
break
}
time.Sleep(5 * time.Second)
}
log.Info("ingress_lxc.network_ready", "vmid", vmid, "ip", opts.FloatingIP)
// Install podman inside the LXC (C-53: idempotent).
_, _ = runRemote(fmt.Sprintf(
"pct exec %s -- bash -c 'command -v podman >/dev/null 2>&1 || (apt-get update -qq && apt-get install -y -qq podman conmon crun fuse-overlayfs nftables 2>&1)' 2>&1",
vmid,
))
// Enable podman-restart.service inside the LXC (research Topic 6).
_, _ = runRemote(fmt.Sprintf("pct exec %s -- systemctl enable --now podman-restart.service 2>/dev/null", vmid))
// Push step-ca root CA into the LXC (C-60: CACertPath).
caPath := certpaths.CACertPath()
caData, caErr := os.ReadFile(caPath)
if caErr != nil {
caData = []byte{}
}
caDelim := "EOF_CA"
_, _ = runRemote(fmt.Sprintf(
"pct exec %s -- bash -c 'mkdir -p /etc/orca && cat > /etc/orca/step-ca-root.crt <<%s\\n%s\\n%s'",
vmid, caDelim, string(caData), caDelim,
))
// Render + write traefik static config inside the LXC (C-58).
staticFiles, err := emitter.TraefikEmitter{}.RenderTraefikStaticConfig(emitter.TraefikStaticOpts{})
if err == nil {
for _, f := range staticFiles {
delim := "EOF_TF"
_, _ = runRemote(fmt.Sprintf(
"pct exec %s -- bash -c 'mkdir -p /etc/traefik/dynamic && cat > %s <<%s\\n%s\\n%s'",
vmid, f.Path, delim, f.Content, delim,
))
}
}
// Render + apply nft DNAT+SNAT INSIDE the LXC (DNATTarget =
// 127.0.0.1 — traefik runs with --network host inside the LXC).
nftFiles, err := emitter.NftEmitter{}.RenderNftConfig(emitter.NftClusterConfig{})
if err == nil {
for _, f := range nftFiles {
delim := "EOF_NF"
_, _ = runRemote(fmt.Sprintf(
"pct exec %s -- bash -c 'mkdir -p /etc/nftables.d && cat > %s <<%s\\n%s\\n%s'",
vmid, f.Path, delim, f.Content, delim,
))
}
_, _ = runRemote(fmt.Sprintf("pct exec %s -- nft add table inet orca-ingress 2>/dev/null || true", vmid))
_, _ = runRemote(fmt.Sprintf("pct exec %s -- nft -f /etc/nftables.d/orca.nft 2>&1", vmid))
}
// Ensure podman orca-traefik container inside the LXC.
traefikExecFn := func(cmd string) ([]byte, error) {
return runRemote(fmt.Sprintf("pct exec %s -- bash -c %s 2>&1", vmid, shellQuote(cmd)))
}
if err := traefik.EnsureTraefikContainerRemote(ctx, "", traefikExecFn); err != nil {
log.Warn("ingress_lxc.traefik_failed", "err", err)
}
log.Info("ingress_lxc.provisioned", "vmid", vmid, "ip", opts.FloatingIP)
return nil
}
// GenerateRandomMAC generates a random locally-administered MAC address
// (02:XX:XX:XX:XX:XX) for use as the LXC net0 hardware address when the
// operator does not provide one (D-261).
func GenerateRandomMAC() (string, error) {
b := make([]byte, 5)
if _, err := rand.Read(b); err != nil {
return "", fmt.Errorf("generate MAC: %w", err)
}
return fmt.Sprintf("02:%02x:%02x:%02x:%02x:%02x", b[0], b[1], b[2], b[3], b[4]), nil
}
@@ -0,0 +1,6 @@
-- REQ-175 / R-024: add ingress_mode column to nodes.
-- Values: '' (legacy/default), 'native' (proxmox native — traefik
-- in LXC), 'floating-ip' (proxmox floating-IP — ingress LXC owns
-- the floating IP). Defaults to empty string for backward
-- compatibility with pre-v0.14 nodes.
ALTER TABLE nodes ADD COLUMN ingress_mode TEXT NOT NULL DEFAULT '';
+25 -2
View File
@@ -145,8 +145,8 @@ assert "34 type_linux_available" \
assert "35 status_deprecated" \ assert "35 status_deprecated" \
'$ORCA status 2>&1 | grep -qi "deprecated"' '$ORCA status 2>&1 | grep -qi "deprecated"'
assert "36 traefik_installed" \ assert "36 traefik_container_running" \
'systemctl is-active orca-traefik 2>/dev/null | grep -q "active" || exit 77' 'podman inspect --format "{{.State.Running}}" orca-traefik 2>/dev/null | grep -q "true" || exit 77'
assert "37 known_hosts_exists" \ assert "37 known_hosts_exists" \
'test -f "$ORCA_HOME/known_hosts" || test -f "$ORCA_HOME/cluster/known_hosts"' 'test -f "$ORCA_HOME/known_hosts" || test -f "$ORCA_HOME/cluster/known_hosts"'
@@ -154,6 +154,29 @@ assert "37 known_hosts_exists" \
assert "38 master_key_exists" \ assert "38 master_key_exists" \
'test -f "$ORCA_HOME/cluster/master.key" || test -f "$ORCA_HOME/cluster/master.key.sealed"' 'test -f "$ORCA_HOME/cluster/master.key" || test -f "$ORCA_HOME/cluster/master.key.sealed"'
# --- v0.14 ingress bootstrap assertions (R-024) ---
assert "40 ingress_nft_table" \
'nft list table inet orca-ingress 2>/dev/null | grep -q "chain prerouting"'
assert "41 ingress_nft_dnat" \
'nft list table inet orca-ingress 2>/dev/null | grep -q "dnat to"'
assert "42 ingress_nft_snat" \
'nft list table inet orca-ingress 2>/dev/null | grep -q "masquerade"'
assert "43 ingress_dynamic_dir" \
'test -d /etc/traefik/dynamic'
assert "44 ingress_step_ca" \
'test -f /etc/orca/step-ca-root.crt'
assert "45 ingress_traefik_yml" \
'test -f /etc/traefik/traefik.yml'
assert "46 ingress_doctor_pass" \
'$ORCA doctor ingress 2>&1 | grep -q "PASS"'
# --- Report --- # --- Report ---
echo "==========================================" echo "=========================================="
+163
View File
@@ -0,0 +1,163 @@
package tests
import (
"log/slog"
"strings"
"testing"
"git.cloudinit.dev/coreci/orca/internal/emitter"
"git.cloudinit.dev/coreci/orca/internal/jobspec"
"git.cloudinit.dev/coreci/orca/internal/proxmox"
"git.cloudinit.dev/coreci/orca/internal/traefik"
)
// TestNftEmitter_PostroutingAndDNATTarget (REQ-173) verifies the nft
// emitter renders the postrouting masquerade chain and supports
// DNATTarget substitution.
func TestNftEmitter_PostroutingAndDNATTarget(t *testing.T) {
files, err := emitter.NftEmitter{}.RenderNftConfig(emitter.NftClusterConfig{
DNATTarget: "10.99.0.10",
EnableSNAT: true,
})
if err != nil {
t.Fatalf("RenderNftConfig: %v", err)
}
c := files[0].Content
if !strings.Contains(c, "chain postrouting") {
t.Errorf("missing postrouting chain:\n%s", c)
}
if !strings.Contains(c, "masquerade") {
t.Errorf("missing masquerade rule:\n%s", c)
}
if !strings.Contains(c, "dnat to 10.99.0.10:8443") {
t.Errorf("missing custom DNAT target:\n%s", c)
}
}
// TestNftEmitter_PriorityMinus10 (research Topic 2) verifies the input
// and forward chains use priority -10 for pve-firewall coexistence.
func TestNftEmitter_PriorityMinus10(t *testing.T) {
files, _ := emitter.NftEmitter{}.RenderNftConfig(emitter.NftClusterConfig{})
c := files[0].Content
if !strings.Contains(c, "hook input priority -10;") {
t.Errorf("input chain should use priority -10:\n%s", c)
}
if !strings.Contains(c, "hook forward priority -10;") {
t.Errorf("forward chain should use priority -10:\n%s", c)
}
}
// TestTraefikEmitter_TLSModel (REQ-172) verifies the dynamic config
// emits tls: {} and does NOT contain certResolver (dropped in v0.14).
func TestTraefikEmitter_TLSModel(t *testing.T) {
spec := &jobspec.WorkloadSpec{
Name: "test-svc",
Kind: "Service",
Ports: []jobspec.PortSpec{{Name: "http"}},
}
node := &emitter.Node{
Hostname: "test-node",
}
files, err := emitter.TraefikEmitter{}.Render(spec, node)
if err != nil {
t.Fatalf("Render: %v", err)
}
c := files[0].Content
if !strings.Contains(c, "tls: {}") {
t.Errorf("missing tls: {} (v0.14 model):\n%s", c)
}
if strings.Contains(c, "certResolver: orca") {
t.Errorf("certResolver: orca should be removed (v0.14):\n%s", c)
}
}
// TestTraefikImageRef verifies the image reference resolution for the
// orca-traefik podman container.
func TestTraefikImageRef(t *testing.T) {
ref := traefik.ImageRef("v0.13.7")
want := "git.cloudinit.dev/coreci/orca-traefik:v0.13.7"
if ref != want {
t.Errorf("ImageRef(v0.13.7) = %q, want %q", ref, want)
}
// Dev build falls back to latest.
ref = traefik.ImageRef("dev")
if ref != "git.cloudinit.dev/coreci/orca-traefik:latest" {
t.Errorf("ImageRef(dev) = %q, want latest", ref)
}
}
// TestProxmox_FloatingIP_LXC_ProvisioningCommands (REQ-176) verifies
// the ProvisionIngressLXC function sends the correct pct create
// command with the right net0 parameters.
func TestProxmox_FloatingIP_LXC_ProvisioningCommands(t *testing.T) {
var cmds []string
execFn := func(cmd string) ([]byte, error) {
cmds = append(cmds, cmd)
// Simulate: pct status returns "absent" on first call, then OK.
if strings.Contains(cmd, "pct status 201") {
return []byte("absent\n"), nil
}
if strings.Contains(cmd, "pct create") {
return []byte(""), nil
}
if strings.Contains(cmd, "pct start 201") {
return []byte(""), nil
}
if strings.Contains(cmd, "hostname -I") {
return []byte("203.0.113.10\n"), nil
}
return []byte(""), nil
}
err := proxmox.ProvisionIngressLXC(nil, execFn, proxmox.FloatingIPOptions{
FloatingIP: "203.0.113.10",
Gateway: "203.0.113.1",
MAC: "02:01:02:03:04:05",
NetPrefix: 24,
LXCTemplate: "ubuntu-24.04",
}, slog.Default())
if err != nil {
t.Fatalf("ProvisionIngressLXC: %v", err)
}
// Verify pct create has the right net0 params.
foundCreate := false
for _, c := range cmds {
if strings.Contains(c, "pct create") {
foundCreate = true
if !strings.Contains(c, "hostname ingress") {
t.Errorf("pct create missing hostname ingress: %s", c)
}
if !strings.Contains(c, "hwaddr=02:01:02:03:04:05") {
t.Errorf("pct create missing hwaddr: %s", c)
}
if !strings.Contains(c, "ip=203.0.113.10/24") {
t.Errorf("pct create missing ip: %s", c)
}
if !strings.Contains(c, "gw=203.0.113.1") {
t.Errorf("pct create missing gw: %s", c)
}
if !strings.Contains(c, "nesting=1,keyctl=1,fuse=1") {
t.Errorf("pct create missing features (research Topic 3): %s", c)
}
}
}
if !foundCreate {
t.Errorf("pct create command not sent\ncommands: %v", cmds)
}
}
// TestProxmox_GenerateRandomMAC (D-261) verifies MAC generation produces
// a valid locally-administered MAC.
func TestProxmox_GenerateRandomMAC(t *testing.T) {
mac, err := proxmox.GenerateRandomMAC()
if err != nil {
t.Fatalf("GenerateRandomMAC: %v", err)
}
if !strings.HasPrefix(mac, "02:") {
t.Errorf("MAC should start with 02: (locally administered): %s", mac)
}
// Verify it's 6 octets.
parts := strings.Split(mac, ":")
if len(parts) != 6 {
t.Errorf("MAC should have 6 octets: %s", mac)
}
}