Compare commits

..

5 Commits

Author SHA1 Message Date
Jon Chery 6e65eadaa5 feat(P6): proxmox floating-IP LXC ingress + interactive prompt (REQ-176)
New internal/proxmox/ingress_lxc.go: ProvisionIngressLXC creates an
Ubuntu LXC named 'ingress' that owns the floating IP (net0
bridge=vmbr0,hwaddr=<mac>,ip=<floating-ip>/<prefix>,gw=<gateway>).
Unprivileged with --features nesting=1,keyctl=1,fuse=1 (research
Topic 3). Installs podman inside, runs orca-traefik container,
applies nft DNAT+SNAT INSIDE the LXC, pushes step-ca root CA.

GenerateRandomMAC: 02:XX:XX:XX:XX:XX for interactive mode (D-261).

Interactive prompting in joinProxmox: when --ingress-mode empty +
!--json, prompt for mode + floating IP + gateway + MAC (auto-
generate + confirm). Validate IP/MAC/gateway/prefix.

Floating-IP routing: calls ProvisionIngressLXC + registers:
  1. PVE host as 'proxmox' node (IngressMode=floating-ip)
  2. Ingress LXC as 'linux' node (name=ingress, addr=<floating-ip>:8443)
     so orca job run pushes traefik dynamic config to it.

---ci---
project: orca
phase: 6
milestone: v0.14
status: execute
---/ci---
2026-08-10 20:19:47 +00:00
Jon Chery 1b7aac71f6 feat(P5): proxmox native ingress mode — LXC + podman traefik (REQ-175)
Add --ingress-mode flag (native default, floating-ip) + --floating-ip,
--gateway, --mac, --net-prefix flags to 'orca node join'.

Native mode (default): provision an unprivileged LXC with
--features nesting=1,keyctl=1,fuse=1 (research Topic 3), install
podman inside it, run orca-traefik container. nft on PVE host DNATs
to the LXC bridge IP (DNATTarget parameterization, C-55: discover
LXC IP before first nft apply, no downtime window).

LXC provisioning: deterministic VMID 200, hostname orca-traefik,
--onboot 1, 2GB RAM. Idempotent (C-53: command -v podman check).
podman-restart.service enabled inside LXC (research Topic 6).

step-ca root CA pushed into LXC via pct exec heredoc.
traefik static config rendered + written into LXC.
nft ruleset rendered with DNATTarget=LXC-IP + applied on PVE host.

Migration 0009_ingress_mode.sql (C-59: NOT 0007 — already taken by
certs_serial_unique). ALTER TABLE nodes ADD COLUMN ingress_mode.
IngressMode field added to model.Node + set on proxmox node record.

---ci---
project: orca
phase: 5
milestone: v0.14
status: execute
---/ci---
2026-08-10 20:16:40 +00:00
Jon Chery ea42a17474 feat(P4): linux node join remote ingress bootstrap (REQ-174)
Add ingress.BootstrapRemoteIngress: renders+writes traefik static
config, renders+writes+applies nft DNAT/SNAT, pushes step-ca root CA,
ensures podman traefik container — all over SSH exec. Uses a heredoc-
based remoteWriteFile with a random delimiter (F9 injection guard).

Wired into linux/bootstrap.go Step 4d, replacing the standalone
EnsureTraefikContainerRemote call with the full ingress stack.

C-60: uses certpaths.CACertPath() (not CAPath).
C-58: mounts host-side traefik.yml (preserves REQ-100 opt-out).
C-55: pre-creates nft table before nft -f.

---ci---
project: orca
phase: 4
milestone: v0.14
status: execute
---/ci---
2026-08-10 20:11:32 +00:00
Jon Chery 5013209e31 feat(P3): nft SNAT+DNAT + orca init ingress bootstrap (REQ-173)
nft emitter (internal/emitter/nft.go):
- Add DNATTarget field (C-51: validated via net.ParseIP; injection
  guard). Default 127.0.0.1; proxmox native uses LXC bridge IP.
- Add EnableSNAT field (default true for zero-value config).
- Add postrouting masquerade chain (research Topic 1):
  ip saddr 127.0.0.0/8 oifname != lo masquerade
- Shift input/forward priority from filter (=0) to -10 (research
  Topic 2: pve-firewall coexistence — avoids same-priority undefined
  evaluation order).

internal/ingress/bootstrap.go (new):
- BootstrapLocalIngress: mkdir dirs, push step-ca root CA (C-60:
  certpaths.CACertPath not CAPath), render+write traefik static
  config (C-58: preserves traefik-on-public-ip opt-out), render+
  write+apply nft ruleset, pre-create table (C-55: avoids first-
  apply flush-table error), ensure podman container. All non-fatal.

init.go: Step 4d now calls ingress.BootstrapLocalIngress (R-024).
doctor_nft.go: assert postrouting masquerade + priority -10.

Tests: nft_test.go — DNATTarget substitution, invalid DNATTarget
rejection (C-51), EnableSNAT=false omits postrouting, priority -10.

---ci---
project: orca
phase: 3
milestone: v0.14
status: execute
---/ci---
2026-08-10 20:09:26 +00:00
Jon Chery dea472f443 feat(P2): podman traefik reconciler + TLS model fix (REQ-172)
Replace internal/traefik/install.go binary+systemd installer with a
podman-container reconciler (R-024). The reconciler is idempotent:
inspect → start-if-stopped → pull+run-if-absent.

Container run flags (research-validated):
  --restart=unless-stopped (not always; research Topic 6)
  --network host (binds 127.0.0.1:8080/8443 on host/LXC loopback)
  -v /etc/traefik/traefik.yml:ro (overrides baked default; C-58)
  -v /etc/traefik/dynamic:ro (orca writes atomically via SSH-push)
  -v /etc/orca/step-ca-root.crt:ro (future mTLS; v0.14 uses tls:{})
  No :Z SELinux flag (research Topic 7)

C-50: ensurePodmanLocal/Remote installs podman if absent.
C-57: removeLegacySystemdUnitLocal/Remote stops+disables+removes
  the v0.13 orca-traefik.service + /usr/local/bin/traefik before
  starting the podman container (upgrade path).
  upgrade.go cutover rewritten to use the reconciler.

TLS model fix (research Topic 4): drop certResolver: orca from
dynamic config (traefik v3.3 only supports acme/tailscale resolvers,
not CA-file-based). Emit tls: {} instead. Real mTLS via dynamic
tls.certificates + clientAuth.caFiles deferred to v0.15 (grill
G-003, confidence 0.55 < 0.60).

Callsites updated:
  init.go: installTraefikLocal → ensureTraefikContainerLocal
  linux/bootstrap.go: traefik.InstallRemote → EnsureTraefikContainerRemote
  proxmox/bootstrap.go: same
  traefik_install.go: wrapper updated

Tests: internal/traefik/install_test.go (new) — ImageRef, podmanRunArgs,
  container-running/stopped/absent paths, legacy systemd removal (C-57).

---ci---
project: orca
phase: 2
milestone: v0.14
status: execute
---/ci---
2026-08-10 20:04:20 +00:00
17 changed files with 1370 additions and 152 deletions
+14
View File
@@ -113,6 +113,20 @@ func runNftChecks(ctx context.Context) []nftCheckResult {
results = append(results, nftCheckResult{Name: "nft:dnat-80", Result: "FAIL", Message: "DNAT :80->127.0.0.1:8080 missing"})
}
// Research Topic 1: postrouting masquerade for DNAT return path.
if strings.Contains(tableStr, "masquerade") {
results = append(results, nftCheckResult{Name: "nft:snat-masquerade", Result: "PASS", Message: "postrouting masquerade (SNAT) present"})
} else {
results = append(results, nftCheckResult{Name: "nft:snat-masquerade", Result: "FAIL", Message: "postrouting masquerade missing (R-024)"})
}
// Research Topic 2: priority -10 on input/forward (pve-firewall coexistence).
if strings.Contains(tableStr, "priority -10") {
results = append(results, nftCheckResult{Name: "nft:priority", Result: "PASS", Message: "input/forward chains at priority -10 (pve-firewall coexistence)"})
} else {
results = append(results, nftCheckResult{Name: "nft:priority", Result: "WARN", Message: "priority -10 not found (may be pre-v0.14 ruleset)"})
}
if strings.Contains(tableStr, "ora_rl") {
results = append(results, nftCheckResult{Name: "nft:rate-limit", Result: "PASS", Message: "rate-limit meter ora_rl present"})
} else {
+22 -16
View File
@@ -1,27 +1,28 @@
package cli
import (
"bufio"
"context"
"crypto/x509"
"encoding/pem"
"fmt"
"bufio"
"os"
"strings"
"os/exec"
"path/filepath"
"time"
"github.com/google/uuid"
"golang.org/x/crypto/ssh"
"github.com/spf13/cobra"
"git.cloudinit.dev/coreci/orca/internal/acl"
"git.cloudinit.dev/coreci/orca/internal/certpaths"
"git.cloudinit.dev/coreci/orca/internal/identity"
"git.cloudinit.dev/coreci/orca/internal/ingress"
"git.cloudinit.dev/coreci/orca/internal/model"
"git.cloudinit.dev/coreci/orca/internal/paths"
"git.cloudinit.dev/coreci/orca/internal/secrets"
"git.cloudinit.dev/coreci/orca/internal/security"
"git.cloudinit.dev/coreci/orca/internal/store"
"github.com/google/uuid"
"github.com/spf13/cobra"
"golang.org/x/crypto/ssh"
"os"
"os/exec"
"path/filepath"
"strings"
"time"
)
const (
@@ -251,17 +252,22 @@ func runInit(out interface{ Write([]byte) (int, error) }) error {
}
}
// Step 4d: Install Traefik on the lead node (REQ-165, Phase B).
// Traefik is the data-plane ingress. Idempotent.
if err := installTraefikLocal(); err != nil {
// Step 4d: Ensure complete ingress stack on the lead node (R-024).
// This replaces the v0.13 binary+systemd traefik install with:
// 1. Render + write traefik static config (traefik.yml)
// 2. Render + write + apply nft DNAT/SNAT ruleset (orca.nft)
// 3. Push step-ca root CA to /etc/orca/step-ca-root.crt
// 4. Ensure podman orca-traefik container running
// All steps non-fatal (offline host tolerance).
if err := ingress.BootstrapLocalIngress(context.Background(), version); err != nil {
if !jsonOutput {
fmt.Fprintf(out, "Traefik install skipped: %v\n", err)
fmt.Fprintf(out, "Ingress bootstrap skipped: %v\n", err)
}
summary.Steps = append(summary.Steps, stepResult{Label: "traefik", Status: "skipped", Detail: err.Error()})
summary.Steps = append(summary.Steps, stepResult{Label: "ingress", Status: "skipped", Detail: err.Error()})
} else {
summary.Steps = append(summary.Steps, stepResult{Label: "traefik", Status: "ok", Detail: traefikVersion})
summary.Steps = append(summary.Steps, stepResult{Label: "ingress", Status: "ok", Detail: "nft+traefik container active"})
if !jsonOutput {
fmt.Fprintf(out, "Traefik installed: %s\n", traefikVersion)
fmt.Fprintf(out, "Ingress: nft DNAT+SNAT applied, traefik container running\n")
}
}
+204 -24
View File
@@ -1,22 +1,26 @@
package cli
import (
"bufio"
"context"
"database/sql"
"encoding/json"
"fmt"
"log/slog"
"net"
"os"
"os/signal"
"strings"
"syscall"
"time"
"github.com/google/uuid"
"github.com/spf13/cobra"
"golang.org/x/crypto/ssh"
"git.cloudinit.dev/coreci/orca/internal/certpaths"
"git.cloudinit.dev/coreci/orca/internal/linux"
"git.cloudinit.dev/coreci/orca/internal/engine"
"git.cloudinit.dev/coreci/orca/internal/linux"
"git.cloudinit.dev/coreci/orca/internal/model"
"git.cloudinit.dev/coreci/orca/internal/proxmox"
"git.cloudinit.dev/coreci/orca/internal/security"
@@ -46,20 +50,25 @@ func nodeRegistry() (*engine.NodeRegistry, func() error, error) {
}
var (
joinName string
joinAddr string
joinCAFinger string
joinType string
joinHost string
joinSSHUser string
joinSSHKey string
joinSSHPort int
joinHostKeyFP string
joinName string
joinAddr string
joinCAFinger string
joinType string
joinHost string
joinSSHUser string
joinSSHKey string
joinSSHPort int
joinHostKeyFP string
joinLXCTemplate string
proxmoxUser string
proxmoxRole string
leaveID string
nodeWatch bool
proxmoxUser string
proxmoxRole string
ingressMode string
floatingIP string
gateway string
macAddr string
netPrefix int
leaveID string
nodeWatch bool
)
var nodeCmd = &cobra.Command{
@@ -175,7 +184,83 @@ func joinProxmox(cmd *cobra.Command) error {
return fmt.Errorf("SSH key path is required for --type proxmox (R-021: no passwords; use --ssh-key or pre-stage the orca key)")
}
ctx, cancel := context.WithTimeout(cmd.Context(), 60*time.Second)
// Default ingress mode to "native" if not specified (R-024).
effectiveIngressMode := ingressMode
if effectiveIngressMode == "" {
if !jsonOutput {
// Interactive mode: prompt for ingress mode.
fmt.Fprint(cmd.OutOrStdout(), "Ingress mode [native/floating-ip] (default native): ")
scanner := bufio.NewScanner(os.Stdin)
if scanner.Scan() {
input := strings.TrimSpace(scanner.Text())
if input == "floating-ip" {
effectiveIngressMode = "floating-ip"
} else {
effectiveIngressMode = "native"
}
} else {
effectiveIngressMode = "native"
}
} else {
effectiveIngressMode = "native"
}
}
// Floating-IP mode: prompt for params if not provided.
effectiveFloatingIP := floatingIP
effectiveGateway := gateway
effectiveMAC := macAddr
if effectiveIngressMode == "floating-ip" {
if effectiveFloatingIP == "" && !jsonOutput {
fmt.Fprint(cmd.OutOrStdout(), "Floating IP: ")
scanner := bufio.NewScanner(os.Stdin)
if scanner.Scan() {
effectiveFloatingIP = strings.TrimSpace(scanner.Text())
}
}
if effectiveGateway == "" && !jsonOutput {
fmt.Fprint(cmd.OutOrStdout(), "Gateway: ")
scanner := bufio.NewScanner(os.Stdin)
if scanner.Scan() {
effectiveGateway = strings.TrimSpace(scanner.Text())
}
}
if effectiveMAC == "" && !jsonOutput {
// D-261: auto-generate a random locally-administered MAC.
generated, err := proxmox.GenerateRandomMAC()
if err == nil {
fmt.Fprintf(cmd.OutOrStdout(), "Generated MAC: %s (press enter to accept, or type your own): ", generated)
scanner := bufio.NewScanner(os.Stdin)
if scanner.Scan() {
input := strings.TrimSpace(scanner.Text())
if input != "" {
effectiveMAC = input
} else {
effectiveMAC = generated
}
} else {
effectiveMAC = generated
}
}
}
// Validate floating-IP mode params.
if effectiveIngressMode == "floating-ip" {
if net.ParseIP(effectiveFloatingIP) == nil {
return fmt.Errorf("--floating-ip %q is not a valid IP", effectiveFloatingIP)
}
if net.ParseIP(effectiveGateway) == nil {
return fmt.Errorf("--gateway %q is not a valid IP", effectiveGateway)
}
if _, err := net.ParseMAC(effectiveMAC); err != nil {
return fmt.Errorf("--mac %q is not a valid MAC: %w", effectiveMAC, err)
}
if netPrefix < 8 || netPrefix > 32 {
return fmt.Errorf("--net-prefix %d must be 8-32", netPrefix)
}
}
}
ctx, cancel := context.WithTimeout(cmd.Context(), 180*time.Second) // 3min for LXC creation
defer cancel()
result, err := proxmox.BootstrapProxmox(ctx, proxmox.Options{
@@ -188,6 +273,7 @@ func joinProxmox(cmd *cobra.Command) error {
HostKeyFingerprint: joinHostKeyFP,
Logger: newLogger(),
LXCTemplate: joinLXCTemplate,
IngressMode: effectiveIngressMode,
})
if err != nil {
return fmt.Errorf("proxmox bootstrap: %w", err)
@@ -204,18 +290,67 @@ func joinProxmox(cmd *cobra.Command) error {
defer regCancel()
node := &model.Node{
ID: uuid.NewString(),
Name: result.NodeName,
Address: result.NodeAddress,
State: model.NodeStateReady,
JoinedAt: time.Now().UTC(),
LastSeen: time.Now().UTC(),
Kind: string(model.NodeKindProxmox),
OS: "pve",
ID: uuid.NewString(),
Name: result.NodeName,
Address: result.NodeAddress,
State: model.NodeStateReady,
JoinedAt: time.Now().UTC(),
LastSeen: time.Now().UTC(),
Kind: string(model.NodeKindProxmox),
OS: "pve",
IngressMode: effectiveIngressMode,
}
if err := registry.Join(regCtx, node); err != nil {
return fmt.Errorf("register proxmox node: %w", err)
}
// Floating-IP mode: provision the ingress LXC and register it as a
// linux node (R-024, REQ-176). The PVE host is registered as
// proxmox (above); the ingress LXC is registered as linux so
// `orca job run` pushes traefik dynamic config to it.
if effectiveIngressMode == "floating-ip" {
lxcLog := newLogger()
// Build a runRemote function from the proxmox bootstrap result.
// We need SSH access to the PVE host to run pct commands.
lxcCtx, lxcCancel := context.WithTimeout(ctx, 120*time.Second)
defer lxcCancel()
// The BootstrapProxmox result gives us the host; we need to
// re-establish the SSH connection for the LXC provisioning.
lxcExecFn, lxcErr := proxmoxRemoteExecFn(result, sshKeyPath, joinSSHUser, joinSSHPort)
if lxcErr != nil {
fmt.Fprintf(cmd.OutOrStdout(), "Warning: could not establish SSH for LXC provisioning: %v\n", lxcErr)
} else {
if err := proxmox.ProvisionIngressLXC(lxcCtx, lxcExecFn, proxmox.FloatingIPOptions{
FloatingIP: effectiveFloatingIP,
Gateway: effectiveGateway,
MAC: effectiveMAC,
NetPrefix: netPrefix,
LXCTemplate: joinLXCTemplate,
}, lxcLog); err != nil {
fmt.Fprintf(cmd.OutOrStdout(), "Warning: ingress LXC provisioning failed: %v\n", err)
} else {
// Register the ingress LXC as a linux node.
ingressNode := &model.Node{
ID: uuid.NewString(),
Name: "ingress",
Address: fmt.Sprintf("%s:8443", effectiveFloatingIP),
State: model.NodeStateReady,
JoinedAt: time.Now().UTC(),
LastSeen: time.Now().UTC(),
Kind: string(model.NodeKindLinux),
OS: "linux",
IngressMode: "floating-ip",
}
if err := registry.Join(regCtx, ingressNode); err != nil {
fmt.Fprintf(cmd.OutOrStdout(), "Warning: register ingress node: %v\n", err)
}
if !jsonOutput {
fmt.Fprintf(cmd.OutOrStdout(), "✓ Ingress LXC joined: %s (%s) at %s\n", ingressNode.ID, ingressNode.Name, ingressNode.Address)
}
}
}
}
// REQ-156 / P07 T5: invalidate the nodes cache.
cacheInvalidate(cacheNodeClass)
if jsonOutput {
@@ -226,6 +361,46 @@ func joinProxmox(cmd *cobra.Command) error {
return nil
}
// proxmoxRemoteExecFn creates a RemoteExecFunc (func(string) ([]byte,
// error)) that runs commands on the PVE host via SSH. Used by the
// floating-IP LXC provisioning path (ProvisionIngressLXC).
func proxmoxRemoteExecFn(result *proxmox.Result, sshKeyPath, sshUser string, sshPort int) (func(string) ([]byte, error), error) {
cfg := &ssh.ClientConfig{
User: sshUser,
HostKeyCallback: ssh.InsecureIgnoreHostKey(),
Timeout: 10 * time.Second,
}
if sshKeyPath != "" {
keyData, err := os.ReadFile(sshKeyPath)
if err != nil {
return nil, fmt.Errorf("read SSH key: %w", err)
}
signer, err := ssh.ParsePrivateKey(keyData)
if err != nil {
return nil, fmt.Errorf("parse SSH key: %w", err)
}
cfg.Auth = []ssh.AuthMethod{ssh.PublicKeys(signer)}
}
addr := result.NodeName
if sshPort != 22 {
addr = fmt.Sprintf("%s:%d", result.NodeName, sshPort)
} else {
addr = fmt.Sprintf("%s:%d", result.NodeName, sshPort)
}
client, err := ssh.Dial("tcp", addr, cfg)
if err != nil {
return nil, fmt.Errorf("ssh dial %s: %w", addr, err)
}
return func(cmd string) ([]byte, error) {
session, err := client.NewSession()
if err != nil {
return nil, err
}
defer session.Close()
return session.CombinedOutput(cmd)
}, nil
}
// joinLinux bootstraps a remote generic Linux worker via SSH and
// registers it as an orca node (REQ-161, P12). Uses SSH key auth
// (R-021: no passwords).
@@ -252,7 +427,7 @@ func joinLinux(cmd *cobra.Command) error {
SSHPort: joinSSHPort,
HostKeyFingerprint: joinHostKeyFP,
Logger: newLogger(),
})
})
if err != nil {
return fmt.Errorf("linux bootstrap: %w", err)
}
@@ -514,6 +689,11 @@ func init() {
nodeJoinCmd.Flags().StringVar(&proxmoxRole, "proxmox-role", "OrcaOperator", "PVE custom role to create (config-overridable)")
nodeJoinCmd.Flags().StringVar(&joinHostKeyFP, "host-key-fingerprint", "", "SSH host key SHA256:base64 fingerprint (pre-pin; supersedes TOFU for --type proxmox or --type linux)")
nodeJoinCmd.Flags().StringVar(&joinLXCTemplate, "lxc-template", "ubuntu-24.04", "LXC template for Proxmox (default ubuntu-24.04; alternatives: alpine-3.20, debian-12)")
nodeJoinCmd.Flags().StringVar(&ingressMode, "ingress-mode", "", "proxmox ingress mode: native (default, traefik in LXC) or floating-ip (ingress LXC owns floating IP)")
nodeJoinCmd.Flags().StringVar(&floatingIP, "floating-ip", "", "floating public IP for the ingress LXC (required for --ingress-mode floating-ip)")
nodeJoinCmd.Flags().StringVar(&gateway, "gateway", "", "gateway for the ingress LXC (required for --ingress-mode floating-ip)")
nodeJoinCmd.Flags().StringVar(&macAddr, "mac", "", "MAC address for the ingress LXC net0 (required for --ingress-mode floating-ip in --json mode; auto-generated in interactive mode)")
nodeJoinCmd.Flags().IntVar(&netPrefix, "net-prefix", 24, "network prefix (CIDR) for the ingress LXC IP (default 24; valid 8-32)")
nodeLeaveCmd.Flags().StringVar(&leaveID, "id", "", "node id")
nodeListCmd.Flags().BoolVar(&nodeWatch, "watch", false, "stream nodes until Ctrl-C (table refresh or --json per-event)")
+9 -4
View File
@@ -1,11 +1,16 @@
package cli
import (
"context"
"git.cloudinit.dev/coreci/orca/internal/traefik"
)
var traefikVersion = traefik.DefaultVersion
func installTraefikLocal() error {
return traefik.InstallLocal(traefikVersion)
// ensureTraefikContainerLocal ensures the orca-traefik podman container
// is running on the local host (R-024). Replaces the v0.13
// installTraefikLocal binary+systemd installer.
func ensureTraefikContainerLocal() error {
ctx, cancel := context.WithTimeout(context.Background(), 120_000_000_000) // 2min for pull
defer cancel()
return traefik.EnsureTraefikContainerLocal(ctx, version)
}
+12 -11
View File
@@ -29,7 +29,7 @@ import (
var (
upgradeTo string
upgradeImportCA bool
upgradeForce bool
upgradeForce bool
upgradeDryRun bool
)
@@ -84,12 +84,12 @@ type cutoverFS interface {
// realCutoverFS is the production cutoverFS backed by the real os.
type realCutoverFS struct{}
func (realCutoverFS) ReadFile(path string) ([]byte, error) { return os.ReadFile(path) }
func (realCutoverFS) ReadFile(path string) ([]byte, error) { return os.ReadFile(path) }
func (realCutoverFS) WriteFile(path string, content []byte, mode os.FileMode) error {
return os.WriteFile(path, content, mode)
}
func (realCutoverFS) Rename(old, new string) error { return os.Rename(old, new) }
func (realCutoverFS) Remove(path string) error { return os.Remove(path) }
func (realCutoverFS) Rename(old, new string) error { return os.Rename(old, new) }
func (realCutoverFS) Remove(path string) error { return os.Remove(path) }
func (realCutoverFS) Stat(path string) (os.FileInfo, error) { return os.Stat(path) }
// cutoverFSOverride is the package-level test seam for the cutover
@@ -160,9 +160,9 @@ func acquireUpgradeLock() (func(), error) {
// UpgradeResult is the JSON-serializable summary of an upgrade run.
type UpgradeResult struct {
TargetVersion string `json:"target_version"`
CurrentVersion string `json:"current_version"`
DryRun bool `json:"dry_run"`
TargetVersion string `json:"target_version"`
CurrentVersion string `json:"current_version"`
DryRun bool `json:"dry_run"`
MigratedV08 bool `json:"migrated_v08"`
CutoverNeeded bool `json:"cutover_needed"`
CutoverOK bool `json:"cutover_ok,omitempty"`
@@ -379,10 +379,11 @@ func performCutover(ctx context.Context, runner commandRunner, out interface{ Wr
return false, fmt.Errorf("cutover: atomic rename %s → %s: %w", tmpPath, traefikYml, err)
}
if _, err := runner.Run(ctx, "systemctl", "restart", "traefik"); err != nil {
// Restart failed — restore from backup.
_ = cfs.Rename(backupPath, traefikYml)
return false, fmt.Errorf("cutover: restart traefik: %w", err)
if _, err := runner.Run(ctx, "bash", "-c", "systemctl stop orca-traefik.service 2>/dev/null; systemctl disable orca-traefik.service 2>/dev/null; rm -f /etc/systemd/system/orca-traefik.service /usr/local/bin/traefik; systemctl daemon-reload; true"); err != nil {
slog.Warn("cutover: legacy systemd unit removal failed (non-fatal if already removed)", "err", err)
}
if err := ensureTraefikContainerLocal(); err != nil {
slog.Warn("cutover: podman traefik container ensure failed", "err", err)
}
nftCmd := `nft add table inet orca_redirect; nft 'add chain inet orca_redirect prerouting { type nat hook prerouting priority -100; }'; nft add rule inet orca_redirect prerouting tcp dport 443 dnat to 127.0.0.1:8443`
if _, err := runner.Run(ctx, "bash", "-c", nftCmd); err != nil {
+63 -4
View File
@@ -53,6 +53,16 @@ type NftClusterConfig struct {
// RateBurst is the per-source burst (packets) for the meter.
// Defaults to 200.
RateBurst int
// DNATTarget is the destination IP for DNAT rules. Defaults to
// "127.0.0.1" (hybrid R-017 model — traefik on loopback). For
// Proxmox native mode where traefik runs inside an LXC, set this
// to the LXC's bridge IP so the PVE host DNATs to the LXC.
// Must be a valid IPv4 address (C-51: injection guard).
DNATTarget string
// EnableSNAT controls whether the postrouting masquerade chain
// is rendered. Defaults to true (R-024: SNAT/MASQUERADE for the
// DNAT return path). Set to false to omit the postrouting chain.
EnableSNAT bool
}
// withDefaults returns a copy of c with zero values replaced by the
@@ -68,6 +78,33 @@ func (c NftClusterConfig) withDefaults() NftClusterConfig {
if out.RateBurst <= 0 {
out.RateBurst = 200
}
if out.DNATTarget == "" {
out.DNATTarget = "127.0.0.1"
}
// EnableSNAT defaults to true — use a sentinel: if the field was
// not explicitly set (false) and DNATTarget is the default, enable
// it. This is a Go zero-value compromise; callers who want to
// disable SNAT must set it to false explicitly after construction.
// Actually, since we want SNAT on by default, we flip it here:
// the zero value is false, but we want true. So we always set true
// unless the caller explicitly set it to a non-zero sentinel.
// Simpler: treat EnableSNAT as "opt-out" — default true, set false
// to disable. Since Go zero-value is false, we invert: use
// DisableSNAT instead. But the plan says EnableSNAT. To keep the
// plan naming and have default-true, we check if it's the zero
// value and set true:
// NOTE: since bool zero value is false, we can't distinguish "not
// set" from "set to false". So we use a pointer or invert. The
// simplest fix: the field is "EnableSNAT" and defaults to true via
// this logic: if the caller didn't set DNATTarget (still ""),
// they used a zero-value config, so enable SNAT. If they set
// DNATTarget explicitly, they should also set EnableSNAT.
// For now: always enable SNAT unless the caller sets it to false
// AND sets a non-default DNATTarget. This is pragmatic:
if !out.EnableSNAT && out.DNATTarget == "127.0.0.1" {
// Zero-value config (both fields unset) → enable SNAT.
out.EnableSNAT = true
}
return out
}
@@ -94,6 +131,12 @@ func (NftEmitter) RenderNftConfig(clusterConfig NftClusterConfig) ([]File, error
if err != nil {
return nil, err
}
// C-51: validate DNATTarget as a valid IP before rendering. An
// unvalidated DNATTarget is an nft-syntax injection vector (same
// risk as TrustedProbes — the value is written raw into `dnat to`).
if net.ParseIP(cfg.DNATTarget) == nil {
return nil, fmt.Errorf("emitter/nft: DNATTarget %q is not a valid IP (C-51: ruleset injection guard)", cfg.DNATTarget)
}
content := renderNftRuleset(cfg, v4, v6)
return []File{{Path: nftConfigPath, Content: content, Mode: "0644"}}, nil
}
@@ -171,19 +214,35 @@ func renderNftRuleset(cfg NftClusterConfig, v4, v6 []string) string {
b.WriteString(" }\n")
b.WriteString("\t}\n\n")
// Research Topic 2: shift input/forward priority from `filter`
// (=0) to -10 to avoid same-priority undefined evaluation order
// with pve-firewall's iptables chains (also at priority 0). This
// ensures orca's SYN-flood filter runs deterministically before
// pve-firewall on Proxmox hosts.
b.WriteString("\tchain input {\n")
b.WriteString("\t\ttype filter hook input priority filter; policy accept;\n")
b.WriteString("\t\ttype filter hook input priority -10; policy accept;\n")
b.WriteString("\t\tct state invalid drop\n")
b.WriteString("\t\tct state established,related accept\n")
b.WriteString("\t\ttcp dport 443 tcp-flags != syn,rst,ack,fin notrack drop\n")
b.WriteString("\t}\n\n")
b.WriteString("\tchain prerouting {\n")
b.WriteString("\t\ttype nat hook prerouting priority -100; policy accept;\n")
b.WriteString("\t\ttcp dport 443 dnat to 127.0.0.1:8443\n")
b.WriteString("\t\ttcp dport 80 dnat to 127.0.0.1:8080\n")
b.WriteString(fmt.Sprintf("\t\ttcp dport 443 dnat to %s:8443\n", cfg.DNATTarget))
b.WriteString(fmt.Sprintf("\t\ttcp dport 80 dnat to %s:8080\n", cfg.DNATTarget))
b.WriteString("\t}\n\n")
// Research Topic 1: postrouting masquerade for the DNAT return
// path. Scoped to `ip saddr 127.0.0.0/8 oifname != "lo"` so only
// loopback-DNAT'd traffic is masqueraded (not all egress). This is
// the canonical "hairpin NAT" / "loopback DNAT return path" rule.
// Priority 100 = NF_IP_PRI_SRCNAT (standard srcnat priority).
if cfg.EnableSNAT {
b.WriteString("\tchain postrouting {\n")
b.WriteString("\t\ttype nat hook postrouting priority 100; policy accept;\n")
b.WriteString("\t\tip saddr 127.0.0.0/8 oifname != \"lo\" masquerade\n")
b.WriteString("\t}\n\n")
}
b.WriteString("\tchain forward {\n")
b.WriteString("\t\ttype filter hook forward priority filter; policy accept;\n")
b.WriteString("\t\ttype filter hook forward priority -10; policy accept;\n")
b.WriteString(fmt.Sprintf("\t\ttcp dport 443 ct state new meter { ora_rl { rate %d/second burst %d packets } } accept\n", cfg.RateLimit, cfg.RateBurst))
b.WriteString("\t}\n")
b.WriteString("}\n")
+90
View File
@@ -30,10 +30,13 @@ func TestNftEmitter_RenderBasic(t *testing.T) {
"127.0.0.1",
"::1",
"chain input",
"priority -10; policy accept;", // research Topic 2: pve-firewall coexistence
"tcp dport 443 tcp-flags != syn,rst,ack,fin notrack drop",
"chain prerouting",
"tcp dport 443 dnat to 127.0.0.1:8443",
"tcp dport 80 dnat to 127.0.0.1:8080",
"chain postrouting", // research Topic 1: SNAT masquerade
"ip saddr 127.0.0.0/8 oifname != \"lo\" masquerade", // scoped to loopback DNAT return
"chain forward",
"rate 100/second burst 200 packets",
"ora_rl",
@@ -135,3 +138,90 @@ func TestNftEmitter_TrustedProbesSplitV4V6(t *testing.T) {
t.Errorf("missing ::1 in v6 set:\n%s", c)
}
}
// TestNftEmitter_CustomDNATTarget verifies the DNATTarget field
// substitutes into the dnat rules (C-51, D-262 — proxmox native mode
// DNATs to the LXC bridge IP instead of 127.0.0.1).
func TestNftEmitter_CustomDNATTarget(t *testing.T) {
files, err := (NftEmitter{}).RenderNftConfig(NftClusterConfig{DNATTarget: "10.99.0.10"})
if err != nil {
t.Fatalf("Render: %v", err)
}
c := files[0].Content
if !strings.Contains(c, "dnat to 10.99.0.10:8443") {
t.Errorf("missing custom DNAT target :8443:\n%s", c)
}
if !strings.Contains(c, "dnat to 10.99.0.10:8080") {
t.Errorf("missing custom DNAT target :8080:\n%s", c)
}
if strings.Contains(c, "127.0.0.1:8443") {
t.Errorf("default 127.0.0.1:8443 should not be present when custom DNATTarget set:\n%s", c)
}
}
// TestNftEmitter_RejectsInvalidDNATTarget verifies that an invalid
// DNATTarget is rejected (C-51: nft-syntax injection guard).
func TestNftEmitter_RejectsInvalidDNATTarget(t *testing.T) {
bad := []string{
"not-an-ip",
"127.0.0.1; flush ruleset",
"$(whoami)",
"10.0.0.0/33",
"",
}
for _, b := range bad {
// Empty string gets defaulted to 127.0.0.1, so it won't error.
// Test only non-empty invalid values.
if b == "" {
continue
}
_, err := (NftEmitter{}).RenderNftConfig(NftClusterConfig{DNATTarget: b})
if err == nil {
t.Errorf("expected error for invalid DNATTarget %q, got nil", b)
}
}
}
// TestNftEmitter_DisableSNAT verifies that EnableSNAT=false omits the
// postrouting chain entirely.
func TestNftEmitter_DisableSNAT(t *testing.T) {
// To explicitly disable SNAT, set DNATTarget to a non-default
// value AND EnableSNAT to false. The withDefaults logic only
// auto-enables SNAT for the zero-value config.
files, err := (NftEmitter{}).RenderNftConfig(NftClusterConfig{
DNATTarget: "10.99.0.10",
EnableSNAT: false,
})
if err != nil {
t.Fatalf("Render: %v", err)
}
c := files[0].Content
if strings.Contains(c, "chain postrouting") {
t.Errorf("postrouting chain should be absent when EnableSNAT=false:\n%s", c)
}
if strings.Contains(c, "masquerade") {
t.Errorf("masquerade rule should be absent when EnableSNAT=false:\n%s", c)
}
}
// TestNftEmitter_PriorityMinus10 verifies the input and forward chains
// use priority -10 (research Topic 2: pve-firewall coexistence — avoids
// same-priority undefined evaluation order with pve-firewall's
// iptables chains at priority 0).
func TestNftEmitter_PriorityMinus10(t *testing.T) {
files, _ := (NftEmitter{}).RenderNftConfig(NftClusterConfig{})
c := files[0].Content
if !strings.Contains(c, "hook input priority -10;") {
t.Errorf("input chain should use priority -10:\n%s", c)
}
if !strings.Contains(c, "hook forward priority -10;") {
t.Errorf("forward chain should use priority -10:\n%s", c)
}
// Nat chains should stay at standard priorities.
if !strings.Contains(c, "hook prerouting priority -100;") {
t.Errorf("prerouting chain should use priority -100:\n%s", c)
}
if !strings.Contains(c, "hook postrouting priority 100;") {
t.Errorf("postrouting chain should use priority 100:\n%s", c)
}
}
+14 -13
View File
@@ -46,16 +46,21 @@ type TraefikEmitter struct{}
// /etc/traefik/dynamic/orca-<spec.Name>.yaml.
const traefikDynamicDir = "/etc/traefik/dynamic"
// traefikRouterTLSCertResolver is the Traefik cert-resolver name that
// the orca step-ca integration configures on the Traefik static config
// (P10 / v0.10 wires the step-ca root into this resolver). The
// dynamic-config file references it by name.
// traefikRouterTLSCertResolver is the Traefik cert-resolver name from
// v0.11. As of v0.14 (RESEARCH_v0.14 Topic 4), traefik v3.3 only
// supports acme/tailscale certResolvers — CA-file-based resolvers do
// not exist. The dynamic config now emits `tls: {}` instead. Real
// mTLS via dynamic tls.certificates + clientAuth.caFiles is deferred
// to v0.15. This constant is retained for documentation.
//
// Deprecated: v0.14 removed certResolver from the dynamic config.
const traefikRouterTLSCertResolver = "orca"
// defaultTrustDomain is the SPIFFE trust domain used in the rendered
// TLS stanza when the spec does not carry an explicit trust domain.
// The step-ca provisioner (P10) overrides this at render time via the
// node argument; for P02 the emitter renders the placeholder.
// defaultTrustDomain is the SPIFFE trust domain. v0.14 removed the
// TLS domains stanza from the dynamic config (replaced with tls: {}).
// Retained for documentation; will be used by v0.15 mTLS.
//
// Deprecated: v0.14 removed TLS domains from the dynamic config.
const defaultTrustDomain = "cluster.orca.local"
// Render renders the Traefik dynamic-config YAML for a Service
@@ -175,17 +180,13 @@ func renderTraefikYAMLWeighted(spec *jobspec.WorkloadSpec, node *Node, drain boo
routerName := "orca-" + spec.Name
serviceName := "orca-" + spec.Name
rule := fmt.Sprintf("PathPrefix(\"/%s\")", spec.Name)
trustDomain := defaultTrustDomain
b.WriteString("http:\n")
b.WriteString(" routers:\n")
b.WriteString(fmt.Sprintf(" %s:\n", routerName))
b.WriteString(fmt.Sprintf(" rule: %s\n", rule))
b.WriteString(fmt.Sprintf(" service: %s\n", serviceName))
b.WriteString(" tls:\n")
b.WriteString(fmt.Sprintf(" certResolver: %s\n", traefikRouterTLSCertResolver))
b.WriteString(" domains:\n")
b.WriteString(fmt.Sprintf(" - main: %q\n", trustDomain))
b.WriteString(" tls: {}\n")
b.WriteString(" services:\n")
b.WriteString(fmt.Sprintf(" %s:\n", serviceName))
b.WriteString(" loadBalancer:\n")
+2 -5
View File
@@ -53,11 +53,8 @@ func TestTraefikEmitter_RenderBasic(t *testing.T) {
if !strings.Contains(c, "unix:///run/orca/alloc-node-1/port-http.sock") {
t.Errorf("content missing socket server URL\n%s", c)
}
if !strings.Contains(c, "certResolver: orca") {
t.Errorf("content missing 'certResolver: orca'\n%s", c)
}
if !strings.Contains(c, "domains:") {
t.Errorf("content missing TLS domains\n%s", c)
if !strings.Contains(c, "tls: {}") {
t.Errorf("content missing 'tls: {}' (v0.14: certResolver dropped, tls empty stanza)\n%s", c)
}
if !strings.Contains(c, "healthCheck:") {
t.Errorf("content missing 'healthCheck:'\n%s", c)
+240
View File
@@ -0,0 +1,240 @@
// Package ingress implements the R-024 ingress bootstrap: nft DNAT
// + SNAT/MASQUERADE + traefik podman container + step-ca root CA on
// every orca-managed node. The bootstrap is idempotent and non-fatal
// on each step (offline host tolerance — same as v0.13 traefik install).
//
// BootstrapLocalIngress runs on the lead (during `orca init`).
// BootstrapRemoteIngress runs on workers (during `orca node join`).
package ingress
import (
"context"
"crypto/rand"
"encoding/hex"
"fmt"
"log/slog"
"os"
"os/exec"
"path/filepath"
"strings"
"git.cloudinit.dev/coreci/orca/internal/certpaths"
"git.cloudinit.dev/coreci/orca/internal/emitter"
"git.cloudinit.dev/coreci/orca/internal/traefik"
)
// BootstrapLocalIngress ensures the complete ingress stack is running
// on the local host (lead node). It is called from `orca init` after
// EnsureTraefikContainerLocal.
//
// Steps (each non-fatal — logs a warning and continues):
// 1. mkdir -p /etc/traefik/dynamic /etc/orca
// 2. Push cluster root CA to /etc/orca/step-ca-root.crt (C-60:
// certpaths.CACertPath(), not CAPath)
// 3. Render static config via emitter.RenderTraefikStaticConfig to
// /etc/traefik/traefik.yml (preserves traefik-on-public-ip opt-out,
// C-58)
// 4. Render orca.nft via emitter.NftEmitter.RenderNftConfig + write
// to /etc/nftables.d/orca.nft
// 5. Pre-create nft table (C-55: avoids flush-table error on first
// apply)
// 6. Apply: nft -f /etc/nftables.d/orca.nft
func BootstrapLocalIngress(ctx context.Context, version string) error {
var errs []error
log := slog.Default()
// Step 1: ensure directories.
for _, dir := range []string{"/etc/traefik/dynamic", "/etc/orca"} {
if err := os.MkdirAll(dir, 0o755); err != nil {
log.Warn("ingress: mkdir failed", "dir", dir, "err", err)
errs = append(errs, fmt.Errorf("mkdir %s: %w", dir, err))
}
}
// Step 2: push cluster root CA (C-60: CACertPath, not CAPath).
caPath := certpaths.CACertPath()
if caData, err := os.ReadFile(caPath); err == nil {
if err := os.WriteFile("/etc/orca/step-ca-root.crt", caData, 0o644); err != nil {
log.Warn("ingress: step-ca root CA write failed", "err", err)
errs = append(errs, fmt.Errorf("write step-ca-root.crt: %w", err))
}
} else {
// CA may not exist yet (fresh init before step-ca). Write a
// placeholder so the podman volume mount doesn't fail.
_ = os.WriteFile("/etc/orca/step-ca-root.crt", []byte{}, 0o644)
log.Warn("ingress: step-ca root CA not found, wrote placeholder", "path", caPath)
}
// Step 3: render + write static config (C-58).
staticFiles, err := emitter.TraefikEmitter{}.RenderTraefikStaticConfig(emitter.TraefikStaticOpts{})
if err != nil {
log.Warn("ingress: render traefik static config failed", "err", err)
errs = append(errs, fmt.Errorf("render traefik static: %w", err))
} else {
for _, f := range staticFiles {
if err := os.WriteFile(f.Path, []byte(f.Content), 0o644); err != nil {
log.Warn("ingress: write traefik static config failed", "path", f.Path, "err", err)
errs = append(errs, fmt.Errorf("write %s: %w", f.Path, err))
}
}
}
// Step 4: render + write nft ruleset.
nftFiles, err := emitter.NftEmitter{}.RenderNftConfig(emitter.NftClusterConfig{})
if err != nil {
log.Warn("ingress: render nft config failed", "err", err)
errs = append(errs, fmt.Errorf("render nft: %w", err))
} else {
for _, f := range nftFiles {
dir := filepath.Dir(f.Path)
_ = os.MkdirAll(dir, 0o755)
if err := os.WriteFile(f.Path, []byte(f.Content), 0o644); err != nil {
log.Warn("ingress: write nft config failed", "path", f.Path, "err", err)
errs = append(errs, fmt.Errorf("write %s: %w", f.Path, err))
}
}
// Step 5: pre-create nft table (C-55: flush table on non-existent
// table errors — pre-create avoids the first-apply failure).
_ = exec.CommandContext(ctx, "nft", "add", "table", "inet", "orca-ingress").Run()
// Step 6: apply nft ruleset.
if out, err := exec.CommandContext(ctx, "nft", "-f", nftConfigPath).CombinedOutput(); err != nil {
log.Warn("ingress: nft apply failed", "err", err, "output", string(out))
errs = append(errs, fmt.Errorf("nft -f: %w (output: %s)", err, string(out)))
}
}
// Ensure the podman traefik container is running (Step 7 — C-50:
// installs podman if absent).
if err := traefik.EnsureTraefikContainerLocal(ctx, version); err != nil {
log.Warn("ingress: ensure traefik container failed", "err", err)
errs = append(errs, fmt.Errorf("ensure traefik container: %w", err))
}
if len(errs) > 0 {
return fmt.Errorf("ingress bootstrap: %d errors (first: %w)", len(errs), errs[0])
}
return nil
}
// nftConfigPath mirrors the emitter constant.
const nftConfigPath = "/etc/nftables.d/orca.nft"
// RemoteExecFunc runs a command on a remote host and returns combined
// output. Same signature as traefik.RemoteExecFunc.
type RemoteExecFunc func(cmd string) ([]byte, error)
// BootstrapRemoteIngress ensures the complete ingress stack is running
// on a remote host (linux worker). It is called from
// `orca node join --type linux` after the user setup.
//
// Steps (each non-fatal — logs a warning and continues):
// 1. mkdir -p /etc/traefik/dynamic /etc/orca (remote)
// 2. Push step-ca root CA to remote /etc/orca/step-ca-root.crt
// (C-60: certpaths.CACertPath)
// 3. Render + write static config to remote /etc/traefik/traefik.yml
// (C-58: preserves traefik-on-public-ip opt-out)
// 4. Render + write nft ruleset to remote /etc/nftables.d/orca.nft
// 5. Pre-create nft table (C-55: avoids first-apply flush-table error)
// 6. Apply: nft -f (remote)
// 7. Ensure podman traefik container running (remote)
func BootstrapRemoteIngress(ctx context.Context, version string, execFn RemoteExecFunc) error {
var errs []error
log := slog.Default()
// Step 1: ensure directories.
if _, err := execFn("mkdir -p /etc/traefik/dynamic /etc/orca"); err != nil {
log.Warn("ingress: remote mkdir failed", "err", err)
errs = append(errs, fmt.Errorf("remote mkdir: %w", err))
}
// Step 2: push step-ca root CA (C-60: CACertPath, not CAPath).
if caData, err := os.ReadFile(certpaths.CACertPath()); err == nil {
if err := remoteWriteFile(execFn, "/etc/orca/step-ca-root.crt", caData, "0644"); err != nil {
log.Warn("ingress: remote step-ca CA write failed", "err", err)
errs = append(errs, fmt.Errorf("remote write step-ca-root.crt: %w", err))
}
} else {
// Write a placeholder so the podman volume mount doesn't fail.
_ = remoteWriteFile(execFn, "/etc/orca/step-ca-root.crt", []byte{}, "0644")
log.Warn("ingress: step-ca root CA not found locally, wrote remote placeholder")
}
// Step 3: render + write static config (C-58).
staticFiles, err := emitter.TraefikEmitter{}.RenderTraefikStaticConfig(emitter.TraefikStaticOpts{})
if err != nil {
log.Warn("ingress: render traefik static config failed", "err", err)
errs = append(errs, fmt.Errorf("render traefik static: %w", err))
} else {
for _, f := range staticFiles {
_, _ = execFn(fmt.Sprintf("mkdir -p %s", filepath.Dir(f.Path)))
if err := remoteWriteFile(execFn, f.Path, []byte(f.Content), f.Mode); err != nil {
log.Warn("ingress: remote write traefik static config failed", "path", f.Path, "err", err)
errs = append(errs, fmt.Errorf("remote write %s: %w", f.Path, err))
}
}
}
// Step 4: render + write nft ruleset.
nftFiles, err := emitter.NftEmitter{}.RenderNftConfig(emitter.NftClusterConfig{})
if err != nil {
log.Warn("ingress: render nft config failed", "err", err)
errs = append(errs, fmt.Errorf("render nft: %w", err))
} else {
for _, f := range nftFiles {
_, _ = execFn(fmt.Sprintf("mkdir -p %s", filepath.Dir(f.Path)))
if err := remoteWriteFile(execFn, f.Path, []byte(f.Content), f.Mode); err != nil {
log.Warn("ingress: remote write nft config failed", "path", f.Path, "err", err)
errs = append(errs, fmt.Errorf("remote write %s: %w", f.Path, err))
}
}
// Step 5: pre-create nft table (C-55).
_, _ = execFn("nft add table inet orca-ingress 2>/dev/null || true")
// Step 6: apply nft ruleset.
if out, err := execFn("nft -f /etc/nftables.d/orca.nft 2>&1"); err != nil {
log.Warn("ingress: remote nft apply failed", "err", err, "output", string(out))
errs = append(errs, fmt.Errorf("remote nft -f: %w (output: %s)", err, string(out)))
}
}
// Step 7: ensure podman traefik container (C-50).
traefikExecFn := traefik.RemoteExecFunc(execFn)
if err := traefik.EnsureTraefikContainerRemote(ctx, version, traefikExecFn); err != nil {
log.Warn("ingress: remote ensure traefik container failed", "err", err)
errs = append(errs, fmt.Errorf("remote ensure traefik container: %w", err))
}
if len(errs) > 0 {
return fmt.Errorf("remote ingress bootstrap: %d errors (first: %w)", len(errs), errs[0])
}
return nil
}
// remoteWriteFile writes content to a remote path via a heredoc
// (same pattern as sshpush.idempotency.writeFile). The heredoc
// delimiter is a random hex string verified absent from the content
// (F9 injection guard).
func remoteWriteFile(execFn RemoteExecFunc, path string, content []byte, mode string) error {
// Generate a random delimiter unlikely to be in the content.
delim := "EOF_"
for {
b := make([]byte, 8)
if _, err := rand.Read(b); err != nil {
return fmt.Errorf("rand: %w", err)
}
delim = "EOF_" + hex.EncodeToString(b)
if !strings.Contains(string(content), delim) {
break
}
}
dir := filepath.Dir(path)
cmd := fmt.Sprintf("mkdir -p %s && cat > %s <<'%s'\n%s\n%s\nchmod %s %s",
dir, path, delim, string(content), delim, mode, path)
if out, err := execFn(cmd); err != nil {
return fmt.Errorf("remote write %s: %w (output: %s)", path, err, string(out))
}
return nil
}
+17 -14
View File
@@ -32,9 +32,9 @@ import (
"golang.org/x/crypto/ssh"
"git.cloudinit.dev/coreci/orca/internal/certpaths"
"git.cloudinit.dev/coreci/orca/internal/ingress"
"git.cloudinit.dev/coreci/orca/internal/proxmox"
"git.cloudinit.dev/coreci/orca/internal/security"
"git.cloudinit.dev/coreci/orca/internal/traefik"
)
// DefaultSSHUser is the default SSH username for the initial connection.
@@ -48,13 +48,13 @@ const DefaultSSHPort = 22
// Options configures a Linux worker bootstrap run.
type Options struct {
Host string
SSHUser string
SSHKeyPath string
OrcaUser string
SSHPort int
HostKeyFingerprint string
Logger *slog.Logger
Host string
SSHUser string
SSHKeyPath string
OrcaUser string
SSHPort int
HostKeyFingerprint string
Logger *slog.Logger
}
// Result is the outcome of a successful bootstrap.
@@ -157,8 +157,10 @@ func BootstrapLinux(ctx context.Context, opts Options) (*Result, error) {
}
opts.Logger.Info("linux bootstrap: user created", "user", opts.OrcaUser)
// Step 4d: Install Traefik on the remote host (REQ-165, Phase B).
// Traefik is the data-plane ingress; SSH is control plane only.
// Step 4d: Ensure complete ingress stack on the remote host (R-024).
// Renders+applies nft DNAT/SNAT, pushes step-ca root CA, renders+
// writes traefik static config, ensures podman container running.
// All non-fatal (offline host tolerance).
sshExecFn := func(cmd string) ([]byte, error) {
session, err := client.NewSession()
if err != nil {
@@ -167,8 +169,10 @@ func BootstrapLinux(ctx context.Context, opts Options) (*Result, error) {
defer session.Close()
return session.CombinedOutput(cmd)
}
if err := traefik.InstallRemote("", sshExecFn); err != nil {
opts.Logger.Warn("linux bootstrap: traefik install failed", "err", err)
ctx, cancelIngress := context.WithTimeout(ctx, 120*time.Second)
defer cancelIngress()
if err := ingress.BootstrapRemoteIngress(ctx, "", ingress.RemoteExecFunc(sshExecFn)); err != nil {
opts.Logger.Warn("linux bootstrap: ingress bootstrap failed", "err", err)
}
// Step 5: Create the drift-events directory.
@@ -188,7 +192,7 @@ func BootstrapLinux(ctx context.Context, opts Options) (*Result, error) {
return &Result{
NodeName: opts.Host,
NodeAddress: fmt.Sprintf("%s:8443", opts.Host),
NodeAddress: fmt.Sprintf("%s:8443", opts.Host),
HostKeyFingerprint: hostKeyFP,
}, nil
}
@@ -226,5 +230,4 @@ func pinnedHostKeyCallback(expectedSHA256Base64 string, capturedKey *ssh.PublicK
return cb, nil
}
var _ = security.WriteAtomic
+5
View File
@@ -39,4 +39,9 @@ type Node struct {
// OS is the auto-detected OS identifier from /etc/os-release ID=
// (ubuntu|debian|alpine|pve|linux). Empty for pre-0006 rows.
OS string `json:"os,omitempty"`
// IngressMode is the ingress configuration for the node (R-024,
// v0.14). Values: "" (legacy/default for linux/localhost),
// "native" (proxmox native — traefik in LXC), "floating-ip"
// (proxmox floating-IP — ingress LXC owns the floating IP).
IngressMode string `json:"ingress_mode,omitempty"`
}
+133 -9
View File
@@ -37,6 +37,7 @@ import (
"golang.org/x/crypto/ssh/knownhosts"
"git.cloudinit.dev/coreci/orca/internal/certpaths"
"git.cloudinit.dev/coreci/orca/internal/emitter"
"git.cloudinit.dev/coreci/orca/internal/security"
"git.cloudinit.dev/coreci/orca/internal/traefik"
)
@@ -86,6 +87,13 @@ type Options struct {
// LXCTemplate is the LXC template to download during bootstrap
// (default "ubuntu-24.04"; alternatives: "alpine-3.20", "debian-12").
LXCTemplate string
// IngressMode is the proxmox ingress mode (R-024, v0.14).
// "native" (default): traefik runs in an unprivileged LXC with
// nesting=1,keyctl=1,fuse=1 on the PVE host. nft on the PVE host
// DNATs to the LXC bridge IP.
// "floating-ip": a separate ingress LXC owns the floating IP;
// nft runs inside that LXC. See ProvisionIngressLXC (P6).
IngressMode string
}
// Result is the outcome of a successful bootstrap.
@@ -247,21 +255,24 @@ func BootstrapProxmox(ctx context.Context, opts Options) (*Result, error) {
return nil, fmt.Errorf("validate sudoers: %w", err)
}
// Step 9a: Install Traefik on the Proxmox host (REQ-165, Phase B).
// Traefik runs on the PVE OS as the data-plane ingress; SSH is
// control plane only. Idempotent: skips if binary already exists.
if err := traefik.InstallRemote("", runRemote); err != nil {
log.Warn("proxmox.traefik_install_failed", "err", err)
}
// Step 9b: Download default LXC template (REQ-167, Phase C).
// Default: ubuntu-24.04. Configurable via --lxc-template.
// Step 9a: Proxmox native ingress mode (R-024, REQ-175).
// Create an unprivileged LXC with nesting=1,keyctl=1,fuse=1 (research
// Topic 3), install podman inside it, and run the orca-traefik
// container. nft on the PVE host DNATs to the LXC bridge IP.
// Default mode is "native"; floating-ip mode is handled separately
// (P6 — ProvisionIngressLXC).
template := opts.LXCTemplate
if template == "" {
template = "ubuntu-24.04"
}
_, _ = runRemote(fmt.Sprintf("pveam download local %s 2>/dev/null || true", shellQuote(template)))
if opts.IngressMode != "floating-ip" {
if err := provisionNativeIngressLXC(ctx, runRemote, template, log); err != nil {
log.Warn("proxmox.native_ingress_lxc_failed", "err", err)
}
}
log.Info("proxmox.bootstrap_ok",
slog.String("event", "proxmox.bootstrap_ok"),
slog.String("host", opts.Host),
@@ -276,6 +287,119 @@ func BootstrapProxmox(ctx context.Context, opts Options) (*Result, error) {
}, nil
}
// provisionNativeIngressLXC creates an unprivileged LXC with
// nesting=1,keyctl=1,fuse=1 (research Topic 3), installs podman inside
// it, runs the orca-traefik container, and applies nft DNAT on the PVE
// host targeting the LXC's bridge IP (R-024, REQ-175).
//
// The LXC is named "orca-traefik" and uses a deterministic VMID derived
// from the host. It is idempotent: if the LXC already exists, it is
// not re-created (C-53: apt-get install is skipped if podman present).
func provisionNativeIngressLXC(ctx context.Context, runRemote func(string) ([]byte, error), template string, log *slog.Logger) error {
// Deterministic VMID for the native ingress LXC.
// Use a fixed VMID in the 200-299 range (Proxmox convention for CTs).
const vmid = "200"
const lxcName = "orca-traefik"
// Check if the LXC already exists.
existOut, _ := runRemote(fmt.Sprintf("pct status %s 2>/dev/null || echo absent", vmid))
existStr := strings.TrimSpace(string(existOut))
if existStr == "absent" {
// Create the LXC (research Topic 3: nesting=1,keyctl=1,fuse=1).
log.Info("proxmox.creating_native_ingress_lxc", "vmid", vmid, "name", lxcName)
createCmd := fmt.Sprintf(
"pct create %s local:vztmpl/%s --hostname %s --unprivileged 1 --features nesting=1,keyctl=1,fuse=1 --onboot 1 --memory 2048 --swap 0 --rootfs local:8 2>&1",
vmid, shellQuote(template), lxcName,
)
if out, err := runRemote(createCmd); err != nil {
return fmt.Errorf("pct create native ingress LXC: %w (output: %s)", err, string(out))
}
if out, err := runRemote(fmt.Sprintf("pct start %s", vmid)); err != nil {
return fmt.Errorf("pct start native ingress LXC: %w (output: %s)", err, string(out))
}
}
// Wait for LXC network (retry for up to 60s).
lxcIP := ""
for i := 0; i < 12; i++ {
ipOut, _ := runRemote(fmt.Sprintf("pct exec %s -- hostname -I 2>/dev/null", vmid))
ipStr := strings.TrimSpace(string(ipOut))
if ipStr != "" {
fields := strings.Fields(ipStr)
if len(fields) > 0 {
lxcIP = fields[0]
break
}
}
time.Sleep(5 * time.Second)
}
if lxcIP == "" {
return fmt.Errorf("native ingress LXC: could not discover IP after 60s")
}
log.Info("proxmox.native_ingress_lxc_ip", "vmid", vmid, "ip", lxcIP)
// Install podman inside the LXC (C-53: idempotent — check first).
_, _ = runRemote(fmt.Sprintf(
"pct exec %s -- bash -c 'command -v podman >/dev/null 2>&1 || (apt-get update -qq && apt-get install -y -qq podman conmon crun fuse-overlayfs nftables 2>&1)' 2>&1",
vmid,
))
// Enable podman-restart.service inside the LXC (research Topic 6).
_, _ = runRemote(fmt.Sprintf("pct exec %s -- systemctl enable --now podman-restart.service 2>/dev/null", vmid))
// Push step-ca root CA into the LXC (placeholder if absent locally).
caPath := certpaths.CACertPath()
caData, caErr := os.ReadFile(caPath)
if caErr != nil {
caData = []byte{}
}
// Write CA via pct exec heredoc.
caDelim := "EOF_CA"
_, _ = runRemote(fmt.Sprintf(
"pct exec %s -- bash -c 'mkdir -p /etc/orca && cat > /etc/orca/step-ca-root.crt <<%s\\n%s\\n%s'",
vmid, caDelim, string(caData), caDelim,
))
// Render + write traefik static config inside the LXC.
staticFiles, err := emitter.TraefikEmitter{}.RenderTraefikStaticConfig(emitter.TraefikStaticOpts{})
if err == nil {
for _, f := range staticFiles {
delim := "EOF_TF"
_, _ = runRemote(fmt.Sprintf(
"pct exec %s -- bash -c 'mkdir -p /etc/traefik/dynamic && cat > %s <<%s\\n%s\\n%s'",
vmid, f.Path, delim, f.Content, delim,
))
}
}
// Ensure podman orca-traefik container inside the LXC.
traefikExecFn := func(cmd string) ([]byte, error) {
return runRemote(fmt.Sprintf("pct exec %s -- bash -c %s 2>&1", vmid, shellQuote(cmd)))
}
if err := traefik.EnsureTraefikContainerRemote(ctx, "", traefikExecFn); err != nil {
log.Warn("proxmox.native_ingress_lxc_traefik_failed", "err", err)
}
// Render + apply nft on the PVE host with DNATTarget = LXC IP.
nftFiles, err := emitter.NftEmitter{}.RenderNftConfig(emitter.NftClusterConfig{
DNATTarget: lxcIP,
})
if err == nil {
for _, f := range nftFiles {
nftDelim := "EOF_NF"
_, _ = runRemote(fmt.Sprintf("mkdir -p /etc/nftables.d && cat > %s <<%s\\n%s\\n%s",
f.Path, nftDelim, f.Content, nftDelim))
}
_, _ = runRemote("nft add table inet orca-ingress 2>/dev/null || true")
if out, err := runRemote("nft -f /etc/nftables.d/orca.nft 2>&1"); err != nil {
log.Warn("proxmox.native_ingress_nft_apply_failed", "err", err, "output", string(out))
}
}
log.Info("proxmox.native_ingress_lxc_ok", "vmid", vmid, "ip", lxcIP)
return nil
}
// sshDialer is the dialer used by BootstrapProxmox. It's a package-level
// variable so tests can override it with a fake SSH server.
var sshDialer sshDialerType = defaultSSHDialer{}
+171
View File
@@ -0,0 +1,171 @@
package proxmox
import (
"context"
"crypto/rand"
"fmt"
"log/slog"
"os"
"strings"
"time"
"git.cloudinit.dev/coreci/orca/internal/certpaths"
"git.cloudinit.dev/coreci/orca/internal/emitter"
"git.cloudinit.dev/coreci/orca/internal/traefik"
)
// FloatingIPOptions carries the parameters for provisioning a
// floating-IP ingress LXC (R-024, REQ-176).
type FloatingIPOptions struct {
// FloatingIP is the public IP assigned to the LXC's eth0.
FloatingIP string
// Gateway is the default gateway for the LXC.
Gateway string
// MAC is the MAC address for the LXC's net0 interface.
MAC string
// NetPrefix is the CIDR prefix for the floating IP (8-32).
NetPrefix int
// LXCTemplate is the LXC template (default "ubuntu-24.04").
LXCTemplate string
// VMID is the LXC container ID (default "201" for the ingress LXC).
VMID string
}
// ProvisionIngressLXC creates an Ubuntu LXC named "ingress" that owns
// the floating IP, installs podman + orca-traefik inside it, applies nft
// DNAT+SNAT inside the LXC, and returns the LXC's IP for node
// registration (R-024, REQ-176).
//
// The LXC is created with:
//
// --unprivileged 1 --features nesting=1,keyctl=1,fuse=1
// --net0 name=eth0,bridge=vmbr0,hwaddr=<mac>,ip=<floating-ip>/<prefix>,gw=<gateway>
// --onboot 1
//
// Inside the LXC, the complete ingress stack is set up: podman
// installed, traefik static config written, nft DNAT:443→127.0.0.1:8443
// + postrouting masquerade applied, orca-traefik podman container
// running with --network host.
//
// Idempotent: if the LXC already exists, it is not re-created (C-53).
func ProvisionIngressLXC(ctx context.Context, runRemote func(string) ([]byte, error), opts FloatingIPOptions, log *slog.Logger) error {
template := opts.LXCTemplate
if template == "" {
template = "ubuntu-24.04"
}
vmid := opts.VMID
if vmid == "" {
vmid = "201"
}
if opts.NetPrefix == 0 {
opts.NetPrefix = 24
}
// Validate required fields.
if opts.FloatingIP == "" || opts.Gateway == "" || opts.MAC == "" {
return fmt.Errorf("ingress_lxc: floating-ip, gateway, and mac are required")
}
// Ensure the template is downloaded.
_, _ = runRemote(fmt.Sprintf("pveam download local %s 2>/dev/null || true", shellQuote(template)))
// Check if the LXC already exists (idempotent — C-53).
existOut, _ := runRemote(fmt.Sprintf("pct status %s 2>/dev/null || echo absent", vmid))
existStr := strings.TrimSpace(string(existOut))
if existStr == "absent" {
log.Info("ingress_lxc.creating", "vmid", vmid, "hostname", "ingress", "ip", opts.FloatingIP)
net0 := fmt.Sprintf("name=eth0,bridge=vmbr0,hwaddr=%s,ip=%s/%d,gw=%s",
opts.MAC, opts.FloatingIP, opts.NetPrefix, opts.Gateway)
createCmd := fmt.Sprintf(
"pct create %s local:vztmpl/%s --hostname ingress --unprivileged 1 --features nesting=1,keyctl=1,fuse=1 --net0 %s --onboot 1 --memory 2048 --swap 0 --rootfs local:8 2>&1",
vmid, shellQuote(template), net0,
)
if out, err := runRemote(createCmd); err != nil {
return fmt.Errorf("pct create ingress LXC: %w (output: %s)", err, string(out))
}
if out, err := runRemote(fmt.Sprintf("pct start %s", vmid)); err != nil {
return fmt.Errorf("pct start ingress LXC: %w (output: %s)", err, string(out))
}
}
// Wait for LXC network (retry for up to 60s).
for i := 0; i < 12; i++ {
ipOut, _ := runRemote(fmt.Sprintf("pct exec %s -- hostname -I 2>/dev/null", vmid))
ipStr := strings.TrimSpace(string(ipOut))
if ipStr != "" {
break
}
time.Sleep(5 * time.Second)
}
log.Info("ingress_lxc.network_ready", "vmid", vmid, "ip", opts.FloatingIP)
// Install podman inside the LXC (C-53: idempotent).
_, _ = runRemote(fmt.Sprintf(
"pct exec %s -- bash -c 'command -v podman >/dev/null 2>&1 || (apt-get update -qq && apt-get install -y -qq podman conmon crun fuse-overlayfs nftables 2>&1)' 2>&1",
vmid,
))
// Enable podman-restart.service inside the LXC (research Topic 6).
_, _ = runRemote(fmt.Sprintf("pct exec %s -- systemctl enable --now podman-restart.service 2>/dev/null", vmid))
// Push step-ca root CA into the LXC (C-60: CACertPath).
caPath := certpaths.CACertPath()
caData, caErr := os.ReadFile(caPath)
if caErr != nil {
caData = []byte{}
}
caDelim := "EOF_CA"
_, _ = runRemote(fmt.Sprintf(
"pct exec %s -- bash -c 'mkdir -p /etc/orca && cat > /etc/orca/step-ca-root.crt <<%s\\n%s\\n%s'",
vmid, caDelim, string(caData), caDelim,
))
// Render + write traefik static config inside the LXC (C-58).
staticFiles, err := emitter.TraefikEmitter{}.RenderTraefikStaticConfig(emitter.TraefikStaticOpts{})
if err == nil {
for _, f := range staticFiles {
delim := "EOF_TF"
_, _ = runRemote(fmt.Sprintf(
"pct exec %s -- bash -c 'mkdir -p /etc/traefik/dynamic && cat > %s <<%s\\n%s\\n%s'",
vmid, f.Path, delim, f.Content, delim,
))
}
}
// Render + apply nft DNAT+SNAT INSIDE the LXC (DNATTarget =
// 127.0.0.1 — traefik runs with --network host inside the LXC).
nftFiles, err := emitter.NftEmitter{}.RenderNftConfig(emitter.NftClusterConfig{})
if err == nil {
for _, f := range nftFiles {
delim := "EOF_NF"
_, _ = runRemote(fmt.Sprintf(
"pct exec %s -- bash -c 'mkdir -p /etc/nftables.d && cat > %s <<%s\\n%s\\n%s'",
vmid, f.Path, delim, f.Content, delim,
))
}
_, _ = runRemote(fmt.Sprintf("pct exec %s -- nft add table inet orca-ingress 2>/dev/null || true", vmid))
_, _ = runRemote(fmt.Sprintf("pct exec %s -- nft -f /etc/nftables.d/orca.nft 2>&1", vmid))
}
// Ensure podman orca-traefik container inside the LXC.
traefikExecFn := func(cmd string) ([]byte, error) {
return runRemote(fmt.Sprintf("pct exec %s -- bash -c %s 2>&1", vmid, shellQuote(cmd)))
}
if err := traefik.EnsureTraefikContainerRemote(ctx, "", traefikExecFn); err != nil {
log.Warn("ingress_lxc.traefik_failed", "err", err)
}
log.Info("ingress_lxc.provisioned", "vmid", vmid, "ip", opts.FloatingIP)
return nil
}
// GenerateRandomMAC generates a random locally-administered MAC address
// (02:XX:XX:XX:XX:XX) for use as the LXC net0 hardware address when the
// operator does not provide one (D-261).
func GenerateRandomMAC() (string, error) {
b := make([]byte, 5)
if _, err := rand.Read(b); err != nil {
return "", fmt.Errorf("generate MAC: %w", err)
}
return fmt.Sprintf("02:%02x:%02x:%02x:%02x:%02x", b[0], b[1], b[2], b[3], b[4]), nil
}
@@ -0,0 +1,6 @@
-- REQ-175 / R-024: add ingress_mode column to nodes.
-- Values: '' (legacy/default), 'native' (proxmox native — traefik
-- in LXC), 'floating-ip' (proxmox floating-IP — ingress LXC owns
-- the floating IP). Defaults to empty string for backward
-- compatibility with pre-v0.14 nodes.
ALTER TABLE nodes ADD COLUMN ingress_mode TEXT NOT NULL DEFAULT '';
+202 -52
View File
@@ -1,78 +1,228 @@
package traefik
import (
"context"
"fmt"
"os/exec"
"strings"
)
// DefaultVersion is the traefik version tag for the orca-traefik image.
const DefaultVersion = "v3.3.0"
func downloadURL(version string) string {
return fmt.Sprintf("https://github.com/traefik/traefik/releases/download/%s/traefik_%s_linux_amd64.tar.gz", version, version)
}
// DefaultImage is the full image reference for the orca-traefik container.
// The tag is resolved at runtime from the orca version (or "latest" for
// dev builds). The image is built from Dockerfile.traefik and published
// per release (REQ-171).
const DefaultImage = "git.cloudinit.dev/coreci/orca-traefik"
func InstallLocal(version string) error {
if version == "" {
version = DefaultVersion
}
if _, err := exec.LookPath("traefik"); err == nil {
ensureDirs()
return nil
}
url := downloadURL(version)
cmd := exec.Command("bash", "-c",
fmt.Sprintf(`curl -fsSL %s | tar -xzf - -C /usr/local/bin/ traefik && chmod +x /usr/local/bin/traefik`, url))
if out, err := cmd.CombinedOutput(); err != nil {
return fmt.Errorf("download traefik %s: %w (output: %s)", version, err, string(out))
}
ensureDirs()
writeSystemdUnit()
_ = exec.Command("systemctl", "daemon-reload").Run()
_ = exec.Command("systemctl", "enable", "--now", "orca-traefik").Run()
return nil
}
// ContainerName is the podman container name for the traefik data plane.
const ContainerName = "orca-traefik"
// RemoteExecFunc runs a command on a remote host and returns combined
// output. It is the same signature used by the v0.13 binary installer
// and by the proxmox/linux bootstrap SSH sessions.
type RemoteExecFunc func(cmd string) ([]byte, error)
func InstallRemote(version string, execFn RemoteExecFunc) error {
if version == "" {
version = DefaultVersion
// ImageRef returns the full image:tag reference for the orca-traefik
// container. If version is empty or "dev"/"0.1.0-dev", it falls back to
// "latest" (dev builds don't have a published tag).
func ImageRef(version string) string {
tag := version
if tag == "" || tag == "dev" || tag == "0.1.0-dev" || strings.HasSuffix(tag, "-dev") {
return fmt.Sprintf("%s:latest", DefaultImage)
}
if out, err := execFn("command -v traefik"); err == nil && len(strings.TrimSpace(string(out))) > 0 {
_, _ = execFn("mkdir -p /etc/traefik/dynamic")
tag = strings.TrimPrefix(tag, "v")
return fmt.Sprintf("%s:v%s", DefaultImage, tag)
}
// podmanRunArgs returns the podman run arguments for the traefik
// container. The container uses --network host so traefik binds
// 127.0.0.1:8080/8443 directly on the host (or LXC) loopback. nft
// DNATs public :443/:80 to those loopback ports (R-017/R-024).
//
// Volume mounts (no SELinux :Z flag — research finding Topic 7):
// - /etc/traefik/traefik.yml:ro — static config (overrides baked default; C-58)
// - /etc/traefik/dynamic:ro — dynamic config (orca writes atomically via SSH-push)
// - /etc/orca/step-ca-root.crt:ro — step-ca root CA (for future mTLS; v0.14 uses tls:{})
func podmanRunArgs(imageRef string) []string {
return []string{
"run", "-d",
"--name", ContainerName,
"--restart=unless-stopped",
"--network", "host",
"-v", "/etc/traefik/traefik.yml:/etc/traefik/traefik.yml:ro",
"-v", "/etc/traefik/dynamic:/etc/traefik/dynamic:ro",
"-v", "/etc/orca/step-ca-root.crt:/etc/orca/step-ca-root.crt:ro",
imageRef,
}
}
// EnsureTraefikContainerLocal ensures the orca-traefik podman container
// is running on the local host. It is idempotent:
// 1. If the container is running → no-op.
// 2. If the container exists but is stopped → start it.
// 3. If the container does not exist → pull the image + run it.
//
// C-50: if podman is not installed, attempts apt-get install. If that
// fails, returns an error with install instructions.
//
// C-57: if a legacy v0.13 systemd service exists (orca-traefik.service),
// it is stopped, disabled, and removed before starting the container.
func EnsureTraefikContainerLocal(ctx context.Context, version string) error {
imageRef := ImageRef(version)
if err := ensurePodmanLocal(ctx); err != nil {
return err
}
if err := removeLegacySystemdUnitLocal(ctx); err != nil {
// Non-fatal: legacy unit may not exist on fresh installs.
_ = err
}
if err := ensureDirsLocal(); err != nil {
return fmt.Errorf("traefik: ensure dirs: %w", err)
}
return reconcileContainerLocal(ctx, imageRef)
}
// EnsureTraefikContainerRemote ensures the orca-traefik podman container
// is running on a remote host (via SSH exec). Same idempotent logic as
// EnsureTraefikContainerLocal but over the provided exec function.
func EnsureTraefikContainerRemote(ctx context.Context, version string, execFn RemoteExecFunc) error {
imageRef := ImageRef(version)
if err := ensurePodmanRemote(execFn); err != nil {
return err
}
if err := removeLegacySystemdUnitRemote(execFn); err != nil {
_ = err // non-fatal
}
if _, err := execFn("mkdir -p /etc/traefik/dynamic /etc/orca"); err != nil {
return fmt.Errorf("traefik: ensure remote dirs: %w", err)
}
return reconcileContainerRemote(execFn, imageRef)
}
// ensurePodmanLocal checks if podman is installed locally and attempts
// to install it if absent (C-50).
func ensurePodmanLocal(ctx context.Context) error {
if _, err := exec.LookPath("podman"); err == nil {
return nil
}
url := downloadURL(version)
installCmd := fmt.Sprintf(`curl -fsSL %s | tar -xzf - -C /usr/local/bin/ traefik && chmod +x /usr/local/bin/traefik && mkdir -p /etc/traefik/dynamic`, url)
if out, err := execFn(installCmd); err != nil {
return fmt.Errorf("download traefik on remote: %w (output: %s)", err, string(out))
// Attempt apt-get install (Ubuntu/Debian).
cmd := exec.CommandContext(ctx, "bash", "-c",
"apt-get update -qq && apt-get install -y -qq podman conmon crun fuse-overlayfs 2>&1")
if out, err := cmd.CombinedOutput(); err != nil {
return fmt.Errorf("podman not found and apt-get install failed: %w (output: %s).\nInstall podman manually: apt-get install podman conmon crun fuse-overlayfs", err, string(out))
}
unit := systemdUnitContent()
_, _ = execFn(fmt.Sprintf(`echo '%s' > /etc/systemd/system/orca-traefik.service`, unit))
_, _ = execFn("systemctl daemon-reload && systemctl enable --now orca-traefik")
return nil
}
func ensureDirs() {
_ = exec.Command("mkdir", "-p", "/etc/traefik/dynamic").Run()
// ensurePodmanRemote checks if podman is installed on the remote host
// and attempts to install it if absent (C-50).
func ensurePodmanRemote(execFn RemoteExecFunc) error {
if out, err := execFn("command -v podman"); err == nil && len(strings.TrimSpace(string(out))) > 0 {
return nil
}
// Attempt apt-get install on the remote host.
cmd := "apt-get update -qq && apt-get install -y -qq podman conmon crun fuse-overlayfs 2>&1"
if out, err := execFn(cmd); err != nil {
return fmt.Errorf("podman not found on remote and apt-get install failed: %w (output: %s)", err, string(out))
}
return nil
}
func writeSystemdUnit() {
_ = exec.Command("bash", "-c", fmt.Sprintf(`echo '%s' > /etc/systemd/system/orca-traefik.service`, systemdUnitContent())).Run()
// removeLegacySystemdUnitLocal stops, disables, and removes the legacy
// v0.13 orca-traefik.service systemd unit + /usr/local/bin/traefik
// binary (C-57). Idempotent — no-op if the unit doesn't exist.
func removeLegacySystemdUnitLocal(ctx context.Context) error {
// Check if the legacy unit exists.
if _, err := exec.CommandContext(ctx, "systemctl", "is-active", "orca-traefik.service").CombinedOutput(); err == nil {
// Unit is active or exists — stop + disable it.
_ = exec.CommandContext(ctx, "systemctl", "stop", "orca-traefik.service").Run()
_ = exec.CommandContext(ctx, "systemctl", "disable", "orca-traefik.service").Run()
}
// Remove the unit file and binary.
_ = exec.CommandContext(ctx, "rm", "-f", "/etc/systemd/system/orca-traefik.service").Run()
_ = exec.CommandContext(ctx, "rm", "-f", "/usr/local/bin/traefik").Run()
_ = exec.CommandContext(ctx, "systemctl", "daemon-reload").Run()
return nil
}
func systemdUnitContent() string {
return `[Unit]
Description=Orca Traefik Data Plane
After=network.target
[Service]
Type=simple
ExecStart=/usr/local/bin/traefik --configFile=/etc/traefik/traefik.yml
Restart=on-failure
RestartSec=5s
[Install]
WantedBy=multi-user.target`
// removeLegacySystemdUnitRemote is the remote SSH variant (C-57).
func removeLegacySystemdUnitRemote(execFn RemoteExecFunc) error {
cmd := `systemctl is-active orca-traefik.service 2>/dev/null && systemctl stop orca-traefik.service 2>/dev/null; systemctl disable orca-traefik.service 2>/dev/null; rm -f /etc/systemd/system/orca-traefik.service /usr/local/bin/traefik; systemctl daemon-reload 2>/dev/null; true`
_, _ = execFn(cmd)
return nil
}
// ensureDirsLocal creates /etc/traefik/dynamic and /etc/orca locally.
func ensureDirsLocal() error {
if err := exec.Command("mkdir", "-p", "/etc/traefik/dynamic", "/etc/orca").Run(); err != nil {
return fmt.Errorf("mkdir: %w", err)
}
return nil
}
// reconcileContainerLocal implements the idempotent pull+run logic
// locally (C-50).
func reconcileContainerLocal(ctx context.Context, imageRef string) error {
// Check if the container is already running.
out, err := exec.CommandContext(ctx, "podman", "inspect", "--format", "{{.State.Running}}", ContainerName).CombinedOutput()
if err == nil {
v := strings.TrimSpace(string(out))
if v == "true" {
return nil // already running
}
// Container exists but is stopped — start it.
if _, err := exec.CommandContext(ctx, "podman", "start", ContainerName).CombinedOutput(); err != nil {
return fmt.Errorf("podman start %s: %w", ContainerName, err)
}
return nil
}
// Container does not exist — pull + run.
if out, err := exec.CommandContext(ctx, "podman", "pull", imageRef).CombinedOutput(); err != nil {
return fmt.Errorf("podman pull %s: %w (output: %s)", imageRef, err, string(out))
}
args := append([]string{}, podmanRunArgs(imageRef)...)
if out, err := exec.CommandContext(ctx, "podman", args...).CombinedOutput(); err != nil {
return fmt.Errorf("podman run: %w (output: %s)", err, string(out))
}
// Enable podman-restart.service for reboot persistence (research Topic 6).
_ = exec.CommandContext(ctx, "systemctl", "enable", "--now", "podman-restart.service").Run()
return nil
}
// reconcileContainerRemote implements the idempotent pull+run logic
// over SSH exec.
func reconcileContainerRemote(execFn RemoteExecFunc, imageRef string) error {
// Check if the container is already running.
out, err := execFn(fmt.Sprintf("podman inspect --format '{{.State.Running}}' %s 2>/dev/null", ContainerName))
if err == nil {
v := strings.TrimSpace(string(out))
if v == "true" {
return nil // already running
}
// Container exists but is stopped — start it.
if _, err := execFn(fmt.Sprintf("podman start %s 2>/dev/null", ContainerName)); err != nil {
return fmt.Errorf("podman start %s: %w", ContainerName, err)
}
return nil
}
// Container does not exist — pull + run.
if out, err := execFn(fmt.Sprintf("podman pull %s", shellQuote(imageRef))); err != nil {
return fmt.Errorf("podman pull %s: %w (output: %s)", imageRef, err, string(out))
}
runArgs := strings.Join(podmanRunArgs(imageRef), " ")
if out, err := execFn(fmt.Sprintf("podman %s", runArgs)); err != nil {
return fmt.Errorf("podman run: %w (output: %s)", err, string(out))
}
// Enable podman-restart.service for reboot persistence (research Topic 6).
_, _ = execFn("systemctl enable --now podman-restart.service 2>/dev/null || true")
return nil
}
// shellQuote wraps a string in single quotes for shell-safe usage.
func shellQuote(s string) string {
return "'" + strings.ReplaceAll(s, "'", "'\\''") + "'"
}
+166
View File
@@ -0,0 +1,166 @@
package traefik
import (
"context"
"strings"
"testing"
)
func TestImageRef(t *testing.T) {
tests := []struct {
version string
want string
}{
{"v0.13.1", "git.cloudinit.dev/coreci/orca-traefik:v0.13.1"},
{"0.13.1", "git.cloudinit.dev/coreci/orca-traefik:v0.13.1"},
{"", "git.cloudinit.dev/coreci/orca-traefik:latest"},
{"dev", "git.cloudinit.dev/coreci/orca-traefik:latest"},
{"0.1.0-dev", "git.cloudinit.dev/coreci/orca-traefik:latest"},
{"v1.2.3-dev", "git.cloudinit.dev/coreci/orca-traefik:latest"},
}
for _, tt := range tests {
got := ImageRef(tt.version)
if got != tt.want {
t.Errorf("ImageRef(%q) = %q, want %q", tt.version, got, tt.want)
}
}
}
func TestPodmanRunArgs(t *testing.T) {
args := podmanRunArgs("git.cloudinit.dev/coreci/orca-traefik:v0.13.1")
joined := strings.Join(args, " ")
checks := []string{
"run -d",
"--name orca-traefik",
"--restart=unless-stopped",
"--network host",
"/etc/traefik/traefik.yml:/etc/traefik/traefik.yml:ro",
"/etc/traefik/dynamic:/etc/traefik/dynamic:ro",
"/etc/orca/step-ca-root.crt:/etc/orca/step-ca-root.crt:ro",
"git.cloudinit.dev/coreci/orca-traefik:v0.13.1",
}
for _, c := range checks {
if !strings.Contains(joined, c) {
t.Errorf("podmanRunArgs missing %q\nfull: %s", c, joined)
}
}
// Ensure no :Z flag (research Topic 7)
if strings.Contains(joined, ":Z") {
t.Errorf("podmanRunArgs should NOT contain :Z SELinux flag\nfull: %s", joined)
}
// Ensure --restart=always is NOT used (research Topic 6)
if strings.Contains(joined, "--restart=always") {
t.Errorf("podmanRunArgs should use --restart=unless-stopped, not --restart=always\nfull: %s", joined)
}
}
func TestEnsureTraefikContainerRemote_ContainerRunning(t *testing.T) {
var cmds []string
execFn := func(cmd string) ([]byte, error) {
cmds = append(cmds, cmd)
if strings.Contains(cmd, "podman inspect") {
return []byte("true\n"), nil
}
return []byte(""), nil
}
err := EnsureTraefikContainerRemote(context.Background(), "v0.13.1", execFn)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
// Should have checked inspect and found it running — no pull/run.
if len(cmds) < 1 {
t.Fatal("expected at least 1 command (inspect)")
}
for _, c := range cmds {
if strings.Contains(c, "podman pull") {
t.Errorf("should not pull when container is running: %s", c)
}
if strings.Contains(c, "podman run") {
t.Errorf("should not run when container is running: %s", c)
}
}
}
func TestEnsureTraefikContainerRemote_ContainerStopped(t *testing.T) {
var cmds []string
execFn := func(cmd string) ([]byte, error) {
cmds = append(cmds, cmd)
if strings.Contains(cmd, "podman inspect") {
return []byte("false\n"), nil // stopped
}
return []byte(""), nil
}
err := EnsureTraefikContainerRemote(context.Background(), "v0.13.1", execFn)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
// Should have started the container.
foundStart := false
for _, c := range cmds {
if strings.Contains(c, "podman start orca-traefik") {
foundStart = true
}
}
if !foundStart {
t.Errorf("expected 'podman start orca-traefik' when container is stopped\ncommands: %v", cmds)
}
}
func TestEnsureTraefikContainerRemote_ContainerAbsent(t *testing.T) {
var cmds []string
execFn := func(cmd string) ([]byte, error) {
cmds = append(cmds, cmd)
if strings.Contains(cmd, "podman inspect") {
return nil, &execError{"inspect failed: no such container"}
}
return []byte(""), nil
}
err := EnsureTraefikContainerRemote(context.Background(), "v0.13.1", execFn)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
// Should have pulled and run.
foundPull := false
foundRun := false
for _, c := range cmds {
if strings.Contains(c, "podman pull") {
foundPull = true
}
if strings.Contains(c, "podman run -d") {
foundRun = true
}
}
if !foundPull {
t.Errorf("expected 'podman pull' when container is absent\ncommands: %v", cmds)
}
if !foundRun {
t.Errorf("expected 'podman run -d' when container is absent\ncommands: %v", cmds)
}
}
func TestEnsureTraefikContainerRemote_LegacySystemdRemoval(t *testing.T) {
var cmds []string
execFn := func(cmd string) ([]byte, error) {
cmds = append(cmds, cmd)
if strings.Contains(cmd, "podman inspect") {
return []byte("true\n"), nil // container running
}
return []byte(""), nil
}
_ = EnsureTraefikContainerRemote(context.Background(), "v0.13.1", execFn)
// Should include legacy systemd unit removal command (C-57).
foundLegacyRemoval := false
for _, c := range cmds {
if strings.Contains(c, "orca-traefik.service") && strings.Contains(c, "stop") {
foundLegacyRemoval = true
}
}
if !foundLegacyRemoval {
t.Errorf("expected legacy systemd unit removal command (C-57)\ncommands: %v", cmds)
}
}
// execError is a simple error type for testing.
type execError struct{ msg string }
func (e *execError) Error() string { return e.msg }