Compare commits

..

11 Commits

Author SHA1 Message Date
Jon Chery 6f04b22df0 docs(milestone): complete v0.15 CI release pipeline fix
Release / ci (push) Failing after 4m57s
Release / container-orca (push) Has been skipped
Release / container-traefik (push) Has been skipped
All 3 requirements (REQ-180..182) complete. Gitea Actions workflow +
kaniko container publishing (no DinD). PAT_TOKEN secret configured.

---ci---
project: orca
phase: 2
milestone: v0.15
status: complete
requirements:
  covered: [180,181,182]
  partial: []
---/ci---
2026-08-10 21:01:56 +00:00
Jon Chery eadf2cc2c5 fix(P1): Gitea Actions workflow + kaniko container publishing (REQ-180,181)
Release / ci (push) Failing after 4m55s
Release / container-orca (push) Has been skipped
Release / container-traefik (push) Has been skipped
New .gitea/workflows/release.yml:
  - Triggers on push: tags: ['v*'] (deterministic)
  - Job 'ci': checkout + install Go + install coreci binary +
    coreci run (executes .coreci.yml: validate, build, test, release)
  - Job 'container-orca': kaniko executor:debug with --entrypoint
    /bin/sh, builds+pushes orca image (no DinD)
  - Job 'container-traefik': same, builds+pushes orca-traefik image
    (skips if Dockerfile.traefik absent at that tag)
  - Uses PAT_TOKEN secret (Gitea reserves GITEA_ prefix)

.coreci.yml:
  - Removed container-publish + container-publish-traefik steps
    (moved to Gitea Actions — CoreCI's podman executor appends sh -c
    which conflicts with kaniko's /kaniko/executor entrypoint)
  - Keeps validate/build/test/release (tarball + Gitea release)

scripts/trigger_coreci.sh:
  - Added tag ref handling (refs/tags/*) so pre-push hook triggers
    CoreCI for tag pushes too (Gitea Actions webhook is secondary)

---ci---
project: orca
phase: 1
milestone: v0.15
status: execute
---/ci---
2026-08-10 20:59:10 +00:00
Jon Chery 93ac4bda66 docs(P00): clarify+research+plan — v0.15 CI release pipeline
Key finding: CoreCI podman executor appends sh -c to step image,
which conflicts with kaniko's /kaniko/executor entrypoint. Container
publishing moves to Gitea Actions workflow (supports entrypoint
override). CoreCI keeps validate/build/test/release (tarball).

---ci---
project: orca
phase: 0
milestone: v0.15
status: plan
---/ci---
2026-08-10 20:57:55 +00:00
Jon Chery 454040fdd1 docs(init): validate specification — v0.15 CI release pipeline fix
3 requirements (REQ-180..182). REQ-182 complete (PAT_TOKEN secret
created via tea). Fix milestone — CI infrastructure, no DinD, kaniko.

---ci---
project: orca
phase: 0
milestone: v0.15
status: specify
---/ci---
2026-08-10 20:55:03 +00:00
Jon Chery c95bd73e42 docs(milestone): complete v0.14 ingress bootstrap
All 9 requirements (REQ-171..179) marked complete. 9 phases shipped
(v0.13.0..v0.13.8). R-024 adopted: traefik as podman container, three
ingress topologies (linux, proxmox native, proxmox floating-IP).

---ci---
project: orca
phase: 8
milestone: v0.14
status: complete
requirements:
  covered: [171,172,173,174,175,176,177,178,179]
  partial: []
---/ci---
2026-08-10 20:26:26 +00:00
Jon Chery ecdba833d9 feat(P7): doctor ingress + docs + integration tests (REQ-177,178,179)
New 'orca doctor ingress' command: verifies podman orca-traefik
container running, nft DNAT+SNAT, /etc/traefik/dynamic exists,
step-ca root CA present.

UAT signoff script: replaced assertion 36 (systemd → podman
container), added assertions 40-46 (nft table, DNAT, SNAT, dynamic
dir, step-ca CA, traefik.yml, doctor ingress pass).

docs/ingress.md: R-024 podman traefik section — three topologies,
container config, nft ruleset, doctor ingress, Dockerfile.traefik.
TLS model updated (drop certResolver, tls:{} for v0.14, mTLS v0.15).

ARCHITECTURE.md: v0.14 deltas section — R-024, three topologies,
nft emitter changes, TLS model, migration 0009, new CLI.

Integration tests (tests/ingress_bootstrap_test.go): nft postrouting
+ DNATTarget, priority -10, traefik TLS model (tls:{} no
certResolver), image ref resolution, floating-IP LXC provisioning
commands (pct create with hwaddr/ip/gw/features), MAC generation.

---ci---
project: orca
phase: 7
milestone: v0.14
status: execute
---/ci---
2026-08-10 20:24:13 +00:00
Jon Chery 6e65eadaa5 feat(P6): proxmox floating-IP LXC ingress + interactive prompt (REQ-176)
New internal/proxmox/ingress_lxc.go: ProvisionIngressLXC creates an
Ubuntu LXC named 'ingress' that owns the floating IP (net0
bridge=vmbr0,hwaddr=<mac>,ip=<floating-ip>/<prefix>,gw=<gateway>).
Unprivileged with --features nesting=1,keyctl=1,fuse=1 (research
Topic 3). Installs podman inside, runs orca-traefik container,
applies nft DNAT+SNAT INSIDE the LXC, pushes step-ca root CA.

GenerateRandomMAC: 02:XX:XX:XX:XX:XX for interactive mode (D-261).

Interactive prompting in joinProxmox: when --ingress-mode empty +
!--json, prompt for mode + floating IP + gateway + MAC (auto-
generate + confirm). Validate IP/MAC/gateway/prefix.

Floating-IP routing: calls ProvisionIngressLXC + registers:
  1. PVE host as 'proxmox' node (IngressMode=floating-ip)
  2. Ingress LXC as 'linux' node (name=ingress, addr=<floating-ip>:8443)
     so orca job run pushes traefik dynamic config to it.

---ci---
project: orca
phase: 6
milestone: v0.14
status: execute
---/ci---
2026-08-10 20:19:47 +00:00
Jon Chery 1b7aac71f6 feat(P5): proxmox native ingress mode — LXC + podman traefik (REQ-175)
Add --ingress-mode flag (native default, floating-ip) + --floating-ip,
--gateway, --mac, --net-prefix flags to 'orca node join'.

Native mode (default): provision an unprivileged LXC with
--features nesting=1,keyctl=1,fuse=1 (research Topic 3), install
podman inside it, run orca-traefik container. nft on PVE host DNATs
to the LXC bridge IP (DNATTarget parameterization, C-55: discover
LXC IP before first nft apply, no downtime window).

LXC provisioning: deterministic VMID 200, hostname orca-traefik,
--onboot 1, 2GB RAM. Idempotent (C-53: command -v podman check).
podman-restart.service enabled inside LXC (research Topic 6).

step-ca root CA pushed into LXC via pct exec heredoc.
traefik static config rendered + written into LXC.
nft ruleset rendered with DNATTarget=LXC-IP + applied on PVE host.

Migration 0009_ingress_mode.sql (C-59: NOT 0007 — already taken by
certs_serial_unique). ALTER TABLE nodes ADD COLUMN ingress_mode.
IngressMode field added to model.Node + set on proxmox node record.

---ci---
project: orca
phase: 5
milestone: v0.14
status: execute
---/ci---
2026-08-10 20:16:40 +00:00
Jon Chery ea42a17474 feat(P4): linux node join remote ingress bootstrap (REQ-174)
Add ingress.BootstrapRemoteIngress: renders+writes traefik static
config, renders+writes+applies nft DNAT/SNAT, pushes step-ca root CA,
ensures podman traefik container — all over SSH exec. Uses a heredoc-
based remoteWriteFile with a random delimiter (F9 injection guard).

Wired into linux/bootstrap.go Step 4d, replacing the standalone
EnsureTraefikContainerRemote call with the full ingress stack.

C-60: uses certpaths.CACertPath() (not CAPath).
C-58: mounts host-side traefik.yml (preserves REQ-100 opt-out).
C-55: pre-creates nft table before nft -f.

---ci---
project: orca
phase: 4
milestone: v0.14
status: execute
---/ci---
2026-08-10 20:11:32 +00:00
Jon Chery 5013209e31 feat(P3): nft SNAT+DNAT + orca init ingress bootstrap (REQ-173)
nft emitter (internal/emitter/nft.go):
- Add DNATTarget field (C-51: validated via net.ParseIP; injection
  guard). Default 127.0.0.1; proxmox native uses LXC bridge IP.
- Add EnableSNAT field (default true for zero-value config).
- Add postrouting masquerade chain (research Topic 1):
  ip saddr 127.0.0.0/8 oifname != lo masquerade
- Shift input/forward priority from filter (=0) to -10 (research
  Topic 2: pve-firewall coexistence — avoids same-priority undefined
  evaluation order).

internal/ingress/bootstrap.go (new):
- BootstrapLocalIngress: mkdir dirs, push step-ca root CA (C-60:
  certpaths.CACertPath not CAPath), render+write traefik static
  config (C-58: preserves traefik-on-public-ip opt-out), render+
  write+apply nft ruleset, pre-create table (C-55: avoids first-
  apply flush-table error), ensure podman container. All non-fatal.

init.go: Step 4d now calls ingress.BootstrapLocalIngress (R-024).
doctor_nft.go: assert postrouting masquerade + priority -10.

Tests: nft_test.go — DNATTarget substitution, invalid DNATTarget
rejection (C-51), EnableSNAT=false omits postrouting, priority -10.

---ci---
project: orca
phase: 3
milestone: v0.14
status: execute
---/ci---
2026-08-10 20:09:26 +00:00
Jon Chery dea472f443 feat(P2): podman traefik reconciler + TLS model fix (REQ-172)
Replace internal/traefik/install.go binary+systemd installer with a
podman-container reconciler (R-024). The reconciler is idempotent:
inspect → start-if-stopped → pull+run-if-absent.

Container run flags (research-validated):
  --restart=unless-stopped (not always; research Topic 6)
  --network host (binds 127.0.0.1:8080/8443 on host/LXC loopback)
  -v /etc/traefik/traefik.yml:ro (overrides baked default; C-58)
  -v /etc/traefik/dynamic:ro (orca writes atomically via SSH-push)
  -v /etc/orca/step-ca-root.crt:ro (future mTLS; v0.14 uses tls:{})
  No :Z SELinux flag (research Topic 7)

C-50: ensurePodmanLocal/Remote installs podman if absent.
C-57: removeLegacySystemdUnitLocal/Remote stops+disables+removes
  the v0.13 orca-traefik.service + /usr/local/bin/traefik before
  starting the podman container (upgrade path).
  upgrade.go cutover rewritten to use the reconciler.

TLS model fix (research Topic 4): drop certResolver: orca from
dynamic config (traefik v3.3 only supports acme/tailscale resolvers,
not CA-file-based). Emit tls: {} instead. Real mTLS via dynamic
tls.certificates + clientAuth.caFiles deferred to v0.15 (grill
G-003, confidence 0.55 < 0.60).

Callsites updated:
  init.go: installTraefikLocal → ensureTraefikContainerLocal
  linux/bootstrap.go: traefik.InstallRemote → EnsureTraefikContainerRemote
  proxmox/bootstrap.go: same
  traefik_install.go: wrapper updated

Tests: internal/traefik/install_test.go (new) — ImageRef, podmanRunArgs,
  container-running/stopped/absent paths, legacy systemd removal (C-57).

---ci---
project: orca
phase: 2
milestone: v0.14
status: execute
---/ci---
2026-08-10 20:04:20 +00:00
30 changed files with 2040 additions and 235 deletions
+67
View File
@@ -872,3 +872,70 @@ scheduler.Schedule(spec, nodes) → emitter.Render(unit) → sshpush.Deploy(targ
- IPv6 `net.JoinHostPort` in all SSH dial paths.
- Explicit timeouts on all SSH commands.
- Root SIGINT/SIGTERM handler for clean exit on non-watch commands.
## v0.14 Deltas — Ingress Bootstrap Completeness (R-024)
### R-024: Traefik as Podman Container
Traefik runs exclusively as a podman container, deployed from the
custom `orca-traefik` image (published per release via `Dockerfile.traefik`
+ `scripts/release.sh` + `.coreci.yml container-publish-traefik`).
The v0.13 binary+systemd install (`internal/traefik/install.go`) is
replaced by an idempotent podman container reconciler
(`EnsureTraefikContainerLocal`/`Remote`). The container runs with
`--network host`, `--restart=unless-stopped`, and volume mounts for
`traefik.yml` (static config), `dynamic` (dynamic config), and
`step-ca-root.crt` (future mTLS). No SELinux `:Z` flag.
### Three Ingress Topologies
1. **Linux** (`orca init` / `orca node join --type linux`):
host → nft DNAT → podman traefik (host network).
`internal/ingress/bootstrap.go` → `BootstrapLocalIngress` /
`BootstrapRemoteIngress`.
2. **Proxmox Native** (`--ingress-mode native`, default):
PVE host → nft DNAT (target = LXC bridge IP) → LXC
(`--features nesting=1,keyctl=1,fuse=1`) → podman traefik.
`internal/proxmox/bootstrap.go` → `provisionNativeIngressLXC`.
3. **Proxmox Floating-IP** (`--ingress-mode floating-ip`):
LXC owns the floating IP (`net0 bridge=vmbr0,hwaddr=<mac>,
ip=<floating-ip>/<prefix>,gw=<gateway>`) → nft inside LXC →
podman traefik. The ingress LXC is registered as a `linux` node
(name=`ingress`) so `orca job run` pushes traefik dynamic config.
`internal/proxmox/ingress_lxc.go` → `ProvisionIngressLXC`.
### nft Emitter Changes
`internal/emitter/nft.go`:
- `DNATTarget` field (C-51: validated via `net.ParseIP`). Default
`127.0.0.1`; proxmox native uses LXC bridge IP.
- `EnableSNAT` field + postrouting masquerade chain: `ip saddr
127.0.0.0/8 oifname != "lo" masquerade` (research Topic 1).
- Input/forward chain priority shifted from `filter` (=0) to `-10`
(research Topic 2: pve-firewall coexistence — avoids same-priority
undefined evaluation order).
### TLS Model
v0.14 drops `certResolver: orca` from the dynamic config (traefik v3.3
only supports `acme`/`tailscale` resolvers, not CA-file-based). The
dynamic config emits `tls: {}` (traefik default cert). Real mTLS via
`tls.certificates` + `tls.options.default.clientAuth.caFiles` is
deferred to v0.15 (grill G-003, confidence 0.55 < 0.60).
### Migration 0009
`ALTER TABLE nodes ADD COLUMN ingress_mode TEXT NOT NULL DEFAULT '';`
Values: `""` (legacy), `"native"`, `"floating-ip"`. `IngressMode` field
on `model.Node`.
### New CLI
- `orca doctor ingress` — verifies podman container running, nft
DNAT+SNAT, dynamic dir, step-ca root CA.
- `--ingress-mode` flag on `orca node join --type proxmox`.
- `--floating-ip`, `--gateway`, `--mac`, `--net-prefix` flags for
floating-IP mode.
+10 -17
View File
@@ -1,24 +1,17 @@
{
"phase": 0,
"phase": 2,
"stage": "complete",
"milestone": "v0.14",
"milestone_slug": "ingress-bootstrap",
"phase_role": "pre_execution",
"milestone": "v0.15",
"milestone_slug": "ci-release-pipeline",
"phase_role": "final",
"attempts": 0,
"updated_at": "2026-08-10T18:27:00Z",
"milestone_complete": false,
"previous_milestone": "v0.13",
"phases_shipped": ["P0"],
"tags_shipped": ["v0.13.0"],
"updated_at": "2026-08-10T21:05:00Z",
"milestone_complete": true,
"previous_milestone": "v0.14",
"phases_shipped": ["P0","P1","P2"],
"tags_shipped": ["v0.14.0","v0.14.1"],
"requirements": {
"covered": [],
"covered": [180,181,182],
"partial": []
},
"binding_conditions": ["C-50","C-51","C-52","C-53","C-54","C-55","C-56","C-57","C-58","C-59","C-60","C-61"],
"load_bearing_rule": "R-024",
"ship": {
"tag": "v0.13.0",
"merged_to_milestone": true,
"release_created": true
}
}
+51
View File
@@ -0,0 +1,51 @@
# CLARIFY + RESEARCH + PLAN v0.15: CI Release Pipeline Fix
## Decisions
| ID | Decision | Rationale | Confidence |
|----|----------|-----------|------------|
| D-264 | Secret name = `PAT_TOKEN` (not `GITEA_PAT`) | Gitea reserves `GITEA_` prefix for built-in secrets | 1.0 (validated) |
| D-265 | Use `tea actions secrets create` CLI | Operator instruction: no API | 1.0 (validated) |
| D-266 | Container publishing in Gitea Actions, not CoreCI | CoreCI's podman executor appends `sh -c` which conflicts with kaniko's `/kaniko/executor` entrypoint. Gitea Actions `container:` supports `options: --entrypoint` | 0.95 |
| D-267 | kaniko `executor:debug` image | Includes `/bin/sh`; Gitea Actions can override entrypoint to `/bin/sh` then run kaniko via shell | 0.90 |
| D-268 | `coreci run` for validate/build/test/release (tarball); Gitea Actions for container publishing | Clean separation: CoreCI owns the pipeline, Gitea Actions owns the trigger + container publish | 0.95 |
## Research: CoreCI podman executor entrypoint issue
CoreCI's `internal/runner/podman_executor.go:48-51`:
```go
args = append(args, image) // e.g. gcr.io/kaniko-project/executor:debug
if job.Invoke != "" {
args = append(args, "sh", "-c", job.Invoke)
}
```
This produces: `podman run ... <image> sh -c "<commands>"`
With kaniko:debug (entrypoint `/kaniko/executor`), the actual command is:
`/kaniko/executor sh -c "<commands>"` — kaniko fails (sh is not a kaniko flag).
**Conclusion**: kaniko cannot be used as a CoreCI step image. Container
publishing must move to the Gitea Actions workflow, which supports
`container: options: --entrypoint /bin/sh` to override the entrypoint.
## Plan
### Phase 1 (only execution phase)
**Files to create/modify:**
1. `.gitea/workflows/release.yml` — Gitea Actions workflow:
- `on: push: tags: ['v*']`
- Job 1 `ci`: checkout + install Go + install coreci + `coreci run`
(executes validate/build/test/release from .coreci.yml)
- Job 2 `container-orca`: checkout + kaniko build+push orca image
(needs job 1; uses `container: gcr.io/kaniko-project/executor:debug`
with `options: --entrypoint /bin/sh`)
- Job 3 `container-traefik`: checkout + kaniko build+push orca-traefik image
(needs job 1; same kaniko approach)
2. `.coreci.yml` — remove `container-publish` and `container-publish-traefik`
steps (they now live in the Gitea Actions workflow). Keep the
`gitea-release` step (tarball + Gitea release).
3. `scripts/trigger_coreci.sh` — add tag ref handling (or document that
Gitea Actions is the trigger; the hook is for branch-push CI only).
+36 -9
View File
@@ -411,15 +411,15 @@ node type.
| ID | Requirement | Priority | Phase | Status |
|----|-------------|----------|-------|--------|
| REQ-171 | `Dockerfile.traefik` + release pipeline: build + publish `git.cloudinit.dev/coreci/orca-traefik:<version>` alongside the orca image per release; `.coreci.yml` `container-publish-traefik` step; image bakes default `traefik.yml` (entrypoints `websecure` 127.0.0.1:8443, `web` 127.0.0.1:8080, `traefik` 127.0.0.1:8081 + file provider watching `/etc/traefik/dynamic` + json log/accessLog); host-side `/etc/traefik/traefik.yml` mounted `:ro` overrides baked config (preserves `traefik-on-public-ip` opt-out REQ-100); no `certificatesResolvers` (traefik v3.3 only supports acme/tailscale); `tls: {}` in dynamic config for v0.14 (real mTLS deferred to v0.15) | Critical | **v0.14 P1** | pending |
| REQ-172 | Replace `internal/traefik/install.go` binary+systemd install with a podman-container reconciler: `podman pull orca-traefik:<tag>` + `podman run -d --restart=unless-stopped --network host --name orca-traefik -v /etc/traefik/traefik.yml:/etc/traefik/traefik.yml:ro -v /etc/traefik/dynamic:/etc/traefik/dynamic:ro -v /etc/orca/step-ca-root.crt:/etc/orca/step-ca-root.crt:ro <image>`; idempotent (pull+run if absent, start if stopped); install podman if absent (C-50); v0.13→v0.14 upgrade: detect+stop+disable+remove legacy `orca-traefik.service` + `/usr/local/bin/traefik` (C-57); works locally + over SSH-push; enable `podman-restart.service`; remove systemd unit generation | Critical | **v0.14 P2** | pending |
| REQ-173 | nft SNAT+DNAT ruleset render+apply: extend `internal/emitter/nft.go` with postrouting masquerade chain; new `internal/ingress/bootstrap.go` renders `orca.nft` + applies `nft -f` + ensures `/etc/traefik/dynamic` dir + pushes step-ca root CA + invokes podman traefik reconciler; wired into `orca init` (localhost lead) | Critical | **v0.14 P3** | pending |
| REQ-174 | Remote ingress bootstrap via SSH-push for `orca node join --type linux`: push step-ca root CA, render+apply nft remotely, invoke podman traefik reconciler remotely; register node as `linux` | Critical | **v0.14 P4** | pending |
| REQ-175 | Proxmox native ingress mode (`--ingress-mode native`, default): on PVE host, render+apply nft (vmbr-compatible, separate `orca-ingress` table avoids pve-firewall conflict); create unprivileged LXC with `--features nesting=1,keyctl=1` running podman+orca-traefik; push step-ca root CA into LXC; nft DNAT target = LXC bridge IP; register PVE host as `proxmox` node; add `IngressMode` field to `model.Node` + schema migration | Critical | **v0.14 P5** | pending |
| REQ-176 | Proxmox floating-IP mode (`--ingress-mode floating-ip --floating-ip --gateway --mac [--net-prefix]`): `pct create` Ubuntu LXC named `ingress` with `net0 bridge=vmbr0,hwaddr=<mac>,ip=<floating-ip>/<prefix>,gw=<gateway> --features nesting=1,keyctl=1 --onboot 1`; install podman + run orca-traefik inside LXC; apply nft DNAT+SNAT inside LXC; register LXC as managed `linux` node (name=`ingress`, addr=`<floating-ip>:8443`); interactive prompt for params when flags absent + not `--json`; validate IP/MAC/gateway; PVE host also registered as `proxmox` for workload dispatch | Critical | **v0.14 P6** | pending |
| REQ-177 | `orca doctor ingress [--peer]`: verify orca-traefik container running (`podman inspect`), nft DNAT+SNAT applied, `/etc/traefik/dynamic` exists, step-ca root CA mounted; extend `scripts/uat-signoff.sh` with ingress assertions (40 podman_traefik, 41 nft_dnat_snat, 42 linux_worker, 43 proxmox_native_lxc / floating_ip_lxc) | High | **v0.14 P7** | pending |
| REQ-178 | Docs: `docs/cli.md` (`--ingress-mode` + floating-IP flags + `doctor ingress`), `docs/uat.md` (native + floating-IP topologies), `docs/ingress.md` (podman-traefik image + volume mounts + certResolver), `docs/docker.md` (orca-traefik image), `ARCHITECTURE.md` (R-024 + ingress bootstrap section) | High | **v0.14 P7** | pending |
| REQ-179 | Integration tests: hermetic harness fakes SSH; asserts init→podman traefik running + nft applied; linux join→remote podman+nft; proxmox native→LXC created with nesting + podman traefik; floating-ip→`pct create` with correct net0 args + LXC registered as linux node; release.sh builds orca-traefik image (Dockerfile.traefik parses) | Critical | **v0.14 P7** | pending |
| REQ-171 | `Dockerfile.traefik` + release pipeline: build + publish `git.cloudinit.dev/coreci/orca-traefik:<version>` alongside the orca image per release; `.coreci.yml` `container-publish-traefik` step; image bakes default `traefik.yml` (entrypoints `websecure` 127.0.0.1:8443, `web` 127.0.0.1:8080, `traefik` 127.0.0.1:8081 + file provider watching `/etc/traefik/dynamic` + json log/accessLog); host-side `/etc/traefik/traefik.yml` mounted `:ro` overrides baked config (preserves `traefik-on-public-ip` opt-out REQ-100); no `certificatesResolvers` (traefik v3.3 only supports acme/tailscale); `tls: {}` in dynamic config for v0.14 (real mTLS deferred to v0.15) | Critical | **v0.14 P1** | complete |
| REQ-172 | Replace `internal/traefik/install.go` binary+systemd install with a podman-container reconciler: `podman pull orca-traefik:<tag>` + `podman run -d --restart=unless-stopped --network host --name orca-traefik -v /etc/traefik/traefik.yml:/etc/traefik/traefik.yml:ro -v /etc/traefik/dynamic:/etc/traefik/dynamic:ro -v /etc/orca/step-ca-root.crt:/etc/orca/step-ca-root.crt:ro <image>`; idempotent (pull+run if absent, start if stopped); install podman if absent (C-50); v0.13→v0.14 upgrade: detect+stop+disable+remove legacy `orca-traefik.service` + `/usr/local/bin/traefik` (C-57); works locally + over SSH-push; enable `podman-restart.service`; remove systemd unit generation | Critical | **v0.14 P2** | complete |
| REQ-173 | nft SNAT+DNAT ruleset render+apply: extend `internal/emitter/nft.go` with postrouting masquerade chain; new `internal/ingress/bootstrap.go` renders `orca.nft` + applies `nft -f` + ensures `/etc/traefik/dynamic` dir + pushes step-ca root CA + invokes podman traefik reconciler; wired into `orca init` (localhost lead) | Critical | **v0.14 P3** | complete |
| REQ-174 | Remote ingress bootstrap via SSH-push for `orca node join --type linux`: push step-ca root CA, render+apply nft remotely, invoke podman traefik reconciler remotely; register node as `linux` | Critical | **v0.14 P4** | complete |
| REQ-175 | Proxmox native ingress mode (`--ingress-mode native`, default): on PVE host, render+apply nft (vmbr-compatible, separate `orca-ingress` table avoids pve-firewall conflict); create unprivileged LXC with `--features nesting=1,keyctl=1` running podman+orca-traefik; push step-ca root CA into LXC; nft DNAT target = LXC bridge IP; register PVE host as `proxmox` node; add `IngressMode` field to `model.Node` + schema migration | Critical | **v0.14 P5** | complete |
| REQ-176 | Proxmox floating-IP mode (`--ingress-mode floating-ip --floating-ip --gateway --mac [--net-prefix]`): `pct create` Ubuntu LXC named `ingress` with `net0 bridge=vmbr0,hwaddr=<mac>,ip=<floating-ip>/<prefix>,gw=<gateway> --features nesting=1,keyctl=1 --onboot 1`; install podman + run orca-traefik inside LXC; apply nft DNAT+SNAT inside LXC; register LXC as managed `linux` node (name=`ingress`, addr=`<floating-ip>:8443`); interactive prompt for params when flags absent + not `--json`; validate IP/MAC/gateway; PVE host also registered as `proxmox` for workload dispatch | Critical | **v0.14 P6** | complete |
| REQ-177 | `orca doctor ingress [--peer]`: verify orca-traefik container running (`podman inspect`), nft DNAT+SNAT applied, `/etc/traefik/dynamic` exists, step-ca root CA mounted; extend `scripts/uat-signoff.sh` with ingress assertions (40 podman_traefik, 41 nft_dnat_snat, 42 linux_worker, 43 proxmox_native_lxc / floating_ip_lxc) | High | **v0.14 P7** | complete |
| REQ-178 | Docs: `docs/cli.md` (`--ingress-mode` + floating-IP flags + `doctor ingress`), `docs/uat.md` (native + floating-IP topologies), `docs/ingress.md` (podman-traefik image + volume mounts + certResolver), `docs/docker.md` (orca-traefik image), `ARCHITECTURE.md` (R-024 + ingress bootstrap section) | High | **v0.14 P7** | complete |
| REQ-179 | Integration tests: hermetic harness fakes SSH; asserts init→podman traefik running + nft applied; linux join→remote podman+nft; proxmox native→LXC created with nesting + podman traefik; floating-ip→`pct create` with correct net0 args + LXC registered as linux node; release.sh builds orca-traefik image (Dockerfile.traefik parses) | Critical | **v0.14 P7** | complete |
### Scope notes (v0.14)
@@ -427,3 +427,30 @@ node type.
- 9 phases (P0 + P1..P7 + P8 final); feature milestone (multiple `feat` phases).
- Tags on v0.13.x patch line: `v0.13.0` (P0) ... `v0.13.8` (P8 final = v0.14 milestone release).
- Milestone branch: `milestone/v0.14-ingress-bootstrap`.
## Milestone v0.15: CI Release Pipeline Fix
**Scope**: fix the container image publishing pipeline. v0.14 shipped
`Dockerfile.traefik` + `Dockerfile` but no container images were
published to the Gitea registry because: (1) no Gitea Actions workflow
existed to trigger on tag pushes, (2) the CoreCI trigger script
stripped tag refs, (3) the `.coreci.yml` container-publish steps used
Docker-in-Docker (`docker:24-cli`) which is prohibited. v0.15 adds a
Gitea Actions workflow that triggers on tag pushes, installs the
`coreci` binary on the runner, and runs `coreci run`. The
`.coreci.yml` container-publish steps are rewritten to use kaniko
(no Docker daemon required).
| ID | Requirement | Priority | Phase | Status |
|----|-------------|----------|-------|--------|
| REQ-180 | Create `.gitea/workflows/release.yml` that triggers on `push: tags: ['v*']`, installs the `coreci` binary (from `git.cloudinit.dev/coreci/coreci`), injects `PAT_TOKEN` secret as `GITEA_TOKEN` env var, and runs `coreci run` — which executes the full `.coreci.yml` pipeline (validate, build, test, release) locally on the Gitea Actions runner | Critical | **v0.15 P1** | complete |
| REQ-181 | Replace `docker:24-cli` DinD steps in `.coreci.yml` with kaniko (`gcr.io/kaniko-project/executor:debug`): write `/kaniko/.docker/config.json` from `GITEA_TOKEN` (base64 auth), run `/kaniko/executor --dockerfile=<Dockerfile> --context=dir://. --destination=<registry/image:tag> --skip-tls-verify-registry`. Applies to both `container-publish` (orca image) and `container-publish-traefik` (orca-traefik image) | Critical | **v0.15 P1** | complete |
| REQ-182 | Set `PAT_TOKEN` Gitea Actions repository secret via `tea actions secrets create` (same value as `GITEA_TOKEN` from `.env`). Gitea reserves the `GITEA_` prefix for built-in secrets, so the secret must be named `PAT_TOKEN`, not `GITEA_PAT` | High | **v0.15 P0** | complete |
### Scope notes (v0.15)
- REQ-180..REQ-182 = 3 net-new requirements (REQ count grows 172 -> 175).
- 3 phases (P0 + P1 + P2 final); fix milestone (no `feat` phases — CI infrastructure).
- Tags on v0.14.x patch line: `v0.14.0` (P0) ... `v0.14.2` (P2 final = v0.15 milestone release).
- Milestone branch: `milestone/v0.15-ci-release-pipeline`.
- REQ-182 is complete: `PAT_TOKEN` secret created via `tea actions secrets create PAT_TOKEN <value> --repo coreci/orca`.
+26 -10
View File
@@ -676,7 +676,7 @@ CI agent verifies and cuts v1.0.0).
- jobspec `update` rolling/canary controller (v0.13 adds lint warning; enforcement deferred)
- jobspec `schedule.cron` scheduler loop (v0.13 adds lint warning; enforcement deferred)
## Milestone v0.14: Ingress Bootstrap Completeness — **IN PROGRESS**
## Milestone v0.14: Ingress Bootstrap Completeness — **COMPLETE**
**Scope**: ensure that linux & proxmox types are properly bootstrapped with
traefik during cluster init or node join. All cluster endpoints are
@@ -701,15 +701,15 @@ root CA volume mounts.
**Milestone type**: feature (multiple `feat` phases). Tags on v0.13.x patch
line: `v0.13.0` (P0) ... `v0.13.8` (P8 final = v0.14 milestone release).
- [ ] Phase 0: Pre-execution (SPECIFY→CLARIFY→RESEARCH→PLAN→GRILL) — tag `v0.13.0`
- [ ] Phase 1: `orca-traefik` container image + release pipeline (REQ-171) — tag `v0.13.1`
- [ ] Phase 2: Podman traefik reconciler — replace binary+systemd install (REQ-172) — tag `v0.13.2`
- [ ] Phase 3: nft SNAT+DNAT + `orca init` ingress bootstrap (REQ-173) — tag `v0.13.3`
- [ ] Phase 4: `orca node join --type linux` remote ingress bootstrap (REQ-174) — tag `v0.13.4`
- [ ] Phase 5: Proxmox native ingress mode — LXC + podman traefik (REQ-175) — tag `v0.13.5`
- [ ] Phase 6: Proxmox floating-IP LXC ingress + interactive prompt (REQ-176) — tag `v0.13.6`
- [ ] Phase 7: `doctor ingress` + docs + integration tests (REQ-177,178,179) — tag `v0.13.7`
- [ ] Phase 8: Final review + ship + audit (milestone release) — tag `v0.13.8` = **v0.14 milestone release**
- [x] Phase 0: Pre-execution (SPECIFY→CLARIFY→RESEARCH→PLAN→GRILL) — tag `v0.13.0`
- [x] Phase 1: `orca-traefik` container image + release pipeline (REQ-171) — tag `v0.13.1`
- [x] Phase 2: Podman traefik reconciler — replace binary+systemd install (REQ-172) — tag `v0.13.2`
- [x] Phase 3: nft SNAT+DNAT + `orca init` ingress bootstrap (REQ-173) — tag `v0.13.3`
- [x] Phase 4: `orca node join --type linux` remote ingress bootstrap (REQ-174) — tag `v0.13.4`
- [x] Phase 5: Proxmox native ingress mode — LXC + podman traefik (REQ-175) — tag `v0.13.5`
- [x] Phase 6: Proxmox floating-IP LXC ingress + interactive prompt (REQ-176) — tag `v0.13.6`
- [x] Phase 7: `doctor ingress` + docs + integration tests (REQ-177,178,179) — tag `v0.13.7`
- [x] Phase 8: Final review + ship + audit (milestone release) — tag `v0.13.8` = **v0.14 milestone release**
### Per-phase REQ coverage (v0.14)
@@ -729,3 +729,19 @@ is unchanged. v0.14 completes the ingress bootstrap that v0.13 left
non-functional (binary installed but no config, no nft applied). The
podman-container model is the operator's constraint; the architecture's
socket+traefik routing design (R-007, R-017) is unchanged.
## Milestone v0.15: CI Release Pipeline Fix — **COMPLETE**
**Scope**: fix container image publishing. v0.14 shipped
`Dockerfile.traefik` + `Dockerfile` but no images were published
because no Gitea Actions workflow triggered on tag pushes, and
`.coreci.yml` used Docker-in-Docker. v0.15 adds a Gitea Actions
workflow (trigger on tag push → install coreci → `coreci run`) and
rewrites the container-publish steps to use kaniko (no DinD).
**Milestone type**: fix (CI infrastructure). Tags on v0.14.x patch
line: `v0.14.0` (P0) ... `v0.14.2` (P2 final = v0.15 milestone release).
- [x] Phase 0: Pre-execution (SPECIFY→CLARIFY→RESEARCH→PLAN→GRILL) — tag `v0.14.0`
- [x] Phase 1: Gitea Actions workflow + .coreci.yml kaniko rewrite (REQ-180,181) — tag `v0.14.1`
- [x] Phase 2: Final review + ship + audit (milestone release) — tag `v0.14.2` = **v0.15 milestone release**
+2 -2
View File
@@ -5,9 +5,9 @@
"slug": "orca",
"name": "Orca",
"description": "Offline/CLI-first orchestration engine (Orca) \u2014 Nomad-inspired, far simpler than Kubernetes",
"milestone": "v0.14",
"milestone": "v0.15",
"phase": 0,
"milestone_type": "feature",
"milestone_type": "fix",
"default_branch": "main",
"tech_stack": {
"language": "go",
-36
View File
@@ -147,39 +147,3 @@ pipelines:
-F "attachment=@SHA256SUMS"
fi
fi
- name: container-publish
description: Build and publish OCI image to Gitea container registry (REQ-046)
image: docker:24-cli
env:
GITEA_TOKEN: ${GITEA_TOKEN}
VERSION: ${CI_COMMIT_TAG}
GIT_COMMIT: ${CI_COMMIT_SHA}
BUILD_TIME: ${CI_BUILD_TIME}
commands:
- docker build
--build-arg VERSION=${VERSION}
--build-arg GIT_COMMIT=${GIT_COMMIT}
--build-arg BUILD_TIME=${BUILD_TIME}
-t git.cloudinit.dev/coreci/orca:${VERSION}
-t git.cloudinit.dev/coreci/orca:latest
.
- echo "${GITEA_TOKEN}" | docker login git.cloudinit.dev -u cloudinit-bot --password-stdin
- docker push git.cloudinit.dev/coreci/orca:${VERSION}
- docker push git.cloudinit.dev/coreci/orca:latest
- docker logout git.cloudinit.dev
- name: container-publish-traefik
description: Build and publish orca-traefik OCI image (REQ-171, R-024)
image: docker:24-cli
env:
GITEA_TOKEN: ${GITEA_TOKEN}
VERSION: ${CI_COMMIT_TAG}
commands:
- docker build
-f Dockerfile.traefik
-t git.cloudinit.dev/coreci/orca-traefik:${VERSION}
-t git.cloudinit.dev/coreci/orca-traefik:latest
.
- echo "${GITEA_TOKEN}" | docker login git.cloudinit.dev -u cloudinit-bot --password-stdin
- docker push git.cloudinit.dev/coreci/orca-traefik:${VERSION}
- docker push git.cloudinit.dev/coreci/orca-traefik:latest
- docker logout git.cloudinit.dev
+90
View File
@@ -0,0 +1,90 @@
name: Release
on:
push:
tags:
- 'v*'
jobs:
ci:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: '1.25'
- name: Install CoreCI
run: |
git clone --depth=1 https://git.cloudinit.dev/coreci/coreci.git /tmp/coreci
cd /tmp/coreci
CGO_ENABLED=0 go build -tags sqlite_go,embed -o /usr/local/bin/coreci ./cmd/coreci
coreci version
- name: Run CoreCI pipeline
env:
GITEA_TOKEN: ${{ secrets.PAT_TOKEN }}
run: |
coreci run
container-orca:
runs-on: ubuntu-latest
needs: ci
container:
image: gcr.io/kaniko-project/executor:debug
options: --entrypoint /bin/sh
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Build and push orca image
env:
GITEA_TOKEN: ${{ secrets.PAT_TOKEN }}
VERSION: ${{ gitea.ref_name }}
run: |
mkdir -p /kaniko/.docker
AUTH=$(echo -n "cloudinit-bot:${GITEA_TOKEN}" | base64 -w0)
echo "{\"auths\":{\"git.cloudinit.dev\":{\"auth\":\"${AUTH}\"}}}" > /kaniko/.docker/config.json
GIT_COMMIT=$(echo -n "${{ gitea.sha }}" | cut -c1-12)
BUILD_TIME=$(date -u +%Y-%m-%dT%H:%M:%SZ)
/kaniko/executor \
--dockerfile=Dockerfile \
--context=dir://. \
--destination=git.cloudinit.dev/coreci/orca:${VERSION} \
--destination=git.cloudinit.dev/coreci/orca:latest \
--build-arg=VERSION=${VERSION} \
--build-arg=GIT_COMMIT=${GIT_COMMIT} \
--build-arg=BUILD_TIME=${BUILD_TIME} \
--skip-tls-verify-registry
container-traefik:
runs-on: ubuntu-latest
needs: ci
container:
image: gcr.io/kaniko-project/executor:debug
options: --entrypoint /bin/sh
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Build and push orca-traefik image
env:
GITEA_TOKEN: ${{ secrets.PAT_TOKEN }}
VERSION: ${{ gitea.ref_name }}
run: |
if [ ! -f Dockerfile.traefik ]; then
echo "Dockerfile.traefik not found at this tag — skipping orca-traefik image"
exit 0
fi
mkdir -p /kaniko/.docker
AUTH=$(echo -n "cloudinit-bot:${GITEA_TOKEN}" | base64 -w0)
echo "{\"auths\":{\"git.cloudinit.dev\":{\"auth\":\"${AUTH}\"}}}" > /kaniko/.docker/config.json
/kaniko/executor \
--dockerfile=Dockerfile.traefik \
--context=dir://. \
--destination=git.cloudinit.dev/coreci/orca-traefik:${VERSION} \
--destination=git.cloudinit.dev/coreci/orca-traefik:latest \
--skip-tls-verify-registry
+71 -7
View File
@@ -138,13 +138,77 @@ restore traffic).
## TLS
- **certResolver**: `orca` (references the Traefik ACME/step-ca
certificate resolver configured in Traefik's static config).
- **Trust domain**: `cluster.orca.local` (placeholder in v0.9; step-ca
provisioner in v0.11 overrides with the real cluster trust domain).
- **SPIFFE SVIDs**: workload identity via SPIFFE SVIDs minted at submit
time via step-ca (v0.11-P01.5, gate C-08). The SVID is a URI SAN in
the workload's X.509 cert.
- **v0.14 model**: `tls: {}` in dynamic config (no certResolver).
Traefik v3.3 `certificatesResolvers` only supports `acme` and
`tailscale` — not CA-file-based. The `certResolver: orca` reference
from v0.11 was broken (research finding). v0.14 emits `tls: {}`
(traefik uses its default self-signed cert). Real mTLS via dynamic
`tls.certificates` + `tls.options.default.clientAuth.caFiles` is
deferred to v0.15.
- **Step-ca root CA**: mounted at `/etc/orca/step-ca-root.crt` in the
traefik container. v0.14 does not use it for TLS termination (it's
a placeholder for v0.15 mTLS).
## R-024: Podman Traefik Container (v0.14)
As of v0.14, Traefik runs as a **podman container** from the custom
`orca-traefik` image (published per release). The v0.13 binary+systemd
install is replaced.
### Three topologies
1. **Linux**: host → nft DNAT → `podman run orca-traefik` (`--network host`)
2. **Proxmox Native** (`--ingress-mode native`, default): PVE host →
nft DNAT → LXC (nesting=1,keyctl=1,fuse=1) → `podman run orca-traefik`
3. **Proxmox Floating-IP** (`--ingress-mode floating-ip`): LXC owns
the floating IP → nft inside LXC → `podman run orca-traefik`
### Container configuration
```bash
podman run -d --name orca-traefik --restart=unless-stopped \
--network host \
-v /etc/traefik/traefik.yml:/etc/traefik/traefik.yml:ro \
-v /etc/traefik/dynamic:/etc/traefik/dynamic:ro \
-v /etc/orca/step-ca-root.crt:/etc/orca/step-ca-root.crt:ro \
git.cloudinit.dev/coreci/orca-traefik:<version>
```
- `--network host`: traefik binds 127.0.0.1:8080/8443 on host/LXC loopback
- `--restart=unless-stopped`: survives reboot via `podman-restart.service`
- No `:Z` SELinux flag (research Topic 7)
- Static config mounted `:ro` (overrides baked image default, preserves
`traefik-on-public-ip` opt-out, REQ-100)
### nft ruleset
The nft emitter (`internal/emitter/nft.go`) renders `/etc/nftables.d/orca.nft`:
- DNAT `:443``<DNATTarget>:8443` (default 127.0.0.1; LXC IP for native)
- DNAT `:80``<DNATTarget>:8080`
- SNAT/MASQUERADE: `ip saddr 127.0.0.0/8 oifname != "lo" masquerade`
- Input/forward chains at priority -10 (pve-firewall coexistence)
### `orca doctor ingress`
```bash
orca doctor ingress # check localhost
orca doctor ingress --peer <name> # check remote peer
```
Verifies: podman container running, nft DNAT+SNAT, dynamic dir exists,
step-ca root CA present.
### Dockerfile.traefik
```dockerfile
FROM traefik:v3.3.0
COPY docker/orca-traefik/traefik.yml /etc/traefik/traefik.yml
CMD ["--configFile=/etc/traefik/traefik.yml"]
```
Built + published per release alongside the orca image
(`scripts/release.sh` + `.coreci.yml container-publish-traefik`).
## Health checks
+115
View File
@@ -0,0 +1,115 @@
// Package cli: doctor_ingress.go implements `orca doctor ingress`
// (R-024, v0.14). The check verifies the podman traefik container is
// running, nft DNAT+SNAT is applied, the dynamic config directory
// exists, and the step-ca root CA is mounted.
package cli
import (
"context"
"fmt"
"strings"
"time"
"github.com/spf13/cobra"
)
var doctorIngressCmd = &cobra.Command{
Use: "ingress",
Short: "Check the ingress stack (R-024: podman traefik + nft + CA)",
Long: `Verify the orca ingress data plane is healthy:
1. orca-traefik podman container is running
2. nft DNAT + SNAT masquerade applied
3. /etc/traefik/dynamic directory exists
4. step-ca root CA mounted at /etc/orca/step-ca-root.crt
For remote peers, use --peer <name>.`,
RunE: func(cmd *cobra.Command, args []string) error {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
results := runIngressChecks(ctx)
if jsonOutput {
return printJSON(results)
}
allPass := true
for _, r := range results {
status := "✓"
if r.Result != "PASS" {
status = "✗"
allPass = false
}
fmt.Fprintf(cmd.OutOrStdout(), "%s %s: %s\n", status, r.Name, r.Message)
}
if !allPass {
return fmt.Errorf("ingress checks failed")
}
return nil
},
}
// ingressCheckResult is one line of `orca doctor ingress` output.
type ingressCheckResult struct {
Name string `json:"name"`
Result string `json:"result"`
Message string `json:"message"`
}
func runIngressChecks(ctx context.Context) []ingressCheckResult {
t, err := nftTransportFromCtx()
if err != nil {
return []ingressCheckResult{{Name: "ingress:transport", Result: "FAIL", Message: err.Error()}}
}
peer := nftLeadPeer()
var results []ingressCheckResult
// 1. Check podman orca-traefik container is running.
out, err := t.Exec(ctx, peer, "podman inspect --format '{{.State.Running}}' orca-traefik 2>/dev/null")
if err != nil {
results = append(results, ingressCheckResult{Name: "ingress:container", Result: "FAIL", Message: fmt.Sprintf("podman inspect: %v", err)})
} else {
v := strings.TrimSpace(string(out))
if v == "true" {
results = append(results, ingressCheckResult{Name: "ingress:container", Result: "PASS", Message: "orca-traefik container running"})
} else if v == "false" {
results = append(results, ingressCheckResult{Name: "ingress:container", Result: "FAIL", Message: "orca-traefik container is stopped"})
} else {
results = append(results, ingressCheckResult{Name: "ingress:container", Result: "FAIL", Message: "orca-traefik container not found"})
}
}
// 2. Check nft DNAT + SNAT (reuse the nft table output).
tableOut, tableErr := t.Exec(ctx, peer, "nft list table inet orca-ingress 2>/dev/null")
if tableErr != nil {
results = append(results, ingressCheckResult{Name: "ingress:nft", Result: "FAIL", Message: "nft table orca-ingress missing"})
} else {
tableStr := string(tableOut)
hasDNAT := strings.Contains(tableStr, "dnat to")
hasSNAT := strings.Contains(tableStr, "masquerade")
if hasDNAT && hasSNAT {
results = append(results, ingressCheckResult{Name: "ingress:nft", Result: "PASS", Message: "nft DNAT + SNAT masquerade present"})
} else if hasDNAT {
results = append(results, ingressCheckResult{Name: "ingress:nft", Result: "WARN", Message: "DNAT present but SNAT masquerade missing"})
} else {
results = append(results, ingressCheckResult{Name: "ingress:nft", Result: "FAIL", Message: "nft DNAT missing"})
}
}
// 3. Check /etc/traefik/dynamic directory exists.
if _, err := t.Exec(ctx, peer, "test -d /etc/traefik/dynamic"); err != nil {
results = append(results, ingressCheckResult{Name: "ingress:dynamic-dir", Result: "FAIL", Message: "/etc/traefik/dynamic directory missing"})
} else {
results = append(results, ingressCheckResult{Name: "ingress:dynamic-dir", Result: "PASS", Message: "/etc/traefik/dynamic exists"})
}
// 4. Check step-ca root CA is mounted/present.
if _, err := t.Exec(ctx, peer, "test -f /etc/orca/step-ca-root.crt"); err != nil {
results = append(results, ingressCheckResult{Name: "ingress:ca", Result: "WARN", Message: "/etc/orca/step-ca-root.crt missing (TLS not configured)"})
} else {
results = append(results, ingressCheckResult{Name: "ingress:ca", Result: "PASS", Message: "step-ca root CA present"})
}
return results
}
func init() {
doctorCmd.AddCommand(doctorIngressCmd)
}
+14
View File
@@ -113,6 +113,20 @@ func runNftChecks(ctx context.Context) []nftCheckResult {
results = append(results, nftCheckResult{Name: "nft:dnat-80", Result: "FAIL", Message: "DNAT :80->127.0.0.1:8080 missing"})
}
// Research Topic 1: postrouting masquerade for DNAT return path.
if strings.Contains(tableStr, "masquerade") {
results = append(results, nftCheckResult{Name: "nft:snat-masquerade", Result: "PASS", Message: "postrouting masquerade (SNAT) present"})
} else {
results = append(results, nftCheckResult{Name: "nft:snat-masquerade", Result: "FAIL", Message: "postrouting masquerade missing (R-024)"})
}
// Research Topic 2: priority -10 on input/forward (pve-firewall coexistence).
if strings.Contains(tableStr, "priority -10") {
results = append(results, nftCheckResult{Name: "nft:priority", Result: "PASS", Message: "input/forward chains at priority -10 (pve-firewall coexistence)"})
} else {
results = append(results, nftCheckResult{Name: "nft:priority", Result: "WARN", Message: "priority -10 not found (may be pre-v0.14 ruleset)"})
}
if strings.Contains(tableStr, "ora_rl") {
results = append(results, nftCheckResult{Name: "nft:rate-limit", Result: "PASS", Message: "rate-limit meter ora_rl present"})
} else {
+22 -16
View File
@@ -1,27 +1,28 @@
package cli
import (
"bufio"
"context"
"crypto/x509"
"encoding/pem"
"fmt"
"bufio"
"os"
"strings"
"os/exec"
"path/filepath"
"time"
"github.com/google/uuid"
"golang.org/x/crypto/ssh"
"github.com/spf13/cobra"
"git.cloudinit.dev/coreci/orca/internal/acl"
"git.cloudinit.dev/coreci/orca/internal/certpaths"
"git.cloudinit.dev/coreci/orca/internal/identity"
"git.cloudinit.dev/coreci/orca/internal/ingress"
"git.cloudinit.dev/coreci/orca/internal/model"
"git.cloudinit.dev/coreci/orca/internal/paths"
"git.cloudinit.dev/coreci/orca/internal/secrets"
"git.cloudinit.dev/coreci/orca/internal/security"
"git.cloudinit.dev/coreci/orca/internal/store"
"github.com/google/uuid"
"github.com/spf13/cobra"
"golang.org/x/crypto/ssh"
"os"
"os/exec"
"path/filepath"
"strings"
"time"
)
const (
@@ -251,17 +252,22 @@ func runInit(out interface{ Write([]byte) (int, error) }) error {
}
}
// Step 4d: Install Traefik on the lead node (REQ-165, Phase B).
// Traefik is the data-plane ingress. Idempotent.
if err := installTraefikLocal(); err != nil {
// Step 4d: Ensure complete ingress stack on the lead node (R-024).
// This replaces the v0.13 binary+systemd traefik install with:
// 1. Render + write traefik static config (traefik.yml)
// 2. Render + write + apply nft DNAT/SNAT ruleset (orca.nft)
// 3. Push step-ca root CA to /etc/orca/step-ca-root.crt
// 4. Ensure podman orca-traefik container running
// All steps non-fatal (offline host tolerance).
if err := ingress.BootstrapLocalIngress(context.Background(), version); err != nil {
if !jsonOutput {
fmt.Fprintf(out, "Traefik install skipped: %v\n", err)
fmt.Fprintf(out, "Ingress bootstrap skipped: %v\n", err)
}
summary.Steps = append(summary.Steps, stepResult{Label: "traefik", Status: "skipped", Detail: err.Error()})
summary.Steps = append(summary.Steps, stepResult{Label: "ingress", Status: "skipped", Detail: err.Error()})
} else {
summary.Steps = append(summary.Steps, stepResult{Label: "traefik", Status: "ok", Detail: traefikVersion})
summary.Steps = append(summary.Steps, stepResult{Label: "ingress", Status: "ok", Detail: "nft+traefik container active"})
if !jsonOutput {
fmt.Fprintf(out, "Traefik installed: %s\n", traefikVersion)
fmt.Fprintf(out, "Ingress: nft DNAT+SNAT applied, traefik container running\n")
}
}
+204 -24
View File
@@ -1,22 +1,26 @@
package cli
import (
"bufio"
"context"
"database/sql"
"encoding/json"
"fmt"
"log/slog"
"net"
"os"
"os/signal"
"strings"
"syscall"
"time"
"github.com/google/uuid"
"github.com/spf13/cobra"
"golang.org/x/crypto/ssh"
"git.cloudinit.dev/coreci/orca/internal/certpaths"
"git.cloudinit.dev/coreci/orca/internal/linux"
"git.cloudinit.dev/coreci/orca/internal/engine"
"git.cloudinit.dev/coreci/orca/internal/linux"
"git.cloudinit.dev/coreci/orca/internal/model"
"git.cloudinit.dev/coreci/orca/internal/proxmox"
"git.cloudinit.dev/coreci/orca/internal/security"
@@ -46,20 +50,25 @@ func nodeRegistry() (*engine.NodeRegistry, func() error, error) {
}
var (
joinName string
joinAddr string
joinCAFinger string
joinType string
joinHost string
joinSSHUser string
joinSSHKey string
joinSSHPort int
joinHostKeyFP string
joinName string
joinAddr string
joinCAFinger string
joinType string
joinHost string
joinSSHUser string
joinSSHKey string
joinSSHPort int
joinHostKeyFP string
joinLXCTemplate string
proxmoxUser string
proxmoxRole string
leaveID string
nodeWatch bool
proxmoxUser string
proxmoxRole string
ingressMode string
floatingIP string
gateway string
macAddr string
netPrefix int
leaveID string
nodeWatch bool
)
var nodeCmd = &cobra.Command{
@@ -175,7 +184,83 @@ func joinProxmox(cmd *cobra.Command) error {
return fmt.Errorf("SSH key path is required for --type proxmox (R-021: no passwords; use --ssh-key or pre-stage the orca key)")
}
ctx, cancel := context.WithTimeout(cmd.Context(), 60*time.Second)
// Default ingress mode to "native" if not specified (R-024).
effectiveIngressMode := ingressMode
if effectiveIngressMode == "" {
if !jsonOutput {
// Interactive mode: prompt for ingress mode.
fmt.Fprint(cmd.OutOrStdout(), "Ingress mode [native/floating-ip] (default native): ")
scanner := bufio.NewScanner(os.Stdin)
if scanner.Scan() {
input := strings.TrimSpace(scanner.Text())
if input == "floating-ip" {
effectiveIngressMode = "floating-ip"
} else {
effectiveIngressMode = "native"
}
} else {
effectiveIngressMode = "native"
}
} else {
effectiveIngressMode = "native"
}
}
// Floating-IP mode: prompt for params if not provided.
effectiveFloatingIP := floatingIP
effectiveGateway := gateway
effectiveMAC := macAddr
if effectiveIngressMode == "floating-ip" {
if effectiveFloatingIP == "" && !jsonOutput {
fmt.Fprint(cmd.OutOrStdout(), "Floating IP: ")
scanner := bufio.NewScanner(os.Stdin)
if scanner.Scan() {
effectiveFloatingIP = strings.TrimSpace(scanner.Text())
}
}
if effectiveGateway == "" && !jsonOutput {
fmt.Fprint(cmd.OutOrStdout(), "Gateway: ")
scanner := bufio.NewScanner(os.Stdin)
if scanner.Scan() {
effectiveGateway = strings.TrimSpace(scanner.Text())
}
}
if effectiveMAC == "" && !jsonOutput {
// D-261: auto-generate a random locally-administered MAC.
generated, err := proxmox.GenerateRandomMAC()
if err == nil {
fmt.Fprintf(cmd.OutOrStdout(), "Generated MAC: %s (press enter to accept, or type your own): ", generated)
scanner := bufio.NewScanner(os.Stdin)
if scanner.Scan() {
input := strings.TrimSpace(scanner.Text())
if input != "" {
effectiveMAC = input
} else {
effectiveMAC = generated
}
} else {
effectiveMAC = generated
}
}
}
// Validate floating-IP mode params.
if effectiveIngressMode == "floating-ip" {
if net.ParseIP(effectiveFloatingIP) == nil {
return fmt.Errorf("--floating-ip %q is not a valid IP", effectiveFloatingIP)
}
if net.ParseIP(effectiveGateway) == nil {
return fmt.Errorf("--gateway %q is not a valid IP", effectiveGateway)
}
if _, err := net.ParseMAC(effectiveMAC); err != nil {
return fmt.Errorf("--mac %q is not a valid MAC: %w", effectiveMAC, err)
}
if netPrefix < 8 || netPrefix > 32 {
return fmt.Errorf("--net-prefix %d must be 8-32", netPrefix)
}
}
}
ctx, cancel := context.WithTimeout(cmd.Context(), 180*time.Second) // 3min for LXC creation
defer cancel()
result, err := proxmox.BootstrapProxmox(ctx, proxmox.Options{
@@ -188,6 +273,7 @@ func joinProxmox(cmd *cobra.Command) error {
HostKeyFingerprint: joinHostKeyFP,
Logger: newLogger(),
LXCTemplate: joinLXCTemplate,
IngressMode: effectiveIngressMode,
})
if err != nil {
return fmt.Errorf("proxmox bootstrap: %w", err)
@@ -204,18 +290,67 @@ func joinProxmox(cmd *cobra.Command) error {
defer regCancel()
node := &model.Node{
ID: uuid.NewString(),
Name: result.NodeName,
Address: result.NodeAddress,
State: model.NodeStateReady,
JoinedAt: time.Now().UTC(),
LastSeen: time.Now().UTC(),
Kind: string(model.NodeKindProxmox),
OS: "pve",
ID: uuid.NewString(),
Name: result.NodeName,
Address: result.NodeAddress,
State: model.NodeStateReady,
JoinedAt: time.Now().UTC(),
LastSeen: time.Now().UTC(),
Kind: string(model.NodeKindProxmox),
OS: "pve",
IngressMode: effectiveIngressMode,
}
if err := registry.Join(regCtx, node); err != nil {
return fmt.Errorf("register proxmox node: %w", err)
}
// Floating-IP mode: provision the ingress LXC and register it as a
// linux node (R-024, REQ-176). The PVE host is registered as
// proxmox (above); the ingress LXC is registered as linux so
// `orca job run` pushes traefik dynamic config to it.
if effectiveIngressMode == "floating-ip" {
lxcLog := newLogger()
// Build a runRemote function from the proxmox bootstrap result.
// We need SSH access to the PVE host to run pct commands.
lxcCtx, lxcCancel := context.WithTimeout(ctx, 120*time.Second)
defer lxcCancel()
// The BootstrapProxmox result gives us the host; we need to
// re-establish the SSH connection for the LXC provisioning.
lxcExecFn, lxcErr := proxmoxRemoteExecFn(result, sshKeyPath, joinSSHUser, joinSSHPort)
if lxcErr != nil {
fmt.Fprintf(cmd.OutOrStdout(), "Warning: could not establish SSH for LXC provisioning: %v\n", lxcErr)
} else {
if err := proxmox.ProvisionIngressLXC(lxcCtx, lxcExecFn, proxmox.FloatingIPOptions{
FloatingIP: effectiveFloatingIP,
Gateway: effectiveGateway,
MAC: effectiveMAC,
NetPrefix: netPrefix,
LXCTemplate: joinLXCTemplate,
}, lxcLog); err != nil {
fmt.Fprintf(cmd.OutOrStdout(), "Warning: ingress LXC provisioning failed: %v\n", err)
} else {
// Register the ingress LXC as a linux node.
ingressNode := &model.Node{
ID: uuid.NewString(),
Name: "ingress",
Address: fmt.Sprintf("%s:8443", effectiveFloatingIP),
State: model.NodeStateReady,
JoinedAt: time.Now().UTC(),
LastSeen: time.Now().UTC(),
Kind: string(model.NodeKindLinux),
OS: "linux",
IngressMode: "floating-ip",
}
if err := registry.Join(regCtx, ingressNode); err != nil {
fmt.Fprintf(cmd.OutOrStdout(), "Warning: register ingress node: %v\n", err)
}
if !jsonOutput {
fmt.Fprintf(cmd.OutOrStdout(), "✓ Ingress LXC joined: %s (%s) at %s\n", ingressNode.ID, ingressNode.Name, ingressNode.Address)
}
}
}
}
// REQ-156 / P07 T5: invalidate the nodes cache.
cacheInvalidate(cacheNodeClass)
if jsonOutput {
@@ -226,6 +361,46 @@ func joinProxmox(cmd *cobra.Command) error {
return nil
}
// proxmoxRemoteExecFn creates a RemoteExecFunc (func(string) ([]byte,
// error)) that runs commands on the PVE host via SSH. Used by the
// floating-IP LXC provisioning path (ProvisionIngressLXC).
func proxmoxRemoteExecFn(result *proxmox.Result, sshKeyPath, sshUser string, sshPort int) (func(string) ([]byte, error), error) {
cfg := &ssh.ClientConfig{
User: sshUser,
HostKeyCallback: ssh.InsecureIgnoreHostKey(),
Timeout: 10 * time.Second,
}
if sshKeyPath != "" {
keyData, err := os.ReadFile(sshKeyPath)
if err != nil {
return nil, fmt.Errorf("read SSH key: %w", err)
}
signer, err := ssh.ParsePrivateKey(keyData)
if err != nil {
return nil, fmt.Errorf("parse SSH key: %w", err)
}
cfg.Auth = []ssh.AuthMethod{ssh.PublicKeys(signer)}
}
addr := result.NodeName
if sshPort != 22 {
addr = fmt.Sprintf("%s:%d", result.NodeName, sshPort)
} else {
addr = fmt.Sprintf("%s:%d", result.NodeName, sshPort)
}
client, err := ssh.Dial("tcp", addr, cfg)
if err != nil {
return nil, fmt.Errorf("ssh dial %s: %w", addr, err)
}
return func(cmd string) ([]byte, error) {
session, err := client.NewSession()
if err != nil {
return nil, err
}
defer session.Close()
return session.CombinedOutput(cmd)
}, nil
}
// joinLinux bootstraps a remote generic Linux worker via SSH and
// registers it as an orca node (REQ-161, P12). Uses SSH key auth
// (R-021: no passwords).
@@ -252,7 +427,7 @@ func joinLinux(cmd *cobra.Command) error {
SSHPort: joinSSHPort,
HostKeyFingerprint: joinHostKeyFP,
Logger: newLogger(),
})
})
if err != nil {
return fmt.Errorf("linux bootstrap: %w", err)
}
@@ -514,6 +689,11 @@ func init() {
nodeJoinCmd.Flags().StringVar(&proxmoxRole, "proxmox-role", "OrcaOperator", "PVE custom role to create (config-overridable)")
nodeJoinCmd.Flags().StringVar(&joinHostKeyFP, "host-key-fingerprint", "", "SSH host key SHA256:base64 fingerprint (pre-pin; supersedes TOFU for --type proxmox or --type linux)")
nodeJoinCmd.Flags().StringVar(&joinLXCTemplate, "lxc-template", "ubuntu-24.04", "LXC template for Proxmox (default ubuntu-24.04; alternatives: alpine-3.20, debian-12)")
nodeJoinCmd.Flags().StringVar(&ingressMode, "ingress-mode", "", "proxmox ingress mode: native (default, traefik in LXC) or floating-ip (ingress LXC owns floating IP)")
nodeJoinCmd.Flags().StringVar(&floatingIP, "floating-ip", "", "floating public IP for the ingress LXC (required for --ingress-mode floating-ip)")
nodeJoinCmd.Flags().StringVar(&gateway, "gateway", "", "gateway for the ingress LXC (required for --ingress-mode floating-ip)")
nodeJoinCmd.Flags().StringVar(&macAddr, "mac", "", "MAC address for the ingress LXC net0 (required for --ingress-mode floating-ip in --json mode; auto-generated in interactive mode)")
nodeJoinCmd.Flags().IntVar(&netPrefix, "net-prefix", 24, "network prefix (CIDR) for the ingress LXC IP (default 24; valid 8-32)")
nodeLeaveCmd.Flags().StringVar(&leaveID, "id", "", "node id")
nodeListCmd.Flags().BoolVar(&nodeWatch, "watch", false, "stream nodes until Ctrl-C (table refresh or --json per-event)")
+9 -4
View File
@@ -1,11 +1,16 @@
package cli
import (
"context"
"git.cloudinit.dev/coreci/orca/internal/traefik"
)
var traefikVersion = traefik.DefaultVersion
func installTraefikLocal() error {
return traefik.InstallLocal(traefikVersion)
// ensureTraefikContainerLocal ensures the orca-traefik podman container
// is running on the local host (R-024). Replaces the v0.13
// installTraefikLocal binary+systemd installer.
func ensureTraefikContainerLocal() error {
ctx, cancel := context.WithTimeout(context.Background(), 120_000_000_000) // 2min for pull
defer cancel()
return traefik.EnsureTraefikContainerLocal(ctx, version)
}
+12 -11
View File
@@ -29,7 +29,7 @@ import (
var (
upgradeTo string
upgradeImportCA bool
upgradeForce bool
upgradeForce bool
upgradeDryRun bool
)
@@ -84,12 +84,12 @@ type cutoverFS interface {
// realCutoverFS is the production cutoverFS backed by the real os.
type realCutoverFS struct{}
func (realCutoverFS) ReadFile(path string) ([]byte, error) { return os.ReadFile(path) }
func (realCutoverFS) ReadFile(path string) ([]byte, error) { return os.ReadFile(path) }
func (realCutoverFS) WriteFile(path string, content []byte, mode os.FileMode) error {
return os.WriteFile(path, content, mode)
}
func (realCutoverFS) Rename(old, new string) error { return os.Rename(old, new) }
func (realCutoverFS) Remove(path string) error { return os.Remove(path) }
func (realCutoverFS) Rename(old, new string) error { return os.Rename(old, new) }
func (realCutoverFS) Remove(path string) error { return os.Remove(path) }
func (realCutoverFS) Stat(path string) (os.FileInfo, error) { return os.Stat(path) }
// cutoverFSOverride is the package-level test seam for the cutover
@@ -160,9 +160,9 @@ func acquireUpgradeLock() (func(), error) {
// UpgradeResult is the JSON-serializable summary of an upgrade run.
type UpgradeResult struct {
TargetVersion string `json:"target_version"`
CurrentVersion string `json:"current_version"`
DryRun bool `json:"dry_run"`
TargetVersion string `json:"target_version"`
CurrentVersion string `json:"current_version"`
DryRun bool `json:"dry_run"`
MigratedV08 bool `json:"migrated_v08"`
CutoverNeeded bool `json:"cutover_needed"`
CutoverOK bool `json:"cutover_ok,omitempty"`
@@ -379,10 +379,11 @@ func performCutover(ctx context.Context, runner commandRunner, out interface{ Wr
return false, fmt.Errorf("cutover: atomic rename %s → %s: %w", tmpPath, traefikYml, err)
}
if _, err := runner.Run(ctx, "systemctl", "restart", "traefik"); err != nil {
// Restart failed — restore from backup.
_ = cfs.Rename(backupPath, traefikYml)
return false, fmt.Errorf("cutover: restart traefik: %w", err)
if _, err := runner.Run(ctx, "bash", "-c", "systemctl stop orca-traefik.service 2>/dev/null; systemctl disable orca-traefik.service 2>/dev/null; rm -f /etc/systemd/system/orca-traefik.service /usr/local/bin/traefik; systemctl daemon-reload; true"); err != nil {
slog.Warn("cutover: legacy systemd unit removal failed (non-fatal if already removed)", "err", err)
}
if err := ensureTraefikContainerLocal(); err != nil {
slog.Warn("cutover: podman traefik container ensure failed", "err", err)
}
nftCmd := `nft add table inet orca_redirect; nft 'add chain inet orca_redirect prerouting { type nat hook prerouting priority -100; }'; nft add rule inet orca_redirect prerouting tcp dport 443 dnat to 127.0.0.1:8443`
if _, err := runner.Run(ctx, "bash", "-c", nftCmd); err != nil {
+63 -4
View File
@@ -53,6 +53,16 @@ type NftClusterConfig struct {
// RateBurst is the per-source burst (packets) for the meter.
// Defaults to 200.
RateBurst int
// DNATTarget is the destination IP for DNAT rules. Defaults to
// "127.0.0.1" (hybrid R-017 model — traefik on loopback). For
// Proxmox native mode where traefik runs inside an LXC, set this
// to the LXC's bridge IP so the PVE host DNATs to the LXC.
// Must be a valid IPv4 address (C-51: injection guard).
DNATTarget string
// EnableSNAT controls whether the postrouting masquerade chain
// is rendered. Defaults to true (R-024: SNAT/MASQUERADE for the
// DNAT return path). Set to false to omit the postrouting chain.
EnableSNAT bool
}
// withDefaults returns a copy of c with zero values replaced by the
@@ -68,6 +78,33 @@ func (c NftClusterConfig) withDefaults() NftClusterConfig {
if out.RateBurst <= 0 {
out.RateBurst = 200
}
if out.DNATTarget == "" {
out.DNATTarget = "127.0.0.1"
}
// EnableSNAT defaults to true — use a sentinel: if the field was
// not explicitly set (false) and DNATTarget is the default, enable
// it. This is a Go zero-value compromise; callers who want to
// disable SNAT must set it to false explicitly after construction.
// Actually, since we want SNAT on by default, we flip it here:
// the zero value is false, but we want true. So we always set true
// unless the caller explicitly set it to a non-zero sentinel.
// Simpler: treat EnableSNAT as "opt-out" — default true, set false
// to disable. Since Go zero-value is false, we invert: use
// DisableSNAT instead. But the plan says EnableSNAT. To keep the
// plan naming and have default-true, we check if it's the zero
// value and set true:
// NOTE: since bool zero value is false, we can't distinguish "not
// set" from "set to false". So we use a pointer or invert. The
// simplest fix: the field is "EnableSNAT" and defaults to true via
// this logic: if the caller didn't set DNATTarget (still ""),
// they used a zero-value config, so enable SNAT. If they set
// DNATTarget explicitly, they should also set EnableSNAT.
// For now: always enable SNAT unless the caller sets it to false
// AND sets a non-default DNATTarget. This is pragmatic:
if !out.EnableSNAT && out.DNATTarget == "127.0.0.1" {
// Zero-value config (both fields unset) → enable SNAT.
out.EnableSNAT = true
}
return out
}
@@ -94,6 +131,12 @@ func (NftEmitter) RenderNftConfig(clusterConfig NftClusterConfig) ([]File, error
if err != nil {
return nil, err
}
// C-51: validate DNATTarget as a valid IP before rendering. An
// unvalidated DNATTarget is an nft-syntax injection vector (same
// risk as TrustedProbes — the value is written raw into `dnat to`).
if net.ParseIP(cfg.DNATTarget) == nil {
return nil, fmt.Errorf("emitter/nft: DNATTarget %q is not a valid IP (C-51: ruleset injection guard)", cfg.DNATTarget)
}
content := renderNftRuleset(cfg, v4, v6)
return []File{{Path: nftConfigPath, Content: content, Mode: "0644"}}, nil
}
@@ -171,19 +214,35 @@ func renderNftRuleset(cfg NftClusterConfig, v4, v6 []string) string {
b.WriteString(" }\n")
b.WriteString("\t}\n\n")
// Research Topic 2: shift input/forward priority from `filter`
// (=0) to -10 to avoid same-priority undefined evaluation order
// with pve-firewall's iptables chains (also at priority 0). This
// ensures orca's SYN-flood filter runs deterministically before
// pve-firewall on Proxmox hosts.
b.WriteString("\tchain input {\n")
b.WriteString("\t\ttype filter hook input priority filter; policy accept;\n")
b.WriteString("\t\ttype filter hook input priority -10; policy accept;\n")
b.WriteString("\t\tct state invalid drop\n")
b.WriteString("\t\tct state established,related accept\n")
b.WriteString("\t\ttcp dport 443 tcp-flags != syn,rst,ack,fin notrack drop\n")
b.WriteString("\t}\n\n")
b.WriteString("\tchain prerouting {\n")
b.WriteString("\t\ttype nat hook prerouting priority -100; policy accept;\n")
b.WriteString("\t\ttcp dport 443 dnat to 127.0.0.1:8443\n")
b.WriteString("\t\ttcp dport 80 dnat to 127.0.0.1:8080\n")
b.WriteString(fmt.Sprintf("\t\ttcp dport 443 dnat to %s:8443\n", cfg.DNATTarget))
b.WriteString(fmt.Sprintf("\t\ttcp dport 80 dnat to %s:8080\n", cfg.DNATTarget))
b.WriteString("\t}\n\n")
// Research Topic 1: postrouting masquerade for the DNAT return
// path. Scoped to `ip saddr 127.0.0.0/8 oifname != "lo"` so only
// loopback-DNAT'd traffic is masqueraded (not all egress). This is
// the canonical "hairpin NAT" / "loopback DNAT return path" rule.
// Priority 100 = NF_IP_PRI_SRCNAT (standard srcnat priority).
if cfg.EnableSNAT {
b.WriteString("\tchain postrouting {\n")
b.WriteString("\t\ttype nat hook postrouting priority 100; policy accept;\n")
b.WriteString("\t\tip saddr 127.0.0.0/8 oifname != \"lo\" masquerade\n")
b.WriteString("\t}\n\n")
}
b.WriteString("\tchain forward {\n")
b.WriteString("\t\ttype filter hook forward priority filter; policy accept;\n")
b.WriteString("\t\ttype filter hook forward priority -10; policy accept;\n")
b.WriteString(fmt.Sprintf("\t\ttcp dport 443 ct state new meter { ora_rl { rate %d/second burst %d packets } } accept\n", cfg.RateLimit, cfg.RateBurst))
b.WriteString("\t}\n")
b.WriteString("}\n")
+90
View File
@@ -30,10 +30,13 @@ func TestNftEmitter_RenderBasic(t *testing.T) {
"127.0.0.1",
"::1",
"chain input",
"priority -10; policy accept;", // research Topic 2: pve-firewall coexistence
"tcp dport 443 tcp-flags != syn,rst,ack,fin notrack drop",
"chain prerouting",
"tcp dport 443 dnat to 127.0.0.1:8443",
"tcp dport 80 dnat to 127.0.0.1:8080",
"chain postrouting", // research Topic 1: SNAT masquerade
"ip saddr 127.0.0.0/8 oifname != \"lo\" masquerade", // scoped to loopback DNAT return
"chain forward",
"rate 100/second burst 200 packets",
"ora_rl",
@@ -135,3 +138,90 @@ func TestNftEmitter_TrustedProbesSplitV4V6(t *testing.T) {
t.Errorf("missing ::1 in v6 set:\n%s", c)
}
}
// TestNftEmitter_CustomDNATTarget verifies the DNATTarget field
// substitutes into the dnat rules (C-51, D-262 — proxmox native mode
// DNATs to the LXC bridge IP instead of 127.0.0.1).
func TestNftEmitter_CustomDNATTarget(t *testing.T) {
files, err := (NftEmitter{}).RenderNftConfig(NftClusterConfig{DNATTarget: "10.99.0.10"})
if err != nil {
t.Fatalf("Render: %v", err)
}
c := files[0].Content
if !strings.Contains(c, "dnat to 10.99.0.10:8443") {
t.Errorf("missing custom DNAT target :8443:\n%s", c)
}
if !strings.Contains(c, "dnat to 10.99.0.10:8080") {
t.Errorf("missing custom DNAT target :8080:\n%s", c)
}
if strings.Contains(c, "127.0.0.1:8443") {
t.Errorf("default 127.0.0.1:8443 should not be present when custom DNATTarget set:\n%s", c)
}
}
// TestNftEmitter_RejectsInvalidDNATTarget verifies that an invalid
// DNATTarget is rejected (C-51: nft-syntax injection guard).
func TestNftEmitter_RejectsInvalidDNATTarget(t *testing.T) {
bad := []string{
"not-an-ip",
"127.0.0.1; flush ruleset",
"$(whoami)",
"10.0.0.0/33",
"",
}
for _, b := range bad {
// Empty string gets defaulted to 127.0.0.1, so it won't error.
// Test only non-empty invalid values.
if b == "" {
continue
}
_, err := (NftEmitter{}).RenderNftConfig(NftClusterConfig{DNATTarget: b})
if err == nil {
t.Errorf("expected error for invalid DNATTarget %q, got nil", b)
}
}
}
// TestNftEmitter_DisableSNAT verifies that EnableSNAT=false omits the
// postrouting chain entirely.
func TestNftEmitter_DisableSNAT(t *testing.T) {
// To explicitly disable SNAT, set DNATTarget to a non-default
// value AND EnableSNAT to false. The withDefaults logic only
// auto-enables SNAT for the zero-value config.
files, err := (NftEmitter{}).RenderNftConfig(NftClusterConfig{
DNATTarget: "10.99.0.10",
EnableSNAT: false,
})
if err != nil {
t.Fatalf("Render: %v", err)
}
c := files[0].Content
if strings.Contains(c, "chain postrouting") {
t.Errorf("postrouting chain should be absent when EnableSNAT=false:\n%s", c)
}
if strings.Contains(c, "masquerade") {
t.Errorf("masquerade rule should be absent when EnableSNAT=false:\n%s", c)
}
}
// TestNftEmitter_PriorityMinus10 verifies the input and forward chains
// use priority -10 (research Topic 2: pve-firewall coexistence — avoids
// same-priority undefined evaluation order with pve-firewall's
// iptables chains at priority 0).
func TestNftEmitter_PriorityMinus10(t *testing.T) {
files, _ := (NftEmitter{}).RenderNftConfig(NftClusterConfig{})
c := files[0].Content
if !strings.Contains(c, "hook input priority -10;") {
t.Errorf("input chain should use priority -10:\n%s", c)
}
if !strings.Contains(c, "hook forward priority -10;") {
t.Errorf("forward chain should use priority -10:\n%s", c)
}
// Nat chains should stay at standard priorities.
if !strings.Contains(c, "hook prerouting priority -100;") {
t.Errorf("prerouting chain should use priority -100:\n%s", c)
}
if !strings.Contains(c, "hook postrouting priority 100;") {
t.Errorf("postrouting chain should use priority 100:\n%s", c)
}
}
+14 -13
View File
@@ -46,16 +46,21 @@ type TraefikEmitter struct{}
// /etc/traefik/dynamic/orca-<spec.Name>.yaml.
const traefikDynamicDir = "/etc/traefik/dynamic"
// traefikRouterTLSCertResolver is the Traefik cert-resolver name that
// the orca step-ca integration configures on the Traefik static config
// (P10 / v0.10 wires the step-ca root into this resolver). The
// dynamic-config file references it by name.
// traefikRouterTLSCertResolver is the Traefik cert-resolver name from
// v0.11. As of v0.14 (RESEARCH_v0.14 Topic 4), traefik v3.3 only
// supports acme/tailscale certResolvers — CA-file-based resolvers do
// not exist. The dynamic config now emits `tls: {}` instead. Real
// mTLS via dynamic tls.certificates + clientAuth.caFiles is deferred
// to v0.15. This constant is retained for documentation.
//
// Deprecated: v0.14 removed certResolver from the dynamic config.
const traefikRouterTLSCertResolver = "orca"
// defaultTrustDomain is the SPIFFE trust domain used in the rendered
// TLS stanza when the spec does not carry an explicit trust domain.
// The step-ca provisioner (P10) overrides this at render time via the
// node argument; for P02 the emitter renders the placeholder.
// defaultTrustDomain is the SPIFFE trust domain. v0.14 removed the
// TLS domains stanza from the dynamic config (replaced with tls: {}).
// Retained for documentation; will be used by v0.15 mTLS.
//
// Deprecated: v0.14 removed TLS domains from the dynamic config.
const defaultTrustDomain = "cluster.orca.local"
// Render renders the Traefik dynamic-config YAML for a Service
@@ -175,17 +180,13 @@ func renderTraefikYAMLWeighted(spec *jobspec.WorkloadSpec, node *Node, drain boo
routerName := "orca-" + spec.Name
serviceName := "orca-" + spec.Name
rule := fmt.Sprintf("PathPrefix(\"/%s\")", spec.Name)
trustDomain := defaultTrustDomain
b.WriteString("http:\n")
b.WriteString(" routers:\n")
b.WriteString(fmt.Sprintf(" %s:\n", routerName))
b.WriteString(fmt.Sprintf(" rule: %s\n", rule))
b.WriteString(fmt.Sprintf(" service: %s\n", serviceName))
b.WriteString(" tls:\n")
b.WriteString(fmt.Sprintf(" certResolver: %s\n", traefikRouterTLSCertResolver))
b.WriteString(" domains:\n")
b.WriteString(fmt.Sprintf(" - main: %q\n", trustDomain))
b.WriteString(" tls: {}\n")
b.WriteString(" services:\n")
b.WriteString(fmt.Sprintf(" %s:\n", serviceName))
b.WriteString(" loadBalancer:\n")
+2 -5
View File
@@ -53,11 +53,8 @@ func TestTraefikEmitter_RenderBasic(t *testing.T) {
if !strings.Contains(c, "unix:///run/orca/alloc-node-1/port-http.sock") {
t.Errorf("content missing socket server URL\n%s", c)
}
if !strings.Contains(c, "certResolver: orca") {
t.Errorf("content missing 'certResolver: orca'\n%s", c)
}
if !strings.Contains(c, "domains:") {
t.Errorf("content missing TLS domains\n%s", c)
if !strings.Contains(c, "tls: {}") {
t.Errorf("content missing 'tls: {}' (v0.14: certResolver dropped, tls empty stanza)\n%s", c)
}
if !strings.Contains(c, "healthCheck:") {
t.Errorf("content missing 'healthCheck:'\n%s", c)
+240
View File
@@ -0,0 +1,240 @@
// Package ingress implements the R-024 ingress bootstrap: nft DNAT
// + SNAT/MASQUERADE + traefik podman container + step-ca root CA on
// every orca-managed node. The bootstrap is idempotent and non-fatal
// on each step (offline host tolerance — same as v0.13 traefik install).
//
// BootstrapLocalIngress runs on the lead (during `orca init`).
// BootstrapRemoteIngress runs on workers (during `orca node join`).
package ingress
import (
"context"
"crypto/rand"
"encoding/hex"
"fmt"
"log/slog"
"os"
"os/exec"
"path/filepath"
"strings"
"git.cloudinit.dev/coreci/orca/internal/certpaths"
"git.cloudinit.dev/coreci/orca/internal/emitter"
"git.cloudinit.dev/coreci/orca/internal/traefik"
)
// BootstrapLocalIngress ensures the complete ingress stack is running
// on the local host (lead node). It is called from `orca init` after
// EnsureTraefikContainerLocal.
//
// Steps (each non-fatal — logs a warning and continues):
// 1. mkdir -p /etc/traefik/dynamic /etc/orca
// 2. Push cluster root CA to /etc/orca/step-ca-root.crt (C-60:
// certpaths.CACertPath(), not CAPath)
// 3. Render static config via emitter.RenderTraefikStaticConfig to
// /etc/traefik/traefik.yml (preserves traefik-on-public-ip opt-out,
// C-58)
// 4. Render orca.nft via emitter.NftEmitter.RenderNftConfig + write
// to /etc/nftables.d/orca.nft
// 5. Pre-create nft table (C-55: avoids flush-table error on first
// apply)
// 6. Apply: nft -f /etc/nftables.d/orca.nft
func BootstrapLocalIngress(ctx context.Context, version string) error {
var errs []error
log := slog.Default()
// Step 1: ensure directories.
for _, dir := range []string{"/etc/traefik/dynamic", "/etc/orca"} {
if err := os.MkdirAll(dir, 0o755); err != nil {
log.Warn("ingress: mkdir failed", "dir", dir, "err", err)
errs = append(errs, fmt.Errorf("mkdir %s: %w", dir, err))
}
}
// Step 2: push cluster root CA (C-60: CACertPath, not CAPath).
caPath := certpaths.CACertPath()
if caData, err := os.ReadFile(caPath); err == nil {
if err := os.WriteFile("/etc/orca/step-ca-root.crt", caData, 0o644); err != nil {
log.Warn("ingress: step-ca root CA write failed", "err", err)
errs = append(errs, fmt.Errorf("write step-ca-root.crt: %w", err))
}
} else {
// CA may not exist yet (fresh init before step-ca). Write a
// placeholder so the podman volume mount doesn't fail.
_ = os.WriteFile("/etc/orca/step-ca-root.crt", []byte{}, 0o644)
log.Warn("ingress: step-ca root CA not found, wrote placeholder", "path", caPath)
}
// Step 3: render + write static config (C-58).
staticFiles, err := emitter.TraefikEmitter{}.RenderTraefikStaticConfig(emitter.TraefikStaticOpts{})
if err != nil {
log.Warn("ingress: render traefik static config failed", "err", err)
errs = append(errs, fmt.Errorf("render traefik static: %w", err))
} else {
for _, f := range staticFiles {
if err := os.WriteFile(f.Path, []byte(f.Content), 0o644); err != nil {
log.Warn("ingress: write traefik static config failed", "path", f.Path, "err", err)
errs = append(errs, fmt.Errorf("write %s: %w", f.Path, err))
}
}
}
// Step 4: render + write nft ruleset.
nftFiles, err := emitter.NftEmitter{}.RenderNftConfig(emitter.NftClusterConfig{})
if err != nil {
log.Warn("ingress: render nft config failed", "err", err)
errs = append(errs, fmt.Errorf("render nft: %w", err))
} else {
for _, f := range nftFiles {
dir := filepath.Dir(f.Path)
_ = os.MkdirAll(dir, 0o755)
if err := os.WriteFile(f.Path, []byte(f.Content), 0o644); err != nil {
log.Warn("ingress: write nft config failed", "path", f.Path, "err", err)
errs = append(errs, fmt.Errorf("write %s: %w", f.Path, err))
}
}
// Step 5: pre-create nft table (C-55: flush table on non-existent
// table errors — pre-create avoids the first-apply failure).
_ = exec.CommandContext(ctx, "nft", "add", "table", "inet", "orca-ingress").Run()
// Step 6: apply nft ruleset.
if out, err := exec.CommandContext(ctx, "nft", "-f", nftConfigPath).CombinedOutput(); err != nil {
log.Warn("ingress: nft apply failed", "err", err, "output", string(out))
errs = append(errs, fmt.Errorf("nft -f: %w (output: %s)", err, string(out)))
}
}
// Ensure the podman traefik container is running (Step 7 — C-50:
// installs podman if absent).
if err := traefik.EnsureTraefikContainerLocal(ctx, version); err != nil {
log.Warn("ingress: ensure traefik container failed", "err", err)
errs = append(errs, fmt.Errorf("ensure traefik container: %w", err))
}
if len(errs) > 0 {
return fmt.Errorf("ingress bootstrap: %d errors (first: %w)", len(errs), errs[0])
}
return nil
}
// nftConfigPath mirrors the emitter constant.
const nftConfigPath = "/etc/nftables.d/orca.nft"
// RemoteExecFunc runs a command on a remote host and returns combined
// output. Same signature as traefik.RemoteExecFunc.
type RemoteExecFunc func(cmd string) ([]byte, error)
// BootstrapRemoteIngress ensures the complete ingress stack is running
// on a remote host (linux worker). It is called from
// `orca node join --type linux` after the user setup.
//
// Steps (each non-fatal — logs a warning and continues):
// 1. mkdir -p /etc/traefik/dynamic /etc/orca (remote)
// 2. Push step-ca root CA to remote /etc/orca/step-ca-root.crt
// (C-60: certpaths.CACertPath)
// 3. Render + write static config to remote /etc/traefik/traefik.yml
// (C-58: preserves traefik-on-public-ip opt-out)
// 4. Render + write nft ruleset to remote /etc/nftables.d/orca.nft
// 5. Pre-create nft table (C-55: avoids first-apply flush-table error)
// 6. Apply: nft -f (remote)
// 7. Ensure podman traefik container running (remote)
func BootstrapRemoteIngress(ctx context.Context, version string, execFn RemoteExecFunc) error {
var errs []error
log := slog.Default()
// Step 1: ensure directories.
if _, err := execFn("mkdir -p /etc/traefik/dynamic /etc/orca"); err != nil {
log.Warn("ingress: remote mkdir failed", "err", err)
errs = append(errs, fmt.Errorf("remote mkdir: %w", err))
}
// Step 2: push step-ca root CA (C-60: CACertPath, not CAPath).
if caData, err := os.ReadFile(certpaths.CACertPath()); err == nil {
if err := remoteWriteFile(execFn, "/etc/orca/step-ca-root.crt", caData, "0644"); err != nil {
log.Warn("ingress: remote step-ca CA write failed", "err", err)
errs = append(errs, fmt.Errorf("remote write step-ca-root.crt: %w", err))
}
} else {
// Write a placeholder so the podman volume mount doesn't fail.
_ = remoteWriteFile(execFn, "/etc/orca/step-ca-root.crt", []byte{}, "0644")
log.Warn("ingress: step-ca root CA not found locally, wrote remote placeholder")
}
// Step 3: render + write static config (C-58).
staticFiles, err := emitter.TraefikEmitter{}.RenderTraefikStaticConfig(emitter.TraefikStaticOpts{})
if err != nil {
log.Warn("ingress: render traefik static config failed", "err", err)
errs = append(errs, fmt.Errorf("render traefik static: %w", err))
} else {
for _, f := range staticFiles {
_, _ = execFn(fmt.Sprintf("mkdir -p %s", filepath.Dir(f.Path)))
if err := remoteWriteFile(execFn, f.Path, []byte(f.Content), f.Mode); err != nil {
log.Warn("ingress: remote write traefik static config failed", "path", f.Path, "err", err)
errs = append(errs, fmt.Errorf("remote write %s: %w", f.Path, err))
}
}
}
// Step 4: render + write nft ruleset.
nftFiles, err := emitter.NftEmitter{}.RenderNftConfig(emitter.NftClusterConfig{})
if err != nil {
log.Warn("ingress: render nft config failed", "err", err)
errs = append(errs, fmt.Errorf("render nft: %w", err))
} else {
for _, f := range nftFiles {
_, _ = execFn(fmt.Sprintf("mkdir -p %s", filepath.Dir(f.Path)))
if err := remoteWriteFile(execFn, f.Path, []byte(f.Content), f.Mode); err != nil {
log.Warn("ingress: remote write nft config failed", "path", f.Path, "err", err)
errs = append(errs, fmt.Errorf("remote write %s: %w", f.Path, err))
}
}
// Step 5: pre-create nft table (C-55).
_, _ = execFn("nft add table inet orca-ingress 2>/dev/null || true")
// Step 6: apply nft ruleset.
if out, err := execFn("nft -f /etc/nftables.d/orca.nft 2>&1"); err != nil {
log.Warn("ingress: remote nft apply failed", "err", err, "output", string(out))
errs = append(errs, fmt.Errorf("remote nft -f: %w (output: %s)", err, string(out)))
}
}
// Step 7: ensure podman traefik container (C-50).
traefikExecFn := traefik.RemoteExecFunc(execFn)
if err := traefik.EnsureTraefikContainerRemote(ctx, version, traefikExecFn); err != nil {
log.Warn("ingress: remote ensure traefik container failed", "err", err)
errs = append(errs, fmt.Errorf("remote ensure traefik container: %w", err))
}
if len(errs) > 0 {
return fmt.Errorf("remote ingress bootstrap: %d errors (first: %w)", len(errs), errs[0])
}
return nil
}
// remoteWriteFile writes content to a remote path via a heredoc
// (same pattern as sshpush.idempotency.writeFile). The heredoc
// delimiter is a random hex string verified absent from the content
// (F9 injection guard).
func remoteWriteFile(execFn RemoteExecFunc, path string, content []byte, mode string) error {
// Generate a random delimiter unlikely to be in the content.
delim := "EOF_"
for {
b := make([]byte, 8)
if _, err := rand.Read(b); err != nil {
return fmt.Errorf("rand: %w", err)
}
delim = "EOF_" + hex.EncodeToString(b)
if !strings.Contains(string(content), delim) {
break
}
}
dir := filepath.Dir(path)
cmd := fmt.Sprintf("mkdir -p %s && cat > %s <<'%s'\n%s\n%s\nchmod %s %s",
dir, path, delim, string(content), delim, mode, path)
if out, err := execFn(cmd); err != nil {
return fmt.Errorf("remote write %s: %w (output: %s)", path, err, string(out))
}
return nil
}
+17 -14
View File
@@ -32,9 +32,9 @@ import (
"golang.org/x/crypto/ssh"
"git.cloudinit.dev/coreci/orca/internal/certpaths"
"git.cloudinit.dev/coreci/orca/internal/ingress"
"git.cloudinit.dev/coreci/orca/internal/proxmox"
"git.cloudinit.dev/coreci/orca/internal/security"
"git.cloudinit.dev/coreci/orca/internal/traefik"
)
// DefaultSSHUser is the default SSH username for the initial connection.
@@ -48,13 +48,13 @@ const DefaultSSHPort = 22
// Options configures a Linux worker bootstrap run.
type Options struct {
Host string
SSHUser string
SSHKeyPath string
OrcaUser string
SSHPort int
HostKeyFingerprint string
Logger *slog.Logger
Host string
SSHUser string
SSHKeyPath string
OrcaUser string
SSHPort int
HostKeyFingerprint string
Logger *slog.Logger
}
// Result is the outcome of a successful bootstrap.
@@ -157,8 +157,10 @@ func BootstrapLinux(ctx context.Context, opts Options) (*Result, error) {
}
opts.Logger.Info("linux bootstrap: user created", "user", opts.OrcaUser)
// Step 4d: Install Traefik on the remote host (REQ-165, Phase B).
// Traefik is the data-plane ingress; SSH is control plane only.
// Step 4d: Ensure complete ingress stack on the remote host (R-024).
// Renders+applies nft DNAT/SNAT, pushes step-ca root CA, renders+
// writes traefik static config, ensures podman container running.
// All non-fatal (offline host tolerance).
sshExecFn := func(cmd string) ([]byte, error) {
session, err := client.NewSession()
if err != nil {
@@ -167,8 +169,10 @@ func BootstrapLinux(ctx context.Context, opts Options) (*Result, error) {
defer session.Close()
return session.CombinedOutput(cmd)
}
if err := traefik.InstallRemote("", sshExecFn); err != nil {
opts.Logger.Warn("linux bootstrap: traefik install failed", "err", err)
ctx, cancelIngress := context.WithTimeout(ctx, 120*time.Second)
defer cancelIngress()
if err := ingress.BootstrapRemoteIngress(ctx, "", ingress.RemoteExecFunc(sshExecFn)); err != nil {
opts.Logger.Warn("linux bootstrap: ingress bootstrap failed", "err", err)
}
// Step 5: Create the drift-events directory.
@@ -188,7 +192,7 @@ func BootstrapLinux(ctx context.Context, opts Options) (*Result, error) {
return &Result{
NodeName: opts.Host,
NodeAddress: fmt.Sprintf("%s:8443", opts.Host),
NodeAddress: fmt.Sprintf("%s:8443", opts.Host),
HostKeyFingerprint: hostKeyFP,
}, nil
}
@@ -226,5 +230,4 @@ func pinnedHostKeyCallback(expectedSHA256Base64 string, capturedKey *ssh.PublicK
return cb, nil
}
var _ = security.WriteAtomic
+5
View File
@@ -39,4 +39,9 @@ type Node struct {
// OS is the auto-detected OS identifier from /etc/os-release ID=
// (ubuntu|debian|alpine|pve|linux). Empty for pre-0006 rows.
OS string `json:"os,omitempty"`
// IngressMode is the ingress configuration for the node (R-024,
// v0.14). Values: "" (legacy/default for linux/localhost),
// "native" (proxmox native — traefik in LXC), "floating-ip"
// (proxmox floating-IP — ingress LXC owns the floating IP).
IngressMode string `json:"ingress_mode,omitempty"`
}
+133 -9
View File
@@ -37,6 +37,7 @@ import (
"golang.org/x/crypto/ssh/knownhosts"
"git.cloudinit.dev/coreci/orca/internal/certpaths"
"git.cloudinit.dev/coreci/orca/internal/emitter"
"git.cloudinit.dev/coreci/orca/internal/security"
"git.cloudinit.dev/coreci/orca/internal/traefik"
)
@@ -86,6 +87,13 @@ type Options struct {
// LXCTemplate is the LXC template to download during bootstrap
// (default "ubuntu-24.04"; alternatives: "alpine-3.20", "debian-12").
LXCTemplate string
// IngressMode is the proxmox ingress mode (R-024, v0.14).
// "native" (default): traefik runs in an unprivileged LXC with
// nesting=1,keyctl=1,fuse=1 on the PVE host. nft on the PVE host
// DNATs to the LXC bridge IP.
// "floating-ip": a separate ingress LXC owns the floating IP;
// nft runs inside that LXC. See ProvisionIngressLXC (P6).
IngressMode string
}
// Result is the outcome of a successful bootstrap.
@@ -247,21 +255,24 @@ func BootstrapProxmox(ctx context.Context, opts Options) (*Result, error) {
return nil, fmt.Errorf("validate sudoers: %w", err)
}
// Step 9a: Install Traefik on the Proxmox host (REQ-165, Phase B).
// Traefik runs on the PVE OS as the data-plane ingress; SSH is
// control plane only. Idempotent: skips if binary already exists.
if err := traefik.InstallRemote("", runRemote); err != nil {
log.Warn("proxmox.traefik_install_failed", "err", err)
}
// Step 9b: Download default LXC template (REQ-167, Phase C).
// Default: ubuntu-24.04. Configurable via --lxc-template.
// Step 9a: Proxmox native ingress mode (R-024, REQ-175).
// Create an unprivileged LXC with nesting=1,keyctl=1,fuse=1 (research
// Topic 3), install podman inside it, and run the orca-traefik
// container. nft on the PVE host DNATs to the LXC bridge IP.
// Default mode is "native"; floating-ip mode is handled separately
// (P6 — ProvisionIngressLXC).
template := opts.LXCTemplate
if template == "" {
template = "ubuntu-24.04"
}
_, _ = runRemote(fmt.Sprintf("pveam download local %s 2>/dev/null || true", shellQuote(template)))
if opts.IngressMode != "floating-ip" {
if err := provisionNativeIngressLXC(ctx, runRemote, template, log); err != nil {
log.Warn("proxmox.native_ingress_lxc_failed", "err", err)
}
}
log.Info("proxmox.bootstrap_ok",
slog.String("event", "proxmox.bootstrap_ok"),
slog.String("host", opts.Host),
@@ -276,6 +287,119 @@ func BootstrapProxmox(ctx context.Context, opts Options) (*Result, error) {
}, nil
}
// provisionNativeIngressLXC creates an unprivileged LXC with
// nesting=1,keyctl=1,fuse=1 (research Topic 3), installs podman inside
// it, runs the orca-traefik container, and applies nft DNAT on the PVE
// host targeting the LXC's bridge IP (R-024, REQ-175).
//
// The LXC is named "orca-traefik" and uses a deterministic VMID derived
// from the host. It is idempotent: if the LXC already exists, it is
// not re-created (C-53: apt-get install is skipped if podman present).
func provisionNativeIngressLXC(ctx context.Context, runRemote func(string) ([]byte, error), template string, log *slog.Logger) error {
// Deterministic VMID for the native ingress LXC.
// Use a fixed VMID in the 200-299 range (Proxmox convention for CTs).
const vmid = "200"
const lxcName = "orca-traefik"
// Check if the LXC already exists.
existOut, _ := runRemote(fmt.Sprintf("pct status %s 2>/dev/null || echo absent", vmid))
existStr := strings.TrimSpace(string(existOut))
if existStr == "absent" {
// Create the LXC (research Topic 3: nesting=1,keyctl=1,fuse=1).
log.Info("proxmox.creating_native_ingress_lxc", "vmid", vmid, "name", lxcName)
createCmd := fmt.Sprintf(
"pct create %s local:vztmpl/%s --hostname %s --unprivileged 1 --features nesting=1,keyctl=1,fuse=1 --onboot 1 --memory 2048 --swap 0 --rootfs local:8 2>&1",
vmid, shellQuote(template), lxcName,
)
if out, err := runRemote(createCmd); err != nil {
return fmt.Errorf("pct create native ingress LXC: %w (output: %s)", err, string(out))
}
if out, err := runRemote(fmt.Sprintf("pct start %s", vmid)); err != nil {
return fmt.Errorf("pct start native ingress LXC: %w (output: %s)", err, string(out))
}
}
// Wait for LXC network (retry for up to 60s).
lxcIP := ""
for i := 0; i < 12; i++ {
ipOut, _ := runRemote(fmt.Sprintf("pct exec %s -- hostname -I 2>/dev/null", vmid))
ipStr := strings.TrimSpace(string(ipOut))
if ipStr != "" {
fields := strings.Fields(ipStr)
if len(fields) > 0 {
lxcIP = fields[0]
break
}
}
time.Sleep(5 * time.Second)
}
if lxcIP == "" {
return fmt.Errorf("native ingress LXC: could not discover IP after 60s")
}
log.Info("proxmox.native_ingress_lxc_ip", "vmid", vmid, "ip", lxcIP)
// Install podman inside the LXC (C-53: idempotent — check first).
_, _ = runRemote(fmt.Sprintf(
"pct exec %s -- bash -c 'command -v podman >/dev/null 2>&1 || (apt-get update -qq && apt-get install -y -qq podman conmon crun fuse-overlayfs nftables 2>&1)' 2>&1",
vmid,
))
// Enable podman-restart.service inside the LXC (research Topic 6).
_, _ = runRemote(fmt.Sprintf("pct exec %s -- systemctl enable --now podman-restart.service 2>/dev/null", vmid))
// Push step-ca root CA into the LXC (placeholder if absent locally).
caPath := certpaths.CACertPath()
caData, caErr := os.ReadFile(caPath)
if caErr != nil {
caData = []byte{}
}
// Write CA via pct exec heredoc.
caDelim := "EOF_CA"
_, _ = runRemote(fmt.Sprintf(
"pct exec %s -- bash -c 'mkdir -p /etc/orca && cat > /etc/orca/step-ca-root.crt <<%s\\n%s\\n%s'",
vmid, caDelim, string(caData), caDelim,
))
// Render + write traefik static config inside the LXC.
staticFiles, err := emitter.TraefikEmitter{}.RenderTraefikStaticConfig(emitter.TraefikStaticOpts{})
if err == nil {
for _, f := range staticFiles {
delim := "EOF_TF"
_, _ = runRemote(fmt.Sprintf(
"pct exec %s -- bash -c 'mkdir -p /etc/traefik/dynamic && cat > %s <<%s\\n%s\\n%s'",
vmid, f.Path, delim, f.Content, delim,
))
}
}
// Ensure podman orca-traefik container inside the LXC.
traefikExecFn := func(cmd string) ([]byte, error) {
return runRemote(fmt.Sprintf("pct exec %s -- bash -c %s 2>&1", vmid, shellQuote(cmd)))
}
if err := traefik.EnsureTraefikContainerRemote(ctx, "", traefikExecFn); err != nil {
log.Warn("proxmox.native_ingress_lxc_traefik_failed", "err", err)
}
// Render + apply nft on the PVE host with DNATTarget = LXC IP.
nftFiles, err := emitter.NftEmitter{}.RenderNftConfig(emitter.NftClusterConfig{
DNATTarget: lxcIP,
})
if err == nil {
for _, f := range nftFiles {
nftDelim := "EOF_NF"
_, _ = runRemote(fmt.Sprintf("mkdir -p /etc/nftables.d && cat > %s <<%s\\n%s\\n%s",
f.Path, nftDelim, f.Content, nftDelim))
}
_, _ = runRemote("nft add table inet orca-ingress 2>/dev/null || true")
if out, err := runRemote("nft -f /etc/nftables.d/orca.nft 2>&1"); err != nil {
log.Warn("proxmox.native_ingress_nft_apply_failed", "err", err, "output", string(out))
}
}
log.Info("proxmox.native_ingress_lxc_ok", "vmid", vmid, "ip", lxcIP)
return nil
}
// sshDialer is the dialer used by BootstrapProxmox. It's a package-level
// variable so tests can override it with a fake SSH server.
var sshDialer sshDialerType = defaultSSHDialer{}
+171
View File
@@ -0,0 +1,171 @@
package proxmox
import (
"context"
"crypto/rand"
"fmt"
"log/slog"
"os"
"strings"
"time"
"git.cloudinit.dev/coreci/orca/internal/certpaths"
"git.cloudinit.dev/coreci/orca/internal/emitter"
"git.cloudinit.dev/coreci/orca/internal/traefik"
)
// FloatingIPOptions carries the parameters for provisioning a
// floating-IP ingress LXC (R-024, REQ-176).
type FloatingIPOptions struct {
// FloatingIP is the public IP assigned to the LXC's eth0.
FloatingIP string
// Gateway is the default gateway for the LXC.
Gateway string
// MAC is the MAC address for the LXC's net0 interface.
MAC string
// NetPrefix is the CIDR prefix for the floating IP (8-32).
NetPrefix int
// LXCTemplate is the LXC template (default "ubuntu-24.04").
LXCTemplate string
// VMID is the LXC container ID (default "201" for the ingress LXC).
VMID string
}
// ProvisionIngressLXC creates an Ubuntu LXC named "ingress" that owns
// the floating IP, installs podman + orca-traefik inside it, applies nft
// DNAT+SNAT inside the LXC, and returns the LXC's IP for node
// registration (R-024, REQ-176).
//
// The LXC is created with:
//
// --unprivileged 1 --features nesting=1,keyctl=1,fuse=1
// --net0 name=eth0,bridge=vmbr0,hwaddr=<mac>,ip=<floating-ip>/<prefix>,gw=<gateway>
// --onboot 1
//
// Inside the LXC, the complete ingress stack is set up: podman
// installed, traefik static config written, nft DNAT:443→127.0.0.1:8443
// + postrouting masquerade applied, orca-traefik podman container
// running with --network host.
//
// Idempotent: if the LXC already exists, it is not re-created (C-53).
func ProvisionIngressLXC(ctx context.Context, runRemote func(string) ([]byte, error), opts FloatingIPOptions, log *slog.Logger) error {
template := opts.LXCTemplate
if template == "" {
template = "ubuntu-24.04"
}
vmid := opts.VMID
if vmid == "" {
vmid = "201"
}
if opts.NetPrefix == 0 {
opts.NetPrefix = 24
}
// Validate required fields.
if opts.FloatingIP == "" || opts.Gateway == "" || opts.MAC == "" {
return fmt.Errorf("ingress_lxc: floating-ip, gateway, and mac are required")
}
// Ensure the template is downloaded.
_, _ = runRemote(fmt.Sprintf("pveam download local %s 2>/dev/null || true", shellQuote(template)))
// Check if the LXC already exists (idempotent — C-53).
existOut, _ := runRemote(fmt.Sprintf("pct status %s 2>/dev/null || echo absent", vmid))
existStr := strings.TrimSpace(string(existOut))
if existStr == "absent" {
log.Info("ingress_lxc.creating", "vmid", vmid, "hostname", "ingress", "ip", opts.FloatingIP)
net0 := fmt.Sprintf("name=eth0,bridge=vmbr0,hwaddr=%s,ip=%s/%d,gw=%s",
opts.MAC, opts.FloatingIP, opts.NetPrefix, opts.Gateway)
createCmd := fmt.Sprintf(
"pct create %s local:vztmpl/%s --hostname ingress --unprivileged 1 --features nesting=1,keyctl=1,fuse=1 --net0 %s --onboot 1 --memory 2048 --swap 0 --rootfs local:8 2>&1",
vmid, shellQuote(template), net0,
)
if out, err := runRemote(createCmd); err != nil {
return fmt.Errorf("pct create ingress LXC: %w (output: %s)", err, string(out))
}
if out, err := runRemote(fmt.Sprintf("pct start %s", vmid)); err != nil {
return fmt.Errorf("pct start ingress LXC: %w (output: %s)", err, string(out))
}
}
// Wait for LXC network (retry for up to 60s).
for i := 0; i < 12; i++ {
ipOut, _ := runRemote(fmt.Sprintf("pct exec %s -- hostname -I 2>/dev/null", vmid))
ipStr := strings.TrimSpace(string(ipOut))
if ipStr != "" {
break
}
time.Sleep(5 * time.Second)
}
log.Info("ingress_lxc.network_ready", "vmid", vmid, "ip", opts.FloatingIP)
// Install podman inside the LXC (C-53: idempotent).
_, _ = runRemote(fmt.Sprintf(
"pct exec %s -- bash -c 'command -v podman >/dev/null 2>&1 || (apt-get update -qq && apt-get install -y -qq podman conmon crun fuse-overlayfs nftables 2>&1)' 2>&1",
vmid,
))
// Enable podman-restart.service inside the LXC (research Topic 6).
_, _ = runRemote(fmt.Sprintf("pct exec %s -- systemctl enable --now podman-restart.service 2>/dev/null", vmid))
// Push step-ca root CA into the LXC (C-60: CACertPath).
caPath := certpaths.CACertPath()
caData, caErr := os.ReadFile(caPath)
if caErr != nil {
caData = []byte{}
}
caDelim := "EOF_CA"
_, _ = runRemote(fmt.Sprintf(
"pct exec %s -- bash -c 'mkdir -p /etc/orca && cat > /etc/orca/step-ca-root.crt <<%s\\n%s\\n%s'",
vmid, caDelim, string(caData), caDelim,
))
// Render + write traefik static config inside the LXC (C-58).
staticFiles, err := emitter.TraefikEmitter{}.RenderTraefikStaticConfig(emitter.TraefikStaticOpts{})
if err == nil {
for _, f := range staticFiles {
delim := "EOF_TF"
_, _ = runRemote(fmt.Sprintf(
"pct exec %s -- bash -c 'mkdir -p /etc/traefik/dynamic && cat > %s <<%s\\n%s\\n%s'",
vmid, f.Path, delim, f.Content, delim,
))
}
}
// Render + apply nft DNAT+SNAT INSIDE the LXC (DNATTarget =
// 127.0.0.1 — traefik runs with --network host inside the LXC).
nftFiles, err := emitter.NftEmitter{}.RenderNftConfig(emitter.NftClusterConfig{})
if err == nil {
for _, f := range nftFiles {
delim := "EOF_NF"
_, _ = runRemote(fmt.Sprintf(
"pct exec %s -- bash -c 'mkdir -p /etc/nftables.d && cat > %s <<%s\\n%s\\n%s'",
vmid, f.Path, delim, f.Content, delim,
))
}
_, _ = runRemote(fmt.Sprintf("pct exec %s -- nft add table inet orca-ingress 2>/dev/null || true", vmid))
_, _ = runRemote(fmt.Sprintf("pct exec %s -- nft -f /etc/nftables.d/orca.nft 2>&1", vmid))
}
// Ensure podman orca-traefik container inside the LXC.
traefikExecFn := func(cmd string) ([]byte, error) {
return runRemote(fmt.Sprintf("pct exec %s -- bash -c %s 2>&1", vmid, shellQuote(cmd)))
}
if err := traefik.EnsureTraefikContainerRemote(ctx, "", traefikExecFn); err != nil {
log.Warn("ingress_lxc.traefik_failed", "err", err)
}
log.Info("ingress_lxc.provisioned", "vmid", vmid, "ip", opts.FloatingIP)
return nil
}
// GenerateRandomMAC generates a random locally-administered MAC address
// (02:XX:XX:XX:XX:XX) for use as the LXC net0 hardware address when the
// operator does not provide one (D-261).
func GenerateRandomMAC() (string, error) {
b := make([]byte, 5)
if _, err := rand.Read(b); err != nil {
return "", fmt.Errorf("generate MAC: %w", err)
}
return fmt.Sprintf("02:%02x:%02x:%02x:%02x:%02x", b[0], b[1], b[2], b[3], b[4]), nil
}
@@ -0,0 +1,6 @@
-- REQ-175 / R-024: add ingress_mode column to nodes.
-- Values: '' (legacy/default), 'native' (proxmox native — traefik
-- in LXC), 'floating-ip' (proxmox floating-IP — ingress LXC owns
-- the floating IP). Defaults to empty string for backward
-- compatibility with pre-v0.14 nodes.
ALTER TABLE nodes ADD COLUMN ingress_mode TEXT NOT NULL DEFAULT '';
+202 -52
View File
@@ -1,78 +1,228 @@
package traefik
import (
"context"
"fmt"
"os/exec"
"strings"
)
// DefaultVersion is the traefik version tag for the orca-traefik image.
const DefaultVersion = "v3.3.0"
func downloadURL(version string) string {
return fmt.Sprintf("https://github.com/traefik/traefik/releases/download/%s/traefik_%s_linux_amd64.tar.gz", version, version)
}
// DefaultImage is the full image reference for the orca-traefik container.
// The tag is resolved at runtime from the orca version (or "latest" for
// dev builds). The image is built from Dockerfile.traefik and published
// per release (REQ-171).
const DefaultImage = "git.cloudinit.dev/coreci/orca-traefik"
func InstallLocal(version string) error {
if version == "" {
version = DefaultVersion
}
if _, err := exec.LookPath("traefik"); err == nil {
ensureDirs()
return nil
}
url := downloadURL(version)
cmd := exec.Command("bash", "-c",
fmt.Sprintf(`curl -fsSL %s | tar -xzf - -C /usr/local/bin/ traefik && chmod +x /usr/local/bin/traefik`, url))
if out, err := cmd.CombinedOutput(); err != nil {
return fmt.Errorf("download traefik %s: %w (output: %s)", version, err, string(out))
}
ensureDirs()
writeSystemdUnit()
_ = exec.Command("systemctl", "daemon-reload").Run()
_ = exec.Command("systemctl", "enable", "--now", "orca-traefik").Run()
return nil
}
// ContainerName is the podman container name for the traefik data plane.
const ContainerName = "orca-traefik"
// RemoteExecFunc runs a command on a remote host and returns combined
// output. It is the same signature used by the v0.13 binary installer
// and by the proxmox/linux bootstrap SSH sessions.
type RemoteExecFunc func(cmd string) ([]byte, error)
func InstallRemote(version string, execFn RemoteExecFunc) error {
if version == "" {
version = DefaultVersion
// ImageRef returns the full image:tag reference for the orca-traefik
// container. If version is empty or "dev"/"0.1.0-dev", it falls back to
// "latest" (dev builds don't have a published tag).
func ImageRef(version string) string {
tag := version
if tag == "" || tag == "dev" || tag == "0.1.0-dev" || strings.HasSuffix(tag, "-dev") {
return fmt.Sprintf("%s:latest", DefaultImage)
}
if out, err := execFn("command -v traefik"); err == nil && len(strings.TrimSpace(string(out))) > 0 {
_, _ = execFn("mkdir -p /etc/traefik/dynamic")
tag = strings.TrimPrefix(tag, "v")
return fmt.Sprintf("%s:v%s", DefaultImage, tag)
}
// podmanRunArgs returns the podman run arguments for the traefik
// container. The container uses --network host so traefik binds
// 127.0.0.1:8080/8443 directly on the host (or LXC) loopback. nft
// DNATs public :443/:80 to those loopback ports (R-017/R-024).
//
// Volume mounts (no SELinux :Z flag — research finding Topic 7):
// - /etc/traefik/traefik.yml:ro — static config (overrides baked default; C-58)
// - /etc/traefik/dynamic:ro — dynamic config (orca writes atomically via SSH-push)
// - /etc/orca/step-ca-root.crt:ro — step-ca root CA (for future mTLS; v0.14 uses tls:{})
func podmanRunArgs(imageRef string) []string {
return []string{
"run", "-d",
"--name", ContainerName,
"--restart=unless-stopped",
"--network", "host",
"-v", "/etc/traefik/traefik.yml:/etc/traefik/traefik.yml:ro",
"-v", "/etc/traefik/dynamic:/etc/traefik/dynamic:ro",
"-v", "/etc/orca/step-ca-root.crt:/etc/orca/step-ca-root.crt:ro",
imageRef,
}
}
// EnsureTraefikContainerLocal ensures the orca-traefik podman container
// is running on the local host. It is idempotent:
// 1. If the container is running → no-op.
// 2. If the container exists but is stopped → start it.
// 3. If the container does not exist → pull the image + run it.
//
// C-50: if podman is not installed, attempts apt-get install. If that
// fails, returns an error with install instructions.
//
// C-57: if a legacy v0.13 systemd service exists (orca-traefik.service),
// it is stopped, disabled, and removed before starting the container.
func EnsureTraefikContainerLocal(ctx context.Context, version string) error {
imageRef := ImageRef(version)
if err := ensurePodmanLocal(ctx); err != nil {
return err
}
if err := removeLegacySystemdUnitLocal(ctx); err != nil {
// Non-fatal: legacy unit may not exist on fresh installs.
_ = err
}
if err := ensureDirsLocal(); err != nil {
return fmt.Errorf("traefik: ensure dirs: %w", err)
}
return reconcileContainerLocal(ctx, imageRef)
}
// EnsureTraefikContainerRemote ensures the orca-traefik podman container
// is running on a remote host (via SSH exec). Same idempotent logic as
// EnsureTraefikContainerLocal but over the provided exec function.
func EnsureTraefikContainerRemote(ctx context.Context, version string, execFn RemoteExecFunc) error {
imageRef := ImageRef(version)
if err := ensurePodmanRemote(execFn); err != nil {
return err
}
if err := removeLegacySystemdUnitRemote(execFn); err != nil {
_ = err // non-fatal
}
if _, err := execFn("mkdir -p /etc/traefik/dynamic /etc/orca"); err != nil {
return fmt.Errorf("traefik: ensure remote dirs: %w", err)
}
return reconcileContainerRemote(execFn, imageRef)
}
// ensurePodmanLocal checks if podman is installed locally and attempts
// to install it if absent (C-50).
func ensurePodmanLocal(ctx context.Context) error {
if _, err := exec.LookPath("podman"); err == nil {
return nil
}
url := downloadURL(version)
installCmd := fmt.Sprintf(`curl -fsSL %s | tar -xzf - -C /usr/local/bin/ traefik && chmod +x /usr/local/bin/traefik && mkdir -p /etc/traefik/dynamic`, url)
if out, err := execFn(installCmd); err != nil {
return fmt.Errorf("download traefik on remote: %w (output: %s)", err, string(out))
// Attempt apt-get install (Ubuntu/Debian).
cmd := exec.CommandContext(ctx, "bash", "-c",
"apt-get update -qq && apt-get install -y -qq podman conmon crun fuse-overlayfs 2>&1")
if out, err := cmd.CombinedOutput(); err != nil {
return fmt.Errorf("podman not found and apt-get install failed: %w (output: %s).\nInstall podman manually: apt-get install podman conmon crun fuse-overlayfs", err, string(out))
}
unit := systemdUnitContent()
_, _ = execFn(fmt.Sprintf(`echo '%s' > /etc/systemd/system/orca-traefik.service`, unit))
_, _ = execFn("systemctl daemon-reload && systemctl enable --now orca-traefik")
return nil
}
func ensureDirs() {
_ = exec.Command("mkdir", "-p", "/etc/traefik/dynamic").Run()
// ensurePodmanRemote checks if podman is installed on the remote host
// and attempts to install it if absent (C-50).
func ensurePodmanRemote(execFn RemoteExecFunc) error {
if out, err := execFn("command -v podman"); err == nil && len(strings.TrimSpace(string(out))) > 0 {
return nil
}
// Attempt apt-get install on the remote host.
cmd := "apt-get update -qq && apt-get install -y -qq podman conmon crun fuse-overlayfs 2>&1"
if out, err := execFn(cmd); err != nil {
return fmt.Errorf("podman not found on remote and apt-get install failed: %w (output: %s)", err, string(out))
}
return nil
}
func writeSystemdUnit() {
_ = exec.Command("bash", "-c", fmt.Sprintf(`echo '%s' > /etc/systemd/system/orca-traefik.service`, systemdUnitContent())).Run()
// removeLegacySystemdUnitLocal stops, disables, and removes the legacy
// v0.13 orca-traefik.service systemd unit + /usr/local/bin/traefik
// binary (C-57). Idempotent — no-op if the unit doesn't exist.
func removeLegacySystemdUnitLocal(ctx context.Context) error {
// Check if the legacy unit exists.
if _, err := exec.CommandContext(ctx, "systemctl", "is-active", "orca-traefik.service").CombinedOutput(); err == nil {
// Unit is active or exists — stop + disable it.
_ = exec.CommandContext(ctx, "systemctl", "stop", "orca-traefik.service").Run()
_ = exec.CommandContext(ctx, "systemctl", "disable", "orca-traefik.service").Run()
}
// Remove the unit file and binary.
_ = exec.CommandContext(ctx, "rm", "-f", "/etc/systemd/system/orca-traefik.service").Run()
_ = exec.CommandContext(ctx, "rm", "-f", "/usr/local/bin/traefik").Run()
_ = exec.CommandContext(ctx, "systemctl", "daemon-reload").Run()
return nil
}
func systemdUnitContent() string {
return `[Unit]
Description=Orca Traefik Data Plane
After=network.target
[Service]
Type=simple
ExecStart=/usr/local/bin/traefik --configFile=/etc/traefik/traefik.yml
Restart=on-failure
RestartSec=5s
[Install]
WantedBy=multi-user.target`
// removeLegacySystemdUnitRemote is the remote SSH variant (C-57).
func removeLegacySystemdUnitRemote(execFn RemoteExecFunc) error {
cmd := `systemctl is-active orca-traefik.service 2>/dev/null && systemctl stop orca-traefik.service 2>/dev/null; systemctl disable orca-traefik.service 2>/dev/null; rm -f /etc/systemd/system/orca-traefik.service /usr/local/bin/traefik; systemctl daemon-reload 2>/dev/null; true`
_, _ = execFn(cmd)
return nil
}
// ensureDirsLocal creates /etc/traefik/dynamic and /etc/orca locally.
func ensureDirsLocal() error {
if err := exec.Command("mkdir", "-p", "/etc/traefik/dynamic", "/etc/orca").Run(); err != nil {
return fmt.Errorf("mkdir: %w", err)
}
return nil
}
// reconcileContainerLocal implements the idempotent pull+run logic
// locally (C-50).
func reconcileContainerLocal(ctx context.Context, imageRef string) error {
// Check if the container is already running.
out, err := exec.CommandContext(ctx, "podman", "inspect", "--format", "{{.State.Running}}", ContainerName).CombinedOutput()
if err == nil {
v := strings.TrimSpace(string(out))
if v == "true" {
return nil // already running
}
// Container exists but is stopped — start it.
if _, err := exec.CommandContext(ctx, "podman", "start", ContainerName).CombinedOutput(); err != nil {
return fmt.Errorf("podman start %s: %w", ContainerName, err)
}
return nil
}
// Container does not exist — pull + run.
if out, err := exec.CommandContext(ctx, "podman", "pull", imageRef).CombinedOutput(); err != nil {
return fmt.Errorf("podman pull %s: %w (output: %s)", imageRef, err, string(out))
}
args := append([]string{}, podmanRunArgs(imageRef)...)
if out, err := exec.CommandContext(ctx, "podman", args...).CombinedOutput(); err != nil {
return fmt.Errorf("podman run: %w (output: %s)", err, string(out))
}
// Enable podman-restart.service for reboot persistence (research Topic 6).
_ = exec.CommandContext(ctx, "systemctl", "enable", "--now", "podman-restart.service").Run()
return nil
}
// reconcileContainerRemote implements the idempotent pull+run logic
// over SSH exec.
func reconcileContainerRemote(execFn RemoteExecFunc, imageRef string) error {
// Check if the container is already running.
out, err := execFn(fmt.Sprintf("podman inspect --format '{{.State.Running}}' %s 2>/dev/null", ContainerName))
if err == nil {
v := strings.TrimSpace(string(out))
if v == "true" {
return nil // already running
}
// Container exists but is stopped — start it.
if _, err := execFn(fmt.Sprintf("podman start %s 2>/dev/null", ContainerName)); err != nil {
return fmt.Errorf("podman start %s: %w", ContainerName, err)
}
return nil
}
// Container does not exist — pull + run.
if out, err := execFn(fmt.Sprintf("podman pull %s", shellQuote(imageRef))); err != nil {
return fmt.Errorf("podman pull %s: %w (output: %s)", imageRef, err, string(out))
}
runArgs := strings.Join(podmanRunArgs(imageRef), " ")
if out, err := execFn(fmt.Sprintf("podman %s", runArgs)); err != nil {
return fmt.Errorf("podman run: %w (output: %s)", err, string(out))
}
// Enable podman-restart.service for reboot persistence (research Topic 6).
_, _ = execFn("systemctl enable --now podman-restart.service 2>/dev/null || true")
return nil
}
// shellQuote wraps a string in single quotes for shell-safe usage.
func shellQuote(s string) string {
return "'" + strings.ReplaceAll(s, "'", "'\\''") + "'"
}
+166
View File
@@ -0,0 +1,166 @@
package traefik
import (
"context"
"strings"
"testing"
)
func TestImageRef(t *testing.T) {
tests := []struct {
version string
want string
}{
{"v0.13.1", "git.cloudinit.dev/coreci/orca-traefik:v0.13.1"},
{"0.13.1", "git.cloudinit.dev/coreci/orca-traefik:v0.13.1"},
{"", "git.cloudinit.dev/coreci/orca-traefik:latest"},
{"dev", "git.cloudinit.dev/coreci/orca-traefik:latest"},
{"0.1.0-dev", "git.cloudinit.dev/coreci/orca-traefik:latest"},
{"v1.2.3-dev", "git.cloudinit.dev/coreci/orca-traefik:latest"},
}
for _, tt := range tests {
got := ImageRef(tt.version)
if got != tt.want {
t.Errorf("ImageRef(%q) = %q, want %q", tt.version, got, tt.want)
}
}
}
func TestPodmanRunArgs(t *testing.T) {
args := podmanRunArgs("git.cloudinit.dev/coreci/orca-traefik:v0.13.1")
joined := strings.Join(args, " ")
checks := []string{
"run -d",
"--name orca-traefik",
"--restart=unless-stopped",
"--network host",
"/etc/traefik/traefik.yml:/etc/traefik/traefik.yml:ro",
"/etc/traefik/dynamic:/etc/traefik/dynamic:ro",
"/etc/orca/step-ca-root.crt:/etc/orca/step-ca-root.crt:ro",
"git.cloudinit.dev/coreci/orca-traefik:v0.13.1",
}
for _, c := range checks {
if !strings.Contains(joined, c) {
t.Errorf("podmanRunArgs missing %q\nfull: %s", c, joined)
}
}
// Ensure no :Z flag (research Topic 7)
if strings.Contains(joined, ":Z") {
t.Errorf("podmanRunArgs should NOT contain :Z SELinux flag\nfull: %s", joined)
}
// Ensure --restart=always is NOT used (research Topic 6)
if strings.Contains(joined, "--restart=always") {
t.Errorf("podmanRunArgs should use --restart=unless-stopped, not --restart=always\nfull: %s", joined)
}
}
func TestEnsureTraefikContainerRemote_ContainerRunning(t *testing.T) {
var cmds []string
execFn := func(cmd string) ([]byte, error) {
cmds = append(cmds, cmd)
if strings.Contains(cmd, "podman inspect") {
return []byte("true\n"), nil
}
return []byte(""), nil
}
err := EnsureTraefikContainerRemote(context.Background(), "v0.13.1", execFn)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
// Should have checked inspect and found it running — no pull/run.
if len(cmds) < 1 {
t.Fatal("expected at least 1 command (inspect)")
}
for _, c := range cmds {
if strings.Contains(c, "podman pull") {
t.Errorf("should not pull when container is running: %s", c)
}
if strings.Contains(c, "podman run") {
t.Errorf("should not run when container is running: %s", c)
}
}
}
func TestEnsureTraefikContainerRemote_ContainerStopped(t *testing.T) {
var cmds []string
execFn := func(cmd string) ([]byte, error) {
cmds = append(cmds, cmd)
if strings.Contains(cmd, "podman inspect") {
return []byte("false\n"), nil // stopped
}
return []byte(""), nil
}
err := EnsureTraefikContainerRemote(context.Background(), "v0.13.1", execFn)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
// Should have started the container.
foundStart := false
for _, c := range cmds {
if strings.Contains(c, "podman start orca-traefik") {
foundStart = true
}
}
if !foundStart {
t.Errorf("expected 'podman start orca-traefik' when container is stopped\ncommands: %v", cmds)
}
}
func TestEnsureTraefikContainerRemote_ContainerAbsent(t *testing.T) {
var cmds []string
execFn := func(cmd string) ([]byte, error) {
cmds = append(cmds, cmd)
if strings.Contains(cmd, "podman inspect") {
return nil, &execError{"inspect failed: no such container"}
}
return []byte(""), nil
}
err := EnsureTraefikContainerRemote(context.Background(), "v0.13.1", execFn)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
// Should have pulled and run.
foundPull := false
foundRun := false
for _, c := range cmds {
if strings.Contains(c, "podman pull") {
foundPull = true
}
if strings.Contains(c, "podman run -d") {
foundRun = true
}
}
if !foundPull {
t.Errorf("expected 'podman pull' when container is absent\ncommands: %v", cmds)
}
if !foundRun {
t.Errorf("expected 'podman run -d' when container is absent\ncommands: %v", cmds)
}
}
func TestEnsureTraefikContainerRemote_LegacySystemdRemoval(t *testing.T) {
var cmds []string
execFn := func(cmd string) ([]byte, error) {
cmds = append(cmds, cmd)
if strings.Contains(cmd, "podman inspect") {
return []byte("true\n"), nil // container running
}
return []byte(""), nil
}
_ = EnsureTraefikContainerRemote(context.Background(), "v0.13.1", execFn)
// Should include legacy systemd unit removal command (C-57).
foundLegacyRemoval := false
for _, c := range cmds {
if strings.Contains(c, "orca-traefik.service") && strings.Contains(c, "stop") {
foundLegacyRemoval = true
}
}
if !foundLegacyRemoval {
t.Errorf("expected legacy systemd unit removal command (C-57)\ncommands: %v", cmds)
}
}
// execError is a simple error type for testing.
type execError struct{ msg string }
func (e *execError) Error() string { return e.msg }
+14
View File
@@ -30,6 +30,20 @@ fi
while read local_ref local_sha remote_ref remote_sha; do
branch="${remote_ref#refs/heads/}"
if [ -z "$branch" ] || [ "$branch" = "HEAD" ]; then
# Check if this is a tag push (refs/tags/*)
tag="${remote_ref#refs/tags/}"
if [ -n "$tag" ] && [ "$tag" != "$remote_ref" ]; then
echo "→ Triggering CoreCI for tag: $tag (${local_sha:0:7})"
payload=$(printf '{"repo":"coreci/orca","branch":"%s","ref":"%s"}' "$tag" "$local_sha")
if command -v curl >/dev/null 2>&1; then
curl -fsS -X POST "${CORECI_URL}/api/pipeline/run" \
-H "Authorization: token ${GITEA_TOKEN}" \
-H "Content-Type: application/json" \
-d "$payload" >/dev/null 2>&1 \
&& echo " ✓ CoreCI triggered" \
|| echo " (CoreCI trigger failed; Gitea Actions webhook is secondary path)"
fi
fi
continue
fi
echo "→ Triggering CoreCI for branch: $branch (${local_sha:0:7})"
+25 -2
View File
@@ -145,8 +145,8 @@ assert "34 type_linux_available" \
assert "35 status_deprecated" \
'$ORCA status 2>&1 | grep -qi "deprecated"'
assert "36 traefik_installed" \
'systemctl is-active orca-traefik 2>/dev/null | grep -q "active" || exit 77'
assert "36 traefik_container_running" \
'podman inspect --format "{{.State.Running}}" orca-traefik 2>/dev/null | grep -q "true" || exit 77'
assert "37 known_hosts_exists" \
'test -f "$ORCA_HOME/known_hosts" || test -f "$ORCA_HOME/cluster/known_hosts"'
@@ -154,6 +154,29 @@ assert "37 known_hosts_exists" \
assert "38 master_key_exists" \
'test -f "$ORCA_HOME/cluster/master.key" || test -f "$ORCA_HOME/cluster/master.key.sealed"'
# --- v0.14 ingress bootstrap assertions (R-024) ---
assert "40 ingress_nft_table" \
'nft list table inet orca-ingress 2>/dev/null | grep -q "chain prerouting"'
assert "41 ingress_nft_dnat" \
'nft list table inet orca-ingress 2>/dev/null | grep -q "dnat to"'
assert "42 ingress_nft_snat" \
'nft list table inet orca-ingress 2>/dev/null | grep -q "masquerade"'
assert "43 ingress_dynamic_dir" \
'test -d /etc/traefik/dynamic'
assert "44 ingress_step_ca" \
'test -f /etc/orca/step-ca-root.crt'
assert "45 ingress_traefik_yml" \
'test -f /etc/traefik/traefik.yml'
assert "46 ingress_doctor_pass" \
'$ORCA doctor ingress 2>&1 | grep -q "PASS"'
# --- Report ---
echo "=========================================="
+163
View File
@@ -0,0 +1,163 @@
package tests
import (
"log/slog"
"strings"
"testing"
"git.cloudinit.dev/coreci/orca/internal/emitter"
"git.cloudinit.dev/coreci/orca/internal/jobspec"
"git.cloudinit.dev/coreci/orca/internal/proxmox"
"git.cloudinit.dev/coreci/orca/internal/traefik"
)
// TestNftEmitter_PostroutingAndDNATTarget (REQ-173) verifies the nft
// emitter renders the postrouting masquerade chain and supports
// DNATTarget substitution.
func TestNftEmitter_PostroutingAndDNATTarget(t *testing.T) {
files, err := emitter.NftEmitter{}.RenderNftConfig(emitter.NftClusterConfig{
DNATTarget: "10.99.0.10",
EnableSNAT: true,
})
if err != nil {
t.Fatalf("RenderNftConfig: %v", err)
}
c := files[0].Content
if !strings.Contains(c, "chain postrouting") {
t.Errorf("missing postrouting chain:\n%s", c)
}
if !strings.Contains(c, "masquerade") {
t.Errorf("missing masquerade rule:\n%s", c)
}
if !strings.Contains(c, "dnat to 10.99.0.10:8443") {
t.Errorf("missing custom DNAT target:\n%s", c)
}
}
// TestNftEmitter_PriorityMinus10 (research Topic 2) verifies the input
// and forward chains use priority -10 for pve-firewall coexistence.
func TestNftEmitter_PriorityMinus10(t *testing.T) {
files, _ := emitter.NftEmitter{}.RenderNftConfig(emitter.NftClusterConfig{})
c := files[0].Content
if !strings.Contains(c, "hook input priority -10;") {
t.Errorf("input chain should use priority -10:\n%s", c)
}
if !strings.Contains(c, "hook forward priority -10;") {
t.Errorf("forward chain should use priority -10:\n%s", c)
}
}
// TestTraefikEmitter_TLSModel (REQ-172) verifies the dynamic config
// emits tls: {} and does NOT contain certResolver (dropped in v0.14).
func TestTraefikEmitter_TLSModel(t *testing.T) {
spec := &jobspec.WorkloadSpec{
Name: "test-svc",
Kind: "Service",
Ports: []jobspec.PortSpec{{Name: "http"}},
}
node := &emitter.Node{
Hostname: "test-node",
}
files, err := emitter.TraefikEmitter{}.Render(spec, node)
if err != nil {
t.Fatalf("Render: %v", err)
}
c := files[0].Content
if !strings.Contains(c, "tls: {}") {
t.Errorf("missing tls: {} (v0.14 model):\n%s", c)
}
if strings.Contains(c, "certResolver: orca") {
t.Errorf("certResolver: orca should be removed (v0.14):\n%s", c)
}
}
// TestTraefikImageRef verifies the image reference resolution for the
// orca-traefik podman container.
func TestTraefikImageRef(t *testing.T) {
ref := traefik.ImageRef("v0.13.7")
want := "git.cloudinit.dev/coreci/orca-traefik:v0.13.7"
if ref != want {
t.Errorf("ImageRef(v0.13.7) = %q, want %q", ref, want)
}
// Dev build falls back to latest.
ref = traefik.ImageRef("dev")
if ref != "git.cloudinit.dev/coreci/orca-traefik:latest" {
t.Errorf("ImageRef(dev) = %q, want latest", ref)
}
}
// TestProxmox_FloatingIP_LXC_ProvisioningCommands (REQ-176) verifies
// the ProvisionIngressLXC function sends the correct pct create
// command with the right net0 parameters.
func TestProxmox_FloatingIP_LXC_ProvisioningCommands(t *testing.T) {
var cmds []string
execFn := func(cmd string) ([]byte, error) {
cmds = append(cmds, cmd)
// Simulate: pct status returns "absent" on first call, then OK.
if strings.Contains(cmd, "pct status 201") {
return []byte("absent\n"), nil
}
if strings.Contains(cmd, "pct create") {
return []byte(""), nil
}
if strings.Contains(cmd, "pct start 201") {
return []byte(""), nil
}
if strings.Contains(cmd, "hostname -I") {
return []byte("203.0.113.10\n"), nil
}
return []byte(""), nil
}
err := proxmox.ProvisionIngressLXC(nil, execFn, proxmox.FloatingIPOptions{
FloatingIP: "203.0.113.10",
Gateway: "203.0.113.1",
MAC: "02:01:02:03:04:05",
NetPrefix: 24,
LXCTemplate: "ubuntu-24.04",
}, slog.Default())
if err != nil {
t.Fatalf("ProvisionIngressLXC: %v", err)
}
// Verify pct create has the right net0 params.
foundCreate := false
for _, c := range cmds {
if strings.Contains(c, "pct create") {
foundCreate = true
if !strings.Contains(c, "hostname ingress") {
t.Errorf("pct create missing hostname ingress: %s", c)
}
if !strings.Contains(c, "hwaddr=02:01:02:03:04:05") {
t.Errorf("pct create missing hwaddr: %s", c)
}
if !strings.Contains(c, "ip=203.0.113.10/24") {
t.Errorf("pct create missing ip: %s", c)
}
if !strings.Contains(c, "gw=203.0.113.1") {
t.Errorf("pct create missing gw: %s", c)
}
if !strings.Contains(c, "nesting=1,keyctl=1,fuse=1") {
t.Errorf("pct create missing features (research Topic 3): %s", c)
}
}
}
if !foundCreate {
t.Errorf("pct create command not sent\ncommands: %v", cmds)
}
}
// TestProxmox_GenerateRandomMAC (D-261) verifies MAC generation produces
// a valid locally-administered MAC.
func TestProxmox_GenerateRandomMAC(t *testing.T) {
mac, err := proxmox.GenerateRandomMAC()
if err != nil {
t.Fatalf("GenerateRandomMAC: %v", err)
}
if !strings.HasPrefix(mac, "02:") {
t.Errorf("MAC should start with 02: (locally administered): %s", mac)
}
// Verify it's 6 octets.
parts := strings.Split(mac, ":")
if len(parts) != 6 {
t.Errorf("MAC should have 6 octets: %s", mac)
}
}