Compare commits
7 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 93ac4bda66 | |||
| 454040fdd1 | |||
| c95bd73e42 | |||
| ecdba833d9 | |||
| 6e65eadaa5 | |||
| 1b7aac71f6 | |||
| ea42a17474 |
@@ -872,3 +872,70 @@ scheduler.Schedule(spec, nodes) → emitter.Render(unit) → sshpush.Deploy(targ
|
||||
- IPv6 `net.JoinHostPort` in all SSH dial paths.
|
||||
- Explicit timeouts on all SSH commands.
|
||||
- Root SIGINT/SIGTERM handler for clean exit on non-watch commands.
|
||||
|
||||
## v0.14 Deltas — Ingress Bootstrap Completeness (R-024)
|
||||
|
||||
### R-024: Traefik as Podman Container
|
||||
|
||||
Traefik runs exclusively as a podman container, deployed from the
|
||||
custom `orca-traefik` image (published per release via `Dockerfile.traefik`
|
||||
+ `scripts/release.sh` + `.coreci.yml container-publish-traefik`).
|
||||
|
||||
The v0.13 binary+systemd install (`internal/traefik/install.go`) is
|
||||
replaced by an idempotent podman container reconciler
|
||||
(`EnsureTraefikContainerLocal`/`Remote`). The container runs with
|
||||
`--network host`, `--restart=unless-stopped`, and volume mounts for
|
||||
`traefik.yml` (static config), `dynamic` (dynamic config), and
|
||||
`step-ca-root.crt` (future mTLS). No SELinux `:Z` flag.
|
||||
|
||||
### Three Ingress Topologies
|
||||
|
||||
1. **Linux** (`orca init` / `orca node join --type linux`):
|
||||
host → nft DNAT → podman traefik (host network).
|
||||
`internal/ingress/bootstrap.go` → `BootstrapLocalIngress` /
|
||||
`BootstrapRemoteIngress`.
|
||||
|
||||
2. **Proxmox Native** (`--ingress-mode native`, default):
|
||||
PVE host → nft DNAT (target = LXC bridge IP) → LXC
|
||||
(`--features nesting=1,keyctl=1,fuse=1`) → podman traefik.
|
||||
`internal/proxmox/bootstrap.go` → `provisionNativeIngressLXC`.
|
||||
|
||||
3. **Proxmox Floating-IP** (`--ingress-mode floating-ip`):
|
||||
LXC owns the floating IP (`net0 bridge=vmbr0,hwaddr=<mac>,
|
||||
ip=<floating-ip>/<prefix>,gw=<gateway>`) → nft inside LXC →
|
||||
podman traefik. The ingress LXC is registered as a `linux` node
|
||||
(name=`ingress`) so `orca job run` pushes traefik dynamic config.
|
||||
`internal/proxmox/ingress_lxc.go` → `ProvisionIngressLXC`.
|
||||
|
||||
### nft Emitter Changes
|
||||
|
||||
`internal/emitter/nft.go`:
|
||||
- `DNATTarget` field (C-51: validated via `net.ParseIP`). Default
|
||||
`127.0.0.1`; proxmox native uses LXC bridge IP.
|
||||
- `EnableSNAT` field + postrouting masquerade chain: `ip saddr
|
||||
127.0.0.0/8 oifname != "lo" masquerade` (research Topic 1).
|
||||
- Input/forward chain priority shifted from `filter` (=0) to `-10`
|
||||
(research Topic 2: pve-firewall coexistence — avoids same-priority
|
||||
undefined evaluation order).
|
||||
|
||||
### TLS Model
|
||||
|
||||
v0.14 drops `certResolver: orca` from the dynamic config (traefik v3.3
|
||||
only supports `acme`/`tailscale` resolvers, not CA-file-based). The
|
||||
dynamic config emits `tls: {}` (traefik default cert). Real mTLS via
|
||||
`tls.certificates` + `tls.options.default.clientAuth.caFiles` is
|
||||
deferred to v0.15 (grill G-003, confidence 0.55 < 0.60).
|
||||
|
||||
### Migration 0009
|
||||
|
||||
`ALTER TABLE nodes ADD COLUMN ingress_mode TEXT NOT NULL DEFAULT '';`
|
||||
Values: `""` (legacy), `"native"`, `"floating-ip"`. `IngressMode` field
|
||||
on `model.Node`.
|
||||
|
||||
### New CLI
|
||||
|
||||
- `orca doctor ingress` — verifies podman container running, nft
|
||||
DNAT+SNAT, dynamic dir, step-ca root CA.
|
||||
- `--ingress-mode` flag on `orca node join --type proxmox`.
|
||||
- `--floating-ip`, `--gateway`, `--mac`, `--net-prefix` flags for
|
||||
floating-IP mode.
|
||||
|
||||
@@ -1,24 +1,17 @@
|
||||
{
|
||||
"phase": 0,
|
||||
"stage": "complete",
|
||||
"milestone": "v0.14",
|
||||
"milestone_slug": "ingress-bootstrap",
|
||||
"stage": "plan",
|
||||
"milestone": "v0.15",
|
||||
"milestone_slug": "ci-release-pipeline",
|
||||
"phase_role": "pre_execution",
|
||||
"attempts": 0,
|
||||
"updated_at": "2026-08-10T18:27:00Z",
|
||||
"updated_at": "2026-08-10T20:55:00Z",
|
||||
"milestone_complete": false,
|
||||
"previous_milestone": "v0.13",
|
||||
"phases_shipped": ["P0"],
|
||||
"tags_shipped": ["v0.13.0"],
|
||||
"previous_milestone": "v0.14",
|
||||
"phases_shipped": [],
|
||||
"tags_shipped": [],
|
||||
"requirements": {
|
||||
"covered": [],
|
||||
"covered": [182],
|
||||
"partial": []
|
||||
},
|
||||
"binding_conditions": ["C-50","C-51","C-52","C-53","C-54","C-55","C-56","C-57","C-58","C-59","C-60","C-61"],
|
||||
"load_bearing_rule": "R-024",
|
||||
"ship": {
|
||||
"tag": "v0.13.0",
|
||||
"merged_to_milestone": true,
|
||||
"release_created": true
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
# CLARIFY + RESEARCH + PLAN v0.15: CI Release Pipeline Fix
|
||||
|
||||
## Decisions
|
||||
|
||||
| ID | Decision | Rationale | Confidence |
|
||||
|----|----------|-----------|------------|
|
||||
| D-264 | Secret name = `PAT_TOKEN` (not `GITEA_PAT`) | Gitea reserves `GITEA_` prefix for built-in secrets | 1.0 (validated) |
|
||||
| D-265 | Use `tea actions secrets create` CLI | Operator instruction: no API | 1.0 (validated) |
|
||||
| D-266 | Container publishing in Gitea Actions, not CoreCI | CoreCI's podman executor appends `sh -c` which conflicts with kaniko's `/kaniko/executor` entrypoint. Gitea Actions `container:` supports `options: --entrypoint` | 0.95 |
|
||||
| D-267 | kaniko `executor:debug` image | Includes `/bin/sh`; Gitea Actions can override entrypoint to `/bin/sh` then run kaniko via shell | 0.90 |
|
||||
| D-268 | `coreci run` for validate/build/test/release (tarball); Gitea Actions for container publishing | Clean separation: CoreCI owns the pipeline, Gitea Actions owns the trigger + container publish | 0.95 |
|
||||
|
||||
## Research: CoreCI podman executor entrypoint issue
|
||||
|
||||
CoreCI's `internal/runner/podman_executor.go:48-51`:
|
||||
```go
|
||||
args = append(args, image) // e.g. gcr.io/kaniko-project/executor:debug
|
||||
if job.Invoke != "" {
|
||||
args = append(args, "sh", "-c", job.Invoke)
|
||||
}
|
||||
```
|
||||
This produces: `podman run ... <image> sh -c "<commands>"`
|
||||
With kaniko:debug (entrypoint `/kaniko/executor`), the actual command is:
|
||||
`/kaniko/executor sh -c "<commands>"` — kaniko fails (sh is not a kaniko flag).
|
||||
|
||||
**Conclusion**: kaniko cannot be used as a CoreCI step image. Container
|
||||
publishing must move to the Gitea Actions workflow, which supports
|
||||
`container: options: --entrypoint /bin/sh` to override the entrypoint.
|
||||
|
||||
## Plan
|
||||
|
||||
### Phase 1 (only execution phase)
|
||||
|
||||
**Files to create/modify:**
|
||||
|
||||
1. `.gitea/workflows/release.yml` — Gitea Actions workflow:
|
||||
- `on: push: tags: ['v*']`
|
||||
- Job 1 `ci`: checkout + install Go + install coreci + `coreci run`
|
||||
(executes validate/build/test/release from .coreci.yml)
|
||||
- Job 2 `container-orca`: checkout + kaniko build+push orca image
|
||||
(needs job 1; uses `container: gcr.io/kaniko-project/executor:debug`
|
||||
with `options: --entrypoint /bin/sh`)
|
||||
- Job 3 `container-traefik`: checkout + kaniko build+push orca-traefik image
|
||||
(needs job 1; same kaniko approach)
|
||||
|
||||
2. `.coreci.yml` — remove `container-publish` and `container-publish-traefik`
|
||||
steps (they now live in the Gitea Actions workflow). Keep the
|
||||
`gitea-release` step (tarball + Gitea release).
|
||||
|
||||
3. `scripts/trigger_coreci.sh` — add tag ref handling (or document that
|
||||
Gitea Actions is the trigger; the hook is for branch-push CI only).
|
||||
@@ -411,15 +411,15 @@ node type.
|
||||
|
||||
| ID | Requirement | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-171 | `Dockerfile.traefik` + release pipeline: build + publish `git.cloudinit.dev/coreci/orca-traefik:<version>` alongside the orca image per release; `.coreci.yml` `container-publish-traefik` step; image bakes default `traefik.yml` (entrypoints `websecure` 127.0.0.1:8443, `web` 127.0.0.1:8080, `traefik` 127.0.0.1:8081 + file provider watching `/etc/traefik/dynamic` + json log/accessLog); host-side `/etc/traefik/traefik.yml` mounted `:ro` overrides baked config (preserves `traefik-on-public-ip` opt-out REQ-100); no `certificatesResolvers` (traefik v3.3 only supports acme/tailscale); `tls: {}` in dynamic config for v0.14 (real mTLS deferred to v0.15) | Critical | **v0.14 P1** | pending |
|
||||
| REQ-172 | Replace `internal/traefik/install.go` binary+systemd install with a podman-container reconciler: `podman pull orca-traefik:<tag>` + `podman run -d --restart=unless-stopped --network host --name orca-traefik -v /etc/traefik/traefik.yml:/etc/traefik/traefik.yml:ro -v /etc/traefik/dynamic:/etc/traefik/dynamic:ro -v /etc/orca/step-ca-root.crt:/etc/orca/step-ca-root.crt:ro <image>`; idempotent (pull+run if absent, start if stopped); install podman if absent (C-50); v0.13→v0.14 upgrade: detect+stop+disable+remove legacy `orca-traefik.service` + `/usr/local/bin/traefik` (C-57); works locally + over SSH-push; enable `podman-restart.service`; remove systemd unit generation | Critical | **v0.14 P2** | pending |
|
||||
| REQ-173 | nft SNAT+DNAT ruleset render+apply: extend `internal/emitter/nft.go` with postrouting masquerade chain; new `internal/ingress/bootstrap.go` renders `orca.nft` + applies `nft -f` + ensures `/etc/traefik/dynamic` dir + pushes step-ca root CA + invokes podman traefik reconciler; wired into `orca init` (localhost lead) | Critical | **v0.14 P3** | pending |
|
||||
| REQ-174 | Remote ingress bootstrap via SSH-push for `orca node join --type linux`: push step-ca root CA, render+apply nft remotely, invoke podman traefik reconciler remotely; register node as `linux` | Critical | **v0.14 P4** | pending |
|
||||
| REQ-175 | Proxmox native ingress mode (`--ingress-mode native`, default): on PVE host, render+apply nft (vmbr-compatible, separate `orca-ingress` table avoids pve-firewall conflict); create unprivileged LXC with `--features nesting=1,keyctl=1` running podman+orca-traefik; push step-ca root CA into LXC; nft DNAT target = LXC bridge IP; register PVE host as `proxmox` node; add `IngressMode` field to `model.Node` + schema migration | Critical | **v0.14 P5** | pending |
|
||||
| REQ-176 | Proxmox floating-IP mode (`--ingress-mode floating-ip --floating-ip --gateway --mac [--net-prefix]`): `pct create` Ubuntu LXC named `ingress` with `net0 bridge=vmbr0,hwaddr=<mac>,ip=<floating-ip>/<prefix>,gw=<gateway> --features nesting=1,keyctl=1 --onboot 1`; install podman + run orca-traefik inside LXC; apply nft DNAT+SNAT inside LXC; register LXC as managed `linux` node (name=`ingress`, addr=`<floating-ip>:8443`); interactive prompt for params when flags absent + not `--json`; validate IP/MAC/gateway; PVE host also registered as `proxmox` for workload dispatch | Critical | **v0.14 P6** | pending |
|
||||
| REQ-177 | `orca doctor ingress [--peer]`: verify orca-traefik container running (`podman inspect`), nft DNAT+SNAT applied, `/etc/traefik/dynamic` exists, step-ca root CA mounted; extend `scripts/uat-signoff.sh` with ingress assertions (40 podman_traefik, 41 nft_dnat_snat, 42 linux_worker, 43 proxmox_native_lxc / floating_ip_lxc) | High | **v0.14 P7** | pending |
|
||||
| REQ-178 | Docs: `docs/cli.md` (`--ingress-mode` + floating-IP flags + `doctor ingress`), `docs/uat.md` (native + floating-IP topologies), `docs/ingress.md` (podman-traefik image + volume mounts + certResolver), `docs/docker.md` (orca-traefik image), `ARCHITECTURE.md` (R-024 + ingress bootstrap section) | High | **v0.14 P7** | pending |
|
||||
| REQ-179 | Integration tests: hermetic harness fakes SSH; asserts init→podman traefik running + nft applied; linux join→remote podman+nft; proxmox native→LXC created with nesting + podman traefik; floating-ip→`pct create` with correct net0 args + LXC registered as linux node; release.sh builds orca-traefik image (Dockerfile.traefik parses) | Critical | **v0.14 P7** | pending |
|
||||
| REQ-171 | `Dockerfile.traefik` + release pipeline: build + publish `git.cloudinit.dev/coreci/orca-traefik:<version>` alongside the orca image per release; `.coreci.yml` `container-publish-traefik` step; image bakes default `traefik.yml` (entrypoints `websecure` 127.0.0.1:8443, `web` 127.0.0.1:8080, `traefik` 127.0.0.1:8081 + file provider watching `/etc/traefik/dynamic` + json log/accessLog); host-side `/etc/traefik/traefik.yml` mounted `:ro` overrides baked config (preserves `traefik-on-public-ip` opt-out REQ-100); no `certificatesResolvers` (traefik v3.3 only supports acme/tailscale); `tls: {}` in dynamic config for v0.14 (real mTLS deferred to v0.15) | Critical | **v0.14 P1** | complete |
|
||||
| REQ-172 | Replace `internal/traefik/install.go` binary+systemd install with a podman-container reconciler: `podman pull orca-traefik:<tag>` + `podman run -d --restart=unless-stopped --network host --name orca-traefik -v /etc/traefik/traefik.yml:/etc/traefik/traefik.yml:ro -v /etc/traefik/dynamic:/etc/traefik/dynamic:ro -v /etc/orca/step-ca-root.crt:/etc/orca/step-ca-root.crt:ro <image>`; idempotent (pull+run if absent, start if stopped); install podman if absent (C-50); v0.13→v0.14 upgrade: detect+stop+disable+remove legacy `orca-traefik.service` + `/usr/local/bin/traefik` (C-57); works locally + over SSH-push; enable `podman-restart.service`; remove systemd unit generation | Critical | **v0.14 P2** | complete |
|
||||
| REQ-173 | nft SNAT+DNAT ruleset render+apply: extend `internal/emitter/nft.go` with postrouting masquerade chain; new `internal/ingress/bootstrap.go` renders `orca.nft` + applies `nft -f` + ensures `/etc/traefik/dynamic` dir + pushes step-ca root CA + invokes podman traefik reconciler; wired into `orca init` (localhost lead) | Critical | **v0.14 P3** | complete |
|
||||
| REQ-174 | Remote ingress bootstrap via SSH-push for `orca node join --type linux`: push step-ca root CA, render+apply nft remotely, invoke podman traefik reconciler remotely; register node as `linux` | Critical | **v0.14 P4** | complete |
|
||||
| REQ-175 | Proxmox native ingress mode (`--ingress-mode native`, default): on PVE host, render+apply nft (vmbr-compatible, separate `orca-ingress` table avoids pve-firewall conflict); create unprivileged LXC with `--features nesting=1,keyctl=1` running podman+orca-traefik; push step-ca root CA into LXC; nft DNAT target = LXC bridge IP; register PVE host as `proxmox` node; add `IngressMode` field to `model.Node` + schema migration | Critical | **v0.14 P5** | complete |
|
||||
| REQ-176 | Proxmox floating-IP mode (`--ingress-mode floating-ip --floating-ip --gateway --mac [--net-prefix]`): `pct create` Ubuntu LXC named `ingress` with `net0 bridge=vmbr0,hwaddr=<mac>,ip=<floating-ip>/<prefix>,gw=<gateway> --features nesting=1,keyctl=1 --onboot 1`; install podman + run orca-traefik inside LXC; apply nft DNAT+SNAT inside LXC; register LXC as managed `linux` node (name=`ingress`, addr=`<floating-ip>:8443`); interactive prompt for params when flags absent + not `--json`; validate IP/MAC/gateway; PVE host also registered as `proxmox` for workload dispatch | Critical | **v0.14 P6** | complete |
|
||||
| REQ-177 | `orca doctor ingress [--peer]`: verify orca-traefik container running (`podman inspect`), nft DNAT+SNAT applied, `/etc/traefik/dynamic` exists, step-ca root CA mounted; extend `scripts/uat-signoff.sh` with ingress assertions (40 podman_traefik, 41 nft_dnat_snat, 42 linux_worker, 43 proxmox_native_lxc / floating_ip_lxc) | High | **v0.14 P7** | complete |
|
||||
| REQ-178 | Docs: `docs/cli.md` (`--ingress-mode` + floating-IP flags + `doctor ingress`), `docs/uat.md` (native + floating-IP topologies), `docs/ingress.md` (podman-traefik image + volume mounts + certResolver), `docs/docker.md` (orca-traefik image), `ARCHITECTURE.md` (R-024 + ingress bootstrap section) | High | **v0.14 P7** | complete |
|
||||
| REQ-179 | Integration tests: hermetic harness fakes SSH; asserts init→podman traefik running + nft applied; linux join→remote podman+nft; proxmox native→LXC created with nesting + podman traefik; floating-ip→`pct create` with correct net0 args + LXC registered as linux node; release.sh builds orca-traefik image (Dockerfile.traefik parses) | Critical | **v0.14 P7** | complete |
|
||||
|
||||
### Scope notes (v0.14)
|
||||
|
||||
@@ -427,3 +427,30 @@ node type.
|
||||
- 9 phases (P0 + P1..P7 + P8 final); feature milestone (multiple `feat` phases).
|
||||
- Tags on v0.13.x patch line: `v0.13.0` (P0) ... `v0.13.8` (P8 final = v0.14 milestone release).
|
||||
- Milestone branch: `milestone/v0.14-ingress-bootstrap`.
|
||||
|
||||
## Milestone v0.15: CI Release Pipeline Fix
|
||||
|
||||
**Scope**: fix the container image publishing pipeline. v0.14 shipped
|
||||
`Dockerfile.traefik` + `Dockerfile` but no container images were
|
||||
published to the Gitea registry because: (1) no Gitea Actions workflow
|
||||
existed to trigger on tag pushes, (2) the CoreCI trigger script
|
||||
stripped tag refs, (3) the `.coreci.yml` container-publish steps used
|
||||
Docker-in-Docker (`docker:24-cli`) which is prohibited. v0.15 adds a
|
||||
Gitea Actions workflow that triggers on tag pushes, installs the
|
||||
`coreci` binary on the runner, and runs `coreci run`. The
|
||||
`.coreci.yml` container-publish steps are rewritten to use kaniko
|
||||
(no Docker daemon required).
|
||||
|
||||
| ID | Requirement | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-180 | Create `.gitea/workflows/release.yml` that triggers on `push: tags: ['v*']`, installs the `coreci` binary (from `git.cloudinit.dev/coreci/coreci`), injects `PAT_TOKEN` secret as `GITEA_TOKEN` env var, and runs `coreci run` — which executes the full `.coreci.yml` pipeline (validate, build, test, release) locally on the Gitea Actions runner | Critical | **v0.15 P1** | pending |
|
||||
| REQ-181 | Replace `docker:24-cli` DinD steps in `.coreci.yml` with kaniko (`gcr.io/kaniko-project/executor:debug`): write `/kaniko/.docker/config.json` from `GITEA_TOKEN` (base64 auth), run `/kaniko/executor --dockerfile=<Dockerfile> --context=dir://. --destination=<registry/image:tag> --skip-tls-verify-registry`. Applies to both `container-publish` (orca image) and `container-publish-traefik` (orca-traefik image) | Critical | **v0.15 P1** | pending |
|
||||
| REQ-182 | Set `PAT_TOKEN` Gitea Actions repository secret via `tea actions secrets create` (same value as `GITEA_TOKEN` from `.env`). Gitea reserves the `GITEA_` prefix for built-in secrets, so the secret must be named `PAT_TOKEN`, not `GITEA_PAT` | High | **v0.15 P0** | complete |
|
||||
|
||||
### Scope notes (v0.15)
|
||||
|
||||
- REQ-180..REQ-182 = 3 net-new requirements (REQ count grows 172 -> 175).
|
||||
- 3 phases (P0 + P1 + P2 final); fix milestone (no `feat` phases — CI infrastructure).
|
||||
- Tags on v0.14.x patch line: `v0.14.0` (P0) ... `v0.14.2` (P2 final = v0.15 milestone release).
|
||||
- Milestone branch: `milestone/v0.15-ci-release-pipeline`.
|
||||
- REQ-182 is complete: `PAT_TOKEN` secret created via `tea actions secrets create PAT_TOKEN <value> --repo coreci/orca`.
|
||||
|
||||
+26
-10
@@ -676,7 +676,7 @@ CI agent verifies and cuts v1.0.0).
|
||||
- jobspec `update` rolling/canary controller (v0.13 adds lint warning; enforcement deferred)
|
||||
- jobspec `schedule.cron` scheduler loop (v0.13 adds lint warning; enforcement deferred)
|
||||
|
||||
## Milestone v0.14: Ingress Bootstrap Completeness — **IN PROGRESS**
|
||||
## Milestone v0.14: Ingress Bootstrap Completeness — **COMPLETE**
|
||||
|
||||
**Scope**: ensure that linux & proxmox types are properly bootstrapped with
|
||||
traefik during cluster init or node join. All cluster endpoints are
|
||||
@@ -701,15 +701,15 @@ root CA volume mounts.
|
||||
**Milestone type**: feature (multiple `feat` phases). Tags on v0.13.x patch
|
||||
line: `v0.13.0` (P0) ... `v0.13.8` (P8 final = v0.14 milestone release).
|
||||
|
||||
- [ ] Phase 0: Pre-execution (SPECIFY→CLARIFY→RESEARCH→PLAN→GRILL) — tag `v0.13.0`
|
||||
- [ ] Phase 1: `orca-traefik` container image + release pipeline (REQ-171) — tag `v0.13.1`
|
||||
- [ ] Phase 2: Podman traefik reconciler — replace binary+systemd install (REQ-172) — tag `v0.13.2`
|
||||
- [ ] Phase 3: nft SNAT+DNAT + `orca init` ingress bootstrap (REQ-173) — tag `v0.13.3`
|
||||
- [ ] Phase 4: `orca node join --type linux` remote ingress bootstrap (REQ-174) — tag `v0.13.4`
|
||||
- [ ] Phase 5: Proxmox native ingress mode — LXC + podman traefik (REQ-175) — tag `v0.13.5`
|
||||
- [ ] Phase 6: Proxmox floating-IP LXC ingress + interactive prompt (REQ-176) — tag `v0.13.6`
|
||||
- [ ] Phase 7: `doctor ingress` + docs + integration tests (REQ-177,178,179) — tag `v0.13.7`
|
||||
- [ ] Phase 8: Final review + ship + audit (milestone release) — tag `v0.13.8` = **v0.14 milestone release**
|
||||
- [x] Phase 0: Pre-execution (SPECIFY→CLARIFY→RESEARCH→PLAN→GRILL) — tag `v0.13.0`
|
||||
- [x] Phase 1: `orca-traefik` container image + release pipeline (REQ-171) — tag `v0.13.1`
|
||||
- [x] Phase 2: Podman traefik reconciler — replace binary+systemd install (REQ-172) — tag `v0.13.2`
|
||||
- [x] Phase 3: nft SNAT+DNAT + `orca init` ingress bootstrap (REQ-173) — tag `v0.13.3`
|
||||
- [x] Phase 4: `orca node join --type linux` remote ingress bootstrap (REQ-174) — tag `v0.13.4`
|
||||
- [x] Phase 5: Proxmox native ingress mode — LXC + podman traefik (REQ-175) — tag `v0.13.5`
|
||||
- [x] Phase 6: Proxmox floating-IP LXC ingress + interactive prompt (REQ-176) — tag `v0.13.6`
|
||||
- [x] Phase 7: `doctor ingress` + docs + integration tests (REQ-177,178,179) — tag `v0.13.7`
|
||||
- [x] Phase 8: Final review + ship + audit (milestone release) — tag `v0.13.8` = **v0.14 milestone release**
|
||||
|
||||
### Per-phase REQ coverage (v0.14)
|
||||
|
||||
@@ -729,3 +729,19 @@ is unchanged. v0.14 completes the ingress bootstrap that v0.13 left
|
||||
non-functional (binary installed but no config, no nft applied). The
|
||||
podman-container model is the operator's constraint; the architecture's
|
||||
socket+traefik routing design (R-007, R-017) is unchanged.
|
||||
|
||||
## Milestone v0.15: CI Release Pipeline Fix — **IN PROGRESS**
|
||||
|
||||
**Scope**: fix container image publishing. v0.14 shipped
|
||||
`Dockerfile.traefik` + `Dockerfile` but no images were published
|
||||
because no Gitea Actions workflow triggered on tag pushes, and
|
||||
`.coreci.yml` used Docker-in-Docker. v0.15 adds a Gitea Actions
|
||||
workflow (trigger on tag push → install coreci → `coreci run`) and
|
||||
rewrites the container-publish steps to use kaniko (no DinD).
|
||||
|
||||
**Milestone type**: fix (CI infrastructure). Tags on v0.14.x patch
|
||||
line: `v0.14.0` (P0) ... `v0.14.2` (P2 final = v0.15 milestone release).
|
||||
|
||||
- [ ] Phase 0: Pre-execution (SPECIFY→CLARIFY→RESEARCH→PLAN→GRILL) — tag `v0.14.0`
|
||||
- [ ] Phase 1: Gitea Actions workflow + .coreci.yml kaniko rewrite (REQ-180,181) — tag `v0.14.1`
|
||||
- [ ] Phase 2: Final review + ship + audit (milestone release) — tag `v0.14.2` = **v0.15 milestone release**
|
||||
|
||||
@@ -5,9 +5,9 @@
|
||||
"slug": "orca",
|
||||
"name": "Orca",
|
||||
"description": "Offline/CLI-first orchestration engine (Orca) \u2014 Nomad-inspired, far simpler than Kubernetes",
|
||||
"milestone": "v0.14",
|
||||
"milestone": "v0.15",
|
||||
"phase": 0,
|
||||
"milestone_type": "feature",
|
||||
"milestone_type": "fix",
|
||||
"default_branch": "main",
|
||||
"tech_stack": {
|
||||
"language": "go",
|
||||
|
||||
+71
-7
@@ -138,13 +138,77 @@ restore traffic).
|
||||
|
||||
## TLS
|
||||
|
||||
- **certResolver**: `orca` (references the Traefik ACME/step-ca
|
||||
certificate resolver configured in Traefik's static config).
|
||||
- **Trust domain**: `cluster.orca.local` (placeholder in v0.9; step-ca
|
||||
provisioner in v0.11 overrides with the real cluster trust domain).
|
||||
- **SPIFFE SVIDs**: workload identity via SPIFFE SVIDs minted at submit
|
||||
time via step-ca (v0.11-P01.5, gate C-08). The SVID is a URI SAN in
|
||||
the workload's X.509 cert.
|
||||
- **v0.14 model**: `tls: {}` in dynamic config (no certResolver).
|
||||
Traefik v3.3 `certificatesResolvers` only supports `acme` and
|
||||
`tailscale` — not CA-file-based. The `certResolver: orca` reference
|
||||
from v0.11 was broken (research finding). v0.14 emits `tls: {}`
|
||||
(traefik uses its default self-signed cert). Real mTLS via dynamic
|
||||
`tls.certificates` + `tls.options.default.clientAuth.caFiles` is
|
||||
deferred to v0.15.
|
||||
- **Step-ca root CA**: mounted at `/etc/orca/step-ca-root.crt` in the
|
||||
traefik container. v0.14 does not use it for TLS termination (it's
|
||||
a placeholder for v0.15 mTLS).
|
||||
|
||||
## R-024: Podman Traefik Container (v0.14)
|
||||
|
||||
As of v0.14, Traefik runs as a **podman container** from the custom
|
||||
`orca-traefik` image (published per release). The v0.13 binary+systemd
|
||||
install is replaced.
|
||||
|
||||
### Three topologies
|
||||
|
||||
1. **Linux**: host → nft DNAT → `podman run orca-traefik` (`--network host`)
|
||||
2. **Proxmox Native** (`--ingress-mode native`, default): PVE host →
|
||||
nft DNAT → LXC (nesting=1,keyctl=1,fuse=1) → `podman run orca-traefik`
|
||||
3. **Proxmox Floating-IP** (`--ingress-mode floating-ip`): LXC owns
|
||||
the floating IP → nft inside LXC → `podman run orca-traefik`
|
||||
|
||||
### Container configuration
|
||||
|
||||
```bash
|
||||
podman run -d --name orca-traefik --restart=unless-stopped \
|
||||
--network host \
|
||||
-v /etc/traefik/traefik.yml:/etc/traefik/traefik.yml:ro \
|
||||
-v /etc/traefik/dynamic:/etc/traefik/dynamic:ro \
|
||||
-v /etc/orca/step-ca-root.crt:/etc/orca/step-ca-root.crt:ro \
|
||||
git.cloudinit.dev/coreci/orca-traefik:<version>
|
||||
```
|
||||
|
||||
- `--network host`: traefik binds 127.0.0.1:8080/8443 on host/LXC loopback
|
||||
- `--restart=unless-stopped`: survives reboot via `podman-restart.service`
|
||||
- No `:Z` SELinux flag (research Topic 7)
|
||||
- Static config mounted `:ro` (overrides baked image default, preserves
|
||||
`traefik-on-public-ip` opt-out, REQ-100)
|
||||
|
||||
### nft ruleset
|
||||
|
||||
The nft emitter (`internal/emitter/nft.go`) renders `/etc/nftables.d/orca.nft`:
|
||||
|
||||
- DNAT `:443` → `<DNATTarget>:8443` (default 127.0.0.1; LXC IP for native)
|
||||
- DNAT `:80` → `<DNATTarget>:8080`
|
||||
- SNAT/MASQUERADE: `ip saddr 127.0.0.0/8 oifname != "lo" masquerade`
|
||||
- Input/forward chains at priority -10 (pve-firewall coexistence)
|
||||
|
||||
### `orca doctor ingress`
|
||||
|
||||
```bash
|
||||
orca doctor ingress # check localhost
|
||||
orca doctor ingress --peer <name> # check remote peer
|
||||
```
|
||||
|
||||
Verifies: podman container running, nft DNAT+SNAT, dynamic dir exists,
|
||||
step-ca root CA present.
|
||||
|
||||
### Dockerfile.traefik
|
||||
|
||||
```dockerfile
|
||||
FROM traefik:v3.3.0
|
||||
COPY docker/orca-traefik/traefik.yml /etc/traefik/traefik.yml
|
||||
CMD ["--configFile=/etc/traefik/traefik.yml"]
|
||||
```
|
||||
|
||||
Built + published per release alongside the orca image
|
||||
(`scripts/release.sh` + `.coreci.yml container-publish-traefik`).
|
||||
|
||||
## Health checks
|
||||
|
||||
|
||||
@@ -0,0 +1,115 @@
|
||||
// Package cli: doctor_ingress.go implements `orca doctor ingress`
|
||||
// (R-024, v0.14). The check verifies the podman traefik container is
|
||||
// running, nft DNAT+SNAT is applied, the dynamic config directory
|
||||
// exists, and the step-ca root CA is mounted.
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
var doctorIngressCmd = &cobra.Command{
|
||||
Use: "ingress",
|
||||
Short: "Check the ingress stack (R-024: podman traefik + nft + CA)",
|
||||
Long: `Verify the orca ingress data plane is healthy:
|
||||
1. orca-traefik podman container is running
|
||||
2. nft DNAT + SNAT masquerade applied
|
||||
3. /etc/traefik/dynamic directory exists
|
||||
4. step-ca root CA mounted at /etc/orca/step-ca-root.crt
|
||||
|
||||
For remote peers, use --peer <name>.`,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
results := runIngressChecks(ctx)
|
||||
if jsonOutput {
|
||||
return printJSON(results)
|
||||
}
|
||||
allPass := true
|
||||
for _, r := range results {
|
||||
status := "✓"
|
||||
if r.Result != "PASS" {
|
||||
status = "✗"
|
||||
allPass = false
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s %s: %s\n", status, r.Name, r.Message)
|
||||
}
|
||||
if !allPass {
|
||||
return fmt.Errorf("ingress checks failed")
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
// ingressCheckResult is one line of `orca doctor ingress` output.
|
||||
type ingressCheckResult struct {
|
||||
Name string `json:"name"`
|
||||
Result string `json:"result"`
|
||||
Message string `json:"message"`
|
||||
}
|
||||
|
||||
func runIngressChecks(ctx context.Context) []ingressCheckResult {
|
||||
t, err := nftTransportFromCtx()
|
||||
if err != nil {
|
||||
return []ingressCheckResult{{Name: "ingress:transport", Result: "FAIL", Message: err.Error()}}
|
||||
}
|
||||
peer := nftLeadPeer()
|
||||
var results []ingressCheckResult
|
||||
|
||||
// 1. Check podman orca-traefik container is running.
|
||||
out, err := t.Exec(ctx, peer, "podman inspect --format '{{.State.Running}}' orca-traefik 2>/dev/null")
|
||||
if err != nil {
|
||||
results = append(results, ingressCheckResult{Name: "ingress:container", Result: "FAIL", Message: fmt.Sprintf("podman inspect: %v", err)})
|
||||
} else {
|
||||
v := strings.TrimSpace(string(out))
|
||||
if v == "true" {
|
||||
results = append(results, ingressCheckResult{Name: "ingress:container", Result: "PASS", Message: "orca-traefik container running"})
|
||||
} else if v == "false" {
|
||||
results = append(results, ingressCheckResult{Name: "ingress:container", Result: "FAIL", Message: "orca-traefik container is stopped"})
|
||||
} else {
|
||||
results = append(results, ingressCheckResult{Name: "ingress:container", Result: "FAIL", Message: "orca-traefik container not found"})
|
||||
}
|
||||
}
|
||||
|
||||
// 2. Check nft DNAT + SNAT (reuse the nft table output).
|
||||
tableOut, tableErr := t.Exec(ctx, peer, "nft list table inet orca-ingress 2>/dev/null")
|
||||
if tableErr != nil {
|
||||
results = append(results, ingressCheckResult{Name: "ingress:nft", Result: "FAIL", Message: "nft table orca-ingress missing"})
|
||||
} else {
|
||||
tableStr := string(tableOut)
|
||||
hasDNAT := strings.Contains(tableStr, "dnat to")
|
||||
hasSNAT := strings.Contains(tableStr, "masquerade")
|
||||
if hasDNAT && hasSNAT {
|
||||
results = append(results, ingressCheckResult{Name: "ingress:nft", Result: "PASS", Message: "nft DNAT + SNAT masquerade present"})
|
||||
} else if hasDNAT {
|
||||
results = append(results, ingressCheckResult{Name: "ingress:nft", Result: "WARN", Message: "DNAT present but SNAT masquerade missing"})
|
||||
} else {
|
||||
results = append(results, ingressCheckResult{Name: "ingress:nft", Result: "FAIL", Message: "nft DNAT missing"})
|
||||
}
|
||||
}
|
||||
|
||||
// 3. Check /etc/traefik/dynamic directory exists.
|
||||
if _, err := t.Exec(ctx, peer, "test -d /etc/traefik/dynamic"); err != nil {
|
||||
results = append(results, ingressCheckResult{Name: "ingress:dynamic-dir", Result: "FAIL", Message: "/etc/traefik/dynamic directory missing"})
|
||||
} else {
|
||||
results = append(results, ingressCheckResult{Name: "ingress:dynamic-dir", Result: "PASS", Message: "/etc/traefik/dynamic exists"})
|
||||
}
|
||||
|
||||
// 4. Check step-ca root CA is mounted/present.
|
||||
if _, err := t.Exec(ctx, peer, "test -f /etc/orca/step-ca-root.crt"); err != nil {
|
||||
results = append(results, ingressCheckResult{Name: "ingress:ca", Result: "WARN", Message: "/etc/orca/step-ca-root.crt missing (TLS not configured)"})
|
||||
} else {
|
||||
results = append(results, ingressCheckResult{Name: "ingress:ca", Result: "PASS", Message: "step-ca root CA present"})
|
||||
}
|
||||
|
||||
return results
|
||||
}
|
||||
|
||||
func init() {
|
||||
doctorCmd.AddCommand(doctorIngressCmd)
|
||||
}
|
||||
+204
-24
@@ -1,22 +1,26 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net"
|
||||
"os"
|
||||
"os/signal"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/spf13/cobra"
|
||||
"golang.org/x/crypto/ssh"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/certpaths"
|
||||
"git.cloudinit.dev/coreci/orca/internal/linux"
|
||||
"git.cloudinit.dev/coreci/orca/internal/engine"
|
||||
"git.cloudinit.dev/coreci/orca/internal/linux"
|
||||
"git.cloudinit.dev/coreci/orca/internal/model"
|
||||
"git.cloudinit.dev/coreci/orca/internal/proxmox"
|
||||
"git.cloudinit.dev/coreci/orca/internal/security"
|
||||
@@ -46,20 +50,25 @@ func nodeRegistry() (*engine.NodeRegistry, func() error, error) {
|
||||
}
|
||||
|
||||
var (
|
||||
joinName string
|
||||
joinAddr string
|
||||
joinCAFinger string
|
||||
joinType string
|
||||
joinHost string
|
||||
joinSSHUser string
|
||||
joinSSHKey string
|
||||
joinSSHPort int
|
||||
joinHostKeyFP string
|
||||
joinName string
|
||||
joinAddr string
|
||||
joinCAFinger string
|
||||
joinType string
|
||||
joinHost string
|
||||
joinSSHUser string
|
||||
joinSSHKey string
|
||||
joinSSHPort int
|
||||
joinHostKeyFP string
|
||||
joinLXCTemplate string
|
||||
proxmoxUser string
|
||||
proxmoxRole string
|
||||
leaveID string
|
||||
nodeWatch bool
|
||||
proxmoxUser string
|
||||
proxmoxRole string
|
||||
ingressMode string
|
||||
floatingIP string
|
||||
gateway string
|
||||
macAddr string
|
||||
netPrefix int
|
||||
leaveID string
|
||||
nodeWatch bool
|
||||
)
|
||||
|
||||
var nodeCmd = &cobra.Command{
|
||||
@@ -175,7 +184,83 @@ func joinProxmox(cmd *cobra.Command) error {
|
||||
return fmt.Errorf("SSH key path is required for --type proxmox (R-021: no passwords; use --ssh-key or pre-stage the orca key)")
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(cmd.Context(), 60*time.Second)
|
||||
// Default ingress mode to "native" if not specified (R-024).
|
||||
effectiveIngressMode := ingressMode
|
||||
if effectiveIngressMode == "" {
|
||||
if !jsonOutput {
|
||||
// Interactive mode: prompt for ingress mode.
|
||||
fmt.Fprint(cmd.OutOrStdout(), "Ingress mode [native/floating-ip] (default native): ")
|
||||
scanner := bufio.NewScanner(os.Stdin)
|
||||
if scanner.Scan() {
|
||||
input := strings.TrimSpace(scanner.Text())
|
||||
if input == "floating-ip" {
|
||||
effectiveIngressMode = "floating-ip"
|
||||
} else {
|
||||
effectiveIngressMode = "native"
|
||||
}
|
||||
} else {
|
||||
effectiveIngressMode = "native"
|
||||
}
|
||||
} else {
|
||||
effectiveIngressMode = "native"
|
||||
}
|
||||
}
|
||||
|
||||
// Floating-IP mode: prompt for params if not provided.
|
||||
effectiveFloatingIP := floatingIP
|
||||
effectiveGateway := gateway
|
||||
effectiveMAC := macAddr
|
||||
if effectiveIngressMode == "floating-ip" {
|
||||
if effectiveFloatingIP == "" && !jsonOutput {
|
||||
fmt.Fprint(cmd.OutOrStdout(), "Floating IP: ")
|
||||
scanner := bufio.NewScanner(os.Stdin)
|
||||
if scanner.Scan() {
|
||||
effectiveFloatingIP = strings.TrimSpace(scanner.Text())
|
||||
}
|
||||
}
|
||||
if effectiveGateway == "" && !jsonOutput {
|
||||
fmt.Fprint(cmd.OutOrStdout(), "Gateway: ")
|
||||
scanner := bufio.NewScanner(os.Stdin)
|
||||
if scanner.Scan() {
|
||||
effectiveGateway = strings.TrimSpace(scanner.Text())
|
||||
}
|
||||
}
|
||||
if effectiveMAC == "" && !jsonOutput {
|
||||
// D-261: auto-generate a random locally-administered MAC.
|
||||
generated, err := proxmox.GenerateRandomMAC()
|
||||
if err == nil {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "Generated MAC: %s (press enter to accept, or type your own): ", generated)
|
||||
scanner := bufio.NewScanner(os.Stdin)
|
||||
if scanner.Scan() {
|
||||
input := strings.TrimSpace(scanner.Text())
|
||||
if input != "" {
|
||||
effectiveMAC = input
|
||||
} else {
|
||||
effectiveMAC = generated
|
||||
}
|
||||
} else {
|
||||
effectiveMAC = generated
|
||||
}
|
||||
}
|
||||
}
|
||||
// Validate floating-IP mode params.
|
||||
if effectiveIngressMode == "floating-ip" {
|
||||
if net.ParseIP(effectiveFloatingIP) == nil {
|
||||
return fmt.Errorf("--floating-ip %q is not a valid IP", effectiveFloatingIP)
|
||||
}
|
||||
if net.ParseIP(effectiveGateway) == nil {
|
||||
return fmt.Errorf("--gateway %q is not a valid IP", effectiveGateway)
|
||||
}
|
||||
if _, err := net.ParseMAC(effectiveMAC); err != nil {
|
||||
return fmt.Errorf("--mac %q is not a valid MAC: %w", effectiveMAC, err)
|
||||
}
|
||||
if netPrefix < 8 || netPrefix > 32 {
|
||||
return fmt.Errorf("--net-prefix %d must be 8-32", netPrefix)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(cmd.Context(), 180*time.Second) // 3min for LXC creation
|
||||
defer cancel()
|
||||
|
||||
result, err := proxmox.BootstrapProxmox(ctx, proxmox.Options{
|
||||
@@ -188,6 +273,7 @@ func joinProxmox(cmd *cobra.Command) error {
|
||||
HostKeyFingerprint: joinHostKeyFP,
|
||||
Logger: newLogger(),
|
||||
LXCTemplate: joinLXCTemplate,
|
||||
IngressMode: effectiveIngressMode,
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("proxmox bootstrap: %w", err)
|
||||
@@ -204,18 +290,67 @@ func joinProxmox(cmd *cobra.Command) error {
|
||||
defer regCancel()
|
||||
|
||||
node := &model.Node{
|
||||
ID: uuid.NewString(),
|
||||
Name: result.NodeName,
|
||||
Address: result.NodeAddress,
|
||||
State: model.NodeStateReady,
|
||||
JoinedAt: time.Now().UTC(),
|
||||
LastSeen: time.Now().UTC(),
|
||||
Kind: string(model.NodeKindProxmox),
|
||||
OS: "pve",
|
||||
ID: uuid.NewString(),
|
||||
Name: result.NodeName,
|
||||
Address: result.NodeAddress,
|
||||
State: model.NodeStateReady,
|
||||
JoinedAt: time.Now().UTC(),
|
||||
LastSeen: time.Now().UTC(),
|
||||
Kind: string(model.NodeKindProxmox),
|
||||
OS: "pve",
|
||||
IngressMode: effectiveIngressMode,
|
||||
}
|
||||
if err := registry.Join(regCtx, node); err != nil {
|
||||
return fmt.Errorf("register proxmox node: %w", err)
|
||||
}
|
||||
|
||||
// Floating-IP mode: provision the ingress LXC and register it as a
|
||||
// linux node (R-024, REQ-176). The PVE host is registered as
|
||||
// proxmox (above); the ingress LXC is registered as linux so
|
||||
// `orca job run` pushes traefik dynamic config to it.
|
||||
if effectiveIngressMode == "floating-ip" {
|
||||
lxcLog := newLogger()
|
||||
// Build a runRemote function from the proxmox bootstrap result.
|
||||
// We need SSH access to the PVE host to run pct commands.
|
||||
lxcCtx, lxcCancel := context.WithTimeout(ctx, 120*time.Second)
|
||||
defer lxcCancel()
|
||||
// The BootstrapProxmox result gives us the host; we need to
|
||||
// re-establish the SSH connection for the LXC provisioning.
|
||||
lxcExecFn, lxcErr := proxmoxRemoteExecFn(result, sshKeyPath, joinSSHUser, joinSSHPort)
|
||||
if lxcErr != nil {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "Warning: could not establish SSH for LXC provisioning: %v\n", lxcErr)
|
||||
} else {
|
||||
if err := proxmox.ProvisionIngressLXC(lxcCtx, lxcExecFn, proxmox.FloatingIPOptions{
|
||||
FloatingIP: effectiveFloatingIP,
|
||||
Gateway: effectiveGateway,
|
||||
MAC: effectiveMAC,
|
||||
NetPrefix: netPrefix,
|
||||
LXCTemplate: joinLXCTemplate,
|
||||
}, lxcLog); err != nil {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "Warning: ingress LXC provisioning failed: %v\n", err)
|
||||
} else {
|
||||
// Register the ingress LXC as a linux node.
|
||||
ingressNode := &model.Node{
|
||||
ID: uuid.NewString(),
|
||||
Name: "ingress",
|
||||
Address: fmt.Sprintf("%s:8443", effectiveFloatingIP),
|
||||
State: model.NodeStateReady,
|
||||
JoinedAt: time.Now().UTC(),
|
||||
LastSeen: time.Now().UTC(),
|
||||
Kind: string(model.NodeKindLinux),
|
||||
OS: "linux",
|
||||
IngressMode: "floating-ip",
|
||||
}
|
||||
if err := registry.Join(regCtx, ingressNode); err != nil {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "Warning: register ingress node: %v\n", err)
|
||||
}
|
||||
if !jsonOutput {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "✓ Ingress LXC joined: %s (%s) at %s\n", ingressNode.ID, ingressNode.Name, ingressNode.Address)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// REQ-156 / P07 T5: invalidate the nodes cache.
|
||||
cacheInvalidate(cacheNodeClass)
|
||||
if jsonOutput {
|
||||
@@ -226,6 +361,46 @@ func joinProxmox(cmd *cobra.Command) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// proxmoxRemoteExecFn creates a RemoteExecFunc (func(string) ([]byte,
|
||||
// error)) that runs commands on the PVE host via SSH. Used by the
|
||||
// floating-IP LXC provisioning path (ProvisionIngressLXC).
|
||||
func proxmoxRemoteExecFn(result *proxmox.Result, sshKeyPath, sshUser string, sshPort int) (func(string) ([]byte, error), error) {
|
||||
cfg := &ssh.ClientConfig{
|
||||
User: sshUser,
|
||||
HostKeyCallback: ssh.InsecureIgnoreHostKey(),
|
||||
Timeout: 10 * time.Second,
|
||||
}
|
||||
if sshKeyPath != "" {
|
||||
keyData, err := os.ReadFile(sshKeyPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read SSH key: %w", err)
|
||||
}
|
||||
signer, err := ssh.ParsePrivateKey(keyData)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("parse SSH key: %w", err)
|
||||
}
|
||||
cfg.Auth = []ssh.AuthMethod{ssh.PublicKeys(signer)}
|
||||
}
|
||||
addr := result.NodeName
|
||||
if sshPort != 22 {
|
||||
addr = fmt.Sprintf("%s:%d", result.NodeName, sshPort)
|
||||
} else {
|
||||
addr = fmt.Sprintf("%s:%d", result.NodeName, sshPort)
|
||||
}
|
||||
client, err := ssh.Dial("tcp", addr, cfg)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("ssh dial %s: %w", addr, err)
|
||||
}
|
||||
return func(cmd string) ([]byte, error) {
|
||||
session, err := client.NewSession()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer session.Close()
|
||||
return session.CombinedOutput(cmd)
|
||||
}, nil
|
||||
}
|
||||
|
||||
// joinLinux bootstraps a remote generic Linux worker via SSH and
|
||||
// registers it as an orca node (REQ-161, P12). Uses SSH key auth
|
||||
// (R-021: no passwords).
|
||||
@@ -252,7 +427,7 @@ func joinLinux(cmd *cobra.Command) error {
|
||||
SSHPort: joinSSHPort,
|
||||
HostKeyFingerprint: joinHostKeyFP,
|
||||
Logger: newLogger(),
|
||||
})
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("linux bootstrap: %w", err)
|
||||
}
|
||||
@@ -514,6 +689,11 @@ func init() {
|
||||
nodeJoinCmd.Flags().StringVar(&proxmoxRole, "proxmox-role", "OrcaOperator", "PVE custom role to create (config-overridable)")
|
||||
nodeJoinCmd.Flags().StringVar(&joinHostKeyFP, "host-key-fingerprint", "", "SSH host key SHA256:base64 fingerprint (pre-pin; supersedes TOFU for --type proxmox or --type linux)")
|
||||
nodeJoinCmd.Flags().StringVar(&joinLXCTemplate, "lxc-template", "ubuntu-24.04", "LXC template for Proxmox (default ubuntu-24.04; alternatives: alpine-3.20, debian-12)")
|
||||
nodeJoinCmd.Flags().StringVar(&ingressMode, "ingress-mode", "", "proxmox ingress mode: native (default, traefik in LXC) or floating-ip (ingress LXC owns floating IP)")
|
||||
nodeJoinCmd.Flags().StringVar(&floatingIP, "floating-ip", "", "floating public IP for the ingress LXC (required for --ingress-mode floating-ip)")
|
||||
nodeJoinCmd.Flags().StringVar(&gateway, "gateway", "", "gateway for the ingress LXC (required for --ingress-mode floating-ip)")
|
||||
nodeJoinCmd.Flags().StringVar(&macAddr, "mac", "", "MAC address for the ingress LXC net0 (required for --ingress-mode floating-ip in --json mode; auto-generated in interactive mode)")
|
||||
nodeJoinCmd.Flags().IntVar(&netPrefix, "net-prefix", 24, "network prefix (CIDR) for the ingress LXC IP (default 24; valid 8-32)")
|
||||
nodeLeaveCmd.Flags().StringVar(&leaveID, "id", "", "node id")
|
||||
nodeListCmd.Flags().BoolVar(&nodeWatch, "watch", false, "stream nodes until Ctrl-C (table refresh or --json per-event)")
|
||||
|
||||
|
||||
@@ -9,11 +9,14 @@ package ingress
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/certpaths"
|
||||
"git.cloudinit.dev/coreci/orca/internal/emitter"
|
||||
@@ -117,3 +120,121 @@ func BootstrapLocalIngress(ctx context.Context, version string) error {
|
||||
|
||||
// nftConfigPath mirrors the emitter constant.
|
||||
const nftConfigPath = "/etc/nftables.d/orca.nft"
|
||||
|
||||
// RemoteExecFunc runs a command on a remote host and returns combined
|
||||
// output. Same signature as traefik.RemoteExecFunc.
|
||||
type RemoteExecFunc func(cmd string) ([]byte, error)
|
||||
|
||||
// BootstrapRemoteIngress ensures the complete ingress stack is running
|
||||
// on a remote host (linux worker). It is called from
|
||||
// `orca node join --type linux` after the user setup.
|
||||
//
|
||||
// Steps (each non-fatal — logs a warning and continues):
|
||||
// 1. mkdir -p /etc/traefik/dynamic /etc/orca (remote)
|
||||
// 2. Push step-ca root CA to remote /etc/orca/step-ca-root.crt
|
||||
// (C-60: certpaths.CACertPath)
|
||||
// 3. Render + write static config to remote /etc/traefik/traefik.yml
|
||||
// (C-58: preserves traefik-on-public-ip opt-out)
|
||||
// 4. Render + write nft ruleset to remote /etc/nftables.d/orca.nft
|
||||
// 5. Pre-create nft table (C-55: avoids first-apply flush-table error)
|
||||
// 6. Apply: nft -f (remote)
|
||||
// 7. Ensure podman traefik container running (remote)
|
||||
func BootstrapRemoteIngress(ctx context.Context, version string, execFn RemoteExecFunc) error {
|
||||
var errs []error
|
||||
log := slog.Default()
|
||||
|
||||
// Step 1: ensure directories.
|
||||
if _, err := execFn("mkdir -p /etc/traefik/dynamic /etc/orca"); err != nil {
|
||||
log.Warn("ingress: remote mkdir failed", "err", err)
|
||||
errs = append(errs, fmt.Errorf("remote mkdir: %w", err))
|
||||
}
|
||||
|
||||
// Step 2: push step-ca root CA (C-60: CACertPath, not CAPath).
|
||||
if caData, err := os.ReadFile(certpaths.CACertPath()); err == nil {
|
||||
if err := remoteWriteFile(execFn, "/etc/orca/step-ca-root.crt", caData, "0644"); err != nil {
|
||||
log.Warn("ingress: remote step-ca CA write failed", "err", err)
|
||||
errs = append(errs, fmt.Errorf("remote write step-ca-root.crt: %w", err))
|
||||
}
|
||||
} else {
|
||||
// Write a placeholder so the podman volume mount doesn't fail.
|
||||
_ = remoteWriteFile(execFn, "/etc/orca/step-ca-root.crt", []byte{}, "0644")
|
||||
log.Warn("ingress: step-ca root CA not found locally, wrote remote placeholder")
|
||||
}
|
||||
|
||||
// Step 3: render + write static config (C-58).
|
||||
staticFiles, err := emitter.TraefikEmitter{}.RenderTraefikStaticConfig(emitter.TraefikStaticOpts{})
|
||||
if err != nil {
|
||||
log.Warn("ingress: render traefik static config failed", "err", err)
|
||||
errs = append(errs, fmt.Errorf("render traefik static: %w", err))
|
||||
} else {
|
||||
for _, f := range staticFiles {
|
||||
_, _ = execFn(fmt.Sprintf("mkdir -p %s", filepath.Dir(f.Path)))
|
||||
if err := remoteWriteFile(execFn, f.Path, []byte(f.Content), f.Mode); err != nil {
|
||||
log.Warn("ingress: remote write traefik static config failed", "path", f.Path, "err", err)
|
||||
errs = append(errs, fmt.Errorf("remote write %s: %w", f.Path, err))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Step 4: render + write nft ruleset.
|
||||
nftFiles, err := emitter.NftEmitter{}.RenderNftConfig(emitter.NftClusterConfig{})
|
||||
if err != nil {
|
||||
log.Warn("ingress: render nft config failed", "err", err)
|
||||
errs = append(errs, fmt.Errorf("render nft: %w", err))
|
||||
} else {
|
||||
for _, f := range nftFiles {
|
||||
_, _ = execFn(fmt.Sprintf("mkdir -p %s", filepath.Dir(f.Path)))
|
||||
if err := remoteWriteFile(execFn, f.Path, []byte(f.Content), f.Mode); err != nil {
|
||||
log.Warn("ingress: remote write nft config failed", "path", f.Path, "err", err)
|
||||
errs = append(errs, fmt.Errorf("remote write %s: %w", f.Path, err))
|
||||
}
|
||||
}
|
||||
|
||||
// Step 5: pre-create nft table (C-55).
|
||||
_, _ = execFn("nft add table inet orca-ingress 2>/dev/null || true")
|
||||
|
||||
// Step 6: apply nft ruleset.
|
||||
if out, err := execFn("nft -f /etc/nftables.d/orca.nft 2>&1"); err != nil {
|
||||
log.Warn("ingress: remote nft apply failed", "err", err, "output", string(out))
|
||||
errs = append(errs, fmt.Errorf("remote nft -f: %w (output: %s)", err, string(out)))
|
||||
}
|
||||
}
|
||||
|
||||
// Step 7: ensure podman traefik container (C-50).
|
||||
traefikExecFn := traefik.RemoteExecFunc(execFn)
|
||||
if err := traefik.EnsureTraefikContainerRemote(ctx, version, traefikExecFn); err != nil {
|
||||
log.Warn("ingress: remote ensure traefik container failed", "err", err)
|
||||
errs = append(errs, fmt.Errorf("remote ensure traefik container: %w", err))
|
||||
}
|
||||
|
||||
if len(errs) > 0 {
|
||||
return fmt.Errorf("remote ingress bootstrap: %d errors (first: %w)", len(errs), errs[0])
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// remoteWriteFile writes content to a remote path via a heredoc
|
||||
// (same pattern as sshpush.idempotency.writeFile). The heredoc
|
||||
// delimiter is a random hex string verified absent from the content
|
||||
// (F9 injection guard).
|
||||
func remoteWriteFile(execFn RemoteExecFunc, path string, content []byte, mode string) error {
|
||||
// Generate a random delimiter unlikely to be in the content.
|
||||
delim := "EOF_"
|
||||
for {
|
||||
b := make([]byte, 8)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
return fmt.Errorf("rand: %w", err)
|
||||
}
|
||||
delim = "EOF_" + hex.EncodeToString(b)
|
||||
if !strings.Contains(string(content), delim) {
|
||||
break
|
||||
}
|
||||
}
|
||||
dir := filepath.Dir(path)
|
||||
cmd := fmt.Sprintf("mkdir -p %s && cat > %s <<'%s'\n%s\n%s\nchmod %s %s",
|
||||
dir, path, delim, string(content), delim, mode, path)
|
||||
if out, err := execFn(cmd); err != nil {
|
||||
return fmt.Errorf("remote write %s: %w (output: %s)", path, err, string(out))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -32,9 +32,9 @@ import (
|
||||
"golang.org/x/crypto/ssh"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/certpaths"
|
||||
"git.cloudinit.dev/coreci/orca/internal/ingress"
|
||||
"git.cloudinit.dev/coreci/orca/internal/proxmox"
|
||||
"git.cloudinit.dev/coreci/orca/internal/security"
|
||||
"git.cloudinit.dev/coreci/orca/internal/traefik"
|
||||
)
|
||||
|
||||
// DefaultSSHUser is the default SSH username for the initial connection.
|
||||
@@ -157,8 +157,10 @@ func BootstrapLinux(ctx context.Context, opts Options) (*Result, error) {
|
||||
}
|
||||
opts.Logger.Info("linux bootstrap: user created", "user", opts.OrcaUser)
|
||||
|
||||
// Step 4d: Ensure orca-traefik podman container on the remote host
|
||||
// (REQ-172, R-024). Replaces v0.13 binary+systemd install.
|
||||
// Step 4d: Ensure complete ingress stack on the remote host (R-024).
|
||||
// Renders+applies nft DNAT/SNAT, pushes step-ca root CA, renders+
|
||||
// writes traefik static config, ensures podman container running.
|
||||
// All non-fatal (offline host tolerance).
|
||||
sshExecFn := func(cmd string) ([]byte, error) {
|
||||
session, err := client.NewSession()
|
||||
if err != nil {
|
||||
@@ -167,10 +169,10 @@ func BootstrapLinux(ctx context.Context, opts Options) (*Result, error) {
|
||||
defer session.Close()
|
||||
return session.CombinedOutput(cmd)
|
||||
}
|
||||
ctx, cancelContainer := context.WithTimeout(ctx, 120*time.Second)
|
||||
defer cancelContainer()
|
||||
if err := traefik.EnsureTraefikContainerRemote(ctx, "", sshExecFn); err != nil {
|
||||
opts.Logger.Warn("linux bootstrap: traefik container ensure failed", "err", err)
|
||||
ctx, cancelIngress := context.WithTimeout(ctx, 120*time.Second)
|
||||
defer cancelIngress()
|
||||
if err := ingress.BootstrapRemoteIngress(ctx, "", ingress.RemoteExecFunc(sshExecFn)); err != nil {
|
||||
opts.Logger.Warn("linux bootstrap: ingress bootstrap failed", "err", err)
|
||||
}
|
||||
|
||||
// Step 5: Create the drift-events directory.
|
||||
|
||||
@@ -39,4 +39,9 @@ type Node struct {
|
||||
// OS is the auto-detected OS identifier from /etc/os-release ID=
|
||||
// (ubuntu|debian|alpine|pve|linux). Empty for pre-0006 rows.
|
||||
OS string `json:"os,omitempty"`
|
||||
// IngressMode is the ingress configuration for the node (R-024,
|
||||
// v0.14). Values: "" (legacy/default for linux/localhost),
|
||||
// "native" (proxmox native — traefik in LXC), "floating-ip"
|
||||
// (proxmox floating-IP — ingress LXC owns the floating IP).
|
||||
IngressMode string `json:"ingress_mode,omitempty"`
|
||||
}
|
||||
|
||||
+133
-11
@@ -37,6 +37,7 @@ import (
|
||||
"golang.org/x/crypto/ssh/knownhosts"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/certpaths"
|
||||
"git.cloudinit.dev/coreci/orca/internal/emitter"
|
||||
"git.cloudinit.dev/coreci/orca/internal/security"
|
||||
"git.cloudinit.dev/coreci/orca/internal/traefik"
|
||||
)
|
||||
@@ -86,6 +87,13 @@ type Options struct {
|
||||
// LXCTemplate is the LXC template to download during bootstrap
|
||||
// (default "ubuntu-24.04"; alternatives: "alpine-3.20", "debian-12").
|
||||
LXCTemplate string
|
||||
// IngressMode is the proxmox ingress mode (R-024, v0.14).
|
||||
// "native" (default): traefik runs in an unprivileged LXC with
|
||||
// nesting=1,keyctl=1,fuse=1 on the PVE host. nft on the PVE host
|
||||
// DNATs to the LXC bridge IP.
|
||||
// "floating-ip": a separate ingress LXC owns the floating IP;
|
||||
// nft runs inside that LXC. See ProvisionIngressLXC (P6).
|
||||
IngressMode string
|
||||
}
|
||||
|
||||
// Result is the outcome of a successful bootstrap.
|
||||
@@ -247,23 +255,24 @@ func BootstrapProxmox(ctx context.Context, opts Options) (*Result, error) {
|
||||
return nil, fmt.Errorf("validate sudoers: %w", err)
|
||||
}
|
||||
|
||||
// Step 9a: Ensure orca-traefik podman container on the Proxmox host
|
||||
// (REQ-172, R-024). Replaces v0.13 binary+systemd install.
|
||||
// In native mode (default for v0.14 P5), the container runs inside
|
||||
// an LXC with nesting. For now, this installs on the PVE host OS.
|
||||
// Idempotent: no-op if container already running.
|
||||
if err := traefik.EnsureTraefikContainerRemote(ctx, "", runRemote); err != nil {
|
||||
log.Warn("proxmox.traefik_container_failed", "err", err)
|
||||
}
|
||||
|
||||
// Step 9b: Download default LXC template (REQ-167, Phase C).
|
||||
// Default: ubuntu-24.04. Configurable via --lxc-template.
|
||||
// Step 9a: Proxmox native ingress mode (R-024, REQ-175).
|
||||
// Create an unprivileged LXC with nesting=1,keyctl=1,fuse=1 (research
|
||||
// Topic 3), install podman inside it, and run the orca-traefik
|
||||
// container. nft on the PVE host DNATs to the LXC bridge IP.
|
||||
// Default mode is "native"; floating-ip mode is handled separately
|
||||
// (P6 — ProvisionIngressLXC).
|
||||
template := opts.LXCTemplate
|
||||
if template == "" {
|
||||
template = "ubuntu-24.04"
|
||||
}
|
||||
_, _ = runRemote(fmt.Sprintf("pveam download local %s 2>/dev/null || true", shellQuote(template)))
|
||||
|
||||
if opts.IngressMode != "floating-ip" {
|
||||
if err := provisionNativeIngressLXC(ctx, runRemote, template, log); err != nil {
|
||||
log.Warn("proxmox.native_ingress_lxc_failed", "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
log.Info("proxmox.bootstrap_ok",
|
||||
slog.String("event", "proxmox.bootstrap_ok"),
|
||||
slog.String("host", opts.Host),
|
||||
@@ -278,6 +287,119 @@ func BootstrapProxmox(ctx context.Context, opts Options) (*Result, error) {
|
||||
}, nil
|
||||
}
|
||||
|
||||
// provisionNativeIngressLXC creates an unprivileged LXC with
|
||||
// nesting=1,keyctl=1,fuse=1 (research Topic 3), installs podman inside
|
||||
// it, runs the orca-traefik container, and applies nft DNAT on the PVE
|
||||
// host targeting the LXC's bridge IP (R-024, REQ-175).
|
||||
//
|
||||
// The LXC is named "orca-traefik" and uses a deterministic VMID derived
|
||||
// from the host. It is idempotent: if the LXC already exists, it is
|
||||
// not re-created (C-53: apt-get install is skipped if podman present).
|
||||
func provisionNativeIngressLXC(ctx context.Context, runRemote func(string) ([]byte, error), template string, log *slog.Logger) error {
|
||||
// Deterministic VMID for the native ingress LXC.
|
||||
// Use a fixed VMID in the 200-299 range (Proxmox convention for CTs).
|
||||
const vmid = "200"
|
||||
const lxcName = "orca-traefik"
|
||||
|
||||
// Check if the LXC already exists.
|
||||
existOut, _ := runRemote(fmt.Sprintf("pct status %s 2>/dev/null || echo absent", vmid))
|
||||
existStr := strings.TrimSpace(string(existOut))
|
||||
if existStr == "absent" {
|
||||
// Create the LXC (research Topic 3: nesting=1,keyctl=1,fuse=1).
|
||||
log.Info("proxmox.creating_native_ingress_lxc", "vmid", vmid, "name", lxcName)
|
||||
createCmd := fmt.Sprintf(
|
||||
"pct create %s local:vztmpl/%s --hostname %s --unprivileged 1 --features nesting=1,keyctl=1,fuse=1 --onboot 1 --memory 2048 --swap 0 --rootfs local:8 2>&1",
|
||||
vmid, shellQuote(template), lxcName,
|
||||
)
|
||||
if out, err := runRemote(createCmd); err != nil {
|
||||
return fmt.Errorf("pct create native ingress LXC: %w (output: %s)", err, string(out))
|
||||
}
|
||||
if out, err := runRemote(fmt.Sprintf("pct start %s", vmid)); err != nil {
|
||||
return fmt.Errorf("pct start native ingress LXC: %w (output: %s)", err, string(out))
|
||||
}
|
||||
}
|
||||
|
||||
// Wait for LXC network (retry for up to 60s).
|
||||
lxcIP := ""
|
||||
for i := 0; i < 12; i++ {
|
||||
ipOut, _ := runRemote(fmt.Sprintf("pct exec %s -- hostname -I 2>/dev/null", vmid))
|
||||
ipStr := strings.TrimSpace(string(ipOut))
|
||||
if ipStr != "" {
|
||||
fields := strings.Fields(ipStr)
|
||||
if len(fields) > 0 {
|
||||
lxcIP = fields[0]
|
||||
break
|
||||
}
|
||||
}
|
||||
time.Sleep(5 * time.Second)
|
||||
}
|
||||
if lxcIP == "" {
|
||||
return fmt.Errorf("native ingress LXC: could not discover IP after 60s")
|
||||
}
|
||||
log.Info("proxmox.native_ingress_lxc_ip", "vmid", vmid, "ip", lxcIP)
|
||||
|
||||
// Install podman inside the LXC (C-53: idempotent — check first).
|
||||
_, _ = runRemote(fmt.Sprintf(
|
||||
"pct exec %s -- bash -c 'command -v podman >/dev/null 2>&1 || (apt-get update -qq && apt-get install -y -qq podman conmon crun fuse-overlayfs nftables 2>&1)' 2>&1",
|
||||
vmid,
|
||||
))
|
||||
|
||||
// Enable podman-restart.service inside the LXC (research Topic 6).
|
||||
_, _ = runRemote(fmt.Sprintf("pct exec %s -- systemctl enable --now podman-restart.service 2>/dev/null", vmid))
|
||||
|
||||
// Push step-ca root CA into the LXC (placeholder if absent locally).
|
||||
caPath := certpaths.CACertPath()
|
||||
caData, caErr := os.ReadFile(caPath)
|
||||
if caErr != nil {
|
||||
caData = []byte{}
|
||||
}
|
||||
// Write CA via pct exec heredoc.
|
||||
caDelim := "EOF_CA"
|
||||
_, _ = runRemote(fmt.Sprintf(
|
||||
"pct exec %s -- bash -c 'mkdir -p /etc/orca && cat > /etc/orca/step-ca-root.crt <<%s\\n%s\\n%s'",
|
||||
vmid, caDelim, string(caData), caDelim,
|
||||
))
|
||||
|
||||
// Render + write traefik static config inside the LXC.
|
||||
staticFiles, err := emitter.TraefikEmitter{}.RenderTraefikStaticConfig(emitter.TraefikStaticOpts{})
|
||||
if err == nil {
|
||||
for _, f := range staticFiles {
|
||||
delim := "EOF_TF"
|
||||
_, _ = runRemote(fmt.Sprintf(
|
||||
"pct exec %s -- bash -c 'mkdir -p /etc/traefik/dynamic && cat > %s <<%s\\n%s\\n%s'",
|
||||
vmid, f.Path, delim, f.Content, delim,
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
// Ensure podman orca-traefik container inside the LXC.
|
||||
traefikExecFn := func(cmd string) ([]byte, error) {
|
||||
return runRemote(fmt.Sprintf("pct exec %s -- bash -c %s 2>&1", vmid, shellQuote(cmd)))
|
||||
}
|
||||
if err := traefik.EnsureTraefikContainerRemote(ctx, "", traefikExecFn); err != nil {
|
||||
log.Warn("proxmox.native_ingress_lxc_traefik_failed", "err", err)
|
||||
}
|
||||
|
||||
// Render + apply nft on the PVE host with DNATTarget = LXC IP.
|
||||
nftFiles, err := emitter.NftEmitter{}.RenderNftConfig(emitter.NftClusterConfig{
|
||||
DNATTarget: lxcIP,
|
||||
})
|
||||
if err == nil {
|
||||
for _, f := range nftFiles {
|
||||
nftDelim := "EOF_NF"
|
||||
_, _ = runRemote(fmt.Sprintf("mkdir -p /etc/nftables.d && cat > %s <<%s\\n%s\\n%s",
|
||||
f.Path, nftDelim, f.Content, nftDelim))
|
||||
}
|
||||
_, _ = runRemote("nft add table inet orca-ingress 2>/dev/null || true")
|
||||
if out, err := runRemote("nft -f /etc/nftables.d/orca.nft 2>&1"); err != nil {
|
||||
log.Warn("proxmox.native_ingress_nft_apply_failed", "err", err, "output", string(out))
|
||||
}
|
||||
}
|
||||
|
||||
log.Info("proxmox.native_ingress_lxc_ok", "vmid", vmid, "ip", lxcIP)
|
||||
return nil
|
||||
}
|
||||
|
||||
// sshDialer is the dialer used by BootstrapProxmox. It's a package-level
|
||||
// variable so tests can override it with a fake SSH server.
|
||||
var sshDialer sshDialerType = defaultSSHDialer{}
|
||||
|
||||
@@ -0,0 +1,171 @@
|
||||
package proxmox
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/certpaths"
|
||||
"git.cloudinit.dev/coreci/orca/internal/emitter"
|
||||
"git.cloudinit.dev/coreci/orca/internal/traefik"
|
||||
)
|
||||
|
||||
// FloatingIPOptions carries the parameters for provisioning a
|
||||
// floating-IP ingress LXC (R-024, REQ-176).
|
||||
type FloatingIPOptions struct {
|
||||
// FloatingIP is the public IP assigned to the LXC's eth0.
|
||||
FloatingIP string
|
||||
// Gateway is the default gateway for the LXC.
|
||||
Gateway string
|
||||
// MAC is the MAC address for the LXC's net0 interface.
|
||||
MAC string
|
||||
// NetPrefix is the CIDR prefix for the floating IP (8-32).
|
||||
NetPrefix int
|
||||
// LXCTemplate is the LXC template (default "ubuntu-24.04").
|
||||
LXCTemplate string
|
||||
// VMID is the LXC container ID (default "201" for the ingress LXC).
|
||||
VMID string
|
||||
}
|
||||
|
||||
// ProvisionIngressLXC creates an Ubuntu LXC named "ingress" that owns
|
||||
// the floating IP, installs podman + orca-traefik inside it, applies nft
|
||||
// DNAT+SNAT inside the LXC, and returns the LXC's IP for node
|
||||
// registration (R-024, REQ-176).
|
||||
//
|
||||
// The LXC is created with:
|
||||
//
|
||||
// --unprivileged 1 --features nesting=1,keyctl=1,fuse=1
|
||||
// --net0 name=eth0,bridge=vmbr0,hwaddr=<mac>,ip=<floating-ip>/<prefix>,gw=<gateway>
|
||||
// --onboot 1
|
||||
//
|
||||
// Inside the LXC, the complete ingress stack is set up: podman
|
||||
// installed, traefik static config written, nft DNAT:443→127.0.0.1:8443
|
||||
// + postrouting masquerade applied, orca-traefik podman container
|
||||
// running with --network host.
|
||||
//
|
||||
// Idempotent: if the LXC already exists, it is not re-created (C-53).
|
||||
func ProvisionIngressLXC(ctx context.Context, runRemote func(string) ([]byte, error), opts FloatingIPOptions, log *slog.Logger) error {
|
||||
template := opts.LXCTemplate
|
||||
if template == "" {
|
||||
template = "ubuntu-24.04"
|
||||
}
|
||||
vmid := opts.VMID
|
||||
if vmid == "" {
|
||||
vmid = "201"
|
||||
}
|
||||
if opts.NetPrefix == 0 {
|
||||
opts.NetPrefix = 24
|
||||
}
|
||||
|
||||
// Validate required fields.
|
||||
if opts.FloatingIP == "" || opts.Gateway == "" || opts.MAC == "" {
|
||||
return fmt.Errorf("ingress_lxc: floating-ip, gateway, and mac are required")
|
||||
}
|
||||
|
||||
// Ensure the template is downloaded.
|
||||
_, _ = runRemote(fmt.Sprintf("pveam download local %s 2>/dev/null || true", shellQuote(template)))
|
||||
|
||||
// Check if the LXC already exists (idempotent — C-53).
|
||||
existOut, _ := runRemote(fmt.Sprintf("pct status %s 2>/dev/null || echo absent", vmid))
|
||||
existStr := strings.TrimSpace(string(existOut))
|
||||
if existStr == "absent" {
|
||||
log.Info("ingress_lxc.creating", "vmid", vmid, "hostname", "ingress", "ip", opts.FloatingIP)
|
||||
net0 := fmt.Sprintf("name=eth0,bridge=vmbr0,hwaddr=%s,ip=%s/%d,gw=%s",
|
||||
opts.MAC, opts.FloatingIP, opts.NetPrefix, opts.Gateway)
|
||||
createCmd := fmt.Sprintf(
|
||||
"pct create %s local:vztmpl/%s --hostname ingress --unprivileged 1 --features nesting=1,keyctl=1,fuse=1 --net0 %s --onboot 1 --memory 2048 --swap 0 --rootfs local:8 2>&1",
|
||||
vmid, shellQuote(template), net0,
|
||||
)
|
||||
if out, err := runRemote(createCmd); err != nil {
|
||||
return fmt.Errorf("pct create ingress LXC: %w (output: %s)", err, string(out))
|
||||
}
|
||||
if out, err := runRemote(fmt.Sprintf("pct start %s", vmid)); err != nil {
|
||||
return fmt.Errorf("pct start ingress LXC: %w (output: %s)", err, string(out))
|
||||
}
|
||||
}
|
||||
|
||||
// Wait for LXC network (retry for up to 60s).
|
||||
for i := 0; i < 12; i++ {
|
||||
ipOut, _ := runRemote(fmt.Sprintf("pct exec %s -- hostname -I 2>/dev/null", vmid))
|
||||
ipStr := strings.TrimSpace(string(ipOut))
|
||||
if ipStr != "" {
|
||||
break
|
||||
}
|
||||
time.Sleep(5 * time.Second)
|
||||
}
|
||||
log.Info("ingress_lxc.network_ready", "vmid", vmid, "ip", opts.FloatingIP)
|
||||
|
||||
// Install podman inside the LXC (C-53: idempotent).
|
||||
_, _ = runRemote(fmt.Sprintf(
|
||||
"pct exec %s -- bash -c 'command -v podman >/dev/null 2>&1 || (apt-get update -qq && apt-get install -y -qq podman conmon crun fuse-overlayfs nftables 2>&1)' 2>&1",
|
||||
vmid,
|
||||
))
|
||||
|
||||
// Enable podman-restart.service inside the LXC (research Topic 6).
|
||||
_, _ = runRemote(fmt.Sprintf("pct exec %s -- systemctl enable --now podman-restart.service 2>/dev/null", vmid))
|
||||
|
||||
// Push step-ca root CA into the LXC (C-60: CACertPath).
|
||||
caPath := certpaths.CACertPath()
|
||||
caData, caErr := os.ReadFile(caPath)
|
||||
if caErr != nil {
|
||||
caData = []byte{}
|
||||
}
|
||||
caDelim := "EOF_CA"
|
||||
_, _ = runRemote(fmt.Sprintf(
|
||||
"pct exec %s -- bash -c 'mkdir -p /etc/orca && cat > /etc/orca/step-ca-root.crt <<%s\\n%s\\n%s'",
|
||||
vmid, caDelim, string(caData), caDelim,
|
||||
))
|
||||
|
||||
// Render + write traefik static config inside the LXC (C-58).
|
||||
staticFiles, err := emitter.TraefikEmitter{}.RenderTraefikStaticConfig(emitter.TraefikStaticOpts{})
|
||||
if err == nil {
|
||||
for _, f := range staticFiles {
|
||||
delim := "EOF_TF"
|
||||
_, _ = runRemote(fmt.Sprintf(
|
||||
"pct exec %s -- bash -c 'mkdir -p /etc/traefik/dynamic && cat > %s <<%s\\n%s\\n%s'",
|
||||
vmid, f.Path, delim, f.Content, delim,
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
// Render + apply nft DNAT+SNAT INSIDE the LXC (DNATTarget =
|
||||
// 127.0.0.1 — traefik runs with --network host inside the LXC).
|
||||
nftFiles, err := emitter.NftEmitter{}.RenderNftConfig(emitter.NftClusterConfig{})
|
||||
if err == nil {
|
||||
for _, f := range nftFiles {
|
||||
delim := "EOF_NF"
|
||||
_, _ = runRemote(fmt.Sprintf(
|
||||
"pct exec %s -- bash -c 'mkdir -p /etc/nftables.d && cat > %s <<%s\\n%s\\n%s'",
|
||||
vmid, f.Path, delim, f.Content, delim,
|
||||
))
|
||||
}
|
||||
_, _ = runRemote(fmt.Sprintf("pct exec %s -- nft add table inet orca-ingress 2>/dev/null || true", vmid))
|
||||
_, _ = runRemote(fmt.Sprintf("pct exec %s -- nft -f /etc/nftables.d/orca.nft 2>&1", vmid))
|
||||
}
|
||||
|
||||
// Ensure podman orca-traefik container inside the LXC.
|
||||
traefikExecFn := func(cmd string) ([]byte, error) {
|
||||
return runRemote(fmt.Sprintf("pct exec %s -- bash -c %s 2>&1", vmid, shellQuote(cmd)))
|
||||
}
|
||||
if err := traefik.EnsureTraefikContainerRemote(ctx, "", traefikExecFn); err != nil {
|
||||
log.Warn("ingress_lxc.traefik_failed", "err", err)
|
||||
}
|
||||
|
||||
log.Info("ingress_lxc.provisioned", "vmid", vmid, "ip", opts.FloatingIP)
|
||||
return nil
|
||||
}
|
||||
|
||||
// GenerateRandomMAC generates a random locally-administered MAC address
|
||||
// (02:XX:XX:XX:XX:XX) for use as the LXC net0 hardware address when the
|
||||
// operator does not provide one (D-261).
|
||||
func GenerateRandomMAC() (string, error) {
|
||||
b := make([]byte, 5)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
return "", fmt.Errorf("generate MAC: %w", err)
|
||||
}
|
||||
return fmt.Sprintf("02:%02x:%02x:%02x:%02x:%02x", b[0], b[1], b[2], b[3], b[4]), nil
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
-- REQ-175 / R-024: add ingress_mode column to nodes.
|
||||
-- Values: '' (legacy/default), 'native' (proxmox native — traefik
|
||||
-- in LXC), 'floating-ip' (proxmox floating-IP — ingress LXC owns
|
||||
-- the floating IP). Defaults to empty string for backward
|
||||
-- compatibility with pre-v0.14 nodes.
|
||||
ALTER TABLE nodes ADD COLUMN ingress_mode TEXT NOT NULL DEFAULT '';
|
||||
+25
-2
@@ -145,8 +145,8 @@ assert "34 type_linux_available" \
|
||||
assert "35 status_deprecated" \
|
||||
'$ORCA status 2>&1 | grep -qi "deprecated"'
|
||||
|
||||
assert "36 traefik_installed" \
|
||||
'systemctl is-active orca-traefik 2>/dev/null | grep -q "active" || exit 77'
|
||||
assert "36 traefik_container_running" \
|
||||
'podman inspect --format "{{.State.Running}}" orca-traefik 2>/dev/null | grep -q "true" || exit 77'
|
||||
|
||||
assert "37 known_hosts_exists" \
|
||||
'test -f "$ORCA_HOME/known_hosts" || test -f "$ORCA_HOME/cluster/known_hosts"'
|
||||
@@ -154,6 +154,29 @@ assert "37 known_hosts_exists" \
|
||||
assert "38 master_key_exists" \
|
||||
'test -f "$ORCA_HOME/cluster/master.key" || test -f "$ORCA_HOME/cluster/master.key.sealed"'
|
||||
|
||||
# --- v0.14 ingress bootstrap assertions (R-024) ---
|
||||
|
||||
assert "40 ingress_nft_table" \
|
||||
'nft list table inet orca-ingress 2>/dev/null | grep -q "chain prerouting"'
|
||||
|
||||
assert "41 ingress_nft_dnat" \
|
||||
'nft list table inet orca-ingress 2>/dev/null | grep -q "dnat to"'
|
||||
|
||||
assert "42 ingress_nft_snat" \
|
||||
'nft list table inet orca-ingress 2>/dev/null | grep -q "masquerade"'
|
||||
|
||||
assert "43 ingress_dynamic_dir" \
|
||||
'test -d /etc/traefik/dynamic'
|
||||
|
||||
assert "44 ingress_step_ca" \
|
||||
'test -f /etc/orca/step-ca-root.crt'
|
||||
|
||||
assert "45 ingress_traefik_yml" \
|
||||
'test -f /etc/traefik/traefik.yml'
|
||||
|
||||
assert "46 ingress_doctor_pass" \
|
||||
'$ORCA doctor ingress 2>&1 | grep -q "PASS"'
|
||||
|
||||
# --- Report ---
|
||||
|
||||
echo "=========================================="
|
||||
|
||||
@@ -0,0 +1,163 @@
|
||||
package tests
|
||||
|
||||
import (
|
||||
"log/slog"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.cloudinit.dev/coreci/orca/internal/emitter"
|
||||
"git.cloudinit.dev/coreci/orca/internal/jobspec"
|
||||
"git.cloudinit.dev/coreci/orca/internal/proxmox"
|
||||
"git.cloudinit.dev/coreci/orca/internal/traefik"
|
||||
)
|
||||
|
||||
// TestNftEmitter_PostroutingAndDNATTarget (REQ-173) verifies the nft
|
||||
// emitter renders the postrouting masquerade chain and supports
|
||||
// DNATTarget substitution.
|
||||
func TestNftEmitter_PostroutingAndDNATTarget(t *testing.T) {
|
||||
files, err := emitter.NftEmitter{}.RenderNftConfig(emitter.NftClusterConfig{
|
||||
DNATTarget: "10.99.0.10",
|
||||
EnableSNAT: true,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("RenderNftConfig: %v", err)
|
||||
}
|
||||
c := files[0].Content
|
||||
if !strings.Contains(c, "chain postrouting") {
|
||||
t.Errorf("missing postrouting chain:\n%s", c)
|
||||
}
|
||||
if !strings.Contains(c, "masquerade") {
|
||||
t.Errorf("missing masquerade rule:\n%s", c)
|
||||
}
|
||||
if !strings.Contains(c, "dnat to 10.99.0.10:8443") {
|
||||
t.Errorf("missing custom DNAT target:\n%s", c)
|
||||
}
|
||||
}
|
||||
|
||||
// TestNftEmitter_PriorityMinus10 (research Topic 2) verifies the input
|
||||
// and forward chains use priority -10 for pve-firewall coexistence.
|
||||
func TestNftEmitter_PriorityMinus10(t *testing.T) {
|
||||
files, _ := emitter.NftEmitter{}.RenderNftConfig(emitter.NftClusterConfig{})
|
||||
c := files[0].Content
|
||||
if !strings.Contains(c, "hook input priority -10;") {
|
||||
t.Errorf("input chain should use priority -10:\n%s", c)
|
||||
}
|
||||
if !strings.Contains(c, "hook forward priority -10;") {
|
||||
t.Errorf("forward chain should use priority -10:\n%s", c)
|
||||
}
|
||||
}
|
||||
|
||||
// TestTraefikEmitter_TLSModel (REQ-172) verifies the dynamic config
|
||||
// emits tls: {} and does NOT contain certResolver (dropped in v0.14).
|
||||
func TestTraefikEmitter_TLSModel(t *testing.T) {
|
||||
spec := &jobspec.WorkloadSpec{
|
||||
Name: "test-svc",
|
||||
Kind: "Service",
|
||||
Ports: []jobspec.PortSpec{{Name: "http"}},
|
||||
}
|
||||
node := &emitter.Node{
|
||||
Hostname: "test-node",
|
||||
}
|
||||
files, err := emitter.TraefikEmitter{}.Render(spec, node)
|
||||
if err != nil {
|
||||
t.Fatalf("Render: %v", err)
|
||||
}
|
||||
c := files[0].Content
|
||||
if !strings.Contains(c, "tls: {}") {
|
||||
t.Errorf("missing tls: {} (v0.14 model):\n%s", c)
|
||||
}
|
||||
if strings.Contains(c, "certResolver: orca") {
|
||||
t.Errorf("certResolver: orca should be removed (v0.14):\n%s", c)
|
||||
}
|
||||
}
|
||||
|
||||
// TestTraefikImageRef verifies the image reference resolution for the
|
||||
// orca-traefik podman container.
|
||||
func TestTraefikImageRef(t *testing.T) {
|
||||
ref := traefik.ImageRef("v0.13.7")
|
||||
want := "git.cloudinit.dev/coreci/orca-traefik:v0.13.7"
|
||||
if ref != want {
|
||||
t.Errorf("ImageRef(v0.13.7) = %q, want %q", ref, want)
|
||||
}
|
||||
// Dev build falls back to latest.
|
||||
ref = traefik.ImageRef("dev")
|
||||
if ref != "git.cloudinit.dev/coreci/orca-traefik:latest" {
|
||||
t.Errorf("ImageRef(dev) = %q, want latest", ref)
|
||||
}
|
||||
}
|
||||
|
||||
// TestProxmox_FloatingIP_LXC_ProvisioningCommands (REQ-176) verifies
|
||||
// the ProvisionIngressLXC function sends the correct pct create
|
||||
// command with the right net0 parameters.
|
||||
func TestProxmox_FloatingIP_LXC_ProvisioningCommands(t *testing.T) {
|
||||
var cmds []string
|
||||
execFn := func(cmd string) ([]byte, error) {
|
||||
cmds = append(cmds, cmd)
|
||||
// Simulate: pct status returns "absent" on first call, then OK.
|
||||
if strings.Contains(cmd, "pct status 201") {
|
||||
return []byte("absent\n"), nil
|
||||
}
|
||||
if strings.Contains(cmd, "pct create") {
|
||||
return []byte(""), nil
|
||||
}
|
||||
if strings.Contains(cmd, "pct start 201") {
|
||||
return []byte(""), nil
|
||||
}
|
||||
if strings.Contains(cmd, "hostname -I") {
|
||||
return []byte("203.0.113.10\n"), nil
|
||||
}
|
||||
return []byte(""), nil
|
||||
}
|
||||
err := proxmox.ProvisionIngressLXC(nil, execFn, proxmox.FloatingIPOptions{
|
||||
FloatingIP: "203.0.113.10",
|
||||
Gateway: "203.0.113.1",
|
||||
MAC: "02:01:02:03:04:05",
|
||||
NetPrefix: 24,
|
||||
LXCTemplate: "ubuntu-24.04",
|
||||
}, slog.Default())
|
||||
if err != nil {
|
||||
t.Fatalf("ProvisionIngressLXC: %v", err)
|
||||
}
|
||||
// Verify pct create has the right net0 params.
|
||||
foundCreate := false
|
||||
for _, c := range cmds {
|
||||
if strings.Contains(c, "pct create") {
|
||||
foundCreate = true
|
||||
if !strings.Contains(c, "hostname ingress") {
|
||||
t.Errorf("pct create missing hostname ingress: %s", c)
|
||||
}
|
||||
if !strings.Contains(c, "hwaddr=02:01:02:03:04:05") {
|
||||
t.Errorf("pct create missing hwaddr: %s", c)
|
||||
}
|
||||
if !strings.Contains(c, "ip=203.0.113.10/24") {
|
||||
t.Errorf("pct create missing ip: %s", c)
|
||||
}
|
||||
if !strings.Contains(c, "gw=203.0.113.1") {
|
||||
t.Errorf("pct create missing gw: %s", c)
|
||||
}
|
||||
if !strings.Contains(c, "nesting=1,keyctl=1,fuse=1") {
|
||||
t.Errorf("pct create missing features (research Topic 3): %s", c)
|
||||
}
|
||||
}
|
||||
}
|
||||
if !foundCreate {
|
||||
t.Errorf("pct create command not sent\ncommands: %v", cmds)
|
||||
}
|
||||
}
|
||||
|
||||
// TestProxmox_GenerateRandomMAC (D-261) verifies MAC generation produces
|
||||
// a valid locally-administered MAC.
|
||||
func TestProxmox_GenerateRandomMAC(t *testing.T) {
|
||||
mac, err := proxmox.GenerateRandomMAC()
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateRandomMAC: %v", err)
|
||||
}
|
||||
if !strings.HasPrefix(mac, "02:") {
|
||||
t.Errorf("MAC should start with 02: (locally administered): %s", mac)
|
||||
}
|
||||
// Verify it's 6 octets.
|
||||
parts := strings.Split(mac, ":")
|
||||
if len(parts) != 6 {
|
||||
t.Errorf("MAC should have 6 octets: %s", mac)
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user