orca/docs/ingress.md

Orca Ingress & Traefik Guide

This document explains how Orca configures ingress via Traefik dynamic configuration. It covers the service→Traefik mapping, the R-007 socket-vs-TCP-bind model, atomic reload, drain, TLS, and a worked example.

Canonical path (v0.9): Orca generates Traefik dynamic configuration files via the TraefikEmitter. The kind: Service workload implies a Traefik route. The emitter renders one YAML file per Service; Traefik watches the dynamic config directory and reloads atomically on change.

The model

A kind: Service jobspec implies a Traefik route (D-175). Job and DaemonSet do not carry a Traefik route by default — a service: block on a Job is rejected by the validator.

When orca job run submits a kind: Service workload, the TraefikEmitter renders a Traefik dynamic config file at:

/etc/traefik/dynamic/orca-<service-name>.yaml

This file contains:

• One router (orca-<name>) with a PathPrefix rule and TLS config.
• One service (orca-<name>) as a loadBalancer with one server per port, pointing at the workload's Unix socket (or TCP port).
• A healthCheck stanza when the health: block is present.

Traefik watches /etc/traefik/dynamic/ via fsnotify and reloads whenever a file changes. Orca writes config atomically (write-tmp + rename) so Traefik sees a single IN_MOVED_TO event and never observes a half-written file.

R-007: socket vs TCP bind

Orca workloads bind to a Unix socket by default, not a TCP port. This is the R-007 security model: loopback-only by default, no network exposure.

Default: Unix socket

When service.bind is empty (default), the workload binds a Unix socket at:

/run/orca/alloc-<alloc-id>/port-<port-name>.sock

systemd creates /run/orca/alloc-<alloc-id>/ via RuntimeDirectory=orca/alloc-<alloc-id> (mode 0750, owned by orca:orca). The Traefik backend server URL is:

servers:
  - url: "unix:///run/orca/alloc-<alloc-id>/port-<port-name>.sock"

TCP opt-in: service.bind: 127.0.0.1

When service.bind: 127.0.0.1 is set, the workload binds a TCP port directly (loopback only). The emitter adds an ExecStartPre marker to the systemd unit so the bind mode is visible:

ExecStartPre=/bin/echo orca: bind 127.0.0.1 port <name> (tcp, R-007 opt-in)

service.bind must be a valid IP address. Empty (socket default) or 127.0.0.1 (TCP opt-in) are the documented values; any other valid IP is accepted but the bind happens in the process, not the emitter.

Generated Traefik YAML

For a Service named web with port http:

http:
  routers:
    orca-web:
      rule: PathPrefix("/web")
      service: orca-web
      tls:
        certResolver: orca
        domains:
          - main: "cluster.orca.local"
  services:
    orca-web:
      loadBalancer:
        servers:
          - url: "unix:///run/orca/alloc-<alloc-id>/port-http.sock"
        healthCheck:
          path: /healthz
          interval: 5s
          timeout: 1s

• One router per Service, named orca-<service-name>.
• Router rule: PathPrefix("/<service-name>").
• TLS: certResolver: orca, trust domain cluster.orca.local (placeholder; step-ca provisioner overrides in v0.11).
• One service per Service, named orca-<service-name>.
• One server per port, URL is unix://<socket-path>.
• healthCheck stanza present when health: block is set (required for Service). Path is /healthz; interval and timeout come from the health: block.

Atomic reload (gate C-10)

Orca writes Traefik config atomically to avoid Traefik observing a half-written file:

1. Write to <path>.tmp via WriteFileIdempotent (write + fsync).
2. mv -f <path>.tmp <path> (atomic POSIX rename).

Traefik's fsnotify watcher sees a single IN_MOVED_TO event and reloads. If the new config is malformed, Traefik logs an error and holds last-good config — the cluster keeps serving traffic on the previous config.

Drain

RenderDrain produces the same Traefik YAML with weight: 0 on every server in the load balancer:

servers:
  - url: "unix:///run/orca/alloc-<alloc-id>/port-http.sock"
    weight: 0

Traefik stops sending traffic to the drained backend. The workload keeps running; drain is reversible (re-submit the normal config to restore traffic).

TLS

• v0.14 model: tls: {} in dynamic config (no certResolver). Traefik v3.3 certificatesResolvers only supports acme and tailscale — not CA-file-based. The certResolver: orca reference from v0.11 was broken (research finding). v0.14 emits tls: {} (traefik uses its default self-signed cert). Real mTLS via dynamic tls.certificates + tls.options.default.clientAuth.caFiles is deferred to v0.15.
• Step-ca root CA: mounted at /etc/orca/step-ca-root.crt in the traefik container. v0.14 does not use it for TLS termination (it's a placeholder for v0.15 mTLS).

R-024: Podman Traefik Container (v0.14)

As of v0.14, Traefik runs as a podman container from the custom orca-traefik image (published per release). The v0.13 binary+systemd install is replaced.

Three topologies

1. Linux: host → nft DNAT → podman run orca-traefik (--network host)
2. Proxmox Native (--ingress-mode native, default): PVE host → nft DNAT → LXC (nesting=1,keyctl=1,fuse=1) → podman run orca-traefik
3. Proxmox Floating-IP (--ingress-mode floating-ip): LXC owns the floating IP → nft inside LXC → podman run orca-traefik

Container configuration

podman run -d --name orca-traefik --restart=unless-stopped \
  --network host \
  -v /etc/traefik/traefik.yml:/etc/traefik/traefik.yml:ro \
  -v /etc/traefik/dynamic:/etc/traefik/dynamic:ro \
  -v /etc/orca/step-ca-root.crt:/etc/orca/step-ca-root.crt:ro \
  git.cloudinit.dev/coreci/orca-traefik:<version>

• --network host: traefik binds 127.0.0.1:8080/8443 on host/LXC loopback
• --restart=unless-stopped: survives reboot via podman-restart.service
• No :Z SELinux flag (research Topic 7)
• Static config mounted :ro (overrides baked image default, preserves traefik-on-public-ip opt-out, REQ-100)

nft ruleset

The nft emitter (internal/emitter/nft.go) renders /etc/nftables.d/orca.nft:

• DNAT :443 → <DNATTarget>:8443 (default 127.0.0.1; LXC IP for native)
• DNAT :80 → <DNATTarget>:8080
• SNAT/MASQUERADE: ip saddr 127.0.0.0/8 oifname != "lo" masquerade
• Input/forward chains at priority -10 (pve-firewall coexistence)

orca doctor ingress

orca doctor ingress              # check localhost
orca doctor ingress --peer <name> # check remote peer

Verifies: podman container running, nft DNAT+SNAT, dynamic dir exists, step-ca root CA present.

Dockerfile.traefik

FROM traefik:v3.3.0
COPY docker/orca-traefik/traefik.yml /etc/traefik/traefik.yml
CMD ["--configFile=/etc/traefik/traefik.yml"]

Built + published per release alongside the orca image (scripts/release.sh + .coreci.yml container-publish-traefik).

Health checks

The health: block (required for Service) maps to the Traefik healthCheck stanza:

health:
  check_type: http
  interval: 5s
  timeout: 1s
  unhealthy_threshold: 2

→

healthCheck:
  path: /healthz
  interval: 5s
  timeout: 1s

Traefik polls each backend's /healthz at the configured interval. An unhealthy backend is removed from the load balancer pool until it passes the health check again.

Worked example

See examples/full-stack/ for a complete multi-service stack with ingress configured:

• web-app.md — frontend Service (socket bind, PathPrefix route)
• api.md — backend API Service (TCP opt-in, 127.0.0.1 bind)
• examples/full-stack/rendered/traefik-dynamic-web-app.yaml — the Traefik config Orca generates

v0.11 forward (limitations)

The following are not yet implemented in v0.9 and will land in v0.11:

• service.host / service.route_id: stored on the ServiceBlock but not yet consumed by the TraefikEmitter. The router rule is hardcoded PathPrefix("/<name>"). Custom host-based routing lands in v0.11.
• Socket activation: real socket-activation (socket unit files, fd passing) lands in v0.11-P08. The current emitter renders the RuntimeDirectory + socket path comments but does not create socket units.
• Transactional update execution: the update: block's rolling/ canary/blue-green plan is computed by the emitter but not yet executed transactionally. Transactional execution lands in v0.11-P10.
• SPIFFE SVID minting: workload identity via step-ca SVIDs lands in v0.11-P01.5 (gate C-08).
• Secrets in env: env: { KEY: { from: "secret:..." } } resolution to EnvironmentFile=/LoadCredential= lands in v0.11-P03.

See also

• docs/cli.md — CLI reference
• docs/jobspec.md — Jobspec reference (service:, health:, ports: blocks)
• examples/full-stack/ — Full-stack example with ingress