orca/docs/ingress.md

# Orca Ingress & Traefik Guide

This document explains how Orca configures ingress via Traefik dynamic
configuration. It covers the service→Traefik mapping, the R-007
socket-vs-TCP-bind model, atomic reload, drain, TLS, and a worked
example.

> **Canonical path (v0.9)**: Orca generates Traefik dynamic
> configuration files via the `TraefikEmitter`. The `kind: Service`
> workload implies a Traefik route. The emitter renders one YAML file
> per Service; Traefik watches the dynamic config directory and reloads
> atomically on change.

## The model

A `kind: Service` jobspec **implies** a Traefik route (D-175). `Job`
and `DaemonSet` do **not** carry a Traefik route by default — a
`service:` block on a `Job` is rejected by the validator.

When `orca job run` submits a `kind: Service` workload, the
`TraefikEmitter` renders a Traefik dynamic config file at:

```
/etc/traefik/dynamic/orca-<service-name>.yaml
```

This file contains:
- One **router** (`orca-<name>`) with a `PathPrefix` rule and TLS config.
- One **service** (`orca-<name>`) as a `loadBalancer` with one **server**
  per port, pointing at the workload's Unix socket (or TCP port).
- A **healthCheck** stanza when the `health:` block is present.

Traefik watches `/etc/traefik/dynamic/` via `fsnotify` and reloads
whenever a file changes. Orca writes config atomically (write-tmp +
rename) so Traefik sees a single `IN_MOVED_TO` event and never observes
a half-written file.

## R-007: socket vs TCP bind

Orca workloads bind to a **Unix socket** by default, not a TCP port.
This is the R-007 security model: loopback-only by default, no network
exposure.

### Default: Unix socket

When `service.bind` is empty (default), the workload binds a Unix
socket at:

```
/run/orca/alloc-<alloc-id>/port-<port-name>.sock
```

systemd creates `/run/orca/alloc-<alloc-id>/` via
`RuntimeDirectory=orca/alloc-<alloc-id>` (mode 0750, owned by
`orca:orca`). The Traefik backend server URL is:

```yaml
servers:
  - url: "unix:///run/orca/alloc-<alloc-id>/port-<port-name>.sock"
```

### TCP opt-in: `service.bind: 127.0.0.1`

When `service.bind: 127.0.0.1` is set, the workload binds a TCP port
directly (loopback only). The emitter adds an `ExecStartPre` marker to
the systemd unit so the bind mode is visible:

```ini
ExecStartPre=/bin/echo orca: bind 127.0.0.1 port <name> (tcp, R-007 opt-in)
```

`service.bind` must be a valid IP address. Empty (socket default) or
`127.0.0.1` (TCP opt-in) are the documented values; any other valid IP
is accepted but the bind happens in the process, not the emitter.

## Generated Traefik YAML

For a Service named `web` with port `http`:

```yaml
http:
  routers:
    orca-web:
      rule: PathPrefix("/web")
      service: orca-web
      tls:
        certResolver: orca
        domains:
          - main: "cluster.orca.local"
  services:
    orca-web:
      loadBalancer:
        servers:
          - url: "unix:///run/orca/alloc-<alloc-id>/port-http.sock"
        healthCheck:
          path: /healthz
          interval: 5s
          timeout: 1s
```

- One router per Service, named `orca-<service-name>`.
- Router rule: `PathPrefix("/<service-name>")`.
- TLS: `certResolver: orca`, trust domain `cluster.orca.local`
  (placeholder; step-ca provisioner overrides in v0.11).
- One service per Service, named `orca-<service-name>`.
- One server per port, URL is `unix://<socket-path>`.
- `healthCheck` stanza present when `health:` block is set (required
  for Service). Path is `/healthz`; interval and timeout come from the
  `health:` block.

## Atomic reload (gate C-10)

Orca writes Traefik config atomically to avoid Traefik observing a
half-written file:

1. Write to `<path>.tmp` via `WriteFileIdempotent` (write + fsync).
2. `mv -f <path>.tmp <path>` (atomic POSIX rename).

Traefik's `fsnotify` watcher sees a single `IN_MOVED_TO` event and
reloads. If the new config is malformed, Traefik logs an error and
**holds last-good config** — the cluster keeps serving traffic on the
previous config.

## Drain

`RenderDrain` produces the same Traefik YAML with `weight: 0` on every
server in the load balancer:

```yaml
servers:
  - url: "unix:///run/orca/alloc-<alloc-id>/port-http.sock"
    weight: 0
```

Traefik stops sending traffic to the drained backend. The workload
keeps running; drain is reversible (re-submit the normal config to
restore traffic).

## TLS

- **v0.14 model**: `tls: {}` in dynamic config (no certResolver).
  Traefik v3.3 `certificatesResolvers` only supports `acme` and
  `tailscale` — not CA-file-based. The `certResolver: orca` reference
  from v0.11 was broken (research finding). v0.14 emits `tls: {}`
  (traefik uses its default self-signed cert). Real mTLS via dynamic
  `tls.certificates` + `tls.options.default.clientAuth.caFiles` is
  deferred to v0.15.
- **Step-ca root CA**: mounted at `/etc/orca/step-ca-root.crt` in the
  traefik container. v0.14 does not use it for TLS termination (it's
  a placeholder for v0.15 mTLS).

## R-024: Podman Traefik Container (v0.14)

As of v0.14, Traefik runs as a **podman container** from the custom
`orca-traefik` image (published per release). The v0.13 binary+systemd
install is replaced.

### Three topologies

1. **Linux**: host → nft DNAT → `podman run orca-traefik` (`--network host`)
2. **Proxmox Native** (`--ingress-mode native`, default): PVE host →
   nft DNAT → LXC (nesting=1,keyctl=1,fuse=1) → `podman run orca-traefik`
3. **Proxmox Floating-IP** (`--ingress-mode floating-ip`): LXC owns
   the floating IP → nft inside LXC → `podman run orca-traefik`

### Container configuration

```bash
podman run -d --name orca-traefik --restart=unless-stopped \
  --network host \
  -v /etc/traefik/traefik.yml:/etc/traefik/traefik.yml:ro \
  -v /etc/traefik/dynamic:/etc/traefik/dynamic:ro \
  -v /etc/orca/step-ca-root.crt:/etc/orca/step-ca-root.crt:ro \
  git.cloudinit.dev/coreci/orca-traefik:<version>
```

- `--network host`: traefik binds 127.0.0.1:8080/8443 on host/LXC loopback
- `--restart=unless-stopped`: survives reboot via `podman-restart.service`
- No `:Z` SELinux flag (research Topic 7)
- Static config mounted `:ro` (overrides baked image default, preserves
  `traefik-on-public-ip` opt-out, REQ-100)

### nft ruleset

The nft emitter (`internal/emitter/nft.go`) renders `/etc/nftables.d/orca.nft`:

- DNAT `:443` → `<DNATTarget>:8443` (default 127.0.0.1; LXC IP for native)
- DNAT `:80` → `<DNATTarget>:8080`
- SNAT/MASQUERADE: `ip saddr 127.0.0.0/8 oifname != "lo" masquerade`
- Input/forward chains at priority -10 (pve-firewall coexistence)

### `orca doctor ingress`

```bash
orca doctor ingress              # check localhost
orca doctor ingress --peer <name> # check remote peer
```

Verifies: podman container running, nft DNAT+SNAT, dynamic dir exists,
step-ca root CA present.

### Dockerfile.traefik

```dockerfile
FROM traefik:v3.3.0
COPY docker/orca-traefik/traefik.yml /etc/traefik/traefik.yml
CMD ["--configFile=/etc/traefik/traefik.yml"]
```

Built + published per release alongside the orca image
(`scripts/release.sh` + `.coreci.yml container-publish-traefik`).

## Health checks

The `health:` block (required for `Service`) maps to the Traefik
`healthCheck` stanza:

```yaml
health:
  check_type: http
  interval: 5s
  timeout: 1s
  unhealthy_threshold: 2
```

→

```yaml
healthCheck:
  path: /healthz
  interval: 5s
  timeout: 1s
```

Traefik polls each backend's `/healthz` at the configured interval. An
unhealthy backend is removed from the load balancer pool until it
passes the health check again.

## Worked example

See [examples/full-stack/](../examples/full-stack/) for a complete
multi-service stack with ingress configured:
- `web-app.md` — frontend Service (socket bind, PathPrefix route)
- `api.md` — backend API Service (TCP opt-in, `127.0.0.1` bind)
- `examples/full-stack/rendered/traefik-dynamic-web-app.yaml` — the
  Traefik config Orca generates

## v0.11 forward (limitations)

The following are not yet implemented in v0.9 and will land in v0.11:

- **`service.host` / `service.route_id`**: stored on the `ServiceBlock`
  but not yet consumed by the `TraefikEmitter`. The router rule is
  hardcoded `PathPrefix("/<name>")`. Custom host-based routing lands in
  v0.11.
- **Socket activation**: real socket-activation (socket unit files, fd
  passing) lands in v0.11-P08. The current emitter renders the
  `RuntimeDirectory` + socket path comments but does not create socket
  units.
- **Transactional update execution**: the `update:` block's rolling/
  canary/blue-green plan is computed by the emitter but not yet
  executed transactionally. Transactional execution lands in
  v0.11-P10.
- **SPIFFE SVID minting**: workload identity via step-ca SVIDs lands in
  v0.11-P01.5 (gate C-08).
- **Secrets in env**: `env: { KEY: { from: "secret:..." } }` resolution
  to `EnvironmentFile=`/`LoadCredential=` lands in v0.11-P03.

## See also

- [docs/cli.md](cli.md) — CLI reference
- [docs/jobspec.md](jobspec.md) — Jobspec reference (`service:`, `health:`, `ports:` blocks)
- [examples/full-stack/](../examples/full-stack/) — Full-stack example with ingress