diff --git a/.ciagent/CHECKPOINT.json b/.ciagent/CHECKPOINT.json index 817b7ff..744dfe1 100644 --- a/.ciagent/CHECKPOINT.json +++ b/.ciagent/CHECKPOINT.json @@ -1,14 +1,14 @@ { - "phase": 3, - "stage": "verify", + "phase": 4, + "stage": "execute", "milestone": "v0.10", "milestone_slug": "docs-cli-examples", "phase_role": "execution", "attempts": 0, - "updated_at": "2026-08-05T21:05:00Z", + "updated_at": "2026-08-05T21:15:00Z", "milestone_complete": false, "ship": { - "tag": "v0.9.2", + "tag": "v0.9.3", "merged_to_main": false, "milestone_branch_deleted": false, "all_phase_branches_deleted": true diff --git a/README.md b/README.md index 2d0b214..87299b5 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,9 @@ Offline/CLI-first orchestration engine inspired by HashiCorp Nomad, far simpler ## Status -**v0.1: Foundation** — see [.ciagent/ROADMAP.md](.ciagent/ROADMAP.md) for the 6-phase plan. +**v0.9: Re-architecture Foundation — COMPLETE** | **v0.10: Docs & Install Hardening — IN PROGRESS** + +See [.ciagent/ROADMAP.md](.ciagent/ROADMAP.md) for the full roadmap. ## Pillars @@ -28,7 +30,10 @@ curl -fsSL https://git.cloudinit.dev/coreci/orca/raw/branch/main/scripts/install curl -fsSL https://git.cloudinit.dev/coreci/orca/raw/branch/main/scripts/install.sh | sudo bash -s -- --system # Pin a specific version -curl -fsSL https://git.cloudinit.dev/coreci/orca/raw/branch/main/scripts/install.sh | bash -s -- --version v0.4.2 +curl -fsSL https://git.cloudinit.dev/coreci/orca/raw/branch/main/scripts/install.sh | bash -s -- --version v0.9.1 + +# Dry-run: check what would be installed without writing +curl -fsSL https://git.cloudinit.dev/coreci/orca/raw/branch/main/scripts/install.sh | bash -s -- --check ``` Then initialize local state and verify: @@ -54,27 +59,67 @@ config, database, and certificates in the namespace dir: ```bash curl -fsSL https://git.cloudinit.dev/coreci/orca/raw/branch/main/scripts/install.sh | bash -# → "updated orca from v0.4.1 to v0.4.2" +# → "updated orca from v0.8.15 to v0.9.1" ``` ## Subcommands -| Command | Description | Status | -|---------|-------------|--------| -| `orca version` | Print version info | ✅ Phase 1 | -| `orca init` | Initialize local orca state | ✅ Phase 1 (stub) | -| `orca status` | Show orca daemon status | ✅ Phase 1 (stub) | -| `orca node` | Node management (`join`, `leave`, `list`) | Phase 2 | -| `orca job` | Job management (`run`, `list`, `stop`, `logs`) | Phase 3 | +| Command | Description | Since | +|---------|-------------|-------| +| `orca init` | Initialize local orca state (full bootstrap) | v0.6 | +| `orca version` | Print version info | v0.1 | +| `orca status` | Show orca daemon status (**deprecated** v0.9) | v0.1 | +| `orca job run` | Run a job from a spec file (`.md`/`.yaml`/`.hcl`) | v0.1 | +| `orca job list` | List all jobs (`--watch` for streaming) | v0.1 | +| `orca job stop` | Stop a running job | v0.1 | +| `orca job logs` | Show task output for a job | v0.1 | +| `orca node join` | Join a node (`--type proxmox` for SSH-push) | v0.2 | +| `orca node leave` | Remove a node from the registry | v0.2 | +| `orca node list` | List all nodes (`--watch` for streaming) | v0.2 | +| `orca node key-reset` | Reset SSH known_hosts entry for a node | v0.8 | +| `orca node capacity` | Manage node capacity (show/set/list) | v0.2 | +| `orca ns list` | List all namespaces | v0.9 | +| `orca ns create` | Create a namespace directory + ns.md | v0.9 | +| `orca ns delete` | Remove an empty namespace | v0.9 | +| `orca ns inspect` | Print effective chain, merged env, constraints | v0.9 | +| `orca ns validate` | Run cycle + missing-parent + schema checks | v0.9 | +| `orca doctor` | Run self-checks (cert/network/db/os/proxmox) | v0.2 | +| `orca audit list` | View audit log entries | v0.1 | +| `orca daemon` | Run the daemon (**deprecated** v0.9) | v0.1 | +| `orca cert` | Manage certificates (**deprecated** v0.9) | v0.2 | + +See [docs/cli.md](docs/cli.md) for the full CLI reference with all flags and examples. + +## Documentation + +| Document | Description | +|----------|-------------| +| [docs/cli.md](docs/cli.md) | CLI reference — every command, flag, and example | +| [docs/jobspec.md](docs/jobspec.md) | Jobspec reference — markdown frontmatter schema | +| [docs/ingress.md](docs/ingress.md) | Ingress guide — Traefik configuration | +| [docs/namespace.md](docs/namespace.md) | Namespace and path layout | +| [docs/install.md](docs/install.md) | Installation guide | +| [docs/docker.md](docs/docker.md) | Docker image guide | +| [docs/security-scanning.md](docs/security-scanning.md) | Security scanning tools | + +## Examples + +| Example | Description | +|---------|-------------| +| [examples/full-stack/](examples/full-stack/) | Full-stack deployment with ingress (5 services + rendered artifacts) | ## Development ```bash -make build # Build binary to ./bin/orca -make test # Run tests with race detection -make lint # Run golangci-lint -make fmt # Format code -make release # Build + create Gitea release (Phase 6) +make build # Build binary to ./bin/orca +make test # Run tests +make test-race # Run tests with race detection +make lint # Run gofmt + go vet + shellcheck +make fmt # Format code +make security-scan # Run gosec + govulncheck + gitleaks +make verify-reqs # Assert ROADMAP ↔ REQUIREMENTS consistency +make changelog # Generate CHANGELOG.md from ---ci--- blocks +make release # Build + create Gitea release (VERSION required) ``` ## Architecture @@ -83,4 +128,4 @@ See [.ciagent/ARCHITECTURE.md](.ciagent/ARCHITECTURE.md) for full architecture d ## License -MIT — see [LICENSE](LICENSE). +MIT — see [LICENSE](LICENSE). \ No newline at end of file diff --git a/docs/namespace.md b/docs/namespace.md index 6fa751b..4e59b8e 100644 --- a/docs/namespace.md +++ b/docs/namespace.md @@ -1,96 +1,177 @@ # Namespace and Paths -Orca stores all on-disk state (SQLite database, CA certs, server certs, -config) under a single **namespace root** directory. This document -describes how that root is resolved and how to override it. +Orca stores all on-disk state under a single **namespace root** +directory. The v0.9 re-architecture introduced a multi-namespace +layout (R-002) where each namespace is a self-contained directory tree +with its own database, jobs, allocs, env, and secrets. A `cluster/` +directory holds cluster-wide artifacts shared across namespaces. -## Default: User-Level (`~/.orca`) +> **v0.9 layout (canonical)**: This document describes the v0.9 +> multi-namespace layout. The v0.8 flat layout (`orca.db`, `ca.crt`, +> `server.crt` at the root) is deprecated and will be removed in +> v0.11. See [v0.8 flat layout](#deprecated-v08-flat-layout) below. -By default, the namespace root is `~/.orca` (i.e., `$HOME/.orca`). -All orca state lives under this directory: +## Namespace root resolution -| Path | Contents | -|------|----------| -| `~/.orca/orca.db` | SQLite database (jobs, nodes, tasks, audit log, capacity) | -| `~/.orca/ca.crt` | CA certificate (PEM, mode 0644) | -| `~/.orca/ca.key` | CA private key (PEM, mode 0600) | -| `~/.orca/server.crt` | Server certificate (PEM, mode 0644) | -| `~/.orca/server.key` | Server private key (PEM, mode 0600) | - -## Override: `ORCA_HOME` Environment Variable (REQ-041) - -Set the `ORCA_HOME` environment variable to change the namespace root -for **all** orca components (database, certs, init, daemon): - -```bash -export ORCA_HOME=/var/lib/orca -orca init # creates /var/lib/orca/ -orca daemon # reads /var/lib/orca/orca.db -orca cert ca-init # writes CA to /var/lib/orca/ -``` - -This is the single source of truth for the namespace root. Every -component that reads or writes on-disk state resolves the root via -`ORCA_HOME` (falling back to `~/.orca` when unset). - -### Use cases - -- **Testing**: point `ORCA_HOME` at a temp directory. -- **Multi-instance**: run multiple orca daemons on the same host with - different `ORCA_HOME` values. -- **Custom layout**: store state on a mounted volume - (`ORCA_HOME=/mnt/orca-data`). - -## System-Level: `--system` Flag (REQ-042) - -The `--system` persistent flag selects the system-level namespace root -`/root/.orca`. This is intended for root-owned system deployments -(where orca runs as a system service under root): - -```bash -sudo orca --system init # creates /root/.orca/ -sudo orca --system daemon # reads /root/.orca/orca.db -sudo orca --system cert ca-init # writes CA to /root/.orca/ -``` - -The `--system` flag is equivalent to setting `ORCA_HOME=/root/.orca`, -but it is a CLI convenience that does not require exporting an env var. -If `ORCA_HOME` is already set to a different value, `--system` returns -an error (to avoid silent namespace mismatches). - -### Path layout - -System-level uses the same directory shape as user-level, just under -`/root/.orca` instead of `~/.orca`: - -| Path | Contents | -|------|----------| -| `/root/.orca/orca.db` | SQLite database | -| `/root/.orca/ca.crt` | CA certificate | -| `/root/.orca/ca.key` | CA private key | -| `/root/.orca/server.crt` | Server certificate | -| `/root/.orca/server.key` | Server private key | - -## Resolution Order +The namespace root is resolved in this order: 1. If `--system` flag is passed → root is `/root/.orca` (errors if `ORCA_HOME` is set to a conflicting value). 2. Else if `ORCA_HOME` is set → root is `$ORCA_HOME`. 3. Else → root is `~/.orca` (`$HOME/.orca`). -## `ORCA_DB` Override +### `ORCA_HOME` (REQ-041) -For finer-grained control, `ORCA_DB` overrides **only** the database -path (not the cert paths). This is primarily a testing affordance. When -`ORCA_DB` is set, certs still resolve under `ORCA_HOME` (or `~/.orca`). +Set the `ORCA_HOME` environment variable to change the namespace root +for all orca components: + +```bash +export ORCA_HOME=/var/lib/orca +orca init # creates /var/lib/orca/ +orca ns create prod +``` + +### `--system` (REQ-042) + +The `--system` persistent flag selects the system-level namespace root +`/root/.orca`: + +```bash +sudo orca --system init # creates /root/.orca/ +sudo orca --system ns list +``` + +If `ORCA_HOME` is already set to a different value, `--system` returns +an error (to avoid silent namespace mismatches). + +## v0.9 multi-namespace layout (R-002) + +``` +$ORCA_HOME/ +├── cluster/ # cluster-wide (NOT a workload namespace) +│ ├── ca.crt, ca.key # step-ca root (R-006, D-101) +│ ├── master.key # AES-256-GCM root (R-011, mode 0600) +│ ├── config.md # Markdown frontmatter config (R-014) +│ ├── known_hosts # SSH known_hosts (D-035) +│ ├── orca_ssh_key # orca SSH private key (D-037) +│ ├── orca_ssh_key.pub # orca SSH public key +│ ├── peers// # per-peer directory +│ ├── txns/ # cluster transaction log (R-016) +│ └── state/ # cluster state +├── _defaults/ # implicit root namespace (always exists) +│ ├── ns.md # namespace frontmatter (kind: Namespace) +│ ├── .env # per-namespace env +│ ├── .env.secrets # encrypted secrets +│ ├── db/orca.db # per-namespace SQLite database +│ ├── jobs/ # submitted jobspecs +│ └── alloc/ # allocation state +├── / # operator-created (e.g., prod, staging) +│ ├── ns.md +│ ├── .env, .env.secrets +│ ├── db/orca.db +│ ├── jobs/, alloc/ +│ └── syncthing/ # Syncthing config (if replicated volumes) +└── orca_cache.db # CLI-side cache (R-008) +``` + +### Key points + +- **`_defaults/`** is the implicit root namespace (D-159). It always + exists. Every namespace inherits from `_defaults` and cannot opt out + (D-185, D-187). +- **`cluster/`** is NOT a workload namespace — it holds cluster-wide + artifacts (CA, master key, SSH keys, known_hosts, peers, txns). +- **Per-namespace DBs**: each namespace has its own + `db/orca.db` (R-002). No namespace column in SQLite. +- **Namespace inheritance**: child namespaces inherit env and + constraints from parents (via `ns.md` frontmatter `parents:` field). + `_defaults` is always appended last in the inheritance chain. +- **`orca ns` subcommands**: `list`, `create`, `delete`, `inspect`, + `validate` — see [docs/cli.md](cli.md#orca-ns). + +### Path reference (`internal/paths/`) + +| Function | Path | Contents | +|----------|------|----------| +| `Root()` | `$ORCA_HOME` | Namespace root | +| `ClusterDir()` | `Root()/cluster` | Cluster-wide artifacts | +| `NamespaceDir(ns)` | `Root()/ns` | Per-namespace directory | +| `NSDb(ns)` | `Root()/ns/db/orca.db` | Per-namespace SQLite DB | +| `NSEnv(ns)` | `Root()/ns/.env` | Per-namespace env | +| `NSSecrets(ns)` | `Root()/ns/.env.secrets` | Encrypted secrets | +| `NSJobs(ns)` | `Root()/ns/jobs` | Jobs dir | +| `NSAlloc(ns)` | `Root()/ns/alloc` | Alloc dir | +| `NSMd(ns)` | `Root()/ns/ns.md` | Namespace frontmatter | +| `DefaultNamespace()` | `_defaults` | Implicit root (D-159) | +| `CACertPath()` | `ClusterDir()/ca.crt` | step-ca root (D-101) | +| `MasterKeyPath()` | `ClusterDir()/master.key` | AES-256-GCM root key | +| `KnownHostsPath()` | `ClusterDir()/known_hosts` | SSH known_hosts | +| `SSHKeyPath()` | `ClusterDir()/orca_ssh_key` | orca SSH private key | +| `ConfigPath()` | `ClusterDir()/config.md` | Markdown config (R-014) | +| `CacheDB()` | `Root()/orca_cache.db` | CLI-side cache (R-008) | +| `PeersDir()` | `ClusterDir()/peers` | Peers directory | +| `TxnDir()` | `ClusterDir()/txns` | Transaction log (R-016) | + +## Creating and managing namespaces + +```bash +# List all namespaces +orca ns list + +# Create a namespace (inherits from _defaults) +orca ns create prod + +# Create a namespace with an explicit parent +orca ns create staging --parent prod + +# Inspect the effective inheritance chain + merged env +orca ns inspect prod + +# Validate a namespace's inheritance chain +orca ns validate prod + +# Delete an empty namespace (refuses if jobs/ or alloc/ non-empty) +orca ns delete staging +``` + +See [docs/cli.md](cli.md#orca-ns) for the full `orca ns` reference. + +## `ORCA_DB` override + +For finer-grained control, `ORCA_DB` overrides only the database path +(not the cert/namespace paths). This is primarily a testing affordance. ```bash export ORCA_DB=/tmp/test.db -orca daemon # uses /tmp/test.db for the DB, ~/.orca/ for certs +orca init # uses /tmp/test.db for the DB, ~/.orca/ for everything else ``` -## See Also +## Deprecated: v0.8 flat layout + +> **Deprecated in v0.9**: The v0.8 flat layout (`orca.db`, `ca.crt`, +> `ca.key`, `server.crt`, `server.key` at the namespace root) is +> superseded by the v0.9 multi-namespace layout (R-002). The v0.8 +> layout is supported during the dual-write window via +> `internal/certpaths` (a thin shim) and will be removed in v0.11. + +The v0.8 flat layout stored all state at the namespace root: + +| Path | Contents | +|------|----------| +| `~/.orca/orca.db` | SQLite database | +| `~/.orca/ca.crt` | CA certificate | +| `~/.orca/ca.key` | CA private key | +| `~/.orca/server.crt` | Server certificate | +| `~/.orca/server.key` | Server private key | + +The v0.9 re-architecture moved these to `cluster/` (CA, SSH keys) and +per-namespace `db/` (SQLite) to support multi-tenancy (R-002). The +`orca doctor --legacy-paths` command (v0.11-P14c) will detect v0.8 +residue and recommend migration. + +## See also - [Install Guide](install.md) — 1-liner install with `install.sh`. -- [Docker Guide](docker.md) — running orca in a container (uses - `ORCA_HOME=/var/lib/orca` inside the image). \ No newline at end of file +- [Docker Guide](docker.md) — running orca in a container. +- [CLI Reference](cli.md) — `orca ns` subcommands. +- [Jobspec Reference](jobspec.md) — markdown frontmatter schema. \ No newline at end of file