orca/docs/uat.md

Orca User Acceptance Testing (UAT) Plan

Version: v0.13 (production hardening round 2) Gate: v1.0.0 production-ready tag is deferred until this UAT passes Signoff: run scripts/uat-signoff.sh on the lead node and paste the output back

Prerequisites

Hardware

Role	OS	Requirements
lead	Ubuntu 22.04 LTS	Operator laptop or VM; SSH key; orca binary (built from v0.13 tag)
pve01	Proxmox VE 8/9	Bare-metal or nested; SSH root access; orca SSH key pre-staged
worker01	Ubuntu 22.04 LTS	VM or bare-metal; SSH root access; orca SSH key pre-staged

Alternative topology (3x Ubuntu, no Proxmox)

If a Proxmox host is unavailable, run the UAT with 3x Ubuntu hosts. Use --type linux for all remote nodes. Proxmox-specific claims (doctor proxmox, PVE role, sudoers) are skipped in this path. The signoff script reports exercised vs. skipped claims.

Pre-staging

1. Build orca from the v0.13 tag:

   git clone https://git.cloudinit.dev/coreci/orca.git
   cd orca && git checkout v0.12.13
   make build
   # binary is at bin/orca
   

2. Generate the orca SSH keypair on the lead:

   ssh-keygen -t ed25519 -f ~/.ssh/orca_ed25519 -N ""
   

3. Pre-stage the orca public key on pve01 and worker01:

   ssh-copy-id -i ~/.ssh/orca_ed25519.pub root@pve01
   ssh-copy-id -i ~/.ssh/orca_ed25519.pub root@worker01
   

4. Pin host-key fingerprints (optional but recommended):

   ssh-keyscan pve01 | ssh-keygen -lf -
   ssh-keyscan worker01 | ssh-keygen -lf -
   

Step-by-step UAT

Step 1: Initialize the cluster

export ORCA_HOME=~/orca-uat
orca init

Expected: cluster directory created, CA cert generated, localhost node registered.

Step 2: Onboard the Proxmox host

orca node join --type proxmox \
  --host pve01 \
  --ssh-user root \
  --ssh-key ~/.ssh/orca_ed25519 \
  --host-key-fingerprint SHA256:<fingerprint>

Expected: SSH bootstrap succeeds, orca user created, PVE role assigned, node registered as ready with kind=proxmox.

Step 3: Onboard the Ubuntu worker

orca node join --type linux \
  --host worker01 \
  --ssh-user root \
  --ssh-key ~/.ssh/orca_ed25519 \
  --host-key-fingerprint SHA256:<fingerprint>

Expected: SSH bootstrap succeeds, orca user created, drift-events dir created, node registered as ready with kind=linux.

Step 4: Verify nodes

orca node list
orca node list --json

Expected: 3 nodes listed (localhost + pve01 + worker01), all ready.

Step 5: Set capacity on remote nodes

orca node capacity set --node pve01 --cpu 4 --memory 8192 --disk 100000
orca node capacity set --node worker01 --cpu 2 --memory 4096 --disk 50000
orca node capacity list

Expected: capacity shown for both remote nodes.

Step 6: Create a namespace

orca ns create prod
orca ns list

Expected: prod namespace listed.

Step 7: Deploy the full stack

Deploy each service from examples/full-stack/:

orca job run examples/full-stack/web-app.md --target pve01
orca job run examples/full-stack/api.md --target pve01
orca job run examples/full-stack/worker.md --target worker01
orca job run examples/full-stack/postgres.md --target pve01
orca job run examples/full-stack/log-shipper.md --target worker01

Expected: each job is scheduled on the target, systemd unit deployed via SSH-push, job status running or complete.

Step 8: Verify deployment

orca job list
orca job list --json

Expected: all 5 jobs listed, with correct target nodes.

On each remote node:

ssh root@pve01 systemctl status 'orca-alloc-*'
ssh root@worker01 systemctl status 'orca-alloc-*'

Step 9: Verify Traefik routes

ssh root@pve01 ls /etc/traefik/dynamic/
ssh root@worker01 ls /etc/traefik/dynamic/

Expected: traefik-dynamic-*.yaml files present on nodes where jobs were deployed.

Step 10: Migrate between hosts

Migrate web-app from pve01 to worker01:

orca job migrate web-app --to worker01

Expected: job drained on pve01, rescheduled on worker01, new systemd unit deployed.

Verify:

orca job list
ssh root@worker01 systemctl status 'orca-alloc-*web-app*'
ssh root@pve01 systemctl status 'orca-alloc-*web-app*' # should be stopped

Step 11: Aggregate logs

orca logs --all-nodes --job web-app --since 5m

Expected: log entries from multiple nodes.

Step 12: ACL enforcement

orca acl grant operator-1 --namespace prod --permissions read,write
orca acl check operator-1 --namespace prod --permission read
orca acl check operator-1 --namespace prod --permission admin

Expected: read+write allowed, admin denied (not granted).

Step 13: Seal/unseal

orca cluster seal --rp-id orca.local
orca cluster unseal
orca secrets set prod TEST_KEY --value "test-value"
orca secrets get prod TEST_KEY

Expected: seal succeeds, unseal succeeds, secrets readable post-unseal.

Step 14: Audit chain

orca doctor audit

Expected: chain head reported, no tamper detected.

Step 15: Doctor modes

orca doctor modes

Expected: all file modes correct, exit 0.

Step 16: OIDC health

orca doctor oidc

Expected: Dex unit active, issuer reachable (or WARN if Dex not installed).

Step 17: Backup and restore

orca backup --out /tmp/uat-backup.tar.gz
orca restore --in /tmp/uat-backup.tar.gz --dry-run

Expected: backup succeeds, restore dry-run succeeds.

Step 18: Drift detection

orca drift show

Expected: no error (empty drift is fine).

Step 19: Transaction idempotency

orca txn apply <some-txn-dir>
orca txn apply <some-txn-dir>  # re-run

Expected: second apply is idempotent (exit 5 or "already applied").

Step 20: Metrics

orca metrics --addr :9100 &
sleep 3
curl -s http://localhost:9100/metrics | grep orca_

Expected: expanded metric set present (orca_jobs_running, orca_audit_chain_head, etc.).

Step 21: Compat check

orca cluster compat-check

Expected: exit 0, all nodes compatible.

Step 22: Run the signoff script

scripts/uat-signoff.sh

Expected: UAT SIGNOFF: N/35 assertions passed, exit 0 iff N==35.

Claim Matrix

#	Claim	UAT Step	Signoff Assertion
1	Cluster initializes from scratch	Step 1	assert_orca_version
2	Proxmox host onboards via SSH	Step 2	assert_proxmox_onboarded
3	Ubuntu worker onboards via --type linux	Step 3	assert_linux_worker_onboarded
4	Node list shows all nodes	Step 4	assert_cluster_initialized
5	Capacity is set on remote nodes	Step 5	assert_capacity_set
6	Namespace created	Step 6	assert_namespace_created
7	Full stack deploys to remote nodes	Step 7	assert_full_stack_running
8	Scheduler deploys to remote (not local)	Step 7	assert_job_deploys_to_remote
9	Traefik routes present	Step 9	assert_traefik_routes
10	Job migrates between hosts	Step 10	assert_migrate_worked
11	Logs aggregate from multiple nodes	Step 11	assert_logs_aggregate
12	ACL grant/check works	Step 12	assert_acl_enforced
13	ACL deny-by-default	Step 12	assert_acl_deny_default
14	acl.json mode 0600	Step 12	assert_acl_file_mode
15	Seal/unseal round-trip	Step 13	assert_seal_unseal_roundtrip
16	Audit chain intact	Step 14	assert_audit_chain_intact
17	Doctor modes passes	Step 15	assert_doctor_modes
18	OIDC health check	Step 16	assert_oidc_health
19	Backup works	Step 17	assert_backup_restore_dryrun
20	Drift visible	Step 18	assert_drift_visible
21	Txn idempotent	Step 19	assert_txn_idempotent
22	Metrics expanded	Step 20	assert_metrics_expanded
23	Compat check passes	Step 21	assert_compat_check_passes
24	No --password in docs/examples	—	assert_no_password_in_docs
25	Go toolchain current	—	assert_go_toolchain_current
26	cli.md matches orca --help	—	assert_cli_md_complete
27	pprof not on all interfaces	—	assert_no_pprof_on_all_interfaces
28	WebAuthn registration requires auth	—	assert_webauthn_reg_requires_auth
29	Audit chain survives concurrency	—	assert_audit_chain_concurrent
30	Concurrent secrets no data loss	—	assert_concurrent_secrets_no_loss
31	Cache invalidated after write	—	assert_cache_invalidated_after_write
32	SQLite no lock under concurrency	—	assert_sqlite_no_lock
33	No injection in logs --job	—	assert_no_injection_in_logs
34	--type linux exists as subcommand	Step 3	assert_type_linux_available
35	orca status deprecated	—	assert_status_deprecated

Signoff procedure

1. Run all steps above on the 3-host cluster
2. Run scripts/uat-signoff.sh on the lead
3. Paste the output back to the CI agent
4. The CI agent verifies 35/35 PASS and cuts v1.0.0