orca/.ciagent/RESEARCH_v0.6.md

Research: Orca v0.6 — Node Bootstrap & Proxmox

Findings grounded in codebase analysis (8 key files read) + verified against golang.org/x/crypto v0.54.0 (probe built clean), Proxmox VE 9.2.3 admin guide (§14.7-14.8 pveum + privileges), sudoers(5) man page (NOEXEC/NOPASSWD), and freedesktop.org os-release spec.

A. SSH library — golang.org/x/crypto/ssh

A.1 go.mod addition

require golang.org/x/crypto v0.54.0

Latest available, compatible with go 1.25. Transitive deps (verified by probe build):

• golang.org/x/crypto v0.54.0 (direct)
• golang.org/x/sys v0.47.0 (indirect — bumps from v0.42.0)
• golang.org/x/term v0.45.0 (indirect — pulled by ssh for PTY)

3 module entries, 0 new heavy deps. Matches D-030 minimal-deps rationale. go.sum gains ~6 lines.

A.2 Minimal API surface

import (
    "crypto/ed25519"
    "crypto/rand"
    "crypto/x509"
    "encoding/pem"
    "net"
    "time"
    "golang.org/x/crypto/ssh"
    "golang.org/x/crypto/ssh/knownhosts"
)

Key functions:

• ssh.Dial(network, addr, config) (*ssh.Client, error) — high-level dialer
• (*ssh.Client).NewSession() (*ssh.Session, error)
• (*ssh.Session).CombinedOutput(cmd) ([]byte, error) — run + capture
• ssh.ClientConfig{User, Auth, HostKeyCallback, Timeout}
• ssh.Password(secret) ssh.AuthMethod — password auth
• ssh.PublicKeys(signer) ssh.AuthMethod — pubkey auth
• ssh.ParsePrivateKey(pem) (ssh.Signer, error) — parse PKCS8 PEM (works with orca's existing key format)
• ssh.NewPublicKey(pub) (ssh.PublicKey, error) + ssh.MarshalAuthorizedKey(pub) []byte — authorized_keys line
• ssh.FixedHostKey(key) ssh.HostKeyCallback — strict pin (subsequent connects)
• knownhosts.New(path) (ssh.HostKeyCallback, error) — TOFU via known_hosts file (cleaner than custom callback; avoids deprecated InsecureIgnoreHostKey)

A.3 Ed25519 keygen (D-037)

Verified end-to-end: ed25519.GenerateKey(rand.Reader) → x509.MarshalPKCS8PrivateKey(priv) → PEM encode → ssh.ParsePrivateKey round-trips cleanly. ssh.MarshalAuthorizedKey produces valid ssh-ed25519 AAAA... line. PKCS8 PEM (orca's existing format) parses with ssh.ParsePrivateKey — no OpenSSH-format marshaller needed. Reuse security.WriteKey/writeAtomic for persistence.

A.4 File upload — cat > file via session, NOT SFTP

SFTP lives in separate module github.com/pkg/sftp — would add a 4th direct dep beyond D-030. The only files orca uploads are:

• ~orca/.ssh/authorized_keys (1-line append)
• /etc/sudoers.d/orca (few lines)

Both are text. Use session.CombinedOutput with heredoc / tee -a. Keeps everything within x/crypto/ssh.

A.5 TOFU host-key handling (D-035)

Use golang.org/x/crypto/ssh/knownhosts.New(path) as the HostKeyCallback. On first connect, the callback writes the host key to ~/.orca/known_hosts (OpenSSH format). On subsequent connects, it verifies and returns an error on mismatch. Avoids ssh.InsecureIgnoreHostKey deprecation — knownhosts.New handles both capture and verify in one callback. On host-key change (reinstall), fail closed with a clear error; operator runs orca node key-reset <node> (future) or manually edits known_hosts.

B. /etc/os-release parsing (D-032)

B.1 Confirmed ID= values

Distro	ID=	ID_LIKE=	Verified
Ubuntu	ubuntu	debian	✅ (this host: Ubuntu 24.04)
Debian	debian	—	✅ (freedesktop spec)
Alpine	alpine	—	✅ (Alpine policy)
Proxmox VE	pve	debian	✅ (PVE ships own os-release)

VARIANT_ID absent on all four target distros — not worth capturing for v0.6.

B.2 Parsing approach

No Go stdlib helper. Trivial: bufio.Scanner + strings.SplitN(line, "=", 2) + strip surrounding quotes. ~15 lines. Returns map[string]string; read ID field. Fallback "linux" if file missing or ID absent (D-032). Read /etc/os-release first; fall back to /usr/lib/os-release for minimal containers. Unknown ID values stored verbatim (not masked) — doctor os can warn.

C. Proxmox VE role & user management

C.1 Realm: orca@pam (NOT orca@pve)

Confirmed by both researchers + PVE User Management docs: since orca node join SSHes in and creates a Linux system user via useradd, the PVE user must be orca@pam (PAM realm maps to host system users). orca@pve would require a separate PVE-internal password and interactive -password prompt over non-PTY SSH (hangs). @pam sidesteps both issues. D-033 refined: orca@pam.

C.2 OrcaOperator PVE role — privilege set

Per D-033 (operator-confirmed): VM.Audit, Datastore.AllocateSpace, SDN.Use. This is a minimal API-level role — the actual management capability comes from the sudoers allowlist (sudo runs as root, bypassing PVE RBAC). The PVE role governs non-sudo API access (future REST client).

Refinement from research: VM.Audit covers containers (CTs) as well as VMs (both live under /vms/{vmid} path; no separate CT.* family). PVE 8→9: privilege set valid on both (no breaking changes to pveum or the core privilege names).

Researcher 2 proposed an expanded 21-privilege set for fuller API-level management. Decision: keep D-033's 3-priv minimal set for v0.6 — the operator explicitly confirmed it, and the sudoers allowlist is the primary management path. The expanded set is noted as a v0.7+ enhancement option if orca adds a direct PVE REST client.

C.3 pveum command sequence (idempotent)

# 1. Role — probe-then-add (pveum role add fails if exists)
pveum role list | grep -q '^OrcaOperator' || \
  pveum role add OrcaOperator --privs "VM.Audit Datastore.AllocateSpace SDN.Use"

# 2. User — probe-then-add (maps to existing Linux system user)
pveum user list | grep -q 'orca@pam' || \
  pveum user add orca@pam -comment "Orca automation user"

# 3. ACL — modify is idempotent (creates or updates)
pveum acl modify / -user orca@pam -role OrcaOperator

Flag syntax: both -privs and --privs work (Perl Getopt::Long). Use --privs (canonical). Privs are space-separated inside quotes (NOT comma-separated).

C.4 sudoers file /etc/sudoers.d/orca (D-033 refined)

Research refinement: exclude pvesh from sudoers — pvesh can reach the /nodes/{node}/execute API endpoint which spawns shell commands server-side, bypassing sudo's NOEXEC tag. Keep pct/qm with NOEXEC; apt-get/dpkg without NOEXEC (they need to spawn child processes for maintainer scripts).

# /etc/sudoers.d/orca — mode 0440, owner root:root
# Orca automation: VM/CT management + package management, no shell escape
orca ALL=(root) NOPASSWD: NOEXEC: /usr/bin/pct, /usr/bin/qm
orca ALL=(root) NOPASSWD: /usr/bin/apt-get, /usr/bin/dpkg

NOEXEC works via Linux seccomp (sudoers man page). pct/qm are Perl scripts run via dynamically-linked /usr/bin/perl → NOEXEC effective. apt-get/dpkg need exec for postinst scripts → no NOEXEC. File mode 0440 or sudo refuses to load. Validate with visudo -cf /etc/sudoers.d/orca after writing; abort bootstrap on validation failure.

Resolve binary paths at runtime via command -v pct etc. before writing the sudoers file (cheap insurance against non-standard installs).

C.5 PVE 8 vs 9

No breaking changes to pveum, privilege names, or sudo defaults between 8 and 9. VM.Monitor removed in 9.0 (OrcaOperator doesn't use it). Privileged container creation needs Sys.Modify in 9.0 (OrcaOperator doesn't have it → intended). Both versions: orca@pam flow identical. Binary paths identical (/usr/bin/{pct,qm,pvesh}).

D. Codebase integration points (confirmed by reading files)

D.1 Files to modify/create per requirement

File	Change	REQ
go.mod / go.sum	Add golang.org/x/crypto v0.54.0; bump sys, add term	REQ-050
internal/model/node.go	Add Kind, OS string fields + NodeKind constants	REQ-049
internal/store/migrations/0006_node_kind_os.sql	NEW: ALTER TABLE nodes ADD COLUMN kind TEXT; ADD COLUMN os TEXT; (nullable, backward-compatible)	REQ-049
internal/store/node_repo.go	Extend INSERT/SELECT/scanNode for kind, os; add GetByName, UpdateLastSeenAndOS helpers	REQ-049
internal/cli/init.go	Full bootstrap: MkdirAll → store.Open (runs migrations) → CAInit → server cert gen (if absent) → detectOS → localhost node upsert	REQ-047,048
internal/cli/node.go	Add --type, --host, --user, --password, --proxmox-user, --proxmox-role flags; bootstrapProxmox branch	REQ-050,051
internal/security/sshkey.go	NEW: GenerateOrLoadSSHKey(dir) — Ed25519 keygen, PKCS8 PEM, 0600/0644 modes	REQ-050
internal/proxmox/bootstrap.go	NEW package: BootstrapProxmox(ctx, opts) — SSH dial, pubkey deploy, useradd, pveum role/user/acl, sudoers write, visudo validate	REQ-050,051
internal/doctor/doctor.go	Add OS() and Proxmox() checks; extend All()	REQ-052
internal/cli/doctor.go	Add doctor os + doctor proxmox subcommands	REQ-052
internal/certpaths/certpaths.go	Add SSHKeyPath, SSHPubPath, KnownHostsPath	REQ-050

D.2 Reuse opportunities (confirmed)

• security.CAInit (ca.go:63) — already idempotent (fast-path loads existing). orca init calls it directly.
• security.GenerateCSR (csr.go) — signature fits: GenerateCSR("localhost", []string{"localhost","127.0.0.1"}).
• security.WriteCert/WriteKey (ca.go:292) — enforce 0644/0600 via writeAtomic; reuse for SSH key.
• store.Open (migrate.go) — runs migrations on open; calling it in orca init auto-applies 0006.
• Migration runner — FS-embedded, sorts lexicographically, idempotent per-file. Adding 0006_*.sql is the entire change.
• doctor.Network() (doctor.go:222) — exact pattern to clone for doctor.Proxmox() (list nodes, filter by kind, 3s timeout per peer, PASS/WARN/FAIL).

D.3 No changes needed

• internal/security/ca.go, csr.go — idempotent already, signatures fit.
• internal/store/migrate.go — runner is generic.
• internal/transport/* — mTLS transport not involved in SSH bootstrap.
• internal/engine/* — NodeRegistry.Join works; new fields are metadata.

E. Pitfalls & gotchas

1. pveum flag is --privs (space-separated), not --privs "a,b,c". Confirmed by both researchers + official docs.
2. orca@pam not orca@pve — PVE-internal realm requires interactive password prompt over non-PTY SSH (hangs). PAM realm maps to the Linux system user orca creates.
3. Exclude pvesh from sudoers — pvesh can trigger API execute endpoint spawning shell commands server-side, bypassing NOEXEC. Use PVE API via OrcaOperator role for API access instead.
4. NOEXEC only on dynamically-linked binaries — pct/qm are Perl scripts via dynamically-linked /usr/bin/perl → effective. apt-get/dpkg need exec → no NOEXEC.
5. sudoers file mode 0440 — or sudo silently refuses to load it. chmod 0440 + visudo -cf validate after write.
6. Migration 0006 NULL handling — scanNode must use sql.NullString for kind/os and map NULL → "" (Go struct fields are string, not *string).
7. localhost node idempotency — NodeRepo.Insert fails on UNIQUE constraint if orca init re-runs. Need GetByName("localhost") check first; if found, UpdateLastSeenAndOS instead of Insert. Don't change id or joined_at (D-036).
8. orca init must not regenerate server cert (D-036) — check certpaths.ServerCertPath() existence before GenerateCSR. CAInit has a fast-path; server cert gen needs an explicit existence check.
9. Password handling (D-031) — --password flag visible in ps//proc briefly. Prefer $ORCA_PROXMOX_PASSWORD env var. Never log the password (slog redaction). Zero the byte slice after use.
10. knownhosts.New for TOFU — avoids deprecated ssh.InsecureIgnoreHostKey. Handles both capture and verify in one callback.
11. PKCS8 PEM parses with ssh.ParsePrivateKey — no need for OpenSSH-format marshaller. Consistent with ca.key/server.key format.
12. /etc/os-release is a symlink on most distros → os.ReadFile follows it. Fall back to /usr/lib/os-release for minimal containers.

F. Persona recommendations (v0.6 roster)

Persona	Active	Reason
lead-developer	✅	Coordination across P01/P02/P03; SSH/bootstrap touches security + cli + store + doctor
backend-engineer	✅	Owns internal/cli/init.go full-bootstrap orchestration + internal/proxmox/bootstrap.go SSH logic
cli-engineer	✅	Owns --type/--host/--password flag wiring, doctor os/doctor proxmox subcommands, init output UX
data-engineer	✅ REACTIVATE	Owns migration 0006 + NodeRepo schema extension (kind/os columns, new helpers)
security-engineer	✅ REACTIVATE	Owns internal/security/sshkey.go, TOFU host-key, sudoers design, password redaction, audit logging
devops-engineer	❌ DEACTIVATE	No install.sh/Dockerfile/.coreci.yml surface in v0.6
network-engineer	❌	No transport/mTLS surface (SSH is point-to-point bootstrap, not mesh)
frontend-engineer	❌	No web UI

Territory overlaps to adjudicate (lead-developer):

• internal/proxmox/bootstrap.go (security-engineer SSH/sudoers logic) vs internal/cli/node.go (cli-engineer flag wiring) — boundary: security package exposes BootstrapProxmox(ctx, opts) error, CLI just calls it.
• internal/doctor/doctor.go Proxmox() reuses SSH client from internal/proxmox (security) but check scaffolding clones doctor.Network() pattern (backend adjudicates since network-engineer deactivated).