acdl/core/hitl_matrix_design.md

ACDL Human-in-the-Loop Matrix + Separation-of-Duties Design (REQ-21)

Status: design authored in Phase 07 (milestone v1.1); v1.9 wires the gates (Phase 42). The spike (Phases 08-10) was dev-only; HITL was not exercised then. v1.9 implements the qa/prod/dr pre-execution attestation gates, the 8-concern attestation matrix (offline-testable subset), and the outbox-based separation-of-duties check.

The vision's "Lower Environments are Autonomous; Higher Environments are Attested" tenet [1] and the "deliberate human attestation — not as a rubber stamp" requirement [1] are the binding constraints.

Gate model (ARCHITECTURE.md §10.1)

Pre-execution gates. The contract is held in a "validated but not applied" state until the human attests. qa, prod, dr are attestation gates. No partial deployment to roll back on rejection (qa, prod); dr is a separate deployment against a separate cluster/region. The canary/deployment-rollback model is explicitly not in scope for v1.

Gitea-specific gate mechanics (D-042)

Gitea has no Environments API and ignores environment: blocks (v1.0 D-013; re-confirmed in RESEARCH TARGET 1). The pre-execution gate is modeled as a workflow_dispatch with approval inputs:

• qa gate: workflow_dispatch with approve_qa: true; the dispatch run's gitea.actor is the QA approver.
• prod gate: workflow_dispatch with approve_prod: true; gitea.actor is the SRE approver.
• dr gate: workflow_dispatch with approve_dr: true; same.

The approver identity of record = gitea.actor of the dispatch run (D-042). There is no other approval-identity signal in Gitea. The real OIDC path (blocked on go-gitea/gitea#36988) does not change this — OIDC authorizes the runner to AWS, it does not change how the platform records the human approver.

On GitHub, the equivalent is github.actor of the workflow_dispatch run; GitHub Environments with required reviewers are the native gate, but the workflow_dispatch approval-input fallback is used for byte-identical Gitea + GitHub workflows.

Reviewer routing (ARCHITECTURE.md §10.2)

Gitea CODEOWNERS routes the right reviewer to the right gate:

• qa → QA team
• prod → SRE team
• dr → SRE team

CODEOWNERS routes; it does not enforce identity distinctness (that is the platform-internal outbox check in core/separation_of_duties.py).

Full 8-concern attestation matrix (§10.4)

The matrix is implemented in v1.9 as core/attestation_matrix.py (REQ-109, D-084). The concerns split into two tiers:

Offline-testable concerns (run for real, no operator input):

• Contract NFRs (the platform's own contract validator).
• Schema validity (jsonschema).
• Policy pass (Checkov/Wiz/Kyverno PolicyCheckResult records).

Operator-supplied concerns (require an uploaded signed evidence artifact, validated for freshness + schema per D-084):

• Functional correctness (e2e suite report).
• Performance baseline (k6 / Gatling / Locust load test report).
• Security posture (Trivy / Snyk / contract-declared scan + Security on-call signature).
• Operational readiness (runbook published, dashboard exists, on-call rotation assigned, alerts configured).
• Incident response (Sev-1 runbook tabletop or live drill completed).
• Capacity / cost (FinOps forecast for next 30d within budget envelope).
• Resilience (DR drill, chaos engineering report, backup verified).
• dr-region deploy (most recent prod-bound dr drill as canary evidence).

The full table (lifted verbatim from §10.4):

Env	Concern	Evidence artifact	Freshness	Source	Attester
qa	Functional correctness	Last successful run of contract-declared validation.e2eSuite with pass rate ≥ 99%	Last 24h	Test runner declared in contract	QA
qa	Performance baseline	Load test report (k6 / Gatling / Locust) showing p99 latency < declared NFR and throughput > declared minimum	Last 7d	Load test runner declared in contract	QA
qa	Security posture	Vulnerability scan (Trivy, Snyk, or contract-declared equivalent) with no criticals/highs, signed by Security on-call	Last 24h	Security scanner + Security team signature	QA
qa	Contract NFRs	Platform-generated report: schema valid, NFR assertions (latency, throughput, error rate) within declared bounds	At submission	Platform contract validator	QA
prod	Operational readiness	Runbook published, dashboard exists, on-call rotation assigned, alerts configured	At submission, validated against last 30d history	Platform + SRE	SRE
prod	Incident response	Sev-1 runbook tabletop or live drill completed	Last 90d	SRE drill record	SRE
prod	Capacity / cost	FinOps forecast for next 30d within budget envelope, cost anomaly baseline stored, budget alert configured	Forecast valid for next 30d	FinOps + SRE	SRE
prod	Resilience	DR drill, chaos engineering report, backup verified	DR: 180d; chaos: 90d; backup: 30d	SRE + Platform	SRE
dr	dr-region deploy with the most recent prod-bound dr drill as canary evidence	dr drill report	Last 180d	SRE	SRE

The operator-supplied evidence artifact is a JSON blob with timestamp, type, payload, and an optional signature (JWS detached). Freshness is validated against the window above. Signature verification runs when NOVA_ATTESTATION_SIGNING_KEY_ID is set; it is skipped + logged when unset (dev/CI — D-089). The matrix fails loud if an operator-supplied concern is missing or expired for prod/dr.

Timeout behavior (§10.5)

Time	State	Action
Submission	PENDING_ATTESTATION	Notify responsible team
1 business day	PENDING_ATTESTATION_WARNING	Notify team + platform on-call (elevated path); emit PENDING_ATTESTATION_TIMEOUT_WARNING event
2 business days	PENDING_ATTESTATION_AUTO_FREEZE	Auto-freeze; require re-submission; emit PENDING_ATTESTATION_AUTO_FREEZE event; new submission linked via supersedes

Implementation: a Gitea on: schedule workflow (runs hourly) that scans the DynamoDB outbox for PENDING_ATTESTATION events with ts older than 1/2 business days and emits the warn/freeze events. Not implemented in v1.9 (roadmap item; the attestation gates themselves are wired, the timeout scanner is future work).

Rejection and rollback (§10.6)

Rejection returns the contract to a HELD state with the rejection reason captured as a PROMOTION_REJECTED event. The consumer fixes the cause and re-submits; the new submission is linked to the rejected one via supersedes (a contract-schema field — schemas/contract.schema.json). The audit chain is extended, not torn up (the "Not a mutable audit log" anti-goal). No partial deployment to roll back at any v1 gate.

Separation of duties (§10.3) — pointer to the .py

The identity-distinctness check is platform-internal, not GitHub-native, not Kyverno (in v1). Sequence:

1. On promotion dev → qa, the platform reads the QA approver's identity from the workflow_dispatch run's gitea.actor (or github.actor) and writes it to the DynamoDB outbox keyed by contractId (attribute approver_qa).
2. On promotion qa → prod, the platform reads the stored approver_qa from the outbox and the new SRE approver's gitea.actor from the prod-dispatch run.
3. If approver_qa == approver_prod, the platform blocks the prod promotion, writes a SEPARATION_OF_DUTIES_VIOLATION event to the evidence stream, and routes a halt artifact to the SRE on-call.
4. The check is implemented in core/separation_of_duties.py (T-7.8). The platform is the only writer to the outbox; the check is in the same process that has authority to block the promotion.

v1.9 implements route_halt_artifact as a real SNS publish (topic acdl-sod-halt, ARN from NOVA_SOD_HALT_TOPIC_ARN) with an outbox-event fallback when the topic ARN is unset (REQ-107). The attestation gate itself is core/hitl_gates.py (attest(contract_id, env, approver, evidence)), which records the approver to the outbox, runs the SoD check on prod, invokes the attestation matrix, and returns (ok, reason).

v1.9 wiring

v1.9 (Phase 41 + Phase 42) wires the gates end-to-end:

• Phase 41 ships the per-environment CI job structure: one job per environment (dev/qa/prod/dr), each pointing at its respective contract (or the same contract + the environment workflow_call input). The qa/prod/dr caller workflows use workflow_dispatch with the approval inputs above; dev is autonomous (no gate). Promotion = running the matching job; no environment: field editing (D-082).
• Phase 42 implements core/hitl_gates.py (the attestation gate), core/attestation_matrix.py (the 8-concern matrix), and the real route_halt_artifact (SNS + outbox fallback). scripts/run_platform.sh calls hitl_gates.attest before apply for qa/prod/dr (dev skips).

Decision trail

• D-042 — approver identity = gitea.actor of the workflow_dispatch run; no Environments API in Gitea. On GitHub, github.actor.
• D-013 (v1.0) — the workflow_dispatch approval-input fallback, re-used for the real platform's pre-execution gate model.
• D-084 (v1.9) — 8-concern attestation matrix: offline-testable concerns run for real; operator-supplied concerns accept signed evidence artifacts validated for freshness + schema.
• D-089 (v1.9) — attestation artifact signature verification is skipped when NOVA_ATTESTATION_SIGNING_KEY_ID is unset (dev/CI); required for prod/dr.