Jon Chery
e1be05287b
---ci--- project: acdl phase: 39 milestone: v1.9 status: execute ---/ci--- Phase 39 — design-doc-refresh-and-p1-1-parameterization: Design docs (REQ-100, REQ-101): - hitl_matrix_design.md: 'dev-only spike'/'v1.2 wires the gates' framing replaced with v1.9 wired-gates reality; 8-concern matrix marked implemented (offline-testable subset + signed evidence artifacts, D-084); v1.9 wiring section cross-references hitl_gates.py + attestation_matrix.py; approver_dr noted. - audit_ledger_design.md: outbox marked shipped+production since v1.8; S3 Object Lock + JWS + async worker + DLQ + daily checkpoints clearly labeled 'Deferred to a future milestone (D-083)'; RPO/RTO table updated; approver fields note v1.9 hitl_gates.attest. P1-1 adapter parameterization (REQ-102, D-085): - ecs-service interface.json: desired_count (default 1), launch_type (FARGATE), family (app) inputs added. - alb interface.json: load_balancer_type (application), target_type (ip). - adapter.py: hardcoded defaults replaced with inputs.get(<name>, <default>); hardcoded 'acdl-microservice-rt'/'acdl-microservice-igw' Name tags derive from the VPC name input. - contract_resolver.py: child_input_map routes wires to the sub-resource that declares the input (desired_count → aws:ecs:service, family → aws:ecs:task_definition, target_type → targetgroup, etc.). - microservice composition.json: wires added for the new inputs. Tests: +21 (test_p1_1_adapter_parameterization.py, test_design_docs_current.py). 371 passed; run_ci.sh green; run_platform.sh --check-only green; v1.1 S3 regression preserved.
175 lines
9.4 KiB
Markdown
175 lines
9.4 KiB
Markdown
# ACDL Human-in-the-Loop Matrix + Separation-of-Duties Design (REQ-21)
|
|
|
|
> **Status:** design authored in Phase 07 (milestone v1.1); **v1.9 wires
|
|
> the gates** (Phase 42). The spike (Phases 08-10) was dev-only; HITL was
|
|
> not exercised then. v1.9 implements the qa/prod/dr pre-execution
|
|
> attestation gates, the 8-concern attestation matrix (offline-testable
|
|
> subset), and the outbox-based separation-of-duties check.
|
|
|
|
The vision's "Lower Environments are Autonomous; Higher Environments are
|
|
Attested" tenet [1] and the "deliberate human attestation — not as a
|
|
rubber stamp" requirement [1] are the binding constraints.
|
|
|
|
## Gate model (ARCHITECTURE.md §10.1)
|
|
|
|
**Pre-execution gates.** The contract is held in a "validated but not
|
|
applied" state until the human attests. qa, prod, dr are attestation
|
|
gates. No partial deployment to roll back on rejection (qa, prod); dr is
|
|
a separate deployment against a separate cluster/region. The
|
|
canary/deployment-rollback model is explicitly not in scope for v1.
|
|
|
|
## Gitea-specific gate mechanics (D-042)
|
|
|
|
Gitea has **no Environments API** and ignores `environment:` blocks
|
|
(v1.0 D-013; re-confirmed in RESEARCH TARGET 1). The pre-execution gate
|
|
is modeled as a `workflow_dispatch` with approval inputs:
|
|
|
|
- **qa gate:** `workflow_dispatch` with `approve_qa: true`; the dispatch
|
|
run's `gitea.actor` is the QA approver.
|
|
- **prod gate:** `workflow_dispatch` with `approve_prod: true`;
|
|
`gitea.actor` is the SRE approver.
|
|
- **dr gate:** `workflow_dispatch` with `approve_dr: true`; same.
|
|
|
|
The approver identity of record = `gitea.actor` of the dispatch run
|
|
(D-042). There is no other approval-identity signal in Gitea. The real
|
|
OIDC path (blocked on go-gitea/gitea#36988) does not change this —
|
|
OIDC authorizes the *runner* to AWS, it does not change how the platform
|
|
records the *human* approver.
|
|
|
|
On GitHub, the equivalent is `github.actor` of the `workflow_dispatch`
|
|
run; GitHub Environments with required reviewers are the native gate,
|
|
but the `workflow_dispatch` approval-input fallback is used for
|
|
byte-identical Gitea + GitHub workflows.
|
|
|
|
## Reviewer routing (ARCHITECTURE.md §10.2)
|
|
|
|
Gitea CODEOWNERS routes the right reviewer to the right gate:
|
|
|
|
- qa → QA team
|
|
- prod → SRE team
|
|
- dr → SRE team
|
|
|
|
CODEOWNERS **routes**; it does **not** enforce identity distinctness (that
|
|
is the platform-internal outbox check in
|
|
`core/separation_of_duties.py`).
|
|
|
|
## Full 8-concern attestation matrix (§10.4)
|
|
|
|
The matrix is implemented in v1.9 as `core/attestation_matrix.py`
|
|
(REQ-109, D-084). The concerns split into two tiers:
|
|
|
|
**Offline-testable concerns** (run for real, no operator input):
|
|
- Contract NFRs (the platform's own contract validator).
|
|
- Schema validity (jsonschema).
|
|
- Policy pass (Checkov/Wiz/Kyverno `PolicyCheckResult` records).
|
|
|
|
**Operator-supplied concerns** (require an uploaded signed evidence
|
|
artifact, validated for freshness + schema per D-084):
|
|
- Functional correctness (e2e suite report).
|
|
- Performance baseline (k6 / Gatling / Locust load test report).
|
|
- Security posture (Trivy / Snyk / contract-declared scan + Security
|
|
on-call signature).
|
|
- Operational readiness (runbook published, dashboard exists, on-call
|
|
rotation assigned, alerts configured).
|
|
- Incident response (Sev-1 runbook tabletop or live drill completed).
|
|
- Capacity / cost (FinOps forecast for next 30d within budget envelope).
|
|
- Resilience (DR drill, chaos engineering report, backup verified).
|
|
- dr-region deploy (most recent prod-bound dr drill as canary evidence).
|
|
|
|
The full table (lifted verbatim from §10.4):
|
|
|
|
| Env | Concern | Evidence artifact | Freshness | Source | Attester |
|
|
|---|---|---|---|---|---|
|
|
| qa | Functional correctness | Last successful run of contract-declared validation.e2eSuite with pass rate ≥ 99% | Last 24h | Test runner declared in contract | QA |
|
|
| qa | Performance baseline | Load test report (k6 / Gatling / Locust) showing p99 latency < declared NFR and throughput > declared minimum | Last 7d | Load test runner declared in contract | QA |
|
|
| qa | Security posture | Vulnerability scan (Trivy, Snyk, or contract-declared equivalent) with no criticals/highs, signed by Security on-call | Last 24h | Security scanner + Security team signature | QA |
|
|
| qa | Contract NFRs | Platform-generated report: schema valid, NFR assertions (latency, throughput, error rate) within declared bounds | At submission | Platform contract validator | QA |
|
|
| prod | Operational readiness | Runbook published, dashboard exists, on-call rotation assigned, alerts configured | At submission, validated against last 30d history | Platform + SRE | SRE |
|
|
| prod | Incident response | Sev-1 runbook tabletop or live drill completed | Last 90d | SRE drill record | SRE |
|
|
| prod | Capacity / cost | FinOps forecast for next 30d within budget envelope, cost anomaly baseline stored, budget alert configured | Forecast valid for next 30d | FinOps + SRE | SRE |
|
|
| prod | Resilience | DR drill, chaos engineering report, backup verified | DR: 180d; chaos: 90d; backup: 30d | SRE + Platform | SRE |
|
|
| dr | dr-region deploy with the most recent prod-bound dr drill as canary evidence | dr drill report | Last 180d | SRE | SRE |
|
|
|
|
The operator-supplied evidence artifact is a JSON blob with `timestamp`,
|
|
`type`, `payload`, and an optional `signature` (JWS detached). Freshness
|
|
is validated against the window above. Signature verification runs when
|
|
`ACDL_ATTESTATION_SIGNING_KEY_ID` is set; it is skipped + logged when
|
|
unset (dev/CI — D-089). The matrix fails loud if an operator-supplied
|
|
concern is missing or expired for prod/dr.
|
|
|
|
## Timeout behavior (§10.5)
|
|
|
|
| Time | State | Action |
|
|
|---|---|---|
|
|
| Submission | PENDING_ATTESTATION | Notify responsible team |
|
|
| 1 business day | PENDING_ATTESTATION_WARNING | Notify team + platform on-call (elevated path); emit `PENDING_ATTESTATION_TIMEOUT_WARNING` event |
|
|
| 2 business days | PENDING_ATTESTATION_AUTO_FREEZE | Auto-freeze; require re-submission; emit `PENDING_ATTESTATION_AUTO_FREEZE` event; new submission linked via `supersedes` |
|
|
|
|
**Implementation:** a Gitea `on: schedule` workflow (runs hourly) that
|
|
scans the DynamoDB outbox for `PENDING_ATTESTATION` events with `ts`
|
|
older than 1/2 business days and emits the warn/freeze events. Not
|
|
implemented in v1.9 (roadmap item; the attestation gates themselves are
|
|
wired, the timeout scanner is future work).
|
|
|
|
## Rejection and rollback (§10.6)
|
|
|
|
Rejection returns the contract to a `HELD` state with the rejection
|
|
reason captured as a `PROMOTION_REJECTED` event. The consumer fixes the
|
|
cause and re-submits; the new submission is linked to the rejected one
|
|
via `supersedes` (a contract-schema field — `schemas/contract.schema.json`).
|
|
The audit chain is **extended, not torn up** (the "Not a mutable audit
|
|
log" anti-goal). No partial deployment to roll back at any v1 gate.
|
|
|
|
## Separation of duties (§10.3) — pointer to the .py
|
|
|
|
The identity-distinctness check is platform-internal, not GitHub-native,
|
|
not Kyverno (in v1). Sequence:
|
|
|
|
1. On promotion dev → qa, the platform reads the QA approver's identity
|
|
from the `workflow_dispatch` run's `gitea.actor` (or `github.actor`)
|
|
and writes it to the DynamoDB outbox keyed by `contractId` (attribute
|
|
`approver_qa`).
|
|
2. On promotion qa → prod, the platform reads the stored `approver_qa`
|
|
from the outbox and the new SRE approver's `gitea.actor` from the
|
|
prod-dispatch run.
|
|
3. If `approver_qa == approver_prod`, the platform blocks the prod
|
|
promotion, writes a `SEPARATION_OF_DUTIES_VIOLATION` event to the
|
|
evidence stream, and routes a halt artifact to the SRE on-call.
|
|
4. The check is implemented in `core/separation_of_duties.py`
|
|
(T-7.8). The platform is the only writer to the outbox; the check is
|
|
in the same process that has authority to block the promotion.
|
|
|
|
v1.9 implements `route_halt_artifact` as a real SNS publish (topic
|
|
`acdl-sod-halt`, ARN from `ACDL_SOD_HALT_TOPIC_ARN`) with an outbox-event
|
|
fallback when the topic ARN is unset (REQ-107). The attestation gate
|
|
itself is `core/hitl_gates.py` (`attest(contract_id, env, approver,
|
|
evidence)`), which records the approver to the outbox, runs the SoD
|
|
check on prod, invokes the attestation matrix, and returns `(ok, reason)`.
|
|
|
|
## v1.9 wiring
|
|
|
|
v1.9 (Phase 41 + Phase 42) wires the gates end-to-end:
|
|
|
|
- **Phase 41** ships the per-environment CI job structure: one job per
|
|
environment (dev/qa/prod/dr), each pointing at its respective contract
|
|
(or the same contract + the `environment` workflow_call input). The
|
|
qa/prod/dr caller workflows use `workflow_dispatch` with the approval
|
|
inputs above; dev is autonomous (no gate). Promotion = running the
|
|
matching job; no `environment:` field editing (D-082).
|
|
- **Phase 42** implements `core/hitl_gates.py` (the attestation gate),
|
|
`core/attestation_matrix.py` (the 8-concern matrix), and the real
|
|
`route_halt_artifact` (SNS + outbox fallback). `scripts/run_platform.sh`
|
|
calls `hitl_gates.attest` before apply for qa/prod/dr (dev skips).
|
|
|
|
## Decision trail
|
|
|
|
- **D-042** — approver identity = `gitea.actor` of the `workflow_dispatch`
|
|
run; no Environments API in Gitea. On GitHub, `github.actor`.
|
|
- **D-013** (v1.0) — the `workflow_dispatch` approval-input fallback,
|
|
re-used for the real platform's pre-execution gate model.
|
|
- **D-084** (v1.9) — 8-concern attestation matrix: offline-testable
|
|
concerns run for real; operator-supplied concerns accept signed
|
|
evidence artifacts validated for freshness + schema.
|
|
- **D-089** (v1.9) — attestation artifact signature verification is
|
|
skipped when `ACDL_ATTESTATION_SIGNING_KEY_ID` is unset (dev/CI);
|
|
required for prod/dr. |