--- phase: 39-43 name: v1.9-design-doc-interpolation-per-env-ci-stubs-p1-1 milestone: v1.9 requirements: [REQ-100, REQ-101, REQ-102, REQ-103, REQ-104, REQ-105, REQ-106, REQ-107, REQ-108, REQ-109, REQ-110, REQ-111] type: feat/docs/fix --- # ACDL v1.9 — Phase Plans > Milestone v1.9. Generated at PLAN stage. Autonomy: full. > Requirements: REQ-100..REQ-111 (see REQUIREMENTS.md). > Decisions: D-080..D-089 (see PROJECT.md + RESEARCH.md RA section). > Versioning: feature milestone — progressive patch versions per phase > (v1.8.1..v1.8.5), tag `v1.9.0` at milestone COMPLETE. ## Wave ordering - **Wave 1 (parallel, 2 tasks):** Phase 39 — design-doc refresh (security-engineer) + P1-1 adapter parameterization (platform-engineer). Disjoint file sets; no merge conflict. - **Wave 2 (sequential):** Phase 40 — contract interpolation. Depends on Phase 39's design-doc context (lightweight). - **Wave 3 (sequential):** Phase 41 — per-env CI jobs. Depends on Phase 40's interpolation + env schema. - **Wave 4 (sequential):** Phase 42 — stub implementation. Depends on Phase 41's HITL job structure. - **Wave 5 (sequential):** Phase 43 — verify + review + audit + complete. --- ## Phase 39 — design-doc-refresh-and-p1-1-parameterization **Requirements:** REQ-100, REQ-101, REQ-102 **Personas:** security-engineer (lead: design docs), platform-engineer (lead: P1-1), backend-engineer (review) **Branch:** `phase/39-design-doc-refresh-and-p1-1` ### Task 39.1 — Refresh hitl_matrix_design.md (REQ-100, security-engineer) - Rewrite the status block: "v1.2 wires the gates" → "v1.9 wires the gates (Phase 42)". - Update "Spike scope note" → "v1.9 scope note": qa/prod/dr now exercised (Phase 41 wires the job structure; Phase 42 wires the attestation gates); dev remains autonomous. - Update §10.4 matrix: mark the offline-testable concerns (contract NFRs, schema validity, policy pass) as **implemented in v1.9** (`core/attestation_matrix.py`); mark operator-supplied concerns as **accept signed evidence artifacts** (D-084). - Add a "v1.9 wiring" section: cross-reference Phase 41's per-env jobs + Phase 42's `hitl_gates.py` + `attestation_matrix.py` + the outbox-based SoD check. - Preserve D-042 (approver identity = `gitea.actor` / `github.actor`) — still accurate. - Verify: `grep -i "dev-only spike" core/hitl_matrix_design.md` returns 0 hits; `grep -i "v1.2 wires" core/hitl_matrix_design.md` returns 0 hits. ### Task 39.2 — Refresh audit_ledger_design.md (REQ-101, security-engineer) - Mark the "Spike scope (D-041)" section as **shipped + production since v1.8** (hash chain + DynamoDB outbox + `acdl-evidence` mirror). - Move the "v1.2 build-out" section (S3 Object Lock + JWS + async worker + DLQ + daily checkpoints) under a clearly-labeled "**Deferred to a future milestone (D-083)**" heading. Keep the content (it's the design for when it ships) but mark it not-v1.9. - Update the RPO/RTO table: spike row → "v1.8+ (production): RPO=0 (sync outbox), RTO=workflow re-run"; v1.2 row → "Future milestone (D-083): RPO=0, RTO=DLQ replay". - Update the outbox item shape: note `approver_qa`/`approver_prod`/`approver_dr` are populated by v1.9's `hitl_gates.attest` (Phase 42). - Verify: `grep -i "Phases 08-10 implement" core/audit_ledger_design.md` returns 0 hits; the deferred section is clearly labeled. ### Task 39.3 — P1-1 adapter parameterization (REQ-102, platform-engineer) - `modules/l1/ecs-service/interface.json`: add inputs `desired_count` (integer, default 1), `launch_type` (string, default "FARGATE"), `family` (string, default "app"). - `modules/l1/alb/interface.json`: add inputs `load_balancer_type` (string, default "application"), `target_type` (string, default "ip"). - `modules/l1/vpc/interface.json`: add input `name` (string, default "app") for the VPC/IGW/RT `Name` tag prefix. - `adapters/terraform/adapter.py`: change hardcoded defaults to `inputs.get("", "")` where the default matches the interface default (safety fallback; the resolver populates from the interface). Remove the hardcoded `Name = "acdl-microservice-rt"` (line 283) → use `inputs.get("name", "app")`-derived tag. - Preserve the v1.1 S3 regression (S3 has none of these inputs → no change). - Tests: `tests/test_p1_1_adapter_parameterization.py` — (a) `desired_count: 3` in contract inputs emits `desired_count = 3`; (b) absent `desired_count` emits `desired_count = 1` via interface default; (c) `target_type: "instance"` emits `target_type = "instance"`; (d) v1.1 S3 regression still passes (byte-identical `main.tf`). - Verify: `pytest tests/test_p1_1_adapter_parameterization.py` passes; `run_platform.sh --check-only` exits 0; `pytest` total count increases; v1.1 S3 regression test passes. ### Task 39.4 — Design doc test (REQ-100/101, backend-engineer) - `tests/test_design_docs_current.py`: assert (a) no stale "dev-only spike" / "v1.2 wires the gates" / "Phases 08-10 implement" framing in either design doc; (b) `audit_ledger_design.md` has a "Deferred to a future milestone" section referencing D-083; (c) `hitl_matrix_design.md` references the v1.9 implementation (`attestation_matrix.py`, `hitl_gates.py`). - Verify: `pytest tests/test_design_docs_current.py` passes. ### Must-haves (Phase 39) - [ ] `core/hitl_matrix_design.md` refreshed (no stale framing). - [ ] `core/audit_ledger_design.md` refreshed (S3 Object Lock marked deferred D-083). - [ ] Adapter has no hardcoded ECS/ALB/VPC defaults (read from inputs). - [ ] `tests/test_p1_1_adapter_parameterization.py` + `tests/test_design_docs_current.py` pass. - [ ] `run_ci.sh` exits 0; `run_platform.sh --check-only` exits 0; v1.1 S3 regression passes. --- ## Phase 40 — contract-interpolation **Requirements:** REQ-103, REQ-104 **Personas:** backend-engineer (lead), platform-engineer (review) **Branch:** `phase/40-contract-interpolation` ### Task 40.1 — Environment JSON schema (REQ-104, backend-engineer) - `schemas/environment.schema.json` (draft 2020-12): required `name` (string), `account_id` (string), `region` (string), `state_backend` (object: `bucket`, `lock_table`), `network` (object: `vpc_cidr`, `azs` array), `runner_role_arn` (string), `autonomy` (enum: full/attested), `confidence_threshold` (number). - `core/environments/dev.json` validates against it. - Add `core/environments/qa.json`, `prod.json`, `dr.json`: `account_id: "000000000000"`, `autonomy: "attested"`, `confidence_threshold` 0.75/0.90/0.95, regions us-east-1, state_backend buckets `acdl-qa-state`/`acdl-prod-state`/`acdl-dr-state`. - `core/environment_check.py`: add `load(env_name, root=None)` returning the parsed env dict; `check()` stays. Add a stderr warning when `account_id == "000000000000"` and `env_name != "dev"` (prompts real binding). - `tests/test_environment_schema.py`: all 4 env files validate; `load("dev")` returns the dict; warning emitted for qa/prod/dr placeholders. - Verify: `pytest tests/test_environment_schema.py` passes. ### Task 40.2 — Interpolation expansion in the resolver (REQ-103, backend-engineer) - `core/contract_resolver.py`: add `_expand_vars(value, context)` — recursively walks dicts/lists/strings; replaces `${env.}` and `${contract.}` tokens by looking up the dotted path in the context dict. Unknown token → `ValueError(f"unresolved interpolation token: {token}")`. - `resolve()`: after schema validation, load the env via `environment_check.load(contract["environment"])`, build `context = {"env": env, "contract": contract}`, expand all string values in `contract["inputs"]` (recursively, per D-087), then proceed to IR resolution. - The expansion is post-schema-validation (schema sees the raw tokens, which are valid strings) and pre-IR-resolution (the resolver sees concrete values). - `tests/test_interpolation.py`: (a) `${env.region}` expands to `us-east-1`; (b) `${env.state_backend.bucket}` expands to `acdl-dev-state`; (c) `${contract.module}` expands to `static-assets`; (d) unknown token raises `ValueError`; (e) nested map value `env: { DB_URL: "acdl-${env.environment}-db" }` expands recursively; (f) `resolve("contracts/static-assets.yaml")` succeeds with expanded values. - Verify: `pytest tests/test_interpolation.py` passes. ### Task 40.3 — Sample contracts use naming patterns (REQ-103, backend-engineer) - `contracts/static-assets.yaml`: `bucket_name: acdl-${env.environment}-${contract.module}-${env.account_id}-${env.region}` (the naming pattern the requirement calls out: region + account id + environment). - `contracts/microservice.yaml`: same pattern for `bucket_name`. - Keep `region: us-east-1` as a literal (or `${env.region}` — both valid; use `${env.region}` to demonstrate). - `tests/test_sample_contracts_interpolate.py`: resolving the sample contracts produces concrete bucket names like `acdl-dev-static-assets-000000000000-us-east-1`. - Verify: `pytest tests/test_sample_contracts_interpolate.py` passes; `run_platform.sh --check-only` exits 0 (resolver expands before adapter). ### Must-haves (Phase 40) - [ ] `schemas/environment.schema.json` exists; 4 env files validate. - [ ] `_expand_vars` in resolver; unknown tokens raise. - [ ] Sample contracts use `${env.*}` + `${contract.*}` naming patterns. - [ ] `tests/test_environment_schema.py` + `tests/test_interpolation.py` + `tests/test_sample_contracts_interpolate.py` pass. - [ ] `run_ci.sh` exits 0; `run_platform.sh --check-only` exits 0. --- ## Phase 41 — per-environment-ci-jobs **Requirements:** REQ-105, REQ-106 **Personas:** backend-engineer (lead), security-engineer (HITL gate review) **Branch:** `phase/41-per-environment-ci-jobs` ### Task 41.1 — Per-env contract files (REQ-105, backend-engineer) - `contracts/static-assets.dev.yaml`, `.qa.yaml`, `.prod.yaml`, `.dr.yaml` — each sets `environment:` to its own name; `inputs.bucket_name` uses `${env.environment}-${contract.module}-${env.account_id}-${env.region}` interpolation (so the file content is near-identical; only `environment:` differs). - `contracts/microservice.{dev,qa,prod,dr}.yaml` — same pattern. - Keep `contracts/static-assets.yaml` + `contracts/microservice.yaml` as the dev default (backwards compat). - `tests/test_per_env_contracts.py`: all 8 per-env files validate against `schemas/contract.schema.json`; each resolves to a stack with the correct environment. - Verify: `pytest tests/test_per_env_contracts.py` passes. ### Task 41.2 — Deploy workflow `environment` input (REQ-106, backend-engineer) - `.github/workflows/deploy.yml` + `.gitea/workflows/deploy.yml` (byte-identical): add `environment` input (`type: string`, default `""`, description "Target environment override (dev/qa/prod/dr); when empty, the contract's environment field is used"). - `scripts/run_platform.sh`: add `--environment ` flag. When set, override the contract's `environment` field at load time (before schema validation per D-088, so interpolation context is consistent). Re-run the onboarding check against the supplied env. - The workflow's "Run the platform pipeline" step passes `--environment ${{ inputs.environment }}` when non-empty. - `tests/test_deploy_workflow_env_input.py`: both deploy workflows declare the `environment` input; byte-identical; `run_platform.sh --environment qa contracts/static-assets.yaml` produces a stack whose env is qa (tested via the resolver directly since run_platform.sh needs AWS for full mode — test the override logic in the resolver). - `core/contract_resolver.py` `resolve()`: accept optional `environment_override` arg; when set, set `contract["environment"] = override` before schema validation + interpolation. - Verify: `pytest tests/test_deploy_workflow_env_input.py` passes; both deploy workflows byte-identical. ### Task 41.3 — Per-env caller workflow docs + HITL gate structure (REQ-106, security-engineer review) - `docs/CONSUMER_GUIDE.md`: add a "Per-environment deployment" section with 4 caller-workflow examples (`.github/workflows/deploy-dev.yml`, `deploy-qa.yml`, `deploy-prod.yml`, `deploy-dr.yml`), each `uses: acdl/.github/workflows/deploy.yml@v1.9` with `environment: ` + `contract: .acdl/..yaml`. Document: "Promotion = running the matching job; no `environment:` field editing." - HITL gate structure (wired in Phase 42, documented here): qa/prod/dr caller workflows use `workflow_dispatch` with approval inputs (`approve_qa`, `approve_prod`, `approve_dr`) per `hitl_matrix_design.md` D-042; `gitea.actor` / `github.actor` is the approver of record. dev is autonomous (no gate). - `tests/test_consumer_guide_per_env_section.py`: the consumer guide has the per-env section with 4 caller examples. - Verify: `pytest tests/test_consumer_guide_per_env_section.py` passes. ### Must-haves (Phase 41) - [ ] 8 per-env contract files exist + validate + resolve. - [ ] Deploy workflow has `environment` input (byte-identical Gitea + GitHub). - [ ] `run_platform.sh --environment ` overrides; resolver supports `environment_override`. - [ ] Consumer guide documents per-env caller workflows + promotion-without-editing. - [ ] `tests/test_per_env_contracts.py` + `tests/test_deploy_workflow_env_input.py` + `tests/test_consumer_guide_per_env_section.py` pass. - [ ] `run_ci.sh` exits 0; both deploy workflows byte-identical. --- ## Phase 42 — stub-implementation **Requirements:** REQ-107, REQ-108, REQ-109, REQ-110, REQ-111 **Personas:** security-engineer (lead), backend-engineer (run_platform wiring), lambda-engineer (SNS topic Terraform) **Branch:** `phase/42-stub-implementation` ### Task 42.1 — route_halt_artifact real (REQ-107, security-engineer + lambda-engineer) - `core/separation_of_duties.py` `route_halt_artifact`: when `ACDL_SOD_HALT_TOPIC_ARN` set, publish to SNS via boto3 (`sns.publish(TopicArn=arn, Message=..., Subject="ACDL SoD halt")`); when unset, fall back to structured stderr emission + a `SEPARATION_OF_DUTIES_VIOLATION` event write via `outbox_writer.write_event` (so the halt is in the audit chain). No silent print-only stub. - `terraform/platform/main.tf`: add `aws_sns_topic.acdl-sod-halt` + a basic access policy (allow the platform Lambda / runner role to publish). Output the topic ARN. - `tests/test_route_halt_artifact.py`: (a) with `ACDL_SOD_HALT_TOPIC_ARN` set, moto-mocked SNS receives the publish; (b) without it, a `SEPARATION_OF_DUTIES_VIOLATION` event is written to the outbox (moto-mocked DynamoDB); (c) stderr emission occurs in both cases. - Verify: `pytest tests/test_route_halt_artifact.py` passes. ### Task 42.2 — HITL attestation gates (REQ-108, security-engineer + backend-engineer) - `core/hitl_gates.py`: `attest(contract_id, env, approver, evidence, outbox_client=None)` → records `approver_qa`/`approver_prod`/`approver_dr` to the outbox item for `contract_id`; runs `separation_of_duties.check(outbox_client, contract_id, approver)` on prod; invokes the attestation matrix (Task 42.3) for the target env; returns `(ok, reason)`. Dev skips (returns `(True, "dev autonomous")`). - `scripts/run_platform.sh`: before apply (for qa/prod/dr), call `hitl_gates.attest` with the approver from `GITHUB_ACTOR`/`GITEA_ACTOR` env. Block on `(ok=False)`. - `tests/test_hitl_gates.py`: (a) dev skips; (b) qa records `approver_qa` (moto outbox); (c) prod records `approver_prod` + SoD blocks when `approver_qa == approver_prod`; (d) prod passes when approvers differ. - Verify: `pytest tests/test_hitl_gates.py` passes. ### Task 42.3 — 8-concern attestation matrix (REQ-109, security-engineer) - `core/attestation_matrix.py`: `check(env, evidence_bundle)` → runs the 8 concerns. Offline-testable concerns (contract NFRs, schema validity, policy pass) run for real. Operator-supplied concerns accept an uploaded signed evidence artifact (JSON with `timestamp`, `type`, `payload`, optional `signature`); validate freshness (within the declared window from `hitl_matrix_design.md` §10.4) + schema (per-concern). Signature verification via KMS when `ACDL_ATTESTATION_SIGNING_KEY_ID` set; skipped + logged when unset (D-089). Fail loud if missing/expired for prod/dr. - `hitl_gates.attest` calls `attestation_matrix.check(env, evidence)` and blocks on any failing concern. - `tests/test_attestation_matrix.py`: (a) offline concerns pass for a valid contract; (b) operator-supplied concern missing → block for prod; (c) operator-supplied concern present + fresh → pass; (d) expired artifact → block; (e) signature skip when key unset (logged). - Verify: `pytest tests/test_attestation_matrix.py` passes. ### Task 42.4 — Wiz real API client (REQ-110, security-engineer) - `adapters/wiz/wiz_adapter.py`: add `WizClient` class — `__init__` reads `WIZ_API_TOKEN` + `WIZ_API_URL`; `fetch_issues(filter_by)` queries the Wiz GraphQL API (`/graphql`, Bearer auth, `issues` query). Translate results → `PolicyCheckResult` records (`engine: "wiz"`, `ruleId: `, `severity: `, `status: FAIL`, `message: `, `resource: <entity.name>`). Graceful degrade: when `WIZ_API_TOKEN` or `WIZ_API_URL` unset → emit the existing single `SKIPPED` `WIZ_NOT_CONFIGURED` record (no network call). Pagination handled via `pageInfo.hasNextPage`. - `tests/test_wiz_adapter_real_client.py`: (a) with a recorded GraphQL fixture, `WizClient` translates issues → `PolicyCheckResult` records; (b) graceful degrade when env unset; (c) pagination follows `endCursor`. - Verify: `pytest tests/test_wiz_adapter_real_client.py` passes. ### Task 42.5 — Kyverno translator fleshed out (REQ-111, security-engineer) - `adapters/kyverno/kyverno_adapter.py`: full `PolicyReport` → `PolicyCheckResult` mapping — handle `pass`/`fail`/`skip`/`warn` results, severity mapping (critical/high/medium/low/info), resource extraction, skip-with-reason handling. Keep the inactive-for-Terraform guard (emits a single `SKIPPED` `KYVERNO_INACTIVE_TF_STACK` record when no K8s manifests). Add a `--kube-version` stub (parsed but not yet used — for future GitOps). - `tests/test_kyverno_adapter.py`: expand — (a) `pass` result → `PolicyCheckResult` with `status: PASS`; (b) `fail` with severity → correct severity mapping; (c) `skip` with reason → `SKIPPED` record; (d) inactive-for-TF guard emits the `KYVERNO_INACTIVE_TF_STACK` record. - Verify: `pytest tests/test_kyverno_adapter.py` passes. ### Must-haves (Phase 42) - [ ] `route_halt_artifact` real (SNS + outbox fallback); SNS topic in Terraform. - [ ] `hitl_gates.py` attests qa/prod/dr; SoD blocks on identity equality. - [ ] `attestation_matrix.py` implements 8 concerns (offline-testable + signed artifacts). - [ ] Wiz adapter real client + graceful degrade. - [ ] Kyverno translator fleshed out + inactive guard preserved. - [ ] All 5 new test files pass; `run_ci.sh` exits 0. --- ## Phase 43 — verify-review-audit-complete **Requirements:** — (milestone gate) **Personas:** lead-developer (lead), all personas (review participation) **Branch:** `phase/43-verify-review-audit-complete` ### Task 43.1 — 4-layer verify - Structural: all new files present (environment.schema.json, 4 env files, 8 per-env contracts, hitl_gates.py, attestation_matrix.py, SNS topic in main.tf, 5+ new test files). - Behavioral: `pytest` passes (count increases from v1.8's 350 by ~30+ new tests); `run_ci.sh` exits 0; `run_platform.sh --check-only` exits 0. - Security: no hardcoded adapter defaults; HITL gates block on SoD violation; attestation matrix fails loud on missing evidence for prod/dr; Wiz degrades gracefully. - Quality: each new feature has dedicated tests (interpolation, per-env jobs, SoD, HITL gates, attestation matrix, Wiz, Kyverno). ### Task 43.2 — Multi-persona review - `ciagent-review` across the v1.9 diff (phases 39–42). Auto-apply P0; flag P1+ for post-hoc. - Reconstruct `.ciagent/REVIEW.md` with v1.9 content (D-086). Note that v1.3–v1.8 reviews were not persisted (no git-history rewrite). ### Task 43.3 — Audit - Reconstruction: git log matches `.ciagent/` files. - File discipline: all `.ciagent/` files valid. - Branch hygiene: stale branches cleaned. - Commit discipline: all commits have `---ci---` blocks. ### Task 43.4 — Complete - Update `.ciagent/REQUIREMENTS.md`: mark REQ-100..REQ-111 complete; add v1.9 traceability table. - Update `.ciagent/ROADMAP.md`: add v1.9 milestone section (complete). - Update `.ciagent/PROJECT.md`: v1.9 status → complete. - Tag `v1.9.0`; update floating `v1.9` + `v1` tags. - Bump `uses:`/`ref:` from `@v1.6` → `@v1.9` in `contracts/*.yaml`, `deploy.yml` checkout `ref:`, `docs/CONSUMER_GUIDE.md` (D-071 successor). - Commit: `docs(milestone): complete v1.9`. ### Must-haves (Phase 43) - [ ] 4-layer verify PASS. - [ ] Review: 0 new P0; P1+ flagged for post-hoc. - [ ] Audit: clean. - [ ] Tag `v1.9.0` created; floating tags updated. - [ ] `uses:`/`ref:` bumped to `@v1.9`. - [ ] REQUIREMENTS.md + ROADMAP.md + PROJECT.md updated. --- *End of PLAN.md.*