acdl/.ciagent/ARCHITECTURE.md

# ACDL — Architecture (v1.1 target)

> Target architecture for the real Agentic Cloud Delivery Platform.
> Source of truth for **how**: `docs/architecture.md` (v0.2) is the upstream
> draft; this file is the ACDL-repo operating copy, refined at phase
> boundaries. Where this file and `docs/vision.md` conflict, the vision wins.

## Status

Architecture is at **v0.2** upstream (`docs/architecture.md`). Milestone v1.1
**finalizes it to v1.0** in Phase 07 by resolving the 11 open decisions
(see `PROJECT.md` open-decision resolutions table). This file records the
locked commitments and the v1.1 spike scope.

## Overview

The platform is **four layers + six cross-cutting concerns**. The sixth
concern — the substrate abstraction (§12) — is first-class, not an
implementation detail. The vision's "Two Consumer Surfaces, One Platform"
tenet binds everything: L3A and L3B converge on the same contract schema,
the same policy envelope, and the same evidence stream.

```
                  ┌──────────── acdl-contracts ────────────┐
  Developer ───▶  │ commit contract.yaml                   │   (L3A)
  Citizen dev ──▶ │ Issue → agent → contract.yaml            │   (L3B)
                  └────────────────┬───────────────────────┘
                                   │ (push)
                                   ▼
                         ┌──────────────────────┐
                         │ central pipeline      │
                         │ (acdl repo, Gitea      │
                         │  Actions / act_runner) │
                         └────────┬─────────────┘
                                  │
        ┌─────────────────────────┼─────────────────────────┐
        ▼                         ▼                           ▼
  contract→IR resolution   policy (Checkov/Kyverno)    confidence signal
        │                         │                           │
        ▼                         ▼                           ▼
  Terraform adapter ──▶ terraform plan ──▶ PolicyCheckResult ──▶ {score,band}
        │                                                                │
        ▼                                                                ▼
  dev (autonomous, ≥0.50)   qa (HITL, ≥0.75)   prod (HITL, ≥0.90)   dr (HITL, ≥0.95)
        │
        ▼
  DynamoDB outbox ──▶ S3 Object Lock (7-yr, source of truth) ──▶ GitHub audit repo (hot index)
        │
        ▼
  acdl-evidence (timeline UI)
```

## Layers

### Layer 1 — Foundational Primitives
Single-purpose, **substrate-agnostic** primitive modules. L1 modules do
not compose with other L1s; L1 takes its environment as input. The L1
interface is defined against the **Target Stack IR**, not against Terraform
directly (the IR is shaped to round-trip to Terraform in v1, per §12.1).

- No inter-L1 references. L1 may call Terraform data sources.
- Semver: interface → MAJOR, behavior → MINOR, lifecycle → PATCH (W3.D).
- Immutability on publication. 12-month deprecation window.
- AI refinement is a flag; the trigger is the W1.A joint condition.

### Layer 2 — Composed Stacks
Combine L1 primitives into deployable shapes. Each codebase maps to one
canonical L2 stack (`multiStack: true` only per W1.B). Shape X
(parameterized module) or Shape Y (thin-composition layer). Hierarchical
composition, max depth 5, only registered L1s. The thin-composition tree's
`wires` field is defined against the IR's relationship type, not a Terraform
module block.

Pipeline quality checks: secrets-in-plaintext, public ingress, IAM
wildcard, KMS key reference, tag compliance, naming convention. Restricted
from thin-composition: IAM principal creation, network boundary creation,
key/secret creation, external data transfer. Auto-promote after 3 observed
usages.

### Layer 3A — Developer Consumer Surface
Tag-based reference to the central pipeline template. Developer-owned
workflow file, no platform auto-sync. L3A and L3B are parallel paths, not a
progression. **W2.A (Path B):** tag for dev/qa, SHA for prod; platform CLI
resolves tag→SHA for prod-bound workflows.

### Layer 3B — Agentic Consumer Surface
Hybrid runtime, skill as markdown, agent as executor. Trust model: trust
and always verify on the platform side. Skill envelope (4 dimensions).
Stateless agents, all state in the platform. `profile: agentic` marker
unlocks `naturalLanguageIntent`, `confidenceAtSubmission`, `agentTrace`.
Initial skill catalog (BA.A): web API, worker, scheduled job, static asset,
basic observability bootstrap.

Environment progression:

| Environment | Autonomy | Attester | Gate |
|---|---|---|---|
| dev | Full autonomy (no HITL) | — | Confidence ≥ 0.50, all six inputs present |
| qa | Held for attestation | QA | GitHub Deployment approval + full QA matrix (§10) |
| prod | Held for attestation | SRE | GitHub Deployment approval + full SRE matrix (§10) |
| dr | Held for attestation | SRE | GitHub Deployment approval + dr-drill evidence |

**Staging is removed.** Dev is the only autonomous environment.

## Cross-cutting concerns

### Central pipeline template (§6)
JSON Schema (draft 2020-12) with a thin domain wrapper. Central repo +
generated client libraries. Multi-stage validation: schema → policy → NFR →
confidence. Distributed enrichment. GitOps reconciler (K8s API; cdlc-gitops
state → CRDs) + Terraform execution layer (§12.5). The pipeline emits one
`PolicyCheckResult` per policy rule; the confidence signal consumes them as
one normalized input.

### Contract schema (§7)
Central repo + generated client libraries. Strict fail-fast at schema
stage, multi-stage validation with reason codes from a published
vocabulary. **W3.E:** per-env mandatory inputs —
- dev: `stack`, `environment`
- qa adds: `validation.e2eSuite`, `validation.loadTest`
- prod adds: `runbook`, `dashboard`, `oncall`
- dr adds: `drDrillRef`
- `inputs` always optional; `profile: agentic` fields optional everywhere.

### Confidence signal (§8)
Six canonical inputs, weighted sum with per-input breakdown. Per-env
thresholds: dev ≥ 0.50, qa ≥ 0.75, prod ≥ 0.90, dr ≥ 0.95. Structured output
`{ score, band, perInput, reasonCodes }`. 1-year storage, no retraining in
v1. Halt with explicit reason on missing input.

Policy input = list of `PolicyCheckResult` records (engine-agnostic).
Severity → penalty: critical → hard override to mandatory block; high →
-0.2; medium → -0.05; low → -0.01; info → 0.0. One critical finding
hard-overrides the score regardless of all other inputs.

**BA.B:** thresholds frozen for v1; tuning begins v1.2 (quarterly FP/FN
tracking; override = Infra & Ops + SRE joint sign-off, itself a
confidence-event).

### Audit and evidence stream (§9)
Tiered ledger: **S3 with Object Lock in compliance mode** (cold, source of
truth, 7-year retention) + **GitHub audit repo** (`acdl-evidence`, hot
query index, not part of the chain). Daily checkpoints. Event schema: JWS
detached signature, `prev_event_hash` chain, controlled-vocabulary
`event_type`. Outbox pattern: local durable outbox + async worker.

Outbox database = **DynamoDB**. RPO = 0 (synchronous write to local outbox
before contract submission ack); RTO = async worker's dead-letter recovery.
Single-region in v1. The outbox also stores per-contract QA and prod
approver identities (the only durable record outside GitHub's audit log).

### Human-in-the-Loop mechanics (§10)
Pre-execution gates. qa, prod, dr are PR-based attestation gates backed by
GitHub Environments with required reviewers. No partial deployment to roll
back on rejection (qa, prod); dr is a separate GitHub Deployment against a
separate cluster/region.

Reviewer routing: GitHub CODEOWNERS + Environment required reviewers
(qa → QA; prod → SRE; dr → SRE). CODEOWNERS routes, does not enforce
identity distinctness.

**Separation of duties** (platform-internal, not GitHub-native, not Kyverno
in v1): on dev→qa promotion the platform writes the QA approver's GitHub
identity to the DynamoDB outbox keyed by `contractId`; on qa→prod it reads
the stored QA approver and the new SRE approver; if equal, it blocks, emits
`SEPARATION_OF_DUTIES_VIOLATION`, and routes a halt artifact to SRE on-call.

Full 8-concern attestation matrix (functional, performance, security
posture, contract NFRs, operational readiness, incident response,
capacity/cost, resilience) — see `docs/architecture.md` §10.4.

Timeout: 1 business day = warn + escalate; 2 business days = auto-freeze +
re-submit (linked via `supersedes`). Rejection returns the contract to HELD;
the audit chain is extended, not torn up.

### Agentic stack (§11)
Hybrid runtime: platform-managed control plane + consumer-owned agent.
Versioned, signed skill catalog over MCP. Skill envelope enforced on
invocation and result submission. Consumer-owned skill execution; the
platform does not run the skill. Stateless agents, all state in the
platform. Skills are reviewed for sensitive data before release (Infra &
Ops owns the review; it is the mandatory release gate).

### Substrate execution (§12) — the binding constraint
**Target Stack IR** (locked): a substrate-neutral description of resources
(typed inputs/outputs/NFRs), relationships (single parent per child),
composition (tree, max depth 5), and policy hooks. The L1 registry, L2
thin-composition tree, contract YML, and PolicyCheckResult schema are all
defined against the IR — none against any specific substrate.

**Substrate adapters** are the only substrate-specific code. An adapter
compiles the IR into a substrate execution plan. **v1 ships exactly one
adapter: the Terraform adapter.** v2+ may add OpenTofu, Pulumi, K8s CRDs
without architectural change.

v1 reality: the IR is shaped to round-trip cleanly to Terraform (nearly
isomorphic). As more adapters appear, the IR gets more expressive and the
adapters gain translation logic; the L1 content, the YML standard, and the
thin-composition tree do not change.

**Terraform adapter (v1):** translates IR-typed L1 interface → Terraform
`variable`/`output` blocks; IR-typed L2 thin-composition tree → Terraform
root module; IR-typed relationships → module references; emits a
`terraform plan` from the IR. The adapter is a thin layer; it does not own
L1/L2 content.

State storage: S3 (state) + DynamoDB (locking), cloud-managed,
single-region in v1.

Policy toolchain: **Checkov** for Terraform plan policy (the L2 checks +
tag/naming); **Kyverno** for K8s-native/platform-internal policy; **OPA**
reserved for cross-resource cases, explicitly last resort.

**Policy result normalization (§12.6):** the confidence signal consumes a
normalized `PolicyCheckResult` schema, not raw engine output.

```json
{
  "contractId": "uuid",
  "evaluatedAt": "ISO-8601",
  "engine": "checkov | kyverno | opa",
  "ruleId": "CKV_AWS_24 | KYVERNO_NO_PRIVILEGED | ...",
  "severity": "critical | high | medium | low | info",
  "result": "pass | fail | skipped | error",
  "message": "human-readable",
  "evidence": { "...engine-specific, opaque to the signal..." },
  "resourceRef": "IR-typed resource identifier"
}
```

Execution layer: GitHub/Gitea Actions in the central pipeline repo. State
locking via DynamoDB. **AWS credentials via OIDC federation — long-lived
credentials are forbidden** (§12.5). The platform does not run
`terraform apply` against a developer's workstation; all execution is in
the central pipeline.

Registry maintenance: L1 publication updates the L1 registry in the same
PR. The registry is the IR-typed contract, not a Terraform-specific
variable schema.

Contract→IR resolution: the contract declares intent in IR-typed terms;
the pipeline resolves it to a target stack (list of L1 instances + inputs +
relationships); the Terraform adapter compiles the target stack to a plan.

## v1.1 spike scope

The spike (Phases 08–10) materializes the **minimum** that proves the IR
commitments hold (no polyglot mess):

- One L1: `l1-s3` (IR-typed interface; the only AWS resource in the spike).
- One L2 thin-composition: `l2-static-asset` (references `l1-s3` only).
- Terraform adapter: IR → `terraform plan` against AWS via OIDC.
- One contract submission → contract→IR → `terraform plan` → Checkov
  `PolicyCheckResult` → confidence signal → evidence event to the DynamoDB
  outbox.
- State: S3 + DynamoDB (real AWS, single-region).

Out of spike scope: full HITL matrix wiring, Kyverno, OPA, MCP skill
catalog, GitOps reconciler, multi-region, prod/dr environments, the 5-skill
L3B catalog. Those are post-spike (v1.2+) platform build-out.

## Gitea API surface (carried from v1.0, refined)

| Capability | Gitea support | ACDL approach (v1.1) |
|------------|---------------|----------------------|
| Org-scoped repo create | `POST /api/v1/orgs/{org}/repos` | Used for any new repos |
| Native Pages | **None** | Serve `acdl-evidence` via raw file URLs (unchanged from v1.0) |
| Environments API | **None**; act_runner ignores `environment:` | Model HITL gates via `workflow_dispatch` approval inputs (v1.0 D-013 pattern) — **refined in Phase 07** for the real pre-execution gate model |
| `repository_dispatch` | Not supported | Cross-repo trigger via `workflow_dispatch` API (unchanged) |
| Reusable workflows | Supported | `acdl/.gitea/workflows/pipeline.yml` via `uses: ...@<ref>` |
| `id-token: write` / OIDC | **Not supported** (RESEARCH TARGET 1, conf 0.95). Gitea docs list `id-token` as an unsupported GitHub-only scope; open proposal go-gitea/gitea#33681; draft PR go-gitea/gitea#36988 unmerged. Even Gitea's own CI uses long-lived AWS keys (issue #37980). | **Spike waiver D-039:** per-run-rotated long-lived key (rotated after each run by `scripts/rotate_spike_key.sh`). Real OIDC deferred to v1.2, blocked on PR #36988. |
| `actions/configure-aws-credentials` | Unusable without OIDC | Spike uses static AWS creds from a (rotated) Gitea Actions secret via the `aws-actions/configure-aws-credentials@v4` `access-key-id`/`secret-access-key` inputs, or plain `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` env vars. v1.2 switches to `role-to-assume` when OIDC lands. |

### Branch pinning rule (refined for W2.A)

- Dev/qa contracts reference the reusable workflow by **tag**
  (`@v1.1-spike`).
- Prod-bound workflows reference by **SHA**; the platform CLI
  (`platform/cli/resolve-tag.ts`, Phase 07) resolves the current tag to its
  SHA. (Spike scope: the CLI is a stub; the real CLI lands in v1.2.)

### Verification toolchain

ACDL has no `package.json`. The verification gate substitutes:
- **typecheck:** `terraform validate`, `python3 -m py_compile`, JSON Schema
  validation (`ajv` or `python -m jsonschema`) against `schemas/`.
- **test:** per-phase `scripts/verify_phaseNN.sh` (Phase 06: archive integrity;
  Phase 07: schema validation + decision-resolution completeness; Phase 08:
  OIDC assume-role + state backend; Phase 09: IR + L1 + adapter `terraform
  plan`; Phase 10: end-to-end contract submission).
- **build:** `terraform init` (real build for the spike).
- See `PERSONAS.md` verification_toolchain.

## Build order (v1.1)

1. Phase 06 — archive demo, reorient repo.
2. Phase 07 — finalize architecture v1.0; author schemas + designs.
3. Phase 08 — AWS OIDC bootstrap (use temp key once, rotate).
4. Phase 09 — IR + `l1-s3` + Terraform adapter → `terraform plan`.
5. Phase 10 — `l2-static-asset` + contract→IR → end-to-end spike.
6. COMPLETE gate — review → ship `v1.2.0` → audit. **DONE.**

## v1.2 build-out scope

v1.2 takes the v1.1 spike (dev-only, `plan`-only, single S3 L1) to a real,
simpler, better-documented platform that delivers a microservice to AWS ECS
Fargate end-to-end. The locked architecture (§1–§12) is unchanged — v1.2
extends the *implementation*, not the design.

### In scope (five axes, user-directed 2026-07-21)

1. **Re-evaluate the current state.** go-gitea/gitea#36988 (OIDC for Gitea
   Actions) re-checked 2026-07-21: still **open** (last updated 2026-05-27,
   not merged). Real OIDC remains deferred to v1.3+; v1.2 extends the D-039
   per-run-rotated-key waiver as **D-047**. The waiver continues to satisfy
   §12.5's *intent* (no *persistently* long-lived key): the spike key is
   rotated after each run by `scripts/rotate_spike_key.sh`, and Phase 12
   tightens the IAM scoping + rotation hygiene.
2. **NFR improvements on the existing spike.** Least-privilege IAM audit of
   `spike_runner_policy.json`; idempotent `create_state_backend.py` /
   `create_iam_user.py`; proper exit codes / error handling; P1-1 redaction
   (two AWS access key IDs in `.ciagent/VERIFY.md` Phase 09 narrative).
3. **Streamline / simplify the current setup.** Consolidate
   `run_spike_plan.sh` + `run_spike_e2e.sh` into one
   `scripts/run_platform.sh`; remove dead code and stale `platform/` paths.
4. **README.md fully up to date on how the platform works.** Reflect v1.1
   complete; document the actual spike flow, `scripts/run_platform.sh`, the
   real repo layout, and the v1.2 objective.
5. **Bootstrap a consumer repo with a basic microservice deployed to ECS
   end-to-end.** New Gitea repo `acdl-consumer-microservice` (org
   `continuous-intelligence`); new IR-typed L1s (`l1-vpc`, `l1-ecs-cluster`,
   `l1-ecs-service`, `l1-iam-role`, `l1-alb`, `l1-ecr`); new
   `l2-microservice` thin-composition; one contract submission →
   `terraform apply` (dev, autonomous per §10, confidence ≥ 0.50) → a live
   ECS Fargate service serving HTTP 200 → evidence event to the DynamoDB
   outbox → acdl-evidence timeline.

### Substrate extension (ECS Fargate)

The Terraform adapter (§12) remains the only substrate-specific code. v1.2
expands the adapter `TYPE_MAP` to cover the six new ECS-shaped IR resource
types. The L1 interface shape (IR-typed inputs/outputs/NFRs, registered in
`modules-ir/registry.json`) is unchanged — only the set of registered L1s
grows. The IR commitments (REQ-28) continue to hold: `modules-ir/`,
`schemas/`, `contracts/`, `acdl_platform/confidence_signal.py`,
`acdl_platform/contract_resolver.py`, `acdl_platform/outbox_writer.py`
remain substrate-agnostic.

### `terraform apply` (dev only)

v1.2 lifts the substrate execution from `plan` to `apply` for the `dev`
environment only. Dev is autonomous per §10 (confidence ≥ 0.50, no HITL).
`apply` for qa/prod/dr remains HITL-gated and out of scope for v1.2. The
apply result (resources created, plan diff) is captured in the evidence
stream as a `terraform.apply` event.

### Out of scope for v1.2 (deferred to v1.3+)

| Feature | Reason |
|---------|--------|
| Real OIDC federation | go-gitea/gitea#36988 still open. v1.2 extends D-039 waiver (D-047); real OIDC is v1.3+. |
| Full HITL matrix wiring (qa/prod/dr) | v1.2 is dev-only autonomous `apply`; HITL wiring is v1.3. |
| Kyverno + OPA policy engines | v1.2 keeps Checkov only; Kyverno/OPA are v1.3. |
| MCP skill catalog + real L3B agent | v1.2 keeps the L3B stub; the 5-skill catalog is v1.3. |
| Audit ledger build-out (S3 Object Lock + JWS + async worker + DLQ + daily checkpoints) | v1.2 keeps the v1.1 outbox; the regulatory ledger is v1.3. |
| Multi-region state / outbox | Single-region in v1 (§9, §12.3); multi-region is v1.3+. |
| Prod/dr environments | v1.2 is dev-only; prod/dr are v1.3. |
| GitOps reconciler (ArgoCD/Flux) | v1.3+. |

## Build order (v1.2)

1. Phase 11 — re-eval #36988 + NFR audit + simplification findings + README rewrite.
2. Phase 12 — NFR harden + simplify (idempotent bootstrap, one `run_platform.sh`, IAM audit, redactions).
3. Phase 13 — six ECS L1s + adapter `TYPE_MAP` expansion.
4. Phase 14 — `l2-microservice` + contract schema extension.
5. Phase 15 — consumer repo + `terraform apply` (dev) → live ECS service.
6. Phase 16 — capstone e2e: consumer commit → live HTTP 200 → evidence → timeline.
7. COMPLETE gate — review → ship `v1.3.0` → audit.