acdl/README.md

# ACDL — Agentic Cloud Delivery Platform

Consumers declare intent; the platform delivers safe production deployment
through an agentic stack — automatically, safely, and with a complete audit
trail. A merged change progresses through lower environments end-to-end
without a platform engineer joining a thread; a non-technical consumer ships
a production deployment by declaring intent, without authoring a workflow,
a configuration file, or a Terraform module.

- **Vision** (the why): [`docs/vision.md`](docs/vision.md)
- **Architecture** (the how): [`docs/architecture.md`](docs/architecture.md) + [`.ciagent/ARCHITECTURE.md`](.ciagent/ARCHITECTURE.md)
- **Decisions**: [`.ciagent/PROJECT.md`](.ciagent/PROJECT.md)
- **Phase plan**: [`.ciagent/ROADMAP.md`](.ciagent/ROADMAP.md)
- **Consumer guide**: [`docs/CONSUMER_GUIDE.md`](docs/CONSUMER_GUIDE.md)

## Repository roles

There are two kinds of repository in the ACDL model:

- **Platform repo (this one).** This is the **source code of the platform**.
  It owns `modules/`, `adapters/`, `acdl_platform/`, `schemas/`, `pipelines/`,
  `scripts/`, and the reusable workflow files. Platform engineers work here.
  A **consumer never clones it.**
- **Consumer repo (yours).** A consumer repo contains only its application
  code and a single `contract.yaml` that references the central pipeline +
  contract. The consumer does not write Terraform, workflow YAML, or adapter
  code — they write a contract YAML file and the platform does the rest.

The rest of this README describes the **platform repo** (how the platform
works, how to run it locally, how it's laid out). If you are a consumer,
jump to the [Consumer guide](docs/CONSUMER_GUIDE.md).

## Status

- **v1.5 (active):** consumer happy path + zero-trust docs + reusable deploy
  workflow. README rewritten so the consumer model is unambiguous. Platform
  flow + consumer guide converted to mermaid. Legacy surface + implementation
  nomenclature removed from docs. Credentials section rewritten for
  zero-trust OIDC + ABAC. A generic `docs/CONSUMER_GUIDE.md` (all L2 modules,
  versioned `uses:`, consumer-scoped prerequisites, run-time platform fetch)
  replaces the module-specific guide. A byte-identical reusable `deploy.yml`
  workflow (Gitea + GitHub) implements `pipelines/deploy.yaml` and is invoked
  by consumer repos via a versioned tag.
- **v1.4 (complete, tag `v1.4.1`):** central pipeline contract + shell
  reproducibility + output streaming. A declarative pipeline contract
  (`schemas/pipeline.schema.json` + `pipelines/ci.yaml`) binds the Gitea
  and GitHub workflows to a single source of truth. `scripts/run_ci.sh`
  mirrors the CI pipeline locally. `scripts/run_platform.sh` streams
  terraform/checkov output by default. L2 compositions re-introduced with
  a `uses:`-based contract resolution mechanism.
- **v1.3 (complete, tag `v1.3.2`):** module documentation. Testing + CI/CD
  pipelines (pytest, `--check-only`, Gitea + GitHub workflows).
- **v1.2 (complete, tag `v1.3.0`):** platform hardening + first real
  consumer deployment. Harden the v1.1 implementation's NFRs, simplify the
  setup, rewrite the docs, and prove the platform delivers real value by
  deploying a basic microservice to AWS ECS Fargate end-to-end (`terraform
  apply`, dev autonomous).
- **v1.1 (complete, tag `v1.2.0`):** architecture finalization + v1
  implementation. Finalized the architecture to v1.0 (resolved all 11 open
  design decisions) and proved the stack commitments hold with one
  end-to-end run (`s3` + `static-asset` + Terraform adapter → real
  `terraform plan` against AWS). Gitea release id 202.

## How the platform works

The platform is **four layers + six cross-cutting concerns**, bound by the
vision's "Two Consumer Surfaces, One Platform" tenet: consumers declare
intent via a contract; the platform delivers the deployment through the
same contract schema, the same policy envelope, and the same evidence
stream.

Consumers have their own repos and consume ACDL by referencing `uses:` the
central pipeline definitions. A consumer declares a contract (module +
environment + inputs); the platform resolves it to a stack instance,
compiles it to Terraform, runs policy checks, computes a confidence signal,
and writes an evidence event to the audit outbox.

### The platform flow (end-to-end)

```mermaid
flowchart TD
    A["contracts/static-asset.yaml<br/>(consumer contract: uses + module + inputs)"] --> B
    B["schema validation<br/>(schemas/contract.schema.json)"] --> C
    C["acdl_platform/contract_resolver.py<br/>→ Target Stack (JSON)"] --> D
    D["stack schema validation<br/>(schemas/stack.schema.json)"] --> E
    E["adapters/terraform/adapter.py<br/>→ terraform/spike/{main,terraform,providers}.tf<br/>(the only substrate-specific code)"] --> F
    F["terraform plan<br/>(real AWS, via the rotated runner key — D-039/D-047)"] --> G
    G["adapters/terraform/policy/checkov_adapter.py<br/>→ PolicyCheckResult (JSON list)<br/>(normalized, engine-agnostic)"] --> H
    H["acdl_platform/confidence_signal.py<br/>→ { score, band, perInput, reasonCodes }<br/>(6 inputs: policy, validation, freshness, source, history, nfrs)"] --> I
    I["acdl_platform/outbox_writer.py<br/>→ DynamoDB outbox (acdl-outbox)<br/>(hash-chained evidence event)"] --> J
    J["acdl-evidence timeline<br/>(acdl-evidence repo, raw-file served)"]
```

The platform validates the architecture's claim that the **stack
commitments do not require a polyglot mess**: the adapter is the only
substrate-specific code. `modules/`, `schemas/`, `contracts/`,
`acdl_platform/confidence_signal.py`, `acdl_platform/contract_resolver.py`,
and `acdl_platform/outbox_writer.py` are all substrate-agnostic (no
`aws_s3_bucket` / `aws_` Terraform terms).

## How to run

### Prerequisites

> These prerequisites are for running the **platform repo** locally. A
> consumer does not need any of these — see the
> [Consumer guide](docs/CONSUMER_GUIDE.md) for the consumer happy path.

- AWS account + the rotated runner key in `.env.secrets` (see
  `scripts/rotate_spike_key.sh`; the bootstrap root key was deactivated
  per D-034 closure).
- `terraform` (pin `1.9.*`), `checkov` (pin `>=3.2,<4`), `python3` + `boto3`
  + `jsonschema`.

### Run the platform pipeline end-to-end

```bash
# 1. Bootstrap the AWS state backend + runner IAM user (one-time, idempotent)
#    (requires the bootstrap root key in env — now deactivated; skip if
#    the state bucket + acdl-spike-runner already exist)
ACDL_BOOTSTRAP_AWS_ACCESS_KEY_ID=... ACDL_BOOTSTRAP_AWS_SECRET_ACCESS_KEY=... \
  python3 terraform/bootstrap/create_state_backend.py
ACDL_BOOTSTRAP_AWS_ACCESS_KEY_ID=... ACDL_BOOTSTRAP_AWS_SECRET_ACCESS_KEY=... \
  python3 terraform/bootstrap/create_iam_user.py  # prints the initial key

# 2. Rotate the runner key (writes .env.secrets, gitignored)
ACDL_BOOTSTRAP_AWS_ACCESS_KEY_ID=... ACDL_BOOTSTRAP_AWS_SECRET_ACCESS_KEY=... \
  bash scripts/rotate_spike_key.sh

# 3. Run the full platform pipeline (contract -> stack -> adapter -> plan ->
#    Checkov -> confidence -> outbox). Output is streamed to stdout by default.
bash scripts/run_platform.sh contracts/static-asset.yaml
# Expected: "=== PLATFORM E2E OK ==="

# Or plan-only (contract -> stack -> adapter -> terraform plan; no Checkov/outbox):
bash scripts/run_platform.sh --plan-only contracts/static-asset.yaml

# Add --quiet to suppress streaming (output to log files only):
bash scripts/run_platform.sh --quiet contracts/static-asset.yaml
```

### Test the platform (offline, no AWS required)

```bash
# Install test dependencies
pip install -r requirements-test.txt

# Run the test suite (all offline — uses moto for DynamoDB mocking)
python3 -m pytest tests/ -v

# Run the platform in check-only mode (offline — no AWS, no Checkov, no outbox)
# Uses the default sample contract (contracts/static-asset.yaml)
bash scripts/run_platform.sh --check-only
# Expected: "=== PLATFORM CHECK OK ==="

# Reproduce the full CI pipeline locally (lint → test → check-only)
bash scripts/run_ci.sh
# Expected: "=== CI PIPELINE OK ==="
```

### CI/CD pipelines

The CI/CD pipeline is defined by a **central pipeline contract** — a
declarative YAML instance (`pipelines/ci.yaml`) validated against a JSON
Schema (`schemas/pipeline.schema.json`). Both forge workflows implement
the same contract:

- `.gitea/workflows/ci.yml` — Gitea Actions (dev environment)
- `.github/workflows/ci.yml` — GitHub Actions (production)

Both workflow files are **byte-identical** — the only difference is the
forge runtime. Both run three stages: **lint** (py_compile), **test**
(pytest), and **check-only** (`run_platform.sh --check-only`). Both
trigger on push to `main` and on pull requests. A test
(`tests/test_pipeline_contract.py`) validates that both workflows conform
to the contract.

`scripts/run_ci.sh` mirrors the CI pipeline locally — running the same
three stages in sequence. This makes the pipeline fully reproducible from
the shell, not just in CI:

```bash
bash scripts/run_ci.sh           # run all 3 stages (lint, test, check-only)
bash scripts/run_ci.sh --quiet    # suppress per-stage banners
```

### Reusable deploy workflow

The deployment pipeline is defined by a **central deployment pipeline
contract** (`pipelines/deploy.yaml`, validated against
`schemas/deploy-pipeline.schema.json`) and exposed to consumer repos as a
**reusable workflow**:

- `.gitea/workflows/deploy.yml` — Gitea Actions (dev environment)
- `.github/workflows/deploy.yml` — GitHub Actions (production)

Both files are **byte-identical** and implement the same stages as
`pipelines/deploy.yaml` (validate-contract → resolve-stack →
terraform-plan → checkov → confidence → apply). A consumer repo invokes
the reusable workflow via a **versioned tag** (floating MAJOR + MINOR, e.g.
`acdl/.gitea/workflows/deploy.yml@v1.4`). The workflow checks out the
consumer repo, then checks out the ACDL platform repo into the runner
workspace, and runs `scripts/run_platform.sh` against the consumer's
contract — the consumer never clones the platform repo or invokes its
scripts locally. See the [Consumer guide](docs/CONSUMER_GUIDE.md) for the
end-to-end happy path.

### Output streaming (run_platform.sh)

`scripts/run_platform.sh` streams output by default so the user can see
what the platform is doing:

- **`--check-only`**: streams the emitted Terraform file content to stdout
- **`--plan-only`** and **full mode**: streams `terraform init`, `terraform
  validate`, and `terraform plan` output via `tee` (visible and logged)
- **Full mode**: prints Checkov compliance results and each
  PolicyCheckResult record with severity, rule ID, and pass/fail status

A `--quiet` flag suppresses streaming (output to log files only) for
backwards-compatible log-only mode.

## Consumer guide

A step-by-step guide for a consumer to create their pipeline and define a
contract that deploys any ACDL module to AWS is at
[`docs/CONSUMER_GUIDE.md`](docs/CONSUMER_GUIDE.md). The guide is generic
across all L2 modules; `static-asset` is the worked example.

## Repository layout

| Path | Purpose | Status |
|------|---------|--------|
| `acdl_platform/` | Platform code: contract resolver, confidence signal, outbox writer, separation of duties, HITL/ledger designs | active |
| `schemas/` | JSON Schemas: stack, contract, PolicyCheckResult, pipeline contract, deploy pipeline contract (draft 2020-12) | active |
| `pipelines/` | Central pipeline contracts: `ci.yaml` (CI), `deploy.yaml` (deployment) | active |
| `adapters/` | Substrate adapters — Terraform adapter (the only substrate-specific code per §12) + Checkov policy adapter | active |
| `terraform/` | State backend (S3 + DynamoDB) + platform TF (`terraform/spike/`) + bootstrap scripts (`terraform/bootstrap/`) | active |
| `modules/` | L1/L2 modules + `registry.json`. L1: s3, vpc, ecs-cluster, ecs-service, iam-role, alb, ecr. L2: microservice, static-asset | active |
| `contracts/` | Sample consumer contracts (e.g. `static-asset.yaml`) | active |
| `scripts/` | Platform run script (`run_platform.sh` with `--check-only`/`--plan-only`/`--quiet`), CI pipeline script (`run_ci.sh`), key rotation | active |
| `tests/` | Pytest suite (all offline — adapter, confidence signal, checkov adapter, outbox writer, pipeline contract, contract resolver, streaming) | active |
| `.gitea/workflows/` | Gitea Actions workflows: `ci.yml` (CI), `deploy.yml` (reusable deploy, invoked by consumer repos) | active |
| `.github/workflows/` | GitHub Actions workflows: `ci.yml` (CI), `deploy.yml` (reusable deploy, invoked by consumer repos) | active |
| `.ciagent/` | CIAgent metadata (config, project, architecture, requirements, roadmap, personas, plans, research, verify, review, audit) | active |
| `docs/` | Upstream vision + architecture sources (`vision.md`, `architecture.md`) + consumer guide | active |

## Environments

| Environment | Autonomy | Gate | Status |
|---|---|---|---|
| dev | Full autonomy (no HITL) | Confidence ≥ 0.50 | v1.1 (`plan`); v1.2 (`apply`) |
| qa | Held for attestation | QA HITL + confidence ≥ 0.75 | v1.3+ |
| prod | Held for attestation | SRE HITL + confidence ≥ 0.90 | v1.3+ |
| dr | Held for attestation | SRE HITL + confidence ≥ 0.95 + dr-drill | v1.3+ |

**Staging does not exist** (Path A locked).

## Credentials & zero-trust

### Default — zero-trust OIDC + attribute-based authorization (the locked target)

Consumer GitHub/Gitea repos are **zero-trust**: they hold **no long-lived
AWS keys** and no static credentials in repo secrets.

- **Authentication** is **OIDC federation** between the forge (GitHub or
  Gitea Actions) and AWS. Each job mints a short-lived STS token; no
  credential is ever stored in the consumer repo or in a forge secret.
- **Authorization** is **attribute-based (ABAC)**, not role-based (RBAC).
  AWS IAM roles and session policies are scoped by two attribute classes:
  - **Repository identity** — the forge claim (e.g.
    `repo:org/consumer-repo:ref:refs/heads/main`) binds the role's trust
    policy to the exact consumer repo + branch that invoked the workflow.
  - **Resource-creation attributes** — every resource the pipeline creates
    is tagged with `acdl:owner=<consumer-repo>` and
    `acdl:contract=<contract-id>`. The session policy grants
    view/update/delete **only on resources whose tags match the calling
    repo**.

  The effect: a consumer's pipeline can only view and update the resources
  it created. Blast radius is contained to that consumer's own stack
  instances — one consumer can never touch another consumer's resources,
  and the consumer cannot escape its own scope.

### Override — static key + managed daily rotation

Where OIDC is not yet available (Gitea Actions OIDC is blocked on
[go-gitea/gitea#36988](https://github.com/go-gitea/gitea/pull/36988), still
open as of 2026-07-21), a static AWS key **may** be used as a documented
override:

- The key is stored in **GitHub Secrets** (consumer repo) for forge runs,
  or in **`.env.secrets`** (gitignored, chmod 600) for local testing.
- The key is rotated by a **platform-managed scheduled pipeline on a daily
  cadence** — rotation is not the consumer's burden in the forge path.
- **When `.env.secrets` is used locally**, rotating the key **out of band is
  the consumer's responsibility**. The platform guarantees daily rotation
  for forge runs; it does not guarantee rotation for locally-held copies.
  The consumer must rotate a local key via `scripts/rotate_spike_key.sh`
  (or equivalent) on their own cadence.

The current per-run-rotated-key flow (waivers D-039 / D-047) is the
present-day instance of this override. The zero-trust OIDC + ABAC model
above is the locked target; the override is time-boxed until the Gitea
OIDC provider merges. `§12.5` forbids long-lived credentials; both the
target and the override satisfy its *intent* (no *persistently* long-lived
key — the forge key's useful lifetime is one workflow run, and the
override is rotated at least daily).