acdl/README.md

# ACDL — Agentic Cloud Delivery Platform

Consumers declare intent; the platform delivers safe production deployment
through an agentic stack — automatically, safely, and with a complete audit
trail. A merged change progresses through lower environments end-to-end
without a platform engineer joining a thread; a non-technical consumer ships
a production deployment by declaring intent, without authoring a workflow,
a configuration file, or an infrastructure module.

- **Consumer guide:** [`docs/consumer-guide.md`](docs/consumer-guide.md)
- **Modules:** [`docs/modules/`](docs/modules/)
- **Contracts:** [`docs/contracts/`](docs/contracts/)
- **Pipeline:** [`docs/pipeline/`](docs/pipeline/)
- **Versioning:** [`docs/pipeline/versioning.md`](docs/pipeline/versioning.md)
- **Environments:** [`docs/environments/`](docs/environments/)
- **Architecture:** [`docs/architecture.md`](docs/architecture.md)
- **Vision:** [`docs/vision.md`](docs/vision.md)

## Repository roles

There are two kinds of repository in the ACDL model:

- **Platform repo (this one).** This is the **source code of the platform**.
  It owns `modules/`, `adapters/`, `core/`, `schemas/`, `pipelines/`,
  `scripts/`, and the reusable workflow files. Platform engineers work here.
  A **consumer never clones it.**
- **Consumer repo (yours).** A consumer repo contains only:
  1. **Its application code** — the service or site being deployed.
  2. **One or more contracts** — small YAML files at `.acdl/contract.yaml`
     that reference the central pipeline, name a module, select an
     environment, and supply module-specific inputs.
  3. **One or more CI definitions** — thin `.github/workflows/*.yml` files
     that `uses:` the central reusable deploy workflow, pointing at the
     appropriate environment + contract.

  The consumer does not write infrastructure modules, workflow YAML beyond
  the thin `uses:` wrapper, or adapter code — they write a contract YAML
  file and the platform does the rest.

The rest of this README describes the **platform repo** (how the platform
works, how to run it locally, how it's laid out). If you are a consumer,
jump to the [Consumer guide](docs/consumer-guide.md).

## Features

A referenceable list of what the platform provides today, for consumers and
platform engineers alike:

- **Contract-driven deploys** — a consumer writes a YAML contract; the
  platform resolves it to a stack, compiles it, and deploys it.
- **Reusable versioned deploy workflow** — consumer repos `uses:` a
  versioned central workflow; no platform code is cloned by the consumer.
- **Module catalog** — primitives (single resources) and modules (patterns
  of primitives) with self-documented inputs/outputs. See
  [docs/modules/](docs/modules/).
- **Zero-trust credentials** — OIDC federation + attribute-based
  authorization (ABAC) by default; no long-lived keys in consumer repos.
- **Security + policy checks** — a security-check stage and a policy-check
  stage run before any infrastructure is created.
- **Confidence signal** — a computed, explainable score gates promotion.
- **Evidence outbox** — every deployment writes a hash-chained evidence
  event to an audit outbox.
- **Shell reproducibility** — `scripts/run_ci.sh` mirrors the CI pipeline
  locally; `scripts/run_platform.sh --check-only` runs offline.
- **Platform-managed environments** — consumers provide no AWS account,
  VPC, subnet, or state bucket; the platform manages environments. See
  [docs/environments/](docs/environments/).
- **Central pipeline contract** — a declarative YAML instance is the single
  source of truth for both the CI and deploy workflows.

## Roadmap

Planned future features (no dates; tracked in the internal roadmap):

- **Dynamic module creation from a contract** — an agentic flow where a
  consumer creates a module directly from the contract file (the
  "composition" mechanism, redesigned).
- **Compliance milestone** — per-module compliance extension points (GDPR,
  SOX, SOC2, HIPAA, DORA) wired into the pipeline.
- **Additional substrate adapters** — beyond the Terraform adapter.
- **Environment self-service** — a consumer-facing flow to request and
  provision a new platform-managed environment (today it is a platform-team
  action).
- **HITL gates for qa / prod / dr** — human attestation + higher confidence
  thresholds for higher environments.
- **OIDC for all platform runners** — zero-trust credentials everywhere.

## How the platform works

The platform is **four layers + six cross-cutting concerns**, bound by the
vision's "Two Consumer Surfaces, One Platform" tenet: consumers declare
intent via a contract; the platform delivers the deployment through the
same contract schema, the same policy envelope, and the same evidence
stream.

Consumers have their own repos and consume ACDL by referencing `uses:` the
central pipeline definitions. A consumer declares a contract (module +
environment + inputs); the platform resolves it to a stack instance,
compiles it, runs security + policy checks, computes a confidence signal,
writes an evidence event to the audit outbox, and applies the
infrastructure.

### The platform flow (end-to-end)

```mermaid
flowchart TD
    A["consumer contract<br/>(uses + module + environment + inputs)"] --> B
    B["schema validation<br/>(contract schema)"] --> C
    C["resolve to Target Stack<br/>(contract resolver)"] --> D
    D["security checks<br/>(adapter)"] --> E
    E["infrastructure plan<br/>(adapter compiles the stack)"] --> F
    F["policy checks<br/>(adapter -&gt; PolicyCheckResult records)"] --> G
    G["confidence signal<br/>(6 inputs: policy, validation,<br/>freshness, source, history, NFRs)"] --> H
    H["evidence event<br/>(hash-chained, to the audit outbox)"] --> I
    I["infrastructure apply<br/>(dev only, autonomous)"]
```

The platform validates the architecture's claim that the **stack
commitments do not require a polyglot mess**: the adapter is the only
substrate-specific code. `modules/`, `schemas/`, `contracts/`,
`core/confidence_signal.py`, `core/contract_resolver.py`, and
`core/outbox_writer.py` are all substrate-agnostic (no `aws_s3_bucket` /
`aws_` infrastructure terms).

## How to run

### Prerequisites

> These prerequisites are for running the **platform repo** locally. A
> consumer does not need any of these — see the
> [Consumer guide](docs/consumer-guide.md) for the consumer happy path.

- A platform-managed environment (see [docs/environments/](docs/environments/)).
  For local testing, `core/environments/dev.json` is provided as the sample.
- AWS credentials for the dev environment (in `.env.secrets`, gitignored;
  see [Credentials & zero-trust](#credentials--zero-trust)).
- `terraform` (pin `1.9.*`), `checkov` (pin `>=3.2,<4`), `python3` + `boto3`
  + `jsonschema`.

### Run the platform pipeline end-to-end

```bash
# 1. Bootstrap the AWS state backend + runner IAM user (one-time, idempotent)
#    (requires the bootstrap root key in env — skip if the state bucket +
#    acdl-spike-runner already exist)
ACDL_BOOTSTRAP_AWS_ACCESS_KEY_ID=... ACDL_BOOTSTRAP_AWS_SECRET_ACCESS_KEY=... \
  python3 terraform/bootstrap/create_state_backend.py
ACDL_BOOTSTRAP_AWS_ACCESS_KEY_ID=... ACDL_BOOTSTRAP_AWS_SECRET_ACCESS_KEY=... \
  python3 terraform/bootstrap/create_iam_user.py  # prints the initial key

# 2. Rotate the runner key (writes .env.secrets, gitignored)
ACDL_BOOTSTRAP_AWS_ACCESS_KEY_ID=... ACDL_BOOTSTRAP_AWS_SECRET_ACCESS_KEY=... \
  bash scripts/rotate_spike_key.sh

# 3. Run the full platform pipeline (contract -> environment check -> stack ->
#    adapter -> security checks -> infrastructure plan -> policy checks ->
#    confidence -> evidence event -> apply). Output is streamed to stdout.
bash scripts/run_platform.sh contracts/static-asset.yaml
# Expected: "=== PLATFORM E2E OK ==="

# Or plan-only (contract -> stack -> adapter -> infrastructure plan; no
# policy checks / outbox):
bash scripts/run_platform.sh --plan-only contracts/static-asset.yaml

# Add --quiet to suppress streaming (output to log files only):
bash scripts/run_platform.sh --quiet contracts/static-asset.yaml
```

### Test the platform (offline, no AWS required)

```bash
# Install test dependencies
pip install -r requirements-test.txt

# Run the test suite (all offline — uses moto for DynamoDB mocking)
python3 -m pytest tests/ -v

# Run the platform in check-only mode (offline — no AWS, no policy checks,
# no outbox). Uses the default sample contract (contracts/static-asset.yaml)
# and the sample dev environment (core/environments/dev.json).
bash scripts/run_platform.sh --check-only
# Expected: "=== PLATFORM CHECK OK ==="

# Reproduce the full CI pipeline locally (lint -> test -> check-only)
bash scripts/run_ci.sh
# Expected: "=== CI PIPELINE OK ==="
```

### CI/CD pipelines

The CI/CD pipeline is defined by a **central pipeline contract** — a
declarative YAML instance (`pipelines/ci.yaml`) validated against a JSON
Schema (`schemas/pipeline.schema.json`). Both platform-runner workflows
implement the same contract:

- `.github/workflows/ci.yml` — GitHub Actions (production)

Both run three stages: **lint** (py_compile), **test** (pytest), and
**check-only** (`run_platform.sh --check-only`). Both trigger on push to
`main` and on pull requests. A test (`tests/test_pipeline_contract.py`)
validates that the workflow conforms to the contract.

`scripts/run_ci.sh` mirrors the CI pipeline locally — running the same
three stages in sequence. This makes the pipeline fully reproducible from
the shell, not just in CI:

```bash
bash scripts/run_ci.sh           # run all 3 stages (lint, test, check-only)
bash scripts/run_ci.sh --quiet    # suppress per-stage banners
```

### Reusable deploy workflow

The deployment pipeline is defined by a **central deployment pipeline
contract** (`pipelines/deploy.yaml`, validated against
`schemas/deploy-pipeline.schema.json`) and exposed to consumer repos as a
**reusable workflow**:

- `.github/workflows/deploy.yml` — GitHub Actions (production)

The workflow implements the same stages as `pipelines/deploy.yaml`
(validate-contract → resolve-stack → security checks → infrastructure plan
→ policy checks → confidence → evidence event → apply). A consumer repo
invokes the reusable workflow via a **versioned tag** (floating MAJOR +
MINOR, e.g. `acdl/.github/workflows/deploy.yml@v1.4`). The workflow checks
out the consumer repo, then checks out the ACDL platform repo into the
runner workspace, and runs `scripts/run_platform.sh` against the consumer's
contract — the consumer never clones the platform repo or invokes its
scripts locally. See the [Consumer guide](docs/consumer-guide.md) for the
end-to-end happy path.

### Output streaming (run_platform.sh)

`scripts/run_platform.sh` streams output by default so the user can see
what the platform is doing:

- **`--check-only`**: streams the emitted infrastructure file content to
  stdout.
- **`--plan-only`** and **full mode**: streams the infrastructure plan
  output via `tee` (visible and logged).
- **Full mode**: prints policy-check results and each `PolicyCheckResult`
  record with severity, rule ID, and pass/fail status.

A `--quiet` flag suppresses streaming (output to log files only) for
backwards-compatible log-only mode.

## Consumer guide

A step-by-step guide for a consumer to create their pipeline and define a
contract that deploys any ACDL module to AWS is at
[`docs/consumer-guide.md`](docs/consumer-guide.md). The guide is generic
across all modules; `static-asset` is the worked example.

## Repository layout

| Path | Purpose | Status |
|------|---------|--------|
| `core/` | Platform code: contract resolver, confidence signal, outbox writer, environment check, environments, separation of duties, HITL/ledger designs | active |
| `schemas/` | JSON Schemas: stack, contract, PolicyCheckResult, pipeline contract, deploy pipeline contract (draft 2020-12) | active |
| `pipelines/` | Central pipeline contracts: `ci.yaml` (CI), `deploy.yaml` (deployment) | active |
| `adapters/` | Substrate adapters — the substrate adapter (the only substrate-specific code per §12) + the policy adapter | active |
| `terraform/` | State backend (S3 + DynamoDB) + platform TF (`terraform/spike/`) + bootstrap scripts (`terraform/bootstrap/`) | active |
| `modules/` | Primitives + modules + `registry.json`. Primitives: s3, vpc, ecs-cluster, ecs-service, iam-role, alb, ecr. Modules: microservice, static-asset | active |
| `contracts/` | Sample consumer contracts (e.g. `static-asset.yaml`) | active |
| `scripts/` | Platform run script (`run_platform.sh` with `--check-only`/`--plan-only`/`--quiet`), CI pipeline script (`run_ci.sh`), key rotation | active |
| `tests/` | Pytest suite (all offline — adapter, confidence signal, policy adapter, outbox writer, pipeline contract, contract resolver, streaming, environment check) | active |
| `.github/workflows/` | GitHub Actions workflows: `ci.yml` (CI), `deploy.yml` (reusable deploy, invoked by consumer repos) | active |
| `docs/` | GitHub Pages documentation site: consumer guide, modules, contracts, pipeline, versioning, environments, architecture, vision | active |

## Credentials & zero-trust

### Default — zero-trust OIDC + attribute-based authorization

Consumer repos are **zero-trust**: they hold **no long-lived AWS keys** and
no static credentials in repo secrets.

- **Authentication** is **OIDC federation** between the platform runners
  (GitHub Actions) and AWS. Each job mints a short-lived STS token; no
  credential is ever stored in the consumer repo or in a runner secret.
- **Authorization** is **attribute-based (ABAC)**, not role-based (RBAC).
  AWS IAM roles and session policies are scoped by two attribute classes:
  - **Repository identity** — the runner claim (e.g.
    `repo:org/consumer-repo:ref:refs/heads/main`) binds the role's trust
    policy to the exact consumer repo + branch that invoked the workflow.
  - **Resource-creation attributes** — every resource the pipeline creates
    is tagged with `acdl:owner=<consumer-repo>` and
    `acdl:contract=<contract-id>`. The session policy grants
    view/update/delete **only on resources whose tags match the calling
    repo**.

  The effect: a consumer's pipeline can only view and update the resources
  it created. Blast radius is contained to that consumer's own stack
  instances — one consumer can never touch another consumer's resources,
  and the consumer cannot escape its own scope.

### Alternative — static AWS key

Where OIDC is not yet available, a static AWS key **may** be used as a
documented alternative:

- The key is stored in **GitHub Secrets** (consumer repo) for platform-runner
  runs, or in **`.env.secrets`** (gitignored, chmod 600) for local testing.
- The platform rotates platform-runner keys on a **daily cadence** —
  rotation is not the consumer's burden in the platform-runner path.
- **When `.env.secrets` is used locally**, rotating the key **out of band is
  the consumer's responsibility**. The platform guarantees daily rotation
  for platform-runner runs; it does not guarantee rotation for
  locally-held copies. The consumer must rotate a local key via
  `scripts/rotate_spike_key.sh` (or equivalent) on their own cadence.

No long-lived credential is permitted persistently — the platform-runner
key's useful lifetime is one workflow run, and the local alternative is
rotated at least daily (platform-runner) or out of band (local).