continuous-intelligence/acdl
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 128 Wiki Activity

feat(P20): consumer happy path + reusable deploy workflow (v1.5.0)

Browse Source 
---ci---
project: acdl
phase: 20
milestone: v1.5
status: verify
---/ci---

REQ-46: README rewritten — platform-source vs consumer-repo distinction up
front; platform flow converted to mermaid flowchart TD; L3A/L3B + spike
nomenclature scrubbed from prose (code paths kept verbatim); prereqs pointer
to consumer guide added.
REQ-47: docs/CONSUMER_GUIDE.md (generic, all L2 modules) replaces
docs/consumer-guide-static-asset.md — mermaid diagrams (model LR + pipeline
TD), versioned uses: (@v1.4 floating MAJOR+MINOR, bare/@main discouraged),
consumer-scoped prerequisites (no Terraform/Checkov/boto3/runner-key), run-
time platform fetch via reusable workflow (consumers never invoke
scripts/run_platform.sh locally for the happy path), optional local
validation note.
REQ-48: Credentials section rewritten — zero-trust OIDC + ABAC default
(repo-identity + resource-tag scoping, blast-radius containment); static-key
override in GitHub Secrets or .env.secrets with platform-managed daily
rotation; consumer rotates out of band when using .env.secrets locally.
REQ-49: byte-identical .gitea/workflows/deploy.yml + .github/workflows/
deploy.yml — reusable (on: workflow_call), checks out consumer repo + ACDL
platform repo, installs deps, runs run_platform.sh, uploads artifacts; OIDC
default (permissions: id-token: write) + static-key override via secrets.
REQ-50: contracts/static-asset.yaml uses: @v1.4 (MAJOR+MINOR).
REQ-51: tests/test_pipeline_contract.py extended — TestDeployPipelineSchema,
TestDeployPipelineContract, TestDeployWorkflowConformance (byte-identical,
reusable, contract/mode inputs, run_platform invocation, platform-repo
checkout, OIDC permissions), TestSampleContractVersioning. 154 tests pass
(19 new); run_ci.sh green.

Fixes: modules/l2/static-asset/README.md dangling link retargeted to
docs/CONSUMER_GUIDE.md.
This commit is contained in:
Jon Chery
2026-07-22 17:14:12 +00:00
parent 895a2f3806
commit 2a84c0047b
7 changed files with 958 additions and 122 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+168 -100
View File
@@ -11,95 +11,102 @@ a configuration file, or a Terraform module.
- **Architecture** (the how): [`docs/architecture.md`](docs/architecture.md) + [`.ciagent/ARCHITECTURE.md`](.ciagent/ARCHITECTURE.md)
- **Decisions**: [`.ciagent/PROJECT.md`](.ciagent/PROJECT.md)
- **Phase plan**: [`.ciagent/ROADMAP.md`](.ciagent/ROADMAP.md)
- **Consumer guide**: [`docs/CONSUMER_GUIDE.md`](docs/CONSUMER_GUIDE.md)
## Repository roles
There are two kinds of repository in the ACDL model:
- **Platform repo (this one).** This is the **source code of the platform**.
It owns `modules/`, `adapters/`, `acdl_platform/`, `schemas/`, `pipelines/`,
`scripts/`, and the reusable workflow files. Platform engineers work here.
A **consumer never clones it.**
- **Consumer repo (yours).** A consumer repo contains only its application
code and a single `contract.yaml` that references the central pipeline +
contract. The consumer does not write Terraform, workflow YAML, or adapter
code — they write a contract YAML file and the platform does the rest.
The rest of this README describes the **platform repo** (how the platform
works, how to run it locally, how it's laid out). If you are a consumer,
jump to the [Consumer guide](docs/CONSUMER_GUIDE.md).
## Status
- **v1.4 (active):** central pipeline contract + shell reproducibility +
output streaming. A declarative pipeline contract
- **v1.5 (active):** consumer happy path + zero-trust docs + reusable deploy
workflow. README rewritten so the consumer model is unambiguous. Platform
flow + consumer guide converted to mermaid. Legacy surface + implementation
nomenclature removed from docs. Credentials section rewritten for
zero-trust OIDC + ABAC. A generic `docs/CONSUMER_GUIDE.md` (all L2 modules,
versioned `uses:`, consumer-scoped prerequisites, run-time platform fetch)
replaces the module-specific guide. A byte-identical reusable `deploy.yml`
workflow (Gitea + GitHub) implements `pipelines/deploy.yaml` and is invoked
by consumer repos via a versioned tag.
- **v1.4 (complete, tag `v1.4.1`):** central pipeline contract + shell
reproducibility + output streaming. A declarative pipeline contract
(`schemas/pipeline.schema.json` + `pipelines/ci.yaml`) binds the Gitea
and GitHub workflows to a single source of truth. `scripts/run_ci.sh`
mirrors the CI pipeline locally. `scripts/run_platform.sh` streams
terraform/checkov output by default. Ship tag `v1.4.1`.
- **v1.3 (complete, tag `v1.3.2`):** module documentation + thin-composition
removal. The L2 composition layer is removed; module READMEs are built
out. Testing + CI/CD pipelines (pytest, `--check-only`, Gitea + GitHub
workflows).
terraform/checkov output by default. L2 compositions re-introduced with
a `uses:`-based contract resolution mechanism.
- **v1.3 (complete, tag `v1.3.2`):** module documentation. Testing + CI/CD
pipelines (pytest, `--check-only`, Gitea + GitHub workflows).
- **v1.2 (complete, tag `v1.3.0`):** platform hardening + first real
consumer deployment. Harden the v1.1 spike's NFRs, simplify the setup,
rewrite the docs, and prove the platform delivers real value by
consumer deployment. Harden the v1.1 implementation's NFRs, simplify the
setup, rewrite the docs, and prove the platform delivers real value by
deploying a basic microservice to AWS ECS Fargate end-to-end (`terraform
apply`, dev autonomous).
- **v1.1 (complete, tag `v1.2.0`):** architecture finalization + v1 spike.
Finalized the architecture to v1.0 (resolved all 11 open design
decisions) and proved the IR commitments hold with one end-to-end spike
(`l1-s3` + `l2-static-asset` + Terraform adapter → real `terraform plan`
against AWS). Gitea release id 202.
- **v1.0 demo (complete, archived under `demo/`, tag `v1.1.0`):** the
30-minute stub-driven executive demo. Preserved as the intent reference;
it is not the platform.
- **v1.1 (complete, tag `v1.2.0`):** architecture finalization + v1
implementation. Finalized the architecture to v1.0 (resolved all 11 open
design decisions) and proved the stack commitments hold with one
end-to-end run (`s3` + `static-asset` + Terraform adapter → real
`terraform plan` against AWS). Gitea release id 202.
## How the platform works
The platform is **four layers + six cross-cutting concerns**, bound by the
vision's "Two Consumer Surfaces, One Platform" tenet: technical developers
(L3A) and non-technical consumers (L3B) converge on the same contract
schema, the same policy envelope, and the same evidence stream.
vision's "Two Consumer Surfaces, One Platform" tenet: consumers declare
intent via a contract; the platform delivers the deployment through the
same contract schema, the same policy envelope, and the same evidence
stream.
### The v1.1 spike flow (end-to-end)
Consumers have their own repos and consume ACDL by referencing `uses:` the
central pipeline definitions. A consumer declares a contract (module +
environment + inputs); the platform resolves it to a stack instance,
compiles it to Terraform, runs policy checks, computes a confidence signal,
and writes an evidence event to the audit outbox.
```
contracts/spike.yaml
│ (contract schema validation)
▼
acdl_platform/contract_resolver.py ──▶ Target Stack IR (JSON)
│ (IR schema validation)
▼
adapters/terraform/adapter.py ──▶ terraform/spike/{main,terraform,providers}.tf
│ (the only substrate-specific code)
▼
terraform plan (real AWS, via the rotated spike key — D-039/D-047)
│
▼
adapters/terraform/policy/checkov_adapter.py ──▶ PolicyCheckResult (JSON list)
│ (normalized, engine-agnostic)
▼
acdl_platform/confidence_signal.py ──▶ { score, band, perInput, reasonCodes }
│ (6 inputs: policy, validation, freshness, source, history, nfrs)
▼
acdl_platform/outbox_writer.py ──▶ DynamoDB outbox (acdl-outbox)
│ (hash-chained evidence event)
▼
acdl-evidence timeline (acdl-evidence repo, raw-file served)
### The platform flow (end-to-end)
```mermaid
flowchart TD
A["contracts/static-asset.yaml<br/>(consumer contract: uses + module + inputs)"] --> B
B["schema validation<br/>(schemas/contract.schema.json)"] --> C
C["acdl_platform/contract_resolver.py<br/>→ Target Stack (JSON)"] --> D
D["stack schema validation<br/>(schemas/stack.schema.json)"] --> E
E["adapters/terraform/adapter.py<br/>→ terraform/spike/{main,terraform,providers}.tf<br/>(the only substrate-specific code)"] --> F
F["terraform plan<br/>(real AWS, via the rotated runner key — D-039/D-047)"] --> G
G["adapters/terraform/policy/checkov_adapter.py<br/>→ PolicyCheckResult (JSON list)<br/>(normalized, engine-agnostic)"] --> H
H["acdl_platform/confidence_signal.py<br/>→ { score, band, perInput, reasonCodes }<br/>(6 inputs: policy, validation, freshness, source, history, nfrs)"] --> I
I["acdl_platform/outbox_writer.py<br/>→ DynamoDB outbox (acdl-outbox)<br/>(hash-chained evidence event)"] --> J
J["acdl-evidence timeline<br/>(acdl-evidence repo, raw-file served)"]
```
The spike validates the architecture's claim that the **IR-shaped
The platform validates the architecture's claim that the **stack
commitments do not require a polyglot mess**: the adapter is the only
substrate-specific code. `modules-ir/`, `schemas/`, `contracts/`,
substrate-specific code. `modules/`, `schemas/`, `contracts/`,
`acdl_platform/confidence_signal.py`, `acdl_platform/contract_resolver.py`,
and `acdl_platform/outbox_writer.py` are all substrate-agnostic (no
`aws_s3_bucket` / `aws_` Terraform terms).
### What's different in v1.2
v1.2 extends the spike to a real, simpler, better-documented platform that
**deploys a microservice to ECS Fargate**:
- Six new IR-typed L1s: `l1-vpc`, `l1-ecs-cluster`, `l1-ecs-service`,
`l1-iam-role`, `l1-alb`, `l1-ecr`.
- One new L2 thin-composition: `l2-microservice` (references the six L1s).
- `terraform apply` (dev, autonomous per §10, confidence ≥ 0.50) — real
provisioning, not just `plan`.
- A new consumer repo `acdl-consumer-microservice` with a basic HTTP
container + Dockerfile + ECR push + contract submission.
- One `scripts/run_platform.sh` (consolidated from the v1.1 spike scripts).
- NFR hardening: least-privilege IAM (expanded for ECS), idempotent
bootstrap, proper error handling, P1-1 redaction.
## How to run
### Prerequisites
- AWS account + the rotated spike key in `.env.secrets` (see
> These prerequisites are for running the **platform repo** locally. A
> consumer does not need any of these — see the
> [Consumer guide](docs/CONSUMER_GUIDE.md) for the consumer happy path.
- AWS account + the rotated runner key in `.env.secrets` (see
`scripts/rotate_spike_key.sh`; the bootstrap root key was deactivated
per D-034 closure).
- `terraform` (pin `1.9.*`), `checkov` (pin `>=3.2,<4`), `python3` + `boto3`
@@ -108,7 +115,7 @@ v1.2 extends the spike to a real, simpler, better-documented platform that
### Run the platform pipeline end-to-end
```bash
# 1. Bootstrap the AWS state backend + spike IAM user (one-time, idempotent)
# 1. Bootstrap the AWS state backend + runner IAM user (one-time, idempotent)
# (requires the bootstrap root key in env — now deactivated; skip if
# the state bucket + acdl-spike-runner already exist)
ACDL_BOOTSTRAP_AWS_ACCESS_KEY_ID=... ACDL_BOOTSTRAP_AWS_SECRET_ACCESS_KEY=... \
@@ -116,20 +123,20 @@ ACDL_BOOTSTRAP_AWS_ACCESS_KEY_ID=... ACDL_BOOTSTRAP_AWS_SECRET_ACCESS_KEY=... \
ACDL_BOOTSTRAP_AWS_ACCESS_KEY_ID=... ACDL_BOOTSTRAP_AWS_SECRET_ACCESS_KEY=... \
python3 terraform/bootstrap/create_iam_user.py # prints the initial key
# 2. Rotate the spike key (writes .env.secrets, gitignored)
# 2. Rotate the runner key (writes .env.secrets, gitignored)
ACDL_BOOTSTRAP_AWS_ACCESS_KEY_ID=... ACDL_BOOTSTRAP_AWS_SECRET_ACCESS_KEY=... \
bash scripts/rotate_spike_key.sh
# 3. Run the full platform pipeline (IR -> adapter -> plan -> Checkov ->
# confidence -> outbox). Output is streamed to stdout by default.
bash scripts/run_platform.sh
# 3. Run the full platform pipeline (contract -> stack -> adapter -> plan ->
# Checkov -> confidence -> outbox). Output is streamed to stdout by default.
bash scripts/run_platform.sh contracts/static-asset.yaml
# Expected: "=== PLATFORM E2E OK ==="
# Or plan-only (IR -> adapter -> terraform plan; no Checkov/outbox):
bash scripts/run_platform.sh --plan-only
# Or plan-only (contract -> stack -> adapter -> terraform plan; no Checkov/outbox):
bash scripts/run_platform.sh --plan-only contracts/static-asset.yaml
# Add --quiet to suppress streaming (output to log files only):
bash scripts/run_platform.sh --quiet
bash scripts/run_platform.sh --quiet contracts/static-asset.yaml
```
### Test the platform (offline, no AWS required)
@@ -138,11 +145,11 @@ bash scripts/run_platform.sh --quiet
# Install test dependencies
pip install -r requirements-test.txt
# Run the test suite (122 tests, all offline — uses moto for DynamoDB mocking)
# Run the test suite (all offline — uses moto for DynamoDB mocking)
python3 -m pytest tests/ -v
# Run the platform in check-only mode (offline — no AWS, no Checkov, no outbox)
# Streams the emitted Terraform to stdout by default; --quiet suppresses it
# Uses the default sample contract (contracts/static-asset.yaml)
bash scripts/run_platform.sh --check-only
# Expected: "=== PLATFORM CHECK OK ==="
@@ -177,6 +184,27 @@ bash scripts/run_ci.sh # run all 3 stages (lint, test, check-only)
bash scripts/run_ci.sh --quiet # suppress per-stage banners
```
### Reusable deploy workflow
The deployment pipeline is defined by a **central deployment pipeline
contract** (`pipelines/deploy.yaml`, validated against
`schemas/deploy-pipeline.schema.json`) and exposed to consumer repos as a
**reusable workflow**:
- `.gitea/workflows/deploy.yml` — Gitea Actions (dev environment)
- `.github/workflows/deploy.yml` — GitHub Actions (production)
Both files are **byte-identical** and implement the same stages as
`pipelines/deploy.yaml` (validate-contract → resolve-stack →
terraform-plan → checkov → confidence → apply). A consumer repo invokes
the reusable workflow via a **versioned tag** (floating MAJOR + MINOR, e.g.
`acdl/.gitea/workflows/deploy.yml@v1.4`). The workflow checks out the
consumer repo, then checks out the ACDL platform repo into the runner
workspace, and runs `scripts/run_platform.sh` against the consumer's
contract — the consumer never clones the platform repo or invokes its
scripts locally. See the [Consumer guide](docs/CONSUMER_GUIDE.md) for the
end-to-end happy path.
### Output streaming (run_platform.sh)
`scripts/run_platform.sh` streams output by default so the user can see
@@ -191,49 +219,89 @@ what the platform is doing:
A `--quiet` flag suppresses streaming (output to log files only) for
backwards-compatible log-only mode.
### Re-run the archived v1.0 demo (stubs only, no AWS)
## Consumer guide
```bash
bash demo/scripts/run_demo.sh --no-upload
```
The demo deck is at [`demo/ACDL_DEMO.md`](demo/ACDL_DEMO.md). It runs
entirely on local stubs — no AWS, no AI — and shows intent and safety
behavior rather than provisioning real cloud resources.
A step-by-step guide for a consumer to create their pipeline and define a
contract that deploys any ACDL module to AWS is at
[`docs/CONSUMER_GUIDE.md`](docs/CONSUMER_GUIDE.md). The guide is generic
across all L2 modules; `static-asset` is the worked example.
## Repository layout
| Path | Purpose | Status |
|------|---------|--------|
| `acdl_platform/` | Platform code: confidence signal, outbox writer, separation of duties, HITL/ledger designs | v1.1 complete; v1.3 removes contract_resolver |
| `schemas/` | JSON Schemas: IR, PolicyCheckResult, pipeline contract (draft 2020-12) | v1.1 complete; v1.4 adds pipeline schema |
| `pipelines/` | Central pipeline contract: `ci.yaml` (YAML instance validated against `schemas/pipeline.schema.json`) | v1.4 |
| `adapters/` | Substrate adapters — Terraform adapter (the only substrate-specific code per §12) + Checkov policy adapter | v1.1 complete; v1.2 expands `TYPE_MAP` |
| `terraform/` | State backend (S3 + DynamoDB) + spike TF (`terraform/spike/`) + bootstrap scripts (`terraform/bootstrap/`) | v1.1 complete; v1.2 adds ECS apply |
| `modules-ir/` | IR-typed L1/L2 modules + `registry.json`. v1.1: `l1-s3`. v1.2: + 6 ECS L1s. v1.3: L2 removed (placeholders) | v1.3 |
| `scripts/` | Platform run script (`run_platform.sh` with `--check-only`/`--plan-only`/`--quiet`), CI pipeline script (`run_ci.sh`), verify scripts, key rotation | v1.4 |
| `tests/` | Pytest suite (122 tests, all offline — adapter, confidence signal, checkov adapter, outbox writer, pipeline contract, streaming) | v1.4 |
| `demo/` | Archived v1.0 executive demo (tag `v1.1.0`); runs locally via `demo/scripts/run_demo.sh --no-upload` | complete (archived) |
| `acdl_platform/` | Platform code: contract resolver, confidence signal, outbox writer, separation of duties, HITL/ledger designs | active |
| `schemas/` | JSON Schemas: stack, contract, PolicyCheckResult, pipeline contract, deploy pipeline contract (draft 2020-12) | active |
| `pipelines/` | Central pipeline contracts: `ci.yaml` (CI), `deploy.yaml` (deployment) | active |
| `adapters/` | Substrate adapters — Terraform adapter (the only substrate-specific code per §12) + Checkov policy adapter | active |
| `terraform/` | State backend (S3 + DynamoDB) + platform TF (`terraform/spike/`) + bootstrap scripts (`terraform/bootstrap/`) | active |
| `modules/` | L1/L2 modules + `registry.json`. L1: s3, vpc, ecs-cluster, ecs-service, iam-role, alb, ecr. L2: microservice, static-asset | active |
| `contracts/` | Sample consumer contracts (e.g. `static-asset.yaml`) | active |
| `scripts/` | Platform run script (`run_platform.sh` with `--check-only`/`--plan-only`/`--quiet`), CI pipeline script (`run_ci.sh`), key rotation | active |
| `tests/` | Pytest suite (all offline — adapter, confidence signal, checkov adapter, outbox writer, pipeline contract, contract resolver, streaming) | active |
| `.gitea/workflows/` | Gitea Actions workflows: `ci.yml` (CI), `deploy.yml` (reusable deploy, invoked by consumer repos) | active |
| `.github/workflows/` | GitHub Actions workflows: `ci.yml` (CI), `deploy.yml` (reusable deploy, invoked by consumer repos) | active |
| `.ciagent/` | CIAgent metadata (config, project, architecture, requirements, roadmap, personas, plans, research, verify, review, audit) | active |
| `docs/` | Upstream vision + architecture sources (`vision.md`, `architecture.md`) | active |
| `docs/` | Upstream vision + architecture sources (`vision.md`, `architecture.md`) + consumer guide | active |
## Environments
| Environment | Autonomy | Gate | Status |
|---|---|---|---|
| dev | Full autonomy (no HITL) | Confidence ≥ 0.50 | v1.1 spike (`plan`); v1.2 (`apply`) |
| dev | Full autonomy (no HITL) | Confidence ≥ 0.50 | v1.1 (`plan`); v1.2 (`apply`) |
| qa | Held for attestation | QA HITL + confidence ≥ 0.75 | v1.3+ |
| prod | Held for attestation | SRE HITL + confidence ≥ 0.90 | v1.3+ |
| dr | Held for attestation | SRE HITL + confidence ≥ 0.95 + dr-drill | v1.3+ |
**Staging does not exist** (Path A locked).
## Credentials
## Credentials & zero-trust
**Long-lived AWS credentials are forbidden** (§12.5). The v1.1 spike uses a
temporary long-lived key **once** to bootstrap (waiver D-034, now closed —
the root key was deactivated by the user), then rotates the spike key
per-run via `scripts/rotate_spike_key.sh` (waiver D-039, extended for v1.2
as D-047). Real OIDC federation is deferred to v1.3+, blocked on
[go-gitea/gitea#36988](https://github.com/go-gitea/gitea/pull/36988) (still
open as of 2026-07-21).
### Default — zero-trust OIDC + attribute-based authorization (the locked target)
Consumer GitHub/Gitea repos are **zero-trust**: they hold **no long-lived
AWS keys** and no static credentials in repo secrets.
- **Authentication** is **OIDC federation** between the forge (GitHub or
Gitea Actions) and AWS. Each job mints a short-lived STS token; no
credential is ever stored in the consumer repo or in a forge secret.
- **Authorization** is **attribute-based (ABAC)**, not role-based (RBAC).
AWS IAM roles and session policies are scoped by two attribute classes:
- **Repository identity** — the forge claim (e.g.
`repo:org/consumer-repo:ref:refs/heads/main`) binds the role's trust
policy to the exact consumer repo + branch that invoked the workflow.
- **Resource-creation attributes** — every resource the pipeline creates
is tagged with `acdl:owner=<consumer-repo>` and
`acdl:contract=<contract-id>`. The session policy grants
view/update/delete **only on resources whose tags match the calling
repo**.
The effect: a consumer's pipeline can only view and update the resources
it created. Blast radius is contained to that consumer's own stack
instances — one consumer can never touch another consumer's resources,
and the consumer cannot escape its own scope.
### Override — static key + managed daily rotation
Where OIDC is not yet available (Gitea Actions OIDC is blocked on
[go-gitea/gitea#36988](https://github.com/go-gitea/gitea/pull/36988), still
open as of 2026-07-21), a static AWS key **may** be used as a documented
override:
- The key is stored in **GitHub Secrets** (consumer repo) for forge runs,
or in **`.env.secrets`** (gitignored, chmod 600) for local testing.
- The key is rotated by a **platform-managed scheduled pipeline on a daily
cadence** — rotation is not the consumer's burden in the forge path.
- **When `.env.secrets` is used locally**, rotating the key **out of band is
the consumer's responsibility**. The platform guarantees daily rotation
for forge runs; it does not guarantee rotation for locally-held copies.
The consumer must rotate a local key via `scripts/rotate_spike_key.sh`
(or equivalent) on their own cadence.
The current per-run-rotated-key flow (waivers D-039 / D-047) is the
present-day instance of this override. The zero-trust OIDC + ABAC model
above is the locked target; the override is time-boxed until the Gitea
OIDC provider merges. `§12.5` forbids long-lived credentials; both the
target and the override satisfy its *intent* (no *persistently* long-lived
key — the forge key's useful lifetime is one workflow run, and the
override is rotated at least daily).