Jon Chery
e050e65158
---ci--- project: acdl phase: 19 milestone: v1.4 status: execute --- Add declarative pipeline contract (schemas/pipeline.schema.json + pipelines/ci.yaml) as single source of truth for both Gitea Actions (dev) and GitHub Actions (production) workflows. Both workflow files are byte-identical and validated against the contract by 32 new tests. Add scripts/run_ci.sh for shell reproducibility — mirrors the CI pipeline locally (lint → test → check-only), exits 0 with 'CI PIPELINE OK'. Update scripts/run_platform.sh to stream output by default: terraform init/validate/plan via tee, Checkov compliance results with per-record severity/rule/pass-fail, and emitted Terraform in --check-only. New --quiet flag for log-only mode. Requirements: REQ-43 (central pipeline contract), REQ-44 (shell reproducibility), REQ-45 (output streaming). 122 tests pass (90 + 32).
239 lines
11 KiB
Markdown
239 lines
11 KiB
Markdown
# ACDL — Agentic Cloud Delivery Platform
|
|
|
|
Consumers declare intent; the platform delivers safe production deployment
|
|
through an agentic stack — automatically, safely, and with a complete audit
|
|
trail. A merged change progresses through lower environments end-to-end
|
|
without a platform engineer joining a thread; a non-technical consumer ships
|
|
a production deployment by declaring intent, without authoring a workflow,
|
|
a configuration file, or a Terraform module.
|
|
|
|
- **Vision** (the why): [`docs/vision.md`](docs/vision.md)
|
|
- **Architecture** (the how): [`docs/architecture.md`](docs/architecture.md) + [`.ciagent/ARCHITECTURE.md`](.ciagent/ARCHITECTURE.md)
|
|
- **Decisions**: [`.ciagent/PROJECT.md`](.ciagent/PROJECT.md)
|
|
- **Phase plan**: [`.ciagent/ROADMAP.md`](.ciagent/ROADMAP.md)
|
|
|
|
## Status
|
|
|
|
- **v1.4 (active):** central pipeline contract + shell reproducibility +
|
|
output streaming. A declarative pipeline contract
|
|
(`schemas/pipeline.schema.json` + `pipelines/ci.yaml`) binds the Gitea
|
|
and GitHub workflows to a single source of truth. `scripts/run_ci.sh`
|
|
mirrors the CI pipeline locally. `scripts/run_platform.sh` streams
|
|
terraform/checkov output by default. Ship tag `v1.4.1`.
|
|
- **v1.3 (complete, tag `v1.3.2`):** module documentation + thin-composition
|
|
removal. The L2 composition layer is removed; module READMEs are built
|
|
out. Testing + CI/CD pipelines (pytest, `--check-only`, Gitea + GitHub
|
|
workflows).
|
|
- **v1.2 (complete, tag `v1.3.0`):** platform hardening + first real
|
|
consumer deployment. Harden the v1.1 spike's NFRs, simplify the setup,
|
|
rewrite the docs, and prove the platform delivers real value by
|
|
deploying a basic microservice to AWS ECS Fargate end-to-end (`terraform
|
|
apply`, dev autonomous).
|
|
- **v1.1 (complete, tag `v1.2.0`):** architecture finalization + v1 spike.
|
|
Finalized the architecture to v1.0 (resolved all 11 open design
|
|
decisions) and proved the IR commitments hold with one end-to-end spike
|
|
(`l1-s3` + `l2-static-asset` + Terraform adapter → real `terraform plan`
|
|
against AWS). Gitea release id 202.
|
|
- **v1.0 demo (complete, archived under `demo/`, tag `v1.1.0`):** the
|
|
30-minute stub-driven executive demo. Preserved as the intent reference;
|
|
it is not the platform.
|
|
|
|
## How the platform works
|
|
|
|
The platform is **four layers + six cross-cutting concerns**, bound by the
|
|
vision's "Two Consumer Surfaces, One Platform" tenet: technical developers
|
|
(L3A) and non-technical consumers (L3B) converge on the same contract
|
|
schema, the same policy envelope, and the same evidence stream.
|
|
|
|
### The v1.1 spike flow (end-to-end)
|
|
|
|
```
|
|
contracts/spike.yaml
|
|
│ (contract schema validation)
|
|
▼
|
|
acdl_platform/contract_resolver.py ──▶ Target Stack IR (JSON)
|
|
│ (IR schema validation)
|
|
▼
|
|
adapters/terraform/adapter.py ──▶ terraform/spike/{main,terraform,providers}.tf
|
|
│ (the only substrate-specific code)
|
|
▼
|
|
terraform plan (real AWS, via the rotated spike key — D-039/D-047)
|
|
│
|
|
▼
|
|
adapters/terraform/policy/checkov_adapter.py ──▶ PolicyCheckResult (JSON list)
|
|
│ (normalized, engine-agnostic)
|
|
▼
|
|
acdl_platform/confidence_signal.py ──▶ { score, band, perInput, reasonCodes }
|
|
│ (6 inputs: policy, validation, freshness, source, history, nfrs)
|
|
▼
|
|
acdl_platform/outbox_writer.py ──▶ DynamoDB outbox (acdl-outbox)
|
|
│ (hash-chained evidence event)
|
|
▼
|
|
acdl-evidence timeline (acdl-evidence repo, raw-file served)
|
|
```
|
|
|
|
The spike validates the architecture's claim that the **IR-shaped
|
|
commitments do not require a polyglot mess**: the adapter is the only
|
|
substrate-specific code. `modules-ir/`, `schemas/`, `contracts/`,
|
|
`acdl_platform/confidence_signal.py`, `acdl_platform/contract_resolver.py`,
|
|
and `acdl_platform/outbox_writer.py` are all substrate-agnostic (no
|
|
`aws_s3_bucket` / `aws_` Terraform terms).
|
|
|
|
### What's different in v1.2
|
|
|
|
v1.2 extends the spike to a real, simpler, better-documented platform that
|
|
**deploys a microservice to ECS Fargate**:
|
|
|
|
- Six new IR-typed L1s: `l1-vpc`, `l1-ecs-cluster`, `l1-ecs-service`,
|
|
`l1-iam-role`, `l1-alb`, `l1-ecr`.
|
|
- One new L2 thin-composition: `l2-microservice` (references the six L1s).
|
|
- `terraform apply` (dev, autonomous per §10, confidence ≥ 0.50) — real
|
|
provisioning, not just `plan`.
|
|
- A new consumer repo `acdl-consumer-microservice` with a basic HTTP
|
|
container + Dockerfile + ECR push + contract submission.
|
|
- One `scripts/run_platform.sh` (consolidated from the v1.1 spike scripts).
|
|
- NFR hardening: least-privilege IAM (expanded for ECS), idempotent
|
|
bootstrap, proper error handling, P1-1 redaction.
|
|
|
|
## How to run
|
|
|
|
### Prerequisites
|
|
|
|
- AWS account + the rotated spike key in `.env.secrets` (see
|
|
`scripts/rotate_spike_key.sh`; the bootstrap root key was deactivated
|
|
per D-034 closure).
|
|
- `terraform` (pin `1.9.*`), `checkov` (pin `>=3.2,<4`), `python3` + `boto3`
|
|
+ `jsonschema`.
|
|
|
|
### Run the platform pipeline end-to-end
|
|
|
|
```bash
|
|
# 1. Bootstrap the AWS state backend + spike IAM user (one-time, idempotent)
|
|
# (requires the bootstrap root key in env — now deactivated; skip if
|
|
# the state bucket + acdl-spike-runner already exist)
|
|
ACDL_BOOTSTRAP_AWS_ACCESS_KEY_ID=... ACDL_BOOTSTRAP_AWS_SECRET_ACCESS_KEY=... \
|
|
python3 terraform/bootstrap/create_state_backend.py
|
|
ACDL_BOOTSTRAP_AWS_ACCESS_KEY_ID=... ACDL_BOOTSTRAP_AWS_SECRET_ACCESS_KEY=... \
|
|
python3 terraform/bootstrap/create_iam_user.py # prints the initial key
|
|
|
|
# 2. Rotate the spike key (writes .env.secrets, gitignored)
|
|
ACDL_BOOTSTRAP_AWS_ACCESS_KEY_ID=... ACDL_BOOTSTRAP_AWS_SECRET_ACCESS_KEY=... \
|
|
bash scripts/rotate_spike_key.sh
|
|
|
|
# 3. Run the full platform pipeline (IR -> adapter -> plan -> Checkov ->
|
|
# confidence -> outbox). Output is streamed to stdout by default.
|
|
bash scripts/run_platform.sh
|
|
# Expected: "=== PLATFORM E2E OK ==="
|
|
|
|
# Or plan-only (IR -> adapter -> terraform plan; no Checkov/outbox):
|
|
bash scripts/run_platform.sh --plan-only
|
|
|
|
# Add --quiet to suppress streaming (output to log files only):
|
|
bash scripts/run_platform.sh --quiet
|
|
```
|
|
|
|
### Test the platform (offline, no AWS required)
|
|
|
|
```bash
|
|
# Install test dependencies
|
|
pip install -r requirements-test.txt
|
|
|
|
# Run the test suite (122 tests, all offline — uses moto for DynamoDB mocking)
|
|
python3 -m pytest tests/ -v
|
|
|
|
# Run the platform in check-only mode (offline — no AWS, no Checkov, no outbox)
|
|
# Streams the emitted Terraform to stdout by default; --quiet suppresses it
|
|
bash scripts/run_platform.sh --check-only
|
|
# Expected: "=== PLATFORM CHECK OK ==="
|
|
|
|
# Reproduce the full CI pipeline locally (lint → test → check-only)
|
|
bash scripts/run_ci.sh
|
|
# Expected: "=== CI PIPELINE OK ==="
|
|
```
|
|
|
|
### CI/CD pipelines
|
|
|
|
The CI/CD pipeline is defined by a **central pipeline contract** — a
|
|
declarative YAML instance (`pipelines/ci.yaml`) validated against a JSON
|
|
Schema (`schemas/pipeline.schema.json`). Both forge workflows implement
|
|
the same contract:
|
|
|
|
- `.gitea/workflows/ci.yml` — Gitea Actions (dev environment)
|
|
- `.github/workflows/ci.yml` — GitHub Actions (production)
|
|
|
|
Both workflow files are **byte-identical** — the only difference is the
|
|
forge runtime. Both run three stages: **lint** (py_compile), **test**
|
|
(pytest), and **check-only** (`run_platform.sh --check-only`). Both
|
|
trigger on push to `main` and on pull requests. A test
|
|
(`tests/test_pipeline_contract.py`) validates that both workflows conform
|
|
to the contract.
|
|
|
|
`scripts/run_ci.sh` mirrors the CI pipeline locally — running the same
|
|
three stages in sequence. This makes the pipeline fully reproducible from
|
|
the shell, not just in CI:
|
|
|
|
```bash
|
|
bash scripts/run_ci.sh # run all 3 stages (lint, test, check-only)
|
|
bash scripts/run_ci.sh --quiet # suppress per-stage banners
|
|
```
|
|
|
|
### Output streaming (run_platform.sh)
|
|
|
|
`scripts/run_platform.sh` streams output by default so the user can see
|
|
what the platform is doing:
|
|
|
|
- **`--check-only`**: streams the emitted Terraform file content to stdout
|
|
- **`--plan-only`** and **full mode**: streams `terraform init`, `terraform
|
|
validate`, and `terraform plan` output via `tee` (visible and logged)
|
|
- **Full mode**: prints Checkov compliance results and each
|
|
PolicyCheckResult record with severity, rule ID, and pass/fail status
|
|
|
|
A `--quiet` flag suppresses streaming (output to log files only) for
|
|
backwards-compatible log-only mode.
|
|
|
|
### Re-run the archived v1.0 demo (stubs only, no AWS)
|
|
|
|
```bash
|
|
bash demo/scripts/run_demo.sh --no-upload
|
|
```
|
|
|
|
The demo deck is at [`demo/ACDL_DEMO.md`](demo/ACDL_DEMO.md). It runs
|
|
entirely on local stubs — no AWS, no AI — and shows intent and safety
|
|
behavior rather than provisioning real cloud resources.
|
|
|
|
## Repository layout
|
|
|
|
| Path | Purpose | Status |
|
|
|------|---------|--------|
|
|
| `acdl_platform/` | Platform code: confidence signal, outbox writer, separation of duties, HITL/ledger designs | v1.1 complete; v1.3 removes contract_resolver |
|
|
| `schemas/` | JSON Schemas: IR, PolicyCheckResult, pipeline contract (draft 2020-12) | v1.1 complete; v1.4 adds pipeline schema |
|
|
| `pipelines/` | Central pipeline contract: `ci.yaml` (YAML instance validated against `schemas/pipeline.schema.json`) | v1.4 |
|
|
| `adapters/` | Substrate adapters — Terraform adapter (the only substrate-specific code per §12) + Checkov policy adapter | v1.1 complete; v1.2 expands `TYPE_MAP` |
|
|
| `terraform/` | State backend (S3 + DynamoDB) + spike TF (`terraform/spike/`) + bootstrap scripts (`terraform/bootstrap/`) | v1.1 complete; v1.2 adds ECS apply |
|
|
| `modules-ir/` | IR-typed L1/L2 modules + `registry.json`. v1.1: `l1-s3`. v1.2: + 6 ECS L1s. v1.3: L2 removed (placeholders) | v1.3 |
|
|
| `scripts/` | Platform run script (`run_platform.sh` with `--check-only`/`--plan-only`/`--quiet`), CI pipeline script (`run_ci.sh`), verify scripts, key rotation | v1.4 |
|
|
| `tests/` | Pytest suite (122 tests, all offline — adapter, confidence signal, checkov adapter, outbox writer, pipeline contract, streaming) | v1.4 |
|
|
| `demo/` | Archived v1.0 executive demo (tag `v1.1.0`); runs locally via `demo/scripts/run_demo.sh --no-upload` | complete (archived) |
|
|
| `.ciagent/` | CIAgent metadata (config, project, architecture, requirements, roadmap, personas, plans, research, verify, review, audit) | active |
|
|
| `docs/` | Upstream vision + architecture sources (`vision.md`, `architecture.md`) | active |
|
|
|
|
## Environments
|
|
|
|
| Environment | Autonomy | Gate | Status |
|
|
|---|---|---|---|
|
|
| dev | Full autonomy (no HITL) | Confidence ≥ 0.50 | v1.1 spike (`plan`); v1.2 (`apply`) |
|
|
| qa | Held for attestation | QA HITL + confidence ≥ 0.75 | v1.3+ |
|
|
| prod | Held for attestation | SRE HITL + confidence ≥ 0.90 | v1.3+ |
|
|
| dr | Held for attestation | SRE HITL + confidence ≥ 0.95 + dr-drill | v1.3+ |
|
|
|
|
**Staging does not exist** (Path A locked).
|
|
|
|
## Credentials
|
|
|
|
**Long-lived AWS credentials are forbidden** (§12.5). The v1.1 spike uses a
|
|
temporary long-lived key **once** to bootstrap (waiver D-034, now closed —
|
|
the root key was deactivated by the user), then rotates the spike key
|
|
per-run via `scripts/rotate_spike_key.sh` (waiver D-039, extended for v1.2
|
|
as D-047). Real OIDC federation is deferred to v1.3+, blocked on
|
|
[go-gitea/gitea#36988](https://github.com/go-gitea/gitea/pull/36988) (still
|
|
open as of 2026-07-21). |