diff --git a/docs/CONSUMER_GUIDE.md b/docs/CONSUMER_GUIDE.md deleted file mode 100644 index 59b926f..0000000 --- a/docs/CONSUMER_GUIDE.md +++ /dev/null @@ -1,359 +0,0 @@ -# Consumer Guide — Declare intent, deploy to AWS - -This guide walks a consumer through creating their pipeline and defining a -contract that deploys any ACDL module to AWS. It is **generic** across all -L2 modules in the registry; `static-asset` is the worked example, but every -step applies to `microservice` and any future L2 composition. - -## The model - -Consumers have their own repos and consume ACDL by referencing `uses:` the -central pipeline definitions. The consumer declares a **contract** (which -module, which environment, which inputs); the ACDL platform owns the -pipelines, modules, Terraform adapter, and evidence stream. - -You do not write Terraform, workflow YAML, or adapter code. You write a -contract YAML file and the platform does the rest. Your repository contains -only your application code and that one contract. - -```mermaid -flowchart LR - A["your repo
(app code + contract.yaml)"] -->|uses: acdl/.gitea/workflows/deploy.yml@v1.4| B - B["ACDL platform runners
(modules/ + pipelines/ + adapters/ + schemas/)"] -->|contract -> resolver -> stack -> adapter
-> terraform plan -> Checkov -> confidence
-> apply -> evidence event to outbox| C - C["your resources in AWS"] -``` - -## Versioning the `uses:` reference - -The central deployment pipeline is **always versioned with floating MAJOR -and MINOR tags** (e.g. `acdl/pipelines/deploy.yaml@v1.4`). Version -constraints cannot be expressed inside the contract, so the tag in -`uses:` is the only immutability lever a consumer has. - -**Unversioned references are discouraged.** Do not use `@main` or a bare -`acdl/pipelines/deploy.yaml` — `main` is constantly updated and can cause -unexpected failures in your deployment. Pinning to a MAJOR+MINOR tag means: - -- **Immutability** — the pipeline behavior you tested is the behavior you - get. Patch fixes flow within the tag; breaking changes land under the - next MINOR tag (`@v1.5`), which you opt into explicitly. -- **Resilience** — your deployment does not break because an unrelated - change landed on `main`. -- **DX** — your setup is stable and reproducible. You upgrade on your - schedule by bumping the tag. - -All examples in this guide use `@v1.4`. When a new MINOR tag is released -(e.g. `@v1.5`), review its changelog and bump your `uses:` reference when -ready. - -## Prerequisites - -These are the **only** prerequisites for a consumer repo. You do **not** -need an AWS account, Terraform, Checkov, boto3, or a rotated runner key — -those are platform-repo concerns, provided by the platform runners. - -- **A consumer GitHub or Gitea repository** for your application code + - `contract.yaml`. -- **An ACDL platform runner available to your org.** The platform team - provides runners with Terraform, Checkov, Python, and the AWS auth - already configured. You do not install any of these. -- **Authorization to reference the central pipeline.** Onboarding grants - your repo the right to `uses: acdl/.gitea/workflows/deploy.yml@v1.4`. - Contact the platform team if you have not been onboarded. - -## Step 1 — Create a consumer repo - -Create a repository for your application. The top level holds your app -code; your contract lives at `.acdl/contract.yaml`. Example for a static -site: - -``` -my-static-site/ - index.html - assets/ - style.css - logo.png - .acdl/ - contract.yaml -``` - -Example for a microservice: - -``` -my-microservice/ - app.py - Dockerfile - .acdl/ - contract.yaml -``` - -Your app code lives at the top level. Your contract lives at -`.acdl/contract.yaml` regardless of the module you deploy. - -## Step 2 — Reference the central pipeline - -In your contract YAML, declare `uses:` pointing at the central ACDL -deployment pipeline with a **versioned tag** (floating MAJOR + MINOR): - -```yaml -uses: acdl/pipelines/deploy.yaml@v1.4 -``` - -This tells the platform to run the standard deployment pipeline: -validate-contract -> resolve-stack -> terraform-plan -> checkov -> -confidence -> apply. - -## Step 3 — Define the contract - -Write `.acdl/contract.yaml`. The `static-asset` example: - -```yaml -uses: acdl/pipelines/deploy.yaml@v1.4 -module: static-asset -environment: dev -inputs: - bucket_name: my-static-site-assets - region: us-east-1 -``` - -A `microservice` example: - -```yaml -uses: acdl/pipelines/deploy.yaml@v1.4 -module: microservice -environment: dev -inputs: - image: my-registry/my-microservice:latest - port: 8080 - env: - LOG_LEVEL: info -``` - -### Contract fields - -| Field | Type | Required | Description | -|-------|------|----------|-------------| -| `uses` | string | yes | Reference to the central deployment pipeline, **versioned** with a floating MAJOR+MINOR tag (e.g. `acdl/pipelines/deploy.yaml@v1.4`). Bare or `@main` references are discouraged. | -| `module` | string | yes | Module name from the registry — any L1 primitive or L2 composition (e.g. `static-asset`, `microservice`, `s3`). See the [module catalog](../modules/README.md). | -| `environment` | enum | yes | `dev` (autonomous), `qa` (QA HITL), `prod` (SRE HITL), `dr` (SRE HITL). | -| `inputs` | object | yes | Module-specific inputs (see below). | - -### Module inputs - -Each module declares its inputs in its `interface.json` (L1) or -`composition.json` (L2). Consult the [module catalog](../modules/README.md) -for the full list, or read the module's own README under `modules/l1//` -or `modules/l2//`. - -**`static-asset` inputs** (the worked example): - -| Input | Type | Required | Description | -|-------|------|----------|-------------| -| `bucket_name` | string | yes | Globally-unique S3 bucket name. | -| `region` | string | yes | AWS region the bucket is created in. | - -The contract is validated against `schemas/contract.schema.json`. An -invalid contract (missing field, unknown module, wrong type) fails at the -validate-contract stage with a clear error. - -## Step 4 — Run the pipeline - -You do **not** run platform scripts locally for the happy path. The -central deploy workflow is a **reusable workflow** that the platform -runners fetch and execute for you. - -### The consumer workflow - -Add a thin workflow file to **your** repo that invokes the reusable ACDL -deploy workflow with a **versioned tag**. For Gitea Actions -(`.gitea/workflows/deploy.yml`): - -```yaml -name: deploy -on: - push: - branches: [main] -jobs: - deploy: - uses: acdl/.gitea/workflows/deploy.yml@v1.4 - with: - contract: .acdl/contract.yaml -``` - -For GitHub Actions (`.github/workflows/deploy.yml`), the `uses:` line is -identical — only the directory differs: - -```yaml -name: deploy -on: - push: - branches: [main] -jobs: - deploy: - uses: acdl/.github/workflows/deploy.yml@v1.4 - with: - contract: .acdl/contract.yaml -``` - -That is the entire consumer-side workflow. When you push to `main`: - -1. The forge resolves `uses: acdl/.gitea/workflows/deploy.yml@v1.4` (or - the GitHub equivalent) to the reusable workflow **at the pinned tag**. -2. A **platform-provided runner** checks out **your** repo (the consumer - repo). -3. The runner checks out the **ACDL platform repo** into the workspace - (`acdl-platform/`) — this is how the pipeline fetches the platform code - at run time. You never clone the platform repo yourself. -4. The runner installs the runtime dependencies (Python, Terraform, - Checkov) that the platform requires. -5. The runner invokes `scripts/run_platform.sh` against your - `.acdl/contract.yaml`. - -You see the streamed output (terraform plan, Checkov results, confidence -signal) in your forge run logs. The `--check-only` and `--plan-only` flags -are platform-side modes visible in the pipeline logs; you do not pass them -yourself — the reusable workflow selects the mode based on the -`environment` in your contract (`dev` = full apply; higher environments -hold for HITL). - -### Local validation (optional) - -A consumer *may* clone the ACDL platform repo to run `--check-only` -against their contract before pushing — this is optional and not required -for the happy path. If you do this, the runtime dependencies (Python, -`jsonschema`, `pyyaml`, `boto3`) must be installed locally, and any AWS -credentials follow the [Credentials](../README.md#credentials--zero-trust) -override model: a static key in `.env.secrets` (gitignored) is rotated -**out of band by you** — the platform guarantees daily rotation for forge -runs, not for locally-held copies. - -```bash -# Optional pre-push validation (clone the platform repo first): -bash scripts/run_platform.sh --check-only path/to/your/.acdl/contract.yaml -# Expected: "=== PLATFORM CHECK OK ===" -``` - -## Step 5 — What the pipeline does - -Each stage of the central deployment pipeline (`pipelines/deploy.yaml`): - -```mermaid -flowchart TD - S1["validate-contract
schema check vs contract.schema.json"] --> S2 - S2["resolve-stack
contract_resolver.py -> Target Stack JSON"] --> S3 - S3["terraform-plan
adapter.py compiles stack -> terraform plan (real AWS)"] --> S4 - S4["checkov
policy checks -> PolicyCheckResult records"] --> S5 - S5["confidence
confidence_signal.py -> score + band (dev >= 0.50)"] --> S6 - S6["apply
dev only: terraform apply + evidence event to outbox"] -``` - -1. **validate-contract** — validates your contract YAML against - `schemas/contract.schema.json`. Fails fast on missing fields, unknown - modules, or wrong types. - -2. **resolve-stack** — the contract resolver - (`acdl_platform/contract_resolver.py`) resolves your contract to a - Target Stack instance. It loads the module's composition, expands its - children, wires your contract inputs to the children's inputs, and - emits a stack JSON instance. - -3. **terraform-plan** — the Terraform adapter - (`adapters/terraform/adapter.py`) compiles the stack to Terraform - (`main.tf`, `terraform.tf`, `providers.tf`) and runs `terraform plan` - against real AWS. You see the plan in your run logs. - -4. **checkov** — Checkov runs policy checks on the emitted Terraform. The - results are normalized to `PolicyCheckResult` records by the Checkov - adapter. Each result has a severity, rule ID, and pass/fail status. - -5. **confidence** — the confidence signal - (`acdl_platform/confidence_signal.py`) computes a score from 6 inputs - (policy, validation, freshness, source, history, NFRs). For `dev`, the - threshold is >= 0.50. If the band is `pass`, the pipeline proceeds. - -6. **apply** — (dev only, autonomous per the environment model) Terraform - applies the plan, creating the resources in your AWS account. An - evidence event (hash-chained) is written to the DynamoDB outbox. - -## Step 6 — What gets created - -After a successful `dev` run, the resources declared by your module's -composition exist in your AWS account, and an evidence event is recorded. - -For the `static-asset` example: - -- **An S3 bucket** named `my-static-site-assets` in `us-east-1` with - versioning enabled. -- **An evidence event** in the DynamoDB outbox (`acdl-outbox` table) with - the contract ID, stack name (`static-asset`), confidence score, and band. -- **A confidence band** of `pass` (score >= 0.50 for dev). - -For other modules, consult the module's README -(`modules/l1//README.md` or `modules/l2//README.md`) for the -exact resources created. - -## Step 7 — Upload your content (static-asset example) - -The platform provisions the infrastructure; you upload your content. For -the `static-asset` module: - -```bash -aws s3 sync ./assets s3://my-static-site-assets/ --acl public-read -``` - -(For a proper static site, configure the bucket for website hosting or -put a CloudFront distribution in front — both are future compliance -extension points for the `static-asset` module.) - -For a `microservice`, the platform provisions the ECS service and ALB; you -push your container image to the ECR repo the platform created. - -## Step 8 — Promote to qa / prod - -Change `environment` in your contract (keeping the same versioned `uses:`): - -```yaml -uses: acdl/pipelines/deploy.yaml@v1.4 -environment: qa # QA HITL gate + confidence >= 0.75 -environment: prod # SRE HITL gate + confidence >= 0.90 -``` - -Higher environments require human attestation (forge deployment approval) -and higher confidence thresholds. The platform enforces separation of -duties (qaApprover != prodApprover) via the DynamoDB outbox. - -| Environment | Autonomy | Gate | -|-------------|----------|------| -| dev | Full autonomy | Confidence >= 0.50 | -| qa | QA HITL | Confidence >= 0.75 | -| prod | SRE HITL | Confidence >= 0.90 | -| dr | SRE HITL | Confidence >= 0.95 + dr-drill | - -## Step 9 — Compliance extensions - -Each module lists compliance extension points for the future compliance -milestone (GDPR, SOX, SOC2, HIPAA, DORA). See each module's README under -`modules/l1//README.md` or `modules/l2//README.md` for the -per-module extension points. Common examples: - -- **KMS key** — shared encryption key for SSE. -- **S3 access logs** — access logging to a separate audit bucket. -- **Object Lock** — 7-year immutable retention for evidence. -- **Public access block** — prevent data exfiltration. - -## Reference - -| Resource | Path | Description | -|----------|------|-------------| -| Central deployment pipeline contract | `pipelines/deploy.yaml` | The pipeline stages your contract references. | -| Reusable deploy workflow (Gitea) | `.gitea/workflows/deploy.yml` | The workflow your repo invokes via `uses:`. | -| Reusable deploy workflow (GitHub) | `.github/workflows/deploy.yml` | The workflow your repo invokes via `uses:`. | -| Contract schema | `schemas/contract.schema.json` | JSON Schema for consumer contracts. | -| Stack schema | `schemas/stack.schema.json` | JSON Schema for the resolved stack instance. | -| Module catalog | `modules/README.md` | All L1 primitives and L2 compositions. | -| Sample contract | `contracts/static-asset.yaml` | The reference example contract (uses `@v1.4`). | -| Contract resolver | `acdl_platform/contract_resolver.py` | Resolves contracts to stack instances. | -| Terraform adapter | `adapters/terraform/adapter.py` | Compiles stack instances to Terraform. | -| Platform pipeline runner | `scripts/run_platform.sh` | The pipeline runner (platform-side; consumers do not invoke it directly). | -| Platform README | `README.md` | How the platform works + how to run the platform repo locally. | -| Credentials & zero-trust | `README.md#credentials--zero-trust` | The OIDC/ABAC default + static-key override model. | \ No newline at end of file diff --git a/docs/_config.yml b/docs/_config.yml new file mode 100644 index 0000000..b3063d4 --- /dev/null +++ b/docs/_config.yml @@ -0,0 +1,33 @@ +title: ACDL — Agentic Cloud Delivery Platform +description: Consumer + platform-engineer documentation for the ACDL platform. +theme: jekyll-rtd-theme +remote_theme: mmistakes/minimal-mistakes@9.0.4 + +exclude: + - internal/ + +defaults: + - scope: + path: "" + values: + layout: single + +nav: + - title: Overview + url: / + - title: Consumer Guide + url: /consumer-guide/ + - title: Modules + url: /modules/ + - title: Contracts + url: /contracts/ + - title: Pipeline + url: /pipeline/ + - title: Versioning + url: /pipeline/versioning/ + - title: Environments + url: /environments/ + - title: Architecture + url: /architecture/ + - title: Vision + url: /vision/ \ No newline at end of file diff --git a/docs/architecture-v1.0.md b/docs/architecture-v1.0.md deleted file mode 100644 index 5fb6943..0000000 --- a/docs/architecture-v1.0.md +++ /dev/null @@ -1,458 +0,0 @@ -# Architecture Document v1.0 - -> **Snapshot status:** v1.0 — taken in ACDL Phase 07 (milestone v1.1). -> All 11 open decisions in §13 are **resolved** — see `PROJECT.md` -> "Open-decision resolutions" table + decisions D-034..D-046. -> The body §§1-12 is copied verbatim from the upstream -> `docs/architecture.md` v0.2; only the header status line, the resolution -> session log, §13, §14, and the new §15 are Phase 07 additions. The -> `act_runner` → `gitea-runner` rename (D-046, 2026-04 in gitea/runner#850) -> is applied; `act_runner` appears only in a "formerly" note. - -# Agentic Cloud Delivery Platform — Architecture Document - -Status: **v1.0** (snapshot taken in ACDL Phase 07, milestone v1.1). All 11 -open decisions in §13 are resolved — see `PROJECT.md` "Open-decision -resolutions" table + decisions D-034..D-046. - -Companion to: Agentic Cloud Delivery Vision [1]. - -Authoring principle: The vision is the source of truth for why [1]; this document is the source of truth for how. Where the two conflict, the vision wins. - -Resolution session log (v1.0 snapshot — see PROJECT.md for full text): - -| ID | Question | Resolution (one-line — see PROJECT.md for rationale) | -|---|---|---| -| W1.A | AI-refinement trigger | ✅ RESOLVED — joint condition: N ≥ 50 consecutive zero-rollback changes AND no L1/L2 incident in 6 months AND Infra & Ops unilateral override. | -| W1.B | Multi-stack edge case rule | ✅ RESOLVED — permitted only for (a) DR-region mirror, (b) time-boxed experimental stack TTL ≤ 30d, (c) explicit Infra & Ops approval with `multiStack.justification`. | -| W2.A | Tag mutability for prod | ✅ RESOLVED — Path B: tag for dev/qa, SHA for prod; platform CLI resolves tag→SHA. | -| W3.D | L1/L2 standard versioning | ✅ RESOLVED — semver (interface→MAJOR, behavior→MINOR, lifecycle→PATCH); L2 pins L1 by `name@semver`; MAJOR bump = new registry entry + 12-month deprecation. | -| W3.E | Schema mandatory vs. optional inputs | ✅ RESOLVED — dev: stack+environment; qa adds validation.e2eSuite+loadTest; prod adds runbook+dashboard+oncall; dr adds drDrillRef; `inputs` always optional; `profile: agentic` fields optional everywhere (naturalLanguageIntent required when profile is agentic). | -| BA.A | Initial L3B skill catalog | ✅ RESOLVED — 5 skills: web API, worker, scheduled job, static asset, basic observability bootstrap; addition criteria: (a) sensitive-data reviewable, (b) single contract submission, (c) documented use case. | -| BA.B | Confidence threshold tuning | ✅ RESOLVED — thresholds frozen for v1; tuning begins v1.2 (quarterly FP/FN tracking; override = Infra & Ops + SRE joint sign-off, itself a confidence-event). | -| BA.C | On-call / operational ownership | ✅ RESOLVED — platform on-call = Infra & Ops; L3A/L3B halt → platform on-call (Sev2); consumer-visible outage → consumer on-call (Sev1) + platform support. | -| BA.D | Cost / capacity governance | ✅ RESOLVED — FinOps owns cloud cost; per-contract monthly reporting; runaway spend hard-halts at 120% of declared budget via the confidence signal; override = FinOps + SRE joint sign-off. | -| BA.E | Consumer onboarding | ✅ RESOLVED — developer (L3A): `getting-started` → contract schema + central pipeline template; citizen (L3B): scoped agent + skill catalog, no workflow authoring; both end in a sandbox dev submission that must pass the confidence gate. | -| BA.F | Cross-platform evolution | ✅ RESOLVED — contract schema, stack, PolicyCheckResult, confidence signal, audit stream are portable (forge-agnostic); forge-specific code = workflow YAML, OIDC trust, CODEOWNERS, Environments; a second forge needs a forge adapter + workflow-template translator, no change to L1/L2/stack/confidence/audit. | -| Q1.3 | OpenTofu timing | ✅ RESOLVED (deferred) — not in v1 or v1.1; the substrate abstraction (§12) makes OpenTofu a future adapter, not an architecture change; revisit when an OpenTofu adapter is requested. | - ---- - -## 0. Purpose - -This document encodes the architectural commitments that realize the vision [1]. The resolution session has closed eight open items; the document is now at v0.2 with eleven open items remaining, listed in Section 13. Every locked commitment is grounded in either a vision tenet or a specific decision made during resolution. - -The structure remains: four layers (L1 primitives, L2 composed stacks, L3A developer surface, L3B agentic surface) plus five cross-cutting concerns (central pipeline, contract schema, confidence signal, audit stream, HITL mechanics), with one addition: the substrate abstraction layer (Section 12) is now a first-class architectural concern, not an implementation detail. - -## 1. Architectural Overview - -The platform remains four layers and five cross-cutting concerns. The substrate abstraction is added as a sixth cross-cutting concern in Section 12 because it is the binding constraint for the L1/L2 model, the central pipeline, and the policy toolchain. - -The vision's "Two Consumer Surfaces, One Platform" tenet [1] remains the constraint that binds all concerns: L3A and L3B converge on the same contract schema, the same policy envelope, and the same evidence stream. - -Locked additions this revision: - -- The environment model is dev (autonomous) → qa (QA HITL) → prod (SRE HITL) → dr (SRE HITL). Staging does not exist. - -- L1/L2 are substrate-agnostic in shape; substrate adapters are the only substrate-specific component. - -## 2. Layer 1 — Foundational Primitives - -Purpose. Single-purpose, substrate-agnostic primitive modules representing the smallest reusable infrastructure pieces. L1 modules do not compose with other L1 modules; L1 takes its environment as input. - -Locked commitments (unchanged from v0.1): - -- No inter-L1 references. L1 may call Terraform data sources. - -- Semver with three triggers (interface → MAJOR, behavior → MINOR, lifecycle → PATCH). - -- Immutability on publication. - -- 12-month deprecation window. - -- AI refinement is a flag. - -✅ RESOLVED (see PROJECT.md W1.A): AI-refinement operational trigger — joint condition: N ≥ 50 consecutive changes with zero rollbacks AND no L1/L2 incident in last 6 months AND Infra & Ops holds a unilateral override. - -✅ RESOLVED (sub-decision): The L1 module's interface field is defined against the Target Stack, not against Terraform's variable block directly. In v1, the stack is shaped to round-trip cleanly to Terraform, but the schema is substrate-agnostic. Pending v1 implementation details in Section 12. - -## 3. Layer 2 — Composed Stacks - -Purpose. Combine L1 primitives into deployable infrastructure shapes. Each codebase maps to one canonical L2 stack; the stack is either a parameterized module (Shape X) or a thin-composition layer (Shape Y). - -Locked commitments (unchanged from v0.1): - -- 1 codebase = 1 L2 stack (default), with multiStack: true for exceptions. - -- Shape X or Shape Y. - -- Hierarchical composition, max depth 5, only registered L1s. - -- Pipeline quality checks: secrets-in-plaintext, public ingress, IAM wildcard, KMS key reference, tag compliance, naming convention. - -- Restricted from thin-composition: IAM principal creation, network boundary creation, key/secret creation, external data transfer. - -- Auto-promote after 3 observed usages. - -✅ RESOLVED (see PROJECT.md W1.B): Multi-stack edge case rule — permitted only for (a) DR-region mirror, (b) time-boxed experimental stack with TTL ≤ 30 days, (c) explicit Infra & Ops approval for a documented reason captured in multiStack.justification. - -✅ RESOLVED (sub-decision): The L2 composition tree's wires field is defined against the stack's relationship type, not against a Terraform module block. The stack → Terraform translation is the Terraform adapter's job (Section 12). The composition pipeline itself is substrate-agnostic. - -## 4. Layer 3A — Developer Consumer Surface - -Locked commitments (unchanged from v0.1): - -- Tag-based reference to the central pipeline template. - -- Developer-owned workflow file, no platform auto-sync. - -- L3A and L3B are parallel paths, not a progression. - -✅ RESOLVED (see PROJECT.md W2.A): Tag mutability for production-bound references — Path B (tag for dev/qa, SHA for prod). The platform provides a CLI command that resolves the current tag to its SHA for prod-bound workflows. - -## 5. Layer 3B — Agentic Consumer Surface - -Locked commitments (unchanged from v0.1): - -- Hybrid runtime, skill as markdown, agent as executor. - -- Trust model: trust and always verify on the platform side. - -- Skill envelope (4 dimensions). - -- Stateless agents, all state in the platform. - -Environment progression — locked (this revision): - -| Environment | Autonomy | Attester | Gate | -|---|---|---|---| -| dev | Full autonomy (no HITL) | — | Confidence signal ≥ 0.50, all six inputs present | -| qa | Held for attestation | QA | GitHub Deployment approval + full QA matrix (see §10) | -| prod | Held for attestation | SRE | GitHub Deployment approval + full SRE matrix (see §10) | -| dr | Held for attestation | SRE | GitHub Deployment approval + dr-drill evidence (see §10) | - -Staging is removed. Dev is the only autonomous environment and absorbs integration, contract, security smoke, and performance smoke validation. The CDLC reference document's environment model is a doc-sync item flagged at the top of this document. - -Profile marker: profile: agentic unlocks L3B-specific fields naturalLanguageIntent, confidenceAtSubmission, agentTrace). - -✅ RESOLVED (see PROJECT.md BA.A): Skill catalog — initial set: web API, worker, scheduled job, static asset, basic observability bootstrap. Addition criteria: (a) reviewable for sensitive data, (b) expressible as a single contract submission, (c) documented use case. - -## 6. Cross-Cutting — Central Pipeline Template - -Locked commitments (unchanged from v0.1): - -- JSON Schema (draft 2020-12) with thin domain-specific wrapper. - -- Central repo + generated client libraries. - -- Multi-stage validation pipeline (schema → policy → NFR → confidence). - -- Distributed enrichment. - -- GitOps reconciler + Terraform execution layer. - -Locked additions this revision: - -- The GitOps reconciler is the platform's K8s API. The cdlc-gitops repository's state materializes into K8s CRDs (ArgoCD Applications or Flux Kustomizations) that the reconciler watches. This is the platform's internal state surface. - -- The pipeline emits a PolicyCheckResult record per policy rule evaluated. The confidence signal consumes these as one normalized input (Section 8). - -✅ RESOLVED (see PROJECT.md W3.D): L1/L2 standard versioning details — semver (interface→MAJOR, behavior→MINOR, lifecycle→PATCH); L2 contracts pin L1 by `name@semver`; the resolver picks the highest compatible; MAJOR bumps require a new registry entry (immutable publication); old entry enters a 12-month deprecation window. - -✅ RESOLVED (see PROJECT.md W3.E): Schema mandatory vs. optional inputs — dev requires stack+environment; qa adds validation.e2eSuite + validation.loadTest; prod adds runbook + dashboard + oncall; dr adds drDrillRef; `inputs` always optional; `profile: agentic` fields optional everywhere (naturalLanguageIntent required when profile is agentic). - -## 7. Cross-Cutting — Contract Schema - -Locked commitments (unchanged from v0.1): - -- Central repo + generated client libraries. - -- Strict fail-fast at schema stage, multi-stage validation pipeline with reason codes from a published vocabulary. - -✅ RESOLVED (see PROJECT.md W3.E): Schema mandatory vs. optional inputs. The CDLC reference contract example [1] is illustrative; the v1 contract schema has explicit per-field mandatory/optional declarations per environment. - -## 8. Cross-Cutting — Confidence Signal - -Locked commitments (unchanged from v0.1): - -- Six canonical inputs. - -- Weighted sum with per-input breakdown. - -- Per-environment thresholds: dev ≥ 0.50, qa ≥ 0.75, prod ≥ 0.90, dr ≥ 0.95. - -- Structured output { score, band, perInput, reasonCodes }. - -- 1-year storage, no algorithm retraining in v1. - -- Halt with explicit reason on missing input. - -Locked additions this revision: - -- The policy check results input is a list of PolicyCheckResult records from the normalized schema (Section 9, 12). The signal does not know which engine produced which result. - -- Severity → score penalty mapping: critical → hard override to mandatory block, high → -0.2, medium → -0.05, low → -0.01, info → 0.0. One critical finding hard-overrides the score regardless of all other inputs. - -✅ RESOLVED (see PROJECT.md BA.B): Threshold tuning policy. Thresholds frozen for v1. Tuning begins v1.2: quarterly FP/FN tracking per environment; override authority = Infra & Ops + SRE joint sign-off; any override is itself a confidence-event in the audit stream. - -## 9. Cross-Cutting — Audit and Evidence Stream - -Locked commitments (unchanged from v0.1): - -- Tiered audit ledger: S3 with Object Lock in compliance mode (cold, source of truth, 7-year retention) + GitHub audit repo (hot, query index, not part of the chain). - -- Daily checkpoints. - -- Event schema: JWS detached signature, prev_event_hash chain, controlled-vocabulary event_type. - -- Outbox pattern with local durable outbox + async worker. - -- Linkage via workflow run ID or agent invocation ID. - -Locked additions this revision: - -- The outbox database is DynamoDB. RPO is zero (synchronous write to local outbox before contract submission ack); RTO is the async worker's recovery from the dead-letter queue. Single-region in v1; multi-region is a v2 concern. - -- The outbox also stores the per-contract QA and prod approver identities (Section 10). The platform-internal identity-distinctness check reads from this outbox. This is the only durable record of the approver identities outside GitHub's audit log. - -## 10. Cross-Cutting — Human-in-the-Loop Mechanics - -Purpose. The human gates at higher environments. The vision's "Lower Environments are Autonomous; Higher Environments are Attested" tenet [1] and the "deliberate human attestation — not as a rubber stamp" requirement [1] are the binding constraints. - -### 10.1 Gate model - -Pre-execution gates. The contract is held in a "validated but not applied" state until the human attests. qa, prod, and dr are PR-based attestation gates backed by GitHub Environments with required reviewers. - -For qa and prod, there is no partial deployment to roll back on rejection. For dr, the same model — promotion to the DR environment is a separate GitHub Deployment, gated by SRE, against a separate cluster/region. The canary/deployment-rollback model is explicitly not in scope for v1. - -### 10.2 Reviewer routing - -GitHub CODEOWNERS + GitHub Environment required reviewers. qa → QA team; prod → SRE team; dr → SRE team. CODEOWNERS is the routing layer; it does not enforce identity distinctness. - -### 10.3 Separation of duties — identity distinctness - -Mechanism is platform-internal, not GitHub-native, not Kyverno (in v1). - -Sequence: - -1. On promotion dev → qa, the platform reads the QA approver's GitHub identity from the GitHub Deployment approval event and writes it to the DynamoDB outbox keyed by contractId. - -2. On promotion qa → prod, the platform reads the stored QA approver identity from the outbox and the new SRE approver identity from the GitHub Deployment approval event. - -3. If qaApprover == prodApprover, the platform blocks the prod promotion, writes a SEPARATION_OF_DUTIES_VIOLATION event to the evidence stream, and routes a halt artifact to the SRE on-call. - -4. The check is implemented in the central pipeline repo, not as an external policy. The platform is the only writer to the outbox; the check is in the same process that has authority to block the promotion. - -### 10.4 Full HITL attestation matrix - -| Env | Concern | Evidence artifact | Freshness | Source | Attester | -|---|---|---|---|---|---| -| qa | Functional correctness | Last successful run of contract-declared validation.e2eSuite with pass rate ≥ 99% | Last 24h | Test runner declared in contract | QA | -| qa | Performance baseline | Load test report (k6 / Gatling / Locust) showing p99 latency < declared NFR and throughput > declared minimum | Last 7d | Load test runner declared in contract | QA | -| qa | Security posture | Vulnerability scan (Trivy, Snyk, or contract-declared equivalent) with no criticals/highs, signed by Security on-call | Last 24h | Security scanner + Security team signature | QA | -| qa | Contract NFRs | Platform-generated report: schema valid, NFR assertions (latency, throughput, error rate) within declared bounds | At submission | Platform contract validator | QA | -| prod | Operational readiness | Runbook published, dashboard exists, on-call rotation assigned, alerts configured | At submission, validated against last 30d history | Platform + SRE | SRE | -| prod | Incident response | Sev-1 runbook tabletop or live drill completed | Last 90d | SRE drill record | SRE | -| prod | Capacity / cost | FinOps forecast for next 30d within budget envelope, cost anomaly baseline stored, budget alert configured | Forecast valid for next 30d | FinOps + SRE | SRE | -| prod | Resilience | DR drill, chaos engineering report, backup verified | DR: 180d; chaos: 90d; backup: 30d | SRE + Platform | SRE | -| dr | dr-region deploy with the most recent prod-bound dr drill as canary evidence | dr drill report | Last 180d | SRE | SRE | - -### 10.5 Timeout behavior - -| Time | State | Action | -|---|---|---| -| Submission | PENDING_ATTESTATION | Notify responsible team | -| 1 business day | PENDING_ATTESTATION_WARNING | Notify team + platform on-call (elevated path); emit PENDING_ATTESTATION_TIMEOUT_WARNING event | -| 2 business days | PENDING_ATTESTATION_AUTO_FREEZE | Auto-freeze; require re-submission; emit PENDING_ATTESTATION_AUTO_FREEZE event; new submission linked via supersedes | - -### 10.6 Rejection and rollback - -Rejection returns the contract to a HELD state with the rejection reason captured as a PROMOTION_REJECTED event. The consumer fixes the cause and re-submits; the new submission is linked to the rejected one via supersedes. The audit chain is extended, not torn up — matching the resolution session's answer. - -There is no partial deployment to roll back at any v1 gate. - -## 11. Cross-Cutting — Agentic Stack - -Locked commitments (unchanged from v0.1): - -- Hybrid runtime, platform-managed control plane + consumer-owned agent. - -- Versioned, signed skill catalog over MCP. - -- Skill envelope enforced on invocation and result submission. - -- Consumer-owned skill execution environment. Platform does not run the skill. - -- Stateless agents, all state in the platform. - -Locked additions this revision: - -- Skills are reviewed for sensitive data before release. Secrets, customer data, internal IPs, and other sensitive payloads are forbidden in skill markdown. The review is owned by Infra & Ops and is the mandatory release gate for any new skill. This is the trade-off for accepting the L3B runtime threat model (skill content is consumer-readable, so the platform must not put anything sensitive in it). - -✅ RESOLVED (see PROJECT.md BA.A): Skill catalog — initial set, addition process, deprecation process per the resolution. - -## 12. Cross-Cutting — L1/L2 Substrate Execution - -Purpose. The technical execution layer for the L1/L2 substrate, including the substrate abstraction that protects v1 from polyglot mess while leaving v2+ room to grow. - -### 12.1 Substrate abstraction (locked this revision) - -L1/L2 are substrate-agnostic in shape. The architecture defines a Target Stack — a substrate-neutral description of: - -- Resources with typed input contracts, typed output contracts, and declared NFRs. - -- Relationships (single parent per child, with a shared keyword for multi-relationship dependencies). - -- Composition (a tree of resources with max depth 5). - -- Policy hooks (the points in the composition where policy checks attach). - -The L1 registry, the L2 thin-composition tree, the YML standard, and the policy check result schema are all defined against the stack schema. None of them is defined against any specific substrate. - -Substrate adapters are the only substrate-specific code. An adapter compiles the stack into a substrate execution plan. v1 ships exactly one adapter: the Terraform adapter. v2+ may add additional adapters (OpenTofu, Pulumi, K8s CRDs) without architectural change. - -v1 implementation reality: the stack is shaped to round-trip cleanly to Terraform because there is no other adapter to differentiate from. The stack and the Terraform output are nearly isomorphic in v1. As additional adapters appear in v2+, the stack gets more expressive (e.g., substrate-specific output types) and the adapters gain translation logic, but the L1 module content, the YML standard, and the composition tree do not change. This is the design that prevents the polyglot mess. - -Why not build the abstraction earlier? Building a substrate-agnostic stack before there is a second adapter to test against is speculative generality. The v1 commitment is: (1) the L1 module interface is defined against the stack schema even though the only adapter is Terraform, and (2) the central pipeline, registry, and policy schema consume the stack-typed contracts. The adapter is the only place where substrate terminology appears in v1. - -### 12.2 Terraform adapter (v1) - -The Terraform adapter: - -- Translates the stack-typed L1 module interface to a Terraform variable block and a Terraform output block. - -- Translates the stack-typed L2 composition tree to a Terraform root module that calls the L1 modules. - -- Translates the stack-typed relationships to Terraform module references. - -- Emits a Terraform plan from the stack. - -The adapter is a thin layer. It does not own L1/L2 content; it only translates. - -### 12.3 State storage - -Locked: S3 (state files) + DynamoDB (state locking), cloud-managed. Single-region in v1. - -### 12.4 Policy toolchain - -Locked: - -- Checkov for Terraform plan policy (the four L2 thin-composition checks: secrets-in-plaintext, public ingress, IAM wildcard, KMS key reference, plus tag and naming convention). Checkov is open-source, has a broad rule catalog, and is GitOps-friendly. - -- Kyverno for K8s-native policy (platform-internal state in the GitOps reconciler, separation-of-dues-adjacent checks if any are added in v2, future CRD validation). - -- OPA/Rego is reserved for cross-resource policy and is explicitly last resort due to Rego complexity. - -### 12.5 Execution layer - -Locked: GitHub Actions. terraform plan and terraform apply run in the central pipeline repo's GitHub Actions workflow. State locking via DynamoDB. AWS credentials via OIDC federation (long-lived credentials are forbidden). The platform does not run terraform apply against a developer's workstation; all execution is in the central pipeline. - -> **ACDL Phase 07 note (D-039):** Gitea Actions (the ACDL forge) does not -> support `id-token: write` / OIDC token issuance as of Gitea 1.27.x / -> gitea-runner v2.1.0 (formerly `act_runner`, renamed 2026-04 in -> gitea/runner#850). The v1.1 spike uses a per-run-rotated long-lived key -> waiver; real OIDC federation is a v1.2 deliverable, blocked on -> go-gitea/gitea#36988. The §12.5 "long-lived credentials are forbidden" -> commitment is the locked target; the waiver is a time-boxed spike -> exception. - -### 12.6 Policy result normalization (locked this revision) - -The confidence signal does not consume raw Checkov or Kyverno output. It consumes a normalized PolicyCheckResult schema produced by substrate-specific adapters. - -Schema (canonical form, lives in the central pipeline repo): - -```json -{ - "contractId": "uuid", - "evaluatedAt": "ISO-8601", - "engine": "checkov | kyverno | opa", - "ruleId": "CKV_AWS_24 | KYVERNO_NO_PRIVILEGED | ...", - "severity": "critical | high | medium | low | info", - "result": "pass | fail | skipped | error", - "message": "human-readable", - "evidence": { "...engine-specific payload, opaque to the signal..." }, - "resourceRef": "stack-typed resource identifier" -} -``` - -The Checkov adapter runs in the same GitHub Actions step as Checkov itself and translates Checkov JSON to PolicyCheckResult records. The Kyverno adapter runs as a controller in the platform's K8s cluster and translates Kyverno PolicyReport CRDs to PolicyCheckResult records. The confidence signal's policy input component is the union of all PolicyCheckResult records, regardless of engine. The signal does not know which engine produced which result — substrate-agnostic over its inputs, matching the L1/L2 model's substrate-agnostic over its outputs. - -### 12.7 Registry maintenance - -Locked: L1 module publication updates the L1 registry in the same PR as the module. Registry and module land together. The registry is the stack-typed contract, not a Terraform-specific variable schema. The L1 registry, the central pipeline, and the policy schema all consume the same stack-typed contract — there is one source of truth for the L1 interface, not multiple substrate-specific copies. - -### 12.8 Contract-schema-to-stack resolution - -The contract schema declares the consumer's intent in stack-typed terms. The central pipeline resolves the contract to a target stack (a list of L1 module instances with their inputs and the relationships between them). The Terraform adapter compiles the target stack to a Terraform execution plan. This resolution is substrate-agnostic — the target stack is in the stack schema. - -## 13. Consolidated Open Design Decisions - -✅ **All 11 decisions are RESOLVED (see PROJECT.md).** The §13 subsections -below preserve the upstream structure with the `🟡 OPEN` markers replaced -by `✅ RESOLVED (see PROJECT.md)`. - -### From Wave 1 (L1/L2 Substrate) - -- (W1.A) AI-refinement trigger. ✅ RESOLVED (see PROJECT.md) — joint condition: N ≥ 50 consecutive zero-rollback changes AND no L1/L2 incident in 6 months AND Infra & Ops unilateral override. - -- (W1.B) Multi-stack edge case rule. ✅ RESOLVED (see PROJECT.md) — permitted only for (a) DR-region mirror, (b) time-boxed experimental stack TTL ≤ 30d, (c) explicit Infra & Ops approval with `multiStack.justification`. - -### From Wave 2 (L3A/L3B) - -- (W2.A) Tag mutability for production-bound references. ✅ RESOLVED (see PROJECT.md) — Path B (tag for dev/qa, SHA for prod) with platform-provided CLI to resolve tag → SHA. - -### From Wave 3 (Technical Execution) - -- (W3.D) L1/L2 standard versioning details. ✅ RESOLVED (see PROJECT.md) — semver (interface→MAJOR, behavior→MINOR, lifecycle→PATCH); L2 pins L1 by `name@semver`; MAJOR bump = new registry entry + 12-month deprecation. - -- (W3.E) Schema mandatory vs. optional inputs. ✅ RESOLVED (see PROJECT.md) — per-env mandatory table (dev: stack+environment; qa adds validation.e2eSuite+loadTest; prod adds runbook+dashboard+oncall; dr adds drDrillRef); `inputs` always optional; `profile: agentic` fields optional everywhere. - -### From Beyond Architecture - -- (BA.A) Skill catalog. ✅ RESOLVED (see PROJECT.md) — 5 skills (web API, worker, scheduled job, static asset, basic observability bootstrap); addition criteria locked. - -- (BA.B) Confidence signal threshold tuning. ✅ RESOLVED (see PROJECT.md) — frozen for v1; tuning begins v1.2 (quarterly FP/FN; override = Infra & Ops + SRE joint sign-off). - -- (BA.C) On-call and operational ownership. ✅ RESOLVED (see PROJECT.md) — platform on-call = Infra & Ops; L3A/L3B halt → Sev2; consumer outage → Sev1. - -- (BA.D) Cost and capacity governance. ✅ RESOLVED (see PROJECT.md) — FinOps owns; per-contract monthly reporting; hard halt at 120% of declared budget via the confidence signal; override = FinOps + SRE joint sign-off. - -- (BA.E) Consumer onboarding. ✅ RESOLVED (see PROJECT.md) — developer (L3A): getting-started → contract schema + central pipeline template; citizen (L3B): scoped agent + skill catalog; both end in a sandbox dev submission that must pass the confidence gate. - -- (BA.F) Cross-platform evolution. ✅ RESOLVED (see PROJECT.md) — contract schema, stack, PolicyCheckResult, confidence signal, audit stream are portable; forge-specific code = workflow YAML, OIDC trust, CODEOWNERS, Environments; a second forge needs a forge adapter + workflow-template translator. - -- (Q1.3) OpenTofu timing. ✅ RESOLVED (deferred — see PROJECT.md) — not in v1 or v1.1; the substrate abstraction makes OpenTofu a future adapter, not an architecture change. - -## 14. Document Status and Next Steps - -Status: **v1.0**. All 11 open items in §13 are resolved. The architecture is -internally consistent; the v1.1 implementation spike (ACDL Phases 08-10) -validates the locked substrate abstraction + contract→stack→adapter path -against real AWS via a per-run-rotated key (D-039; OIDC deferred to v1.2). -The v1.2 build-out (S3 Object Lock, JWS, HITL wiring, L3B skill catalog, -Kyverno/OPA, real OIDC federation, multi-region) is design-authored in -Phase 07 and implemented post-spike. - -Doc-sync items (out of scope of this document but flagged for the same change set): - -- The CDLC reference document's environment model assumes staging exists. Path A invalidates that. The CDLC contract example's targetEnvironments: [staging, production] must be revised to [dev, qa, prod, dr]. - -## 15. Phase 07 authored artifacts - -The 11 resolutions are recorded in `PROJECT.md` (decisions D-034..D-046 + -the "Open-decision resolutions" table). Phase 07 formalizes the locked -commitments into these schema/design files (landed in Waves 2-4 of -Phase 07): - -| REQ | File | Owner persona | -|-----|------|--------------| -| REQ-17 | `schemas/stack.schema.json` | platform-engineer | -| REQ-18 | `schemas/policy_check_result.schema.json` + `adapters/terraform/policy/checkov_adapter.py` | security-engineer | -| REQ-19 | `platform/confidence_signal.py` | backend-engineer + security-engineer (co-authored) | -| REQ-20 | `platform/audit_ledger_design.md` | security-engineer | -| REQ-21 | `platform/hitl_matrix_design.md` + `platform/separation_of_duties.py` | security-engineer | -| REQ-22 | `schemas/contract.schema.json` | backend-engineer | - -The spike scope (D-041, D-043) vs v1.2 build-out boundary for each design -is documented in the respective file. \ No newline at end of file diff --git a/docs/architecture.md b/docs/architecture.md index ae01ea1..f2df1bf 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,452 +1,241 @@ -# Architecture Document v0.2 + Executive Demo Plan +# Architecture -Three things to set up before I deliver the document, because they determine how I write the doc: - -1. What is locked from the resolution session. Eight items: environment model (Path A — dev-only autonomous, no staging), substrate abstraction (Target Stack + adapter pattern, Terraform adapter in v1), policy toolchain (Checkov for Terraform plan, Kyverno for K8s, OPA last resort), separation of duties (CODEOWNERS for routing + DynamoDB outbox for identity distinctness), policy normalization PolicyCheckResult schema with engine adapters), HITL matrix (full 8-concern matrix with evidence, freshness, source), HITL timeout (1d warn, 2d freeze), HITL rollback (pre-execution model, audit chain extended, no partial deploy). - -2. What is still open after the session. Eleven items, listed in the updated Section 13. They are the gating items for v1.0. - -3. Doc-sync item. The CDLC reference document (not in scope of this architecture) describes staging as an environment and uses targetEnvironments: [staging, production] in its contract example. Path A invalidates that. The CDLC document is the responsibility of its own owners, but this architecture flags the inconsistency so the CDLC revision can be scheduled in the same change set. - -The architecture document is below, followed by the executive demo plan, followed by the final three questions I need resolved before v1.0. - ---- - -# Agentic Cloud Delivery Platform — Architecture Document - -Status: Draft v0.2. Section 13 lists 11 open decisions. Eight of the original 15 open items are now locked; seven remain from Section 13 plus four new sub-decisions surfaced during resolution. - -Companion to: Agentic Cloud Delivery Vision [1]. - -Authoring principle: The vision is the source of truth for why [1]; this document is the source of truth for how. Where the two conflict, the vision wins. - -Resolution session log (this revision): - -| ID | Question | Resolution | -|---|---|---| -| Q1 | Environment model | Path A locked. Dev is the only autonomous environment. QA HITL at qa. SRE HITL at prod and dr. Staging does not exist. | -| Q1.2 | Substrate trajectory | Substrate abstraction locked. L1/L2 are defined against a Target Stack. Substrate adapters compile the stack to a substrate execution plan. v1 ships only the Terraform adapter. | -| Q1.3 | OpenTofu timing | 🟡 OPEN (W3.D-adjacent). No specific version or trigger committed. | -| Q2.1 | Policy toolchain | Locked. Checkov for Terraform plan policy. Kyverno for K8s-native and platform-internal policy. OPA/Rego reserved for cross-resource cases; explicitly last resort due to Rego complexity. | -| Q2.2 | Separation of duties | Locked. GitHub CODEOWNERS routes the right reviewer to the right environment. Platform-internal identity record in DynamoDB outbox enforces qaApprover ≠ prodApprover for the same contract. | -| Q2.3 | Policy normalization | Locked. PolicyCheckResult JSON schema is the contract between engines and the confidence signal. Engine-specific adapters translate native output to the schema. | -| Q3 | HITL matrix + timeout + rollback | Locked (full 8-concern matrix in §10). Pre-execution gate model. 1 business day = warn + escalate. 2 business days = auto-freeze + re-submit. Rejection extends the audit chain, no partial deploy to roll back. | -| W1.A | AI-refinement trigger | 🟡 OPEN. Recommendation pending sign-off. | -| W1.B | Multi-stack edge case rule | 🟡 OPEN. Recommendation pending sign-off. | -| W2.A | Tag mutability for prod | 🟡 OPEN. Recommendation pending sign-off. | -| W3.D | L1/L2 standard versioning details | 🟡 OPEN. | -| W3.E | Schema mandatory vs. optional inputs | 🟡 OPEN. | -| BA.A–F | Beyond-architecture questions | 🟡 OPEN (6 items). | - ---- +> **Status:** v1.0 (current). All design decisions are resolved. This is the +> source of truth for *how* the platform works; the [Vision](vision) is the +> source of truth for *why*. ## 0. Purpose -This document encodes the architectural commitments that realize the vision [1]. The resolution session has closed eight open items; the document is now at v0.2 with eleven open items remaining, listed in Section 13. Every locked commitment is grounded in either a vision tenet or a specific decision made during resolution. +This document encodes the architectural commitments that realize the +[vision](vision). Every commitment is grounded in a vision tenet. -The structure remains: four layers (L1 primitives, L2 composed stacks, L3A developer surface, L3B agentic surface) plus five cross-cutting concerns (central pipeline, contract schema, confidence signal, audit stream, HITL mechanics), with one addition: the substrate abstraction layer (Section 12) is now a first-class architectural concern, not an implementation detail. +The platform is **four layers + six cross-cutting concerns**, bound by the +vision's "Two Consumer Surfaces, One Platform" tenet: both surfaces converge +on the same contract schema, the same policy envelope, and the same evidence +stream. ## 1. Architectural Overview -The platform remains four layers and five cross-cutting concerns. The substrate abstraction is added as a sixth cross-cutting concern in Section 12 because it is the binding constraint for the L1/L2 model, the central pipeline, and the policy toolchain. +```mermaid +flowchart TD + A["Consumer surfaces"] --> B["Contract schema"] + B --> C["Central pipeline"] + C --> D["Modules + primitives"] + C --> E["Substrate adapter"] + C --> F["Confidence signal"] + C --> G["Evidence stream"] + D --> E + E --> H["Infrastructure"] + F --> G +``` -The vision's "Two Consumer Surfaces, One Platform" tenet [1] remains the constraint that binds all concerns: L3A and L3B converge on the same contract schema, the same policy envelope, and the same evidence stream. +The four layers: -Locked additions this revision: +1. **Primitives** — single-purpose, substrate-agnostic modules representing + the smallest reusable infrastructure pieces (a VPC, an S3 bucket, an ECS + cluster). A primitive does not reference other primitives; it takes its + environment as input. +2. **Modules** — patterns that combine primitives into deployable + infrastructure shapes (an ECS Fargate microservice, a static-asset site). + A module references registered primitives (max depth 5). +3. **Developer surface** — the developer-owned workflow file + contract. The + developer references the central pipeline via a versioned tag and owns + their workflow file (no platform auto-sync). +4. **Agentic surface** — a hybrid runtime where a consumer declares intent + in natural language and an agent resolves it to a contract submission. + Trust model: trust and always verify on the platform side. Stateless + agents; all state lives in the platform. -- The environment model is dev (autonomous) → qa (QA HITL) → prod (SRE HITL) → dr (SRE HITL). Staging does not exist. +The developer and agentic surfaces are parallel paths, not a progression. +Both end in a contract submission that enters the same pipeline. -- L1/L2 are substrate-agnostic in shape; substrate adapters are the only substrate-specific component. +## 2. Primitives -## 2. Layer 1 — Foundational Primitives - -Purpose. Single-purpose, substrate-agnostic primitive modules representing the smallest reusable infrastructure pieces. L1 modules do not compose with other L1 modules; L1 takes its environment as input. - -Locked commitments (unchanged from v0.1): - -- No inter-L1 references. L1 may call Terraform data sources. - -- Semver with three triggers (interface → MAJOR, behavior → MINOR, lifecycle → PATCH). +Single-purpose, substrate-agnostic modules. Locked commitments: +- No inter-primitive references. A primitive may call substrate data sources. +- Semver with three triggers: interface → MAJOR, behavior → MINOR, + lifecycle → PATCH. - Immutability on publication. - - 12-month deprecation window. +- AI refinement is a flag, triggered by a joint operational condition + (N ≥ 50 consecutive zero-rollback changes, no primitive/module incident in + 6 months, Infra & Ops unilateral override). +- A primitive's interface is defined against the Target Stack (substrate- + agnostic), not against any substrate's variable block directly. -- AI refinement is a flag. +## 3. Modules -🟡 OPEN (W1.A): AI-refinement operational trigger. The criterion for flipping aiRefinement from false to true needs a falsifiable operational signal. Recommendation: joint condition — N ≥ 50 consecutive changes with zero rollbacks AND no L1/L2 incident in the last 6 months AND Infra & Ops holds a unilateral override. Pending sign-off. - -🟡 OPEN (sub-decision surfaced this revision): The L1 module's interface field is defined against the Target Stack, not against Terraform's variable block directly. In v1, the stack is shaped to round-trip cleanly to Terraform, but the schema is substrate-agnostic. Pending v1 implementation details in Section 12. - -## 3. Layer 2 — Composed Stacks - -Purpose. Combine L1 primitives into deployable infrastructure shapes. Each codebase maps to one canonical L2 stack; the stack is either a parameterized module (Shape X) or a thin-composition layer (Shape Y). - -Locked commitments (unchanged from v0.1): - -- 1 codebase = 1 L2 stack (default), with multiStack: true for exceptions. - -- Shape X or Shape Y. - -- Hierarchical composition, max depth 5, only registered L1s. - -- Pipeline quality checks: secrets-in-plaintext, public ingress, IAM wildcard, KMS key reference, tag compliance, naming convention. - -- Restricted from thin-composition: IAM principal creation, network boundary creation, key/secret creation, external data transfer. +Patterns that combine primitives into deployable shapes. Locked commitments: +- One codebase maps to one canonical module (default); `multiStack: true` + is permitted only for (a) a DR-region mirror, (b) a time-boxed + experimental stack (TTL ≤ 30 days), or (c) explicit Infra & Ops approval + with a documented justification. +- A module references registered primitives only (max depth 5). +- Pipeline quality checks: secrets-in-plaintext, public ingress, IAM + wildcard, KMS key reference, tag compliance, naming convention. +- Restricted from module patterns: IAM principal creation, network boundary + creation, key/secret creation, external data transfer. - Auto-promote after 3 observed usages. +- A module's pattern tree wires field is defined against the stack's + relationship type, not against any substrate's module block. The stack → + substrate translation is the substrate adapter's job (§12). The pattern + pipeline itself is substrate-agnostic. -🟡 OPEN (W1.B): Multi-stack edge case rule. The multiStack: true exception needs a falsifiable rule. Recommendation: permitted only for (a) DR-region mirror of the primary stack, (b) time-boxed experimental stack with TTL ≤ 30 days, (c) explicit Infra & Ops approval for a documented reason captured in multiStack.justification. Pending sign-off. - -🟡 OPEN (sub-decision surfaced this revision): The L2 composition tree's wires field is defined against the stack's relationship type, not against a Terraform module block. The stack → Terraform translation is the Terraform adapter's job (Section 12). The composition pipeline itself is substrate-agnostic. - -## 4. Layer 3A — Developer Consumer Surface - -Locked commitments (unchanged from v0.1): +## 4. Developer Surface - Tag-based reference to the central pipeline template. - - Developer-owned workflow file, no platform auto-sync. +- Tag mutability for production-bound references: tag for dev/qa, SHA for + prod. The platform provides a CLI command that resolves the current tag + to its SHA for prod-bound workflows. -- L3A and L3B are parallel paths, not a progression. - -🟡 OPEN (W2.A): Tag mutability for production-bound references. Path A (tag throughout with protection) vs. Path B (tag for dev/qa, SHA for prod). Recommendation: Path B, justified by the vision's "Audit truth lives outside the repository" bet [1] and the "Not a mutable audit log" anti-goal [1]; SHA-pinning is the only guarantee that the exact bytes reviewed in dev/qa are the bytes deployed to prod. The platform provides a CLI command that resolves the current tag to its SHA for prod-bound workflows. Pending sign-off. - -## 5. Layer 3B — Agentic Consumer Surface - -Locked commitments (unchanged from v0.1): - -- Hybrid runtime, skill as markdown, agent as executor. +## 5. Agentic Surface +- Hybrid runtime: skill as markdown, agent as executor. - Trust model: trust and always verify on the platform side. - - Skill envelope (4 dimensions). - -- Stateless agents, all state in the platform. - -Environment progression — locked (this revision): - -| Environment | Autonomy | Attester | Gate | -|---|---|---|---| -| dev | Full autonomy (no HITL) | — | Confidence signal ≥ 0.50, all six inputs present | -| qa | Held for attestation | QA | GitHub Deployment approval + full QA matrix (see §10) | -| prod | Held for attestation | SRE | GitHub Deployment approval + full SRE matrix (see §10) | -| dr | Held for attestation | SRE | GitHub Deployment approval + dr-drill evidence (see §10) | - -Staging is removed. Dev is the only autonomous environment and absorbs integration, contract, security smoke, and performance smoke validation. The CDLC reference document's environment model is a doc-sync item flagged at the top of this document. - -Profile marker: profile: agentic unlocks L3B-specific fields naturalLanguageIntent, confidenceAtSubmission, agentTrace). - -🟡 OPEN (BA.A): Skill catalog. Which skills exist in the initial L3B capability set, who decides what gets added, how are skills deprecated. Pending resolution. +- Stateless agents; all state in the platform. +- Initial skill catalog: web API, worker, scheduled job, static asset, + basic observability bootstrap. Addition criteria: (a) reviewable for + sensitive data, (b) expressible as a single contract submission, + (c) documented use case. +- `profile: agentic` unlocks agentic-specific fields + (`naturalLanguageIntent`, `confidenceAtSubmission`, `agentTrace`). ## 6. Cross-Cutting — Central Pipeline Template -Locked commitments (unchanged from v0.1): - -- JSON Schema (draft 2020-12) with thin domain-specific wrapper. - +- JSON Schema (draft 2020-12) with a thin domain-specific wrapper. - Central repo + generated client libraries. - -- Multi-stage validation pipeline (schema → policy → NFR → confidence). - +- Multi-stage validation pipeline: schema → policy → NFR → confidence. - Distributed enrichment. - -- GitOps reconciler + Terraform execution layer. - -Locked additions this revision: - -- The GitOps reconciler is the platform's K8s API. The cdlc-gitops repository's state materializes into K8s CRDs (ArgoCD Applications or Flux Kustomizations) that the reconciler watches. This is the platform's internal state surface. - -- The pipeline emits a PolicyCheckResult record per policy rule evaluated. The confidence signal consumes these as one normalized input (Section 8). - -🟡 OPEN (W3.D): L1/L2 standard versioning details — semver scheme, pin model, evolution compatibility contract. - -🟡 OPEN (W3.E): Schema mandatory vs. optional inputs — which are required for all consumers, which are required only for higher environments, which are always optional. +- GitOps reconciler + substrate execution layer. +- The pipeline emits a `PolicyCheckResult` record per policy rule evaluated; + the confidence signal consumes these as one normalized input (§8). ## 7. Cross-Cutting — Contract Schema -Locked commitments (unchanged from v0.1): - - Central repo + generated client libraries. - -- Strict fail-fast at schema stage, multi-stage validation pipeline with reason codes from a published vocabulary. - -🟡 OPEN (W3.E): Schema mandatory vs. optional inputs. The CDLC reference contract example [1] is illustrative; the v1 contract schema needs explicit per-field mandatory/optional declarations per environment. +- Strict fail-fast at the schema stage with reason codes from a published + vocabulary. +- Per-environment mandatory fields: dev requires stack + environment; qa + adds `validation.e2eSuite` + `validation.loadTest`; prod adds runbook + + dashboard + oncall; dr adds `drDrillRef`. `inputs` is always optional. + `profile: agentic` fields are optional everywhere (`naturalLanguageIntent` + required when profile is agentic). ## 8. Cross-Cutting — Confidence Signal -Locked commitments (unchanged from v0.1): - -- Six canonical inputs. - +- Six canonical inputs: policy, validation, freshness, source, history, NFRs. - Weighted sum with per-input breakdown. - - Per-environment thresholds: dev ≥ 0.50, qa ≥ 0.75, prod ≥ 0.90, dr ≥ 0.95. - -- Structured output { score, band, perInput, reasonCodes }. - -- 1-year storage, no algorithm retraining in v1. - +- Structured output: `{ score, band, perInput, reasonCodes }`. +- 1-year storage; no algorithm retraining in v1. - Halt with explicit reason on missing input. - -Locked additions this revision: - -- The policy check results input is a list of PolicyCheckResult records from the normalized schema (Section 9, 12). The signal does not know which engine produced which result. - -- Severity → score penalty mapping: critical → hard override to mandatory block, high → -0.2, medium → -0.05, low → -0.01, info → 0.0. One critical finding hard-overrides the score regardless of all other inputs. - -🟡 OPEN (BA.B): Threshold tuning policy. The initial thresholds (dev 0.50, qa 0.75, prod 0.90, dr 0.95) are starting values. The tuning process, false-positive/false-negative tracking, and override authority are pending. +- Severity → score penalty: critical → hard override to mandatory block, + high → -0.2, medium → -0.05, low → -0.01, info → 0.0. One critical finding + hard-overrides the score regardless of all other inputs. +- Thresholds frozen for v1; tuning begins post-v1 with quarterly FP/FN + tracking per environment. Override authority = Infra & Ops + SRE joint + sign-off; any override is itself a confidence-event in the audit stream. ## 9. Cross-Cutting — Audit and Evidence Stream -Locked commitments (unchanged from v0.1): - -- Tiered audit ledger: S3 with Object Lock in compliance mode (cold, source of truth, 7-year retention) + GitHub audit repo (hot, query index, not part of the chain). - -- Daily checkpoints. - -- Event schema: JWS detached signature, prev_event_hash chain, controlled-vocabulary event_type. - -- Outbox pattern with local durable outbox + async worker. - -- Linkage via workflow run ID or agent invocation ID. - -Locked additions this revision: - -- The outbox database is DynamoDB. RPO is zero (synchronous write to local outbox before contract submission ack); RTO is the async worker's recovery from the dead-letter queue. Single-region in v1; multi-region is a v2 concern. - -- The outbox also stores the per-contract QA and prod approver identities (Section 10). The platform-internal identity-distinctness check reads from this outbox. This is the only durable record of the approver identities outside GitHub's audit log. - -🟡 OPEN (BA.C): On-call and operational ownership. The platform's on-call rotation, escalation paths when L3A or L3B halts unexpectedly, and the relationship to consumer on-call. - -## 10. Cross-Cutting — Human-in-the-Loop Mechanics - -Purpose. The human gates at higher environments. The vision's "Lower Environments are Autonomous; Higher Environments are Attested" tenet [1] and the "deliberate human attestation — not as a rubber stamp" requirement [1] are the binding constraints. - -### 10.1 Gate model - -Pre-execution gates. The contract is held in a "validated but not applied" state until the human attests. qa, prod, and dr are PR-based attestation gates backed by GitHub Environments with required reviewers. - -For qa and prod, there is no partial deployment to roll back on rejection. For dr, the same model — promotion to the DR environment is a separate GitHub Deployment, gated by SRE, against a separate cluster/region. The canary/deployment-rollback model is explicitly not in scope for v1. - -### 10.2 Reviewer routing - -GitHub CODEOWNERS + GitHub Environment required reviewers. qa → QA team; prod → SRE team; dr → SRE team. CODEOWNERS is the routing layer; it does not enforce identity distinctness. - -### 10.3 Separation of duties — identity distinctness - -Mechanism is platform-internal, not GitHub-native, not Kyverno (in v1). - -Sequence: - -1. On promotion dev → qa, the platform reads the QA approver's GitHub identity from the GitHub Deployment approval event and writes it to the DynamoDB outbox keyed by contractId. - -2. On promotion qa → prod, the platform reads the stored QA approver identity from the outbox and the new SRE approver identity from the GitHub Deployment approval event. - -3. If qaApprover == prodApprover, the platform blocks the prod promotion, writes a SEPARATION_OF_DUTIES_VIOLATION event to the evidence stream, and routes a halt artifact to the SRE on-call. - -4. The check is implemented in the central pipeline repo, not as an external policy. The platform is the only writer to the outbox; the check is in the same process that has authority to block the promotion. - -### 10.4 Full HITL attestation matrix - -| Env | Concern | Evidence artifact | Freshness | Source | Attester | -|---|---|---|---|---|---| -| qa | Functional correctness | Last successful run of contract-declared validation.e2eSuite with pass rate ≥ 99% | Last 24h | Test runner declared in contract | QA | -| qa | Performance baseline | Load test report (k6 / Gatling / Locust) showing p99 latency < declared NFR and throughput > declared minimum | Last 7d | Load test runner declared in contract | QA | -| qa | Security posture | Vulnerability scan (Trivy, Snyk, or contract-declared equivalent) with no criticals/highs, signed by Security on-call | Last 24h | Security scanner + Security team signature | QA | -| qa | Contract NFRs | Platform-generated report: schema valid, NFR assertions (latency, throughput, error rate) within declared bounds | At submission | Platform contract validator | QA | -| prod | Operational readiness | Runbook published, dashboard exists, on-call rotation assigned, alerts configured | At submission, validated against last 30d history | Platform + SRE | SRE | -| prod | Incident response | Sev-1 runbook tabletop or live drill completed | Last 90d | SRE drill record | SRE | -| prod | Capacity / cost | FinOps forecast for next 30d within budget envelope, cost anomaly baseline stored, budget alert configured | Forecast valid for next 30d | FinOps + SRE | SRE | -| prod | Resilience | DR drill, chaos engineering report, backup verified | DR: 180d; chaos: 90d; backup: 30d | SRE + Platform | SRE | -| dr | dr-region deploy with the most recent prod-bound dr drill as canary evidence | dr drill report | Last 180d | SRE | SRE | - -### 10.5 Timeout behavior - -| Time | State | Action | -|---|---|---| -| Submission | PENDING_ATTESTATION | Notify responsible team | -| 1 business day | PENDING_ATTESTATION_WARNING | Notify team + platform on-call (elevated path); emit PENDING_ATTESTATION_TIMEOUT_WARNING event | -| 2 business days | PENDING_ATTESTATION_AUTO_FREEZE | Auto-freeze; require re-submission; emit PENDING_ATTESTATION_AUTO_FREEZE event; new submission linked via supersedes | - -### 10.6 Rejection and rollback - -Rejection returns the contract to a HELD state with the rejection reason captured as a PROMOTION_REJECTED event. The consumer fixes the cause and re-submits; the new submission is linked to the rejected one via supersedes. The audit chain is extended, not torn up — matching the resolution session's answer. - -There is no partial deployment to roll back at any v1 gate. - -## 11. Cross-Cutting — Agentic Stack - -Locked commitments (unchanged from v0.1): - -- Hybrid runtime, platform-managed control plane + consumer-owned agent. - -- Versioned, signed skill catalog over MCP. - -- Skill envelope enforced on invocation and result submission. - -- Consumer-owned skill execution environment. Platform does not run the skill. - -- Stateless agents, all state in the platform. - -Locked additions this revision: - -- Skills are reviewed for sensitive data before release. Secrets, customer data, internal IPs, and other sensitive payloads are forbidden in skill markdown. The review is owned by Infra & Ops and is the mandatory release gate for any new skill. This is the trade-off for accepting the L3B runtime threat model (skill content is consumer-readable, so the platform must not put anything sensitive in it). - -🟡 OPEN (BA.A): Skill catalog. Initial skill set, addition process, deprecation process. - -## 12. Cross-Cutting — L1/L2 Substrate Execution - -Purpose. The technical execution layer for the L1/L2 substrate, including the substrate abstraction that protects v1 from polyglot mess while leaving v2+ room to grow. - -### 12.1 Substrate abstraction (locked this revision) - -L1/L2 are substrate-agnostic in shape. The architecture defines a Target Stack — a substrate-neutral description of: - -- Resources with typed input contracts, typed output contracts, and declared NFRs. - -- Relationships (single parent per child, with a shared keyword for multi-relationship dependencies). - -- Composition (a tree of resources with max depth 5). - -- Policy hooks (the points in the composition where policy checks attach). - -The L1 registry, the L2 composition tree, the YML standard, and the policy check result schema are all defined against the stack schema. None of them is defined against any specific substrate. - -Substrate adapters are the only substrate-specific code. An adapter compiles the stack into a substrate execution plan. v1 ships exactly one adapter: the Terraform adapter. v2+ may add additional adapters (OpenTofu, Pulumi, K8s CRDs) without architectural change. - -v1 implementation reality: the stack is shaped to round-trip cleanly to Terraform because there is no other adapter to differentiate from. The stack and the Terraform output are nearly isomorphic in v1. As additional adapters appear in v2+, the stack gets more expressive (e.g., substrate-specific output types) and the adapters gain translation logic, but the L1 module content, the YML standard, and the composition tree do not change. This is the design that prevents the polyglot mess. - -Why not build the abstraction earlier? Building a substrate-agnostic stack before there is a second adapter to test against is speculative generality. The v1 commitment is: (1) the L1 module interface is defined against the stack schema even though the only adapter is Terraform, and (2) the central pipeline, registry, and policy schema consume the stack-typed contracts. The adapter is the only place where substrate terminology appears in v1. - -### 12.2 Terraform adapter (v1) - -The Terraform adapter: - -- Translates the stack-typed L1 module interface to a Terraform variable block and a Terraform output block. - -- Translates the stack-typed L2 composition tree to a Terraform root module that calls the L1 modules. - -- Translates the stack-typed relationships to Terraform module references. - -- Emits a Terraform plan from the stack. - -The adapter is a thin layer. It does not own L1/L2 content; it only translates. - -### 12.3 State storage - -Locked: S3 (state files) + DynamoDB (state locking), cloud-managed. Single-region in v1. - -### 12.4 Policy toolchain - -Locked: - -- Checkov for Terraform plan policy (the four L2 thin-composition checks: secrets-in-plaintext, public ingress, IAM wildcard, KMS key reference, plus tag and naming convention). Checkov is open-source, has a broad rule catalog, and is GitOps-friendly. - -- Kyverno for K8s-native policy (platform-internal state in the GitOps reconciler, separation-of-dues-adjacent checks if any are added in v2, future CRD validation). - -- OPA/Rego is reserved for cross-resource policy and is explicitly last resort due to Rego complexity. - -### 12.5 Execution layer - -Locked: GitHub Actions. terraform plan and terraform apply run in the central pipeline repo's GitHub Actions workflow. State locking via DynamoDB. AWS credentials via OIDC federation (long-lived credentials are forbidden). The platform does not run terraform apply against a developer's workstation; all execution is in the central pipeline. - -### 12.6 Policy result normalization (locked this revision) - -The confidence signal does not consume raw Checkov or Kyverno output. It consumes a normalized PolicyCheckResult schema produced by substrate-specific adapters. - -Schema (canonical form, lives in the central pipeline repo): - -```json -{ - "contractId": "uuid", - "evaluatedAt": "ISO-8601", - "engine": "checkov | kyverno | opa", - "ruleId": "CKV_AWS_24 | KYVERNO_NO_PRIVILEGED | ...", - "severity": "critical | high | medium | low | info", - "result": "pass | fail | skipped | error", - "message": "human-readable", - "evidence": { "...engine-specific payload, opaque to the signal..." }, - "resourceRef": "stack-typed resource identifier" -} -``` - -The Checkov adapter runs in the same GitHub Actions step as Checkov itself and translates Checkov JSON to PolicyCheckResult records. The Kyverno adapter runs as a controller in the platform's K8s cluster and translates Kyverno PolicyReport CRDs to PolicyCheckResult records. The confidence signal's policy input component is the union of all PolicyCheckResult records, regardless of engine. The signal does not know which engine produced which result — substrate-agnostic over its inputs, matching the L1/L2 model's substrate-agnostic over its outputs. - -### 12.7 Registry maintenance - -Locked: L1 module publication updates the L1 registry in the same PR as the module. Registry and module land together. The registry is the stack-typed contract, not a Terraform-specific variable schema. The L1 registry, the central pipeline, and the policy schema all consume the same stack-typed contract — there is one source of truth for the L1 interface, not multiple substrate-specific copies. - -### 12.8 Contract-schema-to-stack resolution - -The contract schema declares the consumer's intent in stack-typed terms. The central pipeline resolves the contract to a target stack (a list of L1 module instances with their inputs and the relationships between them). The Terraform adapter compiles the target stack to a Terraform execution plan. This resolution is substrate-agnostic — the target stack is in the stack schema. - -🟡 OPEN (W3.D): L1/L2 standard versioning details, including pin model and evolution compatibility contract. - -## 13. Consolidated Open Design Decisions - -The following 11 decisions remain open. They are the gating items for v1.0. - -### From Wave 1 (L1/L2 Substrate) - -- (W1.A) AI-refinement trigger. Recommendation: joint condition — N ≥ 50 consecutive changes with zero rollbacks AND no L1/L2 incident in last 6 months AND Infra & Ops unilateral override. Pending sign-off. - -- (W1.B) Multi-stack edge case rule. Recommendation: permitted only for (a) DR-region mirror, (b) time-boxed experimental stack with TTL ≤ 30d, (c) explicit Infra & Ops approval with documented justification in multiStack.justification. Pending sign-off. - -### From Wave 2 (L3A/L3B) - -- (W2.A) Tag mutability for production-bound references. Recommendation: Path B (tag for dev/qa, SHA for prod) with platform-provided CLI to resolve tag → SHA. Pending sign-off. - -### From Wave 3 (Technical Execution) - -- (W3.D) L1/L2 standard versioning details. Semver scheme, pin model, evolution compatibility contract. - -- (W3.E) Schema mandatory vs. optional inputs. Per-field mandatory/optional declarations per environment. - -### From Beyond Architecture - -- (BA.A) Skill catalog. Initial L3B skill set, addition process, deprecation process. - -- (BA.B) Confidence signal threshold tuning. Initial thresholds are starting values; tuning process, FP/FN tracking, override authority. - -- (BA.C) On-call and operational ownership. Platform on-call rotation, escalation paths, relationship to consumer on-call. - -- (BA.D) Cost and capacity governance. Cloud cost ownership, consumption reporting, runaway spend detection and halting. - -- (BA.E) Consumer onboarding. Developer and citizen developer onboarding flow, "getting started" path through the contract schema. - -- (BA.F) Cross-platform evolution. What changes if a second source-control system (e.g., GitLab) is added; which architectural decisions are portable. - -## 14. Document Status and Next Steps - -Status: v0.2. Eight of the original 15 open items are locked. Eleven remain open. The architecture is internally consistent for the locked items; resolution of the open items is the path to v1.0. - -Doc-sync items (out of scope of this document but flagged for the same change set): - -- The CDLC reference document's environment model assumes staging exists. Path A invalidates that. The CDLC contract example's targetEnvironments: [staging, production] must be revised to [dev, qa, prod, dr]. - -To finalize to v1.0: - -1. Resolve the 11 open items in Section 13. - -2. Validate the locked substrate abstraction against a real v1 implementation spike (one L1 module, one L2 composition, one Terraform adapter, one contract submission end-to-end). The spike validates that the stack commitments do not require a polyglot mess. - -3. Validate the locked HITL matrix against a tabletop exercise with QA and SRE. - -4. Sign-off pass. - ---- - -# Final Asks — three remaining open questions, then sign-off - -I have three open questions that gate v1.0. Resolve them and I will revise the architecture document to v1.0 and mark it ready for implementation. - -Q1. W1.A + W1.B — AI-refinement trigger and multi-stack edge case rule. The recommendations are in the document. Do you accept them as committed, or do you want to amend? - -Q2. W2.A — Tag mutability for production-bound references. Path A (tag throughout with protection) vs. Path B (tag for dev/qa, SHA for prod). My recommendation is Path B with a platform CLI to resolve tag → SHA. Accept or amend? - -Q3. BA.A — Initial L3B skill catalog. The demo plan uses 3 stub skills deploy-web-api, add-observability, add-basic-auth). For v1.0, the real platform needs a defensible initial skill set. My recommendation: start with the 5 most common infrastructure intents (web API, worker, scheduled job, static asset, basic observability bootstrap) and grow from there. The criteria for addition: a skill must (a) be reviewable for sensitive data per the locked skill-review gate, (b) be expressible as a single contract submission, and (c) have a documented use case. Accept or amend? - -Once these three are resolved, plus the 8 remaining items (W3.D, W3.E, BA.B, BA.C, BA.D, BA.E, BA.F, and the OpenTofu timing sub-decision), the architecture moves to v1.0. - -Sign-off request. Are you ready for me to draft v1.0 once these are resolved, or do you want to amend the v0.2 above first? \ No newline at end of file +- Every delivery action produces an immutable, hash-chained evidence event. +- The audit stream is the platform's certified record of what happened, when, + and why. +- Events are written to a DynamoDB outbox and rendered on an evidence + timeline. + +## 10. Cross-Cutting — HITL Matrix + +Human-in-the-loop gates for higher environments: + +| Environment | Autonomy | Attester | Gate | +|---|---|---|---| +| dev | Full autonomy (no HITL) | — | Confidence ≥ 0.50, all six inputs present | +| qa | Held for attestation | QA | Platform-runner deployment approval + full QA matrix | +| prod | Held for attestation | SRE | Platform-runner deployment approval + full SRE matrix | +| dr | Held for attestation | SRE | Platform-runner deployment approval + dr-drill evidence | + +Staging does not exist. Dev is the only autonomous environment and absorbs +integration, contract, security smoke, and performance smoke validation. + +- Pre-execution gate model. 1 business day = warn + escalate; 2 business + days = auto-freeze + re-submit. Rejection extends the audit chain; no + partial deploy to roll back. +- Separation of duties: the platform-internal identity record in the + DynamoDB outbox enforces `qaApprover ≠ prodApprover` for the same contract. + +## 11. Cross-Cutting — Separation of Duties + +- CODEOWNERS routes the right reviewer to the right environment. +- The DynamoDB outbox enforces identity distinctness across environment + approvers. + +## 12. Cross-Cutting — Substrate Execution + +The technical execution layer. Primitives and modules are substrate-agnostic +in shape; substrate adapters are the only substrate-specific component. + +The architecture defines a **Target Stack** — a substrate-neutral +description of: + +- The resources to create (typed against the stack schema). +- Their relationships (the module's pattern tree). +- Their inputs (wired from the contract). +- Policy hooks (the points in the pattern where policy checks attach). + +The registry, the module pattern tree, the contract schema, and the +`PolicyCheckResult` schema are all defined against the stack schema. None is +defined against any specific substrate. + +**v1 implementation reality:** the stack is shaped to round-trip cleanly to +Terraform because there is no other adapter to differentiate from. As +additional adapters appear, the stack gets more expressive and the adapters +gain translation logic, but the primitive content, the module pattern tree, +and the contract schema do not change. This is the design that prevents a +polyglot mess. + +The substrate adapter: + +- Translates the stack-typed module pattern tree to a substrate root module + that calls the primitive modules. +- Is a thin layer. It does not own primitive/module content; it only + translates. +- Is the only substrate-specific code in the platform. + +Policy checks run on the substrate plan output. Results are normalized to +`PolicyCheckResult` records by a policy adapter. The confidence signal +consumes the union of all `PolicyCheckResult` records, regardless of engine +— substrate-agnostic over its inputs, matching the module model's +substrate-agnosticism over its outputs. + +## 13. Cross-Cutting — Platform Runners + +The platform runs on platform-managed runners (GitHub Actions in +production). Runner-specific code = workflow YAML, OIDC trust, CODEOWNERS, +environments. The contract schema, stack, `PolicyCheckResult`, confidence +signal, and audit stream are portable (runner-agnostic); a second runner +platform needs a runner adapter + workflow-template translator, with no +change to the modules/stack/confidence/audit. + +## 14. Versioning + +- Primitives and modules use semver: interface → MAJOR, behavior → MINOR, + lifecycle → PATCH. +- A module pins primitives by `name@semver`; the resolver picks the highest + compatible. +- A MAJOR bump requires a new registry entry (immutable publication); the + old entry enters a 12-month deprecation window. +- The central deploy pipeline is referenced by a floating MAJOR + MINOR tag + (e.g. `@v1.4`); patch fixes flow within the tag, breaking changes land + under the next MINOR tag. + +See [Versioning](pipeline/versioning) for the consumer-facing details. + +## 15. OpenTofu + +Not in v1. The substrate abstraction (§12) makes OpenTofu a future adapter, +not an architecture change. Revisit when an OpenTofu adapter is requested. \ No newline at end of file diff --git a/docs/consumer-guide.md b/docs/consumer-guide.md new file mode 100644 index 0000000..ddd9665 --- /dev/null +++ b/docs/consumer-guide.md @@ -0,0 +1,313 @@ +# Consumer Guide — Declare intent, deploy to AWS + +This guide walks a consumer through creating their pipeline and defining a +contract that deploys any ACDL module to AWS. It is **generic** across all +modules in the registry; `static-asset` is the worked example, but every +step applies to `microservice` and any future module. + +## The model + +Consumers have their own repos and consume ACDL by referencing `uses:` the +central pipeline definitions. The consumer declares a **contract** (which +module, which environment, which inputs); the ACDL platform owns the +pipelines, modules, substrate adapter, and evidence stream. + +You do not write infrastructure modules, workflow YAML, or adapter code. +You write a contract YAML file and the platform does the rest. Your +repository contains only your application code, your contracts, and your CI +definitions. + +```mermaid +flowchart LR + A["your repo
(app code + contracts + CI definitions)"] -->|uses: acdl/.github/workflows/deploy.yml@v1.4| B + B["platform runners
(modules + pipelines + adapters + schemas)"] -->|contract -> resolver -> stack -> adapter
-> security checks -> infrastructure plan -> policy checks
-> confidence -> apply -> evidence event| C + C["your resources in AWS"] +``` + +## Versioning the `uses:` reference + +The central deployment pipeline is **always versioned with floating MAJOR +and MINOR tags** (e.g. `acdl/pipelines/deploy.yaml@v1.4`). Version +constraints cannot be expressed inside the contract, so the tag in +`uses:` is the only immutability lever a consumer has. See +[Versioning](pipeline/versioning) for the full rationale. + +**Unversioned references are discouraged.** Do not use `@main` or a bare +`acdl/pipelines/deploy.yaml`. + +## Prerequisites + +These are the **only** prerequisites for a consumer repo. You do **not** +need an AWS account, infrastructure tooling, or a runner key — those are +platform-managed. See [Environments](environments/). + +- **A consumer GitHub repository** for your application code + contracts. +- **A platform-managed environment** bound to your repo. The platform team + provisions the AWS account, network, state backend, and IAM role. If no + environment is bound, your first pipeline run emits a friendly onboarding + prompt. See [Environments](environments/). +- **Authorization to reference the central pipeline.** Onboarding grants + your repo the right to `uses: acdl/.github/workflows/deploy.yml@v1.4`. + Contact the platform team if you have not been onboarded. + +## Step 1 — Create a consumer repo + +Create a repository for your application. The top level holds your app +code; your contract lives at `.acdl/contract.yaml`. Example for a static +site: + +``` +my-static-site/ + index.html + assets/ + style.css + logo.png + .acdl/ + contract.yaml + .github/ + workflows/ + deploy.yml +``` + +Example for a microservice: + +``` +my-microservice/ + app.py + Dockerfile + .acdl/ + contract.yaml + .github/ + workflows/ + deploy.yml +``` + +Your app code lives at the top level. Your contract lives at +`.acdl/contract.yaml` regardless of the module you deploy. Your CI +definition lives at `.github/workflows/deploy.yml`. + +## Step 2 — Reference the central pipeline + +In your contract YAML, declare `uses:` pointing at the central ACDL +deployment pipeline with a **versioned tag** (floating MAJOR + MINOR): + +```yaml +uses: acdl/pipelines/deploy.yaml@v1.4 +``` + +This tells the platform to run the standard deployment pipeline: +validate-contract → resolve-stack → security checks → infrastructure plan → +policy checks → confidence → evidence event → apply. + +## Step 3 — Define the contract + +Write `.acdl/contract.yaml`. The `static-asset` example: + +```yaml +uses: acdl/pipelines/deploy.yaml@v1.4 +module: static-asset +environment: dev +inputs: + bucket_name: my-static-site-assets + region: us-east-1 +``` + +A `microservice` example: + +```yaml +uses: acdl/pipelines/deploy.yaml@v1.4 +module: microservice +environment: dev +inputs: + image: my-registry/my-microservice:latest + port: 8080 + env: + LOG_LEVEL: info +``` + +### Contract fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `uses` | string | yes | Reference to the central deployment pipeline, **versioned** with a floating MAJOR+MINOR tag (e.g. `acdl/pipelines/deploy.yaml@v1.4`). Bare or `@main` references are discouraged. See [Versioning](pipeline/versioning). | +| `module` | string | yes | Module name from the registry — any primitive or module (e.g. `static-asset`, `microservice`, `s3`). See the [module catalog](modules/). | +| `environment` | string | yes | The platform-managed environment to deploy to (e.g. `dev`). See [Environments](environments/). | +| `inputs` | object | yes | Module-specific inputs (see the module's README). | + +### Module inputs + +Each module declares its inputs in its `interface.json` (primitives) or +`composition.json` (modules). Consult the [module catalog](modules/) for +the full list, or read the module's own README under `modules/l1//` +or `modules/l2//`. + +The contract is validated against the contract schema. An invalid contract +(missing field, unknown module, wrong type) fails at the validate-contract +stage with a clear error. + +## Step 4 — Run the pipeline + +You do **not** run platform scripts locally for the happy path. The central +deploy workflow is a **reusable workflow** that the platform runners fetch +and execute for you. + +### The consumer CI definition + +Add a thin workflow file to **your** repo that invokes the reusable ACDL +deploy workflow with a **versioned tag** (`.github/workflows/deploy.yml`): + +```yaml +name: deploy +on: + push: + branches: [main] +jobs: + deploy: + uses: acdl/.github/workflows/deploy.yml@v1.4 + with: + contract: .acdl/contract.yaml +``` + +That is the entire consumer-side workflow. When you push to `main`: + +1. The platform runner resolves `uses: acdl/.github/workflows/deploy.yml@v1.4` + to the reusable workflow **at the pinned tag**. +2. A **platform-provided runner** checks out **your** repo. +3. The runner checks out the **ACDL platform repo** into the workspace — + this is how the pipeline fetches the platform code at run time. You + never clone the platform repo yourself. +4. The runner installs the runtime dependencies the platform requires. +5. The runner invokes `scripts/run_platform.sh` against your + `.acdl/contract.yaml`. + +You see the streamed output (infrastructure plan, policy-check results, +confidence signal) in your run logs. The `--check-only` and `--plan-only` +flags are platform-side modes visible in the pipeline logs; you do not pass +them yourself — the reusable workflow selects the mode based on the +`environment` in your contract (`dev` = full apply; higher environments +hold for attestation). + +### Local validation (optional) + +A consumer *may* clone the ACDL platform repo to run `--check-only` against +their contract before pushing — this is optional and not required for the +happy path. If you do this, the runtime dependencies must be installed +locally, and any AWS credentials follow the +[Credentials](../README.md#credentials--zero-trust) override model: a +static key in `.env.secrets` (gitignored) is rotated **out of band by you** +— the platform guarantees daily rotation for platform-runner runs, not for +locally-held copies. + +```bash +bash scripts/run_platform.sh --check-only path/to/your/.acdl/contract.yaml +``` + +## Step 5 — What the pipeline does + +Each stage of the central deployment pipeline: + +```mermaid +flowchart TD + S1["validate-contract
schema check"] --> S2 + S2["resolve-stack
contract -> Target Stack"] --> S3 + S3["security checks
(adapter)"] --> S4 + S4["infrastructure plan
(adapter compiles the stack)"] --> S5 + S5["policy checks
(adapter -> PolicyCheckResult)"] --> S6 + S6["confidence
score + band (dev >= 0.50)"] --> S7 + S7["evidence event
to the audit outbox"] --> S8 + S8["infrastructure apply
(dev only)"] +``` + +1. **validate-contract** — validates your contract YAML against the contract + schema. Fails fast on missing fields, unknown modules, or wrong types. +2. **resolve-stack** — the contract resolver resolves your contract to a + Target Stack instance. It loads the module's pattern, expands its + children, wires your contract inputs to the children's inputs, and emits + a stack JSON instance. +3. **security checks** (adapter) — security checks run on the resolved + stack before any infrastructure is planned. +4. **infrastructure plan** (adapter) — the substrate adapter compiles the + stack to an infrastructure plan. You see the plan in your run logs. +5. **policy checks** (adapter) — policy checks run on the plan. The results + are normalized to `PolicyCheckResult` records. Each result has a + severity, rule ID, and pass/fail status. +6. **confidence** — the confidence signal computes a score from 6 inputs + (policy, validation, freshness, source, history, NFRs). For `dev`, the + threshold is ≥ 0.50. If the band is `pass`, the pipeline proceeds. +7. **evidence event** — a hash-chained evidence event is written to the + audit outbox. +8. **infrastructure apply** (dev only) — the infrastructure plan is applied, + creating the resources in your AWS account. An evidence event for the + apply is recorded. + +## Step 6 — What gets created + +After a successful `dev` run, the resources declared by your module's +pattern exist in your AWS account, and an evidence event is recorded. + +For the `static-asset` example: + +- **An S3 bucket** named `my-static-site-assets` in `us-east-1` with + versioning enabled. +- **An evidence event** in the audit outbox with the contract ID, stack + name (`static-asset`), confidence score, and band. +- **A confidence band** of `pass` (score ≥ 0.50 for dev). + +For other modules, consult the module's README +(`modules/l1//README.md` or `modules/l2//README.md`) for the +exact resources created. + +## Step 7 — Upload your content (static-asset example) + +The platform provisions the infrastructure; you upload your content. For +the `static-asset` module: + +```bash +aws s3 sync ./assets s3://my-static-site-assets/ --acl public-read +``` + +For a `microservice`, the platform provisions the ECS service and ALB; you +push your container image to the ECR repo the platform created. + +## Step 8 — Promote to qa / prod + +Change `environment` in your contract (keeping the same versioned `uses:`): + +```yaml +uses: acdl/pipelines/deploy.yaml@v1.4 +environment: qa # QA attestation + confidence >= 0.75 +``` + +Higher environments require human attestation (a platform-runner deployment +approval) and higher confidence thresholds. See [Environments](environments/) +for the full table. + +## Step 9 — Compliance extensions + +Each module lists compliance extension points for the future compliance +milestone (GDPR, SOX, SOC2, HIPAA, DORA). See each module's README under +`modules/l1//README.md` or `modules/l2//README.md` for the +per-module extension points. Common examples: + +- **KMS key** — shared encryption key for SSE. +- **S3 access logs** — access logging to a separate audit bucket. +- **Object Lock** — 7-year immutable retention for evidence. +- **Public access block** — prevent data exfiltration. + +## Reference + +| Resource | Path | Description | +|----------|------|-------------| +| Central deployment pipeline contract | `pipelines/deploy.yaml` | The pipeline stages your contract references. | +| Reusable deploy workflow | `.github/workflows/deploy.yml` | The workflow your repo invokes via `uses:`. | +| Contract schema | `schemas/contract.schema.json` | JSON Schema for consumer contracts. | +| Stack schema | `schemas/stack.schema.json` | JSON Schema for the resolved stack instance. | +| Module catalog | [modules/](modules/) | All primitives and modules. | +| Sample contract | `contracts/static-asset.yaml` | The reference example contract (uses `@v1.4`). | +| Contract resolver | `core/contract_resolver.py` | Resolves contracts to stack instances. | +| Substrate adapter | `adapters/terraform/adapter.py` | Compiles stack instances to infrastructure. | +| Platform pipeline runner | `scripts/run_platform.sh` | The pipeline runner (platform-side; consumers do not invoke it directly). | +| Environments | [environments/](environments/) | Platform-managed environments + onboarding. | +| Versioning | [pipeline/versioning](pipeline/versioning) | The `uses:` tag + module versioning. | +| Platform README | `README.md` | How the platform works + how to run the platform repo locally. | +| Credentials & zero-trust | `README.md#credentials--zero-trust` | The OIDC/ABAC default + static-key override model. | \ No newline at end of file diff --git a/docs/contracts/index.md b/docs/contracts/index.md new file mode 100644 index 0000000..840002d --- /dev/null +++ b/docs/contracts/index.md @@ -0,0 +1,63 @@ +# Contracts + +A consumer declares intent in a **contract** — a small YAML file that +references the central deploy pipeline, names a module, selects an +environment, and supplies module-specific inputs. The platform validates, +resolves, and deploys it. + +## The contract file + +A consumer repo keeps its contract at `.acdl/contract.yaml`. A minimal +example (the `static-asset` module): + +```yaml +uses: acdl/pipelines/deploy.yaml@v1.4 +module: static-asset +environment: dev +inputs: + bucket_name: my-static-site-assets + region: us-east-1 +``` + +A `microservice` example: + +```yaml +uses: acdl/pipelines/deploy.yaml@v1.4 +module: microservice +environment: dev +inputs: + image: my-registry/my-microservice:latest + port: 8080 + env: + LOG_LEVEL: info +``` + +## Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `uses` | string | yes | Reference to the central deploy pipeline, **versioned** with a floating MAJOR+MINOR tag (e.g. `acdl/pipelines/deploy.yaml@v1.4`). Bare or `@main` references are discouraged. See [Versioning](../pipeline/versioning). | +| `module` | string | yes | Module name from the registry — any primitive or module (e.g. `static-asset`, `microservice`, `s3`). See the [module catalog](../modules/). | +| `environment` | string | yes | The platform-managed environment to deploy to (e.g. `dev`). See [Environments](../environments/). | +| `inputs` | object | yes | Module-specific inputs (see the module's README). | + +## Validation + +The contract is validated against +[`schemas/contract.schema.json`](https://github.com/acdl/acdl/blob/main/schemas/contract.schema.json). +An invalid contract (missing field, unknown module, wrong type) fails at the +validate-contract stage with a clear error. + +## Sample contract + +The reference example is +[`contracts/static-asset.yaml`](https://github.com/acdl/acdl/blob/main/contracts/static-asset.yaml), +which uses `@v1.4` as the canonical versioned `uses:` reference. + +## Multiple contracts + +A consumer repo may contain more than one contract (e.g. one per service or +one per environment). Each contract is a separate deployment; each is +referenced by a CI definition in `.github/workflows/` that invokes the +central reusable workflow with the contract path. See the +[Consumer Guide](../consumer-guide/) for the multi-contract pattern. \ No newline at end of file diff --git a/docs/environments/index.md b/docs/environments/index.md new file mode 100644 index 0000000..f6206cf --- /dev/null +++ b/docs/environments/index.md @@ -0,0 +1,69 @@ +# Environments + +A consumer does **not** provide an AWS account, a VPC, a subnet, an S3 state +bucket, or a runner key. The platform manages environments. + +## What an environment is + +A named environment is a **platform-owned** bundle of: + +- An AWS account (or a scoped partition of one). +- A network (VPC + subnets). +- A state backend (an S3 bucket + DynamoDB lock table for infrastructure + state). +- An IAM role surfaced to the consumer via attribute-based authorization + (ABAC), scoped to the consumer's repository identity and resource tags. + +A consumer selects an environment **by name** in their contract: + +```yaml +environment: dev +``` + +The platform resolves the name to the underlying account/network/state/role +at run time. The consumer never sees the raw credentials. + +## First-run onboarding + +When a consumer pipeline runs for the first time and **no environment is +defined** for the consumer's repo, the platform detects this and emits a +user-friendly onboarding prompt instead of failing opaquely. The prompt +tells the consumer: + +1. That no environment is bound to their repo yet. +2. What the platform will provision on their behalf (account/network/state/ + role). +3. The expected turnaround for the platform team to grant the environment. +4. How to request an environment (contact the platform team). + +The pipeline then exits without attempting a deployment. Once the platform +team binds an environment to the repo, the next pipeline run proceeds +normally. + +## Autonomy by environment + +| Environment | Autonomy | Gate | +|-------------|----------|------| +| dev | Full autonomy | Confidence ≥ 0.50 | +| qa | Held for attestation | QA attestation + confidence ≥ 0.75 | +| prod | Held for attestation | SRE attestation + confidence ≥ 0.90 | +| dr | Held for attestation | SRE attestation + confidence ≥ 0.95 + dr-drill | + +`dev` is the only autonomous environment. Higher environments require human +attestation (a platform-runner deployment approval) and a higher confidence +threshold. Staging does not exist. + +## Onboarding scaffold (current state) + +The platform repo ships a minimal onboarding scaffold: + +- [`core/environments/`](https://github.com/acdl/acdl/blob/main/core/environments/) + — environment definitions (a sample `dev.json`). +- `core/environment_check.py` — checks whether an environment is defined for + a given contract's repo + environment name; prints the friendly onboarding + prompt when none is defined. +- `scripts/run_platform.sh` calls the check before contract validation. + +The scaffold is minimal: the actual provisioning of a new environment is a +platform-team action today. Self-service environment provisioning is on the +[roadmap](../). \ No newline at end of file diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..f0594ed --- /dev/null +++ b/docs/index.md @@ -0,0 +1,78 @@ +# ACDL — Agentic Cloud Delivery Platform + +Consumers declare intent; the platform delivers safe production deployment +through an agentic stack — automatically, safely, and with a complete audit +trail. A merged change progresses through lower environments end-to-end +without a platform engineer joining a thread; a non-technical consumer ships +a production deployment by declaring intent, without authoring a workflow, +a configuration file, or an infrastructure module. + +## Two repositories + +There are two kinds of repository in the ACDL model: + +- **Platform repo (this one).** The source code of the platform. It owns + `modules/`, `adapters/`, `core/`, `schemas/`, `pipelines/`, `scripts/`, + and the reusable workflow files. Platform engineers work here. A consumer + never clones it. +- **Consumer repo (yours).** A consumer repo contains only its application + code, one or more contracts (`.acdl/contract.yaml`), and one or more CI + definitions (a thin `.github/workflows/deploy.yml` that `uses:` the central + reusable workflow, pointing at the appropriate environment + contract). + The consumer does not write infrastructure modules, workflow YAML, or + adapter code. + +## Documentation + +| Section | Audience | What it covers | +|---------|----------|----------------| +| [Consumer Guide](consumer-guide) | Consumers | Step-by-step: create a repo, write a contract, reference the central pipeline, ship a deployment. | +| [Modules](modules/) | Consumers + platform engineers | The module catalog — primitives and modules, their inputs/outputs, and usage. | +| [Contracts](contracts/) | Consumers | The contract schema, fields, and a worked sample. | +| [Pipeline](pipeline/) | Consumers + platform engineers | The central CI + deployment pipeline and its stages. | +| [Versioning](pipeline/versioning) | Consumers + platform engineers | Module versioning + deploy-pipeline versioning (the `uses:` tag). | +| [Environments](environments/) | Consumers | Platform-managed environments and the first-run onboarding flow. | +| [Architecture](architecture) | Platform engineers | The current architecture — layers, cross-cutting concerns, the substrate abstraction. | +| [Vision](vision) | All | The why — the friction the platform absorbs and the north star. | + +## Features + +- **Contract-driven deploys** — a consumer writes a YAML contract; the + platform resolves it to a stack, compiles it, and deploys it. +- **Reusable versioned deploy workflow** — consumer repos `uses:` a + versioned central workflow; no platform code is cloned by the consumer. +- **Module catalog** — primitives (single resources) and modules (patterns + of primitives) with self-documented inputs/outputs. +- **Zero-trust credentials** — OIDC federation + attribute-based + authorization (ABAC) by default; no long-lived keys in consumer repos. +- **Security + policy checks** — a security-check stage and a policy-check + stage run before any infrastructure is created. +- **Confidence signal** — a computed, explainable score gates promotion. +- **Evidence outbox** — every deployment writes a hash-chained evidence + event to an audit outbox. +- **Shell reproducibility** — `scripts/run_ci.sh` mirrors the CI pipeline + locally; `scripts/run_platform.sh --check-only` runs offline. +- **Platform-managed environments** — consumers provide no AWS account, + VPC, subnet, or state bucket; the platform manages environments. + +## Roadmap + +Planned future features (no dates; tracked in the internal roadmap): + +- **Dynamic module creation from a contract** — an agentic flow where a + consumer creates a module directly from the contract file (the "composition" + mechanism, redesigned). +- **Compliance milestone** — per-module compliance extension points (GDPR, + SOX, SOC2, HIPAA, DORA) wired into the pipeline. +- **Additional substrate adapters** — beyond the Terraform adapter. +- **Environment self-service** — a consumer-facing flow to request and + provision a new platform-managed environment. +- **HITL gates for qa / prod / dr** — human attestation + higher confidence + thresholds for higher environments. +- **OIDC for all platform runners** — zero-trust credentials everywhere. + +## Quick links + +- [Consumer Guide](consumer-guide) — start here if you are a consumer. +- [Architecture](architecture) — start here if you are a platform engineer. +- The [README](https://github.com/acdl/acdl) describes the platform repo. \ No newline at end of file diff --git a/docs/modules/index.md b/docs/modules/index.md new file mode 100644 index 0000000..fcf74b7 --- /dev/null +++ b/docs/modules/index.md @@ -0,0 +1,52 @@ +# Modules + +Reusable building blocks for cloud infrastructure. There are two kinds: + +- **Primitives** — a single cloud resource or a small group of related + resources (e.g. a VPC with subnets and routing). Each primitive has an + `interface.json` declaring its inputs and outputs. +- **Modules** — a pattern that references multiple primitives to deploy a + complete stack (e.g. an ECS Fargate microservice). Each module has a + `composition.json` declaring its children and wires. + +The substrate adapter compiles a module instance to infrastructure. Each +module's README documents which resources it creates. + +## Primitives + +| Module | What it creates | Source | +|--------|----------------|--------| +| `s3` | `aws_s3_bucket` — a single S3 bucket | [modules/l1/s3/README.md](https://github.com/acdl/acdl/blob/main/modules/l1/s3/README.md) | +| `vpc` | `aws_vpc` + `aws_subnet` + `aws_route_table` + `aws_internet_gateway` — VPC with subnets and routing | [modules/l1/vpc/README.md](https://github.com/acdl/acdl/blob/main/modules/l1/vpc/README.md) | +| `ecs-cluster` | `aws_ecs_cluster` — ECS Fargate cluster | [modules/l1/ecs-cluster/README.md](https://github.com/acdl/acdl/blob/main/modules/l1/ecs-cluster/README.md) | +| `ecs-service` | `aws_ecs_task_definition` + `aws_ecs_service` — Fargate service with task definition | [modules/l1/ecs-service/README.md](https://github.com/acdl/acdl/blob/main/modules/l1/ecs-service/README.md) | +| `iam-role` | `aws_iam_role` — IAM role with assume-role policy | [modules/l1/iam-role/README.md](https://github.com/acdl/acdl/blob/main/modules/l1/iam-role/README.md) | +| `alb` | `aws_lb` + `aws_lb_target_group` + `aws_lb_listener` — Application Load Balancer | [modules/l1/alb/README.md](https://github.com/acdl/acdl/blob/main/modules/l1/alb/README.md) | +| `ecr` | `aws_ecr_repository` — ECR container image repository | [modules/l1/ecr/README.md](https://github.com/acdl/acdl/blob/main/modules/l1/ecr/README.md) | + +## Modules + +| Module | What it references | Source | +|--------|--------------------|--------| +| `static-asset` | 1 primitive (s3) — a static-asset S3 bucket | [modules/l2/static-asset/README.md](https://github.com/acdl/acdl/blob/main/modules/l2/static-asset/README.md) | +| `microservice` | 6 primitives (vpc, cluster, ecr, iam-role, alb, ecs-service) — an ECS Fargate microservice | [modules/l2/microservice/README.md](https://github.com/acdl/acdl/blob/main/modules/l2/microservice/README.md) | + +## Registry + +Module versions are tracked in +[`registry.json`](https://github.com/acdl/acdl/blob/main/modules/registry.json). +Both primitives and modules are registered. + +## Versioning + +Primitives and modules use semver: interface → MAJOR, behavior → MINOR, +lifecycle → PATCH. A MAJOR bump requires a new registry entry (immutable +publication); the old entry enters a 12-month deprecation window. See +[Versioning](../pipeline/versioning) for the deploy-pipeline versioning. + +## Module patterns (roadmap) + +The current `composition.json` mechanism is a thin pattern layer. A future +redesign will let a consumer dynamically create a module directly from the +contract file (an agentic "composition" flow). That is on the roadmap, not +implemented today. \ No newline at end of file diff --git a/docs/pipeline/index.md b/docs/pipeline/index.md new file mode 100644 index 0000000..5c7ed6d --- /dev/null +++ b/docs/pipeline/index.md @@ -0,0 +1,95 @@ +# Pipeline + +The platform runs two pipelines, both defined by declarative contracts that +are the single source of truth for the workflow files. + +## CI pipeline + +The CI pipeline runs on every push and pull request to `main`. It is defined +by [`pipelines/ci.yaml`](https://github.com/acdl/acdl/blob/main/pipelines/ci.yaml), +validated against +[`schemas/pipeline.schema.json`](https://github.com/acdl/acdl/blob/main/schemas/pipeline.schema.json). +Both platform-runner workflow files implement the same contract and are +byte-identical: + +- `.github/workflows/ci.yml` — GitHub Actions (production) + +Three stages run in sequence: + +1. **lint** — `py_compile` across the platform's Python files. +2. **test** — `pytest` across the offline test suite. +3. **check-only** — `run_platform.sh --check-only` (offline, no AWS). + +`scripts/run_ci.sh` mirrors the CI pipeline locally so the pipeline is fully +reproducible from the shell: + +```bash +bash scripts/run_ci.sh # run all 3 stages +bash scripts/run_ci.sh --quiet # suppress per-stage banners +``` + +## Deployment pipeline + +The deployment pipeline runs when a consumer submits a contract. It is +defined by [`pipelines/deploy.yaml`](https://github.com/acdl/acdl/blob/main/pipelines/deploy.yaml), +validated against +[`schemas/deploy-pipeline.schema.json`](https://github.com/acdl/acdl/blob/main/schemas/deploy-pipeline.schema.json). +It is exposed to consumer repos as a **reusable workflow**: + +- `.github/workflows/deploy.yml` — GitHub Actions (production) + +A consumer repo invokes the reusable workflow via a **versioned tag** +(floating MAJOR + MINOR, e.g. `acdl/.github/workflows/deploy.yml@v1.4`). +The workflow checks out the consumer repo, then checks out the ACDL platform +repo into the runner workspace, and runs `scripts/run_platform.sh` against +the consumer's contract. The consumer never clones the platform repo or +invokes its scripts locally. See the [Consumer Guide](../consumer-guide/) +for the end-to-end happy path. + +## Deployment stages + +```mermaid +flowchart TD + S1["validate-contract
schema check"] --> S2 + S2["resolve-stack
contract -> Target Stack"] --> S3 + S3["security checks
(adapter)"] --> S4 + S4["infrastructure plan
(adapter compiles the stack)"] --> S5 + S5["policy checks
(adapter -> PolicyCheckResult)"] --> S6 + S6["confidence
score + band"] --> S7 + S7["evidence event
to the audit outbox"] --> S8 + S8["infrastructure apply
(dev only)"] +``` + +1. **validate-contract** — validates the contract YAML against the contract + schema. Fails fast on missing fields, unknown modules, or wrong types. +2. **resolve-stack** — the contract resolver resolves the contract to a + Target Stack instance (loads the module's pattern, expands its children, + wires the contract inputs, emits a stack JSON instance). +3. **security checks** (adapter) — security checks run on the resolved + stack before any infrastructure is planned. +4. **infrastructure plan** (adapter) — the substrate adapter compiles the + stack to an infrastructure plan. +5. **policy checks** (adapter) — policy checks run on the plan. Results are + normalized to `PolicyCheckResult` records (severity, rule ID, pass/fail). +6. **confidence** — the confidence signal computes a score from 6 inputs + (policy, validation, freshness, source, history, NFRs). For `dev`, the + threshold is ≥ 0.50. If the band is `pass`, the pipeline proceeds. +7. **evidence event** — a hash-chained evidence event is written to the + audit outbox. +8. **infrastructure apply** (dev only) — the infrastructure plan is applied, + creating the resources. An evidence event for the apply is recorded. + +Higher environments hold for human attestation (see +[Environments](../environments/)). + +## Output streaming + +`scripts/run_platform.sh` streams output by default so the user can see what +the platform is doing: + +- **`--check-only`**: streams the emitted infrastructure file content. +- **`--plan-only`** and **full mode**: streams the infrastructure plan output. +- **Full mode**: prints policy-check results with severity, rule ID, and + pass/fail status. + +A `--quiet` flag suppresses streaming (output to log files only). \ No newline at end of file diff --git a/docs/pipeline/versioning.md b/docs/pipeline/versioning.md new file mode 100644 index 0000000..1e7330a --- /dev/null +++ b/docs/pipeline/versioning.md @@ -0,0 +1,56 @@ +# Versioning + +ACDL uses two versioning schemes: one for modules, one for the deploy +pipeline. Both matter to a consumer. + +## Module versioning + +Primitives and modules use **semver** with three triggers: + +- **interface → MAJOR** — a breaking change to the module's inputs/outputs. +- **behavior → MINOR** — a backward-compatible behavior change. +- **lifecycle → PATCH** — a fix or internal change. + +A MAJOR bump requires a **new registry entry** (immutable publication); the +old entry enters a **12-month deprecation window**. A module pins its +primitives by `name@semver`; the resolver picks the highest compatible. + +Module versions are tracked in +[`registry.json`](https://github.com/acdl/acdl/blob/main/modules/registry.json). + +## Deploy-pipeline versioning (the `uses:` tag) + +The central deploy pipeline is referenced by a **floating MAJOR + MINOR +tag** in a consumer's contract and CI definition: + +```yaml +uses: acdl/pipelines/deploy.yaml@v1.4 +``` + +Version constraints cannot be expressed inside the contract, so the tag in +`uses:` is the only immutability lever a consumer has. + +**Unversioned references are discouraged.** Do not use `@main` or a bare +`acdl/pipelines/deploy.yaml` — `main` is constantly updated and can cause +unexpected failures. Pinning to a MAJOR+MINOR tag means: + +- **Immutability** — the pipeline behavior you tested is the behavior you + get. Patch fixes flow within the tag; breaking changes land under the + next MINOR tag (`@v1.5`), which you opt into explicitly. +- **Resilience** — your deployment does not break because an unrelated + change landed on `main`. +- **Reproducibility** — your setup is stable. You upgrade on your schedule + by bumping the tag. + +## When a new tag is released + +When a new MINOR tag is released (e.g. `@v1.5`), review its changelog and +bump your `uses:` reference when ready. The old tag continues to receive +patch fixes until the next MINOR tag. + +## Production-bound references + +For production-bound workflows, the platform resolves the current tag to its +SHA (tag for dev/qa, SHA for prod). This prevents a silent patch from +changing a production deployment. The platform provides a CLI command for +the tag → SHA resolution. \ No newline at end of file