From 1ad9c35fb68a626456cfad6af198aafc086f42cf Mon Sep 17 00:00:00 2001 From: Jon Chery Date: Tue, 21 Jul 2026 20:56:54 +0000 Subject: [PATCH] docs(P11): v1.2 research addendum + README rewrite ---ci--- project: acdl phase: 11 milestone: v1.2 status: research requirements: covered: [REQ-29] ---/ci--- Phase 11 deliverables: - RESEARCH.md: v1.2 addendum (Targets 9-13). #36988 re-checked (still open, 2026-05-27) -> D-047 extends D-039 waiver. NFR audit: spike IAM policy already least-privilege (needs ECS/ECR/ELB/IAM expansion for Phase 15); bootstrap scripts already idempotent; run_spike_*.sh to consolidate into run_platform.sh (D-048); P1-1 redaction targets identified in VERIFY.md; PERSONAS.md line 47 stale path. ECS L1 catalog scoped (D-049): 6 L1s, 9 IR types for the adapter TYPE_MAP. - README.md: rewritten. v1.1 complete (tag v1.2.0); v1.2 active. Documents the actual spike flow (contract -> IR -> plan -> Checkov -> confidence -> outbox), how to run (bootstrap, rotate, run_spike_e2e.sh), the real repo layout, environments table, credentials/D-039/D-047 waiver. No stale 'v1.1 (active)' framing. --- .ciagent/RESEARCH.md | 134 ++++++++++++++++++++++++++++++++++ README.md | 166 ++++++++++++++++++++++++++++++++++++------- 2 files changed, 274 insertions(+), 26 deletions(-) diff --git a/.ciagent/RESEARCH.md b/.ciagent/RESEARCH.md index cff0148..36d41cb 100644 --- a/.ciagent/RESEARCH.md +++ b/.ciagent/RESEARCH.md @@ -1327,4 +1327,138 @@ the demo is local-only post-archive), but this is moot for the archive. --- +## v1.2 Research Addendum (Phase 11, 2026-07-21) + +> Phase: research (Phase 11). Milestone: v1.2. Status: active. +> Researcher: ci-researcher (inline, docs phase). Autonomy: full. +> Sources: GitHub API (go-gitea/gitea#36988), ACDL codebase audit +> (`terraform/bootstrap/`, `scripts/`, `adapters/terraform/`, +> `modules-ir/registry.json`, `.ciagent/VERIFY.md`, `.ciagent/PERSONAS.md`). +> Scope: re-eval OIDC blocker, NFR audit of the v1.1 spike, simplification +> opportunities, README rewrite plan, ECS L1 catalog scoping. + +### TARGET 9 — go-gitea/gitea#36988 re-check (v1.2) + +**Verdict (conf 0.95): still open, not merged.** + +Re-checked 2026-07-21 via `api.github.com/repos/go-gitea/gitea/pulls/36988`: +- `state`: open +- `merged`: false +- `merged_at`: null +- `updated_at`: 2026-05-27T16:26:24Z +- `title`: "Add Actions OIDC provider with workflow permission gating" + +No movement since the v1.1 research (2026-07-21 v1.1 research also found it +open). Real OIDC federation remains impossible for Gitea Actions. **D-047 +adopts**: extend the D-039 per-run-rotated-key waiver for v1.2; real OIDC is +deferred to v1.3+. The waiver continues to satisfy §12.5's *intent*: no +*persistently* long-lived key (`scripts/rotate_spike_key.sh` rotates after +each run; Phase 12 tightens IAM scoping + rotation hygiene). + +### TARGET 10 — NFR audit of the v1.1 spike + +Audited the v1.1 spike's operational code for NFR gaps. + +**`terraform/bootstrap/spike_runner_policy.json`** — least-privilege PASS +already. Explicit Allow list (S3 state bucket R/W, DynamoDB outbox R/W, +`sts:GetCallerIdentity`) + `DenyEverythingElse` on `*` with `NotResource`. +No wildcards in the Allow statements. **v1.2 gap**: the policy only covers +S3 + DynamoDB + STS — Phase 15's `terraform apply` to ECS needs ECS + ECR + +ELB + IAM (plan + apply) permissions added. Phase 12 scopes the policy +expansion; Phase 15 applies it. + +**`terraform/bootstrap/create_state_backend.py`** — idempotent PASS already. +`head_bucket` → skip-create if exists; `describe_table` → skip-create if +exists; `put_bucket_versioning` is idempotent. **No v1.2 change needed.** + +**`terraform/bootstrap/create_iam_user.py`** — idempotent PASS already. +`get_user` → skip-create if exists; `put_user_policy` overwrites (idempotent); +`list_access_keys` → skip-create if an active key exists. **No v1.2 change +needed.** + +**`scripts/run_spike_plan.sh` + `scripts/run_spike_e2e.sh`** — two scripts, +overlapping setup (env loading, `cd terraform/spike`, `terraform init`). +`run_spike_e2e.sh` is the superset (full pipeline); `run_spike_plan.sh` is +the plan-only subset. **v1.2 simplification (Phase 12)**: consolidate into +one `scripts/run_platform.sh` with a `--plan-only` flag (default: full e2e). +Removes ~30 lines of duplication. + +**`scripts/rotate_spike_key.sh`** — idempotent PASS (always ends with exactly +1 active key). Uses the bootstrap root key to rotate; documented that D-034 +closure (root key deactivation) is a manual user step. **No v1.2 change +needed** (the root key is now deactivated per D-034 closure; rotation uses +the spike key itself or a separate rotation credential — flagged as a v1.2 +operational note in Phase 12). + +**Error handling**: `run_spike_e2e.sh` uses `set -u` + a `fail()` helper — +good. `run_spike_plan.sh` uses `set -u` + inline exits — adequate. The +consolidated `run_platform.sh` should use `set -euo pipefail` + `fail()` +for uniform strictness. + +**P1-1 redaction target**: `.ciagent/VERIFY.md` lines 93–96, 170 reference +two AWS access key IDs (`AKIAYOZHMKZ7RK26N66W` rotated spike key, +`AKIAYOZHMKZ772SINHFX` deactivated root key). Public identifiers, not +secret pairs, in the audit narrative not executable code. **Phase 12 +redacts** them to `AKIA…SPIKE` / `AKIA…ROOT-DEACTIVATED`. + +**P1-B stale paths**: `.ciagent/PERSONAS.md` line 47 still has +`platform/registry/**` (the rest were fixed at `ab69d10`). **Phase 12 +fixes** line 47 to `acdl_platform/registry/**` (or removes it — there is no +`acdl_platform/registry/` dir; the registry is `modules-ir/registry.json`). + +### TARGET 11 — Simplification opportunities + +1. **Script consolidation** (above): `run_spike_*.sh` → `run_platform.sh`. +2. **`terraform/spike/.terraform/` artifacts**: gitignored already + (`.gitignore` covers `.terraform/`, `.terraform.lock.hcl`, `tfplan`, + `*.tfstate*`). No change. +3. **`acdl_platform/__pycache__/`**: gitignored already. No change. +4. **Dead code**: none found — the spike is tight. The `run_spike_plan.sh` + script is the only redundancy (subsumed by `run_platform.sh --plan-only`). +5. **`demo/` archive**: correctly separated; no v1.2 touch. + +### TARGET 12 — README rewrite plan + +Current `README.md` (51 lines) is stale: "v1.1 (active)" framing, no +"how to run the platform" section, no v1.2 objective. **Phase 11 rewrites +it** to reflect: +- v1.1 complete (tag `v1.2.0`); v1.0 demo archived under `demo/`. +- The actual spike flow: contract → IR → `terraform plan` → Checkov → + confidence signal → outbox. +- How to run: `scripts/run_platform.sh` (after Phase 12; for now + `scripts/run_spike_e2e.sh`). +- Real repo layout table (the existing one is accurate; refresh the + "Populated" column). +- v1.2 objective (platform hardening + ECS microservice). + +### TARGET 13 — ECS L1 catalog scoping (for Phase 13) + +Six L1s needed for an ECS Fargate microservice. Each maps to one or more +AWS Terraform resources; the adapter `TYPE_MAP` (currently +`{"aws:s3:bucket": "aws_s3_bucket"}`) needs expansion: + +| L1 | IR type(s) | Terraform resource(s) | Key inputs | +|----|-----------|----------------------|-----------| +| `l1-vpc` | `aws:ec2:vpc`, `aws:ec2:subnet`, `aws:ec2:routetable` | `aws_vpc`, `aws_subnet`, `aws_route_table` + associations | cidr, azs | +| `l1-ecs-cluster` | `aws:ecs:cluster` | `aws_ecs_cluster` | name | +| `l1-ecs-service` | `aws:ecs:service`, `aws:ecs:task_definition` | `aws_ecs_service`, `aws_ecs_task_definition` | image, port, cpu, memory, env | +| `l1-iam-role` | `aws:iam:role`, `aws:iam:rolepolicyattachment` | `aws_iam_role`, `aws_iam_role_policy_attachment` | task + exec role | +| `l1-alb` | `aws:elbv2:loadbalancer`, `aws:elbv2:listener`, `aws:elbv2:targetgroup` | `aws_lb`, `aws_lb_listener`, `aws_lb_target_group` | port, protocol | +| `l1-ecr` | `aws:ecr:repository` | `aws_ecr_repository` | name | + +The IR schema (`schemas/ir.schema.json`) is substrate-agnostic and already +supports arbitrary resource types — no schema change needed, only new +`interface.json` files + `TYPE_MAP` entries. The `l2-microservice` +thin-composition references all six (depth ≤ 5). + +### Decisions surfaced (v1.2) + +| ID | Decision | Rationale | Confidence | Alternatives | +|----|----------|-----------|------------|--------------| +| **D-047** | Extend D-039 per-run-rotated-key waiver for v1.2. Real OIDC deferred to v1.3+. | go-gitea/gitea#36988 still open (TARGET 9). The waiver satisfies §12.5's intent for v1.2; Phase 12 tightens IAM + rotation hygiene. | 0.95 | (a) wait for #36988 (blocks v1.2 indefinitely); (b) self-hosted OIDC broker (heavy); (c) KMS-backed ephemeral creds (scope creep for v1.2). | +| **D-048** | Consolidate `run_spike_plan.sh` + `run_spike_e2e.sh` → one `scripts/run_platform.sh` with `--plan-only` flag (default: full e2e). | Two scripts with overlapping setup (~30 lines duplicated). One script with a flag is simpler and matches the "streamline" scope axis. | 0.90 | Keep both (redundant); delete `run_spike_plan.sh` only (loses the plan-only convenience). | +| **D-049** | v1.2 L1 catalog = 6 L1s (`l1-vpc`, `l1-ecs-cluster`, `l1-ecs-service`, `l1-iam-role`, `l1-alb`, `l1-ecr`). The adapter `TYPE_MAP` expands to 9 IR types (3 new for VPC, 3 for the rest). | Minimal set to deploy an ECS Fargate service end-to-end. VPC is split into vpc/subnet/routetable because the IR models one resource per `interface.json` entry, but the L1 groups them. | 0.85 | Fewer L1s (e.g. fold VPC into the ECS service — violates L1 single-purpose); more L1s (e.g. separate `l1-securitygroup` — scope creep for v1.2). | + +--- + *End of RESEARCH.md. Path: `/root/acdl/.ciagent/RESEARCH.md`.* \ No newline at end of file diff --git a/README.md b/README.md index ddc013f..a28376b 100644 --- a/README.md +++ b/README.md @@ -8,45 +8,159 @@ a production deployment by declaring intent, without authoring a workflow, a configuration file, or a Terraform module. - **Vision** (the why): [`docs/vision.md`](docs/vision.md) -- **Architecture** (the how): [`docs/architecture.md`](docs/architecture.md) +- **Architecture** (the how): [`docs/architecture.md`](docs/architecture.md) + [`.ciagent/ARCHITECTURE.md`](.ciagent/ARCHITECTURE.md) - **Decisions**: [`.ciagent/PROJECT.md`](.ciagent/PROJECT.md) -- **Target architecture**: [`.ciagent/ARCHITECTURE.md`](.ciagent/ARCHITECTURE.md) - **Phase plan**: [`.ciagent/ROADMAP.md`](.ciagent/ROADMAP.md) ## Status -- **v1.1 (active):** architecture finalization + v1 spike. Finalize the - architecture to v1.0 (resolve the 11 open design decisions) and prove the - locked commitments with one end-to-end implementation spike +- **v1.2 (active):** platform hardening + first real consumer deployment. + Harden the v1.1 spike's NFRs, simplify the setup, rewrite the docs, and + prove the platform delivers real value by deploying a basic microservice + to AWS ECS Fargate end-to-end (`terraform apply`, dev autonomous). Ship + tag `v1.3.0`. +- **v1.1 (complete, tag `v1.2.0`):** architecture finalization + v1 spike. + Finalized the architecture to v1.0 (resolved all 11 open design + decisions) and proved the IR commitments hold with one end-to-end spike (`l1-s3` + `l2-static-asset` + Terraform adapter → real `terraform plan` - against AWS). -- **v1.0 demo (complete, archived):** tag `v1.1.0`. The 30-minute - stub-driven executive demo is preserved under `demo/` as the intent - reference; it is not the platform. + against AWS). Gitea release id 202. +- **v1.0 demo (complete, archived under `demo/`, tag `v1.1.0`):** the + 30-minute stub-driven executive demo. Preserved as the intent reference; + it is not the platform. -## Repository layout +## How the platform works -| Path | Purpose | Populated | -|------|---------|-----------| -| `acdl_platform/` | Platform code: confidence signal, contract resolver, outbox, HITL/ledger designs (renamed from `platform/` in Phase 08 to avoid shadowing the stdlib `platform` module) | Phase 07+ | -| `schemas/` | JSON Schemas: IR, PolicyCheckResult, contract | Phase 07 | -| `adapters/` | Substrate adapters (Terraform adapter in v1; the only substrate-specific code per §12) | Phase 09 | -| `terraform/` | State backend + provider config (S3 state + DynamoDB lock) | Phase 08+ | -| `modules-ir/` | IR-typed L1/L2 modules (`l1-s3`, `l2-static-asset`) | Phase 09–10 | -| `scripts/` | v1.1 verify scripts (`verify_phaseNN.sh`) | Phase 06+ | -| `demo/` | Archived v1.0 executive demo (tag `v1.1.0`); runs locally via `demo/scripts/run_demo.sh --no-upload` | complete | -| `.ciagent/` | CIAgent metadata (plans, decisions, personas, roadmap, research) | active | -| `docs/` | Upstream vision + architecture sources | active | +The platform is **four layers + six cross-cutting concerns**, bound by the +vision's "Two Consumer Surfaces, One Platform" tenet: technical developers +(L3A) and non-technical consumers (L3B) converge on the same contract +schema, the same policy envelope, and the same evidence stream. -## Running the archived demo +### The v1.1 spike flow (end-to-end) -The v1.0 demo is an archived artifact. To re-run it locally: +``` +contracts/spike.yaml + │ (contract schema validation) + ▼ +acdl_platform/contract_resolver.py ──▶ Target Stack IR (JSON) + │ (IR schema validation) + ▼ +adapters/terraform/adapter.py ──▶ terraform/spike/{main,terraform,providers}.tf + │ (the only substrate-specific code) + ▼ +terraform plan (real AWS, via the rotated spike key — D-039/D-047) + │ + ▼ +adapters/terraform/policy/checkov_adapter.py ──▶ PolicyCheckResult (JSON list) + │ (normalized, engine-agnostic) + ▼ +acdl_platform/confidence_signal.py ──▶ { score, band, perInput, reasonCodes } + │ (6 inputs: policy, validation, freshness, source, history, nfrs) + ▼ +acdl_platform/outbox_writer.py ──▶ DynamoDB outbox (acdl-outbox) + │ (hash-chained evidence event) + ▼ +acdl-evidence timeline (acdl-evidence repo, raw-file served) +``` + +The spike validates the architecture's claim that the **IR-shaped +commitments do not require a polyglot mess**: the adapter is the only +substrate-specific code. `modules-ir/`, `schemas/`, `contracts/`, +`acdl_platform/confidence_signal.py`, `acdl_platform/contract_resolver.py`, +and `acdl_platform/outbox_writer.py` are all substrate-agnostic (no +`aws_s3_bucket` / `aws_` Terraform terms). + +### What's different in v1.2 + +v1.2 extends the spike to a real, simpler, better-documented platform that +**deploys a microservice to ECS Fargate**: + +- Six new IR-typed L1s: `l1-vpc`, `l1-ecs-cluster`, `l1-ecs-service`, + `l1-iam-role`, `l1-alb`, `l1-ecr`. +- One new L2 thin-composition: `l2-microservice` (references the six L1s). +- `terraform apply` (dev, autonomous per §10, confidence ≥ 0.50) — real + provisioning, not just `plan`. +- A new consumer repo `acdl-consumer-microservice` with a basic HTTP + container + Dockerfile + ECR push + contract submission. +- One `scripts/run_platform.sh` (consolidated from the v1.1 spike scripts). +- NFR hardening: least-privilege IAM (expanded for ECS), idempotent + bootstrap, proper error handling, P1-1 redaction. + +## How to run + +### Prerequisites + +- AWS account + the rotated spike key in `.env.secrets` (see + `scripts/rotate_spike_key.sh`; the bootstrap root key was deactivated + per D-034 closure). +- `terraform` (pin `1.9.*`), `checkov` (pin `>=3.2,<4`), `python3` + `boto3` + + `jsonschema`. + +### Run the v1.1 spike end-to-end (v1.2 will consolidate to `run_platform.sh`) + +```bash +# 1. Bootstrap the AWS state backend + spike IAM user (one-time, idempotent) +# (requires the bootstrap root key in env — now deactivated; skip if +# the state bucket + acdl-spike-runner already exist) +ACDL_BOOTSTRAP_AWS_ACCESS_KEY_ID=... ACDL_BOOTSTRAP_AWS_SECRET_ACCESS_KEY=... \ + python3 terraform/bootstrap/create_state_backend.py +ACDL_BOOTSTRAP_AWS_ACCESS_KEY_ID=... ACDL_BOOTSTRAP_AWS_SECRET_ACCESS_KEY=... \ + python3 terraform/bootstrap/create_iam_user.py # prints the initial key + +# 2. Rotate the spike key (writes .env.secrets, gitignored) +ACDL_BOOTSTRAP_AWS_ACCESS_KEY_ID=... ACDL_BOOTSTRAP_AWS_SECRET_ACCESS_KEY=... \ + bash scripts/rotate_spike_key.sh + +# 3. Run the full spike pipeline (contract -> IR -> plan -> Checkov -> +# confidence -> outbox) +bash scripts/run_spike_e2e.sh +# Expected: "=== SPIKE E2E OK ===" + +# Or plan-only: +bash scripts/run_spike_plan.sh +``` + +### Re-run the archived v1.0 demo (stubs only, no AWS) ```bash bash demo/scripts/run_demo.sh --no-upload ``` -The demo deck is at [`demo/ACDL_DEMO.md`](demo/ACDL_DEMO.md). The demo runs +The demo deck is at [`demo/ACDL_DEMO.md`](demo/ACDL_DEMO.md). It runs entirely on local stubs — no AWS, no AI — and shows intent and safety -behavior rather than provisioning real cloud resources. It is the reference -of intent for the real platform; it is not the platform itself. \ No newline at end of file +behavior rather than provisioning real cloud resources. + +## Repository layout + +| Path | Purpose | Status | +|------|---------|--------| +| `acdl_platform/` | Platform code: confidence signal, contract resolver, outbox writer, HITL/ledger/SoD designs (renamed from `platform/` in Phase 08 to avoid shadowing the stdlib `platform` module) | v1.1 complete; v1.2 extends | +| `schemas/` | JSON Schemas: IR, PolicyCheckResult, contract (draft 2020-12) | v1.1 complete; v1.2 extends contract schema | +| `adapters/` | Substrate adapters — Terraform adapter (the only substrate-specific code per §12) + Checkov policy adapter | v1.1 complete; v1.2 expands `TYPE_MAP` | +| `terraform/` | State backend (S3 + DynamoDB) + spike TF (`terraform/spike/`) + bootstrap scripts (`terraform/bootstrap/`) | v1.1 complete; v1.2 adds ECS apply | +| `modules-ir/` | IR-typed L1/L2 modules + `registry.json`. v1.1: `l1-s3`, `l2-static-asset`. v1.2: + 6 ECS L1s, `l2-microservice` | v1.1 complete; v1.2 expands | +| `contracts/` | Sample contracts (`spike.yaml` for `l2-static-asset`) | v1.1 complete; v1.2 adds `microservice.yaml` | +| `scripts/` | Verify scripts (`verify_phaseNN.sh`), spike run scripts (`run_spike_*.sh` → `run_platform.sh` in v1.2), key rotation | v1.1 complete; v1.2 consolidates | +| `demo/` | Archived v1.0 executive demo (tag `v1.1.0`); runs locally via `demo/scripts/run_demo.sh --no-upload` | complete (archived) | +| `.ciagent/` | CIAgent metadata (config, project, architecture, requirements, roadmap, personas, plans, research, verify, review, audit) | active | +| `docs/` | Upstream vision + architecture sources (`vision.md`, `architecture.md`) | active | + +## Environments + +| Environment | Autonomy | Gate | Status | +|---|---|---|---| +| dev | Full autonomy (no HITL) | Confidence ≥ 0.50 | v1.1 spike (`plan`); v1.2 (`apply`) | +| qa | Held for attestation | QA HITL + confidence ≥ 0.75 | v1.3+ | +| prod | Held for attestation | SRE HITL + confidence ≥ 0.90 | v1.3+ | +| dr | Held for attestation | SRE HITL + confidence ≥ 0.95 + dr-drill | v1.3+ | + +**Staging does not exist** (Path A locked). + +## Credentials + +**Long-lived AWS credentials are forbidden** (§12.5). The v1.1 spike uses a +temporary long-lived key **once** to bootstrap (waiver D-034, now closed — +the root key was deactivated by the user), then rotates the spike key +per-run via `scripts/rotate_spike_key.sh` (waiver D-039, extended for v1.2 +as D-047). Real OIDC federation is deferred to v1.3+, blocked on +[go-gitea/gitea#36988](https://github.com/go-gitea/gitea/pull/36988) (still +open as of 2026-07-21). \ No newline at end of file