Jon Chery
7310da224e
---ci---
phase: 3
milestone: v1.0
status: research
research:
l2_schema: l1s list of {name, inputs: map}; 4 L2s each referencing 5 L1s
contract_schema: stack + inputs + optional public-ingress: bool
state_json_shape: l2, l1s array (name+applied+exit_code), contract
audit_json: JSON array; canonical-JSON SHA-256 hash chain; GENESIS prev_hash
core_script_io: 5 scripts with explicit input/output/exit contracts
personas: no change; backend-engineer owns core scripts + L2 manifests; infra-stub-engineer owns L1 manifests only
---/ci---
ARCHITECTURE.md gains the L2 manifest schema, the L2 list with per-L2
L1 references (5 each, within max-depth-5), the contract.yaml schema,
the state.json shape, the audit.json event + canonical-JSON hash chain,
and a core-script I/O contract table. PERSONAS.md is unchanged for
Phase 03 (backend-engineer and infra-stub-engineer were both already
active in the project-level roster).
254 lines
14 KiB
Markdown
254 lines
14 KiB
Markdown
# ACDL — Architecture (initial)
|
|
|
|
> Initial architecture for the ACDL demo. May be incomplete; refined at phase boundaries.
|
|
|
|
## Overview
|
|
|
|
The demo is a three-repo, stub-driven system that simulates an autonomous cloud delivery platform. No real cloud or AI is used; every "infrastructure" action is a bash/Python stub that emits structured evidence. The platform is driven by either a developer-supplied `contract.yaml` (L3A) or a natural-language GitHub Issue parsed by a keyword script (L3B), then flows through an autonomous Dev stage, manual QA and Prod approval gates, and finally publishes a hash-chained audit trail to a Pages site.
|
|
|
|
```
|
|
┌──────────────── acdl-contracts ─────────────────┐
|
|
Developer ───▶ │ commit contract.yaml Issue (NL intent) │
|
|
└────────────┬───────────────────┬────────────────┘
|
|
│ (push) │ (issue opened)
|
|
▼ ▼
|
|
┌─────────────────┐ ┌──────────────────────┐
|
|
│ reusable │ │ issue workflow → │
|
|
│ pipeline │ │ l3b_agent_stub.py → │
|
|
│ (acdl repo) │ │ contract.yaml → push │
|
|
└────────┬────────┘ └──────────────────────┘
|
|
│
|
|
┌────────────────────┼────────────────────┐
|
|
▼ ▼ ▼
|
|
Dev (autonomous) QA (approval) Prod (approval)
|
|
mock_executor.sh environment gate environment gate
|
|
policy_checker.py
|
|
confidence_signal.py
|
|
│
|
|
▼
|
|
evidence_writer.py ──▶ audit.json (hash-chained) ──▶ acdl-evidence
|
|
│
|
|
▼
|
|
index.html (Pages)
|
|
timeline UI
|
|
```
|
|
|
|
## Components
|
|
|
|
| Name | Description | Boundaries | Depends On |
|
|
|------|-------------|-----------|------------|
|
|
| `acdl` repo | Platform meta repo: reusable workflows, L1/L2 stub modules, core scripts | Owns workflows + stubs; does not hold contracts or evidence | — |
|
|
| L1 modules | Single-purpose infra primitives (EKS Fargate, IAM, Lambda, API Gateway, EventBridge, SQS, S3, CloudWatch) | One folder per L1; `manifest.yaml` + `mock_apply.sh`; do not compose with other L1s | `acdl` repo |
|
|
| L2 modules | Composed stacks (invoice, commodity-price-feed, energy-analytics-api, regulatory-reporting) | Reference L1s by name; max depth 5; expressed as a composition manifest | L1 modules |
|
|
| `mock_executor.sh` | Reads an L2 composition, invokes each L1 `mock_apply.sh`, writes `state.json` | Bash; reads L2 manifest + L1 manifests | L1/L2 modules |
|
|
| `policy_checker.py` | Reads `contract.yaml`; fails on forbidden keys (e.g. `public-ingress: true`) | Python; emits `POLICY_VIOLATION:<REASON>` or pass | contract.yaml |
|
|
| `confidence_signal.py` | Base 0.90; on policy failure drops to 0.40 and echoes reason | Python; calls policy_checker | policy_checker.py |
|
|
| `evidence_writer.py` | Appends an event to `audit.json`, links to previous event via SHA-256 chain | Python; canonical-JSON hashing | audit.json |
|
|
| `l3b_agent_stub.py` | Parses Issue text by keywords, emits `contract.yaml` | Python keyword map; no external APIs | contract.yaml schema |
|
|
| `acdl-contracts` repo | Developer + agentic entry surface; holds contracts + issue workflow | Triggers main pipeline on push | `acdl` reusable workflow |
|
|
| `acdl-evidence` repo | Pages host for `audit.json` + `index.html` timeline | Read-only for the pipeline; written at finalize stage | evidence_writer.py output |
|
|
| Reusable pipeline workflow | Dev → QA → Prod → Finalize stages with environment gates | Gitea Actions; calls core scripts | All core scripts |
|
|
|
|
## Data Flow
|
|
|
|
1. A `contract.yaml` arrives either by direct push (L3A) or by the issue workflow running `l3b_agent_stub.py` (L3B).
|
|
2. Push to `acdl-contracts` triggers the reusable pipeline in the `acdl` repo.
|
|
3. **Dev stage:** `policy_checker.py` validates the contract; `mock_executor.sh` applies the L2 composition's L1s; `confidence_signal.py` computes the score; `evidence_writer.py` records each step. If score < 0.50, the stage fails and evidence records the rejection.
|
|
4. **QA stage:** the workflow pauses on the `qa` environment; a human approves.
|
|
5. **Prod stage:** same gate on the `prod` environment.
|
|
6. **Finalize:** the workflow commits the updated `audit.json` to `acdl-evidence`; Pages republishes `index.html`, which fetches and renders the timeline.
|
|
|
|
## Build Order
|
|
|
|
1. Repo scaffolding: create `acdl-contracts` and `acdl-evidence` in the org; seed `acdl` directory layout.
|
|
2. L1 modules (8 stubs).
|
|
3. L2 modules (4 compositions).
|
|
4. Core scripts (`mock_executor.sh`, `policy_checker.py`, `confidence_signal.py`, `evidence_writer.py`, `l3b_agent_stub.py`).
|
|
5. Reusable pipeline workflow (Dev → QA → Prod → Finalize) + environment gates.
|
|
6. Issue-triggered L3B workflow in `acdl-contracts`.
|
|
7. Evidence UI (`index.html` + Pages config).
|
|
8. Demo dry-run + the four scripted acts.
|
|
|
|
## Gitea API Surface (Phase 01 research)
|
|
|
|
Authoritative findings from the Gitea docs (added in RESEARCH; supersedes any
|
|
GitHub-Pages / GitHub-Environments assumptions carried over from the spec):
|
|
|
|
| Capability | Gitea support | ACDL approach |
|
|
|------------|---------------|---------------|
|
|
| Org-scoped repo create | `POST /api/v1/orgs/{org}/repos` (`CreateRepoOption`) | Used to create `acdl-contracts` + `acdl-evidence` |
|
|
| Native Pages | **None** (no `[pages]` config section) | Serve `acdl-evidence` via raw file URLs: `https://git.cloudinit.dev/continuous-intelligence/acdl-evidence/raw/branch/main/index.html`; `index.html` fetches `audit.json` from the same raw path. Requires `[cors] ENABLED=true` on the server if the UI is loaded cross-origin. |
|
|
| Environments API | **None**; `jobs.<id>.environment` is ignored by act_runner | Model QA/Prod gates as `workflow_dispatch` approval inputs (D-004 / D-013); optionally create `qa` and `prod` branches as a visible stand-in |
|
|
| `repository_dispatch` trigger | **Not supported** | Cross-repo trigger via `workflow_dispatch` API: `POST /api/v1/repos/{owner}/{repo}/actions/workflows/{workflow_id}/dispatches` called from a step using `$GITEA_TOKEN` |
|
|
| Reusable workflows (`workflow_call`) | Supported | `acdl/.gitea/workflows/pipeline.yml` called via `uses: continuous-intelligence/acdl/.gitea/workflows/pipeline.yml@milestone/v1.0-initial` |
|
|
| `workflow_dispatch` | Supported (trigger + API) | Used for the manual-approval fallback and the issue workflow's cross-repo trigger |
|
|
| `issues.opened` trigger | Supported | Drives the L3B issue-trigger workflow in `acdl-contracts` |
|
|
| `act_runner` labels | Single label only (`runs-on: ubuntu-latest`) | All workflows use `runs-on: ubuntu-latest` |
|
|
| Context | `${{ gitea.* }}` and `${{ github.* }}` both work | Workflows use `gitea.*` for clarity |
|
|
|
|
### Branch pinning rule
|
|
|
|
The reusable workflow in the `acdl` repo lives on `milestone/v1.0-initial`
|
|
(that is the repo's default branch). `uses:` references from `acdl-contracts`
|
|
must pin to `@milestone/v1.0-initial`, not `@main` (the `acdl` repo has no
|
|
`main` branch). The new repos `acdl-contracts` and `acdl-evidence` use
|
|
`default_branch: "main"` (D-015) so their default branch exists immediately
|
|
for pushes.
|
|
|
|
### Default verification toolchain
|
|
|
|
There is no `package.json`; ACDL is bash + python stubs. The verification gate
|
|
substitutes `bash -n` and `python -m py_compile` for `npm run typecheck`, and
|
|
per-phase `scripts/verify_phaseNN.sh` for `npm test`. `npm run build` is a
|
|
no-op (no build step). See PERSONAS.md / VERIFICATION note.
|
|
|
|
## L1 module schema (Phase 02 research)
|
|
|
|
Each L1 module lives at `modules/l1/<name>/` with exactly two files:
|
|
|
|
- `manifest.yaml` — declares the L1's identity + a flat `inputs:` map.
|
|
Schema (D-017):
|
|
```yaml
|
|
name: l1-eks-fargate # matches the folder name
|
|
kind: l1 # literal "l1"; substrate-agnostic
|
|
description: <one-line>
|
|
inputs:
|
|
<key>:
|
|
description: <one-line>
|
|
type: string # only "string" allowed (flat, max-depth-1)
|
|
```
|
|
- `mock_apply.sh` — uniform stub per D-007 + D-018:
|
|
```bash
|
|
#!/usr/bin/env bash
|
|
set -euo pipefail
|
|
echo "[L1: <name>] applying..."
|
|
sleep 1
|
|
echo "[L1: <name>] OK"
|
|
exit 0
|
|
```
|
|
`mock_apply.sh` does NOT read input values; the manifest is for traceability
|
|
and for Phase 03's `mock_executor.sh` to enumerate the L1s in an L2.
|
|
|
|
### L1 list (fixed per REQ-02 / D-019)
|
|
|
|
| Folder | Description |
|
|
|--------|-------------|
|
|
| `l1-eks-fargate` | Serverless container compute substrate |
|
|
| `l1-iam-role` | Identity and access role primitive |
|
|
| `l1-lambda` | Event-driven function primitive |
|
|
| `l1-api-gateway` | HTTP routing primitive |
|
|
| `l1-eventbridge` | Event bus primitive |
|
|
| `l1-sqs` | Queue primitive |
|
|
| `l1-s3` | Object store primitive |
|
|
| `l1-cloudwatch` | Observability primitive |
|
|
|
|
L1 modules are single-purpose, substrate-agnostic, max-depth-1 (per
|
|
PROJECT.md Constraints). They do not compose with other L1s.
|
|
|
|
## L2 module schema + core scripts (Phase 03 research)
|
|
|
|
### L2 manifest.yaml schema (D-020)
|
|
|
|
```yaml
|
|
name: l2-commodity-price-feed # matches the folder name
|
|
kind: l2 # literal "l2"
|
|
description: <one-line>
|
|
l1s: # ordered list of L1 references
|
|
- name: l1-eks-fargate # MUST match an existing L1 folder name
|
|
inputs:
|
|
cluster_name: price-feed-cluster
|
|
region: us-east-1
|
|
cpu_arch: arm64
|
|
- name: l1-lambda
|
|
inputs:
|
|
function_name: price-ingest
|
|
runtime: python3.11
|
|
handler: index.handler
|
|
# ... up to 5 L1 references per L2 (max-depth-5 per REQ-05; L2->L1 is depth 1)
|
|
```
|
|
|
|
L2s reference L1s **by name only** (no path); `mock_executor.sh` resolves
|
|
the name to `modules/l1/<name>/`.
|
|
|
|
### L2 list (fixed per REQ-04)
|
|
|
|
| Folder | Description | L1s (per S&P Global Energy / Platts use cases) |
|
|
|--------|-------------|------------------------------------------------|
|
|
| `l2-invoice-service` | Billing + invoicing microservice | `l1-eks-fargate`, `l1-iam-role`, `l1-lambda`, `l1-sqs`, `l1-s3` |
|
|
| `l2-commodity-price-feed` | Real-time price ingestion | `l1-eks-fargate`, `l1-lambda`, `l1-api-gateway`, `l1-eventbridge`, `l1-s3` |
|
|
| `l2-energy-analytics-api` | Historical query API | `l1-eks-fargate`, `l1-api-gateway`, `l1-lambda`, `l1-s3`, `l1-cloudwatch` |
|
|
| `l2-regulatory-reporting` | Compliance + reporting | `l1-eks-fargate`, `l1-iam-role`, `l1-lambda`, `l1-sqs`, `l1-s3` |
|
|
|
|
Each L2 references exactly 5 L1s (within the max-depth-5 constraint; L2→L1
|
|
is depth 1, so depth-5 is generous but the spec caps composition depth at
|
|
5 — the count is 5 to demonstrate a realistic composed stack).
|
|
|
|
### contract.yaml schema (D-021)
|
|
|
|
```yaml
|
|
stack: l2-commodity-price-feed # MUST match an existing L2 folder name
|
|
inputs: # top-level params for the L2 (optional)
|
|
environment: dev
|
|
owner: platform-team
|
|
public-ingress: false # bool; true triggers POLICY_VIOLATION:PUBLIC_INGRESS
|
|
```
|
|
|
|
The `public-ingress` key is the only policy-enforced field in Phase 03.
|
|
Phase 04's pipeline reads `contract.yaml`, runs `policy_checker.py`, then
|
|
`mock_executor.sh` to apply the L2.
|
|
|
|
### state.json shape (D-022)
|
|
|
|
`mock_executor.sh` writes `state.json` to its working directory:
|
|
|
|
```json
|
|
{
|
|
"l2": "l2-commodity-price-feed",
|
|
"l1s": [
|
|
{"name": "l1-eks-fargate", "applied": true, "exit_code": 0},
|
|
{"name": "l1-lambda", "applied": true, "exit_code": 0},
|
|
...
|
|
],
|
|
"contract": {
|
|
"stack": "l2-commodity-price-feed",
|
|
"inputs": {...},
|
|
"public-ingress": false
|
|
}
|
|
}
|
|
```
|
|
|
|
### audit.json event + hash chain (D-023)
|
|
|
|
`audit.json` is a JSON array of event objects. `evidence_writer.py`
|
|
appends one event per call. Hash chain:
|
|
|
|
1. Construct the event dict with `hash` set to empty string.
|
|
2. Serialize via `json.dumps(event, sort_keys=True, separators=(",", ":"))` — canonical JSON (deterministic key order, no whitespace).
|
|
3. Compute `hash = sha256(canonical_json.encode("utf-8")).hexdigest()`.
|
|
4. Set `event["hash"] = hash`.
|
|
5. Append to `audit.json`.
|
|
|
|
Genesis event (when `audit.json` is empty or missing):
|
|
|
|
```json
|
|
{
|
|
"seq": 0,
|
|
"ts": "2026-07-21T13:00:00Z",
|
|
"stage": "genesis",
|
|
"event": "audit log initialized",
|
|
"prev_hash": "GENESIS",
|
|
"hash": "<sha256 of the canonical json of this event with hash empty>"
|
|
}
|
|
```
|
|
|
|
Subsequent events: `seq = prev.seq + 1`, `prev_hash = prev.hash`.
|
|
|
|
### Core script I/O contracts
|
|
|
|
| Script | Input | Output | Exit |
|
|
|--------|-------|--------|------|
|
|
| `mock_executor.sh` | `<contract.yaml path>` (argv[1]); reads L2 manifest from `modules/l2/<contract.stack>/manifest.yaml` | writes `state.json` to cwd; prints per-L1 progress | 0 on all-L1s-pass; non-zero on any L1 failure |
|
|
| `policy_checker.py` | `<contract.yaml path>` (argv[1]) | stdout: `POLICY_PASS` or `POLICY_VIOLATION:PUBLIC_INGRESS` | 0 on pass; 1 on violation |
|
|
| `confidence_signal.py` | `<contract.yaml path>` (argv[1]); calls policy_checker | stdout: `{"score": 0.90|0.40, "reason": "..."}` | 0 always (per D-024; pipeline decides gate) |
|
|
| `evidence_writer.py` | argv: `--stage <dev|qa|prod|finalize|genesis>` `--event "<text>"` `--audit <path to audit.json>` (default `./audit.json`) | appends event to audit.json; prints the new event's hash + seq | 0 on success; 1 on I/O error |
|
|
| `l3b_agent_stub.py` | argv[1] = issue body text (or stdin if no argv); optional `-o <path>` (default stdout) | writes a `contract.yaml` (D-021 schema) with `stack` set by the D-008 keyword map | 0 on success; 1 on empty input | |