CIAgent Orchestrator
932923ee99
Nova Slides Render / render (push) Failing after 22s
---ci--- project: acdl phase: 6 milestone: v1.29 status: complete ---/ci---
724 lines
40 KiB
Markdown
724 lines
40 KiB
Markdown
# Nova — Architecture
|
||
|
||
> **Compressed.** The full v1.0–v1.24 architecture history (v1.1 spike
|
||
> scope, v1.2 build-out, v1.8–v1.16 addenda) is preserved verbatim at
|
||
> `.ciagent/archive/ARCHITECTURE-v1.0-v1.24.md`. This file retains the
|
||
> durable target architecture (§1–§12, the four layers + six cross-cutting
|
||
> concerns) + the three addenda that describe the **current state**:
|
||
> v1.11 (stateless adapter), v1.15 (Nova rebrand — current naming), and
|
||
> v1.17 (telemetry/observability layer + §12.7 Policy Engine Registry).
|
||
> Intermediate addenda (v1.1 spike scope, v1.2 build-out, v1.8/1.9/1.10/
|
||
> 1.12/1.13/1.14/1.16) describe evolved or superseded states and are
|
||
> preserved in the archive snapshot.
|
||
>
|
||
> Source of truth for **how**: `docs/architecture.md` (v0.2) is the
|
||
> upstream draft; this file is the Nova-repo operating copy, refined at
|
||
> phase boundaries. Where this file and `docs/vision.md` conflict, the
|
||
> vision wins.
|
||
|
||
## Status
|
||
|
||
Architecture is at **v0.2** upstream (`docs/architecture.md`). Milestone
|
||
v1.1 **finalized it to v1.0** in Phase 07 by resolving the 11 open
|
||
decisions (see `PROJECT.md` open-decision resolutions table). The v1.11
|
||
addendum (stateless adapter) and the v1.17 addendum (telemetry layer +
|
||
§12.7 Policy Engine Registry) record the current-state refinements.
|
||
|
||
## Overview
|
||
|
||
The platform is **four layers + six cross-cutting concerns**. The sixth
|
||
concern — the engine abstraction (§12) — is first-class, not an
|
||
implementation detail. The vision's "Two Consumer Surfaces, One Platform"
|
||
tenet binds everything: L3A and L3B converge on the same contract schema,
|
||
the same policy envelope, and the same evidence stream.
|
||
|
||
```
|
||
┌──────────── acdl-contracts ────────────┐
|
||
Developer ───▶ │ commit contract.yaml │ (L3A)
|
||
Citizen dev ──▶ │ Issue → agent → contract.yaml │ (L3B)
|
||
└────────────────┬───────────────────────┘
|
||
│ (push)
|
||
▼
|
||
┌──────────────────────┐
|
||
│ central pipeline │
|
||
│ (acdl repo, Gitea │
|
||
│ Actions / act_runner) │
|
||
└────────┬─────────────┘
|
||
│
|
||
┌─────────────────────────┼─────────────────────────┐
|
||
▼ ▼ ▼
|
||
contract→IR resolution policy (Checkov/Kyverno) confidence signal
|
||
│ │ │
|
||
▼ ▼ ▼
|
||
Terraform adapter ──▶ terraform plan ──▶ PolicyCheckResult ──▶ {score,band}
|
||
│ │
|
||
▼ ▼
|
||
dev (autonomous, ≥0.50) qa (HITL, ≥0.75) prod (HITL, ≥0.90) dr (HITL, ≥0.95)
|
||
│
|
||
▼
|
||
DynamoDB outbox ──▶ S3 Object Lock (7-yr, source of truth) ──▶ GitHub audit repo (hot index)
|
||
│
|
||
▼
|
||
acdl-evidence (timeline UI)
|
||
```
|
||
|
||
## Layers
|
||
|
||
### Layer 1 — Foundational Primitives
|
||
Single-purpose, **engine-agnostic** primitive modules. L1 modules do
|
||
not compose with other L1s; L1 takes its environment as input. The L1
|
||
interface is defined against the **Target Stack IR**, not against
|
||
Terraform directly (the IR is shaped to round-trip to Terraform in v1,
|
||
per §12.1).
|
||
|
||
- No inter-L1 references. L1 may call Terraform data sources.
|
||
- Semver: interface → MAJOR, behavior → MINOR, lifecycle → PATCH (W3.D).
|
||
- Immutability on publication. 12-month deprecation window.
|
||
- AI refinement is a flag; the trigger is the W1.A joint condition.
|
||
|
||
### Layer 2 — Composed Stacks
|
||
Combine L1 primitives into deployable shapes. Each codebase maps to one
|
||
canonical L2 stack (`multiStack: true` only per W1.B). Shape X
|
||
(parameterized module) or Shape Y (thin-composition layer). Hierarchical
|
||
composition, max depth 5, only registered L1s. The thin-composition tree's
|
||
`wires` field is defined against the IR's relationship type, not a
|
||
Terraform module block.
|
||
|
||
Pipeline quality checks: secrets-in-plaintext, public ingress, IAM
|
||
wildcard, KMS key reference, tag compliance, naming convention. Restricted
|
||
from thin-composition: IAM principal creation, network boundary creation,
|
||
key/secret creation, external data transfer. Auto-promote after 3 observed
|
||
usages.
|
||
|
||
### Layer 3A — Developer Consumer Surface
|
||
Tag-based reference to the central pipeline template. Developer-owned
|
||
workflow file, no platform auto-sync. L3A and L3B are parallel paths, not a
|
||
progression. **W2.A (Path B):** tag for dev/qa, SHA for prod; platform CLI
|
||
resolves tag→SHA for prod-bound workflows.
|
||
|
||
### Layer 3B — Agentic Consumer Surface
|
||
Hybrid runtime, skill as markdown, agent as executor. Trust model: trust
|
||
and always verify on the platform side. Skill envelope (4 dimensions).
|
||
Stateless agents, all state in the platform. `profile: agentic` marker
|
||
unlocks `naturalLanguageIntent`, `confidenceAtSubmission`, `agentTrace`.
|
||
Initial skill catalog (BA.A): web API, worker, scheduled job, static asset,
|
||
basic observability bootstrap.
|
||
|
||
Environment progression:
|
||
|
||
| Environment | Autonomy | Attester | Gate |
|
||
|---|---|---|---|
|
||
| dev | Full autonomy (no HITL) | — | Confidence ≥ 0.50, all six inputs present |
|
||
| qa | Held for attestation | QA | GitHub Deployment approval + full QA matrix (§10) |
|
||
| prod | Held for attestation | SRE | GitHub Deployment approval + full SRE matrix (§10) |
|
||
| dr | Held for attestation | SRE | GitHub Deployment approval + dr-drill evidence |
|
||
|
||
**Staging is removed.** Dev is the only autonomous environment.
|
||
|
||
## Cross-cutting concerns
|
||
|
||
### Central pipeline template (§6)
|
||
JSON Schema (draft 2020-12) with a thin domain wrapper. Central repo +
|
||
generated client libraries. Multi-stage validation: schema → policy → NFR →
|
||
confidence. Distributed enrichment. GitOps reconciler (K8s API; cdlc-gitops
|
||
state → CRDs) + Terraform execution layer (§12.5). The pipeline emits one
|
||
`PolicyCheckResult` per policy rule; the confidence signal consumes them as
|
||
one normalized input.
|
||
|
||
### Contract schema (§7)
|
||
Central repo + generated client libraries. Strict fail-fast at schema
|
||
stage, multi-stage validation with reason codes from a published
|
||
vocabulary. **W3.E:** per-env mandatory inputs —
|
||
- dev: `stack`, `environment`
|
||
- qa adds: `validation.e2eSuite`, `validation.loadTest`
|
||
- prod adds: `runbook`, `dashboard`, `oncall`
|
||
- dr adds: `drDrillRef`
|
||
- `inputs` always optional; `profile: agentic` fields optional everywhere.
|
||
|
||
### Confidence signal (§8)
|
||
Six canonical inputs, weighted sum with per-input breakdown. Per-env
|
||
thresholds: dev ≥ 0.50, qa ≥ 0.75, prod ≥ 0.90, dr ≥ 0.95. Structured output
|
||
`{ score, band, perInput, reasonCodes }`. 1-year storage, no retraining in
|
||
v1. Halt with explicit reason on missing input.
|
||
|
||
Policy input = list of `PolicyCheckResult` records (engine-agnostic).
|
||
Severity → penalty: critical → hard override to mandatory block; high →
|
||
-0.2; medium → -0.05; low → -0.01; info → 0.0. One critical finding
|
||
hard-overrides the score regardless of all other inputs.
|
||
|
||
**BA.B:** thresholds frozen for v1; tuning begins v1.2 (quarterly FP/FN
|
||
tracking; override = Infra & Ops + SRE joint sign-off, itself a
|
||
confidence-event).
|
||
|
||
### Audit and evidence stream (§9)
|
||
Tiered ledger: **S3 with Object Lock in compliance mode** (cold, source of
|
||
truth, 7-year retention) + **GitHub audit repo** (`acdl-evidence`, hot
|
||
query index, not part of the chain). Daily checkpoints. Event schema: JWS
|
||
detached signature, `prev_event_hash` chain, controlled-vocabulary
|
||
`event_type`. Outbox pattern: local durable outbox + async worker.
|
||
|
||
Outbox database = **DynamoDB**. RPO = 0 (synchronous write to local outbox
|
||
before contract submission ack); RTO = async worker's dead-letter recovery.
|
||
Single-region in v1. The outbox also stores per-contract QA and prod
|
||
approver identities (the only durable record outside GitHub's audit log).
|
||
|
||
> **v1.17 update:** the Decision Ledger (SQLite hash-chain, D-121) is the
|
||
> pilot's audit record. S3 Object Lock / JWS (D-083) is deferred — see
|
||
> the v1.17 addendum below.
|
||
|
||
### Human-in-the-Loop mechanics (§10)
|
||
Pre-execution gates. qa, prod, dr are PR-based attestation gates backed by
|
||
GitHub Environments with required reviewers. No partial deployment to roll
|
||
back on rejection (qa, prod); dr is a separate GitHub Deployment against a
|
||
separate cluster/region.
|
||
|
||
Reviewer routing: GitHub CODEOWNERS + Environment required reviewers
|
||
(qa → QA; prod → SRE; dr → SRE). CODEOWNERS routes, does not enforce
|
||
identity distinctness.
|
||
|
||
**Separation of duties** (platform-internal, not GitHub-native, not Kyverno
|
||
in v1): on dev→qa promotion the platform writes the QA approver's GitHub
|
||
identity to the DynamoDB outbox keyed by `contractId`; on qa→prod it reads
|
||
the stored QA approver and the new SRE approver; if equal, it blocks, emits
|
||
`SEPARATION_OF_DUTIES_VIOLATION`, and routes a halt artifact to SRE on-call.
|
||
|
||
Full 8-concern attestation matrix (functional, performance, security
|
||
posture, contract NFRs, operational readiness, incident response,
|
||
capacity/cost, resilience) — see `docs/architecture.md` §10.4.
|
||
|
||
Timeout: 1 business day = warn + escalate; 2 business days = auto-freeze +
|
||
re-submit (linked via `supersedes`). Rejection returns the contract to HELD;
|
||
the audit chain is extended, not torn up.
|
||
|
||
### Agentic stack (§11)
|
||
Hybrid runtime: platform-managed control plane + consumer-owned agent.
|
||
Versioned, signed skill catalog over MCP. Skill envelope enforced on
|
||
invocation and result submission. Consumer-owned skill execution; the
|
||
platform does not run the skill. Stateless agents, all state in the
|
||
platform. Skills are reviewed for sensitive data before release (Infra &
|
||
Ops owns the review; it is the mandatory release gate).
|
||
|
||
### Engine execution (§12) — the binding constraint
|
||
**Target Stack IR** (locked): an engine-neutral description of resources
|
||
(typed inputs/outputs/NFRs), relationships (single parent per child),
|
||
composition (tree, max depth 5), and policy hooks. The L1 registry, L2
|
||
thin-composition tree, contract YML, and PolicyCheckResult schema are all
|
||
defined against the IR — none against any specific engine.
|
||
|
||
**Engine adapters** are the only engine-specific code. An adapter
|
||
compiles the IR into an engine execution plan. **v1 ships exactly one
|
||
adapter: the Terraform adapter.** v2+ may add OpenTofu, Pulumi, K8s CRDs
|
||
without architectural change.
|
||
|
||
v1 reality: the IR is shaped to round-trip cleanly to Terraform (nearly
|
||
isomorphic). As more adapters appear, the IR gets more expressive and the
|
||
adapters gain translation logic; the L1 content, the YML standard, and
|
||
the thin-composition tree do not change.
|
||
|
||
> **v1.11 update:** the Terraform adapter is now a **stateless assembler**
|
||
> (~80 lines, emits `module "x" { source }` blocks) — see the v1.11
|
||
> addendum below. The §12 "thin layer that translates IR → Terraform
|
||
> variable/output blocks" framing is superseded by the stateless-assembler
|
||
> model; the L1-owns-its-shape invariant is the new contract.
|
||
|
||
State storage: S3 (state) + DynamoDB (locking), cloud-managed,
|
||
single-region in v1.
|
||
|
||
Policy toolchain: **Checkov** for Terraform plan policy (the L2 checks +
|
||
tag/naming); **Kyverno** for K8s-native/platform-internal policy; **OPA**
|
||
reserved for cross-resource cases, explicitly last resort.
|
||
|
||
> **v1.25 update:** the policy toolchain is now unified under the
|
||
> swappable `PolicyEngine` protocol — see §12.7 below. Checkov and Wiz
|
||
> remain as raw-finding adapters feeding into kyverno-json meta-policies.
|
||
|
||
**Policy result normalization (§12.6):** the confidence signal consumes a
|
||
normalized `PolicyCheckResult` schema, not raw engine output.
|
||
|
||
```json
|
||
{
|
||
"contractId": "uuid",
|
||
"evaluatedAt": "ISO-8601",
|
||
"engine": "checkov | kyverno | opa",
|
||
"ruleId": "CKV_AWS_24 | KYVERNO_NO_PRIVILEGED | ...",
|
||
"severity": "critical | high | medium | low | info",
|
||
"result": "pass | fail | skipped | error",
|
||
"message": "human-readable",
|
||
"evidence": { "...engine-specific, opaque to the signal..." },
|
||
"resourceRef": "IR-typed resource identifier"
|
||
}
|
||
```
|
||
|
||
Execution layer: GitHub/Gitea Actions in the central pipeline repo. State
|
||
locking via DynamoDB. **AWS credentials via OIDC federation — long-lived
|
||
credentials are forbidden** (§12.5). The platform does not run
|
||
`terraform apply` against a developer's workstation; all execution is in
|
||
the central pipeline.
|
||
|
||
Registry maintenance: L1 publication updates the L1 registry in the same
|
||
PR. The registry is the IR-typed contract, not a Terraform-specific
|
||
variable schema.
|
||
|
||
Contract→IR resolution: the contract declares intent in IR-typed terms;
|
||
the pipeline resolves it to a target stack (list of L1 instances + inputs +
|
||
relationships); the Terraform adapter compiles the target stack to a plan.
|
||
|
||
---
|
||
|
||
## v1.11 Addendum — Stateless Adapter + Pipeline-Driven Lifecycle Testing (current state)
|
||
|
||
**Stateless adapter (D-098).** `adapters/terraform/adapter.py` rewritten
|
||
from a 918-line monolith (3 constant tables `TYPE_MAP`/`INPUT_MAP`/
|
||
`OUTPUT_MAP`, 39 type-specific branches) to a ~80-line stateless assembler.
|
||
Each L1 module ships a real `terraform/` module dir
|
||
(`versions.tf`/`variables.tf`/`locals.tf`/`main.tf`/`outputs.tf`) owning
|
||
its resource shape, nested blocks, and defaults. The adapter reads the
|
||
registry, emits a root `main.tf` instantiating each L1 as
|
||
`module "x" { source = "..." }` with resolved inputs and wired refs.
|
||
|
||
**Terraform owns lifecycle (D-101).** `scripts/run_platform.sh` gains
|
||
`--apply` and `--destroy` modes. Python never runs terraform.
|
||
`scripts/verify_deploy_microservice.py` is deleted.
|
||
|
||
**Pipeline-driven testing (D-102).** A `modules-lifecycle` pipeline
|
||
(Gitea + GitHub, byte-identical) matrix-runs each L1 module's
|
||
`examples/{simple,complex}.yml` contracts through apply→modify→destroy
|
||
against live AWS. No per-module Python/pytest. The "test" = the pipeline
|
||
cell going green.
|
||
|
||
**Single platform VPC (D-105).** `terraform/platform/main.tf` owns ONE
|
||
VPC; the microservice composition references it via
|
||
`terraform_remote_state` (data source). State keys are deterministic and
|
||
env-aware (`spike/{contract.id}/{contract.environment}/terraform.tfstate`).
|
||
|
||
**NOVA_LIFECYCLE_MODE (v1.12, REQ-134; renamed ACDL→NOVA in v1.15 P2).**
|
||
The lifecycle pipeline defaults to plan-only (fast, no AWS mutation, no
|
||
cost). A CI variable `NOVA_LIFECYCLE_MODE` (default `plan`) overrides to
|
||
`full` for the real apply→modify→destroy. (P2–P4 dual-read fallback to
|
||
`ACDL_LIFECYCLE_MODE`; fallback removed in P5 per the v1.15 addendum.)
|
||
|
||
---
|
||
|
||
## v1.15 Addendum — Nova Rebrand (current naming)
|
||
|
||
**Milestone:** v1.15-Nova. A full rebrand from **ACDL** / "Agentic Cloud
|
||
Delivery Platform" → **Nova** / "The New Dawn of DevSecOps — security as
|
||
a seamless enabler of fast deployments." This is a **Major milestone**
|
||
(breaking): consumer-facing path, env var prefixes, SSM path, AWS tag
|
||
keys, and AWS resource names all change. v1.15 tags run on the **v1.15.x
|
||
minor line**: `v1.15.0` (P0) → `v1.15.4` (P5 final = release). (G-104
|
||
binding.)
|
||
|
||
### Naming conventions (rebranded — current)
|
||
|
||
| Convention | Before (v1.0–v1.14) | After (v1.15+) | Phase |
|
||
|------------|---------------------|-----------------|-------|
|
||
| Project name | `ACDL` / "Agentic Cloud Delivery Platform" | `Nova` / "The New Dawn of DevSecOps" | P1 |
|
||
| Tagline | "Consumers declare intent; the platform delivers safe production deployment through an agentic stack" | (retained) **+** "The New Dawn of DevSecOps — security as a seamless enabler of fast deployments" | P1 |
|
||
| Schema `$id` URL | `https://acdl.cloudinit.dev/schemas/...` | `https://nova.cloudinit.dev/schemas/...` | P1 |
|
||
| Gitea release title | `ACDL vX.Y.Z` | `Nova vX.Y.Z` | P1 (forward only) |
|
||
| Env var prefix | `ACDL_*` (21 vars) | `NOVA_*` (dual-read fallback in P2–P4; removed P5) | P2 |
|
||
| Env loader | scattered `os.environ.get("ACDL_*")` | centralized `core/env.py` `get_env()` (D-108) | P2 |
|
||
| Consumer contract path | `.acdl/contract.yml` | `.nova/contract.yml` | P2 |
|
||
| Checkov custom rule file | `acdl_tagging.py` | `nova_tagging.py` | P2 |
|
||
| Checkov tag-key enforcement | `acdl:*` (hard) | `nova:*` (warn P2, hard P3) | P2/P3 |
|
||
| SSM parameter path | `/acdl/{env}/{contractId}/{output}` | `/nova/{env}/{contractId}/{output}` | P3 |
|
||
| AWS tag keys | `acdl:owner|environment|contract|cost-center|ref` | `nova:owner|environment|contract|cost-center|ref` | P3 |
|
||
| ABAC session policy match | `acdl:*` tags | `nova:*` tags (parallel-tag period) | P3 |
|
||
| DynamoDB tables | `acdl-contracts`, `acdl-change-requests` | `nova-contracts`, `nova-change-requests` (scan+copy) | P4 |
|
||
| Lambda (ingestor) | `acdl-contract-ingestor` (role/policy/function) | `nova-contract-ingestor` | P4 |
|
||
| Secrets Manager secret | `acdl/github-token` | `nova/github-token` | P4 |
|
||
| SNS topic | `acdl-sod-halt` | `nova-sod-halt` | P4 |
|
||
| Security group | `acdl-ecs-sg` | `nova-ecs-sg` | P4 |
|
||
| KMS alias | `alias/acdl-platform` | `alias/nova-platform` | P4 |
|
||
| ECS cluster/service/task | `acdl-microservice` | `nova-microservice` | P4 |
|
||
| ECR repo | `acdl-microservice` | `nova-microservice` (re-push) | P4 |
|
||
| IAM user/policy | `acdl-spike-runner` (+policy) | `nova-spike-runner` (re-bootstrap) | P4 |
|
||
| S3 state bucket | `acdl-tfstate-581513795199-us-east-1` | `nova-tfstate-581513795199-us-east-1` (`-migrate-state`) | P4 |
|
||
| ALB name prefix | `acdl-alb` | `nova-alb` | P4 |
|
||
| Lambda default table names | `CONTRACTS_TABLE` default `acdl-contracts` | default `nova-contracts` (D-111) | P4 |
|
||
|
||
### Unchanged conventions (out of scope)
|
||
|
||
- **S&P Global Energy visual theme** (`sp-theme.json`, deck CSS: #D6002A
|
||
red, Akkurat Pro) — client branding, not the Nova product brand (D-107).
|
||
- **config.json `release.gitea.repo`** = `acdl` — real Gitea repo name
|
||
unchanged (D-105). Doc URLs updated to `nova` for prose only.
|
||
- **Git branch/tag naming** — `milestone/v*`, `phase/*`, `v*` semver; no
|
||
brand name present (D-112: flat-branch convention preserved).
|
||
- **Past Gitea release titles** — existing releases keep `ACDL vX.Y.Z`.
|
||
|
||
> The full migration ordering (P1–P5), capability gate, and rollback
|
||
> runbook are preserved in `.ciagent/archive/ARCHITECTURE-v1.0-v1.24.md`
|
||
> §v1.15 Addendum.
|
||
|
||
---
|
||
|
||
## v1.17 Addendum — Strategic Direction, Leadership Metrics & Unified Story (current telemetry layer)
|
||
|
||
The v1.17 milestone added a telemetry/observability layer, a Decision
|
||
Ledger, a metrics export pipeline, a unified narrative deck, and a
|
||
durable strategic-direction artifact. This addendum documents the
|
||
architecture; the full research findings are in RESEARCH.md §v1.17.
|
||
|
||
### New components
|
||
|
||
| Component | Path | Purpose |
|
||
|-----------|------|---------|
|
||
| Event envelope | `core/metrics/event_envelope.py` | CloudEvents 1.0 envelope + `platform.*` semantic conventions (P1, REQ-187) |
|
||
| Per-run manifest writer | `core/metrics/run_manifest.py` | Emits `nova.run.started/completed/failed` events + writes `metrics/runs/<run_id>.json` (P1, REQ-187) |
|
||
| Decision Ledger (SQLite) | `core/metrics/decision_ledger.py` | Extends `outbox_writer.py` → SQLite append-only hash-chain table; `ai.decision.made` + `attestation.recorded` events + outcome backfill (P1, REQ-188, D-121) |
|
||
| Infracost post-processor | `core/metrics/infracost_adapter.py` | Runs Infracost on plan JSON; emits `nova.cost.estimated{delta_usd}` (P1, REQ-187, D-120) |
|
||
| Metrics collector | `core/metrics/collector.py` | Reads all grounded signals (files + events) → SQLite cold store at `metrics/nova_metrics.db` (P2, REQ-189) |
|
||
| PowerBI export | `core/metrics/powerbi_export.py` | Emits CSV/JSON views to `metrics/powerbi/` (fact + dim + 8 deferred placeholder views) (P3, REQ-190) |
|
||
| Metrics schemas | `schemas/metrics_*.schema.json` | Schemas for all event types + fact/dim tables (P1–P2, REQ-187/189) |
|
||
| Metrics catalog | `docs/METRICS.md` + `docs/metrics/<kpi>.md` | Canonical catalog + per-KPI definition-of-success docs (P4, REQ-195, D-127) |
|
||
| Unified narrative deck | `docs/presentations/nova-no-humans-platform.md` | Merged deck: Problem→Vision→How→Proof→Roadmap; x3 arc at deck+slide level (P5, REQ-196/197, D-130) |
|
||
| Strategic direction | `.ciagent/NORTH_STAR.md` | PO-authored durable vision/objectives/anti-goals/targets; read by CIAgent in every future `/ci-run` (P0, REQ-185/186) |
|
||
|
||
### Modified components
|
||
|
||
| Component | Change | Phase |
|
||
|-----------|--------|-------|
|
||
| `core/outbox_writer.py` | Extended to emit to SQLite append-only hash-chain table (Decision Ledger); `ai.decision.made` + `attestation.recorded` events added (P1, D-121) | P1 |
|
||
| `scripts/run_platform.sh` | Per-run manifest writer invoked; `$WORK/*.json` persisted to `metrics/runs/`; Infracost post-processor invoked after plan (P1) | P1 |
|
||
| `core/hitl_gates.py` | Emits `attestation.recorded` event to Decision Ledger on qa/prod/dr gate (P1, D-132) | P1 |
|
||
| `core/confidence_signal.py` | Emits `nova.confidence.computed` + `nova.ai.decision.made` events (P1, D-122) | P1 |
|
||
| `adapters/terraform/policy/checkov_adapter.py` | Emits `nova.policy.evaluated` event (P1) | P1 |
|
||
| `core/regression_verify.py` | Emits `nova.capability.verified` event; CAP-023 (metrics collector) + CAP-024 (deck structure) added (P1, P6) | P1, P6 |
|
||
| `pyproject.toml` | `addopts` gains `--junitxml=metrics/test-results.xml` + `--json-report` (P1, D-120) | P1 |
|
||
| `docs/presentations/` | Two old decks retired (deleted); unified deck added (P5, D-130) | P5 |
|
||
|
||
### Telemetry/observability layer architecture (D-120)
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────────┐
|
||
│ Nova platform components (existing) │
|
||
│ run_platform.sh · confidence_signal · checkov_adapter · │
|
||
│ hitl_gates · regression_verify · outbox_writer · contract_ingestor │
|
||
└────────────────────┬──────────────────────────────────────────────┘
|
||
│ CloudEvents 1.0 envelope (new emitters, P1)
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────────────┐
|
||
│ metrics/events.jsonl (append-only CloudEvents log) │
|
||
│ metrics/runs/<run_id>.json (per-run manifests) │
|
||
│ metrics/decision_ledger.db (SQLite hash-chain, D-121) │
|
||
│ metrics/test-results.xml (junit, P1) │
|
||
└────────────────────┬──────────────────────────────────────────────┘
|
||
│ collector reads (P2)
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────────────┐
|
||
│ metrics/nova_metrics.db (SQLite cold store, D-126) │
|
||
│ fact_run · fact_capability · fact_policy_check · fact_confidence │
|
||
│ fact_test · fact_decision · fact_cost_estimate │
|
||
│ dim_capability · dim_milestone │
|
||
│ + 8 empty placeholder views (deferred metrics) │
|
||
└────────────────────┬──────────────────────────────────────────────┘
|
||
│ powerbi_export (P3)
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────────────┐
|
||
│ metrics/powerbi/ (CSV/JSON views, folder connector, D-129) │
|
||
│ → PowerBI dashboards (external) │
|
||
└─────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
**Hot path: deferred (D-126).** No live ops dashboard; SQLite is
|
||
cold-only (batch/historical). The hot path activates when live AWS is
|
||
re-provisioned (D-096 lift — the v1.26 milestone lifts this for the pilot
|
||
estate).
|
||
|
||
### NORTH_STAR integration point (REQ-186)
|
||
|
||
`.ciagent/NORTH_STAR.md` is read by CIAgent in context-loading for all
|
||
future milestones. The integration mechanism: a reference from
|
||
`PROJECT.md` + `ARCHITECTURE.md` (this section) + a config entry in
|
||
`config.json` (`strategic_direction_file: ".ciagent/NORTH_STAR.md"`)
|
||
that the run workflow reads at SPECIFY. This ensures the strategic
|
||
direction survives across milestones without being overwritten by status
|
||
updates.
|
||
|
||
### §12.7 — Policy Engine Registry (v1.25, REQ-291 — current)
|
||
|
||
The policy-engine abstraction is first-class: a swappable `PolicyEngine`
|
||
protocol so the engine may change without touching the confidence
|
||
signal, the pipeline, or the `PolicyCheckResult` schema. This is the
|
||
**swap boundary** that keeps the platform's compliance posture
|
||
replaceable (Strategic Objective #2 — provable trust via a replaceable
|
||
substrate, not a vendor lock-in).
|
||
|
||
```
|
||
contract.yml ─┐ ┌─→ list[PolicyCheckResult] ─┐
|
||
stack IR ─────┼─→ PolicyEngine.evaluate ├─→ list[PolicyCheckResult] ─┼─→ confidence_signal
|
||
plan JSON ────┤ (protocol) └─→ list[PolicyCheckResult] ─┘ (engine-agnostic,
|
||
PCR list ─────┘ unchanged)
|
||
│
|
||
▼
|
||
┌─ KyvernoJsonEngine (shells to `kj scan`; engine: "kyverno")
|
||
└─ OpaEngine (future — same protocol; engine: "opa")
|
||
|
||
checkov/wiz ──→ raw findings ──→ (merged PCR list is the meta-policy payload)
|
||
```
|
||
|
||
**The protocol (`core/policy_engine.py`):**
|
||
```python
|
||
class PolicyEngine(Protocol):
|
||
@property
|
||
def name(self) -> str: ...
|
||
def is_configured(self) -> bool: ...
|
||
def evaluate(self, payload, policy_dir: Path, contract_id: str) -> list[dict]: ...
|
||
```
|
||
|
||
**The registry** reads `config.json.policy.engine` (default
|
||
`"kyverno-json"`) and returns the active engine. A `NullEngine` is the
|
||
fallback when the `policy` key is absent (emits `SKIPPED` PCRs —
|
||
backward compatibility for tests that don't set the key). The
|
||
confidence signal is **untouched** — it already consumes
|
||
`list[PolicyCheckResult]` engine-agnostically (§12.6). v1.25 only
|
||
changes *who produces* the PCR list, not *what* the list is.
|
||
|
||
**Engine enum reuse (D-116):** kyverno-json PCR records carry
|
||
`engine: "kyverno"` (no new enum value). The `engine` field records the
|
||
policy-engine *family*, not the specific binary. The K8s Kyverno adapter
|
||
and the kyverno-json engine are distinguished by `ruleId` prefix
|
||
(`KYVERNO_` vs `KJ_`) and `evidence` payload shape (`namespace`/`kind`
|
||
vs `assertion`/`jmespath`).
|
||
|
||
**Defense-in-depth (D-119):** the declarative meta-policy
|
||
`block-on-any-critical` (asserts no PCR has `severity: critical` +
|
||
`result: fail`) is the *source of truth* for "critical = block". The
|
||
`confidence_signal.py` `PENALTY["critical"]: None` hard-override stays
|
||
as the *imperative* safety net — the meta-policy runs *before* the
|
||
confidence signal (produces PCRs that flow in), the hard-override runs
|
||
*inside* it (the last gate). Removing the hard-override would make the
|
||
"critical = block" guarantee depend on a single policy file — a
|
||
regression in provable trust.
|
||
|
||
**Graceful degradation (D-120):** `KyvernoJsonEngine.is_configured()`
|
||
returns false when `which kj` is absent → `evaluate()` returns a single
|
||
`SKIPPED` PCR (`ruleId: "KJ_ENGINE_NOT_CONFIGURED"`). The platform
|
||
functions without the binary (the "platform functions without AI /
|
||
deterministic scripts" tenet holds — kyverno-json is deterministic, not
|
||
AI; the `is_configured()` guard ensures the platform runs even when the
|
||
binary is not installed).
|
||
|
||
### §12.8 — Pilot Estate (v1.26, live)
|
||
|
||
The first real consumer estate is **`nova-blockchain-exchange`** — a
|
||
blockchain stock exchange on a homegrown Proof-of-Authority chain,
|
||
equities only, dev only (D-020/D-200/D-201). The live apply landed on
|
||
2026-08-19 against AWS account `581513795199`. This is the estate that
|
||
activated the Post-Pilot metric denominators (see `docs/METRICS.md`).
|
||
|
||
**The live apply (run id `blkex-pilot-apply-v0.2`):**
|
||
- Target: account `581513795199`, environment `dev`, autonomous (no
|
||
HITL — dev is the only autonomous environment, confidence ≥ 0.50).
|
||
- The microservice L2 composition (ECS Fargate running nginx) + the
|
||
`dynamodb` L1 (the `nova-blkex-ledger-dev` table) + the `s3` L1 (the
|
||
`nova-blkex-blocks-dev-581513795199-us-east-1` bucket).
|
||
- The platform VPC prerequisite (`vpc-0d7c8867e6cc080f1` + 6 subnets +
|
||
the ECS SG) is read via `terraform_remote_state` — the L2 composition
|
||
does not own the network boundary (the "restricted from
|
||
thin-composition" rule from §Layer 2).
|
||
- Confidence signal: score **0.800**, band **pass**; `human_override`
|
||
false; `escalation_reason` absent (clean apply).
|
||
|
||
**The Gitea adapter (SPEC §10 Q1):** Gitea Actions does not support
|
||
cross-repo `uses:`, so the consumer's `deploy.yml` is an **inline
|
||
adapter** — `actions/checkout@v4` the consumer, `actions/checkout@v4`
|
||
`acdl/acdl` @ `ref: v1.25` into `platform/`, then
|
||
`bash platform/scripts/run_platform.sh ...`. The platform's own
|
||
`.github/workflows/deploy.yml` stays as the GitHub Actions reference
|
||
impl (the reusable `workflow_call` workflow). See `adapters/README.md`
|
||
§Consumers for the adapter note.
|
||
|
||
**The Decision Ledger evidence stream** (the apply produces these
|
||
events in order):
|
||
```
|
||
nova.confidence.computed (score 0.800, band pass)
|
||
│
|
||
▼
|
||
nova.ai.decision.made (decision_id blkex-pilot-apply-v0.2,
|
||
chosen_action pass, human_override false)
|
||
│
|
||
▼
|
||
nova.attestation.recorded (dev = no HITL gate; the record exists,
|
||
the gate is a no-op in the autonomous env)
|
||
│
|
||
▼
|
||
nova.run.completed (apply succeeded)
|
||
│
|
||
▼
|
||
nova.outcome.backfilled (outcome pending → succeeded, REQ-317;
|
||
backfilled_at 2026-08-19T03:05:04Z)
|
||
```
|
||
The SQLite hash-chain is valid (0 breaks). S3 Object Lock / JWS
|
||
(D-083) stays deferred — the SQLite Decision Ledger is the pilot's
|
||
audit record (D-204).
|
||
|
||
**Live outputs (account 581513795199):**
|
||
- ALB DNS: `app-254671247.us-east-1.elb.amazonaws.com`
|
||
- ECS service: `arn:aws:ecs:us-east-1:581513795199:service/nova-cluster/nova-microservice`
|
||
- DynamoDB table: `nova-blkex-ledger-dev` (PK `block_index`, PAY_PER_REQUEST)
|
||
- S3 bucket: `nova-blkex-blocks-dev-581513795199-us-east-1` (versioning + SSE)
|
||
|
||
The full evidence (every ARN, the confidence JSON, the Decision Ledger
|
||
rows, the module-completeness gaps the live apply uncovered) is in
|
||
`.ciagent/archive/P4-PILOT-RUN-EVIDENCE-v1.26.md` (archived v1.27).
|
||
|
||
### §12.9 — Secret Rotation (v1.26 P3 W7, SPEC §5.9 — current)
|
||
|
||
The platform-managed scheduled workflow `workflows-src/rotate-aws-key.yml`
|
||
rotates the `NOVA_AWS_*` static key daily (cron `0 0 * * *`) and on
|
||
`workflow_dispatch`. v0.2 scope: the mechanism exists (SPEC §5.9 —
|
||
exists-not-ran); the v0.2 deploy uses the currently-active key. The
|
||
rotation is idempotent — `scripts/rotate_spike_key.sh` deactivates the old
|
||
key only after the new one propagates to the consumer's Actions secret
|
||
store, verified by a post-PUT GET; on upload/verify failure the old key is
|
||
left Active and the run exits non-zero. The synced workflow file is
|
||
forge-agnostic (REQ-230): forge base URL / owner / consumer repo come from
|
||
repository secrets (`NOVA_FORGE_*`, `NOVA_CONSUMER_REPO`), not literals.
|
||
|
||
### §12.10 — Nova-idp Identity Layer (v1.28, current)
|
||
|
||
Nova owns its identity layer end-to-end. Two (optionally three) Lambda
|
||
functions + four DynamoDB tables + one KMS asymmetric signing key + one
|
||
kyverno-json ABAC policy. **No Cognito, no IAM Identity Center (INV-15).**
|
||
The `nova-cli` Lambda layer carries the Nova wheel + `argon2-cffi` +
|
||
`cryptography` + `pyjwt` + the `kj` Go binary, making the same code
|
||
importable in both the CLI and the Lambda (REQ-329 dual-use, NFR-7).
|
||
|
||
**Components:**
|
||
|
||
- `nova-idp-auth` Lambda — sign-up, sign-in, session creation. Argon2id
|
||
password hashing (D-228: bundled abi3 wheel; fail-closed on
|
||
`ImportError`, no pure-Python fallback). DynamoDB: `nova-users`
|
||
(PK `user_id`, Argon2id `password_hash`), `nova-sessions` (PK
|
||
`session_id`, TTL `expires_at`), `nova-password-resets` (PK
|
||
`reset_token`, TTL 15m). Function URL with IAM auth.
|
||
- `nova-idp-token-vend` Lambda — accepts a PAT (or session token),
|
||
validates revocation (`nova-pats.GetItem(jti, ConsistentRead=True)` —
|
||
D-229, 60s SLO), evaluates the kyverno-json ABAC policy at
|
||
`platform/abac/token-vend.policy` (D-227, INV-17), KMS-signs an
|
||
ECDSA P-256 JWT (`ES256`), converts DER→raw ECDSA signature (RFC 7515
|
||
§3.1.3), returns the OIDC token. The `policy_version` (git SHA,
|
||
D-231) is recorded in every `token.vend.allowed/denied` audit event.
|
||
- `nova-idp-jwks` Lambda (optional, separation of concerns) — function
|
||
URL with `AuthType: NONE` (public key only), `Cache-Control: max-age=3600`.
|
||
`kms.get_public_key` → DER SPKI → JWK via `cryptography`. Custom
|
||
domain + WAF via CloudFront is OPTIONAL (`--public-jwks-domain` flag
|
||
on `nova idp setup`, D-230).
|
||
- `nova-pats` DynamoDB table — PK `jti`, GSI1 `sub` (list PATs for
|
||
user), GSI2 `pat_hash` (lookup by hash). Only the hash stored (not
|
||
raw PAT, REQ-343). Revoked PATs retained for audit.
|
||
|
||
**CLI surface (`nova` package, greenfield):**
|
||
|
||
- Entry point: `[project.scripts] nova = "nova.cli:main"` (argparse-only,
|
||
no click/typer — repo convention). `nova/cli.py` auto-discovers
|
||
`nova/<module>.py` subcommands via `pkgutil.iter_modules`, dispatches,
|
||
emits the `cli.invocation` audit event (INV-12) with `mode`,
|
||
`selection_reason`, `credential_type`, `command`, `args`.
|
||
- Each `nova/<module>.py` is ≤50 lines, delegates to `core/` (CAP-034
|
||
AST scan). Subgroups: `nova auth login/revoke/status`, `nova idp
|
||
setup --check/--apply/--verify`, `nova init`, `nova apply --local`.
|
||
- `core/mode_resolver.py` — flag → env (`NOVA_CLIENT_MODE`) → credential
|
||
type → `sys.stdin.isatty()` (D-226). CLI-only; Lambdas don't resolve
|
||
modes. Property-tested with `hypothesis` (REQ-349).
|
||
- `core/env.py:+synthesize_local_env()` — synthesizes a local env dict
|
||
from a contract + `--local` flag (REQ-330). No cloud provisioning.
|
||
|
||
**Packaging (NFR-6, CAP-035):**
|
||
|
||
- CI publishes a wheel to CodeArtifact AND a Lambda layer with identical
|
||
version strings on every merge affecting `core/`/`adapters/`/`nova/`.
|
||
Version mapping recorded in SSM `/nova/layer/nova-cli/version`.
|
||
If either publish fails, the merge is blocked (REQ-323).
|
||
- `nova cli-action` composite action at
|
||
`.github/actions/nova-cli/action.yml`, referenced by both GitHub +
|
||
Gitea (`uses: continuous-intelligence/acdl/.github/actions/nova-cli@v1.28`).
|
||
Python 3.12 pinned. Byte-identical behavior verified by CI matrix
|
||
(REQ-326, NFR-11).
|
||
|
||
**Data flows:**
|
||
|
||
1. Sign-up → `nova-idp-auth` → Argon2id → `nova-users` PutItem → session
|
||
→ `nova-sessions` PutItem → return session token.
|
||
2. Token vend (hot path) → `nova-idp-token-vend` → `nova-pats` strong
|
||
read (revocation) → kyverno-json ABAC eval → if allow → KMS sign →
|
||
DER→raw → return OIDC JWT. Audit at every step.
|
||
3. JWKS fetch → `nova-idp-jwks` → `kms.get_public_key` → DER→JWK →
|
||
`{"keys":[...]}`. Cached 1h at CloudFront (if custom domain) / client.
|
||
4. PAT revoke → `nova auth revoke --pat <jti>` → `nova-pats.UpdateItem(
|
||
status=revoked)` → audit. Strong read on next vend → 403 (within 60s).
|
||
|
||
**`nova idp setup` (REQ-340, NFR-10):** generates a CloudFormation
|
||
template (raw dict → JSON, no troposphere dep), presents for review
|
||
(`$PAGER` + resource summary), requires explicit `y/N` approval before
|
||
`cloudformation deploy --capabilities CAPABILITY_IAM`. `--check` reports
|
||
prerequisites + IAM policy delta; `--verify` runs the KMS round-trip
|
||
test. New IAM grants required: `cloudformation:*`, `codeartifact:*`.
|
||
|
||
### §12.11 — Platform Ops Reposplit (v1.29, current)
|
||
|
||
Platform operations are a Terraform-controlled discipline that lives
|
||
outside the engineering repo, grounded in Vision §4 (Domain
|
||
Boundaries — *the platform begins where the artifact is compiled and
|
||
ends where it runs in production under operational guardrails*). Two
|
||
repos, two ownership surfaces:
|
||
|
||
- **`acdl/acdl` (GitHub)** — engineering. Authors `publish.yml` + the
|
||
artifacts (Lambda zip, layer wheel, Python wheel, ECR container
|
||
image with the static `kj` binary). Each tag `v1.29.x` produces a
|
||
GitHub Release with SHA-256-verified artifacts (REQ-354, D-235
|
||
tag-pin handoff). Engineering ends at the compiled artifact.
|
||
- **`nova-platform-ops` (Gitea-private, OPER-PRIV, REQ-359)** —
|
||
operations. Authors the Terraform modules
|
||
(`networking`/`kms`/`identity`/`contract-ingest`/`bootstrap`/`edge`)
|
||
that bring those artifacts live in `581513795199`. Operations begins
|
||
at the live platform under guardrails. No GitHub mirror; CIAgent has
|
||
no presence there.
|
||
|
||
The handoff between the two repos is the **tag-pin** (D-235):
|
||
`nova-platform-ops` declares `local.nova_platform_version` +
|
||
`local.kj_source_sha` and resolves substrates through a single
|
||
`data.aws_ecr_image.kj_image`.
|
||
|
||
**The `kj` substrate (KJ-LOCKSTEP, REQ-371):** `kj` (a compiled Go
|
||
binary, pinned v0.0.3 in `platform/abac/kj-version.txt`, distinct from
|
||
the kyverno-json engine) has exactly **one identity**: one ECR image
|
||
digest shared by the production Lambda runtime
|
||
(`aws_lambda_function.nova_idp_token_vend.image_uri`) and its
|
||
defensive Fargate fallback
|
||
(`aws_ecs_task_definition.kj.container_definitions[0].image`). A
|
||
`lifecycle.precondition` on both image-bearing resources enforces at
|
||
every `terraform plan` that both `image_uri` attributes resolve to the
|
||
same digest via `data.aws_ecr_image.kj_image`. No second pipeline, no
|
||
second SHA pin (D-238). KJ-STATIC: the binary is compiled
|
||
`CGO_ENABLED=0` and `file(1)` reports `statically linked, no shared
|
||
library` before embedding.
|
||
|
||
**Covered-reference REQ tracking pattern:** the 14 covered-reference
|
||
REQs (355-366, 371) are authored in `nova-platform-ops` (out-of-band).
|
||
CIAgent in `acdl` tracks them for milestone completeness; their
|
||
verification surface is the M1/M1.5/M2 cutover gates documented in
|
||
the operator guide. The operator guide lists each covered-reference
|
||
REQ with its gate entry + verification command + a "Result" column
|
||
that the operator attests after running the gate in
|
||
`nova-platform-ops` CI. P6 audit verifies every covered-reference REQ
|
||
has a non-empty, green Result (grill CF-2/G-5). M1.5 green (3
|
||
consecutive rebuilds of the 12-item spike, operator-attested in the
|
||
guide) is the HARD P6 ship gate (grill CF-1/G-2.1).
|
||
|
||
**Operator guide pointer:** `docs/operator-guide-platform-ops.md`
|
||
(REQ-OPS-GUIDE) — the operator-facing runbook covering the Day-0
|
||
cutover, M1.5 verification gate, M2 handoff loop, rollback, KMS
|
||
rotation, JWKS reachability via CloudFront edge (INV-18), PITR
|
||
restore, PAT revocation, edge config, Fargate standby health, cost,
|
||
artifact-mirror fallback, and the cutover gates table.
|
||
|
||
**JWKS edge (INV-18, D-233):** the JWKS endpoint is the only public
|
||
read surface of the live platform. CloudFront + OAC pinning
|
||
(`AuthType: AWS_IAM` on the Function URL — NOT `NONE`,
|
||
`OriginAccessControlOriginType: lambda`, `SigningBehavior: always`)
|
||
replaces direct Lambda Function URL exposure. Direct Function URL →
|
||
403; via-CloudFront → 200. |