From ca7d41c1adbd984d0b694e0f686d866db8bc5645 Mon Sep 17 00:00:00 2001 From: Jon Chery Date: Tue, 4 Aug 2026 19:12:18 +0000 Subject: [PATCH] =?UTF-8?q?docs(P00):=20research=20findings=20=E2=80=94=20?= =?UTF-8?q?v1.17=20telemetry=20signal=20inventory=20+=20reference=20archit?= =?UTF-8?q?ecture=20+=20metric=20scorecard=20+=20deck=20research?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ---ci--- project: acdl phase: 0 milestone: v1.17 status: research ---/ci--- --- .ciagent/ARCHITECTURE.md | 82 +++++++++++ .ciagent/CHECKPOINT.json | 6 +- .ciagent/PERSONAS.md | 158 ++++++++++++++++++-- .ciagent/RESEARCH.md | 305 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 535 insertions(+), 16 deletions(-) diff --git a/.ciagent/ARCHITECTURE.md b/.ciagent/ARCHITECTURE.md index 2fb56f6..86368c3 100644 --- a/.ciagent/ARCHITECTURE.md +++ b/.ciagent/ARCHITECTURE.md @@ -797,3 +797,85 @@ return `Skipped` when the resources are absent (`NoSuchBucket`/ `ResourceNotFoundException`). `RegressionReport.passed` is `all(r.status in ("Verified", "Skipped"))`. The gate passes at 18 Verified + 4 Skipped (0 Decayed/Broken). + +## v1.17 Addendum — Strategic Direction, Leadership Metrics & Unified Story (2026-08-04) + +The v1.17 milestone adds a telemetry/observability layer, a Decision +Ledger, a metrics export pipeline, a unified narrative deck, and a +durable strategic-direction artifact. This addendum documents the +architecture; the full research findings are in RESEARCH.md §v1.17. + +### New components + +| Component | Path | Purpose | +|-----------|------|---------| +| Event envelope | `core/metrics/event_envelope.py` | CloudEvents 1.0 envelope + `platform.*` semantic conventions (P1, REQ-187) | +| Per-run manifest writer | `core/metrics/run_manifest.py` | Emits `nova.run.started/completed/failed` events + writes `metrics/runs/.json` (P1, REQ-187) | +| Decision Ledger (SQLite) | `core/metrics/decision_ledger.py` | Extends `outbox_writer.py` → SQLite append-only hash-chain table; `ai.decision.made` + `attestation.recorded` events + outcome backfill (P1, REQ-188, D-121) | +| Infracost post-processor | `core/metrics/infracost_adapter.py` | Runs Infracost on plan JSON; emits `nova.cost.estimated{delta_usd}` (P1, REQ-187, D-120) | +| Metrics collector | `core/metrics/collector.py` | Reads all grounded signals (files + events) → SQLite cold store at `metrics/nova_metrics.db` (P2, REQ-189) | +| PowerBI export | `core/metrics/powerbi_export.py` | Emits CSV/JSON views to `metrics/powerbi/` (fact + dim + 8 deferred placeholder views) (P3, REQ-190) | +| Metrics schemas | `schemas/metrics_*.schema.json` | Schemas for all event types + fact/dim tables (P1–P2, REQ-187/189) | +| Metrics catalog | `docs/METRICS.md` + `docs/metrics/.md` | Canonical catalog + per-KPI definition-of-success docs (P4, REQ-195, D-127) | +| Unified narrative deck | `docs/presentations/nova-no-humans-platform.md` | Merged deck: Problem→Vision→How→Proof→Roadmap; x3 arc at deck+slide level (P5, REQ-196/197, D-130) | +| Strategic direction | `.ciagent/NORTH_STAR.md` | PO-authored durable vision/objectives/anti-goals/targets; read by CIAgent in every future `/ci-run` (P0, REQ-185/186) | + +### Modified components + +| Component | Change | Phase | +|-----------|--------|-------| +| `core/outbox_writer.py` | Extended to emit to SQLite append-only hash-chain table (Decision Ledger); `ai.decision.made` + `attestation.recorded` events added (P1, D-121) | P1 | +| `scripts/run_platform.sh` | Per-run manifest writer invoked; `$WORK/*.json` persisted to `metrics/runs/`; Infracost post-processor invoked after plan (P1) | P1 | +| `core/hitl_gates.py` | Emits `attestation.recorded` event to Decision Ledger on qa/prod/dr gate (P1, D-132) | P1 | +| `core/confidence_signal.py` | Emits `nova.confidence.computed` + `nova.ai.decision.made` events (P1, D-122) | P1 | +| `adapters/terraform/policy/checkov_adapter.py` | Emits `nova.policy.evaluated` event (P1) | P1 | +| `core/regression_verify.py` | Emits `nova.capability.verified` event; CAP-023 (metrics collector) + CAP-024 (deck structure) added (P1, P6) | P1, P6 | +| `pyproject.toml` | `addopts` gains `--junitxml=metrics/test-results.xml` + `--json-report` (P1, D-120) | P1 | +| `docs/presentations/` | Two old decks retired (deleted); unified deck added (P5, D-130) | P5 | + +### Telemetry/observability layer architecture (D-120) + +``` +┌─────────────────────────────────────────────────────────────────────┐ +│ Nova platform components (existing) │ +│ run_platform.sh · confidence_signal · checkov_adapter · │ +│ hitl_gates · regression_verify · outbox_writer · contract_ingestor │ +└──────────────────────┬──────────────────────────────────────────────┘ + │ CloudEvents 1.0 envelope (new emitters, P1) + ▼ +┌─────────────────────────────────────────────────────────────────────┐ +│ metrics/events.jsonl (append-only CloudEvents log) │ +│ metrics/runs/.json (per-run manifests) │ +│ metrics/decision_ledger.db (SQLite hash-chain, D-121) │ +│ metrics/test-results.xml (junit, P1) │ +└──────────────────────┬──────────────────────────────────────────────┘ + │ collector reads (P2) + ▼ +┌─────────────────────────────────────────────────────────────────────┐ +│ metrics/nova_metrics.db (SQLite cold store, D-126) │ +│ fact_run · fact_capability · fact_policy_check · fact_confidence │ +│ fact_test · fact_decision · fact_cost_estimate │ +│ dim_capability · dim_milestone │ +│ + 8 empty placeholder views (deferred metrics) │ +└──────────────────────┬──────────────────────────────────────────────┘ + │ powerbi_export (P3) + ▼ +┌─────────────────────────────────────────────────────────────────────┐ +│ metrics/powerbi/ (CSV/JSON views, folder connector, D-129) │ +│ → PowerBI dashboards (external) │ +└─────────────────────────────────────────────────────────────────────┘ +``` + +**Hot path: deferred (D-126).** No live ops dashboard; SQLite is +cold-only (batch/historical). The hot path activates when live AWS is +re-provisioned (D-096 lift). + +### NORTH_STAR integration point (REQ-186) + +`.ciagent/NORTH_STAR.md` is read by CIAgent in context-loading for all +future milestones. The integration mechanism (to be finalized in P4): +a reference from `PROJECT.md` + `ARCHITECTURE.md` (this section) + a +config entry in `config.json` (`strategic_direction_file: +".ciagent/NORTH_STAR.md"`) that the run workflow reads at SPECIFY. This +ensures the strategic direction survives across milestones without +being overwritten by status updates. diff --git a/.ciagent/CHECKPOINT.json b/.ciagent/CHECKPOINT.json index 03dec57..8b842ec 100644 --- a/.ciagent/CHECKPOINT.json +++ b/.ciagent/CHECKPOINT.json @@ -1,12 +1,12 @@ { "phase": 0, - "stage": "clarify", + "stage": "research", "milestone": "v1.17", "phase_role": "pre_execution", "attempts": 0, - "updated_at": "2026-08-04T19:45:00Z", + "updated_at": "2026-08-04T20:15:00Z", "milestone_complete": false, "tag": null, "requirements": ["REQ-185"], - "notes": "CLARIFY validation pass complete. 14 decisions (D-120..D-132) locked. 4 low-severity items deferred to PLAN (attestation denominator wording tightened, REQ-194 owns Attestation Coverage, REQ-186 mechanism TBD in P4, P0 type=docs). No blocking ambiguities. NORTH_STAR draft is PO-approved pending interactive GRILL." + "notes": "RESEARCH complete. Signal inventory (grounding audit), telemetry reference architecture (Nova-native), metric-to-signal scorecard, deferred-decision ledger, deck-storytelling research. ARCHITECTURE.md v1.17 addendum + PERSONAS.md v1.17 roster written. 3 active personas (lead, backend, data); frontend deactivated. 6 assumptions logged." } \ No newline at end of file diff --git a/.ciagent/PERSONAS.md b/.ciagent/PERSONAS.md index 9d96375..3793226 100644 --- a/.ciagent/PERSONAS.md +++ b/.ciagent/PERSONAS.md @@ -1,23 +1,24 @@ --- project: acdl -milestone: v1.16 -generated_at: 2026-07-30 +milestone: v1.17 +generated_at: 2026-08-04 generator: lead-developer verification_toolchain: typecheck: "terraform validate && python3 -m py_compile core/**/*.py && python3 -m jsonschema schemas/*.schema.json" - test: "bash scripts/run_regression.sh # 22-capability gate (D-091/D-118)" + test: "bash scripts/run_regression.sh # 22-capability gate (D-091/D-118) + CAP-023/024 (v1.17)" build: "bash scripts/run_ci.sh # full local CI reproduction (lint+test+check-only)" note: | - Nova (formerly ACDL) has no package.json. The execute/verify/ship - workflows substitute `terraform validate` + `python -m py_compile` + - JSON Schema validation for npm run typecheck, the regression gate - (D-091, 22 capabilities) for npm test, and `bash scripts/run_ci.sh` - for npm run build. v1.11 testing is pipeline-driven (D-102); - v1.16 is NFR-only (no live apply by default; NOVA_LIFECYCLE_MODE= - plan). Roster carries forward from v1.11/v1.14/v1.15 unchanged. - frontend-engineer stays inactive (no frontend; decks are markdown = - lead-developer territory). No custom personas needed (no new - domains — onboarding is backend-engineer + data-engineer territory). + v1.17 adds a telemetry/observability layer (metrics emitters, SQLite + cold store, PowerBI export, Decision Ledger) + a unified narrative + deck + a durable NORTH_STAR.md. Three active personas: lead-developer + (coordination + deck narrative co-author), backend-engineer (event + emitters, outbox_writer extension, Infracost adapter), data-engineer + (SQLite store, schemas, PowerBI views, metrics collector). frontend- + engineer stays deactivated (no Nova web UI — dashboards are PowerBI, + not a Nova-built frontend; decks are markdown = lead-developer + territory). No new custom personas needed — the metrics domain maps + cleanly to data-engineer (schema/store/export) + backend-engineer + (emitters/instrumentation). --- # ACDL — Persona Roster (project-level, v1.11 RESTART) @@ -251,3 +252,134 @@ The regression gate (22 capabilities) must stay **22/22 Verified** throughout v1.16 — simplification must not regress any capability (D-118). P9 (end of Wave 2) and P21 (milestone complete) run the gate; P14 (end of Wave 3) is an offline mid-milestone checkpoint. + +--- + +# v1.17 Persona Roster — Strategic Direction, Leadership Metrics & Unified Story + +> v1.17 adds a telemetry/observability layer (P1–P3), a metrics catalog +> + NORTH_STAR integration (P4), a unified narrative deck (P5), a +> regression capability (P6), and a final review/ship (P7). Three +> active personas; frontend-engineer stays deactivated (no Nova web UI +> — dashboards are PowerBI, not a Nova-built frontend). + +## Active personas + +### lead-developer +- **Domain:** coordination + deck narrative +- **Active:** true +- **Phase-specific:** false +- **Reason:** Owns CIAgent metadata, the NORTH_STAR.md authoring + process (P0), the milestone decomposition, the unified narrative deck + co-authoring (P5 — the deck is markdown, which is lead-developer + territory per the established convention), and the final review/ship + (P7). Arbitrates persona conflicts (e.g., backend vs data on the + emitter/store boundary). +- **Territory:** `.ciagent/NORTH_STAR.md`, `.ciagent/PROJECT.md`, + `.ciagent/REQUIREMENTS.md`, `.ciagent/PLAN.md`, `.ciagent/RESEARCH.md`, + `.ciagent/ARCHITECTURE.md`, `docs/presentations/nova-no-humans-platform.md` + (NEW — unified deck source of truth), `docs/presentations/nova-no-humans-platform-marp.md`, + `docs/presentations/nova-no-humans-platform-talking-points.md`, + `docs/METRICS.md`, `docs/metrics/*.md` (per-KPI definition docs). + +### backend-engineer +- **Domain:** backend (event emitters + instrumentation) +- **Active:** true +- **Phase-specific:** false +- **Reason:** Owns the event emitters (P1): the CloudEvents envelope, + the per-run manifest writer, the `outbox_writer.py` extension to the + SQLite Decision Ledger, the Infracost post-processor, the + `hitl_gates.py` attestation event emission, the `confidence_signal.py` + decision event emission, the `checkov_adapter.py` policy event + emission, and the pytest `--junitxml` addopts change. Also owns the + `regression_verify.py` CAP-023/024 additions (P6). The emitter work + is the bridge between existing Nova components and the new metrics + layer — it touches the code paths that already exist. +- **Territory:** `core/metrics/event_envelope.py` (NEW), + `core/metrics/run_manifest.py` (NEW), + `core/metrics/infracost_adapter.py` (NEW), + `core/metrics/decision_ledger.py` (NEW — extends outbox_writer), + `core/outbox_writer.py` (extend to SQLite), + `core/hitl_gates.py` (emit attestation.recorded), + `core/confidence_signal.py` (emit ai.decision.made), + `adapters/terraform/policy/checkov_adapter.py` (emit policy.evaluated), + `scripts/run_platform.sh` (invoke manifest writer + Infracost), + `core/regression_verify.py` (CAP-023/024), + `pyproject.toml` (addopts --junitxml), + `tests/test_metrics_emitters.py` (NEW), + `tests/test_decision_ledger.py` (NEW). + +### data-engineer +- **Domain:** data (schema, SQLite store, PowerBI export) +- **Active:** true +- **Phase-specific:** false +- **Reason:** Reactivated with a new territory for v1.17: the metrics + collector (P2) and the PowerBI export (P3). Owns the schema design + (metrics_*.schema.json), the SQLite cold store (nova_metrics.db), the + fact/dimension table design, the 8 deferred placeholder views, and + the CSV/JSON export. The data-engineer's schema-first constraint + applies: all event types and fact/dim tables have JSON Schema + definitions before any code is written. The collector reads files + + events → SQLite; the export reads SQLite → CSV/JSON. This is the + heaviest data-territory work since v1.11's terraform modules. +- **Territory:** `core/metrics/collector.py` (NEW), + `core/metrics/powerbi_export.py` (NEW), + `schemas/metrics_*.schema.json` (NEW — event + fact/dim schemas), + `metrics/nova_metrics.db` (NEW — SQLite cold store), + `metrics/powerbi/` (NEW — CSV/JSON export dir), + `docs/METRICS_VIEWS.md` (NEW — schema doc for PowerBI views), + `tests/test_metrics_collector.py` (NEW), + `tests/test_powerbi_export.py` (NEW). + +## Deactivated personas + +### frontend-engineer +- **Domain:** frontend +- **Active:** false +- **Phase-specific:** false +- **Reason:** v1.17 has no Nova web UI. The leadership dashboards are + PowerBI (an external tool that ingests CSV/JSON files), not a + Nova-built frontend. The decks are markdown (lead-developer + territory). frontend-engineer stays deactivated, consistent with + v1.11–v1.16. Reactivates if a future milestone builds a Nova web UI. + +### lambda-engineer, platform-engineer, security-engineer +- **Active:** false (carried forward from v1.11) +- **Reason:** v1.17 does not touch the Lambda (beyond emitting events + from the existing hitl_gates/attestation_matrix), does not do IR- + shaped module authoring, and does not touch security adapters beyond + emitting policy.evaluated events. The existing components are + instrumented, not rewritten. + +## v1.17 phase assignment + +| Phase | Primary persona | Supporting | Territory | +|-------|----------------|------------|-----------| +| P0 pre-execution | lead-developer | — | `.ciagent/NORTH_STAR.md`, `PROJECT.md`, `REQUIREMENTS.md`, `RESEARCH.md`, `ARCHITECTURE.md`, `PERSONAS.md`, `PLAN.md` | +| P1 event-emitters | backend-engineer | data-engineer (schemas) | `core/metrics/event_envelope.py`, `run_manifest.py`, `decision_ledger.py`, `infracost_adapter.py`, `outbox_writer.py`, `hitl_gates.py`, `confidence_signal.py`, `checkov_adapter.py`, `run_platform.sh`, `pyproject.toml` | +| P2 metrics-collector | data-engineer | backend-engineer (event formats) | `core/metrics/collector.py`, `schemas/metrics_*.schema.json`, `metrics/nova_metrics.db` | +| P3 powerbi-export | data-engineer | — | `core/metrics/powerbi_export.py`, `metrics/powerbi/`, `docs/METRICS_VIEWS.md` | +| P4 metrics-catalog + north-star-integration | lead-developer | data-engineer (metric definitions) | `docs/METRICS.md`, `docs/metrics/*.md`, `PROJECT.md`, `ARCHITECTURE.md`, `config.json` | +| P5 deck-rebuild | lead-developer | — | `docs/presentations/nova-no-humans-platform*.md`, retire old decks | +| P6 regression-capability | backend-engineer | data-engineer (CAP-023 schema) | `core/regression_verify.py` (CAP-023, CAP-024) | +| P7 final-review-ship | lead-developer | all active (review) | `.ciagent/**`, review + audit + ship | + +## v1.17 domain priority + +`backend → data → lead` (the emitter work in P1 is the foundation; +data-engineer's collector + export in P2–P3 depends on P1's event +formats; lead-developer's catalog + deck in P4–P5 depends on the +metrics being grounded). + +## v1.17 verification toolchain + +``` +typecheck: terraform validate && python3 -m py_compile core/**/*.py adapters/**/*.py +test: bash scripts/run_regression.sh # 22-capability gate + CAP-023/024 (v1.17) +build: bash scripts/run_ci.sh # full local CI reproduction +``` + +The regression gate (22 capabilities + CAP-023 metrics collector + +CAP-024 deck structure) must pass at P6 and P7. CAP-009 (offline pytest +suite) must remain Verified after the `--junitxml` addopts change +(assumption A5). diff --git a/.ciagent/RESEARCH.md b/.ciagent/RESEARCH.md index 8843f26..755aed5 100644 --- a/.ciagent/RESEARCH.md +++ b/.ciagent/RESEARCH.md @@ -1188,3 +1188,308 @@ stays a future feature (D-113). - A4 (0.85): The regression gate (D-091, D-118) at P9 and P21 confirms "simplify without regressions" — 22/22 capabilities must stay Verified. The gate is the credible control for the simplification wave. + +--- + +# v1.17 Research — Strategic Direction, Leadership Metrics & Unified Story + +> Phase: research (P0). Milestone: v1.17. Status: research. +> Researcher: ci-researcher + explore agent (signal inventory). +> Autonomy: full. Decisions D-120..D-132 locked in the planning +> conversation (PROJECT.md). NORTH_STAR.md drafted (pending GRILL). + +## 1. Telemetry Signal Inventory (grounding audit) + +**Methodology:** every claim below is grounded in a concrete file path + +line number in `/root/acdl`. No speculation. The explore agent performed +a full sweep of the repo. The finding: **Nova has no metrics/telemetry/ +dashboard aggregation layer today.** What exists is a set of discrete, +structured, file-based signal artifacts (JSON reports, JSONL logs, +hash-chained outbox events, PR comments, Checkov JSON) plus unstructured +stdout logs. A metrics milestone must aggregate these existing signals +— it must not invent new ones without first adding emitters. + +### (a) Signals that EXIST TODAY and are STRUCTURED (groundable) + +| Signal | File / Emitter | Schema | Persistent? | +|--------|---------------|--------|-------------| +| Regression report (22 caps, status, duration_ms, gate) | `.ciagent/REGRESSION_REPORT.json` ← `core/regression_verify.py:643-667` | `regression_verify.py:82-91` | **Yes** (committed file) | +| Regression report (markdown mirror) | `.ciagent/REGRESSION_REPORT.md` | same | Yes | +| Checkpoint (milestone/phase/tag/regression summary) | `.ciagent/CHECKPOINT.json` (CIAgent-managed) | ad-hoc | Yes | +| PolicyCheckResult list (per-rule pass/fail/severity/resourceRef) | `$WORK/pcr.json` ← `run_platform.sh:395` + `checkov_adapter.py:50-71` | `schemas/policy_check_result.schema.json` | **No** (ephemeral `/tmp/`) | +| Confidence signal (score, band, perInput, reasonCodes) | `$WORK/signal.json` ← `run_platform.sh:412-426` + `confidence_signal.py:60-65` | `confidence_signal.py:60-65` | No (ephemeral) | +| Outbox event (hash-chained, CONFIDENCE_COMPUTED) | `$WORK/event.json` + `$WORK/outbox_item.json` ← `run_platform.sh:444-459` + `outbox_writer.py:44-56` | `audit_ledger_design.md:44-45,81-97` | No (ephemeral; live DynamoDB torn down D-096) | +| Resolved Target Stack | `$WORK/stack.json` ← `contract_resolver.py:581-603` | `schemas/stack.schema.json` | No (ephemeral) | +| Lambda return bodies (submit/report_error/validate_cr/onboard) | `core/lambda/contract_ingestor.py:171,265,284,392,446` | ad-hoc JSON | No (Lambda not live; local stub only) | +| DynamoDB CMDB rows (submitted/pending contracts) | `nova-contracts` table ← `contract_ingestor.py:160-170,433-445` | ad-hoc | **No** (table torn down D-096) | +| SSM parameters (deploy outputs) | `/nova///` ← `output_publisher.py:123-156` | ad-hoc | No (live AWS, torn down) | +| PR stage comment (mode, runId) | GitHub PR API ← `post_stage_comment.sh:34-48` + `deploy.yml:141` | markdown table | Yes (GitHub) | +| PR deploy-outputs comment | GitHub PR API ← `output_publisher.py:159-189` | markdown table | Yes (GitHub) | +| GitHub issue (deploy failure alert) | GitHub API ← `contract_ingestor.py:179-290` + `deploy.yml:143-152` | issue body | Yes (GitHub) | +| Local E2E result (stack_name, tier, outbox_events, chain_verified, lambda_status) | stdout JSON ← `core/local_emulators.py:498-508,519` | ad-hoc | No (stdout) | +| HITL gate result | `core/hitl_gates.py:87,90` + `run_platform.sh:179-185` | stdout `HITL PASS/BLOCK` | No (stdout) | +| Attestation matrix result | `core/attestation_matrix.py:184,187` | stdout `ATTESTATION PASS/BLOCK` | No (stdout) | +| Cost figures | `.ciagent/COST.md` (manual Cost Explorer query) | markdown table | Yes (manual, not automated) | + +### (b) Signals that EXIST but are UNSTRUCTURED (log-only) + +| Signal | Source | Format | +|--------|--------|--------| +| CI pipeline result | `scripts/run_ci.sh:70-71` | stdout banner `=== CI PIPELINE OK ===` | +| Platform stage banners + summaries | `scripts/run_platform.sh:222,241,258,263,315,383,411,442,463,490,496` | stdout `=== Step N: ... ===` + summary lines | +| Terraform init/validate/plan/apply/destroy logs | `$WORK/tf-*.log` ← `run_platform.sh:320,324,328,352,375` | raw terraform stdout (via `tee`) | +| Lifecycle test results | `scripts/run_lifecycle_test.sh` etc. | exit code only (no report file) | +| Decommission step counts | `scripts/run_decommission.sh:40,54` | stdout `decommission step N: M resources...` | +| Uptime endpoint count | `scripts/run_uptime.sh:72,87` | stdout `uptime: N endpoint(s) to monitor` | +| Onboarding prompt | `core/environment_check.py:57-81` | stdout text block | +| Pytest results | `pyproject.toml:25` (`-v --tb=short`) | stdout only (no junit/json) | +| sync_workflows result | `scripts/sync_workflows.py:56,53` | stdout `OK: 3 workflow pairs match` / `DRIFT: ...` | + +### (c) Proposed executive metrics with NO grounding today (DEFERRED) + +| Proposed metric | Why no grounding | Controlling decision | +|------------------|------------------|---------------------| +| Live infrastructure health (ECS running count, ALB 5xx, RPS) | Live AWS torn down; CAP-013..016 Skipped | **D-096** | +| Live outbox write rate / ledger append latency | DynamoDB outbox table absent | **D-096** | +| Tamper-evident ledger checkpoint count / JWS signature rate | S3 Object Lock + JWS + async worker deferred | **D-083** | +| Onboarding funnel: requested → granted conversion | Only "requested" (pending row) is emitted; no grant event | **D-113, D-114, D-119** | +| Time-to-provision (onboarding SLA) | Real AWS provisioning deferred | **D-113** | +| Cross-account role grant count | Offline-proven only, no live apply | **D-114** | +| Drift detection (scheduled terraform plan -detailed-exitcode) | Needs live AWS workspaces + a scheduler Nova doesn't have | **D-096** + no scheduler | +| GreenOps / carbon (WattTime/Electricity Maps API) | No grounding; new external API | future emitter | +| Predictive vs Reactive ratio | Requires an ML anomaly-forecasting service | future emitter | +| Multi-cloud normalization (Azure/GCP/K8s, FOCUS spec) | Nova is AWS-only | future | +| Red Team MTTR | No red-team program exists | future | +| Self-healing velocity | Nova has no auto-remediator | future emitter | +| SLA / unplanned downtime | Needs live service uptime monitoring against SLOs | **D-096** | +| Per-module lifecycle success rate over time | No structured report file written; only exit code | gap (no decision) | +| Test pass rate / test count time-series | No junit/json reporter configured | gap (add `--junitxml` to addopts) | +| Code coverage trend | `pytest-cov` installed but not in `addopts` | gap | +| Deploy frequency / lead time / MTTR (DORA) | No deploy-event emitter; pipeline runs not counted | gap | +| Policy pass rate time-series | `pcr.json` emitted but ephemeral; not persisted | gap (D-096 blocks live persistence) | +| Confidence score distribution over time | `signal.json` emitted but ephemeral | gap | +| Consumer adoption count / active consumers | `PROJECT.md:487` explicitly states "0 consumer adoption today" | honest scope | +| Cost time-series (automated) | `COST.md` is a one-shot manual query; no automated emitter | gap | + +**Bottom line:** the single richest existing structured signal is +`.ciagent/REGRESSION_REPORT.json` (22 capabilities × {status, tier, +duration_ms, detail} + summary counts + boolean gate). The next richest +is the per-run `$WORK/*.json` family (pcr.json, signal.json, event.json, +stack.json) — but these are **ephemeral** and **not persisted in CI**. +The lowest-friction grounding for a "no-humans" dashboard is therefore: +(1) regression report → capability health, (2) PR comments + GitHub +issues → deploy/failure activity, (3) add `--junitxml` to pytest → test +trend, (4) persist `$WORK/*.json` → policy/confidence/outbox time-series, +(5) extend outbox_writer → Decision Ledger, (6) add Infracost → +pre-apply cost estimates. + +## 2. Telemetry Reference Architecture (Nova-native adaptation) + +The PO provided a full distributed-system telemetry reference +architecture (CloudEvents 1.0 envelope, OpenTelemetry SDK, Kafka/NATS +event bus, Prometheus hot path, ClickHouse warehouse, QLDB decision +ledger, Infracost, drift detection, ML anomaly forecasting). Per +D-120, we adopt the **principles** but implement with **Nova-native +minimal tech**. The mapping: + +| Direction's principle | Nova-native implementation (v1.17) | +|---|---| +| Events are the source of truth; dashboards are projections | Hybrid (D-125): existing file signals stay as files; collector reads them and emits normalized CloudEvents into `metrics/events.jsonl` + SQLite. New emitters emit CloudEvents directly. | +| Every AI action is logged with confidence + alternatives | Decision Ledger (D-121): `outbox_writer.py` extended → SQLite append-only hash-chain table. `ai.decision.made` modeled from confidence_signal (D-122): decision_id=run_id, chosen_action=band, confidence=score, alternatives=perInput, human_override=HITL block. | +| Hot/cold storage split | Cold-only SQLite (D-126): `metrics/nova_metrics.db`. Hot path deferred (no live ops, D-096). | +| Read-only external integrators | Infracost (pre-apply, offline, reads plan JSON). Cloud billing CUR deferred (D-096). Carbon APIs deferred (future). | +| CloudEvents 1.0 envelope | Adopted. `core/metrics/event_envelope.py` defines the envelope + `platform.*` semantic conventions. | +| Decision Ledger = append-only with hash chain + outcome backfill | SQLite append-only table with hash chain (D-121). Outcome backfilled from apply.completed via decision_id → request_id correlation. Honors D-083 (no S3 Object Lock/JWS). | +| Cost governance: mandatory tags + Infracost pre-apply | Nova already enforces `nova:*` tags (nova_tagging.py, hard mode). Infracost added as plan post-processor (D-120). Post-apply CUR deferred (D-096). | +| Definition-of-success docs for every KPI | Per-KPI docs in `docs/metrics/` (D-127). | +| Replay-ability | SQLite store + JSONL event log are replayable by design. | + +### CloudEvents envelope (Nova-native) + +```json +{ + "specversion": "1.0", + "id": "", + "source": "nova.platform", + "type": "nova.run.completed", + "time": "", + "subject": "/", + "datacontenttype": "application/json", + "platform": { + "tenant_id": "acdl", + "run_id": "run-", + "contract_id": "", + "environment": "dev|qa|prod|dr", + "actor": {"type": "confidence-gate", "id": "confidence_signal"}, + "trace_id": "" + }, + "data": { + "duration_ms": 4800, + "stages": ["resolve", "adapt", "validate", "plan", "apply"], + "exit_code": 0, + "confidence": {"score": 0.94, "band": "pass", "perInput": {...}}, + "policy": {"passed": 12, "failed": 0, "skipped": 0}, + "hitl": {"gate": "dev", "result": "autonomous", "block": false}, + "cost_estimate_usd": -12.40, + "decision_id": "run-", + "outcome": "succeeded" + } +} +``` + +### Core event types (Nova-native minimum viable set) + +| Event type | Emitted by | Purpose | Grounding | +|---|---|---|---| +| `nova.run.started` | run_platform.sh | Measures demand; provisioning lead time start | new emitter (P1) | +| `nova.run.completed` | run_platform.sh | Run count, stage durations, exit, MTTR | new emitter (P1) | +| `nova.run.failed` | run_platform.sh | Failure count, MTTR numerator | new emitter (P1) | +| `nova.policy.evaluated` | checkov_adapter.py | Policy pass rate, compliance KPIs | grounded (pcr.json → P1 persists) | +| `nova.confidence.computed` | confidence_signal.py | Confidence distribution, decision accuracy | grounded (signal.json → P1 persists) | +| `nova.ai.decision.made` | outbox_writer.py (extended) | Decision Ledger entry | grounded (D-121, D-122) | +| `nova.attestation.recorded` | hitl_gates.py | Attestation Coverage, human-in-the-loop audit | grounded (D-132) | +| `nova.cost.estimated` | Infracost post-processor | Pre-apply cost estimate | new emitter (P1, Infracost) | +| `nova.capability.verified` | regression_verify.py | Capability health, regression gate | grounded (REGRESSION_REPORT.json) | +| `nova.test.completed` | pytest (junit XML) | Test count, pass rate | new (P1 adds --junitxml) | + +## 3. Metric-to-Signal Scorecard (the "no fabrication" contract) + +| Executive metric (NORTH_STAR target) | Status | Source / formula | Decision | +|---|---|---|---| +| Touchless Resolution Rate ≥99% | grounded (after P1) | runs without operational HITL block ÷ total runs (attestation gates excluded) | D-122, D-132 | +| Human Escalation Frequency <0.1% | grounded (after P1) | operational HITL blocks ÷ total runs (attestation sign-offs excluded) | D-122, D-132 | +| MTTR (p95) <60s | grounded (platform-run) | apply.failed.time → successful retry.time | D-131 | +| Predictive vs Reactive ≥3:1 | **deferred** | requires ML forecasting (future emitter) | future | +| AI Decision Accuracy ≥99.5% | grounded (after decision ledger) | decisions not followed by apply.failed/incident within 5min | D-121, D-122 | +| Drift Auto-Reversal ≥95% | **deferred** | requires drift detection (D-096 + scheduler) | D-096 | +| Cloud Spend Reduction ≥25% | partial | pre-apply estimate grounded (Infracost); actuals deferred (D-096 CUR) | D-120 | +| L1/L2 Ops Hours Avoided ≥70% | derived | formula: run count × manual baseline minutes × blended rate | D-127 | +| Platform ROI ≥250% | derived | formula: (labor savings + cloud savings + avoided downtime) ÷ platform op cost | D-127 | +| Decision Ledger Coverage 100% | grounded (this milestone) | outbox_writer.py → SQLite hash-chain | D-121 | +| Attestation Coverage 100% | grounded | hitl_gates.py + outbox approver_* attributes; prod/dr | D-132 | +| AI-Agent Intent Share ≥40% | future | no AI-agent consumers today; placeholder view | future | +| Capability health (18V+4S) | grounded | REGRESSION_REPORT.json | existing | +| Confidence score distribution | grounded (after P1) | signal.json → decision ledger | D-121 | +| Policy pass rate | grounded (after P1) | pcr.json → persisted | D-120 | +| Test count / pass rate | grounded (after P1) | pytest --junitxml | D-120 | +| Provisioning Lead Time | grounded (after P1) | run.started → run.completed | D-120 | +| Deployment Frequency | grounded (after P1) | count(run.completed) per day | D-120 | +| Deploy-failure alert count | grounded | GitHub issues via Lambda report_error (D-055) | existing | +| Cost figures (actuals) | manual one-shot | COST.md (Cost Explorer query) | existing | +| FTE Hours Saved / TRV | derived | formula over run count + COST.md | D-127 | +| Self-healing velocity | **deferred** | no auto-remediator | future | +| SLA / unplanned downtime | **deferred** | needs live service uptime (D-096) | D-096 | +| GreenOps / carbon | **deferred** | WattTime/Electricity Maps API (future) | future | +| Red Team MTTR | **deferred** | no red-team program | future | +| Multi-cloud normalization | **deferred** | Nova is AWS-only | future | +| Live CUR reconciliation | **deferred** | needs live AWS billing (D-096) | D-096 | + +## 4. Deferred-Decision Ledger (constraints on this milestone) + +| Decision | Scope | Grounding impact | +|----------|-------|------------------| +| D-096 | Live AWS torn down post-v1.11 | BLOCKS all live-AWS metrics (CAP-013..016 Skipped; live outbox; live state bucket; live CUR) | +| D-083 | S3 Object Lock + JWS + async worker deferred | BLOCKS tamper-evident ledger; v1.17 uses SQLite hash-chain instead | +| D-113/D-114/D-119 | Onboarding = request-path only; no auto-grant | BLOCKS onboarding funnel "granted" half | +| D-091/D-118 | Regression gate (D-091) gates milestone completion | ENABLES the strongest metric signal (REGRESSION_REPORT.json) | +| D-092 | Local emulating adapters | ENABLES offline E2E metrics (CAP-011/012) | +| D-055 | report_error Lambda action creates GitHub issues | ENABLES deploy-failure alert metric | +| D-050 | Publish deploy outputs to SSM + GitHub PR comment | ENABLES outputs-published metric | +| D-054/D-043/D-109 | Nova tagging standard (hard mode) | ENABLES tagging-compliance metric | +| D-084 | 8-concern attestation matrix | ENABLES attestation metrics (operator-supplied evidence artifacts) | +| D-089 | Signature verification skipped when signing key unset (dev/CI) | Signature metrics are no-ops in dev | + +## 5. Deck-Storytelling Research (x3 arc + per-slide benefit) + +### The "tell them x3" structure + +The PO's direction: "Tell them what you're going to tell them, then tell +them, then tell them what you told them." Applied at two levels: + +**Deck level (the 5-act arc):** +1. **Opening slide** = "what I'm going to tell you" — the full arc + preview: Problem → Vision → How → Proof → Roadmap. +2. **Body** (acts 1–5) = "tell them" — each act delivers its content. +3. **Closing slide** = "what I told you" — recap of the 5 acts + the ask. + +**Per slide:** +1. **Slide opens** with what it'll cover (1 line: "This slide shows X"). +2. **Slide delivers** the content (bullets, diagram, or table). +3. **Slide closes** with an explicit **"benefit of this stage" callout** + (1 line: "Benefit: you now know Y" or "Why this matters: Z"). + +### Fluidity conventions + +- **Transitions are written, not hand-waved.** Each slide's opening line + references the previous slide's close ("Having seen X, now consider Y"). +- **No disjointed jumps.** If a topic shift is needed, a bridge slide or + a transition sentence carries the audience across. +- **The arc is visible.** A small "act indicator" in the Marp footer + (e.g., `Act 3/5: How it works`) keeps the audience oriented. + +### Existing deck inventory (to be retired) + +Two decks exist today in `docs/presentations/`: +- `how-the-platform-works.md` (32,916 bytes) → marp → html → talking-points +- `the-developer-experience.md` (27,509 bytes) → marp → html → talking-points + +Both follow a 4-step process (source `.md` → Marp → HTML → talking-points) +documented in `docs/presentations/README.md`. Per D-130, both are merged +into one unified narrative deck and retired. + +### Grounded metrics already cited in existing decks + +- "22/22 auto-verifiable capabilities Verified" — **stale** vs current + REGRESSION_REPORT.json (18V+4S post-D-096). The unified deck must + derive this from the report, not copy the stale claim. +- Confidence thresholds: dev ≥0.50, qa ≥0.75, prod ≥0.90, dr ≥0.95 — + grounded in `core/confidence_signal.py:57` (THRESHOLDS). +- RPO = 0 (evidence write synchronous) — grounded in + `core/audit_ledger_design.md:27,103`. +- Cost figures — `how-the-platform-works.md:461`; cites COST.md. +- Confidence signal 6 inputs + weights — grounded in + `core/confidence_signal.py:40-47`. +- "~80-line stateless adapter" vs "918-line monolith" — grounded in + ROADMAP/RESEARCH prose. + +### Planned deck structure (for PLAN to detail) + +The unified deck "Nova — The No-Humans Infrastructure Platform": + +| Act | Slides | Content | Proof source | +|---|---|---|---| +| 1. Problem | 2–3 | The no-humans imperative; why operators are the bottleneck; the trust gap | NORTH_STAR vision | +| 2. Vision/Direction | 2–3 | Nova's vision; 4 strategic objectives; anti-goals; the attestation model (autonomy in operations, human at stage gates) | NORTH_STAR | +| 3. How it works | 3–4 | Contract → resolver → adapter → confidence → HITL gate; the Decision Ledger; the 8-concern attestation matrix | code grounding | +| 4. Proof (metrics) | 3–4 | Capability health (18V+4S); confidence distribution; policy pass rate; Decision Ledger coverage; Attestation Coverage; cost estimates; the grounded/derived/deferred honesty model | metrics export | +| 5. Roadmap/Ask | 2 | 12–18mo targets (committed); deferred metrics (honest); the ask | NORTH_STAR targets | + +Total: ~12–16 slides. Opening = arc preview; closing = recap + ask. + +## 6. Assumptions logged (v1.17) + +- A1 (0.9): No live AWS access during execution (consistent with + v1.11–v1.16). All metrics that require live AWS ship as placeholder + views. The Infracost integration runs offline (reads plan JSON). +- A2 (0.85): The Decision Ledger SQLite hash-chain is sufficient for + v1.17's audit needs. The full tamper-evident ledger (S3 Object Lock + + JWS, D-083) is a future milestone. The hash-chain provides + append-only + integrity verification locally. +- A3 (0.8): The "AI decision" framing (D-122) is honest: Nova's "AI" is + the confidence-gated policy engine (confidence_signal + HITL gate), + not an LLM planner. The deck and METRICS.md must frame this accurately + — overclaiming "AI" would violate the "no fabrication" constraint. +- A4 (0.85): The unified deck's "Proof" section cites only grounded + metrics with real numbers. Deferred metrics are shown as "Planned" + with the `Planned` badge. No + fabricated numbers in any slide. +- A5 (0.8): `--junitxml` + `--json-report` added to pytest addopts + does not break the existing test suite (the flags are additive; pytest + continues to run normally). CAP-009 (offline pytest suite passes) + must remain Verified after the change. +- A6 (0.75): Infracost is available as a CLI tool that can be installed + in the CI environment and run locally. It reads `terraform plan + -out=plan.tfplan` + `terraform show -json plan.tfplan` to produce a + cost estimate. No live AWS access required. If Infracost is not + available, the `cost.estimated` event is omitted (degraded mode, not + a failure).