docs(P00): research findings — v1.17 telemetry signal inventory + reference architecture + metric scorecard + deck research
---ci--- project: acdl phase: 0 milestone: v1.17 status: research ---/ci---
This commit is contained in:
Jon Chery
@@ -797,3 +797,85 @@ return `Skipped` when the resources are absent (`NoSuchBucket`/
|
||||
`ResourceNotFoundException`). `RegressionReport.passed` is
|
||||
`all(r.status in ("Verified", "Skipped"))`. The gate passes at 18
|
||||
Verified + 4 Skipped (0 Decayed/Broken).
|
||||
|
||||
## v1.17 Addendum — Strategic Direction, Leadership Metrics & Unified Story (2026-08-04)
|
||||
|
||||
The v1.17 milestone adds a telemetry/observability layer, a Decision
|
||||
Ledger, a metrics export pipeline, a unified narrative deck, and a
|
||||
durable strategic-direction artifact. This addendum documents the
|
||||
architecture; the full research findings are in RESEARCH.md §v1.17.
|
||||
|
||||
### New components
|
||||
|
||||
| Component | Path | Purpose |
|
||||
|-----------|------|---------|
|
||||
| Event envelope | `core/metrics/event_envelope.py` | CloudEvents 1.0 envelope + `platform.*` semantic conventions (P1, REQ-187) |
|
||||
| Per-run manifest writer | `core/metrics/run_manifest.py` | Emits `nova.run.started/completed/failed` events + writes `metrics/runs/<run_id>.json` (P1, REQ-187) |
|
||||
| Decision Ledger (SQLite) | `core/metrics/decision_ledger.py` | Extends `outbox_writer.py` → SQLite append-only hash-chain table; `ai.decision.made` + `attestation.recorded` events + outcome backfill (P1, REQ-188, D-121) |
|
||||
| Infracost post-processor | `core/metrics/infracost_adapter.py` | Runs Infracost on plan JSON; emits `nova.cost.estimated{delta_usd}` (P1, REQ-187, D-120) |
|
||||
| Metrics collector | `core/metrics/collector.py` | Reads all grounded signals (files + events) → SQLite cold store at `metrics/nova_metrics.db` (P2, REQ-189) |
|
||||
| PowerBI export | `core/metrics/powerbi_export.py` | Emits CSV/JSON views to `metrics/powerbi/` (fact + dim + 8 deferred placeholder views) (P3, REQ-190) |
|
||||
| Metrics schemas | `schemas/metrics_*.schema.json` | Schemas for all event types + fact/dim tables (P1–P2, REQ-187/189) |
|
||||
| Metrics catalog | `docs/METRICS.md` + `docs/metrics/<kpi>.md` | Canonical catalog + per-KPI definition-of-success docs (P4, REQ-195, D-127) |
|
||||
| Unified narrative deck | `docs/presentations/nova-no-humans-platform.md` | Merged deck: Problem→Vision→How→Proof→Roadmap; x3 arc at deck+slide level (P5, REQ-196/197, D-130) |
|
||||
| Strategic direction | `.ciagent/NORTH_STAR.md` | PO-authored durable vision/objectives/anti-goals/targets; read by CIAgent in every future `/ci-run` (P0, REQ-185/186) |
|
||||
|
||||
### Modified components
|
||||
|
||||
| Component | Change | Phase |
|
||||
|-----------|--------|-------|
|
||||
| `core/outbox_writer.py` | Extended to emit to SQLite append-only hash-chain table (Decision Ledger); `ai.decision.made` + `attestation.recorded` events added (P1, D-121) | P1 |
|
||||
| `scripts/run_platform.sh` | Per-run manifest writer invoked; `$WORK/*.json` persisted to `metrics/runs/`; Infracost post-processor invoked after plan (P1) | P1 |
|
||||
| `core/hitl_gates.py` | Emits `attestation.recorded` event to Decision Ledger on qa/prod/dr gate (P1, D-132) | P1 |
|
||||
| `core/confidence_signal.py` | Emits `nova.confidence.computed` + `nova.ai.decision.made` events (P1, D-122) | P1 |
|
||||
| `adapters/terraform/policy/checkov_adapter.py` | Emits `nova.policy.evaluated` event (P1) | P1 |
|
||||
| `core/regression_verify.py` | Emits `nova.capability.verified` event; CAP-023 (metrics collector) + CAP-024 (deck structure) added (P1, P6) | P1, P6 |
|
||||
| `pyproject.toml` | `addopts` gains `--junitxml=metrics/test-results.xml` + `--json-report` (P1, D-120) | P1 |
|
||||
| `docs/presentations/` | Two old decks retired (deleted); unified deck added (P5, D-130) | P5 |
|
||||
|
||||
### Telemetry/observability layer architecture (D-120)
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ Nova platform components (existing) │
|
||||
│ run_platform.sh · confidence_signal · checkov_adapter · │
|
||||
│ hitl_gates · regression_verify · outbox_writer · contract_ingestor │
|
||||
└──────────────────────┬──────────────────────────────────────────────┘
|
||||
│ CloudEvents 1.0 envelope (new emitters, P1)
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ metrics/events.jsonl (append-only CloudEvents log) │
|
||||
│ metrics/runs/<run_id>.json (per-run manifests) │
|
||||
│ metrics/decision_ledger.db (SQLite hash-chain, D-121) │
|
||||
│ metrics/test-results.xml (junit, P1) │
|
||||
└──────────────────────┬──────────────────────────────────────────────┘
|
||||
│ collector reads (P2)
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ metrics/nova_metrics.db (SQLite cold store, D-126) │
|
||||
│ fact_run · fact_capability · fact_policy_check · fact_confidence │
|
||||
│ fact_test · fact_decision · fact_cost_estimate │
|
||||
│ dim_capability · dim_milestone │
|
||||
│ + 8 empty placeholder views (deferred metrics) │
|
||||
└──────────────────────┬──────────────────────────────────────────────┘
|
||||
│ powerbi_export (P3)
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ metrics/powerbi/ (CSV/JSON views, folder connector, D-129) │
|
||||
│ → PowerBI dashboards (external) │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Hot path: deferred (D-126).** No live ops dashboard; SQLite is
|
||||
cold-only (batch/historical). The hot path activates when live AWS is
|
||||
re-provisioned (D-096 lift).
|
||||
|
||||
### NORTH_STAR integration point (REQ-186)
|
||||
|
||||
`.ciagent/NORTH_STAR.md` is read by CIAgent in context-loading for all
|
||||
future milestones. The integration mechanism (to be finalized in P4):
|
||||
a reference from `PROJECT.md` + `ARCHITECTURE.md` (this section) + a
|
||||
config entry in `config.json` (`strategic_direction_file:
|
||||
".ciagent/NORTH_STAR.md"`) that the run workflow reads at SPECIFY. This
|
||||
ensures the strategic direction survives across milestones without
|
||||
being overwritten by status updates.
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"phase": 0,
|
||||
"stage": "clarify",
|
||||
"stage": "research",
|
||||
"milestone": "v1.17",
|
||||
"phase_role": "pre_execution",
|
||||
"attempts": 0,
|
||||
"updated_at": "2026-08-04T19:45:00Z",
|
||||
"updated_at": "2026-08-04T20:15:00Z",
|
||||
"milestone_complete": false,
|
||||
"tag": null,
|
||||
"requirements": ["REQ-185"],
|
||||
"notes": "CLARIFY validation pass complete. 14 decisions (D-120..D-132) locked. 4 low-severity items deferred to PLAN (attestation denominator wording tightened, REQ-194 owns Attestation Coverage, REQ-186 mechanism TBD in P4, P0 type=docs). No blocking ambiguities. NORTH_STAR draft is PO-approved pending interactive GRILL."
|
||||
"notes": "RESEARCH complete. Signal inventory (grounding audit), telemetry reference architecture (Nova-native), metric-to-signal scorecard, deferred-decision ledger, deck-storytelling research. ARCHITECTURE.md v1.17 addendum + PERSONAS.md v1.17 roster written. 3 active personas (lead, backend, data); frontend deactivated. 6 assumptions logged."
|
||||
}
|
||||
+145
-13
@@ -1,23 +1,24 @@
|
||||
---
|
||||
project: acdl
|
||||
milestone: v1.16
|
||||
generated_at: 2026-07-30
|
||||
milestone: v1.17
|
||||
generated_at: 2026-08-04
|
||||
generator: lead-developer
|
||||
verification_toolchain:
|
||||
typecheck: "terraform validate && python3 -m py_compile core/**/*.py && python3 -m jsonschema schemas/*.schema.json"
|
||||
test: "bash scripts/run_regression.sh # 22-capability gate (D-091/D-118)"
|
||||
test: "bash scripts/run_regression.sh # 22-capability gate (D-091/D-118) + CAP-023/024 (v1.17)"
|
||||
build: "bash scripts/run_ci.sh # full local CI reproduction (lint+test+check-only)"
|
||||
note: |
|
||||
Nova (formerly ACDL) has no package.json. The execute/verify/ship
|
||||
workflows substitute `terraform validate` + `python -m py_compile` +
|
||||
JSON Schema validation for npm run typecheck, the regression gate
|
||||
(D-091, 22 capabilities) for npm test, and `bash scripts/run_ci.sh`
|
||||
for npm run build. v1.11 testing is pipeline-driven (D-102);
|
||||
v1.16 is NFR-only (no live apply by default; NOVA_LIFECYCLE_MODE=
|
||||
plan). Roster carries forward from v1.11/v1.14/v1.15 unchanged.
|
||||
frontend-engineer stays inactive (no frontend; decks are markdown =
|
||||
lead-developer territory). No custom personas needed (no new
|
||||
domains — onboarding is backend-engineer + data-engineer territory).
|
||||
v1.17 adds a telemetry/observability layer (metrics emitters, SQLite
|
||||
cold store, PowerBI export, Decision Ledger) + a unified narrative
|
||||
deck + a durable NORTH_STAR.md. Three active personas: lead-developer
|
||||
(coordination + deck narrative co-author), backend-engineer (event
|
||||
emitters, outbox_writer extension, Infracost adapter), data-engineer
|
||||
(SQLite store, schemas, PowerBI views, metrics collector). frontend-
|
||||
engineer stays deactivated (no Nova web UI — dashboards are PowerBI,
|
||||
not a Nova-built frontend; decks are markdown = lead-developer
|
||||
territory). No new custom personas needed — the metrics domain maps
|
||||
cleanly to data-engineer (schema/store/export) + backend-engineer
|
||||
(emitters/instrumentation).
|
||||
---
|
||||
|
||||
# ACDL — Persona Roster (project-level, v1.11 RESTART)
|
||||
@@ -251,3 +252,134 @@ The regression gate (22 capabilities) must stay **22/22 Verified**
|
||||
throughout v1.16 — simplification must not regress any capability
|
||||
(D-118). P9 (end of Wave 2) and P21 (milestone complete) run the gate;
|
||||
P14 (end of Wave 3) is an offline mid-milestone checkpoint.
|
||||
|
||||
---
|
||||
|
||||
# v1.17 Persona Roster — Strategic Direction, Leadership Metrics & Unified Story
|
||||
|
||||
> v1.17 adds a telemetry/observability layer (P1–P3), a metrics catalog
|
||||
> + NORTH_STAR integration (P4), a unified narrative deck (P5), a
|
||||
> regression capability (P6), and a final review/ship (P7). Three
|
||||
> active personas; frontend-engineer stays deactivated (no Nova web UI
|
||||
> — dashboards are PowerBI, not a Nova-built frontend).
|
||||
|
||||
## Active personas
|
||||
|
||||
### lead-developer
|
||||
- **Domain:** coordination + deck narrative
|
||||
- **Active:** true
|
||||
- **Phase-specific:** false
|
||||
- **Reason:** Owns CIAgent metadata, the NORTH_STAR.md authoring
|
||||
process (P0), the milestone decomposition, the unified narrative deck
|
||||
co-authoring (P5 — the deck is markdown, which is lead-developer
|
||||
territory per the established convention), and the final review/ship
|
||||
(P7). Arbitrates persona conflicts (e.g., backend vs data on the
|
||||
emitter/store boundary).
|
||||
- **Territory:** `.ciagent/NORTH_STAR.md`, `.ciagent/PROJECT.md`,
|
||||
`.ciagent/REQUIREMENTS.md`, `.ciagent/PLAN.md`, `.ciagent/RESEARCH.md`,
|
||||
`.ciagent/ARCHITECTURE.md`, `docs/presentations/nova-no-humans-platform.md`
|
||||
(NEW — unified deck source of truth), `docs/presentations/nova-no-humans-platform-marp.md`,
|
||||
`docs/presentations/nova-no-humans-platform-talking-points.md`,
|
||||
`docs/METRICS.md`, `docs/metrics/*.md` (per-KPI definition docs).
|
||||
|
||||
### backend-engineer
|
||||
- **Domain:** backend (event emitters + instrumentation)
|
||||
- **Active:** true
|
||||
- **Phase-specific:** false
|
||||
- **Reason:** Owns the event emitters (P1): the CloudEvents envelope,
|
||||
the per-run manifest writer, the `outbox_writer.py` extension to the
|
||||
SQLite Decision Ledger, the Infracost post-processor, the
|
||||
`hitl_gates.py` attestation event emission, the `confidence_signal.py`
|
||||
decision event emission, the `checkov_adapter.py` policy event
|
||||
emission, and the pytest `--junitxml` addopts change. Also owns the
|
||||
`regression_verify.py` CAP-023/024 additions (P6). The emitter work
|
||||
is the bridge between existing Nova components and the new metrics
|
||||
layer — it touches the code paths that already exist.
|
||||
- **Territory:** `core/metrics/event_envelope.py` (NEW),
|
||||
`core/metrics/run_manifest.py` (NEW),
|
||||
`core/metrics/infracost_adapter.py` (NEW),
|
||||
`core/metrics/decision_ledger.py` (NEW — extends outbox_writer),
|
||||
`core/outbox_writer.py` (extend to SQLite),
|
||||
`core/hitl_gates.py` (emit attestation.recorded),
|
||||
`core/confidence_signal.py` (emit ai.decision.made),
|
||||
`adapters/terraform/policy/checkov_adapter.py` (emit policy.evaluated),
|
||||
`scripts/run_platform.sh` (invoke manifest writer + Infracost),
|
||||
`core/regression_verify.py` (CAP-023/024),
|
||||
`pyproject.toml` (addopts --junitxml),
|
||||
`tests/test_metrics_emitters.py` (NEW),
|
||||
`tests/test_decision_ledger.py` (NEW).
|
||||
|
||||
### data-engineer
|
||||
- **Domain:** data (schema, SQLite store, PowerBI export)
|
||||
- **Active:** true
|
||||
- **Phase-specific:** false
|
||||
- **Reason:** Reactivated with a new territory for v1.17: the metrics
|
||||
collector (P2) and the PowerBI export (P3). Owns the schema design
|
||||
(metrics_*.schema.json), the SQLite cold store (nova_metrics.db), the
|
||||
fact/dimension table design, the 8 deferred placeholder views, and
|
||||
the CSV/JSON export. The data-engineer's schema-first constraint
|
||||
applies: all event types and fact/dim tables have JSON Schema
|
||||
definitions before any code is written. The collector reads files +
|
||||
events → SQLite; the export reads SQLite → CSV/JSON. This is the
|
||||
heaviest data-territory work since v1.11's terraform modules.
|
||||
- **Territory:** `core/metrics/collector.py` (NEW),
|
||||
`core/metrics/powerbi_export.py` (NEW),
|
||||
`schemas/metrics_*.schema.json` (NEW — event + fact/dim schemas),
|
||||
`metrics/nova_metrics.db` (NEW — SQLite cold store),
|
||||
`metrics/powerbi/` (NEW — CSV/JSON export dir),
|
||||
`docs/METRICS_VIEWS.md` (NEW — schema doc for PowerBI views),
|
||||
`tests/test_metrics_collector.py` (NEW),
|
||||
`tests/test_powerbi_export.py` (NEW).
|
||||
|
||||
## Deactivated personas
|
||||
|
||||
### frontend-engineer
|
||||
- **Domain:** frontend
|
||||
- **Active:** false
|
||||
- **Phase-specific:** false
|
||||
- **Reason:** v1.17 has no Nova web UI. The leadership dashboards are
|
||||
PowerBI (an external tool that ingests CSV/JSON files), not a
|
||||
Nova-built frontend. The decks are markdown (lead-developer
|
||||
territory). frontend-engineer stays deactivated, consistent with
|
||||
v1.11–v1.16. Reactivates if a future milestone builds a Nova web UI.
|
||||
|
||||
### lambda-engineer, platform-engineer, security-engineer
|
||||
- **Active:** false (carried forward from v1.11)
|
||||
- **Reason:** v1.17 does not touch the Lambda (beyond emitting events
|
||||
from the existing hitl_gates/attestation_matrix), does not do IR-
|
||||
shaped module authoring, and does not touch security adapters beyond
|
||||
emitting policy.evaluated events. The existing components are
|
||||
instrumented, not rewritten.
|
||||
|
||||
## v1.17 phase assignment
|
||||
|
||||
| Phase | Primary persona | Supporting | Territory |
|
||||
|-------|----------------|------------|-----------|
|
||||
| P0 pre-execution | lead-developer | — | `.ciagent/NORTH_STAR.md`, `PROJECT.md`, `REQUIREMENTS.md`, `RESEARCH.md`, `ARCHITECTURE.md`, `PERSONAS.md`, `PLAN.md` |
|
||||
| P1 event-emitters | backend-engineer | data-engineer (schemas) | `core/metrics/event_envelope.py`, `run_manifest.py`, `decision_ledger.py`, `infracost_adapter.py`, `outbox_writer.py`, `hitl_gates.py`, `confidence_signal.py`, `checkov_adapter.py`, `run_platform.sh`, `pyproject.toml` |
|
||||
| P2 metrics-collector | data-engineer | backend-engineer (event formats) | `core/metrics/collector.py`, `schemas/metrics_*.schema.json`, `metrics/nova_metrics.db` |
|
||||
| P3 powerbi-export | data-engineer | — | `core/metrics/powerbi_export.py`, `metrics/powerbi/`, `docs/METRICS_VIEWS.md` |
|
||||
| P4 metrics-catalog + north-star-integration | lead-developer | data-engineer (metric definitions) | `docs/METRICS.md`, `docs/metrics/*.md`, `PROJECT.md`, `ARCHITECTURE.md`, `config.json` |
|
||||
| P5 deck-rebuild | lead-developer | — | `docs/presentations/nova-no-humans-platform*.md`, retire old decks |
|
||||
| P6 regression-capability | backend-engineer | data-engineer (CAP-023 schema) | `core/regression_verify.py` (CAP-023, CAP-024) |
|
||||
| P7 final-review-ship | lead-developer | all active (review) | `.ciagent/**`, review + audit + ship |
|
||||
|
||||
## v1.17 domain priority
|
||||
|
||||
`backend → data → lead` (the emitter work in P1 is the foundation;
|
||||
data-engineer's collector + export in P2–P3 depends on P1's event
|
||||
formats; lead-developer's catalog + deck in P4–P5 depends on the
|
||||
metrics being grounded).
|
||||
|
||||
## v1.17 verification toolchain
|
||||
|
||||
```
|
||||
typecheck: terraform validate && python3 -m py_compile core/**/*.py adapters/**/*.py
|
||||
test: bash scripts/run_regression.sh # 22-capability gate + CAP-023/024 (v1.17)
|
||||
build: bash scripts/run_ci.sh # full local CI reproduction
|
||||
```
|
||||
|
||||
The regression gate (22 capabilities + CAP-023 metrics collector +
|
||||
CAP-024 deck structure) must pass at P6 and P7. CAP-009 (offline pytest
|
||||
suite) must remain Verified after the `--junitxml` addopts change
|
||||
(assumption A5).
|
||||
|
||||
@@ -1188,3 +1188,308 @@ stays a future feature (D-113).
|
||||
- A4 (0.85): The regression gate (D-091, D-118) at P9 and P21 confirms
|
||||
"simplify without regressions" — 22/22 capabilities must stay Verified.
|
||||
The gate is the credible control for the simplification wave.
|
||||
|
||||
---
|
||||
|
||||
# v1.17 Research — Strategic Direction, Leadership Metrics & Unified Story
|
||||
|
||||
> Phase: research (P0). Milestone: v1.17. Status: research.
|
||||
> Researcher: ci-researcher + explore agent (signal inventory).
|
||||
> Autonomy: full. Decisions D-120..D-132 locked in the planning
|
||||
> conversation (PROJECT.md). NORTH_STAR.md drafted (pending GRILL).
|
||||
|
||||
## 1. Telemetry Signal Inventory (grounding audit)
|
||||
|
||||
**Methodology:** every claim below is grounded in a concrete file path +
|
||||
line number in `/root/acdl`. No speculation. The explore agent performed
|
||||
a full sweep of the repo. The finding: **Nova has no metrics/telemetry/
|
||||
dashboard aggregation layer today.** What exists is a set of discrete,
|
||||
structured, file-based signal artifacts (JSON reports, JSONL logs,
|
||||
hash-chained outbox events, PR comments, Checkov JSON) plus unstructured
|
||||
stdout logs. A metrics milestone must aggregate these existing signals
|
||||
— it must not invent new ones without first adding emitters.
|
||||
|
||||
### (a) Signals that EXIST TODAY and are STRUCTURED (groundable)
|
||||
|
||||
| Signal | File / Emitter | Schema | Persistent? |
|
||||
|--------|---------------|--------|-------------|
|
||||
| Regression report (22 caps, status, duration_ms, gate) | `.ciagent/REGRESSION_REPORT.json` ← `core/regression_verify.py:643-667` | `regression_verify.py:82-91` | **Yes** (committed file) |
|
||||
| Regression report (markdown mirror) | `.ciagent/REGRESSION_REPORT.md` | same | Yes |
|
||||
| Checkpoint (milestone/phase/tag/regression summary) | `.ciagent/CHECKPOINT.json` (CIAgent-managed) | ad-hoc | Yes |
|
||||
| PolicyCheckResult list (per-rule pass/fail/severity/resourceRef) | `$WORK/pcr.json` ← `run_platform.sh:395` + `checkov_adapter.py:50-71` | `schemas/policy_check_result.schema.json` | **No** (ephemeral `/tmp/`) |
|
||||
| Confidence signal (score, band, perInput, reasonCodes) | `$WORK/signal.json` ← `run_platform.sh:412-426` + `confidence_signal.py:60-65` | `confidence_signal.py:60-65` | No (ephemeral) |
|
||||
| Outbox event (hash-chained, CONFIDENCE_COMPUTED) | `$WORK/event.json` + `$WORK/outbox_item.json` ← `run_platform.sh:444-459` + `outbox_writer.py:44-56` | `audit_ledger_design.md:44-45,81-97` | No (ephemeral; live DynamoDB torn down D-096) |
|
||||
| Resolved Target Stack | `$WORK/stack.json` ← `contract_resolver.py:581-603` | `schemas/stack.schema.json` | No (ephemeral) |
|
||||
| Lambda return bodies (submit/report_error/validate_cr/onboard) | `core/lambda/contract_ingestor.py:171,265,284,392,446` | ad-hoc JSON | No (Lambda not live; local stub only) |
|
||||
| DynamoDB CMDB rows (submitted/pending contracts) | `nova-contracts` table ← `contract_ingestor.py:160-170,433-445` | ad-hoc | **No** (table torn down D-096) |
|
||||
| SSM parameters (deploy outputs) | `/nova/<env>/<contractId>/<name>` ← `output_publisher.py:123-156` | ad-hoc | No (live AWS, torn down) |
|
||||
| PR stage comment (mode, runId) | GitHub PR API ← `post_stage_comment.sh:34-48` + `deploy.yml:141` | markdown table | Yes (GitHub) |
|
||||
| PR deploy-outputs comment | GitHub PR API ← `output_publisher.py:159-189` | markdown table | Yes (GitHub) |
|
||||
| GitHub issue (deploy failure alert) | GitHub API ← `contract_ingestor.py:179-290` + `deploy.yml:143-152` | issue body | Yes (GitHub) |
|
||||
| Local E2E result (stack_name, tier, outbox_events, chain_verified, lambda_status) | stdout JSON ← `core/local_emulators.py:498-508,519` | ad-hoc | No (stdout) |
|
||||
| HITL gate result | `core/hitl_gates.py:87,90` + `run_platform.sh:179-185` | stdout `HITL PASS/BLOCK` | No (stdout) |
|
||||
| Attestation matrix result | `core/attestation_matrix.py:184,187` | stdout `ATTESTATION PASS/BLOCK` | No (stdout) |
|
||||
| Cost figures | `.ciagent/COST.md` (manual Cost Explorer query) | markdown table | Yes (manual, not automated) |
|
||||
|
||||
### (b) Signals that EXIST but are UNSTRUCTURED (log-only)
|
||||
|
||||
| Signal | Source | Format |
|
||||
|--------|--------|--------|
|
||||
| CI pipeline result | `scripts/run_ci.sh:70-71` | stdout banner `=== CI PIPELINE OK ===` |
|
||||
| Platform stage banners + summaries | `scripts/run_platform.sh:222,241,258,263,315,383,411,442,463,490,496` | stdout `=== Step N: ... ===` + summary lines |
|
||||
| Terraform init/validate/plan/apply/destroy logs | `$WORK/tf-*.log` ← `run_platform.sh:320,324,328,352,375` | raw terraform stdout (via `tee`) |
|
||||
| Lifecycle test results | `scripts/run_lifecycle_test.sh` etc. | exit code only (no report file) |
|
||||
| Decommission step counts | `scripts/run_decommission.sh:40,54` | stdout `decommission step N: M resources...` |
|
||||
| Uptime endpoint count | `scripts/run_uptime.sh:72,87` | stdout `uptime: N endpoint(s) to monitor` |
|
||||
| Onboarding prompt | `core/environment_check.py:57-81` | stdout text block |
|
||||
| Pytest results | `pyproject.toml:25` (`-v --tb=short`) | stdout only (no junit/json) |
|
||||
| sync_workflows result | `scripts/sync_workflows.py:56,53` | stdout `OK: 3 workflow pairs match` / `DRIFT: ...` |
|
||||
|
||||
### (c) Proposed executive metrics with NO grounding today (DEFERRED)
|
||||
|
||||
| Proposed metric | Why no grounding | Controlling decision |
|
||||
|------------------|------------------|---------------------|
|
||||
| Live infrastructure health (ECS running count, ALB 5xx, RPS) | Live AWS torn down; CAP-013..016 Skipped | **D-096** |
|
||||
| Live outbox write rate / ledger append latency | DynamoDB outbox table absent | **D-096** |
|
||||
| Tamper-evident ledger checkpoint count / JWS signature rate | S3 Object Lock + JWS + async worker deferred | **D-083** |
|
||||
| Onboarding funnel: requested → granted conversion | Only "requested" (pending row) is emitted; no grant event | **D-113, D-114, D-119** |
|
||||
| Time-to-provision (onboarding SLA) | Real AWS provisioning deferred | **D-113** |
|
||||
| Cross-account role grant count | Offline-proven only, no live apply | **D-114** |
|
||||
| Drift detection (scheduled terraform plan -detailed-exitcode) | Needs live AWS workspaces + a scheduler Nova doesn't have | **D-096** + no scheduler |
|
||||
| GreenOps / carbon (WattTime/Electricity Maps API) | No grounding; new external API | future emitter |
|
||||
| Predictive vs Reactive ratio | Requires an ML anomaly-forecasting service | future emitter |
|
||||
| Multi-cloud normalization (Azure/GCP/K8s, FOCUS spec) | Nova is AWS-only | future |
|
||||
| Red Team MTTR | No red-team program exists | future |
|
||||
| Self-healing velocity | Nova has no auto-remediator | future emitter |
|
||||
| SLA / unplanned downtime | Needs live service uptime monitoring against SLOs | **D-096** |
|
||||
| Per-module lifecycle success rate over time | No structured report file written; only exit code | gap (no decision) |
|
||||
| Test pass rate / test count time-series | No junit/json reporter configured | gap (add `--junitxml` to addopts) |
|
||||
| Code coverage trend | `pytest-cov` installed but not in `addopts` | gap |
|
||||
| Deploy frequency / lead time / MTTR (DORA) | No deploy-event emitter; pipeline runs not counted | gap |
|
||||
| Policy pass rate time-series | `pcr.json` emitted but ephemeral; not persisted | gap (D-096 blocks live persistence) |
|
||||
| Confidence score distribution over time | `signal.json` emitted but ephemeral | gap |
|
||||
| Consumer adoption count / active consumers | `PROJECT.md:487` explicitly states "0 consumer adoption today" | honest scope |
|
||||
| Cost time-series (automated) | `COST.md` is a one-shot manual query; no automated emitter | gap |
|
||||
|
||||
**Bottom line:** the single richest existing structured signal is
|
||||
`.ciagent/REGRESSION_REPORT.json` (22 capabilities × {status, tier,
|
||||
duration_ms, detail} + summary counts + boolean gate). The next richest
|
||||
is the per-run `$WORK/*.json` family (pcr.json, signal.json, event.json,
|
||||
stack.json) — but these are **ephemeral** and **not persisted in CI**.
|
||||
The lowest-friction grounding for a "no-humans" dashboard is therefore:
|
||||
(1) regression report → capability health, (2) PR comments + GitHub
|
||||
issues → deploy/failure activity, (3) add `--junitxml` to pytest → test
|
||||
trend, (4) persist `$WORK/*.json` → policy/confidence/outbox time-series,
|
||||
(5) extend outbox_writer → Decision Ledger, (6) add Infracost →
|
||||
pre-apply cost estimates.
|
||||
|
||||
## 2. Telemetry Reference Architecture (Nova-native adaptation)
|
||||
|
||||
The PO provided a full distributed-system telemetry reference
|
||||
architecture (CloudEvents 1.0 envelope, OpenTelemetry SDK, Kafka/NATS
|
||||
event bus, Prometheus hot path, ClickHouse warehouse, QLDB decision
|
||||
ledger, Infracost, drift detection, ML anomaly forecasting). Per
|
||||
D-120, we adopt the **principles** but implement with **Nova-native
|
||||
minimal tech**. The mapping:
|
||||
|
||||
| Direction's principle | Nova-native implementation (v1.17) |
|
||||
|---|---|
|
||||
| Events are the source of truth; dashboards are projections | Hybrid (D-125): existing file signals stay as files; collector reads them and emits normalized CloudEvents into `metrics/events.jsonl` + SQLite. New emitters emit CloudEvents directly. |
|
||||
| Every AI action is logged with confidence + alternatives | Decision Ledger (D-121): `outbox_writer.py` extended → SQLite append-only hash-chain table. `ai.decision.made` modeled from confidence_signal (D-122): decision_id=run_id, chosen_action=band, confidence=score, alternatives=perInput, human_override=HITL block. |
|
||||
| Hot/cold storage split | Cold-only SQLite (D-126): `metrics/nova_metrics.db`. Hot path deferred (no live ops, D-096). |
|
||||
| Read-only external integrators | Infracost (pre-apply, offline, reads plan JSON). Cloud billing CUR deferred (D-096). Carbon APIs deferred (future). |
|
||||
| CloudEvents 1.0 envelope | Adopted. `core/metrics/event_envelope.py` defines the envelope + `platform.*` semantic conventions. |
|
||||
| Decision Ledger = append-only with hash chain + outcome backfill | SQLite append-only table with hash chain (D-121). Outcome backfilled from apply.completed via decision_id → request_id correlation. Honors D-083 (no S3 Object Lock/JWS). |
|
||||
| Cost governance: mandatory tags + Infracost pre-apply | Nova already enforces `nova:*` tags (nova_tagging.py, hard mode). Infracost added as plan post-processor (D-120). Post-apply CUR deferred (D-096). |
|
||||
| Definition-of-success docs for every KPI | Per-KPI docs in `docs/metrics/` (D-127). |
|
||||
| Replay-ability | SQLite store + JSONL event log are replayable by design. |
|
||||
|
||||
### CloudEvents envelope (Nova-native)
|
||||
|
||||
```json
|
||||
{
|
||||
"specversion": "1.0",
|
||||
"id": "<uuid>",
|
||||
"source": "nova.platform",
|
||||
"type": "nova.run.completed",
|
||||
"time": "<ISO8601>",
|
||||
"subject": "<contractId>/<env>",
|
||||
"datacontenttype": "application/json",
|
||||
"platform": {
|
||||
"tenant_id": "acdl",
|
||||
"run_id": "run-<epoch>",
|
||||
"contract_id": "<uuid>",
|
||||
"environment": "dev|qa|prod|dr",
|
||||
"actor": {"type": "confidence-gate", "id": "confidence_signal"},
|
||||
"trace_id": "<run_id>"
|
||||
},
|
||||
"data": {
|
||||
"duration_ms": 4800,
|
||||
"stages": ["resolve", "adapt", "validate", "plan", "apply"],
|
||||
"exit_code": 0,
|
||||
"confidence": {"score": 0.94, "band": "pass", "perInput": {...}},
|
||||
"policy": {"passed": 12, "failed": 0, "skipped": 0},
|
||||
"hitl": {"gate": "dev", "result": "autonomous", "block": false},
|
||||
"cost_estimate_usd": -12.40,
|
||||
"decision_id": "run-<epoch>",
|
||||
"outcome": "succeeded"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Core event types (Nova-native minimum viable set)
|
||||
|
||||
| Event type | Emitted by | Purpose | Grounding |
|
||||
|---|---|---|---|
|
||||
| `nova.run.started` | run_platform.sh | Measures demand; provisioning lead time start | new emitter (P1) |
|
||||
| `nova.run.completed` | run_platform.sh | Run count, stage durations, exit, MTTR | new emitter (P1) |
|
||||
| `nova.run.failed` | run_platform.sh | Failure count, MTTR numerator | new emitter (P1) |
|
||||
| `nova.policy.evaluated` | checkov_adapter.py | Policy pass rate, compliance KPIs | grounded (pcr.json → P1 persists) |
|
||||
| `nova.confidence.computed` | confidence_signal.py | Confidence distribution, decision accuracy | grounded (signal.json → P1 persists) |
|
||||
| `nova.ai.decision.made` | outbox_writer.py (extended) | Decision Ledger entry | grounded (D-121, D-122) |
|
||||
| `nova.attestation.recorded` | hitl_gates.py | Attestation Coverage, human-in-the-loop audit | grounded (D-132) |
|
||||
| `nova.cost.estimated` | Infracost post-processor | Pre-apply cost estimate | new emitter (P1, Infracost) |
|
||||
| `nova.capability.verified` | regression_verify.py | Capability health, regression gate | grounded (REGRESSION_REPORT.json) |
|
||||
| `nova.test.completed` | pytest (junit XML) | Test count, pass rate | new (P1 adds --junitxml) |
|
||||
|
||||
## 3. Metric-to-Signal Scorecard (the "no fabrication" contract)
|
||||
|
||||
| Executive metric (NORTH_STAR target) | Status | Source / formula | Decision |
|
||||
|---|---|---|---|
|
||||
| Touchless Resolution Rate ≥99% | grounded (after P1) | runs without operational HITL block ÷ total runs (attestation gates excluded) | D-122, D-132 |
|
||||
| Human Escalation Frequency <0.1% | grounded (after P1) | operational HITL blocks ÷ total runs (attestation sign-offs excluded) | D-122, D-132 |
|
||||
| MTTR (p95) <60s | grounded (platform-run) | apply.failed.time → successful retry.time | D-131 |
|
||||
| Predictive vs Reactive ≥3:1 | **deferred** | requires ML forecasting (future emitter) | future |
|
||||
| AI Decision Accuracy ≥99.5% | grounded (after decision ledger) | decisions not followed by apply.failed/incident within 5min | D-121, D-122 |
|
||||
| Drift Auto-Reversal ≥95% | **deferred** | requires drift detection (D-096 + scheduler) | D-096 |
|
||||
| Cloud Spend Reduction ≥25% | partial | pre-apply estimate grounded (Infracost); actuals deferred (D-096 CUR) | D-120 |
|
||||
| L1/L2 Ops Hours Avoided ≥70% | derived | formula: run count × manual baseline minutes × blended rate | D-127 |
|
||||
| Platform ROI ≥250% | derived | formula: (labor savings + cloud savings + avoided downtime) ÷ platform op cost | D-127 |
|
||||
| Decision Ledger Coverage 100% | grounded (this milestone) | outbox_writer.py → SQLite hash-chain | D-121 |
|
||||
| Attestation Coverage 100% | grounded | hitl_gates.py + outbox approver_* attributes; prod/dr | D-132 |
|
||||
| AI-Agent Intent Share ≥40% | future | no AI-agent consumers today; placeholder view | future |
|
||||
| Capability health (18V+4S) | grounded | REGRESSION_REPORT.json | existing |
|
||||
| Confidence score distribution | grounded (after P1) | signal.json → decision ledger | D-121 |
|
||||
| Policy pass rate | grounded (after P1) | pcr.json → persisted | D-120 |
|
||||
| Test count / pass rate | grounded (after P1) | pytest --junitxml | D-120 |
|
||||
| Provisioning Lead Time | grounded (after P1) | run.started → run.completed | D-120 |
|
||||
| Deployment Frequency | grounded (after P1) | count(run.completed) per day | D-120 |
|
||||
| Deploy-failure alert count | grounded | GitHub issues via Lambda report_error (D-055) | existing |
|
||||
| Cost figures (actuals) | manual one-shot | COST.md (Cost Explorer query) | existing |
|
||||
| FTE Hours Saved / TRV | derived | formula over run count + COST.md | D-127 |
|
||||
| Self-healing velocity | **deferred** | no auto-remediator | future |
|
||||
| SLA / unplanned downtime | **deferred** | needs live service uptime (D-096) | D-096 |
|
||||
| GreenOps / carbon | **deferred** | WattTime/Electricity Maps API (future) | future |
|
||||
| Red Team MTTR | **deferred** | no red-team program | future |
|
||||
| Multi-cloud normalization | **deferred** | Nova is AWS-only | future |
|
||||
| Live CUR reconciliation | **deferred** | needs live AWS billing (D-096) | D-096 |
|
||||
|
||||
## 4. Deferred-Decision Ledger (constraints on this milestone)
|
||||
|
||||
| Decision | Scope | Grounding impact |
|
||||
|----------|-------|------------------|
|
||||
| D-096 | Live AWS torn down post-v1.11 | BLOCKS all live-AWS metrics (CAP-013..016 Skipped; live outbox; live state bucket; live CUR) |
|
||||
| D-083 | S3 Object Lock + JWS + async worker deferred | BLOCKS tamper-evident ledger; v1.17 uses SQLite hash-chain instead |
|
||||
| D-113/D-114/D-119 | Onboarding = request-path only; no auto-grant | BLOCKS onboarding funnel "granted" half |
|
||||
| D-091/D-118 | Regression gate (D-091) gates milestone completion | ENABLES the strongest metric signal (REGRESSION_REPORT.json) |
|
||||
| D-092 | Local emulating adapters | ENABLES offline E2E metrics (CAP-011/012) |
|
||||
| D-055 | report_error Lambda action creates GitHub issues | ENABLES deploy-failure alert metric |
|
||||
| D-050 | Publish deploy outputs to SSM + GitHub PR comment | ENABLES outputs-published metric |
|
||||
| D-054/D-043/D-109 | Nova tagging standard (hard mode) | ENABLES tagging-compliance metric |
|
||||
| D-084 | 8-concern attestation matrix | ENABLES attestation metrics (operator-supplied evidence artifacts) |
|
||||
| D-089 | Signature verification skipped when signing key unset (dev/CI) | Signature metrics are no-ops in dev |
|
||||
|
||||
## 5. Deck-Storytelling Research (x3 arc + per-slide benefit)
|
||||
|
||||
### The "tell them x3" structure
|
||||
|
||||
The PO's direction: "Tell them what you're going to tell them, then tell
|
||||
them, then tell them what you told them." Applied at two levels:
|
||||
|
||||
**Deck level (the 5-act arc):**
|
||||
1. **Opening slide** = "what I'm going to tell you" — the full arc
|
||||
preview: Problem → Vision → How → Proof → Roadmap.
|
||||
2. **Body** (acts 1–5) = "tell them" — each act delivers its content.
|
||||
3. **Closing slide** = "what I told you" — recap of the 5 acts + the ask.
|
||||
|
||||
**Per slide:**
|
||||
1. **Slide opens** with what it'll cover (1 line: "This slide shows X").
|
||||
2. **Slide delivers** the content (bullets, diagram, or table).
|
||||
3. **Slide closes** with an explicit **"benefit of this stage" callout**
|
||||
(1 line: "Benefit: you now know Y" or "Why this matters: Z").
|
||||
|
||||
### Fluidity conventions
|
||||
|
||||
- **Transitions are written, not hand-waved.** Each slide's opening line
|
||||
references the previous slide's close ("Having seen X, now consider Y").
|
||||
- **No disjointed jumps.** If a topic shift is needed, a bridge slide or
|
||||
a transition sentence carries the audience across.
|
||||
- **The arc is visible.** A small "act indicator" in the Marp footer
|
||||
(e.g., `Act 3/5: How it works`) keeps the audience oriented.
|
||||
|
||||
### Existing deck inventory (to be retired)
|
||||
|
||||
Two decks exist today in `docs/presentations/`:
|
||||
- `how-the-platform-works.md` (32,916 bytes) → marp → html → talking-points
|
||||
- `the-developer-experience.md` (27,509 bytes) → marp → html → talking-points
|
||||
|
||||
Both follow a 4-step process (source `.md` → Marp → HTML → talking-points)
|
||||
documented in `docs/presentations/README.md`. Per D-130, both are merged
|
||||
into one unified narrative deck and retired.
|
||||
|
||||
### Grounded metrics already cited in existing decks
|
||||
|
||||
- "22/22 auto-verifiable capabilities Verified" — **stale** vs current
|
||||
REGRESSION_REPORT.json (18V+4S post-D-096). The unified deck must
|
||||
derive this from the report, not copy the stale claim.
|
||||
- Confidence thresholds: dev ≥0.50, qa ≥0.75, prod ≥0.90, dr ≥0.95 —
|
||||
grounded in `core/confidence_signal.py:57` (THRESHOLDS).
|
||||
- RPO = 0 (evidence write synchronous) — grounded in
|
||||
`core/audit_ledger_design.md:27,103`.
|
||||
- Cost figures — `how-the-platform-works.md:461`; cites COST.md.
|
||||
- Confidence signal 6 inputs + weights — grounded in
|
||||
`core/confidence_signal.py:40-47`.
|
||||
- "~80-line stateless adapter" vs "918-line monolith" — grounded in
|
||||
ROADMAP/RESEARCH prose.
|
||||
|
||||
### Planned deck structure (for PLAN to detail)
|
||||
|
||||
The unified deck "Nova — The No-Humans Infrastructure Platform":
|
||||
|
||||
| Act | Slides | Content | Proof source |
|
||||
|---|---|---|---|
|
||||
| 1. Problem | 2–3 | The no-humans imperative; why operators are the bottleneck; the trust gap | NORTH_STAR vision |
|
||||
| 2. Vision/Direction | 2–3 | Nova's vision; 4 strategic objectives; anti-goals; the attestation model (autonomy in operations, human at stage gates) | NORTH_STAR |
|
||||
| 3. How it works | 3–4 | Contract → resolver → adapter → confidence → HITL gate; the Decision Ledger; the 8-concern attestation matrix | code grounding |
|
||||
| 4. Proof (metrics) | 3–4 | Capability health (18V+4S); confidence distribution; policy pass rate; Decision Ledger coverage; Attestation Coverage; cost estimates; the grounded/derived/deferred honesty model | metrics export |
|
||||
| 5. Roadmap/Ask | 2 | 12–18mo targets (committed); deferred metrics (honest); the ask | NORTH_STAR targets |
|
||||
|
||||
Total: ~12–16 slides. Opening = arc preview; closing = recap + ask.
|
||||
|
||||
## 6. Assumptions logged (v1.17)
|
||||
|
||||
- A1 (0.9): No live AWS access during execution (consistent with
|
||||
v1.11–v1.16). All metrics that require live AWS ship as placeholder
|
||||
views. The Infracost integration runs offline (reads plan JSON).
|
||||
- A2 (0.85): The Decision Ledger SQLite hash-chain is sufficient for
|
||||
v1.17's audit needs. The full tamper-evident ledger (S3 Object Lock +
|
||||
JWS, D-083) is a future milestone. The hash-chain provides
|
||||
append-only + integrity verification locally.
|
||||
- A3 (0.8): The "AI decision" framing (D-122) is honest: Nova's "AI" is
|
||||
the confidence-gated policy engine (confidence_signal + HITL gate),
|
||||
not an LLM planner. The deck and METRICS.md must frame this accurately
|
||||
— overclaiming "AI" would violate the "no fabrication" constraint.
|
||||
- A4 (0.85): The unified deck's "Proof" section cites only grounded
|
||||
metrics with real numbers. Deferred metrics are shown as "Planned"
|
||||
with the `<span class="badge planned">Planned</span>` badge. No
|
||||
fabricated numbers in any slide.
|
||||
- A5 (0.8): `--junitxml` + `--json-report` added to pytest addopts
|
||||
does not break the existing test suite (the flags are additive; pytest
|
||||
continues to run normally). CAP-009 (offline pytest suite passes)
|
||||
must remain Verified after the change.
|
||||
- A6 (0.75): Infracost is available as a CLI tool that can be installed
|
||||
in the CI environment and run locally. It reads `terraform plan
|
||||
-out=plan.tfplan` + `terraform show -json plan.tfplan` to produce a
|
||||
cost estimate. No live AWS access required. If Infracost is not
|
||||
available, the `cost.estimated` event is omitted (degraded mode, not
|
||||
a failure).
|
||||
|
||||
Reference in New Issue
Block a user