acdl/docs/METRICS.md

Nova Metrics Catalog

This is the canonical catalog of every executive KPI in Nova's leadership metrics layer. Each metric carries a status:

• grounded — cites a source file + schema (the metric is computed from a real emitted signal)
• derived — documented formula over grounded inputs
• deferred — cites a blocking decision ID (D-096/D-083/D-113/etc.); ships as an empty PowerBI placeholder view with a documented schema

Hard constraint (NORTH_STAR): DO NOT make anything up. No fabricated numbers. Every metric either has a real source or is explicitly deferred.

---

Zero-Touch Efficiency & AI Autonomy (REQ-191)

Touchless Resolution Rate

• Target: ≥ 99% across production estates (Post-Pilot)
• Status: partial (pipeline grounded; denominator = 1 run post-pilot)
• Formula: runs completing without operational HITL block ÷ total runs (attestation gates excluded — they're designed controls, not escalations)
• Source: metrics/nova_metrics.db fact_run (hitl_block column)
• Definition-of-success: docs/metrics/touchless_resolution_rate.md

Human Escalation Frequency

• Target: < 0.1% of platform actions (Post-Pilot)
• Status: partial (pipeline grounded; denominator = 1 run post-pilot, 0 escalations)
• Formula: operational HITL blocks ÷ total runs (attestation sign-offs excluded)
• Source: metrics/nova_metrics.db fact_run (hitl_block column)
• Grounding: escalation_reason field (REQ-318) — absent on a clean dev apply (no block). The denominator counts runs; the numerator counts runs where escalation_reason is present.
• Definition-of-success: docs/metrics/human_escalation_frequency.md

AI Decision Accuracy

• Target: ≥ 99.5% (no rollback, no follow-up incident within 5 min)
• Status: partial (pipeline grounded; denominator = 1 decision post-pilot)
• Formula: decisions not followed by apply.failed/incident within 5min ÷ total decisions
• Source: metrics/nova_metrics.db fact_decision (outcome column)
• Grounding: fact_decision.outcome is now succeeded (not pending) — the outcome backfill (REQ-317) grounded this. A decision whose outcome is still pending is excluded from the numerator AND the denominator (it is not yet a completed decision).
• Definition-of-success: docs/metrics/ai_decision_accuracy.md

Post-Pilot Activation (v1.26 P4)

The three Post-Pilot targets above were previously documented as "denominator = 0 today" — no real consumer estate had run through the platform end-to-end. The v1.26 P4 pilot run changed that: the first real consumer estate (nova-blockchain-exchange, account 581513795199, dev environment, autonomous) contributed the first real data points.

• Run id: blkex-pilot-apply-v0.2 (2026-08-19)
• AI Decision Accuracy: 1 decision (blkex-pilot-apply-v0.2), outcome pending → succeeded (REQ-317 backfill). Numerator = 1 (no apply.failed, no incident), denominator = 1. Future runs accumulate into this denominator.
• Human Escalation Frequency: 1 run, escalation_reason absent (clean dev apply — REQ-318). Numerator = 0 escalations, denominator = 1.
• Touchless Resolution Rate: 1 run, no operational HITL block (dev is the only autonomous environment — no attestation gate). Numerator = 1, denominator = 1.

The denominators are now non-zero. Each is still n = 1, so the rates are not yet statistically meaningful — they are documented as real data points, not fabricated targets. See .ciagent/P4-PILOT-RUN-EVIDENCE.md for the full evidence stream (confidence 0.800 pass, Decision Ledger hash chain valid).

MTTD / MTTR (platform-run)

• Target: < 60 seconds (p95)
• Status: grounded (platform-run MTTR)
• Formula: apply.failed.time → successful retry.time
• Source: metrics/nova_metrics.db fact_run (started_at, completed_at)
• Note: infra-incident MTTR deferred (no incident detection system)
• Definition-of-success: docs/metrics/mttr.md

Confidence-Gate Halt Rate (REQ-212)

• Target: not a committed target (operational signal)
• Status: grounded
• Formula: runs where confidence band = halt ÷ total runs
• Source: metrics/nova_metrics.db fact_confidence (band column)
• Definition-of-success: docs/metrics/confidence_gate_halt_rate.md

---

Velocity (REQ-192)

Provisioning Lead Time

• Target: not a committed target (operational signal)
• Status: grounded (after P1)
• Formula: apply.completed.time − intent.received.time
• Source: metrics/nova_metrics.db fact_run (started_at, completed_at)
• Definition-of-success: docs/metrics/provisioning_lead_time.md

Deployment Frequency

• Target: not a committed target (operational signal)
• Status: grounded (after P1)
• Formula: count(run.completed) per day
• Source: metrics/nova_metrics.db fact_run
• Definition-of-success: docs/metrics/deployment_frequency.md

Self-Healing Velocity — DEFERRED

• Status: deferred (no auto-remediator)
• Blocking decision: future emitter
• Placeholder view: placeholder_predictive_reactive.csv

---

Financial & Cost ROI (REQ-193)

Cost Savings via Infracost Estimates

• Target: ≥ 25% on pilot estates (partial)
• Status: partial (pre-apply estimate grounded; actual-spend deferred D-096)
• Formula: sum(cost_estimate.delta_usd) where delta < 0
• Source: metrics/nova_metrics.db fact_cost_estimate
• Definition-of-success: docs/metrics/cost_savings.md

FTE Hours Saved (Toil Reallocation Value)

• Target: ≥ 70% of pre-Nova FTE allocation (derived)
• Status: derived
• Formula: run count × manual baseline minutes × blended rate
• Source: metrics/nova_metrics.db fact_run (count) + manual baseline
• Note: computed on N internal runs today; production-denominator activates post-pilot
• Definition-of-success: docs/metrics/fte_hours_saved.md

Platform ROI

• Target: ≥ 250% measured annually (derived)
• Status: derived
• Formula: (FTE hours saved × blended rate + cloud savings + avoided downtime) ÷ platform op cost
• Source: derived from fact_run + fact_cost_estimate + manual baseline
• Note: computed on N internal runs today; production-denominator activates post-pilot
• Definition-of-success: docs/metrics/platform_roi.md

Live CUR Reconciliation — DEFERRED

• Status: deferred (D-096)
• Placeholder view: placeholder_live_cur_reconciliation.csv

---

Reliability, Security & Compliance (REQ-194)

Zero-Trust Policy Compliance Rate

• Target: not a committed target (operational signal)
• Status: grounded (after P1)
• Formula: 1 − count(assets WHERE last_scan.status ≠ pass) ÷ count(assets)
• Source: metrics/nova_metrics.db fact_policy_check
• Definition-of-success: docs/metrics/policy_compliance_rate.md

Attestation Coverage

• Target: 100% of prod/dr promotions attested by a human
• Status: grounded
• Formula: prod/dr promotions attested ÷ total prod/dr promotions
• Source: metrics/decision_ledger.db (attestation.recorded events) + hitl_gates.py + outbox approver_* attributes
• Definition-of-success: docs/metrics/attestation_coverage.md

SLA / Unplanned Downtime — DEFERRED

• Status: deferred (D-096)
• Placeholder view: placeholder_sla_downtime.csv

Patch Remediation Rate — DEFERRED

• Status: deferred (no patch remediation system)
• Placeholder view: (future)

---

Trust Substrate (REQ-211)

Decision Ledger Coverage

• Target: 100% of AI actions with backfilled outcome
• Status: grounded (this milestone builds it)
• Formula: count(decision_ledger rows with outcome ≠ 'pending') ÷ count(decision_ledger rows)
• Source: metrics/decision_ledger.db + core/metrics/decision_ledger.py
• Definition-of-success: docs/metrics/decision_ledger_coverage.md

Trust Snapshot

• Status: grounded (P4 tool)
• Source: core/metrics/trust_snapshot.py → metrics/TRUST_SNAPSHOT.md
• Contents: Decision Ledger Coverage, Attestation Coverage, Capability Health, AI Decision Accuracy, Confidence-Gate Halt Rate, chain-integrity verdict, snapshot hash

---

Deferred Metrics (8 placeholder views)

Metric	Blocking Decision	Placeholder View
Live Infrastructure Health	D-096	placeholder_live_infra_health.csv
Live Outbox Write Rate	D-096	placeholder_live_outbox_rate.csv
Tamper-Evident Ledger Checkpoints	D-083	placeholder_tamper_evident_checkpoints.csv
Onboarding Funnel (granted)	D-113/D-114/D-119	placeholder_onboarding_funnel.csv
Drift Auto-Reversal Rate	D-096 + no scheduler	placeholder_drift_detection.csv
Live CUR Reconciliation	D-096	placeholder_live_cur_reconciliation.csv
SLA / Unplanned Downtime	D-096	placeholder_sla_downtime.csv
Predictive vs Reactive Ratio	future emitter	placeholder_predictive_reactive.csv

See docs/METRICS_DEFERRED_ROADMAP.md for the activation path for each.

---

v1.25 — Swappable Policy Engine

The policy engine that produces the PolicyCheckResult records feeding the confidence signal is swappable (NORTH_STAR Strategic Objective #2 — provable trust via a replaceable substrate, not a vendor lock-in). The PolicyEngine protocol (core/policy_engine.py) is the swap boundary; config.json.policy.engine selects the active engine (default "kyverno-json"). A future OpaEngine implements the same protocol without touching the confidence signal, the PCR schema, or the pipeline. See .ciagent/ARCHITECTURE.md §12.7 for the registry diagram.