acdl/docs/METRICS.md

# Nova Metrics Catalog


This is the canonical catalog of every executive KPI in Nova's
leadership metrics layer. Each metric carries a **status**:

- **grounded** — cites a source file + schema (the metric is computed
  from a real emitted signal)
- **derived** — documented formula over grounded inputs
- **deferred** — cites a blocking decision ID (D-096/D-083/D-113/etc.);
  ships as an empty PowerBI placeholder view with a documented schema

**Hard constraint (NORTH_STAR):** DO NOT make anything up. No fabricated
numbers. Every metric either has a real source or is explicitly deferred.

---

## Zero-Touch Efficiency & AI Autonomy (REQ-191)

### Touchless Resolution Rate
- **Target:** ≥ 99% across production estates (Post-Pilot)
- **Status:** partial (pipeline grounded; denominator = 1 run post-pilot)
- **Formula:** runs completing without *operational* HITL block ÷ total runs
  (attestation gates excluded — they're designed controls, not escalations)
- **Source:** `metrics/nova_metrics.db` `fact_run` (hitl_block column)
- **Definition-of-success:** `docs/metrics/touchless_resolution_rate.md`

### Human Escalation Frequency
- **Target:** < 0.1% of platform actions (Post-Pilot)
- **Status:** partial (pipeline grounded; denominator = 1 run post-pilot, 0 escalations)
- **Formula:** operational HITL blocks ÷ total runs (attestation sign-offs
  excluded)
- **Source:** `metrics/nova_metrics.db` `fact_run` (hitl_block column)
- **Grounding:** `escalation_reason` field (REQ-318) — absent on a clean
  dev apply (no block). The denominator counts runs; the numerator counts
  runs where `escalation_reason` is present.
- **Definition-of-success:** `docs/metrics/human_escalation_frequency.md`

### AI Decision Accuracy
- **Target:** ≥ 99.5% (no rollback, no follow-up incident within 5 min)
- **Status:** partial (pipeline grounded; denominator = 1 decision post-pilot)
- **Formula:** decisions not followed by apply.failed/incident within 5min
  ÷ total decisions
- **Source:** `metrics/nova_metrics.db` `fact_decision` (outcome column)
- **Grounding:** `fact_decision.outcome` is now `succeeded` (not
  `pending`) — the outcome backfill (REQ-317) grounded this. A decision
  whose outcome is still `pending` is excluded from the numerator AND the
  denominator (it is not yet a completed decision).
- **Definition-of-success:** `docs/metrics/ai_decision_accuracy.md`

#### Post-Pilot Activation (v1.26 P4)

The three Post-Pilot targets above were previously documented as
"denominator = 0 today" — no real consumer estate had run through the
platform end-to-end. The v1.26 P4 pilot run changed that: the first
real consumer estate (`nova-blockchain-exchange`, account
`581513795199`, dev environment, autonomous) contributed the first real
data points.

- **Run id:** `blkex-pilot-apply-v0.2` (2026-08-19)
- **AI Decision Accuracy:** 1 decision (`blkex-pilot-apply-v0.2`),
  outcome `pending → succeeded` (REQ-317 backfill). Numerator = 1
  (no apply.failed, no incident), denominator = 1. Future runs
  accumulate into this denominator.
- **Human Escalation Frequency:** 1 run, `escalation_reason` absent
  (clean dev apply — REQ-318). Numerator = 0 escalations, denominator
  = 1.
- **Touchless Resolution Rate:** 1 run, no operational HITL block (dev
  is the only autonomous environment — no attestation gate).
  Numerator = 1, denominator = 1.

The denominators are now non-zero. Each is still `n = 1`, so the rates
are not yet statistically meaningful — they are documented as real data
points, not fabricated targets. See `.ciagent/P4-PILOT-RUN-EVIDENCE.md`
for the full evidence stream (confidence 0.800 pass, Decision Ledger
hash chain valid).

### MTTD / MTTR (platform-run)
- **Target:** < 60 seconds (p95)
- **Status:** grounded (platform-run MTTR)
- **Formula:** apply.failed.time → successful retry.time
- **Source:** `metrics/nova_metrics.db` `fact_run` (started_at, completed_at)
- **Note:** infra-incident MTTR deferred (no incident detection system)
- **Definition-of-success:** `docs/metrics/mttr.md`

### Confidence-Gate Halt Rate (REQ-212)
- **Target:** not a committed target (operational signal)
- **Status:** grounded
- **Formula:** runs where confidence band = halt ÷ total runs
- **Source:** `metrics/nova_metrics.db` `fact_confidence` (band column)
- **Definition-of-success:** `docs/metrics/confidence_gate_halt_rate.md`

---

## Velocity (REQ-192)

### Provisioning Lead Time
- **Target:** not a committed target (operational signal)
- **Status:** grounded (after P1)
- **Formula:** apply.completed.time − intent.received.time
- **Source:** `metrics/nova_metrics.db` `fact_run` (started_at, completed_at)
- **Definition-of-success:** `docs/metrics/provisioning_lead_time.md`

### Deployment Frequency
- **Target:** not a committed target (operational signal)
- **Status:** grounded (after P1)
- **Formula:** count(run.completed) per day
- **Source:** `metrics/nova_metrics.db` `fact_run`
- **Definition-of-success:** `docs/metrics/deployment_frequency.md`

### Self-Healing Velocity — DEFERRED
- **Status:** deferred (no auto-remediator)
- **Blocking decision:** future emitter
- **Placeholder view:** `placeholder_predictive_reactive.csv`

---

## Financial & Cost ROI (REQ-193)

### Cost Savings via Infracost Estimates
- **Target:** ≥ 25% on pilot estates (partial)
- **Status:** partial (pre-apply estimate grounded; actual-spend deferred D-096)
- **Formula:** sum(cost_estimate.delta_usd) where delta < 0
- **Source:** `metrics/nova_metrics.db` `fact_cost_estimate`
- **Definition-of-success:** `docs/metrics/cost_savings.md`

### FTE Hours Saved (Toil Reallocation Value)
- **Target:** ≥ 70% of pre-Nova FTE allocation (derived)
- **Status:** derived
- **Formula:** run count × manual baseline minutes × blended rate
- **Source:** `metrics/nova_metrics.db` `fact_run` (count) + manual baseline
- **Note:** computed on N internal runs today; production-denominator
  activates post-pilot
- **Definition-of-success:** `docs/metrics/fte_hours_saved.md`

### Platform ROI
- **Target:** ≥ 250% measured annually (derived)
- **Status:** derived
- **Formula:** (FTE hours saved × blended rate + cloud savings + avoided
  downtime) ÷ platform op cost
- **Source:** derived from fact_run + fact_cost_estimate + manual baseline
- **Note:** computed on N internal runs today; production-denominator
  activates post-pilot
- **Definition-of-success:** `docs/metrics/platform_roi.md`

### Live CUR Reconciliation — DEFERRED
- **Status:** deferred (D-096)
- **Placeholder view:** `placeholder_live_cur_reconciliation.csv`

---

## Reliability, Security & Compliance (REQ-194)

### Zero-Trust Policy Compliance Rate
- **Target:** not a committed target (operational signal)
- **Status:** grounded (after P1)
- **Formula:** 1 − count(assets WHERE last_scan.status ≠ pass) ÷ count(assets)
- **Source:** `metrics/nova_metrics.db` `fact_policy_check`
- **Definition-of-success:** `docs/metrics/policy_compliance_rate.md`

### Attestation Coverage
- **Target:** 100% of prod/dr promotions attested by a human
- **Status:** grounded
- **Formula:** prod/dr promotions attested ÷ total prod/dr promotions
- **Source:** `metrics/decision_ledger.db` (attestation.recorded events) +
  `hitl_gates.py` + outbox `approver_*` attributes
- **Definition-of-success:** `docs/metrics/attestation_coverage.md`

### SLA / Unplanned Downtime — DEFERRED
- **Status:** deferred (D-096)
- **Placeholder view:** `placeholder_sla_downtime.csv`

### Patch Remediation Rate — DEFERRED
- **Status:** deferred (no patch remediation system)
- **Placeholder view:** (future)

---

## Trust Substrate (REQ-211)

### Decision Ledger Coverage
- **Target:** 100% of AI actions with backfilled outcome
- **Status:** grounded (this milestone builds it)
- **Formula:** count(decision_ledger rows with outcome ≠ 'pending') ÷
  count(decision_ledger rows)
- **Source:** `metrics/decision_ledger.db` + `core/metrics/decision_ledger.py`
- **Definition-of-success:** `docs/metrics/decision_ledger_coverage.md`

### Trust Snapshot
- **Status:** grounded (P4 tool)
- **Source:** `core/metrics/trust_snapshot.py` → `metrics/TRUST_SNAPSHOT.md`
- **Contents:** Decision Ledger Coverage, Attestation Coverage, Capability
  Health, AI Decision Accuracy, Confidence-Gate Halt Rate, chain-integrity
  verdict, snapshot hash

---

## Deferred Metrics (8 placeholder views)

| Metric | Blocking Decision | Placeholder View |
|--------|-----------------|------------------|
| Live Infrastructure Health | D-096 | `placeholder_live_infra_health.csv` |
| Live Outbox Write Rate | D-096 | `placeholder_live_outbox_rate.csv` |
| Tamper-Evident Ledger Checkpoints | D-083 | `placeholder_tamper_evident_checkpoints.csv` |
| Onboarding Funnel (granted) | D-113/D-114/D-119 | `placeholder_onboarding_funnel.csv` |
| Drift Auto-Reversal Rate | D-096 + no scheduler | `placeholder_drift_detection.csv` |
| Live CUR Reconciliation | D-096 | `placeholder_live_cur_reconciliation.csv` |
| SLA / Unplanned Downtime | D-096 | `placeholder_sla_downtime.csv` |
| Predictive vs Reactive Ratio | future emitter | `placeholder_predictive_reactive.csv` |

See `docs/METRICS_DEFERRED_ROADMAP.md` for the activation path for each.

---

## v1.25 — Swappable Policy Engine

The policy engine that produces the `PolicyCheckResult` records feeding
the confidence signal is **swappable** (NORTH_STAR Strategic Objective #2
— provable trust via a replaceable substrate, not a vendor lock-in).
The `PolicyEngine` protocol (`core/policy_engine.py`) is the swap
boundary; `config.json.policy.engine` selects the active engine
(default `"kyverno-json"`). A future `OpaEngine` implements the same
protocol without touching the confidence signal, the PCR schema, or
the pipeline. See `.ciagent/ARCHITECTURE.md` §12.7 for the registry
diagram.