Jon Chery
38b51f3e6d
regression/ policies (3): cap-013-adapter-dedup, cap-023-metrics-collector, cap-024-deck-structure — declarative mirrors of core/regression_verify.py over capability-inventory JSON. The imperative regression_verify.py is kept (drives CI gate); the policies are the declarative mirror (IDEATE I1 quality improvement). tests: test_regression_policies.py + clean/drifted fixtures. Skip-without-kj. docs: adapters/README.md (new kyverno-json row + PolicyEngine Protocol section with how-to-add-OpaEngine), adapters/kyverno-json/README.md (engine, install, policy directory layout, 4 categories, severity convention), schemas/README.md (D-116 engine enum reuse note), modules/STANDARDS.md §10 Policy Authoring Standard, docs/METRICS.md (swappable engine narrative). ---ci--- project: acdl phase: 4 milestone: v1.25 status: execute phase_role: execution requirements: covered: [REQ-304, REQ-305, REQ-306, REQ-307] partial: [] ---/ci---
191 lines
7.9 KiB
Markdown
191 lines
7.9 KiB
Markdown
# Nova Metrics Catalog
|
||
|
||
|
||
This is the canonical catalog of every executive KPI in Nova's
|
||
leadership metrics layer. Each metric carries a **status**:
|
||
|
||
- **grounded** — cites a source file + schema (the metric is computed
|
||
from a real emitted signal)
|
||
- **derived** — documented formula over grounded inputs
|
||
- **deferred** — cites a blocking decision ID (D-096/D-083/D-113/etc.);
|
||
ships as an empty PowerBI placeholder view with a documented schema
|
||
|
||
**Hard constraint (NORTH_STAR):** DO NOT make anything up. No fabricated
|
||
numbers. Every metric either has a real source or is explicitly deferred.
|
||
|
||
---
|
||
|
||
## Zero-Touch Efficiency & AI Autonomy (REQ-191)
|
||
|
||
### Touchless Resolution Rate
|
||
- **Target:** ≥ 99% across production estates (Post-Pilot)
|
||
- **Status:** partial (pipeline grounded; denominator = 0 today)
|
||
- **Formula:** runs completing without *operational* HITL block ÷ total runs
|
||
(attestation gates excluded — they're designed controls, not escalations)
|
||
- **Source:** `metrics/nova_metrics.db` `fact_run` (hitl_block column)
|
||
- **Definition-of-success:** `docs/metrics/touchless_resolution_rate.md`
|
||
|
||
### Human Escalation Frequency
|
||
- **Target:** < 0.1% of platform actions (Post-Pilot)
|
||
- **Status:** partial (pipeline grounded; denominator = 0 today)
|
||
- **Formula:** operational HITL blocks ÷ total runs (attestation sign-offs
|
||
excluded)
|
||
- **Source:** `metrics/nova_metrics.db` `fact_run` (hitl_block column)
|
||
- **Definition-of-success:** `docs/metrics/human_escalation_frequency.md`
|
||
|
||
### AI Decision Accuracy
|
||
- **Target:** ≥ 99.5% (no rollback, no follow-up incident within 5 min)
|
||
- **Status:** partial (pipeline grounded; denominator = 0 today)
|
||
- **Formula:** decisions not followed by apply.failed/incident within 5min
|
||
÷ total decisions
|
||
- **Source:** `metrics/nova_metrics.db` `fact_decision` (outcome column)
|
||
- **Definition-of-success:** `docs/metrics/ai_decision_accuracy.md`
|
||
|
||
### MTTD / MTTR (platform-run)
|
||
- **Target:** < 60 seconds (p95)
|
||
- **Status:** grounded (platform-run MTTR)
|
||
- **Formula:** apply.failed.time → successful retry.time
|
||
- **Source:** `metrics/nova_metrics.db` `fact_run` (started_at, completed_at)
|
||
- **Note:** infra-incident MTTR deferred (no incident detection system)
|
||
- **Definition-of-success:** `docs/metrics/mttr.md`
|
||
|
||
### Confidence-Gate Halt Rate (REQ-212)
|
||
- **Target:** not a committed target (operational signal)
|
||
- **Status:** grounded
|
||
- **Formula:** runs where confidence band = halt ÷ total runs
|
||
- **Source:** `metrics/nova_metrics.db` `fact_confidence` (band column)
|
||
- **Definition-of-success:** `docs/metrics/confidence_gate_halt_rate.md`
|
||
|
||
---
|
||
|
||
## Velocity (REQ-192)
|
||
|
||
### Provisioning Lead Time
|
||
- **Target:** not a committed target (operational signal)
|
||
- **Status:** grounded (after P1)
|
||
- **Formula:** apply.completed.time − intent.received.time
|
||
- **Source:** `metrics/nova_metrics.db` `fact_run` (started_at, completed_at)
|
||
- **Definition-of-success:** `docs/metrics/provisioning_lead_time.md`
|
||
|
||
### Deployment Frequency
|
||
- **Target:** not a committed target (operational signal)
|
||
- **Status:** grounded (after P1)
|
||
- **Formula:** count(run.completed) per day
|
||
- **Source:** `metrics/nova_metrics.db` `fact_run`
|
||
- **Definition-of-success:** `docs/metrics/deployment_frequency.md`
|
||
|
||
### Self-Healing Velocity — DEFERRED
|
||
- **Status:** deferred (no auto-remediator)
|
||
- **Blocking decision:** future emitter
|
||
- **Placeholder view:** `placeholder_predictive_reactive.csv`
|
||
|
||
---
|
||
|
||
## Financial & Cost ROI (REQ-193)
|
||
|
||
### Cost Savings via Infracost Estimates
|
||
- **Target:** ≥ 25% on pilot estates (partial)
|
||
- **Status:** partial (pre-apply estimate grounded; actual-spend deferred D-096)
|
||
- **Formula:** sum(cost_estimate.delta_usd) where delta < 0
|
||
- **Source:** `metrics/nova_metrics.db` `fact_cost_estimate`
|
||
- **Definition-of-success:** `docs/metrics/cost_savings.md`
|
||
|
||
### FTE Hours Saved (Toil Reallocation Value)
|
||
- **Target:** ≥ 70% of pre-Nova FTE allocation (derived)
|
||
- **Status:** derived
|
||
- **Formula:** run count × manual baseline minutes × blended rate
|
||
- **Source:** `metrics/nova_metrics.db` `fact_run` (count) + manual baseline
|
||
- **Note:** computed on N internal runs today; production-denominator
|
||
activates post-pilot
|
||
- **Definition-of-success:** `docs/metrics/fte_hours_saved.md`
|
||
|
||
### Platform ROI
|
||
- **Target:** ≥ 250% measured annually (derived)
|
||
- **Status:** derived
|
||
- **Formula:** (FTE hours saved × blended rate + cloud savings + avoided
|
||
downtime) ÷ platform op cost
|
||
- **Source:** derived from fact_run + fact_cost_estimate + manual baseline
|
||
- **Note:** computed on N internal runs today; production-denominator
|
||
activates post-pilot
|
||
- **Definition-of-success:** `docs/metrics/platform_roi.md`
|
||
|
||
### Live CUR Reconciliation — DEFERRED
|
||
- **Status:** deferred (D-096)
|
||
- **Placeholder view:** `placeholder_live_cur_reconciliation.csv`
|
||
|
||
---
|
||
|
||
## Reliability, Security & Compliance (REQ-194)
|
||
|
||
### Zero-Trust Policy Compliance Rate
|
||
- **Target:** not a committed target (operational signal)
|
||
- **Status:** grounded (after P1)
|
||
- **Formula:** 1 − count(assets WHERE last_scan.status ≠ pass) ÷ count(assets)
|
||
- **Source:** `metrics/nova_metrics.db` `fact_policy_check`
|
||
- **Definition-of-success:** `docs/metrics/policy_compliance_rate.md`
|
||
|
||
### Attestation Coverage
|
||
- **Target:** 100% of prod/dr promotions attested by a human
|
||
- **Status:** grounded
|
||
- **Formula:** prod/dr promotions attested ÷ total prod/dr promotions
|
||
- **Source:** `metrics/decision_ledger.db` (attestation.recorded events) +
|
||
`hitl_gates.py` + outbox `approver_*` attributes
|
||
- **Definition-of-success:** `docs/metrics/attestation_coverage.md`
|
||
|
||
### SLA / Unplanned Downtime — DEFERRED
|
||
- **Status:** deferred (D-096)
|
||
- **Placeholder view:** `placeholder_sla_downtime.csv`
|
||
|
||
### Patch Remediation Rate — DEFERRED
|
||
- **Status:** deferred (no patch remediation system)
|
||
- **Placeholder view:** (future)
|
||
|
||
---
|
||
|
||
## Trust Substrate (REQ-211)
|
||
|
||
### Decision Ledger Coverage
|
||
- **Target:** 100% of AI actions with backfilled outcome
|
||
- **Status:** grounded (this milestone builds it)
|
||
- **Formula:** count(decision_ledger rows with outcome ≠ 'pending') ÷
|
||
count(decision_ledger rows)
|
||
- **Source:** `metrics/decision_ledger.db` + `core/metrics/decision_ledger.py`
|
||
- **Definition-of-success:** `docs/metrics/decision_ledger_coverage.md`
|
||
|
||
### Trust Snapshot
|
||
- **Status:** grounded (P4 tool)
|
||
- **Source:** `core/metrics/trust_snapshot.py` → `metrics/TRUST_SNAPSHOT.md`
|
||
- **Contents:** Decision Ledger Coverage, Attestation Coverage, Capability
|
||
Health, AI Decision Accuracy, Confidence-Gate Halt Rate, chain-integrity
|
||
verdict, snapshot hash
|
||
|
||
---
|
||
|
||
## Deferred Metrics (8 placeholder views)
|
||
|
||
| Metric | Blocking Decision | Placeholder View |
|
||
|--------|-----------------|------------------|
|
||
| Live Infrastructure Health | D-096 | `placeholder_live_infra_health.csv` |
|
||
| Live Outbox Write Rate | D-096 | `placeholder_live_outbox_rate.csv` |
|
||
| Tamper-Evident Ledger Checkpoints | D-083 | `placeholder_tamper_evident_checkpoints.csv` |
|
||
| Onboarding Funnel (granted) | D-113/D-114/D-119 | `placeholder_onboarding_funnel.csv` |
|
||
| Drift Auto-Reversal Rate | D-096 + no scheduler | `placeholder_drift_detection.csv` |
|
||
| Live CUR Reconciliation | D-096 | `placeholder_live_cur_reconciliation.csv` |
|
||
| SLA / Unplanned Downtime | D-096 | `placeholder_sla_downtime.csv` |
|
||
| Predictive vs Reactive Ratio | future emitter | `placeholder_predictive_reactive.csv` |
|
||
|
||
See `docs/METRICS_DEFERRED_ROADMAP.md` for the activation path for each.
|
||
|
||
---
|
||
|
||
## v1.25 — Swappable Policy Engine
|
||
|
||
The policy engine that produces the `PolicyCheckResult` records feeding
|
||
the confidence signal is **swappable** (NORTH_STAR Strategic Objective #2
|
||
— provable trust via a replaceable substrate, not a vendor lock-in).
|
||
The `PolicyEngine` protocol (`core/policy_engine.py`) is the swap
|
||
boundary; `config.json.policy.engine` selects the active engine
|
||
(default `"kyverno-json"`). A future `OpaEngine` implements the same
|
||
protocol without touching the confidence signal, the PCR schema, or
|
||
the pipeline. See `.ciagent/ARCHITECTURE.md` §12.7 for the registry
|
||
diagram. |