Jon Chery
b054849a99
P4 (Wave 3, docs) — REQ-186, 191, 192, 193, 194, 195, 204, 210, 211, 212, 213 New docs: - docs/METRICS.md — canonical KPI catalog (grounded/derived/deferred) - docs/metrics/*.md — 13 per-KPI definition-of-success docs (D-127) - docs/METRICS_DEFERRED_ROADMAP.md — 8 deferred metrics + hot-path plan + re-eval triggers (REQ-210) - docs/NO_HUMANS_THESIS.md — thesis defensibility brief (REQ-213) New tools: - core/metrics/trust_snapshot.py — 5 trust metrics + chain-integrity verdict + snapshot hash (REQ-211) - scripts/check_north_star_diff.sh — CI check for NORTH_STAR strategic section changes (REQ-204) Modified: - .ciagent/config.json — strategic_direction_file: .ciagent/NORTH_STAR.md (REQ-186) ---ci--- project: acdl phase: 4 milestone: v1.17 status: execute ---/ci---
21 lines
998 B
Markdown
21 lines
998 B
Markdown
# AI Decision Accuracy — Definition of Success
|
|
|
|
> KPI: AI Decision Accuracy
|
|
> Target: ≥ 99.5% (no rollback, no follow-up incident within 5 min of action)
|
|
|
|
**What this number means:** the percentage of AI decisions (confidence-
|
|
gated policy engine outcomes) that were NOT followed by an apply failure
|
|
or incident within 5 minutes. A high-confidence decision that later
|
|
caused an incident does NOT count as accurate.
|
|
|
|
**How it's computed:** `count(decisions WHERE outcome = 'succeeded' AND
|
|
no incident within 5min)` ÷ `total decisions`. Correlation via
|
|
`decision_id` → `run_id` → subsequent `apply.failed` or `incident.detected`
|
|
events.
|
|
|
|
**What "good" looks like:** ≥ 99.5% means fewer than 1 in 200 decisions
|
|
cause a secondary failure. The 0.5% allowance is for novel edge cases.
|
|
|
|
**D-122 honesty:** Nova's "AI" is the confidence-gated policy engine
|
|
(confidence_signal + HITL gate), not an LLM planner. The Decision Ledger
|
|
captures this real decision path — not a fabricated "AI agent." |