Jon Chery
b054849a99
P4 (Wave 3, docs) — REQ-186, 191, 192, 193, 194, 195, 204, 210, 211, 212, 213 New docs: - docs/METRICS.md — canonical KPI catalog (grounded/derived/deferred) - docs/metrics/*.md — 13 per-KPI definition-of-success docs (D-127) - docs/METRICS_DEFERRED_ROADMAP.md — 8 deferred metrics + hot-path plan + re-eval triggers (REQ-210) - docs/NO_HUMANS_THESIS.md — thesis defensibility brief (REQ-213) New tools: - core/metrics/trust_snapshot.py — 5 trust metrics + chain-integrity verdict + snapshot hash (REQ-211) - scripts/check_north_star_diff.sh — CI check for NORTH_STAR strategic section changes (REQ-204) Modified: - .ciagent/config.json — strategic_direction_file: .ciagent/NORTH_STAR.md (REQ-186) ---ci--- project: acdl phase: 4 milestone: v1.17 status: execute ---/ci---
23 lines
1.1 KiB
Markdown
23 lines
1.1 KiB
Markdown
# Touchless Resolution Rate — Definition of Success
|
|
|
|
> KPI: Touchless Resolution Rate
|
|
> Target: ≥ 99% across production estates (Post-Pilot)
|
|
|
|
**What this number means:** the percentage of platform runs that complete
|
|
end-to-end without an operational HITL block. An operational HITL block
|
|
is a confidence-driven escalation (the AI's confidence was too low to
|
|
proceed). Attestation gates (qa/prod/dr sign-offs) are NOT counted as
|
|
escalations — they are designed controls, not autonomy failures.
|
|
|
|
**How it's computed:** `runs WHERE hitl_block = 0 AND environment = 'dev'`
|
|
÷ `total runs` (dev environment only, where attestation gates don't apply).
|
|
For production estates: `runs WHERE hitl_block = 0` ÷ `total runs`
|
|
excluding attestation-gate sign-offs.
|
|
|
|
**What "good" looks like:** ≥ 99% means fewer than 1 in 100 runs require
|
|
human intervention due to low confidence. The 1% allowance is for
|
|
genuine edge cases (novel failure modes, blast-radius exceedances).
|
|
|
|
**What would be "gamer metrics":** counting attestation gates as
|
|
"touchless" (they're not — they're human by design) or counting only
|
|
dev runs (cherry-picking the easiest environment). |