docs(P04 W2): pilot-run docs (REQ-321) — adapters/README, METRICS, ARCHITECTURE §12.8, consumer onboarding
- adapters/README.md: fixed stale TYPE_MAP/INPUT_MAP refs (the adapter is a stateless assembler); added the blockchain-exchange consumer row + the Gitea adapter note (SPEC §10 Q1 — no cross-repo uses:) - docs/METRICS.md: Post-Pilot denominators activated (AI Decision Accuracy + Human Escalation Frequency + the third metric now have non-zero data from the blkex-pilot-apply-v0.2 run) - .ciagent/ARCHITECTURE.md §12.8: Pilot Estate (v1.26 live) — the first real consumer estate, the live apply, the Gitea adapter, the evidence stream - .ciagent/nova-blockchain-exchange/README.md: consumer onboarding guide (deploy invocation, secrets, contract shape, verification) ---ci--- project: acdl phase: 4 milestone: v1.26 status: execute wave: W2 ---
This commit is contained in:
+37
-3
@@ -19,7 +19,7 @@ numbers. Every metric either has a real source or is explicitly deferred.
|
||||
|
||||
### Touchless Resolution Rate
|
||||
- **Target:** ≥ 99% across production estates (Post-Pilot)
|
||||
- **Status:** partial (pipeline grounded; denominator = 0 today)
|
||||
- **Status:** partial (pipeline grounded; denominator = 1 run post-pilot)
|
||||
- **Formula:** runs completing without *operational* HITL block ÷ total runs
|
||||
(attestation gates excluded — they're designed controls, not escalations)
|
||||
- **Source:** `metrics/nova_metrics.db` `fact_run` (hitl_block column)
|
||||
@@ -27,20 +27,54 @@ numbers. Every metric either has a real source or is explicitly deferred.
|
||||
|
||||
### Human Escalation Frequency
|
||||
- **Target:** < 0.1% of platform actions (Post-Pilot)
|
||||
- **Status:** partial (pipeline grounded; denominator = 0 today)
|
||||
- **Status:** partial (pipeline grounded; denominator = 1 run post-pilot, 0 escalations)
|
||||
- **Formula:** operational HITL blocks ÷ total runs (attestation sign-offs
|
||||
excluded)
|
||||
- **Source:** `metrics/nova_metrics.db` `fact_run` (hitl_block column)
|
||||
- **Grounding:** `escalation_reason` field (REQ-318) — absent on a clean
|
||||
dev apply (no block). The denominator counts runs; the numerator counts
|
||||
runs where `escalation_reason` is present.
|
||||
- **Definition-of-success:** `docs/metrics/human_escalation_frequency.md`
|
||||
|
||||
### AI Decision Accuracy
|
||||
- **Target:** ≥ 99.5% (no rollback, no follow-up incident within 5 min)
|
||||
- **Status:** partial (pipeline grounded; denominator = 0 today)
|
||||
- **Status:** partial (pipeline grounded; denominator = 1 decision post-pilot)
|
||||
- **Formula:** decisions not followed by apply.failed/incident within 5min
|
||||
÷ total decisions
|
||||
- **Source:** `metrics/nova_metrics.db` `fact_decision` (outcome column)
|
||||
- **Grounding:** `fact_decision.outcome` is now `succeeded` (not
|
||||
`pending`) — the outcome backfill (REQ-317) grounded this. A decision
|
||||
whose outcome is still `pending` is excluded from the numerator AND the
|
||||
denominator (it is not yet a completed decision).
|
||||
- **Definition-of-success:** `docs/metrics/ai_decision_accuracy.md`
|
||||
|
||||
#### Post-Pilot Activation (v1.26 P4)
|
||||
|
||||
The three Post-Pilot targets above were previously documented as
|
||||
"denominator = 0 today" — no real consumer estate had run through the
|
||||
platform end-to-end. The v1.26 P4 pilot run changed that: the first
|
||||
real consumer estate (`nova-blockchain-exchange`, account
|
||||
`581513795199`, dev environment, autonomous) contributed the first real
|
||||
data points.
|
||||
|
||||
- **Run id:** `blkex-pilot-apply-v0.2` (2026-08-19)
|
||||
- **AI Decision Accuracy:** 1 decision (`blkex-pilot-apply-v0.2`),
|
||||
outcome `pending → succeeded` (REQ-317 backfill). Numerator = 1
|
||||
(no apply.failed, no incident), denominator = 1. Future runs
|
||||
accumulate into this denominator.
|
||||
- **Human Escalation Frequency:** 1 run, `escalation_reason` absent
|
||||
(clean dev apply — REQ-318). Numerator = 0 escalations, denominator
|
||||
= 1.
|
||||
- **Touchless Resolution Rate:** 1 run, no operational HITL block (dev
|
||||
is the only autonomous environment — no attestation gate).
|
||||
Numerator = 1, denominator = 1.
|
||||
|
||||
The denominators are now non-zero. Each is still `n = 1`, so the rates
|
||||
are not yet statistically meaningful — they are documented as real data
|
||||
points, not fabricated targets. See `.ciagent/P4-PILOT-RUN-EVIDENCE.md`
|
||||
for the full evidence stream (confidence 0.800 pass, Decision Ledger
|
||||
hash chain valid).
|
||||
|
||||
### MTTD / MTTR (platform-run)
|
||||
- **Target:** < 60 seconds (p95)
|
||||
- **Status:** grounded (platform-run MTTR)
|
||||
|
||||
Reference in New Issue
Block a user