Compare commits
3 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Praxis CI
|
81d43666c7 | ||
|
Praxis CI
|
fb26d3388e | ||
|
Praxis CI
|
5290d4d05d |
@@ -1,408 +0,0 @@
|
||||
# Praxis — v0.5 Milestone Audit (Final Phase P3)
|
||||
|
||||
> **Phase:** 3 — Review + Ship (FINAL PHASE audit, v0.5 milestone)
|
||||
> **Milestone:** v0.5 (Live Assist — on-the-job voice companion)
|
||||
> **Branch:** `phase/03-final-review-ship` (current; 2 commits ahead of `milestone/v0.5-live-assist` tip `b621cb6` — the P3 verify + P0-fix commits; this audit IS the P3 work)
|
||||
> **Auditor:** CIAgent ci-doc-verifier (mechanical, autonomy `full`, single-project mode, slug `praxis`)
|
||||
> **Date:** 2026-08-04
|
||||
> **Mode:** P3 final milestone audit per the audit workflow — verifies the entire v0.5 milestone is healthy before the milestone merge to main
|
||||
> **Codebase state at audit:** HEAD = `2627923` (P3 verify commit); 9 commits `v0.1.9..HEAD` (v0.4 completion + P0 merge + ship, P1 merge + ship, P2 merge + ship, P3 verify + P0-fix); working tree clean (no auto-fixes applied — this audit surfaces, does not fix, per the audit charter)
|
||||
> **Inputs:** git log (`v0.1.9..HEAD` = 9 commits, `--all` = 105 commits), `.ciagent/` files (30), `---ci---` blocks (all 9 v0.5-range commits verified), REVIEW-v0.5.md (multi-persona code review, APPROVE_WITH_NOTES), VERIFY-P1-v0.5.md + VERIFY-P2-v0.5.md, tag verification, branch/merge topology, GRILL-v0.5.md (39 decisions, 2 MUSTs, 1 escalation), reflog topology reconstruction
|
||||
|
||||
---
|
||||
|
||||
## 1. Audit Summary
|
||||
|
||||
| # | Check | Result | Notes |
|
||||
|---|-------|--------|-------|
|
||||
| 1 | Reconstruction test | ✅ PASS | git log `v0.1.9..HEAD` (9 commits) covers P0/P1/P2/P3; all `---ci---` blocks correct (project/phase/milestone/status); all 16 v0.5 REQ-IDs appear in commit `requirements.covered` blocks; tags v0.1.10/v0.1.11/v0.1.12 exist + annotated + point to correct commits; phase progression matches ROADMAP.md |
|
||||
| 2 | `.ciagent/` file discipline | ⚠️ NEEDS_ATTENTION | All 13 expected files present + current; v0.4 reference files retained; **5 stale-status fields** found (PROJECT.md:4, ROADMAP.md:3-4, REQUIREMENTS.md:3-4 + 16 REQ status fields, CHECKPOINT.json phase/stage, config.json status) — same stale-status drift class as the v0.4 audit; NOT auto-fixed (audit surfaces, does not fix) |
|
||||
| 3 | Branch hygiene | ✅ PASS | main → milestone/v0.5 → phase/03 hierarchy correct; v0.5 phase branches (phase/00, phase/01-assist-core-guardrail, phase/02-integration-techdebt-nfr) created + deleted post-merge (confirmed via reflog); old v0.2/v0.3/v0.4 phase branches retained (housekeeping pattern) |
|
||||
| 4 | Commit discipline | ✅ PASS | All 9 commits have `---ci---` blocks (count = 9 = commit count); correct `project: praxis`, `milestone: v0.5`; conventional-commit prefixes (`docs(ship)`, `feat(P01)`, `feat(P02)`, `docs(P00)`, `verify(P03)`, `fix(P03)`); no secrets in commit messages |
|
||||
|
||||
**Final verdict: NEEDS_ATTENTION** — reconstruction + branch + commit discipline all PASS; file discipline has 5 stale-status fields (stale-status drift, not logic/data/scope errors) that the ship orchestrator should fix before/during ship. No critical issues. The milestone is healthy and ready for ship after the stale-status fields are advanced.
|
||||
|
||||
---
|
||||
|
||||
## 2. Check 1 — Reconstruction Test
|
||||
|
||||
### 2.1 Git log phase-by-phase vs ROADMAP.md
|
||||
|
||||
`git log --oneline v0.1.9..HEAD` (9 commits, oldest → newest):
|
||||
|
||||
```
|
||||
ba928cf docs(milestone): complete v0.4-operator-tier — v0.1.9 tagged, release created, merged to main [v0.4 complete — base of v0.5 range]
|
||||
5290d4d docs(P00): complete v0.5 phase 0 pre-execution — v0.1.10 tagged [P0]
|
||||
fb26d33 docs(ship): phase 0 complete — v0.1.10 tagged, release #443 created [P0 ship]
|
||||
81d4366 feat(P01): complete assist core + guardrail phase — v0.1.11 tagged [P1]
|
||||
38b97ee docs(ship): P1 complete — v0.1.11 tagged, release #451 created [P1 ship]
|
||||
bdcf793 feat(P02): complete integration + tech-debt + NFR measurement phase — v0.1.12 tagged [P2]
|
||||
b621cb6 docs(ship): P2 complete — v0.1.12 tagged, release #452 created [P2 ship]
|
||||
5373df2 fix(P03): P0 — guardrail processor must buffer LLM text before TTS (REQ-ASSIST-03) [P3 P0-fix]
|
||||
2627923 verify(P03): code review — v0.5 Live Assist (5 personas, APPROVE_WITH_NOTES) [P3 verify — HEAD]
|
||||
```
|
||||
|
||||
**Phase progression vs ROADMAP.md:**
|
||||
- Phase 0 — Pre-Execution: complete — tagged v0.1.10 ✅ matches `5290d4d`/`fb26d33`
|
||||
- Phase 1 — Assist Core + Guardrail: complete — tagged v0.1.11 ✅ matches `81d4366`/`38b97ee`
|
||||
- Phase 2 — Integration + Tech-Debt + NFR Measurement: complete — tagged v0.1.12 ✅ matches `bdcf793`/`b621cb6`
|
||||
- Final Phase (P3) — Review + Ship: in-progress (this audit) ✅ matches `5373df2`/`2627923` on `phase/03-final-review-ship`
|
||||
|
||||
**Note on v0.5 merge topology:** Unlike v0.4 (which used `feat(milestone): merge phase/NN` squash-merge commits with separate phase branches retained), v0.5 committed phase work directly to `milestone/v0.5-live-assist` as `feat(P01)`/`feat(P02)`/`docs(P00)` commits (single-parent, linear). The reflog confirms v0.5 DID use phase branches during execution (`phase/00-pre-execution`, `phase/01-assist-core-guardrail`, `phase/02-integration-techdebt-nfr`) — they were created, worked on, then deleted post-merge to milestone. This is a **different but valid** merge pattern (linear direct-commit vs squash-merge). The phase work is fully traceable via the `feat(PNN)`/`docs(PNN)` commit prefixes + `---ci---` blocks. Non-blocking — the v0.4 squash-merge pattern is preferred for phase-boundary integrity, but the v0.5 linear pattern preserves full traceability.
|
||||
|
||||
### 2.2 `---ci---` blocks vs declared phase/stage/milestone
|
||||
|
||||
All 9 `v0.1.9..HEAD` commits carry `---ci---` blocks (`git log v0.1.9..HEAD --pretty=%B | grep -c "^---ci---"` = 9 = commit count). Verified each block:
|
||||
|
||||
| Commit | phase | milestone | status | requirements.covered | Match |
|
||||
|--------|-------|-----------|--------|----------------------|-------|
|
||||
| `ba928cf` (v0.4 complete) | 3 | v0.4 | complete | [8 v0.4 REQs] | ✅ (v0.4 carry-over — base of range) |
|
||||
| `5290d4d` (P0 merge) | 0 | v0.5 | complete | [16 v0.5 REQs] | ✅ all 16 |
|
||||
| `fb26d33` (P0 ship) | 0 | v0.5 | complete | tag v0.1.10 | ✅ |
|
||||
| `81d4366` (P1 merge) | 1 | v0.5 | complete | [12 REQs: 3 ASSIST + 3 NFR + 6 IDEATE] | ✅ 12 REQs |
|
||||
| `38b97ee` (P1 ship) | 1 | v0.5 | complete | tag v0.1.11 | ✅ |
|
||||
| `bdcf793` (P2 merge) | 2 | v0.5 | complete | [4 REQs: NFR-ASSIST-01 + IDEATE-04/06/07] | ✅ 4 REQs |
|
||||
| `b621cb6` (P2 ship) | 2 | v0.5 | complete | tag v0.1.12 | ✅ |
|
||||
| `5373df2` (P3 P0-fix) | 3 | v0.5 | verify | (lessons block) | ✅ |
|
||||
| `2627923` (P3 verify) | 3 | v0.5 | verify | (verdict block) | ✅ |
|
||||
|
||||
All blocks declare `project: praxis` (matches config.json `active_project`). ✅
|
||||
|
||||
**REQ coverage reconciliation:**
|
||||
- P0 merge claims all 16 (planning — all REQs activated)
|
||||
- P1 merge claims 12 (the P1-implemented REQs: 3 ASSIST + 3 NFR-ASSIST + 6 IDEATE)
|
||||
- P2 merge claims 4 (the P2-implemented REQs: NFR-ASSIST-01 + IDEATE-04/06/07)
|
||||
- 12 + 4 = 16 ✅ — all 16 v0.5 REQ-IDs covered across P1+P2 (no overlap, no gaps)
|
||||
|
||||
### 2.3 All 16 v0.5 REQ-IDs covered in commit blocks
|
||||
|
||||
`git log v0.1.9..HEAD --pretty=%B | grep -oE "REQ-(ASSIST|NFR-ASSIST|IDEATE)-[0-9]+" | sort -u` returns all 16:
|
||||
|
||||
| REQ-ID | Phase claimed | Verified |
|
||||
|--------|----------------|----------|
|
||||
| REQ-ASSIST-01 | P1 | ✅ P1 merge `81d4366` |
|
||||
| REQ-ASSIST-02 | P1 | ✅ P1 merge `81d4366` |
|
||||
| REQ-ASSIST-03 | P1 | ✅ P1 merge `81d4366` |
|
||||
| REQ-NFR-ASSIST-01 | P2 | ✅ P2 merge `bdcf793` |
|
||||
| REQ-NFR-ASSIST-02 | P1 | ✅ P1 merge `81d4366` |
|
||||
| REQ-NFR-ASSIST-03 | P1 | ✅ P1 merge `81d4366` |
|
||||
| REQ-NFR-ASSIST-04 | P1 | ✅ P1 merge `81d4366` |
|
||||
| REQ-IDEATE-01 | P1 | ✅ P1 merge `81d4366` |
|
||||
| REQ-IDEATE-02 | P1 | ✅ P1 merge `81d4366` |
|
||||
| REQ-IDEATE-03 | P1 | ✅ P1 merge `81d4366` |
|
||||
| REQ-IDEATE-04 | P2 | ✅ P2 merge `bdcf793` |
|
||||
| REQ-IDEATE-05 | P1 | ✅ P1 merge `81d4366` |
|
||||
| REQ-IDEATE-06 | P2 | ✅ P2 merge `bdcf793` |
|
||||
| REQ-IDEATE-07 | P2 | ✅ P2 merge `bdcf793` |
|
||||
| REQ-IDEATE-08 | P1 | ✅ P1 merge `81d4366` |
|
||||
| REQ-IDEATE-09 | P1 | ✅ P1 merge `81d4366` |
|
||||
|
||||
**16/16 covered.** ✅ Independently confirmed by REVIEW-v0.5.md REQ coverage table (16/16 COVERED), VERIFY-P1-v0.5.md (12/12), VERIFY-P2-v0.5.md (4/4), CHECKPOINT.json `p1_requirements_covered` (12) + `p2_requirements_covered` (4).
|
||||
|
||||
### 2.4 Tags v0.1.10, v0.1.11, v0.1.12 exist and point to the right commits
|
||||
|
||||
`git tag -l v0.1.10 v0.1.11 v0.1.12` → all three exist. `git cat-file -t` → all `tag` (annotated). `git rev-list -n1 <tag>`:
|
||||
|
||||
| Tag | Commit | Phase | Correct? |
|
||||
|-----|--------|-------|----------|
|
||||
| v0.1.10 | `5290d4d` | P0 merge (pre-execution) | ✅ |
|
||||
| v0.1.11 | `81d4366` | P1 merge (assist core + guardrail) | ✅ |
|
||||
| v0.1.12 | `bdcf793` | P2 merge (integration + tech-debt + NFR) | ✅ |
|
||||
|
||||
Tag sequence v0.1.9 (main, v0.4) < v0.1.10 < v0.1.11 < v0.1.12 — strictly increasing, no skips. ✅
|
||||
Next tag v0.1.13 (= v0.5 milestone release) not yet created — correct, ship is delegated to the orchestrator. ✅
|
||||
|
||||
### 2.5 CHECKPOINT.json vs actual state
|
||||
|
||||
**Current state (NOT auto-fixed by this audit):**
|
||||
```json
|
||||
{
|
||||
"phase": 2,
|
||||
"stage": "complete",
|
||||
"milestone": "v0.5",
|
||||
"phase_role": "execution",
|
||||
"tag": "v0.1.12",
|
||||
"next_tag": "v0.1.13",
|
||||
"p1_requirements_covered": [12 REQs],
|
||||
"p2_requirements_covered": [4 REQs],
|
||||
"p1_verify": "APPROVE_WITH_NOTES",
|
||||
"p2_verify": "APPROVE_WITH_NOTES",
|
||||
"p2_tests": "469 passed, 45 skipped, 0 failed",
|
||||
"grill_musts_resolved": ["G-049", "G-067"],
|
||||
"grill_escalations": ["ESCALATION-01"],
|
||||
"v0.4_p1_plus_addressed": 8
|
||||
}
|
||||
```
|
||||
|
||||
**Drift:** `phase: 2, stage: complete, phase_role: execution` reflects the P2-complete state but does NOT account for P3 in progress. The actual state is P3 (final review) in-progress — the verify commit `2627923` is on `phase/03-final-review-ship`. The `active_requirements` + `p1/p2_requirements_covered` + `grill_musts_resolved` + `v0.4_p1_plus_addressed` fields are all correct and consistent with the git log + REVIEW-v0.5.md. **Only the phase/stage/phase_role fields are stale.** See §3.4 for the fix recommendation (not applied — audit surfaces, does not fix).
|
||||
|
||||
**Reconstruction test verdict: PASS.** The git log tells the same story as PROJECT.md, ROADMAP.md, REQUIREMENTS.md, and CHECKPOINT.json — modulo the stale-status fields documented in §3.
|
||||
|
||||
---
|
||||
|
||||
## 3. Check 2 — `.ciagent/` File Discipline
|
||||
|
||||
### 3.1 All expected files exist
|
||||
|
||||
| File | Exists | Size | Notes |
|
||||
|------|--------|------|-------|
|
||||
| PROJECT.md | ✅ | 59.7 KB / 292 lines | v0.5 scope (D-058..D-073); ⚠️ status line stale (§3.4) |
|
||||
| ROADMAP.md | ✅ | 11.9 KB / 161 lines | v0.5 phases 0-2 complete, P3 in-progress; ⚠️ status line stale (§3.4) |
|
||||
| REQUIREMENTS.md | ✅ | 33.2 KB / 359 lines | 16 v0.5 REQs (3 ASSIST + 4 NFR + 9 IDEATE); 4 v0.6 backlog; ⚠️ status line + REQ status fields stale (§3.4) |
|
||||
| ARCHITECTURE.md | ✅ | 56.3 KB / 913 lines | v0.5 Live Assist Mode section at line 753 ✅ |
|
||||
| PERSONAS.md | ✅ | 47.7 KB / 692 lines | v0.5 roster (5 active: lead/voice/backend/security/data; 2 deactivated: devops/frontend) ✅ |
|
||||
| RESEARCH-v0.5-live-assist.md | ✅ | 77.8 KB / 760 lines | ✅ matches expected 760 lines |
|
||||
| PLAN-v0.5-live-assist.md | ✅ | 108 KB / 1075 lines | ✅ matches expected 1075 lines |
|
||||
| GRILL-v0.5.md | ✅ | 94.6 KB / 627 lines | ✅ matches expected 627 lines; 39 decisions, 2 MUSTs (G-049, G-067), 1 escalation (ESCALATION-01) |
|
||||
| VERIFY-P1-v0.5.md | ✅ | 37.3 KB / 493 lines | P1 verification, APPROVE_WITH_NOTES, 12/12 REQ, 5 P1+ |
|
||||
| VERIFY-P2-v0.5.md | ✅ | 33.5 KB / 502 lines | P2 verification, APPROVE_WITH_NOTES, 4/4 REQ, 3 P1+ |
|
||||
| REVIEW-v0.5.md | ✅ | 32.4 KB / 321 lines | P3 multi-persona review, APPROVE_WITH_NOTES, 5/5 personas PASS, 1 P0 fix, 8 P1+ |
|
||||
| CHECKPOINT.json | ✅ | 1.4 KB / 26 lines | ⚠️ phase/stage/phase_role stale (§3.4); all other fields current |
|
||||
| config.json | ✅ | 3.1 KB / 114 lines | active_project=praxis, milestone=v0.5, autonomy=full; ⚠️ status field stale (§3.4) |
|
||||
|
||||
All 13 expected files present. ✅
|
||||
|
||||
### 3.2 v0.4 / v0.3 / v0.2 / v0.1 reference files retained (not deleted)
|
||||
|
||||
| File | Exists | Notes |
|
||||
|------|--------|-------|
|
||||
| RESEARCH.md (v0.1) | ✅ | reference |
|
||||
| RESEARCH-vc.md (v0.3) | ✅ | reference |
|
||||
| RESEARCH-v0.3-anonymization-irt-scenarios.md | ✅ | reference |
|
||||
| RESEARCH-v0.4-operator-tier.md | ✅ | reference |
|
||||
| GRILL.md (v0.1) | ✅ | reference |
|
||||
| GRILL-v0.3.md | ✅ | reference |
|
||||
| GRILL-v0.4.md | ✅ | reference |
|
||||
| PLAN.md (v0.3) | ✅ | reference |
|
||||
| PLAN-v0.4-operator-tier.md | ✅ | reference |
|
||||
| VERIFY.md (v0.3 P1) | ✅ | reference |
|
||||
| VERIFY-P1.md (v0.4) | ✅ | reference |
|
||||
| VERIFY-P2.md (v0.4) | ✅ | reference |
|
||||
| REVIEW.md (v0.4) | ✅ | reference |
|
||||
| AUDIT.md (v0.3 + v0.4 sections preserved) | ✅ | reference |
|
||||
|
||||
Prior-milestone reference artifacts retained — no destructive deletion. ✅
|
||||
|
||||
### 3.3 Internal consistency (no contradictions)
|
||||
|
||||
- PROJECT.md §v0.5 scope (3 ASSIST + 4 NFR + 9 IDEATE = 16 REQs) ↔ REQUIREMENTS.md v0.5 active section (16 REQs) ↔ CHECKPOINT.json `active_requirements` (16) ↔ ROADMAP.md phase deliverables ↔ REVIEW-v0.5.md REQ coverage (16/16). **Consistent.** ✅
|
||||
- PROJECT.md out-of-scope list ↔ REQUIREMENTS.md v0.5 out-of-scope list — identical items. ✅
|
||||
- ROADMAP.md v0.5 phases ↔ actual git commits (`feat(P01)`, `feat(P02)`, `docs(P00)`). ✅
|
||||
- GRILL-v0.5.md MUSTs (G-049, G-067) ↔ CHECKPOINT.json `grill_musts_resolved` (["G-049", "G-067"]) ↔ REVIEW-v0.5.md grill MUSTs honored (2/2). ✅
|
||||
- CHECKPOINT.json `v0.4_p1_plus_addressed: 8` ↔ REVIEW-v0.5.md "8 v0.4 P1+ Tech-Debt Wave (all addressed)". ✅
|
||||
- No stale "v0.4 is active" references in v0.5 files (v0.4 consistently marked complete, tagged v0.1.9, merged to main). ✅
|
||||
|
||||
### 3.4 Stale-status fields found (NOT auto-fixed — audit surfaces, does not fix)
|
||||
|
||||
These are the same class of stale-status drift the v0.4 P3 audit found + auto-fixed. This audit does NOT auto-fix (per the audit charter: "audit surfaces, doesn't fix"). The ship orchestrator should advance these before/during ship.
|
||||
|
||||
| File:Line | Current (stale) | Expected (current) | Severity |
|
||||
|-----------|-----------------|---------------------|----------|
|
||||
| PROJECT.md:4 | `Status: phase 0 — pre-execution (active milestone)` | `Status: phase 3 — final review (active milestone); P0-P2 complete (v0.1.10/v0.1.11/v0.1.12 tagged)` | important (stale) |
|
||||
| ROADMAP.md:3 | `Milestone: v0.5 ... — active, phase 0 pre-execution` | `Milestone: v0.5 ... — active, phase 3 final review` | important (stale) |
|
||||
| ROADMAP.md:4 | `Status: phase 0 pre-execution (SPECIFY → ... → SHIP)` | `Status: phase 3 final review; P0-P2 complete (v0.1.10/v0.1.11/v0.1.12 tagged)` | important (stale) |
|
||||
| ROADMAP.md:15 | `Phase 0 — Pre-Execution (active)` | `Phase 0 — Pre-Execution (complete — tagged v0.1.10)` | important (stale) |
|
||||
| ROADMAP.md:19 | `Status: active (SPECIFY complete → CLARIFY → ...)` | `Status: complete (v0.1.10 tagged, release #443 created)` | important (stale) |
|
||||
| REQUIREMENTS.md:3 | `Milestone: v0.5 ... — active, phase 0` | `Milestone: v0.5 ... — active, phase 3 final review` | important (stale) |
|
||||
| REQUIREMENTS.md:4 | `Status: phase 0 pre-execution — v0.4 complete ...` | `Status: phase 3 final review; P0-P2 complete — 16/16 v0.5 REQ covered` | important (stale) |
|
||||
| REQUIREMENTS.md:14-16 | 3 REQ-ASSIST-* status `active` | `complete` (P1 merge `81d4366` covered them) | important (stale) |
|
||||
| REQUIREMENTS.md:22-25 | 4 REQ-NFR-ASSIST-* status `research-grounded` | `complete` (NFR-ASSIST-01 in P2; 02/03/04 in P1) | important (stale) |
|
||||
| REQUIREMENTS.md:37-70 | 9 REQ-IDEATE-* status `active` | `complete` (IDEATE-04/06/07 in P2; 01/02/03/05/08/09 in P1) | important (stale) |
|
||||
| CHECKPOINT.json:2-5 | `phase: 2, stage: complete, phase_role: execution` | `phase: 3, stage: in_progress, phase_role: final_review` | important (stale) |
|
||||
| config.json:7 | `"status": "phase-0-active"` | `"status": "phase-3-final-review"` | nit (stale — config.json status is informational; CHECKPOINT.json is the source of truth) |
|
||||
|
||||
**All 12 stale-status fields were set during P0 SPECIFY and never advanced as P1/P2 shipped.** This is the exact same drift pattern the v0.4 P3 audit documented (v0.4 AUDIT.md §B.4 / §G). The v0.4 audit auto-fixed them; this v0.5 audit surfaces them for the ship orchestrator per the audit charter ("audit surfaces, doesn't fix"). None are logic/data/scope errors — all are status-field drift. The authoritative state lives in the git log + tags + REVIEW-v0.5.md + VERIFY-P1/P2-v0.5.md, all of which are consistent.
|
||||
|
||||
**File discipline verdict: NEEDS_ATTENTION** (5 files with stale-status fields; all 13 expected files present + current content; no contradictions; v0.4 reference files retained).
|
||||
|
||||
---
|
||||
|
||||
## 4. Check 3 — Branch Hygiene
|
||||
|
||||
### 4.1 Branch hierarchy
|
||||
|
||||
```
|
||||
main (ba928cf — v0.4 merged)
|
||||
└─ milestone/v0.5-live-assist (b621cb6 — P2 ship, == base of phase/03)
|
||||
└─ phase/03-final-review-ship (2627923 — P3 verify, CURRENT, 2 commits ahead)
|
||||
```
|
||||
|
||||
- `main` → `milestone/v0.5-live-assist` → `phase/03-final-review-ship`: hierarchy correct ✅
|
||||
- `git merge-base --is-ancestor main milestone/v0.5-live-assist` → ✅
|
||||
- `git merge-base --is-ancestor milestone/v0.5-live-assist phase/03-final-review-ship` → ✅
|
||||
- `milestone/v0.5-live-assist` exists, points to P2 ship commit `b621cb6` (latest P2 ship). ✅
|
||||
- `phase/03-final-review-ship` is the current branch (marked `*` in `git branch -vv`), 2 commits ahead of milestone (P0-fix `5373df2` + verify `2627923`), not yet merged. ✅
|
||||
|
||||
### 4.2 v0.5 phase branches created + deleted post-merge
|
||||
|
||||
The v0.5 milestone used phase branches during execution (confirmed via reflog):
|
||||
- `phase/00-pre-execution` (tip `3649344` per reflog) — worked on, merged to milestone, deleted
|
||||
- `phase/01-assist-core-guardrail` (tip `fb26d33` per reflog) — worked on, merged to milestone, deleted
|
||||
- `phase/02-integration-techdebt-nfr` (tip `38b97ee` per reflog) — worked on, merged to milestone, deleted
|
||||
|
||||
`git branch -a` confirms none of these exist locally or on remote — they were deleted post-merge. ✅ This matches the audit prompt's expectation ("Old phase branches phase/00, phase/01, phase/02 are DELETED").
|
||||
|
||||
**Merge pattern note:** v0.5 used a **linear direct-commit** pattern (phase work committed as `feat(P01)`/`feat(P02)`/`docs(P00)` directly to `milestone/v0.5-live-assist`, single-parent) rather than v0.4's `feat(milestone): merge phase/NN` squash-merge pattern. The reflog shows the phase branches existed during execution, so the work was done on phase branches then merged (likely fast-forward or squash-then-delete). The result is a linear milestone history with `feat(PNN)` commits. This is valid + traceable (the `---ci---` blocks + commit prefixes preserve phase boundaries). Non-blocking — the v0.4 squash-merge pattern with retained phase branches is preferred for explicit phase-boundary integrity, but the v0.5 pattern is acceptable.
|
||||
|
||||
### 4.3 Old phase branches from prior milestones (informational, non-blocking)
|
||||
|
||||
`git branch -a` shows retained phase branches from v0.2/v0.3/v0.4:
|
||||
- `phase/01-lxc-deploy` (v0.2)
|
||||
- `phase/01-mastery-core` (v0.3)
|
||||
- `phase/01-operator-foundation` (v0.4)
|
||||
- `phase/02-cohort-dashboard` (v0.4)
|
||||
- `phase/02-final-review-ship` (v0.3 — shared name, points to v0.3 tip `056ce01`)
|
||||
- `remotes/origin/phase/01-minimal-voice-loop` (v0.1)
|
||||
|
||||
These are retained per the housekeeping pattern (branches kept for traceability across milestones). Not v0.5-stale. The v0.4 audit (§C.4) noted the same retention + recommended optional cleanup post-merge-to-main. Non-blocking.
|
||||
|
||||
### 4.4 No stale/dangling branches for v0.5
|
||||
|
||||
`git branch -vv` shows no orphaned v0.5 phase branches (they were deleted post-merge per §4.2). ✅
|
||||
|
||||
**Branch hygiene verdict: PASS.**
|
||||
|
||||
---
|
||||
|
||||
## 5. Check 4 — Commit Discipline
|
||||
|
||||
### 5.1 Every phase has a ship commit with `---ci---` block
|
||||
|
||||
| Phase | Ship commit | `---ci---` | Tag |
|
||||
|-------|-------------|-----------|-----|
|
||||
| P0 | `fb26d33` docs(ship): phase 0 complete | ✅ phase:0, milestone:v0.5, status:complete | v0.1.10 |
|
||||
| P1 | `38b97ee` docs(ship): P1 complete | ✅ phase:1, milestone:v0.5, status:complete | v0.1.11 |
|
||||
| P2 | `b621cb6` docs(ship): P2 complete | ✅ phase:2, milestone:v0.5, status:complete | v0.1.12 |
|
||||
|
||||
✅
|
||||
|
||||
### 5.2 All 9 commits have `---ci---` blocks with required fields
|
||||
|
||||
`git log v0.1.9..HEAD --pretty=%B | grep -c "^---ci---"` = 9 = number of commits `v0.1.9..HEAD`. No missing blocks. ✅
|
||||
|
||||
Field distribution (`git log v0.1.9..HEAD --pretty=%B | grep -E "^project:|^phase:|^milestone:|^status:" | sort | uniq -c`):
|
||||
- `project: praxis` × 9 (all commits) ✅
|
||||
- `milestone: v0.5` × 8 + `milestone: v0.4` × 1 (the v0.4 completion base commit `ba928cf`) ✅
|
||||
- `phase: 0` × 2, `phase: 1` × 2, `phase: 2` × 2, `phase: 3` × 3 (P0-fix + verify + ... ) ✅
|
||||
- `status: complete` × 7, `status: verify` × 2 (P3 commits) ✅
|
||||
|
||||
The P1/P2 merge commits (`81d4366`, `bdcf793`) carry full `---ci---` blocks with `requirements.covered` + `requirements.partial: []`. The ship commits carry `project/phase/milestone/status`. The P3 commits (`5373df2`, `2627923`) carry `phase_role: final_review` + `verdict`/`lessons`. ✅
|
||||
|
||||
### 5.3 Conventional-commit format
|
||||
|
||||
All 9 commits use conventional prefixes:
|
||||
- `docs(milestone):` — v0.4 completion (`ba928cf`) ✅
|
||||
- `docs(P00):` / `docs(ship):` — P0 merge + ship (`5290d4d`, `fb26d33`) ✅
|
||||
- `feat(P01):` / `docs(ship):` — P1 merge + ship (`81d4366`, `38b97ee`) ✅
|
||||
- `feat(P02):` / `docs(ship):` — P2 merge + ship (`bdcf793`, `b621cb6`) ✅
|
||||
- `fix(P03):` / `verify(P03):` — P3 P0-fix + verify (`5373df2`, `2627923`) ✅
|
||||
|
||||
Consistent with the v0.2/v0.3/v0.4 style (with the v0.5-specific `PNN` scope instead of `milestone`). ✅
|
||||
|
||||
### 5.4 No secrets in commit messages
|
||||
|
||||
`git log v0.1.9..HEAD --pretty=%B | grep -iE "GITEA_TOKEN|password|OLLAMA_API_KEY|DEEPGRAM_API_KEY|CARTESIA_API_KEY|PRAXIS_PG_PASSWORD|PRAXIS_COOKIE_SECRET"` returned one hit: `cookie-secret` in the P2 merge commit body — this is a **false positive** (it describes the tech-debt fix "cookie-secret length validation", not a secret value). No actual secret values (tokens, passwords, keys) found in any commit message. ✅
|
||||
|
||||
### 5.5 Tag sequence
|
||||
|
||||
v0.1.9 (main, v0.4) < v0.1.10 (P0) < v0.1.11 (P1) < v0.1.12 (P2) < v0.1.13 (next, not yet created = v0.5 milestone release). Strictly increasing, no skips. All annotated. ✅
|
||||
|
||||
**Commit discipline verdict: PASS.**
|
||||
|
||||
---
|
||||
|
||||
## 6. Issues Found
|
||||
|
||||
### 6.1 Critical issues
|
||||
|
||||
**None.** No reconstruction mismatch, no missing files, no broken branch hierarchy, no missing REQ coverage, no unaddressed grill MUSTs, no secrets in commits.
|
||||
|
||||
### 6.2 Important issues (stale-status drift — not auto-fixed)
|
||||
|
||||
12 stale-status fields across 5 files (PROJECT.md, ROADMAP.md, REQUIREMENTS.md, CHECKPOINT.json, config.json) — all set during P0 SPECIFY, never advanced as P1/P2 shipped. Same drift class as the v0.4 P3 audit. See §3.4 for the full table. **Severity: important** (stale docs, but the authoritative state in git log + tags + REVIEW/VERIFY is correct + consistent).
|
||||
|
||||
### 6.3 Nits
|
||||
|
||||
- `config.json:7` `"status": "phase-0-active"` — informational field, CHECKPOINT.json is the source of truth. Severity: nit.
|
||||
- v0.5 used a linear direct-commit merge pattern (phase branches deleted post-merge) rather than v0.4's squash-merge-with-retained-branches pattern. Both are valid; the v0.4 pattern is preferred for explicit phase-boundary integrity. Severity: nit (process variation, non-blocking).
|
||||
|
||||
### 6.4 Non-issues (verified clean)
|
||||
|
||||
- All 16 v0.5 REQ-IDs covered in commit blocks + REVIEW-v0.5.md + VERIFY-P1/P2-v0.5.md + CHECKPOINT.json. ✅
|
||||
- Both grill MUSTs (G-049, G-067) resolved + documented in REVIEW-v0.5.md + CHECKPOINT.json. ✅
|
||||
- ESCALATION-01 (PIPEDA) documented as OPEN for human legal review — correctly escalated, not a CI-resolvable issue. ✅
|
||||
- 8 v0.4 P1+ tech-debt findings all addressed in P2 SLICE-12 (REVIEW-v0.5.md §"8 v0.4 P1+ Tech-Debt Wave"). ✅
|
||||
- 1 P0 fix applied during P3 (guardrail processor buffers LLM text before TTS — REQ-ASSIST-03 safety-critical). ✅
|
||||
- 469 tests pass, 45 skipped (all env-gated), 0 failed (REVIEW-v0.5.md). ✅
|
||||
|
||||
---
|
||||
|
||||
## 7. Recommendations
|
||||
|
||||
Non-blocking, for the ship orchestrator (post-audit):
|
||||
|
||||
1. **Advance the 12 stale-status fields** before/during ship (the same fix the v0.4 P3 audit applied):
|
||||
- PROJECT.md:4 → `phase 3 — final review; P0-P2 complete (v0.1.10/v0.1.11/v0.1.12 tagged)`
|
||||
- ROADMAP.md:3-4, 15, 19 → `phase 3 final review` + Phase 0 `complete — tagged v0.1.10`
|
||||
- REQUIREMENTS.md:3-4 → `phase 3 final review; 16/16 v0.5 REQ covered`; lines 14-70 → all 16 v0.5 REQs `complete`
|
||||
- CHECKPOINT.json → `phase: 3, stage: in_progress, phase_role: final_review` (tag remains v0.1.12, requirements/grill/test fields unchanged)
|
||||
- config.json:7 → `"status": "phase-3-final-review"` (optional — informational)
|
||||
2. **Ship**: tag `v0.1.13` (= v0.5 milestone release), merge `milestone/v0.5-live-assist` → `main`, create Gitea release. The audit found no blockers; the orchestrator delegates to ship after this audit.
|
||||
3. **On ship**: advance CHECKPOINT.json to `phase: 3, stage: complete, milestone_complete: true, milestone_merged_to_main: true, tag: v0.1.13` (the audit recommends setting it to `in_progress` now; ship advances it to `complete`).
|
||||
4. **Carry-forward the 8 P1+ items** (from REVIEW-v0.5.md §P1+ Flagged) to the v0.6 backlog: (1) PII retention cleanup not scheduled, (2) scenario-tag prompt injection unsanitized, (3) end_session_assist doesn't persist turn/block counts, (4) WebRTC reconnect offer-event not wired, (5) no concurrent shift-start race test, (6) cache I/O on every session-end hook, (7) nightly_trend bypasses PraxisStore API, (8) nightly_trend fn_candidates include truncated tts_text. All non-blocking with mitigations present.
|
||||
5. **ESCALATION-01 (PIPEDA)** remains OPEN for human legal review before the assist surface goes live. The engineering mitigations (consent disclosure D-070 + PII redaction REQ-IDEATE-05 + local SQLite + 30-day retention) are implemented regardless. This is a post-ship human action item, not a CI-resolvable issue.
|
||||
6. **Branch cleanup (optional, post-merge-to-main)**: the prior-milestone phase branches (`phase/01-lxc-deploy`, `phase/01-mastery-core`, `phase/02-final-review-ship` from v0.3, `phase/01-operator-foundation`, `phase/02-cohort-dashboard` from v0.4) are retained per housekeeping pattern; consider deleting after v0.5 merges to main if a cleanup pass is desired. Not blocking.
|
||||
7. **For v0.6**: consider restoring the v0.4 squash-merge pattern (retained phase branches + `feat(milestone): merge phase/NN` commits) for explicit phase-boundary integrity. The v0.5 linear direct-commit pattern is valid but loses the explicit merge-commit phase boundaries.
|
||||
|
||||
---
|
||||
|
||||
## 8. Final Verdict
|
||||
|
||||
# ⚠️ NEEDS_ATTENTION
|
||||
|
||||
The v0.5 milestone (Live Assist — On-the-Job Voice Companion) is **healthy and ready for milestone ship (v0.1.13 = v0.5)** after the ship orchestrator advances 12 stale-status fields:
|
||||
|
||||
- **Reconstruction (PASS):** git log (9 commits v0.1.9..HEAD covers P0/P1/P2/P3) matches ROADMAP phase progression; `---ci---` blocks match declared phase/milestone; tags v0.1.10/v0.1.11/v0.1.12 annotated + point to correct commits; all 16 v0.5 REQ-IDs covered in commit blocks (12 in P1 + 4 in P2 = 16, no overlap/gaps).
|
||||
- **File discipline (NEEDS_ATTENTION):** all 13 expected `.ciagent/` files present + current content; v0.4/v0.3/v0.2/v0.1 reference files retained; internally consistent (16 REQs across PROJECT/REQUIREMENTS/CHECKPOINT/ROADMAP/REVIEW); **12 stale-status fields** across 5 files (PROJECT/ROADMAP/REQUIREMENTS/CHECKPOINT/config.json) — same drift class as v0.4 P3 audit; NOT auto-fixed (audit surfaces, does not fix).
|
||||
- **Branch hygiene (PASS):** main → milestone/v0.5 → phase/03 hierarchy correct; v0.5 phase branches created + deleted post-merge (confirmed via reflog); old v0.2/v0.3/v0.4 phase branches retained (housekeeping pattern).
|
||||
- **Commit discipline (PASS):** all 9 commits have `---ci---` blocks; correct `project: praxis` + `milestone: v0.5`; conventional-commit prefixes; no secrets in commit messages; tag sequence strictly increasing.
|
||||
|
||||
**No critical issues.** The 12 stale-status fields are documentation drift (status fields set during P0 SPECIFY, never advanced), not logic/data/scope errors. The authoritative state lives in the git log + tags + REVIEW-v0.5.md + VERIFY-P1/P2-v0.5.md + CHECKPOINT.json's non-status fields, all of which are consistent + correct.
|
||||
|
||||
The v0.5 implementation is independently verified by:
|
||||
- **REVIEW-v0.5.md** (P3 multi-persona code review): APPROVE_WITH_NOTES, 5/5 personas PASS, 1 P0 fix applied (guardrail processor safety-critical), 8 P1+ flagged (all non-blocking carry-forward to v0.6)
|
||||
- **VERIFY-P1-v0.5.md**: APPROVE_WITH_NOTES, 12/12 REQ, 5 P1+
|
||||
- **VERIFY-P2-v0.5.md**: APPROVE_WITH_NOTES, 4/4 REQ, 3 P1+
|
||||
- **GRILL-v0.5.md**: 39 decisions, 2 MUSTs (G-049, G-067) resolved, 1 escalation (ESCALATION-01 PIPEDA — OPEN for human legal review)
|
||||
- **Tests**: 469 pytest pass / 45 skip / 0 fail; npm build succeeds
|
||||
|
||||
The orchestrator delegates to ship after this audit. Do NOT ship from this audit. Advance the 12 stale-status fields first (recommendation #1).
|
||||
|
||||
---
|
||||
|
||||
---ci---
|
||||
project: praxis
|
||||
phase: 3
|
||||
milestone: v0.5
|
||||
status: audit
|
||||
phase_role: final_review
|
||||
verdict: NEEDS_ATTENTION
|
||||
checks:
|
||||
reconstruction: PASS
|
||||
file_discipline: NEEDS_ATTENTION
|
||||
branch_hygiene: PASS
|
||||
commit_discipline: PASS
|
||||
requirements_coverage: 16/16
|
||||
grill_musts_honored: 2/2
|
||||
stale_status_fields: 12
|
||||
auto_fixes: none
|
||||
critical_issues: none
|
||||
recommendations:
|
||||
- advance 12 stale-status fields (PROJECT/ROADMAP/REQUIREMENTS/CHECKPOINT/config.json) before/during ship
|
||||
- ship: tag v0.1.13, merge milestone/v0.5 → main, create release
|
||||
- on ship: advance CHECKPOINT to phase 3 complete + milestone_complete true
|
||||
- carry-forward 8 P1+ items to v0.6 backlog
|
||||
- ESCALATION-01 PIPEDA remains OPEN for human legal review before assist go-live
|
||||
- optional branch cleanup post-merge
|
||||
- consider restoring v0.4 squash-merge pattern for v0.6
|
||||
---/ci---
|
||||
+18
-18
@@ -1,24 +1,24 @@
|
||||
{
|
||||
"phase": 3,
|
||||
"phase": 0,
|
||||
"stage": "complete",
|
||||
"milestone": "v0.5",
|
||||
"phase_role": "final",
|
||||
"phase_role": "pre_execution",
|
||||
"attempts": 0,
|
||||
"updated_at": "2026-08-04T13:40:00Z",
|
||||
"milestone_complete": true,
|
||||
"milestone_merged_to_main": true,
|
||||
"next_milestone": "v0.6",
|
||||
"tag": "v0.1.13",
|
||||
"release_url": "https://git.cloudinit.dev/coreci/praxis/releases/tag/v0.1.13",
|
||||
"release_status": "created",
|
||||
"requirements": {
|
||||
"covered": ["REQ-ASSIST-01", "REQ-ASSIST-02", "REQ-ASSIST-03", "REQ-NFR-ASSIST-01", "REQ-NFR-ASSIST-02", "REQ-NFR-ASSIST-03", "REQ-NFR-ASSIST-04", "REQ-IDEATE-01", "REQ-IDEATE-02", "REQ-IDEATE-03", "REQ-IDEATE-04", "REQ-IDEATE-05", "REQ-IDEATE-06", "REQ-IDEATE-07", "REQ-IDEATE-08", "REQ-IDEATE-09"],
|
||||
"deferred": ["REQ-IDEATE-10", "REQ-IDEATE-11", "REQ-IDEATE-12", "REQ-IDEATE-13"]
|
||||
},
|
||||
"updated_at": "2026-08-04T12:40:00Z",
|
||||
"milestone_complete": false,
|
||||
"milestone_merged_to_main": false,
|
||||
"next_milestone": "v0.5",
|
||||
"active_requirements": ["REQ-ASSIST-01", "REQ-ASSIST-02", "REQ-ASSIST-03", "REQ-NFR-ASSIST-01", "REQ-NFR-ASSIST-02", "REQ-NFR-ASSIST-03", "REQ-NFR-ASSIST-04", "REQ-IDEATE-01", "REQ-IDEATE-02", "REQ-IDEATE-03", "REQ-IDEATE-04", "REQ-IDEATE-05", "REQ-IDEATE-06", "REQ-IDEATE-07", "REQ-IDEATE-08", "REQ-IDEATE-09"],
|
||||
"v0.6_backlog": ["REQ-IDEATE-10", "REQ-IDEATE-11", "REQ-IDEATE-12", "REQ-IDEATE-13"],
|
||||
"review_verdict": "APPROVE_WITH_NOTES",
|
||||
"audit_verdict": "NEEDS_ATTENTION",
|
||||
"p0_fixes": 1,
|
||||
"p1_plus_flagged": 8,
|
||||
"escalations": ["ESCALATION-01"]
|
||||
"tag_base": "v0.1.x",
|
||||
"tag": "v0.1.10",
|
||||
"next_tag": "v0.1.11",
|
||||
"release_url": "https://git.cloudinit.dev/coreci/praxis/releases/tag/v0.1.10",
|
||||
"release_status": "created",
|
||||
"ideate": true,
|
||||
"ideate_result": {"total": 13, "accepted_v0.5": 9, "accepted_v0.6": 4, "skipped": 0},
|
||||
"grill_verdict": "proceed_with_conditions",
|
||||
"grill_confidence": 0.70,
|
||||
"grill_musts": ["G-049", "G-067"],
|
||||
"grill_escalations": ["ESCALATION-01"]
|
||||
}
|
||||
+2
-2
@@ -1,7 +1,7 @@
|
||||
# Praxis — Voice-first AI Apprenticeship Platform
|
||||
|
||||
**Milestone:** v0.5 (Live Assist — on-the-job voice companion) — complete
|
||||
**Status:** milestone released as v0.1.13 (merged to main) — 16/16 v0.5 REQ covered; v0.4 complete (v0.1.9); v0.3 complete (v0.1.5)
|
||||
**Milestone:** v0.5 (Live Assist — on-the-job voice companion)
|
||||
**Status:** phase 0 — pre-execution (active milestone)
|
||||
**Autonomy:** full
|
||||
**Previous milestone:** v0.4 (Operator tier — cohort dashboard, auth, Postgres) — complete, tagged v0.1.9, release created, merged to main
|
||||
|
||||
|
||||
+21
-21
@@ -1,28 +1,28 @@
|
||||
# Praxis — Requirements
|
||||
|
||||
**Milestone:** v0.5 (Live Assist — on-the-job voice companion) — complete
|
||||
**Status:** milestone released as v0.1.13 (merged to main) — 16/16 v0.5 REQ covered; v0.4 complete (released as v0.1.9, 8/8 REQ covered); v0.3 complete (released as v0.1.5, 13/13 REQ covered)
|
||||
**Milestone:** v0.5 (Live Assist — on-the-job voice companion) — active, phase 0
|
||||
**Status:** phase 0 pre-execution — v0.4 complete (released as v0.1.9, merged to main, 8/8 v0.4 REQ covered); v0.3 complete (released as v0.1.5, 13/13 v0.3 REQ covered)
|
||||
|
||||
Formal requirements with REQ-IDs. Scoped to the active milestone unless noted. v0.1/v0.2/v0.3/v0.4/v0.5 requirements (complete) are retained for reference with their final status. Later-milestone requirements are marked `deferred`.
|
||||
Formal requirements with REQ-IDs. Scoped to the active milestone unless noted. v0.1/v0.2/v0.3/v0.4 requirements (complete) are retained for reference with their final status. Later-milestone requirements are marked `deferred`.
|
||||
|
||||
## v0.5 Active Requirements (complete — released as v0.1.13, retained for reference)
|
||||
## v0.5 Active Requirements
|
||||
|
||||
### Live Assist (v0.5 core)
|
||||
|
||||
| REQ-ID | Requirement | Priority | Phase | Status |
|
||||
|--------|-------------|----------|-------|--------|
|
||||
| REQ-ASSIST-01 | Hands-free voice companion invocable while working — distinct from the practice voice loop (v0.1). Always-listening or wake-word/hotkey-activated, short coaching turns interleaved with real work. Reuses the v0.1 voice pipeline (Pipecat + Deepgram + Cartesia + Ollama Cloud) in a new "assist" mode. | must | P1 | complete |
|
||||
| REQ-ASSIST-02 | Context-aware — knows the learner's current scenario/skill path. Binds to the learner's active path week (D-037) + scenario context so coaching is relevant to the job they're doing. Carries forward learner state from SQLite (D-007 preserved). | must | P1 | complete |
|
||||
| REQ-ASSIST-03 | Guardrails: coaches, does not do the job; never lies to real customers. Safety-critical: the AI is in the learner's ear during real customer interactions. Extends D-019 guardrail layer with Live-Assist-specific ruleset. Never impersonates, never gives parrot-able answers, never claims false authority. | must | P1 | complete |
|
||||
| REQ-ASSIST-01 | Hands-free voice companion invocable while working — distinct from the practice voice loop (v0.1). Always-listening or wake-word/hotkey-activated, short coaching turns interleaved with real work. Reuses the v0.1 voice pipeline (Pipecat + Deepgram + Cartesia + Ollama Cloud) in a new "assist" mode. | must | P1 | active |
|
||||
| REQ-ASSIST-02 | Context-aware — knows the learner's current scenario/skill path. Binds to the learner's active path week (D-037) + scenario context so coaching is relevant to the job they're doing. Carries forward learner state from SQLite (D-007 preserved). | must | P1 | active |
|
||||
| REQ-ASSIST-03 | Guardrails: coaches, does not do the job; never lies to real customers. Safety-critical: the AI is in the learner's ear during real customer interactions. Extends D-019 guardrail layer with Live-Assist-specific ruleset. Never impersonates, never gives parrot-able answers, never claims false authority. | must | P1 | active |
|
||||
|
||||
## v0.5 Non-Functional Requirements (complete)
|
||||
## v0.5 Non-Functional Requirements
|
||||
|
||||
| REQ-ID | Requirement | Target | Phase | Status |
|
||||
|--------|-------------|--------|-------|--------|
|
||||
| REQ-NFR-ASSIST-01 | Live Assist voice round-trip latency | **< 600ms target (C-8); estimated ~655ms (Piper + lean prompt — D-065, D-066). AT RISK — accept ~650ms for pilot if trending down; <600ms hardening in v0.6.** Wake-word → first-audio is a separate ~850-1150ms budget (warm WebRTC — D-067). Must not degrade the practice pipeline (assist is a separate mode, not concurrent — D-061). | P1 | complete (p95 measurement in P2) |
|
||||
| REQ-NFR-ASSIST-02 | Hands-free invocation on $100 Android | **Picovoice Porcupine on-device (offline, ~1MB RAM, <4% core — verified). Battery ~4-9% per 8h shift (estimated, needs Phase-1 measurement — R-ASSIST-14). Foreground service of type `microphone` (Android 14+). Built-in wake word for v0.5 pilot (D-064 — MAU pricing has no recurring free tier, R-ASSIST-01); custom "Hey Praxis" post-pilot; Vosk fallback. Tap-to-talk fallback for battery-saving / wake-word failure / noisy environments.** | P1 | complete (tap-to-talk only per D-071; wake-word deferred to v0.6) |
|
||||
| REQ-NFR-ASSIST-03 | Live Assist guardrail enforcement | **3-layer guardrail (D-060, D-068): (1) coaching-mode system prompt (ask guiding questions, never give the answer, never claim false authority, never impersonate); (2) regex output filter (DIRECT_SCRIPT_RE + IMPERATIVE_RE + FALSE_AUTHORITY_RE + IMPERSONATION_RE; COACHING_QUESTION_RE allowed) with one retry on block + canned coaching fallback; (3) audit log (turns table guardrail_verdict JSON + cohort guardrail_block_rate safety signal for operators). Consent disclosure: foreground-service notification + learner-facing "Assist is on — those around you may be recorded" at shift start (D-070). Output filter false-negative residual risk mitigated by defense-in-depth + post-v0.5 LLM-as-judge.** | P1 | complete (FN 13.3% ≤ 20% pilot threshold per G-067) |
|
||||
| REQ-NFR-ASSIST-04 | Live Assist session model | **Shift-bounded (learner starts/ends a shift; assist turns within — D-062). Auto-end after 8h via `PRAXIS_ASSIST_MAX_SHIFT_HOURS=8` (D-069). Aggregates as `session_type=assist` in v0.4 cohort pipeline (no schema change — new metric strings: assist_shifts_count, assist_turns_count, assist_avg_turns_per_shift, assist_active_learners_count, assist_guardrail_block_rate). Does NOT update mastery (D-063 — `schedule_mastery=False` for assist shifts). k-anonymity ≥ 10 applies to assist metrics (D-034 carry-forward).** | P1 | complete |
|
||||
| REQ-NFR-ASSIST-01 | Live Assist voice round-trip latency | **< 600ms target (C-8); estimated ~655ms (Piper + lean prompt — D-065, D-066). AT RISK — accept ~650ms for pilot if trending down; <600ms hardening in v0.6.** Wake-word → first-audio is a separate ~850-1150ms budget (warm WebRTC — D-067). Must not degrade the practice pipeline (assist is a separate mode, not concurrent — D-061). | P1 | research-grounded (R-ASSIST-02) |
|
||||
| REQ-NFR-ASSIST-02 | Hands-free invocation on $100 Android | **Picovoice Porcupine on-device (offline, ~1MB RAM, <4% core — verified). Battery ~4-9% per 8h shift (estimated, needs Phase-1 measurement — R-ASSIST-14). Foreground service of type `microphone` (Android 14+). Built-in wake word for v0.5 pilot (D-064 — MAU pricing has no recurring free tier, R-ASSIST-01); custom "Hey Praxis" post-pilot; Vosk fallback. Tap-to-talk fallback for battery-saving / wake-word failure / noisy environments.** | P1 | research-grounded (R-ASSIST-01/04/05/13/14) |
|
||||
| REQ-NFR-ASSIST-03 | Live Assist guardrail enforcement | **3-layer guardrail (D-060, D-068): (1) coaching-mode system prompt (ask guiding questions, never give the answer, never claim false authority, never impersonate); (2) regex output filter (DIRECT_SCRIPT_RE + IMPERATIVE_RE + FALSE_AUTHORITY_RE + IMPERSONATION_RE; COACHING_QUESTION_RE allowed) with one retry on block + canned coaching fallback; (3) audit log (turns table guardrail_verdict JSON + cohort guardrail_block_rate safety signal for operators). Consent disclosure: foreground-service notification + learner-facing "Assist is on — those around you may be recorded" at shift start (D-070). Output filter false-negative residual risk mitigated by defense-in-depth + post-v0.5 LLM-as-judge.** | P1 | research-grounded (R-ASSIST-06/07/08) |
|
||||
| REQ-NFR-ASSIST-04 | Live Assist session model | **Shift-bounded (learner starts/ends a shift; assist turns within — D-062). Auto-end after 8h via `PRAXIS_ASSIST_MAX_SHIFT_HOURS=8` (D-069). Aggregates as `session_type=assist` in v0.4 cohort pipeline (no schema change — new metric strings: assist_shifts_count, assist_turns_count, assist_avg_turns_per_shift, assist_active_learners_count, assist_guardrail_block_rate). Does NOT update mastery (D-063 — `schedule_mastery=False` for assist shifts). k-anonymity ≥ 10 applies to assist metrics (D-034 carry-forward).** | P1 | research-grounded |
|
||||
|
||||
_NFRs refined from `pending-research` to `research-grounded` after the v0.5 RESEARCH stage (see RESEARCH-v0.5-live-assist.md). Targets are research-derived; Phase-1 measurement may further refine R-ASSIST-02 (latency) and R-ASSIST-14 (battery)._
|
||||
|
||||
@@ -34,40 +34,40 @@ _Generated by the IDEATE stage (3-tier analysis: mechanical git-mining + backend
|
||||
|
||||
| REQ-ID | Requirement | Priority | Phase | Status |
|
||||
|--------|-------------|----------|-------|--------|
|
||||
| REQ-IDEATE-01 | Guardrail output-filter tuning corpus + adversarial bypass test (pre-ship). Build a synthetic corpus (LLM-generate coaching vs direct-answer responses, label, tune the regex patterns DIRECT_SCRIPT_RE/IMPERATIVE_RE/FALSE_AUTHORITY_RE/IMPERSONATION_RE). Add an adversarial-bypass test with paraphrased direct answers designed to slip past the regex. Proactively mitigates R-ASSIST-06/07 (false-positive + false-negative risks) before the guardrail ships blind on its two most safety-critical metrics. Relates to the v0.1 latent safety-trap lesson (misspelled `_DEBRIFF_LEGAL_REDIRECT` — the rewrite/fallback path was never exercised by tests). | must | P1 | complete |
|
||||
| REQ-IDEATE-02 | In-loop guardrail processor pipeline test + GuardrailContext.role 'assist' extension. (1) Add a pipeline-integration test that inserts the LiveAssistGuardrail as a post-LLM Pipecat frame processor between llm and tts (the existing test_guardrail.py only tests `check()` standalone). (2) Extend the `GuardrailContext.role` Literal to include `'assist'` (currently `system|user|assistant|debrief` — the LiveAssistGuardrail hits an interface gap). Both are structural coverage holes Phase 1 will hit immediately. | must | P1 | complete |
|
||||
| REQ-IDEATE-09 | Audit-log completeness on abrupt shift end. Log the assist turn incrementally — persist the ASR transcript + LLM response + guardrail verdict before/at TTS start, not after playback completes — so abrupt termination (battery death R-ASSIST-14, power loss mid-turn) still leaves an audit trail. For a safety-critical surface (REQ-ASSIST-03), an incomplete audit log undermines the guardrail_block_rate safety signal and the operator's ability to investigate incidents. | must | P1 | complete |
|
||||
| REQ-IDEATE-01 | Guardrail output-filter tuning corpus + adversarial bypass test (pre-ship). Build a synthetic corpus (LLM-generate coaching vs direct-answer responses, label, tune the regex patterns DIRECT_SCRIPT_RE/IMPERATIVE_RE/FALSE_AUTHORITY_RE/IMPERSONATION_RE). Add an adversarial-bypass test with paraphrased direct answers designed to slip past the regex. Proactively mitigates R-ASSIST-06/07 (false-positive + false-negative risks) before the guardrail ships blind on its two most safety-critical metrics. Relates to the v0.1 latent safety-trap lesson (misspelled `_DEBRIFF_LEGAL_REDIRECT` — the rewrite/fallback path was never exercised by tests). | must | P1 | active |
|
||||
| REQ-IDEATE-02 | In-loop guardrail processor pipeline test + GuardrailContext.role 'assist' extension. (1) Add a pipeline-integration test that inserts the LiveAssistGuardrail as a post-LLM Pipecat frame processor between llm and tts (the existing test_guardrail.py only tests `check()` standalone). (2) Extend the `GuardrailContext.role` Literal to include `'assist'` (currently `system|user|assistant|debrief` — the LiveAssistGuardrail hits an interface gap). Both are structural coverage holes Phase 1 will hit immediately. | must | P1 | active |
|
||||
| REQ-IDEATE-09 | Audit-log completeness on abrupt shift end. Log the assist turn incrementally — persist the ASR transcript + LLM response + guardrail verdict before/at TTS start, not after playback completes — so abrupt termination (battery death R-ASSIST-14, power loss mid-turn) still leaves an audit trail. For a safety-critical surface (REQ-ASSIST-03), an incomplete audit log undermines the guardrail_block_rate safety signal and the operator's ability to investigate incidents. | must | P1 | active |
|
||||
|
||||
### Chaos & Resilience (IDEATE-03, 08)
|
||||
|
||||
| REQ-ID | Requirement | Priority | Phase | Status |
|
||||
|--------|-------------|----------|-------|--------|
|
||||
| REQ-IDEATE-03 | Mode-conflict enforcement: assist vs practice mutual exclusivity. Add a server-side guard (reject shift-start if a practice session is active, or vice versa) + a chaos test invoking assist during an active practice session. D-061 states assist is a separate mode (not concurrent), but nothing currently enforces mutual exclusivity — the server-side assist API and the practice /pipecat/webrtc endpoint are independent with no shared state guarding against a second connection. | must | P1 | complete |
|
||||
| REQ-IDEATE-08 | WebRTC mid-shift drop + reconnect logic. Specify the reconnect state machine (does the foreground service auto-reconnect? what does the learner experience during the gap? does the in-flight assist turn retry or fail?) + add a chaos test (kill the WebRTC connection mid-shift, verify reconnect + turn recovery). R-ASSIST-09 names the risk; D-067 mandates warm WebRTC with 30s heartbeat but the reconnect logic is unspecified. | must | P1 | complete |
|
||||
| REQ-IDEATE-03 | Mode-conflict enforcement: assist vs practice mutual exclusivity. Add a server-side guard (reject shift-start if a practice session is active, or vice versa) + a chaos test invoking assist during an active practice session. D-061 states assist is a separate mode (not concurrent), but nothing currently enforces mutual exclusivity — the server-side assist API and the practice /pipecat/webrtc endpoint are independent with no shared state guarding against a second connection. | must | P1 | active |
|
||||
| REQ-IDEATE-08 | WebRTC mid-shift drop + reconnect logic. Specify the reconnect state machine (does the foreground service auto-reconnect? what does the learner experience during the gap? does the in-flight assist turn retry or fail?) + add a chaos test (kill the WebRTC connection mid-shift, verify reconnect + turn recovery). R-ASSIST-09 names the risk; D-067 mandates warm WebRTC with 30s heartbeat but the reconnect logic is unspecified. | must | P1 | active |
|
||||
|
||||
### Security & Privacy (IDEATE-05)
|
||||
|
||||
| REQ-ID | Requirement | Priority | Phase | Status |
|
||||
|--------|-------------|----------|-------|--------|
|
||||
| REQ-IDEATE-05 | Customer-speech PII handling in the assist turns audit log (STRIDE information-disclosure). The ambient mic (R-ASSIST-08) captures BOTH the learner and the real customer; ASR transcribes both; the turns table stores transcribed text. The customer is a third party — their transcribed speech is third-party PII in SQLite. v0.5 needs an explicit policy: (a) strip customer turns from the audit log, (b) store only the learner's utterances, or (c) document that the audit log contains customer speech + apply consent-disclosure (D-070) + retention limits. Intersects with the R-ASSIST-08 legal review (D-073). | must | P1 | complete |
|
||||
| REQ-IDEATE-05 | Customer-speech PII handling in the assist turns audit log (STRIDE information-disclosure). The ambient mic (R-ASSIST-08) captures BOTH the learner and the real customer; ASR transcribes both; the turns table stores transcribed text. The customer is a third party — their transcribed speech is third-party PII in SQLite. v0.5 needs an explicit policy: (a) strip customer turns from the audit log, (b) store only the learner's utterances, or (c) document that the audit log contains customer speech + apply consent-disclosure (D-070) + retention limits. Intersects with the R-ASSIST-08 legal review (D-073). | must | P1 | active |
|
||||
|
||||
### Spec Refinement (IDEATE-04)
|
||||
|
||||
| REQ-ID | Requirement | Priority | Phase | Status |
|
||||
|--------|-------------|----------|-------|--------|
|
||||
| REQ-IDEATE-04 | Measurable NFR targets for REQ-NFR-ASSIST-01 and REQ-NFR-ASSIST-03. (1) Latency: specify 'p95 assist-turn latency ≤ 650ms in Phase-1 measurement (pilot tolerance per D-072); <600ms hardening deferred to v0.6' — resolves the ambiguity in REQ-NFR-ASSIST-01's current text. (2) Guardrail: specify 'false-positive rate < 5% on the tuning corpus (REQ-IDEATE-01); false-negative rate measured + trended nightly' — makes REQ-NFR-ASSIST-03 verifiable. | must | P1 | complete |
|
||||
| REQ-IDEATE-04 | Measurable NFR targets for REQ-NFR-ASSIST-01 and REQ-NFR-ASSIST-03. (1) Latency: specify 'p95 assist-turn latency ≤ 650ms in Phase-1 measurement (pilot tolerance per D-072); <600ms hardening deferred to v0.6' — resolves the ambiguity in REQ-NFR-ASSIST-01's current text. (2) Guardrail: specify 'false-positive rate < 5% on the tuning corpus (REQ-IDEATE-01); false-negative rate measured + trended nightly' — makes REQ-NFR-ASSIST-03 verifiable. | must | P1 | active |
|
||||
|
||||
### Process / Tech Debt (IDEATE-06)
|
||||
|
||||
| REQ-ID | Requirement | Priority | Phase | Status |
|
||||
|--------|-------------|----------|-------|--------|
|
||||
| REQ-IDEATE-06 | Carry-forward the 8 v0.4 P1+ findings into the v0.5 backlog as a 'tech-debt wave'. Especially: (1) aggregation in-memory cache lost on restart (REVIEW.md P1+ #7 — directly corrupts v0.5 assist_active_learners_count after a server restart); (2) cookie-secret length validation (P1+ #3); (3) set_credential_status enum/f-string SQL (P1+ #4/#8). High-value, low-effort — folding into the v0.5 PLAN as a dedicated wave. | should | P1 | complete |
|
||||
| REQ-IDEATE-06 | Carry-forward the 8 v0.4 P1+ findings into the v0.5 backlog as a 'tech-debt wave'. Especially: (1) aggregation in-memory cache lost on restart (REVIEW.md P1+ #7 — directly corrupts v0.5 assist_active_learners_count after a server restart); (2) cookie-secret length validation (P1+ #3); (3) set_credential_status enum/f-string SQL (P1+ #4/#8). High-value, low-effort — folding into the v0.5 PLAN as a dedicated wave. | should | P1 | active |
|
||||
|
||||
### Cost (IDEATE-07)
|
||||
|
||||
| REQ-ID | Requirement | Priority | Phase | Status |
|
||||
|--------|-------------|----------|-------|--------|
|
||||
| REQ-IDEATE-07 | Assist per-turn cost tracking + C-3 budget impact verification. Extend server/cost.py to log per-assist-turn cost (each assist turn is a separate gemma4:cloud invocation). Add a Phase-1 budget check: estimate monthly assist cost per learner (e.g., 20 turns/shift × 20 shifts/month = 400 extra LLM calls) and flag if it pushes the total over the C-3 ≤ $3/active learner/month target. Extends REQ-NFR-COST-01 (v0.1 cost logging) to the new assist surface. | should | P1 | complete |
|
||||
| REQ-IDEATE-07 | Assist per-turn cost tracking + C-3 budget impact verification. Extend server/cost.py to log per-assist-turn cost (each assist turn is a separate gemma4:cloud invocation). Add a Phase-1 budget check: estimate monthly assist cost per learner (e.g., 20 turns/shift × 20 shifts/month = 400 extra LLM calls) and flag if it pushes the total over the C-3 ≤ $3/active learner/month target. Extends REQ-NFR-COST-01 (v0.1 cost logging) to the new assist surface. | should | P1 | active |
|
||||
|
||||
## v0.6 Backlog (IDEATE-10..13, accepted for v0.6)
|
||||
|
||||
|
||||
@@ -1,322 +0,0 @@
|
||||
# Praxis — v0.5 Milestone Review (Final Phase P3)
|
||||
|
||||
> **Reviewer:** ci-code-reviewer (multi-persona: lead-developer, voice-engineer, backend-engineer, security-engineer, data-engineer)
|
||||
> **Scope:** full v0.5 milestone diff — `git diff v0.1.9..milestone/v0.5-live-assist` (63 files, +10,785/-85 LOC) — covers Phase 0 (planning) + P1 (assist core + guardrail) + P2 (integration + tech-debt + NFR measurement)
|
||||
> **Branch:** `phase/03-final-review-ship` (from `milestone/v0.5-live-assist`)
|
||||
> **Date:** 2026-08-04
|
||||
> **Method:** code inspection (all v0.5 source + tests), test execution, security grep, grill MUST verification, adversarial analysis, PIPEDA escalation review
|
||||
> **Prior verification:** VERIFY-P1-v0.5.md (APPROVE_WITH_NOTES, 5 P1+), VERIFY-P2-v0.5.md (APPROVE_WITH_NOTES, 3 P1+), GRILL-v0.5.md (39 decisions, 2 MUSTs resolved, 1 escalation)
|
||||
|
||||
## Summary
|
||||
- **Verdict: APPROVE_WITH_NOTES**
|
||||
- **Personas:** lead-developer **PASS**, voice-engineer **PASS**, backend-engineer **PASS**, security-engineer **PASS**, data-engineer **PASS**
|
||||
- **P0 fixes applied:** 1 (guardrail processor streaming-before-check — REQ-ASSIST-03 safety-critical)
|
||||
- **P1+ flagged:** 8 (5 from P1 VERIFY + 3 from P2 VERIFY — all non-blocking, all carry-forward to v0.6)
|
||||
- **Total v0.5 REQ coverage:** 16/16 (3 ASSIST + 4 NFR + 9 IDEATE)
|
||||
- **Grill MUSTs honored:** 2/2 (G-049 in-loop retry validation, G-067 adversarial FN threshold)
|
||||
- **ESCALATION-01 (PIPEDA):** OPEN — flagged for human legal review before assist surface goes live
|
||||
|
||||
## Test Results
|
||||
|
||||
| Suite | Result | Notes |
|
||||
|-------|--------|-------|
|
||||
| `python3 -m pytest tests/` | **469 passed, 45 skipped, 0 failed** (106.57s) | Post-P0-fix; 36 v0.4 skips + 9 P2 PG-skipped; all env-gated (PRAXIS_PG_DSN unset, live voice keys, W3C interop) |
|
||||
| `cd client && npm run build` | **PASS** | 665KB / 187KB gzip, 499ms |
|
||||
| `python3 -c "import server.assist.context; ..."` | **PASS** | All 13 assist modules + 25 exports importable |
|
||||
| `python3 -m pytest tests/test_guardrail_tuning.py` | **5 passed** | FP 0.0%, direct FN 0.0%, false-authority 0%, adversarial FN 13.3% (≤20% G-067) |
|
||||
| Security grep (f-string SQL, hardcoded secrets, PII in Postgres) | **PASS** | No injection vectors; no secrets; no raw PII in operator tier |
|
||||
|
||||
---
|
||||
|
||||
## P0 Fix Applied
|
||||
|
||||
### P0-1 — Guardrail processor streamed blocked text to TTS before the check (REQ-ASSIST-03)
|
||||
|
||||
**File:** `server/assist/guardrail_processor.py:83-90` (pre-fix)
|
||||
**Issue:** The in-loop `LiveAssistGuardrailProcessor` pushed `TextFrame` chunks through to TTS **as they arrived** (streaming), then ran the guardrail `check()` on `LLMFullResponseEndFrame` (after the full response). For a safety-critical surface (REQ-ASSIST-03 — the AI is in the learner's ear during real customer interactions), this means the LLM's direct-answer text would be **spoken to the learner before the guardrail could block it**. The code comment even acknowledged this: *"In a full implementation, we'd buffer + emit only the filtered text."*
|
||||
|
||||
This defeats the entire guardrail surface: a "you should say sorry to the customer" response would reach the learner's ear, the learner would parrot it to the real customer (R-ASSIST-07 — the project-killing risk), and then the canned fallback would play afterward — too late. The guardrail `check()` returning `allowed=False` would log the verdict + emit `CANNED_FALLBACK`, but the blocked text was already spoken.
|
||||
|
||||
**Severity:** P0 — safety-critical. This is the single most important requirement in v0.5 (REQ-ASSIST-03). The grill's G-067 binding (adversarial FN threshold) is moot if the blocked text reaches TTS regardless of the verdict.
|
||||
|
||||
**Fix applied (commit `5373df2`):** Buffer `TextFrame` chunks (do not push to TTS) until `LLMFullResponseEndFrame`. On the end frame, run the guardrail check:
|
||||
- **allowed** → push the buffered text as a single `TextFrame` to TTS (not streamed chunk-by-chunk)
|
||||
- **blocked + retry-eligible** → inject `RETRY_INSTRUCTION` (no text to TTS; the LLM re-runs)
|
||||
- **blocked + hard violation** → push `CANNED_FALLBACK` to TTS
|
||||
|
||||
This adds ~200-500ms of latency (buffering 1-3 sentences) but is **required for safety** — a blocked direct answer must never reach the learner's ear. The latency cost is flagged for v0.6 hardening if it pushes p95 >650ms (D-072 pilot tolerance). The existing e2e test (`test_guardrail_blocks_direct_answer_e2e`) already asserted `CANNED_FALLBACK` was pushed — but it didn't assert the blocked text was *not* pushed (the mock `push_frame` accepted everything). The fix + updated test now verify the safety-critical invariant: only allowed text or `CANNED_FALLBACK` reaches TTS.
|
||||
|
||||
**Test updated:** `tests/test_assist_pipeline.py::test_processor_passes_allowed_text_through` — now asserts the buffered text is pushed as a single `TextFrame` on `LLMFullResponseEndFrame` (not streamed chunk-by-chunk), reflecting the safety-critical behavior.
|
||||
|
||||
**Post-fix test run:** 469 passed, 45 skipped, 0 failed. The fix is verified.
|
||||
|
||||
---
|
||||
|
||||
## Persona 1 — Lead-Developer (Coordination + Architecture Coherence)
|
||||
|
||||
### Findings (all PASS — post-P0-fix)
|
||||
|
||||
1. **D-071 (tap-to-talk only) honored:** `client/src/AssistControl.tsx` (139 LOC) implements tap-to-talk (press+hold to speak, release to send). No wake-word, no Porcupine, no foreground service. The wake-word is deferred to v0.6. The component is below the frontend-engineer reactivation threshold (~100-150 LOC). ✅
|
||||
|
||||
2. **D-063 (assist ≠ mastery) honored:** `AssistSession.end()` (server/assist/session.py:168-192) does NOT call `run_mastery_flow()`. `_build_session_outcome()` sets `rubric_scores=[]` + `"session_type": "assist"`. The cohort aggregation `_aggregate_assist` branch computes NO mastery metrics (no gate_open_rate, no median_mastery_score). The mastery view (`server/operator/mastery.py`) excludes assist metrics. Explicitly tested (`test_d063_assist_does_not_update_mastery`). ✅
|
||||
|
||||
3. **D-072 (≤650ms pilot tolerance) honored:** `AssistLatencyMetrics` (server/assist/latency_metrics.py) computes p95 with `within_target = (p95 < 600)` + `within_pilot = (p95 <= 650)`. The boundary test (p95 == 650 → within_pilot=True, within_target=False) confirms the ≤ vs < distinction. The measurement is infrastructure (mock records), not a live latency assertion (correct — latency depends on live voice services). ✅
|
||||
|
||||
4. **2-phase split coherent:** P1 (assist core + guardrail, 12 REQs, 24 tasks) is independently shippable — a learner can start a shift, tap-to-talk, get coaching with guardrails, end the shift. P2 (integration + tech-debt + NFR measurement, 4 REQs, 9 tasks) layers on operator visibility + cost + measurement. The aggregation cache tech-debt (v0.4 P1+ #7) was in P2 SLICE-12, on the critical path for correct assist metrics (G-051). ✅
|
||||
|
||||
5. **Architecture coherence:** The assist surface is additive — new `server/assist/` package, new `server/guardrails/live_assist.py`, new SQLite migration 0004 (additive), new React route `/assist`. The v0.1-v0.4 surfaces (practice voice loop, mastery, VC, operator dashboard) are unchanged. The assist pipeline reuses `_build_transport`, `_build_stt`, `_build_llm` from `server/pipeline.py` (FIXED, not rewritten). ✅
|
||||
|
||||
### Lead-developer verdict: PASS — binding constraints honored, architecture coherent, 2-phase split clean.
|
||||
|
||||
---
|
||||
|
||||
## Persona 2 — Voice-Engineer (Assist Pipeline + Latency + WebRTC + Tap-to-Talk)
|
||||
|
||||
### Find (PASS — post-P0-fix)
|
||||
|
||||
1. **build_assist_pipeline reuses v0.1 services (D-061):** `server/assist/pipeline.py:83` imports `_build_llm, _build_stt, _build_transport` from `server/pipeline.py`. The pipeline structure is correct: `transport.input → stt → latency_observer → user_aggregator → llm → latency_observer → guardrail_processor → tts → latency_observer → transport.output → assistant_aggregator`. The `LiveAssistGuardrailProcessor` is between `llm` and `tts` (D-060 layer 2). ✅
|
||||
|
||||
2. **Piper TTS default (D-065):** `_build_tts_assist()` defaults to Piper (`_build_tts_piper()`). Falls back to Cartesia if `PRAXIS_ASSIST_TTS=cartesia`. The existing `_build_tts()` (Cartesia, practice path) is unchanged. ✅
|
||||
|
||||
3. **≤150-token assist prompt (D-066):** `AssistContextBinder.bind()` constructs the system prompt from `COACHING_INSTRUCTION` (~80 tokens) + context-binding (~50 tokens) + `VOICE_CONCISENESS` (~20 tokens). The word-budget assertion (`_MAX_PROMPT_WORDS = 200`) truncates the context-binding section if exceeded. ✅
|
||||
|
||||
4. **Warm WebRTC (D-067):** `WarmWebRTCManager` opens a connection at shift start, runs a 30s heartbeat (`_HEARTBEAT_INTERVAL_S = 30`), closes at shift-end. The reconnect state machine (`connected → reconnecting → disconnected`) waits 30s for a new offer; the shift is NOT auto-ended on disconnect (the 8h auto-end still fires). ✅
|
||||
|
||||
5. **P0 fix correctness (post-fix):** The guardrail processor now buffers `TextFrame` chunks + only pushes allowed text (or `CANNED_FALLBACK`) to TTS on `LLMFullResponseEndFrame`. This is the safety-critical behavior — a blocked direct answer never reaches the learner's ear. The latency cost (~200-500ms buffering) is flagged for v0.6 hardening if p95 >650ms. The retry mechanism (G-049) injects `RETRY_INSTRUCTION` via `llm_context.add_message()` — validated by the spike test. ✅
|
||||
|
||||
6. **Tap-to-talk client (D-071):** `AssistControl.tsx` provides Start/End Shift buttons + a press-to-talk button + context declaration (path week + scenario tag) + consent disclosure banner. Routed at `/assist`. `npm run build` succeeds. ✅
|
||||
|
||||
### Voice-engineer verdict: PASS — pipeline reuses v0.1 services, Piper default, warm WebRTC, guardrail processor now safety-correct (post-P0-fix).
|
||||
|
||||
---
|
||||
|
||||
## Persona 3 — Backend-Engineer (Assist Session API + Context-Binding + Aggregator + __main__.py)
|
||||
|
||||
### Findings (all PASS — post-P0-fix)
|
||||
|
||||
1. **Assist session API (3 routes):** `POST /api/assist/shift/start` (mode-conflict → bind context → create session → return shift_id + context + consent_disclosure), `POST /api/assist/shift/end` (end session → return turn_count + guardrail_block_count), `GET /api/assist/shift/active` (return active shift or `{active: false}`). All use `HARDCODED_LEARNER_ID = "learner-1"` (D-007). Routes registered before StaticFiles. ✅
|
||||
|
||||
2. **Mode-conflict guard (REQ-IDEATE-03):** `enforce_mutual_exclusivity()` checks the *other* type (`other_type = "practice" if requested_type == "assist" else "assist"`). The `get_active_session()` query filters on `ended_at IS NULL` — ended sessions don't trigger the conflict. Enforced in both directions (assist-during-practice → 409; practice-during-assist → 409). The existing `/pipecat/webrtc` endpoint (practice) calls the guard with `'practice'`; the new `/api/assist/shift/start` + `/api/assist/webrtc` endpoints call it with `'assist'`. ✅
|
||||
|
||||
3. **Context-binding (D-059, D-066):** `AssistContextBinder.bind()` reads `progress.current_week` + `theta` from SQLite (parameterized queries via aiosqlite — no SQL injection). The path YAML is read with `yaml.safe_load` (no arbitrary object construction). Missing learner state → defaults (week=1, theta=0.0, focus=generic). The prompt is never empty. ✅
|
||||
|
||||
4. **__main__.py wiring:** Assist routes + WebRTC endpoint + lifecycle monitor wired in `lifespan`. `app.state.assist_webrtc_manager = WarmWebRTCManager()`, `app.state.assist_shifts = {}`, `ShiftLifecycleManager` started. The mode-conflict guard is enforced on both the practice `/pipecat/webrtc` endpoint + the assist `/api/assist/webrtc` endpoint. ✅
|
||||
|
||||
5. **Cohort aggregation assist branch (D-062):** `aggregate_session()` branches on `session_type`: `'assist'` → `_aggregate_assist`, else → `_aggregate_practice`. The assist branch computes 5 core metrics + p95 + cost, uses the SAME k-anon suppression (≥10), the SAME 7-day rolling window, + the SAME idempotent upsert. No schema change (D-062 — metric is free-form TEXT). D-063: no mastery metrics in the assist branch. ✅
|
||||
|
||||
6. **Cost tracking (REQ-IDEATE-07):** `derive_assist_turn_cost()` (server/cost.py) computes per-turn cost (LLM tokens + Piper TTS chars). `check_c3_budget()` (server/assist/budget_check.py) is diagnostic (not enforced per D-012) — returns `flag=True` if over $3 but does not raise. The cents→USD conversion is correct (divide by 100). ✅
|
||||
|
||||
### Backend-engineer verdict: PASS — API routes correct, mode-conflict enforced both directions, context-binding safe, aggregation branch clean, __main__.py wiring complete.
|
||||
|
||||
---
|
||||
|
||||
## Persona 4 — Security-Engineer (LiveAssistGuardrail 3-Layer + PII + Consent + PIPEDA)
|
||||
|
||||
### Findings (all PASS — post-P0-fix)
|
||||
|
||||
1. **3-layer guardrail (D-060, D-068):**
|
||||
- **Layer 1 (coaching-mode system prompt):** `COACHING_INSTRUCTION` is a fixed prefix in `AssistContextBinder.bind()` — always prepended, never replaced. The `scenario_tag` is inserted into the context-binding section, but the coaching instruction is immutable. ✅
|
||||
- **Layer 2 (regex output filter):** `LiveAssistGuardrail.check()` runs 6 regex patterns (DIRECT_SCRIPT_RE, INDIRECT_SCRIPT_RE, IMPERATIVE_RE, FALSE_AUTHORITY_RE, IMPERSONATION_RE, COACHING_QUESTION_RE). The `INDIRECT_SCRIPT_RE` is an addition beyond the plan (catches adversarial paraphrases — this is how the adversarial FN rate was reduced to 13.3%). The regex compilation is at module load (not per-call) — correct for performance. **Post-P0-fix:** the in-loop processor now buffers text + only pushes allowed text or `CANNED_FALLBACK` to TTS — the guardrail actually prevents blocked text from reaching the learner's ear. ✅
|
||||
- **Layer 3 (audit log):** `guardrail_verdict_json` is written to the turns table for every assist turn (incremental write per REQ-IDEATE-09). The verdict is JSON-serialized + persisted before TTS playback completes. ✅
|
||||
|
||||
2. **G-067 (adversarial FN threshold) resolved:** `ADVERSARIAL_FN_THRESHOLD = 0.20` (≤20% acceptable for pilot). Measured: 13.3% (4/30). The threshold + rationale are documented: "acceptable for pilot because defense-in-depth (prompt + regex + audit) + the v0.6 LLM-as-judge (REQ-IDEATE-10) mitigate the residual risk." ✅
|
||||
|
||||
3. **G-049 (in-loop retry validation) resolved:** The spike test (`test_g049_guardrail_processor_spike.py`, 6 tests) verified `LLMFullResponseEndFrame` is a real Frame type + `LLMContext.add_message` can inject `RETRY_INSTRUCTION`. The retry mechanism is implemented: retry-eligible blocks inject the retry instruction; hard violations (false-authority, impersonation) get `CANNED_FALLBACK` immediately (no retry). ✅
|
||||
|
||||
4. **PII policy (REQ-IDEATE-05):** `redact_pii()` redacts phone numbers, emails, card numbers, SIN-like numbers before writing to the turns table. Applied in `AssistSession.log_assist_turn()` + `log_assist_turn_partial()`. The policy is option (c): retain with redaction + consent + 30-day retention. No raw PII in Postgres (D-031 — local SQLite only). ✅
|
||||
|
||||
5. **Consent disclosure (D-070):** `CONSENT_DISCLOSURE_TEXT` is surfaced to the client in the `/api/assist/shift/start` response. The disclosure mentions mic active, those around you may be recorded, local consent laws, and how to stop. ✅
|
||||
|
||||
6. **STRIDE summary (post-P0-fix):**
|
||||
- **Spoofing:** LOW (D-007 single-learner, mode-conflict guard both directions)
|
||||
- **Tampering:** LOW (3-layer defense, each tamper-resistant; **post-P0-fix: Layer 2 now actually prevents blocked text from reaching TTS**)
|
||||
- **Repudiation:** LOW (incremental append-first audit log — REQ-IDEATE-09)
|
||||
- **Info Disclosure:** MEDIUM (customer-speech PII — mitigated by redaction + consent + local SQLite; PIPEDA legal review pending ESCALATION-01; nightly cleanup not scheduled — P1+)
|
||||
- **Denial of Service:** LOW (8h auto-end + 30s heartbeat + single-learner)
|
||||
- **Elevation of Privilege:** LOW (D-063 enforced — no mastery update on assist)
|
||||
|
||||
### Security-engineer verdict: PASS — 3-layer guardrail is safety-correct post-P0-fix, PII redacted, consent disclosed, G-049 + G-067 resolved. PIPEDA escalation remains open (ESCALATION-01).
|
||||
|
||||
---
|
||||
|
||||
## Persona 5 — Data-Engineer (SQLite Migration + Aggregation Cache + Cohort Metrics)
|
||||
|
||||
### Findings (all PASS)
|
||||
|
||||
1. **SQLite migration 0004 (additive):** `db/migrations/0004_assist.sql` adds `session_type TEXT NOT NULL DEFAULT 'practice'` (existing sessions unaffected), `guardrail_verdict_json TEXT` (nullable — only assist turns populate), + `idx_sessions_active_by_type` index (for the mode-conflict check). Idempotent (`CREATE INDEX IF NOT EXISTS`). ✅
|
||||
|
||||
2. **Aggregation cache persistence (v0.4 P1+ #7):** `server/cohort/learner_cache.py` persists the distinct-learner set to a SQLite `cohort_learner_cache` table. `_load_learner_cache` on startup, `_save_learner_cache` on each session, `_clear_learner_cache` by the nightly job. The cache survives restart (verified by `test_p2_techdebt_aggregation_cache_survives_restart`, PG-skipped). This was the highest-value tech-debt fix for v0.5 — it directly corrupts `assist_active_learners_count` after a restart. ✅
|
||||
|
||||
3. **Assist cohort metrics (D-062):** The 5 core metrics + p95 + cost are computed in `_aggregate_assist`: `assist_shifts_count`, `assist_turns_count`, `assist_avg_turns_per_shift`, `assist_active_learners_count`, `assist_guardrail_block_rate`, `assist_p95_latency_ms`, `assist_avg_cost_per_shift`. k-anon suppression (≥10) applies identically to practice. No schema change (D-062 — metric is free-form TEXT). ✅
|
||||
|
||||
4. **k-anon boundary tests:** 9 learners → suppressed, 10 → not suppressed (same threshold as practice). The assist branch uses the SAME `_bump_active_learners` + `K_ANON_THRESHOLD = 10` as practice. ✅
|
||||
|
||||
5. **Nightly trend (REQ-IDEATE-04):** `GuardrailMetrics.nightly_trend()` reads assist turns from the last 24h, re-runs the guardrail, classifies coaching/neutral, + identifies FN candidates. Off-voice-path (called by the nightly job, not the assist pipeline). The `fn_candidates` include truncated `tts_text` (AI-generated coaching, not customer PII — `asr_text` is redacted). ✅
|
||||
|
||||
6. **ZoneInfo DST (v0.4 P1+ #6):** `server/cohort/nightly.py` uses `ZoneInfo("America/Winnipeg")` — correctly handles CST (UTC-6) in winter + CDT (UTC-5) in summer. Verified by summer/winter/spring-forward tests. ✅
|
||||
|
||||
### Data-engineer verdict: PASS — migration additive, cache persistence fixes the restart corruption, assist metrics correct, k-anon enforced, nightly trend off-voice-path.
|
||||
|
||||
---
|
||||
|
||||
## Grill MUSTs Honored (2/2)
|
||||
|
||||
| MUST | Honored | Evidence |
|
||||
|------|---------|----------|
|
||||
| G-049 (in-loop guardrail retry validation) | YES | `tests/test_g049_guardrail_processor_spike.py` (6 tests): LLMFullResponseEndFrame is a real Frame, LLMContext.add_message injects RETRY_INSTRUCTION, retry-eligible vs hard-violation distinction. The processor implements the validated pattern. |
|
||||
| G-067 (R-ASSIST-07 adversarial FN threshold) | YES | `tests/test_guardrail_tuning.py`: `ADVERSARIAL_FN_THRESHOLD = 0.20`, measured 13.3% (4/30), threshold + rationale documented. The test asserts `fn <= 0.20` (PASSES). |
|
||||
|
||||
---
|
||||
|
||||
## REQ Coverage (16/16)
|
||||
|
||||
| REQ-ID | Phase | Covered by | Status |
|
||||
|--------|-------|-----------|--------|
|
||||
| REQ-ASSIST-01 | P1 | build_assist_pipeline + tap-to-talk client + __main__.py wiring | ✅ COVERED |
|
||||
| REQ-ASSIST-02 | P1 | AssistContextBinder (path week + scenario tag + theta from SQLite) | ✅ COVERED |
|
||||
| REQ-ASSIST-03 | P1 | LiveAssistGuardrail 3-layer + tuning corpus + adversarial test + e2e (post-P0-fix) | ✅ COVERED |
|
||||
| REQ-NFR-ASSIST-01 | P2 | AssistLatencyMetrics (p95/p50/p99 + D-072 within_target/within_pilot) | ✅ COVERED |
|
||||
| REQ-NFR-ASSIST-02 | P1 | tap-to-talk only (D-071 — no wake-word in v0.5) | ✅ COVERED |
|
||||
| REQ-NFR-ASSIST-03 | P1 | 3-layer guardrail + tuning corpus + adversarial test | ✅ COVERED |
|
||||
| REQ-NFR-ASSIST-04 | P1 | shift-bounded session model + 8h auto-end + aggregation as session_type=assist | ✅ COVERED |
|
||||
| REQ-IDEATE-01 | P1 | guardrail tuning corpus (151 entries) + adversarial bypass test | ✅ COVERED |
|
||||
| REQ-IDEATE-02 | P1 | in-loop guardrail processor pipeline test + GuardrailContext.role 'assist' | ✅ COVERED |
|
||||
| REQ-IDEATE-03 | P1 | mode-conflict enforcement (assist vs practice mutual exclusivity, 409 both directions) | ✅ COVERED |
|
||||
| REQ-IDEATE-04 | P1+P2 | measurable NFR targets (p95 ≤650ms, FP<5%, FN measured + trended nightly) | ✅ COVERED |
|
||||
| REQ-IDEATE-05 | P1 | customer-speech PII policy (retain with redaction + consent + 30-day retention) | ✅ COVERED |
|
||||
| REQ-IDEATE-06 | P2 | 8 v0.4 P1+ tech-debt wave (all addressed with fix + test) | ✅ COVERED |
|
||||
| REQ-IDEATE-07 | P2 | assist per-turn cost tracking + C-3 budget check (diagnostic) | ✅ COVERED |
|
||||
| REQ-IDEATE-08 | P1 | WebRTC mid-shift drop + reconnect logic (state machine + chaos test) | ✅ COVERED |
|
||||
| REQ-IDEATE-09 | P1 | audit-log incremental write (partial turn on TranscriptionFrame, complete on LLMFullResponseEndFrame) | ✅ COVERED |
|
||||
|
||||
---
|
||||
|
||||
## 8 v0.4 P1+ Tech-Debt Wave (all addressed in P2 SLICE-12)
|
||||
|
||||
| P1+ ID | Finding | P2 Fix | Test | Verified |
|
||||
|--------|---------|--------|------|----------|
|
||||
| #1 | Argon2id blocking event loop | `asyncio.to_thread(verify_password/hash_password)` | `test_login_argon2id_offloaded_to_thread` | ✅ |
|
||||
| #2 | Rate limit 429 not tested in mock path | Mock-based 429 test (6th attempt → 429) | `test_login_rate_limit_429_after_5_attempts` | ✅ |
|
||||
| #3 | No PRAXIS_COOKIE_SECRET length validation | `elif len(secret) < 32: logger.warning(...)` | 3 cookie-secret tests | ✅ |
|
||||
| #4 | set_credential_status status not validated | `if status not in ("active", "revoked"): raise ValueError` | `test_set_credential_status_invalid_raises_value_error` | ✅ |
|
||||
| #5 | Credential revocation lacks audit log | `log.info("credential revoked: operator=%s cred_id=%s", ...)` | `test_credential_revocation_logs_audit_event` | ✅ |
|
||||
| #6 | Nightly scheduler fixed UTC-5 offset | `ZoneInfo("America/Winnipeg")` | 4 zoneinfo tests | ✅ |
|
||||
| #7 | Aggregation cache lost on restart | SQLite `cohort_learner_cache` persistence | `test_p2_techdebt_aggregation_cache_survives_restart` | ✅ |
|
||||
| #8 | set_credential_status f-string SQL | Two explicit parameterized queries | `test_set_credential_status_no_fstring_in_sql` | ✅ |
|
||||
|
||||
---
|
||||
|
||||
## P1+ Findings Flagged for Post-Hoc Review (8 — all non-blocking, carry-forward to v0.6)
|
||||
|
||||
### From P1 VERIFY (5 P1+):
|
||||
|
||||
1. **P1-1 (MEDIUM — Info Disclosure): PII retention cleanup not scheduled** — `server/assist/pii_policy.py:24` (`RETENTION_DAYS = 30`). The 30-day retention is documented but no scheduled task deletes turns older than 30 days. Defense-in-depth (consent + local SQLite) is the primary protection. **Deferred to v0.6** — add a nightly retention-cleanup task.
|
||||
|
||||
2. **P1-2 (LOW — Security): Scenario-tag prompt injection (unsanitized input)** — `server/assist/context.py:134`. The `scenario_tag` is inserted into the system prompt via f-string without sanitization. Low risk: single-learner (D-007, self-injection only), coaching instruction is a fixed prefix, Layer 2 regex still filters output. **Deferred to v0.6** — sanitize the `scenario_tag` (strip newlines, cap length, validate against a known scenario list).
|
||||
|
||||
3. **P1-3 (LOW — Correctness): end_session_assist doesn't persist turn/block counts** — `db/store.py:193`. The counts flow to the aggregation hook via `session_outcome` (in-memory), but the sessions table has no `turn_count`/`guardrail_block_count` columns. Server-restart edge case loses the counts. **Mitigated** by the cache persistence (P2 SLICE-12) — the cache survives restart.
|
||||
|
||||
4. **P1-4 (LOW — Maintainability): WebRTC reconnect offer-event not wired** — `server/assist/webrtc.py:149-152`. The reconnect state machine waits 30s for a new offer, but the mechanism for a new offer to arrive during the wait is not wired (the `/api/assist/webrtc` endpoint always calls `manager.open()`, not `manager.reconnect()`). The shift is NOT auto-ended on disconnect; the 8h auto-end still fires. **Deferred to v0.6** — wire the endpoint to call `reconnect()` if a shift is in 'reconnecting' state.
|
||||
|
||||
5. **P1-5 (LOW — Testing): No concurrent shift-start race test** — `server/assist/routes.py:78-82`. The `active_shifts` dict on `app.state` is a plain dict (no lock). Low risk: single-learner (D-007), no concurrent requests expected in pilot. The DB-level mode-conflict guard catches concurrent starts. **Deferred to v0.6** — add a concurrent-shift-start test.
|
||||
|
||||
### From P2 VERIFY (3 P1+):
|
||||
|
||||
6. **P2-1 (LOW — Performance): Cache I/O on every session-end hook** — `server/cohort/aggregator.py:320-355`. `_bump_active_learners` calls `_load_learner_cache` (first call per path/window) + `_save_learner_cache` (every call). The `_load_learner_cache` loads the ENTIRE cache. Pilot scale (~100 learners) is <10ms per hook; off-voice-path. **Deferred to v0.6** — load only the specific (path, window) learners; batch the saves.
|
||||
|
||||
7. **P2-2 (LOW — Maintainability): nightly_trend bypasses PraxisStore API** — `server/assist/guardrail_metrics.py:153-167`. The `nightly_trend` reads from the turns table via a direct `aiosqlite.connect(store.db_path)` connection, bypassing the `PraxisStore` API. Deliberate choice (documented) — the store abstraction is leaked. **Deferred to v0.6** — add a `list_recent_assist_turns(hours: int)` method to `PraxisStore`.
|
||||
|
||||
8. **P2-3 (LOW — Security): nightly_trend fn_candidates include truncated tts_text** — `server/assist/guardrail_metrics.py:202, 210`. The `fn_candidates` dict includes `tts_text` (truncated to 200 chars). The `tts_text` is AI-generated coaching (not customer PII — `asr_text` is redacted). The `fn_candidates` are returned to the caller (nightly job), not logged directly. **Deferred to v0.6** — ensure the nightly job does not log the `tts_text` from `fn_candidates`.
|
||||
|
||||
---
|
||||
|
||||
## ESCALATION-01 (PIPEDA Consent-Law Review) — Status: OPEN
|
||||
|
||||
**Per GRILL-v0.5.md ESCALATION-01 (confidence 0.55 — below 0.60 threshold):**
|
||||
|
||||
The ambient mic captures the real customer (a third party); ASR transcribes their speech; the turns table stores it (REQ-IDEATE-05). Canada's PIPEDA + provincial one-party/two-party consent laws govern recording. D-073 defers the legal review. The disclosure (D-070) is shown to the *learner*, not the *customer* — it is the engineering mitigation, not a legal determination.
|
||||
|
||||
**Engineering mitigations implemented (D-070, REQ-IDEATE-05):**
|
||||
- Consent disclosure surfaced to the learner in the `/api/assist/shift/start` response + displayed in the client (`AssistControl.tsx` consent banner).
|
||||
- PII redaction (`redact_pii()`) applied to `asr_text` before storage (phone, email, card, SIN-like numbers).
|
||||
- 30-day retention documented (`RETENTION_DAYS = 30` in `get_pii_policy()`).
|
||||
- Local SQLite only (not Postgres — D-031, no raw PII in the operator tier).
|
||||
- The PII policy returns `"legal_review": "pending — D-073"`.
|
||||
|
||||
**The CI cannot resolve a legal question under full autonomy.** This is the de facto stop trigger for the assist surface (G-072). The disclosure is ethically required + implemented regardless of the legal review.
|
||||
|
||||
**Action required (before assist surface goes live):** Human legal review of Canada PIPEDA + provincial consent law for ambient recording during coaching. Determine:
|
||||
1. Does the pilot province require one-party consent (learner's consent sufficient — D-070 covers) or two-party consent (customer must consent — Praxis cannot notify the customer)?
|
||||
2. If one-party: the disclosure (D-070) is sufficient. Proceed.
|
||||
3. If two-party: the assist surface may need geographic restriction (one-party provinces only) or customer-facing consent (out of scope for v0.5).
|
||||
4. If a PIPEDA privacy policy / data handling agreement is required: the PII policy (REQ-IDEATE-05) may need to be formalized into a PIPEDA-compliant policy before ship.
|
||||
|
||||
**Status: OPEN — flagged for human attention. The milestone ships with the engineering mitigations in place; the legal determination is a post-ship human action item.**
|
||||
|
||||
---
|
||||
|
||||
## Milestone Readiness Assessment
|
||||
|
||||
The v0.5 milestone (Live Assist — On-the-Job Voice Companion) is **APPROVE_WITH_NOTES** and ready for ship (v0.1.13 = v0.5 milestone release), subject to the ESCALATION-01 human action item.
|
||||
|
||||
**Ready:**
|
||||
- All 16 REQ-IDs covered (3 ASSIST + 4 NFR + 9 IDEATE).
|
||||
- All 2 grill MUSTs honored (G-049, G-067).
|
||||
- All 8 v0.4 P1+ tech-debt findings addressed (fix + test).
|
||||
- 469 tests pass, 45 skipped (all env-gated), 0 failed.
|
||||
- 1 P0 fix applied (guardrail processor safety-critical — REQ-ASSIST-03).
|
||||
- Client build succeeds.
|
||||
- The assist surface is additive (clean revert to v0.1.9 = v0.4).
|
||||
|
||||
**Flagged (non-blocking):**
|
||||
- 8 P1+ findings deferred to v0.6 (all LOW/MEDIUM, all with mitigations present).
|
||||
- ESCALATION-01 (PIPEDA) — human legal review required before the assist surface goes live.
|
||||
|
||||
**Ship notes (per G-046, G-051, G-065, G-069, G-073, G-078):**
|
||||
- IDEATE expanded scope +128% (7→16 REQs). All additions are risk-reduction. Future ideation must maintain discipline.
|
||||
- P1 shipped with assist metrics incorrect (aggregation cache tech-debt) — fixed in P2 SLICE-12 before operator dashboard visibility.
|
||||
- Tap-to-talk UX (D-071) is the lowest-confidence assumption (0.60, unvalidated). v0.5 pilot validates adoption; v0.6 adds wake-word if low.
|
||||
- Post-ship safety signal escalation (nightly FN trend spike → human) is a v0.6+ governance gap. v0.5 ships the measurement; v0.6 adds the LLM-as-judge + the escalation response.
|
||||
- v0.5 validates the coaching/guardrail/context-binding value, not the hands-free UX (tap-to-talk is the pilot validation; wake-word is v0.6).
|
||||
- The guardrail tuning corpus (REQ-IDEATE-01) is synthetic (LLM-generated), not a human red-team prompt set. Accepted limitation for pilot.
|
||||
- **P0 fix added ~200-500ms latency (buffering LLM text before TTS). If p95 >650ms in Phase-1 live measurement, v0.6 hardening is required (streaming guardrail with early-exit on first direct-answer pattern, or a faster LLM).**
|
||||
|
||||
---
|
||||
|
||||
## Bottom Line
|
||||
|
||||
The v0.5 milestone (Live Assist — On-the-Job Voice Companion) is **APPROVE_WITH_NOTES**. All 5 personas pass. All 16 REQs are covered. All 2 grill MUSTs are honored. All 8 v0.4 P1+ tech-debt findings are addressed. One P0 fix was applied (guardrail processor safety-critical — the in-loop processor now buffers LLM text before TTS, ensuring blocked direct answers never reach the learner's ear). Eight P1+ items are flagged for v0.6 post-hoc review (all non-blocking, all with mitigations present).
|
||||
|
||||
The implementation is correct (D-063 enforced, mode-conflict both directions, k-anon ≥10, p95 percentile nearest-rank), secure (3-layer guardrail safety-correct post-P0-fix, PII redacted, consent disclosed, no raw PII in Postgres), performant (guardrail regex compiled at module load, aggregation off-voice-path, cache persistence survives restart), maintainable (clean `server/assist/` package, consistent naming, comprehensive docstrings), and adversarially sound (non-configurable privacy controls, guardrail tuning corpus + adversarial test, incremental audit-log for abrupt termination).
|
||||
|
||||
The PIPEDA legal review (ESCALATION-01) remains the open risk for human attention before the assist surface goes live. The engineering mitigations (consent disclosure + PII redaction + local SQLite + 30-day retention documented) are implemented regardless.
|
||||
|
||||
The milestone is ready for ship (v0.1.13 = v0.5). The orchestrator delegates to ship after this review, with the ESCALATION-01 human action item flagged for the assist surface go-live decision.
|
||||
|
||||
---
|
||||
|
||||
---ci---
|
||||
project: praxis
|
||||
phase: 3
|
||||
milestone: v0.5
|
||||
status: verify
|
||||
phase_role: final_review
|
||||
verdict: APPROVE_WITH_NOTES
|
||||
personas:
|
||||
lead-developer: PASS
|
||||
voice-engineer: PASS
|
||||
backend-engineer: PASS
|
||||
security-engineer: PASS
|
||||
data-engineer: PASS
|
||||
p0_fixes_applied:
|
||||
- guardrail processor buffered LLM text before TTS (REQ-ASSIST-03 safety-critical)
|
||||
p1_plus_flagged: 8
|
||||
req_coverage: 16/16
|
||||
grill_musts_honored: 2/2
|
||||
escalation_01_pipeda: OPEN
|
||||
lessons:
|
||||
- P0 fix applied: guardrail processor must buffer LLM text before TTS (REQ-ASSIST-03)
|
||||
- G-049 + G-067 MUSTs resolved with binding evidence (adversarial FN 13.3% ≤ 20% threshold)
|
||||
- 8 v0.4 P1+ tech-debt wave addressed (all with fix + test)
|
||||
- ESCALATION-01 PIPEDA remains open for human legal review before assist go-live
|
||||
---/ci---
|
||||
+6
-30
@@ -1,7 +1,7 @@
|
||||
# Praxis — Roadmap
|
||||
|
||||
**Milestone:** v0.5 (Live Assist — on-the-job voice companion) — complete
|
||||
**Status:** milestone released as v0.1.13 (merged to main) — 16/16 v0.5 REQ covered
|
||||
**Milestone:** v0.5 (Live Assist — on-the-job voice companion) — active, phase 0 pre-execution
|
||||
**Status:** phase 0 pre-execution (SPECIFY → CLARIFY → RESEARCH → IDEATE → PLAN → GRILL → SHIP)
|
||||
**Previous milestone:** v0.4 (Operator tier — cohort dashboard, auth, Postgres) — complete, tagged v0.1.9, release created, merged to main
|
||||
|
||||
## Milestone Philosophy
|
||||
@@ -12,11 +12,11 @@ The key distinction from the practice surface is **real-customer interaction**:
|
||||
|
||||
## v0.5 Phases
|
||||
|
||||
### Phase 0 — Pre-Execution (complete — tagged v0.1.10, release #443)
|
||||
### Phase 0 — Pre-Execution (active)
|
||||
|
||||
**Branch:** `phase/00-pre-execution` → merged to `milestone/v0.5-live-assist`
|
||||
**Branch:** `phase/00-pre-execution` → `milestone/v0.5-live-assist`
|
||||
**Ship target:** `v0.1.10` (next available patch on the v0.1.x line — NFR/docs milestone type)
|
||||
**Status:** complete (v0.1.10 tagged, Gitea release #443 created)
|
||||
**Status:** active (SPECIFY complete → CLARIFY → RESEARCH → IDEATE → PLAN → GRILL → SHIP)
|
||||
|
||||
Pipeline stages: SPECIFY → CLARIFY → RESEARCH → **IDEATE** (--ideate flag) → PLAN → GRILL → SHIP
|
||||
|
||||
@@ -28,31 +28,7 @@ Pipeline stages: SPECIFY → CLARIFY → RESEARCH → **IDEATE** (--ideate flag)
|
||||
- ARCHITECTURE.md (Live Assist mode added to v0.4 topology — assist voice loop + context-binding + guardrail extension)
|
||||
- PERSONAS.md (v0.5 roster — voice-engineer reactivated for hands-free/latency; backend-engineer for context-binding + guardrails; security-engineer retained for REQ-ASSIST-03 safety surface)
|
||||
- GRILL-v0.5.md (adversarial review — real-customer interaction warrants grill)
|
||||
- Phase 1 + Phase 2 plans (vertical slices with wave ordering)
|
||||
|
||||
### Phase 1 — Assist Core + Guardrail (complete — tagged v0.1.11, release #451)
|
||||
|
||||
**Branch:** `phase/01-assist-core-guardrail` → merged to `milestone/v0.5-live-assist`
|
||||
**Ship target:** `v0.1.11` (patch release, feature milestone type)
|
||||
**Status:** complete (v0.1.11 tagged, Gitea release #451 created; 409 pass, 36 skip, 0 fail; 12/16 REQ covered; APPROVE_WITH_NOTES, 5 P1+ flagged; G-049 + G-067 MUSTs resolved)
|
||||
|
||||
**Goal:** The assist voice loop + 3-layer guardrail + context-binding + shift-bounded session model + warm WebRTC + tap-to-talk client. The safety-critical, on-voice-path surface. Independently shippable (a learner can start a shift, tap-to-talk, get coaching with guardrails, end the shift).
|
||||
|
||||
### Phase 2 — Integration + Tech-Debt + NFR Measurement (complete — tagged v0.1.12, release #452)
|
||||
|
||||
**Branch:** `phase/02-integration-techdebt-nfr` → merged to `milestone/v0.5-live-assist`
|
||||
**Ship target:** `v0.1.12` (patch release, feature milestone type)
|
||||
**Status:** complete (v0.1.12 tagged, Gitea release #452 created; 469 pass, 45 skip, 0 fail; 4/16 REQ covered; APPROVE_WITH_NOTES, 3 P1+ flagged; 8 v0.4 P1+ tech-debt findings addressed)
|
||||
|
||||
**Goal:** Cohort aggregation assist metrics (5 new metrics, no schema change), assist per-turn cost tracking + C-3 budget check, NFR measurement (p95 latency ≤650ms pilot, guardrail FP<5% / FN measured + trended nightly), v0.4 P1+ tech-debt wave (8 findings).
|
||||
|
||||
### Final Phase (P3) — Review + Ship (complete — tagged v0.1.13, release created, merged to main)
|
||||
|
||||
**Branch:** `phase/03-final-review-ship` → merged to `milestone/v0.5-live-assist` → merged to `main`
|
||||
**Ship target:** final patch = v0.5 milestone release
|
||||
**Status:** complete (v0.1.13 tagged, Gitea release created, merged to main; review APPROVE_WITH_NOTES, 1 P0 fixed, 8 P1+ flagged for v0.6; audit NEEDS_ATTENTION — 12 stale-status fields advanced, no critical issues)
|
||||
|
||||
**Goal:** Multi-persona code review, project audit, milestone merge to main, milestone release. P0 fix: guardrail processor must buffer LLM text before TTS (REQ-ASSIST-03 safety-critical). ESCALATION-01 (PIPEDA consent-law review) remains OPEN for human legal review before assist surface go-live.
|
||||
- Phase 1 plan (vertical slices with wave ordering)
|
||||
|
||||
## v0.4 Milestone (complete — reference)
|
||||
|
||||
|
||||
@@ -1,503 +0,0 @@
|
||||
# P2 Verification Report — v0.5 Live Assist (Phase 2: Integration + Tech-Debt + NFR Measurement)
|
||||
|
||||
> **Phase:** P2 (Integration + Tech-Debt + NFR Measurement)
|
||||
> **Milestone:** v0.5
|
||||
> **Branch:** `phase/02-integration-techdebt-nfr`
|
||||
> **Status:** verify — 4-layer verification complete
|
||||
> **Date:** 2026-08-04
|
||||
> **Verifier:** ci-code-reviewer (correctness, testing, security, performance, maintainability, adversarial)
|
||||
> **REQ-IDs covered (4):** REQ-NFR-ASSIST-01, REQ-IDEATE-04, REQ-IDEATE-06, REQ-IDEATE-07
|
||||
|
||||
---
|
||||
|
||||
## Verdict: APPROVE_WITH_NOTES
|
||||
|
||||
P2 (Integration + Tech-Debt + NFR Measurement) passes all 4 verification layers.
|
||||
All 469 tests pass (45 skipped — all env-gated: Postgres + live voice-service keys
|
||||
+ W3C interop), 0 failures. The 4 P2 REQ-IDs are covered by 69 new tests (60 run in
|
||||
CI without Postgres; 9 PG-skipped). All 8 v0.4 P1+ findings (REVIEW.md) are
|
||||
addressed with fixes + tests. All 5 P1-VERIFIER findings (VERIFY-P1-v0.5.md) are
|
||||
reviewed with documented dispositions. No P0 issues found. 3 P1+ findings are
|
||||
flagged for post-hoc review at the final phase (none block ship).
|
||||
|
||||
---
|
||||
|
||||
## Layer 1: Structural Verification — PASS
|
||||
|
||||
### 1.1 File existence (all P2 plan files present on disk)
|
||||
|
||||
| File | Status |
|
||||
|------|--------|
|
||||
| `server/assist/latency_metrics.py` | ✅ exists (131 lines — AssistLatencyMetrics: p95/p50/p99 + D-072 summary) |
|
||||
| `server/assist/guardrail_metrics.py` | ✅ exists (274 lines — GuardrailMetrics: FP/FN rates + nightly_trend) |
|
||||
| `server/assist/budget_check.py` | ✅ exists (77 lines — check_c3_budget: C-3 diagnostic) |
|
||||
| `server/cohort/learner_cache.py` | ✅ exists (259 lines — SQLite cache persistence for P1+ #7) |
|
||||
| `server/cohort/aggregator.py` | ✅ extended (+180 lines — _aggregate_assist branch, 5 core metrics + p95 + cost) |
|
||||
| `server/cost.py` | ✅ extended (+52 lines — derive_assist_turn_cost, LLM + Piper TTS) |
|
||||
| `server/assist/session.py` | ✅ extended (+28 lines — latency_metrics + assist_cost_cents accumulator) |
|
||||
| `server/auth/cookies.py` | ✅ extended (+18 lines — cookie-secret <32 bytes WARNING, P1+ #3) |
|
||||
| `server/auth/routes.py` | ✅ extended (+11 lines — argon2id offload to asyncio.to_thread, P1+ #1) |
|
||||
| `server/cohort/nightly.py` | ✅ extended (+29 lines — ZoneInfo("America/Winnipeg") + cache clear, P1+ #6/#7) |
|
||||
| `server/operator/credentials.py` | ✅ extended (+8 lines — credential revocation audit log, P1+ #5) |
|
||||
| `server/operator/cohort.py` | ✅ extended (+23 lines — assist_shifts_count + assist_turns_count in cohort view) |
|
||||
| `server/operator/failure_patterns.py` | ✅ extended (+21 lines — assist_guardrail_block_rate safety signal) |
|
||||
| `server/operator/mastery.py` | ✅ extended (+10 lines — D-063 comment: assist metrics excluded from mastery view) |
|
||||
| `db/pg_store.py` | ✅ extended (+33 lines — set_credential_status enum validation + parameterized queries, P1+ #4/#8) |
|
||||
| `tests/test_nfr_measurement.py` | ✅ exists (290 lines, 12 tests — SLICE-09) |
|
||||
| `tests/test_cohort_assist_aggregation.py` | ✅ exists (416 lines, 19 tests — SLICE-10) |
|
||||
| `tests/test_assist_cost.py` | ✅ exists (228 lines, 13 tests — SLICE-11) |
|
||||
| `tests/test_credential_status_techdebt.py` | ✅ exists (106 lines, 5 tests — SLICE-12, TASK-12-03) |
|
||||
| `tests/test_p2_assist_integration.py` | ✅ exists (353 lines, 9 tests — SLICE-12, TASK-12-05, PG-skipped) |
|
||||
| `tests/test_auth.py` | ✅ extended (+279 lines, +7 tests — SLICE-12, TASK-12-02 + TASK-12-04) |
|
||||
| `tests/test_cohort_nightly.py` | ✅ extended (+65 lines, +4 tests — SLICE-12, TASK-12-04) |
|
||||
|
||||
### 1.2 Import resolution
|
||||
|
||||
```
|
||||
python3 -c "import server.assist.latency_metrics; import server.assist.guardrail_metrics;
|
||||
import server.assist.budget_check; import server.cohort.learner_cache;
|
||||
import server.cohort.aggregator; import server.cost; import server.auth.cookies;
|
||||
import server.auth.routes; import server.cohort.nightly; import server.operator.credentials;
|
||||
import db.pg_store"
|
||||
→ ALL IMPORTS OK
|
||||
```
|
||||
|
||||
All declared exports resolve: `AssistLatencyMetrics`, `TARGET_MS`, `PILOT_TOLERANCE_MS`,
|
||||
`GuardrailMetrics`, `FP_TARGET`, `FN_TARGET`, `check_c3_budget`, `C3_TARGET_USD`,
|
||||
`derive_assist_turn_cost`, `_aggregate_assist`, `_load_learner_cache`,
|
||||
`_save_learner_cache`, `_clear_learner_cache`, `_count_distinct_learners` — all importable.
|
||||
|
||||
### 1.3 No stubs / TODOs / placeholders
|
||||
|
||||
`grep -rE "TODO|FIXME|XXX|NotImplemented|pass # stub|raise NotImplementedError"` in
|
||||
`server/assist/latency_metrics.py`, `server/assist/guardrail_metrics.py`,
|
||||
`server/assist/budget_check.py`, `server/cohort/learner_cache.py` → **No matches.**
|
||||
All P2 code is fully implemented.
|
||||
|
||||
---
|
||||
|
||||
## Layer 2: Behavioral Verification — PASS
|
||||
|
||||
### 2.1 Full test suite
|
||||
|
||||
```
|
||||
python3 -m pytest tests/ --tb=no
|
||||
→ 469 passed, 45 skipped, 5 warnings in 107.28s
|
||||
```
|
||||
|
||||
**Matches the expected baseline exactly: 469 passed, 45 skipped, 0 failed.**
|
||||
Breakdown: 409 (P1 baseline) + 60 new P2 tests (run in CI) = 469 passed.
|
||||
36 (P1 skips) + 9 new P2 PG-skipped = 45 skipped. All skips are env-gated
|
||||
(PRAXIS_PG_DSN not set → 33 Postgres tests; live voice-service keys not
|
||||
provisioned → 11 live audio tests; PRAXIS_RUN_VC_INTEROP not set → 1 interop test).
|
||||
No unexpected skips or failures.
|
||||
|
||||
### 2.2 P2-specific tests
|
||||
|
||||
```
|
||||
python3 -m pytest tests/test_nfr_measurement.py tests/test_cohort_assist_aggregation.py
|
||||
tests/test_assist_cost.py tests/test_credential_status_techdebt.py
|
||||
tests/test_p2_assist_integration.py tests/test_auth.py tests/test_cohort_nightly.py
|
||||
→ 87 passed, 9 skipped, 2 warnings in 9.78s
|
||||
```
|
||||
|
||||
**69 new P2 tests** (60 run + 9 PG-skipped). Breakdown:
|
||||
|
||||
| Test file | Tests | Coverage |
|
||||
|-----------|-------|----------|
|
||||
| test_nfr_measurement.py | 12 | AssistLatencyMetrics p95/p50/p99, D-072 within_target/within_pilot, boundary 650, empty metrics; GuardrailMetrics FP/FN/adversarial rates, nightly_trend on mock turns, excludes practice |
|
||||
| test_cohort_assist_aggregation.py | 19 | _aggregate_assist 5 core metrics + p95 + cost, k-anon 9/10 boundary, idempotent, block_rate=blocks/turns, zero-turns no div-by-zero, practice branch unchanged, no PII in upserts, hook dispatch, dashboard endpoints return assist rows, mastery excludes assist (D-063) |
|
||||
| test_assist_cost.py | 13 | derive_assist_turn_cost (LLM + Piper), Piper zero TTS, Cartesia fallback, same rates as derive_cost, derive_cost unchanged, shift-end sum, check_c3_budget within/exceeds/with-practice/zero/diagnostic, C-3 target $3 |
|
||||
| test_credential_status_techdebt.py | 5 | set_credential_status revoked parameterized, active clears revoked_at, invalid → ValueError, no f-string, revoke+reactivate round-trip |
|
||||
| test_p2_assist_integration.py | 9 (PG-skipped) | k-anon threshold e2e, p95 in aggregates, cost in session_outcome, C-3 check, cache survives restart, cookie-secret warning, credential enum, argon2id offloaded, no per-learner data |
|
||||
| test_auth.py | +7 | cookie-secret short/32/long warning, argon2id verify offloaded, rehash offloaded, 429 mock test, credential revocation audit log |
|
||||
| test_cohort_nightly.py | +4 | zoneinfo America/Winnipeg, summer CDT UTC-5, winter CST UTC-6, spring-forward transition |
|
||||
|
||||
### 2.3 REQ coverage matrix (4 P2 REQs)
|
||||
|
||||
| REQ-ID | Covered | Test file(s) | Evidence |
|
||||
|--------|---------|--------------|----------|
|
||||
| REQ-NFR-ASSIST-01 | ✅ covered | test_nfr_measurement.py, test_p2_assist_integration.py | p95 assist-turn latency measurement — AssistLatencyMetrics (p95/p50/p99 + D-072 within_target/within_pilot); assist_p95_latency_ms in cohort aggregates |
|
||||
| REQ-IDEATE-04 | ✅ covered | test_nfr_measurement.py, test_guardrail_tuning.py (P1 carry-forward) | Measurable NFR targets — p95 ≤650ms pilot (D-072) + guardrail FP<5% / FN<5% measured + trended nightly (GuardrailMetrics.nightly_trend) |
|
||||
| REQ-IDEATE-06 | ✅ covered | test_credential_status_techdebt.py, test_auth.py, test_cohort_nightly.py, test_p2_assist_integration.py | v0.4 P1+ tech-debt wave — all 8 findings addressed (see §4 below) |
|
||||
| REQ-IDEATE-07 | ✅ covered | test_assist_cost.py, test_p2_assist_integration.py | Assist per-turn cost tracking + C-3 budget check — derive_assist_turn_cost + check_c3_budget (diagnostic, not enforced per D-012) |
|
||||
|
||||
**All 4 P2 REQ-IDs are covered by at least one test file. No gaps.**
|
||||
|
||||
---
|
||||
|
||||
## Layer 3: Security Verification (STRIDE) — PASS
|
||||
|
||||
**Scope:** P2 additions — `server/assist/latency_metrics.py`, `server/assist/guardrail_metrics.py`,
|
||||
`server/assist/budget_check.py`, `server/cohort/learner_cache.py`, `server/cohort/aggregator.py`
|
||||
(assist branch), `server/cost.py` (assist turn cost), `server/auth/cookies.py` (cookie-secret),
|
||||
`db/pg_store.py` (credential status), `server/auth/routes.py` (argon2id offload),
|
||||
`server/cohort/nightly.py` (zoneinfo), `server/operator/credentials.py` (audit log).
|
||||
|
||||
### 3.1 Tech-debt security fixes — do they close the v0.4 P1+ security findings?
|
||||
|
||||
| v0.4 P1+ | Fix | Closes? |
|
||||
|----------|-----|---------|
|
||||
| #1 (argon2id blocking) | `asyncio.to_thread(verify_password, ...)` + `asyncio.to_thread(hash_password, ...)` in login handler | ✅ YES — argon2id no longer blocks the event loop (verified by test_login_argon2id_offloaded_to_thread + test_login_rehash_offloaded_to_thread) |
|
||||
| #3 (cookie-secret length) | `elif len(secret) < 32: logger.warning(...)` in cookies.py | ✅ YES — short secret logs WARNING with remediation guidance (verified by 3 cookie-secret tests) |
|
||||
| #4 (credential status enum) | `if status not in ("active", "revoked"): raise ValueError` in pg_store.py | ✅ YES — invalid status raises ValueError before the query (verified by test_set_credential_status_invalid_raises_value_error) |
|
||||
| #5 (revocation audit log) | `log.info("credential revoked: operator=%s cred_id=%s", op.id, cred_id)` in credentials.py | ✅ YES — revocation event logged with operator + cred_id (verified by test_credential_revocation_logs_audit_event) |
|
||||
| #8 (f-string SQL) | Two explicit parameterized queries (no f-string interpolation) in pg_store.py | ✅ YES — no f-string in SQL; $1/$2 bound parameters (verified by test_set_credential_status_no_fstring_in_sql) |
|
||||
|
||||
### 3.2 Aggregation cache persistence (P1+ #7) — new info-disclosure vector?
|
||||
|
||||
**No.** The `cohort_learner_cache.db` SQLite file stores `(path, window_start, learner_ref)`
|
||||
tuples. The `learner_ref` is an opaque string (D-031 — not raw PII, just an opaque
|
||||
identifier for distinct counting). The cache file lives next to `praxis.db` (D-007 —
|
||||
learner-local SQLite, not Postgres). The cache is on the learner's device, not in the
|
||||
operator tier. This is consistent with the existing architecture — **no new
|
||||
info-disclosure vector**.
|
||||
|
||||
The cache file is created with `CREATE TABLE IF NOT EXISTS` (idempotent). If the file
|
||||
is corrupted, the try/except catches the error + returns empty (graceful degradation).
|
||||
The cache is cleared by the nightly job after reconciliation (no stale entries
|
||||
accumulate).
|
||||
|
||||
### 3.3 Cost tracking — does it log sensitive data?
|
||||
|
||||
**No.** The cost tracking is pure computation:
|
||||
- `derive_assist_turn_cost()` takes LLM token counts + TTS character counts → returns
|
||||
`CostBreakdown` with `derived_cents`. No PII in, no PII out.
|
||||
- `check_c3_budget()` takes usage estimates (turns/shift, shifts/month, cost/turn) →
|
||||
returns a diagnostic dict. No PII.
|
||||
- The `assist_cost_cents` in `session_outcome` is an integer (cents) — not PII.
|
||||
- The cost module logs nothing (it's a pure function). The only logs in the P2
|
||||
modules are: learner_cache logs counts + db_path (not learner refs);
|
||||
guardrail_metrics logs turn counts (not tts_text); credentials logs operator id +
|
||||
cred_id (the audit event, not PII).
|
||||
|
||||
**Cost is tokens + cents, not PII.** Correct.
|
||||
|
||||
### 3.4 Nightly trend — tts_text in fn_candidates
|
||||
|
||||
The `nightly_trend()` function reads `tts_text` from the turns table and includes it
|
||||
(truncated to 200 chars) in the `fn_candidates` dict. The `tts_text` is the AI's
|
||||
coaching response (not customer PII — the `asr_text` is redacted via `redact_pii()`
|
||||
before storage per REQ-IDEATE-05). The `fn_candidates` are returned to the caller
|
||||
(the nightly job), not logged directly by this module. The `log.info` call at line 228
|
||||
logs only counts (total_turns, blocked, allowed_coaching, allowed_neutral,
|
||||
fn_candidates count) — not the tts_text itself.
|
||||
|
||||
**Disposition:** LOW — the tts_text is AI-generated coaching, not customer PII; the
|
||||
fn_candidates are diagnostic (not stored in Postgres); the log contains only counts.
|
||||
|
||||
### STRIDE Summary
|
||||
|
||||
| Threat | Severity | Disposition |
|
||||
|--------|----------|-------------|
|
||||
| Spoofing | LOW | accept (D-007 single-learner; operator auth unchanged from v0.4) |
|
||||
| Tampering | LOW | accept (credential status enum validation prevents invalid states; parameterized queries prevent SQL injection) |
|
||||
| Repudiation | LOW | accept (credential revocation audit log added; cost tracking is diagnostic) |
|
||||
| Info Disclosure | LOW | accept (cache stores opaque learner_ref, not PII; cost is tokens+cents; nightly_trend tts_text is AI-generated, not customer PII) |
|
||||
| Denial of Service | LOW | accept (argon2id offloaded to thread; nightly_trend is off-voice-path) |
|
||||
| Elevation of Privilege | LOW | accept (D-063 enforced — assist metrics excluded from mastery view) |
|
||||
|
||||
**Layer 3 verdict: PASS** (no HIGH or MEDIUM-severity threats; all LOW accepted).
|
||||
|
||||
---
|
||||
|
||||
## Layer 4: Quality Verification (Multi-persona code review) — PASS
|
||||
|
||||
### Correctness
|
||||
|
||||
- **p95 computation (nearest-rank):** `_percentile(values, 95.0)` uses
|
||||
`rank = ceil(0.95 * n)`, `idx = rank - 1`. Verified: 100 records (80 at 500..579,
|
||||
15 at 610..624, 5 at 700..704) → p95 = 624.0 (index 94), p99 = 703.0 (index 98).
|
||||
Correct. The D-072 thresholds are correctly applied: `within_target = (p95 < 600)`,
|
||||
`within_pilot = (p95 <= 650)`. The boundary test (p95 == 650 → within_pilot=True,
|
||||
within_target=False) confirms the ≤ vs < distinction. ✅
|
||||
- **FP/FN rate computation:** `false_positive_rate()` counts coaching responses
|
||||
blocked (allowed=False when should be True). `false_negative_rate()` counts direct
|
||||
answers allowed (allowed=True when should be False). Verified: FP 0.0% (0/50),
|
||||
FN 0.0% (0/51), adversarial FN 13.3% (4/30). The rates are `misclassified / total`
|
||||
with `total = 0 → rate = 0.0` (no division by zero). ✅
|
||||
- **C-3 budget check math:** `turns_per_month = turns_per_shift * shifts_per_month`;
|
||||
`monthly_assist_cost_usd = (turns_per_month * cost_per_turn_cents) / 100.0` (cents
|
||||
→ USD); `total_with_practice = monthly_assist_cost + practice_cost`;
|
||||
`within_budget = total <= 3.0`; `flag = not within_budget`. Verified: 20×20×0.05¢
|
||||
= $0.20 (within), 100×30×0.15¢ = $4.50 (exceeds). The cents→USD conversion is
|
||||
correct (divide by 100). ✅
|
||||
- **D-063 enforcement (assist ≠ mastery):** `_aggregate_assist()` computes NO mastery
|
||||
metrics (no gate_open_rate, no median_mastery_score, no rubric_criterion_mean).
|
||||
The mastery view (`server/operator/mastery.py`) excludes assist metrics
|
||||
(`_is_mastery_metric` returns False for assist_*). Verified by
|
||||
`test_aggregate_assist_no_mastery_metrics` + `test_mastery_endpoint_excludes_assist_metrics`. ✅
|
||||
- **k-anon suppression (assist):** The assist branch uses the SAME `K_ANON_THRESHOLD
|
||||
= 10` + the SAME `_bump_active_learners` as practice. Boundary tests: 9 learners →
|
||||
suppressed, 10 → not suppressed. ✅
|
||||
- **block_rate = blocks / turns (no div-by-zero):** `block_rate = (blocks / turn_count)
|
||||
if turn_count > 0 else 0.0`. Verified by `test_assist_zero_turns_block_rate_is_zero`. ✅
|
||||
- **ZoneInfo DST:** `CT = ZoneInfo("America/Winnipeg")` correctly handles CST (UTC-6)
|
||||
in winter + CDT (UTC-5) in summer. The `now.astimezone(CT)` conversion is correct.
|
||||
Verified by summer/winter/spring-forward tests. ✅
|
||||
- **Credential status enum:** `if status not in ("active", "revoked"): raise ValueError`.
|
||||
The 'active' status clears `revoked_at = NULL` (re-activation). Verified by 5 tests. ✅
|
||||
|
||||
### Testing
|
||||
|
||||
- **69 new P2 tests** — comprehensive coverage of all 4 P2 REQ-IDs.
|
||||
- **Coverage gaps:** None identified for P2 scope. The 9 PG-skipped integration tests
|
||||
have mock-based equivalents (test_cohort_assist_aggregation.py covers the same
|
||||
logic without Postgres). The 429 mock test (P1+ #2) fills the CI-coverage gap.
|
||||
- **Flaky tests:** None observed. The zoneinfo tests use fixed dates (2026-08-04,
|
||||
2027-01-15, 2027-03-14) — no time mocking issues. The cache-survives-restart test
|
||||
(PG-skipped) uses a temp file + clears the in-memory cache to simulate restart.
|
||||
- **Edge cases covered:** empty latency metrics (p95=None), zero turns (block_rate=0),
|
||||
zero usage (cost=0), p95 exactly 650 (within_pilot=True boundary), invalid credential
|
||||
status (ValueError), short cookie secret (WARNING), corrupted cache file (graceful
|
||||
degradation via try/except).
|
||||
|
||||
### Security
|
||||
|
||||
- **Input validation:** `set_credential_status` validates the status enum before the
|
||||
query. `check_c3_budget` takes numeric inputs (no injection vector). The
|
||||
`nightly_trend` query uses parameterized SQL (`t.created_at >= ?` with `(cutoff,)`).
|
||||
- **SQL injection:** The f-string SQL in `set_credential_status` (P1+ #8) is replaced
|
||||
with two explicit parameterized queries. No f-string interpolation in any SQL.
|
||||
- **Secrets:** No secrets in P2 code. The cookie-secret validation logs a WARNING but
|
||||
does not reject the secret (backward compat — pilot). Post-pilot this should be a
|
||||
hard error.
|
||||
|
||||
### Performance
|
||||
|
||||
- **nightly_trend is off-voice-path:** The `GuardrailMetrics.nightly_trend()` is called
|
||||
by the nightly job (server/cohort/nightly.py), NOT by the assist pipeline. The assist
|
||||
pipeline does NOT call nightly_trend. The nightly job runs at 03:00 CT (low activity).
|
||||
The nightly_trend reads from SQLite (local, not Postgres) — no network latency. ✅
|
||||
- **Aggregation hook is off-voice-path:** The `_aggregate_assist` function is called by
|
||||
the on-session-end hook (asyncio.create_task — fire-and-forget), NOT on the voice
|
||||
path. The C-8 latency budget is unaffected. ✅
|
||||
- **Cache persistence I/O:** The `_bump_active_learners` function calls
|
||||
`_load_learner_cache` (on first call per path/window) + `_save_learner_cache` (on
|
||||
every call). This is O(n) per hook where n = total cached learners. For pilot scale
|
||||
(~100 learners), this is <10ms — negligible. For scale, this would be a performance
|
||||
concern (see P1+ finding below). The I/O is off-voice-path (async fire-and-forget). ✅
|
||||
- **Regex compilation:** The guardrail regex patterns are compiled at module load (not
|
||||
per-call). The nightly_trend re-runs the guardrail on each turn — O(turns) per night.
|
||||
For pilot scale (~400 turns/month), this is <1s — negligible. ✅
|
||||
|
||||
### Maintainability
|
||||
|
||||
- **Assist aggregation follows existing cohort patterns:** `_aggregate_assist` uses the
|
||||
SAME `_bump_active_learners`, `_upsert_cell`, `_running_mean`, `_rolling_window`,
|
||||
`K_ANON_THRESHOLD` as `_aggregate_practice`. The new `_bump_assist_turns` helper
|
||||
follows the `_bump_counter` pattern. The branch dispatch in `aggregate_session` is
|
||||
clean (if session_type == 'assist' → _aggregate_assist, else → _aggregate_practice). ✅
|
||||
- **Naming:** `AssistLatencyMetrics`, `GuardrailMetrics`, `check_c3_budget`,
|
||||
`derive_assist_turn_cost`, `_aggregate_assist` — descriptive, follow the existing
|
||||
v0.1-v0.4 naming conventions. ✅
|
||||
- **Structure:** The P2 modules follow the existing package patterns
|
||||
(`server/assist/`, `server/cohort/`). The `learner_cache.py` is a new module in
|
||||
`server/cohort/` (the cache persistence is a cohort concern). ✅
|
||||
- **Coupling:** The latency metrics + cost tracking are loosely coupled to the
|
||||
AssistSession (injected via attributes). The guardrail metrics depend on the
|
||||
LiveAssistGuardrail (imported, not injected — acceptable for a diagnostic). The
|
||||
learner_cache depends on aiosqlite (direct connection, not via PraxisStore — a
|
||||
deliberate choice documented in the code). ✅
|
||||
- **Documentation:** Every P2 module has a comprehensive docstring explaining the
|
||||
design decisions (D-062, D-063, D-068, D-072, D-012, C-3, REQ-IDEATE-04/06/07
|
||||
references). Every test file has a docstring mapping to REQ-IDs + tasks. ✅
|
||||
|
||||
### Adversarial
|
||||
|
||||
- **Can the budget check be gamed?** `check_c3_budget` is a pure computation with
|
||||
explicit parameters (turns_per_shift, shifts_per_month, cost_per_turn_cents). The
|
||||
caller provides the parameters. A learner can't directly control the token count
|
||||
(the LLM generates the response). A learner could make more turns (increasing cost),
|
||||
but that's legitimate usage. The budget check is diagnostic (not enforced per
|
||||
D-012) — gaming it doesn't matter (it's just a measurement). ✅
|
||||
- **Can the latency metrics be spoofed?** The `LatencyRecord` is created by the
|
||||
`LatencyObserver` in the pipeline (server/latency.py). The learner doesn't control
|
||||
the latency measurement — it's measured server-side. The `AssistLatencyMetrics`
|
||||
collects records from the pipeline (the `record()` method is called by the pipeline
|
||||
code, not the API). A learner can't inject fake records. ✅
|
||||
- **Can the cache persistence be corrupted?** The cache SQLite file uses
|
||||
`INSERT OR IGNORE` (idempotent). If the file is corrupted, the try/except catches
|
||||
the error + returns empty (graceful degradation). The nightly job reconciles from
|
||||
`mastery_gate_events` (the source of truth) + clears the cache. A learner with
|
||||
filesystem access could delete the cache file — but the cache is an intermediate
|
||||
state (the nightly job is the source of truth). ✅
|
||||
- **Can the credential status enum be bypassed?** The `set_credential_status` function
|
||||
validates the status before the query. The only caller is the
|
||||
`revoke_credential` endpoint, which always passes 'revoked'. A future caller passing
|
||||
an invalid status gets `ValueError`. ✅
|
||||
|
||||
---
|
||||
|
||||
## 8 v0.4 P1+ Findings Verification (REVIEW.md — all addressed)
|
||||
|
||||
| P1+ ID | Finding | P2 Fix | Test | Verified |
|
||||
|--------|---------|--------|------|----------|
|
||||
| #1 | Argon2id blocking event loop | `asyncio.to_thread(verify_password, ...)` + `asyncio.to_thread(hash_password, ...)` in `server/auth/routes.py` | `test_login_argon2id_offloaded_to_thread`, `test_login_rehash_offloaded_to_thread` | ✅ YES |
|
||||
| #2 | Rate limit 429 not tested in mock path | Mock-based 429 test (6th attempt → 429) in `tests/test_auth.py` | `test_login_rate_limit_429_after_5_attempts` | ✅ YES |
|
||||
| #3 | No PRAXIS_COOKIE_SECRET length validation | `elif len(secret) < 32: logger.warning(...)` in `server/auth/cookies.py` | `test_cookie_secret_short_logs_warning_accepted`, `test_cookie_secret_32_bytes_no_warning`, `test_cookie_secret_long_no_warning` | ✅ YES |
|
||||
| #4 | set_credential_status status not validated | `if status not in ("active", "revoked"): raise ValueError` in `db/pg_store.py` | `test_set_credential_status_invalid_raises_value_error` | ✅ YES |
|
||||
| #5 | Credential revocation lacks audit log | `log.info("credential revoked: operator=%s cred_id=%s", op.id, cred_id)` in `server/operator/credentials.py` | `test_credential_revocation_logs_audit_event` | ✅ YES |
|
||||
| #6 | Nightly scheduler fixed UTC-5 offset | `CT = ZoneInfo("America/Winnipeg")` in `server/cohort/nightly.py` | `test_nightly_scheduler_uses_zoneinfo_america_winnipeg`, `test_nightly_scheduler_dst_summer_cdt`, `test_nightly_scheduler_dst_winter_cst`, `test_nightly_scheduler_dst_transition_spring_2027` | ✅ YES |
|
||||
| #7 | Aggregation cache lost on restart | SQLite `cohort_learner_cache` table persistence in `server/cohort/learner_cache.py`; `_load_learner_cache` on startup, `_save_learner_cache` on each session, `_clear_learner_cache` by nightly job | `test_p2_techdebt_aggregation_cache_survives_restart` (PG-skipped) | ✅ YES |
|
||||
| #8 | set_credential_status f-string SQL | Two explicit parameterized queries (no f-string) in `db/pg_store.py` | `test_set_credential_status_no_fstring_in_sql`, `test_set_credential_status_revoked_uses_parameterized_query` | ✅ YES |
|
||||
|
||||
**All 8 v0.4 P1+ findings are addressed with a fix + at least one test.**
|
||||
|
||||
---
|
||||
|
||||
## 5 P1-VERIFIER Findings Review (VERIFY-P1-v0.5.md — all reviewed)
|
||||
|
||||
| P1+ ID | Finding | P2 Disposition | Resolved? |
|
||||
|--------|---------|----------------|-----------|
|
||||
| P1-1 (MEDIUM — Info Disclosure) | PII retention cleanup not scheduled | **Deferred to v0.6** — the 30-day retention is documented in `get_pii_policy()` + the consent disclosure (D-070) is the primary mitigation. The nightly cleanup task is not a P2 tech-debt item (the P2 plan covers the 8 v0.4 P1+ findings, not P1-VERIFIER findings). Left for v0.6 nightly cleanup. | ✅ Reviewed (deferred with rationale) |
|
||||
| P1-2 (LOW — Security) | Scenario-tag prompt injection (unsanitized input) | **Deferred to v0.6** — single-learner self-injection only (D-007); Layer 2 regex still filters output; coaching instruction is a fixed prefix. Not in the P2 plan scope. | ✅ Reviewed (deferred with rationale) |
|
||||
| P1-3 (LOW — Correctness) | end_session_assist doesn't persist turn/block counts | **Mitigated** — the counts flow to the aggregation hook via `session_outcome` (the in-memory `AssistSession` holds them; the aggregation reads them). The restart edge case is mitigated by the cache persistence (TASK-12-01 — the cache survives restart). | ✅ Resolved (mitigated by cache persistence) |
|
||||
| P1-4 (LOW — Maintainability) | WebRTC reconnect offer-event not wired | **Deferred to v0.6** — the reconnect state machine is tested + correct; the shift is NOT auto-ended on disconnect; the 8h auto-end still fires. Not in the P2 plan scope. | ✅ Reviewed (deferred with rationale) |
|
||||
| P1-5 (LOW — Testing) | No concurrent shift-start race test | **Deferred to v0.6** — single-learner (D-007); no concurrent requests expected in pilot; the DB-level mode-conflict guard catches concurrent starts. Not in the P2 plan scope. | ✅ Reviewed (deferred with rationale) |
|
||||
|
||||
**All 5 P1-VERIFIER findings are reviewed with documented dispositions:**
|
||||
- P1-3 is **resolved** (mitigated by the cache persistence from TASK-12-01).
|
||||
- P1-1, P1-2, P1-4, P1-5 are **deferred to v0.6** with documented rationale (low risk,
|
||||
documented mitigations present, not in P2 plan scope). None block ship.
|
||||
|
||||
---
|
||||
|
||||
## P0 Fixes Applied
|
||||
|
||||
**None.** No P0 issues (broken tests, missing REQ coverage, security holes, logic
|
||||
errors causing incorrect behavior) were found across any of the 4 verification layers.
|
||||
The P2 implementation is correct, tested, and secure. No auto-fixes were necessary.
|
||||
|
||||
---
|
||||
|
||||
## P1+ Findings Flagged for Post-Hoc Review
|
||||
|
||||
### P2-1 (LOW — Performance): Cache I/O on every session-end hook
|
||||
|
||||
**File:** `server/cohort/aggregator.py:320-355` (`_bump_active_learners`)
|
||||
**Issue:** The `_bump_active_learners` function calls `_load_learner_cache` (on first
|
||||
call per path/window) + `_save_learner_cache` (on every call). The `_load_learner_cache`
|
||||
loads the ENTIRE cache from SQLite (all rows across all path/window pairs), not just
|
||||
the learners for the specific (path, window). The `_save_learner_cache` writes to
|
||||
SQLite on every session-end hook.
|
||||
**Risk:** LOW — the hook is off-voice-path (async fire-and-forget); pilot scale
|
||||
(~100 learners) is <10ms per hook; the nightly job reconciles. For scale (1000+
|
||||
learners), this would be a performance concern.
|
||||
**Recommendation:** (a) Load only the learners for the specific (path, window) — use
|
||||
`_count_distinct_learners` instead of `_load_learner_cache` for the seed. (b) Batch
|
||||
the saves (write every 5 minutes or on shift-end, not on every session). v0.6.
|
||||
**Disposition:** Flag for v0.6 post-hoc review.
|
||||
|
||||
### P2-2 (LOW — Maintainability): nightly_trend bypasses PraxisStore API
|
||||
|
||||
**File:** `server/assist/guardrail_metrics.py:153-167` (`nightly_trend`)
|
||||
**Issue:** The `nightly_trend` function reads from the turns table via a direct
|
||||
`aiosqlite.connect(store.db_path)` connection, bypassing the `PraxisStore` API. This
|
||||
is a deliberate choice (documented: "we read directly via aiosqlite to avoid adding
|
||||
a method to the store surface for a diagnostic"), but it means the store abstraction
|
||||
is leaked.
|
||||
**Risk:** LOW — the nightly_trend is a diagnostic (off-voice-path, one-off read per
|
||||
night). The direct connection is closed after the read. No correctness issue.
|
||||
**Recommendation:** Add a `list_recent_assist_turns(hours: int)` method to
|
||||
`PraxisStore` in v0.6 to maintain the abstraction. P2 finding.
|
||||
**Disposition:** Flag for v0.6 post-hoc review.
|
||||
|
||||
### P2-3 (LOW — Security): nightly_trend fn_candidates include truncated tts_text
|
||||
|
||||
**File:** `server/assist/guardrail_metrics.py:202, 210` (`fn_candidates`)
|
||||
**Issue:** The `fn_candidates` dict includes `tts_text` (truncated to 200 chars). The
|
||||
`tts_text` is the AI's coaching response (not customer PII — the `asr_text` is
|
||||
redacted via `redact_pii()` before storage per REQ-IDEATE-05). The `fn_candidates`
|
||||
are returned to the caller (the nightly job), not logged directly by this module.
|
||||
However, the nightly job might log them.
|
||||
**Risk:** LOW — the `tts_text` is AI-generated coaching, not customer PII. The
|
||||
`fn_candidates` are diagnostic (not stored in Postgres). The `log.info` call in this
|
||||
module logs only counts, not the tts_text.
|
||||
**Recommendation:** Ensure the nightly job does not log the `tts_text` from
|
||||
`fn_candidates` (or redact it). v0.6.
|
||||
**Disposition:** Flag for v0.6 post-hoc review.
|
||||
|
||||
---
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
1. **The 8 v0.4 P1+ tech-debt wave is the right pattern for milestone-to-milestone
|
||||
debt repayment.** Folding the 8 findings into the P2 plan as a dedicated slice
|
||||
(SLICE-12) ensured they were addressed with fixes + tests, not lost. The
|
||||
cookie-secret validation, credential status enum, argon2id offload, zoneinfo
|
||||
scheduler, and cache persistence are all high-value, low-effort fixes that
|
||||
close real (if non-blocking) issues. This is the correct pattern for future
|
||||
milestones.
|
||||
|
||||
2. **The cache persistence (P1+ #7) is the highest-value tech-debt fix for v0.5.**
|
||||
The v0.4 P1+ #7 finding (aggregation cache lost on restart) directly corrupts
|
||||
v0.5's `assist_active_learners_count` after a server restart. The SQLite
|
||||
`cohort_learner_cache` table persistence ensures the distinct-learner set
|
||||
survives restarts. This is the correct fix — the cache is an intermediate state
|
||||
(the nightly job is the source of truth), but the persistence prevents
|
||||
under-counting between restart + nightly reconcile.
|
||||
|
||||
3. **The D-072 pilot tolerance (≤650ms) is correctly encoded as a measurement, not
|
||||
an assertion.** The `AssistLatencyMetrics` class provides the measurement
|
||||
infrastructure (p95/p50/p99 + within_target/within_pilot flags). The tests
|
||||
assert the infrastructure works against mock records, NOT that the actual
|
||||
latency is under budget (that's a Phase-1 live measurement). This is the
|
||||
correct pattern for NFRs that can't be verified in CI (latency depends on
|
||||
live voice-service latency, not mockable).
|
||||
|
||||
4. **The C-3 budget check is correctly diagnostic (not enforced).** D-012 says
|
||||
no enforced ceiling in the pilot. The `check_c3_budget` function returns a
|
||||
dict with `flag=True` when over budget, but does NOT raise an exception. The
|
||||
caller logs the flag + continues. This is the correct pattern for cost
|
||||
controls in a pilot — measure + alert, don't block.
|
||||
|
||||
5. **The D-063 enforcement (assist ≠ mastery) is cleanly maintained in P2.** The
|
||||
`_aggregate_assist` branch computes NO mastery metrics. The mastery view
|
||||
excludes assist metrics. The `test_aggregate_assist_no_mastery_metrics` +
|
||||
`test_mastery_endpoint_excludes_assist_metrics` tests verify the absence.
|
||||
This continues the P1 pattern (make the absence testable) into the aggregation
|
||||
layer.
|
||||
|
||||
---
|
||||
|
||||
## Final Test Count
|
||||
|
||||
```
|
||||
python3 -m pytest tests/ --tb=no
|
||||
→ 469 passed, 45 skipped, 5 warnings in 107.28s
|
||||
```
|
||||
|
||||
- **469 passed** (60 new P2 tests + 409 existing v0.1-v0.5 P1 tests)
|
||||
- **45 skipped** (all env-gated: PRAXIS_PG_DSN not set → 33 Postgres tests [24 v0.4 + 9 P2];
|
||||
live voice-service keys not provisioned → 11 live audio tests; PRAXIS_RUN_VC_INTEROP
|
||||
not set → 1 interop test)
|
||||
- **0 failed**
|
||||
- **0 errors**
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
| Layer | Result |
|
||||
|-------|--------|
|
||||
| Layer 1: Structural | PASS (all files exist, imports resolve, no stubs, exports present) |
|
||||
| Layer 2: Behavioral | PASS (469 passed, 45 skipped, 0 failed; 69 new P2 tests; 4/4 REQs covered) |
|
||||
| Layer 3: Security (STRIDE) | PASS (no HIGH/MEDIUM threats; all LOW accepted; 5 v0.4 security P1+ closed) |
|
||||
| Layer 4: Quality | PASS (correctness, testing, security, performance, maintainability, adversarial — all reviewed) |
|
||||
|
||||
| Verification Item | Result |
|
||||
|--------------------|--------|
|
||||
| 4 P2 REQ coverage | ✅ ALL COVERED (REQ-NFR-ASSIST-01, REQ-IDEATE-04, REQ-IDEATE-06, REQ-IDEATE-07) |
|
||||
| 8 v0.4 P1+ findings | ✅ ALL ADDRESSED (8/8 with fix + test) |
|
||||
| 5 P1-VERIFIER findings | ✅ ALL REVIEWED (P1-3 resolved; P1-1/P1-2/P1-4/P1-5 deferred to v0.6 with rationale) |
|
||||
| P0 fixes applied | 0 (none needed) |
|
||||
| P1+ findings flagged | 3 (all LOW — non-blocking, flagged for v0.6 post-hoc review) |
|
||||
|
||||
**Verdict: APPROVE_WITH_NOTES**
|
||||
|
||||
P2 (Integration + Tech-Debt + NFR Measurement) is ready to ship as `v0.1.12`. The
|
||||
3 P1+ findings are flagged for v0.6 post-hoc review (none block ship). All 8 v0.4
|
||||
P1+ findings are addressed. All 5 P1-VERIFIER findings are reviewed. The 4 P2 REQs
|
||||
are covered. The PIPEDA legal review (ESCALATION-01 from P1) remains the open risk
|
||||
for human attention.
|
||||
@@ -4,7 +4,7 @@
|
||||
"slug": "praxis",
|
||||
"name": "Praxis",
|
||||
"milestone": "v0.5",
|
||||
"status": "milestone-complete"
|
||||
"status": "phase-0-active"
|
||||
}
|
||||
],
|
||||
"active_project": "praxis",
|
||||
|
||||
+6
-27
@@ -221,34 +221,13 @@ class PgStore:
|
||||
return dict(row) if row else None
|
||||
|
||||
async def set_credential_status(self, cred_id: str, status: str) -> None:
|
||||
"""Set a credential's status (TASK-12-03, P1+ #4/#8 from v0.4 REVIEW).
|
||||
|
||||
Validates `status` against the allowed enum ('active', 'revoked') +
|
||||
uses two explicit parameterized queries (no f-string interpolation in
|
||||
SQL — P1+ #8 code smell fix). 'revoked' sets revoked_at=now(); 'active'
|
||||
clears revoked_at=NULL (re-activation).
|
||||
|
||||
P1+ #4: the status field is now validated (raises ValueError on invalid
|
||||
status — previously accepted any string).
|
||||
P1+ #8: the f-string interpolation (`, revoked_at = now()` or empty)
|
||||
is replaced with two explicit parameterized queries.
|
||||
"""
|
||||
if status not in ("active", "revoked"):
|
||||
raise ValueError(f"Invalid credential status: {status!r}")
|
||||
extra = ", revoked_at = now()" if status == "revoked" else ""
|
||||
async with self.pool.acquire() as conn:
|
||||
if status == "revoked":
|
||||
await conn.execute(
|
||||
"UPDATE issued_credentials SET status = $1, revoked_at = now() "
|
||||
"WHERE id = $2",
|
||||
status, cred_id,
|
||||
)
|
||||
else:
|
||||
# 'active' clears revoked_at (re-activation).
|
||||
await conn.execute(
|
||||
"UPDATE issued_credentials SET status = $1, revoked_at = NULL "
|
||||
"WHERE id = $2",
|
||||
status, cred_id,
|
||||
)
|
||||
await conn.execute(
|
||||
f"UPDATE issued_credentials SET status = $1{extra} WHERE id = $2",
|
||||
status,
|
||||
cred_id,
|
||||
)
|
||||
|
||||
async def list_credentials(self, operator_id: str | None = None) -> list[dict]:
|
||||
async with self.pool.acquire() as conn:
|
||||
|
||||
@@ -1,77 +0,0 @@
|
||||
"""C-3 budget check for assist cost (TASK-11-02, REQ-IDEATE-07, C-3, D-012).
|
||||
|
||||
Estimates the monthly assist cost per learner + compares against the C-3
|
||||
target (≤ $3/active learner/month — relaxed for the Canada pilot per D-012,
|
||||
but the architecture must not preclude it).
|
||||
|
||||
This is a DIAGNOSTIC check (not enforced — D-012 says no enforced ceiling in
|
||||
the pilot). It's logged at shift-end + reported in the P2 verification. The
|
||||
operator can review the log to understand the cost impact of assist usage.
|
||||
|
||||
R-ASSIST-14 mitigation: the budget check helps the operator understand the
|
||||
cost impact of assist usage. If the total (practice + assist) exceeds $3, the
|
||||
`flag` is True (diagnostic — the pilot continues, but the operator is alerted).
|
||||
|
||||
Example (from the plan):
|
||||
20 turns/shift × 20 shifts/month = 400 extra LLM calls. At ~$0.0005/turn
|
||||
(gemma4:cloud pilot rates), that's ~$0.20/month — well under $3. But if the
|
||||
turns are longer or the model is more expensive, the cost could approach
|
||||
the ceiling.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
# C-3 target: ≤ $3/active learner/month (relaxed for pilot per D-012, but the
|
||||
# architecture must not preclude it).
|
||||
C3_TARGET_USD = 3.0
|
||||
|
||||
|
||||
def check_c3_budget(
|
||||
assist_turns_per_shift: int,
|
||||
shifts_per_month: int,
|
||||
cost_per_turn_cents: float,
|
||||
practice_cost_per_month_usd: float = 0.0,
|
||||
) -> dict[str, Any]:
|
||||
"""Estimate the monthly assist cost + compare against the C-3 target.
|
||||
|
||||
Args:
|
||||
assist_turns_per_shift: average assist turns per shift.
|
||||
shifts_per_month: number of assist shifts per month.
|
||||
cost_per_turn_cents: average cost per assist turn (cents) — from
|
||||
derive_assist_turn_cost().derived_cents.
|
||||
practice_cost_per_month_usd: the existing practice cost/month (USD) —
|
||||
added to the assist cost to get the total. Default 0 (assist-only).
|
||||
|
||||
Returns:
|
||||
{
|
||||
monthly_assist_cost: float (USD),
|
||||
practice_cost_per_month: float (USD),
|
||||
total_with_practice: float (USD),
|
||||
c3_target: 3.0,
|
||||
within_budget: bool, # total <= c3_target
|
||||
flag: bool, # total > c3_target (diagnostic — not enforced)
|
||||
turns_per_month: int,
|
||||
}
|
||||
|
||||
D-012: the check is diagnostic (not enforced). `flag=True` means the
|
||||
total exceeds $3 — the operator is alerted, but the pilot continues.
|
||||
"""
|
||||
turns_per_month = assist_turns_per_shift * shifts_per_month
|
||||
# cost_per_turn_cents is in CENTS → divide by 100 for USD.
|
||||
monthly_assist_cost_usd = (turns_per_month * float(cost_per_turn_cents)) / 100.0
|
||||
total_with_practice = monthly_assist_cost_usd + float(practice_cost_per_month_usd)
|
||||
within_budget = total_with_practice <= C3_TARGET_USD
|
||||
return {
|
||||
"monthly_assist_cost": round(monthly_assist_cost_usd, 4),
|
||||
"practice_cost_per_month": round(float(practice_cost_per_month_usd), 4),
|
||||
"total_with_practice": round(total_with_practice, 4),
|
||||
"c3_target": C3_TARGET_USD,
|
||||
"within_budget": within_budget,
|
||||
"flag": not within_budget, # flag=True if over budget (diagnostic)
|
||||
"turns_per_month": turns_per_month,
|
||||
}
|
||||
|
||||
|
||||
__all__ = ["check_c3_budget", "C3_TARGET_USD"]
|
||||
@@ -1,274 +0,0 @@
|
||||
"""GuardrailMetrics — false-positive / false-negative measurement (TASK-09-02, REQ-IDEATE-04).
|
||||
|
||||
Measures the two guardrail NFR targets from REQ-IDEATE-04:
|
||||
- false_positive_rate: the FP rate on the tuning corpus (coaching responses
|
||||
blocked). Target < 5% (REQ-IDEATE-04). Measured at test time
|
||||
(test_guardrail_tuning.py) + reported here for the P2 verification.
|
||||
- false_negative_rate: the FN rate on the direct-answer + adversarial corpus
|
||||
(direct answers allowed). Measured at test time + trended nightly.
|
||||
|
||||
The nightly trend (`nightly_trend`) samples the last 24h of assist turns from
|
||||
the local SQLite turns table, re-runs the LiveAssistGuardrail on the `tts_text`
|
||||
(the LLM response that was actually played to the learner), and reports any
|
||||
`fn_candidates` — turns where the guardrail allowed the text but the text
|
||||
contains direct-answer patterns (a heuristic re-check, not a full LLM-as-judge
|
||||
which is v0.6 per REQ-IDEATE-10).
|
||||
|
||||
D-068 mitigation: the regex is the first line, not the only line. The nightly
|
||||
trend + the v0.6 LLM-as-judge (REQ-IDEATE-10) are the defense-in-depth. This
|
||||
nightly trend is a diagnostic (logged, not stored in Postgres — it's not a
|
||||
cohort metric). The operator can review the log to spot guardrail regressions.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import datetime as _dt
|
||||
import json
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from server.guardrails.live_assist import LiveAssistGuardrail
|
||||
from server.services.base import GuardrailContext
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
# REQ-IDEATE-04 targets.
|
||||
FP_TARGET = 0.05 # < 5% false-positive rate on coaching corpus
|
||||
FN_TARGET = 0.05 # < 5% false-negative rate on direct-answer corpus
|
||||
|
||||
|
||||
class GuardrailMetrics:
|
||||
"""Measures the LiveAssistGuardrail FP/FN rates (TASK-09-02, REQ-IDEATE-04).
|
||||
|
||||
Constructed with the tuning corpus (tests/guardrail_corpus.py) for the
|
||||
FP/FN rate computation. The nightly_trend() method takes a PraxisStore
|
||||
(SQLite) to sample recent assist turns.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
coaching_corpus: list[dict] | None = None,
|
||||
direct_corpus: list[dict] | None = None,
|
||||
adversarial_corpus: list[dict] | None = None,
|
||||
) -> None:
|
||||
# Lazy-import the corpus to avoid a circular import at module load
|
||||
# (tests/guardrail_corpus.py is a test fixture).
|
||||
if coaching_corpus is None or direct_corpus is None:
|
||||
from tests.guardrail_corpus import (
|
||||
ADVERSARIAL_RESPONSES,
|
||||
COACHING_RESPONSES,
|
||||
DIRECT_ANSWER_RESPONSES,
|
||||
)
|
||||
self._coaching = coaching_corpus or COACHING_RESPONSES
|
||||
self._direct = direct_corpus or DIRECT_ANSWER_RESPONSES
|
||||
self._adversarial = adversarial_corpus or ADVERSARIAL_RESPONSES
|
||||
else:
|
||||
self._coaching = coaching_corpus
|
||||
self._direct = direct_corpus
|
||||
self._adversarial = adversarial_corpus or []
|
||||
self._guardrail = LiveAssistGuardrail()
|
||||
self._ctx = GuardrailContext(role="assist")
|
||||
|
||||
async def _check(self, text: str) -> bool:
|
||||
"""Return True if the guardrail allows `text` (allowed=True)."""
|
||||
verdict = await self._guardrail.check(text, self._ctx)
|
||||
return bool(verdict.allowed)
|
||||
|
||||
async def false_positive_rate(self) -> tuple[float, int, int]:
|
||||
"""FP rate on the coaching corpus (coaching responses blocked).
|
||||
|
||||
A false positive = a coaching response that the guardrail blocked
|
||||
(allowed=False when it should have been allowed=True). Target < 5%
|
||||
(REQ-IDEATE-04).
|
||||
"""
|
||||
misclassified = 0
|
||||
total = 0
|
||||
for entry in self._coaching:
|
||||
total += 1
|
||||
allowed = await self._check(entry["text"])
|
||||
if not allowed: # blocked a coaching response → FP
|
||||
misclassified += 1
|
||||
rate = misclassified / total if total else 0.0
|
||||
return rate, misclassified, total
|
||||
|
||||
async def false_negative_rate(self) -> tuple[float, int, int]:
|
||||
"""FN rate on the direct-answer corpus (direct answers allowed).
|
||||
|
||||
A false negative = a direct-answer response that the guardrail allowed
|
||||
(allowed=True when it should have been allowed=False). Target < 5%
|
||||
(REQ-IDEATE-04).
|
||||
"""
|
||||
misclassified = 0
|
||||
total = 0
|
||||
for entry in self._direct:
|
||||
total += 1
|
||||
allowed = await self._check(entry["text"])
|
||||
if allowed: # allowed a direct answer → FN
|
||||
misclassified += 1
|
||||
rate = misclassified / total if total else 0.0
|
||||
return rate, misclassified, total
|
||||
|
||||
async def adversarial_false_negative_rate(self) -> tuple[float, int, int]:
|
||||
"""FN rate on the adversarial corpus (paraphrased direct answers).
|
||||
|
||||
This is the G-067 residual-risk set. The threshold is ≤ 20% for pilot
|
||||
(documented in test_guardrail_tuning.py). Reported here for the P2
|
||||
verification matrix; NOT asserted against the 5% target (the adversarial
|
||||
set is explicitly the residual-risk set, not the tuning target).
|
||||
"""
|
||||
misclassified = 0
|
||||
total = 0
|
||||
for entry in self._adversarial:
|
||||
total += 1
|
||||
allowed = await self._check(entry["text"])
|
||||
if allowed: # allowed a paraphrased direct answer → FN
|
||||
misclassified += 1
|
||||
rate = misclassified / total if total else 0.0
|
||||
return rate, misclassified, total
|
||||
|
||||
async def nightly_trend(self, store: Any) -> dict[str, Any]:
|
||||
"""Sample the last 24h of assist turns + re-run the guardrail (TASK-09-02).
|
||||
|
||||
Reads assist turns from the local SQLite turns table (joined to sessions
|
||||
on session_type='assist'), re-runs the LiveAssistGuardrail on each
|
||||
`tts_text`, and reports `fn_candidates` — turns where the guardrail
|
||||
allowed the text but the text contains direct-answer heuristic patterns.
|
||||
|
||||
This is the "trended nightly" part of REQ-IDEATE-04. It's a diagnostic
|
||||
(logged, not stored in Postgres — not a cohort metric). The heuristic
|
||||
re-check is a simple direct-answer pattern match (not a full LLM-as-judge
|
||||
— that's v0.6 per REQ-IDEATE-10).
|
||||
|
||||
Returns:
|
||||
{total_turns, blocked, allowed_coaching, allowed_neutral,
|
||||
fn_candidates: [{turn_seq, tts_text, reason}], window_hours: 24}
|
||||
"""
|
||||
cutoff = (_dt.datetime.now(_dt.timezone.utc) - _dt.timedelta(hours=24)).isoformat()
|
||||
# Query assist turns from the last 24h. The PraxisStore (SQLite) holds
|
||||
# the turns table; we read directly via aiosqlite to avoid adding a
|
||||
# method to the store surface for a diagnostic.
|
||||
rows: list[dict[str, Any]] = []
|
||||
try:
|
||||
import aiosqlite
|
||||
async with aiosqlite.connect(store.db_path) as db:
|
||||
db.row_factory = aiosqlite.Row
|
||||
cur = await db.execute(
|
||||
"SELECT t.id, t.seq, t.tts_text, t.guardrail_verdict_json, "
|
||||
"t.created_at, s.session_type "
|
||||
"FROM turns t JOIN sessions s ON t.session_id = s.id "
|
||||
"WHERE s.session_type = 'assist' "
|
||||
"AND t.tts_text IS NOT NULL "
|
||||
"AND t.created_at >= ? "
|
||||
"ORDER BY t.seq",
|
||||
(cutoff,),
|
||||
)
|
||||
async for r in cur:
|
||||
rows.append(dict(r))
|
||||
except Exception:
|
||||
log.exception("nightly_trend: failed to read assist turns from %s",
|
||||
getattr(store, "db_path", "?"))
|
||||
return {
|
||||
"total_turns": 0, "blocked": 0, "allowed_coaching": 0,
|
||||
"allowed_neutral": 0, "fn_candidates": [], "window_hours": 24,
|
||||
"error": "failed to read turns",
|
||||
}
|
||||
|
||||
total = len(rows)
|
||||
blocked = 0
|
||||
allowed_coaching = 0
|
||||
allowed_neutral = 0
|
||||
fn_candidates: list[dict[str, Any]] = []
|
||||
|
||||
for r in rows:
|
||||
tts_text = r.get("tts_text") or ""
|
||||
verdict_json = r.get("guardrail_verdict_json")
|
||||
try:
|
||||
verdict = json.loads(verdict_json) if verdict_json else {}
|
||||
except Exception:
|
||||
verdict = {}
|
||||
allowed = bool(verdict.get("allowed", True))
|
||||
if not allowed:
|
||||
blocked += 1
|
||||
continue
|
||||
# The guardrail allowed this text. Re-run the guardrail to confirm
|
||||
# (regression detection) + apply a heuristic direct-answer check.
|
||||
re_allowed = await self._check(tts_text)
|
||||
if not re_allowed:
|
||||
# The guardrail now blocks what it previously allowed → a
|
||||
# regression (or the corpus tuning changed). Flag it.
|
||||
fn_candidates.append({
|
||||
"turn_seq": r.get("seq"),
|
||||
"tts_text": tts_text[:200], # truncate for the log
|
||||
"reason": "guardrail regression: previously allowed, now blocked",
|
||||
})
|
||||
continue
|
||||
# Heuristic direct-answer check (defense-in-depth — not the LLM-as-judge).
|
||||
if _heuristic_direct_answer(tts_text):
|
||||
fn_candidates.append({
|
||||
"turn_seq": r.get("seq"),
|
||||
"tts_text": tts_text[:200],
|
||||
"reason": "heuristic direct-answer pattern detected",
|
||||
})
|
||||
continue
|
||||
# Classify allowed responses as coaching or neutral.
|
||||
if _looks_like_coaching_question(tts_text):
|
||||
allowed_coaching += 1
|
||||
else:
|
||||
allowed_neutral += 1
|
||||
|
||||
result = {
|
||||
"total_turns": total,
|
||||
"blocked": blocked,
|
||||
"allowed_coaching": allowed_coaching,
|
||||
"allowed_neutral": allowed_neutral,
|
||||
"fn_candidates": fn_candidates,
|
||||
"window_hours": 24,
|
||||
}
|
||||
log.info(
|
||||
"guardrail nightly trend: %d turns, %d blocked, %d allowed_coaching, "
|
||||
"%d allowed_neutral, %d fn_candidates",
|
||||
total, blocked, allowed_coaching, allowed_neutral, len(fn_candidates),
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
# ── Heuristic direct-answer detection (nightly trend defense-in-depth) ──────
|
||||
# A simple pattern check for the nightly trend. This is NOT the guardrail itself
|
||||
# (the guardrail is the 6-regex LiveAssistGuardrail). This is a secondary
|
||||
# heuristic to catch direct-answer patterns the guardrail may have allowed —
|
||||
# it's the "trended nightly" detection surface per REQ-IDEATE-04. The v0.6
|
||||
# LLM-as-judge (REQ-IDEATE-10) will replace this with a semantic classifier.
|
||||
|
||||
_DIRECT_ANSWER_HEURISTIC_PATTERNS = (
|
||||
"you should say",
|
||||
"tell the customer",
|
||||
"the answer is",
|
||||
"here's what to say",
|
||||
"what you should do is",
|
||||
"say this:",
|
||||
"respond with:",
|
||||
)
|
||||
|
||||
|
||||
def _heuristic_direct_answer(text: str) -> bool:
|
||||
"""Heuristic check for direct-answer patterns (nightly trend only)."""
|
||||
lower = text.lower()
|
||||
return any(p in lower for p in _DIRECT_ANSWER_HEURISTIC_PATTERNS)
|
||||
|
||||
|
||||
def _looks_like_coaching_question(text: str) -> bool:
|
||||
"""Heuristic: does the text look like a coaching question?"""
|
||||
stripped = text.strip()
|
||||
if stripped.endswith("?"):
|
||||
return True
|
||||
coaching_starters = ("what ", "how ", "why ", "have you ", "can you ", "could you ")
|
||||
lower = stripped.lower()
|
||||
return any(lower.startswith(s) for s in coaching_starters)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"GuardrailMetrics",
|
||||
"FP_TARGET",
|
||||
"FN_TARGET",
|
||||
]
|
||||
@@ -79,21 +79,17 @@ class LiveAssistGuardrailProcessor(FrameProcessor):
|
||||
await self.push_frame(frame, direction)
|
||||
return
|
||||
|
||||
# BUFFER LLM text chunks: do NOT push to TTS yet. The guardrail check
|
||||
# runs on LLMFullResponseEndFrame (after the full LLM response). Only
|
||||
# the allowed text (or CANNED_FALLBACK) is pushed to TTS. This is
|
||||
# REQ-ASSIST-03 — the guardrail MUST prevent direct-answer text from
|
||||
# reaching the learner's ear before the check completes. Streaming the
|
||||
# blocked text through to TTS would defeat the guardrail's purpose
|
||||
# (the learner would hear + parrot the direct answer before the canned
|
||||
# fallback plays). The latency cost of buffering (~200-500ms for 1-3
|
||||
# sentences) is acceptable for safety; the C-8 pilot tolerance (D-072)
|
||||
# is flagged for v0.6 hardening if the added latency pushes p95 >650ms.
|
||||
# Accumulate LLM text chunks.
|
||||
if isinstance(frame, TextFrame):
|
||||
self._accumulated_text += frame.text
|
||||
# Pass through for now; the verdict is applied on LLMFullResponseEndFrame.
|
||||
# (In a full implementation, we'd buffer + emit only the filtered text.
|
||||
# For the pilot, we pass through + rely on the end-frame check to log
|
||||
# the verdict + emit the canned fallback if blocked.)
|
||||
await self.push_frame(frame, direction)
|
||||
return
|
||||
|
||||
# On LLM full response end: run the guardrail check on the full text.
|
||||
# On LLM full response end: run the guardrail check.
|
||||
if isinstance(frame, LLMFullResponseEndFrame):
|
||||
response_text = self._accumulated_text
|
||||
verdict = await self.guardrail.check(
|
||||
@@ -101,11 +97,7 @@ class LiveAssistGuardrailProcessor(FrameProcessor):
|
||||
)
|
||||
|
||||
if verdict.allowed:
|
||||
# Allowed → push the buffered text to TTS + log the verdict.
|
||||
# (Buffered, not streamed — REQ-ASSIST-03 requires the guardrail
|
||||
# check to complete before any text reaches TTS.)
|
||||
if response_text:
|
||||
await self.push_frame(TextFrame(text=response_text), direction)
|
||||
# Allowed → log the verdict + complete the turn.
|
||||
await self._log_verdict(verdict, response_text)
|
||||
await self.push_frame(frame, direction)
|
||||
self._accumulated_text = ""
|
||||
|
||||
@@ -1,131 +0,0 @@
|
||||
"""AssistLatencyMetrics — p95 assist-turn latency measurement (TASK-09-01, D-072, REQ-IDEATE-04).
|
||||
|
||||
Collects per-turn LatencyRecord objects (from the LatencyObserver — server/latency.py)
|
||||
and computes the 95th percentile of `e2e_asr_to_tts_ms` (ASR transcript-ready → TTS
|
||||
first-audio — the C-8 latency budget).
|
||||
|
||||
D-072 binding (pilot tolerance):
|
||||
- target_ms = 600 (C-8 < 600ms — the hard target; v0.6 hardening)
|
||||
- pilot_tolerance_ms = 650 (≤ 650ms acceptable for pilot per D-072)
|
||||
- within_target = (p95 < 600) — the v0.6 hardening goal
|
||||
- within_pilot = (p95 <= 650) — the pilot acceptance gate
|
||||
|
||||
The metrics are collected per shift (one AssistLatencyMetrics instance per
|
||||
AssistSession) and reported at shift-end in the `session_outcome` dict, which
|
||||
flows to the cohort aggregation (SLICE-10 — `assist_p95_latency_ms` metric).
|
||||
|
||||
This module does NOT assert that the actual latency is under budget — that is a
|
||||
Phase-1 live measurement, not a CI test. This module provides the measurement
|
||||
infrastructure (collect → percentile → summary). The test (TASK-09-03) asserts
|
||||
the infrastructure works against mock records.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import statistics
|
||||
from typing import Any
|
||||
|
||||
from server.latency import LatencyRecord
|
||||
|
||||
# D-072 binding thresholds (pilot tolerance).
|
||||
TARGET_MS = 600 # C-8 hard target (< 600ms — v0.6 hardening goal)
|
||||
PILOT_TOLERANCE_MS = 650 # D-072 pilot acceptance (≤ 650ms)
|
||||
|
||||
|
||||
def _percentile(values: list[float], pct: float) -> float | None:
|
||||
"""Compute the `pct`-th percentile (0..100) of `values` using nearest-rank.
|
||||
|
||||
Returns None if `values` is empty. Uses the nearest-rank method (the same
|
||||
method used by numpy's default 'linear' interpolation for integer ranks):
|
||||
rank = ceil(pct/100 * N), 1-indexed; index = rank - 1 (clamped to [0, N-1]).
|
||||
This is the standard p95 computation for latency SLOs (Google SRE book §6).
|
||||
"""
|
||||
if not values:
|
||||
return None
|
||||
s = sorted(values)
|
||||
n = len(s)
|
||||
if n == 1:
|
||||
return s[0]
|
||||
# Nearest-rank: rank = ceil(pct/100 * n), then index = rank - 1.
|
||||
import math
|
||||
rank = max(1, math.ceil((pct / 100.0) * n))
|
||||
idx = min(rank - 1, n - 1)
|
||||
return s[idx]
|
||||
|
||||
|
||||
class AssistLatencyMetrics:
|
||||
"""Collects per-turn latency records + computes p95/p50/p99 (TASK-09-01).
|
||||
|
||||
One instance per assist shift. The LatencyObserver (server/latency.py) holds
|
||||
the live per-turn records; at shift-end the session code calls `record()` for
|
||||
each completed turn, then `summary()` to get the aggregate dict.
|
||||
|
||||
D-072: the summary reports both `within_target` (p95 < 600ms — the v0.6 goal)
|
||||
and `within_pilot` (p95 ≤ 650ms — the pilot acceptance gate). If
|
||||
`within_pilot` is False, the shift is flagged for the operator via the cohort
|
||||
aggregation (`assist_p95_latency_ms` metric — SLICE-10).
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._records: list[LatencyRecord] = []
|
||||
|
||||
def record(self, record: LatencyRecord) -> None:
|
||||
"""Append a latency record (one per completed assist turn)."""
|
||||
self._records.append(record)
|
||||
|
||||
@property
|
||||
def count(self) -> int:
|
||||
return len(self._records)
|
||||
|
||||
def _e2e_values(self) -> list[float]:
|
||||
"""The non-None e2e_asr_to_tts_ms values across all records."""
|
||||
out: list[float] = []
|
||||
for r in self._records:
|
||||
v = r.e2e_asr_to_tts_ms
|
||||
if v is not None:
|
||||
out.append(float(v))
|
||||
return out
|
||||
|
||||
def p50(self) -> float | None:
|
||||
"""The median e2e latency (ms), or None if no records."""
|
||||
return _percentile(self._e2e_values(), 50.0)
|
||||
|
||||
def p95(self) -> float | None:
|
||||
"""The 95th percentile e2e latency (ms), or None if no records."""
|
||||
return _percentile(self._e2e_values(), 95.0)
|
||||
|
||||
def p99(self) -> float | None:
|
||||
"""The 99th percentile e2e latency (ms), or None if no records."""
|
||||
return _percentile(self._e2e_values(), 99.0)
|
||||
|
||||
def summary(self) -> dict[str, Any]:
|
||||
"""Return the shift-end latency summary dict (D-072).
|
||||
|
||||
Fields:
|
||||
p50, p95, p99: the percentiles (ms) or None if no records.
|
||||
count: number of recorded turns.
|
||||
target_ms: 600 (C-8 hard target).
|
||||
pilot_tolerance_ms: 650 (D-072 pilot acceptance).
|
||||
within_target: p95 < 600 (the v0.6 hardening goal).
|
||||
within_pilot: p95 <= 650 (the pilot acceptance gate).
|
||||
"""
|
||||
p50 = self.p50()
|
||||
p95 = self.p95()
|
||||
p99 = self.p99()
|
||||
return {
|
||||
"p50": p50,
|
||||
"p95": p95,
|
||||
"p99": p99,
|
||||
"count": self.count,
|
||||
"target_ms": TARGET_MS,
|
||||
"pilot_tolerance_ms": PILOT_TOLERANCE_MS,
|
||||
"within_target": (p95 is not None and p95 < TARGET_MS),
|
||||
"within_pilot": (p95 is not None and p95 <= PILOT_TOLERANCE_MS),
|
||||
}
|
||||
|
||||
|
||||
__all__ = [
|
||||
"AssistLatencyMetrics",
|
||||
"TARGET_MS",
|
||||
"PILOT_TOLERANCE_MS",
|
||||
]
|
||||
@@ -21,7 +21,6 @@ from typing import Any
|
||||
|
||||
from db.store import PraxisStore, HARDCODED_LEARNER_ID
|
||||
from server.assist.context import AssistContext
|
||||
from server.assist.latency_metrics import AssistLatencyMetrics
|
||||
from server.assist.pii_policy import redact_pii
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
@@ -51,15 +50,6 @@ class AssistSession:
|
||||
self.turn_count: int = 0
|
||||
self.guardrail_block_count: int = 0
|
||||
self.shift_started_at: _dt.datetime = _dt.datetime.now(_dt.timezone.utc)
|
||||
# Per-shift latency metrics (TASK-09-01, D-072). The assist pipeline's
|
||||
# LatencyObserver holds the live records; at shift-end the pipeline code
|
||||
# calls record() for each completed turn, then summary() flows to the
|
||||
# session_outcome → cohort aggregation (assist_p95_latency_ms metric).
|
||||
self.latency_metrics = AssistLatencyMetrics()
|
||||
# TASK-11-01: per-shift assist cost accumulator (cents). Each turn's
|
||||
# cost is added via add_assist_turn_cost(); the total flows to the
|
||||
# session_outcome as assist_cost_cents for the C-3 budget check.
|
||||
self.assist_cost_cents: int = 0
|
||||
|
||||
async def start(self) -> str:
|
||||
"""Create the assist shift session row. Returns the session id."""
|
||||
@@ -156,15 +146,6 @@ class AssistSession:
|
||||
if not guardrail_verdict.get("allowed", True):
|
||||
self.guardrail_block_count += 1
|
||||
|
||||
def add_assist_turn_cost(self, cost_cents: int) -> None:
|
||||
"""Accumulate per-turn assist cost (TASK-11-01, REQ-IDEATE-07).
|
||||
|
||||
Called by the assist pipeline after each turn's cost is derived via
|
||||
derive_assist_turn_cost(). The total flows to the session_outcome as
|
||||
assist_cost_cents (for the C-3 budget check — TASK-11-02).
|
||||
"""
|
||||
self.assist_cost_cents += int(cost_cents)
|
||||
|
||||
async def end(self, outcome: str = "completed") -> dict[str, Any]:
|
||||
"""End the shift: update the session row + fire the aggregation hook.
|
||||
|
||||
@@ -193,7 +174,6 @@ class AssistSession:
|
||||
|
||||
def _build_session_outcome(self, outcome: str) -> dict[str, Any]:
|
||||
"""Construct the session_outcome dict for the aggregation hook (D-062)."""
|
||||
latency_summary = self.latency_metrics.summary()
|
||||
return {
|
||||
"learner_ref": self.learner_id,
|
||||
"path": self.context.path_slug,
|
||||
@@ -205,14 +185,6 @@ class AssistSession:
|
||||
"branch_path": [],
|
||||
"assist_turn_count": self.turn_count,
|
||||
"guardrail_blocks": self.guardrail_block_count,
|
||||
# D-072 (TASK-09-01): p95 latency flows to the cohort aggregation
|
||||
# as assist_p95_latency_ms. None if no completed turns.
|
||||
"assist_p95_latency_ms": latency_summary.get("p95"),
|
||||
"assist_p50_latency_ms": latency_summary.get("p50"),
|
||||
"assist_p99_latency_ms": latency_summary.get("p99"),
|
||||
"assist_within_pilot": latency_summary.get("within_pilot", False),
|
||||
# TASK-11-01: per-shift assist cost (sum of per-turn costs in cents).
|
||||
"assist_cost_cents": getattr(self, "assist_cost_cents", 0),
|
||||
"timestamp": _now_iso(),
|
||||
}
|
||||
|
||||
|
||||
@@ -37,12 +37,6 @@ def get_session_middleware_kwargs() -> dict:
|
||||
If PRAXIS_COOKIE_SECRET is unset, generate an ephemeral random secret
|
||||
and log a WARNING (dev only — sessions won't survive a restart and this
|
||||
MUST NOT be used in pilot/production).
|
||||
|
||||
TASK-12-02 (P1+ #3 from v0.4 REVIEW): if the secret is set but <32 bytes,
|
||||
log a WARNING (the HMAC signature is weakened). The secret is still
|
||||
accepted (backward compat — the pilot may have a short secret), but the
|
||||
warning is logged. In production (post-pilot), this should be a hard
|
||||
error (`raise RuntimeError`). For v0.5 pilot, the warning is sufficient.
|
||||
"""
|
||||
secret = os.environ.get("PRAXIS_COOKIE_SECRET", "").strip()
|
||||
if not secret:
|
||||
@@ -52,18 +46,6 @@ def get_session_middleware_kwargs() -> dict:
|
||||
"Sessions will NOT survive a server restart. This is dev-only; set "
|
||||
"PRAXIS_COOKIE_SECRET (>=32 bytes) for pilot/production."
|
||||
)
|
||||
elif len(secret) < 32:
|
||||
# TASK-12-02 (P1+ #3): a short non-empty secret weakens the HMAC
|
||||
# signature. Log a WARNING with the remediation guidance. The secret
|
||||
# is still accepted (backward compat — pilot); post-pilot this should
|
||||
# be a hard error.
|
||||
logger.warning(
|
||||
"PRAXIS_COOKIE_SECRET is <32 bytes (%d bytes) — HMAC signature weakened. "
|
||||
"Use 'openssl rand -base64 48' to generate a >=32-byte secret. "
|
||||
"The secret is accepted for pilot (backward compat); post-pilot this "
|
||||
"should be a hard error.",
|
||||
len(secret),
|
||||
)
|
||||
secure = _env_bool("PRAXIS_COOKIE_SECURE", True)
|
||||
if not secure:
|
||||
logger.warning(
|
||||
|
||||
@@ -11,8 +11,6 @@ client also clears its cookie. No sessions table.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request, status
|
||||
from pydantic import BaseModel
|
||||
|
||||
@@ -78,11 +76,7 @@ async def login(body: LoginBody, request: Request) -> LoginResponse:
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="invalid credentials",
|
||||
)
|
||||
# TASK-12-04 (P1+ #1): offload argon2id verification to a thread so the
|
||||
# ~100-300ms hashing duration does not block the event loop. R-AUTH-02:
|
||||
# acceptable for single-operator pilot, but offloading is low-effort +
|
||||
# correct for any future multi-operator load.
|
||||
if not await asyncio.to_thread(verify_password, row["password_hash"], body.password):
|
||||
if not verify_password(row["password_hash"], body.password):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="invalid credentials",
|
||||
@@ -91,8 +85,7 @@ async def login(body: LoginBody, request: Request) -> LoginResponse:
|
||||
request.session["operator_id"] = op_id
|
||||
await pg_store.update_last_login(op_id)
|
||||
if needs_rehash(row["password_hash"]):
|
||||
# Offload the rehash too (same ~100-300ms blocking concern).
|
||||
new_hash = await asyncio.to_thread(hash_password, body.password)
|
||||
new_hash = hash_password(body.password)
|
||||
async with pg_store.pool.acquire() as conn:
|
||||
await conn.execute(
|
||||
"UPDATE operators SET password_hash = $1 WHERE id = $2",
|
||||
|
||||
+2
-178
@@ -48,10 +48,6 @@ def _distinct_learners(sessions: list[dict[str, Any]]) -> int:
|
||||
async def aggregate_session(pg_store: PgStore, session_outcome: dict[str, Any]) -> None:
|
||||
"""Compute + upsert k-anonymized aggregates for one session outcome.
|
||||
|
||||
Branches on `session_type` (D-062):
|
||||
- 'assist' → _aggregate_assist (assist metrics, no mastery — D-063)
|
||||
- else → _aggregate_practice (the existing v0.4 practice logic)
|
||||
|
||||
Reads the affected path's recent session set (from cohort_aggregates or
|
||||
an in-memory accumulator), recomputes the metric cells for the 7-day
|
||||
window, applies k-anon suppression, and upserts each cell idempotently.
|
||||
@@ -60,20 +56,6 @@ async def aggregate_session(pg_store: PgStore, session_outcome: dict[str, Any])
|
||||
produces the same aggregate. The caller (hook.py) passes one session at
|
||||
a time; the nightly job (nightly.py) recomputes the full window.
|
||||
"""
|
||||
session_type = session_outcome.get("session_type", "practice")
|
||||
if session_type == "assist":
|
||||
await _aggregate_assist(pg_store, session_outcome)
|
||||
else:
|
||||
await _aggregate_practice(pg_store, session_outcome)
|
||||
|
||||
|
||||
async def _aggregate_practice(pg_store: PgStore, session_outcome: dict[str, Any]) -> None:
|
||||
"""The v0.4 practice aggregation logic (renamed for clarity — D-062).
|
||||
|
||||
Computes: sessions_count, active_learners_count, gate_open_rate,
|
||||
median_mastery_score, failure_mode_frequency, rubric_criterion_means,
|
||||
week_distribution. k-anon suppression (≥10 distinct learners).
|
||||
"""
|
||||
path = session_outcome.get("path") or session_outcome.get("path_id") or "unknown"
|
||||
learner_ref = session_outcome.get("learner_ref") or "unknown"
|
||||
outcome = session_outcome.get("outcome", "fail")
|
||||
@@ -170,122 +152,6 @@ async def _aggregate_practice(pg_store: PgStore, session_outcome: dict[str, Any]
|
||||
)
|
||||
|
||||
|
||||
# ── Assist aggregation (D-062, D-063, TASK-10-01) ────────────────────────────
|
||||
# Assist metrics use the SAME k-anonymity suppression (≥10 distinct learners),
|
||||
# the SAME 7-day rolling window, + the SAME idempotent upsert as practice.
|
||||
# No schema change to cohort_aggregates (the `metric` column is free-form TEXT
|
||||
# — D-062). D-063: assist does NOT update mastery (no rubric scores, no
|
||||
# gate_open_rate — those are practice-only metrics).
|
||||
|
||||
# The 5 core assist metrics (REQ-NFR-ASSIST-04) + p95 latency + cost:
|
||||
# assist_shifts_count — count of assist shifts in the window
|
||||
# assist_turns_count — total assist turns across all shifts
|
||||
# assist_avg_turns_per_shift — running mean of turns per shift
|
||||
# assist_active_learners_count — distinct learners with assist shifts
|
||||
# assist_guardrail_block_rate — guardrail_blocks / assist_turns_count
|
||||
# assist_p95_latency_ms — D-072 p95 latency (from SLICE-09)
|
||||
# assist_avg_cost_per_shift — per-shift assist cost (from SLICE-11, optional)
|
||||
|
||||
|
||||
async def _aggregate_assist(pg_store: PgStore, session_outcome: dict[str, Any]) -> None:
|
||||
"""Aggregate one assist shift outcome (D-062, D-063, TASK-10-01).
|
||||
|
||||
Upserts the 5 core assist metrics + p95 latency (+ optional avg cost).
|
||||
k-anon suppression applies (≥10 distinct learners — D-034 carry-forward).
|
||||
Idempotent upsert (ON CONFLICT). No schema change (D-062 — metric is TEXT).
|
||||
|
||||
D-063: assist does NOT update mastery. This function computes NO mastery
|
||||
metrics (no rubric scores, no gate_open_rate). The practice branch owns
|
||||
mastery; the assist branch owns assist-only metrics.
|
||||
"""
|
||||
path = session_outcome.get("path") or session_outcome.get("path_id") or "unknown"
|
||||
learner_ref = session_outcome.get("learner_ref") or "unknown"
|
||||
turn_count = int(session_outcome.get("assist_turn_count", 0))
|
||||
blocks = int(session_outcome.get("guardrail_blocks", 0))
|
||||
p95_latency = session_outcome.get("assist_p95_latency_ms")
|
||||
p95_latency_f = float(p95_latency) if p95_latency is not None else None
|
||||
cost_cents = int(session_outcome.get("assist_cost_cents", 0) or 0)
|
||||
ts = session_outcome.get("timestamp")
|
||||
|
||||
window_start, window_end = _rolling_window(
|
||||
_dt.datetime.fromisoformat(ts) if isinstance(ts, str) else None
|
||||
)
|
||||
|
||||
# Distinct-learner count for k-anon (same in-memory cache as practice).
|
||||
active_count = await _bump_active_learners(pg_store, path, window_start, learner_ref)
|
||||
shifts_count = await _bump_counter(pg_store, path, "assist_shifts_count",
|
||||
window_start, window_end)
|
||||
turns_total = await _bump_assist_turns(pg_store, path, window_start, turn_count)
|
||||
|
||||
suppressed = active_count < K_ANON_THRESHOLD
|
||||
|
||||
# assist_shifts_count
|
||||
await _upsert_cell(pg_store, path, "assist_shifts_count", window_start, window_end,
|
||||
float(shifts_count) if not suppressed else None,
|
||||
active_count, suppressed)
|
||||
|
||||
# assist_active_learners_count
|
||||
await _upsert_cell(pg_store, path, "assist_active_learners_count",
|
||||
window_start, window_end,
|
||||
float(active_count) if not suppressed else None,
|
||||
active_count, suppressed)
|
||||
|
||||
# assist_turns_count
|
||||
await _upsert_cell(pg_store, path, "assist_turns_count", window_start, window_end,
|
||||
float(turns_total) if not suppressed else None,
|
||||
active_count, suppressed)
|
||||
|
||||
# assist_avg_turns_per_shift — running mean of turns per shift
|
||||
avg_turns = await _running_mean(pg_store, path, "assist_avg_turns_per_shift",
|
||||
window_start, window_end, float(turn_count),
|
||||
active_count)
|
||||
await _upsert_cell(pg_store, path, "assist_avg_turns_per_shift",
|
||||
window_start, window_end,
|
||||
avg_turns if not suppressed else None,
|
||||
active_count, suppressed)
|
||||
|
||||
# assist_guardrail_block_rate = blocks / turns (0 if no turns yet)
|
||||
block_rate = (blocks / turn_count) if turn_count > 0 else 0.0
|
||||
# Running mean of per-shift block rates (so the window value is the mean
|
||||
# across shifts, not just the latest shift's rate).
|
||||
avg_block_rate = await _running_mean(pg_store, path, "assist_guardrail_block_rate",
|
||||
window_start, window_end, block_rate,
|
||||
active_count)
|
||||
await _upsert_cell(pg_store, path, "assist_guardrail_block_rate",
|
||||
window_start, window_end,
|
||||
avg_block_rate if not suppressed else None,
|
||||
active_count, suppressed)
|
||||
|
||||
# assist_p95_latency_ms (D-072 — from SLICE-09). Running mean of per-shift
|
||||
# p95 so the window value is the mean p95 across shifts (a trend signal).
|
||||
if p95_latency_f is not None:
|
||||
avg_p95 = await _running_mean(pg_store, path, "assist_p95_latency_ms",
|
||||
window_start, window_end, p95_latency_f,
|
||||
active_count)
|
||||
await _upsert_cell(pg_store, path, "assist_p95_latency_ms",
|
||||
window_start, window_end,
|
||||
avg_p95 if not suppressed else None,
|
||||
active_count, suppressed)
|
||||
|
||||
# assist_avg_cost_per_shift (TASK-11-01 — optional, useful for C-3 check).
|
||||
# Running mean of per-shift cost in cents.
|
||||
if cost_cents > 0:
|
||||
avg_cost = await _running_mean(pg_store, path, "assist_avg_cost_per_shift",
|
||||
window_start, window_end, float(cost_cents),
|
||||
active_count)
|
||||
await _upsert_cell(pg_store, path, "assist_avg_cost_per_shift",
|
||||
window_start, window_end,
|
||||
avg_cost if not suppressed else None,
|
||||
active_count, suppressed)
|
||||
|
||||
log.debug(
|
||||
"aggregate_assist path=%s learner=%s turns=%d blocks=%d p95=%s "
|
||||
"window=%s..%s active=%d suppressed=%s",
|
||||
path, learner_ref, turn_count, blocks, p95_latency_f,
|
||||
window_start, window_end, active_count, suppressed,
|
||||
)
|
||||
|
||||
|
||||
# ── Internal cell upsert + counter helpers ──────────────────────────────────
|
||||
# The PgStore.upsert_cohort_aggregate is idempotent (ON CONFLICT). We use a
|
||||
# small in-memory cache on the PgStore instance (created lazily) to track
|
||||
@@ -323,35 +189,12 @@ async def _bump_active_learners(pg_store: PgStore, path: str,
|
||||
|
||||
Returns the current distinct count (after adding this learner). The
|
||||
nightly job reconciles the true count from mastery_gate_events.
|
||||
|
||||
TASK-12-01 (P1+ #7): on the first call for a (path, window), the in-memory
|
||||
cache is seeded from the persisted SQLite cache (cohort_learner_cache) so
|
||||
the distinct count survives a server restart. The cache is persisted
|
||||
periodically via _save_learner_cache() (called by the hook on shift-end).
|
||||
"""
|
||||
cache = _cache(pg_store)
|
||||
key = _ck(path, "__learners__", window_start)
|
||||
learners: set[str] = cache.get(key)
|
||||
if learners is None:
|
||||
# First call for this (path, window) since restart → seed from the
|
||||
# persisted SQLite cache (TASK-12-01). If the cache is empty (fresh
|
||||
# install or first run), this starts a new set.
|
||||
try:
|
||||
from server.cohort.learner_cache import _count_distinct_learners, _load_learner_cache
|
||||
persisted = await _load_learner_cache(pg_store)
|
||||
# Merge any persisted learners for this (path, window).
|
||||
learners = persisted.get(key, set()).copy()
|
||||
except Exception:
|
||||
log.debug("cohort_learner_cache: load failed (fresh start?) — using empty set")
|
||||
learners = set()
|
||||
learners: set[str] = cache.get(key, set())
|
||||
learners.add(learner_ref)
|
||||
cache[key] = learners
|
||||
# Persist the updated set to SQLite (TASK-12-01 — survives restart).
|
||||
try:
|
||||
from server.cohort.learner_cache import _save_learner_cache
|
||||
await _save_learner_cache(pg_store, {key: learners})
|
||||
except Exception:
|
||||
log.debug("cohort_learner_cache: save failed (non-fatal — nightly reconciles)")
|
||||
return len(learners)
|
||||
|
||||
|
||||
@@ -368,19 +211,6 @@ async def _bump_mode_counter(pg_store: PgStore, path: str, metric: str,
|
||||
return await _bump_counter(pg_store, path, metric, window_start, window_end)
|
||||
|
||||
|
||||
async def _bump_assist_turns(pg_store: PgStore, path: str,
|
||||
window_start: _dt.date, turn_count: int) -> int:
|
||||
"""Accumulate assist turns across shifts in the window (TASK-10-01).
|
||||
|
||||
The counter is a running total of assist turns across all shifts in the
|
||||
(path, window). Each shift contributes its `assist_turn_count`.
|
||||
"""
|
||||
cache = _cache(pg_store)
|
||||
key = _ck(path, "assist_turns_count", window_start)
|
||||
cache[key] = cache.get(key, 0) + int(turn_count)
|
||||
return cache[key]
|
||||
|
||||
|
||||
async def _running_mean(pg_store: PgStore, path: str, metric: str,
|
||||
window_start: _dt.date, window_end: _dt.date,
|
||||
value: float, _active_count: int) -> float:
|
||||
@@ -397,10 +227,4 @@ async def _running_mean(pg_store: PgStore, path: str, metric: str,
|
||||
return new_mean
|
||||
|
||||
|
||||
__all__ = [
|
||||
"aggregate_session",
|
||||
"_aggregate_practice",
|
||||
"_aggregate_assist",
|
||||
"K_ANON_THRESHOLD",
|
||||
"_rolling_window",
|
||||
]
|
||||
__all__ = ["aggregate_session", "K_ANON_THRESHOLD", "_rolling_window"]
|
||||
@@ -1,259 +0,0 @@
|
||||
"""Cohort learner cache persistence (TASK-12-01, P1+ #7 from v0.4 REVIEW).
|
||||
|
||||
The v0.4 P1+ #7 finding: the `_agg_cache` on PgStore (aggregator.py:296-304)
|
||||
tracks running counters + distinct learner sets in-memory. On restart, the
|
||||
cache is lost — the next hook starts fresh, `active_learners_count` may reset
|
||||
to 1 (under-counting until nightly reconcile). This directly corrupts v0.5's
|
||||
`assist_active_learners_count` after a server restart.
|
||||
|
||||
Mitigation (TASK-12-01): persist the distinct-learner set to a small SQLite
|
||||
table (`cohort_learner_cache`) keyed by (path, window_start, learner_ref).
|
||||
The hook reads the cache from SQLite on startup + updates it on each session.
|
||||
The nightly job reconciles from `mastery_gate_events` (the source of truth) +
|
||||
clears the cache.
|
||||
|
||||
This is a low-effort, high-value fix (directly corrupts v0.5 assist metrics
|
||||
after a restart). The cache is a diagnostic/intermediate state — the nightly
|
||||
reconciliation from mastery_gate_events remains the source of truth.
|
||||
|
||||
Schema (additive — a new SQLite table, no change to the main praxis.db schema
|
||||
in db/migrations/):
|
||||
CREATE TABLE IF NOT EXISTS cohort_learner_cache (
|
||||
path TEXT NOT NULL,
|
||||
window_start TEXT NOT NULL, -- ISO date
|
||||
learner_ref TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL,
|
||||
PRIMARY KEY (path, window_start, learner_ref)
|
||||
);
|
||||
|
||||
The table is keyed by (path, window_start, learner_ref) — each distinct
|
||||
learner per (path, window) is one row. The distinct count = COUNT(*) per
|
||||
(path, window_start). The cache survives restarts (SQLite is durable).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as _dt
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import aiosqlite
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
# The cache SQLite file lives next to the main praxis.db (D-007 — learner-local
|
||||
# SQLite). A separate file avoids touching the main schema/migrations.
|
||||
_DEFAULT_CACHE_DB_PATH = os.environ.get(
|
||||
"PRAXIS_COHORT_CACHE_PATH",
|
||||
str(Path(os.environ.get("PRAXIS_DB_PATH", "praxis.db")).parent / "cohort_learner_cache.db"),
|
||||
)
|
||||
|
||||
_CREATE_TABLE_SQL = """
|
||||
CREATE TABLE IF NOT EXISTS cohort_learner_cache (
|
||||
path TEXT NOT NULL,
|
||||
window_start TEXT NOT NULL,
|
||||
learner_ref TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL,
|
||||
PRIMARY KEY (path, window_start, learner_ref)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_cache_path_window
|
||||
ON cohort_learner_cache (path, window_start);
|
||||
"""
|
||||
|
||||
|
||||
def _cache_db_path(store: Any = None) -> str | None:
|
||||
"""Resolve the cache DB path. Returns None if the path is not a real string
|
||||
(e.g., a MagicMock in tests) — the caller checks for None + skips the I/O.
|
||||
|
||||
A MagicMock auto-creates attributes, so `getattr(store, 'cohort_cache_db_path')`
|
||||
returns a MagicMock (not None) for a mocked store that didn't explicitly set
|
||||
the attribute. We detect this by checking isinstance(str) + the repr, and
|
||||
return None to skip the I/O (the in-memory cache is the source of truth for
|
||||
mocked tests).
|
||||
"""
|
||||
candidate = None
|
||||
if store is not None:
|
||||
# Use object.__getattribute__ to avoid MagicMock's auto-attribute
|
||||
# creation — only return the attribute if it was explicitly set.
|
||||
try:
|
||||
candidate = object.__getattribute__(store, "cohort_cache_db_path")
|
||||
except AttributeError:
|
||||
candidate = None
|
||||
if not isinstance(candidate, str) or not candidate:
|
||||
# Fall back to the default path ONLY for real stores (not mocks). A
|
||||
# real PgStore doesn't have `cohort_cache_db_path` set by default, so
|
||||
# we use the default. A MagicMock also doesn't have it set explicitly,
|
||||
# but we detect mocks via the type check above (candidate is a MagicMock
|
||||
# → not a str → candidate is None → we skip).
|
||||
if candidate is None and not _is_mock(store):
|
||||
candidate = _DEFAULT_CACHE_DB_PATH
|
||||
else:
|
||||
return None # mocked store or invalid path — skip I/O
|
||||
if "<MagicMock" in candidate:
|
||||
return None # safety: a MagicMock repr slipped through
|
||||
return candidate
|
||||
|
||||
|
||||
def _is_mock(store: Any) -> bool:
|
||||
"""Detect unittest.mock.Mock/MagicMock (so we skip cache I/O in tests)."""
|
||||
if store is None:
|
||||
return False
|
||||
return "Mock" in type(store).__name__ or "mock" in type(store).__module__
|
||||
|
||||
|
||||
async def _init_cache_db(db_path: str | None = None) -> None:
|
||||
"""Create the cache table if it doesn't exist (idempotent)."""
|
||||
p = db_path or _cache_db_path()
|
||||
if p is None:
|
||||
return # mocked store — skip I/O
|
||||
async with aiosqlite.connect(p) as db:
|
||||
await db.executescript(_CREATE_TABLE_SQL)
|
||||
await db.commit()
|
||||
|
||||
|
||||
async def _load_learner_cache(store: Any) -> dict:
|
||||
"""Load the distinct-learner sets from SQLite on startup (TASK-12-01).
|
||||
|
||||
Returns a dict shaped like the in-memory cache's `__learners__` entries:
|
||||
{ (path, "__learners__", window_start): set(learner_ref, ...) }
|
||||
|
||||
The store parameter is accepted for interface symmetry with the plan's
|
||||
signature, but the cache lives in a dedicated SQLite file (not the
|
||||
PraxisStore's praxis.db) so the cache is decoupled from the learner store.
|
||||
The `store` may carry a `cohort_cache_db_path` attribute to override the
|
||||
default path (used by tests). If the path is not a real string (e.g., a
|
||||
MagicMock in tests), returns {} (no-op — the in-memory cache starts fresh).
|
||||
"""
|
||||
db_path = _cache_db_path(store)
|
||||
if db_path is None:
|
||||
return {} # mocked store — skip I/O, start fresh
|
||||
try:
|
||||
await _init_cache_db(db_path)
|
||||
except Exception:
|
||||
log.exception("cohort_learner_cache: failed to init %s", db_path)
|
||||
return {}
|
||||
cache: dict[tuple[str, str, _dt.date], set[str]] = {}
|
||||
try:
|
||||
async with aiosqlite.connect(db_path) as db:
|
||||
cur = await db.execute(
|
||||
"SELECT path, window_start, learner_ref FROM cohort_learner_cache"
|
||||
)
|
||||
async for row in cur:
|
||||
path, ws_iso, learner_ref = row
|
||||
ws = _dt.date.fromisoformat(ws_iso)
|
||||
key = (path, "__learners__", ws)
|
||||
cache.setdefault(key, set()).add(learner_ref)
|
||||
except Exception:
|
||||
log.exception("cohort_learner_cache: failed to load from %s", db_path)
|
||||
return {}
|
||||
log.info("cohort_learner_cache: loaded %d (path, window) learner sets from %s",
|
||||
len(cache), db_path)
|
||||
return cache
|
||||
|
||||
|
||||
async def _save_learner_cache(store: Any, cache: dict) -> None:
|
||||
"""Save the distinct-learner sets to SQLite (TASK-12-01).
|
||||
|
||||
Called periodically (every 5 minutes or on shift-end). Upserts each
|
||||
(path, window_start, learner_ref) row idempotently (INSERT OR IGNORE —
|
||||
the distinct set is a set, so re-inserting an existing row is a no-op).
|
||||
If the store's cache path is not a real string (e.g., a MagicMock in
|
||||
tests), this is a no-op (the in-memory cache is the source of truth for
|
||||
the test).
|
||||
"""
|
||||
db_path = _cache_db_path(store)
|
||||
if db_path is None:
|
||||
return # mocked store — skip I/O
|
||||
try:
|
||||
await _init_cache_db(db_path)
|
||||
except Exception:
|
||||
log.exception("cohort_learner_cache: failed to init %s", db_path)
|
||||
return
|
||||
now_iso = _dt.datetime.now(_dt.timezone.utc).isoformat()
|
||||
rows: list[tuple[str, str, str, str]] = []
|
||||
for key, learners in cache.items():
|
||||
if not isinstance(learners, set):
|
||||
continue
|
||||
# key = (path, "__learners__", window_start)
|
||||
path, _metric, ws = key
|
||||
ws_iso = ws.isoformat() if isinstance(ws, _dt.date) else str(ws)
|
||||
for learner_ref in learners:
|
||||
rows.append((path, ws_iso, learner_ref, now_iso))
|
||||
if not rows:
|
||||
return
|
||||
try:
|
||||
async with aiosqlite.connect(db_path) as db:
|
||||
await db.executemany(
|
||||
"INSERT OR IGNORE INTO cohort_learner_cache "
|
||||
"(path, window_start, learner_ref, updated_at) VALUES (?, ?, ?, ?)",
|
||||
rows,
|
||||
)
|
||||
await db.commit()
|
||||
except Exception:
|
||||
log.exception("cohort_learner_cache: failed to save %d rows to %s",
|
||||
len(rows), db_path)
|
||||
return
|
||||
log.info("cohort_learner_cache: saved %d learner rows to %s", len(rows), db_path)
|
||||
|
||||
|
||||
async def _clear_learner_cache(store: Any, path: str | None = None,
|
||||
window_start: _dt.date | None = None) -> None:
|
||||
"""Clear the cache (called by the nightly job after reconciliation).
|
||||
|
||||
If path + window_start are given, clears only that (path, window). If
|
||||
neither is given, clears the entire cache (full nightly reconciliation).
|
||||
"""
|
||||
db_path = _cache_db_path(store)
|
||||
if db_path is None:
|
||||
return # mocked store — skip I/O
|
||||
try:
|
||||
async with aiosqlite.connect(db_path) as db:
|
||||
if path is not None and window_start is not None:
|
||||
await db.execute(
|
||||
"DELETE FROM cohort_learner_cache "
|
||||
"WHERE path = ? AND window_start = ?",
|
||||
(path, window_start.isoformat()),
|
||||
)
|
||||
else:
|
||||
await db.execute("DELETE FROM cohort_learner_cache")
|
||||
await db.commit()
|
||||
except Exception:
|
||||
log.exception("cohort_learner_cache: failed to clear %s", db_path)
|
||||
|
||||
|
||||
async def _count_distinct_learners(store: Any, path: str,
|
||||
window_start: _dt.date) -> int:
|
||||
"""Count distinct learners for (path, window) from the cache (TASK-12-01).
|
||||
|
||||
This is the persisted count — survives restarts. Used by the aggregator
|
||||
to initialize the in-memory cache on startup (so active_learners_count
|
||||
is not reset to 1 after a restart).
|
||||
"""
|
||||
db_path = _cache_db_path(store)
|
||||
if db_path is None:
|
||||
return 0 # mocked store — no persisted cache
|
||||
try:
|
||||
await _init_cache_db(db_path)
|
||||
async with aiosqlite.connect(db_path) as db:
|
||||
cur = await db.execute(
|
||||
"SELECT COUNT(DISTINCT learner_ref) FROM cohort_learner_cache "
|
||||
"WHERE path = ? AND window_start = ?",
|
||||
(path, window_start.isoformat()),
|
||||
)
|
||||
row = await cur.fetchone()
|
||||
return int(row[0]) if row else 0
|
||||
except Exception:
|
||||
log.exception("cohort_learner_cache: failed to count for path=%s window=%s",
|
||||
path, window_start)
|
||||
return 0
|
||||
|
||||
|
||||
__all__ = [
|
||||
"_load_learner_cache",
|
||||
"_save_learner_cache",
|
||||
"_clear_learner_cache",
|
||||
"_count_distinct_learners",
|
||||
"_init_cache_db",
|
||||
]
|
||||
@@ -19,16 +19,12 @@ import logging
|
||||
import statistics
|
||||
from collections import Counter, defaultdict
|
||||
from typing import Any
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
from db.pg_store import PgStore
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
# TASK-12-04 (P1+ #6): use zoneinfo.ZoneInfo("America/Winnipeg") for proper
|
||||
# DST handling (CST UTC-6 in winter + CDT UTC-5 in summer). The v0.4 fixed
|
||||
# UTC-5 offset drifted ≤1h across DST boundaries; this is the correct fix.
|
||||
CT = ZoneInfo("America/Winnipeg")
|
||||
CT = _dt.timezone(_dt.timedelta(hours=-5), "CT")
|
||||
NIGHTLY_HOUR = 3
|
||||
NIGHTLY_MINUTE = 0
|
||||
|
||||
@@ -36,15 +32,16 @@ NIGHTLY_MINUTE = 0
|
||||
def seconds_until_next_03_ct(now: _dt.datetime | None = None) -> float:
|
||||
"""Seconds from `now` until the next 03:00 America/Winnipeg (CT).
|
||||
|
||||
TASK-12-04 (P1+ #6): uses zoneinfo.ZoneInfo("America/Winnipeg") for proper
|
||||
DST handling (CST UTC-6 in winter + CDT UTC-5 in summer). The v0.4 fixed
|
||||
UTC-5 offset is replaced with the timezone-aware computation.
|
||||
America/Winnipeg observes CST (UTC-6) in winter + CDT (UTC-5) in summer.
|
||||
We approximate CT as a fixed UTC-5 offset (the pilot is in summer CDT
|
||||
and the scheduler drift of ≤1h over DST boundaries is acceptable for a
|
||||
nightly reconciliation job — the on-session-end hook keeps data fresh).
|
||||
A future hardening would use zoneinfo.ZoneInfo("America/Winnipeg") with
|
||||
proper DST handling.
|
||||
"""
|
||||
now = now or _dt.datetime.now(CT)
|
||||
if now.tzinfo is None:
|
||||
now = now.replace(tzinfo=CT)
|
||||
else:
|
||||
now = now.astimezone(CT)
|
||||
next_run = now.replace(hour=NIGHTLY_HOUR, minute=NIGHTLY_MINUTE,
|
||||
second=0, microsecond=0)
|
||||
if next_run <= now:
|
||||
@@ -184,18 +181,6 @@ class NightlyScheduler:
|
||||
|
||||
log.info("nightly reconcile: recomputed %d (path, window) cells", len(by_path_window))
|
||||
|
||||
# TASK-12-01 (P1+ #7): clear the cohort_learner_cache after
|
||||
# reconciliation. The nightly job is the source of truth (it recomputes
|
||||
# from mastery_gate_events); the cache is an intermediate state that
|
||||
# should be cleared so the next hook starts fresh from the reconciled
|
||||
# aggregates. This prevents stale cache entries from accumulating.
|
||||
try:
|
||||
from server.cohort.learner_cache import _clear_learner_cache
|
||||
await _clear_learner_cache(pg_store)
|
||||
log.info("nightly reconcile: cleared cohort_learner_cache (TASK-12-01)")
|
||||
except Exception:
|
||||
log.debug("nightly reconcile: cohort_learner_cache clear failed (non-fatal)")
|
||||
|
||||
async def reconcile_now(self, pg_store: PgStore) -> None:
|
||||
"""Public hook for tests / ad-hoc reconciliation (no clock wait)."""
|
||||
await self._reconcile(pg_store)
|
||||
|
||||
+1
-51
@@ -108,54 +108,4 @@ def derive_cost(
|
||||
)
|
||||
|
||||
|
||||
def derive_assist_turn_cost(
|
||||
llm_input_tokens: int = 0,
|
||||
llm_output_tokens: int = 0,
|
||||
tts_characters: int = 0,
|
||||
tts_provider: str = "piper",
|
||||
rates: dict[str, float] | None = None,
|
||||
) -> CostBreakdown:
|
||||
"""Derive the per-assist-turn cost in cents (TASK-11-01, REQ-IDEATE-07).
|
||||
|
||||
An assist turn is a short coaching exchange — a single gemma4:cloud LLM
|
||||
call + Piper TTS (D-065 — Piper is the assist default). No debrief tokens
|
||||
(assist has no debrief — D-063) + no Deepgram audio minutes (the assist
|
||||
turn's ASR is accounted in the shift's Deepgram minutes, not per-turn —
|
||||
the per-turn cost is the LLM + TTS only).
|
||||
|
||||
Uses the same load_rates() + the same CostBreakdown dataclass as
|
||||
derive_cost(). The per-turn cost is logged via
|
||||
AssistSession.add_assist_turn_cost() + aggregated at shift-end as
|
||||
assist_cost_cents in the session_outcome (for the C-3 budget check —
|
||||
TASK-11-02).
|
||||
|
||||
The existing derive_cost() is unchanged (practice sessions keep their
|
||||
cost logging — backward compat).
|
||||
"""
|
||||
r = rates or load_rates()
|
||||
|
||||
# LLM (gemma4:cloud) — the assist coaching call.
|
||||
llm_tokens = llm_input_tokens + llm_output_tokens
|
||||
llm_cents = (llm_tokens / 1000.0) * r.get("gemma4_cloud_per_1k_tokens_cents", 0.5)
|
||||
|
||||
# TTS (Piper default for assist — D-065; Cartesia fallback).
|
||||
tts_rate_key = (
|
||||
"piper_per_1k_chars_cents" if tts_provider == "piper"
|
||||
else "cartesia_per_1k_chars_cents"
|
||||
)
|
||||
tts_cents = (tts_characters / 1000.0) * r.get(tts_rate_key, 3.0)
|
||||
|
||||
total = int(round(llm_cents + tts_cents))
|
||||
return CostBreakdown(
|
||||
llm_input_tokens=llm_input_tokens,
|
||||
llm_output_tokens=llm_output_tokens,
|
||||
deepgram_audio_minutes=0.0, # assist ASR accounted at shift level
|
||||
tts_characters=tts_characters,
|
||||
debrief_input_tokens=0, # assist has no debrief (D-063)
|
||||
debrief_output_tokens=0,
|
||||
rates=r,
|
||||
derived_cents=total,
|
||||
)
|
||||
|
||||
|
||||
__all__ = ["CostBreakdown", "derive_cost", "derive_assist_turn_cost", "load_rates"]
|
||||
__all__ = ["CostBreakdown", "derive_cost", "load_rates"]
|
||||
@@ -1,15 +1,10 @@
|
||||
"""GET /api/operator/cohort — practice + assist volume view (TASK-08-01, TASK-10-02, D-053, D-057).
|
||||
"""GET /api/operator/cohort — practice volume view (TASK-08-01, D-053, D-057).
|
||||
|
||||
Auth-gated (Depends(current_operator)). Returns k-anonymized practice + assist
|
||||
volume aggregates from cohort_aggregates: sessions_count + active_learners_count
|
||||
(practice) + assist_shifts_count + assist_turns_count (assist) per path.
|
||||
Suppressed cells have value=null + cell_suppressed=true; the frontend renders
|
||||
\"— (<10 learners)\". No per-learner drill-down (R-DASH-02).
|
||||
Auth-gated (Depends(current_operator)). Returns k-anonymized practice-volume
|
||||
aggregates from cohort_aggregates: sessions_count + active_learners_count per
|
||||
path. Suppressed cells have value=null + cell_suppressed=true; the frontend
|
||||
renders \"— (<10 learners)\". No per-learner drill-down (R-DASH-02).
|
||||
last_updated = max(updated_at) for freshness (REQ-NFR-DASH-02).
|
||||
|
||||
D-062: assist metrics are new metric strings in the same cohort_aggregates
|
||||
table (no schema change). The view returns practice + assist volume
|
||||
side-by-side so operators see both modes per path.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -29,13 +24,7 @@ from server.operator._common import (
|
||||
|
||||
router = APIRouter(prefix="/api/operator", tags=["operator-cohort"])
|
||||
|
||||
# Practice volume metrics (v0.4) + assist volume metrics (v0.5 — TASK-10-02).
|
||||
PRACTICE_METRICS = {
|
||||
"sessions_count",
|
||||
"active_learners_count",
|
||||
"assist_shifts_count",
|
||||
"assist_turns_count",
|
||||
}
|
||||
PRACTICE_METRICS = {"sessions_count", "active_learners_count"}
|
||||
|
||||
|
||||
@router.get("/cohort", response_model=ViewResponse)
|
||||
|
||||
@@ -9,7 +9,6 @@ the credential asserts (D-043).
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as _dt
|
||||
import logging
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request, status
|
||||
from pydantic import BaseModel
|
||||
@@ -20,8 +19,6 @@ from server.operator._common import require_pg_store
|
||||
|
||||
router = APIRouter(prefix="/api/operator", tags=["operator-credentials"])
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class CredentialOut(BaseModel):
|
||||
id: str
|
||||
@@ -75,11 +72,6 @@ async def revoke_credential(
|
||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail="credential not found")
|
||||
await pg_store.set_credential_status(cred_id, "revoked")
|
||||
# TASK-12-04 (P1+ #5): application-level audit log for credential revocation.
|
||||
# The revoking operator_id + cred_id are logged. No audit_log table (the
|
||||
# log is sufficient for pilot — D-056 stateless cookies + revoked_at
|
||||
# timestamp are the primary audit trail).
|
||||
log.info("credential revoked: operator=%s cred_id=%s", op.id, cred_id)
|
||||
return OkResponse(ok=True, id=cred_id, status="revoked")
|
||||
|
||||
|
||||
|
||||
@@ -1,14 +1,9 @@
|
||||
"""GET /api/operator/failure-patterns — failure patterns + safety signals (TASK-08-03, TASK-10-02, D-053).
|
||||
"""GET /api/operator/failure-patterns — failure patterns view (TASK-08-03, D-053).
|
||||
|
||||
Auth-gated. Returns failure pattern metrics: failure_mode frequency (cells
|
||||
with metric prefix `failure_mode:`) + branch outcome distribution (cells
|
||||
with metric prefix `branch:`) + the assist guardrail block rate safety signal
|
||||
(TASK-10-02 — `assist_guardrail_block_rate`). Weak-spot rubric criteria
|
||||
(mean < 3.0) are highlighted by the frontend. All k-anonymized.
|
||||
|
||||
The `assist_guardrail_block_rate` is a safety signal for operators: a sudden
|
||||
spike signals either a prompt regression or learners pushing boundaries. High
|
||||
block rate = flag for operator review.
|
||||
with metric prefix `branch:`). Weak-spot rubric criteria (mean < 3.0) are
|
||||
highlighted by the frontend. All k-anonymized.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -28,17 +23,9 @@ from server.operator._common import (
|
||||
|
||||
router = APIRouter(prefix="/api/operator", tags=["operator-failure-patterns"])
|
||||
|
||||
# The assist guardrail block-rate safety signal (TASK-10-02, D-060 layer 3).
|
||||
ASSIST_GUARDRAIL_BLOCK_RATE = "assist_guardrail_block_rate"
|
||||
|
||||
|
||||
def _is_failure_metric(metric: str) -> bool:
|
||||
# Failure patterns (v0.4) + the assist guardrail block-rate safety signal (v0.5).
|
||||
return (
|
||||
metric.startswith("failure_mode:")
|
||||
or metric.startswith("branch:")
|
||||
or metric == ASSIST_GUARDRAIL_BLOCK_RATE
|
||||
)
|
||||
return metric.startswith("failure_mode:") or metric.startswith("branch:")
|
||||
|
||||
|
||||
@router.get("/failure-patterns", response_model=ViewResponse)
|
||||
|
||||
@@ -1,13 +1,8 @@
|
||||
"""GET /api/operator/mastery — mastery progression view (TASK-08-02, D-053, D-063).
|
||||
"""GET /api/operator/mastery — mastery progression view (TASK-08-02, D-053).
|
||||
|
||||
Auth-gated. Returns mastery progression metrics: gate_open_rate,
|
||||
median_mastery_score, rubric_criterion_means (cells with metric prefix
|
||||
`rubric_criterion_mean:`). All k-anonymized (suppressed if < 10).
|
||||
|
||||
D-063 (binding): assist does NOT update mastery. This view is unchanged from
|
||||
v0.4 — assist metrics (assist_shifts_count, assist_turns_count) are NOT
|
||||
mastery metrics and are NOT included here. They appear in the cohort view
|
||||
(TASK-10-02). The assist metrics are separate from practice/mastery metrics.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -31,9 +26,6 @@ MASTERY_METRICS = {"gate_open_rate", "median_mastery_score"}
|
||||
|
||||
|
||||
def _is_mastery_metric(metric: str) -> bool:
|
||||
# D-063: assist metrics are NOT mastery metrics. Only practice mastery
|
||||
# metrics (gate_open_rate, median_mastery_score, rubric_criterion_mean:*)
|
||||
# are included in this view.
|
||||
return metric in MASTERY_METRICS or metric.startswith("rubric_criterion_mean:")
|
||||
|
||||
|
||||
|
||||
@@ -1,228 +0,0 @@
|
||||
"""Assist cost tracking tests (TASK-11-03, REQ-IDEATE-07, C-3, D-012).
|
||||
|
||||
Tests:
|
||||
- derive_assist_turn_cost() computes the per-turn cost (LLM + Piper TTS).
|
||||
- The shift-end assist_cost_cents is the sum of per-turn costs.
|
||||
- check_c3_budget() with 20 turns/shift × 20 shifts/month → within budget.
|
||||
- check_c3_budget() with 100 turns/shift × 30 shifts/month → may exceed (flag=True).
|
||||
- Existing derive_cost() unchanged (practice cost tests still pass).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from server.assist.budget_check import C3_TARGET_USD, check_c3_budget
|
||||
from server.cost import CostBreakdown, derive_assist_turn_cost, derive_cost, load_rates
|
||||
|
||||
|
||||
# ── derive_assist_turn_cost ─────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_derive_assist_turn_cost_basic():
|
||||
"""Per-turn cost computed from LLM tokens + Piper TTS chars."""
|
||||
b = derive_assist_turn_cost(
|
||||
llm_input_tokens=300,
|
||||
llm_output_tokens=100,
|
||||
tts_characters=400,
|
||||
tts_provider="piper",
|
||||
)
|
||||
assert b.derived_cents >= 0
|
||||
assert b.llm_input_tokens == 300
|
||||
assert b.llm_output_tokens == 100
|
||||
assert b.tts_characters == 400
|
||||
# No debrief (D-063) + no Deepgram minutes (accounted at shift level).
|
||||
assert b.debrief_input_tokens == 0
|
||||
assert b.debrief_output_tokens == 0
|
||||
assert b.deepgram_audio_minutes == 0.0
|
||||
|
||||
|
||||
def test_derive_assist_turn_cost_piper_zero_tts():
|
||||
"""Piper self-hosted TTS is $0 marginal cost (D-065 — Piper is assist default)."""
|
||||
b = derive_assist_turn_cost(
|
||||
llm_input_tokens=300,
|
||||
llm_output_tokens=100,
|
||||
tts_characters=10000,
|
||||
tts_provider="piper",
|
||||
)
|
||||
# Piper rate is 0.0 per 1k chars → TTS contributes 0; only LLM cost.
|
||||
# LLM: (300+100)/1000 * 0.5 = 0.2 cents → rounds to 0.
|
||||
assert b.derived_cents >= 0
|
||||
|
||||
|
||||
def test_derive_assist_turn_cost_cartesia_fallback():
|
||||
"""Cartesia TTS fallback (non-default for assist — D-065 prefers Piper)."""
|
||||
b = derive_assist_turn_cost(
|
||||
llm_input_tokens=300,
|
||||
llm_output_tokens=100,
|
||||
tts_characters=1000,
|
||||
tts_provider="cartesia",
|
||||
)
|
||||
# Cartesia rate is 3.0 per 1k chars → 1000 chars = 3.0 cents TTS.
|
||||
assert b.derived_cents > 0
|
||||
|
||||
|
||||
def test_derive_assist_turn_cost_uses_same_rates_as_derive_cost():
|
||||
"""derive_assist_turn_cost uses the same load_rates() + CostBreakdown."""
|
||||
rates = load_rates()
|
||||
b = derive_assist_turn_cost(
|
||||
llm_input_tokens=1000,
|
||||
llm_output_tokens=500,
|
||||
tts_characters=500,
|
||||
tts_provider="piper",
|
||||
rates=rates,
|
||||
)
|
||||
assert b.rates is rates
|
||||
assert isinstance(b, CostBreakdown)
|
||||
|
||||
|
||||
def test_derive_cost_unchanged():
|
||||
"""Existing derive_cost() unchanged (practice cost tests still pass)."""
|
||||
b = derive_cost(
|
||||
llm_input_tokens=500,
|
||||
llm_output_tokens=200,
|
||||
deepgram_audio_minutes=2.0,
|
||||
tts_characters=800,
|
||||
debrief_input_tokens=300,
|
||||
debrief_output_tokens=150,
|
||||
tts_provider="cartesia",
|
||||
)
|
||||
assert b.derived_cents > 0
|
||||
assert b.deepgram_audio_minutes == 2.0
|
||||
assert b.debrief_input_tokens == 300
|
||||
|
||||
|
||||
# ── Shift-end assist_cost_cents aggregation ─────────────────────────────────
|
||||
|
||||
|
||||
def test_shift_end_assist_cost_is_sum_of_per_turn_costs():
|
||||
"""AssistSession.assist_cost_cents is the sum of per-turn costs."""
|
||||
from server.assist.session import AssistSession
|
||||
from server.assist.context import AssistContext
|
||||
|
||||
# Construct an AssistSession without calling start() (we only test the
|
||||
# cost accumulator, not the DB lifecycle).
|
||||
ctx = AssistContext(
|
||||
system_prompt="",
|
||||
current_week=1,
|
||||
scenario_tag="refund",
|
||||
theta=0.0,
|
||||
coaching_focus="empathy",
|
||||
path_slug="customer_service",
|
||||
)
|
||||
session = AssistSession.__new__(AssistSession)
|
||||
session.assist_cost_cents = 0
|
||||
session.turn_count = 0
|
||||
session.guardrail_block_count = 0
|
||||
session.latency_metrics = None # not needed for this test
|
||||
|
||||
# Simulate 3 turns with per-turn costs.
|
||||
for turn_cost in [2, 3, 1]:
|
||||
session.add_assist_turn_cost(turn_cost)
|
||||
|
||||
assert session.assist_cost_cents == 6 # 2 + 3 + 1
|
||||
|
||||
|
||||
# ── check_c3_budget ────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_c3_budget_within_budget_typical_usage():
|
||||
"""20 turns/shift × 20 shifts/month at ~$0.0005/turn → within budget.
|
||||
|
||||
Example from the plan: 400 turns/month at ~$0.0005/turn = ~$0.20/month —
|
||||
well under the $3 C-3 target.
|
||||
"""
|
||||
# cost_per_turn_cents = 0.05 cents ($0.0005) — gemma4:cloud pilot rate.
|
||||
result = check_c3_budget(
|
||||
assist_turns_per_shift=20,
|
||||
shifts_per_month=20,
|
||||
cost_per_turn_cents=0.05,
|
||||
)
|
||||
assert result["turns_per_month"] == 400
|
||||
# 400 * 0.05 / 100 = $0.20/month
|
||||
assert result["monthly_assist_cost"] < 1.0
|
||||
assert result["total_with_practice"] < C3_TARGET_USD
|
||||
assert result["within_budget"] is True
|
||||
assert result["flag"] is False
|
||||
assert result["c3_target"] == C3_TARGET_USD == 3.0
|
||||
|
||||
|
||||
def test_c3_budget_exceeds_with_high_usage():
|
||||
"""100 turns/shift × 30 shifts/month at higher cost → may exceed (flag=True).
|
||||
|
||||
3000 turns/month at 0.15 cents/turn = $4.50/month → exceeds $3.
|
||||
"""
|
||||
result = check_c3_budget(
|
||||
assist_turns_per_shift=100,
|
||||
shifts_per_month=30,
|
||||
cost_per_turn_cents=0.15,
|
||||
)
|
||||
assert result["turns_per_month"] == 3000
|
||||
# 3000 * 0.15 / 100 = $4.50/month → over $3
|
||||
assert result["monthly_assist_cost"] > C3_TARGET_USD
|
||||
assert result["within_budget"] is False
|
||||
assert result["flag"] is True # diagnostic flag (not enforced)
|
||||
|
||||
|
||||
def test_c3_budget_with_practice_cost():
|
||||
"""total_with_practice = assist + practice cost."""
|
||||
result = check_c3_budget(
|
||||
assist_turns_per_shift=20,
|
||||
shifts_per_month=20,
|
||||
cost_per_turn_cents=0.05,
|
||||
practice_cost_per_month_usd=1.5,
|
||||
)
|
||||
# assist = $0.20, practice = $1.50 → total = $1.70 (within $3)
|
||||
assert result["practice_cost_per_month"] == 1.5
|
||||
assert result["total_with_practice"] < C3_TARGET_USD
|
||||
assert result["within_budget"] is True
|
||||
|
||||
|
||||
def test_c3_budget_with_practice_cost_exceeds():
|
||||
"""Assist + practice cost exceeds $3 → flag=True (diagnostic)."""
|
||||
result = check_c3_budget(
|
||||
assist_turns_per_shift=50,
|
||||
shifts_per_month=30,
|
||||
cost_per_turn_cents=0.10,
|
||||
practice_cost_per_month_usd=2.0,
|
||||
)
|
||||
# assist = 1500 * 0.10 / 100 = $1.50, practice = $2.00 → total = $3.50
|
||||
assert result["total_with_practice"] > C3_TARGET_USD
|
||||
assert result["within_budget"] is False
|
||||
assert result["flag"] is True
|
||||
|
||||
|
||||
def test_c3_budget_zero_usage():
|
||||
"""0 turns → zero cost, within budget."""
|
||||
result = check_c3_budget(
|
||||
assist_turns_per_shift=0,
|
||||
shifts_per_month=0,
|
||||
cost_per_turn_cents=0.05,
|
||||
)
|
||||
assert result["turns_per_month"] == 0
|
||||
assert result["monthly_assist_cost"] == 0.0
|
||||
assert result["within_budget"] is True
|
||||
assert result["flag"] is False
|
||||
|
||||
|
||||
def test_c3_budget_is_diagnostic_not_enforced():
|
||||
"""D-012: the check is diagnostic (not enforced). flag=True does not raise.
|
||||
|
||||
The check_c3_budget() function returns a dict with flag=True when over
|
||||
budget, but does NOT raise an exception (D-012 — no enforced ceiling in
|
||||
the pilot). The caller logs the flag + continues.
|
||||
"""
|
||||
result = check_c3_budget(
|
||||
assist_turns_per_shift=1000,
|
||||
shifts_per_month=30,
|
||||
cost_per_turn_cents=1.0,
|
||||
)
|
||||
# 30000 turns * 1.0 cent / 100 = $300/month → way over $3
|
||||
assert result["flag"] is True
|
||||
assert result["within_budget"] is False
|
||||
# No exception raised — the function returns a dict (diagnostic, not enforced).
|
||||
|
||||
|
||||
def test_c3_target_is_3_usd():
|
||||
"""C-3 target is ≤ $3/active learner/month (C-3, D-012)."""
|
||||
assert C3_TARGET_USD == 3.0
|
||||
@@ -125,13 +125,7 @@ def _make_processor(session=None, llm_context=None) -> LiveAssistGuardrailProces
|
||||
|
||||
|
||||
def test_processor_passes_allowed_text_through():
|
||||
"""Allowed coaching text → buffered, then pushed to TTS as one frame (REQ-ASSIST-03).
|
||||
|
||||
The guardrail MUST complete its check before any text reaches TTS. This means
|
||||
text is buffered (not streamed) + pushed as a single TextFrame on
|
||||
LLMFullResponseEndFrame after the guardrail allows it. This is the safety-critical
|
||||
behavior: a blocked response never reaches TTS.
|
||||
"""
|
||||
"""Allowed coaching text → pass through to TTS (no block)."""
|
||||
proc = _make_processor()
|
||||
|
||||
async def _run():
|
||||
@@ -145,14 +139,8 @@ def test_processor_passes_allowed_text_through():
|
||||
await proc.process_frame(end_frame, direction=1)
|
||||
|
||||
asyncio.run(_run())
|
||||
# The buffered text was pushed as a single TextFrame (not streamed chunk-by-chunk).
|
||||
pushed_texts = [
|
||||
call.args[0].text for call in proc.push_frame.await_args_list
|
||||
if hasattr(call.args[0], "text")
|
||||
]
|
||||
assert "What do you think the customer needs?" in pushed_texts
|
||||
# The LLMFullResponseEndFrame was also pushed (to signal TTS the response is done).
|
||||
assert proc.push_frame.await_count >= 2 # 1 buffered text + 1 end frame
|
||||
# The TextFrames were pushed (passed through).
|
||||
assert proc.push_frame.await_count >= 3 # 2 text + 1 end frame
|
||||
|
||||
|
||||
def test_processor_blocks_direct_answer():
|
||||
|
||||
+1
-278
@@ -89,65 +89,6 @@ def test_cookie_secret_unset_generates_random(monkeypatch):
|
||||
assert len(kw["secret_key"]) >= 32
|
||||
|
||||
|
||||
def test_cookie_secret_short_logs_warning_accepted(monkeypatch, caplog):
|
||||
"""TASK-12-02 (P1+ #3): a secret <32 bytes logs a WARNING but is accepted.
|
||||
|
||||
A short non-empty secret (e.g., 'x') weakens the HMAC signature. The
|
||||
secret is still accepted (backward compat — pilot); post-pilot this
|
||||
should be a hard error. The WARNING is logged with remediation guidance.
|
||||
"""
|
||||
from loguru import logger as _logger
|
||||
|
||||
monkeypatch.setenv("PRAXIS_COOKIE_SECRET", "short-secret") # 11 bytes < 32
|
||||
monkeypatch.setenv("PRAXIS_COOKIE_SECURE", "true")
|
||||
# Capture loguru warnings.
|
||||
msgs: list[str] = []
|
||||
sink_id = _logger.add(lambda m: msgs.append(str(m)), level="WARNING")
|
||||
try:
|
||||
kw = get_session_middleware_kwargs()
|
||||
finally:
|
||||
_logger.remove(sink_id)
|
||||
# The short secret is accepted (backward compat — no hard error in pilot).
|
||||
assert kw["secret_key"] == "short-secret"
|
||||
# A WARNING about the short secret was logged.
|
||||
assert any("<32 bytes" in m for m in msgs), \
|
||||
"short PRAXIS_COOKIE_SECRET should log a <32 bytes WARNING"
|
||||
|
||||
|
||||
def test_cookie_secret_32_bytes_no_warning(monkeypatch, caplog):
|
||||
"""TASK-12-02: a secret >=32 bytes logs no <32 bytes warning."""
|
||||
from loguru import logger as _logger
|
||||
|
||||
monkeypatch.setenv("PRAXIS_COOKIE_SECRET", "x" * 32) # exactly 32 bytes
|
||||
monkeypatch.setenv("PRAXIS_COOKIE_SECURE", "true")
|
||||
msgs: list[str] = []
|
||||
sink_id = _logger.add(lambda m: msgs.append(str(m)), level="WARNING")
|
||||
try:
|
||||
kw = get_session_middleware_kwargs()
|
||||
finally:
|
||||
_logger.remove(sink_id)
|
||||
assert kw["secret_key"] == "x" * 32
|
||||
# No <32 bytes warning (the secret is exactly 32 bytes).
|
||||
assert not any("<32 bytes" in m for m in msgs), \
|
||||
"32-byte secret should NOT log a <32 bytes warning"
|
||||
|
||||
|
||||
def test_cookie_secret_long_no_warning(monkeypatch):
|
||||
"""TASK-12-02: a secret >32 bytes logs no warning."""
|
||||
from loguru import logger as _logger
|
||||
|
||||
monkeypatch.setenv("PRAXIS_COOKIE_SECRET", "x" * 64) # 64 bytes
|
||||
monkeypatch.setenv("PRAXIS_COOKIE_SECURE", "true")
|
||||
msgs: list[str] = []
|
||||
sink_id = _logger.add(lambda m: msgs.append(str(m)), level="WARNING")
|
||||
try:
|
||||
kw = get_session_middleware_kwargs()
|
||||
finally:
|
||||
_logger.remove(sink_id)
|
||||
assert kw["secret_key"] == "x" * 64
|
||||
assert not any("<32 bytes" in m for m in msgs)
|
||||
|
||||
|
||||
# ── current_operator dependency ─────────────────────────────────────────────
|
||||
|
||||
|
||||
@@ -366,222 +307,4 @@ def test_rate_limit_login_decorator():
|
||||
|
||||
|
||||
def test_limiter_is_in_memory():
|
||||
assert getattr(limiter, "_storage_uri", "memory://") == "memory://" or limiter._storage is not None
|
||||
|
||||
|
||||
# ── TASK-12-04 (P1+ #1/#2/#5): argon2id offload + 429 mock test + audit log ──
|
||||
|
||||
|
||||
def test_login_argon2id_offloaded_to_thread():
|
||||
"""TASK-12-04 (P1+ #1): verify_password is offloaded to asyncio.to_thread.
|
||||
|
||||
The login handler should call verify_password via asyncio.to_thread (not
|
||||
directly) so the ~100-300ms argon2id hashing does not block the event loop.
|
||||
We verify by patching asyncio.to_thread to record the call.
|
||||
"""
|
||||
import asyncio as _asyncio
|
||||
|
||||
op = {
|
||||
"id": "44444444-4444-4444-4444-444444444444",
|
||||
"username": "erin",
|
||||
"display_name": "Erin",
|
||||
"role": "operator",
|
||||
"is_active": True,
|
||||
"password_hash": hash_password("pw"),
|
||||
}
|
||||
store = _mock_store(operator_row=op)
|
||||
store.get_operator_by_username = AsyncMock(return_value=op)
|
||||
store.update_last_login = AsyncMock()
|
||||
store.pool = MagicMock()
|
||||
conn = MagicMock()
|
||||
conn.execute = AsyncMock()
|
||||
cm = MagicMock()
|
||||
cm.__aenter__ = AsyncMock(return_value=conn)
|
||||
cm.__aexit__ = AsyncMock(return_value=None)
|
||||
store.pool.acquire = MagicMock(return_value=cm)
|
||||
|
||||
to_thread_calls: list = []
|
||||
real_to_thread = _asyncio.to_thread
|
||||
|
||||
async def _spy_to_thread(func, *args, **kwargs):
|
||||
to_thread_calls.append((func, args, kwargs))
|
||||
return await real_to_thread(func, *args, **kwargs)
|
||||
|
||||
import server.auth.routes as _routes_mod
|
||||
orig = _routes_mod.asyncio.to_thread
|
||||
_routes_mod.asyncio.to_thread = _spy_to_thread
|
||||
try:
|
||||
app = _make_app_with_store(store)
|
||||
with TestClient(app) as client:
|
||||
r = client.post("/api/operator/login", json={"username": "erin", "password": "pw"})
|
||||
assert r.status_code == 200
|
||||
finally:
|
||||
_routes_mod.asyncio.to_thread = orig
|
||||
|
||||
# verify_password should have been called via asyncio.to_thread.
|
||||
assert to_thread_calls, "login should offload verify_password to asyncio.to_thread"
|
||||
func = to_thread_calls[0][0]
|
||||
assert func.__name__ == "verify_password", (
|
||||
f"expected verify_password offloaded, got {func.__name__}"
|
||||
)
|
||||
|
||||
|
||||
def test_login_rehash_offloaded_to_thread():
|
||||
"""TASK-12-04 (P1+ #1): hash_password (rehash) is also offloaded to thread."""
|
||||
from argon2 import PasswordHasher
|
||||
|
||||
weak_hasher = PasswordHasher(time_cost=1, memory_cost=8, parallelism=1)
|
||||
op = {
|
||||
"id": "55555555-5555-5555-5555-555555555555",
|
||||
"username": "frank",
|
||||
"display_name": "Frank",
|
||||
"role": "operator",
|
||||
"is_active": True,
|
||||
"password_hash": weak_hasher.hash("pw"),
|
||||
}
|
||||
store = _mock_store(operator_row=op)
|
||||
store.get_operator_by_username = AsyncMock(return_value=op)
|
||||
store.update_last_login = AsyncMock()
|
||||
store.pool = MagicMock()
|
||||
conn = MagicMock()
|
||||
conn.execute = AsyncMock()
|
||||
cm = MagicMock()
|
||||
cm.__aenter__ = AsyncMock(return_value=conn)
|
||||
cm.__aexit__ = AsyncMock(return_value=None)
|
||||
store.pool.acquire = MagicMock(return_value=cm)
|
||||
|
||||
import asyncio as _asyncio
|
||||
import server.auth.routes as _routes_mod
|
||||
|
||||
to_thread_calls: list = []
|
||||
real_to_thread = _routes_mod.asyncio.to_thread
|
||||
|
||||
async def _spy_to_thread(func, *args, **kwargs):
|
||||
to_thread_calls.append((func, args, kwargs))
|
||||
return await real_to_thread(func, *args, **kwargs)
|
||||
|
||||
_routes_mod.asyncio.to_thread = _spy_to_thread
|
||||
try:
|
||||
app = _make_app_with_store(store)
|
||||
with TestClient(app) as client:
|
||||
r = client.post("/api/operator/login", json={"username": "frank", "password": "pw"})
|
||||
assert r.status_code == 200
|
||||
finally:
|
||||
_routes_mod.asyncio.to_thread = real_to_thread
|
||||
|
||||
# Both verify_password + hash_password should be offloaded.
|
||||
func_names = [c[0].__name__ for c in to_thread_calls]
|
||||
assert "verify_password" in func_names
|
||||
assert "hash_password" in func_names, "rehash should offload hash_password to thread"
|
||||
|
||||
|
||||
def test_login_rate_limit_429_after_5_attempts():
|
||||
"""TASK-12-04 (P1+ #2): mock-based 429 test — 6th login attempt → 429.
|
||||
|
||||
The full 6th-attempt→429 path is in the PG-requiring integration test; this
|
||||
adds a mock-based test for CI coverage without Postgres. slowapi's in-memory
|
||||
limiter tracks per-IP; 5/minute → 6th attempt gets 429.
|
||||
"""
|
||||
from slowapi.errors import RateLimitExceeded
|
||||
from slowapi.middleware import SlowAPIMiddleware
|
||||
from slowapi import _rate_limit_exceeded_handler
|
||||
|
||||
op = {
|
||||
"id": "66666666-6666-6666-6666-666666666666",
|
||||
"username": "grace",
|
||||
"display_name": "Grace",
|
||||
"role": "operator",
|
||||
"is_active": True,
|
||||
"password_hash": hash_password("pw"),
|
||||
}
|
||||
store = _mock_store(operator_row=op)
|
||||
store.get_operator_by_username = AsyncMock(return_value=op)
|
||||
store.update_last_login = AsyncMock()
|
||||
store.pool = MagicMock()
|
||||
conn = MagicMock()
|
||||
conn.execute = AsyncMock()
|
||||
cm = MagicMock()
|
||||
cm.__aenter__ = AsyncMock(return_value=conn)
|
||||
cm.__aexit__ = AsyncMock(return_value=None)
|
||||
store.pool.acquire = MagicMock(return_value=cm)
|
||||
|
||||
app = _make_app_with_store(store)
|
||||
app.state.limiter = limiter
|
||||
app.add_middleware(SlowAPIMiddleware)
|
||||
app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler)
|
||||
|
||||
with TestClient(app) as client:
|
||||
# 5 attempts should succeed (or 401 for wrong password — both count).
|
||||
statuses: list[int] = []
|
||||
for _ in range(5):
|
||||
r = client.post(
|
||||
"/api/operator/login", json={"username": "grace", "password": "pw"}
|
||||
)
|
||||
statuses.append(r.status_code)
|
||||
# The 5 attempts should not be 429 (within the 5/minute limit).
|
||||
assert all(s != 429 for s in statuses), f"first 5 should not be 429: {statuses}"
|
||||
# 6th attempt → 429 (rate limit exceeded).
|
||||
r6 = client.post(
|
||||
"/api/operator/login", json={"username": "grace", "password": "pw"}
|
||||
)
|
||||
assert r6.status_code == 429, (
|
||||
f"6th login attempt should be rate-limited (429), got {r6.status_code}"
|
||||
)
|
||||
|
||||
|
||||
def test_credential_revocation_logs_audit_event():
|
||||
"""TASK-12-04 (P1+ #5): credential revocation logs operator + cred_id.
|
||||
|
||||
The revoke_credential endpoint should log an application-level audit event
|
||||
(no audit_log table — the log is sufficient for pilot per D-056).
|
||||
"""
|
||||
import logging as _logging
|
||||
|
||||
from server.operator.credentials import router as creds_router
|
||||
|
||||
op = {
|
||||
"id": "77777777-7777-7777-7777-777777777777",
|
||||
"username": "heidi",
|
||||
"display_name": "Heidi",
|
||||
"role": "operator",
|
||||
}
|
||||
store = MagicMock()
|
||||
store.get_credential = AsyncMock(return_value={"id": "cred-xyz", "status": "active"})
|
||||
store.set_credential_status = AsyncMock()
|
||||
|
||||
app = FastAPI()
|
||||
app.state.pg_store = store
|
||||
app.add_middleware(SessionMiddleware, secret_key="test-secret-1234567890abcdef")
|
||||
app.include_router(creds_router)
|
||||
# Stub auth.
|
||||
from server.auth.dependencies import current_operator
|
||||
from server.auth.models import Operator
|
||||
|
||||
async def _stub_op():
|
||||
return Operator(id=op["id"], username=op["username"],
|
||||
display_name=op["display_name"], role=op["role"])
|
||||
app.dependency_overrides[current_operator] = _stub_op
|
||||
|
||||
# Capture the audit log.
|
||||
cred_log = _logging.getLogger("server.operator.credentials")
|
||||
records: list[_logging.LogRecord] = []
|
||||
handler = _logging.Handler()
|
||||
handler.emit = records.append # type: ignore[method-assign]
|
||||
cred_log.addHandler(handler)
|
||||
cred_log.setLevel(_logging.INFO)
|
||||
try:
|
||||
with TestClient(app) as client:
|
||||
r = client.post("/api/operator/credentials/cred-xyz/revoke")
|
||||
assert r.status_code == 200
|
||||
assert r.json()["status"] == "revoked"
|
||||
finally:
|
||||
cred_log.removeHandler(handler)
|
||||
|
||||
# The audit log should contain the operator id + cred_id.
|
||||
audit_msgs = [r.getMessage() for r in records if r.levelno >= _logging.INFO]
|
||||
assert any("credential revoked" in m for m in audit_msgs), \
|
||||
f"revocation should log 'credential revoked': {audit_msgs}"
|
||||
assert any("cred-xyz" in m for m in audit_msgs), \
|
||||
f"audit log should contain cred_id: {audit_msgs}"
|
||||
assert any(op["id"] in m for m in audit_msgs), \
|
||||
f"audit log should contain operator id: {audit_msgs}"
|
||||
assert getattr(limiter, "_storage_uri", "memory://") == "memory://" or limiter._storage is not None
|
||||
@@ -1,416 +0,0 @@
|
||||
"""Cohort assist aggregation tests (TASK-10-03, D-062, D-063, REQ-NFR-ASSIST-04).
|
||||
|
||||
Tests the _aggregate_assist branch in server/cohort/aggregator.py with a
|
||||
mocked PgStore (no Postgres required). Verifies:
|
||||
- _aggregate_assist() upserts the 5 core assist metrics + p95 latency.
|
||||
- k-anon suppression: <10 distinct learners → suppressed.
|
||||
- Idempotent upsert: same session_outcome twice → same aggregate.
|
||||
- assist_guardrail_block_rate = blocks / turns.
|
||||
- The practice branch (_aggregate_practice) is unchanged (backward compat).
|
||||
- The dashboard endpoints return assist rows (cohort + failure-patterns).
|
||||
|
||||
D-063 (binding): assist does NOT update mastery. The _aggregate_assist branch
|
||||
computes NO mastery metrics (no rubric scores, no gate_open_rate).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as _dt
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from server.cohort.aggregator import (
|
||||
K_ANON_THRESHOLD,
|
||||
_aggregate_assist,
|
||||
_aggregate_practice,
|
||||
aggregate_session,
|
||||
)
|
||||
from server.cohort.hook import on_session_end
|
||||
|
||||
|
||||
def _mock_pg_store():
|
||||
store = MagicMock()
|
||||
store.upsert_cohort_aggregate = AsyncMock()
|
||||
return store
|
||||
|
||||
|
||||
def _assist_outcome(
|
||||
learner_ref: str,
|
||||
path: str = "customer_service",
|
||||
turn_count: int = 20,
|
||||
blocks: int = 2,
|
||||
p95_latency_ms: float | None = 580.0,
|
||||
cost_cents: int = 20,
|
||||
outcome: str = "completed",
|
||||
) -> dict:
|
||||
return {
|
||||
"learner_ref": learner_ref,
|
||||
"path": path,
|
||||
"scenario_id": f"assist:refund",
|
||||
"outcome": outcome,
|
||||
"session_type": "assist",
|
||||
"rubric_scores": [], # D-063: no rubric scores for assist
|
||||
"failure_mode": None,
|
||||
"branch_path": [],
|
||||
"assist_turn_count": turn_count,
|
||||
"guardrail_blocks": blocks,
|
||||
"assist_p95_latency_ms": p95_latency_ms,
|
||||
"assist_p50_latency_ms": 500.0,
|
||||
"assist_p99_latency_ms": 620.0,
|
||||
"assist_within_pilot": True,
|
||||
"assist_cost_cents": cost_cents,
|
||||
"timestamp": _dt.datetime.now(_dt.timezone.utc).isoformat(),
|
||||
}
|
||||
|
||||
|
||||
def _practice_outcome(learner_ref: str, path: str = "customer_service") -> dict:
|
||||
return {
|
||||
"learner_ref": learner_ref,
|
||||
"path": path,
|
||||
"scenario_id": f"{path}_v01",
|
||||
"outcome": "pass",
|
||||
"session_type": "practice",
|
||||
"rubric_scores": [{"criterion_id": "empathy", "score": 4.0}],
|
||||
"failure_mode": None,
|
||||
"branch_path": ["accept"],
|
||||
"timestamp": _dt.datetime.now(_dt.timezone.utc).isoformat(),
|
||||
}
|
||||
|
||||
|
||||
# ── Assist metrics upserted ─────────────────────────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_aggregate_assist_upserts_5_core_metrics():
|
||||
"""_aggregate_assist upserts the 5 core assist metrics (REQ-NFR-ASSIST-04)."""
|
||||
store = _mock_pg_store()
|
||||
await _aggregate_assist(store, _assist_outcome("learner-1"))
|
||||
metrics = {c.args[1] for c in store.upsert_cohort_aggregate.call_args_list}
|
||||
assert "assist_shifts_count" in metrics
|
||||
assert "assist_active_learners_count" in metrics
|
||||
assert "assist_turns_count" in metrics
|
||||
assert "assist_avg_turns_per_shift" in metrics
|
||||
assert "assist_guardrail_block_rate" in metrics
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_aggregate_assist_upserts_p95_latency():
|
||||
"""_aggregate_assist upserts assist_p95_latency_ms (D-072, TASK-09-01)."""
|
||||
store = _mock_pg_store()
|
||||
await _aggregate_assist(store, _assist_outcome("learner-1", p95_latency_ms=580.0))
|
||||
metrics = {c.args[1] for c in store.upsert_cohort_aggregate.call_args_list}
|
||||
assert "assist_p95_latency_ms" in metrics
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_aggregate_assist_upserts_avg_cost():
|
||||
"""_aggregate_assist upserts assist_avg_cost_per_shift (TASK-11-01)."""
|
||||
store = _mock_pg_store()
|
||||
await _aggregate_assist(store, _assist_outcome("learner-1", cost_cents=25))
|
||||
metrics = {c.args[1] for c in store.upsert_cohort_aggregate.call_args_list}
|
||||
assert "assist_avg_cost_per_shift" in metrics
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_aggregate_assist_no_mastery_metrics():
|
||||
"""D-063: _aggregate_assist computes NO mastery metrics."""
|
||||
store = _mock_pg_store()
|
||||
await _aggregate_assist(store, _assist_outcome("learner-1"))
|
||||
metrics = {c.args[1] for c in store.upsert_cohort_aggregate.call_args_list}
|
||||
# No mastery metrics should be present.
|
||||
assert "gate_open_rate" not in metrics
|
||||
assert "median_mastery_score" not in metrics
|
||||
assert not any(m.startswith("rubric_criterion_mean:") for m in metrics)
|
||||
# No practice metrics either (assist is a separate branch).
|
||||
assert "sessions_count" not in metrics
|
||||
|
||||
|
||||
# ── k-anonymity suppression ──────────────────────────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_assist_9_learners_suppressed():
|
||||
"""<10 distinct learners → all assist cells suppressed."""
|
||||
store = _mock_pg_store()
|
||||
for i in range(9):
|
||||
await _aggregate_assist(store, _assist_outcome(f"learner-{i}"))
|
||||
suppressed = [c for c in store.upsert_cohort_aggregate.call_args_list if c.args[6] is True]
|
||||
non_suppressed = [c for c in store.upsert_cohort_aggregate.call_args_list if c.args[6] is False]
|
||||
assert suppressed, "assist cells should be suppressed with <10 learners"
|
||||
assert not non_suppressed, "no assist cell should be non-suppressed with 9 learners"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_assist_10_learners_not_suppressed():
|
||||
"""≥10 distinct learners → assist cells not suppressed."""
|
||||
store = _mock_pg_store()
|
||||
for i in range(10):
|
||||
await _aggregate_assist(store, _assist_outcome(f"learner-{i}"))
|
||||
non_suppressed = [c for c in store.upsert_cohort_aggregate.call_args_list if c.args[6] is False]
|
||||
assert non_suppressed, "assist cells should NOT be suppressed at 10 learners"
|
||||
for c in non_suppressed:
|
||||
assert c.args[4] is not None, "non-suppressed cell value must not be None"
|
||||
|
||||
|
||||
# ── Idempotent upsert ───────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_assist_idempotent_same_outcome_twice():
|
||||
"""Re-running with the same outcome produces consistent upserts (idempotent)."""
|
||||
store = _mock_pg_store()
|
||||
outcome = _assist_outcome("learner-x")
|
||||
await _aggregate_assist(store, outcome)
|
||||
first_call_count = store.upsert_cohort_aggregate.call_count
|
||||
await _aggregate_assist(store, outcome)
|
||||
second_call_count = store.upsert_cohort_aggregate.call_count
|
||||
# Both runs produce upsert calls (the DB ON CONFLICT makes them idempotent).
|
||||
assert second_call_count >= first_call_count
|
||||
assert store.upsert_cohort_aggregate.called
|
||||
|
||||
|
||||
# ── assist_guardrail_block_rate = blocks / turns ─────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_assist_guardrail_block_rate_computed():
|
||||
"""assist_guardrail_block_rate = blocks / turns (safety signal)."""
|
||||
store = _mock_pg_store()
|
||||
# 10 learners so the cell is not suppressed (we can read the value).
|
||||
for i in range(10):
|
||||
await _aggregate_assist(store, _assist_outcome(f"learner-{i}", turn_count=20, blocks=2))
|
||||
block_rate_cells = [
|
||||
c for c in store.upsert_cohort_aggregate.call_args_list
|
||||
if c.args[1] == "assist_guardrail_block_rate" and c.args[6] is False
|
||||
]
|
||||
assert block_rate_cells, "should have a non-suppressed assist_guardrail_block_rate cell"
|
||||
# The running mean of per-shift block rates (2/20 = 0.1) → ~0.1.
|
||||
rate = block_rate_cells[-1].args[4]
|
||||
assert rate is not None
|
||||
assert 0.05 <= rate <= 0.15 # ~0.1 with running-mean drift
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_assist_zero_turns_block_rate_is_zero():
|
||||
"""0 turns → block_rate = 0.0 (no division by zero)."""
|
||||
store = _mock_pg_store()
|
||||
for i in range(10):
|
||||
await _aggregate_assist(store, _assist_outcome(f"learner-{i}", turn_count=0, blocks=0))
|
||||
block_rate_cells = [
|
||||
c for c in store.upsert_cohort_aggregate.call_args_list
|
||||
if c.args[1] == "assist_guardrail_block_rate" and c.args[6] is False
|
||||
]
|
||||
assert block_rate_cells
|
||||
rate = block_rate_cells[-1].args[4]
|
||||
assert rate == 0.0
|
||||
|
||||
|
||||
# ── Practice branch unchanged (backward compat) ─────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_aggregate_session_dispatches_to_practice():
|
||||
"""aggregate_session with session_type='practice' → _aggregate_practice."""
|
||||
store = _mock_pg_store()
|
||||
await aggregate_session(store, _practice_outcome("learner-1"))
|
||||
metrics = {c.args[1] for c in store.upsert_cohort_aggregate.call_args_list}
|
||||
# Practice metrics should be present.
|
||||
assert "sessions_count" in metrics
|
||||
assert "active_learners_count" in metrics
|
||||
# Assist metrics should NOT be present (practice branch).
|
||||
assert "assist_shifts_count" not in metrics
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_aggregate_session_dispatches_to_assist():
|
||||
"""aggregate_session with session_type='assist' → _aggregate_assist."""
|
||||
store = _mock_pg_store()
|
||||
await aggregate_session(store, _assist_outcome("learner-1"))
|
||||
metrics = {c.args[1] for c in store.upsert_cohort_aggregate.call_args_list}
|
||||
assert "assist_shifts_count" in metrics
|
||||
assert "sessions_count" not in metrics
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_aggregate_session_default_is_practice():
|
||||
"""aggregate_session with no session_type → practice (backward compat)."""
|
||||
store = _mock_pg_store()
|
||||
outcome = _practice_outcome("learner-1")
|
||||
outcome.pop("session_type") # omit session_type → default practice
|
||||
await aggregate_session(store, outcome)
|
||||
metrics = {c.args[1] for c in store.upsert_cohort_aggregate.call_args_list}
|
||||
assert "sessions_count" in metrics
|
||||
assert "assist_shifts_count" not in metrics
|
||||
|
||||
|
||||
# ── No PII in assist upsert calls ───────────────────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_assist_no_pii_in_upsert_calls():
|
||||
"""No raw learner_ref leaks into assist aggregate cell args (D-031)."""
|
||||
store = _mock_pg_store()
|
||||
await _aggregate_assist(store, _assist_outcome("learner-sensitive-id-1234"))
|
||||
for c in store.upsert_cohort_aggregate.call_args_list:
|
||||
for arg in c.args:
|
||||
assert "learner-sensitive-id-1234" not in str(arg), \
|
||||
"raw learner_ref must not leak into assist aggregate cell args"
|
||||
assert isinstance(c.args[5], int) # cell_count is an int
|
||||
|
||||
|
||||
# ── Hook dispatches assist correctly ───────────────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_hook_dispatches_assist_session():
|
||||
"""on_session_end with session_type='assist' → _aggregate_assist (no error)."""
|
||||
store = _mock_pg_store()
|
||||
await on_session_end(store, _assist_outcome("learner-1"))
|
||||
assert store.upsert_cohort_aggregate.called
|
||||
metrics = {c.args[1] for c in store.upsert_cohort_aggregate.call_args_list}
|
||||
assert "assist_shifts_count" in metrics
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_hook_assist_no_postgres_is_noop():
|
||||
"""on_session_end with no Postgres → no-op (assist hook)."""
|
||||
await on_session_end(None, _assist_outcome("learner-1"))
|
||||
|
||||
|
||||
# ── Dashboard endpoints return assist rows ──────────────────────────────────
|
||||
|
||||
|
||||
def _make_app_with_assist_rows(rows: list[dict]):
|
||||
"""Build a minimal FastAPI app with the cohort + failure-patterns routers
|
||||
+ a mocked pg_store returning `rows`."""
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
from server.operator.cohort import router as cohort_router
|
||||
from server.operator.failure_patterns import router as failure_router
|
||||
|
||||
app = FastAPI()
|
||||
pg_store = MagicMock()
|
||||
pg_store.pool = MagicMock()
|
||||
conn = MagicMock()
|
||||
conn.fetch = AsyncMock(return_value=rows)
|
||||
cm = MagicMock()
|
||||
cm.__aenter__ = AsyncMock(return_value=conn)
|
||||
cm.__aexit__ = AsyncMock(return_value=None)
|
||||
pg_store.pool.acquire = MagicMock(return_value=cm)
|
||||
app.state.pg_store = pg_store
|
||||
# Bypass auth for these tests by stubbing current_operator.
|
||||
from server.auth.dependencies import current_operator
|
||||
from server.auth.models import Operator
|
||||
|
||||
async def _stub_op():
|
||||
return Operator(id="op-1", username="tester", display_name="T", role="operator")
|
||||
app.dependency_overrides[current_operator] = _stub_op
|
||||
app.include_router(cohort_router)
|
||||
app.include_router(failure_router)
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def _assist_metric_row(metric: str, value: float, suppressed: bool = False) -> dict:
|
||||
return {
|
||||
"path": "customer_service",
|
||||
"metric": metric,
|
||||
"window_start": _dt.date.today() - _dt.timedelta(days=6),
|
||||
"window_end": _dt.date.today(),
|
||||
"value": value if not suppressed else None,
|
||||
"cell_count": 12,
|
||||
"cell_suppressed": suppressed,
|
||||
"updated_at": _dt.datetime.now(_dt.timezone.utc),
|
||||
}
|
||||
|
||||
|
||||
def test_cohort_endpoint_returns_assist_rows():
|
||||
"""GET /api/operator/cohort returns assist_shifts_count + assist_turns_count."""
|
||||
rows = [
|
||||
_assist_metric_row("sessions_count", 15.0),
|
||||
_assist_metric_row("active_learners_count", 12.0),
|
||||
_assist_metric_row("assist_shifts_count", 8.0),
|
||||
_assist_metric_row("assist_turns_count", 160.0),
|
||||
]
|
||||
client = _make_app_with_assist_rows(rows)
|
||||
r = client.get("/api/operator/cohort")
|
||||
assert r.status_code == 200
|
||||
data = r.json()
|
||||
metrics = {c["metric"] for v in data["views"] for c in v["metrics"]}
|
||||
assert "assist_shifts_count" in metrics
|
||||
assert "assist_turns_count" in metrics
|
||||
assert "sessions_count" in metrics # practice still present
|
||||
|
||||
|
||||
def test_failure_patterns_endpoint_returns_guardrail_block_rate():
|
||||
"""GET /api/operator/failure-patterns returns assist_guardrail_block_rate."""
|
||||
rows = [
|
||||
_assist_metric_row("failure_mode:missed_apology", 3.0),
|
||||
_assist_metric_row("branch:escalate", 5.0),
|
||||
_assist_metric_row("assist_guardrail_block_rate", 0.08),
|
||||
]
|
||||
client = _make_app_with_assist_rows(rows)
|
||||
r = client.get("/api/operator/failure-patterns")
|
||||
assert r.status_code == 200
|
||||
data = r.json()
|
||||
metrics = {c["metric"] for v in data["views"] for c in v["metrics"]}
|
||||
assert "assist_guardrail_block_rate" in metrics
|
||||
assert "failure_mode:missed_apology" in metrics # practice failure patterns still present
|
||||
|
||||
|
||||
def test_mastery_endpoint_excludes_assist_metrics():
|
||||
"""D-063: GET /api/operator/mastery does NOT return assist metrics."""
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
from server.auth.dependencies import current_operator
|
||||
from server.auth.models import Operator
|
||||
from server.operator.mastery import router as mastery_router
|
||||
|
||||
rows = [
|
||||
_assist_metric_row("gate_open_rate", 0.5),
|
||||
_assist_metric_row("median_mastery_score", 3.8),
|
||||
_assist_metric_row("assist_shifts_count", 8.0), # should be EXCLUDED
|
||||
_assist_metric_row("assist_guardrail_block_rate", 0.08), # EXCLUDED
|
||||
]
|
||||
app = FastAPI()
|
||||
pg_store = MagicMock()
|
||||
pg_store.pool = MagicMock()
|
||||
conn = MagicMock()
|
||||
conn.fetch = AsyncMock(return_value=rows)
|
||||
cm = MagicMock()
|
||||
cm.__aenter__ = AsyncMock(return_value=conn)
|
||||
cm.__aexit__ = AsyncMock(return_value=None)
|
||||
pg_store.pool.acquire = MagicMock(return_value=cm)
|
||||
app.state.pg_store = pg_store
|
||||
|
||||
async def _stub_op():
|
||||
return Operator(id="op-1", username="tester", display_name="T", role="operator")
|
||||
app.dependency_overrides[current_operator] = _stub_op
|
||||
app.include_router(mastery_router)
|
||||
client = TestClient(app)
|
||||
r = client.get("/api/operator/mastery")
|
||||
assert r.status_code == 200
|
||||
data = r.json()
|
||||
metrics = {c["metric"] for v in data["views"] for c in v["metrics"]}
|
||||
assert "gate_open_rate" in metrics
|
||||
assert "median_mastery_score" in metrics
|
||||
# D-063: assist metrics must NOT appear in the mastery view.
|
||||
assert "assist_shifts_count" not in metrics
|
||||
assert "assist_guardrail_block_rate" not in metrics
|
||||
|
||||
|
||||
def test_cohort_endpoint_suppressed_assist_cells():
|
||||
"""Suppressed assist cells have value=null + cell_suppressed=true (k-anon)."""
|
||||
rows = [
|
||||
_assist_metric_row("assist_shifts_count", 0.0, suppressed=True),
|
||||
_assist_metric_row("assist_turns_count", 0.0, suppressed=True),
|
||||
]
|
||||
client = _make_app_with_assist_rows(rows)
|
||||
r = client.get("/api/operator/cohort")
|
||||
assert r.status_code == 200
|
||||
data = r.json()
|
||||
for v in data["views"]:
|
||||
for c in v["metrics"]:
|
||||
if c["metric"] in ("assist_shifts_count", "assist_turns_count"):
|
||||
assert c["cell_suppressed"] is True
|
||||
assert c["value"] is None
|
||||
@@ -44,71 +44,6 @@ def test_seconds_until_next_03_ct_exactly_03_rolls_to_tomorrow():
|
||||
assert secs >= 86390 # ~24h
|
||||
|
||||
|
||||
# ── TASK-12-04 (P1+ #6): zoneinfo DST-aware scheduler ───────────────────────
|
||||
|
||||
|
||||
def test_nightly_scheduler_uses_zoneinfo_america_winnipeg():
|
||||
"""TASK-12-04 (P1+ #6): CT is zoneinfo.ZoneInfo('America/Winnipeg') (DST-aware).
|
||||
|
||||
The v0.4 fixed UTC-5 offset is replaced with ZoneInfo("America/Winnipeg")
|
||||
which correctly handles CST (UTC-6) in winter + CDT (UTC-5) in summer.
|
||||
"""
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
assert isinstance(CT, ZoneInfo), f"CT should be a ZoneInfo, got {type(CT)}"
|
||||
assert str(CT) == "America/Winnipeg", f"CT should be America/Winnipeg, got {CT}"
|
||||
|
||||
|
||||
def test_nightly_scheduler_dst_summer_cdt():
|
||||
"""TASK-12-04 (P1+ #6): summer (August) → CDT (UTC-5).
|
||||
|
||||
In August 2026, America/Winnipeg is on CDT (UTC-5). A 01:00 local time
|
||||
should be 06:00 UTC. The scheduler computes seconds until 03:00 local.
|
||||
"""
|
||||
# 2026-08-04 is summer → CDT (UTC-5).
|
||||
now_local = _dt.datetime(2026, 8, 4, 1, 0, tzinfo=CT)
|
||||
# 01:00 CDT = 06:00 UTC.
|
||||
assert now_local.utcoffset() == _dt.timedelta(hours=-5), (
|
||||
f"August should be CDT (UTC-5), got offset {now_local.utcoffset()}"
|
||||
)
|
||||
secs = seconds_until_next_03_ct(now_local)
|
||||
# 01:00 → 03:00 = 2h = 7200s.
|
||||
assert 7190 <= secs <= 7200
|
||||
|
||||
|
||||
def test_nightly_scheduler_dst_winter_cst():
|
||||
"""TASK-12-04 (P1+ #6): winter (January) → CST (UTC-6).
|
||||
|
||||
In January 2027, America/Winnipeg is on CST (UTC-6). A 01:00 local time
|
||||
should be 07:00 UTC. The v0.4 fixed UTC-5 offset would have been wrong
|
||||
by 1h in winter; the ZoneInfo correctly handles the DST transition.
|
||||
"""
|
||||
# 2027-01-15 is winter → CST (UTC-6).
|
||||
now_local = _dt.datetime(2027, 1, 15, 1, 0, tzinfo=CT)
|
||||
assert now_local.utcoffset() == _dt.timedelta(hours=-6), (
|
||||
f"January should be CST (UTC-6), got offset {now_local.utcoffset()}"
|
||||
)
|
||||
secs = seconds_until_next_03_ct(now_local)
|
||||
# 01:00 → 03:00 = 2h = 7200s.
|
||||
assert 7190 <= secs <= 7200
|
||||
|
||||
|
||||
def test_nightly_scheduler_dst_transition_spring_2027():
|
||||
"""TASK-12-04 (P1+ #6): DST spring forward — 2027-03-14 02:00 → 03:00 CDT.
|
||||
|
||||
On 2027-03-14, DST springs forward at 02:00 local (CST → CDT). The ZoneInfo
|
||||
correctly handles the transition (the 02:00 hour is skipped). The scheduler
|
||||
should still compute a valid seconds-until-03:00.
|
||||
"""
|
||||
# 2027-03-14 01:00 CST (before spring forward) → 03:00 CDT is 1h later
|
||||
# (the 02:00 hour is skipped → 01:59 CST → 03:00 CDT).
|
||||
now_local = _dt.datetime(2027, 3, 14, 1, 0, tzinfo=CT)
|
||||
secs = seconds_until_next_03_ct(now_local)
|
||||
# 01:00 CST → 03:00 CDT is 1h (the 02:00 hour is skipped).
|
||||
# The exact value depends on the DST transition; assert it's ≤ 2h.
|
||||
assert 0 < secs <= 7200, f"spring-forward seconds should be <= 2h, got {secs}"
|
||||
|
||||
|
||||
# ── Reconciliation recomputes all windows ──────────────────────────────────
|
||||
|
||||
|
||||
|
||||
@@ -1,106 +0,0 @@
|
||||
"""Mock-based tests for set_credential_status enum + f-string SQL fix
|
||||
(TASK-12-03, P1+ #4/#8 from v0.4 REVIEW).
|
||||
|
||||
These tests do NOT require Postgres (they use a mock asyncpg pool). They
|
||||
verify:
|
||||
- 'revoked' uses a parameterized query with revoked_at=now() (no f-string).
|
||||
- 'active' clears revoked_at=NULL (re-activation).
|
||||
- Invalid status → ValueError (enum validation — P1+ #4).
|
||||
- No f-string interpolation in the SQL (P1+ #8 code smell fix).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from db.pg_store import PgStore
|
||||
|
||||
|
||||
def _mock_pool_with_conn():
|
||||
"""Build a mock asyncpg pool + conn that records execute() calls."""
|
||||
pool = MagicMock()
|
||||
conn = MagicMock()
|
||||
conn.execute = AsyncMock()
|
||||
cm = MagicMock()
|
||||
cm.__aenter__ = AsyncMock(return_value=conn)
|
||||
cm.__aexit__ = AsyncMock(return_value=None)
|
||||
pool.acquire = MagicMock(return_value=cm)
|
||||
return pool, conn
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_set_credential_status_revoked_uses_parameterized_query():
|
||||
"""TASK-12-03 (P1+ #8): 'revoked' uses a parameterized query (no f-string)."""
|
||||
pool, conn = _mock_pool_with_conn()
|
||||
store = PgStore(pool)
|
||||
await store.set_credential_status("cred-1", "revoked")
|
||||
# Exactly one execute call.
|
||||
assert conn.execute.await_count == 1
|
||||
sql, status_arg, cred_arg = conn.execute.await_args.args
|
||||
# No f-string interpolation — the SQL is a literal with $1, $2.
|
||||
assert "revoked_at = now()" in sql
|
||||
assert "$1" in sql and "$2" in sql
|
||||
assert status_arg == "revoked"
|
||||
assert cred_arg == "cred-1"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_set_credential_status_active_clears_revoked_at():
|
||||
"""TASK-12-03: 'active' clears revoked_at=NULL (re-activation)."""
|
||||
pool, conn = _mock_pool_with_conn()
|
||||
store = PgStore(pool)
|
||||
await store.set_credential_status("cred-1", "active")
|
||||
assert conn.execute.await_count == 1
|
||||
sql, status_arg, cred_arg = conn.execute.await_args.args
|
||||
assert "revoked_at = NULL" in sql
|
||||
assert status_arg == "active"
|
||||
assert cred_arg == "cred-1"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_set_credential_status_invalid_raises_value_error():
|
||||
"""TASK-12-03 (P1+ #4): invalid status → ValueError (enum validation)."""
|
||||
pool, conn = _mock_pool_with_conn()
|
||||
store = PgStore(pool)
|
||||
for bad_status in ("pending", "suspended", "deleted", "", "REVOKED", "active "):
|
||||
with pytest.raises(ValueError, match="Invalid credential status"):
|
||||
await store.set_credential_status("cred-1", bad_status)
|
||||
# No execute call should have been made (validation happens before the query).
|
||||
assert conn.execute.await_count == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_set_credential_status_no_fstring_in_sql():
|
||||
"""TASK-12-03 (P1+ #8): no f-string interpolation in the SQL (code smell fix).
|
||||
|
||||
The SQL must be a literal string (no f-string {extra} interpolation). The
|
||||
status + cred_id are bound parameters ($1, $2), not interpolated.
|
||||
"""
|
||||
pool, conn = _mock_pool_with_conn()
|
||||
store = PgStore(pool)
|
||||
await store.set_credential_status("cred-1", "revoked")
|
||||
sql = conn.execute.await_args.args[0]
|
||||
# The SQL must NOT contain an f-string-interpolated extra clause. The old
|
||||
# code had f"UPDATE ... SET status = $1{extra} WHERE id = $2" where extra
|
||||
# was ', revoked_at = now()' or ''. The new code has two explicit queries.
|
||||
# Verify the SQL is a literal (no {extra}-style interpolation artifacts).
|
||||
assert "{extra}" not in sql
|
||||
assert "UPDATE issued_credentials SET status = $1, revoked_at = now()" in sql
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_set_credential_status_revoked_then_active():
|
||||
"""TASK-12-03: revoke then re-activate (active clears revoked_at)."""
|
||||
pool, conn = _mock_pool_with_conn()
|
||||
store = PgStore(pool)
|
||||
# Revoke.
|
||||
await store.set_credential_status("cred-1", "revoked")
|
||||
revoke_sql = conn.execute.await_args.args[0]
|
||||
assert "revoked_at = now()" in revoke_sql
|
||||
# Re-activate (active clears revoked_at).
|
||||
conn.execute.reset_mock()
|
||||
await store.set_credential_status("cred-1", "active")
|
||||
active_sql = conn.execute.await_args.args[0]
|
||||
assert "revoked_at = NULL" in active_sql
|
||||
@@ -1,290 +0,0 @@
|
||||
"""NFR measurement tests (TASK-09-03, REQ-NFR-ASSIST-01, REQ-IDEATE-04, D-072).
|
||||
|
||||
Tests the measurement infrastructure (NOT the actual latency — that's a Phase-1
|
||||
live measurement, not a CI test):
|
||||
- AssistLatencyMetrics: p95/p50/p99 computed correctly from mock records.
|
||||
D-072: within_target = (p95 < 600), within_pilot = (p95 <= 650).
|
||||
- GuardrailMetrics: false_positive_rate on the tuning corpus, false_negative_rate
|
||||
on the direct-answer corpus, nightly_trend on mock turns.
|
||||
|
||||
D-072 binding: the pilot tolerance is ≤ 650ms. The target is < 600ms (C-8). The
|
||||
test ASSERTS that the measurement infrastructure works (percentiles + flags),
|
||||
not that the actual latency is under budget.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import datetime as _dt
|
||||
import json
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from server.assist.guardrail_metrics import GuardrailMetrics
|
||||
from server.assist.latency_metrics import (
|
||||
PILOT_TOLERANCE_MS,
|
||||
TARGET_MS,
|
||||
AssistLatencyMetrics,
|
||||
)
|
||||
from server.latency import LatencyRecord
|
||||
|
||||
|
||||
# ── AssistLatencyMetrics ────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _record(e2e_ms: float) -> LatencyRecord:
|
||||
"""Build a LatencyRecord with a specific e2e_asr_to_tts_ms value."""
|
||||
# e2e = tts_first_audio_ms - transcript_ready_ms. Use a non-zero base
|
||||
# because LatencyRecord.e2e_asr_to_tts_ms guards on truthiness (0.0 is falsy).
|
||||
base = 100.0
|
||||
return LatencyRecord(
|
||||
transcript_ready_ms=base,
|
||||
tts_first_audio_ms=base + e2e_ms,
|
||||
)
|
||||
|
||||
|
||||
def test_latency_empty_returns_none():
|
||||
m = AssistLatencyMetrics()
|
||||
assert m.p50() is None
|
||||
assert m.p95() is None
|
||||
assert m.p99() is None
|
||||
s = m.summary()
|
||||
assert s["count"] == 0
|
||||
assert s["p95"] is None
|
||||
assert s["within_target"] is False # no records → not within target
|
||||
assert s["within_pilot"] is False
|
||||
|
||||
|
||||
def test_latency_p95_p50_p99_computed():
|
||||
"""100 mock records: some <600ms, some 600-650ms, some >650ms.
|
||||
|
||||
Verifies p50/p95/p99 are computed correctly + the within_target/within_pilot
|
||||
flags reflect the p95 against the D-072 thresholds.
|
||||
"""
|
||||
m = AssistLatencyMetrics()
|
||||
# 80 records < 600ms (within target), 15 records 600-650ms (within pilot),
|
||||
# 5 records > 650ms (over pilot tolerance).
|
||||
for i in range(80):
|
||||
m.record(_record(500.0 + i)) # 500..579ms
|
||||
for i in range(15):
|
||||
m.record(_record(610.0 + i)) # 610..624ms
|
||||
for i in range(5):
|
||||
m.record(_record(700.0 + i)) # 700..704ms
|
||||
|
||||
s = m.summary()
|
||||
assert s["count"] == 100
|
||||
assert s["p50"] is not None
|
||||
assert s["p95"] is not None
|
||||
assert s["p99"] is not None
|
||||
# p50 should be in the < 600ms range (median of the 80 < 600ms records).
|
||||
assert s["p50"] < 600.0
|
||||
# p95: nearest-rank index = ceil(0.95 * 100) - 1 = 94 (0-indexed) → the 95th
|
||||
# sorted value. 80 records are 500..579, 15 are 610..624, 5 are 700..704.
|
||||
# Sorted: [500..579 (80), 610..624 (15), 700..704 (5)]. Index 94 → 610..624
|
||||
# range (index 80..94 = the 610..624 set; index 94 = 624.0).
|
||||
assert 610.0 <= s["p95"] <= 625.0
|
||||
# p99: index = ceil(0.99 * 100) - 1 = 98 → the 99th sorted value (700..704).
|
||||
assert s["p99"] >= 700.0
|
||||
# D-072: within_target = (p95 < 600). p95 is ~624 → not within target.
|
||||
assert s["within_target"] is False
|
||||
# D-072: within_pilot = (p95 <= 650). p95 is ~624 → within pilot.
|
||||
assert s["within_pilot"] is True
|
||||
# D-072 thresholds documented in the summary.
|
||||
assert s["target_ms"] == TARGET_MS == 600
|
||||
assert s["pilot_tolerance_ms"] == PILOT_TOLERANCE_MS == 650
|
||||
|
||||
|
||||
def test_latency_within_target_when_p95_under_600():
|
||||
"""All records < 600ms → within_target=True, within_pilot=True."""
|
||||
m = AssistLatencyMetrics()
|
||||
for i in range(20):
|
||||
m.record(_record(400.0 + i)) # 400..419ms
|
||||
s = m.summary()
|
||||
assert s["p95"] < 600.0
|
||||
assert s["within_target"] is True
|
||||
assert s["within_pilot"] is True
|
||||
|
||||
|
||||
def test_latency_over_pilot_when_p95_over_650():
|
||||
"""All records > 650ms → within_target=False, within_pilot=False."""
|
||||
m = AssistLatencyMetrics()
|
||||
for i in range(20):
|
||||
m.record(_record(700.0 + i)) # 700..719ms
|
||||
s = m.summary()
|
||||
assert s["p95"] > 650.0
|
||||
assert s["within_target"] is False
|
||||
assert s["within_pilot"] is False
|
||||
|
||||
|
||||
def test_latency_pilot_boundary_exactly_650():
|
||||
"""D-072 boundary: p95 == 650 → within_pilot=True (≤ is inclusive)."""
|
||||
m = AssistLatencyMetrics()
|
||||
# 20 records all exactly 650ms → p95 = 650.0
|
||||
for _ in range(20):
|
||||
m.record(_record(650.0))
|
||||
s = m.summary()
|
||||
assert s["p95"] == 650.0
|
||||
assert s["within_pilot"] is True # ≤ 650 (inclusive)
|
||||
assert s["within_target"] is False # < 600 (strict)
|
||||
|
||||
|
||||
def test_latency_d072_thresholds_documented():
|
||||
"""D-072: the pilot tolerance (≤650ms) + target (<600ms) are documented."""
|
||||
assert TARGET_MS == 600
|
||||
assert PILOT_TOLERANCE_MS == 650
|
||||
assert PILOT_TOLERANCE_MS > TARGET_MS # pilot tolerance is more lenient
|
||||
|
||||
|
||||
# ── GuardrailMetrics ────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_guardrail_fp_rate_on_coaching_corpus():
|
||||
"""FP rate on the tuning corpus < 5% (REQ-IDEATE-04 target)."""
|
||||
gm = GuardrailMetrics()
|
||||
rate, mis, total = await gm.false_positive_rate()
|
||||
print(f"\n[nfr] guardrail FP rate: {rate:.1%} ({mis}/{total})")
|
||||
assert rate < 0.05, (
|
||||
f"guardrail FP rate {rate:.1%} exceeds 5% target — the regex is "
|
||||
f"over-matching coaching responses. {mis}/{total} blocked."
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_guardrail_fn_rate_on_direct_corpus():
|
||||
"""FN rate on the direct-answer corpus < 5% (REQ-IDEATE-04 target)."""
|
||||
gm = GuardrailMetrics()
|
||||
rate, mis, total = await gm.false_negative_rate()
|
||||
print(f"\n[nfr] guardrail FN rate: {rate:.1%} ({mis}/{total})")
|
||||
assert rate < 0.05, (
|
||||
f"guardrail FN rate {rate:.1%} exceeds 5% target — the regex is "
|
||||
f"under-matching direct answers. {mis}/{total} allowed."
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_guardrail_adversarial_fn_measured():
|
||||
"""Adversarial FN rate measured + reported (G-067 — ≤ 20% pilot threshold).
|
||||
|
||||
This test does NOT assert the 5% target (the adversarial set is the
|
||||
residual-risk set, not the tuning target). It asserts the measurement
|
||||
infrastructure works + the rate is within the G-067 pilot threshold (≤ 20%).
|
||||
"""
|
||||
gm = GuardrailMetrics()
|
||||
rate, mis, total = await gm.adversarial_false_negative_rate()
|
||||
print(f"\n[nfr] guardrail adversarial FN rate: {rate:.1%} ({mis}/{total})")
|
||||
# G-067: ≤ 20% pilot threshold (the binding contract from GRILL-v0.5).
|
||||
assert rate <= 0.20, (
|
||||
f"adversarial FN rate {rate:.1%} exceeds G-067 ≤20% threshold — "
|
||||
f"re-tune the regex or escalate. {mis}/{total} slipped through."
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_guardrail_nightly_trend_on_mock_turns(tmp_path: Path):
|
||||
"""nightly_trend() samples 24h of assist turns + reports fn_candidates.
|
||||
|
||||
Seeds a temp SQLite store with assist turns (some coaching, some with
|
||||
direct-answer heuristic patterns) + verifies the nightly trend detects
|
||||
fn_candidates.
|
||||
"""
|
||||
from db.migrate import apply_migrations
|
||||
from db.store import PraxisStore
|
||||
|
||||
db = tmp_path / "test_nfr_nightly.db"
|
||||
apply_migrations(db)
|
||||
store = PraxisStore(db)
|
||||
await store.init()
|
||||
|
||||
# Seed an assist session + turns.
|
||||
session_id = await store.start_session_typed(
|
||||
"learner-1", "assist:refund", session_type="assist"
|
||||
)
|
||||
# Turn 1: a coaching response (allowed, no fn_candidate).
|
||||
await store.log_turn_with_verdict(
|
||||
session_id, 0, role="assistant",
|
||||
asr_text="customer wants refund",
|
||||
tts_text="What do you think the customer needs right now?",
|
||||
latency_ms=580.0,
|
||||
guardrail_verdict_json=json.dumps({"allowed": True, "category": "coaching"}),
|
||||
)
|
||||
# Turn 2: a direct-answer response that slipped past the guardrail
|
||||
# (allowed=True in the verdict, but the heuristic catches it).
|
||||
await store.log_turn_with_verdict(
|
||||
session_id, 1, role="assistant",
|
||||
asr_text="what should I say",
|
||||
tts_text="You should say: I'm sorry, here's a refund.",
|
||||
latency_ms=590.0,
|
||||
guardrail_verdict_json=json.dumps({"allowed": True, "category": "coaching"}),
|
||||
)
|
||||
# Turn 3: a blocked response (guardrail caught it).
|
||||
await store.log_turn_with_verdict(
|
||||
session_id, 2, role="assistant",
|
||||
asr_text="help me",
|
||||
tts_text="Tell the customer: we will issue a full refund now.",
|
||||
latency_ms=570.0,
|
||||
guardrail_verdict_json=json.dumps({"allowed": False, "category": "blocked_direct_script"}),
|
||||
)
|
||||
|
||||
gm = GuardrailMetrics()
|
||||
trend = await gm.nightly_trend(store)
|
||||
|
||||
assert trend["total_turns"] == 3
|
||||
assert trend["blocked"] == 1
|
||||
# Turn 2 should be flagged as an fn_candidate. The guardrail re-check may
|
||||
# catch it as a regression (it now blocks what it previously allowed) OR
|
||||
# the heuristic may catch it as a direct-answer pattern. Either way, it
|
||||
# must appear in fn_candidates.
|
||||
assert len(trend["fn_candidates"]) >= 1
|
||||
seqs = [c.get("turn_seq") for c in trend["fn_candidates"]]
|
||||
assert 1 in seqs, "turn 2 (direct-answer that slipped past) must be flagged"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_guardrail_nightly_trend_empty_store(tmp_path: Path):
|
||||
"""nightly_trend() on an empty store returns zeros + no fn_candidates."""
|
||||
from db.migrate import apply_migrations
|
||||
from db.store import PraxisStore
|
||||
|
||||
db = tmp_path / "test_nfr_nightly_empty.db"
|
||||
apply_migrations(db)
|
||||
store = PraxisStore(db)
|
||||
await store.init()
|
||||
|
||||
gm = GuardrailMetrics()
|
||||
trend = await gm.nightly_trend(store)
|
||||
assert trend["total_turns"] == 0
|
||||
assert trend["blocked"] == 0
|
||||
assert trend["fn_candidates"] == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_guardrail_nightly_trend_excludes_practice_turns(tmp_path: Path):
|
||||
"""nightly_trend() only samples assist turns (not practice turns)."""
|
||||
from db.migrate import apply_migrations
|
||||
from db.store import PraxisStore
|
||||
|
||||
db = tmp_path / "test_nfr_nightly_practice.db"
|
||||
apply_migrations(db)
|
||||
store = PraxisStore(db)
|
||||
await store.init()
|
||||
|
||||
# Seed a practice session (NOT assist) with a turn.
|
||||
practice_id = await store.start_session_typed(
|
||||
"learner-1", "cs_refund_ca_v01", session_type="practice"
|
||||
)
|
||||
await store.log_turn_with_verdict(
|
||||
practice_id, 0, role="assistant",
|
||||
asr_text="hello",
|
||||
tts_text="You should say sorry.",
|
||||
latency_ms=500.0,
|
||||
guardrail_verdict_json=json.dumps({"allowed": True, "category": "coaching"}),
|
||||
)
|
||||
|
||||
gm = GuardrailMetrics()
|
||||
trend = await gm.nightly_trend(store)
|
||||
# Practice turns must NOT appear in the assist nightly trend.
|
||||
assert trend["total_turns"] == 0
|
||||
assert trend["fn_candidates"] == []
|
||||
@@ -1,353 +0,0 @@
|
||||
"""P2 integration test — assist aggregation → endpoint → cost → NFR (TASK-12-05).
|
||||
|
||||
Requires Postgres (skips if PRAXIS_PG_DSN not set). End-to-end P2 integration:
|
||||
1. Seed 12 mock assist shifts (12 distinct learners — above k-anon threshold).
|
||||
2. Run the aggregation hook for each → cohort_aggregates populated with assist metrics.
|
||||
3. GET /api/operator/cohort (with auth cookie) → returns assist volume (non-suppressed).
|
||||
4. GET /api/operator/failure-patterns → returns assist_guardrail_block_rate.
|
||||
5. Seed 5 more assist shifts from 5 NEW distinct learners for a different path →
|
||||
GET /api/operator/cohort for that path → suppressed cells (5 < 10).
|
||||
6. Verify assist_p95_latency_ms is in the aggregates.
|
||||
7. Verify assist_cost_cents is in the session_outcome.
|
||||
8. Verify the C-3 budget check runs at shift-end.
|
||||
9. Verify the tech-debt fixes: aggregation cache survives restart (mock),
|
||||
cookie-secret warning, credential status enum, argon2id offloaded.
|
||||
|
||||
G-038 differencing-attack e2e: k-anon threshold enforced (12 not suppressed,
|
||||
5 suppressed). No per-learner data in any response.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import datetime as _dt
|
||||
import os
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not os.environ.get("PRAXIS_PG_DSN"),
|
||||
reason="PRAXIS_PG_DSN not set — P2 assist integration tests skipped.",
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def pg_pool():
|
||||
import asyncpg
|
||||
|
||||
pool = await asyncpg.create_pool(
|
||||
dsn=os.environ["PRAXIS_PG_DSN"], min_size=1, max_size=5, command_timeout=10,
|
||||
)
|
||||
try:
|
||||
yield pool
|
||||
finally:
|
||||
await pool.close()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def pg_store(pg_pool):
|
||||
from db.pg_migrate import apply_pg_migrations
|
||||
from db.pg_store import PgStore
|
||||
|
||||
await apply_pg_migrations(pg_pool)
|
||||
# Clean cohort_aggregates + operators for an isolated run.
|
||||
async with pg_pool.acquire() as conn:
|
||||
await conn.execute("TRUNCATE cohort_aggregates, operators, issued_credentials")
|
||||
return PgStore(pg_pool)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def authed_client(pg_store):
|
||||
"""A TestClient with auth + the operator routers wired to pg_store."""
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
from starlette.middleware.sessions import SessionMiddleware
|
||||
|
||||
from server.auth.dependencies import current_operator
|
||||
from server.auth.models import Operator
|
||||
from server.operator.cohort import router as cohort_router
|
||||
from server.operator.failure_patterns import router as failure_router
|
||||
from server.operator.mastery import router as mastery_router
|
||||
|
||||
app = FastAPI()
|
||||
app.state.pg_store = pg_store
|
||||
app.add_middleware(SessionMiddleware, secret_key="test-secret-1234567890abcdef1234567890")
|
||||
app.include_router(cohort_router)
|
||||
app.include_router(failure_router)
|
||||
app.include_router(mastery_router)
|
||||
|
||||
# Stub auth — every request is operator "integration-tester".
|
||||
async def _stub_op():
|
||||
return Operator(id="op-1", username="tester", display_name="T", role="operator")
|
||||
app.dependency_overrides[current_operator] = _stub_op
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def _assist_outcome(
|
||||
learner_ref: str,
|
||||
path: str = "customer_service",
|
||||
turn_count: int = 20,
|
||||
blocks: int = 2,
|
||||
p95_latency_ms: float = 580.0,
|
||||
cost_cents: int = 20,
|
||||
) -> dict:
|
||||
return {
|
||||
"learner_ref": learner_ref,
|
||||
"path": path,
|
||||
"scenario_id": "assist:refund",
|
||||
"outcome": "completed",
|
||||
"session_type": "assist",
|
||||
"rubric_scores": [],
|
||||
"failure_mode": None,
|
||||
"branch_path": [],
|
||||
"assist_turn_count": turn_count,
|
||||
"guardrail_blocks": blocks,
|
||||
"assist_p95_latency_ms": p95_latency_ms,
|
||||
"assist_p50_latency_ms": 500.0,
|
||||
"assist_p99_latency_ms": 620.0,
|
||||
"assist_within_pilot": True,
|
||||
"assist_cost_cents": cost_cents,
|
||||
"timestamp": _dt.datetime.now(_dt.timezone.utc).isoformat(),
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_p2_assist_aggregation_k_anon_threshold(pg_store, authed_client):
|
||||
"""1-4: 12 assist shifts (12 learners) → non-suppressed; 5 → suppressed."""
|
||||
from server.cohort.aggregator import aggregate_session
|
||||
|
||||
# 1. Seed 12 assist shifts for 'customer_service' (12 distinct learners).
|
||||
for i in range(12):
|
||||
await aggregate_session(pg_store, _assist_outcome(f"learner-{i}"))
|
||||
|
||||
# 2. Verify cohort_aggregates has assist metrics.
|
||||
async with pg_store.pool.acquire() as conn:
|
||||
rows = await conn.fetch(
|
||||
"SELECT metric, value, cell_count, cell_suppressed "
|
||||
"FROM cohort_aggregates WHERE path = 'customer_service' "
|
||||
"AND metric LIKE 'assist_%'"
|
||||
)
|
||||
metrics = {r["metric"]: r for r in rows}
|
||||
assert "assist_shifts_count" in metrics
|
||||
assert "assist_turns_count" in metrics
|
||||
assert "assist_active_learners_count" in metrics
|
||||
assert "assist_guardrail_block_rate" in metrics
|
||||
# 12 learners → not suppressed.
|
||||
assert metrics["assist_active_learners_count"]["cell_suppressed"] is False
|
||||
assert metrics["assist_active_learners_count"]["value"] == 12.0
|
||||
|
||||
# 3. GET /api/operator/cohort → returns assist volume (non-suppressed).
|
||||
r = authed_client.get("/api/operator/cohort")
|
||||
assert r.status_code == 200
|
||||
cohort_metrics = {
|
||||
c["metric"]: c for v in r.json()["views"] if v["path"] == "customer_service"
|
||||
for c in v["metrics"]
|
||||
}
|
||||
assert "assist_shifts_count" in cohort_metrics
|
||||
assert cohort_metrics["assist_shifts_count"]["cell_suppressed"] is False
|
||||
|
||||
# 4. GET /api/operator/failure-patterns → returns assist_guardrail_block_rate.
|
||||
r = authed_client.get("/api/operator/failure-patterns")
|
||||
assert r.status_code == 200
|
||||
fp_metrics = {
|
||||
c["metric"]: c for v in r.json()["views"] if v["path"] == "customer_service"
|
||||
for c in v["metrics"]
|
||||
}
|
||||
assert "assist_guardrail_block_rate" in fp_metrics
|
||||
|
||||
# 5. Seed 5 assist shifts for a DIFFERENT path (5 NEW learners) → suppressed.
|
||||
for i in range(5):
|
||||
await aggregate_session(pg_store, _assist_outcome(f"new-learner-{i}", path="retail_sales"))
|
||||
|
||||
r = authed_client.get("/api/operator/cohort")
|
||||
retail_metrics = {
|
||||
c["metric"]: c for v in r.json()["views"] if v["path"] == "retail_sales"
|
||||
for c in v["metrics"]
|
||||
}
|
||||
assert "assist_shifts_count" in retail_metrics
|
||||
# 5 < 10 → suppressed.
|
||||
assert retail_metrics["assist_shifts_count"]["cell_suppressed"] is True
|
||||
assert retail_metrics["assist_shifts_count"]["value"] is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_p2_assist_p95_latency_in_aggregates(pg_store):
|
||||
"""6: assist_p95_latency_ms is in the aggregates (D-072)."""
|
||||
from server.cohort.aggregator import aggregate_session
|
||||
|
||||
for i in range(12):
|
||||
await aggregate_session(pg_store, _assist_outcome(f"learner-{i}", p95_latency_ms=580.0))
|
||||
|
||||
async with pg_store.pool.acquire() as conn:
|
||||
row = await conn.fetchrow(
|
||||
"SELECT value, cell_suppressed FROM cohort_aggregates "
|
||||
"WHERE path = 'customer_service' AND metric = 'assist_p95_latency_ms'"
|
||||
)
|
||||
assert row is not None
|
||||
assert row["cell_suppressed"] is False
|
||||
assert row["value"] is not None
|
||||
# The running mean of per-shift p95 (580.0) → ~580.
|
||||
assert 570.0 <= float(row["value"]) <= 590.0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_p2_assist_cost_cents_in_session_outcome():
|
||||
"""7: assist_cost_cents is in the session_outcome (TASK-11-01)."""
|
||||
from server.assist.session import AssistSession
|
||||
from server.assist.context import AssistContext
|
||||
|
||||
ctx = AssistContext(
|
||||
system_prompt="", current_week=1, scenario_tag="refund",
|
||||
theta=0.0, coaching_focus="empathy", path_slug="customer_service",
|
||||
)
|
||||
session = AssistSession.__new__(AssistSession)
|
||||
session.assist_cost_cents = 0
|
||||
session.turn_count = 3
|
||||
session.guardrail_block_count = 0
|
||||
session.latency_metrics = MagicMock()
|
||||
session.latency_metrics.summary = MagicMock(return_value={
|
||||
"p50": 500.0, "p95": 580.0, "p99": 620.0, "count": 3,
|
||||
"target_ms": 600, "pilot_tolerance_ms": 650,
|
||||
"within_target": True, "within_pilot": True,
|
||||
})
|
||||
session.context = ctx
|
||||
session.learner_id = "learner-1"
|
||||
session.session_id = "test-session"
|
||||
|
||||
# Add 3 turns of cost.
|
||||
session.add_assist_turn_cost(5)
|
||||
session.add_assist_turn_cost(10)
|
||||
session.add_assist_turn_cost(3)
|
||||
|
||||
outcome = session._build_session_outcome("completed")
|
||||
assert outcome["assist_cost_cents"] == 18 # 5 + 10 + 3
|
||||
assert outcome["session_type"] == "assist"
|
||||
assert outcome["assist_p95_latency_ms"] == 580.0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_p2_c3_budget_check_runs():
|
||||
"""8: the C-3 budget check runs + reports within_budget (TASK-11-02)."""
|
||||
from server.assist.budget_check import C3_TARGET_USD, check_c3_budget
|
||||
|
||||
# 20 turns/shift × 20 shifts/month at 0.05 cents/turn → $0.20/month.
|
||||
result = check_c3_budget(
|
||||
assist_turns_per_shift=20,
|
||||
shifts_per_month=20,
|
||||
cost_per_turn_cents=0.05,
|
||||
)
|
||||
assert result["within_budget"] is True
|
||||
assert result["total_with_practice"] <= C3_TARGET_USD
|
||||
assert result["c3_target"] == 3.0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_p2_techdebt_aggregation_cache_survives_restart(pg_store, tmp_path):
|
||||
"""9a: aggregation cache survives a restart (TASK-12-01, P1+ #7)."""
|
||||
from server.cohort.aggregator import aggregate_session
|
||||
from server.cohort.learner_cache import (
|
||||
_clear_learner_cache,
|
||||
_count_distinct_learners,
|
||||
_load_learner_cache,
|
||||
)
|
||||
|
||||
# Point the cache to a temp file.
|
||||
pg_store.cohort_cache_db_path = str(tmp_path / "cache.db")
|
||||
|
||||
# Seed 10 learners.
|
||||
for i in range(10):
|
||||
await aggregate_session(pg_store, _assist_outcome(f"learner-{i}"))
|
||||
|
||||
# The persisted cache should have 10 distinct learners for this path.
|
||||
window_start = (_dt.datetime.now(_dt.timezone.utc).date() - _dt.timedelta(days=6))
|
||||
count = await _count_distinct_learners(pg_store, "customer_service", window_start)
|
||||
assert count == 10
|
||||
|
||||
# Simulate a restart: clear the in-memory cache + reload from SQLite.
|
||||
if hasattr(pg_store, "_agg_cache"):
|
||||
del pg_store._agg_cache
|
||||
loaded = await _load_learner_cache(pg_store)
|
||||
key = ("customer_service", "__learners__", window_start)
|
||||
assert key in loaded
|
||||
assert len(loaded[key]) == 10 # survived the "restart"
|
||||
|
||||
# Clear the cache (nightly reconciliation).
|
||||
await _clear_learner_cache(pg_store)
|
||||
count_after_clear = await _count_distinct_learners(pg_store, "customer_service", window_start)
|
||||
assert count_after_clear == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_p2_techdebt_cookie_secret_warning(monkeypatch):
|
||||
"""9b: cookie-secret <32 bytes logs a WARNING (TASK-12-02, P1+ #3)."""
|
||||
from loguru import logger as _logger
|
||||
from server.auth.cookies import get_session_middleware_kwargs
|
||||
|
||||
monkeypatch.setenv("PRAXIS_COOKIE_SECRET", "short") # 5 bytes < 32
|
||||
monkeypatch.setenv("PRAXIS_COOKIE_SECURE", "true")
|
||||
msgs: list[str] = []
|
||||
sink_id = _logger.add(lambda m: msgs.append(str(m)), level="WARNING")
|
||||
try:
|
||||
kw = get_session_middleware_kwargs()
|
||||
finally:
|
||||
_logger.remove(sink_id)
|
||||
assert kw["secret_key"] == "short" # accepted (backward compat)
|
||||
assert any("<32 bytes" in m for m in msgs)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_p2_techdebt_credential_status_enum():
|
||||
"""9c: set_credential_status enum validation (TASK-12-03, P1+ #4)."""
|
||||
from db.pg_store import PgStore
|
||||
|
||||
pool = MagicMock()
|
||||
conn = MagicMock()
|
||||
conn.execute = AsyncMock()
|
||||
cm = MagicMock()
|
||||
cm.__aenter__ = AsyncMock(return_value=conn)
|
||||
cm.__aexit__ = AsyncMock(return_value=None)
|
||||
pool.acquire = MagicMock(return_value=cm)
|
||||
|
||||
store = PgStore(pool)
|
||||
# Invalid status → ValueError.
|
||||
with pytest.raises(ValueError, match="Invalid credential status"):
|
||||
await store.set_credential_status("cred-1", "deleted")
|
||||
# Valid statuses work.
|
||||
await store.set_credential_status("cred-1", "revoked")
|
||||
await store.set_credential_status("cred-1", "active")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_p2_techdebt_argon2id_offloaded():
|
||||
"""9d: argon2id verify_password offloaded to asyncio.to_thread (P1+ #1)."""
|
||||
import asyncio as _asyncio
|
||||
import server.auth.routes as _routes_mod
|
||||
from server.auth.passwords import hash_password
|
||||
|
||||
# The login handler should use asyncio.to_thread for verify_password.
|
||||
# Verify the module imports asyncio + the handler references to_thread.
|
||||
assert hasattr(_routes_mod, "asyncio")
|
||||
assert _asyncio.to_thread is _routes_mod.asyncio.to_thread
|
||||
|
||||
# Functional check: verify_password is callable via to_thread.
|
||||
h = hash_password("pw")
|
||||
result = await _asyncio.to_thread(_routes_mod.verify_password, h, "pw")
|
||||
assert result is True
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_p2_no_per_learner_data_in_responses(pg_store, authed_client):
|
||||
"""No per-learner data in any dashboard response (D-031, G-038)."""
|
||||
from server.cohort.aggregator import aggregate_session
|
||||
|
||||
for i in range(12):
|
||||
await aggregate_session(pg_store, _assist_outcome(f"learner-sensitive-{i}"))
|
||||
|
||||
for endpoint in ("/api/operator/cohort", "/api/operator/failure-patterns", "/api/operator/mastery"):
|
||||
r = authed_client.get(endpoint)
|
||||
assert r.status_code == 200
|
||||
# No learner ref should appear in the response.
|
||||
text = r.text
|
||||
assert "learner-sensitive-" not in text, \
|
||||
f"per-learner data leaked in {endpoint} response"
|
||||
Reference in New Issue
Block a user