Compare commits
13 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| c28f5113c5 | |||
| 0a951029fd | |||
| 46b46479ca | |||
| e8a05adcd1 | |||
| c4c20a3722 | |||
| e39521d51d | |||
| 131545b70a | |||
| fb109337a5 | |||
| b0cb6280d7 | |||
| 6ada2560ba | |||
| 745dd88dfb | |||
| acbe8692ae | |||
| 6ab40c6f25 |
@@ -346,367 +346,4 @@ checks:
|
||||
auto_fixes:
|
||||
- REQUIREMENTS.md stale v0.2 duplicate header removed
|
||||
- REQUIREMENTS.md REQ-DASH-01 row updated to deferred-to-v0.4
|
||||
---/ci---
|
||||
|
||||
---
|
||||
|
||||
# Praxis — v0.4 Milestone Audit (Final Phase P3)
|
||||
|
||||
> **Phase:** 3 — Review + Ship (FINAL PHASE audit, v0.4 milestone)
|
||||
> **Milestone:** v0.4 (Operator tier — cohort dashboard, auth, Postgres)
|
||||
> **Branch:** `phase/03-final-review-ship` (current; == `milestone/v0.4-operator-tier` tip `889892c` — P2 ship commit, no P3 implementation commits yet — this audit IS the P3 work)
|
||||
> **Auditor:** CIAgent ci-doc-verifier (mechanical, autonomy `full`, single-project mode, slug `praxis`)
|
||||
> **Date:** 2026-08-04
|
||||
> **Mode:** P3 final milestone audit per run.md Step 5 — verifies the entire v0.4 milestone is healthy before the milestone merge to main
|
||||
> **Codebase state at audit:** HEAD = `889892c` (phase 2 ship); 6 commits `main..HEAD` (P0 merge + ship, P1 merge + ship, P2 merge + ship); working tree had 4 stale-status-field fixes applied by this audit (see §Auto-Fixes)
|
||||
> **Inputs:** git log (`main..HEAD` = 6 commits, `--all` = 92 commits), `.ciagent/` files (24), `---ci---` blocks (all v0.4 commits verified), REVIEW.md (multi-persona code review, APPROVE_WITH_NOTES), VERIFY-P1.md + VERIFY-P2.md, tag verification, branch/merge topology, GRILL-v0.4.md (6 MUST binding decisions), grill-MUST codebase verification
|
||||
|
||||
## v0.4 Milestone Audit — 2026-08-04 (Final Phase P3)
|
||||
|
||||
### Verdict: HEALTHY
|
||||
### Reconstruction test: PASS
|
||||
### .ciagent/ file discipline: PASS (after 4 stale-status fixes)
|
||||
### Branch hygiene: PASS
|
||||
### Commit discipline: PASS
|
||||
### Requirements coverage: 8/8
|
||||
### Grill MUSTs honored: 6/6
|
||||
### Critical issues: none (4 stale-status-field auto-fixes applied)
|
||||
### Recommendations: 4 (non-blocking, for ship orchestrator)
|
||||
|
||||
---
|
||||
|
||||
## A. Check 1 — Reconstruction Test
|
||||
|
||||
### A.1 Git log phase-by-phase vs ROADMAP.md
|
||||
|
||||
`git log main..HEAD --oneline` (6 commits, oldest → newest):
|
||||
|
||||
```
|
||||
6ab40c6 docs(milestone): merge phase/00 pre-execution → milestone/v0.4-operator-tier [P0]
|
||||
acbe869 docs(ship): phase 0 complete — v0.1.6 tagged, release created [P0 ship]
|
||||
00e39a3 feat(milestone): merge phase/01 operator-foundation → milestone/v0.4-operator-tier [P1]
|
||||
d3a6751 docs(ship): phase 1 complete — v0.1.7 tagged, release created [P1 ship]
|
||||
ec6fcc6 feat(milestone): merge phase/02 cohort-dashboard → milestone/v0.4-operator-tier [P2]
|
||||
889892c docs(ship): phase 2 complete — v0.1.8 tagged, release created [P2 ship]
|
||||
```
|
||||
|
||||
ROADMAP.md phase statuses (post-fix):
|
||||
- Phase 0 — Pre-Execution: **complete — tagged v0.1.6** ✅ matches `6ab40c6`/`acbe869`
|
||||
- Phase 1 — Operator Foundation: **complete — tagged v0.1.7** ✅ matches `00e39a3`/`d3a6751`
|
||||
- Phase 2 — Cohort Dashboard: **complete — tagged v0.1.8** ✅ matches `ec6fcc6`/`889892c`
|
||||
- Final Phase (P3) — Review + Ship: **planned** (this audit) ✅ current branch `phase/03-final-review-ship`
|
||||
|
||||
### A.2 `---ci---` blocks vs declared phase/stage/milestone
|
||||
|
||||
All 6 `main..HEAD` commits carry `---ci---` blocks (`git log main..HEAD --pretty=%B | grep -c "^---ci---"` = 6). Verified each block:
|
||||
|
||||
| Commit | phase | milestone | status | requirements.covered | Match |
|
||||
|--------|-------|-----------|--------|----------------------|-------|
|
||||
| `6ab40c6` (P0 merge) | 0 | v0.4 | complete | `[]` | ✅ |
|
||||
| `acbe869` (P0 ship) | 0 | v0.4 | complete | tag v0.1.6 | ✅ |
|
||||
| `00e39a3` (P1 merge) | 1 | v0.4 | complete | [REQ-MT-01, REQ-AUTH-01, REQ-NFR-AUTH-01, REQ-NFR-MT-01, REQ-MT-02] | ✅ 5 REQs |
|
||||
| `d3a6751` (P1 ship) | 1 | v0.4 | complete | tag v0.1.7 | ✅ |
|
||||
| `ec6fcc6` (P2 merge) | 2 | v0.4 | complete | [REQ-DASH-01, REQ-NFR-DASH-01, REQ-NFR-DASH-02, REQ-MT-02] | ✅ 4 REQs |
|
||||
| `889892c` (P2 ship) | 2 | v0.4 | complete | tag v0.1.8 | ✅ |
|
||||
|
||||
All blocks declare `project: praxis` (matches config.json `active_project`). ✅
|
||||
|
||||
### A.3 CHECKPOINT.json vs actual state
|
||||
|
||||
**Before fix:** `{phase: 2, stage: "complete", phase_role: "execution", tag: v0.1.8}` — reflected P2-complete state but did not account for P3 in progress.
|
||||
|
||||
**After fix:** `{phase: 3, stage: "in_progress", phase_role: "final_review", tag: v0.1.8, requirements.covered: [8 REQs]}` — now correctly reflects P3 (final review) in progress with all 8 v0.4 REQs covered by P0-P2. ✅ Matches the audit prompt's expected "P3 in progress" state.
|
||||
|
||||
### A.4 REQUIREMENTS.md REQ statuses vs commit claims
|
||||
|
||||
**Before fix:** all 8 v0.4 REQs marked `active` (stale — set during P0 SPECIFY, never advanced as P1/P2 shipped).
|
||||
|
||||
**After fix:** all 8 v0.4 REQs marked `complete` — consistent with:
|
||||
- P1 merge commit claims `covered: [REQ-MT-01, REQ-AUTH-01, REQ-NFR-AUTH-01, REQ-NFR-MT-01, REQ-MT-02]`
|
||||
- P2 merge commit claims `covered: [REQ-DASH-01, REQ-NFR-DASH-01, REQ-NFR-DASH-02, REQ-MT-02]`
|
||||
- CHECKPOINT.json `requirements.covered` = all 8
|
||||
- REVIEW.md REQ coverage table = 8/8 COVERED
|
||||
- VERIFY-P1.md = 5/5, VERIFY-P2.md = 4/4
|
||||
|
||||
✅ Consistent (post-fix). No `partial` status anywhere — all marked `complete`/`covered`.
|
||||
|
||||
### A.5 All 8 v0.4 REQ-IDs covered somewhere in the git log
|
||||
|
||||
`git log --all --pretty=%B | grep -E "REQ-(MT-01|MT-02|AUTH-01|DASH-01|NFR-AUTH-01|NFR-MT-01|NFR-DASH-01|NFR-DASH-02)"` returns all 8 unique IDs across P1+P2 merge commits:
|
||||
|
||||
| REQ-ID | Phase claimed | Verified |
|
||||
|--------|----------------|----------|
|
||||
| REQ-MT-01 | P1 | ✅ P1 merge `00e39a3` |
|
||||
| REQ-AUTH-01 | P1 | ✅ P1 merge `00e39a3` |
|
||||
| REQ-NFR-AUTH-01 | P1 | ✅ P1 merge `00e39a3` |
|
||||
| REQ-NFR-MT-01 | P1 | ✅ P1 merge `00e39a3` |
|
||||
| REQ-MT-02 | P1+P2 | ✅ P1 merge (schema) + P2 merge (pipeline) |
|
||||
| REQ-DASH-01 | P2 | ✅ P2 merge `ec6fcc6` |
|
||||
| REQ-NFR-DASH-01 | P2 | ✅ P2 merge `ec6fcc6` |
|
||||
| REQ-NFR-DASH-02 | P2 | ✅ P2 merge `ec6fcc6` |
|
||||
|
||||
All 8/8 covered. ✅
|
||||
|
||||
### A.6 Tags v0.1.6, v0.1.7, v0.1.8 exist and point to the right commits
|
||||
|
||||
`git tag -l v0.1.6 v0.1.7 v0.1.8` → all three exist (annotated). `git rev-list -n1 <tag>`:
|
||||
|
||||
| Tag | Commit | Phase | Correct? |
|
||||
|-----|--------|-------|----------|
|
||||
| v0.1.6 | `6ab40c6` | P0 merge (pre-execution) | ✅ |
|
||||
| v0.1.7 | `00e39a3` | P1 merge (operator foundation) | ✅ |
|
||||
| v0.1.8 | `ec6fcc6` | P2 merge (cohort dashboard) | ✅ |
|
||||
|
||||
Tag sequence v0.1.5 (main, v0.3) < v0.1.6 < v0.1.7 < v0.1.8 — strictly increasing, no skips. ✅
|
||||
Next tag v0.1.9 (= v0.4 milestone release) not yet created — correct, ship is delegated to the orchestrator. ✅
|
||||
|
||||
**Reconstruction test verdict: PASS.** The git log tells the same story as PROJECT.md, ROADMAP.md, REQUIREMENTS.md, and CHECKPOINT.json (after the 4 stale-status fixes).
|
||||
|
||||
---
|
||||
|
||||
## B. Check 2 — `.ciagent/` File Discipline
|
||||
|
||||
### B.1 All expected files exist
|
||||
|
||||
| File | Exists | Notes |
|
||||
|------|--------|-------|
|
||||
| PROJECT.md | ✅ | v0.4 scope (D-050..D-057), 8 REQs, status updated |
|
||||
| ROADMAP.md | ✅ | v0.4 phases 0-2 complete, P3 planned; status updated |
|
||||
| REQUIREMENTS.md | ✅ | 8 v0.4 REQs now `complete` (post-fix); v0.3 retained |
|
||||
| ARCHITECTURE.md | ✅ | operator Postgres + auth + dashboard + aggregation topology |
|
||||
| PERSONAS.md | ✅ | v0.4 roster (frontend + data-engineer reactivated) |
|
||||
| PLAN-v0.4-operator-tier.md | ✅ | 2 execution phases, 10 slices, 52 tasks |
|
||||
| RESEARCH-v0.4-operator-tier.md | ✅ | 7 domains, 20 risks, confidence 0.70-0.95 |
|
||||
| GRILL-v0.4.md | ✅ | 41 challenges, 6 MUST binding decisions |
|
||||
| VERIFY-P1.md | ✅ | P1 verification, APPROVE_WITH_NOTES, 5/5 REQ, 4/4 grill MUSTs |
|
||||
| VERIFY-P2.md | ✅ | P2 verification, APPROVE_WITH_NOTES, 4/4 REQ, 2/2 grill MUSTs |
|
||||
| REVIEW.md | ✅ | P3 multi-persona review, APPROVE_WITH_NOTES, 6/6 personas PASS |
|
||||
| config.json | ✅ | active_project=praxis, milestone=v0.4, autonomy=full |
|
||||
| CHECKPOINT.json | ✅ | updated to phase 3 / final_review / in_progress (post-fix) |
|
||||
|
||||
All 13 expected files present. ✅
|
||||
|
||||
### B.2 v0.3 files retained for reference (not deleted)
|
||||
|
||||
| File | Exists |
|
||||
|------|--------|
|
||||
| RESEARCH.md (v0.1) | ✅ |
|
||||
| RESEARCH-vc.md (v0.3) | ✅ |
|
||||
| RESEARCH-v0.3-anonymization-irt-scenarios.md | ✅ |
|
||||
| GRILL.md (v0.1) | ✅ |
|
||||
| GRILL-v0.3.md | ✅ |
|
||||
| PLAN.md (v0.3) | ✅ |
|
||||
| VERIFY.md (v0.3 P1) | ✅ |
|
||||
| AUDIT.md (v0.3 section preserved) | ✅ |
|
||||
|
||||
v0.3/v0.1 reference artifacts retained — no destructive deletion. ✅
|
||||
|
||||
### B.3 Internal consistency (no contradictions)
|
||||
|
||||
- PROJECT.md §v0.4 scope (8 REQs: REQ-MT-01/02, REQ-AUTH-01, REQ-DASH-01 + 4 NFRs) ↔ REQUIREMENTS.md v0.4 active section (8 REQs) ↔ CHECKPOINT.json `requirements.covered` (8) ↔ ROADMAP.md phase deliverables. **Consistent.** ✅
|
||||
- PROJECT.md out-of-scope list ↔ REQUIREMENTS.md out-of-scope list — identical items. ✅
|
||||
- ROADMAP.md v0.4 phases ↔ actual git branches (`phase/00..03`). ✅
|
||||
- No stale "v0.3 is active" references in v0.4 files (post-fix: PROJECT.md/ROADMAP.md/REQUIREMENTS.md status lines updated to P3 final review). ✅
|
||||
|
||||
### B.4 Stale references found and fixed
|
||||
|
||||
| File:Line | Before | After | Severity |
|
||||
|-----------|--------|-------|----------|
|
||||
| PROJECT.md:4 | `Status: phase 0 — specify (active milestone)` | `Status: phase 3 — final review (active milestone); P0-P2 complete (v0.1.6/v0.1.7/v0.1.8 tagged)` | important (stale) |
|
||||
| ROADMAP.md:4 | `Status: phase 0 — specify (active milestone)` | `Status: phase 3 — final review (active milestone); P0-P2 complete (v0.1.6/v0.1.7/v0.1.8 tagged)` | important (stale) |
|
||||
| REQUIREMENTS.md:4 | `Status: phase 0 — specify (active milestone)` | `Status: phase 3 — final review (active milestone); P0-P2 complete — 8/8 v0.4 REQ covered` | important (stale) |
|
||||
| REQUIREMENTS.md:14-36 | 8 v0.4 REQs `active` | 8 v0.4 REQs `complete` | important (stale) |
|
||||
| CHECKPOINT.json | `phase:2, stage:complete, phase_role:execution` | `phase:3, stage:in_progress, phase_role:final_review` | important (stale) |
|
||||
|
||||
All 5 stale-status fields were set during P0 SPECIFY and never advanced as P1/P2 shipped. Fixed by this audit (see §Auto-Fixes). These are audit-able inconsistencies (stale status fields) explicitly permitted by the audit charter — no scope changes, no REQ additions/removals, no milestone redefinitions.
|
||||
|
||||
**File discipline verdict: PASS (after 4 stale-status fixes).**
|
||||
|
||||
---
|
||||
|
||||
## C. Check 3 — Branch Hygiene
|
||||
|
||||
### C.1 Branch hierarchy
|
||||
|
||||
```
|
||||
main (d0f37e1 — v0.3 merged)
|
||||
└─ milestone/v0.4-operator-tier (889892c — P2 ship, == HEAD)
|
||||
├─ phase/00-pre-execution (3649344) → merged (6ab40c6)
|
||||
├─ phase/01-operator-foundation (c28f511) → merged (00e39a3)
|
||||
├─ phase/02-cohort-dashboard (f7cd162) → merged (ec6fcc6)
|
||||
└─ phase/03-final-review-ship (889892c) → CURRENT (not yet merged)
|
||||
```
|
||||
|
||||
- `main` → `milestone/v0.4-operator-tier` → `phase/NN-*`: hierarchy correct. ✅
|
||||
- `milestone/v0.4-operator-tier` exists, points to P2 ship commit `889892c` (latest P2 ship). ✅
|
||||
- `phase/03-final-review-ship` is the current branch (marked `*` in `git branch -vv`), not yet merged. ✅
|
||||
|
||||
### C.2 Phase merges to milestone (squash pattern)
|
||||
|
||||
| Phase branch | Merge commit | Type | Notes |
|
||||
|--------------|--------------|------|-------|
|
||||
| phase/00 | `6ab40c6` docs(milestone): merge phase/00 | squash-style | ✅ |
|
||||
| phase/01 | `00e39a3` feat(milestone): merge phase/01 | squash-style | ✅ |
|
||||
| phase/02 | `ec6fcc6` feat(milestone): merge phase/02 | squash-style | ✅ |
|
||||
|
||||
All 3 execution phases merged to `milestone/v0.4-operator-tier` with single merge commits (squash pattern — consistent with v0.2 milestone; improves on v0.3's fast-forward warning from the prior audit). ✅
|
||||
|
||||
### C.3 No stale/dangling branches for v0.4
|
||||
|
||||
`git branch -vv` shows no orphaned v0.4 phase branches. The phase branches (`phase/00..02`) are retained (not deleted) post-merge — consistent with the v0.1/v0.2/v0.3 retention pattern (branches kept for traceability). ✅
|
||||
|
||||
### C.4 Stale branches from prior milestones (informational, non-blocking)
|
||||
|
||||
- `phase/01-lxc-deploy` (v0.2), `phase/01-mastery-core` (v0.3), `phase/02-final-review-ship` (v0.3), `milestone/v0.1-praxis`, `milestone/v0.2-lxc-deploy`, `milestone/v0.3-mastery-scoring` — retained from prior milestones (consistent housekeeping pattern; not v0.4-stale).
|
||||
|
||||
**Branch hygiene verdict: PASS.**
|
||||
|
||||
---
|
||||
|
||||
## D. Check 4 — Commit Discipline
|
||||
|
||||
### D.1 Every phase has a ship commit with `---ci---` block
|
||||
|
||||
| Phase | Ship commit | `---ci---` | Tag |
|
||||
|-------|-------------|-----------|-----|
|
||||
| P0 | `acbe869` docs(ship): phase 0 complete | ✅ phase:0, milestone:v0.4, status:complete, tag:v0.1.6 | v0.1.6 |
|
||||
| P1 | `d3a6751` docs(ship): phase 1 complete | ✅ phase:1, milestone:v0.4, status:complete, tag:v0.1.7 | v0.1.7 |
|
||||
| P2 | `889892c` docs(ship): phase 2 complete | ✅ phase:2, milestone:v0.4, status:complete, tag:v0.1.8 | v0.1.8 |
|
||||
|
||||
✅
|
||||
|
||||
### D.2 Execution commits have `---ci---` blocks with required fields
|
||||
|
||||
The squash-merge commits (`6ab40c6`, `00e39a3`, `ec6fcc6`) carry full `---ci---` blocks with: `project`, `phase`, `milestone`, `status`, `requirements.covered`, `requirements.partial`. The ship commits carry `project`, `phase`, `milestone`, `status`, `tag`, `release`. All 6 `main..HEAD` commits have `---ci---` blocks (count = 6). ✅
|
||||
|
||||
### D.3 No commits missing `---ci---` blocks
|
||||
|
||||
`git log main..HEAD --pretty=%B | grep -c "^---ci---"` = 6 = number of commits `main..HEAD`. No missing blocks. ✅
|
||||
|
||||
### D.4 Tag sequence
|
||||
|
||||
v0.1.5 (main, v0.3) < v0.1.6 (P0) < v0.1.7 (P1) < v0.1.8 (P2) < v0.1.9 (next, not yet created = v0.4 milestone release). Strictly increasing, no skips. ✅
|
||||
|
||||
### D.5 Commit message prefixes
|
||||
|
||||
All 6 commits use conventional prefixes: `docs(ship)`, `docs(milestone)`, `feat(milestone)`. Consistent with the v0.2/v0.3 style. ✅
|
||||
|
||||
**Commit discipline verdict: PASS.**
|
||||
|
||||
---
|
||||
|
||||
## E. Check 5 — Requirements Coverage (8/8)
|
||||
|
||||
All 8 v0.4 REQ-IDs covered by at least one phase commit (P1 or P2). No `partial` coverage — all marked `covered`/`complete`.
|
||||
|
||||
| REQ-ID | Phase | Covered by commit | Status |
|
||||
|--------|-------|-------------------|--------|
|
||||
| REQ-MT-01 | P1 | `00e39a3` | covered → complete (post-fix) |
|
||||
| REQ-AUTH-01 | P1 | `00e39a3` | covered → complete (post-fix) |
|
||||
| REQ-NFR-AUTH-01 | P1 | `00e39a3` | covered → complete (post-fix) |
|
||||
| REQ-NFR-MT-01 | P1 | `00e39a3` | covered → complete (post-fix) |
|
||||
| REQ-MT-02 | P1+P2 | `00e39a3` (schema) + `ec6fcc6` (pipeline) | covered → complete (post-fix) |
|
||||
| REQ-DASH-01 | P2 | `ec6fcc6` | covered → complete (post-fix) |
|
||||
| REQ-NFR-DASH-01 | P2 | `ec6fcc6` | covered → complete (post-fix) |
|
||||
| REQ-NFR-DASH-02 | P2 | `ec6fcc6` | covered → complete (post-fix) |
|
||||
|
||||
**Coverage: 8/8.** ✅ REVIEW.md independently confirms 8/8 COVERED with per-REQ evidence (lines 227-234). VERIFY-P1.md confirms 5/5, VERIFY-P2.md confirms 4/4.
|
||||
|
||||
---
|
||||
|
||||
## F. Check 6 — Grill MUSTs Honored (6/6)
|
||||
|
||||
All 6 grill binding decisions (G-008, G-011, G-027, G-031, G-038, G-041) verified in the codebase. GRILL-v0.4.md exists with the full grill report (41 challenges, 6 MUST, proceed-with-conditions).
|
||||
|
||||
| MUST | Decision | Honored | Codebase evidence |
|
||||
|------|----------|---------|-------------------|
|
||||
| G-008 | Backup-restore drill task (pg_restore --clean --if-exists, verify 5 tables + counts) | YES | `tests/test_backup_restore.py` (seeds 5 tables, pg_dump, drop, pg_restore, verify counts); `scripts/backup-pg.sh` has restore-drill comments |
|
||||
| G-011 | Verification endpoint two-store fallback (Postgres → SQLite for v0.3 creds → SQLite-only if no PG) | YES | `server/vc/verification.py` `_lookup_credential` + `_lookup_public_key` implement (a)/(b)/(c); `__main__.py:209-211` docstring documents the binding contract; tests G-011b (`test_verification_fallback_sqlite_when_pg_missing_credential`) + G-011c (`test_verification_sqlite_only_when_no_pg`) |
|
||||
| G-027 | VC migration "no v0.3 active key" first-boot path (skip archive, generate fresh only) | YES | `server/vc/migrate_keys.py:80-87` if `v03_row is None` → `archived_key_id=None`, skips archive; `test_migration_g027_first_boot_no_v03_key` + e2e `test_g027_first_boot_no_v03_key` |
|
||||
| G-031 | R-AUTH-01 reframe (k-anon defense-in-depth = PRIMARY, cookie-secure flag = SECONDARY) | YES | `server/auth/cookies.py` docstring (lines 7-12) + WARNING text (lines 51-57) frame the ordering; `.env.example:86-88` + `.ciagent/.env.secrets.example:28` document it |
|
||||
| G-038 | Differencing-attack test (10 learners in window A, 9 in B → dropped learner not isolatable) | YES | `tests/test_cohort_aggregation.py:175 test_g038_differencing_attack_cannot_isolate_dropped_learner` (unit, runs without PG) + `tests/test_p2_aggregation_integration.py:210 test_g038_differencing_attack_api_layer` (e2e, skips without PG) |
|
||||
| G-041 | SPA fallback via custom StaticFiles subclass (NOT catch-all route) | YES | `server/__main__.py:279` `class SpaStaticFiles(StaticFiles)` with `get_response` 404→index.html; `test_assets_served_by_staticfiles_not_spa_fallback` confirms assets served by StaticFiles not fallback |
|
||||
|
||||
**Grill MUSTs honored: 6/6.** ✅ REVIEW.md lines 240-245 independently confirms 6/6 with evidence. VERIFY-P1.md confirms 4/4 P1-applicable (G-008, G-011, G-027, G-031); VERIFY-P2.md confirms 2/2 P2-applicable (G-038, G-041).
|
||||
|
||||
---
|
||||
|
||||
## G. Auto-Fixes Applied
|
||||
|
||||
This audit applied 4 stale-status-field fixes (audit-able inconsistencies explicitly permitted by the audit charter — no scope/REQ/milestone changes):
|
||||
|
||||
1. **PROJECT.md:4** — status line `phase 0 — specify` → `phase 3 — final review; P0-P2 complete (v0.1.6/v0.1.7/v0.1.8 tagged)`
|
||||
2. **ROADMAP.md:4** — status line `phase 0 — specify` → `phase 3 — final review; P0-P2 complete (v0.1.6/v0.1.7/v0.1.8 tagged)`
|
||||
3. **REQUIREMENTS.md:4 + lines 14-36** — status line `phase 0 — specify` → `phase 3 — final review; P0-P2 complete — 8/8 v0.4 REQ covered`; all 8 v0.4 REQ status fields `active` → `complete`
|
||||
4. **CHECKPOINT.json** — `phase:2, stage:complete, phase_role:execution` → `phase:3, stage:in_progress, phase_role:final_review` (tag remains v0.1.8, requirements.covered unchanged = 8 REQs)
|
||||
|
||||
**Rationale:** These status fields were set during P0 SPECIFY and never advanced as P1/P2 shipped. They are stale-status drift, not scope changes. Fixing them aligns the documentation with the actual git state (P0-P2 complete, P3 in progress) and with the REVIEW.md/VERIFY-P1.md/VERIFY-P2.md claims. This is the same class of fix the v0.3 P2 audit applied (REQUIREMENTS.md stale headers).
|
||||
|
||||
---
|
||||
|
||||
## H. Critical Issues Found
|
||||
|
||||
**None.** No reconstruction mismatch, no missing files, no broken branch hierarchy, no missing REQ coverage, no unaddressed grill MUSTs. The 4 auto-fixed items were stale-status drift, not logic/data/scope errors.
|
||||
|
||||
The v0.4 implementation is independently verified by:
|
||||
- **REVIEW.md** (P3 multi-persona code review): APPROVE_WITH_NOTES, 6/6 personas PASS, 0 P0 issues, 8 P1+ flagged (all non-blocking carry-forward)
|
||||
- **VERIFY-P1.md**: APPROVE_WITH_NOTES, 5/5 REQ, 4/4 grill MUSTs, 0 P0
|
||||
- **VERIFY-P2.md**: APPROVE_WITH_NOTES, 4/4 REQ, 2/2 grill MUSTs, 0 P0
|
||||
- **Tests**: 317 pytest pass / 36 skip / 0 fail; 17/17 vitest pass; npm build + typecheck clean
|
||||
|
||||
---
|
||||
|
||||
## I. Recommendations
|
||||
|
||||
Non-blocking, for the ship orchestrator (post-audit):
|
||||
|
||||
1. **Ship**: tag `v0.1.9` (= v0.4 milestone release), merge `milestone/v0.4-operator-tier` → `main`, create Gitea release. The audit found no blockers; the orchestrator delegates to ship after this audit.
|
||||
2. **On ship**: update CHECKPOINT.json to `phase:3, stage:complete, milestone_complete:true, milestone_merged_to_main:true, tag:v0.1.9` (the audit set it to `in_progress` — ship should advance it to `complete`).
|
||||
3. **Carry-forward the 8 P1+ items** (from REVIEW.md §P1+ Flagged) to the next milestone's backlog: (1) argon2id blocking event loop, (2) rate-limit 429 mock test, (3) cookie-secret length validation, (4) credential-status enum check, (5) revocation audit log, (6) nightly scheduler DST via zoneinfo, (7) aggregation cache persistence, (8) `set_credential_status` f-string SQL refactor. All non-blocking with mitigations present.
|
||||
4. **Branch cleanup (optional, post-merge-to-main)**: the prior-milestone phase branches (`phase/01-lxc-deploy`, `phase/01-mastery-core`, `phase/02-final-review-ship` from v0.3) are retained per housekeeping pattern; consider deleting after v0.4 merges to main if a cleanup pass is desired. Not blocking.
|
||||
|
||||
---
|
||||
|
||||
## J. Final Verdict
|
||||
|
||||
# ✅ HEALTHY
|
||||
|
||||
The v0.4 milestone (Operator Tier — Cohort Dashboard + Auth + Postgres) is **healthy and ready for milestone ship (v0.1.9 = v0.4)**:
|
||||
|
||||
- **Reconstruction (PASS):** git log (6 commits P0-P2) matches ROADMAP phase statuses, `---ci---` blocks match declared phase/milestone, tags v0.1.6/v0.1.7/v0.1.8 point to correct commits, all 8 REQs covered in commits.
|
||||
- **File discipline (PASS after fix):** all 13 expected `.ciagent/` files present; v0.3 reference files retained; internally consistent; 4 stale-status fields fixed (PROJECT/ROADMAP/REQUIREMENTS/CHECKPOINT).
|
||||
- **Branch hygiene (PASS):** main → milestone/v0.4 → phase/NN-* hierarchy correct; P0/P1/P2 squash-merged to milestone; P3 current (not yet merged); no stale v0.4 branches.
|
||||
- **Commit discipline (PASS):** all 6 commits have `---ci---` blocks; conventional prefixes; tag sequence strictly increasing.
|
||||
- **Requirements coverage (8/8):** all 8 v0.4 REQ-IDs covered (5 in P1, 4 in P2, MT-02 spans both); all `complete` (post-fix), no `partial`.
|
||||
- **Grill MUSTs honored (6/6):** G-008, G-011, G-027, G-031, G-038, G-041 all verified in the codebase with tests.
|
||||
|
||||
The orchestrator delegates to ship after this audit. Do NOT ship from this audit.
|
||||
|
||||
---
|
||||
|
||||
---ci---
|
||||
project: praxis
|
||||
phase: 3
|
||||
milestone: v0.4
|
||||
status: audit
|
||||
phase_role: final_review
|
||||
verdict: HEALTHY
|
||||
checks:
|
||||
reconstruction: PASS
|
||||
file_discipline: PASS-after-fix
|
||||
branch_hygiene: PASS
|
||||
commit_discipline: PASS
|
||||
requirements_coverage: 8/8
|
||||
grill_musts_honored: 6/6
|
||||
auto_fixes:
|
||||
- PROJECT.md stale status (phase 0 → phase 3 final review)
|
||||
- ROADMAP.md stale status (phase 0 → phase 3 final review)
|
||||
- REQUIREMENTS.md 8 v0.4 REQs active → complete + status line
|
||||
- CHECKPOINT.json phase 2 complete → phase 3 in_progress
|
||||
critical_issues: none
|
||||
recommendations:
|
||||
- ship: tag v0.1.9, merge milestone/v0.4 → main, create release
|
||||
- on ship: advance CHECKPOINT to phase 3 complete + milestone_complete true
|
||||
- carry-forward 8 P1+ items to next milestone backlog
|
||||
- optional branch cleanup post-merge
|
||||
---/ci---
|
||||
@@ -1,19 +1,19 @@
|
||||
{
|
||||
"phase": 3,
|
||||
"stage": "in_progress",
|
||||
"phase": 0,
|
||||
"stage": "complete",
|
||||
"milestone": "v0.4",
|
||||
"phase_role": "final_review",
|
||||
"phase_role": "pre_execution",
|
||||
"attempts": 0,
|
||||
"updated_at": "2026-08-04T12:00:00Z",
|
||||
"updated_at": "2026-08-04T02:30:00Z",
|
||||
"milestone_complete": false,
|
||||
"milestone_merged_to_main": false,
|
||||
"tag": "v0.1.8",
|
||||
"release_url": "https://git.cloudinit.dev/coreci/praxis/releases/tag/v0.1.8",
|
||||
"tag": "v0.1.6",
|
||||
"release_url": "https://git.cloudinit.dev/coreci/praxis/releases/tag/v0.1.6",
|
||||
"release_status": "created",
|
||||
"next_milestone": null,
|
||||
"requirements": {
|
||||
"covered": ["REQ-MT-01", "REQ-AUTH-01", "REQ-NFR-AUTH-01", "REQ-NFR-MT-01", "REQ-MT-02", "REQ-DASH-01", "REQ-NFR-DASH-01", "REQ-NFR-DASH-02"],
|
||||
"active": [],
|
||||
"covered": [],
|
||||
"active": ["REQ-MT-01", "REQ-MT-02", "REQ-AUTH-01", "REQ-DASH-01", "REQ-NFR-AUTH-01", "REQ-NFR-MT-01", "REQ-NFR-DASH-01", "REQ-NFR-DASH-02"],
|
||||
"deferred": []
|
||||
}
|
||||
}
|
||||
+1
-1
@@ -1,7 +1,7 @@
|
||||
# Praxis — Voice-first AI Apprenticeship Platform
|
||||
|
||||
**Milestone:** v0.4 (Operator tier — cohort dashboard, auth, Postgres)
|
||||
**Status:** phase 3 — final review (active milestone); P0-P2 complete (v0.1.6/v0.1.7/v0.1.8 tagged)
|
||||
**Status:** phase 0 — specify (active milestone)
|
||||
**Autonomy:** full
|
||||
**Previous milestone:** v0.3 (Mastery scoring + competency rubrics + verifiable credentials) — complete, tagged v0.1.5, release #380
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Praxis — Requirements
|
||||
|
||||
**Milestone:** v0.4 (Operator tier — cohort dashboard, auth, Postgres)
|
||||
**Status:** phase 3 — final review (active milestone); P0-P2 complete — 8/8 v0.4 REQ covered (v0.1.6/v0.1.7/v0.1.8 tagged); v0.3 complete — released as v0.1.5 (13/13 v0.3 REQ covered)
|
||||
**Status:** phase 0 — specify (active milestone); v0.3 complete — released as v0.1.5 (13/13 v0.3 REQ covered)
|
||||
|
||||
Formal requirements with REQ-IDs. Scoped to the active milestone unless noted. v0.1/v0.2/v0.3 requirements (complete) are retained for reference with their final status. Later-milestone requirements are marked `deferred`.
|
||||
|
||||
@@ -11,29 +11,29 @@ Formal requirements with REQ-IDs. Scoped to the active milestone unless noted. v
|
||||
|
||||
| REQ-ID | Requirement | Priority | Phase | Status |
|
||||
|--------|-------------|----------|-------|--------|
|
||||
| REQ-MT-01 | Operator-tier Postgres store — cohort aggregations, operator accounts, issued credentials, mastery-gate audit log. Separate from learner-local SQLite (D-007 preserved for learner surface). Migration path: SQLite stays for learner; Postgres added for operator. Postgres 16, persistent volume, internal Docker network only (D-040). | must | P1 | complete |
|
||||
| REQ-MT-02 | Cohort aggregation pipeline — on-session-end hook + nightly reconciliation job writes k-anonymized aggregates to Postgres from learner sessions (D-045). No raw learner PII in Postgres. | must | P1 | complete |
|
||||
| REQ-MT-01 | Operator-tier Postgres store — cohort aggregations, operator accounts, issued credentials, mastery-gate audit log. Separate from learner-local SQLite (D-007 preserved for learner surface). Migration path: SQLite stays for learner; Postgres added for operator. Postgres 16, persistent volume, internal Docker network only (D-040). | must | P1 | active |
|
||||
| REQ-MT-02 | Cohort aggregation pipeline — on-session-end hook + nightly reconciliation job writes k-anonymized aggregates to Postgres from learner sessions (D-045). No raw learner PII in Postgres. | must | P1 | active |
|
||||
|
||||
### Operator Auth (v0.4)
|
||||
|
||||
| REQ-ID | Requirement | Priority | Phase | Status |
|
||||
|--------|-------------|----------|-------|--------|
|
||||
| REQ-AUTH-01 | Operator-tier auth — session-based, single `operator` role in v0.4. Operator accounts in Postgres. Login endpoint + session cookie. Protects cohort dashboard + credential issuance. argon2id passwords, httpOnly+secure cookie, SameSite=Strict, 8h expiry, login rate-limited 5/min (D-041). | must | P1 | complete |
|
||||
| REQ-AUTH-01 | Operator-tier auth — session-based, single `operator` role in v0.4. Operator accounts in Postgres. Login endpoint + session cookie. Protects cohort dashboard + credential issuance. argon2id passwords, httpOnly+secure cookie, SameSite=Strict, 8h expiry, login rate-limited 5/min (D-041). | must | P1 | active |
|
||||
|
||||
### Cohort Dashboard (v0.4)
|
||||
|
||||
| REQ-ID | Requirement | Priority | Phase | Status |
|
||||
|--------|-------------|----------|-------|--------|
|
||||
| REQ-DASH-01 | Anonymized cohort view (practice, mastery progression, failure patterns) for training operators — k-anonymity ≥ 10, 7-day aggregation window (D-034). Operator UI (React) under `/operator/*`, served by same FastAPI server (`/api/operator/*` prefix), reuses v0.2 StaticFiles (D-044). No separate SPA build — same `client/dist`. | must | P2 | complete |
|
||||
| REQ-DASH-01 | Anonymized cohort view (practice, mastery progression, failure patterns) for training operators — k-anonymity ≥ 10, 7-day aggregation window (D-034). Operator UI (React) under `/operator/*`, served by same FastAPI server (`/api/operator/*` prefix), reuses v0.2 StaticFiles (D-044). No separate SPA build — same `client/dist`. | must | P2 | active |
|
||||
|
||||
## v0.4 Non-Functional Requirements
|
||||
|
||||
| REQ-ID | Requirement | Target | Phase | Status |
|
||||
|--------|-------------|--------|-------|--------|
|
||||
| REQ-NFR-AUTH-01 | Operator auth — passwords hashed (argon2id), session cookie httpOnly + secure + SameSite=Strict, login rate-limited (5/min), 8h expiry | must | P1 | complete |
|
||||
| REQ-NFR-MT-01 | Postgres-in-LXC — operator Postgres runs as a second Docker service in the existing LXC CT (D-040) without destabilizing the learner-facing praxis service. Internal Docker network only (not exposed to bridge). | must | P1 | complete |
|
||||
| REQ-NFR-DASH-01 | Cohort dashboard k-anonymity ≥ 10 — any cohort view cell with < 10 learners is suppressed | must | P2 | complete |
|
||||
| REQ-NFR-DASH-02 | Cohort dashboard freshness — aggregates ≤ 24h stale (nightly reconciliation + on-session-end hook per D-045) | must | P2 | complete |
|
||||
| REQ-NFR-AUTH-01 | Operator auth — passwords hashed (argon2id), session cookie httpOnly + secure + SameSite=Strict, login rate-limited (5/min), 8h expiry | must | P1 | active |
|
||||
| REQ-NFR-MT-01 | Postgres-in-LXC — operator Postgres runs as a second Docker service in the existing LXC CT (D-040) without destabilizing the learner-facing praxis service. Internal Docker network only (not exposed to bridge). | must | P1 | active |
|
||||
| REQ-NFR-DASH-01 | Cohort dashboard k-anonymity ≥ 10 — any cohort view cell with < 10 learners is suppressed | must | P2 | active |
|
||||
| REQ-NFR-DASH-02 | Cohort dashboard freshness — aggregates ≤ 24h stale (nightly reconciliation + on-session-end hook per D-045) | must | P2 | active |
|
||||
|
||||
## v0.4 Out of Scope (still deferred)
|
||||
|
||||
|
||||
+167
-188
@@ -1,255 +1,234 @@
|
||||
# Praxis — v0.4 Milestone Review (Final Phase P3)
|
||||
# Praxis v0.3 — Multi-Persona Code Review (P0 Pre-Execution + P1 Mastery Core)
|
||||
|
||||
> **Reviewer:** ci-code-reviewer (multi-persona: correctness, testing, security, performance, maintainability, adversarial)
|
||||
> **Scope:** full v0.4 milestone diff — `git diff main..HEAD` (74 files, +12,361/-819 LOC) — covers P1 (operator foundation) + P2 (cohort dashboard)
|
||||
> **Branch:** `phase/03-final-review-ship` (from `milestone/v0.4-operator-tier`)
|
||||
> **Reviewer:** ci-code-reviewer persona
|
||||
> **Scope:** all v0.3 changes (P0 pre-execution grill amendments + P1 mastery core + VC issuance, SLICE-01 → SLICE-09)
|
||||
> **Lenses:** Correctness, Testing, Security, Performance, Maintainability, Adversarial
|
||||
> **Date:** 2026-08-04
|
||||
> **Method:** code inspection (all v0.4 source + tests), test execution, security grep, grill MUST verification, adversarial analysis
|
||||
|
||||
## Summary
|
||||
- **Verdict: APPROVE_WITH_NOTES**
|
||||
- **Personas:** correctness **PASS**, testing **PASS**, security **PASS**, performance **PASS**, maintainability **PASS**, adversarial **PASS**
|
||||
- **P0 fixes applied:** 0 (none needed — no P0 issues found across all 6 personas)
|
||||
- **P1+ flagged:** 8 (4 from P1 VERIFY + 4 from P2 VERIFY — all non-blocking, all carry-forward)
|
||||
- **Total v0.4 REQ coverage:** 8/8 (REQ-MT-01, REQ-AUTH-01, REQ-NFR-AUTH-01, REQ-NFR-MT-01, REQ-MT-02, REQ-DASH-01, REQ-NFR-DASH-01, REQ-NFR-DASH-02)
|
||||
- **Grill MUSTs honored:** 6/6 (G-008 backup drill, G-011 two-store fallback, G-027 first-boot path, G-031 R-AUTH-01 reframe, G-038 differencing-attack test, G-041 SPA fallback subclass)
|
||||
|
||||
## Test Results
|
||||
|
||||
| Suite | Result | Notes |
|
||||
|-------|--------|-------|
|
||||
| `python3 -m pytest tests/` | **317 passed, 36 skipped, 0 failed** (90.28s) | Postgres-requiring tests skip gracefully (PRAXIS_PG_DSN unset); voice-service-key skips pre-existing |
|
||||
| `cd client && npx vitest run` | **17/17 passed** | Dashboard auth gate, login (200/401/429), sparkline (4 cases), suppressedLabel, formatFreshness, no-PII-in-DOM |
|
||||
| `cd client && npm run build` | **PASS** | 168 modules, 414ms, 662KB / 186KB gzip |
|
||||
| `cd client && npm run typecheck` | **PASS** | tsc -b --noEmit clean |
|
||||
| `python3 -c "import server.__main__"` | **PASS** | All v0.4 modules load, logs "SPA fallback enabled" |
|
||||
| `docker compose config` | **PASS** | Validates; postgres has no `ports:` (D-040 honored) |
|
||||
| Security grep (f-string SQL, hardcoded secrets, missing auth deps) | **PASS** | No injection vectors; no secrets in code; all /api/operator/* auth-gated |
|
||||
> **Authority:** PLAN.md + REQUIREMENTS.md + VERIFY.md (APPROVE_WITH_NOTES) + GRILL-v0.3.md (4 MUST) + PERSONAS.md (v0.3 roster)
|
||||
> **Test baseline:** 238 passed, 10 skipped (matches VERIFY.md L2.1)
|
||||
> **Final verdict:** **APPROVE_WITH_NOTES** — 0 P0 fixes applied; 5 P1 flags + 2 P2 notes for post-hoc review
|
||||
|
||||
---
|
||||
|
||||
## Persona 1 — Correctness
|
||||
## Review Methodology
|
||||
|
||||
### Findings (all PASS — no P0)
|
||||
|
||||
1. **k-anon threshold (exactly 10):** `K_ANON_THRESHOLD = 10` is a module constant in `server/cohort/aggregator.py:32`. Suppression logic `suppressed = active_count < K_ANON_THRESHOLD` (line 87). Boundary tests pass: 9 → suppressed (`test_9_learners_suppressed`), 10 → not suppressed (`test_10_learners_not_suppressed`), 11 → not suppressed (`test_11_learners_not_suppressed`). The threshold is NOT env-configurable (correct for a privacy control — adversarial persona confirms). ✅
|
||||
|
||||
2. **VC key migration (archive-before-active, G-027 first-boot):** `server/vc/migrate_keys.py` implements the R-VC-MIG-01 ordering correctly:
|
||||
- Step 2 (`_archive_v03_public_key`, line 86) runs BEFORE step 3 (`_generate_fresh_v04_key`, line 90).
|
||||
- G-027 first-boot path (line 80-87): if `v03_row is None` → `archived_key_id=None`, skips archive, generates fresh key only. Test: `test_migration_g027_first_boot_no_v03_key`.
|
||||
- Idempotent (line 74-76): if `get_active_signing_key_row()` returns non-None → returns `{None, None}` (no-op). Test: `test_migration_idempotent_when_active_key_exists`.
|
||||
- `init_issuer_key` uses `ON CONFLICT (id) DO NOTHING` → cannot replay to overwrite. ✅
|
||||
|
||||
3. **Auth flow (login/logout/me, cookie lifecycle, rate limit):**
|
||||
- Login (`routes.py:58`): rate-limited, `verify_password`, sets `request.session["operator_id"]`, updates `last_login_at`, rehashes if `needs_rehash`.
|
||||
- Logout (`routes.py:104`): `Depends(current_operator)`, clears session.
|
||||
- Me (`routes.py:112`): `Depends(current_operator)`, returns operator info.
|
||||
- Inactive operator (`dependencies.py:40`): 401 + `session.clear()` (invalidates cookie). ✅
|
||||
|
||||
4. **SPA fallback (SpaStaticFiles subclass, G-041):** `server/__main__.py:279-289` defines `class SpaStaticFiles(StaticFiles)` with `get_response` override that returns `FileResponse("index.html")` ONLY on 404 (non-file paths). This is the custom subclass mandated by G-041, NOT a `@app.get("/{path:path}")` catch-all (which would shadow asset serving). Test: `test_assets_served_by_staticfiles_not_spa_fallback` confirms `/assets/index.js` returns javascript content, not index.html. ✅
|
||||
|
||||
5. **Nightly scheduler timing (03:00 CT):** `seconds_until_next_03_ct` (nightly.py:32) computes seconds until 03:00 CT correctly. Tests: `test_seconds_until_next_03_ct_future_today` + `test_seconds_until_next_03_ct_past_today_wraps_tomorrow`. Fixed UTC-5 offset is a documented DST approximation (P1+-02 from VERIFY-P2). ✅
|
||||
|
||||
6. **Race conditions (aggregation hook fire-and-forget, pool access):**
|
||||
- Hook: `session_recorder.py:161` uses `asyncio.create_task(self._run_cohort_aggregation(session_outcome))` — fire-and-forget, off the voice path.
|
||||
- Hook failure: `hook.py:37` `except Exception: log.exception(...)` — no propagation; nightly reconciles.
|
||||
- Pool access: all PgStore methods use `async with self.pool.acquire() as conn` — no leaked connections. ✅
|
||||
|
||||
### Correctness verdict: PASS — no logic errors, off-by-ones, or missing edge cases found.
|
||||
Each focus file from the task brief was read in full and cross-referenced against its covering tests, the grill MUST conditions, and the VERIFY.md findings. The 4 grill MUST conditions were independently re-verified in code (not just trusting VERIFY.md). SQL was audited for parameterization. The IRT and scenario-selection code were checked for the claimed O(1) / O(n) complexity. The VC crypto path was checked for argument-order correctness in PyNaCl calls (`VerifyKey.verify(smessage, signature)` — confirmed correct at `issuer.py:156`).
|
||||
|
||||
---
|
||||
|
||||
## Persona 2 — Testing
|
||||
## Per-Persona Findings
|
||||
|
||||
### Findings (all PASS — no P0)
|
||||
### 1. Correctness (lead-developer + backend-engineer lens)
|
||||
|
||||
1. **Postgres-requiring tests skip gracefully:** 36 skips total — all `test_pg_store.py` (12), `test_p1_auth_integration.py`, `test_p1_vc_migration_e2e.py`, `test_backup_restore.py`, `test_p2_aggregation_integration.py` (3) skip with clear messages when `PRAXIS_PG_DSN` is unset. No hard CI dependency on Postgres. ✅
|
||||
#### `server/mastery/mastery_score.py` — gate logic
|
||||
|
||||
2. **G-038 differencing-attack test:** `tests/test_cohort_aggregation.py:175 test_g038_differencing_attack_cannot_isolate_dropped_learner` — seeds 10 learners in window A, 9 in window B (learner-9 dropped), asserts:
|
||||
- Window A has non-suppressed cells (10 ≥ threshold).
|
||||
- Window B has ALL cells suppressed (9 < threshold), NO non-suppressed cells.
|
||||
- Suppressed cells have `value=None` (differencing-attack defense — subtraction impossible).
|
||||
- No `learner-9` ref leaks in any aggregate cell arg.
|
||||
API e2e layer: `test_p2_aggregation_integration.py::test_g038_differencing_attack_api_layer` (skips without Postgres, logic verified at unit layer). ✅
|
||||
- **Gate logic (D-032):** `check_gate` at `mastery_score.py:78-86` implements `distinct_passed_count >= 3 AND path_score >= 3.5` — correct. Constants `_GATE_REQUIRED_DISTINCT = 3` and `_GATE_REQUIRED_SCORE = 3.5` are module-level (single source of truth).
|
||||
- **Conjunctive floor:** `compute_scenario_score` at `mastery_score.py:48-54` enforces every criterion ≥ 2 (or the criterion's `conjunctive_floor` if higher) AND mean ≥ 3.0. Professionalism floor (≥2) is honored via `rubric_schema.RubricCriterion.conjunctive_floor`.
|
||||
- **Determinism:** Pure function, no I/O, `round(total, 6)` for stable float comparison. Verified by `test_mastery_integration.py::test_mastery_flow_is_deterministic`.
|
||||
- **Verdict:** ✅ correct.
|
||||
|
||||
3. **R-VC-MIG-01 e2e test:** `tests/test_p1_vc_migration_e2e.py` (skips without Postgres) — seeds v0.3 VC, runs migration, verifies v0.3 VC against archived superseded key, issues v0.4 VC, verifies, tampers, confirms idempotency. Mock-based equivalent: `test_vc_migration.py::test_migration_archives_before_activating_r_vc_mig_01` (instrumented ordering test). ✅
|
||||
#### `server/mastery/irt.py` — theta update + cold-start
|
||||
|
||||
4. **Graceful degradation (server starts without Postgres):** `lifespan` in `__main__.py:78-90` — if `PRAXIS_PG_DSN` unset, logs WARNING, sets `pg_pool=None`, `pg_store=None`, yields. `/health` returns 200, auth routes return 503, learner voice loop (SQLite) unaffected. ✅
|
||||
- **P_success:** `1 / (1 + exp(-(θ−b)))` — standard 1PL/Rasch logistic. Correct.
|
||||
- **update_theta:** Kalman-like Gaussian-approximation update at `irt.py:38-55`:
|
||||
- `prior_precision = 1/σ²`, `info = P(1−P)` (Fisher information for Bernoulli), `new_precision = prior_precision + info`, `new_σ² = 1/new_precision`, `new_θ = θ + new_σ² × (outcome − P)`.
|
||||
- This is the standard 1PL Bayesian update. Correct. σ² shrinks monotonically as observations accumulate.
|
||||
- **Cold-start (R-IRT-01):** `select_scenario` at `irt.py:57-90` falls back to difficulty-based matching when `observations < 5`. Target difficulty = `round(θ + logit(target_p))` clamped to [1,5]. Sound.
|
||||
- **Verdict:** ✅ correct. O(1) per `update_theta` call (verified — single math computation, no loops).
|
||||
|
||||
5. **Voice UI at / unchanged (R-DASH-03, R-DASH-05):** `test_p2_spa_fallback.py::test_root_serves_voice_ui` (200, text/html, `<div id="root">`). `client/src/App.tsx` route `/` → `<VoiceSession />`, `*` → `<VoiceSession />`. All v0.1-v0.3 tests still pass (317 passed, 0 failed). ✅
|
||||
#### `server/vc/issuer.py` — JCS + Ed25519
|
||||
|
||||
6. **Mock-based equivalents exist for all Postgres-requiring paths:** `test_auth.py` (mocked PgStore, 310 LOC), `test_vc_migration.py` (mocked stores, 354 LOC), `test_create_operator.py` (mocked PgStore, 217 LOC), `test_cohort_aggregation.py` (mocked PgStore, 246 LOC). ✅
|
||||
- **JCS canonicalization:** `canonicaljson.encode_canonical_json` at `issuer.py:103-104` — RFC 8785-aligned, deterministic. Tested by `test_vc_issuer.py::test_jcs_canonicalization_determinism` + `test_jcs_key_ordering_is_sorted`.
|
||||
- **eddsa-jcs-2022 proof:** `_compute_hash_data` at `issuer.py:118-125` = `SHA256(canonical_proof) || SHA256(canonical_doc)`. Signed with `signing_key.sign(hash_data).signature` (detached signature). Correct per the cryptosuite spec.
|
||||
- **verify_proof:** at `issuer.py:141-159` reconstructs the same hash and calls `verify_key.verify(hash_data, sig)`. PyNaCl's `VerifyKey.verify(smessage, signature)` arg order is **correct** (verified against the library signature: `verify(self, smessage, signature=None)`). Raises `BadSignatureError` on mismatch → caught → returns False.
|
||||
- **Tamper detection:** re-canonicalizes the unsecured doc (without `proof`) + proof options (without `proofValue`) — any byte flip in the payload changes the canonical bytes → hash mismatch → verify fails. Tested by `test_vc_issuer.py::test_tamper_detection_flipped_byte_fails` + `test_vc_integration.py::test_tamper_payload_verify_fails`.
|
||||
- **Verdict:** ✅ correct. 19 VC tests pass.
|
||||
|
||||
7. **Rate limit 429 path:** Tested at decorator level in mock suite (`test_rate_limit_login_decorator`); full 6th-attempt→429 path is in PG-requiring `test_p1_auth_integration.py`. **P1+ carry-forward** (P1 VERIFY P1+-02): add a mock-based 429 test for CI coverage without Postgres. Non-blocking.
|
||||
#### `server/vc/status_list.py` — bitstring revocation
|
||||
|
||||
### Testing verdict: PASS — comprehensive coverage, graceful skips, G-038 + R-VC-MIG-01 explicitly tested.
|
||||
- **set/get_status:** bit-twiddling at `status_list.py:35-52` is correct (`byte_pos = idx >> 3`, `bit_pos = idx & 7`).
|
||||
- **get_status bounds check:** `status_list.py:50` returns False if `byte_pos >= len(buf)` — defensive, good.
|
||||
- **allocate_slot:** O(n) scan over the allocation bitstring at `status_list.py:54-72`. For `_MIN_BITS = 131072` (16KB), this is fine in practice (pilot scale). Expansion path (doubling) at `status_list.py:66-72` is correct.
|
||||
- **REQ-NFR-VC-02 (revocation latency):** status list fetched from SQLite on every verify call (`verification.py:47-48`) — no cache. Confirmed.
|
||||
- **Verdict:** ✅ correct.
|
||||
|
||||
---
|
||||
#### `server/session_recorder.py` — mastery flow wiring
|
||||
|
||||
## Persona 3 — Security
|
||||
- **Sequencing:** `run_mastery_flow` at `session_recorder.py:154-311` correctly sequences: extract → score → IRT update → progress upsert → gate event record → VC issuance.
|
||||
- **scoring_inconclusive path:** at `session_recorder.py:185-192` short-circuits all downstream steps and surfaces `retry_advised: True`. No score, no gate event, no progress change, no IRT update. Grill Axis 4 MUST #3 satisfied. Tested by `test_mastery_integration.py::test_mastery_flow_scoring_inconclusive_no_score_no_gate_event`.
|
||||
- **VC issuance:** `session_recorder.py:276-293` — `path_complete = gate_open and new_week >= 6`; on True, lazy-imports `server.vc.issuer.issue_credential`. `ImportError` swallowed (SLICE-09-independent ship); `Exception` logged (issuance failure doesn't crash mastery flow). Grill Axis 8 MUST satisfied.
|
||||
- **Outer guard:** `_run_mastery_flow_guarded` at `session_recorder.py:148-152` wraps the whole flow in try/except — mastery failure never crashes session end. Good isolation.
|
||||
- **P1 finding (P1-4, carried from VERIFY.md):** `compute_path_score` at `session_recorder.py:209-211` uses only the current session's score, not the cumulative mean over all passing sessions. The gate still works (distinct-count is the primary gate; the score threshold is secondary and the current-session score is a reasonable proxy). The in-code comment at `session_recorder.py:212-213` acknowledges this. Flag for v0.4: fold in prior passing scores from `mastery_progress.scenarios_passed_json`.
|
||||
- **Verdict:** ✅ correct (with P1-4 noted).
|
||||
|
||||
### Findings (all PASS — no P0)
|
||||
### 2. Testing (backend-engineer + lead-developer lens)
|
||||
|
||||
1. **Auth: argon2id params (OWASP):** `server/auth/passwords.py:14` `_ph = PasswordHasher()` — defaults (time_cost=3, memory_cost=64MiB=65536 KiB, parallelism=4) exceed all OWASP minimums (46MiB/t=1, 19MiB/t=2, 12MiB/t=3, etc.). `verify_password` catches `VerifyMismatchError` → False (no exception, uniform 401 path). `needs_rehash` delegates to `check_needs_rehash`. ✅
|
||||
#### Grill MUST conditions — independently re-verified in code
|
||||
|
||||
2. **Signed cookies (HMAC-SHA256, httpOnly+secure+SameSite):** `server/auth/cookies.py` returns SessionMiddleware kwargs: `https_only=secure` (Starlette's `https_only` param, not `secure` — verified correct via fix `0a95102`), `same_site="strict"`, `max_age=28800` (8h), `session_cookie="praxis_op"`, `path="/"`. itsdangerous HMAC-SHA256 under the hood. ✅
|
||||
| # | Grill MUST | Test evidence (verified in code) | Verdict |
|
||||
|---|-----------|----------------------------------|---------|
|
||||
| Axis 3 #1 | VC interop test exists | `tests/test_vc_interop.py` (153 LOC): JCS canonicalization is valid JSON, signature is 64-byte base64, W3C VC 2.0 schema conformance (@context, type, issuer, validFrom/validUntil, credentialSubject, credentialTier, proof fields). Staging-gated `test_full_w3c_vc_interop_validation` for extended self-check. | ✅ covered (P1-3: live external-verifier run is post-hoc) |
|
||||
| Axis 3 #2 | Key-rotation drill test exists | `tests/test_vc_key_rotation_drill.py::test_key_rotation_operational_drill` — issues N with key A, rotates to B, issues M with B, verifies all, revokes one each. Plus `test_vc_integration.py::test_key_rotation_old_vc_still_verifies`. | ✅ covered |
|
||||
| Axis 4 #1 | `credentialTier: "formative"` in payload | `test_vc_issuer.py::test_credential_tier_is_formative_in_payload` asserts both payload-level and credentialSubject-level. `test_vc_integration.py::test_issue_and_verify_valid` asserts response `credentialTier == "formative"`. | ✅ covered |
|
||||
| Axis 4 #3 | `scoring_inconclusive` fallback | `test_mastery_integration.py::test_mastery_flow_scoring_inconclusive_no_score_no_gate_event` — 3 bad-quote responses → inconclusive, no ability/progress/gate-event rows. `test_evidence_extractor_integration.py` covers the extractor-level inconclusive path. | ✅ covered |
|
||||
|
||||
3. **R-AUTH-01 / G-031 reframe:** `cookies.py` docstring (lines 7-12) + WARNING text (lines 51-57) correctly frame the **k-anon defense-in-depth as the PRIMARY mitigation** ("cohort dashboard reads only k-anonymized aggregates → sniffed cookie leaks no PII") and the config flag as **SECONDARY** ("operational convenience for when TLS arrives"). G-031 honored. ✅
|
||||
**4/4 grill MUST conditions tested.** Matches VERIFY.md L2.5.
|
||||
|
||||
4. **SQL injection (all PgStore queries parameterized):** Verified all PgStore methods use asyncpg `$1, $2, ...` parameterized bindings. Grep for `f"(SELECT|INSERT|UPDATE|DELETE|FROM)` found:
|
||||
- `db/pg_store.py:227` `f"UPDATE issued_credentials SET status = $1{extra} WHERE id = $2"` — `extra` is a hardcoded constant (`, revoked_at = now()` or empty) derived from `status == "revoked"` comparison, NOT user input. `status` and `cred_id` are bound parameters. **SAFE** (P1+-04 code smell, non-blocking).
|
||||
- `tests/test_backup_restore.py` f-strings interpolate hardcoded table names (not user input). SAFE. ✅
|
||||
#### Untested critical paths
|
||||
|
||||
5. **k-anon (write-time suppression, no per-learner drill-down, no PII):** Suppression applied in `aggregator.py:87` BEFORE `upsert_cohort_aggregate` (write-time, auditable). No per-learner drill-down: endpoints return only (path, metric, value, cell_count, cell_suppressed, updated_at). `test_no_per_learner_data_in_cohort_response` confirms no `learner_ref` string in cohort/mastery/failure responses. No raw PII in Postgres aggregates (D-031): only opaque `learner_ref` for distinct counting. ✅
|
||||
- **P1 gap (new finding): HTTP route wiring untested.** The `/vc/verify/{credential_id}` route at `server/__main__.py:124-136` is NOT tested via FastAPI TestClient / ASGI transport. The underlying `verify_credential()` function is well-tested (`test_vc_integration.py`, `test_vc_key_rotation_drill.py`), but the route registration, 404-on-not-found behavior, and the `_store.init()` call in the route handler are untested. A route-registration regression (e.g., route mounted after StaticFiles catch-all at `__main__.py:146`, shadowing the API route) would not be caught. Recommended: add one `httpx.AsyncClient` + ASGI transport test that hits `GET /vc/verify/<unknown>` → 404 and `GET /vc/verify/<valid>` → 200 with the formative tier.
|
||||
- **P2 gap: status list expansion path untested.** `BitstringStatusList.allocate_slot` at `status_list.py:66-72` doubles the bitstring when all slots are full. This expansion branch is not exercised by any test (pilot scale never fills 131072 slots). Low risk, but worth a unit test that forces expansion with a tiny `_MIN_BITS` override.
|
||||
- **P2 gap: `get_status` on uninitialized list.** If `get_status(idx)` is called before any `set_status` or `allocate_slot`, `_load` initializes an all-zero bitstring → returns False. This is correct behavior but untested explicitly.
|
||||
|
||||
6. **VC key migration (v0.3 private key NOT migrated, v0.4 encrypted at rest):** `migrate_keys.py:45` `init_issuer_key(v03_key_id, v03_public_key, b"")` — empty bytes for private_key_enc (only public key archived). Fresh v0.4 key encrypted via `_encrypt_private_key(signing_key, root_key)` (nacl.SecretBox, line 56). `issuer_keys.private_key_enc` is BYTEA in Postgres. ✅
|
||||
### 3. Security (security-engineer lens)
|
||||
|
||||
7. **Secret handling (.env.secrets gitignored, no secrets in code):** `.gitignore` has `.env.secrets`, `.env.*` ignored, `!.ciagent/.env.secrets.example` whitelisted. Grep for `os.environ["PRAXIS_PG_PASSWORD"]` / `os.environ["PRAXIS_COOKIE_SECRET"]` / `os.environ["PRAXIS_BOOTSTRAP` found only in test (`test_p2_spa_fallback.py:47` sets a test secret). No secrets committed. ✅
|
||||
#### `server/vc/verification.py` — public endpoint injection
|
||||
|
||||
8. **Cookie PII check:** The signed cookie (`praxis_op`) payload contains ONLY `{operator_id: "<uuid>"}`. No username, display_name, role, or learner data in the cookie. Verified by inspecting `routes.py:85` (sets `operator_id`) and `dependencies.py:33` (reads `operator_id`). ✅
|
||||
- **credential_id injection:** The `credential_id` path parameter at `__main__.py:125` flows to `store.get_credential(cred_id)` at `store.py:372-381`, which uses a parameterized query (`WHERE id = ?`). No SQL injection. FastAPI does not apply a regex constraint on the path param, but SQLite handles arbitrary strings safely (returns None for non-matching ids → 404).
|
||||
- **No PII leak:** `verification.py:53-73` returns only `{valid, status, issuer, credential{id,type,validFrom,validUntil}, mastery{skill,level,path,rubricScore,scenariosPassed,completedWeeks}, credentialTier, verifiedAt}`. `credentialSubject.id` is `urn:uuid:<learner_ref>` (opaque). No email/name/phone/address. Confirmed.
|
||||
- **Verdict:** ✅ secure (no injection vector).
|
||||
|
||||
### Security verdict: PASS — no injection vectors, no PII leaks, auth stack solid, secrets handled correctly.
|
||||
#### `server/mastery/evidence_extractor.py` — LLM prompt injection
|
||||
|
||||
---
|
||||
- **Vector:** transcript turns injected verbatim into the user message at `evidence_extractor.py:86`. A malicious learner could attempt prompt injection in spoken turns ("ignore previous instructions...").
|
||||
- **Mitigations (all verified in code):**
|
||||
1. System prompt is fixed and authoritative (`evidence_extractor.py:78-84`).
|
||||
2. Output is JSON-schema-validated (`_parse_evidence_json` at `evidence_extractor.py:96-119` rejects non-list, unknown `criterion_id`, schema-invalid items).
|
||||
3. **Fuzzy-match gate** at `evidence_extractor.py:180` — an injected "quote" that isn't in the transcript is rejected. This is the strongest mitigation: even if the LLM obeys an injection, the forged quote must actually appear in the learner's spoken turns to pass.
|
||||
- **Verdict:** ✅ secure. The fuzzy-match gate blocks the highest-impact injection (faking evidence to boost a score).
|
||||
|
||||
## Persona 4 — Performance
|
||||
#### `db/store.py` — SQL injection in new async methods
|
||||
|
||||
### Findings (all PASS — no P0)
|
||||
- **Audit:** all 14 v0.3 async methods (`get_ability`, `upsert_ability`, `get_progress`, `upsert_progress`, `record_gate_event`, `list_gate_events`, `init_issuer_key`, `get_active_signing_key_row`, `get_public_key_row`, `set_issuer_key_superseded`, `insert_credential`, `get_credential`, `set_credential_status`, `get_status_list`, `upsert_status_list`) use `?` placeholder parameterization. No f-string SQL, no string concatenation in queries. Grep for `f".*SELECT|f".*INSERT|f".*UPDATE|f".*WHERE` in `server/` and `db/` returned zero matches.
|
||||
- **Verdict:** ✅ no SQL injection.
|
||||
|
||||
1. **asyncpg pool (min 1, max 10):** `__main__.py:94-99` `create_pool(dsn, min_size=1, max_size=10, command_timeout=10)`. D-050 honored. Appropriate for single-instance pilot with low-frequency operator queries. `command_timeout=10` prevents slow queries from blocking. ✅
|
||||
### 4. Performance (backend-engineer lens)
|
||||
|
||||
2. **Aggregation hook non-blocking (asyncio.create_task):** `session_recorder.py:161` `asyncio.create_task(self._run_cohort_aggregation(session_outcome))` — fire-and-forget, off the voice path (C-8, D-054). Voice loop latency unaffected. ✅
|
||||
#### `server/mastery/irt.py` — O(1) verification
|
||||
|
||||
3. **Nightly job doesn't block the event loop:** `nightly.py:81-95` `_run_loop` uses `asyncio.sleep(secs)` (cooperative). Reconciliation (`_reconcile`) is a sequence of `await pg_store.upsert_cohort_aggregate(...)` calls (yields between each). Runs at 03:00 CT (low activity). ✅
|
||||
- **`update_theta`:** 1 division, 1 multiplication, 1 exp, 1 subtraction — O(1). Confirmed. REQ-NFR-IRT-01 (<100ms) trivially satisfied (sub-microsecond).
|
||||
- **`P_success`:** O(1).
|
||||
- **`select_scenario` cold-start:** O(n) over path scenarios (n ≈ 6 in v0.3). Fine.
|
||||
- **Verdict:** ✅ O(1) per update as required.
|
||||
|
||||
4. **SPA fallback doesn't add latency to API routes:** API routers (`auth_router`, `cohort_router`, `mastery_router`, `failure_router`, `credentials_router`) are mounted (`__main__.py:259-268`) BEFORE the SPA StaticFiles mount (`__main__.py:297`). FastAPI matches API routes first — no fallback overhead on API paths. ✅
|
||||
#### `server/scenarios/library.py` — `select_for_theta` O(n) verification
|
||||
|
||||
5. **argon2id hashing is sync (~100-300ms):** `verify_password` + `hash_password` (rehash) are sync calls in the async login handler (`routes.py:79, 88`). Blocks the event loop ~100-300ms per login. **Acceptable for single-operator pilot** (R-AUTH-02 — low frequency, single operator). **P1+ carry-forward** (P1 VERIFY P1+-01): offload to `asyncio.to_thread` if login frequency increases or multi-operator. Non-blocking. ✅
|
||||
- **`select_for_theta` at `library.py:143-167`:** single `for e in entries` loop with `abs(e.difficulty - target_b)` — O(n), NOT O(n²). No nested loops. `list_by_path` at `library.py:126-133` is also O(n) (one pass, though it calls `self.get(e.id)` per entry which is cached after first load).
|
||||
- **Minor note (P2):** `list_by_path` at `library.py:129-130` calls `self.get(e.id)` (which loads + caches the scenario YAML) for every entry just to read `s.path`. For n=6 this is negligible, but for a large library this could be optimized by storing `path` in the `IndexEntry` itself (the manifest already has it). Not a v0.3 concern.
|
||||
- **Verdict:** ✅ O(n), not O(n²).
|
||||
|
||||
6. **Voice loop (WebRTC → Pipecat) does NOT touch Postgres:** Uses SQLite (D-007 preserved). No perf impact on the <600ms latency budget (C-8). ✅
|
||||
### 5. Maintainability (lead-developer lens)
|
||||
|
||||
### Performance verdict: PASS — no blocking calls on the voice path, pool sizing appropriate, async patterns correct.
|
||||
#### `server/mastery/` module organization
|
||||
|
||||
---
|
||||
- Clean separation: `rubric_schema.py` (model), `rubric_loader.py` (I/O), `rubric_scorer.py` (deterministic scoring), `evidence_extractor.py` (LLM extraction), `mastery_score.py` (gate logic), `irt.py` (IRT engine). Each module is single-responsibility, <120 LOC, typed, with `__all__` exports.
|
||||
- **Verdict:** ✅ well-organized.
|
||||
|
||||
## Persona 5 — Maintainability
|
||||
#### `server/vc/` module organization
|
||||
|
||||
### Findings (all PASS — no P0)
|
||||
- Clean separation: `issuer.py` (payload + signing + issuance), `issuer_keys.py` (key management + encryption), `status_list.py` (revocation), `verification.py` (public verify + revoke). `CREDENTIAL_TIER = "formative"` is a module-level constant in `issuer.py:34` — single source of truth.
|
||||
- **Minor coupling smell (P2):** `issuer_keys._fetch_private_key_enc` at `issuer_keys.py:92-99` reaches into `store._connect()` (a private method) instead of using a public `store.get_private_key_enc(key_id)` method. This couples `issuer_keys` to `PraxisStore`'s internal connection management. Not a bug, but a small abstraction leak. Recommended: add a public `store.get_issuer_key_row(key_id)` method that returns the full row.
|
||||
- **Verdict:** ✅ well-organized (with P2 coupling note).
|
||||
|
||||
1. **IssuerKeyStore protocol clean:** `server/vc/issuer_keys.py:26-44` — `@runtime_checkable class IssuerKeyStore(Protocol)` with 4 methods. Both `PraxisStore` (SQLite, v0.3) and `PgStore` (Postgres, v0.4) implement it (duck-typed). `isinstance(store, IssuerKeyStore)` succeeds for both. Clean dependency inversion — `verification.py` depends on the protocol, not concrete stores. ✅
|
||||
### 6. Adversarial (security-engineer + red-team lens)
|
||||
|
||||
2. **SpaStaticFiles subclass clean:** `__main__.py:279-289` — 11-line override, `get_response` catches 404 → `FileResponse("index.html")`. Well-commented with G-041 rationale. ✅
|
||||
#### `/vc/verify` public endpoint — rate-limiting
|
||||
|
||||
3. **3 dashboard view components consistent:** `PracticeVolume.tsx`, `MasteryProgression.tsx`, `FailurePatterns.tsx` all share `_viewCommon.ts` (Cell type, suppressedLabel, formatFreshness) and follow the same fetch→render pattern. Server-side: `cohort.py`, `mastery.py`, `failure_patterns.py` all use `_common.py` (require_pg_store, all_recent_aggregates, group_by_path). ✅
|
||||
- **P1 (carried from VERIFY.md P1-1):** Endpoint is public + unauthenticated (D-043, by design — third-party verifiers must reach it). No rate limiting in v0.3. A flood of verify requests would each hit SQLite (`get_credential` + `get_public_key_row` + `get_status_list` = 3 queries per verify). Acceptable for pilot (single-deploy, low traffic). Flag for v0.4: add slowapi rate-limit (60 req/min/IP) on `/vc/verify/*`.
|
||||
|
||||
4. **Router mounting order (API before SPA fallback before StaticFiles):** `__main__.py:256-298` — auth_router → cohort_router → mastery_router → failure_router → credentials_router → SpaStaticFiles mount. Documented in comments. ✅
|
||||
#### Issuer key management — `PRAXIS_VC_ISSUER_KEY` fallback
|
||||
|
||||
5. **Naming, structure, coupling:** `server/auth/` package (passwords, cookies, rate_limit, dependencies, routes, models) — clear separation. `db/pg_store.py` — single class with clear method groups (operator CRUD, cohort, issuer keys, credentials, gate events). No god-class. `learner_ref` is opaque (not FK) per D-031. Consistent `get_*_row` / `set_*` / `insert_*` / `upsert_*` conventions. ✅
|
||||
- **P1 (carried from VERIFY.md P1-2):** `_load_root_key` at `issuer_keys.py:25-31` silently falls back to `nacl.utils.random(...)` if `PRAXIS_VC_ISSUER_KEY` is unset. On a deploy where the env var is missing:
|
||||
- First boot: `init_issuer_key` generates a key, encrypts with the random root key, stores ciphertext. Issuance works *within this process*.
|
||||
- Restart: new random root key → `get_active_signing_key` decrypts the old ciphertext with the new key → `nacl.secret.SecretBox.decrypt` raises `CryptoError` → issuance fails with a confusing error.
|
||||
- **Old VCs still verify** (public key is stored unencrypted) — no data loss, no security hole.
|
||||
- This is a **P1 operational footgun**, not a P0. The failure mode is "new issuance breaks after restart" not "credentials become invalid" or "keys leak." Recommended v0.4 fix: fail fast at startup if `PRAXIS_VC_ISSUER_KEY` is unset (raise `RuntimeError`), or persist the root key to a secrets manager on first init.
|
||||
|
||||
### Maintainability verdict: PASS — clean protocols, consistent structure, good separation of concerns.
|
||||
|
||||
---
|
||||
|
||||
## Persona 6 — Adversarial
|
||||
|
||||
### Findings (all PASS — no P0)
|
||||
|
||||
1. **What if an attacker calls /api/operator/cohort with a path that doesn't exist?** The endpoint takes NO path parameter — it returns all paths' aggregates from the last 30 days. A non-existent path simply returns no rows (no error, no leak). The attacker cannot probe for specific paths. ✅
|
||||
|
||||
2. **What if k-anon threshold is lowered via config?** `K_ANON_THRESHOLD = 10` is a **module constant** in `aggregator.py:32`, NOT configurable via env. Changing it requires a code change + redeploy. This is **correct for a privacy control** — it should not be runtime-configurable (an operator with env access should not be able to weaken k-anon). ✅
|
||||
|
||||
3. **What if the aggregation hook runs before Postgres is healthy?** The hook (`hook.py:27-32`) checks `pg_store is None` → no-op + WARNING. If Postgres is unhealthy mid-session, `upsert_cohort_aggregate` raises → caught by `hook.py:37` `except Exception: log.exception(...)` → nightly job reconciles. No crash path. ✅
|
||||
|
||||
4. **What if PRAXIS_COOKIE_SECRET is weak?** `cookies.py:41-48` checks `if not secret` (empty) → generates ephemeral random + WARNING. However, it does NOT validate `len(secret) >= 32` — a short non-empty secret (e.g., "x") would be accepted, weakening the HMAC signature. **P1+ carry-forward** (P1 VERIFY P1+-03): add `len(secret) >= 32` check with WARNING. Non-blocking — `.env.secrets.example` documents `openssl rand -base64 48` generation. ✅
|
||||
|
||||
5. **What if Postgres is exposed despite the internal Docker network?** `docker-compose.yml:59-82` — postgres service has NO `ports:` mapping (D-040 honored). An attacker would need to compromise the LXC CT or the `praxis-net` bridge. Mitigated by network isolation. ✅
|
||||
|
||||
6. **What if an attacker forges a cookie?** SessionMiddleware validates the itsdangerous HMAC-SHA256 signature on every request. A forged cookie without the correct `PRAXIS_COOKIE_SECRET` fails signature validation → `request.session` is empty → `current_operator` returns 401. ✅
|
||||
|
||||
7. **Migration replay attack?** `init_issuer_key` uses `ON CONFLICT (id) DO NOTHING` → re-running migration cannot overwrite an existing key. An attacker with DB access could insert a key directly, but DB access is already game-over. Not a v0.4 concern. ✅
|
||||
|
||||
### Adversarial verdict: PASS — no exploitable attack paths found. Privacy controls are non-configurable (correct). Weak cookie secret is a P1+ carry-forward.
|
||||
- **No other adversarial vectors found.** Issuance is server-side only (learner code never calls `issue_credential` directly — only `session_recorder.run_mastery_flow` after gate-open). Key rotation marks old keys `superseded`, not deleted — old VCs verify against archived public keys. Tested by `test_vc_key_rotation_drill.py`.
|
||||
|
||||
---
|
||||
|
||||
## P0 Fixes Applied
|
||||
|
||||
**None.** No P0 issues (broken tests, missing REQ coverage, security holes, logic errors causing incorrect behavior) were found across any of the 6 personas. The v0.4 implementation is correct, secure, complete, and well-tested. All 6 grill MUSTs are honored. All 8 REQs are covered. No auto-fixes were necessary.
|
||||
**None.** No P0 (critical bug / security hole) fixes were required. The codebase passes all 238 tests, all 4 grill MUST conditions are satisfied and tested, all SQL is parameterized, the VC crypto path is correct (PyNaCl arg order verified), the IRT and gate logic are mathematically sound, and the `scoring_inconclusive` fallback correctly avoids silent fail-to-zero.
|
||||
|
||||
The two issues flagged as P1 in VERIFY.md (rate-limiting, root-key fallback) were re-confirmed as **P1, not P0**:
|
||||
- Rate-limiting: acceptable for pilot scale, no security hole (public verify is read-only, no PII leak).
|
||||
- Root-key fallback: operational footgun, not a security hole (old VCs remain valid; only new issuance breaks after restart with missing env).
|
||||
|
||||
---
|
||||
|
||||
## P1+ Flagged for Post-Hoc Review
|
||||
## P1+ Flags (post-hoc review — non-blocking for v0.1.4 ship)
|
||||
|
||||
The following 8 non-blocking issues are flagged for the next milestone's backlog. All have mitigations present in the v0.4 code. None block ship.
|
||||
|
||||
### From P1 VERIFY (4 P1+):
|
||||
|
||||
1. **Argon2id blocking event loop** (`server/auth/routes.py:79,88`): `verify_password` + `hash_password` (rehash) are sync calls in the async login handler, blocking ~100-300ms. Acceptable for single-operator pilot (R-AUTH-02). If login frequency increases, offload to `asyncio.to_thread`. **Non-blocking.**
|
||||
|
||||
2. **Rate limit 429 not tested in mock path** (`tests/test_auth.py:303`): only the decorator factory is tested in the mock-based suite; the full 6th-attempt→429 path is in the PG-requiring integration test. Add a mock-based 429 test for CI coverage without Postgres. **Non-blocking.**
|
||||
|
||||
3. **No PRAXIS_COOKIE_SECRET length validation** (`server/auth/cookies.py:41`): only checks non-empty, not >=32 bytes. A short secret weakens the HMAC signature. Add `len(secret) >= 32` check with WARNING. **Non-blocking.**
|
||||
|
||||
4. **`set_credential_status` status field not validated** (`db/pg_store.py:223`): accepts any string for `status` (no enum check). Currently only called with "revoked" from operator code, but a future caller could pass arbitrary strings. Consider a CHECK constraint on the `issued_credentials.status` column or a Python enum. **Non-blocking.**
|
||||
|
||||
### From P2 VERIFY (4 P1+):
|
||||
|
||||
5. **Credential revocation lacks application-level audit log** (`server/operator/credentials.py`): the `revoke_credential` endpoint sets `status='revoked'` + `revoked_at=now()` but does NOT log the revocation event at the application level, and the revoking `operator_id` is not recorded. Mitigation: `revoked_at` timestamp + signed session cookie. Recommended: add `log.info("credential revoked: operator=%s cred_id=%s", op.id, cred_id)` + consider an `audit_log` table. **Non-blocking.**
|
||||
|
||||
6. **Nightly scheduler uses fixed UTC-5 offset (not true America/Winnipeg DST)** (`server/cohort/nightly.py:27`): CT approximated as fixed UTC-5. America/Winnipeg observes CST (UTC-6) in winter + CDT (UTC-5) in summer. Scheduler drifts ≤1h across DST boundaries — acceptable for a nightly reconciliation job. Documented in comments. Recommended: replace with `zoneinfo.ZoneInfo("America/Winnipeg")`. **Non-blocking.**
|
||||
|
||||
7. **Aggregation in-memory cache is per-PgStore-instance (lost on restart)** (`server/cohort/aggregator.py:162-170`): the `_agg_cache` on PgStore tracks running counters + distinct learner sets. On restart, the cache is lost — the next hook starts fresh, `active_learners_count` may reset to 1 (under-counting until nightly reconcile). Risk is low — nightly reconciliation recomputes from `mastery_gate_events` (source of truth), and under-counting → over-suppression (privacy-safe but value-destroying). **Non-blocking.**
|
||||
|
||||
8. **`set_credential_status` uses f-string interpolation in SQL (code smell)** (`db/pg_store.py:227`): the `extra` variable (`, revoked_at = now()` or empty) is interpolated via f-string. While `extra` is a hardcoded constant (not user input) and `status`/`cred_id` are parameterized, f-strings in SQL are a code smell. Recommended: refactor to two explicit queries. (Same as P1+ #4 — listed in both VERIFY reports.) **Non-blocking.**
|
||||
| ID | Flag | Severity | Location | Recommended action | Origin |
|
||||
|----|------|----------|----------|--------------------|--------|
|
||||
| **P1-1** | `/vc/verify` public + unauthenticated, no rate limiting → DoS vector (3 SQLite queries per verify) | P1 | `server/vc/verification.py`, `server/__main__.py:124` | v0.4: add slowapi rate-limit (60 req/min/IP) on `/vc/verify/*`. Acceptable for pilot. | VERIFY.md P1-1 (re-confirmed) |
|
||||
| **P1-2** | `_load_root_key()` silent random fallback when `PRAXIS_VC_ISSUER_KEY` unset → cross-restart issuance breaks silently (old VCs still verify) | P1 | `server/vc/issuer_keys.py:25-31` | v0.4: fail fast at startup if env unset (raise `RuntimeError`), or persist root key to secrets manager. | VERIFY.md P1-2 (re-confirmed) |
|
||||
| **P1-3** | VC interop test validates W3C schema + crypto format but does not invoke a live external W3C verifier (grill Axis 3 MUST #1 strictest bar) | P1 | `tests/test_vc_interop.py:128-153` | Before v0.3 milestone ship (v0.1.5): schedule staging run with `@digitalcredentials/vc` or `digitalbazaar/vc-verifier`. Schema + format validation is sufficient for v0.1.4 patch ship. | VERIFY.md P1-3 (re-confirmed) |
|
||||
| **P1-4** | `compute_path_score` uses only current session's score, not cumulative mean over all passing sessions | P1 | `server/session_recorder.py:209-211` | v0.4: fold in prior passing scores from `mastery_progress.scenarios_passed_json`. Gate still works (distinct-count is primary). | VERIFY.md P1-4 (re-confirmed) |
|
||||
| **P1-5 (new)** | HTTP route `/vc/verify/{credential_id}` wiring untested (no TestClient/ASGI test) — route registration, 404 behavior, `_store.init()` in handler not exercised | P1 | `server/__main__.py:124-136`, `tests/` | v0.4 (or before v0.1.5): add one `httpx.AsyncClient` + ASGI transport test: `GET /vc/verify/<unknown>` → 404, `GET /vc/verify/<valid>` → 200 with `credentialTier: formative`. Catches route-shadowing regressions (StaticFiles catch-all at `__main__.py:146` could shadow API routes if ordering changes). | New finding |
|
||||
| **P2-1** | No max-transcript-length guard in evidence extraction → long sessions could exceed model context window | P2 | `server/mastery/evidence_extractor.py:75-93` | Future: truncation or chunking for >30-min sessions. Not a v0.3 blocker. | VERIFY.md P2-1 (carried) |
|
||||
| **P2-2 (new)** | `BitstringStatusList.allocate_slot` expansion branch (doubling when full) untested; `issuer_keys._fetch_private_key_enc` reaches into `store._connect()` (private method) — abstraction leak | P2 | `server/vc/status_list.py:66-72`, `server/vc/issuer_keys.py:92-99` | Future: add a forced-expansion unit test with tiny `_MIN_BITS`; add a public `store.get_issuer_key_row(key_id)` method to remove the private-method coupling. | New finding |
|
||||
|
||||
---
|
||||
|
||||
## Carry-forward from P1/P2 VERIFY (P1+ items)
|
||||
## Final Verdict: **APPROVE_WITH_NOTES**
|
||||
|
||||
### P1 VERIFY P1+ (4):
|
||||
1. Argon2id blocking event loop (`server/auth/routes.py:79,88`) — offload to `asyncio.to_thread` if login frequency increases.
|
||||
2. Rate limit 429 not tested in mock path (`tests/test_auth.py:303`) — add mock-based 429 test.
|
||||
3. No PRAXIS_COOKIE_SECRET length validation (`server/auth/cookies.py:41`) — add `len(secret) >= 32` check.
|
||||
4. `set_credential_status` status field not validated (`db/pg_store.py:223`) — add CHECK constraint or Python enum.
|
||||
v0.3 (P0 + P1) is verified across all 6 persona lenses:
|
||||
|
||||
### P2 VERIFY P1+ (4):
|
||||
1. Credential revocation lacks application-level audit log (`server/operator/credentials.py`) — add `log.info` + consider `audit_log` table.
|
||||
2. Nightly scheduler fixed UTC-5 offset (`server/cohort/nightly.py:27`) — use `zoneinfo.ZoneInfo("America/Winnipeg")`.
|
||||
3. Aggregation in-memory cache lost on restart (`server/cohort/aggregator.py:162-170`) — document or persist distinct-learner set.
|
||||
4. `set_credential_status` f-string SQL code smell (`db/pg_store.py:227`) — refactor to two explicit queries. (Overlaps with P1+ #4.)
|
||||
- ✅ **Correctness:** gate logic (D-032 ≥3 distinct AND ≥3.5), IRT Kalman update, JCS+Ed25519 signing/verification, status list bit-twiddling, mastery flow wiring, `scoring_inconclusive` short-circuit — all correct. PyNaCl `VerifyKey.verify(smessage, signature)` arg order confirmed.
|
||||
- ✅ **Testing:** 238 passed / 10 skipped. 4/4 grill MUST conditions independently re-verified as tested. P1-5 flags the untested HTTP route wiring (function-level tests are sufficient for v0.1.4).
|
||||
- ✅ **Security:** no SQL injection (all 14 new async methods parameterized), no PII leak on `/vc/verify`, LLM prompt injection mitigated by fuzzy-match gate. P1-1 (rate-limit) and P1-2 (root-key fallback) re-confirmed as P1, not P0.
|
||||
- ✅ **Performance:** `irt.update_theta` is O(1); `library.select_for_theta` is O(n) (not O(n²)); `status_list.allocate_slot` is O(n) over 131072 bits (acceptable).
|
||||
- ✅ **Maintainability:** `server/mastery/` and `server/vc/` are cleanly separated, single-responsibility, typed, <120 LOC per module. Minor P2 coupling note on `issuer_keys._fetch_private_key_enc`.
|
||||
- ✅ **Adversarial:** issuance is server-side only (gated by mastery flow); key rotation archives (not deletes) old keys; public verify is read-only with no PII. P1-1/P1-2 are the only attack-surface flags, both acceptable for pilot.
|
||||
|
||||
**0 P0 fixes applied.** No critical bugs or security holes found. The 5 P1 flags + 2 P2 notes are non-blocking and tracked for v0.4 / the v0.1.5 milestone ship. The v0.1.4 patch ship is **unblocked**.
|
||||
|
||||
**Recommended next steps:**
|
||||
1. Proceed to P2 (final audit + milestone ship).
|
||||
2. Before v0.1.5: schedule the live external-verifier interop run (P1-3) + add the HTTP route test (P1-5).
|
||||
3. v0.4: address P1-1 (rate-limit), P1-2 (root-key fail-fast), P1-4 (path-score cumulative mean).
|
||||
|
||||
---
|
||||
|
||||
## REQ Coverage (8/8)
|
||||
|
||||
| REQ-ID | Phase | Covered by | Status |
|
||||
|--------|-------|-----------|--------|
|
||||
| REQ-MT-01 | P1 | docker-compose postgres + asyncpg pool + PgStore + IssuerKeyStore protocol + verification swap | ✅ COVERED |
|
||||
| REQ-AUTH-01 | P1 | argon2id + signed cookies + rate limit + current_operator dep + bootstrap CLI | ✅ COVERED |
|
||||
| REQ-NFR-AUTH-01 | P1 | argon2id (PasswordHasher defaults), httpOnly+secure+SameSite=Strict, 5/min rate limit, 8h expiry | ✅ COVERED |
|
||||
| REQ-NFR-MT-01 | P1 | postgres internal network only (no ports), 6GB CT, graceful degradation, voice loop unaffected | ✅ COVERED |
|
||||
| REQ-MT-02 | P1+P2 | schema (P1 SLICE-01) + pipeline (P2 SLICE-07 aggregator + hook + nightly) | ✅ COVERED |
|
||||
| REQ-DASH-01 | P2 | 4 endpoints + React UI + SPA fallback | ✅ COVERED |
|
||||
| REQ-NFR-DASH-01 | P2 | write-time suppression + query value=null + display "— (<10 learners)" + G-038 | ✅ COVERED |
|
||||
| REQ-NFR-DASH-02 | P2 | nightly job + on-session-end hook + last_updated freshness | ✅ COVERED |
|
||||
|
||||
## Grill MUSTs Honored (6/6)
|
||||
|
||||
| MUST | Honored | Evidence |
|
||||
|------|---------|----------|
|
||||
| G-008 (backup drill) | YES | `tests/test_backup_restore.py` seeds 5 tables, pg_dump, drop, pg_restore --clean --if-exists, verify counts. `scripts/backup-pg.sh` has restore drill comments. |
|
||||
| G-011 (two-store fallback) | YES | `server/vc/verification.py` `_lookup_credential` + `_lookup_public_key` implement (a)/(b)/(c). Tests: G-011b + G-011c. |
|
||||
| G-027 (first-boot no v0.3 key) | YES | `migrate_keys.py:80-87` if v03_row is None → archived_key_id=None, skip archive. Tests: `test_migration_g027_first_boot_no_v03_key` + e2e. |
|
||||
| G-031 (R-AUTH-01 reframe) | YES | `cookies.py` docstring + WARNING: "primary R-AUTH-01 mitigation is k-anon defense-in-depth... this flag is the secondary mitigation." |
|
||||
| G-038 (differencing-attack test) | YES | `test_g038_differencing_attack_cannot_isolate_dropped_learner` — 10 in A, 9 in B → B fully suppressed, dropped learner not isolatable. |
|
||||
| G-041 (SPA fallback subclass) | YES | `__main__.py:279-289` `class SpaStaticFiles(StaticFiles)` with `get_response` 404→index.html. NOT a catch-all route. `test_assets_served_by_staticfiles_not_spa_fallback`. |
|
||||
|
||||
```yaml
|
||||
---ci---
|
||||
phase: 2
|
||||
milestone: v0.3
|
||||
status: review
|
||||
requirements_covered:
|
||||
- REQ-MAST-01
|
||||
- REQ-MAST-02
|
||||
- REQ-MAST-03
|
||||
- REQ-MAST-04
|
||||
- REQ-SCEN-02
|
||||
- REQ-SCEN-03
|
||||
- REQ-SCEN-04
|
||||
- REQ-PATH-02
|
||||
- REQ-NFR-MAST-01
|
||||
- REQ-NFR-MAST-02
|
||||
- REQ-NFR-VC-01
|
||||
- REQ-NFR-VC-02
|
||||
- REQ-NFR-IRT-01
|
||||
requirements_total: 13
|
||||
requirements_covered_count: 13
|
||||
requirements_pending_count: 0
|
||||
grill_must_satisfied: 4
|
||||
grill_must_total: 4
|
||||
grill_must_tested: 4
|
||||
p0_fixes_applied: 0
|
||||
p1_flags: 5
|
||||
p2_notes: 2
|
||||
verdict: APPROVE_WITH_NOTES
|
||||
personas_run:
|
||||
- correctness
|
||||
- testing
|
||||
- security
|
||||
- performance
|
||||
- maintainability
|
||||
- adversarial
|
||||
tests_passed: 238
|
||||
tests_skipped: 10
|
||||
---
|
||||
|
||||
## Bottom Line
|
||||
|
||||
The v0.4 milestone (Operator Tier — Cohort Dashboard + Auth + Postgres) is **APPROVE_WITH_NOTES**. All 6 personas pass. All 8 REQs are covered. All 6 grill MUSTs are honored. Zero P0 issues. Eight P1+ items flagged for post-hoc review (all non-blocking, all with mitigations present, all carry-forward to the next milestone's backlog).
|
||||
|
||||
The implementation is correct (k-anon threshold exactly 10, archive-before-active, G-027 first-boot), secure (argon2id exceeding OWASP, parameterized SQL, k-anon defense-in-depth, no PII in Postgres), performant (async fire-and-forget hook, pool sizing appropriate, voice loop untouched), maintainable (clean protocols, consistent structure, good separation), and adversarially sound (non-configurable privacy controls, no exploitable attack paths).
|
||||
|
||||
The milestone is ready for ship (v0.1.9 = v0.4). The orchestrator delegates to ship after this review.
|
||||
```
|
||||
+5
-5
@@ -1,7 +1,7 @@
|
||||
# Praxis — Roadmap
|
||||
|
||||
**Milestone:** v0.4 (Operator tier — cohort dashboard, auth, Postgres) — active
|
||||
**Status:** phase 3 — final review (active milestone); P0-P2 complete (v0.1.6/v0.1.7/v0.1.8 tagged)
|
||||
**Status:** phase 0 — specify (active milestone)
|
||||
**Previous milestone:** v0.3 (Mastery scoring + competency rubrics + verifiable credentials) — complete, tagged v0.1.5, release #380, merged to main
|
||||
|
||||
## Milestone Philosophy
|
||||
@@ -28,19 +28,19 @@ Pipeline stages: SPECIFY → CLARIFY → RESEARCH → PLAN → GRILL → SHIP
|
||||
- GRILL-v0.4.md (adversarial review — auth + PII surface warrants grill)
|
||||
- Phase 1 + Phase 2 plans (vertical slices with wave ordering)
|
||||
|
||||
### Phase 1 — Operator Foundation (Postgres + Auth) (complete — tagged v0.1.7, release created)
|
||||
### Phase 1 — Operator Foundation (Postgres + Auth) (planned)
|
||||
|
||||
**Branch:** `phase/01-operator-foundation` → merged to `milestone/v0.4-operator-tier`
|
||||
**Ship target:** `v0.1.7` (patch release, feature milestone type)
|
||||
**Status:** complete (v0.1.7 tagged, Gitea release created; 272 pass, 33 skip, 0 fail; 5/5 REQ covered; APPROVE_WITH_NOTES, 4 P1+ flagged)
|
||||
**Status:** planned
|
||||
|
||||
**Goal:** Operator-tier Postgres 16 running as a second Docker service in the existing LXC CT (internal network only), operator auth (argon2id session cookies, single `operator` role, login rate-limited), VC issuer key store migrated to Postgres + secrets. Foundation for the cohort dashboard in P2. No UI yet — API + DB + auth only.
|
||||
|
||||
### Phase 2 — Cohort Dashboard + Aggregation (complete — tagged v0.1.8, release created)
|
||||
### Phase 2 — Cohort Dashboard + Aggregation (planned)
|
||||
|
||||
**Branch:** `phase/02-cohort-dashboard` → merged to `milestone/v0.4-operator-tier`
|
||||
**Ship target:** `v0.1.8` (patch release, feature milestone type)
|
||||
**Status:** complete (v0.1.8 tagged, Gitea release created; 317 pass, 36 skip, 0 fail; 4/4 REQ covered; APPROVE_WITH_NOTES, 4 P1+ flagged)
|
||||
**Status:** planned
|
||||
|
||||
**Goal:** Cohort aggregation pipeline (on-session-end hook + nightly reconciliation, k-anonymity ≥ 10, 7-day windows) + React cohort dashboard under `/operator/*` (served by same FastAPI, reuses v0.2 StaticFiles) + `/api/operator/*` endpoints (auth-gated). Dashboard shows anonymized practice/mastery/failure-pattern views with cells < 10 learners suppressed.
|
||||
|
||||
|
||||
@@ -1,405 +0,0 @@
|
||||
# Praxis — v0.4 Phase 2 Verification (Cohort Dashboard + Aggregation)
|
||||
|
||||
## Summary
|
||||
- Verdict: **APPROVE_WITH_NOTES**
|
||||
- Layers: structural **PASS**, behavioral **PASS**, security **PASS**, quality **PASS**
|
||||
- REQ coverage: **4/4** (REQ-DASH-01, REQ-NFR-DASH-01, REQ-NFR-DASH-02, REQ-MT-02 pipeline completion)
|
||||
- Grill MUSTs honored: **2/2** (G-038 differencing-attack test, G-041 SPA fallback via custom StaticFiles subclass)
|
||||
- P0 fixes applied: **0** (none needed — no P0 issues found)
|
||||
- P1+ flagged: **4** (non-blocking, for post-hoc review in P3)
|
||||
|
||||
> Phase 2 (P2) of the v0.4 milestone covers SLICE-07..10 (23 tasks): cohort aggregation pipeline, operator API endpoints, React cohort dashboard, and P2 integration. 4 commits since `milestone/v0.4-operator-tier`: c396ded (SLICE-07), a7f7c4e (SLICE-08), d39bd14 (SLICE-09), de2020e (SLICE-10).
|
||||
>
|
||||
> This report supersedes the prior TASK-10-05 verification matrix (preserved in §REQ-ID Coverage Matrix below).
|
||||
|
||||
---
|
||||
|
||||
## Layer 1 — Structural
|
||||
|
||||
### 1.1 File existence (all P2 files present)
|
||||
|
||||
| File | Exists | LOC | Notes |
|
||||
|------|--------|-----|-------|
|
||||
| `server/cohort/__init__.py` | YES | 0 | package marker |
|
||||
| `server/cohort/aggregator.py` | YES | 230 | k-anon suppression, 7-day window, metric cells |
|
||||
| `server/cohort/hook.py` | YES | 44 | fire-and-forget on_session_end, no-op if no Postgres |
|
||||
| `server/cohort/nightly.py` | YES | 232 | NightlyScheduler, 03:00 CT, R-DASH-04 retry |
|
||||
| `server/operator/__init__.py` | YES | 0 | package marker |
|
||||
| `server/operator/_common.py` | YES | 93 | shared Cell/PathView/ViewResponse models, require_pg_store, all_recent_aggregates |
|
||||
| `server/operator/cohort.py` | YES | 42 | GET /api/operator/cohort (practice volume) |
|
||||
| `server/operator/mastery.py` | YES | 45 | GET /api/operator/mastery (mastery progression) |
|
||||
| `server/operator/failure_patterns.py` | YES | 44 | GET /api/operator/failure-patterns |
|
||||
| `server/operator/credentials.py` | YES | 78 | GET /api/operator/credentials + POST /{id}/revoke |
|
||||
| `client/src/operator/Login.tsx` | YES | 93 | login form, 429 handling, keyboard-accessible |
|
||||
| `client/src/operator/Dashboard.tsx` | YES | 120 | auth gate, 3 view tabs, freshness, logout |
|
||||
| `client/src/operator/Sparkline.tsx` | YES | 49 | inline SVG polyline, zero deps |
|
||||
| `client/src/operator/views/PracticeVolume.tsx` | YES | 81 | practice volume view + sparklines |
|
||||
| `client/src/operator/views/MasteryProgression.tsx` | YES | 84 | mastery progression view |
|
||||
| `client/src/operator/views/FailurePatterns.tsx` | YES | 94 | failure patterns view |
|
||||
| `client/src/operator/views/_viewCommon.ts` | YES | 60 | shared Cell type, suppressedLabel, formatFreshness |
|
||||
| `client/src/operator/__tests__/Dashboard.test.tsx` | YES | 193 | 17 vitest tests |
|
||||
| `tests/test_cohort_aggregation.py` | YES | 246 | k-anon threshold, idempotency, G-038 |
|
||||
| `tests/test_cohort_nightly.py` | YES | 199 | scheduler timing, R-DASH-04, reconcile |
|
||||
| `tests/test_operator_endpoints.py` | YES | 304 | 401/200 auth, suppressed cells, revoke, R-DASH-02 |
|
||||
| `tests/test_p2_aggregation_integration.py` | YES | 236 | e2e aggregation→endpoint (skips without Postgres) |
|
||||
| `tests/test_p2_spa_fallback.py` | YES | 128 | 9 SPA fallback assertions (G-041) |
|
||||
| `client/vitest.config.ts` | YES | 13 | vitest config |
|
||||
| `client/src/App.tsx` (extended) | YES | 27 | BrowserRouter routes, voice UI at / unchanged |
|
||||
| `client/src/VoiceSession.tsx` | YES | 177 | extracted voice session (unchanged behavior) |
|
||||
| `server/session_recorder.py` (extended) | YES | +52 | aggregation hook chained, off voice path |
|
||||
| `server/__main__.py` (extended) | YES | +61 | operator routers + SpaStaticFiles + nightly scheduler |
|
||||
|
||||
### 1.2 Imports resolve
|
||||
- `python3 -c "import server.__main__"` → **OK** (server imports cleanly, logs "SPA fallback enabled")
|
||||
- `python3 -c "import server.cohort.aggregator, server.cohort.hook, server.cohort.nightly, server.operator.cohort, server.operator.mastery, server.operator.failure_patterns, server.operator.credentials"` → **OK** (all 7 new P2 modules import)
|
||||
|
||||
### 1.3 No stubs/TODOs in new P2 code
|
||||
- `grep -r "TODO|FIXME|stub|placeholder|NotImplemented" server/cohort/ server/operator/` → **No matches** (zero stubs, zero TODOs in new P2 server code)
|
||||
|
||||
### 1.4 Deps + build
|
||||
- `pip install -e . --break-system-packages` → **OK** (praxis-server 0.1.0 installed; P1 deps asyncpg/argon2-cffi/slowapi present)
|
||||
- `docker compose config` → **OK** (validates, praxis-data volume present)
|
||||
- `cd client && npm run build` → **OK** (vite v8.2.0, 168 modules, built in 547ms; bundle 662KB / 186KB gzip — within react-router-dom budget)
|
||||
- `cd client && npm run typecheck` → **OK** (tsc -b --noEmit, no errors)
|
||||
|
||||
### 1.5 Router mount order (critical for R-DASH-03)
|
||||
Verified in `server/__main__.py` diff (lines 256-298):
|
||||
1. `app.include_router(auth_router)` — `/api/operator/login|logout|me`
|
||||
2. `app.include_router(cohort_router)` — `/api/operator/cohort`
|
||||
3. `app.include_router(mastery_router)` — `/api/operator/mastery`
|
||||
4. `app.include_router(failure_router)` — `/api/operator/failure-patterns`
|
||||
5. `app.include_router(credentials_router)` — `/api/operator/credentials`
|
||||
6. `app.mount("/", SpaStaticFiles(...), name="spa")` — SPA fallback (AFTER all API routes)
|
||||
|
||||
**Order is correct**: API routes take precedence over the SPA fallback mount. R-DASH-03 verified.
|
||||
|
||||
**Layer 1 verdict: PASS** — all structural checks pass.
|
||||
|
||||
---
|
||||
|
||||
## Layer 2 — Behavioral
|
||||
|
||||
### 2.1 Test results
|
||||
|
||||
| Suite | Result | Notes |
|
||||
|-------|--------|-------|
|
||||
| `python3 -m pytest tests/` | **317 passed, 36 skipped, 0 failed** | matches expected (Postgres-requiring tests skip gracefully — PRAXIS_PG_DSN unset) |
|
||||
| `cd client && npx vitest run` | **17/17 passed** | Dashboard auth gate, login form (200/401/429), sparkline (empty/dot/polyline/flat), suppressedLabel, formatFreshness, no-PII-in-DOM |
|
||||
| `cd client && npm run build` | **PASS** | 168 modules, 547ms |
|
||||
| `cd client && npm run typecheck` | **PASS** | tsc clean |
|
||||
| P2-specific (`test_p2_spa_fallback.py` + `test_operator_endpoints.py` + `test_cohort_aggregation.py` + `test_cohort_nightly.py`) | **45/45 passed** | full P2 unit + SPA fallback coverage |
|
||||
| `test_p2_aggregation_integration.py` | **3 skipped** | gracefully skipped (no PRAXIS_PG_DSN) — e2e aggregation→endpoint path covered by unit tests with mocked PgStore |
|
||||
|
||||
### 2.2 P2 SLICE acceptance criteria
|
||||
|
||||
**SLICE-07 (aggregation pipeline):**
|
||||
- ✅ k-anon threshold exactly 10 — `test_k_anon_threshold_at_10` asserts `K_ANON_THRESHOLD == 10`; `test_9_learners_suppressed` (9 → suppressed), `test_10_learners_not_suppressed` (10 → not suppressed, value non-null), `test_11_learners_not_suppressed` (11 → not suppressed)
|
||||
- ✅ Idempotent upsert — `test_idempotent_same_session_twice` (ON CONFLICT at DB layer)
|
||||
- ✅ 7-day window — `test_rolling_window_7_days` (2026-08-04 → start=2026-07-29, 6-day span)
|
||||
- ✅ All metrics computed — `test_multiple_metrics_computed` (sessions_count, active_learners_count, gate_open_rate, median_mastery_score, rubric_criterion_mean:*, failure_mode:*, branch:*)
|
||||
- ✅ No PII in upserts — `test_no_pii_in_upsert_calls` (raw learner_ref not in any cell arg; cell_count is int)
|
||||
- ✅ Hook non-blocking — `server/cohort/hook.py` uses `asyncio.create_task` in `session_recorder.py:161`; hook swallows exceptions (`test_hook_failure_logs_does_not_raise`)
|
||||
- ✅ Hook no-op without Postgres — `test_hook_no_postgres_is_noop`
|
||||
- ✅ Nightly scheduler timing — `test_seconds_until_next_03_ct_future_today` + `test_seconds_until_next_03_ct_past_today_wraps_tomorrow`
|
||||
- ✅ R-DASH-04 nightly failure retry — `test_r_dash_04_nightly_failure_does_not_crash_scheduler`
|
||||
- ✅ Nightly reconcile recomputes — `test_reconcile_recomputes_all_paths`
|
||||
- ✅ Scheduler lifecycle — `test_scheduler_start_stop_lifecycle`
|
||||
|
||||
**SLICE-08 (operator API endpoints):**
|
||||
- ✅ All 4 endpoints auth-gated (401 without cookie) — `test_cohort_401_without_cookie`, `test_mastery_401_without_cookie`, `test_failure_patterns_401_without_cookie`, `test_credentials_401_without_cookie`, `test_revoke_401_without_cookie`
|
||||
- ✅ All 4 endpoints 200 with cookie — `test_cohort_200_with_cookie`, `test_mastery_200_with_cookie`, `test_failure_patterns_200_with_cookie`, `test_credentials_200_with_cookie`
|
||||
- ✅ Suppressed cells value=null — `test_suppressed_cells_value_null` (cell_suppressed=true → value=null)
|
||||
- ✅ last_updated = max(updated_at) — `test_last_updated_is_max`
|
||||
- ✅ Credential revoke — `test_credential_revoke_sets_status_revoked` (status='revoked', set_credential_status awaited) + `test_credential_revoke_404_unknown` (404 for unknown)
|
||||
- ✅ No per-learner data (R-DASH-02) — `test_no_per_learner_data_in_cohort_response` (no "learner-1", no "learner_ref" in response)
|
||||
- ✅ 503 when no Postgres — `test_cohort_503_no_postgres` (graceful degradation)
|
||||
|
||||
**SLICE-09 (React dashboard):**
|
||||
- ✅ react-router-dom@^7 added (`client/package.json`)
|
||||
- ✅ BrowserRouter wrapper + route switch — `client/src/App.tsx`: `/` → VoiceSession (unchanged), `/operator/login` → Login, `/operator/dashboard` → Dashboard, `*` → VoiceSession (fallback)
|
||||
- ✅ Login form — Login.tsx, 429 handling (`test shows rate-limit message on 429`), keyboard-accessible (label associations)
|
||||
- ✅ Dashboard shell + auth gate — Dashboard.tsx, 401 on /me → redirect (`test redirects to /operator/login on 401`), 3 view tabs, freshness indicator, logout
|
||||
- ✅ Inline SVG sparkline — Sparkline.tsx (49 LOC, zero deps), empty/dot/polyline/flat-line cases tested
|
||||
- ✅ 3 view components — PracticeVolume, MasteryProgression, FailurePatterns (read-only, no drill-down)
|
||||
- ✅ Suppressed cell display — "— (<10 learners)" (`suppressedLabel` test)
|
||||
- ✅ Freshness indicator — formatFreshness (m/h/d ago)
|
||||
- ✅ No PII in DOM — `test does not render learner_ref fields`
|
||||
|
||||
**SLICE-10 (P2 integration):**
|
||||
- ✅ SPA fallback (G-041) — custom `SpaStaticFiles` subclass in `__main__.py:279-289`, NOT a catch-all route; 9 assertions in `test_p2_spa_fallback.py` all pass
|
||||
- ✅ Voice UI at `/` unchanged (R-DASH-05) — `test_root_serves_voice_ui` (200, text/html, `<div id="root">`)
|
||||
- ✅ API routes return JSON not HTML — `test_api_operator_cohort_is_json_not_html`, `test_health_is_json`, `test_vc_verify_nonexistent_is_404`
|
||||
- ✅ Assets served by StaticFiles — `test_assets_served_by_staticfiles_not_spa_fallback` (`/assets/index.js` → javascript content-type, not index.html)
|
||||
- ✅ Nightly scheduler starts in lifespan — `server/__main__.py:116` `await nightly.start(app.state.pg_store)`; cancelled on shutdown (`await nightly.stop()` line 121)
|
||||
- ✅ E2e aggregation→endpoint — `test_p2_aggregation_integration.py::test_aggregation_to_endpoint_e2e` (skips without Postgres; logic covered by unit tests with mocked store)
|
||||
|
||||
### 2.3 REQ coverage
|
||||
|
||||
| REQ-ID | Covered by | Status |
|
||||
|--------|-----------|--------|
|
||||
| **REQ-DASH-01** (cohort dashboard, 3 views, k-anon, React under /operator/*) | SLICE-08 (4 endpoints), SLICE-09 (React UI), SLICE-10 (integration). `test_operator_endpoints.py` (all 4 endpoints 200/401), `Dashboard.test.tsx` (auth gate, login, 3 views), `test_p2_spa_fallback.py` (SPA serves /operator/*) | **COVERED** |
|
||||
| **REQ-NFR-DASH-01** (k-anonymity ≥ 10) | SLICE-07 (write-time suppression in `aggregator.py`), SLICE-08 (query returns value=null for suppressed), SLICE-09 (display "— (<10 learners)"), SLICE-10 (e2e). `test_cohort_aggregation.py` (threshold at 10, 9/10/11 learners), `test_operator_endpoints.py::test_suppressed_cells_value_null`, `Dashboard.test.tsx::suppressedLabel`, G-038 differencing-attack | **COVERED** |
|
||||
| **REQ-NFR-DASH-02** (freshness ≤ 24h) | SLICE-07 (nightly job + on-session-end hook), SLICE-10 (e2e). `test_cohort_nightly.py` (scheduler timing, reconcile, R-DASH-04), `test_operator_endpoints.py::test_last_updated_is_max`, `test_p2_aggregation_integration.py::test_nightly_reconciliation_updates_last_updated` (skips without Postgres) | **COVERED** |
|
||||
| **REQ-MT-02** (pipeline completion — schema P1, pipeline P2) | SLICE-07 (aggregator + hook + nightly), SLICE-10 (e2e). `test_cohort_aggregation.py` (idempotent, multiple metrics, hook no-op/failure), `test_cohort_nightly.py` (reconcile), `test_p2_aggregation_integration.py::test_aggregation_to_endpoint_e2e` (skips without Postgres) | **COVERED** |
|
||||
|
||||
**4/4 P2 REQ-IDs covered.**
|
||||
|
||||
### 2.4 Grill MUSTs honored
|
||||
|
||||
**G-038 (differencing-attack test) — HONORED:**
|
||||
- Unit layer: `test_cohort_aggregation.py::test_g038_differencing_attack_cannot_isolate_dropped_learner` — seeds 10 learners in window A, 9 in window B (learner-9 dropped), asserts window B is FULLY suppressed (value=NULL) so the dropped learner's contribution is not recoverable via subtraction. Verifies no per-learner ref leaks in either window's aggregate cells.
|
||||
- API e2e layer: `test_p2_aggregation_integration.py::test_g038_differencing_attack_api_layer` — 10 learners on path diff_a, 9 on diff_b, asserts "a-9" not in response text and diff_b cells all suppressed with value=None. (Skips without Postgres — logic verified at unit layer.)
|
||||
|
||||
**G-041 (SPA fallback via custom StaticFiles subclass) — HONORED:**
|
||||
- Implementation: `server/__main__.py:279-289` defines `class SpaStaticFiles(StaticFiles)` with `get_response` override that returns `FileResponse("index/dist/index.html")` only on 404 (non-file paths). This is the custom subclass approach mandated by G-041, NOT a `@app.get("/{path:path}")` catch-all (which would shadow asset serving per the grill's analysis).
|
||||
- Test: `test_p2_spa_fallback.py::test_assets_served_by_staticfiles_not_spa_fallback` verifies `/assets/index.js` returns javascript content (not index.html) — the critical assertion 8 from TASK-10-04.
|
||||
|
||||
### 2.5 Voice UI at `/` unchanged (R-DASH-03, R-DASH-05)
|
||||
|
||||
- **Server**: `SpaStaticFiles` mount at `/` with `html=True` serves `index.html` for `/` (unchanged from v0.3 StaticFiles behavior). API routes registered before the mount take precedence. `test_root_serves_voice_ui` confirms 200 + text/html + `<div id="root">`.
|
||||
- **Client**: `client/src/App.tsx` route `/` → `<VoiceSession />` (the existing voice session UI, extracted from the old App.tsx to VoiceSession.tsx — behavior unchanged). The `*` catch-all also serves VoiceSession (R-DASH-05: unknown routes fall back to learner surface, not a 404).
|
||||
- **No regression**: 317 passed, 0 failed — all v0.1/v0.2/v0.3 tests still pass.
|
||||
|
||||
**Voice UI at `/` unchanged: CONFIRMED.**
|
||||
|
||||
**Layer 2 verdict: PASS** — all behavioral checks pass.
|
||||
|
||||
---
|
||||
|
||||
## Layer 3 — Security (STRIDE)
|
||||
|
||||
### Spoofing
|
||||
- **Operator endpoints auth-gated via `current_operator` dependency.**
|
||||
- Verified: all 4 operator routers (`cohort.py`, `mastery.py`, `failure_patterns.py`, `credentials.py`) import `current_operator` from `server.auth.dependencies` and apply `op: Operator = Depends(current_operator)` on every endpoint.
|
||||
- Test coverage: 5 tests assert 401 without cookie (`test_cohort_401_without_cookie`, `test_mastery_401_without_cookie`, `test_failure_patterns_401_without_cookie`, `test_credentials_401_without_cookie`, `test_revoke_401_without_cookie`).
|
||||
- **Disposition: low (accept).** No bypass path found — every `/api/operator/*` route (except `/login` which is rate-limited, not auth-gated) requires the dependency.
|
||||
|
||||
### Tampering
|
||||
- **Aggregation pipeline — k-anon suppression at write time.**
|
||||
- `server/cohort/aggregator.py:87` `suppressed = active_count < K_ANON_THRESHOLD` (K_ANON_THRESHOLD=10, module constant). Suppression applied before `upsert_cohort_aggregate` — value set to `None` when suppressed (lines 90, 94, 103, etc.).
|
||||
- Nightly reconciliation (`nightly.py:127`) re-applies the same threshold: `suppressed = active_count < K_ANON_THRESHOLD`.
|
||||
- Suppression cannot be bypassed via the API: endpoints read `cohort_aggregates` rows as-is (no post-processing that could un-suppress); suppressed cells have `value=null` in the DB (enforced at write time).
|
||||
- **Disposition: low (accept).** Write-time suppression is server-side, not display-only.
|
||||
|
||||
### Repudiation
|
||||
- **Credential revoke (POST /api/operator/credentials/{id}/revoke).**
|
||||
- The revoke endpoint sets `status='revoked'` + `revoked_at=now()` in Postgres (`pg_store.py:224` `extra = ", revoked_at = now()" if status == 'revoked'`). The `revoked_at` timestamp is an audit trail.
|
||||
- **GAP (P1+ flagged)**: The revoke endpoint does NOT log the revocation event at the application level, and the `operator_id` of the revoking operator is available via `current_operator` but is NOT recorded against the credential revocation. The `issued_credentials.operator_id` column tracks the *issuer*, not the *revoker*. There is no revocation audit log linking operator→action→credential→timestamp.
|
||||
- Mitigation: the `revoked_at` timestamp + the signed session cookie (which records `operator_id` in `request.session`) provide a partial audit trail, but correlating them requires cross-referencing session logs.
|
||||
- **Disposition: medium (mitigate — P1+ flagged).** Add application-level logging of revocation events (operator_id, credential_id, timestamp) in P3.
|
||||
|
||||
### Info Disclosure
|
||||
- **k-anonymity ≥ 10 enforced (REQ-NFR-DASH-01).**
|
||||
- Write-time suppression: cells with < 10 distinct learners → `cell_suppressed=TRUE`, `value=NULL`. Verified by `test_9_learners_suppressed`, `test_10_learners_not_suppressed`.
|
||||
- No per-learner drill-down (R-DASH-02): endpoints return only aggregate cells (path, metric, value, cell_count, cell_suppressed) — no `learner_ref` in cohort/mastery/failure responses. Verified by `test_no_per_learner_data_in_cohort_response` (no "learner_ref" string, no "learner-1" in response).
|
||||
- G-038 differencing-attack defense: window B (9 learners) is fully suppressed (value=NULL), so subtracting B from A is not possible. Verified at unit + API layers.
|
||||
- No PII in Postgres aggregates (D-031): only opaque `learner_ref` for distinct counting, never stored in aggregate cells. Verified by `test_no_pii_in_upsert_calls`.
|
||||
- **Disposition: low (accept).** k-anon defense-in-depth is sound; G-038 explicitly tested.
|
||||
|
||||
### Denial of Service
|
||||
- **Aggregation hook is async fire-and-forget (non-blocking).**
|
||||
- `server/session_recorder.py:161` `asyncio.create_task(self._run_cohort_aggregation(session_outcome))` — hook runs off the voice path (C-8, D-054). Voice loop latency unaffected.
|
||||
- `server/cohort/hook.py:37` `except Exception: log.exception(...)` — hook failure does not propagate; nightly job reconciles.
|
||||
- `test_hook_failure_logs_does_not_raise` confirms no exception propagation.
|
||||
- Nightly job doesn't block the event loop: `NightlyScheduler._run_loop` uses `asyncio.sleep(secs)` (cooperative); reconciliation is a sequence of `await pg_store.upsert_cohort_aggregate(...)` calls (yields between each).
|
||||
- **Disposition: low (accept).** Hook failure → log + nightly reconcile (R-DASH-04). No crash path.
|
||||
|
||||
### Elevation of Privilege
|
||||
- **Single operator role. No RBAC bypass.**
|
||||
- All 4 operator endpoints + credential management use `Depends(current_operator)`. The `current_operator` dependency (`server/auth/dependencies.py`) checks `request.session["operator_id"]` → fetches operator → checks `is_active=True` → returns `Operator`. No role-based dispatch exists (single role).
|
||||
- The `current_operator` dependency never trusts the client (D-057) — it validates the signed session cookie server-side.
|
||||
- **Disposition: low (accept).** No RBAC to bypass; single operator role; auth-gated everywhere.
|
||||
|
||||
**Layer 3 verdict: PASS** — all STRIDE categories low except Repudiation (medium, mitigated, P1+ flagged). No high-severity findings.
|
||||
|
||||
---
|
||||
|
||||
## Layer 4 — Quality (multi-persona review)
|
||||
|
||||
### Correctness
|
||||
- **k-anon threshold (exactly 10):** `K_ANON_THRESHOLD = 10` module constant; 9 → suppressed, 10 → not suppressed, 11 → not suppressed. Tests cover all three boundaries. ✅
|
||||
- **Aggregation idempotency:** ON CONFLICT upsert at the DB layer (PgStore); hook is deterministic (same learner produces same distinct-count + counter state in cache). `test_idempotent_same_session_twice` passes. ✅
|
||||
- **Nightly scheduler timing:** `seconds_until_next_03_ct` computes seconds until 03:00 CT (fixed UTC-5 offset, documented DST approximation — acceptable for nightly reconciliation). `test_seconds_until_next_03_ct_future_today` + `test_seconds_until_next_03_ct_past_today_wraps_tomorrow` pass. ✅
|
||||
- **SPA fallback (G-041):** Custom `SpaStaticFiles` subclass, NOT catch-all route. Serves assets normally (JS/CSS), falls back to index.html only on 404. `test_assets_served_by_staticfiles_not_spa_fallback` confirms assets are not shadowed. ✅
|
||||
|
||||
### Testing
|
||||
- **Coverage gaps:** Postgres-requiring tests (`test_p2_aggregation_integration.py`, `test_pg_store.py`) skip gracefully when `PRAXIS_PG_DSN` unset — 36 skipped total, 0 failed. The e2e aggregation→endpoint→dashboard path is covered by unit tests with mocked PgStore (45/45 P2 tests pass). ✅
|
||||
- **Client tests (vitest):** 17/17 pass — auth gate, login (200/401/429), sparkline (4 cases), suppressedLabel, formatFreshness, no-PII-in-DOM. ✅
|
||||
- **G-038 differencing-attack coverage:** Unit layer (`test_g038_differencing_attack_cannot_isolate_dropped_learner`) + API e2e layer (`test_g038_differencing_attack_api_layer`). The unit test is the primary proof (runs without Postgres); the e2e test is a bonus that skips without Postgres. ✅
|
||||
|
||||
### Security
|
||||
- **SQL injection in PgStore queries:** All queries use asyncpg parameterized placeholders (`$1`, `$2`, etc.). Verified in `pg_store.py` (operator CRUD, cohort upsert, credential methods, gate events) and `server/operator/_common.py::all_recent_aggregates` (`WHERE window_start >= $1`). One f-string interpolation in `set_credential_status` (`f"UPDATE ... SET status = $1{extra} WHERE id = $2"`) — but `extra` is a hardcoded constant (`, revoked_at = now()` or empty) derived from the `status` value comparison, NOT user input. Safe. ✅
|
||||
- **k-anon suppression enforced server-side:** Suppression is applied in `aggregator.py` (write time) and re-applied in `nightly.py` (reconcile). The API endpoints read cells as-is — no client-side or display-only suppression. ✅
|
||||
- **No PII in API responses:** Cohort/mastery/failure endpoints return only (path, metric, value, cell_count, cell_suppressed, updated_at). Credentials endpoint returns (id, learner_ref, vc_type, status, issued_at, revoked_at) — `learner_ref` is an opaque string (D-031), not PII. ✅
|
||||
|
||||
### Performance
|
||||
- **Aggregation hook non-blocking:** `asyncio.create_task` in `session_recorder.py:161` — fire-and-forget, off the voice path (C-8). ✅
|
||||
- **Nightly job doesn't block event loop:** `asyncio.sleep(secs)` + sequential `await` calls (cooperative). Runs at 03:00 CT (low activity). ✅
|
||||
- **SPA fallback doesn't add latency to API routes:** API routes are registered before the StaticFiles mount — FastAPI matches API routes first (no fallback overhead). ✅
|
||||
|
||||
### Maintainability
|
||||
- **SpaStaticFiles subclass:** Clean 11-line override (`get_response` catches 404 → FileResponse). Well-commented with G-041 rationale. ✅
|
||||
- **3 view components consistent:** All 3 (PracticeVolume, MasteryProgression, FailurePatterns) share `_viewCommon.ts` (Cell type, suppressedLabel, formatFreshness) and follow the same fetch→render pattern. ✅
|
||||
- **Router mounting order:** API routes → SPA fallback mount. Documented in `__main__.py:256-298` comments. ✅
|
||||
|
||||
### Adversarial
|
||||
- **What if an attacker calls /api/operator/cohort with a path that doesn't exist?** The endpoint takes no path parameter — it returns all paths' aggregates from the last 30 days. A non-existent path simply returns no rows (no error, no leak). ✅
|
||||
- **What if k-anon threshold is lowered via config?** `K_ANON_THRESHOLD = 10` is a module constant in `aggregator.py`, NOT configurable via env. Changing it requires a code change + redeploy. This is correct for a privacy control — it should not be runtime-configurable. ✅
|
||||
- **What if the aggregation hook runs before Postgres is healthy?** The hook checks `pg_store is None` → no-op + WARNING (`hook.py:27-32`). If Postgres is unhealthy mid-session, `upsert_cohort_aggregate` raises → caught by `hook.py:37` `except Exception: log.exception(...)` → nightly job reconciles. ✅
|
||||
|
||||
**Layer 4 verdict: PASS** — no quality issues found. Code is clean, well-commented, consistently structured, and adversarially sound.
|
||||
|
||||
---
|
||||
|
||||
## P0 Fixes Applied
|
||||
|
||||
**None.** No P0 issues (broken tests, missing REQ coverage, security holes) were found. The P2 implementation is correct, complete, and secure.
|
||||
|
||||
---
|
||||
|
||||
## P1+ Flagged for Post-Hoc Review
|
||||
|
||||
The following non-blocking issues are flagged for review in the final phase (P3):
|
||||
|
||||
### P1+-01: Credential revocation lacks application-level audit log (Repudiation)
|
||||
- **File:** `server/operator/credentials.py`
|
||||
- **Issue:** The `revoke_credential` endpoint sets `status='revoked'` + `revoked_at=now()` in Postgres but does NOT log the revocation event at the application level, and the revoking `operator_id` (available via `current_operator`) is not recorded against the revocation action. The `issued_credentials.operator_id` column tracks the *issuer*, not the *revoker*.
|
||||
- **Risk:** An operator who revokes a credential leaves a DB timestamp but no application log linking *who* revoked *which* credential *when*. Correlating requires cross-referencing session logs.
|
||||
- **Mitigation present:** `revoked_at` timestamp in DB + signed session cookie (operator_id in session).
|
||||
- **Recommended fix (P3):** Add `log.info("credential revoked: operator=%s cred_id=%s", op.id, cred_id)` in `revoke_credential`, and consider an `audit_log` table or `revoked_by_operator_id` column on `issued_credentials`.
|
||||
|
||||
### P1+-02: Nightly scheduler uses fixed UTC-5 offset (not true America/Winnipeg DST)
|
||||
- **File:** `server/cohort/nightly.py:27` `CT = _dt.timezone(_dt.timedelta(hours=-5), "CT")`
|
||||
- **Issue:** The CT timezone is approximated as a fixed UTC-5 offset. America/Winnipeg observes CST (UTC-6) in winter + CDT (UTC-5) in summer. The scheduler will drift by 1 hour across DST boundaries (the nightly job runs at 02:00 or 04:00 local instead of 03:00).
|
||||
- **Risk:** Low — the nightly job runs once/day; a 1-hour drift is acceptable for a reconciliation job (on-session-end hook keeps data fresh ≤ 24h).
|
||||
- **Mitigation present:** Documented in `nightly.py:36-41` comments ("drift of ≤1h over DST boundaries is acceptable... a future hardening would use zoneinfo.ZoneInfo").
|
||||
- **Recommended fix (P3):** Replace `CT` constant with `zoneinfo.ZoneInfo("America/Winnipeg")` for proper DST handling.
|
||||
|
||||
### P1+-03: Aggregation in-memory cache is per-PgStore-instance (lost on restart)
|
||||
- **File:** `server/cohort/aggregator.py:162-170` `_cache(pg_store)`
|
||||
- **Issue:** The aggregator maintains a per-PgStore-instance in-memory cache (`_agg_cache`) for running counters + distinct learner sets. On server restart, the cache is lost — the next on-session-end hook starts fresh, and the active_learners_count may reset to 1 (under-counting distinct learners until the nightly job reconciles from `mastery_gate_events`).
|
||||
- **Risk:** Low — the nightly job reconciles the true distinct count from the audit log (`mastery_gate_events`). Between restart and nightly reconcile, cells may be incorrectly suppressed (under-count → over-suppression, which is privacy-safe but value-destroying).
|
||||
- **Mitigation present:** Nightly reconciliation recomputes from `mastery_gate_events` (the source of truth).
|
||||
- **Recommended fix (P3):** Document that the in-memory cache is best-effort + nightly reconcile is authoritative, OR persist the distinct-learner set to Postgres (adds a table — may not be worth the complexity for pilot scale).
|
||||
|
||||
### P1+-04: `set_credential_status` uses f-string interpolation in SQL (code smell, not vulnerability)
|
||||
- **File:** `db/pg_store.py:227` `f"UPDATE issued_credentials SET status = $1{extra} WHERE id = $2"`
|
||||
- **Issue:** The `extra` variable (`, revoked_at = now()` or empty string) is interpolated via f-string into the SQL query. While `extra` is a hardcoded constant (not user input) and `status`/`cred_id` are parameterized, f-strings in SQL are a code smell that future maintainers might copy incorrectly.
|
||||
- **Risk:** None (current code is safe — `extra` is derived from `status == "revoked"` comparison, not user input).
|
||||
- **Recommended fix (P3):** Refactor to two explicit queries: `UPDATE ... SET status = $1 WHERE id = $2` and `UPDATE ... SET status = $1, revoked_at = now() WHERE id = $2`, eliminating the f-string.
|
||||
|
||||
---
|
||||
|
||||
## REQ-ID Coverage Matrix (from TASK-10-05, preserved)
|
||||
|
||||
### REQ-DASH-01 — Cohort dashboard (3 views + auth gate)
|
||||
| Test file | Test | What it verifies |
|
||||
|-----------|------|------------------|
|
||||
| tests/test_operator_endpoints.py | test_cohort_200_with_cookie | GET /api/operator/cohort returns practice volume |
|
||||
| tests/test_operator_endpoints.py | test_mastery_200_with_cookie | GET /api/operator/mastery returns mastery progression |
|
||||
| tests/test_operator_endpoints.py | test_failure_patterns_200_with_cookie | GET /api/operator/failure-patterns returns failure data |
|
||||
| tests/test_operator_endpoints.py | test_credentials_200_with_cookie | GET /api/operator/credentials lists VCs |
|
||||
| tests/test_operator_endpoints.py | test_cohort_401_without_cookie (+ 4 others) | All endpoints auth-gated (401) |
|
||||
| client/src/operator/__tests__/Dashboard.test.tsx | Dashboard auth gate | React auth gate redirects on 401 from /me |
|
||||
| client/src/operator/__tests__/Dashboard.test.tsx | Login form | POST /api/operator/login → dashboard |
|
||||
| tests/test_p2_spa_fallback.py | test_operator_dashboard_spa_fallback | /operator/dashboard serves index.html (SPA) |
|
||||
| tests/test_p2_spa_fallback.py | test_operator_login_spa_fallback | /operator/login serves index.html (SPA) |
|
||||
|
||||
### REQ-NFR-DASH-01 — k-anonymity ≥ 10 (write-time suppression + query + display + e2e)
|
||||
| Test file | Test | What it verifies |
|
||||
|-----------|------|------------------|
|
||||
| tests/test_cohort_aggregation.py | test_k_anon_threshold_at_10 | K_ANON_THRESHOLD == 10 |
|
||||
| tests/test_cohort_aggregation.py | test_9_learners_suppressed | 9 learners → cell_suppressed=TRUE, value=NULL |
|
||||
| tests/test_cohort_aggregation.py | test_10_learners_not_suppressed | 10 learners → non-suppressed, value non-null |
|
||||
| tests/test_cohort_aggregation.py | test_11_learners_not_suppressed | 11 learners → non-suppressed |
|
||||
| tests/test_cohort_aggregation.py | test_no_pii_in_upsert_calls | No raw learner_ref in aggregate cell args |
|
||||
| tests/test_cohort_aggregation.py | test_g038_differencing_attack_cannot_isolate_dropped_learner | G-038: 10 in window A, 9 in B → dropped learner not isolatable |
|
||||
| tests/test_operator_endpoints.py | test_suppressed_cells_value_null | API: suppressed cells have value=null |
|
||||
| tests/test_operator_endpoints.py | test_no_per_learner_data_in_cohort_response | API: no per-learner data (R-DASH-02) |
|
||||
| client/src/operator/__tests__/Dashboard.test.tsx | suppressedLabel | UI: suppressed cells render "— (<10 learners)" |
|
||||
| tests/test_p2_aggregation_integration.py | test_aggregation_to_endpoint_e2e | E2e: 12 learners non-suppressed, 5 suppressed (skips without Postgres) |
|
||||
| tests/test_p2_aggregation_integration.py | test_g038_differencing_attack_api_layer | G-038 e2e at API layer (skips without Postgres) |
|
||||
|
||||
### REQ-NFR-DASH-02 — Freshness ≤ 24h (nightly job + on-session-end hook)
|
||||
| Test file | Test | What it verifies |
|
||||
|-----------|------|------------------|
|
||||
| tests/test_cohort_nightly.py | test_seconds_until_next_03_ct_future_today | Scheduler computes correct seconds until 03:00 CT |
|
||||
| tests/test_cohort_nightly.py | test_seconds_until_next_03_ct_past_today_wraps_tomorrow | Wraps to next day correctly |
|
||||
| tests/test_cohort_nightly.py | test_reconcile_recomputes_all_paths | Nightly recomputes all (path, window) cells |
|
||||
| tests/test_cohort_nightly.py | test_r_dash_04_nightly_failure_does_not_crash_scheduler | R-DASH-04: failure logs + retries |
|
||||
| tests/test_cohort_nightly.py | test_scheduler_start_stop_lifecycle | Scheduler starts + stops cleanly |
|
||||
| tests/test_operator_endpoints.py | test_last_updated_is_max | API: last_updated = max(updated_at) |
|
||||
| tests/test_p2_aggregation_integration.py | test_nightly_reconciliation_updates_last_updated | E2e: nightly reconcile refreshes last_updated (skips without Postgres) |
|
||||
| tests/test_p2_aggregation_integration.py | test_aggregation_to_endpoint_e2e (assertion 8) | E2e: last_updated ≤ 24h (skips without Postgres) |
|
||||
|
||||
### REQ-MT-02 — Cohort aggregation pipeline (schema in P1, pipeline in P2)
|
||||
| Test file | Test | What it verifies |
|
||||
|-----------|------|------------------|
|
||||
| tests/test_cohort_aggregation.py | test_multiple_metrics_computed | Pipeline computes all metric types |
|
||||
| tests/test_cohort_aggregation.py | test_idempotent_same_session_twice | Idempotent upsert |
|
||||
| tests/test_cohort_aggregation.py | test_rolling_window_7_days | 7-day rolling window computation |
|
||||
| tests/test_cohort_aggregation.py | test_hook_no_postgres_is_noop | Graceful no-op without Postgres |
|
||||
| tests/test_cohort_aggregation.py | test_hook_failure_logs_does_not_raise | Hook failure does not propagate |
|
||||
| tests/test_cohort_nightly.py | test_reconcile_no_events_no_op | Nightly no-op when no events |
|
||||
| tests/test_p2_aggregation_integration.py | test_aggregation_to_endpoint_e2e | Full pipeline e2e (skips without Postgres) |
|
||||
|
||||
### G-038 (binding — differencing-attack test)
|
||||
| Test file | Test | What it verifies |
|
||||
|-----------|------|------------------|
|
||||
| tests/test_cohort_aggregation.py | test_g038_differencing_attack_cannot_isolate_dropped_learner | Unit: 10 in A, 9 in B → B suppressed, dropped learner not isolatable |
|
||||
| tests/test_p2_aggregation_integration.py | test_g038_differencing_attack_api_layer | E2e at API layer (skips without Postgres) |
|
||||
|
||||
### G-041 (binding — SPA fallback via custom StaticFiles subclass)
|
||||
| Test file | Test | What it verifies |
|
||||
|-----------|------|------------------|
|
||||
| tests/test_p2_spa_fallback.py | test_root_serves_voice_ui | Voice UI at / unchanged (R-DASH-05) |
|
||||
| tests/test_p2_spa_fallback.py | test_operator_dashboard_spa_fallback | /operator/dashboard → index.html |
|
||||
| tests/test_p2_spa_fallback.py | test_assets_served_by_staticfiles_not_spa_fallback | /assets/index.js served by StaticFiles (NOT catch-all) — G-041 critical assertion |
|
||||
| tests/test_p2_spa_fallback.py | test_api_operator_cohort_is_json_not_html | API routes return JSON (not index.html) |
|
||||
| tests/test_p2_spa_fallback.py | test_health_is_json | /health JSON |
|
||||
|
||||
### R-DASH-05 (voice UI at / unchanged)
|
||||
| Test file | Test | What it verifies |
|
||||
|-----------|------|------------------|
|
||||
| tests/test_p2_spa_fallback.py | test_root_serves_voice_ui | / → index.html with <div id="root"> |
|
||||
| client/src/operator/__tests__/Dashboard.test.tsx | (no PII in dashboard DOM) | Voice UI path unchanged |
|
||||
|
||||
---
|
||||
|
||||
## Test Results Summary
|
||||
|
||||
| Suite | Pass | Skip | Fail |
|
||||
|-------|------|------|------|
|
||||
| `python3 -m pytest tests/` (full) | 317 | 36 | 0 |
|
||||
| `tests/test_p2_spa_fallback.py` | 9 | 0 | 0 |
|
||||
| `tests/test_operator_endpoints.py` | 15 | 0 | 0 |
|
||||
| `tests/test_cohort_aggregation.py` | 12 | 0 | 0 |
|
||||
| `tests/test_cohort_nightly.py` | 9 | 0 | 0 |
|
||||
| `tests/test_p2_aggregation_integration.py` | 0 | 3 | 0 (Postgres-requiring, skip gracefully) |
|
||||
| `cd client && npx vitest run` | 17 | 0 | 0 |
|
||||
| `cd client && npm run build` | PASS | — | — |
|
||||
| `cd client && npm run typecheck` | PASS | — | — |
|
||||
| `pip install -e . --break-system-packages` | PASS | — | — |
|
||||
| `docker compose config` | PASS | — | — |
|
||||
| `python3 -c "import server.__main__"` | PASS | — | — |
|
||||
| `python3 -c "import ...all P2 modules"` | PASS | — | — |
|
||||
|
||||
---
|
||||
|
||||
## Voice UI at `/` Unchanged — Confirmation
|
||||
|
||||
**CONFIRMED.** Three layers of evidence:
|
||||
|
||||
1. **Server (`server/__main__.py`):** The `SpaStaticFiles` mount at `/` with `html=True` serves `index.html` for `/` — identical to the v0.3 `StaticFiles` behavior. The custom subclass only changes behavior for *non-file* paths (404 → index.html), not for `/` (which StaticFiles already serves as index.html with `html=True`). `test_root_serves_voice_ui` confirms 200 + text/html + `<div id="root">`.
|
||||
|
||||
2. **Client (`client/src/App.tsx`):** Route `/` → `<VoiceSession />`. The VoiceSession component was extracted from the old App.tsx (behavior unchanged — same voice session UI). The `*` catch-all also serves VoiceSession (R-DASH-05: unknown routes fall back to learner surface).
|
||||
|
||||
3. **Test suite:** 317 passed, 0 failed — all v0.1/v0.2/v0.3 tests (voice loop, WebRTC, scenarios, mastery, VC) still pass. No regression in the learner surface.
|
||||
|
||||
---
|
||||
|
||||
## Bottom Line
|
||||
|
||||
Phase 2 (Cohort Dashboard + Aggregation) is **APPROVE_WITH_NOTES**. All 4 layers pass. All 4 P2 REQ-IDs are covered. Both grill MUSTs (G-038 differencing-attack test, G-041 SPA fallback via custom StaticFiles subclass) are honored. Zero P0 issues. Four P1+ issues flagged for post-hoc review in P3 (credential revocation audit log, nightly scheduler DST, in-memory cache persistence, f-string SQL code smell) — all non-blocking, all with mitigations present.
|
||||
|
||||
The P2 implementation is shippable as `v0.1.8` pending the final P3 review + ship phase.
|
||||
Generated
+2
-2674
File diff suppressed because it is too large
Load Diff
+3
-9
@@ -9,27 +9,21 @@
|
||||
"typecheck": "tsc -b --noEmit",
|
||||
"lint": "oxlint",
|
||||
"preview": "vite preview",
|
||||
"test": "vitest run",
|
||||
"test:watch": "vitest"
|
||||
"test": "echo 'client: no unit tests yet (v0.1 uses e2e smoke via server tests)' && exit 0"
|
||||
},
|
||||
"dependencies": {
|
||||
"@pipecat-ai/client-js": "^1.13.0",
|
||||
"@pipecat-ai/small-webrtc-transport": "^1.10.6",
|
||||
"react": "^19.2.8",
|
||||
"react-dom": "^19.2.8",
|
||||
"react-router-dom": "^7.1.0"
|
||||
"react-dom": "^19.2.8"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@testing-library/jest-dom": "^6.9.1",
|
||||
"@testing-library/react": "^16.3.2",
|
||||
"@types/node": "^24.13.3",
|
||||
"@types/react": "^19.2.17",
|
||||
"@types/react-dom": "^19.2.3",
|
||||
"@vitejs/plugin-react": "^6.0.4",
|
||||
"jsdom": "^25.0.1",
|
||||
"oxlint": "^1.75.0",
|
||||
"typescript": "~6.0.2",
|
||||
"vite": "^8.2.0",
|
||||
"vitest": "^3.2.7"
|
||||
"vite": "^8.2.0"
|
||||
}
|
||||
}
|
||||
|
||||
+175
-21
@@ -1,27 +1,181 @@
|
||||
/**
|
||||
* Praxis — top-level route switch (SLICE-09 TASK-09-02, D-044, R-DASH-05).
|
||||
* Praxis v0.1 — full session UX (SLICE-05 TASK-05-04).
|
||||
*
|
||||
* Routes:
|
||||
* / → existing voice session UI (unchanged)
|
||||
* /operator/login → operator Login form
|
||||
* /operator/dashboard → operator Dashboard (auth-gated)
|
||||
* * → voice session UI (SPA fallback for unknown routes)
|
||||
*
|
||||
* R-DASH-05: the voice UI at `/` is unchanged. The catch-all serves the
|
||||
* voice UI (not a 404) so unknown routes fall back to the learner surface.
|
||||
* Three views: start → live → debrief. Replaces the SLICE-02 minimal page.
|
||||
* - Start: scenario title + disclaimer acknowledgement + Start button
|
||||
* - Live: turn indicators (learner/AI), interrupt feedback, latency readout
|
||||
* - Debrief: debrief text + audio replay control + latency/cost summary
|
||||
*/
|
||||
import { Routes, Route } from 'react-router-dom'
|
||||
import VoiceSession from './VoiceSession'
|
||||
import Login from './operator/Login'
|
||||
import Dashboard from './operator/Dashboard'
|
||||
import { useVoiceSession } from './useVoiceSession'
|
||||
import { useEffect, useState } from 'react'
|
||||
import './App.css'
|
||||
|
||||
type View = 'start' | 'live' | 'debrief'
|
||||
|
||||
function App() {
|
||||
const { state, error, transcripts, latency, start, stop } = useVoiceSession()
|
||||
const [view, setView] = useState<View>('start')
|
||||
const [acknowledged, setAcknowledged] = useState(false)
|
||||
|
||||
useEffect(() => {
|
||||
if (state === 'connected' && view === 'start') {
|
||||
setView('live')
|
||||
}
|
||||
if (state === 'idle' && view === 'live') {
|
||||
setView('debrief')
|
||||
}
|
||||
}, [state, view])
|
||||
|
||||
const handleStart = async () => {
|
||||
await start()
|
||||
}
|
||||
|
||||
const handleEnd = async () => {
|
||||
await stop()
|
||||
setView('debrief')
|
||||
}
|
||||
|
||||
const handleRestart = () => {
|
||||
setView('start')
|
||||
setAcknowledged(false)
|
||||
}
|
||||
|
||||
export default function App() {
|
||||
return (
|
||||
<Routes>
|
||||
<Route path="/" element={<VoiceSession />} />
|
||||
<Route path="/operator/login" element={<Login />} />
|
||||
<Route path="/operator/dashboard" element={<Dashboard />} />
|
||||
<Route path="*" element={<VoiceSession />} />
|
||||
</Routes>
|
||||
<section id="praxis-session">
|
||||
<header>
|
||||
<h1>Praxis</h1>
|
||||
<p className="subtitle">Customer Service role-play — v0.1</p>
|
||||
</header>
|
||||
|
||||
{view === 'start' && (
|
||||
<div className="view view--start">
|
||||
<div className="scenario-card">
|
||||
<h2>Angry customer requesting refund on a damaged product</h2>
|
||||
<p className="scenario-desc">
|
||||
You are a customer service agent. An angry customer (Jordan) is
|
||||
demanding a refund for a cracked product. Handle the
|
||||
conversation. You'll receive a coaching debrief at the end.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div className="disclaimer">
|
||||
<label className="disclaimer-check">
|
||||
<input
|
||||
type="checkbox"
|
||||
checked={acknowledged}
|
||||
onChange={(e) => setAcknowledged(e.target.checked)}
|
||||
/>
|
||||
<span>
|
||||
This is an AI practice session for training purposes. It is
|
||||
not a real conversation and no real company is involved.
|
||||
</span>
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<div className="controls">
|
||||
<button
|
||||
type="button"
|
||||
className="start"
|
||||
disabled={!acknowledged || state === 'connecting'}
|
||||
onClick={() => void handleStart()}
|
||||
>
|
||||
{state === 'connecting' ? 'Connecting…' : 'Start session'}
|
||||
</button>
|
||||
</div>
|
||||
|
||||
{error && <div className="error">{error}</div>}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{view === 'live' && (
|
||||
<div className="view view--live">
|
||||
<div className="status">
|
||||
<span className={`badge badge--${state}`}>{state}</span>
|
||||
{latency && (
|
||||
<span className="latency">
|
||||
<span className="latency-label">{latency.label}:</span>{' '}
|
||||
<span className={latency.e2eMs !== null && latency.e2eMs <= 600 ? 'ok' : 'over'}>
|
||||
{latency.e2eMs !== null ? `${latency.e2eMs.toFixed(0)} ms` : '—'}
|
||||
</span>
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
|
||||
<div className="controls">
|
||||
<button type="button" className="stop" onClick={() => void handleEnd()}>
|
||||
End session
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<div className="transcript">
|
||||
<h2>Live transcript</h2>
|
||||
{transcripts.length === 0 ? (
|
||||
<p className="muted">Speak to the AI customer…</p>
|
||||
) : (
|
||||
<ul>
|
||||
{transcripts.map((t, i) => (
|
||||
<li key={i} className={`turn turn--${t.role}`}>
|
||||
<span className="role">{t.role === 'user' ? 'You' : 'AI'}</span>
|
||||
<span className="text">{t.text}</span>
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{error && <div className="error">{error}</div>}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{view === 'debrief' && (
|
||||
<div className="view view--debrief">
|
||||
<h2>Session debrief</h2>
|
||||
<p className="muted">
|
||||
Your coaching debrief would appear here, generated from your turns
|
||||
+ the branch outcome. In a live run (with API keys), the debrief
|
||||
is spoken in the same voice as the role-play.
|
||||
</p>
|
||||
|
||||
{latency && (
|
||||
<div className="summary">
|
||||
<h3>Latency summary</h3>
|
||||
<p>
|
||||
{latency.label}:{' '}
|
||||
<span className={latency.e2eMs !== null && latency.e2eMs <= 600 ? 'ok' : 'over'}>
|
||||
{latency.e2eMs !== null ? `${latency.e2eMs.toFixed(0)} ms` : '—'}
|
||||
</span>
|
||||
{latency.e2eMs !== null && (
|
||||
<span className="budget">
|
||||
{' '}(budget 600ms — {latency.e2eMs <= 600 ? 'within' : 'over'})
|
||||
</span>
|
||||
)}
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{transcripts.length > 0 && (
|
||||
<div className="transcript">
|
||||
<h3>Turns this session</h3>
|
||||
<ul>
|
||||
{transcripts.map((t, i) => (
|
||||
<li key={i} className={`turn turn--${t.role}`}>
|
||||
<span className="role">{t.role === 'user' ? 'You' : 'AI'}</span>
|
||||
<span className="text">{t.text}</span>
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
</div>
|
||||
)}
|
||||
|
||||
<div className="controls">
|
||||
<button type="button" className="start" onClick={handleRestart}>
|
||||
Start a new session
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
</section>
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
export default App
|
||||
|
||||
@@ -1,177 +0,0 @@
|
||||
/**
|
||||
* Praxis v0.1 — voice session UX (extracted for React Router, SLICE-09 TASK-09-02).
|
||||
*
|
||||
* Three views: start → live → debrief. Reuses useVoiceSession. This is the
|
||||
* existing voice UI, now mounted at `/` and as the catch-all fallback.
|
||||
*/
|
||||
import { useVoiceSession } from './useVoiceSession'
|
||||
import { useEffect, useState } from 'react'
|
||||
import './App.css'
|
||||
|
||||
type View = 'start' | 'live' | 'debrief'
|
||||
|
||||
export default function VoiceSession() {
|
||||
const { state, error, transcripts, latency, start, stop } = useVoiceSession()
|
||||
const [view, setView] = useState<View>('start')
|
||||
const [acknowledged, setAcknowledged] = useState(false)
|
||||
|
||||
useEffect(() => {
|
||||
if (state === 'connected' && view === 'start') {
|
||||
setView('live')
|
||||
}
|
||||
if (state === 'idle' && view === 'live') {
|
||||
setView('debrief')
|
||||
}
|
||||
}, [state, view])
|
||||
|
||||
const handleStart = async () => {
|
||||
await start()
|
||||
}
|
||||
|
||||
const handleEnd = async () => {
|
||||
await stop()
|
||||
setView('debrief')
|
||||
}
|
||||
|
||||
const handleRestart = () => {
|
||||
setView('start')
|
||||
setAcknowledged(false)
|
||||
}
|
||||
|
||||
return (
|
||||
<section id="praxis-session">
|
||||
<header>
|
||||
<h1>Praxis</h1>
|
||||
<p className="subtitle">Customer Service role-play — v0.1</p>
|
||||
</header>
|
||||
|
||||
{view === 'start' && (
|
||||
<div className="view view--start">
|
||||
<div className="scenario-card">
|
||||
<h2>Angry customer requesting refund on a damaged product</h2>
|
||||
<p className="scenario-desc">
|
||||
You are a customer service agent. An angry customer (Jordan) is
|
||||
demanding a refund for a cracked product. Handle the
|
||||
conversation. You'll receive a coaching debrief at the end.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div className="disclaimer">
|
||||
<label className="disclaimer-check">
|
||||
<input
|
||||
type="checkbox"
|
||||
checked={acknowledged}
|
||||
onChange={(e) => setAcknowledged(e.target.checked)}
|
||||
/>
|
||||
<span>
|
||||
This is an AI practice session for training purposes. It is
|
||||
not a real conversation and no real company is involved.
|
||||
</span>
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<div className="controls">
|
||||
<button
|
||||
type="button"
|
||||
className="start"
|
||||
disabled={!acknowledged || state === 'connecting'}
|
||||
onClick={() => void handleStart()}
|
||||
>
|
||||
{state === 'connecting' ? 'Connecting…' : 'Start session'}
|
||||
</button>
|
||||
</div>
|
||||
|
||||
{error && <div className="error">{error}</div>}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{view === 'live' && (
|
||||
<div className="view view--live">
|
||||
<div className="status">
|
||||
<span className={`badge badge--${state}`}>{state}</span>
|
||||
{latency && (
|
||||
<span className="latency">
|
||||
<span className="latency-label">{latency.label}:</span>{' '}
|
||||
<span className={latency.e2eMs !== null && latency.e2eMs <= 600 ? 'ok' : 'over'}>
|
||||
{latency.e2eMs !== null ? `${latency.e2eMs.toFixed(0)} ms` : '—'}
|
||||
</span>
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
|
||||
<div className="controls">
|
||||
<button type="button" className="stop" onClick={() => void handleEnd()}>
|
||||
End session
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<div className="transcript">
|
||||
<h2>Live transcript</h2>
|
||||
{transcripts.length === 0 ? (
|
||||
<p className="muted">Speak to the AI customer…</p>
|
||||
) : (
|
||||
<ul>
|
||||
{transcripts.map((t, i) => (
|
||||
<li key={i} className={`turn turn--${t.role}`}>
|
||||
<span className="role">{t.role === 'user' ? 'You' : 'AI'}</span>
|
||||
<span className="text">{t.text}</span>
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{error && <div className="error">{error}</div>}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{view === 'debrief' && (
|
||||
<div className="view view--debrief">
|
||||
<h2>Session debrief</h2>
|
||||
<p className="muted">
|
||||
Your coaching debrief would appear here, generated from your turns
|
||||
+ the branch outcome. In a live run (with API keys), the debrief
|
||||
is spoken in the same voice as the role-play.
|
||||
</p>
|
||||
|
||||
{latency && (
|
||||
<div className="summary">
|
||||
<h3>Latency summary</h3>
|
||||
<p>
|
||||
{latency.label}:{' '}
|
||||
<span className={latency.e2eMs !== null && latency.e2eMs <= 600 ? 'ok' : 'over'}>
|
||||
{latency.e2eMs !== null ? `${latency.e2eMs.toFixed(0)} ms` : '—'}
|
||||
</span>
|
||||
{latency.e2eMs !== null && (
|
||||
<span className="budget">
|
||||
{' '}(budget 600ms — {latency.e2eMs <= 600 ? 'within' : 'over'})
|
||||
</span>
|
||||
)}
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{transcripts.length > 0 && (
|
||||
<div className="transcript">
|
||||
<h3>Turns this session</h3>
|
||||
<ul>
|
||||
{transcripts.map((t, i) => (
|
||||
<li key={i} className={`turn turn--${t.role}`}>
|
||||
<span className="role">{t.role === 'user' ? 'You' : 'AI'}</span>
|
||||
<span className="text">{t.text}</span>
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
</div>
|
||||
)}
|
||||
|
||||
<div className="controls">
|
||||
<button type="button" className="start" onClick={handleRestart}>
|
||||
Start a new session
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
</section>
|
||||
)
|
||||
}
|
||||
+2
-5
@@ -1,13 +1,10 @@
|
||||
import { StrictMode } from 'react'
|
||||
import { createRoot } from 'react-dom/client'
|
||||
import { BrowserRouter } from 'react-router-dom'
|
||||
import './index.css'
|
||||
import App from './App.tsx'
|
||||
|
||||
createRoot(document.getElementById('root')!).render(
|
||||
<StrictMode>
|
||||
<BrowserRouter>
|
||||
<App />
|
||||
</BrowserRouter>
|
||||
<App />
|
||||
</StrictMode>,
|
||||
)
|
||||
)
|
||||
|
||||
@@ -1,120 +0,0 @@
|
||||
/**
|
||||
* Operator Dashboard shell + auth gate (SLICE-09 TASK-09-04, D-057, D-053).
|
||||
*
|
||||
* On mount: GET /api/operator/me. 401 → redirect to /operator/login (UX-only
|
||||
* route guard — the server is the authority per D-057). 200 → render the
|
||||
* dashboard with operator name, 3 view tabs, freshness indicator, logout.
|
||||
*/
|
||||
import { useEffect, useState } from 'react'
|
||||
import { useNavigate } from 'react-router-dom'
|
||||
import PracticeVolume from './views/PracticeVolume'
|
||||
import MasteryProgression from './views/MasteryProgression'
|
||||
import FailurePatterns from './views/FailurePatterns'
|
||||
import '../App.css'
|
||||
|
||||
type Tab = 'practice' | 'mastery' | 'failure'
|
||||
|
||||
interface OperatorInfo {
|
||||
id: string
|
||||
username: string
|
||||
display_name: string | null
|
||||
role: string
|
||||
}
|
||||
|
||||
export default function Dashboard() {
|
||||
const [op, setOp] = useState<OperatorInfo | null>(null)
|
||||
const [tab, setTab] = useState<Tab>('practice')
|
||||
const [authed, setAuthed] = useState<boolean | null>(null)
|
||||
const navigate = useNavigate()
|
||||
|
||||
useEffect(() => {
|
||||
let cancelled = false
|
||||
;(async () => {
|
||||
try {
|
||||
const r = await fetch('/api/operator/me', { credentials: 'include' })
|
||||
if (cancelled) return
|
||||
if (r.status === 200) {
|
||||
const body = await r.json()
|
||||
setOp(body.operator)
|
||||
setAuthed(true)
|
||||
} else {
|
||||
setAuthed(false)
|
||||
navigate('/operator/login', { replace: true })
|
||||
}
|
||||
} catch {
|
||||
if (!cancelled) setAuthed(false)
|
||||
}
|
||||
})()
|
||||
return () => {
|
||||
cancelled = true
|
||||
}
|
||||
}, [navigate])
|
||||
|
||||
const handleLogout = async () => {
|
||||
try {
|
||||
await fetch('/api/operator/logout', {
|
||||
method: 'POST',
|
||||
credentials: 'include',
|
||||
})
|
||||
} catch {
|
||||
// best-effort — navigate to login regardless
|
||||
}
|
||||
navigate('/operator/login', { replace: true })
|
||||
}
|
||||
|
||||
if (authed === false) return null
|
||||
if (authed === null || !op) {
|
||||
return (
|
||||
<section id="praxis-dashboard">
|
||||
<p className="muted">Loading dashboard…</p>
|
||||
</section>
|
||||
)
|
||||
}
|
||||
|
||||
return (
|
||||
<section id="praxis-dashboard">
|
||||
<header>
|
||||
<h1>Praxis Operator Dashboard</h1>
|
||||
<p className="subtitle">
|
||||
Signed in as {op.display_name || op.username}
|
||||
</p>
|
||||
<div className="controls">
|
||||
<button type="button" className="stop" onClick={handleLogout}>
|
||||
Sign out
|
||||
</button>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<nav className="view-tabs" role="tablist">
|
||||
<button
|
||||
role="tab"
|
||||
aria-selected={tab === 'practice'}
|
||||
className={tab === 'practice' ? 'tab active' : 'tab'}
|
||||
onClick={() => setTab('practice')}
|
||||
>
|
||||
Practice Volume
|
||||
</button>
|
||||
<button
|
||||
role="tab"
|
||||
aria-selected={tab === 'mastery'}
|
||||
className={tab === 'mastery' ? 'tab active' : 'tab'}
|
||||
onClick={() => setTab('mastery')}
|
||||
>
|
||||
Mastery Progression
|
||||
</button>
|
||||
<button
|
||||
role="tab"
|
||||
aria-selected={tab === 'failure'}
|
||||
className={tab === 'failure' ? 'tab active' : 'tab'}
|
||||
onClick={() => setTab('failure')}
|
||||
>
|
||||
Failure Patterns
|
||||
</button>
|
||||
</nav>
|
||||
|
||||
{tab === 'practice' && <PracticeVolume />}
|
||||
{tab === 'mastery' && <MasteryProgression />}
|
||||
{tab === 'failure' && <FailurePatterns />}
|
||||
</section>
|
||||
)
|
||||
}
|
||||
@@ -1,93 +0,0 @@
|
||||
/**
|
||||
* Operator Login form (SLICE-09 TASK-09-03, D-041, D-057).
|
||||
*
|
||||
* POST /api/operator/login on submit. On success → navigate to
|
||||
* /operator/dashboard. On 401 → show error. On 429 → show rate-limit retry
|
||||
* message. Keyboard-accessible (label associations, focus management).
|
||||
*/
|
||||
import { useState, useRef, useEffect } from 'react'
|
||||
import { useNavigate } from 'react-router-dom'
|
||||
|
||||
export default function Login() {
|
||||
const [username, setUsername] = useState('')
|
||||
const [password, setPassword] = useState('')
|
||||
const [error, setError] = useState<string | null>(null)
|
||||
const [submitting, setSubmitting] = useState(false)
|
||||
const navigate = useNavigate()
|
||||
const userRef = useRef<HTMLInputElement>(null)
|
||||
|
||||
useEffect(() => {
|
||||
userRef.current?.focus()
|
||||
}, [])
|
||||
|
||||
const handleSubmit = async (e: React.FormEvent) => {
|
||||
e.preventDefault()
|
||||
setError(null)
|
||||
setSubmitting(true)
|
||||
try {
|
||||
const r = await fetch('/api/operator/login', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
credentials: 'include',
|
||||
body: JSON.stringify({ username, password }),
|
||||
})
|
||||
if (r.status === 200) {
|
||||
navigate('/operator/dashboard')
|
||||
return
|
||||
}
|
||||
if (r.status === 401) {
|
||||
setError('Invalid username or password.')
|
||||
} else if (r.status === 429) {
|
||||
setError('Too many attempts. Try again in a minute.')
|
||||
} else if (r.status === 503) {
|
||||
setError('Operator sign-in is unavailable right now.')
|
||||
} else {
|
||||
setError(`Login failed (HTTP ${r.status}).`)
|
||||
}
|
||||
} catch (err) {
|
||||
setError('Network error — unable to reach the server.')
|
||||
} finally {
|
||||
setSubmitting(false)
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<section id="praxis-login">
|
||||
<header>
|
||||
<h1>Praxis Operator</h1>
|
||||
<p className="subtitle">Sign in to view the cohort dashboard</p>
|
||||
</header>
|
||||
|
||||
<form className="login-form" onSubmit={handleSubmit}>
|
||||
<label htmlFor="login-username">Username</label>
|
||||
<input
|
||||
id="login-username"
|
||||
ref={userRef}
|
||||
type="text"
|
||||
autoComplete="username"
|
||||
value={username}
|
||||
onChange={(e) => setUsername(e.target.value)}
|
||||
required
|
||||
disabled={submitting}
|
||||
/>
|
||||
|
||||
<label htmlFor="login-password">Password</label>
|
||||
<input
|
||||
id="login-password"
|
||||
type="password"
|
||||
autoComplete="current-password"
|
||||
value={password}
|
||||
onChange={(e) => setPassword(e.target.value)}
|
||||
required
|
||||
disabled={submitting}
|
||||
/>
|
||||
|
||||
<button type="submit" className="start" disabled={submitting}>
|
||||
{submitting ? 'Signing in…' : 'Sign in'}
|
||||
</button>
|
||||
|
||||
{error && <div className="error" role="alert">{error}</div>}
|
||||
</form>
|
||||
</section>
|
||||
)
|
||||
}
|
||||
@@ -1,49 +0,0 @@
|
||||
/**
|
||||
* Inline SVG sparkline (SLICE-09 TASK-09-05, RESEARCH-v0.4 §4.3).
|
||||
*
|
||||
* Zero-dep ~50 LOC. Renders a polyline from `data`. Handles empty (renders
|
||||
* nothing), single point (dot), all-same (flat line). stroke=currentColor.
|
||||
* No axes/tooltips — sparklines are compact trend indicators.
|
||||
*/
|
||||
interface SparklineProps {
|
||||
data: number[]
|
||||
width?: number
|
||||
height?: number
|
||||
}
|
||||
|
||||
export default function Sparkline({ data, width = 60, height = 20 }: SparklineProps) {
|
||||
if (!data || data.length === 0) {
|
||||
return null
|
||||
}
|
||||
if (data.length === 1) {
|
||||
return (
|
||||
<svg width={width} height={height} viewBox={`0 0 ${width} ${height}`} aria-hidden="true">
|
||||
<circle cx={width / 2} cy={height / 2} r={1.5} fill="currentColor" />
|
||||
</svg>
|
||||
)
|
||||
}
|
||||
const min = Math.min(...data)
|
||||
const max = Math.max(...data)
|
||||
const span = max - min || 1
|
||||
const pad = 2
|
||||
const w = width - pad * 2
|
||||
const h = height - pad * 2
|
||||
const stepX = w / (data.length - 1)
|
||||
const points = data.map((v, i) => {
|
||||
const x = pad + i * stepX
|
||||
const y = pad + h - ((v - min) / span) * h
|
||||
return `${x.toFixed(2)},${y.toFixed(2)}`
|
||||
})
|
||||
return (
|
||||
<svg width={width} height={height} viewBox={`0 0 ${width} ${height}`} aria-hidden="true">
|
||||
<polyline
|
||||
points={points.join(' ')}
|
||||
fill="none"
|
||||
stroke="currentColor"
|
||||
strokeWidth={1.25}
|
||||
strokeLinejoin="round"
|
||||
strokeLinecap="round"
|
||||
/>
|
||||
</svg>
|
||||
)
|
||||
}
|
||||
@@ -1,193 +0,0 @@
|
||||
/**
|
||||
* Operator dashboard unit tests (SLICE-09 TASK-09-07).
|
||||
*
|
||||
* Covers: auth gate (401 on /me → redirect to /operator/login), login form
|
||||
* (submit → POST /login → navigate to dashboard), suppressed cell display
|
||||
* ("— (<10 learners)"), sparkline renders SVG polyline, freshness indicator,
|
||||
* no PII in rendered DOM.
|
||||
*/
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'
|
||||
import { render, screen, waitFor, fireEvent } from '@testing-library/react'
|
||||
import { MemoryRouter, Routes, Route } from 'react-router-dom'
|
||||
import Login from '../Login'
|
||||
import Dashboard from '../Dashboard'
|
||||
import Sparkline from '../Sparkline'
|
||||
import { suppressedLabel, formatFreshness } from '../views/_viewCommon'
|
||||
import type { Cell } from '../views/_viewCommon'
|
||||
|
||||
function renderAt(path: string) {
|
||||
return render(
|
||||
<MemoryRouter initialEntries={[path]}>
|
||||
<Routes>
|
||||
<Route path="/operator/login" element={<Login />} />
|
||||
<Route path="/operator/dashboard" element={<Dashboard />} />
|
||||
<Route path="*" element={<div data-testid="fallback" />} />
|
||||
</Routes>
|
||||
</MemoryRouter>,
|
||||
)
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
vi.stubGlobal('fetch', vi.fn())
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
vi.unstubAllGlobals()
|
||||
vi.restoreAllMocks()
|
||||
})
|
||||
|
||||
// ── Auth gate ──────────────────────────────────────────────────────────────
|
||||
|
||||
describe('Dashboard auth gate', () => {
|
||||
it('redirects to /operator/login on 401 from /me', async () => {
|
||||
;(global.fetch as any).mockResolvedValue({ status: 401 })
|
||||
renderAt('/operator/dashboard')
|
||||
await waitFor(() => {
|
||||
expect(screen.queryByText(/Praxis Operator Dashboard/i)).not.toBeInTheDocument()
|
||||
})
|
||||
})
|
||||
|
||||
it('renders dashboard on 200 from /me', async () => {
|
||||
;(global.fetch as any).mockResolvedValue({
|
||||
status: 200,
|
||||
json: async () => ({ operator: { id: '1', username: 'alice', display_name: 'Alice', role: 'operator' } }),
|
||||
})
|
||||
renderAt('/operator/dashboard')
|
||||
await waitFor(() => {
|
||||
expect(screen.getByText(/Praxis Operator Dashboard/i)).toBeInTheDocument()
|
||||
expect(screen.getByText(/Signed in as Alice/i)).toBeInTheDocument()
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
// ── Login form ────────────────────────────────────────────────────────────
|
||||
|
||||
describe('Login form', () => {
|
||||
it('renders username + password fields + submit', () => {
|
||||
renderAt('/operator/login')
|
||||
expect(screen.getByLabelText(/Username/i)).toBeInTheDocument()
|
||||
expect(screen.getByLabelText(/Password/i)).toBeInTheDocument()
|
||||
expect(screen.getByRole('button', { name: /Sign in/i })).toBeInTheDocument()
|
||||
})
|
||||
|
||||
it('submits POST /api/operator/login and navigates on success', async () => {
|
||||
;(global.fetch as any).mockResolvedValue({ status: 200 })
|
||||
renderAt('/operator/login')
|
||||
fireEvent.change(screen.getByLabelText(/Username/i), { target: { value: 'alice' } })
|
||||
fireEvent.change(screen.getByLabelText(/Password/i), { target: { value: 'pw' } })
|
||||
fireEvent.click(screen.getByRole('button', { name: /Sign in/i }))
|
||||
await waitFor(() => {
|
||||
expect(global.fetch).toHaveBeenCalledWith(
|
||||
'/api/operator/login',
|
||||
expect.objectContaining({ method: 'POST' }),
|
||||
)
|
||||
})
|
||||
})
|
||||
|
||||
it('shows error on 401', async () => {
|
||||
;(global.fetch as any).mockResolvedValue({ status: 401 })
|
||||
renderAt('/operator/login')
|
||||
fireEvent.change(screen.getByLabelText(/Username/i), { target: { value: 'a' } })
|
||||
fireEvent.change(screen.getByLabelText(/Password/i), { target: { value: 'b' } })
|
||||
fireEvent.click(screen.getByRole('button', { name: /Sign in/i }))
|
||||
await waitFor(() => {
|
||||
expect(screen.getByText(/Invalid username or password/i)).toBeInTheDocument()
|
||||
})
|
||||
})
|
||||
|
||||
it('shows rate-limit message on 429', async () => {
|
||||
;(global.fetch as any).mockResolvedValue({ status: 429 })
|
||||
renderAt('/operator/login')
|
||||
fireEvent.change(screen.getByLabelText(/Username/i), { target: { value: 'a' } })
|
||||
fireEvent.change(screen.getByLabelText(/Password/i), { target: { value: 'b' } })
|
||||
fireEvent.click(screen.getByRole('button', { name: /Sign in/i }))
|
||||
await waitFor(() => {
|
||||
expect(screen.getByText(/Too many attempts/i)).toBeInTheDocument()
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
// ── Sparkline ─────────────────────────────────────────────────────────────
|
||||
|
||||
describe('Sparkline', () => {
|
||||
it('renders nothing for empty data', () => {
|
||||
const { container } = render(<Sparkline data={[]} />)
|
||||
expect(container.querySelector('svg')).toBeNull()
|
||||
})
|
||||
|
||||
it('renders a dot for single point', () => {
|
||||
const { container } = render(<Sparkline data={[5]} />)
|
||||
expect(container.querySelector('circle')).not.toBeNull()
|
||||
})
|
||||
|
||||
it('renders a polyline for multiple points', () => {
|
||||
const { container } = render(<Sparkline data={[1, 2, 3, 4, 5]} />)
|
||||
const poly = container.querySelector('polyline')
|
||||
expect(poly).not.toBeNull()
|
||||
expect(poly?.getAttribute('points')).toBeTruthy()
|
||||
})
|
||||
|
||||
it('renders a flat line for all-same values', () => {
|
||||
const { container } = render(<Sparkline data={[3, 3, 3, 3]} />)
|
||||
expect(container.querySelector('polyline')).not.toBeNull()
|
||||
})
|
||||
})
|
||||
|
||||
// ── Suppressed cell display + freshness ──────────────────────────────────
|
||||
|
||||
describe('suppressedLabel', () => {
|
||||
it('shows "— (<10 learners)" for suppressed cells', () => {
|
||||
const cell: Cell = {
|
||||
metric: 'sessions_count', window_start: null, window_end: null,
|
||||
value: null, cell_count: 5, cell_suppressed: true, updated_at: null,
|
||||
}
|
||||
expect(suppressedLabel(cell)).toBe('— (<10 learners)')
|
||||
})
|
||||
|
||||
it('shows the value for non-suppressed cells', () => {
|
||||
const cell: Cell = {
|
||||
metric: 'sessions_count', window_start: null, window_end: null,
|
||||
value: 12, cell_count: 12, cell_suppressed: false, updated_at: null,
|
||||
}
|
||||
expect(suppressedLabel(cell)).toBe('12')
|
||||
})
|
||||
})
|
||||
|
||||
describe('formatFreshness', () => {
|
||||
it('shows — for null lastUpdated', () => {
|
||||
expect(formatFreshness(null)).toBe('—')
|
||||
})
|
||||
|
||||
it('shows minutes ago for < 1h', () => {
|
||||
const thirtyMinAgo = new Date(Date.now() - 30 * 60_000).toISOString()
|
||||
expect(formatFreshness(thirtyMinAgo)).toMatch(/m ago/)
|
||||
})
|
||||
|
||||
it('shows hours ago for 1-24h', () => {
|
||||
const twoHoursAgo = new Date(Date.now() - 2 * 3_600_000).toISOString()
|
||||
expect(formatFreshness(twoHoursAgo)).toMatch(/h ago/)
|
||||
})
|
||||
|
||||
it('shows days ago for > 24h', () => {
|
||||
const twoDaysAgo = new Date(Date.now() - 48 * 3_600_000).toISOString()
|
||||
expect(formatFreshness(twoDaysAgo)).toMatch(/d ago/)
|
||||
})
|
||||
})
|
||||
|
||||
// ── No PII in rendered DOM ────────────────────────────────────────────────
|
||||
|
||||
describe('No PII in dashboard DOM', () => {
|
||||
it('does not render learner_ref fields', async () => {
|
||||
;(global.fetch as any).mockResolvedValue({
|
||||
status: 200,
|
||||
json: async () => ({ operator: { id: '1', username: 'alice', display_name: 'Alice', role: 'operator' } }),
|
||||
})
|
||||
const { container } = renderAt('/operator/dashboard')
|
||||
await waitFor(() => {
|
||||
expect(screen.getByText(/Praxis Operator Dashboard/i)).toBeInTheDocument()
|
||||
})
|
||||
// No learner-ref label or per-learner data should appear in the dashboard shell.
|
||||
expect(container.textContent).not.toMatch(/learner_ref/i)
|
||||
expect(container.textContent).not.toMatch(/learner-1/i)
|
||||
})
|
||||
})
|
||||
@@ -1,94 +0,0 @@
|
||||
/**
|
||||
* Failure Patterns view (SLICE-09 TASK-09-06, D-053, REQ-NFR-DASH-01).
|
||||
*
|
||||
* Top failure_modes by frequency (sorted table), rubric criteria with
|
||||
* mean < 3.0 (highlighted weak-spots), branch outcome distribution.
|
||||
* Suppressed cells → "— (<10 learners)".
|
||||
*/
|
||||
import { useEffect, useState } from 'react'
|
||||
import { fetchView, formatFreshness, suppressedLabel } from './_viewCommon'
|
||||
import type { ViewResponse } from './_viewCommon'
|
||||
|
||||
export default function FailurePatterns() {
|
||||
const [data, setData] = useState<ViewResponse | null>(null)
|
||||
const [error, setError] = useState<string | null>(null)
|
||||
const [loading, setLoading] = useState(true)
|
||||
|
||||
useEffect(() => {
|
||||
let cancelled = false
|
||||
;(async () => {
|
||||
try {
|
||||
const r = await fetchView('/api/operator/failure-patterns')
|
||||
if (!cancelled) setData(r)
|
||||
} catch (e) {
|
||||
if (!cancelled) setError(String(e))
|
||||
} finally {
|
||||
if (!cancelled) setLoading(false)
|
||||
}
|
||||
})()
|
||||
return () => {
|
||||
cancelled = true
|
||||
}
|
||||
}, [])
|
||||
|
||||
if (loading) return <p className="muted">Loading failure patterns…</p>
|
||||
if (error) return <div className="error">Failed to load: {error}</div>
|
||||
if (!data || data.views.length === 0) {
|
||||
return (
|
||||
<div className="view view--failure">
|
||||
<p className="muted">No failure-pattern data available yet.</p>
|
||||
<p className="muted">Last updated: {formatFreshness(data?.last_updated ?? null)}</p>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="view view--failure">
|
||||
<p className="muted">Last updated: {formatFreshness(data.last_updated)}</p>
|
||||
{data.views.map((v) => {
|
||||
const modes = v.metrics
|
||||
.filter((c) => c.metric.startsWith('failure_mode:'))
|
||||
.sort((a, b) => (b.value ?? 0) - (a.value ?? 0))
|
||||
const branches = v.metrics.filter((c) => c.metric.startsWith('branch:'))
|
||||
return (
|
||||
<div key={v.path} className="cohort-section">
|
||||
<h3>{v.path}</h3>
|
||||
<h4>Failure modes by frequency</h4>
|
||||
<table className="cohort-table">
|
||||
<thead><tr><th>Mode</th><th>Frequency</th></tr></thead>
|
||||
<tbody>
|
||||
{modes.length === 0 ? (
|
||||
<tr><td colSpan={2} className="muted">No failure modes recorded.</td></tr>
|
||||
) : (
|
||||
modes.map((c) => (
|
||||
<tr key={c.metric}>
|
||||
<td>{c.metric.replace('failure_mode:', '')}</td>
|
||||
<td>{suppressedLabel(c)}</td>
|
||||
</tr>
|
||||
))
|
||||
)}
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h4>Branch outcome distribution</h4>
|
||||
<table className="cohort-table">
|
||||
<thead><tr><th>Branch</th><th>Count</th></tr></thead>
|
||||
<tbody>
|
||||
{branches.length === 0 ? (
|
||||
<tr><td colSpan={2} className="muted">No branch data recorded.</td></tr>
|
||||
) : (
|
||||
branches.map((c) => (
|
||||
<tr key={c.metric}>
|
||||
<td>{c.metric.replace('branch:', '')}</td>
|
||||
<td>{suppressedLabel(c)}</td>
|
||||
</tr>
|
||||
))
|
||||
)}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
)
|
||||
})}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -1,84 +0,0 @@
|
||||
/**
|
||||
* Mastery Progression view (SLICE-09 TASK-09-06, D-053, REQ-NFR-DASH-01).
|
||||
*
|
||||
* Gate-open rate, median mastery score, rubric criterion means (table +
|
||||
* sparkline). Suppressed cells → "— (<10 learners)".
|
||||
*/
|
||||
import { useEffect, useState } from 'react'
|
||||
import Sparkline from '../Sparkline'
|
||||
import { fetchView, formatFreshness, suppressedLabel, valuesForSparkline } from './_viewCommon'
|
||||
import type { ViewResponse } from './_viewCommon'
|
||||
|
||||
export default function MasteryProgression() {
|
||||
const [data, setData] = useState<ViewResponse | null>(null)
|
||||
const [error, setError] = useState<string | null>(null)
|
||||
const [loading, setLoading] = useState(true)
|
||||
|
||||
useEffect(() => {
|
||||
let cancelled = false
|
||||
;(async () => {
|
||||
try {
|
||||
const r = await fetchView('/api/operator/mastery')
|
||||
if (!cancelled) setData(r)
|
||||
} catch (e) {
|
||||
if (!cancelled) setError(String(e))
|
||||
} finally {
|
||||
if (!cancelled) setLoading(false)
|
||||
}
|
||||
})()
|
||||
return () => {
|
||||
cancelled = true
|
||||
}
|
||||
}, [])
|
||||
|
||||
if (loading) return <p className="muted">Loading mastery progression…</p>
|
||||
if (error) return <div className="error">Failed to load: {error}</div>
|
||||
if (!data || data.views.length === 0) {
|
||||
return (
|
||||
<div className="view view--mastery">
|
||||
<p className="muted">No mastery data available yet.</p>
|
||||
<p className="muted">Last updated: {formatFreshness(data?.last_updated ?? null)}</p>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="view view--mastery">
|
||||
<p className="muted">Last updated: {formatFreshness(data.last_updated)}</p>
|
||||
{data.views.map((v) => {
|
||||
const gate = v.metrics.find((c) => c.metric === 'gate_open_rate')
|
||||
const median = v.metrics.find((c) => c.metric === 'median_mastery_score')
|
||||
const critMeans = v.metrics.filter((c) => c.metric.startsWith('rubric_criterion_mean:'))
|
||||
return (
|
||||
<div key={v.path} className="cohort-section">
|
||||
<h3>{v.path}</h3>
|
||||
<table className="cohort-table">
|
||||
<thead>
|
||||
<tr><th>Metric</th><th>Value</th><th>Trend</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>Gate-open rate</td>
|
||||
<td>{gate ? suppressedLabel(gate) : '—'}</td>
|
||||
<td><Sparkline data={valuesForSparkline(v.metrics, 'gate_open_rate')} /></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Median mastery score</td>
|
||||
<td>{median ? suppressedLabel(median) : '—'}</td>
|
||||
<td><Sparkline data={valuesForSparkline(v.metrics, 'median_mastery_score')} /></td>
|
||||
</tr>
|
||||
{critMeans.map((c) => (
|
||||
<tr key={c.metric}>
|
||||
<td>{c.metric.replace('rubric_criterion_mean:', '')}</td>
|
||||
<td>{suppressedLabel(c)}</td>
|
||||
<td><Sparkline data={valuesForSparkline(v.metrics, c.metric)} /></td>
|
||||
</tr>
|
||||
))}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
)
|
||||
})}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -1,81 +0,0 @@
|
||||
/**
|
||||
* Practice Volume view (SLICE-09 TASK-09-06, D-053, REQ-NFR-DASH-01).
|
||||
*
|
||||
* Read-only table of sessions/day per path + active learners, with sparklines.
|
||||
* Suppressed cells → "— (<10 learners)". No per-learner drill-down (R-DASH-02).
|
||||
*/
|
||||
import { useEffect, useState } from 'react'
|
||||
import Sparkline from '../Sparkline'
|
||||
import { fetchView, formatFreshness, suppressedLabel, valuesForSparkline } from './_viewCommon'
|
||||
import type { Cell, ViewResponse } from './_viewCommon'
|
||||
|
||||
const SUPPRESSED_PLACEHOLDER: Cell = {
|
||||
metric: '', window_start: null, window_end: null,
|
||||
value: null, cell_count: 0, cell_suppressed: true, updated_at: null,
|
||||
}
|
||||
|
||||
export default function PracticeVolume() {
|
||||
const [data, setData] = useState<ViewResponse | null>(null)
|
||||
const [error, setError] = useState<string | null>(null)
|
||||
const [loading, setLoading] = useState(true)
|
||||
|
||||
useEffect(() => {
|
||||
let cancelled = false
|
||||
;(async () => {
|
||||
try {
|
||||
const r = await fetchView('/api/operator/cohort')
|
||||
if (!cancelled) setData(r)
|
||||
} catch (e) {
|
||||
if (!cancelled) setError(String(e))
|
||||
} finally {
|
||||
if (!cancelled) setLoading(false)
|
||||
}
|
||||
})()
|
||||
return () => {
|
||||
cancelled = true
|
||||
}
|
||||
}, [])
|
||||
|
||||
if (loading) return <p className="muted">Loading practice volume…</p>
|
||||
if (error) return <div className="error">Failed to load: {error}</div>
|
||||
if (!data || data.views.length === 0) {
|
||||
return (
|
||||
<div className="view view--practice">
|
||||
<p className="muted">No practice data available yet.</p>
|
||||
<p className="muted">Last updated: {formatFreshness(data?.last_updated ?? null)}</p>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="view view--practice">
|
||||
<p className="muted">Last updated: {formatFreshness(data.last_updated)}</p>
|
||||
<table className="cohort-table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Path</th>
|
||||
<th>Sessions (trend)</th>
|
||||
<th>Active learners</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{data.views.map((v) => {
|
||||
const sessions = v.metrics.filter((c) => c.metric === 'sessions_count')
|
||||
const active = v.metrics.find((c) => c.metric === 'active_learners_count')
|
||||
return (
|
||||
<tr key={v.path}>
|
||||
<td>{v.path}</td>
|
||||
<td>
|
||||
{suppressedLabel(sessions[sessions.length - 1] ?? SUPPRESSED_PLACEHOLDER)}
|
||||
{' '}
|
||||
<Sparkline data={valuesForSparkline(v.metrics, 'sessions_count')} />
|
||||
</td>
|
||||
<td>{active ? suppressedLabel(active) : '—'}</td>
|
||||
</tr>
|
||||
)
|
||||
})}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -1,60 +0,0 @@
|
||||
/**
|
||||
* Shared types + helpers for operator dashboard views (SLICE-09 TASK-09-06).
|
||||
*/
|
||||
|
||||
export interface Cell {
|
||||
metric: string
|
||||
window_start: string | null
|
||||
window_end: string | null
|
||||
value: number | null
|
||||
cell_count: number
|
||||
cell_suppressed: boolean
|
||||
updated_at: string | null
|
||||
}
|
||||
|
||||
export interface PathView {
|
||||
path: string
|
||||
metrics: Cell[]
|
||||
}
|
||||
|
||||
export interface ViewResponse {
|
||||
views: PathView[]
|
||||
last_updated: string | null
|
||||
}
|
||||
|
||||
export async function fetchView(endpoint: string): Promise<ViewResponse> {
|
||||
const r = await fetch(endpoint, { credentials: 'include' })
|
||||
if (!r.ok) {
|
||||
throw new Error(`HTTP ${r.status}`)
|
||||
}
|
||||
return (await r.json()) as ViewResponse
|
||||
}
|
||||
|
||||
export function formatFreshness(lastUpdated: string | null): string {
|
||||
if (!lastUpdated) return '—'
|
||||
const ts = Date.parse(lastUpdated)
|
||||
if (Number.isNaN(ts)) return '—'
|
||||
const hoursAgo = (Date.now() - ts) / 3_600_000
|
||||
if (hoursAgo < 1) return `${Math.round(hoursAgo * 60)}m ago`
|
||||
if (hoursAgo < 24) return `${hoursAgo.toFixed(1)}h ago`
|
||||
return `${(hoursAgo / 24).toFixed(1)}d ago`
|
||||
}
|
||||
|
||||
export function suppressedLabel(cell: Cell): string {
|
||||
return cell.cell_suppressed ? '— (<10 learners)' : String(cell.value ?? '—')
|
||||
}
|
||||
|
||||
export function groupMetricsByPath(views: PathView[]): Map<string, Cell[]> {
|
||||
const m = new Map<string, Cell[]>()
|
||||
for (const v of views) {
|
||||
m.set(v.path, v.metrics)
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
export function valuesForSparkline(cells: Cell[] | undefined, metric: string): number[] {
|
||||
if (!cells) return []
|
||||
return cells
|
||||
.filter((c) => c.metric === metric && c.value !== null)
|
||||
.map((c) => c.value as number)
|
||||
}
|
||||
@@ -1 +0,0 @@
|
||||
import '@testing-library/jest-dom/vitest'
|
||||
@@ -1,13 +0,0 @@
|
||||
/// <reference types="vitest" />
|
||||
import { defineConfig } from 'vite'
|
||||
import react from '@vitejs/plugin-react'
|
||||
|
||||
export default defineConfig({
|
||||
plugins: [react()],
|
||||
test: {
|
||||
environment: 'jsdom',
|
||||
globals: true,
|
||||
setupFiles: ['./src/test-setup.ts'],
|
||||
include: ['src/**/*.test.{ts,tsx}'],
|
||||
},
|
||||
})
|
||||
+9
-50
@@ -31,6 +31,7 @@ except ImportError: # pragma: no cover
|
||||
from fastapi import FastAPI, HTTPException, Request
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from fastapi.responses import JSONResponse
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
from pipecat.transports.smallwebrtc.connection import SmallWebRTCConnection
|
||||
from slowapi.errors import RateLimitExceeded
|
||||
from slowapi import _rate_limit_exceeded_handler
|
||||
@@ -41,19 +42,11 @@ from db.store import PraxisStore
|
||||
from server.auth.cookies import get_session_middleware_kwargs
|
||||
from server.auth.rate_limit import limiter
|
||||
from server.auth.routes import router as auth_router
|
||||
from server.cohort.nightly import NightlyScheduler
|
||||
from server.operator.cohort import router as cohort_router
|
||||
from server.operator.credentials import router as credentials_router
|
||||
from server.operator.failure_patterns import router as failure_router
|
||||
from server.operator.mastery import router as mastery_router
|
||||
from server.pipeline import build_pipeline
|
||||
from server.vc.issuer_keys import _load_root_key
|
||||
from server.vc.migrate_keys import migrate_issuer_keys
|
||||
from server.vc.verification import verify_credential
|
||||
from starlette.middleware.sessions import SessionMiddleware
|
||||
from starlette.responses import FileResponse
|
||||
from starlette.staticfiles import StaticFiles
|
||||
from starlette.exceptions import HTTPException as StarletteHTTPException
|
||||
|
||||
_store = PraxisStore()
|
||||
|
||||
@@ -99,8 +92,6 @@ async def lifespan(app: FastAPI):
|
||||
)
|
||||
app.state.pg_pool = pool
|
||||
app.state.pg_store = PgStore(pool)
|
||||
nightly = NightlyScheduler()
|
||||
app.state.nightly_scheduler = nightly
|
||||
try:
|
||||
applied = await apply_pg_migrations(pool)
|
||||
if applied:
|
||||
@@ -111,14 +102,10 @@ async def lifespan(app: FastAPI):
|
||||
# first boot, idempotent. Non-fatal on failure (v0.3 SQLite path
|
||||
# remains intact for verification).
|
||||
await _maybe_migrate_issuer_keys()
|
||||
# v0.4 P2 (D-054, REQ-NFR-DASH-02): start the nightly reconciliation
|
||||
# scheduler at 03:00 CT. Cancelled on shutdown.
|
||||
await nightly.start(app.state.pg_store)
|
||||
logger.info("Nightly cohort reconciliation scheduler started (03:00 CT)")
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
await nightly.stop()
|
||||
pass
|
||||
finally:
|
||||
await pool.close()
|
||||
logger.info("Postgres pool closed")
|
||||
@@ -258,44 +245,16 @@ async def _maybe_migrate_issuer_keys() -> None:
|
||||
# the router (routes-before-static-mount constraint, carry-forward v0.2).
|
||||
app.include_router(auth_router)
|
||||
|
||||
# ── Operator API cohort endpoints (TASK-10-02, D-053, D-057) ──────────
|
||||
# Auth-gated via Depends(current_operator) inside each router. Mounted
|
||||
# BEFORE the SPA StaticFiles fallback so /api/operator/* is matched by the
|
||||
# API routers, not the SPA fallback.
|
||||
app.include_router(cohort_router)
|
||||
app.include_router(mastery_router)
|
||||
app.include_router(failure_router)
|
||||
app.include_router(credentials_router)
|
||||
|
||||
|
||||
# ── SPA StaticFiles fallback (G-041 binding, TASK-10-01, R-DASH-03/05) ─
|
||||
# Custom StaticFiles subclass that returns index.html for non-file paths
|
||||
# (SPA client-side routing). G-041 OVERRIDES the plan's catch-all route —
|
||||
# a @app.get("/{path:path}") catch-all before StaticFiles would shadow
|
||||
# asset serving (assertion 8 in TASK-10-04). This subclass serves assets
|
||||
# normally (JS/CSS) and falls back to index.html for client-side routes
|
||||
# (/operator/dashboard, /operator/login). API routes registered above take
|
||||
# precedence over the mount.
|
||||
class SpaStaticFiles(StaticFiles):
|
||||
async def get_response(self, path: str, scope):
|
||||
try:
|
||||
return await super().get_response(path, scope)
|
||||
except (StarletteHTTPException, HTTPException) as e:
|
||||
if getattr(e, "status_code", None) == 404:
|
||||
import os
|
||||
index = os.path.join(self.directory, "index.html")
|
||||
if os.path.isfile(index):
|
||||
return FileResponse(index)
|
||||
raise
|
||||
|
||||
|
||||
# Mount client/dist at "/" AFTER all API routes so they take precedence.
|
||||
# html=True serves index.html for "/" (SPA root). The SpaStaticFiles
|
||||
# subclass serves index.html for unknown paths (React Router routes).
|
||||
# ── Static client serving (D-023, REQ-DEPLOY-13) ──────────────────────
|
||||
# Mount client/dist as StaticFiles at "/" AFTER all API routes so they
|
||||
# take precedence. html=True serves index.html for "/" (SPA root).
|
||||
# The client has no React Router (single-view state machine: start→live
|
||||
# →debrief), so no SPA fallback fallback route is needed per RESEARCH.md Q3.
|
||||
_CLIENT_DIST = _env("PRAXIS_CLIENT_DIST", "client/dist")
|
||||
if os.path.isdir(_CLIENT_DIST):
|
||||
app.mount("/", SpaStaticFiles(directory=_CLIENT_DIST, html=True), name="spa")
|
||||
logger.info(f"Serving client from {_CLIENT_DIST} (SPA fallback enabled)")
|
||||
app.mount("/", StaticFiles(directory=_CLIENT_DIST, html=True), name="client")
|
||||
logger.info(f"Serving client from {_CLIENT_DIST}")
|
||||
else:
|
||||
logger.warning(f"Client dist not found at {_CLIENT_DIST} — API-only mode")
|
||||
|
||||
|
||||
@@ -1,230 +0,0 @@
|
||||
"""Cohort aggregation logic + k-anonymity suppression (TASK-07-01, D-034, D-045).
|
||||
|
||||
Computes k-anonymized aggregates for the affected (path, metric, window_start)
|
||||
bins and upserts them to cohort_aggregates via PgStore. Suppression is at
|
||||
write time (auditable — RESEARCH-v0.4 §3.1): COUNT(DISTINCT learner_ref) < 10
|
||||
=> cell_suppressed=TRUE, value=NULL.
|
||||
|
||||
Metrics computed (per 7-day rolling window, per path):
|
||||
sessions_count, active_learners_count, gate_open_rate,
|
||||
median_mastery_score, failure_mode_frequency,
|
||||
rubric_criterion_means, week_distribution.
|
||||
|
||||
The session_outcome dict contains: learner_ref (opaque — D-031), path,
|
||||
scenario_id, outcome (pass/fail), rubric_scores, failure_mode, branch_path,
|
||||
timestamp.
|
||||
|
||||
No raw learner PII in Postgres (D-031): only aggregates + opaque learner_ref
|
||||
for distinct counting.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as _dt
|
||||
import logging
|
||||
import statistics
|
||||
from typing import Any
|
||||
|
||||
from db.pg_store import PgStore
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
K_ANON_THRESHOLD = 10
|
||||
|
||||
|
||||
def _rolling_window(now: _dt.datetime | None = None) -> tuple[_dt.date, _dt.date]:
|
||||
"""Return the 7-day rolling window (start, end) for `now`.
|
||||
|
||||
window_start = today - 6 days, window_end = today (inclusive 7-day span).
|
||||
"""
|
||||
today = (now or _dt.datetime.now(_dt.timezone.utc)).date()
|
||||
return today - _dt.timedelta(days=6), today
|
||||
|
||||
|
||||
def _distinct_learners(sessions: list[dict[str, Any]]) -> int:
|
||||
return len({s["learner_ref"] for s in sessions if s.get("learner_ref")})
|
||||
|
||||
|
||||
async def aggregate_session(pg_store: PgStore, session_outcome: dict[str, Any]) -> None:
|
||||
"""Compute + upsert k-anonymized aggregates for one session outcome.
|
||||
|
||||
Reads the affected path's recent session set (from cohort_aggregates or
|
||||
an in-memory accumulator), recomputes the metric cells for the 7-day
|
||||
window, applies k-anon suppression, and upserts each cell idempotently.
|
||||
|
||||
Idempotent (ON CONFLICT upsert) — re-running with the same outcome
|
||||
produces the same aggregate. The caller (hook.py) passes one session at
|
||||
a time; the nightly job (nightly.py) recomputes the full window.
|
||||
"""
|
||||
path = session_outcome.get("path") or session_outcome.get("path_id") or "unknown"
|
||||
learner_ref = session_outcome.get("learner_ref") or "unknown"
|
||||
outcome = session_outcome.get("outcome", "fail")
|
||||
rubric_scores = session_outcome.get("rubric_scores") or []
|
||||
failure_mode = session_outcome.get("failure_mode")
|
||||
branch_path = session_outcome.get("branch_path") or []
|
||||
scenario_id = session_outcome.get("scenario_id")
|
||||
ts = session_outcome.get("timestamp")
|
||||
|
||||
window_start, window_end = _rolling_window(
|
||||
_dt.datetime.fromisoformat(ts) if isinstance(ts, str) else None
|
||||
)
|
||||
|
||||
# Distinct-learner count for k-anon: this session's learner + any others
|
||||
# already recorded for the same (path, window). For the per-session hook
|
||||
# we accumulate by appending to a sessions_count cell + tracking distinct
|
||||
# learner_refs via active_learners_count. The nightly job recomputes from
|
||||
# the mastery_gate_events + session log (full reconciliation).
|
||||
#
|
||||
# For the on-session-end hook we cannot cheaply know all distinct learners
|
||||
# without a raw-events table (which we deliberately do not maintain for PII
|
||||
# reasons — D-031). We instead maintain a single active_learners_count
|
||||
# counter per (path, window) and the nightly job reconciles the true
|
||||
# distinct count from mastery_gate_events. The hook uses the running
|
||||
# counter; if it is < K_ANON_THRESHOLD we suppress.
|
||||
active_count = await _bump_active_learners(pg_store, path, window_start, learner_ref)
|
||||
sessions_count = await _bump_counter(pg_store, path, "sessions_count", window_start, window_end)
|
||||
|
||||
suppressed = active_count < K_ANON_THRESHOLD
|
||||
|
||||
await _upsert_cell(pg_store, path, "sessions_count", window_start, window_end,
|
||||
float(sessions_count) if not suppressed else None,
|
||||
active_count, suppressed)
|
||||
|
||||
await _upsert_cell(pg_store, path, "active_learners_count", window_start, window_end,
|
||||
float(active_count) if not suppressed else None,
|
||||
active_count, suppressed)
|
||||
|
||||
# gate_open_rate: 1.0 if this session passed, 0.0 otherwise (running mean
|
||||
# reconciled by nightly). Stored as the fraction of pass outcomes seen.
|
||||
passed = 1.0 if outcome == "pass" else 0.0
|
||||
gate_open_rate = await _running_mean(pg_store, path, "gate_open_rate",
|
||||
window_start, window_end, passed, active_count)
|
||||
await _upsert_cell(pg_store, path, "gate_open_rate", window_start, window_end,
|
||||
gate_open_rate if not suppressed else None,
|
||||
active_count, suppressed)
|
||||
|
||||
# median_mastery_score (from rubric scores) — running median reconciled nightly
|
||||
if rubric_scores:
|
||||
scores = [float(r.get("score", r.get("weighted_mean", 0.0))) for r in rubric_scores]
|
||||
scenario_mean = statistics.mean(scores) if scores else 0.0
|
||||
median_val = await _running_mean(pg_store, path, "median_mastery_score",
|
||||
window_start, window_end, scenario_mean, active_count)
|
||||
await _upsert_cell(pg_store, path, "median_mastery_score", window_start, window_end,
|
||||
median_val if not suppressed else None,
|
||||
active_count, suppressed)
|
||||
|
||||
# rubric_criterion_means — one cell per criterion id
|
||||
for r in rubric_scores:
|
||||
cid = r.get("criterion_id") or r.get("id") or "unknown"
|
||||
score = float(r.get("score", 0.0))
|
||||
mean_val = await _running_mean(pg_store, path, f"rubric_criterion_mean:{cid}",
|
||||
window_start, window_end, score, active_count)
|
||||
await _upsert_cell(pg_store, path, f"rubric_criterion_mean:{cid}",
|
||||
window_start, window_end,
|
||||
mean_val if not suppressed else None,
|
||||
active_count, suppressed)
|
||||
|
||||
# failure_mode_frequency — one cell per observed mode
|
||||
if failure_mode:
|
||||
freq = await _bump_mode_counter(pg_store, path, f"failure_mode:{failure_mode}",
|
||||
window_start, window_end)
|
||||
await _upsert_cell(pg_store, path, f"failure_mode:{failure_mode}",
|
||||
window_start, window_end,
|
||||
float(freq) if not suppressed else None,
|
||||
active_count, suppressed)
|
||||
|
||||
# week_distribution — branch_path captures the path-week; record one cell
|
||||
# per branch outcome seen.
|
||||
if branch_path:
|
||||
last_branch = branch_path[-1] if isinstance(branch_path, list) else str(branch_path)
|
||||
freq = await _bump_mode_counter(pg_store, path, f"branch:{last_branch}",
|
||||
window_start, window_end)
|
||||
await _upsert_cell(pg_store, path, f"branch:{last_branch}",
|
||||
window_start, window_end,
|
||||
float(freq) if not suppressed else None,
|
||||
active_count, suppressed)
|
||||
|
||||
log.debug(
|
||||
"aggregate_session path=%s learner=%s outcome=%s window=%s..%s "
|
||||
"active=%d suppressed=%s",
|
||||
path, learner_ref, outcome, window_start, window_end,
|
||||
active_count, suppressed,
|
||||
)
|
||||
|
||||
|
||||
# ── Internal cell upsert + counter helpers ──────────────────────────────────
|
||||
# The PgStore.upsert_cohort_aggregate is idempotent (ON CONFLICT). We use a
|
||||
# small in-memory cache on the PgStore instance (created lazily) to track
|
||||
# per-(path, metric, window) running counters + distinct learner sets. The
|
||||
# nightly job bypasses this cache and recomputes from mastery_gate_events.
|
||||
|
||||
|
||||
def _cache(pg_store: PgStore) -> dict:
|
||||
cache = getattr(pg_store, "_agg_cache", None)
|
||||
if not isinstance(cache, dict):
|
||||
cache = {}
|
||||
try:
|
||||
pg_store._agg_cache = cache # type: ignore[attr-defined]
|
||||
except Exception:
|
||||
pass
|
||||
return cache
|
||||
|
||||
|
||||
def _ck(path: str, metric: str, window_start: _dt.date) -> tuple:
|
||||
return (path, metric, window_start)
|
||||
|
||||
|
||||
async def _upsert_cell(pg_store: PgStore, path: str, metric: str,
|
||||
window_start: _dt.date, window_end: _dt.date,
|
||||
value: float | None, cell_count: int,
|
||||
suppressed: bool) -> None:
|
||||
await pg_store.upsert_cohort_aggregate(
|
||||
path, metric, window_start, window_end, value, cell_count, suppressed,
|
||||
)
|
||||
|
||||
|
||||
async def _bump_active_learners(pg_store: PgStore, path: str,
|
||||
window_start: _dt.date, learner_ref: str) -> int:
|
||||
"""Track distinct learner_refs per (path, window) in the in-memory cache.
|
||||
|
||||
Returns the current distinct count (after adding this learner). The
|
||||
nightly job reconciles the true count from mastery_gate_events.
|
||||
"""
|
||||
cache = _cache(pg_store)
|
||||
key = _ck(path, "__learners__", window_start)
|
||||
learners: set[str] = cache.get(key, set())
|
||||
learners.add(learner_ref)
|
||||
cache[key] = learners
|
||||
return len(learners)
|
||||
|
||||
|
||||
async def _bump_counter(pg_store: PgStore, path: str, metric: str,
|
||||
window_start: _dt.date, window_end: _dt.date) -> int:
|
||||
cache = _cache(pg_store)
|
||||
key = _ck(path, metric, window_start)
|
||||
cache[key] = cache.get(key, 0) + 1
|
||||
return cache[key]
|
||||
|
||||
|
||||
async def _bump_mode_counter(pg_store: PgStore, path: str, metric: str,
|
||||
window_start: _dt.date, window_end: _dt.date) -> int:
|
||||
return await _bump_counter(pg_store, path, metric, window_start, window_end)
|
||||
|
||||
|
||||
async def _running_mean(pg_store: PgStore, path: str, metric: str,
|
||||
window_start: _dt.date, window_end: _dt.date,
|
||||
value: float, _active_count: int) -> float:
|
||||
"""Incremental running mean per (path, metric, window)."""
|
||||
cache = _cache(pg_store)
|
||||
k = _ck(path, metric, window_start)
|
||||
n_key = _ck(path, metric + "__n__", window_start)
|
||||
n = cache.get(n_key, 0)
|
||||
prev = cache.get(k, 0.0)
|
||||
new_n = n + 1
|
||||
new_mean = prev + (value - prev) / new_n
|
||||
cache[k] = new_mean
|
||||
cache[n_key] = new_n
|
||||
return new_mean
|
||||
|
||||
|
||||
__all__ = ["aggregate_session", "K_ANON_THRESHOLD", "_rolling_window"]
|
||||
@@ -1,44 +0,0 @@
|
||||
"""On-session-end async aggregation hook (TASK-07-02, D-054).
|
||||
|
||||
Fire-and-forget: designed to be chained as an `asyncio.create_task` after
|
||||
the mastery flow. Failures log + the nightly job reconciles (no exception
|
||||
propagation to the caller — the session-end response returns immediately).
|
||||
|
||||
If `pg_store` is None (no Postgres), no-op + log WARNING.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from db.pg_store import PgStore
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
async def on_session_end(pg_store: PgStore | None, session_outcome: dict[str, Any]) -> None:
|
||||
"""Aggregate one session outcome. Non-blocking, fire-and-forget (D-054).
|
||||
|
||||
Failures are logged but never raised — the caller (session_recorder) has
|
||||
already returned its response; aggregation is off the voice path. The
|
||||
nightly job (nightly.py) reconciles any missed/hook-failed sessions.
|
||||
"""
|
||||
if pg_store is None:
|
||||
log.warning(
|
||||
"cohort aggregation skipped (no Postgres) for session %s",
|
||||
session_outcome.get("scenario_id"),
|
||||
)
|
||||
return
|
||||
try:
|
||||
from server.cohort.aggregator import aggregate_session
|
||||
|
||||
await aggregate_session(pg_store, session_outcome)
|
||||
except Exception:
|
||||
log.exception(
|
||||
"cohort aggregation hook failed for session %s — nightly job will reconcile",
|
||||
session_outcome.get("scenario_id"),
|
||||
)
|
||||
|
||||
|
||||
__all__ = ["on_session_end"]
|
||||
@@ -1,232 +0,0 @@
|
||||
"""Nightly reconciliation scheduler (TASK-07-03, D-054, REQ-NFR-DASH-02).
|
||||
|
||||
In-process asyncio scheduler (no APScheduler — RESEARCH-v0.4 §3.4). Loops:
|
||||
compute seconds until next 03:00 CT (America/Winnipeg — Canada pilot) →
|
||||
asyncio.sleep → reconcile all 7-day windows → repeat. Resumes after restart.
|
||||
Failures log + retry next night (R-DASH-04).
|
||||
|
||||
Reconciliation recomputes all (path, metric, window_start) cells from the
|
||||
mastery_gate_events audit log + re-applies k-anonymity suppression. This
|
||||
guarantees REQ-NFR-DASH-02 (freshness ≤ 24h — the nightly job runs at least
|
||||
once/day) and reconciles any hook failures.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import datetime as _dt
|
||||
import logging
|
||||
import statistics
|
||||
from collections import Counter, defaultdict
|
||||
from typing import Any
|
||||
|
||||
from db.pg_store import PgStore
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
CT = _dt.timezone(_dt.timedelta(hours=-5), "CT")
|
||||
NIGHTLY_HOUR = 3
|
||||
NIGHTLY_MINUTE = 0
|
||||
|
||||
|
||||
def seconds_until_next_03_ct(now: _dt.datetime | None = None) -> float:
|
||||
"""Seconds from `now` until the next 03:00 America/Winnipeg (CT).
|
||||
|
||||
America/Winnipeg observes CST (UTC-6) in winter + CDT (UTC-5) in summer.
|
||||
We approximate CT as a fixed UTC-5 offset (the pilot is in summer CDT
|
||||
and the scheduler drift of ≤1h over DST boundaries is acceptable for a
|
||||
nightly reconciliation job — the on-session-end hook keeps data fresh).
|
||||
A future hardening would use zoneinfo.ZoneInfo("America/Winnipeg") with
|
||||
proper DST handling.
|
||||
"""
|
||||
now = now or _dt.datetime.now(CT)
|
||||
if now.tzinfo is None:
|
||||
now = now.replace(tzinfo=CT)
|
||||
next_run = now.replace(hour=NIGHTLY_HOUR, minute=NIGHTLY_MINUTE,
|
||||
second=0, microsecond=0)
|
||||
if next_run <= now:
|
||||
next_run += _dt.timedelta(days=1)
|
||||
return (next_run - now).total_seconds()
|
||||
|
||||
|
||||
class NightlyScheduler:
|
||||
"""In-process asyncio scheduler for nightly cohort reconciliation.
|
||||
|
||||
Started as an asyncio task in the app lifespan (TASK-10-02). Cancel on
|
||||
shutdown. R-DASH-04: a reconciliation failure logs + retries the next
|
||||
night (the loop continues).
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._task: asyncio.Task | None = None
|
||||
self._stopped = False
|
||||
|
||||
async def start(self, pg_store: PgStore) -> asyncio.Task:
|
||||
"""Begin the nightly loop. Returns the running task."""
|
||||
self._stopped = False
|
||||
self._task = asyncio.create_task(self._run_loop(pg_store))
|
||||
return self._task
|
||||
|
||||
async def stop(self) -> None:
|
||||
"""Cancel the running loop (graceful shutdown)."""
|
||||
self._stopped = True
|
||||
if self._task is not None:
|
||||
self._task.cancel()
|
||||
try:
|
||||
await self._task
|
||||
except (asyncio.CancelledError, Exception):
|
||||
pass
|
||||
self._task = None
|
||||
|
||||
async def _run_loop(self, pg_store: PgStore) -> None:
|
||||
while not self._stopped:
|
||||
try:
|
||||
secs = seconds_until_next_03_ct()
|
||||
log.info("nightly scheduler: next run in %.0fs (03:00 CT)", secs)
|
||||
await asyncio.sleep(secs)
|
||||
if self._stopped:
|
||||
return
|
||||
await self._reconcile(pg_store)
|
||||
except asyncio.CancelledError:
|
||||
return
|
||||
except Exception:
|
||||
log.exception("nightly reconciliation failed — retry next night (R-DASH-04)")
|
||||
# brief sleep to avoid a tight error loop if the clock is broken
|
||||
await asyncio.sleep(60)
|
||||
|
||||
async def _reconcile(self, pg_store: PgStore) -> None:
|
||||
"""Recompute all 7-day windows for all paths from mastery_gate_events.
|
||||
|
||||
Reads recent gate events (the audit log, REQ-NFR-MAST-02), groups by
|
||||
(path, window_start), recomputes each metric cell, applies k-anon
|
||||
suppression, and upserts. Idempotent — re-running produces the same
|
||||
aggregates (ON CONFLICT upsert).
|
||||
"""
|
||||
events = await _load_recent_events(pg_store)
|
||||
if not events:
|
||||
log.info("nightly reconcile: no recent gate events; nothing to recompute")
|
||||
return
|
||||
|
||||
# Group by path → window_start → list[events]
|
||||
by_path_window: dict[tuple[str, _dt.date], list[dict[str, Any]]] = defaultdict(list)
|
||||
today = _dt.datetime.now(_dt.timezone.utc).date()
|
||||
window_start = today - _dt.timedelta(days=6)
|
||||
for ev in events:
|
||||
ev_date = _coerce_date(ev.get("recorded_at"))
|
||||
if ev_date is None or ev_date < window_start:
|
||||
continue
|
||||
path = ev.get("path_id") or "unknown"
|
||||
by_path_window[(path, window_start)].append(ev)
|
||||
|
||||
from server.cohort.aggregator import K_ANON_THRESHOLD, _rolling_window
|
||||
|
||||
ws, we = _rolling_window()
|
||||
for (path, _), evs in by_path_window.items():
|
||||
learners = {e.get("learner_ref") for e in evs if e.get("learner_ref")}
|
||||
active_count = len(learners)
|
||||
suppressed = active_count < K_ANON_THRESHOLD
|
||||
|
||||
# sessions_count
|
||||
await pg_store.upsert_cohort_aggregate(
|
||||
path, "sessions_count", ws, we,
|
||||
None if suppressed else float(len(evs)),
|
||||
active_count, suppressed,
|
||||
)
|
||||
# active_learners_count
|
||||
await pg_store.upsert_cohort_aggregate(
|
||||
path, "active_learners_count", ws, we,
|
||||
None if suppressed else float(active_count),
|
||||
active_count, suppressed,
|
||||
)
|
||||
# gate_open_rate
|
||||
gate_opens = sum(1 for e in evs if (e.get("gate_outcome") or "") == "open")
|
||||
rate = gate_opens / len(evs) if evs else 0.0
|
||||
await pg_store.upsert_cohort_aggregate(
|
||||
path, "gate_open_rate", ws, we,
|
||||
None if suppressed else rate,
|
||||
active_count, suppressed,
|
||||
)
|
||||
# median_mastery_score + rubric_criterion_means from rubric_scores_jsonb
|
||||
score_rows: list[float] = []
|
||||
crit_scores: dict[str, list[float]] = defaultdict(list)
|
||||
for e in evs:
|
||||
scores = e.get("rubric_scores") or []
|
||||
if isinstance(scores, str):
|
||||
import json as _json
|
||||
try:
|
||||
scores = _json.loads(scores)
|
||||
except Exception:
|
||||
scores = []
|
||||
for r in scores:
|
||||
if isinstance(r, dict):
|
||||
cid = r.get("criterion_id") or r.get("id") or "unknown"
|
||||
s = r.get("score") or r.get("weighted_mean")
|
||||
if s is not None:
|
||||
crit_scores[cid].append(float(s))
|
||||
score_rows.append(float(s))
|
||||
if score_rows:
|
||||
med = statistics.median(score_rows)
|
||||
await pg_store.upsert_cohort_aggregate(
|
||||
path, "median_mastery_score", ws, we,
|
||||
None if suppressed else med,
|
||||
active_count, suppressed,
|
||||
)
|
||||
for cid, vals in crit_scores.items():
|
||||
mean_v = statistics.mean(vals) if vals else 0.0
|
||||
await pg_store.upsert_cohort_aggregate(
|
||||
path, f"rubric_criterion_mean:{cid}", ws, we,
|
||||
None if suppressed else mean_v,
|
||||
active_count, suppressed,
|
||||
)
|
||||
|
||||
log.info("nightly reconcile: recomputed %d (path, window) cells", len(by_path_window))
|
||||
|
||||
async def reconcile_now(self, pg_store: PgStore) -> None:
|
||||
"""Public hook for tests / ad-hoc reconciliation (no clock wait)."""
|
||||
await self._reconcile(pg_store)
|
||||
|
||||
|
||||
async def _load_recent_events(pg_store: PgStore) -> list[dict[str, Any]]:
|
||||
"""Load mastery_gate_events from the last 7 days.
|
||||
|
||||
Uses the PgStore pool directly (no extra method on PgStore to keep the
|
||||
surface minimal). Returns rows as dicts with decoded rubric_scores.
|
||||
"""
|
||||
async with pg_store.pool.acquire() as conn:
|
||||
rows = await conn.fetch(
|
||||
"SELECT learner_ref, scenario_id, path_id, gate_outcome, "
|
||||
"rubric_scores_jsonb, recorded_at "
|
||||
"FROM mastery_gate_events "
|
||||
"WHERE recorded_at >= now() - interval '7 days' "
|
||||
"ORDER BY recorded_at"
|
||||
)
|
||||
out: list[dict[str, Any]] = []
|
||||
for r in rows:
|
||||
d = dict(r)
|
||||
scores = d.get("rubric_scores_jsonb")
|
||||
if hasattr(scores, "resolve"):
|
||||
try:
|
||||
import json as _json
|
||||
d["rubric_scores"] = _json.loads(scores.resolve()) if scores else []
|
||||
except Exception:
|
||||
d["rubric_scores"] = []
|
||||
else:
|
||||
d["rubric_scores"] = scores
|
||||
out.append(d)
|
||||
return out
|
||||
|
||||
|
||||
def _coerce_date(val: Any) -> _dt.date | None:
|
||||
if val is None:
|
||||
return None
|
||||
if isinstance(val, _dt.datetime):
|
||||
return val.date()
|
||||
if isinstance(val, _dt.date):
|
||||
return val
|
||||
try:
|
||||
return _dt.datetime.fromisoformat(str(val)).date()
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
__all__ = ["NightlyScheduler", "seconds_until_next_03_ct", "CT"]
|
||||
@@ -1,93 +0,0 @@
|
||||
"""Shared helpers for operator API endpoints (SLICE-08).
|
||||
|
||||
Common response models + the recent-aggregates query used by all 3 cohort
|
||||
view endpoints (cohort, mastery, failure-patterns). Kept here to avoid
|
||||
duplicating the Pydantic models + pool query across 3 files.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as _dt
|
||||
from typing import Any
|
||||
|
||||
from fastapi import HTTPException, Request, status
|
||||
from pydantic import BaseModel
|
||||
|
||||
|
||||
class Cell(BaseModel):
|
||||
metric: str
|
||||
window_start: _dt.date | None = None
|
||||
window_end: _dt.date | None = None
|
||||
value: float | None = None
|
||||
cell_count: int = 0
|
||||
cell_suppressed: bool = False
|
||||
updated_at: _dt.datetime | None = None
|
||||
|
||||
|
||||
class PathView(BaseModel):
|
||||
path: str
|
||||
metrics: list[Cell]
|
||||
|
||||
|
||||
class ViewResponse(BaseModel):
|
||||
views: list[PathView]
|
||||
last_updated: _dt.datetime | None = None
|
||||
|
||||
|
||||
async def require_pg_store(request: Request):
|
||||
pg_store = getattr(request.app.state, "pg_store", None)
|
||||
if pg_store is None:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
|
||||
detail="operator tier unavailable (no Postgres)",
|
||||
)
|
||||
return pg_store
|
||||
|
||||
|
||||
async def all_recent_aggregates(pg_store, since: _dt.date) -> list[dict[str, Any]]:
|
||||
async with pg_store.pool.acquire() as conn:
|
||||
rows = await conn.fetch(
|
||||
"SELECT path, metric, window_start, window_end, value, "
|
||||
"cell_count, cell_suppressed, updated_at "
|
||||
"FROM cohort_aggregates WHERE window_start >= $1 "
|
||||
"ORDER BY path, metric, window_start",
|
||||
since,
|
||||
)
|
||||
return [dict(r) for r in rows]
|
||||
|
||||
|
||||
def cell_from_row(row: dict[str, Any]) -> Cell:
|
||||
return Cell(
|
||||
metric=row.get("metric", ""),
|
||||
window_start=row.get("window_start"),
|
||||
window_end=row.get("window_end"),
|
||||
value=float(row["value"]) if row.get("value") is not None else None,
|
||||
cell_count=int(row.get("cell_count") or 0),
|
||||
cell_suppressed=bool(row.get("cell_suppressed") or False),
|
||||
updated_at=row.get("updated_at"),
|
||||
)
|
||||
|
||||
|
||||
def group_by_path(
|
||||
rows: list[dict[str, Any]],
|
||||
metric_filter: set[str] | None = None,
|
||||
) -> tuple[list[PathView], _dt.datetime | None]:
|
||||
by_path: dict[str, list[dict[str, Any]]] = {}
|
||||
last_updated: _dt.datetime | None = None
|
||||
for r in rows:
|
||||
if metric_filter is not None and r.get("metric") not in metric_filter:
|
||||
continue
|
||||
by_path.setdefault(r["path"], []).append(r)
|
||||
ua = r.get("updated_at")
|
||||
if isinstance(ua, _dt.datetime) and (last_updated is None or ua > last_updated):
|
||||
last_updated = ua
|
||||
views = [PathView(path=p, metrics=[cell_from_row(c) for c in cells])
|
||||
for p, cells in by_path.items()]
|
||||
return views, last_updated
|
||||
|
||||
|
||||
__all__ = [
|
||||
"Cell", "PathView", "ViewResponse",
|
||||
"require_pg_store", "all_recent_aggregates",
|
||||
"cell_from_row", "group_by_path",
|
||||
]
|
||||
@@ -1,42 +0,0 @@
|
||||
"""GET /api/operator/cohort — practice volume view (TASK-08-01, D-053, D-057).
|
||||
|
||||
Auth-gated (Depends(current_operator)). Returns k-anonymized practice-volume
|
||||
aggregates from cohort_aggregates: sessions_count + active_learners_count per
|
||||
path. Suppressed cells have value=null + cell_suppressed=true; the frontend
|
||||
renders \"— (<10 learners)\". No per-learner drill-down (R-DASH-02).
|
||||
last_updated = max(updated_at) for freshness (REQ-NFR-DASH-02).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as _dt
|
||||
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
|
||||
from server.auth.dependencies import current_operator
|
||||
from server.auth.models import Operator
|
||||
from server.operator._common import (
|
||||
ViewResponse,
|
||||
all_recent_aggregates,
|
||||
group_by_path,
|
||||
require_pg_store,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/api/operator", tags=["operator-cohort"])
|
||||
|
||||
PRACTICE_METRICS = {"sessions_count", "active_learners_count"}
|
||||
|
||||
|
||||
@router.get("/cohort", response_model=ViewResponse)
|
||||
async def cohort_view(
|
||||
request: Request,
|
||||
op: Operator = Depends(current_operator),
|
||||
) -> ViewResponse:
|
||||
pg_store = await require_pg_store(request)
|
||||
since = _dt.date.today() - _dt.timedelta(days=30)
|
||||
rows = await all_recent_aggregates(pg_store, since)
|
||||
views, last_updated = group_by_path(rows, PRACTICE_METRICS)
|
||||
return ViewResponse(views=views, last_updated=last_updated)
|
||||
|
||||
|
||||
__all__ = ["router"]
|
||||
@@ -1,78 +0,0 @@
|
||||
"""GET/POST /api/operator/credentials — VC management (TASK-08-04, D-057).
|
||||
|
||||
Auth-gated. GET lists issued VCs from Postgres issued_credentials (operator's
|
||||
issuance log). POST /{id}/revoke revokes a VC (status='revoked',
|
||||
revoked_at=now()). Revoked credentials fail verification. No PII beyond what
|
||||
the credential asserts (D-043).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as _dt
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request, status
|
||||
from pydantic import BaseModel
|
||||
|
||||
from server.auth.dependencies import current_operator
|
||||
from server.auth.models import Operator
|
||||
from server.operator._common import require_pg_store
|
||||
|
||||
router = APIRouter(prefix="/api/operator", tags=["operator-credentials"])
|
||||
|
||||
|
||||
class CredentialOut(BaseModel):
|
||||
id: str
|
||||
learner_ref: str
|
||||
vc_type: str | None = None
|
||||
status: str
|
||||
issued_at: _dt.datetime | None = None
|
||||
revoked_at: _dt.datetime | None = None
|
||||
|
||||
|
||||
class CredentialListResponse(BaseModel):
|
||||
credentials: list[CredentialOut]
|
||||
|
||||
|
||||
class OkResponse(BaseModel):
|
||||
ok: bool = True
|
||||
id: str
|
||||
status: str
|
||||
|
||||
|
||||
@router.get("/credentials", response_model=CredentialListResponse)
|
||||
async def list_credentials(
|
||||
request: Request,
|
||||
op: Operator = Depends(current_operator),
|
||||
) -> CredentialListResponse:
|
||||
pg_store = await require_pg_store(request)
|
||||
rows = await pg_store.list_credentials()
|
||||
creds = [
|
||||
CredentialOut(
|
||||
id=str(r["id"]),
|
||||
learner_ref=r["learner_ref"],
|
||||
vc_type=r.get("vc_type"),
|
||||
status=r.get("status", "active"),
|
||||
issued_at=r.get("issued_at"),
|
||||
revoked_at=r.get("revoked_at"),
|
||||
)
|
||||
for r in rows
|
||||
]
|
||||
return CredentialListResponse(credentials=creds)
|
||||
|
||||
|
||||
@router.post("/credentials/{cred_id}/revoke", response_model=OkResponse)
|
||||
async def revoke_credential(
|
||||
cred_id: str,
|
||||
request: Request,
|
||||
op: Operator = Depends(current_operator),
|
||||
) -> OkResponse:
|
||||
pg_store = await require_pg_store(request)
|
||||
row = await pg_store.get_credential(cred_id)
|
||||
if row is None:
|
||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail="credential not found")
|
||||
await pg_store.set_credential_status(cred_id, "revoked")
|
||||
return OkResponse(ok=True, id=cred_id, status="revoked")
|
||||
|
||||
|
||||
__all__ = ["router"]
|
||||
@@ -1,44 +0,0 @@
|
||||
"""GET /api/operator/failure-patterns — failure patterns view (TASK-08-03, D-053).
|
||||
|
||||
Auth-gated. Returns failure pattern metrics: failure_mode frequency (cells
|
||||
with metric prefix `failure_mode:`) + branch outcome distribution (cells
|
||||
with metric prefix `branch:`). Weak-spot rubric criteria (mean < 3.0) are
|
||||
highlighted by the frontend. All k-anonymized.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as _dt
|
||||
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
|
||||
from server.auth.dependencies import current_operator
|
||||
from server.auth.models import Operator
|
||||
from server.operator._common import (
|
||||
ViewResponse,
|
||||
all_recent_aggregates,
|
||||
group_by_path,
|
||||
require_pg_store,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/api/operator", tags=["operator-failure-patterns"])
|
||||
|
||||
|
||||
def _is_failure_metric(metric: str) -> bool:
|
||||
return metric.startswith("failure_mode:") or metric.startswith("branch:")
|
||||
|
||||
|
||||
@router.get("/failure-patterns", response_model=ViewResponse)
|
||||
async def failure_patterns_view(
|
||||
request: Request,
|
||||
op: Operator = Depends(current_operator),
|
||||
) -> ViewResponse:
|
||||
pg_store = await require_pg_store(request)
|
||||
since = _dt.date.today() - _dt.timedelta(days=30)
|
||||
rows = await all_recent_aggregates(pg_store, since)
|
||||
failure_rows = [r for r in rows if _is_failure_metric(r.get("metric", ""))]
|
||||
views, last_updated = group_by_path(failure_rows)
|
||||
return ViewResponse(views=views, last_updated=last_updated)
|
||||
|
||||
|
||||
__all__ = ["router"]
|
||||
@@ -1,45 +0,0 @@
|
||||
"""GET /api/operator/mastery — mastery progression view (TASK-08-02, D-053).
|
||||
|
||||
Auth-gated. Returns mastery progression metrics: gate_open_rate,
|
||||
median_mastery_score, rubric_criterion_means (cells with metric prefix
|
||||
`rubric_criterion_mean:`). All k-anonymized (suppressed if < 10).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as _dt
|
||||
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
|
||||
from server.auth.dependencies import current_operator
|
||||
from server.auth.models import Operator
|
||||
from server.operator._common import (
|
||||
ViewResponse,
|
||||
all_recent_aggregates,
|
||||
group_by_path,
|
||||
require_pg_store,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/api/operator", tags=["operator-mastery"])
|
||||
|
||||
MASTERY_METRICS = {"gate_open_rate", "median_mastery_score"}
|
||||
|
||||
|
||||
def _is_mastery_metric(metric: str) -> bool:
|
||||
return metric in MASTERY_METRICS or metric.startswith("rubric_criterion_mean:")
|
||||
|
||||
|
||||
@router.get("/mastery", response_model=ViewResponse)
|
||||
async def mastery_view(
|
||||
request: Request,
|
||||
op: Operator = Depends(current_operator),
|
||||
) -> ViewResponse:
|
||||
pg_store = await require_pg_store(request)
|
||||
since = _dt.date.today() - _dt.timedelta(days=30)
|
||||
rows = await all_recent_aggregates(pg_store, since)
|
||||
mastery_rows = [r for r in rows if _is_mastery_metric(r.get("metric", ""))]
|
||||
views, last_updated = group_by_path(mastery_rows)
|
||||
return ViewResponse(views=views, last_updated=last_updated)
|
||||
|
||||
|
||||
__all__ = ["router"]
|
||||
@@ -16,7 +16,6 @@ No auth — learner_id is the hardcoded 'learner-1' (D-007).
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import datetime as _dt
|
||||
import json
|
||||
import logging
|
||||
import uuid
|
||||
@@ -28,10 +27,6 @@ from server.cost import CostBreakdown, derive_cost
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _now_iso() -> str:
|
||||
return _dt.datetime.now(_dt.timezone.utc).isoformat()
|
||||
|
||||
|
||||
class SessionRecorder:
|
||||
"""Records a voice session to SQLite (TASK-04-03)."""
|
||||
|
||||
@@ -40,12 +35,10 @@ class SessionRecorder:
|
||||
store: PraxisStore,
|
||||
learner_id: str = HARDCODED_LEARNER_ID,
|
||||
scenario_id: str = "cs_refund_ca_v01",
|
||||
pg_store: Any = None,
|
||||
) -> None:
|
||||
self.store = store
|
||||
self.learner_id = learner_id
|
||||
self.scenario_id = scenario_id
|
||||
self.pg_store = pg_store
|
||||
self.session_id: str | None = None
|
||||
self._turn_seq = 0
|
||||
# Cost inputs accumulated over the session.
|
||||
@@ -150,53 +143,8 @@ class SessionRecorder:
|
||||
asyncio.create_task(
|
||||
self._run_mastery_flow_guarded(mastery_deps)
|
||||
)
|
||||
|
||||
# v0.4 P2 (D-054): fire-and-forget cohort aggregation hook. Runs in
|
||||
# parallel with the mastery flow — aggregation only needs the session
|
||||
# outcome (available after session end), not the mastery scoring
|
||||
# result. Rubric-dependent metrics are reconciled by the nightly job.
|
||||
# Off the voice path (C-8, D-054). No-op if pg_store is None.
|
||||
if self.pg_store is not None:
|
||||
session_outcome = self._build_session_outcome(outcome)
|
||||
asyncio.create_task(self._run_cohort_aggregation(session_outcome))
|
||||
return breakdown
|
||||
|
||||
def _build_session_outcome(self, outcome: str) -> dict[str, Any]:
|
||||
"""Construct the session_outcome dict for the aggregation hook."""
|
||||
rubric_scores: list[dict[str, Any]] = []
|
||||
if self.mastery_result and isinstance(self.mastery_result, dict):
|
||||
rubric_scores = list(self.mastery_result.get("rubric_scores") or [])
|
||||
return {
|
||||
"learner_ref": self.learner_id,
|
||||
"path": self._path_slug(),
|
||||
"scenario_id": self.scenario_id,
|
||||
"outcome": outcome,
|
||||
"rubric_scores": rubric_scores,
|
||||
"failure_mode": self._failure_mode(),
|
||||
"branch_path": list(self._branch_path),
|
||||
"timestamp": _now_iso(),
|
||||
}
|
||||
|
||||
def _path_slug(self) -> str:
|
||||
# The scenario_id encodes the path loosely; default to customer_service.
|
||||
if self.scenario_id and self.scenario_id.startswith("cs_"):
|
||||
return "customer_service"
|
||||
return "default"
|
||||
|
||||
def _failure_mode(self) -> str | None:
|
||||
if self.mastery_result and isinstance(self.mastery_result, dict):
|
||||
return self.mastery_result.get("failure_mode")
|
||||
return None
|
||||
|
||||
async def _run_cohort_aggregation(self, session_outcome: dict[str, Any]) -> None:
|
||||
"""Fire-and-forget wrapper around the cohort aggregation hook (D-054)."""
|
||||
try:
|
||||
from server.cohort.hook import on_session_end
|
||||
|
||||
await on_session_end(self.pg_store, session_outcome)
|
||||
except Exception:
|
||||
log.exception("cohort aggregation dispatch failed for session %s", self.session_id)
|
||||
|
||||
async def _run_mastery_flow_guarded(self, deps: "MasteryFlowDeps") -> None:
|
||||
try:
|
||||
await self.run_mastery_flow(deps)
|
||||
|
||||
@@ -1,246 +0,0 @@
|
||||
"""Cohort aggregation unit tests (TASK-07-05) — mocked PgStore, no Postgres.
|
||||
|
||||
Covers: k-anonymity suppression (9 vs 10 vs 11 learners), idempotent upsert,
|
||||
7-day window computation, multiple metrics, no PII in upsert calls.
|
||||
|
||||
G-038 (binding — differencing-attack test): seed 10 learners in window A and
|
||||
9 in window B (one dropped), verify the API/aggregation cannot isolate the
|
||||
dropped learner — both windows show k-anonymized aggregates with no
|
||||
per-learner data leaks.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as _dt
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from server.cohort.aggregator import (
|
||||
K_ANON_THRESHOLD,
|
||||
_rolling_window,
|
||||
aggregate_session,
|
||||
)
|
||||
from server.cohort.hook import on_session_end
|
||||
|
||||
|
||||
def _mock_pg_store():
|
||||
store = MagicMock()
|
||||
store.upsert_cohort_aggregate = AsyncMock()
|
||||
return store
|
||||
|
||||
|
||||
def _session(learner_ref: str, path: str = "customer_service",
|
||||
outcome: str = "pass", rubric_scores=None,
|
||||
failure_mode=None, branch_path=None) -> dict:
|
||||
return {
|
||||
"learner_ref": learner_ref,
|
||||
"path": path,
|
||||
"scenario_id": f"{path}_v01",
|
||||
"outcome": outcome,
|
||||
"rubric_scores": rubric_scores or [
|
||||
{"criterion_id": "empathy", "score": 4.0},
|
||||
{"criterion_id": "resolution", "score": 3.5},
|
||||
],
|
||||
"failure_mode": failure_mode,
|
||||
"branch_path": branch_path or ["accept"],
|
||||
"timestamp": _dt.datetime.now(_dt.timezone.utc).isoformat(),
|
||||
}
|
||||
|
||||
|
||||
# ── k-anonymity threshold ───────────────────────────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_k_anon_threshold_at_10():
|
||||
assert K_ANON_THRESHOLD == 10
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_9_learners_suppressed():
|
||||
store = _mock_pg_store()
|
||||
for i in range(9):
|
||||
await aggregate_session(store, _session(f"learner-{i}"))
|
||||
suppressed_calls = [
|
||||
c for c in store.upsert_cohort_aggregate.call_args_list
|
||||
if c.args[6] is True # cell_suppressed
|
||||
]
|
||||
non_suppressed = [
|
||||
c for c in store.upsert_cohort_aggregate.call_args_list
|
||||
if c.args[6] is False
|
||||
]
|
||||
assert suppressed_calls, "cells should be suppressed with <10 learners"
|
||||
assert not non_suppressed, "no cell should be non-suppressed with 9 learners"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_10_learners_not_suppressed():
|
||||
store = _mock_pg_store()
|
||||
for i in range(10):
|
||||
await aggregate_session(store, _session(f"learner-{i}"))
|
||||
non_suppressed = [
|
||||
c for c in store.upsert_cohort_aggregate.call_args_list
|
||||
if c.args[6] is False
|
||||
]
|
||||
assert non_suppressed, "cells should NOT be suppressed at exactly 10 learners"
|
||||
# value should be non-null for non-suppressed cells
|
||||
for c in non_suppressed:
|
||||
assert c.args[4] is not None, "non-suppressed cell value must not be None"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_11_learners_not_suppressed():
|
||||
store = _mock_pg_store()
|
||||
for i in range(11):
|
||||
await aggregate_session(store, _session(f"learner-{i}"))
|
||||
non_suppressed = [
|
||||
c for c in store.upsert_cohort_aggregate.call_args_list
|
||||
if c.args[6] is False
|
||||
]
|
||||
assert non_suppressed, "11 learners should NOT be suppressed"
|
||||
|
||||
|
||||
# ── Idempotent upsert ──────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_idempotent_same_session_twice():
|
||||
store = _mock_pg_store()
|
||||
outcome = _session("learner-x")
|
||||
await aggregate_session(store, outcome)
|
||||
await aggregate_session(store, outcome)
|
||||
# Re-running with the same outcome produces additional upsert calls but
|
||||
# the ON CONFLICT in PgStore makes them idempotent at the DB layer. The
|
||||
# hook itself is deterministic — the same learner produces the same
|
||||
# distinct-count + counter state in the cache.
|
||||
# Assert at least one upsert happened (the contract is DB-level idempotency).
|
||||
assert store.upsert_cohort_aggregate.called
|
||||
|
||||
|
||||
# ── 7-day window computation ───────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_rolling_window_7_days():
|
||||
now = _dt.datetime(2026, 8, 4, 12, 0, tzinfo=_dt.timezone.utc)
|
||||
start, end = _rolling_window(now)
|
||||
assert (end - start).days == 6 # 7-day inclusive span
|
||||
assert end == now.date()
|
||||
assert start == _dt.date(2026, 7, 29)
|
||||
|
||||
|
||||
# ── Multiple metrics ───────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_multiple_metrics_computed():
|
||||
store = _mock_pg_store()
|
||||
await aggregate_session(store, _session("learner-1", rubric_scores=[
|
||||
{"criterion_id": "empathy", "score": 4.0},
|
||||
{"criterion_id": "resolution", "score": 3.0},
|
||||
], failure_mode="missed_apology", branch_path=["escalate"]))
|
||||
metrics = {c.args[1] for c in store.upsert_cohort_aggregate.call_args_list}
|
||||
assert "sessions_count" in metrics
|
||||
assert "active_learners_count" in metrics
|
||||
assert "gate_open_rate" in metrics
|
||||
assert "median_mastery_score" in metrics
|
||||
assert "rubric_criterion_mean:empathy" in metrics
|
||||
assert "failure_mode:missed_apology" in metrics
|
||||
assert "branch:escalate" in metrics
|
||||
|
||||
|
||||
# ── No PII in upsert calls ─────────────────────────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_no_pii_in_upsert_calls():
|
||||
store = _mock_pg_store()
|
||||
await aggregate_session(store, _session("learner-sensitive-id-1234"))
|
||||
for c in store.upsert_cohort_aggregate.call_args_list:
|
||||
# path, metric, window_start, window_end, value, cell_count, suppressed
|
||||
# No argument should contain the raw learner_ref string as PII.
|
||||
for arg in c.args:
|
||||
assert "learner-sensitive-id-1234" not in str(arg), \
|
||||
"raw learner_ref must not leak into aggregate cell args"
|
||||
# cell_count is the distinct-learner count (an integer), not the ref.
|
||||
assert isinstance(c.args[5], int)
|
||||
|
||||
|
||||
# ── G-038: Differencing-attack test (binding) ──────────────────────────────
|
||||
# Seed 10 learners in window A, 9 in window B (one dropped). Verify the
|
||||
# aggregation/API cannot isolate the dropped learner — both windows produce
|
||||
# k-anonymized aggregates with no per-learner data leaks.
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_g038_differencing_attack_cannot_isolate_dropped_learner():
|
||||
"""G-038 binding: 10 learners in window A, 9 in window B (one dropped).
|
||||
|
||||
A differencing attack tries to subtract window B's aggregate from
|
||||
window A's to recover the dropped learner's contribution. With k-anon
|
||||
write-time suppression, window B (9 learners) is FULLY suppressed
|
||||
(value=NULL, cell_suppressed=TRUE), so the attacker cannot subtract
|
||||
anything — the dropped learner's contribution is not recoverable.
|
||||
"""
|
||||
store_a = _mock_pg_store()
|
||||
store_b = _mock_pg_store()
|
||||
|
||||
# Window A: 10 distinct learners → non-suppressed
|
||||
for i in range(10):
|
||||
await aggregate_session(store_a, _session(f"learner-{i}"))
|
||||
# Window B: 9 distinct learners (learner-9 dropped) → suppressed
|
||||
for i in range(9):
|
||||
await aggregate_session(store_b, _session(f"learner-{i}"))
|
||||
|
||||
a_cells = list(store_a.upsert_cohort_aggregate.call_args_list)
|
||||
b_cells = list(store_b.upsert_cohort_aggregate.call_args_list)
|
||||
|
||||
# Window A: at least some non-suppressed cells (10 >= threshold)
|
||||
a_non_suppressed = [c for c in a_cells if c.args[6] is False]
|
||||
assert a_non_suppressed, "window A (10 learners) should have non-suppressed cells"
|
||||
|
||||
# Window B: ALL cells suppressed (9 < threshold)
|
||||
b_suppressed = [c for c in b_cells if c.args[6] is True]
|
||||
b_non_suppressed = [c for c in b_cells if c.args[6] is False]
|
||||
assert b_suppressed, "window B (9 learners) must have suppressed cells"
|
||||
assert not b_non_suppressed, \
|
||||
"window B (9 learners) must have NO non-suppressed cells (differencing blocked)"
|
||||
|
||||
# The critical differencing-attack defense: window B's suppressed cells
|
||||
# have value=NULL, so subtracting B from A is not possible — the attacker
|
||||
# cannot recover learner-9's contribution.
|
||||
for c in b_suppressed:
|
||||
assert c.args[4] is None, \
|
||||
"suppressed cell value must be NULL (differencing-attack defense)"
|
||||
|
||||
# No per-learner data leaks in either window's aggregate cells.
|
||||
for cells in (a_cells, b_cells):
|
||||
for c in cells:
|
||||
for arg in c.args:
|
||||
assert "learner-9" not in str(arg), \
|
||||
"dropped learner's ref must not appear in any aggregate cell"
|
||||
|
||||
|
||||
# ── Hook (TASK-07-02) ──────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_hook_no_postgres_is_noop():
|
||||
# No exception, just a warning log.
|
||||
await on_session_end(None, _session("learner-1"))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_hook_failure_logs_does_not_raise(monkeypatch):
|
||||
store = _mock_pg_store()
|
||||
store.upsert_cohort_aggregate = AsyncMock(side_effect=RuntimeError("boom"))
|
||||
# Must not raise — the hook swallows + logs; nightly reconciles.
|
||||
await on_session_end(store, _session("learner-1"))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_hook_idempotent():
|
||||
store = _mock_pg_store()
|
||||
outcome = _session("learner-1")
|
||||
await on_session_end(store, outcome)
|
||||
await on_session_end(store, outcome)
|
||||
assert store.upsert_cohort_aggregate.called
|
||||
@@ -1,199 +0,0 @@
|
||||
"""Nightly reconciliation + hook integration tests (TASK-07-06) — mocked PgStore.
|
||||
|
||||
Covers: scheduler timing (seconds until 03:00 CT), reconciliation recomputes
|
||||
all windows, hook failure + nightly reconciliation = correct final state,
|
||||
R-DASH-04 (nightly failure logs + retries next night).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as _dt
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from server.cohort.nightly import (
|
||||
CT,
|
||||
NightlyScheduler,
|
||||
seconds_until_next_03_ct,
|
||||
)
|
||||
|
||||
|
||||
# ── Scheduler timing ───────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_seconds_until_next_03_ct_future_today():
|
||||
# 01:00 CT → next 03:00 CT is in 2h
|
||||
now = _dt.datetime(2026, 8, 4, 1, 0, tzinfo=CT)
|
||||
secs = seconds_until_next_03_ct(now)
|
||||
assert 7190 <= secs <= 7200 # ~2h
|
||||
|
||||
|
||||
def test_seconds_until_next_03_ct_past_today_wraps_tomorrow():
|
||||
# 04:00 CT → next 03:00 CT is tomorrow (23h)
|
||||
now = _dt.datetime(2026, 8, 4, 4, 0, tzinfo=CT)
|
||||
secs = seconds_until_next_03_ct(now)
|
||||
assert 82790 <= secs <= 82810 # ~23h
|
||||
|
||||
|
||||
def test_seconds_until_next_03_ct_exactly_03_rolls_to_tomorrow():
|
||||
now = _dt.datetime(2026, 8, 4, 3, 0, 0, tzinfo=CT)
|
||||
secs = seconds_until_next_03_ct(now)
|
||||
# exactly 03:00:00 → next run is tomorrow (0 secs would mean "now", but
|
||||
# the scheduler sleeps then runs, so it must be ~24h)
|
||||
assert secs >= 86390 # ~24h
|
||||
|
||||
|
||||
# ── Reconciliation recomputes all windows ──────────────────────────────────
|
||||
|
||||
|
||||
class _FakeRecord(dict):
|
||||
"""Mimics an asyncpg Record — dict(record) returns the dict."""
|
||||
pass
|
||||
|
||||
|
||||
def _mock_pg_store_with_events(events):
|
||||
store = MagicMock()
|
||||
store.upsert_cohort_aggregate = AsyncMock()
|
||||
conn = MagicMock()
|
||||
rows = [_FakeRecord(e) for e in events]
|
||||
conn.fetch = AsyncMock(return_value=rows)
|
||||
cm = MagicMock()
|
||||
cm.__aenter__ = AsyncMock(return_value=conn)
|
||||
cm.__aexit__ = AsyncMock(return_value=None)
|
||||
store.pool = MagicMock()
|
||||
store.pool.acquire = MagicMock(return_value=cm)
|
||||
return store
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_reconcile_recomputes_all_paths():
|
||||
events = [
|
||||
{"learner_ref": "l1", "path_id": "customer_service", "gate_outcome": "open",
|
||||
"rubric_scores_jsonb": '[{"criterion_id":"empathy","score":4.0}]',
|
||||
"recorded_at": _dt.datetime.now(_dt.timezone.utc)},
|
||||
{"learner_ref": "l2", "path_id": "customer_service", "gate_outcome": "open",
|
||||
"rubric_scores_jsonb": '[{"criterion_id":"empathy","score":3.0}]',
|
||||
"recorded_at": _dt.datetime.now(_dt.timezone.utc)},
|
||||
{"learner_ref": "l3", "path_id": "sales", "gate_outcome": "closed",
|
||||
"rubric_scores_jsonb": '[]',
|
||||
"recorded_at": _dt.datetime.now(_dt.timezone.utc)},
|
||||
]
|
||||
store = _mock_pg_store_with_events(events)
|
||||
sched = NightlyScheduler()
|
||||
await sched.reconcile_now(store)
|
||||
# upserts should cover both paths × multiple metrics
|
||||
paths = {c.args[0] for c in store.upsert_cohort_aggregate.call_args_list}
|
||||
assert "customer_service" in paths
|
||||
assert "sales" in paths
|
||||
metrics = {c.args[1] for c in store.upsert_cohort_aggregate.call_args_list}
|
||||
assert "sessions_count" in metrics
|
||||
assert "active_learners_count" in metrics
|
||||
assert "gate_open_rate" in metrics
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_reconcile_suppresses_below_threshold():
|
||||
# 3 distinct learners → suppressed
|
||||
events = [
|
||||
{"learner_ref": f"l{i}", "path_id": "p", "gate_outcome": "open",
|
||||
"rubric_scores_jsonb": "[]",
|
||||
"recorded_at": _dt.datetime.now(_dt.timezone.utc)}
|
||||
for i in range(3)
|
||||
]
|
||||
store = _mock_pg_store_with_events(events)
|
||||
sched = NightlyScheduler()
|
||||
await sched.reconcile_now(store)
|
||||
suppressed = [c for c in store.upsert_cohort_aggregate.call_args_list if c.args[6] is True]
|
||||
non_suppressed = [c for c in store.upsert_cohort_aggregate.call_args_list if c.args[6] is False]
|
||||
assert suppressed, "3 learners must be suppressed"
|
||||
assert not non_suppressed, "no cell should be non-suppressed with 3 learners"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_reconcile_no_events_no_op():
|
||||
store = _mock_pg_store_with_events([])
|
||||
sched = NightlyScheduler()
|
||||
await sched.reconcile_now(store)
|
||||
store.upsert_cohort_aggregate.assert_not_called()
|
||||
|
||||
|
||||
# ── Hook failure → nightly reconciles ──────────────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_hook_failure_then_nightly_reconciles_correct_state():
|
||||
"""A hook failure leaves no aggregate; the nightly job recomputes from
|
||||
mastery_gate_events and produces the correct final state."""
|
||||
events = [
|
||||
{"learner_ref": f"l{i}", "path_id": "p", "gate_outcome": "open",
|
||||
"rubric_scores_jsonb": "[]",
|
||||
"recorded_at": _dt.datetime.now(_dt.timezone.utc)}
|
||||
for i in range(10)
|
||||
]
|
||||
store = _mock_pg_store_with_events(events)
|
||||
# Simulate hook failure: upsert raises first time, then nightly runs.
|
||||
# (In production the hook + nightly use the same store; here we just
|
||||
# verify the nightly path produces correct aggregates independently.)
|
||||
sched = NightlyScheduler()
|
||||
await sched.reconcile_now(store)
|
||||
non_suppressed = [c for c in store.upsert_cohort_aggregate.call_args_list if c.args[6] is False]
|
||||
assert non_suppressed, "nightly should produce non-suppressed cells for 10 learners"
|
||||
|
||||
|
||||
# ── R-DASH-04: nightly failure logs + retries ──────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_r_dash_04_nightly_failure_does_not_crash_scheduler():
|
||||
"""R-DASH-04: a reconciliation failure logs + the scheduler continues.
|
||||
|
||||
The scheduler loop (_run_loop) catches exceptions from _reconcile and
|
||||
retries the next night. We simulate this by invoking the loop with a
|
||||
broken store and confirming the loop catches + continues.
|
||||
"""
|
||||
store = MagicMock()
|
||||
store.upsert_cohort_aggregate = AsyncMock(side_effect=RuntimeError("db down"))
|
||||
store.pool = MagicMock()
|
||||
cm = MagicMock()
|
||||
cm.__aenter__ = AsyncMock(side_effect=RuntimeError("pool down"))
|
||||
cm.__aexit__ = AsyncMock(return_value=None)
|
||||
store.pool.acquire = MagicMock(return_value=cm)
|
||||
sched = NightlyScheduler()
|
||||
import server.cohort.nightly as nightly_mod
|
||||
orig = nightly_mod.seconds_until_next_03_ct
|
||||
calls = []
|
||||
def _fake_secs():
|
||||
calls.append(1)
|
||||
return 0.01
|
||||
nightly_mod.seconds_until_next_03_ct = _fake_secs
|
||||
try:
|
||||
task = await sched.start(store)
|
||||
await _sleep(0.1)
|
||||
await sched.stop()
|
||||
# The loop ran at least once despite the failure (R-DASH-04).
|
||||
assert len(calls) >= 1
|
||||
finally:
|
||||
nightly_mod.seconds_until_next_03_ct = orig
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_scheduler_start_stop_lifecycle():
|
||||
store = _mock_pg_store_with_events([])
|
||||
sched = NightlyScheduler()
|
||||
# Patch seconds_until to be tiny so the loop is testable.
|
||||
import server.cohort.nightly as nightly_mod
|
||||
orig = nightly_mod.seconds_until_next_03_ct
|
||||
nightly_mod.seconds_until_next_03_ct = lambda: 0.01
|
||||
try:
|
||||
task = await sched.start(store)
|
||||
await _sleep(0.05)
|
||||
await sched.stop()
|
||||
assert task.cancelled() or task.done()
|
||||
finally:
|
||||
nightly_mod.seconds_until_next_03_ct = orig
|
||||
|
||||
|
||||
async def _sleep(t: float) -> None:
|
||||
import asyncio
|
||||
await asyncio.sleep(t)
|
||||
@@ -1,304 +0,0 @@
|
||||
"""Operator API endpoint unit tests (TASK-08-05) — mocked PgStore.
|
||||
|
||||
Covers: 401 without cookie, 200 with valid cookie, suppressed cells have
|
||||
value=null, last_updated is max(updated_at), credential revoke works, no
|
||||
per-learner data in responses (R-DASH-02).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as _dt
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
from starlette.middleware.sessions import SessionMiddleware
|
||||
|
||||
from server.auth.models import Operator
|
||||
from server.auth.passwords import hash_password
|
||||
from server.auth.rate_limit import reset_login_rate_limit
|
||||
from server.auth.routes import router as auth_router
|
||||
from server.operator.cohort import router as cohort_router
|
||||
from server.operator.credentials import router as credentials_router
|
||||
from server.operator.failure_patterns import router as failure_router
|
||||
from server.operator.mastery import router as mastery_router
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _reset_limiter():
|
||||
reset_login_rate_limit()
|
||||
yield
|
||||
reset_login_rate_limit()
|
||||
|
||||
|
||||
class _FakeRecord(dict):
|
||||
pass
|
||||
|
||||
|
||||
def _mock_pg_store(aggregates=None, credentials=None):
|
||||
store = MagicMock()
|
||||
# Operator lookup for current_operator dependency.
|
||||
store.get_operator_by_id = AsyncMock(return_value={
|
||||
"id": "11111111-1111-1111-1111-111111111111",
|
||||
"username": "alice",
|
||||
"display_name": "Alice",
|
||||
"role": "operator",
|
||||
"is_active": True,
|
||||
})
|
||||
store.update_last_login = AsyncMock()
|
||||
store.get_operator_by_username = AsyncMock(return_value={
|
||||
"id": "11111111-1111-1111-1111-111111111111",
|
||||
"username": "alice",
|
||||
"display_name": "Alice",
|
||||
"role": "operator",
|
||||
"is_active": True,
|
||||
"password_hash": hash_password("pw"),
|
||||
})
|
||||
# Cohort aggregates query (all_recent_aggregates).
|
||||
aggregates = aggregates or []
|
||||
conn = MagicMock()
|
||||
conn.fetch = AsyncMock(return_value=[_FakeRecord(r) for r in aggregates])
|
||||
cm = MagicMock()
|
||||
cm.__aenter__ = AsyncMock(return_value=conn)
|
||||
cm.__aexit__ = AsyncMock(return_value=None)
|
||||
store.pool = MagicMock()
|
||||
store.pool.acquire = MagicMock(return_value=cm)
|
||||
# Credentials.
|
||||
store.list_credentials = AsyncMock(return_value=credentials or [])
|
||||
store.get_credential = AsyncMock(return_value=credentials[0] if credentials else None)
|
||||
store.set_credential_status = AsyncMock()
|
||||
return store
|
||||
|
||||
|
||||
def _make_app(store) -> FastAPI:
|
||||
app = FastAPI()
|
||||
app.state.pg_store = store
|
||||
app.add_middleware(SessionMiddleware, secret_key="test-secret-1234567890abcdef")
|
||||
app.include_router(auth_router)
|
||||
app.include_router(cohort_router)
|
||||
app.include_router(mastery_router)
|
||||
app.include_router(failure_router)
|
||||
app.include_router(credentials_router)
|
||||
return app
|
||||
|
||||
|
||||
def _login(client) -> None:
|
||||
r = client.post("/api/operator/login", json={"username": "alice", "password": "pw"})
|
||||
assert r.status_code == 200, r.text
|
||||
|
||||
|
||||
# ── 401 without cookie ─────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_cohort_401_without_cookie():
|
||||
app = _make_app(_mock_pg_store())
|
||||
with TestClient(app) as client:
|
||||
r = client.get("/api/operator/cohort")
|
||||
assert r.status_code == 401
|
||||
|
||||
|
||||
def test_mastery_401_without_cookie():
|
||||
app = _make_app(_mock_pg_store())
|
||||
with TestClient(app) as client:
|
||||
r = client.get("/api/operator/mastery")
|
||||
assert r.status_code == 401
|
||||
|
||||
|
||||
def test_failure_patterns_401_without_cookie():
|
||||
app = _make_app(_mock_pg_store())
|
||||
with TestClient(app) as client:
|
||||
r = client.get("/api/operator/failure-patterns")
|
||||
assert r.status_code == 401
|
||||
|
||||
|
||||
def test_credentials_401_without_cookie():
|
||||
app = _make_app(_mock_pg_store())
|
||||
with TestClient(app) as client:
|
||||
r = client.get("/api/operator/credentials")
|
||||
assert r.status_code == 401
|
||||
|
||||
|
||||
def test_revoke_401_without_cookie():
|
||||
app = _make_app(_mock_pg_store())
|
||||
with TestClient(app) as client:
|
||||
r = client.post("/api/operator/credentials/abc/revoke")
|
||||
assert r.status_code == 401
|
||||
|
||||
|
||||
# ── 200 with valid cookie ──────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_cohort_200_with_cookie():
|
||||
now = _dt.datetime.now(_dt.timezone.utc)
|
||||
agg = [
|
||||
{"path": "customer_service", "metric": "sessions_count",
|
||||
"window_start": _dt.date.today(), "window_end": _dt.date.today(),
|
||||
"value": 12.0, "cell_count": 12, "cell_suppressed": False,
|
||||
"updated_at": now},
|
||||
]
|
||||
app = _make_app(_mock_pg_store(aggregates=agg))
|
||||
with TestClient(app) as client:
|
||||
_login(client)
|
||||
r = client.get("/api/operator/cohort")
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
assert any(v["path"] == "customer_service" for v in body["views"])
|
||||
|
||||
|
||||
def test_mastery_200_with_cookie():
|
||||
agg = [
|
||||
{"path": "p", "metric": "gate_open_rate",
|
||||
"window_start": _dt.date.today(), "window_end": _dt.date.today(),
|
||||
"value": 0.5, "cell_count": 10, "cell_suppressed": False,
|
||||
"updated_at": _dt.datetime.now(_dt.timezone.utc)},
|
||||
]
|
||||
app = _make_app(_mock_pg_store(aggregates=agg))
|
||||
with TestClient(app) as client:
|
||||
_login(client)
|
||||
r = client.get("/api/operator/mastery")
|
||||
assert r.status_code == 200
|
||||
|
||||
|
||||
def test_failure_patterns_200_with_cookie():
|
||||
agg = [
|
||||
{"path": "p", "metric": "failure_mode:missed_apology",
|
||||
"window_start": _dt.date.today(), "window_end": _dt.date.today(),
|
||||
"value": 3.0, "cell_count": 10, "cell_suppressed": False,
|
||||
"updated_at": _dt.datetime.now(_dt.timezone.utc)},
|
||||
]
|
||||
app = _make_app(_mock_pg_store(aggregates=agg))
|
||||
with TestClient(app) as client:
|
||||
_login(client)
|
||||
r = client.get("/api/operator/failure-patterns")
|
||||
assert r.status_code == 200
|
||||
|
||||
|
||||
def test_credentials_200_with_cookie():
|
||||
cred = {
|
||||
"id": "11111111-1111-1111-1111-111111111111",
|
||||
"learner_ref": "learner-1",
|
||||
"vc_type": "MasteryCredential",
|
||||
"status": "active",
|
||||
"issued_at": _dt.datetime.now(_dt.timezone.utc),
|
||||
"revoked_at": None,
|
||||
}
|
||||
app = _make_app(_mock_pg_store(credentials=[cred]))
|
||||
with TestClient(app) as client:
|
||||
_login(client)
|
||||
r = client.get("/api/operator/credentials")
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
assert len(body["credentials"]) == 1
|
||||
|
||||
|
||||
# ── Suppressed cells have value=null ───────────────────────────────────────
|
||||
|
||||
|
||||
def test_suppressed_cells_value_null():
|
||||
agg = [
|
||||
{"path": "p", "metric": "sessions_count",
|
||||
"window_start": _dt.date.today(), "window_end": _dt.date.today(),
|
||||
"value": None, "cell_count": 5, "cell_suppressed": True,
|
||||
"updated_at": _dt.datetime.now(_dt.timezone.utc)},
|
||||
]
|
||||
app = _make_app(_mock_pg_store(aggregates=agg))
|
||||
with TestClient(app) as client:
|
||||
_login(client)
|
||||
r = client.get("/api/operator/cohort")
|
||||
assert r.status_code == 200
|
||||
cell = r.json()["views"][0]["metrics"][0]
|
||||
assert cell["cell_suppressed"] is True
|
||||
assert cell["value"] is None
|
||||
|
||||
|
||||
# ── last_updated is max(updated_at) ────────────────────────────────────────
|
||||
|
||||
|
||||
def test_last_updated_is_max():
|
||||
t1 = _dt.datetime(2026, 8, 1, 12, 0, tzinfo=_dt.timezone.utc)
|
||||
t2 = _dt.datetime(2026, 8, 3, 12, 0, tzinfo=_dt.timezone.utc)
|
||||
agg = [
|
||||
{"path": "p", "metric": "sessions_count",
|
||||
"window_start": _dt.date.today(), "window_end": _dt.date.today(),
|
||||
"value": 1.0, "cell_count": 10, "cell_suppressed": False,
|
||||
"updated_at": t1},
|
||||
{"path": "p", "metric": "active_learners_count",
|
||||
"window_start": _dt.date.today(), "window_end": _dt.date.today(),
|
||||
"value": 10.0, "cell_count": 10, "cell_suppressed": False,
|
||||
"updated_at": t2},
|
||||
]
|
||||
app = _make_app(_mock_pg_store(aggregates=agg))
|
||||
with TestClient(app) as client:
|
||||
_login(client)
|
||||
r = client.get("/api/operator/cohort")
|
||||
assert r.status_code == 200
|
||||
assert r.json()["last_updated"] is not None
|
||||
|
||||
|
||||
# ── Credential revoke ──────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_credential_revoke_sets_status_revoked():
|
||||
cred = {
|
||||
"id": "22222222-2222-2222-2222-222222222222",
|
||||
"learner_ref": "learner-1",
|
||||
"vc_type": "MasteryCredential",
|
||||
"status": "active",
|
||||
"issued_at": _dt.datetime.now(_dt.timezone.utc),
|
||||
"revoked_at": None,
|
||||
}
|
||||
store = _mock_pg_store(credentials=[cred])
|
||||
app = _make_app(store)
|
||||
with TestClient(app) as client:
|
||||
_login(client)
|
||||
r = client.post("/api/operator/credentials/22222222-2222-2222-2222-222222222222/revoke")
|
||||
assert r.status_code == 200
|
||||
assert r.json()["status"] == "revoked"
|
||||
store.set_credential_status.assert_awaited_once_with(
|
||||
"22222222-2222-2222-2222-222222222222", "revoked",
|
||||
)
|
||||
|
||||
|
||||
def test_credential_revoke_404_unknown():
|
||||
store = _mock_pg_store(credentials=None)
|
||||
store.get_credential = AsyncMock(return_value=None)
|
||||
app = _make_app(store)
|
||||
with TestClient(app) as client:
|
||||
_login(client)
|
||||
r = client.post("/api/operator/credentials/nonexistent/revoke")
|
||||
assert r.status_code == 404
|
||||
|
||||
|
||||
# ── No per-learner data in cohort responses (R-DASH-02) ───────────────────
|
||||
|
||||
|
||||
def test_no_per_learner_data_in_cohort_response():
|
||||
agg = [
|
||||
{"path": "p", "metric": "sessions_count",
|
||||
"window_start": _dt.date.today(), "window_end": _dt.date.today(),
|
||||
"value": 10.0, "cell_count": 10, "cell_suppressed": False,
|
||||
"updated_at": _dt.datetime.now(_dt.timezone.utc)},
|
||||
]
|
||||
app = _make_app(_mock_pg_store(aggregates=agg))
|
||||
with TestClient(app) as client:
|
||||
_login(client)
|
||||
r = client.get("/api/operator/cohort")
|
||||
body_text = r.text
|
||||
# No per-learner refs in the response (only path + metric + aggregates).
|
||||
assert "learner-1" not in body_text
|
||||
assert "learner_ref" not in body_text
|
||||
|
||||
|
||||
# ── 503 when no Postgres ───────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_cohort_503_no_postgres():
|
||||
app = FastAPI()
|
||||
app.state.pg_store = None
|
||||
app.add_middleware(SessionMiddleware, secret_key="test-secret-1234567890abcdef")
|
||||
app.include_router(auth_router)
|
||||
app.include_router(cohort_router)
|
||||
with TestClient(app) as client:
|
||||
r = client.get("/api/operator/cohort")
|
||||
assert r.status_code == 503
|
||||
@@ -1,236 +0,0 @@
|
||||
"""P2 integration test — aggregation → endpoint → response (TASK-10-03).
|
||||
|
||||
Requires Postgres (skips if PRAXIS_PG_DSN not set). End-to-end:
|
||||
1. Seed 15 mock sessions (12 distinct learners — above k-anon threshold).
|
||||
2. Run the aggregation hook for each → cohort_aggregates populated.
|
||||
3. GET /api/operator/cohort (with auth cookie) → non-suppressed cells.
|
||||
4. Seed 5 sessions (5 NEW learners) for a different path → suppressed cells.
|
||||
5. Run nightly reconciliation → all windows recomputed → last_updated updated.
|
||||
6. GET /api/operator/mastery → mastery progression data.
|
||||
7. GET /api/operator/failure-patterns → failure pattern data.
|
||||
8. Verify last_updated ≤ 24h old (REQ-NFR-DASH-02).
|
||||
|
||||
G-038 differencing-attack e2e: also verified at the API layer here.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import datetime as _dt
|
||||
import os
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not os.environ.get("PRAXIS_PG_DSN"),
|
||||
reason="PRAXIS_PG_DSN not set — P2 aggregation integration tests skipped.",
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def pg_pool():
|
||||
import asyncpg
|
||||
|
||||
pool = await asyncpg.create_pool(
|
||||
dsn=os.environ["PRAXIS_PG_DSN"], min_size=1, max_size=5, command_timeout=10,
|
||||
)
|
||||
try:
|
||||
yield pool
|
||||
finally:
|
||||
await pool.close()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def pg_store(pg_pool):
|
||||
from db.pg_migrate import apply_pg_migrations
|
||||
from db.pg_store import PgStore
|
||||
|
||||
await apply_pg_migrations(pg_pool)
|
||||
# Clean cohort_aggregates + operators for an isolated run.
|
||||
async with pg_pool.acquire() as conn:
|
||||
await conn.execute("DELETE FROM cohort_aggregates")
|
||||
await conn.execute("DELETE FROM operators WHERE username = 'p2intop'")
|
||||
await conn.execute("DELETE FROM issued_credentials")
|
||||
return PgStore(pg_pool)
|
||||
|
||||
|
||||
def _session(learner_ref: str, path: str = "customer_service",
|
||||
outcome: str = "pass") -> dict:
|
||||
return {
|
||||
"learner_ref": learner_ref,
|
||||
"path": path,
|
||||
"scenario_id": f"{path}_v01",
|
||||
"outcome": outcome,
|
||||
"rubric_scores": [
|
||||
{"criterion_id": "empathy", "score": 4.0},
|
||||
{"criterion_id": "resolution", "score": 3.5},
|
||||
],
|
||||
"failure_mode": "missed_apology" if outcome == "fail" else None,
|
||||
"branch_path": ["accept"],
|
||||
"timestamp": _dt.datetime.now(_dt.timezone.utc).isoformat(),
|
||||
}
|
||||
|
||||
|
||||
async def _seed_and_aggregate(pg_store, sessions):
|
||||
from server.cohort.hook import on_session_end
|
||||
|
||||
for s in sessions:
|
||||
await on_session_end(pg_store, s)
|
||||
|
||||
|
||||
async def _login_cookie(client, pg_store) -> None:
|
||||
from server.auth.passwords import hash_password
|
||||
|
||||
op_id = await pg_store.insert_operator("p2intop", hash_password("pw"), "P2 Int")
|
||||
# Login via the test client.
|
||||
r = client.post("/api/operator/login", json={"username": "p2intop", "password": "pw"})
|
||||
assert r.status_code == 200, r.text
|
||||
|
||||
|
||||
def _make_client(pg_store):
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
from starlette.middleware.sessions import SessionMiddleware
|
||||
|
||||
from server.auth.rate_limit import reset_login_rate_limit
|
||||
from server.auth.routes import router as auth_router
|
||||
from server.operator.cohort import router as cohort_router
|
||||
from server.operator.credentials import router as credentials_router
|
||||
from server.operator.failure_patterns import router as failure_router
|
||||
from server.operator.mastery import router as mastery_router
|
||||
|
||||
reset_login_rate_limit()
|
||||
app = FastAPI()
|
||||
app.state.pg_store = pg_store
|
||||
app.add_middleware(SessionMiddleware, secret_key="test-secret-1234567890abcdef")
|
||||
app.include_router(auth_router)
|
||||
app.include_router(cohort_router)
|
||||
app.include_router(mastery_router)
|
||||
app.include_router(failure_router)
|
||||
app.include_router(credentials_router)
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
# ── Main e2e test ─────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_aggregation_to_endpoint_e2e(pg_store):
|
||||
"""12 distinct learners → non-suppressed; 5 distinct → suppressed."""
|
||||
# 1. Seed 12 distinct learners across 15 sessions for 'customer_service'.
|
||||
sessions = []
|
||||
for i in range(12):
|
||||
sessions.append(_session(f"learner-{i}", "customer_service", "pass"))
|
||||
for i in range(3):
|
||||
sessions.append(_session(f"learner-{i}", "customer_service", "fail"))
|
||||
await _seed_and_aggregate(pg_store, sessions)
|
||||
|
||||
# 2. Seed 5 distinct learners for 'sales' (below threshold).
|
||||
sales_sessions = [_session(f"sales-{i}", "sales", "pass") for i in range(5)]
|
||||
await _seed_and_aggregate(pg_store, sales_sessions)
|
||||
|
||||
client = _make_client(pg_store)
|
||||
with client:
|
||||
await _login_cookie(client, pg_store)
|
||||
|
||||
# 3. GET /api/operator/cohort → non-suppressed for customer_service.
|
||||
r = client.get("/api/operator/cohort")
|
||||
assert r.status_code == 200, r.text
|
||||
body = r.json()
|
||||
paths = {v["path"] for v in body["views"]}
|
||||
assert "customer_service" in paths
|
||||
# 4. sales path cells should be suppressed (5 < 10).
|
||||
sales_view = next((v for v in body["views"] if v["path"] == "sales"), None)
|
||||
if sales_view:
|
||||
suppressed = [c for c in sales_view["metrics"] if c["cell_suppressed"]]
|
||||
assert suppressed, "sales (5 learners) must be suppressed"
|
||||
|
||||
# customer_service (12 learners) should have non-suppressed cells.
|
||||
cs_view = next((v for v in body["views"] if v["path"] == "customer_service"), None)
|
||||
assert cs_view is not None
|
||||
non_suppressed = [c for c in cs_view["metrics"] if not c["cell_suppressed"]]
|
||||
assert non_suppressed, "customer_service (12 learners) should have non-suppressed cells"
|
||||
|
||||
# 6. GET /api/operator/mastery
|
||||
r = client.get("/api/operator/mastery")
|
||||
assert r.status_code == 200
|
||||
|
||||
# 7. GET /api/operator/failure-patterns
|
||||
r = client.get("/api/operator/failure-patterns")
|
||||
assert r.status_code == 200
|
||||
|
||||
# 8. last_updated ≤ 24h (REQ-NFR-DASH-02)
|
||||
if body.get("last_updated"):
|
||||
ts = _dt.datetime.fromisoformat(body["last_updated"].replace("Z", "+00:00"))
|
||||
age = _dt.datetime.now(_dt.timezone.utc) - ts
|
||||
assert age < _dt.timedelta(hours=24), "freshness must be ≤ 24h"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_nightly_reconciliation_updates_last_updated(pg_store):
|
||||
from server.cohort.nightly import NightlyScheduler
|
||||
|
||||
# Seed a few events via the aggregation hook first.
|
||||
sessions = [_session(f"r-learner-{i}", "recon_path", "pass") for i in range(11)]
|
||||
await _seed_and_aggregate(pg_store, sessions)
|
||||
|
||||
# Run nightly reconciliation.
|
||||
sched = NightlyScheduler()
|
||||
# mastery_gate_events is the source for nightly — seed a gate event.
|
||||
async with pg_store.pool.acquire() as conn:
|
||||
await conn.execute("DELETE FROM mastery_gate_events")
|
||||
for i in range(11):
|
||||
await conn.execute(
|
||||
"INSERT INTO mastery_gate_events (learner_ref, scenario_id, path_id, "
|
||||
"gate_outcome, rubric_scores_jsonb, source) "
|
||||
"VALUES ($1, $2, $3, $4, $5::jsonb, 'sync')",
|
||||
f"r-learner-{i}", "recon_v01", "recon_path", "open",
|
||||
'[{"criterion_id":"empathy","score":4.0}]',
|
||||
)
|
||||
await sched.reconcile_now(pg_store)
|
||||
|
||||
client = _make_client(pg_store)
|
||||
with client:
|
||||
await _login_cookie(client, pg_store)
|
||||
r = client.get("/api/operator/cohort")
|
||||
assert r.status_code == 200
|
||||
# last_updated should be very recent after reconciliation.
|
||||
body = r.json()
|
||||
if body.get("last_updated"):
|
||||
ts = _dt.datetime.fromisoformat(body["last_updated"].replace("Z", "+00:00"))
|
||||
age = _dt.datetime.now(_dt.timezone.utc) - ts
|
||||
assert age < _dt.timedelta(minutes=1), "nightly reconcile should refresh last_updated"
|
||||
|
||||
|
||||
# ── G-038 e2e: differencing-attack at the API layer ────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_g038_differencing_attack_api_layer(pg_store):
|
||||
"""G-038: 10 learners in window A, 9 in window B. Verify GET /cohort
|
||||
cannot isolate the dropped learner — window B is fully suppressed."""
|
||||
# Window A: 10 learners on path 'diff_a'.
|
||||
a_sessions = [_session(f"a-{i}", "diff_a", "pass") for i in range(10)]
|
||||
await _seed_and_aggregate(pg_store, a_sessions)
|
||||
|
||||
# Window B: 9 learners on path 'diff_b' (learner a-9 dropped).
|
||||
b_sessions = [_session(f"a-{i}", "diff_b", "pass") for i in range(9)]
|
||||
await _seed_and_aggregate(pg_store, b_sessions)
|
||||
|
||||
client = _make_client(pg_store)
|
||||
with client:
|
||||
await _login_cookie(client, pg_store)
|
||||
r = client.get("/api/operator/cohort")
|
||||
assert r.status_code == 200
|
||||
body_text = r.text
|
||||
# The dropped learner's ref must not appear anywhere in the response.
|
||||
assert "a-9" not in body_text, "dropped learner must not be isolatable via API"
|
||||
|
||||
# diff_b cells must all be suppressed (9 < 10).
|
||||
body = r.json()
|
||||
diff_b = next((v for v in body["views"] if v["path"] == "diff_b"), None)
|
||||
assert diff_b is not None
|
||||
for c in diff_b["metrics"]:
|
||||
assert c["cell_suppressed"] is True, "window B (9 learners) must be fully suppressed"
|
||||
assert c["value"] is None
|
||||
@@ -1,128 +0,0 @@
|
||||
"""P2 integration test — SPA fallback + voice UI coexist (TASK-10-04, G-041).
|
||||
|
||||
Tests against the running app (TestClient). Verifies:
|
||||
1. GET / → 200 text/html with <div id="root"> (voice UI loads).
|
||||
2. GET /operator/dashboard → 200 text/html (SPA fallback serves index.html).
|
||||
3. GET /operator/login → 200 text/html (SPA fallback).
|
||||
4. GET /api/operator/cohort → JSON (API route, not SPA fallback).
|
||||
5. GET /health → JSON (API route).
|
||||
6. GET /pipecat/webrtc → 405 (POST only, route exists — not SPA fallback).
|
||||
7. GET /vc/verify/nonexistent → 404 (API route, not SPA fallback).
|
||||
8. GET /assets/index.js → served by StaticFiles (not SPA fallback).
|
||||
|
||||
R-DASH-03 verified: SPA fallback serves index.html for client-side routes;
|
||||
API routes + StaticFiles assets are unaffected. R-DASH-05: voice UI at /
|
||||
unchanged.
|
||||
|
||||
G-041: the SPA fallback uses a custom StaticFiles subclass (SpaStaticFiles),
|
||||
NOT a catch-all route — assets are served normally, index.html is the
|
||||
fallback only for non-file paths.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client_with_dist(tmp_path):
|
||||
"""Build a client/dist with index.html + an asset, then import the app."""
|
||||
dist = tmp_path / "dist"
|
||||
dist.mkdir()
|
||||
(dist / "index.html").write_text(
|
||||
'<!doctype html><html><body><div id="root"></div></body></html>',
|
||||
encoding="utf-8",
|
||||
)
|
||||
assets = dist / "assets"
|
||||
assets.mkdir()
|
||||
(assets / "index.js").write_text("console.log('app');", encoding="utf-8")
|
||||
|
||||
# Set the env var + reload the app module so the StaticFiles mount sees it.
|
||||
os.environ["PRAXIS_CLIENT_DIST"] = str(dist)
|
||||
os.environ["PRAXIS_COOKIE_SECRET"] = "x" * 48
|
||||
os.environ["PRAXIS_COOKIE_SECURE"] = "false"
|
||||
# Drop any PG DSN so we don't try to connect during the lifespan.
|
||||
os.environ.pop("PRAXIS_PG_DSN", None)
|
||||
|
||||
import importlib
|
||||
import server.__main__ as main_mod
|
||||
|
||||
importlib.reload(main_mod)
|
||||
with TestClient(main_mod.app) as c:
|
||||
yield c
|
||||
|
||||
# Cleanup env.
|
||||
os.environ.pop("PRAXIS_CLIENT_DIST", None)
|
||||
|
||||
|
||||
def test_root_serves_voice_ui(client_with_dist):
|
||||
r = client_with_dist.get("/")
|
||||
assert r.status_code == 200
|
||||
assert "text/html" in r.headers.get("content-type", "")
|
||||
assert "<div id=\"root\">" in r.text
|
||||
|
||||
|
||||
def test_operator_dashboard_spa_fallback(client_with_dist):
|
||||
r = client_with_dist.get("/operator/dashboard")
|
||||
assert r.status_code == 200
|
||||
assert "text/html" in r.headers.get("content-type", "")
|
||||
assert "<div id=\"root\">" in r.text
|
||||
|
||||
|
||||
def test_operator_login_spa_fallback(client_with_dist):
|
||||
r = client_with_dist.get("/operator/login")
|
||||
assert r.status_code == 200
|
||||
assert "text/html" in r.headers.get("content-type", "")
|
||||
assert "<div id=\"root\">" in r.text
|
||||
|
||||
|
||||
def test_api_operator_cohort_is_json_not_html(client_with_dist):
|
||||
# Without auth → 401 JSON (not index.html). Proves the API route wins.
|
||||
r = client_with_dist.get("/api/operator/cohort")
|
||||
assert r.status_code in (401, 503)
|
||||
assert "application/json" in r.headers.get("content-type", "")
|
||||
# Critically NOT html.
|
||||
assert "<div id=\"root\">" not in r.text
|
||||
|
||||
|
||||
def test_health_is_json(client_with_dist):
|
||||
r = client_with_dist.get("/health")
|
||||
assert r.status_code == 200
|
||||
assert "application/json" in r.headers.get("content-type", "")
|
||||
|
||||
|
||||
def test_pipecat_webrtc_post_route_exists(client_with_dist):
|
||||
# The POST route exists and responds (not index.html). A GET falls through
|
||||
# to the SPA fallback (serves index.html) — acceptable: the POST route is
|
||||
# the real voice-loop entrypoint; a GET is a client-side navigation attempt.
|
||||
# We assert the POST route is wired (returns 4xx/5xx, not HTML).
|
||||
r = client_with_dist.post("/pipecat/webrtc", json={"sdp": "", "type": "offer"})
|
||||
assert r.status_code in (400, 422, 500)
|
||||
assert "<div id=\"root\">" not in r.text
|
||||
|
||||
|
||||
def test_vc_verify_nonexistent_is_404(client_with_dist):
|
||||
r = client_with_dist.get("/vc/verify/nonexistent-id-xyz")
|
||||
assert r.status_code == 404
|
||||
assert "application/json" in r.headers.get("content-type", "")
|
||||
assert "<div id=\"root\">" not in r.text
|
||||
|
||||
|
||||
def test_assets_served_by_staticfiles_not_spa_fallback(client_with_dist):
|
||||
r = client_with_dist.get("/assets/index.js")
|
||||
assert r.status_code == 200
|
||||
ct = r.headers.get("content-type", "")
|
||||
assert "javascript" in ct or "text/plain" in ct
|
||||
assert "console.log" in r.text
|
||||
|
||||
|
||||
def test_unknown_non_asset_path_serves_index_html(client_with_dist):
|
||||
"""An unknown path that is NOT an asset + NOT an API route → SPA fallback."""
|
||||
r = client_with_dist.get("/some/unknown/route")
|
||||
assert r.status_code == 200
|
||||
assert "<div id=\"root\">" in r.text
|
||||
Reference in New Issue
Block a user