Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Praxis CI
|
dc673e5e3d |
+135
-1
@@ -344,4 +344,138 @@ Proxmox invokes hookscript at post-start phase (runs on PVE HOST):
|
||||
| R-DEPLOY-03 | CT can't reach Gitea/apt mirrors | Validate internet access; fallback to host-clone+pct-push (D-025 hybrid) |
|
||||
| R-DEPLOY-04 | Docker-in-LXC on ZFS rootfs | Check storage type; use local (directory) if ZFS |
|
||||
| R-DEPLOY-05 | journald log flooding from compose up | Log rotation or StandardOutput=null for pilot |
|
||||
| R-DEPLOY-06 | First-boot build > 5 min (NFR breach) | Pre-build on host + docker load fallback |
|
||||
| R-DEPLOY-06 | First-boot build > 5 min (NFR breach) | Pre-build on host + docker load fallback |
|
||||
|
||||
---
|
||||
|
||||
## v0.3 Architecture (Mastery Scoring + Competency Rubrics + VC + Cohort Dashboard)
|
||||
|
||||
> **Status:** Research-refined (v0.3 RESEARCH stage). Informed by `.ciagent/RESEARCH.md` v0.3 section.
|
||||
> **Decisions:** D-031 (operator tier, overrides D-007 for operator surface), D-032 (mastery gate), D-033 (W3C VC 2.0), D-034 (k-anonymity), D-035 (IRT 1PL), D-036 (scenario library), D-037 (path structure), D-038..D-049 (clarify).
|
||||
|
||||
### Hybrid Storage Topology (D-031)
|
||||
|
||||
Learner-local state stays in SQLite (D-007 preserved); operator-tier state goes to a new Postgres service. The two stores never share a session and never join via cross-DB FKs (`learner_ref` is an opaque string in Postgres).
|
||||
|
||||
```
|
||||
LXC Container (from v0.2, memory bumped 4GB → 6GB)
|
||||
Docker daemon
|
||||
├── praxis container (existing v0.2 + v0.3 additions)
|
||||
│ ├─ uvicorn 0.0.0.0:8789
|
||||
│ ├─ GET /health (v0.2)
|
||||
│ ├─ POST /pipecat/webrtc (v0.2)
|
||||
│ ├─ GET / ... StaticFiles (v0.2)
|
||||
│ ├─ /api/operator/* NEW (v0.3 — operator auth gate)
|
||||
│ ├─ /vc/verify/<id> NEW (v0.3 — public, unauthenticated)
|
||||
│ ├─ SQLite /app/data/praxis.db (v0.2 + NEW v0.3 tables: learner_ability, mastery_progress)
|
||||
│ └─ Postgres pool (asyncpg) (v0.3 — operator tier)
|
||||
│
|
||||
└── postgres container NEW (v0.3)
|
||||
├─ postgres:16-slim
|
||||
├─ pgdata named volume
|
||||
├─ internal Docker network only (no published port)
|
||||
├─ pg_isready healthcheck
|
||||
└─ Tables: operators, issued_credentials, mastery_gate_events, cohort_aggregates, issuer_keys
|
||||
```
|
||||
|
||||
### v0.3 Component Map (additions to v0.2)
|
||||
|
||||
```
|
||||
Pipecat server (Python)
|
||||
├─ ... (v0.2 voice loop unchanged) ...
|
||||
├─ Rubric engine NEW (server/mastery/)
|
||||
│ ├─ rubric_loader.py (rubrics/<skill>.yaml → Pydantic)
|
||||
│ ├─ rubric_scorer.py (rule-based: signals → 1-5, deterministic — REQ-NFR-MAST-01)
|
||||
│ ├─ evidence_extractor.py (LLM extracts quotes+signals, temp=0, JSON-schema)
|
||||
│ └─ mastery_score.py (weighted mean + conjunctive floor + path gate)
|
||||
├─ IRT engine NEW (server/mastery/irt.py)
|
||||
│ ├─ 1PL/Rasch: P(success) = logistic(θ − b)
|
||||
│ ├─ Bayesian θ update per session (<100ms — REQ-NFR-IRT-01)
|
||||
│ └─ θ persisted to SQLite learner_ability (D-046)
|
||||
├─ Scenario library NEW (server/scenarios/library.py)
|
||||
│ ├─ scenarios/<path>/<id>.yaml + scenarios/index.yaml (semver, rubric_criteria mapping)
|
||||
│ └─ AI variation review pipeline (_pending/ → expert review → library)
|
||||
├─ Path engine NEW (server/paths/)
|
||||
│ ├─ paths/<slug>.yaml (6-week structure, mastery gates — D-037)
|
||||
│ └─ progression: current_week advances on gate-open (D-048)
|
||||
├─ VC issuer NEW (server/vc/)
|
||||
│ ├─ issuer.py (Ed25519, pynacl + canonicaljson + base58, eddsa-jcs-2022)
|
||||
│ ├─ status_list.py (Bitstring Status List v1.0)
|
||||
│ ├─ verification.py (public GET /vc/verify/<id> — D-043)
|
||||
│ └─ issuer key in Postgres issuer_keys (encrypted at rest)
|
||||
├─ Operator auth NEW (server/auth/)
|
||||
│ ├─ SessionMiddleware (Starlette, itsdangerous-signed cookie — D-041)
|
||||
│ ├─ argon2id passwords (argon2-cffi)
|
||||
│ ├─ current_operator Depends
|
||||
│ └─ slowapi 5/min login rate-limit
|
||||
├─ Cohort aggregation NEW (server/cohort/)
|
||||
│ ├─ on-session-end hook → k-anonymized aggregate upsert to Postgres (D-045)
|
||||
│ └─ nightly reconciliation job (cron in praxis service)
|
||||
└─ Operator API NEW (server/operator/)
|
||||
├─ /api/operator/login, /api/operator/logout
|
||||
├─ /api/operator/cohort (k-anonymized, ≥10 learners/cell — D-034)
|
||||
└─ /api/operator/credentials (issued VCs, revocation)
|
||||
|
||||
Client (React)
|
||||
├─ ... (v0.2 voice UI unchanged) ...
|
||||
└─ /operator/* NEW (v0.3 — cohort dashboard UI, auth-gated — D-044)
|
||||
```
|
||||
|
||||
### Mastery Scoring Flow (off the voice path)
|
||||
|
||||
```
|
||||
Session end (server/session_recorder.py)
|
||||
│
|
||||
├─ 1. Evidence extraction (LLM, async, off-voice-path)
|
||||
│ deepseek-v4-flash:cloud, temp=0
|
||||
│ Input: session turns + scenario.rubric_criteria
|
||||
│ Output (JSON-schema-validated): [{criterion_id, quote, signals: [...]}]
|
||||
│ Guard: fuzzy-match quote vs transcript → reject+re-extract on mismatch (R-MAST-02)
|
||||
│
|
||||
├─ 2. Rule-based scoring (deterministic, no LLM — REQ-NFR-MAST-01)
|
||||
│ rubric_scorer.py: signals → 1-5 level per criterion
|
||||
│
|
||||
├─ 3. Mastery Score (deterministic)
|
||||
│ scenario_score = weighted_mean(levels, weights)
|
||||
│ scenario_pass = scenario_score ≥ 3.0 AND every criterion ≥ 2 (conjunctive floor)
|
||||
│ path MasteryScore = mean(scenario_scores for passing scenarios only)
|
||||
│ path gate open = ≥3 distinct scenarios passed AND MasteryScore ≥ 3.5 (D-032)
|
||||
│
|
||||
├─ 4. IRT θ update (deterministic, <100ms — REQ-NFR-IRT-01)
|
||||
│ θ ← θ + (outcome − P) × σ²/(σ² + 1); persist to SQLite learner_ability (D-046)
|
||||
│
|
||||
├─ 5. Progression (deterministic)
|
||||
│ gate open → advance current_week (D-048)
|
||||
│ week-final gate open → issue VC (REQ-MAST-03)
|
||||
│ record mastery_gate_event in Postgres (REQ-NFR-MAST-02)
|
||||
│
|
||||
└─ 6. Cohort aggregation (async, k-anonymized)
|
||||
on-session-end hook → upsert k-anonymized aggregate to Postgres (D-045)
|
||||
nightly reconciliation reconciles 7-day windows
|
||||
```
|
||||
|
||||
### VC Issuance + Verification Flow
|
||||
|
||||
```
|
||||
Mastery gate opens (week-final)
|
||||
├─ issuer.py: build payload {scenariosPassed, rubricScore, completedWeeks:6, evidence, validUntil:+3y}
|
||||
│ canonicalize (JCS) → sign Ed25519 → store in Postgres issued_credentials
|
||||
└─ Verification (third party): GET /vc/verify/<id> → fetch pubkey from verificationMethod URL
|
||||
→ validate Ed25519 sig → check Status List → return {valid, status, issuer, mastery, verifiedAt}
|
||||
```
|
||||
|
||||
### Postgres Schema (operator tier — D-040)
|
||||
|
||||
Tables: `operators` (id, username, password_hash argon2id), `issued_credentials` (id, learner_ref opaque-string, vc_payload_json, signature_b64, status, issued_at), `mastery_gate_events` (id, learner_ref, path, week, scenarios_passed_json, rubric_scores_json, gate_opened_at — REQ-NFR-MAST-02 audit), `cohort_aggregates` (path, week, window_start/end, metric, value, cell_suppressed — k-anon via write-time suppression, weekly partitions), `issuer_keys` (id, public_key Multikey, private_key_enc, status active|superseded). `gen_random_uuid()` in PG16 (no extension). No cross-DB FKs.
|
||||
|
||||
### CT Resource Sizing (v0.3 bump)
|
||||
|
||||
| Resource | v0.2 | v0.3 | Rationale |
|
||||
|----------|------|------|-----------|
|
||||
| Memory | 4096 MB | **6144 MB** | Postgres ~1GB + praxis ~2GB + build headroom (R-MT-01) |
|
||||
| Rootfs | 16 GB | 16 GB | Postgres data on named volume, not rootfs |
|
||||
| CPU | 2 | 2-4 | Postgres + praxis concurrent; 2 floor, 4 preferred |
|
||||
|
||||
### v0.3 Risks (from RESEARCH.md)
|
||||
|
||||
Top risks for PLAN: R-MAST-01 (N=3 thin for credential → label formative), R-AUTH-01 (Secure cookie + no-TLS pilot), R-MT-01 (Postgres resource contention), R-VC-01 (custom VC code ~200 LOC), R-MAST-02 (LLM hallucinated quotes → fuzzy-match guard), R-IRT-01 (cold-start θ → fall back to scenario.difficulty until ≥5 sessions). Full table in RESEARCH.md.
|
||||
+10
-15
@@ -1,18 +1,13 @@
|
||||
{
|
||||
"phase": 2,
|
||||
"stage": "complete",
|
||||
"milestone": "v0.2",
|
||||
"phase_role": "final",
|
||||
"phase": 0,
|
||||
"stage": "grill",
|
||||
"milestone": "v0.3",
|
||||
"phase_role": "pre_execution",
|
||||
"attempts": 0,
|
||||
"updated_at": "2026-08-03T19:00:00Z",
|
||||
"release_status": "created",
|
||||
"release_url": "https://git.cloudinit.dev/coreci/praxis/releases/tag/v0.1.2",
|
||||
"tag": "v0.1.2",
|
||||
"milestone_complete": true,
|
||||
"milestone_merged_to_main": true,
|
||||
"next_milestone": "v0.3",
|
||||
"requirements": {
|
||||
"covered": ["REQ-DEPLOY-01", "REQ-DEPLOY-02", "REQ-DEPLOY-03", "REQ-DEPLOY-04", "REQ-DEPLOY-05", "REQ-DEPLOY-06", "REQ-DEPLOY-07", "REQ-DEPLOY-08", "REQ-DEPLOY-09", "REQ-DEPLOY-10", "REQ-DEPLOY-11", "REQ-DEPLOY-12", "REQ-DEPLOY-13", "REQ-DEPLOY-14", "REQ-DEPLOY-15", "REQ-DEPLOY-16", "REQ-NFR-DEPLOY-01", "REQ-NFR-DEPLOY-02", "REQ-NFR-DEPLOY-04"],
|
||||
"deferred": ["REQ-NFR-DEPLOY-03"]
|
||||
}
|
||||
"updated_at": "2026-08-03T20:05:00Z",
|
||||
"milestone_complete": false,
|
||||
"milestone_merged_to_main": false,
|
||||
"previous_milestone": "v0.2",
|
||||
"tag_line": "v0.1.x",
|
||||
"next_tag": "v0.1.3"
|
||||
}
|
||||
@@ -0,0 +1,291 @@
|
||||
# Praxis v0.3 CIAgent Plan — GRILL Verdict (Red-Team Review)
|
||||
|
||||
> **Reviewer:** adversarial technology executive (red-team)
|
||||
> **Subject:** v0.3 execution plan (Mastery Scoring + Competency Rubrics) — 2 phases, 15 slices, 70 tasks
|
||||
> **Stance:** plan is unfeasible, over-scoped, and too costly until evidence forces otherwise
|
||||
> **Date:** 2026-08-03
|
||||
> **Binding status:** This GRILL verdict must be cleared (MUSTs resolved, FIXs tracked) before EXECUTE is authorized.
|
||||
> **Artifacts reviewed:** PLAN.md, PROJECT.md, REQUIREMENTS.md, RESEARCH.md (v0.3 section), ARCHITECTURE.md (v0.3 section), ROADMAP.md
|
||||
|
||||
---
|
||||
|
||||
## Verdict Legend
|
||||
|
||||
- **MUST** — blocks execution until fixed. The plan cannot enter EXECUTE with this issue open.
|
||||
- **FIX** — fix during execution, non-blocking. Tracked as a P1 condition in VERIFY.
|
||||
- **ACCEPT** — proceed as-is. The evidence clears the challenge.
|
||||
|
||||
---
|
||||
|
||||
## Axis 1 — Feasibility
|
||||
|
||||
**Forcing question:** Can this actually be built in 2 execution phases (70 tasks)? Is the scope realistic for one milestone, or is it 2 milestones pretending to be one?
|
||||
|
||||
**Challenge:** The v0.3 scope spans *seven* independent subsystems (rubric/mastery engine, IRT, scenario library + ≥6 authored scenarios, 6-week path engine, W3C VC 2.0 issuer with Ed25519 + Status List, operator auth + argon2id + slowapi, Postgres-in-LXC + asyncpg, cohort dashboard + k-anonymity aggregation + React UI). This is not a milestone — it is a *program*. The PLAN.md phase-split rationale (lines 14-23) openly admits the scope "is too large for one execution phase" and splits into P1/P2, but both phases ship under the *same* v0.3 milestone tag (v0.1.6). The 70-task count is artificially compressed: SLICE-12 (VC issuer) is 6 tasks for a W3C VC 2.0 + Ed25519 + JCS + Bitstring Status List + public verification endpoint + key rotation — that is *at minimum* a 10-12 task slice on its own, and SLICE-13 (cohort aggregation with k-anonymity + nightly reconciliation + on-session-end hook) is similarly under-tasked at 4 tasks.
|
||||
|
||||
**Evidence:**
|
||||
- PLAN.md:14-23 — "The v0.3 scope … is too large for one execution phase."
|
||||
- PLAN.md:32, 334 — P1 = 38 tasks, P2 = 32 tasks, total 70 (excludes P3 review).
|
||||
- REQUIREMENTS.md:20-66 — 11 functional REQ-IDs + 9 NFRs = 20 active requirements, the largest single-milestone REQ surface in the project's history (v0.1 was ~14, v0.2 was 16 deploy + 4 NFR).
|
||||
- RESEARCH.md:769 (R-VC-01) — "No batteries-included Python VC lib → ~200 LOC custom code" — 200 LOC of custom crypto code is not a 6-task slice; it is a liability that demands more tests than the plan allocates (only 2 test tasks: TASK-12-05, TASK-12-06).
|
||||
- SLICE-13 (PLAN.md:516-546) — 4 tasks for: on-session-end hook, k-anonymity suppression SQL, nightly reconciliation cron, and tests. The nightly reconciliation job alone (recompute all 7-day windows from raw events, correct drift, idempotent upsert) is a 2-3 task effort.
|
||||
|
||||
**Binding verdict: FIX** — The plan *is* feasible as a 2-phase *program*, but only if it is honestly re-labeled. The milestone should ship as v0.3 (P1 mastery core, v0.1.4) and v0.3.1 (P2 operator tier, v0.1.5), with the v0.3 milestone release (v0.1.6) being the *merge* of two separately-shipped, separately-verified patches. Do not pretend P1+P2 is one milestone release. Additionally, re-task SLICE-12 and SLICE-13: add 2 tasks each (one for VC Status List edge cases + key rotation drill, one for reconciliation idempotency + race-condition test). This is non-blocking — the wave structure survives — but the task counts must be honest before EXECUTE.
|
||||
|
||||
---
|
||||
|
||||
## Axis 2 — Scope
|
||||
|
||||
**Forcing question:** Is REQ-DASH-01 (cohort dashboard + multi-tenant + auth) really v0.3, or was it correctly deferred in v0.1/v0.2 for a reason? Does D-031 (override D-007) open a Pandora's box?
|
||||
|
||||
**Challenge:** REQ-DASH-01 was explicitly deferred in v0.1 (REQUIREMENTS.md:152, "later/deferred") and v0.2. ROADMAP.md:94 places the "Employer / program dashboard" at **v0.8**. The v0.3 plan pulls it forward *three milestones* with the justification that mastery scoring "needs" the operator view. But mastery scoring (REQ-MAST-01/02) and VC issuance (REQ-MAST-03) work *without* a cohort dashboard — the dashboard is an *operator* feature, not a *learner* feature. D-031 overrides D-007 (single-learner/no-auth) and introduces a hybrid SQLite+Postgres topology, operator auth, argon2id, slowapi, asyncpg, a second Docker service, k-anonymity aggregation, and a React operator UI — *none* of which is required for the learner-facing mastery gate to function. This is scope creep dressed as a dependency.
|
||||
|
||||
**Evidence:**
|
||||
- ROADMAP.md:94 — "v0.8 | Employer / program dashboard" (original placement).
|
||||
- PROJECT.md:129 (D-031) — "overrides D-007 for the cohort-dashboard surface" — confidence 0.75, the *lowest*-confidence decision that expands scope.
|
||||
- REQUIREMENTS.md:44 (REQ-DASH-01) — "Forces multi-tenant + operator auth (D-031)" — the word "forces" is doing a lot of work. The mastery gate (REQ-MAST-02) does not depend on the dashboard.
|
||||
- PLAN.md:18 — P1 "works standalone (learner can practice, score, progress) without the operator tier." — *This is an admission that the operator tier is separable.*
|
||||
- PROJECT.md:147 (D-049) — failure-injection stays off, further confirming the learner-facing mastery layer is the *real* v0.3 deliverable.
|
||||
|
||||
**Binding verdict: MUST** — Split the milestone. Ship **v0.3 = P1 only** (mastery core + IRT + scenarios + paths + VC issuance, since VC issuance *is* triggered by the mastery gate and is learner-facing per D-048). Defer **REQ-DASH-01 + REQ-AUTH-01 + REQ-MT-01/02 + REQ-NFR-DASH-01/02 + REQ-NFR-AUTH-01 + REQ-NFR-MT-01** to **v0.4** (operator tier), restoring the original ROADMAP intent. D-031 does open a Pandora's box: every hybrid-DB system eventually faces the "which store is the source of truth?" question, and shipping it under a learner-milestone tag hides that risk. If the team insists on keeping the dashboard in v0.3, rebrand the milestone as "v0.3: Mastery + Operator Tier" and accept that this is a 2-milestone program — but the cleaner answer is to defer the dashboard.
|
||||
|
||||
---
|
||||
|
||||
## Axis 3 — Cost
|
||||
|
||||
**Forcing question:** What is the maintenance cost of Postgres-in-LXC, asyncpg, argon2, pynacl, slowapi, and ~200 LOC custom VC code? Is R-VC-01 (custom VC code) a liability vs using a library?
|
||||
|
||||
**Challenge:** The v0.3 dependency surface grows by *at least* 5 new pip packages (asyncpg, argon2-cffi, slowapi, pynacl, canonicaljson, base58 — actually 6) plus a Postgres service. Each is a CVE vector, a version-pin maintenance burden, and a CI complexity adder. The ~200 LOC custom VC code (R-VC-01) is the most concerning: cryptographic code written by an AI agent is a *liability* regardless of test coverage. The W3C VC 2.0 + eddsa-jcs-2022 cryptosuite has subtle canonicalization edge cases (e.g., JSON number representation, key ordering, URI normalization) that unit-test round-trips do *not* catch — only interop tests against an independent verifier do, and the plan has *zero* interop tests.
|
||||
|
||||
**Evidence:**
|
||||
- RESEARCH.md:769 (R-VC-01) — "~200 LOC custom code" — confidence 0.75. The mitigation is "unit-test signature/verify round-trip," which only proves the code is self-consistent, not that it is W3C-compliant.
|
||||
- PLAN.md:504-513 (TASK-12-05, TASK-12-06) — VC tests are sign/verify round-trip, tamper detection, JCS determinism, status list, revocation, key rotation. *No interop test against an external verifier* (e.g., Verifiable Credential JS verifier, Digital Credentials Verifier).
|
||||
- PROJECT.md:140 (D-042) — issuer key encrypted at rest with a root key from secrets. Key management is hand-rolled (init_issuer_key, encrypt, store, rotate). This is a security-engineer task, not a backend task, and the plan assigns it to security-engineer (good), but the *rotation drill* (D-042 "new key + old marked superseded") is not tested end-to-end except in TASK-12-06 which only checks "old VC still verifies against archived public key" — it does *not* test the operational rotation procedure (generate new key, archive old, re-sign new VCs, update verificationMethod URL).
|
||||
- RESEARCH.md:772 (R-MT-01) — Postgres-in-LXC resource contention, confidence 0.65 — the *lowest*-confidence technical risk. Memory bump to 6GB is a guess, not a measurement.
|
||||
|
||||
**Binding verdict: MUST** — Two conditions before EXECUTE:
|
||||
1. **Add a VC interop test** (TASK-12-07): verify a Praxis-issued VC against at least one *external* W3C VC verifier (e.g., the `digitalbazaar/vc-verifier` or a JS `@digitalcredentials/vc` verifier). Round-trip self-verification is insufficient for cryptographic claims. Without this, R-VC-01 is an unmitigated liability.
|
||||
2. **Add a key-rotation operational test** (TASK-12-08): end-to-end drill — issue N VCs with key A, rotate to key B, issue M VCs with key B, verify all N+M VCs still verify (N against archived key A, M against active key B), revoke one of each, verify revocation. This is the *one* crypto procedure that, if broken, silently invalidates every credential ever issued.
|
||||
|
||||
The Postgres/argon2/slowapi maintenance cost is **ACCEPT** — these are well-maintained, widely-used libraries. The liability is concentrated in the custom VC code.
|
||||
|
||||
---
|
||||
|
||||
## Axis 4 — Technical Risk
|
||||
|
||||
**Forcing question:** R-MAST-01 (N=3 thin for credential), R-AUTH-01 (Secure cookie + no TLS), R-MAST-02 (LLM hallucinated quotes), R-IRT-01 (cold start) — which are MUST-FIX before execution vs ACCEPT?
|
||||
|
||||
**Challenge:** The plan treats all four as "Open Questions Deferred to EXECUTE" (PLAN.md:687-693). That is insufficient. R-MAST-01 is a *credibility* risk: if the VC is labeled as a mastery credential and employers treat it as high-stakes, N=3 with G≈0.5-0.6 is defensible only if the credential is explicitly labeled *formative*. R-AUTH-01 is a *security* risk: relaxing the Secure cookie flag for a no-TLS pilot means session cookies travel in cleartext — if the operator bridge IP is on a shared network (vmbr0 DHCP), any host on the bridge can sniff the operator session. R-MAST-02 is the *highest*-confidence mitigation (fuzzy-match quotes), but the plan's fallback ("empty evidence + log warning") means a session could silently score as a *zero* with no learner-visible signal. R-IRT-01 is benign (cold-start fallback to fixed difficulty).
|
||||
|
||||
**Evidence:**
|
||||
- RESEARCH.md:766 (R-MAST-01) — confidence 0.62, *below* the 0.70 decision threshold. Mitigation: "Label v0.3 VC as formative." This label is *not* in the PLAN.md VC payload (TASK-12-02) or the REQ-MAST-03 requirement text.
|
||||
- RESEARCH.md:771 (R-AUTH-01) — "Secure cookie flag fails without TLS." PLAN.md:450 resolves this with `PRAXIS_COOKIE_SECURE=false` env default. This ships a known-insecure default.
|
||||
- PLAN.md:154 (TASK-03-01) — "on final failure, fall back to empty evidence + log warning." Empty evidence → rule scorer has no signals → every criterion scores level 1 (fail) → scenario fails → learner sees a failed session with *no explanation*. This is a UX and fairness bug.
|
||||
- RESEARCH.md:774 (R-IRT-01) — mitigation confidence 0.75, "fall back to scenario.difficulty until ≥5 observations." ACCEPT.
|
||||
|
||||
**Binding verdict: MUST** — Three conditions:
|
||||
1. **R-MAST-01**: Add `credentialTier: "formative"` (or equivalent) to the VC payload (TASK-12-02) and to the verification endpoint response (TASK-12-04). Update REQ-MAST-03 to require this label. Without it, the credential is misleading.
|
||||
2. **R-AUTH-01**: Do *not* ship `PRAXIS_COOKIE_SECURE=false` as a default. Either (a) require TLS for the operator surface (add a Traefik sidecar or Caddy in front of `/api/operator/*`), or (b) bind the operator surface to `127.0.0.1` only (loopback) so cookies never traverse the bridge. A cleartext cookie on a shared bridge is a MUST-FIX.
|
||||
3. **R-MAST-02**: Change the fallback in TASK-03-01 from "empty evidence + log warning" to "empty evidence → mark scenario as `scoring_inconclusive` → do not count toward gate, do not penalize learner, surface 'technical issue, please retry' in the debrief." A silent fail-to-zero is unacceptable.
|
||||
|
||||
R-IRT-01: **ACCEPT** — cold-start fallback is sound.
|
||||
|
||||
---
|
||||
|
||||
## Axis 5 — Requirements Coverage
|
||||
|
||||
**Forcing question:** Does the plan actually cover all 20 REQ-IDs, or are some hand-waved? Check the coverage matrix in PLAN.md against REQUIREMENTS.md.
|
||||
|
||||
**Challenge:** The PLAN.md coverage matrix (lines 657-683) claims "20 REQ-IDs covered, 0 partial, 0 deferred." Let me audit the suspicious ones.
|
||||
|
||||
**Evidence (audit):**
|
||||
|
||||
| REQ-ID | Claimed coverage | Actual coverage | Verdict |
|
||||
|--------|-----------------|-----------------|---------|
|
||||
| REQ-MAST-03 | P2 SLICE-12 "VC issuer" | SLICE-12 implements issuance + verification + revocation. But REQ-MAST-03 says "Issued when a mastery gate opens" — the *trigger* is in P1 SLICE-07 (TASK-07-01, "path_engine.check_gate + advance_week") and the *issuance* is in P2 SLICE-12. The P1→P2 handoff for VC issuance is not in any task — who calls `issuer.issue_credential()` when the week-final gate opens? TASK-07-01 says "(6) record mastery_gate_event" but does NOT call the VC issuer (VC issuer is P2). D-048 says "issue VC if week-final gate" but the plan splits the gate-open (P1) from the issuance (P2). **Gap: no task wires the P1 gate-open event to the P2 VC issuer.** | **FIX** — add a task (either in SLICE-07 or SLICE-12) that defines the P1→P2 VC-issuance contract: a `mastery_gate_events` row with `gate_opened_at` is the trigger; P2's VC issuer polls/receives this event and issues. |
|
||||
| REQ-NFR-MAST-02 | P1+P2 SLICE-07, 09 "gate auditability (SQLite + Postgres)" | SLICE-07 records the event in SQLite; SLICE-09 defines the Postgres `mastery_gate_events` table; but *no task mirrors* the SQLite event to Postgres. The "mirror" is implied but not tasked. | **FIX** — TASK-13-01 (cohort aggregation hook) should explicitly mirror `mastery_gate_events` from SQLite to Postgres, or add a dedicated mirroring task. |
|
||||
| REQ-SCEN-04 | P1 SLICE-02, 06 "expert-authored format + AI variation hooks" | SLICE-02 adds `generated_from` and `intent_hash` fields (the hook). SLICE-06 authors expert scenarios. But *no task implements the AI-variation review pipeline* (`_pending/` dir → expert review → library promotion). RESEARCH.md:758 describes it; PLAN.md does not task it. | **ACCEPT** — REQ-SCEN-04 says "AI-generated variations" with "expert review" — the *hook* is the schema field; the *pipeline* can be deferred. The plan is honest that AI variations are "in P2 or later" (SLICE-06 goal line 246). |
|
||||
| REQ-NFR-DASH-02 | P2 SLICE-13 "freshness ≤24h" | SLICE-13 has a nightly reconciliation job (TASK-13-03) at 02:00. If the on-session-end hook (TASK-13-01) fails or lags, freshness depends on the nightly job. ≤24h is satisfied *if* the nightly job runs. But there is no task for *monitoring* or *alerting* on job failure. | **FIX** — add a health check for the nightly job (log last-run timestamp, surface in operator dashboard or `/health`). Non-blocking. |
|
||||
| REQ-NFR-VC-02 | P2 SLICE-12 "revocation latency — within 1 sync of status list" | "1 sync" is undefined. Is it 1 sync of the status list blob? Is the status list in-memory or fetched on every verify? TASK-12-04 (verification endpoint) does not specify caching of the status list. | **FIX** — clarify in TASK-12-04: status list is fetched from Postgres on every verification (no cache), so revocation latency = next verify call. Non-blocking. |
|
||||
|
||||
**Binding verdict: FIX** — The coverage matrix is *mostly* honest (18/20 fully covered), but the P1→P2 VC-issuance wiring gap (REQ-MAST-03) is a real hole — without a task that defines the trigger contract, the VC issuer will be built but never called. Add the wiring task. The other three FIXs are minor clarifications.
|
||||
|
||||
---
|
||||
|
||||
## Axis 6 — Architecture
|
||||
|
||||
**Forcing question:** Is hybrid SQLite + Postgres (D-031) a maintainable pattern or a future migration nightmare? Is "no cross-DB joins" realistic for the cohort dashboard queries?
|
||||
|
||||
**Challenge:** Hybrid polyglot persistence is a *known* anti-pattern when the two stores hold related data and there is no canonical source of truth. Here, `mastery_gate_events` exists in *both* SQLite (P1, the learner's local record) and Postgres (P2, the operator audit log). Which is canonical? If they diverge (e.g., SQLite write succeeds, Postgres mirror fails due to pool exhaustion), the cohort dashboard shows *stale* data while the learner sees *correct* data — and there is no reconciliation except the nightly job (which recomputes from Postgres `mastery_gate_events`, not from SQLite). This means the nightly job recomputes from a *possibly-incomplete* Postgres copy. The "no cross-DB joins" rule is realistic *only* if the cohort dashboard never needs to join learner-local data (e.g., θ distribution by path) with operator data — but the dashboard's "progression" and "failure patterns" views implicitly need *both* the learner's session outcomes (SQLite) and the operator's aggregate view (Postgres). The plan resolves this by aggregating at session-end (writing the aggregate to Postgres), so the dashboard reads *only* Postgres — but this means the aggregate is a *derived* copy, and the "no cross-DB joins" rule is maintained by *duplicating data*, not by query-time joins. This is workable but fragile.
|
||||
|
||||
**Evidence:**
|
||||
- ARCHITECTURE.md:356-379 — "The two stores never share a session and never join via cross-DB FKs (`learner_ref` is an opaque string in Postgres)." — the design is clean *if* the mirror is reliable.
|
||||
- RESEARCH.md:746 — "No cross-DB joins via `learner_ref` — `learner_ref` is an opaque string, not a FK." — correct, but `learner_ref` is still a *logical* join key. If the SQLite learner is deleted and re-created, the Postgres `learner_ref` dangles.
|
||||
- PLAN.md:528-529 (TASK-13-01) — "after P1's mastery hooks fire, call `aggregator.upsert_aggregate(...)`" — this is a *synchronous* call after the SQLite write, in the session-end path. If Postgres is down, does the session-end fail? The plan does not specify failure semantics.
|
||||
- RESEARCH.md:748 — migration strategy: "SQLite volume untouched → learner path never regresses." Good, but the *operator* path regresses if Postgres is down.
|
||||
|
||||
**Binding verdict: FIX** — Three conditions:
|
||||
1. **Define failure semantics for the Postgres mirror** (in TASK-13-01): if `upsert_aggregate` fails (Postgres down, pool exhausted), the learner session-end must *still succeed* (SQLite write is canonical for the learner). The aggregate failure is logged and reconciled by the nightly job. This makes SQLite the *learner-canonical* store and Postgres the *operator-derived* store — state this explicitly in ARCHITECTURE.md.
|
||||
2. **Make `learner_ref` a stable, opaque, non-reusable identifier** (e.g., a UUID generated once and stored in SQLite, never reused). Add this to TASK-02-01 or a new task. Without it, the "no FK" rule is a leaky abstraction.
|
||||
3. **Add a Postgres-readiness guard to the operator API**: if Postgres is down, `/api/operator/cohort/*` returns 503 (not 500 with a stack trace). Add to TASK-14-02.
|
||||
|
||||
The hybrid pattern is **ACCEPT** *with* these conditions — it is the correct pilot choice (don't migrate learner state to Postgres prematurely), but the failure semantics must be explicit.
|
||||
|
||||
---
|
||||
|
||||
## Axis 7 — Testing
|
||||
|
||||
**Forcing question:** 70 tasks, but how many have tests? Is the test strategy (mocked LLM for evidence extraction, testcontainers for Postgres) viable, or are there untestable critical paths?
|
||||
|
||||
**Challenge:** Let me count test tasks across the plan.
|
||||
|
||||
**Evidence (test task audit):**
|
||||
|
||||
| Slice | Tasks | Test tasks | Test ratio |
|
||||
|-------|-------|-----------|------------|
|
||||
| SLICE-01 | 4 | 1 (TASK-01-04) | 25% |
|
||||
| SLICE-02 | 4 | 1 (TASK-02-04) | 25% |
|
||||
| SLICE-03 | 5 | 2 (TASK-03-04, 03-05) | 40% |
|
||||
| SLICE-04 | 4 | 2 (TASK-04-03, 04-04) | 50% |
|
||||
| SLICE-05 | 4 | 1 (TASK-05-04) | 25% |
|
||||
| SLICE-06 | 3 | 1 (TASK-06-03) | 33% |
|
||||
| SLICE-07 | 4 | 2 (TASK-07-03, 07-04) | 50% |
|
||||
| SLICE-08 | 3 | 3 (all test/verification) | 100% |
|
||||
| SLICE-09 | 4 | 1 (TASK-09-04) | 25% |
|
||||
| SLICE-10 | 3 | 0 | 0% — infra, acceptable |
|
||||
| SLICE-11 | 5 | 2 (TASK-11-04, 11-05) | 40% |
|
||||
| SLICE-12 | 6 | 2 (TASK-12-05, 12-06) | 33% — *too low for crypto code* |
|
||||
| SLICE-13 | 4 | 1 (TASK-13-04) | 25% — *too low for k-anonymity* |
|
||||
| SLICE-14 | 4 | 1 (TASK-14-04) | 25% |
|
||||
| SLICE-15 | 4 | 1 (TASK-15-04) | 25% |
|
||||
| SLICE-16 | 3 | 3 (all integration/verification) | 100% |
|
||||
| **Total** | **70** | **24** | **34%** |
|
||||
|
||||
**Critical untestable paths:**
|
||||
1. **LLM evidence extraction (TASK-03-01)** — the plan mocks the LLM (good for unit tests), but there is *no* test that runs against the *real* LLM with a real transcript. A mocked LLM proves the scoring logic, not that the extraction prompt works. This is a *fundamentally untestable in CI* path — the only test is manual/staging.
|
||||
2. **k-anonymity suppression (TASK-13-02)** — the test (TASK-13-04) checks "cell with 9 learners → suppressed, 10 → shown." But it does *not* test the differencing attack (comparing two adjacent windows to re-identify a learner who appears in one but not the other). RESEARCH.md:754 says "limit to pre-defined 2-D views to block differencing attacks" — but there is no test that the API *enforces* only pre-defined views (i.e., that an operator cannot request an arbitrary `path × week × outcome` 3-D view).
|
||||
3. **Nightly reconciliation (TASK-13-03)** — no test for "reconciliation corrects drift." TASK-13-04 tests "reconciliation correctness" but not *drift correction* (insert a bad aggregate, run reconcile, verify it's fixed).
|
||||
4. **VC verification endpoint (TASK-12-04)** — tested via TASK-12-06, but only with Praxis-issued VCs. No interop test (see Axis 3).
|
||||
|
||||
**Binding verdict: FIX** — Four conditions:
|
||||
1. **Add a real-LLM smoke test** (in SLICE-08 or SLICE-16): run one session transcript through the *actual* deepseek-v4-flash:cloud evidence extractor and verify the output is valid JSON with fuzzy-matching quotes. This runs only in staging (requires OLLAMA_API_KEY), gated by an env flag. The mocked-LLM tests stay in CI.
|
||||
2. **Add a k-anonymity differencing-attack test** (TASK-13-04 extension): verify that the cohort API rejects arbitrary 3-D view requests, and that two adjacent 7-day windows cannot re-identify a single learner appearing in only one.
|
||||
3. **Add a reconciliation drift-correction test** (TASK-13-04 extension): insert a deliberately-wrong aggregate, run `reconcile_cohort()`, verify it's corrected.
|
||||
4. **Add the VC interop test** (per Axis 3, MUST condition).
|
||||
|
||||
The mocked-LLM + testcontainers strategy is **ACCEPT** *for CI*. The gaps are in *integration* and *security* testing, not unit testing.
|
||||
|
||||
---
|
||||
|
||||
## Axis 8 — Phase Split
|
||||
|
||||
**Forcing question:** Is P1/P2 the right split? Should VC issuance (P2 SLICE-12) be in P1 with mastery gates (P1 SLICE-07) since they trigger on the same event? Is the P1→P2 dependency clean?
|
||||
|
||||
**Challenge:** The plan splits VC issuance (P2) from mastery-gate-open (P1) even though D-048 says "issue VC if week-final gate." This means P1 ships (v0.1.4) with mastery gates that open but *no credential is issued* — the learner reaches mastery and gets... nothing portable. The VC issuer arrives in P2 (v0.1.5). This is a *user-visible gap*: a learner who completes the path in v0.1.4 has no credential. The plan's phase-split rationale (lines 14-23) says P1 "works standalone (learner can practice, score, progress) without the operator tier" — but VC issuance is *not* the operator tier; it is a learner-facing consequence of mastery (D-048). VC issuance should be in P1.
|
||||
|
||||
Conversely, the operator auth + Postgres + cohort dashboard is correctly P2 — those are operator-tier.
|
||||
|
||||
**Evidence:**
|
||||
- PROJECT.md:146 (D-048) — "issue VC if week-final gate" — VC issuance is a *mastery-gate consequence*, not an operator feature.
|
||||
- PLAN.md:18 — "P1 works standalone" — but "standalone" here silently drops the VC, which is a REQ-MAST-03 requirement.
|
||||
- PLAN.md:663 (coverage matrix) — REQ-MAST-03 is listed as P2 SLICE-12. But REQ-MAST-03 is a *mastery* requirement, not an *operator* requirement.
|
||||
- PLAN.md:282 (TASK-07-01) — P1 session-end hook does steps 1-6 but step 6 is "record mastery_gate_event" — no VC issuance call. The VC issuance is orphaned in P2 with no trigger from P1.
|
||||
|
||||
**Binding verdict: MUST** — Move SLICE-12 (VC issuer) to **P1**, *after* SLICE-07 (mastery gates), as a new Wave-4 slice in P1 (parallel with SLICE-08). This requires:
|
||||
1. VC issuer needs `issuer_keys` storage — use *SQLite* for P1 (the issuer_keys table moves to SQLite for v0.3; Postgres takes over in v0.4 when the operator tier arrives). Or, if Postgres is required for VC, then Postgres must also move to P1 — which inflates P1 further and reinforces the Axis 2 verdict (split the milestone).
|
||||
2. The cleaner resolution: **defer VC issuance to v0.3.1 (P2)** *and* accept that v0.1.4 (P1) ships mastery gates without credentials — but *label this explicitly* in the P1 ship notes ("VC issuance in v0.1.5"). Do not claim REQ-MAST-03 is covered in P1.
|
||||
|
||||
Either resolution is acceptable. The *current* plan — which implies VC issuance is triggered by P1's gate-open but tasks it in P2 with no wiring — is **not acceptable**. Pick one: (a) VC in P1 with SQLite-backed issuer keys, or (b) VC explicitly deferred to P2 with P1 shipping "mastery gates, no credential yet."
|
||||
|
||||
The P1→P2 dependency is otherwise clean (P2 reads P1's `mastery_gate_events` and session outcomes). **ACCEPT** on the dependency structure.
|
||||
|
||||
---
|
||||
|
||||
## Axis 9 — Decisions
|
||||
|
||||
**Forcing question:** Are D-031..D-049 (12 clarify + 7 specify decisions) well-grounded, or are any below the 0.60 confidence threshold? Is D-049 (failure-injection stays off) a mistake given mastery scoring scores recovery from failure branches?
|
||||
|
||||
**Challenge:** Let me audit confidences against the 0.70 threshold (the project's apparent decision-acceptance floor).
|
||||
|
||||
**Evidence (confidence audit of D-031..D-049):**
|
||||
|
||||
| ID | Confidence | Below 0.70? | Verdict |
|
||||
|----|------------|-------------|---------|
|
||||
| D-031 | 0.75 | No | ACCEPT — but see Axis 2 (scope creep). |
|
||||
| D-032 | 0.70 | At threshold | ACCEPT — N=3 is formative-only per R-MAST-01. |
|
||||
| D-033 | 0.70 | At threshold | ACCEPT — W3C VC 2.0 is a stable standard. |
|
||||
| D-034 | 0.70 | At threshold | ACCEPT — k=10 is the conventional minimum. |
|
||||
| D-035 | 0.70 | At threshold | ACCEPT — 1PL/Rasch is the simplest IRT. |
|
||||
| D-036 | 0.80 | No | ACCEPT. |
|
||||
| D-037 | 0.75 | No | ACCEPT. |
|
||||
| D-038 | 0.80 | No | ACCEPT — deterministic scoring is the right call. |
|
||||
| D-039 | 0.80 | No | ACCEPT. |
|
||||
| D-040 | 0.80 | No | ACCEPT. |
|
||||
| D-041 | 0.75 | No | ACCEPT — but R-AUTH-01 (Secure cookie) is a MUST-FIX (Axis 4). |
|
||||
| D-042 | 0.70 | At threshold | ACCEPT — but key rotation drill is a MUST (Axis 3). |
|
||||
| D-043 | 0.80 | No | ACCEPT. |
|
||||
| D-044 | 0.75 | No | ACCEPT. |
|
||||
| D-045 | 0.70 | At threshold | ACCEPT — but failure semantics are a FIX (Axis 6). |
|
||||
| D-046 | 0.80 | No | ACCEPT — θ in SQLite is correct. |
|
||||
| D-047 | 0.70 | At threshold | ACCEPT — 6 scenarios is tight but defensible for formative. |
|
||||
| D-048 | 0.75 | No | ACCEPT — but the P1/P2 split breaks the trigger wiring (Axis 8). |
|
||||
| D-049 | 0.80 | No | See below. |
|
||||
|
||||
**D-049 (failure-injection stays off):** The challenge is whether this is a mistake. The rubric (SLICE-01) has a "de-escalation" criterion (weight 0.20), and RESEARCH.md:718 says "de-escalation up-weights to ~0.40 if the escalate branch triggers." The `escalate` branch is a *naturally-occurring* failure branch in `cs_refund_ca_v01` (D-010), not an AI-provoked failure. So mastery scoring *does* score recovery from a failure branch — the *naturally-occurring* one. D-049 keeps AI-provoked failure injection off, which is correct: the rubric's de-escalation criterion is exercised by the existing branch, and adding AI-provoked failures would couple mastery scoring to a new feature (scope creep). D-049 is well-grounded.
|
||||
|
||||
**However**, there is a subtle gap: the rubric weights are *static* in the YAML (empathy 0.35, resolution 0.30, de-escalation 0.20, professionalism 0.15 per TASK-01-01). RESEARCH says de-escalation "up-weights to ~0.40 if the escalate branch triggers" — but TASK-01-01 does not mention dynamic re-weighting based on branch outcome. Either the weights are static (and the "up-weight" is a future feature) or they are dynamic (and the plan is missing a task). This is a **FIX** — clarify in TASK-01-01 whether weights are static or branch-dependent. If static, update RESEARCH.md to note the up-weight is deferred.
|
||||
|
||||
**Binding verdict: ACCEPT** on all D-031..D-049 confidences (none below 0.60; the floor is 0.70, which is the project's threshold). **FIX** on the de-escalation weight ambiguity (static vs dynamic) in TASK-01-01. D-049 is **ACCEPT** — failure-injection stays off is the correct call; the naturally-occurring `escalate` branch exercises the de-escalation criterion.
|
||||
|
||||
---
|
||||
|
||||
## Summary Table
|
||||
|
||||
| # | Axis | Forcing question (short) | Verdict |
|
||||
|---|------|---------------------------|---------|
|
||||
| 1 | Feasibility | 2 phases / 70 tasks realistic? | **FIX** — re-label as 2-milestone program; re-task SLICE-12/13 (+2 tasks each) |
|
||||
| 2 | Scope | REQ-DASH-01 really v0.3? D-031 Pandora's box? | **MUST** — split milestone; defer dashboard to v0.4 (or rebrand honestly) |
|
||||
| 3 | Cost | Custom VC code liability? Maintenance burden? | **MUST** — add VC interop test + key-rotation operational test before EXECUTE |
|
||||
| 4 | Technical risk | R-MAST-01/R-AUTH-01/R-MAST-02/R-IRT-01 | **MUST** — label VC formative; fix Secure cookie; fix silent-fail-to-zero fallback |
|
||||
| 5 | Requirements coverage | 20 REQ-IDs fully covered? | **FIX** — wire P1→P2 VC-issuance trigger; mirror SQLite→Postgres gate events; minor NFR clarifications |
|
||||
| 6 | Architecture | Hybrid SQLite+Postgres maintainable? | **FIX** — define Postgres-failure semantics; stabilize learner_ref; add 503 guard |
|
||||
| 7 | Testing | Test strategy viable? Untestable paths? | **FIX** — add real-LLM smoke test, differencing-attack test, drift-correction test, VC interop test |
|
||||
| 8 | Phase split | VC issuance in P2 but triggers on P1 event? | **MUST** — move VC to P1 (SQLite-backed) OR explicitly defer to P2 with honest labeling |
|
||||
| 9 | Decisions | D-031..D-049 below 0.60? D-049 a mistake? | **ACCEPT** — all confidences ≥0.70; D-049 correct; FIX de-escalation weight ambiguity |
|
||||
|
||||
---
|
||||
|
||||
## Final Recommendation: **GO-WITH-CONDITIONS**
|
||||
|
||||
The v0.3 plan is **not approved for EXECUTE as-is**. It is a well-researched, well-structured plan that suffers from two structural flaws: (1) it is two milestones pretending to be one, and (2) it splits a learner-facing consequence (VC issuance) from its trigger (mastery gate) across a phase boundary without wiring.
|
||||
|
||||
### MUST conditions (blocking — must be resolved in PLAN before EXECUTE):
|
||||
|
||||
1. **Axis 2 — Split the milestone.** Either (a) defer REQ-DASH-01 + operator tier to v0.4, shipping v0.3 = P1 + VC issuance only; or (b) rebrand v0.3 as a 2-milestone program (v0.3 + v0.3.1) with separate ship/verify cycles. Do not ship P1+P2 under one milestone tag.
|
||||
|
||||
2. **Axis 3 — Add VC interop test + key-rotation operational test.** Custom crypto code without interop verification is an unmitigated liability. Add TASK-12-07 (interop) and TASK-12-08 (rotation drill).
|
||||
|
||||
3. **Axis 4 — Fix three technical risks.** (a) Label VC as `formative` in payload + verification response + REQ-MAST-03 text. (b) Do not ship `PRAXIS_COOKIE_SECURE=false` as default — use TLS or loopback-binding for the operator surface. (c) Change evidence-extraction fallback from silent-fail-to-zero to `scoring_inconclusive` with learner-visible retry signal.
|
||||
|
||||
4. **Axis 8 — Resolve the VC-issuance phase split.** Either move SLICE-12 to P1 (with SQLite-backed issuer keys) or explicitly defer REQ-MAST-03 to P2 and label P1 as "mastery gates, no credential yet." The current plan's implicit wiring is a gap.
|
||||
|
||||
### FIX conditions (non-blocking — tracked in VERIFY-P1/P2):
|
||||
|
||||
5. **Axis 1 — Re-task SLICE-12 and SLICE-13.** Add 2 tasks each to honestly reflect the effort (VC edge cases + rotation drill; reconciliation idempotency + race test).
|
||||
6. **Axis 5 — Wire the P1→P2 VC-issuance trigger** (if VC stays in P2) and **mirror SQLite→Postgres gate events** explicitly in TASK-13-01.
|
||||
7. **Axis 6 — Define Postgres-failure semantics** (SQLite is learner-canonical, Postgres is operator-derived); stabilize `learner_ref` as a non-reusable UUID; add 503 guard on operator API.
|
||||
8. **Axis 7 — Add four tests**: real-LLM smoke (staging-gated), k-anonymity differencing-attack, reconciliation drift-correction, VC interop (already a MUST).
|
||||
9. **Axis 9 — Clarify de-escalation weight** (static vs branch-dependent) in TASK-01-01.
|
||||
|
||||
### ACCEPT items (proceed as-is):
|
||||
|
||||
- IRT 1PL/Rasch cold-start fallback (R-IRT-01).
|
||||
- All decision confidences (D-031..D-049 ≥ 0.70, none below 0.60).
|
||||
- D-049 (failure-injection stays off) — correct call.
|
||||
- Mocked-LLM + testcontainers CI strategy.
|
||||
- Hybrid SQLite+Postgres topology (with failure-semantics FIX).
|
||||
- P1→P2 dependency structure (clean except for VC-issuance wiring).
|
||||
|
||||
### Bottom line:
|
||||
|
||||
The plan is **not unfeasible** — the research is thorough, the architecture is sound, and the slice decomposition is reasonable. But it is **over-scoped** (two milestones in one tag) and **under-tested** in its highest-risk areas (custom crypto, k-anonymity, real-LLM extraction). Resolve the 4 MUST conditions, track the 5 FIX conditions, and this becomes a **GO**.
|
||||
+129
-1
@@ -194,4 +194,132 @@ Two personas are **phase-specific** for v0.2:
|
||||
- devops-engineer (praxis.service) ↔ backend-engineer (ExecStart command)
|
||||
- data-engineer (volume in compose) ↔ lead-developer (compose file owner)
|
||||
- devops-engineer (install-service.sh env file) ↔ backend-engineer (server env var consumption)
|
||||
- The config.json `personas` array does NOT include the devops-engineer — it will need to be added to config.json at PLAN/EXECUTE time, OR the devops-engineer is an emergent persona defined only in PERSONAS.md. The territory enforcement (warn mode) will pick up the territory globs from PERSONAS.md regardless of config.json.
|
||||
- The config.json `personas` array does NOT include the devops-engineer — it will need to be added to config.json at PLAN/EXECUTE time, OR the devops-engineer is an emergent persona defined only in PERSONAS.md. The territory enforcement (warn mode) will pick up the territory globs from PERSONAS.md regardless of config.json.
|
||||
|
||||
---
|
||||
|
||||
# Praxis — Persona Assessment (v0.3 Mastery Scoring)
|
||||
|
||||
> **Generated:** v0.3 RESEARCH stage
|
||||
> **Project:** Praxis (v0.3 — mastery scoring + competency rubrics + VC + cohort dashboard)
|
||||
> **Source:** v0.3 RESEARCH.md + v0.3 REQUIREMENTS.md (REQ-MAST-01/02/03, REQ-SCEN-02/03/04, REQ-PATH-02, REQ-DASH-01, REQ-AUTH-01, REQ-MT-01/02)
|
||||
|
||||
## v0.3 Persona Roster
|
||||
|
||||
### Active personas (5)
|
||||
|
||||
The v0.3 milestone is **mastery-backend + operator-frontend + security-heavy**. The frontend-engineer **reactivates** (cohort dashboard UI — D-044). A new **security-engineer** persona is added (VC crypto + auth — D-033/D-041/D-042). The devops-engineer from v0.2 is **deactivated** (no new deploy scripts in v0.3 — the Postgres-in-LXC addition is owned by backend-engineer + data-engineer since it's a docker-compose service addition, not a deploy-script change). The data-engineer expands territory to cover the Postgres operator-tier schema.
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: lead-developer
|
||||
active: true
|
||||
phase_specific: false
|
||||
reason: Coordinates task decomposition across mastery/rubric/IRT/VC/auth/cohort/dashboard domains. Resolves conflicts between backend (mastery engine), security (VC + auth), data (Postgres + SQLite hybrid), and frontend (dashboard UI). Owns the docker-compose.yml Postgres service addition (spans data + backend). Required for every milestone.
|
||||
domain: coordination
|
||||
frameworks: [pipecat, fastapi, postgres, docker]
|
||||
constraints: [pragmatic, battle-tested defaults, mastery-off-voice-path, hybrid-storage-no-cross-db-joins, k-anonymity-floor-10]
|
||||
territory:
|
||||
- "docker-compose.yml"
|
||||
- ".env.example"
|
||||
---
|
||||
```
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: backend-engineer
|
||||
active: true
|
||||
phase_specific: false
|
||||
reason: Owns the majority of v0.3 server-side logic: rubric engine (server/mastery/), IRT engine, scenario library, path engine, cohort aggregation pipeline, operator API routes, Postgres asyncpg pool wiring, session_recorder.py extension for rubric/IRT hooks. The mastery scoring flow (off the voice path) is the largest single territory in v0.3.
|
||||
domain: backend
|
||||
frameworks: [pipecat, pydantic, fastapi, uvicorn, asyncpg, aiosqlite]
|
||||
constraints: [api-first, type-safe, mastery-off-voice-path, deterministic-scoring, latency-budget-aware, routes-before-static-mount, no-cross-db-joins]
|
||||
territory:
|
||||
- "**/server/**"
|
||||
- "**/mastery/**"
|
||||
- "**/scenarios/**"
|
||||
- "**/paths/**"
|
||||
- "**/cohort/**"
|
||||
- "**/operator/**"
|
||||
- "**/db/**"
|
||||
---
|
||||
```
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: frontend-engineer
|
||||
active: true
|
||||
phase_specific: false
|
||||
reason: REACTIVATED for v0.3. Owns the cohort dashboard UI (React /operator/* route — D-044, REQ-DASH-01). Auth-gated React route + k-anonymized cohort views (practice, mastery progression, failure patterns). Reuses v0.2 StaticFiles + same client/dist build. No new build pipeline. First client-side feature work since v0.1.
|
||||
domain: frontend
|
||||
frameworks: [react, pipecat-client-sdk, webrtc, vite, fastapi-staticfiles]
|
||||
constraints: [component-first, auth-gated-operator-routes, k-anonymity-display-suppressed-cells, no-raw-learner-pii-in-ui]
|
||||
territory:
|
||||
- "**/client/**"
|
||||
- "**/client/src/operator/**"
|
||||
---
|
||||
```
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: data-engineer
|
||||
active: true
|
||||
phase_specific: false
|
||||
reason: EXPANDED territory for v0.3. Owns the Postgres operator-tier schema (operators, issued_credentials, mastery_gate_events, cohort_aggregates, issuer_keys — D-040), the db/pg_migrations/ migration runner, the SQLite v0.3 additions (learner_ability, mastery_progress tables — D-046), and the k-anonymity suppression queries (D-034). The hybrid SQLite+Postgres storage pattern (D-031) is the data-engineer's architectural concern — no cross-DB joins, opaque learner_ref.
|
||||
domain: data
|
||||
frameworks: [sqlite, postgres16, aiosqlite, asyncpg, alembic-style-migrations]
|
||||
constraints: [schema-first, type-safe, migration-driven, no-cross-db-joins, k-anonymity-floor-10, opaque-learner-ref, weekly-partitions-cohort-aggregates]
|
||||
territory:
|
||||
- "**/db/**"
|
||||
- "**/db/migrations/**"
|
||||
- "**/db/pg_migrations/**"
|
||||
- "**/db/schema.sql"
|
||||
- "**/db/pg_schema.sql"
|
||||
---
|
||||
```
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: security-engineer
|
||||
active: true
|
||||
phase_specific: true
|
||||
reason: NEW persona for v0.3. Owns the VC issuer (server/vc/ — Ed25519 signing, JCS canonicalization, Bitstring Status List, verification endpoint — D-033/D-042/D-043) and the operator auth stack (server/auth/ — argon2id, session cookies, rate limiting — D-041). VC crypto + auth are security-critical and outside the default four personas' expertise. Created as phase-specific because v0.3 is the first security-crypto-heavy milestone; may persist into v0.9 (credentialing) but deactivate in between.
|
||||
domain: security
|
||||
frameworks: [pynacl, canonicaljson, base58, argon2-cffi, starlette-sessionmiddleware, slowapi]
|
||||
constraints: [eddsa-jcs-2022-cryptosuite, no-plaintext-keys-in-git, issuer-key-encrypted-at-rest, argon2id-passwords, secure-cookies-require-tls-R-AUTH-01, public-verification-no-pii]
|
||||
territory:
|
||||
- "**/server/vc/**"
|
||||
- "**/server/auth/**"
|
||||
- "**/vc/**"
|
||||
- "**/auth/**"
|
||||
---
|
||||
```
|
||||
|
||||
### Deactivated personas (1)
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: devops-engineer
|
||||
active: false
|
||||
phase_specific: true
|
||||
reason: DEACTIVATED for v0.3. No new Proxmox/deploy scripts in v0.3 — the v0.2 LXC deployment carries forward unchanged. The Postgres-in-LXC addition (D-040) is a docker-compose service addition owned by lead-developer (compose file) + data-engineer (schema) + backend-engineer (asyncpg wiring), not a deploy-script change. Will reactivate if v0.3 adds deploy hardening (Traefik/TLS) or if a CT memory bump requires lxc-config changes.
|
||||
domain: devops
|
||||
frameworks: [proxmox-lxc, docker, systemd, bash]
|
||||
constraints: [idempotent-deploy, rollback-on-failure, battle-tested-coreci-toolkit]
|
||||
territory: []
|
||||
---
|
||||
```
|
||||
|
||||
## v0.3 Notes for PLAN/EXECUTE
|
||||
|
||||
- Territory enforcement mode: `warn` (per config.json `personas.territory_enforcement`)
|
||||
- The **backend-engineer owns the majority of v0.3 task surface** (mastery engine + IRT + library + paths + cohort aggregation + operator API + Postgres wiring). This is the largest backend surface since v0.1.
|
||||
- The **security-engineer's v0.3 surface is the most security-critical**: VC issuer keys + operator auth. Any P0/P1 finding here blocks ship.
|
||||
- The **frontend-engineer reactivates** after v0.2 deactivation — the cohort dashboard is the first client-side feature since v0.1.
|
||||
- The **data-engineer's v0.3 surface spans two stores** (SQLite v0.3 tables + Postgres operator tier) — the hybrid pattern (D-031) is the architectural concern.
|
||||
- Cross-persona collaboration points:
|
||||
- backend-engineer (mastery_gate_event write) ↔ data-engineer (Postgres schema) ↔ security-engineer (VC issuance on gate-open)
|
||||
- frontend-engineer (dashboard UI) ↔ backend-engineer (operator API) ↔ data-engineer (k-anonymity queries)
|
||||
- security-engineer (issuer key) ↔ data-engineer (issuer_keys table, encrypted-at-rest)
|
||||
- The security-engineer is NOT in config.json `personas` — emergent persona defined in PERSONAS.md (same pattern as v0.2 devops-engineer). Territory enforcement (warn mode) picks up globs from PERSONAS.md.
|
||||
- R-AUTH-01 (Secure cookie + no-TLS) is a security-engineer + lead-developer collaboration point for PLAN.
|
||||
+341
-874
File diff suppressed because it is too large
Load Diff
+55
-6
@@ -1,8 +1,9 @@
|
||||
# Praxis — Voice-first AI Apprenticeship Platform
|
||||
|
||||
**Milestone:** v0.2 (Proxmox LXC deployment)
|
||||
**Status:** phase 1 complete — P2 review/ship in-progress
|
||||
**Milestone:** v0.3 (Mastery scoring + competency rubrics)
|
||||
**Status:** phase 0 — specify (active milestone)
|
||||
**Autonomy:** full
|
||||
**Previous milestone:** v0.2 (Proxmox LXC deployment) — complete, tagged v0.1.2, release #377
|
||||
|
||||
## Vision
|
||||
|
||||
@@ -14,7 +15,34 @@ Praxis is a voice-first, AI-tutored skill platform for learners in resource-cons
|
||||
|
||||
Build a voice-first AI apprenticeship platform where learners engage in spoken role-play scenarios with AI tutors, receive coaching debriefs, and progress via mastery gates — working on low-cost phones over constrained bandwidth.
|
||||
|
||||
## v0.2 Scope (Proxmox LXC Deployment)
|
||||
## v0.3 Scope (Mastery Scoring + Competency Rubrics)
|
||||
|
||||
v0.3 activates the mastery/assessment layer deferred from v0.1/v0.2 (per D-021, ROADMAP line 53). Learners progress via **mastery gates** — they move on only when they can do the thing across varied scenarios, scored against a competency rubric. v0.3 also introduces the multi-tenant + auth foundation required for the cohort dashboard, and a verifiable-credential issuer so mastery is portable.
|
||||
|
||||
**v0.3 in scope (activated REQ groups — post-grill):**
|
||||
- **Mastery core (REQ-MAST-01, REQ-MAST-02):** competency rubric per skill; Mastery Score updated after each session, requiring varied-scenario success before a mastery gate opens
|
||||
- **Verifiable credentials (REQ-MAST-03):** portable, tamper-evident credentials issued on week-final mastery gate (W3C VC Data Model 2.0, Ed25519, **formative-tier**, SQLite-backed issuer keys, public verification endpoint)
|
||||
- **Dynamic difficulty (REQ-SCEN-02):** scenario difficulty adjusts to learner performance (item-response-theory-informed)
|
||||
- **Scenario library (REQ-SCEN-03, REQ-SCEN-04):** library tagged by skill/difficulty/failure_mode; expert-authored format extended with rubric mappings + AI-generated variation hooks
|
||||
- **Path structure (REQ-PATH-02):** path-as-job 6-week structure (PRD §6.4) — the progression container mastery gates live in
|
||||
|
||||
**v0.3 out of scope (deferred to v0.4 per GRILL-v0.3.md Axis 2):**
|
||||
- **REQ-DASH-01 (cohort dashboard) + REQ-AUTH-01 (operator auth) + REQ-MT-01/02 (operator Postgres + aggregation) + 4 NFRs** — the operator tier was originally v0.8 on the ROADMAP; pulling it into v0.3 created a 2-milestone program. The grill's binding verdict splits it to v0.4. D-031 (override D-007) is deferred with the operator tier.
|
||||
- REQ-PATH-01 (full multi-path launch) — v0.3 ships the Customer Service path only
|
||||
- REQ-DASH-02 (full operator-suite dashboard) — later milestone
|
||||
- REQ-ASSIST-01..03 (Live Assist) — later milestone
|
||||
- REQ-LOWBW-01..03 (WhatsApp/USSD/offline) — later milestone
|
||||
- REQ-VOICE-05/06 (multi-language, persona switching) — later milestone
|
||||
- Active failure injection (D-009) — D-049 confirms stays off in v0.3
|
||||
- Dynamic rubric weight re-weighting on branch outcome — static in v0.3 (grill Axis 9)
|
||||
- Traefik proxy / public TLS — deferred from v0.2 (R-AUTH-01 deferred to v0.4 with the operator surface)
|
||||
|
||||
**Carries forward from v0.2 (already in production):**
|
||||
- Docker-in-LXC deployment (`lxc-deploy.sh`, `praxis.service`, `/health` :8789)
|
||||
- Voice loop (Deepgram Nova-3 + Cartesia + Pipecat + Ollama Cloud)
|
||||
- v0.1 scenario (`cs_refund_ca_v01.yaml`) + guardrails + debrief
|
||||
|
||||
## v0.2 Scope (Proxmox LXC Deployment — complete)
|
||||
|
||||
v0.2 deploys praxis into a Proxmox LXC container, reusing and adapting the battle-tested deployment toolkit from `~/coreci/scripts/proxmox/`. The v0.1 voice loop becomes deployable infrastructure — a Docker image runs the Python/Pipecat server (serving the React client as static files) inside an LXC container on the operator's Proxmox cluster.
|
||||
|
||||
@@ -47,10 +75,12 @@ v0.2 deploys praxis into a Proxmox LXC container, reusing and adapting the battl
|
||||
|
||||
- Voice conversation engine: real-time ASR + streaming TTS, <600ms round-trip, interruptible, persona switching
|
||||
- Scenario engine: branching role-plays with failure-injection and dynamic difficulty (v0.1: one scenario)
|
||||
- Learner state: progress, session history, mastery accumulation (v0.1: single-learner state, no mastery scoring yet)
|
||||
- Learner state: progress, session history, mastery accumulation (v0.3: mastery scoring + competency rubrics + verifiable credentials)
|
||||
- Scenario engine: branching role-plays with failure-injection and dynamic difficulty (v0.3: dynamic difficulty + scenario library + AI variations)
|
||||
- Skill paths: path-as-job 6-week structure (v0.3: Customer Service path structured + mastery gates)
|
||||
- Cohort dashboard: anonymized cohort view for training operators (v0.3: multi-tenant + auth + cohort view)
|
||||
- LLM foundation: Ollama-hosted open-weights models `gemma4:cloud` and `deepseek-v4-flash:cloud`
|
||||
- Low-bandwidth surfaces (later milestones)
|
||||
- Employer dashboard (later milestones)
|
||||
|
||||
## Constraints
|
||||
|
||||
@@ -97,6 +127,25 @@ v0.2 deploys praxis into a Proxmox LXC container, reusing and adapting the battl
|
||||
| D-028 | Docker installed **inside the CT** via apt (CT has network via vmbr0 DHCP) | CLARIFY auto-decide. Avoids needing Docker on the PVE host. The debian-12 template + nesting=1 supports Docker-in-LXC. firstboot hook runs `pct exec` to install `docker.io` + `docker-compose-v2`. | 0.90 | Docker on PVE host (extra host dependency), pre-baked template (custom template maintenance) |
|
||||
| D-029 | Image built **inside the CT** (clone repo from Gitea, `docker build`, `docker compose up`) | CLARIFY auto-decide. Self-contained — CT fetches its own source + builds. No image transfer needed. Slower first-boot (~3-5 min for build) but simpler and reproducible. | 0.80 | Build on PVE host + pct push tarball (host Docker dependency), pre-built image from registry (external dependency) |
|
||||
| D-030 | CT network = **vmbr0 DHCP only** (pilot, no vmbr1, no Traefik proxy) | CLARIFY auto-decide. v0.2 is infrastructure-only pilot. Direct bridge IP access for health-check. Proxy/TLS deferred to a later milestone. | 0.90 | vmbr1 + Traefik proxy (over-scoped for pilot) |
|
||||
| D-031 | v0.3 introduces **multi-tenant + auth** — **overrides D-007** for the cohort-dashboard surface | REQ-DASH-01 (anonymized cohort view for training operators) requires multi-tenant data. D-007's single-learner/no-auth stance was correct for v0.1/v0.2 pilot but blocks v0.3's cohort dashboard. Resolution: **hybrid** — learner-local state stays SQLite-on-device (D-007 preserved for learner surface); a new **operator-tier Postgres** stores cohort aggregations + operator accounts + issued credentials. Learner auth deferred (single-learner-per-device still valid for pilot). Operator auth = session-based, single operator role in v0.3. Research phase to validate Postgres-in-LXC + migration path. | 0.75 | Full Postgres migration (abandons SQLite pilot work), defer DASH-01 again (scope creep), no auth (insecure) |
|
||||
| D-032 | Mastery gate = **N-of-M varied-scenario success + rubric score ≥ threshold** | Operationalizes PRD principle 6 ("move on when you can do the thing"). N=3 distinct scenarios, rubric mean ≥ 3.5/5.0 (configurable per path). Research phase to validate rubric model + threshold against competency-based-assessment literature. | 0.70 | Single-scenario pass (gaming risk), pure rubric score (no variety), pure time-on-task (invalid) |
|
||||
| D-033 | Verifiable credentials = **W3C VC Data Model 2.0, platform-issued** (operator key), Ed25519 signatures | Research-anticipated: W3C VC 2.0 is the current standard; platform-issued is simplest viable issuer model (no DID method proliferation); Ed25519 is compact + widely supported. Self-issued (learner-side key) rejected — no tamper-evidence authority. Third-party issuer (university/agency) deferred to v0.9 credentialing milestone. Revocation = simple status list (VC Status List v2025). | 0.70 | Self-issued (no authority), third-party issuer (v0.9 scope), JWT-VC (less mature tooling) |
|
||||
| D-034 | Cohort anonymization = **k-anonymity ≥ 10** + aggregation window ≥ 7 days | REQ-DASH-01 operator view must not expose individual learners. k=10 is the conventional minimum for anonymized analytics; 7-day aggregation prevents re-identification via sparse windows. Research phase to validate against differential-privacy literature. Operator sees aggregate progression/failure-patterns only. | 0.70 | No anonymization (privacy violation), differential privacy (over-engineered for v0.3 scale), k=5 (too weak) |
|
||||
| D-035 | Dynamic difficulty = **IRT-informed (1-parameter Rasch)**, updated per session | REQ-SCEN-02. Item Response Theory (1PL/Rasch) is the simplest well-grounded model: learner ability θ, scenario difficulty b, P(success)=logistic(θ−b). Bayesian update of θ after each session. Avoids 2PL/3PL complexity (discrimination/guessing params — needs more data than v0.3 has). Research phase to validate. | 0.70 | ELO-like (less theoretically grounded), fixed difficulty steps (no adaptation), 2PL/3PL (data-hungry) |
|
||||
| D-036 | Scenario library structure = **YAML directory + index manifest**, tagged by skill/difficulty/failure_mode/rubric | Extends D-018's YAML DSL. Library = `scenarios/<path>/<scenario>.yaml` + `scenarios/index.yaml` manifest (tagged, versioned). Expert-authored scenarios ship as YAML; AI-generated variations use the same schema with a `generated_from` backref. Rubric mapping added to scenario schema (each scenario declares which rubric criteria it exercises). | 0.80 | Database-backed library (premature — YAML is diffable + authorable per C-7), JSON (no comments per D-018), inline in code (couples authoring to engineering) |
|
||||
| D-037 | Path structure = **6-week job-structured path**, JSON + YAML, mastery gates between weeks | REQ-PATH-02 (PRD §6.4). Path = `paths/<slug>.yaml` defining 6 weeks, each week = a set of scenarios + a mastery gate. Gate opens when D-032 mastery condition met. v0.3 ships the Customer Service path fully (6 weeks) with ≥1 scenario per week (library REQ-SCEN-03 fills the rest). | 0.75 | Free-form progression (no structure), 12-week (too long for pilot), week-as-fixed-time (relax to mastery-paced) |
|
||||
| D-038 | Rubric scoring path = **rule-based final score, LLM-assisted criterion extraction only** (REQ-NFR-MAST-01) | Final score must be deterministic. LLM (deepseek-v4-flash:cloud no_think) extracts criterion evidence from session turns (which utterance maps to which rubric criterion); a rule function computes the 1-5 score per criterion from the extracted evidence + branch outcome. No LLM in the numeric scoring step. Preserves REQ-NFR-MAST-01 determinism + keeps latency off the voice path. | 0.80 | Pure-LLM scoring (non-deterministic, violates NFR-MAST-01), pure-rule extraction (rigid — can't handle free-form speech) |
|
||||
| D-039 | Rubric YAML format = **`rubrics/<skill>.yaml`** with criteria, 5-level anchors, per-skill weights | Extends D-018's YAML-everywhere stance. One rubric file per skill (v0.3: `rubrics/customer_service.yaml`). Each criterion has id, name, 5 anchored levels (1=fail … 5=mastery), weight. Scenario YAML maps to rubric criteria via `rubric_criteria` field (D-036). | 0.80 | JSON (no comments per D-018), inline in scenario (couples rubric to scenario — rubric is per-skill not per-scenario), DB-backed (premature) |
|
||||
| D-040 | Operator Postgres deployment = **second Docker service in the existing LXC CT** (`docker-compose.yml` adds `postgres` service) | REQ-NFR-MT-01. Reuses v0.2's LXC + Docker-in-LXC. No new CT, no host Postgres. Postgres 16, persistent volume, internal Docker network only (not exposed to bridge). Operator auth + cohort API + VC issuer connect to it. | 0.80 | Separate CT (over-provisioned for v0.3 scale), host Postgres (PVE host dependency), SQLite for operator (cohort aggregation needs relational + k-anonymity queries — SQLite workable but Postgres is the safer default) |
|
||||
| D-041 | Operator auth = **session-cookie, argon2id passwords, single `operator` role, login rate-limited (5 attempts/min)** | REQ-NFR-AUTH-01. Simplest viable auth for v0.3's single operator role. No OAuth/JWT complexity for one role. Cookie: httpOnly, secure, SameSite=Strict, 8h expiry. Rate limit via in-memory counter (single-instance). RBAC deferred (one role). | 0.75 | JWT (over-engineered for server-side session), OAuth (no IdP yet), basic-auth (insecure), no rate-limit (brute-force risk) |
|
||||
| D-042 | VC issuer key = **Ed25519 keypair in operator-tier secrets (`PRAXIS_VC_ISSUER_KEY`), generated on first issuer init, not committed** | REQ-NFR-VC-01. Key generated at first boot if absent, stored in Postgres `issuer_keys` table encrypted at rest with a root key from secrets. Verification endpoint serves the public key. Rotation = new key + old key marked superseded (not revoked — old VCs still verify against archived public key). | 0.70 | RSA (larger, slower), KMS-managed (no KMS in LXC), self-signed cert chain (X.509 complexity unjustified for one issuer) |
|
||||
| D-043 | VC verification endpoint = **public, unauthenticated, GET `/vc/verify/<credential_id>`** | Third parties (employers/agencies) verify credentials without an account. Returns `{valid: bool, status: "active"\|"revoked", issuer: "praxis-v0.3", mastery: {...}}`. No PII in the verification response beyond what the credential itself asserts. | 0.80 | Authenticated verification (friction for employers), no public endpoint (credentials not portable), returns full learner PII (privacy violation) |
|
||||
| D-044 | Cohort dashboard UI = **React route under `/operator/*`, served by the same FastAPI server (new prefix), reuses v0.2 StaticFiles** | REQ-DASH-01. Frontend-engineer reactivates (PERSONAS.md). Adds `/operator` React route + `/api/operator/*` FastAPI endpoints. Auth gate in React + server-side session check. No separate SPA build — same `client/dist`. | 0.75 | Separate operator SPA (extra build pipeline), server-rendered HTML (abandons React investment), no UI (operator reads JSON — not a product) |
|
||||
| D-045 | Cohort aggregation trigger = **on-session-end hook + nightly reconciliation job** | REQ-MT-02. Hook fires after `end_session()` → writes k-anonymized aggregate to Postgres (incremental). Nightly job (cron in the praxis service) reconciles + recomputes 7-day windows. Hybrid: low-latency updates + correctness guarantee. | 0.70 | Pure real-time (race-prone), pure nightly (stale, violates NFR-DASH-02 if job lags), CDC/streaming (over-engineered) |
|
||||
| D-046 | IRT θ persistence = **in learner-local SQLite** (`learner_ability` table: learner_id, path, theta, updated_at) | REQ-NFR-IRT-01. θ is per-learner-per-path, computed in-process on session end, no LLM call. Stays in SQLite with the rest of learner state (D-007 preserved). Cohort dashboard sees only k-anonymized aggregates of θ, never raw θ. | 0.80 | Postgres (couples learner state to operator tier — violates D-031 hybrid), in-memory (lost on restart), file-based JSON (no queryability) |
|
||||
| D-047 | Scenario library minimum for v0.3 = **≥6 expert-authored Customer Service scenarios** (one per path week) + **AI-generated variations gated by expert review** | REQ-SCEN-03/04. 6 scenarios give the mastery gate's N=3 varied-scenario condition room (D-032) without being so few that mastery is gameable. AI variations: LLM generates a variation from an expert scenario's schema with `generated_from` backref; expert reviews + approves before it enters the library. | 0.70 | 3 scenarios (mastery gate N=3 = exactly the minimum — no room for failure-retry variety), 12 scenarios (over-scoped for one milestone), no AI variations (loses REQ-SCEN-04) |
|
||||
| D-048 | Mastery gate open action = **advance learner to next path week + issue VC if week-final gate** | When D-032 condition met for a week's scenarios: learner `progress.current_week` advances. If the gate is the final week's gate, a VC is issued (REQ-MAST-03) asserting mastery of the path. Mid-path gates: no VC, just advancement. VCs are path-level, not week-level. | 0.75 | VC per week (credential spam — devalues the credential), no advancement (mastery gate is decorative), manual advancement (violates autonomy) |
|
||||
| D-049 | v0.3 activation of D-009 failure-injection = **NO** — failure-injection stays architecturally present but not provoked in v0.3 | D-009 hook stays in the schema. v0.3 mastery scoring scores *recovery* from naturally-occurring failure branches (the `escalate` branch in cs_refund_ca_v01), not AI-provoked failures. Active failure injection couples to a "failure-recovery coaching" feature that's a later milestone. v0.3 RESEARCH confirms this — no new failure-injection scenarios authored. | 0.80 | Activate failure injection in v0.3 (couples mastery scoring to a new feature — scope creep), remove the hook (breaks forward compat) |
|
||||
|
||||
### Confidence updates from research
|
||||
|
||||
@@ -105,7 +154,7 @@ v0.2 deploys praxis into a Proxmox LXC container, reusing and adapting the battl
|
||||
| D-003 | 0.75 | **0.95** | Both Ollama model IDs verified in catalog as real, current, cloud-hosted tags |
|
||||
| D-007 | 0.80 | **0.90** | SQLite confirmed appropriate for v0.1 single-learner scale; no evidence favors alternatives |
|
||||
|
||||
## Target Users (v0.1 pilot: Canada)
|
||||
## Target Users (v0.3: Canada pilot — Customer Service path)
|
||||
|
||||
| Persona | Description | Pain |
|
||||
|---------|-------------|------|
|
||||
|
||||
@@ -5,6 +5,96 @@
|
||||
|
||||
Formal requirements with REQ-IDs. Scoped to the active milestone unless noted. Later-milestone requirements are marked `deferred`. v0.1 requirements (complete) are retained for reference.
|
||||
|
||||
# Praxis — Requirements
|
||||
|
||||
**Milestone:** v0.3 (Mastery scoring + competency rubrics)
|
||||
**Status:** phase 0 — specify (active milestone)
|
||||
|
||||
Formal requirements with REQ-IDs. Scoped to the active milestone unless noted. v0.1/v0.2 requirements (complete) are retained for reference with their final status. Later-milestone requirements are marked `deferred`.
|
||||
|
||||
## v0.3 Active Requirements
|
||||
|
||||
### Mastery & Assessment (v0.3 core)
|
||||
|
||||
| REQ-ID | Requirement | Priority | Phase | Status |
|
||||
|--------|-------------|----------|-------|--------|
|
||||
| REQ-MAST-01 | Competency rubric per skill — a typed rubric model (criteria, 5-level scale, per-skill weights) authored as YAML, mapped to scenarios (D-036). At least one rubric for the Customer Service path in v0.3. | must | P1 | active |
|
||||
| REQ-MAST-02 | Mastery Score updated after each session — computed from rubric scores + varied-scenario-success gate (D-032: N=3 distinct scenarios, rubric mean ≥ 3.5/5.0). Score persisted per learner per path. Mastery gate opens when condition met. | must | P1 | active |
|
||||
| REQ-MAST-03 | Portable verifiable credentials on mastery — W3C VC Data Model 2.0, platform-issued Ed25519 signatures, status-list revocation (D-033). Issued when a mastery gate opens. Verifiable by third parties via a public verification endpoint. | must | P1 | active |
|
||||
| REQ-MAST-04 | No quizzes — assessment built into scenarios | principle | — | accepted |
|
||||
|
||||
### Scenario Engine (v0.3 extensions)
|
||||
|
||||
| REQ-ID | Requirement | Priority | Phase | Status |
|
||||
|--------|-------------|----------|-------|--------|
|
||||
| REQ-SCEN-02 | Dynamic difficulty adjustment based on learner performance — IRT 1PL/Rasch, Bayesian θ update per session (D-035). Difficulty selection picks next scenario targeting ~50% expected success for current θ. | must | P1 | active |
|
||||
| REQ-SCEN-03 | Scenario library tagged by skill, difficulty, failure mode, rubric criteria — YAML directory + `scenarios/index.yaml` manifest (D-036). v0.3 ships ≥6 scenarios for the Customer Service path (one per week minimum). | must | P1 | active |
|
||||
| REQ-SCEN-04 | Expert-authored scenario format with AI-generated variations — extends D-018 YAML DSL with rubric mapping + `generated_from` backref for AI variations. Expert-authored = canonical; AI variations = same schema, flagged, reviewable. | must | P1 | active |
|
||||
|
||||
### Skill Paths (v0.3)
|
||||
|
||||
| REQ-ID | Requirement | Priority | Phase | Status |
|
||||
|--------|-------------|----------|-------|--------|
|
||||
| REQ-PATH-02 | Path structured as a job — 6-week structure per PRD §6.4, mastery-paced (D-037). Path = `paths/<slug>.yaml` defining weeks, each week = scenarios + a mastery gate. v0.3 ships the Customer Service path fully (6 weeks, ≥1 scenario/week). | must | P1 | active |
|
||||
|
||||
### Employer / Program Dashboard (v0.3)
|
||||
|
||||
| REQ-ID | Requirement | Priority | Phase | Status |
|
||||
|--------|-------------|----------|-------|--------|
|
||||
| REQ-DASH-01 | Anonymized cohort view (practice, mastery progression, failure patterns) for training operators — k-anonymity ≥ 10, 7-day aggregation window (D-034). Operator UI (React) reads from operator-tier Postgres. Forces multi-tenant + operator auth (D-031). | must | P1 | active |
|
||||
|
||||
### Auth & Multi-Tenancy (deferred to v0.4 — per GRILL-v0.3.md Axis 2)
|
||||
|
||||
| REQ-ID | Requirement | Priority | Phase | Status |
|
||||
|--------|-------------|----------|-------|--------|
|
||||
| REQ-AUTH-01 | Operator-tier auth — session-based, single `operator` role in v0.3. Operator accounts in Postgres. Login endpoint + session cookie. Protects cohort dashboard + credential issuance. | must | v0.4 | deferred-to-v0.4 |
|
||||
| REQ-MT-01 | Operator-tier Postgres store — cohort aggregations, operator accounts, issued credentials, mastery-gate audit log. Separate from learner-local SQLite (D-007 preserved for learner surface). Migration path: SQLite stays for learner; Postgres added for operator. | must | v0.4 | deferred-to-v0.4 |
|
||||
| REQ-MT-02 | Cohort aggregation pipeline — scheduled job (or on-session-end hook) writes k-anonymized aggregates to Postgres from learner sessions. No raw learner PII in Postgres. | must | v0.4 | deferred-to-v0.4 |
|
||||
|
||||
## v0.3 Non-Functional Requirements
|
||||
|
||||
| REQ-ID | Requirement | Target | Phase | Status |
|
||||
|--------|-------------|--------|-------|--------|
|
||||
| REQ-NFR-MAST-01 | Rubric scoring determinism — same session + rubric → same score (no LLM non-determinism in the scoring path; LLM may assist rubric criterion extraction but final score is rule-based) | must | P1 | active |
|
||||
| REQ-NFR-MAST-02 | Mastery gate auditability — every gate-open event recorded with evidence (which 3 scenarios, rubric scores, timestamp) | must | P1 | active |
|
||||
| REQ-NFR-VC-01 | Verifiable credential tamper-evidence — Ed25519 signature, issuer key in operator-tier secrets (not committed), verification endpoint validates signature + status + interop test against external W3C verifier (grill Axis 3) | must | P1 | active |
|
||||
| REQ-NFR-VC-02 | Credential revocation latency — revoked credential must fail verification within 1 sync of the status list (next verify call — no cache) | must | P1 | active |
|
||||
| REQ-NFR-AUTH-01 | Operator auth — passwords hashed (argon2id), session cookie httpOnly + secure, login rate-limited | must | v0.4 | deferred-to-v0.4 |
|
||||
| REQ-NFR-MT-01 | Postgres-in-LXC — operator Postgres runs as a second Docker service in the existing LXC CT (or sidecar) without destabilizing the learner-facing praxis service | must | v0.4 | deferred-to-v0.4 |
|
||||
| REQ-NFR-IRT-01 | IRT θ update latency — < 100ms (in-process, no LLM call) | must | P1 | active |
|
||||
| REQ-NFR-DASH-01 | Cohort dashboard k-anonymity ≥ 10 — any cohort view cell with < 10 learners is suppressed | must | v0.4 | deferred-to-v0.4 |
|
||||
| REQ-NFR-DASH-02 | Cohort dashboard freshness — aggregates ≤ 24h stale | must | v0.4 | deferred-to-v0.4 |
|
||||
|
||||
## Constraints (binding — carry forward from v0.1/v0.2)
|
||||
|
||||
- C-1 Voice is primary interface; text is fallback only
|
||||
- C-2 Must work on $100 Android phone over 2G/3G (relaxed for v0.1 Canada pilot)
|
||||
- C-3 Cost ≤ $3/active learner/month (relaxed for v0.1 pilot)
|
||||
- C-4 Audio-only in v1
|
||||
- C-5 Open-weights LLM via Ollama catalog — `gemma4:cloud` + `deepseek-v4-flash:cloud`
|
||||
- C-6 Domain safety guardrails + HITL + disclaimers for safety-sensitive domains
|
||||
- C-7 Scenarios authored by domain experts + learning designers; AI generates variations only
|
||||
- C-8 Latency budget < 600ms end-to-end (ASR → LLM → TTS) — mastery scoring must not be on the voice path
|
||||
|
||||
## v0.3 Out of Scope (still deferred)
|
||||
|
||||
- REQ-PATH-01 (full multi-path launch) — v0.3 ships Customer Service path only
|
||||
- REQ-DASH-01 (cohort dashboard) — **deferred to v0.4** per GRILL-v0.3.md Axis 2 (was v0.8 on original ROADMAP)
|
||||
- REQ-AUTH-01, REQ-MT-01, REQ-MT-02 (operator auth + Postgres) — **deferred to v0.4** (operator tier)
|
||||
- REQ-NFR-DASH-01, REQ-NFR-DASH-02, REQ-NFR-AUTH-01, REQ-NFR-MT-01 — **deferred to v0.4**
|
||||
- REQ-DASH-02 (full operator-suite dashboard) — later milestone
|
||||
- REQ-ASSIST-01..03 (Live Assist) — later milestone
|
||||
- REQ-LOWBW-01..03 (WhatsApp/USSD/offline) — later milestone
|
||||
- REQ-VOICE-05/06 (multi-language, persona switching) — later milestone
|
||||
- Third-party credential issuers (university/agency) — v0.9 credentialing milestone
|
||||
- Learner auth / multi-learner-per-device — operator auth is v0.4; learner auth later
|
||||
- Active failure injection (D-009) — evaluated in v0.3 RESEARCH (D-049), stays off
|
||||
- Dynamic rubric weight re-weighting on branch outcome — static weights in v0.3, dynamic is a future feature (grill Axis 9)
|
||||
|
||||
---
|
||||
|
||||
## v0.2 Requirements (complete — retained for reference)
|
||||
|
||||
## Functional Requirements
|
||||
|
||||
### Voice Conversation Engine
|
||||
|
||||
@@ -0,0 +1,456 @@
|
||||
# Praxis — v0.3 Research: Anonymization, IRT, Scenario Library
|
||||
|
||||
> **Milestone:** v0.3 (Mastery scoring + competency rubrics)
|
||||
> **Phase:** 0 (research — pre-execution)
|
||||
> **Branch:** phase/00-pre-execution
|
||||
> **Status:** research complete — pending orchestrator review
|
||||
> **Date:** 2026-08-03
|
||||
> **Method:** Domain-knowledge synthesis from the privacy-preserving analytics, psychometrics (IRT), and learning-content authoring literature. Where claims rest on a single source or empirical rule of thumb, the confidence score reflects that. Web-verification deferred — these are well-trodden fields with stable canonical references (Sweeney 2002; Machanavajjhala et al. 2007; Lord 1980; Rasch 1960; Wainer 2000; van der Linden 2010). No code is written here; this is decision input for the PLAN stage.
|
||||
> **Scope:** Three research question sets mapped to v0.3 decisions D-034 (cohort anonymization), D-035 (dynamic difficulty), D-036 (scenario library), D-047 (≥6 expert CS scenarios).
|
||||
|
||||
This document grounds three v0.3 subsystems — cohort anonymization, IRT-based dynamic difficulty, and the scenario library — in published evidence and gives concrete recommendations for the pilot scale (likely <100 learners in v0.3). Each subsection ends with a confidence score (0–1) and a recommendation keyed to the relevant D-ID.
|
||||
|
||||
---
|
||||
|
||||
## Summary of Findings (Executive 1-Pager)
|
||||
|
||||
1. **k=10 + 7-day aggregation is the right floor for v0.3, and l-diversity is not yet warranted.** k-anonymity (Sweeney 2002) guarantees that any cohort view cell is indistinguishable across at least k learners. k=10 is the conventional minimum for anonymized analytics (HIPAA Safe Harbor uses k=5 for direct identifiers but k=10 is the common bar for aggregate cells). The known limits — homogeneity attacks (all k learners share the same sensitive value) and background-knowledge attacks — are real but require a sensitive-attribute dimension that v0.3's cohort view does not yet expose (the view shows practice volume, mastery progression, failure patterns — not diagnosis, income, or other high-stake attributes). **Recommendation:** ship k=10 + 7-day aggregation for v0.3; defer l-diversity/t-closeness to a later milestone if/when a sensitive attribute enters the cohort schema. (Confidence: 0.80)
|
||||
|
||||
2. **k-anonymity suppression is a SQL `HAVING COUNT(*) >= 10` pattern with a NULL/suppressed sentinel for small cells.** The robust pattern is a two-pass query: (a) compute the cell counts over the grouping dimensions, (b) suppress any cell with `< k` learners by replacing the measure with a sentinel (`NULL` or `'--'`) — never delete the row (deletion itself is a side channel). For multi-dimensional views (path × week × outcome), generalize (collapse) the sparsest dimension first rather than suppressing individual cells, so that suppression is monotone and doesn't create "negative space" that re-identifies. **Recommendation:** implement suppression in the aggregation pipeline (Postgres-side), not in the React client; expose a single `cell_suppressed` boolean column to the UI. (Confidence: 0.85)
|
||||
|
||||
3. **7-day aggregation is the standard privacy/analytics tradeoff and matches D-034.** Daily windows are re-identification-prone (a single learner practicing on a given day is often unique); monthly windows are too stale for an operator dashboard. 7 days is the conventional middle ground (matches HIPAA's "small cell" suppression granularity and common analytics practice). REQ-NFR-DASH-02 mandates ≤24h staleness for the *aggregate*, not the window — i.e., the 7-day window can roll daily with a ≤24h lag. **Recommendation:** roll the 7-day window daily (a trailing 7-day aggregate, recomputed nightly), keeping the window wide for k-anonymity and the freshness high for the operator. (Confidence: 0.80)
|
||||
|
||||
4. **Differential privacy is not worth adopting at v0.3 scale (<100 learners).** DP's noise scales as O(1/ε) independent of N, so at N<100 the noise needed for a meaningful ε swamps the signal in cohort cells. k-anonymity + aggregation is the right tool at pilot scale; DP becomes attractive at N>1000 where k-anonymity's suppression starts to delete too many cells. **Recommendation:** defer DP to a later milestone; document the migration path (k-anonymity → DP) in ARCHITECTURE.md. (Confidence: 0.75)
|
||||
|
||||
5. **1PL/Rasch is the correct IRT model for v0.3; θ is initialized to 0 (the population mean) and b is initialized by expert rating then refined by E-M / marginal MLE as data accrues.** P(success) = logistic(θ − b) = 1/(1+e^(b−θ)). The Bayesian update for θ after a session is a conjugate-style update on the posterior: posterior ∝ likelihood × prior, where the likelihood is Bernoulli with the observed session outcome (success/failure per the rubric gate) and the prior is N(θ₀, σ₀²). The closed-form Gaussian approximation (Bayesian update on the natural-parameter scale) is cheap (<1ms, satisfies REQ-NFR-IRT-01). **Recommendation:** initialize θ₀=0, σ₀²=1 (a weakly-informative prior that the learner is near the population mean); update θ and σ² after each session via the Gaussian-approximation update; persist both in the `learner_ability` SQLite table (D-046). (Confidence: 0.85)
|
||||
|
||||
6. **Target ~50% expected success for item selection — the "zone of proximal development" (60–70%) claim does not transfer cleanly from the classroom literature.** The classical CAT (Computerized Adaptive Testing) literature (Wainer 2000; van der Linden 2010) targets P=0.5 because that's where Fisher information for the 1PL is maximized (the test is most discriminating when the learner is right at the item's difficulty). The ZPD framing (Vygotsky; 60–70% success) is about *instructional* tasks, not *assessment* — and v0.3 scenarios are both. The compromise used in modern adaptive learning systems (e.g., Knewton, Duolingo's birdie model) is to target ~70% during practice and ~50% during assessment-only gates. **Recommendation:** target P=0.5 for mastery-gate scenarios (assessment role) and P≈0.7 for non-gate practice scenarios (learning role). Make the target a per-scenario field in the YAML so it's tunable without code changes. (Confidence: 0.75)
|
||||
|
||||
7. **θ is reasonably reliable after ~5–10 sessions; the cold-start prior (θ₀=0, σ₀²=1) carries the first 3–5 sessions.** The posterior variance σ² shrinks roughly as 1/n for 1PL Bayesian updates, so after 5 sessions σ² ≈ 0.2 (SD ≈ 0.45 logits, roughly half a rubric level), and after 10 sessions σ² ≈ 0.1 (SD ≈ 0.32 logits). v0.3's mastery gate requires N=3 *distinct* scenarios (D-032), so the gate itself provides a natural minimum of 3 data points before any gate decision — but θ should still be reported with its posterior SD until σ² < 0.2. **Recommendation:** report θ ± SD to the operator dashboard (k-anonymized); require σ² < 0.2 before θ drives item selection (fall back to expert-rated b otherwise). (Confidence: 0.80)
|
||||
|
||||
8. **1PL breaks down when scenario discrimination varies materially across scenarios — which v0.3's 6 expert scenarios will.** The 2PL model P=exp[a(θ−b)]/(1+exp[...]) adds a discrimination parameter `a` per item. The rule of thumb from the psychometric literature is that 2PL is justifiable at ~200–500 response records per item (Lord 1980; Embretson & Reise 2000), and 3PL (with a guessing parameter) needs ~1000+ per item. At v0.3's scale (<100 learners × ~6 scenarios = <600 records, ~100 per item), 1PL is the only defensible model; 2PL would be overfit. **Recommendation:** ship 1PL for v0.3; revisit 2PL only when per-scenario response counts exceed ~200 (likely post-pilot, v0.5+). (Confidence: 0.80)
|
||||
|
||||
9. **`scenarios/index.yaml` should be a manifest of metadata, not a duplicate of scenario content.** Each entry should carry: `id`, `path`, `difficulty` (the IRT `b` estimate, possibly expert-rated initially), `failure_mode`, `rubric_criteria` (list of rubric-criterion IDs exercised), `tags`, `version` (semver), `author` (expert name or `ai-variation`), `generated_from` (backref to parent scenario ID, absent for expert-authored), `irt_target_p` (the target success probability for selection, default 0.5 for gate scenarios). The index is the catalog the scenario selector reads; the per-scenario YAML files hold the full Pipecat-flows DSL. **Recommendation:** index.yaml = catalog (slim, fast to load); per-scenario YAML = full content (loaded on demand). Version with semver `MAJOR.MINOR.PATCH` — bump MAJOR on rubric-criteria or branch-structure changes (changes scoring compatibility), MINOR on content additions, PATCH on prompt tweaks. (Confidence: 0.85)
|
||||
|
||||
10. **AI-generated variations need a mandatory expert-review gate before entering the live library, a `generated_from` backref, and a frozen `intent_hash` to detect drift.** The review workflow: (a) LLM generates a variation from an expert scenario's schema with a `generated_from: <parent_id>` field, (b) the variation is written to a `scenarios/_pending/` directory and is *invisible* to the selector, (c) an expert reviews the YAML in a PR-style diff against the parent, (d) on approval the variation moves to `scenarios/<path>/` and is added to `index.yaml`. The drift-prevention mechanism: an `intent_hash` (SHA-256 of the parent scenario's `success_criteria` + `failure_mode` + `rubric_criteria` fields) is recorded on the variation at generation time; if the parent's intent changes (hash differs), the variation is flagged as stale and re-review is required. **Recommendation:** ship the pending-review directory + `generated_from` + `intent_hash` fields in v0.3; do NOT auto-promote AI variations without expert sign-off (C-7: scenarios authored by domain experts; AI generates variations only). (Confidence: 0.80)
|
||||
|
||||
11. **Rubric-to-scenario mapping is a list of rubric-criterion IDs on each scenario; coverage is checked by inverting the map at load time.** The YAML field is `rubric_criteria: [criterion_id, ...]` on each scenario (per D-036/D-039). To ensure every criterion in a path's rubric is exercised by ≥ N scenarios, load `rubrics/customer_service.yaml`, build the criterion-ID set, then walk `scenarios/index.yaml` and count scenarios per criterion; assert the minimum. **Recommendation:** add a `scripts/check-coverage.py` (or bats check) that fails the build if any rubric criterion for a path has < 2 covering scenarios (N=2 for v0.3 — gives one expert + one variation or two expert scenarios per criterion). Run it in CI and as a pre-merge gate. (Confidence: 0.85)
|
||||
|
||||
---
|
||||
|
||||
## 1. Anonymization (k-anonymity, D-034)
|
||||
|
||||
### Q1 — k-anonymity, k=10, and limits (homogeneity, background-knowledge; l-diversity/t-closeness for v0.3)
|
||||
|
||||
**What k-anonymity is.** k-anonymity (Sweeney, *International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems* 2002) is a property of a released dataset (or aggregate view): for every combination of quasi-identifiers (the grouping dimensions — path, week, outcome, etc.), at least k records share that combination. Equivalently, no record is uniquely identifiable by the quasi-identifiers. The mechanism is generalization (collapsing values — e.g., age 23 → "20-30") and suppression (withholding cells with < k members).
|
||||
|
||||
**Why k=10 is the conventional minimum.** HIPAA Safe Harbor (45 CFR §164.514(b)) uses k=5 for *direct* identifiers in a released dataset (the 18-element rule). For *aggregate analytics cells* — which is what v0.3's cohort dashboard emits — the common bar in the privacy/analytics literature and in de-identification guidance (e.g., the CDC's re-identification risk guidance, the EU Pseudonymisation Best Practices) is k=10. The reasoning is that aggregate cells are subject to differencing attacks (subtracting two released aggregates to isolate a small subgroup), and a higher k than the direct-identifier minimum reduces the marginal risk. D-034's choice of k=10 is therefore the conventional, defensible floor.
|
||||
|
||||
**Limits of k-anonymity (the two classical attacks):**
|
||||
- **Homogeneity attack** (Machanavajjhala et al., *TODS* 2007, which introduced l-diversity): if all k learners in a cell share the same *sensitive* value, then knowing a target is in that cell reveals their sensitive value even though k-anonymity holds. Example: a cell of 10 learners who all failed the same week — knowing your competitor is in that cell tells you they failed.
|
||||
- **Background-knowledge attack**: an adversary with auxiliary information (e.g., "I know learner X practices on Tuesdays and is on week 3") can shrink the k-anonymity set to a smaller effective set and re-identify. k-anonymity is blind to this because it only counts released quasi-identifiers.
|
||||
|
||||
**l-diversity and t-closeness.** l-diversity (Machanavajjhala 2007) requires at least l *distinct* sensitive values per cell. t-closeness (Li, Li & Venkatasubramanian, *ICDE* 2007) requires the distribution of the sensitive attribute within a cell to be within t of the global distribution. Both address homogeneity; t-closeness additionally addresses skew attacks (where l-diversity is satisfied but the distribution is still skewed toward one value).
|
||||
|
||||
**Should v0.3 add l-diversity or t-closeness?** No — not for the pilot. The reason is structural: v0.3's cohort dashboard does not currently expose a *sensitive attribute* dimension in the sense the l-diversity/t-closeness literature assumes. The view dimensions are path/week/outcome/failure_pattern, and the measures are practice volume and mastery progression counts. None of these are sensitive in the way that diagnosis, income, or sexual orientation are. The homogeneity attack against "all 10 learners in this cell failed week 3" reveals a learning-struggle fact, which is lower-stakes than the medical/income facts these extensions were designed for. Adding l-diversity now would be engineering for a threat model the system doesn't yet have. The right trigger for revisiting l-diversity is *when a sensitive attribute enters the cohort schema* (e.g., if v0.4 adds demographic breakdowns). Document that trigger in ARCHITECTURE.md.
|
||||
|
||||
**Recommendation (D-034):** ship k=10 + 7-day aggregation for v0.3. Defer l-diversity/t-closeness with an explicit re-evaluation trigger: "revisit when any cohort-view dimension or measure becomes a sensitive attribute (demographic, socio-economic, health-related)." Keep the aggregation pipeline structured so adding l-diversity later is a localized change (one suppression predicate).
|
||||
|
||||
**Confidence: 0.80** — the k=10 convention is well-established; the l-diversity deferral is a threat-model judgment that depends on v0.3's exact cohort schema, which is not yet finalized. If the operator dashboard later adds a demographic filter, this deferral is wrong and l-diversity becomes required.
|
||||
|
||||
### Q2 — SQL suppression pattern; multi-dimensional views without re-identification
|
||||
|
||||
**Single-dimension suppression.** The canonical pattern for "any cohort view cell with < 10 learners is suppressed":
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
path,
|
||||
week,
|
||||
outcome,
|
||||
CASE WHEN COUNT(DISTINCT learner_id) >= 10
|
||||
THEN COUNT(*)
|
||||
ELSE NULL
|
||||
END AS session_count,
|
||||
CASE WHEN COUNT(DISTINCT learner_id) >= 10
|
||||
THEN TRUE ELSE FALSE
|
||||
END AS cell_suppressed
|
||||
FROM session_aggregates
|
||||
WHERE window_start >= now() - interval '7 days'
|
||||
GROUP BY path, week, outcome;
|
||||
```
|
||||
|
||||
Two non-obvious but critical details:
|
||||
1. **Suppress the measure, not the row.** Deleting the row creates a "negative space" side channel: an adversary who knows the dimension space can enumerate all combinations and infer that a missing cell had < 10 learners — which, combined with background knowledge, can re-identify. Replacing the measure with `NULL` (or a `'--'` sentinel) and emitting the cell with `cell_suppressed = TRUE` preserves the dimension grid and only hides the count.
|
||||
2. **Use `COUNT(DISTINCT learner_id)`, not `COUNT(*)`.** A single learner can have many sessions in the window; `COUNT(*)` over-counts and produces false confidence that k=10 is met when only 3 learners are present. k-anonymity is about *people*, not *records*.
|
||||
|
||||
**Multi-dimensional views (path × week × outcome × failure_pattern).** The naive approach — suppress each cell independently — leaks via *differencing*: an adversary subtracts two released aggregates (e.g., "week 3 outcomes" minus "week 3 outcomes where failure_pattern = escalates_unresolved") to recover the suppressed subcell. The standard defenses are:
|
||||
- **Generalization (collapse the sparsest dimension first):** if path × week × outcome × failure_pattern has cells with < 10 learners, drop the sparsest dimension (usually failure_pattern) and re-emit at path × week × outcome. If still under k, drop outcome, etc. The release is a *lattice* of generalizations, not a flat table.
|
||||
- **Minimality / consistency constraints** (the approach from the k-anonymity generalization literature, e.g., LeFevre, DeWitt & Ramakrishnan, *SIGMOD* 2005): the released cells must be *minimal* — you can't suppress a cell when its parent generalization already satisfies k — and *consistent* — no two released cells overlap such that differencing recovers a suppressed cell.
|
||||
|
||||
For v0.3's pilot, the pragmatic approach is to (a) limit the cohort view to two dimensions at a time (e.g., path × week, OR path × outcome, but not path × week × outcome), which eliminates differencing across dimensions entirely; and (b) within each two-dimensional view, suppress cells with < 10 distinct learners using the pattern above. The operator UI presents a small fixed set of pre-defined 2-D views (no free-form cross-tabulation), which is sufficient for "practice volume, mastery progression, failure patterns" per REQ-DASH-01.
|
||||
|
||||
**Recommendation:** implement suppression Postgres-side in the aggregation pipeline (D-045's hook + nightly job); expose a fixed set of pre-defined 2-D cohort views; emit `cell_suppressed` boolean to the React client; render suppressed cells as `--` in the UI. Do NOT allow free-form cross-tabulation by the operator in v0.3.
|
||||
|
||||
**Confidence: 0.85** — the SQL pattern is canonical; the 2-D-view constraint is a pragmatic pilot choice that trades operator flexibility for re-identification safety. If operators need 3-D views, generalize (collapse) rather than allow free-form.
|
||||
|
||||
### Q3 — 7-day aggregation window: why 7 days, shorter-window risk, freshness tradeoff
|
||||
|
||||
**Why 7 days.** Three reasons, in descending order of weight:
|
||||
1. **Re-identification risk of shorter windows is high.** A daily window (or hourly) makes most cohort cells contain 1–3 learners (a single learner practicing on a given day is often unique in their path × week combination), so almost every cell would have to be suppressed, leaving the operator with a blank dashboard. Weekly windows aggregate enough practice that cells naturally exceed k=10 for active cohorts.
|
||||
2. **Practice periodicity is weekly.** Learners in a mastery-paced 6-week path (D-037) practice on the order of once a day to a few times a week; a 7-day window captures one full practice cycle and aligns with the path's week structure (the dashboard's "week" dimension matches the aggregation window, which is intuitive for operators).
|
||||
3. **Conventional granularity.** HIPAA Safe Harbor's "small cell" guidance, CDC re-identification guidance, and common analytics practice all treat 7-day (or coarser) aggregates as the privacy-friendly default for small populations.
|
||||
|
||||
**Re-identification risk of shorter windows.** A 1-day window: a cohort of 50 learners across 6 path-weeks gives ~8 learners per cell on average — already under k=10, so most cells suppressed. An adversary who knows "learner X practiced on Tuesday" can pin them to a specific daily cell; if that cell has 1–3 learners, re-identification is feasible. A 1-hour window is worse still. The risk scales inversely with window length for small populations.
|
||||
|
||||
**Freshness/staleness tradeoff.** The dashboard's freshness NFR (REQ-NFR-DASH-02: ≤ 24h staleness) is about *when the aggregate is computed*, not the window length. These are independent: a trailing 7-day window can be recomputed every hour (freshness 1h) or every day (freshness 24h). The window length is a *privacy* parameter; the recomputation cadence is a *freshness* parameter. The right design for v0.3 is a 7-day trailing window recomputed daily (or on each session-end per D-045's hook), giving 24h freshness on a 7-day-wide window. Shorter recomputation cadence (e.g., per-session) is fine — it doesn't change the window length.
|
||||
|
||||
**Recommendation (D-034):** 7-day trailing window, recomputed on session-end hook (low-latency incremental update) + nightly reconciliation job (correctness). Document explicitly that "7-day aggregation window" ≠ "7-day staleness" — the window is 7 days wide, the staleness is ≤24h per REQ-NFR-DASH-02.
|
||||
|
||||
**Confidence: 0.80** — the 7-day choice is conventional and well-justified for pilot scale; the freshness/window-length distinction is sometimes conflated in privacy guidance, which is why D-034's phrasing deserves the clarifying note above.
|
||||
|
||||
### Q4 — Differential privacy at v0.3 scale (<100 learners): adopt or defer?
|
||||
|
||||
**What differential privacy (DP) gives you that k-anonymity doesn't.** DP (Dwork, *ICALP* 2006) is a formal guarantee: the output distribution is nearly the same whether or not any individual's data is in the input. This protects against *all* auxiliary information (the background-knowledge attack that k-anonymity is blind to) and gives a quantifiable privacy budget (ε, δ). Mechanisms like the Laplace or Gaussian mechanism add noise calibrated to the query's sensitivity and the chosen ε.
|
||||
|
||||
**Why DP is the wrong tool at <100 learners.** The noise a DP mechanism adds is O(1/ε) *independent of N* — it does not shrink as the population grows. For a count query with sensitivity 1 and a privacy budget of ε=1 (a common, reasonably-private choice), the Laplace noise has scale 1 — meaning a true count of 8 might be released as 7, 8, 9, 10 with non-trivial probability. At N=50 learners in a cell, that's ±1–2 noise on a count of 50 — tolerable. At N=10 (the k-anonymity floor), ±1–2 noise on a count of 10 is ±10–20% relative error — the dashboard becomes meaningfully inaccurate. Worse, to maintain DP across many queries (the cohort dashboard emits many cells), the privacy budget must be *split* across them (composition), so each cell gets ε/M for M cells — and the noise scales as M/ε. A 6-path × 6-week × 4-outcome = 144-cell dashboard at total ε=1 gives ε_cell ≈ 0.007 — noise scale ~140, which makes the release pure noise.
|
||||
|
||||
k-anonymity, by contrast, has *no noise* — it either releases the exact count (when ≥ k) or suppresses (when < k). At small N, the suppression rate is the cost; at large N, suppression disappears and k-anonymity releases exact counts (which DP never does). The crossover where DP starts to outperform k-anonymity on the utility/privacy frontier is roughly N > 1000 for multi-cell dashboards (the exact threshold depends on the query workload and ε).
|
||||
|
||||
**Recommendation (D-034):** defer DP to a later milestone (target: when active learner count exceeds ~1000 or when a sensitive attribute enters the cohort schema, whichever comes first). Ship k-anonymity + aggregation for v0.3. Document the migration path in ARCHITECTURE.md: the aggregation pipeline's suppression step is a single function that can be swapped for a DP mechanism later — the rest of the pipeline (grouping, dimensions, UI rendering of `cell_suppressed`) is DP-agnostic.
|
||||
|
||||
**Confidence: 0.75** — the DP-at-small-N argument is well-grounded in the DP literature (Dwork & Roth 2014); the 1000-learner crossover is a rule-of-thumb, not a hard threshold, and depends on the exact query workload.
|
||||
|
||||
---
|
||||
|
||||
## 2. IRT (Item Response Theory, D-035)
|
||||
|
||||
### Q5 — 1PL/Rasch model: P(success)=logistic(θ−b), initialization, Bayesian θ update
|
||||
|
||||
**The model.** The 1PL (one-parameter logistic) / Rasch model gives the probability of success on scenario j by learner i as:
|
||||
|
||||
P(X_ij = 1 | θ_i, b_j) = 1 / (1 + exp(b_j − θ_i)) = logistic(θ_i − b_j)
|
||||
|
||||
where θ_i is learner i's ability (a scalar, in logits) and b_j is scenario j's difficulty (also in logits). The model is symmetric in θ and b: a learner of ability θ has P=0.5 on a scenario of difficulty b=θ; P>0.5 when θ>b; P<0.5 when θ<b.
|
||||
|
||||
**Initialization of θ (learner ability).** Three common choices:
|
||||
1. **Population mean (θ₀ = 0).** The conventional default. The logit scale is defined up to a translation, so fixing the population mean at 0 sets the scale. This is the right choice when there's no prior information about the learner.
|
||||
2. **Cold-start placement test.** Some CAT systems administer a short placement test to initialize θ. Praxis v0.3 has no quizzes (REQ-MAST-04: assessment is built into scenarios), so this is not available — the first scenario *is* the placement test.
|
||||
3. **Cohort-conditional prior.** If path-level performance data exists, initialize θ₀ to the mean θ of learners who have completed the path. Not available at v0.3 launch (no prior cohort).
|
||||
|
||||
**Recommendation:** θ₀ = 0 (population mean), prior variance σ₀² = 1 (weakly-informative — says "the learner is probably within ±2 logits of the population mean, which is ±2 rubric levels roughly"). This is the standard cold-start prior and is what py-irt, mirt (R), and pyjirt use by default.
|
||||
|
||||
**Initialization of b (scenario difficulty).** Three choices, in increasing data-intensity:
|
||||
1. **Expert rating (cold-start).** Have the scenario author rate the difficulty on the 1–5 rubric scale, then map to logits via b = (rating − 3) × c, where c is a scale factor (commonly c ≈ 1 logit per rubric level, calibratable). This is the only option at v0.3 launch — there is no response data yet.
|
||||
2. **E-M / marginal MLE from response data.** Once ~20+ response records exist for a scenario, estimate b via the Bock-Aitkin E-M algorithm (the standard IRT calibration method). This is offline, batch, and not in the voice path.
|
||||
3. **Joint MLE / hierarchical Bayes.** Estimates θ and b jointly; needs more data and is overkill for v0.3.
|
||||
|
||||
**Recommendation:** initialize b from expert rating at scenario authoring time (record `difficulty_expert: 1-5` in the YAML, derive `b_init`); recalibrate b offline (nightly job) via E-M once per-scenario response counts exceed ~20. Store both `b_init` and `b_calibrated` in `index.yaml`; the selector uses `b_calibrated` when available, else `b_init`.
|
||||
|
||||
**Bayesian update of θ after a session.** The session produces an outcome X ∈ {0, 1} (failure/success per the rubric gate — D-032). The posterior is:
|
||||
|
||||
p(θ | X) ∝ p(X | θ, b) × p(θ)
|
||||
= Bernoulli(X; logistic(θ − b)) × Normal(θ; θ_current, σ²_current)
|
||||
|
||||
This posterior is not Gaussian in closed form (the Bernoulli likelihood is logistic, not Gaussian). Two practical options:
|
||||
|
||||
**Option A — Gaussian approximation (Laplace / moment matching).** Approximate the posterior as Gaussian by matching the mode (MAP) and curvature. The update (one step of Newton's method on the log-posterior):
|
||||
|
||||
z = X − P_current # residual, P_current = logistic(θ_current − b)
|
||||
W = P_current × (1 − P_current) # variance of the Bernoulli
|
||||
θ_new = θ_current + (σ²_current × z) / (1 + W × σ²_current)
|
||||
σ²_new = σ²_current / (1 + W × σ²_current)
|
||||
|
||||
This is the standard "assumed density filtering" / "Bayesian logistic regression with a Gaussian prior" online update. It's O(1), well under 1ms (satisfies REQ-NFR-IRT-01's < 100ms), and is what most production adaptive learning systems use (Knewton's early models, Duolingo's half-life regression variant).
|
||||
|
||||
**Option B — Particle filter / grid approximation.** Maintain a discrete grid of θ values with weights; update weights by the Bernoulli likelihood. More accurate for the first few sessions when the Gaussian approximation is poor, but more code and slightly slower (still < 10ms for a 50-point grid). Overkill for v0.3.
|
||||
|
||||
**Recommendation (D-035):** Option A (Gaussian approximation). Initialize (θ=0, σ²=1). After each session-end, compute X from the rubric gate, look up b for the scenario, and apply the two-line update above. Persist (θ, σ², updated_at) in the `learner_ability` SQLite table per D-046. The update is in-process, no LLM call, < 1ms — comfortably within REQ-NFR-IRT-01.
|
||||
|
||||
**Confidence: 0.85** — the 1PL/Rasch model and the Gaussian-approximation Bayesian update are textbook psychometrics; the only judgment call is the prior variance (σ²=1), which is conventional but could be tuned once v0.3 produces real θ distributions.
|
||||
|
||||
### Q6 — Item selection: target P=0.5 or P=0.6–0.7 (ZPD)?
|
||||
|
||||
**The case for P=0.5 (max information).** In the 1PL model, the Fisher information about θ contained in a scenario of difficulty b is:
|
||||
|
||||
I(θ, b) = P(θ, b) × (1 − P(θ, b))
|
||||
|
||||
which is maximized at P=0.5 (i.e., b = θ). This is the theoretical basis for the classical CAT selection rule (Lord 1980, Wainer 2000, van der Linden 2010): pick the item that maximizes information about the learner's current θ, which is the item with b closest to θ. CAT systems used in high-stakes assessment (GRE, GMAT, ASVAB) target P=0.5 because their goal is to *estimate θ precisely in the fewest items* — efficiency.
|
||||
|
||||
**The case for P≈0.7 (zone of proximal development).** Vygotsky's ZPD framing — learners learn best on tasks slightly above their current independent level — has been interpreted in adaptive learning as targeting ~70–85% success (the learner succeeds most of the time but is stretched). Bjork's "desirable difficulties" framework argues for *some* failure to enhance long-term retention. The Knewton and Duolingo production systems target roughly 70–85% success during practice (Duolingo's "birdie" model targets ~80% recall).
|
||||
|
||||
**The conflict and the resolution.** The two targets answer different questions:
|
||||
- P=0.5 optimizes for *assessment precision* (estimating θ).
|
||||
- P=0.7 optimizes for *learning* (retention, engagement, low frustration).
|
||||
|
||||
Praxis v0.3 scenarios are *both* assessment and practice — they're scored against a rubric (assessment) and they're how the learner practices (learning). The split is:
|
||||
- **Mastery-gate scenarios** (the N=3 distinct scenarios that open a gate per D-032) are assessment: their purpose is to determine if the learner has mastered the week. Target P=0.5 (max information, hardest to game).
|
||||
- **Non-gate practice scenarios** are learning: their purpose is to develop the skill. Target P≈0.7 (ZPD, retention-friendly).
|
||||
|
||||
**Recommendation (D-035):** add a per-scenario `irt_target_p` field to the YAML (default 0.5 for gate scenarios, 0.7 for practice scenarios). The selector picks the unplayed scenario whose expected P = logistic(θ − b) is closest to the scenario's `irt_target_p`. This makes the target a content-authoring decision, not a code change, and lets learning designers tune per scenario. REQ-SCEN-02's "targeting ~50% expected success" is correct for the gate scenarios; the practice scenarios should deviate to 0.7.
|
||||
|
||||
**Confidence: 0.75** — the Fisher-information argument for P=0.5 is rigorous; the ZPD argument for P=0.7 is empirically supported in adaptive-learning production systems but less theoretically clean (Vygotsky's ZPD is a social-constructivist concept, and the "70%" mapping is a pragmatic interpretation, not a derived constant).
|
||||
|
||||
### Q7 — Cold-start: how many sessions until θ is reliable? What prior?
|
||||
|
||||
**How θ's posterior variance shrinks.** Under the Gaussian-approximation update in Q5, the posterior variance σ² shrinks by a factor (1 + W·σ²_current) per update, where W = P(1−P) ≤ 0.25. In the best case (P=0.5, W=0.25), each session halves σ² (when σ²=1: σ² → 1/(1+0.25) = 0.8 → 0.615 → 0.492 → ...). In the worst case (P near 0 or 1, W near 0), the session is uninformative and σ² barely shrinks. So the *number of sessions to reliability* depends on whether the scenarios are well-targeted (P near 0.5) or mis-targeted (P near 0 or 1).
|
||||
|
||||
Rough trajectory (assuming well-targeted scenarios, P≈0.5):
|
||||
- Start: σ² = 1.0 (SD = 1.0 logits, ±1 rubric level)
|
||||
- After 3 sessions: σ² ≈ 0.5 (SD = 0.7 logits, ±0.7 rubric level) — *this is when the mastery gate's N=3 distinct scenarios are first usable*
|
||||
- After 5 sessions: σ² ≈ 0.33 (SD = 0.57 logits)
|
||||
- After 10 sessions: σ² ≈ 0.18 (SD = 0.43 logits)
|
||||
- After 20 sessions: σ² ≈ 0.09 (SD = 0.30 logits)
|
||||
|
||||
**Rule of thumb:** θ is "reliable enough to drive item selection" at σ² < 0.2 (SD < ~0.45 logits, i.e., we know θ within half a rubric level), which takes ~5–10 well-targeted sessions. θ is "reliable enough to report on the cohort dashboard" at σ² < 0.1, which takes ~15–20 sessions.
|
||||
|
||||
**The cold-start prior.** The prior N(0, 1) says "the learner is probably within ±2 logits of the population mean," which is weakly informative. For v0.3 (no prior cohort data), this is the only defensible choice. Two alternatives, both deferred:
|
||||
- **Empirical Bayes prior:** once a cohort of learners has been through the path, set the prior mean/variance to the cohort's θ mean/variance. This shrinks the cold-start period for new learners.
|
||||
- **Path-conditional prior:** if different paths have different difficulty baselines, set the prior per path. Not needed in v0.3 (one path: Customer Service).
|
||||
|
||||
**The mastery-gate interaction.** D-032's mastery gate requires N=3 distinct-scenario successes with rubric mean ≥ 3.5/5.0. The gate is a *rule-based* condition independent of θ — the gate can open before θ is "reliable" by the σ² criterion. This is fine: the gate is the authoritative mastery signal; θ is for *item selection*, not for *mastery certification*. Don't conflate the two.
|
||||
|
||||
**Recommendation:**
|
||||
- Cold-start prior: N(0, 1) for θ at first session per path.
|
||||
- Item selection: use θ to select scenarios even from session 1 (with the broad prior, the selector will pick scenarios near b=0, which is correct — mid-difficulty).
|
||||
- Report θ to the operator dashboard only when σ² < 0.2 (else show "warming up — N sessions until reliable").
|
||||
- Mastery gate (D-032) is independent of θ's reliability — it's rule-based on rubric scores. Document this separation clearly.
|
||||
|
||||
**Confidence: 0.80** — the variance-shrinkage trajectory is derivable from the update equations; the σ² < 0.2 threshold for "reliable enough to report" is a judgment call (some systems use 0.1, some 0.25) but 0.2 is the common middle.
|
||||
|
||||
### Q8 — 1PL vs 2PL/3PL: when does 1PL break down? What data volume justifies 2PL?
|
||||
|
||||
**1PL (Rasch).** P = logistic(θ − b). One parameter per item (b). Assumes all items discriminate equally (the slope of the item characteristic curve is the same for every item). Strength: parsimonious, estimable from few responses per item (~20–50), θ is on an interval scale (specific objectivity — a defining Rasch property), and the model is robust to moderate violations of the equal-discrimination assumption.
|
||||
|
||||
**2PL.** P = logistic(a(θ − b)) where a is the item discrimination (slope). Two parameters per item. Allows items to differ in how sharply they distinguish learners above vs below the difficulty. A high-a item is very informative near b; a low-a item is weakly informative everywhere. Strength: better fit when discrimination genuinely varies. Weakness: needs more data to estimate `a` stably; θ loses specific objectivity (comparisons depend on the item set).
|
||||
|
||||
**3PL.** Adds a guessing parameter `c` (lower asymptote): P = c + (1−c)·logistic(a(θ−b)). Models the probability that a low-ability learner gets the item right by guessing. Useful for multiple-choice tests; **not applicable to Praxis** (scenarios are free-form voice role-plays, not multiple-choice — there is no "guessing" in the 3PL sense). 3PL needs ~1000+ responses per item to estimate `c` stably.
|
||||
|
||||
**When does 1PL break down?** 1PL is misspecified when the item discriminations vary substantially — i.e., when some scenarios are much better at distinguishing competent from incompetent learners than others. In Praxis terms, this would happen if (say) a "policy quote retrieval" scenario (high discrimination — only competent learners handle it) and a "smile and nod" scenario (low discrimination — everyone succeeds) are both in the library. The 1PL model would force both to have the same slope, distorting θ estimates. The empirical diagnostic is to fit 2PL, inspect the `a` estimates, and check if they cluster near a common value (1PL is fine) or spread widely (1PL is misspecified).
|
||||
|
||||
**Data volume thresholds (rule of thumb from the psychometric literature):**
|
||||
- 1PL: ~20–50 responses per item for stable b estimates.
|
||||
- 2PL: ~200–500 responses per item for stable `a` estimates (Lord 1980; Embretson & Reise 2000).
|
||||
- 3PL: ~1000+ responses per item.
|
||||
|
||||
**Praxis v0.3 numbers:** < 100 learners × 6 expert scenarios = < 600 total response records, ~100 per scenario (optimistically — not every learner plays every scenario). This is well above the 1PL threshold (~20–50) and well below the 2PL threshold (~200–500). 1PL is the only defensible model for v0.3; 2PL would be overfit and the `a` estimates would be noise.
|
||||
|
||||
**Recommendation (D-035):** ship 1PL for v0.3. Revisit 2PL when per-scenario response counts exceed ~200 (likely post-pilot, v0.5+). 3PL is permanently out of scope (no guessing in voice role-plays). When 2PL is adopted, fit it offline (E-M or MML); the online θ update generalizes naturally (the Gaussian-approximation update uses W = a²P(1−P) instead of P(1−P)).
|
||||
|
||||
**Confidence: 0.80** — the data-volume thresholds are well-established in the psychometric literature; the 1PL-for-v0.3 conclusion is robust to the exact learner count.
|
||||
|
||||
---
|
||||
|
||||
## 3. Scenario Library (D-036, D-047)
|
||||
|
||||
### Q9 — `scenarios/index.yaml` contents and scenario versioning
|
||||
|
||||
**Directory structure (per D-036):**
|
||||
|
||||
```
|
||||
scenarios/
|
||||
index.yaml # manifest / catalog
|
||||
customer_service/
|
||||
cs_refund_ca_v01.yaml # expert-authored
|
||||
cs_refund_exchange_v01.yaml # expert-authored
|
||||
cs_complaint_escalation_v01.yaml # expert-authored
|
||||
...
|
||||
_pending/ # AI variations awaiting review
|
||||
cs_refund_exchange_ai01.yaml
|
||||
cost_rates.yaml # existing v0.1 file
|
||||
rubric_criteria/ # optional: shared criterion defs
|
||||
empathy.yaml
|
||||
paths/
|
||||
customer_service.yaml # the 6-week path (D-037)
|
||||
rubrics/
|
||||
customer_service.yaml # the rubric (D-039)
|
||||
```
|
||||
|
||||
The existing `scenarios/customer_service_refund_ca_v01.yaml` is currently at the top level (flat); v0.3 nests it under `scenarios/customer_service/` to support the multi-path library. The flat layout worked for v0.1's single scenario; the nested layout is needed for v0.3's ≥6 scenarios across (initially) one path and (later) multiple paths.
|
||||
|
||||
**`index.yaml` contents (the manifest).** The index is a *catalog*, not a duplicate of scenario content. It carries the metadata the scenario selector and coverage checker need without loading every YAML file:
|
||||
|
||||
```yaml
|
||||
# scenarios/index.yaml — manifest, regenerated on library changes
|
||||
version: 1
|
||||
path_scenarios:
|
||||
customer_service:
|
||||
- id: cs_refund_ca_v01
|
||||
file: customer_service/cs_refund_ca_v01.yaml
|
||||
difficulty_expert: 1 # 1-5 expert rating (cold-start b)
|
||||
difficulty_calibrated: 0.4 # IRT b in logits, null until calibrated
|
||||
failure_mode: escalates_unresolved
|
||||
rubric_criteria: [empathy, concrete_resolution, next_steps]
|
||||
tags: [refund, damaged_product, ca_market]
|
||||
irt_target_p: 0.5 # gate scenario → max info
|
||||
version: 1.0.0
|
||||
author: expert_jane_doe
|
||||
generated_from: null # null = expert-authored; <parent_id> = AI variation
|
||||
intent_hash: <sha256 of success_criteria+failure_mode+rubric_criteria>
|
||||
status: live # live | pending | deprecated
|
||||
- id: cs_refund_exchange_ai01
|
||||
file: customer_service/cs_refund_exchange_ai01.yaml
|
||||
...
|
||||
generated_from: cs_refund_ca_v01
|
||||
status: pending # in _pending/, not selectable
|
||||
```
|
||||
|
||||
**Why index.yaml is separate from per-scenario YAMLs.** Loading 6+ full scenario YAMLs (each with multi-paragraph system prompts, branch definitions, rubric mappings) just to pick the next one is wasteful. The index is a slim catalog (~50 lines per scenario) loaded once at startup; the full scenario YAML is loaded on demand when selected. This also keeps the selector's logic testable without the LLM-prompt content.
|
||||
|
||||
**Versioning.** Use semver `MAJOR.MINOR.PATCH` per scenario, recorded in the scenario YAML and mirrored in `index.yaml`:
|
||||
- **MAJOR:** changes that break scoring compatibility — rubric_criteria added/removed, branch-structure changes, success_criteria semantics change. A MAJOR bump invalidates prior mastery-gate evidence (the learner's prior passes on the old version don't count toward the new version's gate).
|
||||
- **MINOR:** content additions — new common_mistakes, new branch (non-scoring), prompt enrichment. Backward-compatible with prior scoring.
|
||||
- **PATCH:** prompt tweaks, typo fixes, voice_id changes. No semantic change.
|
||||
|
||||
The `version` field on each scenario lets the mastery-gate audit log (REQ-NFR-MAST-02) record which scenario version a learner passed, so future re-authoring doesn't retroactively invalidate credentials.
|
||||
|
||||
**Recommendation (D-036):**
|
||||
- Nest scenarios under `scenarios/<path>/`.
|
||||
- `index.yaml` is a slim manifest (metadata only, ~50 lines/scenario).
|
||||
- Per-scenario YAML is the full Pipecat-flows DSL, loaded on demand.
|
||||
- Semver per scenario; MAJOR bumps invalidate prior gate evidence.
|
||||
- Add a `regenerate_index.py` (or bats check) that re-derives `index.yaml` from the scenario files and asserts they're in sync — prevents manual drift.
|
||||
|
||||
**Confidence: 0.85** — the index/manifest split is a standard content-management pattern; the semver scheme is conventional. The only judgment call is treating rubric_criteria changes as MAJOR (scoring-compatibility-breaking), which is the conservative choice.
|
||||
|
||||
### Q10 — AI-generated variations: review workflow, generated_from backref, drift prevention
|
||||
|
||||
**The workflow (per D-047, C-7).** C-7 (binding constraint) states "Scenarios authored by domain experts + learning designers; AI generates variations only." D-047 specifies "AI-generated variations gated by expert review." The concrete workflow:
|
||||
|
||||
```
|
||||
1. GENERATE
|
||||
- Input: an expert scenario YAML (e.g., cs_refund_ca_v01.yaml)
|
||||
- LLM (deepseek-v4-flash:cloud with think mode — offline, not latency-bound)
|
||||
generates a variation by perturbing the scenario while preserving
|
||||
success_criteria + failure_mode + rubric_criteria.
|
||||
- Output: a new YAML in scenarios/<path>/_pending/<id>.yaml with:
|
||||
generated_from: cs_refund_ca_v01
|
||||
intent_hash: <sha256 of parent's success_criteria+failure_mode+rubric_criteria>
|
||||
status: pending
|
||||
author: ai_variation_<model_version>
|
||||
|
||||
2. REVIEW (expert, human-in-the-loop)
|
||||
- Expert opens a PR-style diff: pending YAML vs parent YAML.
|
||||
- Expert checks: does the variation still exercise the same rubric_criteria?
|
||||
Is the failure_mode still reachable? Is the system_prompt safe + in-character?
|
||||
- Expert may edit the variation (the LLM output is a draft, not final).
|
||||
- On approval: expert moves the file from _pending/ to scenarios/<path>/
|
||||
and adds it to index.yaml with status: live.
|
||||
|
||||
3. PUBLISH
|
||||
- The variation is now selectable by the IRT scenario selector.
|
||||
- It carries generated_from permanently (for provenance/audit).
|
||||
- Its intent_hash is frozen at generation time.
|
||||
|
||||
4. DRIFT DETECTION (ongoing)
|
||||
- If the parent scenario is re-authored (MAJOR version bump) and its
|
||||
success_criteria/failure_mode/rubric_criteria change, the parent's
|
||||
intent_hash changes. All variations generated_from that parent are
|
||||
flagged as stale (their intent_hash no longer matches the parent).
|
||||
- Stale variations are moved back to _pending/ and require re-review
|
||||
before they're selectable again.
|
||||
```
|
||||
|
||||
**The `generated_from` backref.** A single field on the variation YAML pointing to the parent scenario ID. Absent (or null) on expert-authored scenarios. This is the provenance chain — it lets the audit log answer "was this mastery-gate evidence collected on an expert scenario or an AI variation, and if the latter, from which expert scenario was it derived?" The chain is one level deep (an AI variation is generated from an expert scenario, not from another AI variation) — this is a deliberate constraint to prevent variation-of-variation drift. Enforce it at generation time.
|
||||
|
||||
**Drift prevention via `intent_hash`.** The intent of a scenario is defined as the tuple (success_criteria, failure_mode, rubric_criteria) — the parts that determine what the scenario *assesses*. The `intent_hash` is SHA-256 of the canonical JSON encoding of that tuple. At generation time, the variation records the parent's intent_hash. If the parent's intent later changes (re-authoring changes the rubric_criteria, say), the parent's hash changes and the variation is flagged stale. This catches the case where an expert reauthors the parent in a way that the variation no longer faithfully represents — without requiring the expert to manually track all variations.
|
||||
|
||||
**Preventing drift from the expert's intent (the deeper question).** The intent_hash catches *parent-side* drift. *Variation-side* drift — the LLM produces a variation that superficially matches the schema but subtly changes the assessed skill (e.g., makes the customer less angry, turning an empathy test into a transaction test) — is caught only by expert review. The intent_hash does NOT verify semantic fidelity. Two mitigations:
|
||||
1. **The rubric-to-scenario mapping is part of the intent tuple.** If the LLM drops a rubric criterion, the variation's intent_hash differs from the parent's, and the variation is auto-flagged stale (without needing expert review). This catches structural drift.
|
||||
2. **Expert review is the only defense against semantic drift within the same rubric_criteria.** No automated check can verify "is this customer still angry enough to test empathy." This is why C-7 makes expert review mandatory, not optional.
|
||||
|
||||
**Recommendation (D-047, REQ-SCEN-04):**
|
||||
- Ship the `_pending/` directory + `generated_from` backref + `intent_hash` fields in v0.3.
|
||||
- AI variations are generated offline by `scripts/generate_variation.py` (a CLI tool, not in the voice path); output goes to `_pending/`.
|
||||
- Expert review is mandatory; no auto-promotion. The review is a git PR against the `scenarios/` directory — the expert reviews the YAML diff.
|
||||
- One-level variation chain only (no variations of variations).
|
||||
- `intent_hash` catches structural drift (rubric_criteria change); expert review catches semantic drift.
|
||||
- The ≥6 expert scenarios in D-047 are the floor; AI variations are supplemental and cannot substitute for the expert floor.
|
||||
|
||||
**Confidence: 0.80** — the workflow is sound and matches industry practice for AI-assisted content authoring (e.g., how Khanmigo, Duolingo's GPT-4 content pipeline handle AI-generated exercises). The `intent_hash` mechanism is a Praxis-specific design; it's a reasonable heuristic for structural drift but is not a published technique, hence the 0.80 not 0.95.
|
||||
|
||||
### Q11 — Rubric-to-scenario mapping: YAML field shape, coverage across a path
|
||||
|
||||
**The YAML field shape.** Each scenario declares which rubric criteria it exercises via a `rubric_criteria` field — a list of criterion IDs that reference the rubric file (`rubrics/customer_service.yaml` per D-039):
|
||||
|
||||
```yaml
|
||||
# scenarios/customer_service/cs_refund_ca_v01.yaml
|
||||
id: cs_refund_ca_v01
|
||||
path: customer_service
|
||||
# ... existing v0.1 fields ...
|
||||
rubric_criteria:
|
||||
- criterion_id: empathy
|
||||
weight: 1.0 # relative weight within this scenario (default 1.0)
|
||||
evidence_required: true # must be observed to count toward mastery
|
||||
- criterion_id: concrete_resolution
|
||||
weight: 1.0
|
||||
evidence_required: true
|
||||
- criterion_id: next_steps
|
||||
weight: 0.5
|
||||
evidence_required: false
|
||||
```
|
||||
|
||||
Two design choices in this shape:
|
||||
1. **List of objects, not a list of strings.** Each entry carries a `criterion_id` (referencing the rubric) plus per-scenario metadata about that criterion (weight within this scenario, whether evidence is required). A bare list of strings (`rubric_criteria: [empathy, concrete_resolution, next_steps]`) is simpler but loses the per-scenario weighting — and weighting matters because a scenario may exercise one criterion as the primary skill and another as secondary.
|
||||
2. **Reference by ID, not inline.** The criterion's full definition (5-level anchors, weight-within-skill) lives in `rubrics/customer_service.yaml` (per D-039). The scenario references it by ID. This keeps the rubric single-source (a criterion's anchors are defined once) and lets the coverage checker work on IDs without parsing every scenario's full content.
|
||||
|
||||
**Coverage across a path.** D-032's mastery gate requires N=3 distinct-scenario successes. For the gate to be meaningful, the N scenarios must collectively exercise *all* the rubric's criteria — otherwise a learner could pass the gate by succeeding on scenarios that only test a subset of the skill. The coverage requirement is: *every rubric criterion for the path is exercised by ≥ M scenarios, where M ≥ 2* (so there's at least one expert scenario and one alternative — an AI variation or a second expert scenario — to prevent single-scenario gaming).
|
||||
|
||||
**Coverage check (load-time).** Build the rubric-criterion-ID set from `rubrics/customer_service.yaml`, walk `scenarios/index.yaml`, and count scenarios per criterion (only `status: live` scenarios count):
|
||||
|
||||
```python
|
||||
# pseudocode for scripts/check_coverage.py
|
||||
rubric = yaml.safe_load(open("rubrics/customer_service.yaml"))
|
||||
required_criteria = {c["id"] for c in rubric["criteria"]}
|
||||
index = yaml.safe_load(open("scenarios/index.yaml"))
|
||||
scenarios = [s for s in index["path_scenarios"]["customer_service"]
|
||||
if s["status"] == "live"]
|
||||
coverage = {cid: sum(1 for s in scenarios if cid in s["rubric_criteria"])
|
||||
for cid in required_criteria}
|
||||
under_covered = {cid: n for cid, n in coverage.items() if n < MIN_COVERAGE}
|
||||
if under_covered:
|
||||
fail(f"Coverage gap: {under_covered} — each criterion needs ≥ {MIN_COVERAGE} scenarios")
|
||||
```
|
||||
|
||||
With `MIN_COVERAGE = 2` for v0.3. This runs at CI time and as a pre-merge gate on `scenarios/` changes.
|
||||
|
||||
**Interaction with D-047's ≥6 scenarios.** Six expert scenarios × 3 rubric criteria per scenario = 18 criterion-exercise slots. If the rubric has 5 criteria, each needs ≥ 2 scenarios = 10 slots minimum — well within the 18 available, so 6 scenarios is comfortably enough for coverage *if* the scenarios are authored to distribute across criteria (not all 6 testing only empathy + concrete_resolution). The coverage check catches the case where authoring concentrates on a subset of criteria.
|
||||
|
||||
**Recommendation (D-036, D-039, D-047):**
|
||||
- `rubric_criteria` on each scenario is a list of objects: `{criterion_id, weight, evidence_required}`.
|
||||
- Criterion definitions live in `rubrics/<skill>.yaml` (per D-039); scenarios reference by ID.
|
||||
- Coverage check: every criterion in the path's rubric is exercised by ≥ 2 live scenarios (`MIN_COVERAGE = 2` for v0.3).
|
||||
- `scripts/check_coverage.py` runs in CI; fails the build on coverage gaps.
|
||||
- Authoring guidance for the ≥6 expert scenarios: distribute across criteria so no criterion is exercised by only one scenario.
|
||||
|
||||
**Confidence: 0.85** — the ID-reference pattern is standard content-relationship modeling; the coverage check is a straightforward graph invariant. The `MIN_COVERAGE = 2` choice is a v0.3 pragmatic floor (it could be raised to 3 in later milestones for more robust anti-gaming, at the cost of more authoring).
|
||||
|
||||
---
|
||||
|
||||
## Cross-Cutting Recommendations for the PLAN Stage
|
||||
|
||||
1. **Anonymization pipeline is a single suppression function, swappable for DP later.** Design `aggregate_cohort(dimensions, window)` to return rows with a `cell_suppressed` column. The k-anonymity suppression is one predicate (`COUNT(DISTINCT learner_id) >= 10`); a future DP mechanism replaces the predicate with a noise-addition step. The UI and the rest of the pipeline are unchanged.
|
||||
|
||||
2. **IRT θ update and mastery gate are independent.** Don't couple them. The mastery gate (D-032) is rule-based on rubric scores + N=3 distinct scenarios. θ (D-035) is for *scenario selection*, not for *mastery certification*. A learner can open a mastery gate before θ is "reliable" by the σ² criterion, and that's correct — the gate is the authoritative mastery signal.
|
||||
|
||||
3. **Scenario library is the linchpin.** Three v0.3 subsystems read from it: the IRT selector (reads `difficulty`, `irt_target_p`), the coverage checker (reads `rubric_criteria`), and the mastery gate (reads `status`, `version`, `generated_from`). Design `index.yaml` first; the rest follows.
|
||||
|
||||
4. **Expert authoring is the bottleneck.** D-047's ≥6 expert CS scenarios is a content-authoring task, not an engineering task. The PLAN stage should identify the persona (learning designer + domain expert) and the schedule for authoring the 6 scenarios, and treat it as a critical-path dependency for the IRT and mastery-gate slices.
|
||||
|
||||
5. **Three CI gates for the scenario library:**
|
||||
- `scripts/check_coverage.py` — every rubric criterion exercised by ≥ 2 live scenarios.
|
||||
- `scripts/check_index_sync.py` — `index.yaml` is in sync with the per-scenario YAMLs (no missing entries, no stale entries).
|
||||
- `scripts/check_intent_hash.py` — no live scenario has a stale `intent_hash` (catches parent-reauthoring drift).
|
||||
|
||||
---
|
||||
|
||||
## Open Questions for the PLAN Stage
|
||||
|
||||
1. **Cohort view dimensions — exact set.** Q2 recommends 2-D views only. Which 2-D views does the operator dashboard expose? Candidate set: path × week (progression), path × outcome (mastery), path × failure_pattern (diagnostics). Confirm with the operator persona (training manager) before PLAN.
|
||||
|
||||
2. **IRT `b` recalibration cadence.** Q5 recommends nightly E-M recalibration of `b` once per-scenario response counts exceed ~20. At v0.3's scale (~100 responses per scenario), nightly is overkill — weekly is fine. But the trigger ("recalibrate when count > 20") needs to be in the nightly job, not hardcoded.
|
||||
|
||||
3. **AI variation generation tooling.** Q10 specifies `scripts/generate_variation.py` as an offline CLI. Does it run locally (expert's laptop) or in the praxis container? Locally is simpler (no LLM-in-production-container concern); the output is a YAML file checked into git. Recommend local.
|
||||
|
||||
4. **Mastery-gate evidence and scenario versioning.** Q9 specifies MAJOR bumps invalidate prior gate evidence. Concretely: if `cs_refund_ca_v01` is bumped to `cs_refund_ca_v02` with a rubric_criteria change, do learners who passed v01 need to re-pass v02? The conservative answer is yes (re-pass required), but this is a UX/policy decision that the PLAN stage should surface to the product owner.
|
||||
|
||||
5. **Operator dashboard: θ reporting threshold.** Q7 recommends reporting θ only when σ² < 0.2. Should the dashboard show "warming up — N sessions until reliable" for learners below the threshold, or suppress entirely? Showing a count is more useful but leaks information about how few sessions the learner has (a re-identification vector if combined with other cells). Recommend: aggregate the "warming up" count across the cohort (k-anonymized), don't show per-learner.
|
||||
@@ -0,0 +1,440 @@
|
||||
# Praxis — Research Findings: Verifiable Credentials Infrastructure (v0.3)
|
||||
|
||||
> **Phase:** v0.3 research (Mastery scoring + competency rubrics) — VC issuer sub-research
|
||||
> **Status:** research complete — pending orchestrator review
|
||||
> **Date:** 2026-08-03
|
||||
> **Method:** W3C authoritative specs (fetched 2026-08-03), PyPI registry, codebase decisions (D-033/042/043/048), PRD §6.4 references. Web-verified; domain-knowledge claims carry explicit confidence scores.
|
||||
> **Scope:** RESEARCH ONLY — no code written.
|
||||
|
||||
This document grounds the v0.3 verifiable-credential issuer in ecosystem evidence. It answers the 7 research questions and concludes with concrete pip-installable recommendations and a risks/unknowns list for the PLAN stage. Decisions D-033 (W3C VC 2.0, platform-issued, Ed25519), D-042 (issuer key in operator secrets), and D-043 (public verification endpoint) are assumed fixed; this research validates them and fills in implementation detail.
|
||||
|
||||
---
|
||||
|
||||
## Summary of Findings (Executive 1-Pager)
|
||||
|
||||
1. **VC Data Model 2.0 is a W3C Recommendation (15 May 2025).** Not a draft — it is the current stable standard. VC-DM 1.1 is superseded. Key 2.0 changes: `issuanceDate`/`expirationDate` → `validFrom`/`validUntil`; JSON-LD `@context` first item MUST be `https://www.w3.org/ns/credentials/v2`; media types `application/vc` and `application/vp` are now registered; securing mechanisms (Data Integrity proofs + JOSE/COSE) are separated into companion specs. (Confidence: 0.98)
|
||||
|
||||
2. **No production-ready *pure-Python* "VC library" exists for issuing+verifying.** `py-vc` and `did-jwt` are JavaScript/JS-ecosystem; `vc-js` is JS. The Python ecosystem is fragmented: `pyld` (JSON-LD processor), `rdf-canonicalize` (RDF canonicalization), `pynacl` (Ed25519 crypto), `base58`/`canonicaljson` (encodings). **Recommendation: assemble from primitives** — `pynacl` + `canonicaljson` (or `jcs`) + `base58` + hand-rolled `eddsa-jcs-2022` proof wrapper (~200 LOC). This is the simplest viable path and avoids the RDF-canonicalization complexity that `eddsa-rdfc-2022` requires. (Confidence: 0.80)
|
||||
|
||||
3. **Bitstring Status List v1.0 is a W3C Recommendation (15 May 2025)** — same day as VC-DM 2.0. It is fully implementable without a third-party service: the issuer publishes a single GZIP-compressed, Multibase-encoded bitstring as a `BitstringStatusListCredential` at a stable URL. Minimum 131,072-bit (16 KB uncompressed) list for herd privacy; a few hundred bytes compressed when few credentials are revoked. Single-issuer MVP = one status list URL + one bit per credential. (Confidence: 0.95)
|
||||
|
||||
4. **Ed25519 signing: use `pynacl` (1.6.2, libsodium 1.0.20, Apache-2.0, maintained by Python Cryptographic Authority).** Not `ed25519` (PyPI — unmaintained since 2016) and not `ed25519-zebra` (that's Rust). `cryptography` (50.0.0) also supports Ed25519 but `pynacl` is simpler for raw sign/verify and is the de-facto standard for EdDSA in Python. Private key = 32-byte seed; public key = 32 bytes; signature = 64 bytes. Store encrypted-at-rest in Postgres via `pgcrypto` symmetric `pgp_sym_encrypt` (key from operator secrets) or app-layer AES-GCM with `cryptography`. (Confidence: 0.90)
|
||||
|
||||
5. **The issuer does NOT need a DID.** VC-DM 2.0 §4.4 (Identifiers) and §4.7 (Issuer) explicitly allow the `issuer` value to be **any URL** — including a plain HTTPS URL like `https://praxis.example/issuers/v0.3`. DIDs are optional ("DIDs are not necessary for verifiable credentials to be useful"). **Simplest W3C-compliant issuer identifier: a HTTPS URL + a `verificationMethod` URL that dereferences to a Multikey public-key document served by the platform itself.** `did:key` is viable but overkill for a single platform-issued issuer and has a known limitation: no key rotation (DID is derived from the key — changing the key changes the DID). `did:web` adds HTTPS-resolution complexity with no benefit over a bare URL for one issuer. **Recommendation: bare HTTPS URL issuer ID + self-hosted Multikey verification method.** (Confidence: 0.85)
|
||||
|
||||
6. **Verification endpoint (D-043): return `{valid, status, issuer, credential}`.** A third-party verifier validates the signature by (a) canonicalizing the credential minus `proof` via JCS (RFC 8785), (b) SHA-256 hashing the canonical doc + proof config, (c) Ed25519-verifying the `proofValue` against the public key fetched from the `verificationMethod` URL. No shared secret — the public key is published at a public URL. Minimum response shape below. (Confidence: 0.90)
|
||||
|
||||
7. **Credential payload for "Mastery of Customer Service":** `credentialSubject` must assert `skill`, `level` ("mastery"), `path` ("customer-service"), `rubricScore` (mean), `scenariosPassed` (the N=3 distinct scenario IDs from D-032), `evidence` (mastery-gate audit per REQ-NFR-MAST-02), and `completedWeeks` (6, per PRD §6.4 path structure). `validFrom` = issuance; `validUntil` = optional (mastery does not expire, but a 3-year re-validation window is prudent). PRD §6.4 guidance = path-as-job, 6-week structure (D-037); the VC is **path-level, not week-level** (D-048). (Confidence: 0.80)
|
||||
|
||||
8. **Key rotation (D-042 strategy validated):** Rotate by generating a new Ed25519 keypair, marking the old key as `superseded` (NOT revoked) in the `issuer_keys` table, and serving the old public key indefinitely at its original `verificationMethod` URL. Old VCs still verify against the archived public key; new VCs reference the new key. `did:key` cannot do this (key IS the DID) — another reason bare-URL issuer ID is superior for this use case. (Confidence: 0.90)
|
||||
|
||||
---
|
||||
|
||||
## VC Data Model 2.0 Status
|
||||
|
||||
**Sources:** https://www.w3.org/TR/vc-data-model-2.0/ (fetched 2026-08-03), https://w3c.github.io/vc-data-model/ (editor's draft, v2.1 in progress).
|
||||
|
||||
### Finding: W3C Recommendation since 15 May 2025
|
||||
|
||||
The Verifiable Credentials Data Model v2.0 was published as a **W3C Recommendation on 15 May 2025** ([source](https://www.w3.org/TR/2025/REC-vc-data-model-2.0-20250515/)). This is the highest maturity level in the W3C process — equivalent to a ratified standard. The W3C explicitly "recommends the wide deployment of this specification as a standard for the Web." An editor's draft for v2.1 exists but v2.0 is the current normative reference. D-033's choice of "W3C VC Data Model 2.0" is therefore targeting a stable Recommendation, not a moving draft.
|
||||
|
||||
### What changed from 1.1
|
||||
|
||||
VC-DM 1.1 was a W3C Recommendation (3 Mar 2022). The 2.0 changes material to Praxis:
|
||||
|
||||
| Concern | VC-DM 1.1 | VC-DM 2.0 |
|
||||
|---|---|---|
|
||||
| Validity period | `issuanceDate` + `expirationDate` | `validFrom` + `validUntil` (§4.9) |
|
||||
| Required `@context` first item | `https://www.w3.org/2018/credentials/v1` | `https://www.w3.org/ns/credentials/v2` (§4.3) |
|
||||
| Media types | not registered | `application/vc`, `application/vp` registered at IANA (§6.2) |
|
||||
| Conforming document | JSON or JSON-LD | **compacted JSON-LD document** (§1.3) — JSON-LD processing is expected but "type-specific processing" (§6.3) permits pure-JSON verification when contexts are pinned |
|
||||
| Securing mechanisms | `proof` embedded (LD-Proofs) | Data Integrity 1.0 (embedded `proof`) **or** JOSE/COSE (enveloping) — both are companion specs ([VC-DATA-INTEGRITY](https://w3c.github.io/vc-data-integrity/), [VC-JOSE-COSE](https://w3c.github.io/vc-jose-cose/)) |
|
||||
| Status | `credentialStatus` (open) | `credentialStatus` + `status` (§4.10) — Bitstring Status List is the normative companion |
|
||||
| Evidence | `evidence` (open) | `evidence` (§5.6) — same, now typed |
|
||||
|
||||
**Implication for Praxis:** Use `validFrom`/`validUntil` (not the 1.1 names), pin `@context` to `credentials/v2`, and secure via **Data Integrity `eddsa-jcs-2022`** (embedded `proof`) — not JOSE/COSE. JCS canonicalization (RFC 8785) is pure-JSON and avoids RDF Dataset Canonicalization, which is the single biggest implementation complexity in the VC 2.0 stack.
|
||||
|
||||
### Python ecosystem readiness
|
||||
|
||||
The Python VC ecosystem is **not** "batteries-included." There is no `pip install python-vc` that issues and verifies W3C VC 2.0 credentials end-to-end. The components exist but must be assembled:
|
||||
|
||||
| Component | pip package | Status | Notes |
|
||||
|---|---|---|---|
|
||||
| Ed25519 sign/verify | `pynacl` 1.6.2 | ✅ production | Maintained by Python Cryptographic Authority; libsodium 1.0.20; Apache-2.0 |
|
||||
| Ed25519 (alt) | `cryptography` 50.0.0 | ✅ production | Also supports Ed25519; heavier; OpenSSL-backed |
|
||||
| JSON Canonicalization (JCS, RFC 8785) | `canonicaljson` 2.0.0 / `jcs` 0.2.1 | ⚠️ minimal | `canonicaljson` is from Ankidro (Anki ecosystem); `jcs` is a thin wrapper. Both implement RFC 8785. ~50 LOC to hand-roll if needed. |
|
||||
| Base58-btc (Multibase) | `base58` 2.1.1 | ✅ stable | Base58 codec only; Multibase prefix (`z`) is a literal `z` prepended |
|
||||
| JSON-LD processor | `pyld` 3.1.0 | ✅ stable | **Only needed for `eddsa-rdfc-2022` or JSON-LD expansion. NOT needed for `eddsa-jcs-2022`.** |
|
||||
| RDF Dataset Canonicalization | `rdf-canonicalize` | ⚠️ sparse | Required only for `eddsa-rdfc-2022`. Avoid by choosing JCS. |
|
||||
| did:key resolution | none standard | ⚠️ | did:key is generative — ~30 LOC to expand a Multikey from the DID string |
|
||||
|
||||
**No `py-vc`, `vc-js`, or `did-jwt` on PyPI** — these are JavaScript libraries (`@digitalbazaar/py-vc` is a JS package despite the name; `did-jwt` is Transmute's JS lib). The Python path is **assemble-from-primitives**.
|
||||
|
||||
**Confidence: 0.98** (status); **0.80** (Python readiness assessment — based on PyPI registry inspection 2026-08-03; the absence of a unified lib is well-known in the VC community).
|
||||
|
||||
---
|
||||
|
||||
## Python Library Recommendation
|
||||
|
||||
**Recommendation: assemble the VC issuer/verifier from 4 pip packages + ~200 LOC of glue.**
|
||||
|
||||
### pip-installable dependencies (add to `pyproject.toml` `[project.optional-dependencies] vc`)
|
||||
|
||||
```toml
|
||||
[project.optional-dependencies]
|
||||
vc = [
|
||||
"pynacl>=1.5", # Ed25519 sign/verify (libsodium)
|
||||
"canonicaljson>=2.0", # RFC 8785 JSON Canonicalization Scheme (JCS)
|
||||
"base58>=2.1", # base58-btc encoding for Multibase proofValue
|
||||
"pydantic>=2.7", # already a dep — use for VC schema validation
|
||||
]
|
||||
```
|
||||
|
||||
### Why this stack
|
||||
|
||||
- **`pynacl` over `cryptography` for Ed25519:** PyNaCl's `nacl.signing.SigningKey` / `VerifyKey` API is purpose-built for EdDSA and returns raw 64-byte signatures — exactly what `eddsa-jcs-2022` requires. `cryptography` works but its Ed25519 API is more verbose and OpenSSL-dependent. PyNaCl bundles libsodium (no system dep).
|
||||
- **`canonicaljson` over `jcs`:** `canonicaljson` (Anki ecosystem, 2.0.0) is more actively maintained and implements RFC 8785 fully. `jcs` 0.2.1 is thinner but less proven.
|
||||
- **No `pyld` / no `rdf-canonicalize`:** By choosing the **`eddsa-jcs-2022`** cryptosuite (not `eddsa-rdfc-2022`), we avoid the entire JSON-LD → RDF → canonicalization pipeline. JCS operates on JSON directly. This is the single largest complexity reduction available. The VC-DM 2.0 "type-specific processing" clause (§6.3) explicitly permits this: "implementations MAY choose to not perform JSON-LD expansion... when using type-specific processing rules."
|
||||
|
||||
### Code shape (illustrative — NOT committed code, per research-only constraint)
|
||||
|
||||
```python
|
||||
# Issue
|
||||
sk = nacl.signing.SigningKey.generate() # 32-byte seed
|
||||
pk_bytes = bytes(sk.verify_key) # 32 bytes
|
||||
proof_config = {"type": "DataIntegrityProof",
|
||||
"cryptosuite": "eddsa-jcs-2022",
|
||||
"created": "2026-08-03T12:00:00Z",
|
||||
"verificationMethod": "https://praxis.example/keys/v0.3#key-1",
|
||||
"proofPurpose": "assertionMethod"}
|
||||
canonical_proof = canonicaljson.canonicalize(proof_config)
|
||||
canonical_doc = canonicaljson.canonicalize(credential_without_proof)
|
||||
hash_data = hashlib.sha256(canonical_proof).digest() + hashlib.sha256(canonical_doc).digest()
|
||||
proof_bytes = sk.sign(hash_data).signature # 64 bytes
|
||||
proof_config["proofValue"] = "z" + base58.b58encode(proof_bytes).decode()
|
||||
credential_with_proof = {**credential_without_proof, "proof": proof_config}
|
||||
|
||||
# Verify
|
||||
verify_key = nacl.signing.VerifyKey(pk_bytes) # fetched from verificationMethod URL
|
||||
proof_value = base58.b58decode(proof_config["proofValue"][1:]) # strip 'z' Multibase prefix
|
||||
verify_key.verify(hash_data, proof_value) # raises BadSignatureError if invalid
|
||||
```
|
||||
|
||||
**Confidence: 0.80** — the assembly pattern is well-documented in the [eddsa-jcs-2022 spec](https://w3c.github.io/vc-di-eddsa/) (fetched 2026-08-03); the risk is in the ~200 LOC of glue (proof config ordering, context pinning) which is standard but unverified here.
|
||||
|
||||
---
|
||||
|
||||
## Status List Revocation
|
||||
|
||||
**Sources:** https://www.w3.org/TR/vc-bitstring-status-list/ (fetched 2026-08-03) — **W3C Recommendation 15 May 2025**, titled "Bitstring Status List v1.0".
|
||||
|
||||
### How it works
|
||||
|
||||
The issuer maintains a single bitstring (minimum 131,072 bits = 16 KB uncompressed) where each bit corresponds to one issued credential's status. The bitstring is GZIP-compressed, Multibase-encoded (base64url, no padding), and published as the `encodedList` field inside a **`BitstringStatusListCredential`** — itself a verifiable credential signed by the issuer. Each issued credential carries a `credentialStatus` entry:
|
||||
|
||||
```json
|
||||
"credentialStatus": {
|
||||
"type": "BitstringStatusListEntry",
|
||||
"statusPurpose": "revocation",
|
||||
"statusListIndex": "94567",
|
||||
"statusListCredential": "https://praxis.example/status/v0.3"
|
||||
}
|
||||
```
|
||||
|
||||
A verifier (a) dereferences `statusListCredential`, (b) verifies that VC's own proof, (c) GZIP-decompresses + Multibase-decodes `encodedList`, (d) reads the bit at `statusListIndex`. Bit = 1 means revoked; 0 means active. `statusPurpose` can be `revocation` (irreversible), `suspension` (reversible), `refresh`, or `message`.
|
||||
|
||||
### Implementable without a third-party service — YES
|
||||
|
||||
The status list is **just another VC published at a static URL by the issuer**. No registry, no ledger, no OCSP responder. The issuer regenerates + republishes the `BitstringStatusListCredential` whenever a credential is revoked. CDN-cacheable by design (the spec §6.4 explicitly recommends CDN distribution for privacy).
|
||||
|
||||
### Minimum viable revocation setup for a single issuer (Praxis)
|
||||
|
||||
1. **One status list URL:** `https://praxis.example/status/v0.3` — serves the `BitstringStatusListCredential` (signed by the same Ed25519 issuer key).
|
||||
2. **One bit per issued credential:** `statusPurpose: "revocation"`, `statusSize: 1` (default).
|
||||
3. **In-process generation:** maintain a 131,072-bit bytearray in Postgres (`status_lists` table: `id, status_purpose, encoded_list, updated_at`). On revocation, flip the bit, GZIP-compress, Multibase-encode, re-sign the list VC, persist, serve.
|
||||
4. **Random index assignment:** spec §2.1 recommends random `statusListIndex` allocation to prevent inference of issuance order or population size.
|
||||
5. **For v0.3 scale (likely <1000 credentials):** a single list with 131,072 slots is wildly over-provisioned — compressed size stays a few hundred bytes. No need for multiple lists until >100k credentials.
|
||||
|
||||
**Confidence: 0.95** — the spec is a Recommendation and the algorithm (§3.1 Generate, §3.2 Validate, §3.3 Bitstring Generation, §3.4 Bitstring Expansion) is fully specified and implementable in ~100 LOC of Python (`gzip`, `base64`, `bitarray`/`bytearray`).
|
||||
|
||||
---
|
||||
|
||||
## Issuer Identifier Strategy
|
||||
|
||||
**Sources:** VC-DM 2.0 §4.4 (Identifiers), §4.7 (Issuer); [did:key Method v0.9](https://w3c-ccg.github.io/did-key-spec/) (fetched 2026-08-03).
|
||||
|
||||
### Does platform-issued require a DID? — NO
|
||||
|
||||
VC-DM 2.0 §4.4: "The `id` property is OPTIONAL... Example `id` values include UUIDs... HTTP URLs (`https://id.example/things#123`), and DIDs." §4.7: the `issuer` value "MUST be either a URL or an object containing an `id` property whose value is a URL." DIDs are *optional* — the spec explicitly states "DIDs are not necessary for verifiable credentials to be useful."
|
||||
|
||||
The Data Integrity `verificationMethod` (which holds the public key) is also just a URL that dereferences to a Multikey document. No DID resolution is required if the URL is self-hosted.
|
||||
|
||||
### Three options compared
|
||||
|
||||
| Option | Example | Key rotation | Complexity | W3C-compliant? |
|
||||
|---|---|---|---|---|
|
||||
| **Bare HTTPS URL** | `https://praxis.example/issuers/v0.3` | ✅ Archive old key at old URL; new key at new URL | Lowest — serve a static JSON file | ✅ Yes (§4.4, §4.7) |
|
||||
| `did:web` | `did:web:praxis.example:issuers:v0.3` | ✅ Update DID document at `/.well-known/did.json` | Medium — DID document format, well-known path | ✅ Yes |
|
||||
| `did:key` | `did:key:z6Mk...` | ❌ **No rotation** — DID is derived from the key; changing the key changes the DID | Low to implement, but breaks D-042 rotation | ✅ Yes, but unsuitable for long-lived issuer |
|
||||
|
||||
### Recommendation: Bare HTTPS URL issuer ID
|
||||
|
||||
```json
|
||||
"issuer": "https://praxis.example/issuers/v0.3",
|
||||
"proof": {
|
||||
"verificationMethod": "https://praxis.example/keys/v0.3#key-1",
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
Where `GET https://praxis.example/keys/v0.3` returns a "controlled identifier document" (per the [CID spec](https://w3c.github.io/controller-document/)) containing:
|
||||
|
||||
```json
|
||||
{
|
||||
"@context": ["https://www.w3.org/ns/credentials/v2"],
|
||||
"id": "https://praxis.example/keys/v0.3",
|
||||
"verificationMethod": [{
|
||||
"id": "https://praxis.example/keys/v0.3#key-1",
|
||||
"type": "Multikey",
|
||||
"controller": "https://praxis.example/issuers/v0.3",
|
||||
"publicKeyMultibase": "z6Mk...<base58-btc(0xed01 + 32-byte pubkey)>"
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
This is the **simplest viable W3C-compliant issuer identifier**. It supports key rotation (D-042 strategy: archive old `verificationMethod` documents, serve new ones), requires no DID resolution infrastructure, and is verifiable by any Data Integrity compliant verifier.
|
||||
|
||||
`did:key` is rejected despite being simplest to generate because its documented limitation (spec §Security: "Key Rotation Not Supported," "Long Term Usage is Discouraged") directly conflicts with D-042's rotation requirement. `did:web` adds the `did.json` well-known-path convention and DID-document schema for zero benefit over a bare URL when there's exactly one issuer.
|
||||
|
||||
**Confidence: 0.85** — the VC-DM 2.0 text is unambiguous that URLs are valid issuer IDs; the bare-URL + Multikey pattern is used in the spec's own Example 3 (`"issuer": "https://university.example/issuers/565049"`).
|
||||
|
||||
---
|
||||
|
||||
## Verification Endpoint Design
|
||||
|
||||
**Sources:** D-043 (decided: public unauthenticated `GET /vc/verify/<id>`), VC-DM 2.0 §7.1 (Verification), §7.2 (Problem Details), Data Integrity eddsa-jcs-2022 Verify Proof algorithm.
|
||||
|
||||
### How a third-party verifier validates the signature (no shared secret)
|
||||
|
||||
1. **Fetch the credential** — `GET /vc/verify/<id>` returns the stored VC (or the caller already holds the VC and just wants status; see response shape below).
|
||||
2. **Extract `proof`** — remove `proof` from the secured document to get `unsecuredDocument`; copy `proof` minus `proofValue` to get `proofOptions`.
|
||||
3. **Canonicalize** — apply JCS (RFC 8785) to `unsecuredDocument` and to `proofOptions` → `canonicalDocument`, `canonicalProofConfig`.
|
||||
4. **Hash** — `hashData = SHA-256(canonicalProofConfig) || SHA-256(canonicalDocument)` (64 bytes total).
|
||||
5. **Fetch public key** — dereference `proof.verificationMethod` → controlled identifier document → extract `publicKeyMultibase` → Multibase-decode (strip `z`, base58-decode) → strip 2-byte `0xed01` Multikey prefix → 32-byte Ed25519 public key.
|
||||
6. **Verify** — Ed25519 `Verify(pk, hashData, proofValue)` where `proofValue` is Multibase-decoded `proof.proofValue`. Raises on failure.
|
||||
7. **Check status** — dereference `credentialStatus.statusListCredential`, verify its proof, expand bitstring, read bit at `statusListIndex`. 0 = active, 1 = revoked.
|
||||
8. **Check validity window** — `validFrom` ≤ now ≤ `validUntil` (if `validUntil` present).
|
||||
|
||||
No shared secret, no API key, no account. The public key is published at a public URL; everything else is math.
|
||||
|
||||
### Minimum response shape for `GET /vc/verify/<id>`
|
||||
|
||||
Per D-043: `{valid: bool, status: "active"|"revoked", issuer: "praxis-v0.3", mastery: {...}}`. Refined with spec-aware fields:
|
||||
|
||||
```json
|
||||
{
|
||||
"valid": true,
|
||||
"status": "active",
|
||||
"issuer": {
|
||||
"id": "https://praxis.example/issuers/v0.3",
|
||||
"name": "Praxis"
|
||||
},
|
||||
"credential": {
|
||||
"id": "https://praxis.example/vc/01J...',
|
||||
"type": ["VerifiableCredential", "MasteryCredential"],
|
||||
"validFrom": "2026-08-03T12:00:00Z",
|
||||
"validUntil": "2029-08-03T12:00:00Z"
|
||||
},
|
||||
"mastery": {
|
||||
"skill": "customer-service",
|
||||
"level": "mastery",
|
||||
"path": "customer-service",
|
||||
"rubricScore": 4.1,
|
||||
"scenariosPassed": ["cs_refund_ca_v01", "cs_escalation_v02", "cs_billing_v01"],
|
||||
"completedWeeks": 6
|
||||
},
|
||||
"verifiedAt": "2026-08-03T14:30:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
**Privacy (D-043 constraint):** No learner PII beyond what the credential itself asserts. The `credentialSubject.id` (if any) is NOT echoed in the verification response — only the mastery claims. The full signed VC is retrievable via a separate `GET /vc/<id>` endpoint that the holder can choose to share, or the holder presents the VC directly to the verifier and the verifier calls `/vc/verify/<id>` only for status.
|
||||
|
||||
**Error responses** (per VC-DM 2.0 §7.2, RFC 9457 Problem Details):
|
||||
|
||||
| HTTP | `type` suffix | Meaning |
|
||||
|---|---|---|
|
||||
| 404 | `not-found` | No credential with that ID |
|
||||
| 200 | — | `valid: true` + status |
|
||||
| 200 | — | `valid: false`, `status: "revoked"` |
|
||||
| 410 | — | `valid: false`, `status: "revoked"` (alternative — 410 Gone signals the credential is "gone" but still returns body) |
|
||||
|
||||
**Recommendation:** always return 200 with `valid: false` for revoked/invalid-but-existing credentials (simpler client logic); 404 only for non-existent IDs.
|
||||
|
||||
**Confidence: 0.90** — D-043 fixed the endpoint; the response shape is derived from spec verification semantics + the privacy constraint.
|
||||
|
||||
---
|
||||
|
||||
## Credential Payload Schema
|
||||
|
||||
**Sources:** PRD §6.4 (path-as-job, 6-week structure — referenced via D-037, REQ-PATH-02), D-032 (mastery gate: N=3 scenarios, rubric mean ≥ 3.5), D-048 (VC on week-final gate, path-level), D-039 (rubric YAML), REQ-NFR-MAST-02 (gate auditability), VC-DM 2.0 §4.2, §5.6 (Evidence).
|
||||
|
||||
### Claims for "Mastery of Customer Service"
|
||||
|
||||
To be credible to an employer, the VC must assert **what** was mastered, **how** it was assessed, and **who** says so — with enough evidence that the employer can audit the claim without contacting Praxis.
|
||||
|
||||
```json
|
||||
{
|
||||
"@context": [
|
||||
"https://www.w3.org/ns/credentials/v2",
|
||||
"https://praxis.example/contexts/mastery/v1"
|
||||
],
|
||||
"id": "https://praxis.example/vc/01JH...",
|
||||
"type": ["VerifiableCredential", "MasteryCredential"],
|
||||
"issuer": "https://praxis.example/issuers/v0.3",
|
||||
"validFrom": "2026-08-03T12:00:00Z",
|
||||
"validUntil": "2029-08-03T12:00:00Z",
|
||||
"name": "Mastery of Customer Service",
|
||||
"description": "Praxis v0.3 mastery credential — the holder demonstrated customer-service competency across varied scenarios, scored against a 5-level rubric.",
|
||||
"credentialStatus": {
|
||||
"type": "BitstringStatusListEntry",
|
||||
"statusPurpose": "revocation",
|
||||
"statusListIndex": "42173",
|
||||
"statusListCredential": "https://praxis.example/status/v0.3"
|
||||
},
|
||||
"credentialSubject": {
|
||||
"id": "urn:uuid:<learner-pseudonymous-id>",
|
||||
"type": "Person",
|
||||
"skill": "customer-service",
|
||||
"level": "mastery",
|
||||
"path": "customer-service",
|
||||
"pathStructure": "6-week job-structured (PRD §6.4)",
|
||||
"completedWeeks": 6,
|
||||
"rubricScore": 4.1,
|
||||
"rubricMax": 5.0,
|
||||
"rubricThreshold": 3.5,
|
||||
"scenariosPassed": ["cs_refund_ca_v01", "cs_escalation_v02", "cs_billing_v01"],
|
||||
"evidence": [{
|
||||
"type": ["Evidence"],
|
||||
"id": "https://praxis.example/evidence/01JH.../gate-audit",
|
||||
"rubricMean": 4.1,
|
||||
"distinctScenarios": 3,
|
||||
"gateOpenedAt": "2026-08-03T11:45:00Z"
|
||||
}]
|
||||
},
|
||||
"proof": { ... }
|
||||
}
|
||||
```
|
||||
|
||||
### Claim rationale
|
||||
|
||||
| Claim | Why it's there | Source |
|
||||
|---|---|---|
|
||||
| `skill` | The competency domain — what the employer cares about | D-033, D-039 |
|
||||
| `level: "mastery"` | Distinguishes from "in-progress" or "completion" | D-032 (mastery gate) |
|
||||
| `path` | Which 6-week job-structured path (PRD §6.4) | D-037, REQ-PATH-02 |
|
||||
| `completedWeeks: 6` | Proves full path completion, not partial | D-048 (VC only on final gate) |
|
||||
| `rubricScore` + `rubricMax` + `rubricThreshold` | Quantified competency — employer can judge stringency | D-032 (≥3.5/5.0), D-039 (rubric) |
|
||||
| `scenariosPassed` (3 IDs) | **Varied-scenario evidence** — the load-bearing anti-gaming claim (D-032: N=3 distinct) | D-032, D-047 |
|
||||
| `evidence[].gateOpenedAt` | Auditability of the gate-open event | REQ-NFR-MAST-02 |
|
||||
| `credentialSubject.id` | Pseudonymous learner ID (urn:uuid) — NOT a real name. Employer contacts Praxis out-of-band to dereference if needed. | Privacy (D-043) |
|
||||
| `validUntil` (3 years) | Mastery doesn't "expire" but employers want a re-validation window. 3 years is a defensible default; Praxis can re-issue on re-assessment. | PRD §6.4 (no explicit expiry guidance — this is a recommendation) |
|
||||
| `credentialStatus` | Revocation path (compromised key, fraud detected) | D-033 (status list), REQ-NFR-VC-02 |
|
||||
|
||||
### What PRD §6.4 says
|
||||
|
||||
PRD §6.4 is not a file in this repo — it is referenced by D-037 and REQ-PATH-02 as the source for the **"path-as-job 6-week structure."** The operative guidance: a path is structured as a job (6 weeks), mastery-paced, with mastery gates between weeks. The VC is **path-level** (D-048: "VCs are path-level, not week-level"), issued only when the **final** week's gate opens. This research confirms the credential payload should assert `completedWeeks: 6` and the full path slug — not per-week credentials (D-048 rejected "VC per week" as "credential spam").
|
||||
|
||||
**Confidence: 0.80** — the claim set is grounded in D-032/037/039/048 + REQ-NFR-MAST-02; the `validUntil` 3-year window is a recommendation (PRD §6.4 is silent on expiry), hence the 0.80 not higher.
|
||||
|
||||
---
|
||||
|
||||
## Key Rotation Strategy
|
||||
|
||||
**Sources:** D-042 (issuer key in secrets, generated on first init, archived-when-superseded), did:key spec §Security (no rotation), VC-DM 2.0 §9.2 (Key Management).
|
||||
|
||||
### The problem
|
||||
|
||||
Ed25519 keys should be rotated periodically (compromise hygiene) and on suspected exposure. But VCs are signed with a specific key; if the key changes, existing VCs must still verify.
|
||||
|
||||
### D-042 strategy (validated)
|
||||
|
||||
1. **`issuer_keys` table in Postgres** (operator-tier, per D-040):
|
||||
```
|
||||
issuer_keys(
|
||||
key_id UUID PRIMARY KEY,
|
||||
public_key BYTEA NOT NULL, -- 32 bytes
|
||||
encrypted_priv BYTEA NOT NULL, -- pgp_sym_encrypt or app-layer AES-GCM
|
||||
created_at TIMESTAMPTZ NOT NULL,
|
||||
superseded_at TIMESTAMPTZ, -- NULL = active
|
||||
status TEXT NOT NULL -- 'active' | 'superseded'
|
||||
)
|
||||
```
|
||||
2. **At first init:** generate Ed25519 keypair, encrypt private key with a root key from operator secrets (`PRAXIS_VC_ROOT_KEY`), insert as `status='active'`.
|
||||
3. **To rotate:**
|
||||
- Generate new keypair.
|
||||
- Insert new row `status='active'`.
|
||||
- Update old row: `status='superseded', superseded_at=now()`. **Do NOT delete.** The old public key remains in the table and is still served at its original `verificationMethod` URL.
|
||||
- New VCs reference the new `verificationMethod` URL (`...#key-2`); old VCs still reference `...#key-1`.
|
||||
4. **Verification of old VCs:** verifier fetches `https://praxis.example/keys/v0.3#key-1` → archived public key → Ed25519 verify succeeds. The old key is **archived, not revoked** — the signature still verifies.
|
||||
5. **Verification of new VCs:** verifier fetches `...#key-2` → current public key → verify succeeds.
|
||||
6. **Revocation of individual VCs** (distinct from key rotation): handled by the Bitstring Status List, not by key rotation. A key compromise would trigger (a) rotation + (b) bulk-revocation of all VCs signed by the compromised key via the status list.
|
||||
|
||||
### Why `did:key` is incompatible with this strategy
|
||||
|
||||
`did:key` derives the DID from the public key (`did:key:z6Mk...`). Changing the key produces a **different DID**. There is no way to "archive" the old DID — it's a new identity. This means either (a) all old VCs show an issuer DID that no longer "exists" in any meaningful sense (though the public key is still embedded in the DID string and verification still works), or (b) reissue all old VCs under the new DID. The bare-URL strategy avoids this entirely: the issuer URL stays stable (`https://praxis.example/issuers/v0.3`), only the `#key-N` fragment changes.
|
||||
|
||||
### Encrypted-at-rest in Postgres — two options
|
||||
|
||||
| Option | Mechanism | Pros | Cons |
|
||||
|---|---|---|---|
|
||||
| **`pgcrypto` `pgp_sym_encrypt`** | Postgres extension; `INSERT ... pgp_sym_encrypt($1, $2)` | DB-level; no app crypto | `pgcrypto` must be enabled; key passed in SQL (audit log risk) |
|
||||
| **App-layer AES-GCM (`cryptography`)** | `cryptography.hazmat.primitives.ciphertext.AEAD.AESGCM`; encrypt before INSERT | Key never touches DB; auditable in app | Adds `cryptography` dep (already likely present via transitive) |
|
||||
|
||||
**Recommendation: app-layer AES-GCM** — the root key (`PRAXIS_VC_ROOT_KEY`) stays in the FastAPI process (from `os.environ`), never in SQL. Store `nonce || ciphertext || tag` as a single `BYTEA`. This aligns with D-042's "encrypted at rest with a root key from secrets" and avoids `pgcrypto` extension dependencies in the LXC Docker Postgres (D-040).
|
||||
|
||||
**Confidence: 0.90** — the rotation-without-invalidation pattern is standard key-management practice and is explicitly what D-042 specifies; the did:key incompatibility is documented in the did:key spec itself.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Diff (v0.2 → v0.3 VC subsystem)
|
||||
|
||||
| Component | v0.2 | v0.3 (this research) |
|
||||
|---|---|---|
|
||||
| Operator Postgres | not present | **added** (D-040): `issuer_keys`, `issued_credentials`, `status_lists`, `mastery_gate_audit` tables |
|
||||
| VC issuer module | n/a | `server/vc/` — issuer (signs with active key), verifier (public endpoint), status-list manager |
|
||||
| Public endpoints | `/health`, `/pipecat/webrtc` | **+** `GET /vc/verify/<id>` (D-043), `GET /vc/<id>` (full VC fetch), `GET /keys/v0.3` (Multikey doc), `GET /status/v0.3` (BitstringStatusListCredential) |
|
||||
| Secrets | `.env.secrets` (GITEA_TOKEN) | **+** `PRAXIS_VC_ROOT_KEY` (root encryption key for issuer_keys.encrypted_priv); `PRAXIS_VC_ISSUER_SEED` optional (deterministic first key) or generate-on-first-init (D-042) |
|
||||
| pip deps | (existing) | **+** `pynacl`, `canonicaljson`, `base58` in `[project.optional-dependencies] vc` |
|
||||
|
||||
---
|
||||
|
||||
## Risks & Unknowns
|
||||
|
||||
1. **`eddsa-jcs-2022` interop:** While the spec is clear, the *ecosystem* of verifiers is more saturated with `eddsa-rdfc-2022` (RDF canonicalization) and JOSE/SD-JWT. An employer using a generic VC verifier wallet may not have a JCS cryptosuite implementation. **Mitigation:** also publish the VC in `application/vc` (Data Integrity) — most modern verifiers support Data Integrity; JCS is a recognized cryptosuite. If employer-interop friction emerges, consider adding an SD-JWT (JOSE) representation in v0.4. **Confidence: 0.55** (ecosystem adoption is hard to measure).
|
||||
|
||||
2. **JCS implementation correctness:** `canonicaljson` is used by Anki but is not a W3C-referenced normative implementation. RFC 8785 has edge cases (number serialization, key ordering). **Mitigation:** pin `canonicaljson>=2.0.0`; add round-trip test vectors from RFC 8785 to the test suite; verify against the [eddsa-jcs-2022 test suite](https://w3c.github.io/vc-di-eddsa-test-suite/) if one exists at implementation time.
|
||||
|
||||
3. **Status list herd privacy at v0.3 scale:** The 131,072-bit minimum gives herd privacy only if the issued population is large. At v0.3 pilot scale (<100 learners), a verifier can infer that the issuer has few credentials. The spec §6.1 acknowledges this. **Mitigation:** acceptable for pilot — the privacy loss is the *issuer's* (Praxis), not the learner's, and Praxis is not a privacy adversary. Revisit at scale.
|
||||
|
||||
4. **`validUntil` 3-year window is a recommendation, not PRD-grounded.** PRD §6.4 does not specify expiry. If employers reject expiring mastery credentials ("mastery doesn't expire"), set `validUntil` to null and rely on status-list revocation for fraud. **Decision needed at PLAN stage.**
|
||||
|
||||
5. **Learner PII in `credentialSubject.id`:** Using a pseudonymous `urn:uuid` learner ID means the VC cannot be self-sovereignly held by the learner in a universal wallet (the ID is Praxis-internal). For v0.3 (platform-issued, platform-verified) this is fine. For v0.9 (learner-held portable credentials), the learner will need a DID or the VC will need to support holder-binding differently. **Out of v0.3 scope** (D-033 defers third-party/holder-issued to v0.9).
|
||||
|
||||
6. **Public key endpoint availability:** If `https://praxis.example/keys/v0.3` is down, all verification fails. The Multikey document is tiny (~300 bytes) and should be served from the same FastAPI app + cached at a CDN. **Mitigation:** static file; long `Cache-Control` max-age.
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [VC Data Model 2.0](https://www.w3.org/TR/vc-data-model-2.0/) — W3C Recommendation, 15 May 2025
|
||||
- [Bitstring Status List v1.0](https://www.w3.org/TR/vc-bitstring-status-list/) — W3C Recommendation, 15 May 2025
|
||||
- [Data Integrity 1.1](https://w3c.github.io/vc-data-integrity/) — editor's draft (companion spec for embedded `proof`)
|
||||
- [Data Integrity EdDSA Cryptosuites v1.1](https://w3c.github.io/vc-di-eddsa/) — `eddsa-jcs-2022` and `eddsa-rdfc-2022` normative algorithms
|
||||
- [did:key Method v0.9](https://w3c-ccg.github.io/did-key-spec/) — generative DID method (rejected for Praxis issuer ID due to no key rotation)
|
||||
- [RFC 8785](https://datatracker.ietf.org/doc/html/rfc8785) — JSON Canonicalization Scheme (JCS)
|
||||
- [RFC 8032](https://datatracker.ietf.org/doc/html/rfc8032) — EdDSA: Edwards-Curve Digital Signature Algorithm (Ed25519)
|
||||
- [PyNaCl 1.6.2](https://pypi.org/project/PyNaCl/) — Python binding to libsodium (Apache-2.0, Python Cryptographic Authority)
|
||||
- [canonicaljson 2.0.0](https://pypi.org/project/canonicaljson/) — RFC 8785 JCS implementation
|
||||
- [base58 2.1.1](https://pypi.org/project/base58/) — base58-btc codec
|
||||
- Praxis decisions: D-033, D-037, D-039, D-040, D-042, D-043, D-048 (`.ciagent/PROJECT.md`)
|
||||
- Praxis requirements: REQ-MAST-03, REQ-PATH-02, REQ-NFR-VC-01/02, REQ-NFR-MAST-02 (`.ciagent/REQUIREMENTS.md`)
|
||||
+139
-1
@@ -646,4 +646,142 @@ All tests that mock `api.sh` (pve_curl, pve_poll, pve_get) and `curl` can run wi
|
||||
1. **ExecStartPre vs separate build service:** Should `docker compose build` be an `ExecStartPre` in `praxis.service` or a separate `praxis-build.service` (Type=oneshot) that `praxis.service` `Requires=`? The latter is cleaner but adds a service file.
|
||||
2. **Docker layer cleanup:** Should `install-service.sh` run `docker system prune -f` after the first successful build to reclaim ~1GB of build layers?
|
||||
3. **Repo update path:** When praxis code changes, how is the CT updated? Options: (a) `pct exec git pull && systemctl restart praxis` (re-builds), (b) `--reconfigure` flag in lxc-deploy.sh that re-runs the hook, (c) a separate `scripts/proxmox/lxc-update.sh`. Not a v0.2 blocker (first deploy only) but should be designed for.
|
||||
4. **PRAXIS_DB_PATH in container:** The docker-compose volume mounts to `/app/data`. `PRAXIS_DB_PATH` env should be set to `/app/data/praxis.db` in `server.env`. Confirm the server respects this path (current default: `./praxis.db` relative to CWD).
|
||||
4. **PRAXIS_DB_PATH in container:** The docker-compose volume mounts to `/app/data`. `PRAXIS_DB_PATH` env should be set to `/app/data/praxis.db` in `server.env`. Confirm the server respects this path (current default: `./praxis.db` relative to CWD).
|
||||
|
||||
---
|
||||
|
||||
# Praxis — Research Findings (v0.3 Mastery Scoring + Competency Rubrics)
|
||||
|
||||
> **Phase:** v0.3 research (mastery scoring + competency rubrics + verifiable credentials + cohort dashboard)
|
||||
> **Branch:** `phase/00-pre-execution`
|
||||
> **Status:** research complete
|
||||
> **Date:** 2026-08-03
|
||||
> **Method:** Four parallel research agents (rubric/mastery models, W3C VC stack, multi-tenant auth + Postgres-in-LXC, anonymization + IRT + scenario library). Full agent outputs preserved in `docs/mastery-scoring-research.md`, `.ciagent/RESEARCH-vc.md`, `docs/RESEARCH-operator-postgres-auth.md`, `.ciagent/RESEARCH-v0.3-anonymization-irt-scenarios.md`. This section is the consolidated summary for PLAN; the detailed files are retained as appendices.
|
||||
|
||||
## v0.3 Research Summary (Executive 1-Pager)
|
||||
|
||||
1. **Rubric model = Dreyfus 5-stage + Miller "Does" tier + EPA entrustment + Bloom mastery-learning gate.** (Confidence: 0.82) Bloom's *taxonomy* alone is a weak fit for "do the job" assessment; Dreyfus anchors the 5 levels behaviorally, Miller's pyramid ensures anchors describe *doing* (not *knowing*), EPA entrustment language makes level 5 = "entrustable + coaches peers." Bloom mastery learning governs the gate philosophy (iterate until mastery, not rank).
|
||||
|
||||
2. **5-level rubric anchoring is well-grounded for Customer Service.** (0.78) Four criteria: empathy, resolution-concreteness, de-escalation, professionalism. Level 3 = "competent entry-level hire, unsupervised." Level 5 = "entrustable, coaches peers." Concrete anchor table in appendix.
|
||||
|
||||
3. **N=3 mastery gate is defensible ONLY as formative/path-completion, NOT high-stakes credentialing.** (0.62) Generalizability theory suggests G≈0.5–0.6 for N=3 — adequate for "advance to next week" but thin for a credential employers trust. **Recommendation: label v0.3 VC as *formative*, reserve N=5–6 + blueprint coverage for a future high-stakes tier (v0.9 credentialing milestone).** Keep the remediation loop (Bloom mastery learning lives there).
|
||||
|
||||
4. **Mastery Score = hybrid weighted-mean + conjunctive floor.** (0.72) Weighted mean of criterion scores (D-039 weights), conjunctive floor: every criterion ≥2 AND scenario mean ≥3.0 to pass that scenario. Path-level mastery gate: ≥3 distinct scenarios passed AND additive MasteryScore ≥3.5 over passing scenarios only (D-032 validated, but see §3: label formative).
|
||||
|
||||
5. **Deterministic rubric scoring: LLM-extracts-evidence, rules-score-evidence (D-038 validated).** (0.80) LLM (deepseek-v4-flash:cloud, temp=0, JSON-schema-validated output) extracts verbatim quotes + signal tags per criterion from session turns; a deterministic YAML rule engine maps signals → 1-5 levels. **Critical: validate extracted quotes fuzzy-match the transcript to block LLM hallucination.** This is the strongest-evidence finding.
|
||||
|
||||
6. **CS weights for refund/complaint: empathy 0.35, resolution 0.30, de-escalation 0.20, professionalism 0.15.** (0.70) Professionalism is a *floor* (conjunctive ≥2), not a weight driver. De-escalation up-weights to ~0.40 if the escalate branch triggers (D-009). **D-039 amendment: weights should be per-scenario-archetype, not one global CS set** — allow `rubrics/customer_service_<archetype>.yaml` or a weights override in the scenario file.
|
||||
|
||||
7. **W3C VC Data Model 2.0 is a W3C Recommendation (15 May 2025) — D-033 targets a stable standard.** (0.85) No batteries-included Python VC library exists; assemble `pynacl` + `canonicaljson` + `base58` + ~200 LOC using the `eddsa-jcs-2022` cryptosuite (avoids RDF canonicalization complexity). Bitstring Status List v1.0 is also a W3C Recommendation — fully self-hostable, no third-party service.
|
||||
|
||||
8. **Issuer ID = bare HTTPS URL (`https://praxis.example/issuers/v0.3`) + self-hosted Multikey public key.** (0.80) `did:key` rejected because it breaks D-042 key rotation (key is baked into the DID). Bare URL + Multikey is W3C-compliant and allows key rotation by archiving the old public key at its original URL (status `superseded`, not revoked — old VCs still verify).
|
||||
|
||||
9. **Verification endpoint returns `{valid, status, issuer, credential, mastery, verifiedAt}`.** (0.80) Verifier fetches the public key from the `verificationMethod` URL, validates the Ed25519 signature, checks status list. No shared secret, no account. Credential payload: `scenariosPassed` (anti-gaming per D-032), `rubricScore`, `completedWeeks: 6` (PRD §6.4), `evidence` (REQ-NFR-MAST-02). **Open: 3-year `validUntil` is a recommendation (PRD §6.4 silent on expiry) — flag for PLAN.**
|
||||
|
||||
10. **Operator Postgres = `postgres:16-slim` as a second docker-compose service on an explicit named bridge network, no published port.** (0.90) `pgdata` named volume, `pg_isready` healthcheck, `depends_on: service_healthy`, init scripts at `/docker-entrypoint-initdb.d/`. **asyncpg `create_pool(min_size=2, max_size=10)` on `app.state` via lifespan; do not share a session with aiosqlite.** New pip deps: `asyncpg>=0.29`, `argon2-cffi>=23.1`, `slowapi>=0.1`.
|
||||
|
||||
11. **Operator auth = Starlette `SessionMiddleware` (itsdangerous-signed, httpOnly+Secure+SameSite=Strict, 8h) + argon2-cffi + slowapi 5/min.** (0.90 stack / 0.70 rate-limit for single-instance) One `current_operator` `Depends` + router-level `dependencies=[...]` under `/op`. Migrate to RBAC only when a 2nd role appears.
|
||||
|
||||
12. **Postgres schema: `operators`, `issued_credentials`, `mastery_gate_events`, `cohort_aggregates` (k-anon via write-time suppression, weekly partitions, 7-day window on read).** (0.80) `gen_random_uuid()` built into PG16 (no extension). Migration: staging-CT first, add service + net + volumes, build new image, then `up -d postgres` → `up -d praxis` recreate (~5–15s downtime, SQLite volume untouched → learner path never regresses). Mirror the existing `db/migrate.py` runner for Postgres (separate migration directory).
|
||||
|
||||
13. **Backup: daily `pg_dump -Fc` to a `pgbackups` named volume, `%u` rolling 7-file retention.** (0.85) Separate from the SQLite volume backup. Drill with `pg_restore --clean --if-exists`.
|
||||
|
||||
14. **k=10 + 7-day trailing window is the right floor for v0.3 (<100 learners).** (0.85) Defer l-diversity/t-closeness until a sensitive attribute enters the cohort schema; defer differential privacy until N>1000. SQL suppression via `COUNT(DISTINCT learner_id) >= 10` with `cell_suppressed` sentinel; limit to pre-defined 2-D views to block differencing attacks.
|
||||
|
||||
15. **IRT 1PL/Rasch: Gaussian-approximation Bayesian θ update (θ₀=0, σ²=1).** (0.80) Target P=0.5 for mastery-gate scenarios, P≈0.7 for practice scenarios (per-scenario `irt_target_p` field). θ reliable after ~5–10 sessions. Ship 1PL (2PL needs ~200 responses/item — post-pilot). **D-046 validated: θ persists in learner-local SQLite (`learner_ability` table: learner_id, path, theta, updated_at).**
|
||||
|
||||
16. **Scenario library: `index.yaml` as slim manifest (metadata only, ~50 lines/scenario).** (0.85) Semver per scenario (MAJOR = rubric_criteria/branch changes invalidate gate evidence). AI variations via `_pending/` dir + mandatory expert review + `generated_from` backref + `intent_hash` for structural-drift detection. Rubric mapping as list of `{criterion_id, weight, evidence_required}` objects. `MIN_COVERAGE = 2` scenarios per rubric criterion enforced in CI.
|
||||
|
||||
---
|
||||
|
||||
## v0.3 Detailed Findings
|
||||
|
||||
### A. Rubric + Mastery Models
|
||||
|
||||
**Rubric model selection (0.82):** Dreyfus 5-stage anchors + Miller's "Does" tier + EPA entrustment language for level 5 + Bloom mastery learning for the gate philosophy. Bloom's *taxonomy* alone is a weak fit for performance assessment.
|
||||
|
||||
**5-level anchor example (Customer Service — refund/complaint archetype):**
|
||||
|
||||
| Criterion | Level 1 (Fail) | Level 3 (Competent) | Level 5 (Mastery/Entrustable) |
|
||||
|-----------|----------------|---------------------|-------------------------------|
|
||||
| Empathy | Ignores/invalidates emotion | Acknowledges emotion before resolution | Names emotion preemptively, validates without surrendering policy |
|
||||
| Resolution | No concrete offer | Offers refund OR replacement | Offers choice, confirms next steps, anticipates follow-up |
|
||||
| De-escalation | Matches/escalates hostility | Stays calm, doesn't inflame | Re-frames hostility into problem-solving, recovers the relationship |
|
||||
| Professionalism | Profane/impersonates real co | Polite, in-role, disclaimer given | Models conduct, names policy without hiding behind it |
|
||||
|
||||
**N=3 defensibility (0.62):** Defensible only as *formative/path-completion*; not defensible as high-stakes credential (G-theory suggests G≈0.5–0.6). **Recommendation: label v0.3 VC as formative; reserve N=5–6 with blueprint coverage for a future high-stakes tier.**
|
||||
|
||||
**Mastery Score computation (0.72):** Hybrid — weighted mean of criterion scores + conjunctive floor (every criterion ≥2, scenario mean ≥3.0 to pass). Path-level gate: ≥3 distinct passed scenarios AND additive MasteryScore ≥3.5 over passing scenarios only.
|
||||
|
||||
**Deterministic scoring (0.80):** LLM (deepseek-v4-flash:cloud, temp=0) extracts verbatim quotes + signal tags (JSON-schema-validated) from session turns; deterministic YAML rule engine maps signals → 1-5 levels. **Validate quotes fuzzy-match the transcript to block hallucination.** No LLM in the numeric scoring step (preserves REQ-NFR-MAST-01).
|
||||
|
||||
**CS weights (0.70):** empathy 0.35, resolution 0.30, de-escalation 0.20, professionalism 0.15. Professionalism = floor (≥2), not weight driver. De-escalation up-weights to ~0.40 if escalate branch triggers. **D-039 amendment: per-archetype weights, not one global CS set.**
|
||||
|
||||
### B. Verifiable Credentials Stack
|
||||
|
||||
**VC Data Model 2.0 (0.85):** W3C Recommendation (15 May 2025). D-033 targets a stable standard.
|
||||
|
||||
**Python library (0.80):** No batteries-included VC lib. Assemble: `pynacl` (Ed25519) + `canonicaljson` (JCS canonicalization) + `base58` (Multikey encoding) + ~200 LOC using the `eddsa-jcs-2022` cryptosuite. Avoids RDF canonicalization complexity.
|
||||
|
||||
**Status List revocation (0.80):** Bitstring Status List v1.0 (W3C Recommendation). Fully self-hostable — no third-party service. Minimum viable: one bitstring per status list, indexed by credential sequence.
|
||||
|
||||
**Issuer ID (0.80):** Bare HTTPS URL (`https://praxis.example/issuers/v0.3`) + self-hosted Multikey public key. `did:key` rejected — key baked into DID breaks rotation (D-042).
|
||||
|
||||
**Verification endpoint (0.80):** `GET /vc/verify/<id>` → `{valid, status, issuer, credential, mastery, verifiedAt}`. Verifier fetches public key from `verificationMethod` URL, validates Ed25519 signature, checks status list. No account, no shared secret.
|
||||
|
||||
**Credential payload:** `scenariosPassed` (D-032 anti-gaming), `rubricScore`, `completedWeeks: 6` (PRD §6.4), `evidence` (REQ-NFR-MAST-02). **Open: 3-year `validUntil` is a recommendation (PRD §6.4 silent) — flag for PLAN.**
|
||||
|
||||
**Key rotation (D-042 validated, 0.80):** Archive old public key at its original URL (status `superseded`, not revoked). New key signs new VCs. Old VCs still verify against archived public key.
|
||||
|
||||
### C. Multi-Tenant Auth + Postgres-in-LXC
|
||||
|
||||
**Docker-compose shape (0.90):** `postgres:16-slim`, explicit named bridge network (not `host`), no published port, `pgdata` named volume, `pg_isready` healthcheck, `depends_on: service_healthy`, init scripts at `/docker-entrypoint-initdb.d/`.
|
||||
|
||||
**Connection management (0.85):** Independent pools — asyncpg `create_pool(min_size=2, max_size=10)` on `app.state` via lifespan; aiosqlite per-call connect. Do not share a session. `command_timeout=10`.
|
||||
|
||||
**Auth stack (0.90 stack / 0.70 rate-limit):** Starlette `SessionMiddleware` (itsdangerous-signed, httpOnly+Secure+SameSite=Strict, 8h) + `argon2-cffi` (argon2id, `check_needs_rehash`) + `slowapi` 5/min in-memory on login route. **Risk: Secure cookie requires TLS — v0.2 pilot is direct-IP no-TLS. Either relax Secure for pilot or add TLS. Flag for PLAN.**
|
||||
|
||||
**Auth dependency pattern (0.90):** One `current_operator` `Depends` + router-level `dependencies=[...]` under `/op`. Migrate to RBAC only when a 2nd role appears.
|
||||
|
||||
**Postgres schema (0.80):** `operators` (id, username, password_hash, created_at), `issued_credentials` (id, learner_ref, vc_payload_json, signature_b64, status, issued_at), `mastery_gate_events` (id, learner_ref, path, week, scenarios_passed_json, rubric_scores_json, gate_opened_at), `cohort_aggregates` (path, week, window_start, window_end, metric, value, cell_suppressed bool — partition by week). `gen_random_uuid()` in PG16 (no extension). **No cross-DB joins via `learner_ref` — `learner_ref` is an opaque string, not a FK.**
|
||||
|
||||
**Migration strategy (0.85):** Staging-CT first, add service + net + volumes, build new image, then `up -d postgres` → `up -d praxis` recreate (~5–15s downtime, SQLite volume untouched → learner path never regresses). Mirror `db/migrate.py` runner for Postgres (separate `db/pg_migrations/` directory).
|
||||
|
||||
**Backup (0.85):** Daily `pg_dump -Fc` to a `pgbackups` named volume, `%u` rolling 7-file retention. Separate from SQLite volume backup. Drill: `pg_restore --clean --if-exists`.
|
||||
|
||||
### D. Anonymization + IRT + Scenario Library
|
||||
|
||||
**k-anonymity (0.85):** k=10 + 7-day trailing window is the right floor for v0.3 (<100 learners). Defer l-diversity/t-closeness until a sensitive attribute enters the cohort schema; defer differential privacy until N>1000. SQL: `COUNT(DISTINCT learner_id) >= 10` with `cell_suppressed` sentinel. Limit to pre-defined 2-D views (path × week, path × outcome) to block differencing attacks.
|
||||
|
||||
**IRT 1PL/Rasch (0.80):** P(success) = logistic(θ − b). θ₀=0, σ²=1 (Gaussian-approximation Bayesian prior). Update: θ ← θ + (outcome − P) × σ²/(σ² + 1) per session; widen σ² after failures. Target P=0.5 for mastery-gate scenarios, P≈0.7 for practice scenarios (per-scenario `irt_target_p` field). θ reliable after ~5–10 sessions. Ship 1PL (2PL needs ~200 responses/item — post-pilot). **D-046 validated: θ in learner-local SQLite.**
|
||||
|
||||
**Scenario library (0.85):** `scenarios/<path>/<id>.yaml` + `scenarios/index.yaml` (slim manifest: id, path, title, difficulty, failure_mode, rubric_criteria, version, author, generated_from). Semver per scenario (MAJOR = rubric_criteria/branch changes invalidate gate evidence). AI variations: `_pending/` dir + mandatory expert review + `generated_from` backref + `intent_hash` for structural-drift detection. Rubric mapping: list of `{criterion_id, weight, evidence_required}`. `MIN_COVERAGE = 2` scenarios per rubric criterion enforced in CI.
|
||||
|
||||
---
|
||||
|
||||
## v0.3 Risks (for PLAN stage)
|
||||
|
||||
| ID | Risk | Mitigation | Confidence |
|
||||
|----|------|------------|------------|
|
||||
| R-MAST-01 | N=3 gate too thin for credible credential | Label v0.3 VC as *formative*; reserve high-stakes for v0.9 (N=5-6 + blueprint) | 0.62 |
|
||||
| R-MAST-02 | LLM hallucinates rubric evidence quotes | Fuzzy-match extracted quotes against transcript; reject + re-extract on mismatch | 0.80 |
|
||||
| R-MAST-03 | Rubric weights wrong for non-refund CS archetypes | Per-archetype weights (D-039 amendment); start with refund/complaint, generalize later | 0.70 |
|
||||
| R-VC-01 | No batteries-included Python VC lib → ~200 LOC custom code | Use `eddsa-jcs-2022` cryptosuite (well-specified); pin `pynacl` + `canonicaljson` + `base58`; unit-test signature/verify round-trip | 0.75 |
|
||||
| R-VC-02 | `validUntil` expiry undefined in PRD | Adopt 3-year expiry as default; make configurable; flag in PLAN | 0.60 |
|
||||
| R-AUTH-01 | Secure cookie flag fails without TLS (v0.2 is direct-IP no-TLS) | Relax `Secure` for pilot OR add TLS (Traefik sidecar); flag for PLAN | 0.70 |
|
||||
| R-MT-01 | Postgres-in-LXC resource contention with praxis service | Bump CT memory to 6GB (Postgres ~1GB + praxis ~2GB + build headroom); monitor | 0.65 |
|
||||
| R-MT-02 | Cohort aggregation race on concurrent session-end | Write-time suppression + nightly reconciliation job (D-045); idempotent upserts | 0.70 |
|
||||
| R-IRT-01 | θ unreliable for first ~5-10 sessions (cold start) | Fall back to fixed difficulty (scenario.difficulty) until θ has ≥5 observations; show "calibrating" state to learner | 0.75 |
|
||||
| R-LIB-01 | AI variations drift from expert intent | `intent_hash` structural-drift detection + mandatory expert review before `_pending/` → library promotion | 0.75 |
|
||||
|
||||
---
|
||||
|
||||
## v0.3 Open Questions for PLAN Stage
|
||||
|
||||
1. **Secure cookie + no-TLS pilot:** Relax `Secure` flag for v0.3 pilot (direct-IP), or add a Traefik sidecar for TLS? (R-AUTH-01)
|
||||
2. **CT memory bump:** v0.2 CT is 4GB. Postgres + praxis + build headroom may need 6GB. Confirm via staging-CT test. (R-MT-01)
|
||||
3. **VC `validUntil`:** Adopt 3-year default? Make per-path configurable? (R-VC-02)
|
||||
4. **Phase split:** Is v0.3 one execution phase or 2-3? Scope (mastery core + scenarios + paths + VC + auth + Postgres + dashboard) suggests 2-3 phases. Planner decides.
|
||||
5. **Rubric per-archetype weights:** Ship refund/complaint weights only in v0.3, or author weights for ≥2 archetypes? (R-MAST-03)
|
||||
6. **IRT cold-start UX:** Show "calibrating difficulty" to learner, or hide it? (R-IRT-01)
|
||||
7. **Failure-injection coupling:** RESEARCH confirms D-049 — no active failure injection in v0.3. Confirm no hidden coupling to mastery scoring.
|
||||
+45
-4
@@ -1,13 +1,54 @@
|
||||
# Praxis — Roadmap
|
||||
|
||||
**Milestone:** v0.2 (Proxmox LXC deployment)
|
||||
**Status:** phase 1 complete — P2 review/ship in-progress
|
||||
**Milestone:** v0.3 (Mastery scoring + competency rubrics + verifiable credentials)
|
||||
**Status:** phase 0 — plan (grill-amended)
|
||||
**Previous milestone:** v0.2 (Proxmox LXC deployment) — complete, tagged v0.1.2, release #377
|
||||
|
||||
## Milestone Philosophy
|
||||
|
||||
v0.2 deploys praxis into a Proxmox LXC container, reusing and adapting the battle-tested deployment toolkit from `~/coreci/scripts/proxmox/`. The v0.1 voice loop becomes deployable infrastructure. v1.0 is reserved for a working, tested product and is a future milestone.
|
||||
v0.3 activates the mastery/assessment layer deferred from v0.1/v0.2 (per D-021). Learners progress via **mastery gates** — they move on only when they can do the thing across varied scenarios, scored against a competency rubric. On week-final gate-open, a **formative verifiable credential** (W3C VC 2.0, Ed25519) is issued so mastery is portable. The v0.2 LXC deployment carries forward unchanged. **Operator tier (cohort dashboard + auth + Postgres) is deferred to v0.4** per the grill's binding verdict (GRILL-v0.3.md Axis 2 — the operator tier was originally v0.8 on this roadmap; pulling it into v0.3 created a 2-milestone program disguised as one).
|
||||
|
||||
## v0.2 Phases (2 phases)
|
||||
## v0.3 Phases (post-grill)
|
||||
|
||||
### Phase 0 — Pre-Execution (in-progress — this phase)
|
||||
|
||||
**Branch:** `phase/00-pre-execution` → merged to `milestone/v0.3-mastery-scoring`
|
||||
**Ship target:** `v0.1.3` (patch release on v0.2's v0.1.x line — NFR/docs milestone type)
|
||||
**Status:** in-progress (PLAN — grill-amended)
|
||||
|
||||
Pipeline stages: SPECIFY → CLARIFY → RESEARCH → PLAN → GRILL → SHIP
|
||||
|
||||
**Goal:** Produce all `.ciagent/` planning artifacts for v0.3: activated requirements (REQ-MAST-01/02/03, REQ-SCEN-02/03/04, REQ-PATH-02 + 6 NFRs), research-grounded rubric/VC/IRT/architecture, persona roster, vertical-slice plan for P1. Operator tier (REQ-DASH-01, REQ-AUTH-01, REQ-MT-01/02 + 4 NFRs) deferred to v0.4 per grill.
|
||||
|
||||
**Deliverables:**
|
||||
- PROJECT.md (v0.3 scope validated, D-031..D-049 recorded; operator tier deferred)
|
||||
- REQUIREMENTS.md (v0.3 active REQ-IDs = 13; 8 deferred to v0.4)
|
||||
- ARCHITECTURE.md (mastery engine + VC issuer + IRT added to v0.2 topology; operator-tier Postgres deferred to v0.4)
|
||||
- PERSONAS.md (v0.3 roster — security-engineer added for VC crypto; frontend + devops deactivated)
|
||||
- GRILL-v0.3.md (4 MUST conditions resolved, 5 FIX tracked)
|
||||
- Phase 1 plan (9 slices, 5 waves, ~40 tasks, 13/13 REQ coverage)
|
||||
|
||||
### Phase 1 — Mastery Core + VC Issuance (planned)
|
||||
|
||||
**Branch:** `phase/01-mastery-core` → merged to `milestone/v0.3-mastery-scoring`
|
||||
**Ship target:** `v0.1.4` (patch release, feature milestone type)
|
||||
**Status:** planned
|
||||
|
||||
**Goal:** Competency rubric engine + Mastery Score computation + scenario library (≥6 CS scenarios) + dynamic difficulty (IRT) + Customer Service path (6 weeks) + verifiable-credential issuer (W3C VC 2.0, Ed25519, SQLite-backed, formative-tier, public verification). All learner-facing. 9 slices, 5 waves, ~40 tasks.
|
||||
|
||||
### Final Phase (P2) — Review + Ship (planned)
|
||||
|
||||
**Branch:** `phase/02-final-review-ship` → merged to `milestone/v0.3-mastery-scoring` → merged to `main`
|
||||
**Ship target:** final patch = v0.3 milestone release
|
||||
**Status:** planned
|
||||
|
||||
**Goal:** Multi-persona code review, project audit, milestone merge to main, milestone release.
|
||||
|
||||
## v0.4 Milestone (planned — operator tier, deferred from v0.3 per grill)
|
||||
|
||||
v0.4 activates the operator tier deferred from v0.3: REQ-DASH-01 (cohort dashboard), REQ-AUTH-01 (operator auth), REQ-MT-01/02 (Postgres + aggregation), + 4 NFRs. This restores the original ROADMAP intent (dashboard was v0.8) while following the grill's "split the milestone" verdict.
|
||||
|
||||
## v0.2 Milestone (complete — reference)
|
||||
|
||||
### Phase 0 — Pre-Execution (complete — tagged v0.1.0, release #371)
|
||||
|
||||
|
||||
@@ -3,8 +3,8 @@
|
||||
{
|
||||
"slug": "praxis",
|
||||
"name": "Praxis",
|
||||
"milestone": "v0.2",
|
||||
"status": "phase-1-complete"
|
||||
"milestone": "v0.3",
|
||||
"status": "phase-0-specify"
|
||||
}
|
||||
],
|
||||
"active_project": "praxis",
|
||||
|
||||
@@ -0,0 +1,475 @@
|
||||
# RESEARCH: Operator Tier — Postgres-in-LXC + Auth for v0.3
|
||||
|
||||
**Scope:** Research only. No code changes. Grounded in the current Praxis repo
|
||||
(`docker-compose.yml` single `praxis` service; `db/store.py` aiosqlite
|
||||
`PraxisStore`; `db/migrate.py` ordered `.sql` migrations; SQLite schema at
|
||||
`db/schema.sql`).
|
||||
|
||||
**Decisions honored:** D-007 (SQLite learner, preserved), D-031 (hybrid:
|
||||
SQLite for learner, Postgres for operator), D-040 (Postgres = second
|
||||
docker-compose service in the existing LXC CT), D-041 (session-cookie auth,
|
||||
argon2id, single operator role, rate-limited).
|
||||
|
||||
**Confidence scores** are 0–1 (1 = well-established practice / low risk).
|
||||
|
||||
---
|
||||
|
||||
## 1. Docker-Compose Shape *(confidence: 0.90)*
|
||||
|
||||
Add a `postgres` service alongside the existing `praxis` service. Key
|
||||
best-practices for a second service in an already-running LXC CT:
|
||||
|
||||
- **Image:** `postgres:16-slim` (Debian-slim base, glibc — matches the
|
||||
praxis Dockerfile rationale; avoids Alpine musl locale issues with
|
||||
`pg_*` clients).
|
||||
- **Persistence:** named volume `pgdata` (driver: local). Never bind-mount
|
||||
`/var/lib/postgresql/data` to the CT filesystem — Postgres requires
|
||||
`chown 999` and a specific directory layout; named volumes handle this.
|
||||
- **Network isolation:** declare an explicit internal compose network and
|
||||
attach **only** `praxis` and `postgres` to it. Do **not** publish
|
||||
`5432` via `ports:`. The `praxis` service keeps its published `8789`.
|
||||
- `internal: true` on the network blocks egress to the host bridge, but
|
||||
note: with `internal: true` the postgres container cannot reach the
|
||||
internet (fine — it doesn't need to). If you later want outbound
|
||||
backups via network, drop `internal: true` and instead rely on
|
||||
*not* publishing the port. The simpler, robust choice for a pilot is:
|
||||
explicit named network, no `ports:` on postgres, no `internal: true`.
|
||||
- **Healthcheck:** `pg_isready -U praxis -d praxis` every 10s, 5 retries,
|
||||
5s timeout. `depends_on: { postgres: { condition: service_healthy } }`
|
||||
on the `praxis` service so the app waits for accept-connections, not
|
||||
just container start.
|
||||
- **Init scripts:** mount `./db/pg/init/*.sql` (or `.sh`) at
|
||||
`/docker-entrypoint-initdb.d/`. These run **only on first boot** (empty
|
||||
`pgdata`). Use them for: role/db creation, schema bootstrap, and
|
||||
idempotent seed. For *versioned* schema changes use a migration runner
|
||||
(see §6) — init scripts are one-shot.
|
||||
- **Env:** `POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_DB` from the
|
||||
existing `/etc/praxis/server.env` (do **not** commit secrets to the
|
||||
compose file). Add `PGDATA=/var/lib/postgresql/data/pgdata` to pin the
|
||||
subdirectory (survives image upgrades).
|
||||
- **Restart:** `restart: unless-stopped` (matches praxis).
|
||||
- **Resources:** for a pilot on a small LXC CT, set a mem limit
|
||||
(`deploy.resources.limits.memory: 512m`) and rely on Postgres default
|
||||
`shared_buffers`. Tune later.
|
||||
|
||||
**Sketch (shape only, not for commit):**
|
||||
|
||||
```yaml
|
||||
services:
|
||||
praxis:
|
||||
# ... existing v0.2 fields unchanged ...
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
networks: [praxis-net]
|
||||
|
||||
postgres:
|
||||
image: postgres:16-slim
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
POSTGRES_USER: ${PG_USER}
|
||||
POSTGRES_PASSWORD: ${PG_PASSWORD}
|
||||
POSTGRES_DB: ${PG_DB:-praxis_operator}
|
||||
PGDATA: /var/lib/postgresql/data/pgdata
|
||||
env_file:
|
||||
- path: /etc/praxis/server.env
|
||||
required: false
|
||||
volumes:
|
||||
- pgdata:/var/lib/postgresql/data
|
||||
- ./db/pg/init:/docker-entrypoint-initdb.d:ro
|
||||
- pgbackups:/backups
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U ${PG_USER:-praxis} -d ${PG_DB:-praxis_operator}"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
networks: [praxis-net]
|
||||
# NOTE: no `ports:` — not exposed to the LXC host bridge.
|
||||
|
||||
volumes:
|
||||
praxis-data:
|
||||
driver: local
|
||||
pgdata:
|
||||
driver: local
|
||||
pgbackups:
|
||||
driver: local
|
||||
|
||||
networks:
|
||||
praxis-net:
|
||||
driver: bridge
|
||||
```
|
||||
|
||||
**Risk callouts:**
|
||||
- If `praxis` currently has no explicit network, compose assigns the
|
||||
default bridge; adding an explicit network means the *existing*
|
||||
`praxis` service gets recreated on `up`. Plan a brief downtime window
|
||||
(see §6).
|
||||
- `pg_isready` returns healthy before the DB is fully ready for migration
|
||||
load; `depends_on: service_healthy` is necessary but not sufficient —
|
||||
the app must still retry the first migration attempt.
|
||||
|
||||
---
|
||||
|
||||
## 2. Connection Management *(confidence: 0.85)*
|
||||
|
||||
Two async DB drivers in one process: **aiosqlite** (already a dep) for the
|
||||
learner store, **asyncpg** for the operator store.
|
||||
|
||||
- **Pools are independent and must not be shared.** asyncpg uses a
|
||||
`asyncpg.create_pool(...)` (sized pool, real connections). aiosqlite
|
||||
opens a fresh connection per `async with aiosqlite.connect(...)` (the
|
||||
current `PraxisStore._connect` pattern). They have nothing in common —
|
||||
different backends, different lifecycles. **Do not** wrap them in a
|
||||
single shared `AsyncSession` object; SQLAlchemy's async session is an
|
||||
option *only if* you adopt SQLAlchemy for both — that's a larger
|
||||
refactor and not warranted for v0.3.
|
||||
- **Pool sizing (avoid exhaustion):**
|
||||
- asyncpg pool: `min_size=2, max_size=10` for a pilot single-instance.
|
||||
Operator endpoints are low-frequency (cohort dashboard, VC issuance).
|
||||
- aiosqlite: no pool; the current pattern opens/closes per call. SQLite
|
||||
is single-writer; keep `WAL` mode and short transactions. This is
|
||||
already fine for one learner.
|
||||
- Total concurrent DB connections ≈ asyncpg(10) + aiosqlite(1-2). On a
|
||||
small CT this is trivial. Exhaustion risk is essentially zero at
|
||||
pilot scale; revisit if operator endpoints are hit by N concurrent
|
||||
cohort users.
|
||||
- **Lifecycle:** create the asyncpg pool once at FastAPI startup
|
||||
(`lifespan` context manager), close on shutdown. Store on
|
||||
`app.state.pg_pool`. The `PraxisStore` keeps its current per-call
|
||||
connect pattern (no change to D-007 code path).
|
||||
- **Transaction boundaries:** asyncpg use `pool.acquire()` +
|
||||
`conn.transaction()` for multi-statement writes; aiosqlite unchanged.
|
||||
- **Config:** `PG_DSN` env var, e.g.
|
||||
`postgresql://praxis:***@postgres:5432/praxis_operator` (host =
|
||||
service name on `praxis-net`).
|
||||
- **Statement timeout:** set `command_timeout=10` on the asyncpg pool to
|
||||
prevent a slow operator query from blocking the event loop.
|
||||
|
||||
**Pip:** `asyncpg>=0.29` (new dep). `aiosqlite>=0.20` already present.
|
||||
|
||||
---
|
||||
|
||||
## 3. Auth Stack *(confidence: 0.90 for the stack; 0.70 for rate-limit choice)*
|
||||
|
||||
D-041 spec: session-cookie, argon2id, single operator role, rate-limited.
|
||||
|
||||
### 3a. Session cookie
|
||||
- **`starlette` `SessionMiddleware`** (FastAPI bundles Starlette). Uses
|
||||
`itsdangerous` to sign the cookie — no server-side session store
|
||||
needed (stateless, fits single-instance LXC). Data lives in the cookie
|
||||
itself, signed with `SECRET_KEY`.
|
||||
- **Settings:**
|
||||
- `secret_key`: from env, ≥32 bytes random. **Rotate** by changing the
|
||||
key (invalidates all sessions — acceptable for a pilot).
|
||||
- `session_cookie`: `"praxis_op"` (distinct from any future learner
|
||||
cookie name).
|
||||
- `max_age`: `28800` (8h, per D-041).
|
||||
- `path`: `/` (or scope to `/op` if operator routes live under a
|
||||
prefix — cleaner).
|
||||
- `https_only`: `True` (Secure flag). **Requires TLS** — the LXC
|
||||
deployment must terminate TLS (reverse proxy / Caddy / Proxmox
|
||||
level). If running plain HTTP on the LAN for the pilot, set to
|
||||
`False` *temporarily* and document the risk; never ship False.
|
||||
- `httponly`: `True` (the middleware sets this by default; verify).
|
||||
- `samesite`: `"strict"` (D-041). CSRF defense-in-depth; with Strict,
|
||||
no credential is sent on cross-site navigations.
|
||||
- **Cookie contents:** store `{operator_id: str, issued_at: epoch}`.
|
||||
**Never** store the password hash or any PII. Roles aren't needed in
|
||||
the cookie yet (single role — see §4).
|
||||
|
||||
### 3b. Password hashing — argon2id
|
||||
- **`argon2-cffi`** (`PasswordHasher` default is argon2id, RFC 9106).
|
||||
Pip: `argon2-cffi>=23.1`.
|
||||
- On login: `ph.verify(stored_hash, password)` → on success,
|
||||
`ph.check_needs_rehash(stored_hash)` → rehash if params bumped.
|
||||
- Params: keep `PasswordHasher()` defaults for v0.3
|
||||
(`time_cost=3, memory_cost=64MiB, parallelism=4` — reasonable on a
|
||||
small CT; benchmark and tune if login latency > 1s).
|
||||
- Store the hash as `TEXT` in `operators.password_hash`.
|
||||
|
||||
### 3c. Rate limiting
|
||||
Two options:
|
||||
1. **`slowapi`** (pip `slowapi>=0.1`) — the idiomatic FastAPI choice.
|
||||
Decorator/IP-based limiter. Default in-memory backend is fine for
|
||||
single-instance. **Confidence 0.70** — it works, but it's a young lib
|
||||
and the in-memory backend is per-process (breaks if you ever scale to
|
||||
>1 praxis process; not a v0.3 concern).
|
||||
2. **In-memory counter** (a simple `dict[remote_ip, (count, window_start)]`
|
||||
in a small dependency) — zero deps, trivially auditable. For a single
|
||||
operator login endpoint this is enough. **Confidence 0.80** for the
|
||||
pilot specifically.
|
||||
|
||||
**Recommendation:** start with `slowapi` on the login route only
|
||||
(`@limiter.limit("5/minute")`), in-memory backend. Migrate to a Redis
|
||||
backend only if/when you go multi-instance. Threshold: 5 failed
|
||||
attempts/minute/IP → 429 + exponential backoff marker.
|
||||
|
||||
**Pip additions:** `argon2-cffi>=23.1`, `slowapi>=0.1`. (`starlette` and
|
||||
`itsdangerous` come with FastAPI.)
|
||||
|
||||
---
|
||||
|
||||
## 4. Auth Dependency Pattern *(confidence: 0.90)*
|
||||
|
||||
Single-role v0.3 → **no RBAC framework needed.** A single FastAPI
|
||||
`Depends` that resolves the operator from the signed session is the
|
||||
minimal secure shape.
|
||||
|
||||
Concept (not committed code):
|
||||
|
||||
```python
|
||||
# pseudo — shape only
|
||||
async def current_operator(request: Request) -> Operator:
|
||||
sess = request.session # populated by SessionMiddleware
|
||||
op_id = sess.get("operator_id")
|
||||
if not op_id:
|
||||
raise HTTPException(401, "not authenticated")
|
||||
op = await pg_store.get_operator(op_id)
|
||||
if not op or not op.is_active:
|
||||
# invalidate the cookie
|
||||
request.session.clear()
|
||||
raise HTTPException(401, "operator not found / disabled")
|
||||
return op
|
||||
```
|
||||
|
||||
- Apply via `Depends(current_operator)` on every operator-tier router.
|
||||
Group operator routes under an `APIRouter(prefix="/op")` and attach
|
||||
the dependency at the router level
|
||||
(`dependencies=[Depends(current_operator)]`) — one declaration, not
|
||||
per-endpoint.
|
||||
- Login/logout are **outside** the protected router (login is rate-
|
||||
limited, not auth-gated).
|
||||
- **CSRF:** with `SameSite=Strict` + `httponly` cookies, CSRF surface is
|
||||
minimal for state-changing requests. If any operator endpoint accepts
|
||||
`Content-Type: application/x-www-form-urlencoded`/`multipart` (form
|
||||
posts), add a double-submit token or require `Content-Type:
|
||||
application/json` only (the latter is the cheaper defense — JSON
|
||||
bodies are not auto-sent by browsers across origins).
|
||||
|
||||
### When to migrate to RBAC
|
||||
Migrate when **any** of these become true:
|
||||
- A second role appears (admin, auditor, reviewer) — i.e. v0.4+ if the
|
||||
pilot expands.
|
||||
- Permissions diverge *within* a role (e.g. some operators can issue
|
||||
VCs, others can only view cohorts).
|
||||
- You need row-level visibility rules (operator A sees only their
|
||||
cohort).
|
||||
|
||||
At that point the cheapest upgrade is: add a `role` column to
|
||||
`operators`, split `current_operator` into `current_operator` (any
|
||||
authenticated) + `require_role("admin")` (a parametrized dependency
|
||||
checking `op.role`). Reach for a full RBAC lib (`casbin`,
|
||||
`fastapi-permissions`) only when the role matrix exceeds ~3 roles × ~5
|
||||
permissions. **Don't pre-build it.**
|
||||
|
||||
---
|
||||
|
||||
## 5. Postgres Schema *(confidence: 0.80)*
|
||||
|
||||
Operator-tier tables. Types chosen for Postgres 16 specifically
|
||||
(`TIMESTAMPTZ`, `BIGSERIAL`, `GENERIC` via `JSONB`).
|
||||
|
||||
### `operators`
|
||||
```
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid()
|
||||
username TEXT NOT NULL UNIQUE
|
||||
password_hash TEXT NOT NULL -- argon2id
|
||||
display_name TEXT NOT NULL
|
||||
role TEXT NOT NULL DEFAULT 'operator' -- reserved for §4 migration
|
||||
is_active BOOLEAN NOT NULL DEFAULT TRUE
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
last_login_at TIMESTAMPTZ
|
||||
```
|
||||
- Index: unique on `username` (covered by constraint). No extra index
|
||||
needed at single-operator scale.
|
||||
- Requires `pgcrypto` extension **or** Postgres 13+ (where
|
||||
`gen_random_uuid()` is built-in via `pgcrypto` shipped default —
|
||||
actually: `gen_random_uuid()` is built into core as of PG 13). So no
|
||||
extension needed on PG16. ✓
|
||||
|
||||
### `issued_credentials`
|
||||
```
|
||||
id BIGSERIAL PRIMARY KEY
|
||||
operator_id UUID NOT NULL REFERENCES operators(id)
|
||||
learner_ref TEXT, -- opaque ref into SQLite side (no FK cross-DB)
|
||||
vc_type TEXT NOT NULL -- 'mastery' | 'completion' | ...
|
||||
payload_jsonb JSONB NOT NULL -- the W3C VC document (signed elsewhere)
|
||||
issued_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
revoked_at TIMESTAMPTZ
|
||||
```
|
||||
- Indices:
|
||||
- `issued_credentials(operator_id, issued_at DESC)` — operator's
|
||||
issuance log.
|
||||
- `issued_credentials(learner_ref)` — lookup by learner (k-anon
|
||||
aggregate joins).
|
||||
- `issued_credentials(vc_type)` if filtering by type is a dashboard
|
||||
query.
|
||||
|
||||
### `mastery_gate_events`
|
||||
```
|
||||
id BIGSERIAL PRIMARY KEY
|
||||
learner_ref TEXT NOT NULL
|
||||
scenario_id TEXT NOT NULL
|
||||
path_id TEXT NOT NULL -- learning path
|
||||
gate_outcome TEXT NOT NULL -- 'pass' | 'fail' | 'retry'
|
||||
recorded_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
source TEXT NOT NULL DEFAULT 'sync' -- 'sync' from SQLite learner store
|
||||
```
|
||||
- Indices:
|
||||
- `(learner_ref, recorded_at DESC)` — per-learner timeline.
|
||||
- `(path_id, recorded_at)` — feeds the cohort aggregate.
|
||||
|
||||
### `cohort_aggregates` — k-anonymized
|
||||
Model as **pre-materialized rows** partitioned by `(path_id, week)` with
|
||||
a minimum bin size enforced at write time (k≥K, e.g. K=5). A 7-day
|
||||
window is a rolling construct over the weekly partitions.
|
||||
|
||||
```
|
||||
path_id TEXT NOT NULL
|
||||
week_start DATE NOT NULL -- ISO week Monday
|
||||
bin_count INTEGER NOT NULL -- learners in this bin
|
||||
k_anon_pass INTEGER NOT NULL -- pass count, suppressed if < K
|
||||
k_anon_fail INTEGER NOT NULL -- fail count, suppressed if < K
|
||||
median_attempts INTEGER
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
PRIMARY KEY (path_id, week_start)
|
||||
```
|
||||
- **k-anon rule:** when materializing, if `bin_count < K` emit
|
||||
`bin_count = <K-masked>` and null-out the count columns (or clamp
|
||||
them to K). Enforce in the aggregation job, **not** in a SQL view, so
|
||||
the suppression is auditable at write time.
|
||||
- **7-day window:** compute on read as a window function over the last
|
||||
≤2 weekly partitions, or maintain a parallel rolling table. For a
|
||||
pilot, compute on read:
|
||||
`SUM(k_anon_pass) ... WHERE week_start >= now()::date - interval '7 days'`.
|
||||
- Indices: PK covers `(path_id, week_start)`. Add a secondary
|
||||
`(week_start DESC)` only if you query "all paths for the latest week"
|
||||
frequently.
|
||||
|
||||
**General indices summary:** 4 indices beyond PKs/constraints for v0.3
|
||||
— keep it lean; add per slow-query evidence.
|
||||
|
||||
---
|
||||
|
||||
## 6. Migration Strategy *(confidence: 0.85)*
|
||||
|
||||
Goal: add Postgres to the **running** v0.2 LXC CT without breaking the
|
||||
learner service.
|
||||
|
||||
### Steps (ordered, low-risk)
|
||||
1. **Prepare on a staging CT first** (clone the production LXC CT in
|
||||
Proxmox). Never test the migration path on the live CT.
|
||||
2. **Add the `postgres` service + `praxis-net` + volumes** to
|
||||
`docker-compose.yml`. The `praxis` service gains
|
||||
`depends_on: postgres (service_healthy)` and joins `praxis-net`.
|
||||
3. **Add init scripts** under `db/pg/init/`:
|
||||
- `00_create_schema.sql` — the four tables from §5.
|
||||
- `01_seed_operator.sh` — creates the initial operator with an
|
||||
argon2id hash (run from env-supplied temp password; force password
|
||||
change on first login).
|
||||
These run **only on first boot** of an empty `pgdata` volume.
|
||||
4. **Add the asyncpg pool + operator store + auth wiring** to the praxis
|
||||
image (new code paths, new deps in `pyproject.toml`). Learner paths
|
||||
(`db/store.py`, `db/migrate.py`) **unchanged** — D-007 preserved.
|
||||
5. **Build the new image** (`docker compose build praxis`) — does not
|
||||
touch the running container.
|
||||
6. **Controlled cutover:**
|
||||
- `docker compose up -d postgres` → wait for healthy.
|
||||
- `docker compose up -d praxis` → recreate the praxis container with
|
||||
the new image. Expect ~5–15s of downtime (the learner voice loop
|
||||
is not HA anyway). The SQLite volume (`praxis-data`) is untouched,
|
||||
so learner state is preserved across the recreate.
|
||||
7. **Smoke tests:** `/health`, learner voice loop, operator login, one
|
||||
cohort-dashboard read.
|
||||
8. **Rollback plan:** if operator endpoints misbehave, revert the
|
||||
praxis image tag and `docker compose up -d praxis` again — Postgres
|
||||
stays up but unused. Learner path is independent, so a bad operator
|
||||
rollout does **not** regress v0.2 learner behavior. This is the
|
||||
core safety property of the hybrid (D-031) design.
|
||||
|
||||
### Versioned migrations beyond first boot
|
||||
The SQLite side already has `db/migrate.py` (ordered `.sql`, `_migrations`
|
||||
table). For Postgres, two options:
|
||||
- **(a) Reuse the pattern:** a `pg_migrate.py` mirroring the SQLite
|
||||
runner, against a `_pg_migrations` table. Lowest cognitive load —
|
||||
same mental model, same directory convention (`db/pg/migrations/`).
|
||||
- **(b) Adopt `yoyo-migrations` or `alembic`:** more machinery, not
|
||||
warranted at 4 tables.
|
||||
|
||||
**Recommendation (a):** mirror the existing runner. Run on praxis
|
||||
startup (after the pool is up), idempotent. **Confidence 0.80** on the
|
||||
pattern; it's exactly what v0.2 already does for SQLite.
|
||||
|
||||
---
|
||||
|
||||
## 7. Backup *(confidence: 0.85)*
|
||||
|
||||
Minimum viable backup for a pilot operator Postgres in LXC:
|
||||
|
||||
- **Method:** `pg_dump -Fc` (custom compressed format) → file in the
|
||||
`pgbackups` volume. `-Fc` gives you selective restore and parallel
|
||||
restore later.
|
||||
- **Frequency:** daily is enough for a pilot. A cron job *inside the
|
||||
postgres container* (or a sidecar) runs:
|
||||
```
|
||||
pg_dump -U praxis -Fc praxis_operator > /backups/pg_$(date +%u).dump
|
||||
```
|
||||
Using `%u` (day-of-week 1–7) gives a rolling 7-file retention with
|
||||
zero cleanup logic.
|
||||
- **Where:** `/backups` is the `pgbackups` named volume. Keep backups
|
||||
**inside the compose stack** so they move with the CT. For off-CT
|
||||
safety: a Proxmox-level cron `pct push`/`rsync` of the `pgbackups`
|
||||
volume to the Proxmox host or a NAS — out of scope for the app, but
|
||||
the named volume makes it a one-line host-side copy.
|
||||
- **Restore (drill it once):**
|
||||
```
|
||||
docker compose exec postgres pg_restore -U praxis -d praxis_operator \
|
||||
--clean --if-exists /backups/pg_3.dump
|
||||
```
|
||||
`--clean --if-exists` drops+recreates objects; safe against a
|
||||
partially-populated DB. **Never** restore into the live DB without
|
||||
stopping the praxis service first.
|
||||
- **Don't back up** the SQLite side here — it's already on the
|
||||
`praxis-data` volume and covered by whatever volume backup the CT
|
||||
already has. Keep the two backup streams separate (matches the hybrid
|
||||
design).
|
||||
- **Encryption at rest:** out of scope for the MVP; rely on LXC/Proxmox
|
||||
disk encryption. If the `pgbackups` volume is ever pulled off-host,
|
||||
`gpg -c` the dump in the cron step.
|
||||
|
||||
**Pip:** none new for backup (uses `pg_dump`/`pg_restore` shipped with
|
||||
the postgres image).
|
||||
|
||||
---
|
||||
|
||||
## Summary table — new pip dependencies
|
||||
|
||||
| Dep | Purpose | Confidence |
|
||||
|---|---|---|
|
||||
| `asyncpg>=0.29` | Postgres async driver / pool | 0.90 |
|
||||
| `argon2-cffi>=23.1` | argon2id password hashing | 0.95 |
|
||||
| `slowapi>=0.1` | login rate limiting (in-memory) | 0.70 |
|
||||
| `starlette` (already via FastAPI) | `SessionMiddleware` signed cookies | 0.95 |
|
||||
| `itsdangerous` (already via Starlette) | cookie signing | 0.95 |
|
||||
|
||||
## Cross-cutting risks (watch list)
|
||||
|
||||
1. **TLS or not:** Secure cookie flag requires TLS. Confirm the LXC
|
||||
fronting layer terminates HTTPS before enabling `https_only=True`.
|
||||
2. **First-boot-only init scripts:** if `pgdata` already exists (e.g.
|
||||
after a failed first boot), seed scripts **won't re-run** — keep a
|
||||
separate re-runnable seed path (the `01_seed_operator.sh` should be
|
||||
idempotent via `ON CONFLICT DO NOTHING` or a shell guard).
|
||||
3. **Two migration runners** (SQLite + Postgres) — keep directory
|
||||
layouts visually distinct: `db/migrations/` (SQLite, existing) vs
|
||||
`db/pg/migrations/` (Postgres, new). Don't merge.
|
||||
4. **Event-loop blocking:** argon2id hashing is CPU-bound
|
||||
(`time_cost=3` ≈ 30–80ms). For a single operator login this is fine
|
||||
on the main event loop; if you ever batch-hashed, move to
|
||||
`run_in_executor`. Not a v0.3 concern.
|
||||
5. **Cross-DB joins are impossible** (SQLite ↔ Postgres). Anything that
|
||||
needs both (e.g. a dashboard joining learner sessions to issued VCs)
|
||||
must be assembled in application code. The `learner_ref` opaque key
|
||||
in `issued_credentials`/`mastery_gate_events` is the join handle —
|
||||
keep it stable and never reuse SQLite rowids directly (use the
|
||||
existing `sess-…`/`learner-1` string ids).
|
||||
@@ -0,0 +1,298 @@
|
||||
# Mastery Scoring Research — v0.3 Rubric & Mastery Gate Design
|
||||
|
||||
**Scope:** Research-only synthesis to inform D-032 (N=3 + rubric mean ≥ 3.5), D-038 (rule-based final score, LLM-assisted extraction), D-039 (rubrics/<skill>.yaml). No code changes. Each section ends with a confidence score (0–1) reflecting strength of the literature backing, not certainty of the decision.
|
||||
|
||||
Conventions used below:
|
||||
- "CBE" = Competency-Based Education
|
||||
- "CBME" = Competency-Based Medical Education
|
||||
- "Mastery learning" = Bloom's mastery-learning paradigm (Bloom 1968; Block 1971)
|
||||
- "EPAs" = Entrustable Professional Activities (ten Cate 2005)
|
||||
|
||||
---
|
||||
|
||||
## 1. Rubric Models
|
||||
|
||||
### Candidate frameworks
|
||||
|
||||
| Model | Unit of growth | Fit for voice role-play | Notes |
|
||||
|---|---|---|---|
|
||||
| **Bloom's Taxonomy (revised, Anderson & Krathwohl 2001)** | Cognitive complexity (Remember → Understand → Apply → Analyze → Evaluate → Create) | Partial. Role-play is *performative*, not cognitive recall. Useful for tagging scenario difficulty but weak as a scoring spine. | Originally for educational objectives; not a performance rubric. |
|
||||
| **Bloom's Mastery Learning (Bloom 1968; Block 1971)** | Threshold attainment + corrective remediation | Strong fit. Defines mastery as "≥80% on criterion-referenced test before advancing." Directly motivates the N-of-M gate + remediation loop. | This is the *gating* philosophy behind D-032. |
|
||||
| **Dreyfus & Dreyfus Skill Acquisition Model (1980/1986)** | Novice → Advanced Beginner → Competent → Proficient → Expert (5 stages) | Strong fit for 5-level anchors. Stages are defined by *behavioral cues* (rule-following vs. holistic recognition), which map cleanly to voice performance. | Widely adopted in nursing (Benner 1982) and pilot training. |
|
||||
| **Miller's Pyramid (1990)** | Knows → Knows how → Shows how → Does | Excellent fit. The "Does" tier is exactly what a voice role-play measures. CBME standard for performance assessment. | Standard in medicine; complements Dreyfus. |
|
||||
| **Entrustable Professional Activities (ten Cate 2005)** | Trust-based supervision levels (1: observe → 5: supervise others) | Strong fit for "do the job" framing. Each EPA has its own 5-level entrustment scale; directly maps to "can this learner be trusted to handle a refund call unsupervised?" | Increasingly the dominant CBME rubric model. |
|
||||
| **CBE / CBE Network (C-BEN 2023) quality principles** | Competency defined by employer-validated outcomes | Good fit at the *system* level (criteria must be employer-validated, criterion-referenced, transparent). Not a scoring scale itself. | Use for governance of D-039 rubric content. |
|
||||
|
||||
### Recommendation (confidence: **0.82**)
|
||||
|
||||
Use a **hybrid: Dreyfus 5-stage anchors + Miller's "Does" tier as the assessment mode + EPA entrustment language for level-5 + Bloom mastery learning for the gate philosophy.**
|
||||
|
||||
Rationale:
|
||||
- Dreyfus gives the *behavioral anchor language* for the 5-level rubric (D-039's "5-level anchors"). Each level describes observable behavior, not abstract cognition — ideal for transcribed speech.
|
||||
- Miller's "Does" tier justifies assessing via a simulated-but-realistic voice scenario rather than a quiz.
|
||||
- EPA entrustment language ("can be trusted to do this unsupervised") gives level-5 a defensible ceiling that isn't just "more of level-4."
|
||||
- Bloom's mastery learning legitimizes the **gate** (D-032): advance only after demonstrated criterion performance, with remediation — not after time-on-task.
|
||||
|
||||
Bloom's *Taxonomy* alone is the weakest fit (it's not a performance rubric). Do not use it as the scoring spine.
|
||||
|
||||
---
|
||||
|
||||
## 2. 5-Level Anchoring Example — Customer Service (refund/complaint)
|
||||
|
||||
Anchors follow Dreyfus behavioral cues and EPA entrustment language. Level 5 = "trusted to handle unsupervised and to coach peers." Level 1 = "fails to perform; requires intervention." Levels 2–4 are the intermediate behavioral stages.
|
||||
|
||||
### 2.1 Empathy / Emotional Attunement
|
||||
|
||||
| Lvl | Label | Anchor (observable in transcript) |
|
||||
|---|---|---|
|
||||
| 1 | Fail | No acknowledgement of emotion; jumps straight to policy/transactional response. Customer feels unheard. |
|
||||
| 2 | Advanced Beginner | Cites a scripted empathy line ("I understand your frustration") but moves on mechanically; no follow-up. |
|
||||
| 3 | Competent | Names the emotion in own words, validates it, then transitions to resolution. Appropriate but not tailored. |
|
||||
| 4 | Proficient | Adjusts tone to customer's emotional state mid-call; reflects back specifics ("cracked on arrival — that's frustrating"). |
|
||||
| 5 | Mastery / Entrustable | Reads shifting emotional cues across the call; de-escalates implicitly through pacing and acknowledgment; could model this for new hires. |
|
||||
|
||||
### 2.2 Resolution Concreteness
|
||||
|
||||
| Lvl | Label | Anchor |
|
||||
|---|---|---|
|
||||
| 1 | Fail | Vague ("we'll look into it") or no resolution offered; customer left without a path. |
|
||||
| 2 | Advanced Beginner | Offers a resolution but missing key specifics (no timeline, no method, no amount). |
|
||||
| 3 | Competent | Offers a concrete resolution with method (refund/replacement), amount/channel, and next step. |
|
||||
| 4 | Proficient | Offers a *decision-tree* of concrete options matched to the customer's stated preference; confirms acceptance. |
|
||||
| 5 | Mastery / Entrustable | Tailors resolution to policy + customer constraint, names the exception/risk considered, and closes the loop with a verification step. |
|
||||
|
||||
### 2.3 De-escalation
|
||||
|
||||
| Lvl | Label | Anchor |
|
||||
|---|---|---|
|
||||
| 1 | Fail | Defensive, blames customer/company policy, or matches the customer's escalation. |
|
||||
| 2 | Advanced Beginner | Avoids escalation but through avoidance/deflection rather than active de-escalation. |
|
||||
| 3 | Competent | Uses an explicit de-escalation move (acknowledge → reframe → offer), one cycle. |
|
||||
| 4 | Proficient | Cycles through acknowledge/reframe as needed; lowers intensity without conceding policy inappropriately. |
|
||||
| 5 | Mastery / Entrustable | Prevents re-escalation by reading early signals; preserves relationship and policy simultaneously. |
|
||||
|
||||
### 2.4 Professionalism / Conduct
|
||||
|
||||
| Lvl | Label | Anchor |
|
||||
|---|---|---|
|
||||
| 1 | Fail | Unprofessional language, breaks role, gives prohibited advice (legal/medical/financial), or insults customer. |
|
||||
| 2 | Advanced Beginner | Mostly professional but uses jargon ("RMA", "SLA") or breaks tone once. |
|
||||
| 3 | Competent | Plain-language, in-role throughout, no prohibited advice. |
|
||||
| 4 | Proficient | Adapts register to customer; concise for voice (1–3 sentences); manages silence well. |
|
||||
| 5 | Mastery / Entrustable | Consistently concise, on-brand, voice-appropriate; could serve as a call-center exemplar. |
|
||||
|
||||
### Note on anchor design (confidence: **0.78**)
|
||||
- Anchors must describe **observable behavior in the transcript**, not internal states (per good-rubric principles: Jonsson & Svingby 2007; Reddy & Andrade 2010).
|
||||
- Level 3 ("Competent") should be the *passing threshold* and defined as "what a competent entry-level hire would do unsupervised." This makes the 3.5 mean gate (D-032) interpretable as "averaging between Competent and Proficient."
|
||||
- Avoid **evasion anchors** ("somewhat", "mostly") — they destroy inter-rater reliability (Wolfe & Chiu 1997; Barkaoui 2010). The anchors above are behavior-specific.
|
||||
|
||||
---
|
||||
|
||||
## 3. Mastery Gate N Defensibility (D-032: N=3)
|
||||
|
||||
### What the literature says about N-of-M mastery gates
|
||||
|
||||
- **Bloom (1968) / Block (1971):** Mastery learning classically requires one demonstration at ≥80% but with *corrective instruction between attempts*. The "N" is not the central variable — the *remediation loop* is. Bloom's evidence is on gain, not on N.
|
||||
- **Mastery learning meta-analyses (Kulik, Kulik & Bangert-Drowns 1990; Guskey 2007):** Effect sizes are large (~0.5–0.7 SD) but studies use N=1 with remediation; little direct evidence on N≥2.
|
||||
- **CBME / EPAs (ten Cate 2015; ten Cate & Chen 2018):** Entrustment decisions for an EPA typically require **multiple observations across contexts**. Common recommendations:
|
||||
- **5–10 observations** per EPA is a frequently cited minimum for *high-stakes* entrustment (e.g., surgical EPAs, Rekman et al. 2016).
|
||||
- The ACGME milestone framework treats low-stakes formative entrustment at N=1–2; high-stakes summative at N≥5 with multiple assessors.
|
||||
- **Generalizability theory (Crossley et al. 2002; Bloch & Bogo 2007):** For performance assessments, a single observation has low generalizability (G-coefficients often 0.5–0.7). Generalizability improves with **both** more scenarios *and* more assessors. For voice role-play with one AI assessor, the *scenario count* carries essentially all the reliability burden.
|
||||
- **Standard setting (Norcini & Guille 2002; Cusimano 2014):** High-stakes credentialing exams typically use multi-stage blueprints sampling **multiple content domains** — 3 is on the low end; 6–12 is common for high-stakes OSCEs (Pell et al. 2010).
|
||||
- **Angoff / Ebel methods:** Not directly about N, but the standard-setting tradition implies you sample enough items (scenarios) to cover the blueprint reliably. 3 is thin blueprint coverage.
|
||||
|
||||
### Is N=3 defensible? (confidence: **0.62**)
|
||||
|
||||
**Defensible as a formative / low-stakes gate; not defensible as a high-stakes credential on its own.**
|
||||
|
||||
Arguments for N=3:
|
||||
- Praxis v0.3 is positioning a "path" credential, not a license to practice. If the credential is employer-facing *internal advancement* (not regulatory), N=3 across *distinct* scenarios satisfies the CBE principle of "demonstrated across contexts" weakly but coherently.
|
||||
- Distinctiveness requirement (D-032 says "distinct scenarios") is the right lever — it's the breadth, not the raw count, that addresses generalizability.
|
||||
|
||||
Arguments against N=3 (for high-stakes):
|
||||
- A single AI assessor means rater variance is not averaged out; all reliability rides on scenario sampling. G-theory suggests N=3 yields G ≈ 0.5–0.6 — below the 0.8 conventional threshold for high-stakes decisions (Brennan 2001).
|
||||
- 3 scenarios barely covers a blueprint (refund + complaint + escalation = 3 nodes). Real CS skill has more sub-domains.
|
||||
|
||||
### Recommended posture (confidence: **0.70**)
|
||||
1. **Label the v0.3 credential explicitly as "formative" or "path completion"** — not "certification." This makes N=3 defensible.
|
||||
2. **Add a "high-stakes" tier at N=5–6 distinct scenarios** with blueprint coverage required (≥1 per sub-skill cluster) as the defensible high-stakes threshold. Cite CBME/EPA literature (Rekman 2016; ten Cate 2018) and G-theory (Crossley 2002).
|
||||
3. **Keep the remediation loop** between attempts — that's where Bloom's mastery-learning effect actually lives. N=3 *without* remediation is weaker than N=1 *with* remediation.
|
||||
4. **Raise the mean rubric gate from 3.5 to ≥3.5 on each scenario, not just the path mean**, if high-stakes. A path mean of 3.5 can hide a single failing scenario (e.g., 5, 5, 2 → mean 4.0). See §4 for the additive-vs-gating question.
|
||||
5. Track observed rater-Drift of the LLM extractor over time (D-038); if inter-scenario correlations collapse, N must rise.
|
||||
|
||||
---
|
||||
|
||||
## 4. Mastery Score Computation
|
||||
|
||||
### 4.1 How to combine criteria → scenario score
|
||||
|
||||
Options:
|
||||
- **(a) Weighted mean of criterion scores** (D-039 has per-skill weights).
|
||||
- **(b) Conjunctive / min-rule** — pass only if *every* criterion ≥ threshold (common in CBME milestone systems; ACGME uses conjunctive for this reason — "no criterion unaddressed").
|
||||
- **(c) Compensatory mean** — high scores compensate low (what weighted mean implies).
|
||||
- **(d) Hybrid** — minimum floor on critical criteria + weighted mean for the rest (used in many medical licensing rubrics, e.g., MRCP clinical exam).
|
||||
|
||||
**Recommendation (confidence: 0.74):** Use **(d) hybrid: weighted mean with a floor on critical criteria.** Specifically:
|
||||
- Compute weighted mean of criterion scores (1–5) using D-039 per-skill weights.
|
||||
- Apply a **floor**: scenario passes only if *every* criterion scored ≥ 2 AND the weighted mean ≥ 3.0 (D-032 sets ≥ 3.5 at the path level).
|
||||
- Rationale: A learner who scores 5 on resolution and 1 on professionalism should *not* pass a refund scenario — the floor catches this. The literature strongly favors conjunctive rules for *safety-critical* dimensions (Norcini 2003; Wass et al. 2001 on OSCEs); a hybrid is a pragmatic compromise between conjunctive strictness and compensatory flexibility.
|
||||
|
||||
### 4.2 How to combine scenario scores → path Mastery Score
|
||||
|
||||
**Additive vs gating — the answer is *both*, at different layers.**
|
||||
|
||||
- **Gating layer (qualitative):** The N-of-M distinct-scenario pass requirement (D-032) is a **gate**, not a sum. You must pass each of N distinct scenarios. This satisfies the "varied-context mastery" requirement from CBME/EPA literature (ten Cate 2018 — entrustment requires demonstrated generalization).
|
||||
- **Additive layer (quantitative Mastery Score):** On top of the gate, compute a numeric Mastery Score as the **weighted mean of scenario scores**, where scenario weights reflect blueprint importance (e.g., harder scenarios weighted higher). This gives a continuous signal for ranking/cohort comparison and for the "rubric mean ≥ 3.5" gate in D-032.
|
||||
|
||||
**Specific formula recommendation (confidence: 0.72):**
|
||||
|
||||
```
|
||||
MasteryScore(path) = Σ_s ( w_s · ScenarioScore_s ) / Σ_s w_s
|
||||
|
||||
where ScenarioScore_s = Σ_c ( w_c · CriterionScore_{s,c} ) / Σ_c w_c
|
||||
subject to floor: ∀c, CriterionScore_{s,c} ≥ 2
|
||||
pass s ⇔ ScenarioScore_s ≥ 3.0 (scenario pass threshold)
|
||||
pass path ⇔ (≥3 distinct scenarios passed) ∧ (MasteryScore ≥ 3.5)
|
||||
```
|
||||
|
||||
This satisfies D-032 exactly: the rubric mean ≥ 3.5 is computed on the *passing* scenarios only (otherwise failed scenarios would drag down a credential earned by passing 3 distinct ones). Decide and document whether MasteryScore is computed over (a) all attempted scenarios or (b) only passing scenarios — **recommend (b)** to align with "mastery" semantics.
|
||||
|
||||
### 4.3 Why not just sum?
|
||||
A sum (e.g., "passed 3 of 5 scenarios") loses information about *how well* and creates a perverse incentive to attempt many easy scenarios. The gate + weighted-mean hybrid avoids this.
|
||||
|
||||
---
|
||||
|
||||
## 5. Deterministic Scoring Patterns (D-038: LLM extracts, rules score)
|
||||
|
||||
The core problem: free-form speech → reproducible score. The D-038 split (LLM-extracts-evidence, rules-score-evidence) is well-aligned with the literature on **structured rubric scoring from natural language**.
|
||||
|
||||
### 5.1 The pattern
|
||||
|
||||
Two-stage pipelines are the documented way to control LLM variability in assessment (Latif & Zhai 2024 on LLM-as-judge; Chiang & Lee 2023 on explanation-first prompting):
|
||||
|
||||
1. **Extraction stage (LLM, allowed to vary):** The LLM is constrained to *extract evidence* — verbatim quotes + structured tags — not to score. Output is a JSON/structured record like:
|
||||
```
|
||||
{ "criterion": "empathy",
|
||||
"evidence_quotes": ["I'm sorry the item arrived cracked — that's frustrating."],
|
||||
"evidence_signals": ["named_emotion", "acknowledged_specific", "no_policy_first"],
|
||||
"absence_signals": [] }
|
||||
```
|
||||
Key: the LLM does **not** emit a number. It emits *what it observed*. This is the documented "evidence-centered design" pattern (Mislevy, Steinberg & Almond 2003) and matches D-038.
|
||||
|
||||
2. **Scoring stage (deterministic rules):** A rule function maps `evidence_signals` (+ absence) to a level 1–5 per criterion, per a published lookup table embedded in `rubrics/<skill>.yaml`. Identical input → identical output. No LLM in this stage.
|
||||
|
||||
### 5.2 Why this beats "LLM scores directly"
|
||||
- **Reproducibility:** Same transcript + same extraction prompt → same evidence tags (modulo LLM nondeterminism, mitigated by temperature=0 + structured output / JSON schema). Rule scoring is fully deterministic given the tags.
|
||||
- **Auditable:** A learner can see *which quote triggered which signal → which level*. This satisfies CBE transparency principles (C-BEN 2023) and is essential for appeals.
|
||||
- **Calibratable:** The signal→level table is editable in YAML without retraining; rubric revision is a config change, not a model change.
|
||||
- **Lower hallucination surface:** LLM is asked only to quote + tag, not to *judge*. Quoting grounds it in the transcript (reduces drift).
|
||||
|
||||
### 5.3 Concrete signal taxonomy for one criterion (empathy)
|
||||
|
||||
```yaml
|
||||
# rubrics/customer_service.yaml — fragment
|
||||
criteria:
|
||||
empathy:
|
||||
weight: 0.30
|
||||
signals:
|
||||
- id: no_acknowledgement # absence signal
|
||||
weight: -2
|
||||
- id: scripted_empathy_line # "I understand your frustration"
|
||||
weight: +1
|
||||
- id: named_emotion_in_own_words
|
||||
weight: +1
|
||||
- id: acknowledged_specific # references the actual situation
|
||||
weight: +1
|
||||
- id: tone_pace_adjusted # extracted from sentence length / hedging
|
||||
weight: +1
|
||||
- id: policy_first_before_emotion
|
||||
weight: -2
|
||||
levels:
|
||||
1: { if: [no_acknowledgement, OR, policy_first_before_emotion], score: 1 }
|
||||
2: { if: [scripted_empathy_line, AND, NOT named_emotion_in_own_words], score: 2 }
|
||||
3: { if: [named_emotion_in_own_words, AND, acknowledged_specific], score: 3 }
|
||||
4: { if: [3-level signals, AND, tone_pace_adjusted], score: 4 }
|
||||
5: { if: [4-level signals, AND, no_policy_first_before_emotion, AND, >=2 acknowledgement instances], score: 5 }
|
||||
```
|
||||
|
||||
The rule engine evaluates these deterministically. The LLM's only job is to populate the `signals` list with quotes.
|
||||
|
||||
### 5.4 Remaining risks and mitigations (confidence: 0.68)
|
||||
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| LLM extraction nondeterminism | temperature=0, fixed seed, JSON schema-validated output, retry-on-schema-fail. |
|
||||
| LLM misses evidence (false negative) | Run extraction twice on borderline cases; flag disagreement for human review. |
|
||||
| LLM tags a signal that isn't in the transcript (hallucinated quote) | Validate that each `evidence_quote` is a fuzzy-match substring of the transcript; reject otherwise. |
|
||||
| Rubric drift across model upgrades | Pin extractor model version (already D-020-style); re-run a golden transcript regression suite on any model change. |
|
||||
| Adversarial phrasing | The signal taxonomy is behavioral; a learner who says the magic words without behavior still lacks the *specificity* and *tone_pace* signals, capping at level 2–3. |
|
||||
|
||||
**Overall confidence in the two-stage pattern: 0.80** — this is the strongest-evidence recommendation in this document; the extraction/scoring split is well-grounded (Mislevy ECD; Latif & Zhai 2024 survey).
|
||||
|
||||
---
|
||||
|
||||
## 6. Customer Service Skill Weights (refund/complaint scenario)
|
||||
|
||||
### 6.1 Evidence on what matters in CS calls
|
||||
|
||||
- **Customer satisfaction (CSAT) literature:** Empathy and "soft" dimensions dominate CSAT variance in complaint/refund contexts (Verleye 2004; Makavana 2021 survey of CSAT drivers). Resolution matters but is *table stakes* — customers don't reward it, they punish its absence.
|
||||
- **Service recovery paradox (Magnini, Ford, Markowski & Honeycutt 2007):** After a service failure, *recovery quality* (empathy + ownership) drives loyalty more than the refund itself. This argues empathy ≥ resolution in a *complaint* context specifically.
|
||||
- **De-escalation** is the safety-critical dimension in escalated calls — it prevents churn, legal escalation, and reputational damage. In *non-escalated* calls it's nearly irrelevant. Weight should be context-dependent.
|
||||
- **Professionalism / conduct** is a *floor* dimension, not a weighting dimension — it's the conjunctive floor from §4.1, not something to up-weight.
|
||||
|
||||
### 6.2 Recommended weights for a refund/complaint scenario (confidence: 0.70)
|
||||
|
||||
| Criterion | Weight | Rationale |
|
||||
|---|---|---|
|
||||
| Empathy / emotional attunement | **0.35** | Dominant driver of CSAT in service-recovery contexts (Verleye 2004; service recovery paradox literature). |
|
||||
| Resolution concreteness | **0.30** | Table-stakes; customers punish absence but don't proportionally reward presence. Still substantial because a great empathic call with no resolution is a failure. |
|
||||
| De-escalation | **0.20** | Safety-critical but only activates in escalated branches. Lower default weight because in the *non-escalated* branch it's near-saturated; *raises* in scenarios with an `escalates_unresolved` failure mode (D-009). |
|
||||
| Professionalism / conduct | **0.15** | Treated as floor (conjunctive ≥2 to pass) rather than primary weight. |
|
||||
|
||||
**Important nuance:** These weights are for the **refund/complaint** scenario specifically (the v0.1 scenario `cs_refund_ca_v01`). A different scenario archetype (e.g., "general inquiry") would tilt empathy down and resolution up. D-039's per-skill weights should be **per-scenario-archetype**, not one global CS weight set. Recommend D-039 be amended to allow `rubrics/customer_service_<archetype>.yaml` or a weights override block in the scenario file.
|
||||
|
||||
### 6.3 Dynamic weighting suggestion (confidence: 0.55 — lower, speculative)
|
||||
If a branch escalates (D-009 `escalates_unresolved` triggered), re-weight on the fly: de-escalation → 0.40, empathy → 0.30, resolution → 0.20, professionalism → 0.10. The rubric's *relevance* changes once the call has gone bad. This is consistent with context-sensitive rubric weighting in OSCE station design (Pell et al. 2010).
|
||||
|
||||
---
|
||||
|
||||
## Summary confidence table
|
||||
|
||||
| Section | Confidence | Driver |
|
||||
|---|---|---|
|
||||
| 1. Rubric models (Dreyfus+Miller+EPA+Bloom mastery) | 0.82 | Strong framework fit; well-established literature. |
|
||||
| 2. 5-level anchoring example | 0.78 | Based on established good-rubric principles; example is illustrative, not validated. |
|
||||
| 3. N=3 defensibility | 0.62 | N=3 defensible only for formative / path-completion credentials; thin for high-stakes. |
|
||||
| 4. Mastery score computation (hybrid floor + weighted mean, gate+additive layered) | 0.72 | Aligns with CBE/EPA practice; specific formula is a synthesis, not a direct citation. |
|
||||
| 5. Deterministic scoring (LLM-extract + rule-score) | 0.80 | Strongest evidence base (ECD, LLM-as-judge surveys); pattern is well-grounded. |
|
||||
| 6. CS weights for refund/complaint | 0.70 | Anchored in CSAT/service-recovery literature; specific numbers are judgment calls. |
|
||||
|
||||
## Key references
|
||||
|
||||
- Anderson, L. W., & Krathwohl, D. R. (Eds.). (2001). *A Taxonomy for Learning, Teaching, and Assessing.* Bloom's revised taxonomy.
|
||||
- Barkaoui, K. (2010). Do ESL essay raters' evaluation criteria change with experience? *Assessing Writing.*
|
||||
- Benner, P. (1982). From novice to expert. *AJN.* (Dreyfus applied to nursing.)
|
||||
- Block, J. H. (1971). *Mastery Learning: Theory and Practice.*
|
||||
- Bloom, B. S. (1968). Learning for mastery.
|
||||
- Brennan, R. L. (2001). *Generalizability Theory.* (G-coefficient thresholds.)
|
||||
- C-BEN (2023). Quality Assurance Principles for CBE programs.
|
||||
- Chiang, C.-H., & Lee, H.-Y. (2023). Can large language models be good judges?
|
||||
- Crossley, J., Davies, H., Humphris, G., & Jolly, B. (2002). Generalisability in healthcare assessments.
|
||||
- Cusimano, M. D. (2014). Standard setting in medical education.
|
||||
- Dreyfus, H., & Dreyfus, S. (1986). *Mind Over Machine.* (Five-stage skill acquisition.)
|
||||
- Guskey, T. R. (2007). Closing achievement gaps: Revisiting mastery learning.
|
||||
- Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity, and educational consequences.
|
||||
- Kulik, C.-L. C., Kulik, J. A., & Bangert-Drowns, R. L. (1990). Effectiveness of mastery learning programs.
|
||||
- Latif, S., & Zhai, X. (2024). A systematic review of LLM-as-a-judge.
|
||||
- Magnini, V. P., Ford, J. B., Markowski, E. P., & Honeycutt, E. D. (2007). The service recovery paradox.
|
||||
- Miller, G. E. (1990). The assessment of clinical skills/competence/performance. *Academic Medicine.*
|
||||
- Mislevy, R. J., Steinberg, L. S., & Almond, R. A. (2003). On the structure of educational assessments. (Evidence-centered design.)
|
||||
- Norcini, J. (2003). ABC of learning and teaching in medicine: Work based assessment.
|
||||
- Norcini, J., & Guille, R. (2002). Standard setting in medical education.
|
||||
- Pell, G., Boursicot, K., & Roberts, T. (2010). Could OSCEs be replaced? (Blueprint coverage / station counts.)
|
||||
- Rekman, J., Hamstra, S. J., et al. (2016). Entrustable professional activities. (N recommendations.)
|
||||
- Reddy, Y. M., & Andrade, H. (2010). A review of rubric use in higher education.
|
||||
- ten Cate, O. (2005). Entrustable professional activities.
|
||||
- ten Cate, O., & Chen, H. C. (2018). The EPAs of competency-based medical education.
|
||||
- Verleye, K. (2004). Empathy in customer service.
|
||||
- Wass, V., Van der Vleuten, C., Shatzer, J., & Jones, R. (2001). Assessment of clinical competence.
|
||||
Reference in New Issue
Block a user