coreci/praxis
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 15 Wiki Activity

docs(milestone): merge phase/00 pre-execution → milestone/v0.3-mastery-scoring

Browse Source 
Phase 0 complete. v0.3 mastery-scoring planning artifacts shipped.
Pipeline: SPECIFY → CLARIFY → RESEARCH → PLAN → GRILL.
13 REQ-IDs active (7 functional + 6 NFR); 8 deferred to v0.4 (operator tier).
Decisions D-031..D-049 (19 total, all >=0.70 confidence).
4 MUST grill conditions resolved, 5 FIX tracked.

---ci---
project: praxis
phase: 0
milestone: v0.3
status: complete
requirements:
  covered: [REQ-MAST-01, REQ-MAST-02, REQ-MAST-03, REQ-SCEN-02, REQ-SCEN-03, REQ-SCEN-04, REQ-PATH-02]
  partial: []
---/ci---
This commit is contained in:
Praxis CI
2026-08-03 19:58:38 +00:00
parent bea2af13d4
commit dc673e5e3d
14 changed files with 2906 additions and 904 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/RESEARCH-operator-postgres-auth.md
+475
View File
@@ -0,0 +1,475 @@
# RESEARCH: Operator Tier — Postgres-in-LXC + Auth for v0.3
**Scope:** Research only. No code changes. Grounded in the current Praxis repo
(`docker-compose.yml` single `praxis` service; `db/store.py` aiosqlite
`PraxisStore`; `db/migrate.py` ordered `.sql` migrations; SQLite schema at
`db/schema.sql`).
**Decisions honored:** D-007 (SQLite learner, preserved), D-031 (hybrid:
SQLite for learner, Postgres for operator), D-040 (Postgres = second
docker-compose service in the existing LXC CT), D-041 (session-cookie auth,
argon2id, single operator role, rate-limited).
**Confidence scores** are 0–1 (1 = well-established practice / low risk).
---
## 1. Docker-Compose Shape *(confidence: 0.90)*
Add a `postgres` service alongside the existing `praxis` service. Key
best-practices for a second service in an already-running LXC CT:
- **Image:** `postgres:16-slim` (Debian-slim base, glibc — matches the
praxis Dockerfile rationale; avoids Alpine musl locale issues with
`pg_*` clients).
- **Persistence:** named volume `pgdata` (driver: local). Never bind-mount
`/var/lib/postgresql/data` to the CT filesystem — Postgres requires
`chown 999` and a specific directory layout; named volumes handle this.
- **Network isolation:** declare an explicit internal compose network and
attach **only** `praxis` and `postgres` to it. Do **not** publish
`5432` via `ports:`. The `praxis` service keeps its published `8789`.
- `internal: true` on the network blocks egress to the host bridge, but
note: with `internal: true` the postgres container cannot reach the
internet (fine — it doesn't need to). If you later want outbound
backups via network, drop `internal: true` and instead rely on
*not* publishing the port. The simpler, robust choice for a pilot is:
explicit named network, no `ports:` on postgres, no `internal: true`.
- **Healthcheck:** `pg_isready -U praxis -d praxis` every 10s, 5 retries,
5s timeout. `depends_on: { postgres: { condition: service_healthy } }`
on the `praxis` service so the app waits for accept-connections, not
just container start.
- **Init scripts:** mount `./db/pg/init/*.sql` (or `.sh`) at
`/docker-entrypoint-initdb.d/`. These run **only on first boot** (empty
`pgdata`). Use them for: role/db creation, schema bootstrap, and
idempotent seed. For *versioned* schema changes use a migration runner
(see §6) — init scripts are one-shot.
- **Env:** `POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_DB` from the
existing `/etc/praxis/server.env` (do **not** commit secrets to the
compose file). Add `PGDATA=/var/lib/postgresql/data/pgdata` to pin the
subdirectory (survives image upgrades).
- **Restart:** `restart: unless-stopped` (matches praxis).
- **Resources:** for a pilot on a small LXC CT, set a mem limit
(`deploy.resources.limits.memory: 512m`) and rely on Postgres default
`shared_buffers`. Tune later.
**Sketch (shape only, not for commit):**
```yaml
services:
praxis:
# ... existing v0.2 fields unchanged ...
depends_on:
postgres:
condition: service_healthy
networks: [praxis-net]
postgres:
image: postgres:16-slim
restart: unless-stopped
environment:
POSTGRES_USER: ${PG_USER}
POSTGRES_PASSWORD: ${PG_PASSWORD}
POSTGRES_DB: ${PG_DB:-praxis_operator}
PGDATA: /var/lib/postgresql/data/pgdata
env_file:
- path: /etc/praxis/server.env
required: false
volumes:
- pgdata:/var/lib/postgresql/data
- ./db/pg/init:/docker-entrypoint-initdb.d:ro
- pgbackups:/backups
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${PG_USER:-praxis} -d ${PG_DB:-praxis_operator}"]
interval: 10s
timeout: 5s
retries: 5
networks: [praxis-net]
# NOTE: no `ports:` — not exposed to the LXC host bridge.
volumes:
praxis-data:
driver: local
pgdata:
driver: local
pgbackups:
driver: local
networks:
praxis-net:
driver: bridge
```
**Risk callouts:**
- If `praxis` currently has no explicit network, compose assigns the
default bridge; adding an explicit network means the *existing*
`praxis` service gets recreated on `up`. Plan a brief downtime window
(see §6).
- `pg_isready` returns healthy before the DB is fully ready for migration
load; `depends_on: service_healthy` is necessary but not sufficient —
the app must still retry the first migration attempt.
---
## 2. Connection Management *(confidence: 0.85)*
Two async DB drivers in one process: **aiosqlite** (already a dep) for the
learner store, **asyncpg** for the operator store.
- **Pools are independent and must not be shared.** asyncpg uses a
`asyncpg.create_pool(...)` (sized pool, real connections). aiosqlite
opens a fresh connection per `async with aiosqlite.connect(...)` (the
current `PraxisStore._connect` pattern). They have nothing in common —
different backends, different lifecycles. **Do not** wrap them in a
single shared `AsyncSession` object; SQLAlchemy's async session is an
option *only if* you adopt SQLAlchemy for both — that's a larger
refactor and not warranted for v0.3.
- **Pool sizing (avoid exhaustion):**
- asyncpg pool: `min_size=2, max_size=10` for a pilot single-instance.
Operator endpoints are low-frequency (cohort dashboard, VC issuance).
- aiosqlite: no pool; the current pattern opens/closes per call. SQLite
is single-writer; keep `WAL` mode and short transactions. This is
already fine for one learner.
- Total concurrent DB connections ≈ asyncpg(10) + aiosqlite(1-2). On a
small CT this is trivial. Exhaustion risk is essentially zero at
pilot scale; revisit if operator endpoints are hit by N concurrent
cohort users.
- **Lifecycle:** create the asyncpg pool once at FastAPI startup
(`lifespan` context manager), close on shutdown. Store on
`app.state.pg_pool`. The `PraxisStore` keeps its current per-call
connect pattern (no change to D-007 code path).
- **Transaction boundaries:** asyncpg use `pool.acquire()` +
`conn.transaction()` for multi-statement writes; aiosqlite unchanged.
- **Config:** `PG_DSN` env var, e.g.
`postgresql://praxis:***@postgres:5432/praxis_operator` (host =
service name on `praxis-net`).
- **Statement timeout:** set `command_timeout=10` on the asyncpg pool to
prevent a slow operator query from blocking the event loop.
**Pip:** `asyncpg>=0.29` (new dep). `aiosqlite>=0.20` already present.
---
## 3. Auth Stack *(confidence: 0.90 for the stack; 0.70 for rate-limit choice)*
D-041 spec: session-cookie, argon2id, single operator role, rate-limited.
### 3a. Session cookie
- **`starlette` `SessionMiddleware`** (FastAPI bundles Starlette). Uses
`itsdangerous` to sign the cookie — no server-side session store
needed (stateless, fits single-instance LXC). Data lives in the cookie
itself, signed with `SECRET_KEY`.
- **Settings:**
- `secret_key`: from env, ≥32 bytes random. **Rotate** by changing the
key (invalidates all sessions — acceptable for a pilot).
- `session_cookie`: `"praxis_op"` (distinct from any future learner
cookie name).
- `max_age`: `28800` (8h, per D-041).
- `path`: `/` (or scope to `/op` if operator routes live under a
prefix — cleaner).
- `https_only`: `True` (Secure flag). **Requires TLS** — the LXC
deployment must terminate TLS (reverse proxy / Caddy / Proxmox
level). If running plain HTTP on the LAN for the pilot, set to
`False` *temporarily* and document the risk; never ship False.
- `httponly`: `True` (the middleware sets this by default; verify).
- `samesite`: `"strict"` (D-041). CSRF defense-in-depth; with Strict,
no credential is sent on cross-site navigations.
- **Cookie contents:** store `{operator_id: str, issued_at: epoch}`.
**Never** store the password hash or any PII. Roles aren't needed in
the cookie yet (single role — see §4).
### 3b. Password hashing — argon2id
- **`argon2-cffi`** (`PasswordHasher` default is argon2id, RFC 9106).
Pip: `argon2-cffi>=23.1`.
- On login: `ph.verify(stored_hash, password)` → on success,
`ph.check_needs_rehash(stored_hash)` → rehash if params bumped.
- Params: keep `PasswordHasher()` defaults for v0.3
(`time_cost=3, memory_cost=64MiB, parallelism=4` — reasonable on a
small CT; benchmark and tune if login latency > 1s).
- Store the hash as `TEXT` in `operators.password_hash`.
### 3c. Rate limiting
Two options:
1. **`slowapi`** (pip `slowapi>=0.1`) — the idiomatic FastAPI choice.
Decorator/IP-based limiter. Default in-memory backend is fine for
single-instance. **Confidence 0.70** — it works, but it's a young lib
and the in-memory backend is per-process (breaks if you ever scale to
>1 praxis process; not a v0.3 concern).
2. **In-memory counter** (a simple `dict[remote_ip, (count, window_start)]`
in a small dependency) — zero deps, trivially auditable. For a single
operator login endpoint this is enough. **Confidence 0.80** for the
pilot specifically.
**Recommendation:** start with `slowapi` on the login route only
(`@limiter.limit("5/minute")`), in-memory backend. Migrate to a Redis
backend only if/when you go multi-instance. Threshold: 5 failed
attempts/minute/IP → 429 + exponential backoff marker.
**Pip additions:** `argon2-cffi>=23.1`, `slowapi>=0.1`. (`starlette` and
`itsdangerous` come with FastAPI.)
---
## 4. Auth Dependency Pattern *(confidence: 0.90)*
Single-role v0.3 → **no RBAC framework needed.** A single FastAPI
`Depends` that resolves the operator from the signed session is the
minimal secure shape.
Concept (not committed code):
```python
# pseudo — shape only
async def current_operator(request: Request) -> Operator:
sess = request.session # populated by SessionMiddleware
op_id = sess.get("operator_id")
if not op_id:
raise HTTPException(401, "not authenticated")
op = await pg_store.get_operator(op_id)
if not op or not op.is_active:
# invalidate the cookie
request.session.clear()
raise HTTPException(401, "operator not found / disabled")
return op
```
- Apply via `Depends(current_operator)` on every operator-tier router.
Group operator routes under an `APIRouter(prefix="/op")` and attach
the dependency at the router level
(`dependencies=[Depends(current_operator)]`) — one declaration, not
per-endpoint.
- Login/logout are **outside** the protected router (login is rate-
limited, not auth-gated).
- **CSRF:** with `SameSite=Strict` + `httponly` cookies, CSRF surface is
minimal for state-changing requests. If any operator endpoint accepts
`Content-Type: application/x-www-form-urlencoded`/`multipart` (form
posts), add a double-submit token or require `Content-Type:
application/json` only (the latter is the cheaper defense — JSON
bodies are not auto-sent by browsers across origins).
### When to migrate to RBAC
Migrate when **any** of these become true:
- A second role appears (admin, auditor, reviewer) — i.e. v0.4+ if the
pilot expands.
- Permissions diverge *within* a role (e.g. some operators can issue
VCs, others can only view cohorts).
- You need row-level visibility rules (operator A sees only their
cohort).
At that point the cheapest upgrade is: add a `role` column to
`operators`, split `current_operator` into `current_operator` (any
authenticated) + `require_role("admin")` (a parametrized dependency
checking `op.role`). Reach for a full RBAC lib (`casbin`,
`fastapi-permissions`) only when the role matrix exceeds ~3 roles × ~5
permissions. **Don't pre-build it.**
---
## 5. Postgres Schema *(confidence: 0.80)*
Operator-tier tables. Types chosen for Postgres 16 specifically
(`TIMESTAMPTZ`, `BIGSERIAL`, `GENERIC` via `JSONB`).
### `operators`
```
id UUID PRIMARY KEY DEFAULT gen_random_uuid()
username TEXT NOT NULL UNIQUE
password_hash TEXT NOT NULL -- argon2id
display_name TEXT NOT NULL
role TEXT NOT NULL DEFAULT 'operator' -- reserved for §4 migration
is_active BOOLEAN NOT NULL DEFAULT TRUE
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
last_login_at TIMESTAMPTZ
```
- Index: unique on `username` (covered by constraint). No extra index
needed at single-operator scale.
- Requires `pgcrypto` extension **or** Postgres 13+ (where
`gen_random_uuid()` is built-in via `pgcrypto` shipped default —
actually: `gen_random_uuid()` is built into core as of PG 13). So no
extension needed on PG16. ✓
### `issued_credentials`
```
id BIGSERIAL PRIMARY KEY
operator_id UUID NOT NULL REFERENCES operators(id)
learner_ref TEXT, -- opaque ref into SQLite side (no FK cross-DB)
vc_type TEXT NOT NULL -- 'mastery' | 'completion' | ...
payload_jsonb JSONB NOT NULL -- the W3C VC document (signed elsewhere)
issued_at TIMESTAMPTZ NOT NULL DEFAULT now()
revoked_at TIMESTAMPTZ
```
- Indices:
- `issued_credentials(operator_id, issued_at DESC)` — operator's
issuance log.
- `issued_credentials(learner_ref)` — lookup by learner (k-anon
aggregate joins).
- `issued_credentials(vc_type)` if filtering by type is a dashboard
query.
### `mastery_gate_events`
```
id BIGSERIAL PRIMARY KEY
learner_ref TEXT NOT NULL
scenario_id TEXT NOT NULL
path_id TEXT NOT NULL -- learning path
gate_outcome TEXT NOT NULL -- 'pass' | 'fail' | 'retry'
recorded_at TIMESTAMPTZ NOT NULL DEFAULT now()
source TEXT NOT NULL DEFAULT 'sync' -- 'sync' from SQLite learner store
```
- Indices:
- `(learner_ref, recorded_at DESC)` — per-learner timeline.
- `(path_id, recorded_at)` — feeds the cohort aggregate.
### `cohort_aggregates` — k-anonymized
Model as **pre-materialized rows** partitioned by `(path_id, week)` with
a minimum bin size enforced at write time (k≥K, e.g. K=5). A 7-day
window is a rolling construct over the weekly partitions.
```
path_id TEXT NOT NULL
week_start DATE NOT NULL -- ISO week Monday
bin_count INTEGER NOT NULL -- learners in this bin
k_anon_pass INTEGER NOT NULL -- pass count, suppressed if < K
k_anon_fail INTEGER NOT NULL -- fail count, suppressed if < K
median_attempts INTEGER
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
PRIMARY KEY (path_id, week_start)
```
- **k-anon rule:** when materializing, if `bin_count < K` emit
`bin_count = <K-masked>` and null-out the count columns (or clamp
them to K). Enforce in the aggregation job, **not** in a SQL view, so
the suppression is auditable at write time.
- **7-day window:** compute on read as a window function over the last
≤2 weekly partitions, or maintain a parallel rolling table. For a
pilot, compute on read:
`SUM(k_anon_pass) ... WHERE week_start >= now()::date - interval '7 days'`.
- Indices: PK covers `(path_id, week_start)`. Add a secondary
`(week_start DESC)` only if you query "all paths for the latest week"
frequently.
**General indices summary:** 4 indices beyond PKs/constraints for v0.3
— keep it lean; add per slow-query evidence.
---
## 6. Migration Strategy *(confidence: 0.85)*
Goal: add Postgres to the **running** v0.2 LXC CT without breaking the
learner service.
### Steps (ordered, low-risk)
1. **Prepare on a staging CT first** (clone the production LXC CT in
Proxmox). Never test the migration path on the live CT.
2. **Add the `postgres` service + `praxis-net` + volumes** to
`docker-compose.yml`. The `praxis` service gains
`depends_on: postgres (service_healthy)` and joins `praxis-net`.
3. **Add init scripts** under `db/pg/init/`:
- `00_create_schema.sql` — the four tables from §5.
- `01_seed_operator.sh` — creates the initial operator with an
argon2id hash (run from env-supplied temp password; force password
change on first login).
These run **only on first boot** of an empty `pgdata` volume.
4. **Add the asyncpg pool + operator store + auth wiring** to the praxis
image (new code paths, new deps in `pyproject.toml`). Learner paths
(`db/store.py`, `db/migrate.py`) **unchanged** — D-007 preserved.
5. **Build the new image** (`docker compose build praxis`) — does not
touch the running container.
6. **Controlled cutover:**
- `docker compose up -d postgres` → wait for healthy.
- `docker compose up -d praxis` → recreate the praxis container with
the new image. Expect ~5–15s of downtime (the learner voice loop
is not HA anyway). The SQLite volume (`praxis-data`) is untouched,
so learner state is preserved across the recreate.
7. **Smoke tests:** `/health`, learner voice loop, operator login, one
cohort-dashboard read.
8. **Rollback plan:** if operator endpoints misbehave, revert the
praxis image tag and `docker compose up -d praxis` again — Postgres
stays up but unused. Learner path is independent, so a bad operator
rollout does **not** regress v0.2 learner behavior. This is the
core safety property of the hybrid (D-031) design.
### Versioned migrations beyond first boot
The SQLite side already has `db/migrate.py` (ordered `.sql`, `_migrations`
table). For Postgres, two options:
- **(a) Reuse the pattern:** a `pg_migrate.py` mirroring the SQLite
runner, against a `_pg_migrations` table. Lowest cognitive load —
same mental model, same directory convention (`db/pg/migrations/`).
- **(b) Adopt `yoyo-migrations` or `alembic`:** more machinery, not
warranted at 4 tables.
**Recommendation (a):** mirror the existing runner. Run on praxis
startup (after the pool is up), idempotent. **Confidence 0.80** on the
pattern; it's exactly what v0.2 already does for SQLite.
---
## 7. Backup *(confidence: 0.85)*
Minimum viable backup for a pilot operator Postgres in LXC:
- **Method:** `pg_dump -Fc` (custom compressed format) → file in the
`pgbackups` volume. `-Fc` gives you selective restore and parallel
restore later.
- **Frequency:** daily is enough for a pilot. A cron job *inside the
postgres container* (or a sidecar) runs:
```
pg_dump -U praxis -Fc praxis_operator > /backups/pg_$(date +%u).dump
```
Using `%u` (day-of-week 1–7) gives a rolling 7-file retention with
zero cleanup logic.
- **Where:** `/backups` is the `pgbackups` named volume. Keep backups
**inside the compose stack** so they move with the CT. For off-CT
safety: a Proxmox-level cron `pct push`/`rsync` of the `pgbackups`
volume to the Proxmox host or a NAS — out of scope for the app, but
the named volume makes it a one-line host-side copy.
- **Restore (drill it once):**
```
docker compose exec postgres pg_restore -U praxis -d praxis_operator \
--clean --if-exists /backups/pg_3.dump
```
`--clean --if-exists` drops+recreates objects; safe against a
partially-populated DB. **Never** restore into the live DB without
stopping the praxis service first.
- **Don't back up** the SQLite side here — it's already on the
`praxis-data` volume and covered by whatever volume backup the CT
already has. Keep the two backup streams separate (matches the hybrid
design).
- **Encryption at rest:** out of scope for the MVP; rely on LXC/Proxmox
disk encryption. If the `pgbackups` volume is ever pulled off-host,
`gpg -c` the dump in the cron step.
**Pip:** none new for backup (uses `pg_dump`/`pg_restore` shipped with
the postgres image).
---
## Summary table — new pip dependencies
| Dep | Purpose | Confidence |
|---|---|---|
| `asyncpg>=0.29` | Postgres async driver / pool | 0.90 |
| `argon2-cffi>=23.1` | argon2id password hashing | 0.95 |
| `slowapi>=0.1` | login rate limiting (in-memory) | 0.70 |
| `starlette` (already via FastAPI) | `SessionMiddleware` signed cookies | 0.95 |
| `itsdangerous` (already via Starlette) | cookie signing | 0.95 |
## Cross-cutting risks (watch list)
1. **TLS or not:** Secure cookie flag requires TLS. Confirm the LXC
fronting layer terminates HTTPS before enabling `https_only=True`.
2. **First-boot-only init scripts:** if `pgdata` already exists (e.g.
after a failed first boot), seed scripts **won't re-run** — keep a
separate re-runnable seed path (the `01_seed_operator.sh` should be
idempotent via `ON CONFLICT DO NOTHING` or a shell guard).
3. **Two migration runners** (SQLite + Postgres) — keep directory
layouts visually distinct: `db/migrations/` (SQLite, existing) vs
`db/pg/migrations/` (Postgres, new). Don't merge.
4. **Event-loop blocking:** argon2id hashing is CPU-bound
(`time_cost=3` ≈ 30–80ms). For a single operator login this is fine
on the main event loop; if you ever batch-hashed, move to
`run_in_executor`. Not a v0.3 concern.
5. **Cross-DB joins are impossible** (SQLite ↔ Postgres). Anything that
needs both (e.g. a dashboard joining learner sessions to issued VCs)
must be assembled in application code. The `learner_ref` opaque key
in `issued_credentials`/`mastery_gate_events` is the join handle —
keep it stable and never reuse SQLite rowids directly (use the
existing `sess-…`/`learner-1` string ids).
docs/mastery-scoring-research.md
+298
View File
@@ -0,0 +1,298 @@
# Mastery Scoring Research — v0.3 Rubric & Mastery Gate Design
**Scope:** Research-only synthesis to inform D-032 (N=3 + rubric mean ≥ 3.5), D-038 (rule-based final score, LLM-assisted extraction), D-039 (rubrics/<skill>.yaml). No code changes. Each section ends with a confidence score (0–1) reflecting strength of the literature backing, not certainty of the decision.
Conventions used below:
- "CBE" = Competency-Based Education
- "CBME" = Competency-Based Medical Education
- "Mastery learning" = Bloom's mastery-learning paradigm (Bloom 1968; Block 1971)
- "EPAs" = Entrustable Professional Activities (ten Cate 2005)
---
## 1. Rubric Models
### Candidate frameworks
| Model | Unit of growth | Fit for voice role-play | Notes |
|---|---|---|---|
| **Bloom's Taxonomy (revised, Anderson & Krathwohl 2001)** | Cognitive complexity (Remember → Understand → Apply → Analyze → Evaluate → Create) | Partial. Role-play is *performative*, not cognitive recall. Useful for tagging scenario difficulty but weak as a scoring spine. | Originally for educational objectives; not a performance rubric. |
| **Bloom's Mastery Learning (Bloom 1968; Block 1971)** | Threshold attainment + corrective remediation | Strong fit. Defines mastery as "≥80% on criterion-referenced test before advancing." Directly motivates the N-of-M gate + remediation loop. | This is the *gating* philosophy behind D-032. |
| **Dreyfus & Dreyfus Skill Acquisition Model (1980/1986)** | Novice → Advanced Beginner → Competent → Proficient → Expert (5 stages) | Strong fit for 5-level anchors. Stages are defined by *behavioral cues* (rule-following vs. holistic recognition), which map cleanly to voice performance. | Widely adopted in nursing (Benner 1982) and pilot training. |
| **Miller's Pyramid (1990)** | Knows → Knows how → Shows how → Does | Excellent fit. The "Does" tier is exactly what a voice role-play measures. CBME standard for performance assessment. | Standard in medicine; complements Dreyfus. |
| **Entrustable Professional Activities (ten Cate 2005)** | Trust-based supervision levels (1: observe → 5: supervise others) | Strong fit for "do the job" framing. Each EPA has its own 5-level entrustment scale; directly maps to "can this learner be trusted to handle a refund call unsupervised?" | Increasingly the dominant CBME rubric model. |
| **CBE / CBE Network (C-BEN 2023) quality principles** | Competency defined by employer-validated outcomes | Good fit at the *system* level (criteria must be employer-validated, criterion-referenced, transparent). Not a scoring scale itself. | Use for governance of D-039 rubric content. |
### Recommendation (confidence: **0.82**)
Use a **hybrid: Dreyfus 5-stage anchors + Miller's "Does" tier as the assessment mode + EPA entrustment language for level-5 + Bloom mastery learning for the gate philosophy.**
Rationale:
- Dreyfus gives the *behavioral anchor language* for the 5-level rubric (D-039's "5-level anchors"). Each level describes observable behavior, not abstract cognition — ideal for transcribed speech.
- Miller's "Does" tier justifies assessing via a simulated-but-realistic voice scenario rather than a quiz.
- EPA entrustment language ("can be trusted to do this unsupervised") gives level-5 a defensible ceiling that isn't just "more of level-4."
- Bloom's mastery learning legitimizes the **gate** (D-032): advance only after demonstrated criterion performance, with remediation — not after time-on-task.
Bloom's *Taxonomy* alone is the weakest fit (it's not a performance rubric). Do not use it as the scoring spine.
---
## 2. 5-Level Anchoring Example — Customer Service (refund/complaint)
Anchors follow Dreyfus behavioral cues and EPA entrustment language. Level 5 = "trusted to handle unsupervised and to coach peers." Level 1 = "fails to perform; requires intervention." Levels 2–4 are the intermediate behavioral stages.
### 2.1 Empathy / Emotional Attunement
| Lvl | Label | Anchor (observable in transcript) |
|---|---|---|
| 1 | Fail | No acknowledgement of emotion; jumps straight to policy/transactional response. Customer feels unheard. |
| 2 | Advanced Beginner | Cites a scripted empathy line ("I understand your frustration") but moves on mechanically; no follow-up. |
| 3 | Competent | Names the emotion in own words, validates it, then transitions to resolution. Appropriate but not tailored. |
| 4 | Proficient | Adjusts tone to customer's emotional state mid-call; reflects back specifics ("cracked on arrival — that's frustrating"). |
| 5 | Mastery / Entrustable | Reads shifting emotional cues across the call; de-escalates implicitly through pacing and acknowledgment; could model this for new hires. |
### 2.2 Resolution Concreteness
| Lvl | Label | Anchor |
|---|---|---|
| 1 | Fail | Vague ("we'll look into it") or no resolution offered; customer left without a path. |
| 2 | Advanced Beginner | Offers a resolution but missing key specifics (no timeline, no method, no amount). |
| 3 | Competent | Offers a concrete resolution with method (refund/replacement), amount/channel, and next step. |
| 4 | Proficient | Offers a *decision-tree* of concrete options matched to the customer's stated preference; confirms acceptance. |
| 5 | Mastery / Entrustable | Tailors resolution to policy + customer constraint, names the exception/risk considered, and closes the loop with a verification step. |
### 2.3 De-escalation
| Lvl | Label | Anchor |
|---|---|---|
| 1 | Fail | Defensive, blames customer/company policy, or matches the customer's escalation. |
| 2 | Advanced Beginner | Avoids escalation but through avoidance/deflection rather than active de-escalation. |
| 3 | Competent | Uses an explicit de-escalation move (acknowledge → reframe → offer), one cycle. |
| 4 | Proficient | Cycles through acknowledge/reframe as needed; lowers intensity without conceding policy inappropriately. |
| 5 | Mastery / Entrustable | Prevents re-escalation by reading early signals; preserves relationship and policy simultaneously. |
### 2.4 Professionalism / Conduct
| Lvl | Label | Anchor |
|---|---|---|
| 1 | Fail | Unprofessional language, breaks role, gives prohibited advice (legal/medical/financial), or insults customer. |
| 2 | Advanced Beginner | Mostly professional but uses jargon ("RMA", "SLA") or breaks tone once. |
| 3 | Competent | Plain-language, in-role throughout, no prohibited advice. |
| 4 | Proficient | Adapts register to customer; concise for voice (1–3 sentences); manages silence well. |
| 5 | Mastery / Entrustable | Consistently concise, on-brand, voice-appropriate; could serve as a call-center exemplar. |
### Note on anchor design (confidence: **0.78**)
- Anchors must describe **observable behavior in the transcript**, not internal states (per good-rubric principles: Jonsson & Svingby 2007; Reddy & Andrade 2010).
- Level 3 ("Competent") should be the *passing threshold* and defined as "what a competent entry-level hire would do unsupervised." This makes the 3.5 mean gate (D-032) interpretable as "averaging between Competent and Proficient."
- Avoid **evasion anchors** ("somewhat", "mostly") — they destroy inter-rater reliability (Wolfe & Chiu 1997; Barkaoui 2010). The anchors above are behavior-specific.
---
## 3. Mastery Gate N Defensibility (D-032: N=3)
### What the literature says about N-of-M mastery gates
- **Bloom (1968) / Block (1971):** Mastery learning classically requires one demonstration at ≥80% but with *corrective instruction between attempts*. The "N" is not the central variable — the *remediation loop* is. Bloom's evidence is on gain, not on N.
- **Mastery learning meta-analyses (Kulik, Kulik & Bangert-Drowns 1990; Guskey 2007):** Effect sizes are large (~0.5–0.7 SD) but studies use N=1 with remediation; little direct evidence on N≥2.
- **CBME / EPAs (ten Cate 2015; ten Cate & Chen 2018):** Entrustment decisions for an EPA typically require **multiple observations across contexts**. Common recommendations:
- **5–10 observations** per EPA is a frequently cited minimum for *high-stakes* entrustment (e.g., surgical EPAs, Rekman et al. 2016).
- The ACGME milestone framework treats low-stakes formative entrustment at N=1–2; high-stakes summative at N≥5 with multiple assessors.
- **Generalizability theory (Crossley et al. 2002; Bloch & Bogo 2007):** For performance assessments, a single observation has low generalizability (G-coefficients often 0.5–0.7). Generalizability improves with **both** more scenarios *and* more assessors. For voice role-play with one AI assessor, the *scenario count* carries essentially all the reliability burden.
- **Standard setting (Norcini & Guille 2002; Cusimano 2014):** High-stakes credentialing exams typically use multi-stage blueprints sampling **multiple content domains** — 3 is on the low end; 6–12 is common for high-stakes OSCEs (Pell et al. 2010).
- **Angoff / Ebel methods:** Not directly about N, but the standard-setting tradition implies you sample enough items (scenarios) to cover the blueprint reliably. 3 is thin blueprint coverage.
### Is N=3 defensible? (confidence: **0.62**)
**Defensible as a formative / low-stakes gate; not defensible as a high-stakes credential on its own.**
Arguments for N=3:
- Praxis v0.3 is positioning a "path" credential, not a license to practice. If the credential is employer-facing *internal advancement* (not regulatory), N=3 across *distinct* scenarios satisfies the CBE principle of "demonstrated across contexts" weakly but coherently.
- Distinctiveness requirement (D-032 says "distinct scenarios") is the right lever — it's the breadth, not the raw count, that addresses generalizability.
Arguments against N=3 (for high-stakes):
- A single AI assessor means rater variance is not averaged out; all reliability rides on scenario sampling. G-theory suggests N=3 yields G ≈ 0.5–0.6 — below the 0.8 conventional threshold for high-stakes decisions (Brennan 2001).
- 3 scenarios barely covers a blueprint (refund + complaint + escalation = 3 nodes). Real CS skill has more sub-domains.
### Recommended posture (confidence: **0.70**)
1. **Label the v0.3 credential explicitly as "formative" or "path completion"** — not "certification." This makes N=3 defensible.
2. **Add a "high-stakes" tier at N=5–6 distinct scenarios** with blueprint coverage required (≥1 per sub-skill cluster) as the defensible high-stakes threshold. Cite CBME/EPA literature (Rekman 2016; ten Cate 2018) and G-theory (Crossley 2002).
3. **Keep the remediation loop** between attempts — that's where Bloom's mastery-learning effect actually lives. N=3 *without* remediation is weaker than N=1 *with* remediation.
4. **Raise the mean rubric gate from 3.5 to ≥3.5 on each scenario, not just the path mean**, if high-stakes. A path mean of 3.5 can hide a single failing scenario (e.g., 5, 5, 2 → mean 4.0). See §4 for the additive-vs-gating question.
5. Track observed rater-Drift of the LLM extractor over time (D-038); if inter-scenario correlations collapse, N must rise.
---
## 4. Mastery Score Computation
### 4.1 How to combine criteria → scenario score
Options:
- **(a) Weighted mean of criterion scores** (D-039 has per-skill weights).
- **(b) Conjunctive / min-rule** — pass only if *every* criterion ≥ threshold (common in CBME milestone systems; ACGME uses conjunctive for this reason — "no criterion unaddressed").
- **(c) Compensatory mean** — high scores compensate low (what weighted mean implies).
- **(d) Hybrid** — minimum floor on critical criteria + weighted mean for the rest (used in many medical licensing rubrics, e.g., MRCP clinical exam).
**Recommendation (confidence: 0.74):** Use **(d) hybrid: weighted mean with a floor on critical criteria.** Specifically:
- Compute weighted mean of criterion scores (1–5) using D-039 per-skill weights.
- Apply a **floor**: scenario passes only if *every* criterion scored ≥ 2 AND the weighted mean ≥ 3.0 (D-032 sets ≥ 3.5 at the path level).
- Rationale: A learner who scores 5 on resolution and 1 on professionalism should *not* pass a refund scenario — the floor catches this. The literature strongly favors conjunctive rules for *safety-critical* dimensions (Norcini 2003; Wass et al. 2001 on OSCEs); a hybrid is a pragmatic compromise between conjunctive strictness and compensatory flexibility.
### 4.2 How to combine scenario scores → path Mastery Score
**Additive vs gating — the answer is *both*, at different layers.**
- **Gating layer (qualitative):** The N-of-M distinct-scenario pass requirement (D-032) is a **gate**, not a sum. You must pass each of N distinct scenarios. This satisfies the "varied-context mastery" requirement from CBME/EPA literature (ten Cate 2018 — entrustment requires demonstrated generalization).
- **Additive layer (quantitative Mastery Score):** On top of the gate, compute a numeric Mastery Score as the **weighted mean of scenario scores**, where scenario weights reflect blueprint importance (e.g., harder scenarios weighted higher). This gives a continuous signal for ranking/cohort comparison and for the "rubric mean ≥ 3.5" gate in D-032.
**Specific formula recommendation (confidence: 0.72):**
```
MasteryScore(path) = Σ_s ( w_s · ScenarioScore_s ) / Σ_s w_s
where ScenarioScore_s = Σ_c ( w_c · CriterionScore_{s,c} ) / Σ_c w_c
subject to floor: ∀c, CriterionScore_{s,c} ≥ 2
pass s ⇔ ScenarioScore_s ≥ 3.0 (scenario pass threshold)
pass path ⇔ (≥3 distinct scenarios passed) ∧ (MasteryScore ≥ 3.5)
```
This satisfies D-032 exactly: the rubric mean ≥ 3.5 is computed on the *passing* scenarios only (otherwise failed scenarios would drag down a credential earned by passing 3 distinct ones). Decide and document whether MasteryScore is computed over (a) all attempted scenarios or (b) only passing scenarios — **recommend (b)** to align with "mastery" semantics.
### 4.3 Why not just sum?
A sum (e.g., "passed 3 of 5 scenarios") loses information about *how well* and creates a perverse incentive to attempt many easy scenarios. The gate + weighted-mean hybrid avoids this.
---
## 5. Deterministic Scoring Patterns (D-038: LLM extracts, rules score)
The core problem: free-form speech → reproducible score. The D-038 split (LLM-extracts-evidence, rules-score-evidence) is well-aligned with the literature on **structured rubric scoring from natural language**.
### 5.1 The pattern
Two-stage pipelines are the documented way to control LLM variability in assessment (Latif & Zhai 2024 on LLM-as-judge; Chiang & Lee 2023 on explanation-first prompting):
1. **Extraction stage (LLM, allowed to vary):** The LLM is constrained to *extract evidence* — verbatim quotes + structured tags — not to score. Output is a JSON/structured record like:
```
{ "criterion": "empathy",
"evidence_quotes": ["I'm sorry the item arrived cracked — that's frustrating."],
"evidence_signals": ["named_emotion", "acknowledged_specific", "no_policy_first"],
"absence_signals": [] }
```
Key: the LLM does **not** emit a number. It emits *what it observed*. This is the documented "evidence-centered design" pattern (Mislevy, Steinberg & Almond 2003) and matches D-038.
2. **Scoring stage (deterministic rules):** A rule function maps `evidence_signals` (+ absence) to a level 1–5 per criterion, per a published lookup table embedded in `rubrics/<skill>.yaml`. Identical input → identical output. No LLM in this stage.
### 5.2 Why this beats "LLM scores directly"
- **Reproducibility:** Same transcript + same extraction prompt → same evidence tags (modulo LLM nondeterminism, mitigated by temperature=0 + structured output / JSON schema). Rule scoring is fully deterministic given the tags.
- **Auditable:** A learner can see *which quote triggered which signal → which level*. This satisfies CBE transparency principles (C-BEN 2023) and is essential for appeals.
- **Calibratable:** The signal→level table is editable in YAML without retraining; rubric revision is a config change, not a model change.
- **Lower hallucination surface:** LLM is asked only to quote + tag, not to *judge*. Quoting grounds it in the transcript (reduces drift).
### 5.3 Concrete signal taxonomy for one criterion (empathy)
```yaml
# rubrics/customer_service.yaml — fragment
criteria:
empathy:
weight: 0.30
signals:
- id: no_acknowledgement # absence signal
weight: -2
- id: scripted_empathy_line # "I understand your frustration"
weight: +1
- id: named_emotion_in_own_words
weight: +1
- id: acknowledged_specific # references the actual situation
weight: +1
- id: tone_pace_adjusted # extracted from sentence length / hedging
weight: +1
- id: policy_first_before_emotion
weight: -2
levels:
1: { if: [no_acknowledgement, OR, policy_first_before_emotion], score: 1 }
2: { if: [scripted_empathy_line, AND, NOT named_emotion_in_own_words], score: 2 }
3: { if: [named_emotion_in_own_words, AND, acknowledged_specific], score: 3 }
4: { if: [3-level signals, AND, tone_pace_adjusted], score: 4 }
5: { if: [4-level signals, AND, no_policy_first_before_emotion, AND, >=2 acknowledgement instances], score: 5 }
```
The rule engine evaluates these deterministically. The LLM's only job is to populate the `signals` list with quotes.
### 5.4 Remaining risks and mitigations (confidence: 0.68)
| Risk | Mitigation |
|---|---|
| LLM extraction nondeterminism | temperature=0, fixed seed, JSON schema-validated output, retry-on-schema-fail. |
| LLM misses evidence (false negative) | Run extraction twice on borderline cases; flag disagreement for human review. |
| LLM tags a signal that isn't in the transcript (hallucinated quote) | Validate that each `evidence_quote` is a fuzzy-match substring of the transcript; reject otherwise. |
| Rubric drift across model upgrades | Pin extractor model version (already D-020-style); re-run a golden transcript regression suite on any model change. |
| Adversarial phrasing | The signal taxonomy is behavioral; a learner who says the magic words without behavior still lacks the *specificity* and *tone_pace* signals, capping at level 2–3. |
**Overall confidence in the two-stage pattern: 0.80** — this is the strongest-evidence recommendation in this document; the extraction/scoring split is well-grounded (Mislevy ECD; Latif & Zhai 2024 survey).
---
## 6. Customer Service Skill Weights (refund/complaint scenario)
### 6.1 Evidence on what matters in CS calls
- **Customer satisfaction (CSAT) literature:** Empathy and "soft" dimensions dominate CSAT variance in complaint/refund contexts (Verleye 2004; Makavana 2021 survey of CSAT drivers). Resolution matters but is *table stakes* — customers don't reward it, they punish its absence.
- **Service recovery paradox (Magnini, Ford, Markowski & Honeycutt 2007):** After a service failure, *recovery quality* (empathy + ownership) drives loyalty more than the refund itself. This argues empathy ≥ resolution in a *complaint* context specifically.
- **De-escalation** is the safety-critical dimension in escalated calls — it prevents churn, legal escalation, and reputational damage. In *non-escalated* calls it's nearly irrelevant. Weight should be context-dependent.
- **Professionalism / conduct** is a *floor* dimension, not a weighting dimension — it's the conjunctive floor from §4.1, not something to up-weight.
### 6.2 Recommended weights for a refund/complaint scenario (confidence: 0.70)
| Criterion | Weight | Rationale |
|---|---|---|
| Empathy / emotional attunement | **0.35** | Dominant driver of CSAT in service-recovery contexts (Verleye 2004; service recovery paradox literature). |
| Resolution concreteness | **0.30** | Table-stakes; customers punish absence but don't proportionally reward presence. Still substantial because a great empathic call with no resolution is a failure. |
| De-escalation | **0.20** | Safety-critical but only activates in escalated branches. Lower default weight because in the *non-escalated* branch it's near-saturated; *raises* in scenarios with an `escalates_unresolved` failure mode (D-009). |
| Professionalism / conduct | **0.15** | Treated as floor (conjunctive ≥2 to pass) rather than primary weight. |
**Important nuance:** These weights are for the **refund/complaint** scenario specifically (the v0.1 scenario `cs_refund_ca_v01`). A different scenario archetype (e.g., "general inquiry") would tilt empathy down and resolution up. D-039's per-skill weights should be **per-scenario-archetype**, not one global CS weight set. Recommend D-039 be amended to allow `rubrics/customer_service_<archetype>.yaml` or a weights override block in the scenario file.
### 6.3 Dynamic weighting suggestion (confidence: 0.55 — lower, speculative)
If a branch escalates (D-009 `escalates_unresolved` triggered), re-weight on the fly: de-escalation → 0.40, empathy → 0.30, resolution → 0.20, professionalism → 0.10. The rubric's *relevance* changes once the call has gone bad. This is consistent with context-sensitive rubric weighting in OSCE station design (Pell et al. 2010).
---
## Summary confidence table
| Section | Confidence | Driver |
|---|---|---|
| 1. Rubric models (Dreyfus+Miller+EPA+Bloom mastery) | 0.82 | Strong framework fit; well-established literature. |
| 2. 5-level anchoring example | 0.78 | Based on established good-rubric principles; example is illustrative, not validated. |
| 3. N=3 defensibility | 0.62 | N=3 defensible only for formative / path-completion credentials; thin for high-stakes. |
| 4. Mastery score computation (hybrid floor + weighted mean, gate+additive layered) | 0.72 | Aligns with CBE/EPA practice; specific formula is a synthesis, not a direct citation. |
| 5. Deterministic scoring (LLM-extract + rule-score) | 0.80 | Strongest evidence base (ECD, LLM-as-judge surveys); pattern is well-grounded. |
| 6. CS weights for refund/complaint | 0.70 | Anchored in CSAT/service-recovery literature; specific numbers are judgment calls. |
## Key references
- Anderson, L. W., & Krathwohl, D. R. (Eds.). (2001). *A Taxonomy for Learning, Teaching, and Assessing.* Bloom's revised taxonomy.
- Barkaoui, K. (2010). Do ESL essay raters' evaluation criteria change with experience? *Assessing Writing.*
- Benner, P. (1982). From novice to expert. *AJN.* (Dreyfus applied to nursing.)
- Block, J. H. (1971). *Mastery Learning: Theory and Practice.*
- Bloom, B. S. (1968). Learning for mastery.
- Brennan, R. L. (2001). *Generalizability Theory.* (G-coefficient thresholds.)
- C-BEN (2023). Quality Assurance Principles for CBE programs.
- Chiang, C.-H., & Lee, H.-Y. (2023). Can large language models be good judges?
- Crossley, J., Davies, H., Humphris, G., & Jolly, B. (2002). Generalisability in healthcare assessments.
- Cusimano, M. D. (2014). Standard setting in medical education.
- Dreyfus, H., & Dreyfus, S. (1986). *Mind Over Machine.* (Five-stage skill acquisition.)
- Guskey, T. R. (2007). Closing achievement gaps: Revisiting mastery learning.
- Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity, and educational consequences.
- Kulik, C.-L. C., Kulik, J. A., & Bangert-Drowns, R. L. (1990). Effectiveness of mastery learning programs.
- Latif, S., & Zhai, X. (2024). A systematic review of LLM-as-a-judge.
- Magnini, V. P., Ford, J. B., Markowski, E. P., & Honeycutt, E. D. (2007). The service recovery paradox.
- Miller, G. E. (1990). The assessment of clinical skills/competence/performance. *Academic Medicine.*
- Mislevy, R. J., Steinberg, L. S., & Almond, R. A. (2003). On the structure of educational assessments. (Evidence-centered design.)
- Norcini, J. (2003). ABC of learning and teaching in medicine: Work based assessment.
- Norcini, J., & Guille, R. (2002). Standard setting in medical education.
- Pell, G., Boursicot, K., & Roberts, T. (2010). Could OSCEs be replaced? (Blueprint coverage / station counts.)
- Rekman, J., Hamstra, S. J., et al. (2016). Entrustable professional activities. (N recommendations.)
- Reddy, Y. M., & Andrade, H. (2010). A review of rubric use in higher education.
- ten Cate, O. (2005). Entrustable professional activities.
- ten Cate, O., & Chen, H. C. (2018). The EPAs of competency-based medical education.
- Verleye, K. (2004). Empathy in customer service.
- Wass, V., Van der Vleuten, C., Shatzer, J., & Jones, R. (2001). Assessment of clinical competence.