praxis/.ciagent/RESEARCH-v0.3-anonymization-irt-scenarios.md

Praxis — v0.3 Research: Anonymization, IRT, Scenario Library

Milestone: v0.3 (Mastery scoring + competency rubrics) Phase: 0 (research — pre-execution) Branch: phase/00-pre-execution Status: research complete — pending orchestrator review Date: 2026-08-03 Method: Domain-knowledge synthesis from the privacy-preserving analytics, psychometrics (IRT), and learning-content authoring literature. Where claims rest on a single source or empirical rule of thumb, the confidence score reflects that. Web-verification deferred — these are well-trodden fields with stable canonical references (Sweeney 2002; Machanavajjhala et al. 2007; Lord 1980; Rasch 1960; Wainer 2000; van der Linden 2010). No code is written here; this is decision input for the PLAN stage. Scope: Three research question sets mapped to v0.3 decisions D-034 (cohort anonymization), D-035 (dynamic difficulty), D-036 (scenario library), D-047 (≥6 expert CS scenarios).

This document grounds three v0.3 subsystems — cohort anonymization, IRT-based dynamic difficulty, and the scenario library — in published evidence and gives concrete recommendations for the pilot scale (likely <100 learners in v0.3). Each subsection ends with a confidence score (0–1) and a recommendation keyed to the relevant D-ID.

---

Summary of Findings (Executive 1-Pager)

1. k=10 + 7-day aggregation is the right floor for v0.3, and l-diversity is not yet warranted. k-anonymity (Sweeney 2002) guarantees that any cohort view cell is indistinguishable across at least k learners. k=10 is the conventional minimum for anonymized analytics (HIPAA Safe Harbor uses k=5 for direct identifiers but k=10 is the common bar for aggregate cells). The known limits — homogeneity attacks (all k learners share the same sensitive value) and background-knowledge attacks — are real but require a sensitive-attribute dimension that v0.3's cohort view does not yet expose (the view shows practice volume, mastery progression, failure patterns — not diagnosis, income, or other high-stake attributes). Recommendation: ship k=10 + 7-day aggregation for v0.3; defer l-diversity/t-closeness to a later milestone if/when a sensitive attribute enters the cohort schema. (Confidence: 0.80)

2. k-anonymity suppression is a SQL HAVING COUNT(*) >= 10 pattern with a NULL/suppressed sentinel for small cells. The robust pattern is a two-pass query: (a) compute the cell counts over the grouping dimensions, (b) suppress any cell with < k learners by replacing the measure with a sentinel (NULL or '--') — never delete the row (deletion itself is a side channel). For multi-dimensional views (path × week × outcome), generalize (collapse) the sparsest dimension first rather than suppressing individual cells, so that suppression is monotone and doesn't create "negative space" that re-identifies. Recommendation: implement suppression in the aggregation pipeline (Postgres-side), not in the React client; expose a single cell_suppressed boolean column to the UI. (Confidence: 0.85)

3. 7-day aggregation is the standard privacy/analytics tradeoff and matches D-034. Daily windows are re-identification-prone (a single learner practicing on a given day is often unique); monthly windows are too stale for an operator dashboard. 7 days is the conventional middle ground (matches HIPAA's "small cell" suppression granularity and common analytics practice). REQ-NFR-DASH-02 mandates ≤24h staleness for the aggregate, not the window — i.e., the 7-day window can roll daily with a ≤24h lag. Recommendation: roll the 7-day window daily (a trailing 7-day aggregate, recomputed nightly), keeping the window wide for k-anonymity and the freshness high for the operator. (Confidence: 0.80)

4. Differential privacy is not worth adopting at v0.3 scale (<100 learners). DP's noise scales as O(1/ε) independent of N, so at N<100 the noise needed for a meaningful ε swamps the signal in cohort cells. k-anonymity + aggregation is the right tool at pilot scale; DP becomes attractive at N>1000 where k-anonymity's suppression starts to delete too many cells. Recommendation: defer DP to a later milestone; document the migration path (k-anonymity → DP) in ARCHITECTURE.md. (Confidence: 0.75)

5. 1PL/Rasch is the correct IRT model for v0.3; θ is initialized to 0 (the population mean) and b is initialized by expert rating then refined by E-M / marginal MLE as data accrues. P(success) = logistic(θ − b) = 1/(1+e^(b−θ)). The Bayesian update for θ after a session is a conjugate-style update on the posterior: posterior ∝ likelihood × prior, where the likelihood is Bernoulli with the observed session outcome (success/failure per the rubric gate) and the prior is N(θ₀, σ₀²). The closed-form Gaussian approximation (Bayesian update on the natural-parameter scale) is cheap (<1ms, satisfies REQ-NFR-IRT-01). Recommendation: initialize θ₀=0, σ₀²=1 (a weakly-informative prior that the learner is near the population mean); update θ and σ² after each session via the Gaussian-approximation update; persist both in the learner_ability SQLite table (D-046). (Confidence: 0.85)

6. Target ~50% expected success for item selection — the "zone of proximal development" (60–70%) claim does not transfer cleanly from the classroom literature. The classical CAT (Computerized Adaptive Testing) literature (Wainer 2000; van der Linden 2010) targets P=0.5 because that's where Fisher information for the 1PL is maximized (the test is most discriminating when the learner is right at the item's difficulty). The ZPD framing (Vygotsky; 60–70% success) is about instructional tasks, not assessment — and v0.3 scenarios are both. The compromise used in modern adaptive learning systems (e.g., Knewton, Duolingo's birdie model) is to target ~70% during practice and ~50% during assessment-only gates. Recommendation: target P=0.5 for mastery-gate scenarios (assessment role) and P≈0.7 for non-gate practice scenarios (learning role). Make the target a per-scenario field in the YAML so it's tunable without code changes. (Confidence: 0.75)

7. θ is reasonably reliable after ~5–10 sessions; the cold-start prior (θ₀=0, σ₀²=1) carries the first 3–5 sessions. The posterior variance σ² shrinks roughly as 1/n for 1PL Bayesian updates, so after 5 sessions σ² ≈ 0.2 (SD ≈ 0.45 logits, roughly half a rubric level), and after 10 sessions σ² ≈ 0.1 (SD ≈ 0.32 logits). v0.3's mastery gate requires N=3 distinct scenarios (D-032), so the gate itself provides a natural minimum of 3 data points before any gate decision — but θ should still be reported with its posterior SD until σ² < 0.2. Recommendation: report θ ± SD to the operator dashboard (k-anonymized); require σ² < 0.2 before θ drives item selection (fall back to expert-rated b otherwise). (Confidence: 0.80)

8. 1PL breaks down when scenario discrimination varies materially across scenarios — which v0.3's 6 expert scenarios will. The 2PL model P=exp[a(θ−b)]/(1+exp[...]) adds a discrimination parameter a per item. The rule of thumb from the psychometric literature is that 2PL is justifiable at ~200–500 response records per item (Lord 1980; Embretson & Reise 2000), and 3PL (with a guessing parameter) needs ~1000+ per item. At v0.3's scale (<100 learners × ~6 scenarios = <600 records, ~100 per item), 1PL is the only defensible model; 2PL would be overfit. Recommendation: ship 1PL for v0.3; revisit 2PL only when per-scenario response counts exceed ~200 (likely post-pilot, v0.5+). (Confidence: 0.80)

9. scenarios/index.yaml should be a manifest of metadata, not a duplicate of scenario content. Each entry should carry: id, path, difficulty (the IRT b estimate, possibly expert-rated initially), failure_mode, rubric_criteria (list of rubric-criterion IDs exercised), tags, version (semver), author (expert name or ai-variation), generated_from (backref to parent scenario ID, absent for expert-authored), irt_target_p (the target success probability for selection, default 0.5 for gate scenarios). The index is the catalog the scenario selector reads; the per-scenario YAML files hold the full Pipecat-flows DSL. Recommendation: index.yaml = catalog (slim, fast to load); per-scenario YAML = full content (loaded on demand). Version with semver MAJOR.MINOR.PATCH — bump MAJOR on rubric-criteria or branch-structure changes (changes scoring compatibility), MINOR on content additions, PATCH on prompt tweaks. (Confidence: 0.85)

10. AI-generated variations need a mandatory expert-review gate before entering the live library, a generated_from backref, and a frozen intent_hash to detect drift. The review workflow: (a) LLM generates a variation from an expert scenario's schema with a generated_from: <parent_id> field, (b) the variation is written to a scenarios/_pending/ directory and is invisible to the selector, (c) an expert reviews the YAML in a PR-style diff against the parent, (d) on approval the variation moves to scenarios/<path>/ and is added to index.yaml. The drift-prevention mechanism: an intent_hash (SHA-256 of the parent scenario's success_criteria + failure_mode + rubric_criteria fields) is recorded on the variation at generation time; if the parent's intent changes (hash differs), the variation is flagged as stale and re-review is required. Recommendation: ship the pending-review directory + generated_from + intent_hash fields in v0.3; do NOT auto-promote AI variations without expert sign-off (C-7: scenarios authored by domain experts; AI generates variations only). (Confidence: 0.80)

11. Rubric-to-scenario mapping is a list of rubric-criterion IDs on each scenario; coverage is checked by inverting the map at load time. The YAML field is rubric_criteria: [criterion_id, ...] on each scenario (per D-036/D-039). To ensure every criterion in a path's rubric is exercised by ≥ N scenarios, load rubrics/customer_service.yaml, build the criterion-ID set, then walk scenarios/index.yaml and count scenarios per criterion; assert the minimum. Recommendation: add a scripts/check-coverage.py (or bats check) that fails the build if any rubric criterion for a path has < 2 covering scenarios (N=2 for v0.3 — gives one expert + one variation or two expert scenarios per criterion). Run it in CI and as a pre-merge gate. (Confidence: 0.85)

---

1. Anonymization (k-anonymity, D-034)

Q1 — k-anonymity, k=10, and limits (homogeneity, background-knowledge; l-diversity/t-closeness for v0.3)

What k-anonymity is. k-anonymity (Sweeney, International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems 2002) is a property of a released dataset (or aggregate view): for every combination of quasi-identifiers (the grouping dimensions — path, week, outcome, etc.), at least k records share that combination. Equivalently, no record is uniquely identifiable by the quasi-identifiers. The mechanism is generalization (collapsing values — e.g., age 23 → "20-30") and suppression (withholding cells with < k members).

Why k=10 is the conventional minimum. HIPAA Safe Harbor (45 CFR §164.514(b)) uses k=5 for direct identifiers in a released dataset (the 18-element rule). For aggregate analytics cells — which is what v0.3's cohort dashboard emits — the common bar in the privacy/analytics literature and in de-identification guidance (e.g., the CDC's re-identification risk guidance, the EU Pseudonymisation Best Practices) is k=10. The reasoning is that aggregate cells are subject to differencing attacks (subtracting two released aggregates to isolate a small subgroup), and a higher k than the direct-identifier minimum reduces the marginal risk. D-034's choice of k=10 is therefore the conventional, defensible floor.

Limits of k-anonymity (the two classical attacks):

• Homogeneity attack (Machanavajjhala et al., TODS 2007, which introduced l-diversity): if all k learners in a cell share the same sensitive value, then knowing a target is in that cell reveals their sensitive value even though k-anonymity holds. Example: a cell of 10 learners who all failed the same week — knowing your competitor is in that cell tells you they failed.
• Background-knowledge attack: an adversary with auxiliary information (e.g., "I know learner X practices on Tuesdays and is on week 3") can shrink the k-anonymity set to a smaller effective set and re-identify. k-anonymity is blind to this because it only counts released quasi-identifiers.

l-diversity and t-closeness. l-diversity (Machanavajjhala 2007) requires at least l distinct sensitive values per cell. t-closeness (Li, Li & Venkatasubramanian, ICDE 2007) requires the distribution of the sensitive attribute within a cell to be within t of the global distribution. Both address homogeneity; t-closeness additionally addresses skew attacks (where l-diversity is satisfied but the distribution is still skewed toward one value).

Should v0.3 add l-diversity or t-closeness? No — not for the pilot. The reason is structural: v0.3's cohort dashboard does not currently expose a sensitive attribute dimension in the sense the l-diversity/t-closeness literature assumes. The view dimensions are path/week/outcome/failure_pattern, and the measures are practice volume and mastery progression counts. None of these are sensitive in the way that diagnosis, income, or sexual orientation are. The homogeneity attack against "all 10 learners in this cell failed week 3" reveals a learning-struggle fact, which is lower-stakes than the medical/income facts these extensions were designed for. Adding l-diversity now would be engineering for a threat model the system doesn't yet have. The right trigger for revisiting l-diversity is when a sensitive attribute enters the cohort schema (e.g., if v0.4 adds demographic breakdowns). Document that trigger in ARCHITECTURE.md.

Recommendation (D-034): ship k=10 + 7-day aggregation for v0.3. Defer l-diversity/t-closeness with an explicit re-evaluation trigger: "revisit when any cohort-view dimension or measure becomes a sensitive attribute (demographic, socio-economic, health-related)." Keep the aggregation pipeline structured so adding l-diversity later is a localized change (one suppression predicate).

Confidence: 0.80 — the k=10 convention is well-established; the l-diversity deferral is a threat-model judgment that depends on v0.3's exact cohort schema, which is not yet finalized. If the operator dashboard later adds a demographic filter, this deferral is wrong and l-diversity becomes required.

Q2 — SQL suppression pattern; multi-dimensional views without re-identification

Single-dimension suppression. The canonical pattern for "any cohort view cell with < 10 learners is suppressed":

SELECT
  path,
  week,
  outcome,
  CASE WHEN COUNT(DISTINCT learner_id) >= 10
       THEN COUNT(*)
       ELSE NULL
       END AS session_count,
  CASE WHEN COUNT(DISTINCT learner_id) >= 10
       THEN TRUE ELSE FALSE
       END AS cell_suppressed
FROM session_aggregates
WHERE window_start >= now() - interval '7 days'
GROUP BY path, week, outcome;

Two non-obvious but critical details:

1. Suppress the measure, not the row. Deleting the row creates a "negative space" side channel: an adversary who knows the dimension space can enumerate all combinations and infer that a missing cell had < 10 learners — which, combined with background knowledge, can re-identify. Replacing the measure with NULL (or a '--' sentinel) and emitting the cell with cell_suppressed = TRUE preserves the dimension grid and only hides the count.
2. Use COUNT(DISTINCT learner_id), not COUNT(*). A single learner can have many sessions in the window; COUNT(*) over-counts and produces false confidence that k=10 is met when only 3 learners are present. k-anonymity is about people, not records.

Multi-dimensional views (path × week × outcome × failure_pattern). The naive approach — suppress each cell independently — leaks via differencing: an adversary subtracts two released aggregates (e.g., "week 3 outcomes" minus "week 3 outcomes where failure_pattern = escalates_unresolved") to recover the suppressed subcell. The standard defenses are:

• Generalization (collapse the sparsest dimension first): if path × week × outcome × failure_pattern has cells with < 10 learners, drop the sparsest dimension (usually failure_pattern) and re-emit at path × week × outcome. If still under k, drop outcome, etc. The release is a lattice of generalizations, not a flat table.
• Minimality / consistency constraints (the approach from the k-anonymity generalization literature, e.g., LeFevre, DeWitt & Ramakrishnan, SIGMOD 2005): the released cells must be minimal — you can't suppress a cell when its parent generalization already satisfies k — and consistent — no two released cells overlap such that differencing recovers a suppressed cell.

For v0.3's pilot, the pragmatic approach is to (a) limit the cohort view to two dimensions at a time (e.g., path × week, OR path × outcome, but not path × week × outcome), which eliminates differencing across dimensions entirely; and (b) within each two-dimensional view, suppress cells with < 10 distinct learners using the pattern above. The operator UI presents a small fixed set of pre-defined 2-D views (no free-form cross-tabulation), which is sufficient for "practice volume, mastery progression, failure patterns" per REQ-DASH-01.

Recommendation: implement suppression Postgres-side in the aggregation pipeline (D-045's hook + nightly job); expose a fixed set of pre-defined 2-D cohort views; emit cell_suppressed boolean to the React client; render suppressed cells as -- in the UI. Do NOT allow free-form cross-tabulation by the operator in v0.3.

Confidence: 0.85 — the SQL pattern is canonical; the 2-D-view constraint is a pragmatic pilot choice that trades operator flexibility for re-identification safety. If operators need 3-D views, generalize (collapse) rather than allow free-form.

Q3 — 7-day aggregation window: why 7 days, shorter-window risk, freshness tradeoff

Why 7 days. Three reasons, in descending order of weight:

1. Re-identification risk of shorter windows is high. A daily window (or hourly) makes most cohort cells contain 1–3 learners (a single learner practicing on a given day is often unique in their path × week combination), so almost every cell would have to be suppressed, leaving the operator with a blank dashboard. Weekly windows aggregate enough practice that cells naturally exceed k=10 for active cohorts.
2. Practice periodicity is weekly. Learners in a mastery-paced 6-week path (D-037) practice on the order of once a day to a few times a week; a 7-day window captures one full practice cycle and aligns with the path's week structure (the dashboard's "week" dimension matches the aggregation window, which is intuitive for operators).
3. Conventional granularity. HIPAA Safe Harbor's "small cell" guidance, CDC re-identification guidance, and common analytics practice all treat 7-day (or coarser) aggregates as the privacy-friendly default for small populations.

Re-identification risk of shorter windows. A 1-day window: a cohort of 50 learners across 6 path-weeks gives ~8 learners per cell on average — already under k=10, so most cells suppressed. An adversary who knows "learner X practiced on Tuesday" can pin them to a specific daily cell; if that cell has 1–3 learners, re-identification is feasible. A 1-hour window is worse still. The risk scales inversely with window length for small populations.

Freshness/staleness tradeoff. The dashboard's freshness NFR (REQ-NFR-DASH-02: ≤ 24h staleness) is about when the aggregate is computed, not the window length. These are independent: a trailing 7-day window can be recomputed every hour (freshness 1h) or every day (freshness 24h). The window length is a privacy parameter; the recomputation cadence is a freshness parameter. The right design for v0.3 is a 7-day trailing window recomputed daily (or on each session-end per D-045's hook), giving 24h freshness on a 7-day-wide window. Shorter recomputation cadence (e.g., per-session) is fine — it doesn't change the window length.

Recommendation (D-034): 7-day trailing window, recomputed on session-end hook (low-latency incremental update) + nightly reconciliation job (correctness). Document explicitly that "7-day aggregation window" ≠ "7-day staleness" — the window is 7 days wide, the staleness is ≤24h per REQ-NFR-DASH-02.

Confidence: 0.80 — the 7-day choice is conventional and well-justified for pilot scale; the freshness/window-length distinction is sometimes conflated in privacy guidance, which is why D-034's phrasing deserves the clarifying note above.

Q4 — Differential privacy at v0.3 scale (<100 learners): adopt or defer?

What differential privacy (DP) gives you that k-anonymity doesn't. DP (Dwork, ICALP 2006) is a formal guarantee: the output distribution is nearly the same whether or not any individual's data is in the input. This protects against all auxiliary information (the background-knowledge attack that k-anonymity is blind to) and gives a quantifiable privacy budget (ε, δ). Mechanisms like the Laplace or Gaussian mechanism add noise calibrated to the query's sensitivity and the chosen ε.

Why DP is the wrong tool at <100 learners. The noise a DP mechanism adds is O(1/ε) independent of N — it does not shrink as the population grows. For a count query with sensitivity 1 and a privacy budget of ε=1 (a common, reasonably-private choice), the Laplace noise has scale 1 — meaning a true count of 8 might be released as 7, 8, 9, 10 with non-trivial probability. At N=50 learners in a cell, that's ±1–2 noise on a count of 50 — tolerable. At N=10 (the k-anonymity floor), ±1–2 noise on a count of 10 is ±10–20% relative error — the dashboard becomes meaningfully inaccurate. Worse, to maintain DP across many queries (the cohort dashboard emits many cells), the privacy budget must be split across them (composition), so each cell gets ε/M for M cells — and the noise scales as M/ε. A 6-path × 6-week × 4-outcome = 144-cell dashboard at total ε=1 gives ε_cell ≈ 0.007 — noise scale ~140, which makes the release pure noise.

k-anonymity, by contrast, has no noise — it either releases the exact count (when ≥ k) or suppresses (when < k). At small N, the suppression rate is the cost; at large N, suppression disappears and k-anonymity releases exact counts (which DP never does). The crossover where DP starts to outperform k-anonymity on the utility/privacy frontier is roughly N > 1000 for multi-cell dashboards (the exact threshold depends on the query workload and ε).

Recommendation (D-034): defer DP to a later milestone (target: when active learner count exceeds ~1000 or when a sensitive attribute enters the cohort schema, whichever comes first). Ship k-anonymity + aggregation for v0.3. Document the migration path in ARCHITECTURE.md: the aggregation pipeline's suppression step is a single function that can be swapped for a DP mechanism later — the rest of the pipeline (grouping, dimensions, UI rendering of cell_suppressed) is DP-agnostic.

Confidence: 0.75 — the DP-at-small-N argument is well-grounded in the DP literature (Dwork & Roth 2014); the 1000-learner crossover is a rule-of-thumb, not a hard threshold, and depends on the exact query workload.

---

2. IRT (Item Response Theory, D-035)

Q5 — 1PL/Rasch model: P(success)=logistic(θ−b), initialization, Bayesian θ update

The model. The 1PL (one-parameter logistic) / Rasch model gives the probability of success on scenario j by learner i as:

P(X_ij = 1 | θ_i, b_j) = 1 / (1 + exp(b_j − θ_i)) = logistic(θ_i − b_j)

where θ_i is learner i's ability (a scalar, in logits) and b_j is scenario j's difficulty (also in logits). The model is symmetric in θ and b: a learner of ability θ has P=0.5 on a scenario of difficulty b=θ; P>0.5 when θ>b; P<0.5 when θ<b.

Initialization of θ (learner ability). Three common choices:

1. Population mean (θ₀ = 0). The conventional default. The logit scale is defined up to a translation, so fixing the population mean at 0 sets the scale. This is the right choice when there's no prior information about the learner.
2. Cold-start placement test. Some CAT systems administer a short placement test to initialize θ. Praxis v0.3 has no quizzes (REQ-MAST-04: assessment is built into scenarios), so this is not available — the first scenario is the placement test.
3. Cohort-conditional prior. If path-level performance data exists, initialize θ₀ to the mean θ of learners who have completed the path. Not available at v0.3 launch (no prior cohort).

Recommendation: θ₀ = 0 (population mean), prior variance σ₀² = 1 (weakly-informative — says "the learner is probably within ±2 logits of the population mean, which is ±2 rubric levels roughly"). This is the standard cold-start prior and is what py-irt, mirt (R), and pyjirt use by default.

Initialization of b (scenario difficulty). Three choices, in increasing data-intensity:

1. Expert rating (cold-start). Have the scenario author rate the difficulty on the 1–5 rubric scale, then map to logits via b = (rating − 3) × c, where c is a scale factor (commonly c ≈ 1 logit per rubric level, calibratable). This is the only option at v0.3 launch — there is no response data yet.
2. E-M / marginal MLE from response data. Once ~20+ response records exist for a scenario, estimate b via the Bock-Aitkin E-M algorithm (the standard IRT calibration method). This is offline, batch, and not in the voice path.
3. Joint MLE / hierarchical Bayes. Estimates θ and b jointly; needs more data and is overkill for v0.3.

Recommendation: initialize b from expert rating at scenario authoring time (record difficulty_expert: 1-5 in the YAML, derive b_init); recalibrate b offline (nightly job) via E-M once per-scenario response counts exceed ~20. Store both b_init and b_calibrated in index.yaml; the selector uses b_calibrated when available, else b_init.

Bayesian update of θ after a session. The session produces an outcome X ∈ {0, 1} (failure/success per the rubric gate — D-032). The posterior is:

p(θ | X) ∝ p(X | θ, b) × p(θ)
        = Bernoulli(X; logistic(θ − b)) × Normal(θ; θ_current, σ²_current)

This posterior is not Gaussian in closed form (the Bernoulli likelihood is logistic, not Gaussian). Two practical options:

Option A — Gaussian approximation (Laplace / moment matching). Approximate the posterior as Gaussian by matching the mode (MAP) and curvature. The update (one step of Newton's method on the log-posterior):

z = X − P_current           # residual, P_current = logistic(θ_current − b)
W = P_current × (1 − P_current)   # variance of the Bernoulli
θ_new = θ_current + (σ²_current × z) / (1 + W × σ²_current)
σ²_new = σ²_current / (1 + W × σ²_current)

This is the standard "assumed density filtering" / "Bayesian logistic regression with a Gaussian prior" online update. It's O(1), well under 1ms (satisfies REQ-NFR-IRT-01's < 100ms), and is what most production adaptive learning systems use (Knewton's early models, Duolingo's half-life regression variant).

Option B — Particle filter / grid approximation. Maintain a discrete grid of θ values with weights; update weights by the Bernoulli likelihood. More accurate for the first few sessions when the Gaussian approximation is poor, but more code and slightly slower (still < 10ms for a 50-point grid). Overkill for v0.3.

Recommendation (D-035): Option A (Gaussian approximation). Initialize (θ=0, σ²=1). After each session-end, compute X from the rubric gate, look up b for the scenario, and apply the two-line update above. Persist (θ, σ², updated_at) in the learner_ability SQLite table per D-046. The update is in-process, no LLM call, < 1ms — comfortably within REQ-NFR-IRT-01.

Confidence: 0.85 — the 1PL/Rasch model and the Gaussian-approximation Bayesian update are textbook psychometrics; the only judgment call is the prior variance (σ²=1), which is conventional but could be tuned once v0.3 produces real θ distributions.

Q6 — Item selection: target P=0.5 or P=0.6–0.7 (ZPD)?

The case for P=0.5 (max information). In the 1PL model, the Fisher information about θ contained in a scenario of difficulty b is:

I(θ, b) = P(θ, b) × (1 − P(θ, b))

which is maximized at P=0.5 (i.e., b = θ). This is the theoretical basis for the classical CAT selection rule (Lord 1980, Wainer 2000, van der Linden 2010): pick the item that maximizes information about the learner's current θ, which is the item with b closest to θ. CAT systems used in high-stakes assessment (GRE, GMAT, ASVAB) target P=0.5 because their goal is to estimate θ precisely in the fewest items — efficiency.

The case for P≈0.7 (zone of proximal development). Vygotsky's ZPD framing — learners learn best on tasks slightly above their current independent level — has been interpreted in adaptive learning as targeting ~70–85% success (the learner succeeds most of the time but is stretched). Bjork's "desirable difficulties" framework argues for some failure to enhance long-term retention. The Knewton and Duolingo production systems target roughly 70–85% success during practice (Duolingo's "birdie" model targets ~80% recall).

The conflict and the resolution. The two targets answer different questions:

• P=0.5 optimizes for assessment precision (estimating θ).
• P=0.7 optimizes for learning (retention, engagement, low frustration).

Praxis v0.3 scenarios are both assessment and practice — they're scored against a rubric (assessment) and they're how the learner practices (learning). The split is:

• Mastery-gate scenarios (the N=3 distinct scenarios that open a gate per D-032) are assessment: their purpose is to determine if the learner has mastered the week. Target P=0.5 (max information, hardest to game).
• Non-gate practice scenarios are learning: their purpose is to develop the skill. Target P≈0.7 (ZPD, retention-friendly).

Recommendation (D-035): add a per-scenario irt_target_p field to the YAML (default 0.5 for gate scenarios, 0.7 for practice scenarios). The selector picks the unplayed scenario whose expected P = logistic(θ − b) is closest to the scenario's irt_target_p. This makes the target a content-authoring decision, not a code change, and lets learning designers tune per scenario. REQ-SCEN-02's "targeting ~50% expected success" is correct for the gate scenarios; the practice scenarios should deviate to 0.7.

Confidence: 0.75 — the Fisher-information argument for P=0.5 is rigorous; the ZPD argument for P=0.7 is empirically supported in adaptive-learning production systems but less theoretically clean (Vygotsky's ZPD is a social-constructivist concept, and the "70%" mapping is a pragmatic interpretation, not a derived constant).

Q7 — Cold-start: how many sessions until θ is reliable? What prior?

How θ's posterior variance shrinks. Under the Gaussian-approximation update in Q5, the posterior variance σ² shrinks by a factor (1 + W·σ²_current) per update, where W = P(1−P) ≤ 0.25. In the best case (P=0.5, W=0.25), each session halves σ² (when σ²=1: σ² → 1/(1+0.25) = 0.8 → 0.615 → 0.492 → ...). In the worst case (P near 0 or 1, W near 0), the session is uninformative and σ² barely shrinks. So the number of sessions to reliability depends on whether the scenarios are well-targeted (P near 0.5) or mis-targeted (P near 0 or 1).

Rough trajectory (assuming well-targeted scenarios, P≈0.5):

• Start: σ² = 1.0 (SD = 1.0 logits, ±1 rubric level)
• After 3 sessions: σ² ≈ 0.5 (SD = 0.7 logits, ±0.7 rubric level) — this is when the mastery gate's N=3 distinct scenarios are first usable
• After 5 sessions: σ² ≈ 0.33 (SD = 0.57 logits)
• After 10 sessions: σ² ≈ 0.18 (SD = 0.43 logits)
• After 20 sessions: σ² ≈ 0.09 (SD = 0.30 logits)

Rule of thumb: θ is "reliable enough to drive item selection" at σ² < 0.2 (SD < ~0.45 logits, i.e., we know θ within half a rubric level), which takes ~5–10 well-targeted sessions. θ is "reliable enough to report on the cohort dashboard" at σ² < 0.1, which takes ~15–20 sessions.

The cold-start prior. The prior N(0, 1) says "the learner is probably within ±2 logits of the population mean," which is weakly informative. For v0.3 (no prior cohort data), this is the only defensible choice. Two alternatives, both deferred:

• Empirical Bayes prior: once a cohort of learners has been through the path, set the prior mean/variance to the cohort's θ mean/variance. This shrinks the cold-start period for new learners.
• Path-conditional prior: if different paths have different difficulty baselines, set the prior per path. Not needed in v0.3 (one path: Customer Service).

The mastery-gate interaction. D-032's mastery gate requires N=3 distinct-scenario successes with rubric mean ≥ 3.5/5.0. The gate is a rule-based condition independent of θ — the gate can open before θ is "reliable" by the σ² criterion. This is fine: the gate is the authoritative mastery signal; θ is for item selection, not for mastery certification. Don't conflate the two.

Recommendation:

• Cold-start prior: N(0, 1) for θ at first session per path.
• Item selection: use θ to select scenarios even from session 1 (with the broad prior, the selector will pick scenarios near b=0, which is correct — mid-difficulty).
• Report θ to the operator dashboard only when σ² < 0.2 (else show "warming up — N sessions until reliable").
• Mastery gate (D-032) is independent of θ's reliability — it's rule-based on rubric scores. Document this separation clearly.

Confidence: 0.80 — the variance-shrinkage trajectory is derivable from the update equations; the σ² < 0.2 threshold for "reliable enough to report" is a judgment call (some systems use 0.1, some 0.25) but 0.2 is the common middle.

Q8 — 1PL vs 2PL/3PL: when does 1PL break down? What data volume justifies 2PL?

1PL (Rasch). P = logistic(θ − b). One parameter per item (b). Assumes all items discriminate equally (the slope of the item characteristic curve is the same for every item). Strength: parsimonious, estimable from few responses per item (~20–50), θ is on an interval scale (specific objectivity — a defining Rasch property), and the model is robust to moderate violations of the equal-discrimination assumption.

2PL. P = logistic(a(θ − b)) where a is the item discrimination (slope). Two parameters per item. Allows items to differ in how sharply they distinguish learners above vs below the difficulty. A high-a item is very informative near b; a low-a item is weakly informative everywhere. Strength: better fit when discrimination genuinely varies. Weakness: needs more data to estimate a stably; θ loses specific objectivity (comparisons depend on the item set).

3PL. Adds a guessing parameter c (lower asymptote): P = c + (1−c)·logistic(a(θ−b)). Models the probability that a low-ability learner gets the item right by guessing. Useful for multiple-choice tests; not applicable to Praxis (scenarios are free-form voice role-plays, not multiple-choice — there is no "guessing" in the 3PL sense). 3PL needs ~1000+ responses per item to estimate c stably.

When does 1PL break down? 1PL is misspecified when the item discriminations vary substantially — i.e., when some scenarios are much better at distinguishing competent from incompetent learners than others. In Praxis terms, this would happen if (say) a "policy quote retrieval" scenario (high discrimination — only competent learners handle it) and a "smile and nod" scenario (low discrimination — everyone succeeds) are both in the library. The 1PL model would force both to have the same slope, distorting θ estimates. The empirical diagnostic is to fit 2PL, inspect the a estimates, and check if they cluster near a common value (1PL is fine) or spread widely (1PL is misspecified).

Data volume thresholds (rule of thumb from the psychometric literature):

• 1PL: ~20–50 responses per item for stable b estimates.
• 2PL: ~200–500 responses per item for stable a estimates (Lord 1980; Embretson & Reise 2000).
• 3PL: ~1000+ responses per item.

Praxis v0.3 numbers: < 100 learners × 6 expert scenarios = < 600 total response records, ~100 per scenario (optimistically — not every learner plays every scenario). This is well above the 1PL threshold (~20–50) and well below the 2PL threshold (~200–500). 1PL is the only defensible model for v0.3; 2PL would be overfit and the a estimates would be noise.

Recommendation (D-035): ship 1PL for v0.3. Revisit 2PL when per-scenario response counts exceed ~200 (likely post-pilot, v0.5+). 3PL is permanently out of scope (no guessing in voice role-plays). When 2PL is adopted, fit it offline (E-M or MML); the online θ update generalizes naturally (the Gaussian-approximation update uses W = a²P(1−P) instead of P(1−P)).

Confidence: 0.80 — the data-volume thresholds are well-established in the psychometric literature; the 1PL-for-v0.3 conclusion is robust to the exact learner count.

---

3. Scenario Library (D-036, D-047)

Q9 — scenarios/index.yaml contents and scenario versioning

Directory structure (per D-036):

scenarios/
  index.yaml                                   # manifest / catalog
  customer_service/
    cs_refund_ca_v01.yaml                      # expert-authored
    cs_refund_exchange_v01.yaml                # expert-authored
    cs_complaint_escalation_v01.yaml           # expert-authored
    ...
    _pending/                                  # AI variations awaiting review
      cs_refund_exchange_ai01.yaml
  cost_rates.yaml                              # existing v0.1 file
  rubric_criteria/                             # optional: shared criterion defs
    empathy.yaml
paths/
  customer_service.yaml                        # the 6-week path (D-037)
rubrics/
  customer_service.yaml                        # the rubric (D-039)

The existing scenarios/customer_service_refund_ca_v01.yaml is currently at the top level (flat); v0.3 nests it under scenarios/customer_service/ to support the multi-path library. The flat layout worked for v0.1's single scenario; the nested layout is needed for v0.3's ≥6 scenarios across (initially) one path and (later) multiple paths.

index.yaml contents (the manifest). The index is a catalog, not a duplicate of scenario content. It carries the metadata the scenario selector and coverage checker need without loading every YAML file:

# scenarios/index.yaml — manifest, regenerated on library changes
version: 1
path_scenarios:
  customer_service:
    - id: cs_refund_ca_v01
      file: customer_service/cs_refund_ca_v01.yaml
      difficulty_expert: 1            # 1-5 expert rating (cold-start b)
      difficulty_calibrated: 0.4      # IRT b in logits, null until calibrated
      failure_mode: escalates_unresolved
      rubric_criteria: [empathy, concrete_resolution, next_steps]
      tags: [refund, damaged_product, ca_market]
      irt_target_p: 0.5               # gate scenario → max info
      version: 1.0.0
      author: expert_jane_doe
      generated_from: null            # null = expert-authored; <parent_id> = AI variation
      intent_hash: <sha256 of success_criteria+failure_mode+rubric_criteria>
      status: live                    # live | pending | deprecated
    - id: cs_refund_exchange_ai01
      file: customer_service/cs_refund_exchange_ai01.yaml
      ...
      generated_from: cs_refund_ca_v01
      status: pending                 # in _pending/, not selectable

Why index.yaml is separate from per-scenario YAMLs. Loading 6+ full scenario YAMLs (each with multi-paragraph system prompts, branch definitions, rubric mappings) just to pick the next one is wasteful. The index is a slim catalog (~50 lines per scenario) loaded once at startup; the full scenario YAML is loaded on demand when selected. This also keeps the selector's logic testable without the LLM-prompt content.

Versioning. Use semver MAJOR.MINOR.PATCH per scenario, recorded in the scenario YAML and mirrored in index.yaml:

• MAJOR: changes that break scoring compatibility — rubric_criteria added/removed, branch-structure changes, success_criteria semantics change. A MAJOR bump invalidates prior mastery-gate evidence (the learner's prior passes on the old version don't count toward the new version's gate).
• MINOR: content additions — new common_mistakes, new branch (non-scoring), prompt enrichment. Backward-compatible with prior scoring.
• PATCH: prompt tweaks, typo fixes, voice_id changes. No semantic change.

The version field on each scenario lets the mastery-gate audit log (REQ-NFR-MAST-02) record which scenario version a learner passed, so future re-authoring doesn't retroactively invalidate credentials.

Recommendation (D-036):

• Nest scenarios under scenarios/<path>/.
• index.yaml is a slim manifest (metadata only, ~50 lines/scenario).
• Per-scenario YAML is the full Pipecat-flows DSL, loaded on demand.
• Semver per scenario; MAJOR bumps invalidate prior gate evidence.
• Add a regenerate_index.py (or bats check) that re-derives index.yaml from the scenario files and asserts they're in sync — prevents manual drift.

Confidence: 0.85 — the index/manifest split is a standard content-management pattern; the semver scheme is conventional. The only judgment call is treating rubric_criteria changes as MAJOR (scoring-compatibility-breaking), which is the conservative choice.

Q10 — AI-generated variations: review workflow, generated_from backref, drift prevention

The workflow (per D-047, C-7). C-7 (binding constraint) states "Scenarios authored by domain experts + learning designers; AI generates variations only." D-047 specifies "AI-generated variations gated by expert review." The concrete workflow:

1. GENERATE
   - Input: an expert scenario YAML (e.g., cs_refund_ca_v01.yaml)
   - LLM (deepseek-v4-flash:cloud with think mode — offline, not latency-bound)
     generates a variation by perturbing the scenario while preserving
     success_criteria + failure_mode + rubric_criteria.
   - Output: a new YAML in scenarios/<path>/_pending/<id>.yaml with:
       generated_from: cs_refund_ca_v01
       intent_hash: <sha256 of parent's success_criteria+failure_mode+rubric_criteria>
       status: pending
       author: ai_variation_<model_version>

2. REVIEW (expert, human-in-the-loop)
   - Expert opens a PR-style diff: pending YAML vs parent YAML.
   - Expert checks: does the variation still exercise the same rubric_criteria?
     Is the failure_mode still reachable? Is the system_prompt safe + in-character?
   - Expert may edit the variation (the LLM output is a draft, not final).
   - On approval: expert moves the file from _pending/ to scenarios/<path>/
     and adds it to index.yaml with status: live.

3. PUBLISH
   - The variation is now selectable by the IRT scenario selector.
   - It carries generated_from permanently (for provenance/audit).
   - Its intent_hash is frozen at generation time.

4. DRIFT DETECTION (ongoing)
   - If the parent scenario is re-authored (MAJOR version bump) and its
     success_criteria/failure_mode/rubric_criteria change, the parent's
     intent_hash changes. All variations generated_from that parent are
     flagged as stale (their intent_hash no longer matches the parent).
   - Stale variations are moved back to _pending/ and require re-review
     before they're selectable again.

The generated_from backref. A single field on the variation YAML pointing to the parent scenario ID. Absent (or null) on expert-authored scenarios. This is the provenance chain — it lets the audit log answer "was this mastery-gate evidence collected on an expert scenario or an AI variation, and if the latter, from which expert scenario was it derived?" The chain is one level deep (an AI variation is generated from an expert scenario, not from another AI variation) — this is a deliberate constraint to prevent variation-of-variation drift. Enforce it at generation time.

Drift prevention via intent_hash. The intent of a scenario is defined as the tuple (success_criteria, failure_mode, rubric_criteria) — the parts that determine what the scenario assesses. The intent_hash is SHA-256 of the canonical JSON encoding of that tuple. At generation time, the variation records the parent's intent_hash. If the parent's intent later changes (re-authoring changes the rubric_criteria, say), the parent's hash changes and the variation is flagged stale. This catches the case where an expert reauthors the parent in a way that the variation no longer faithfully represents — without requiring the expert to manually track all variations.

Preventing drift from the expert's intent (the deeper question). The intent_hash catches parent-side drift. Variation-side drift — the LLM produces a variation that superficially matches the schema but subtly changes the assessed skill (e.g., makes the customer less angry, turning an empathy test into a transaction test) — is caught only by expert review. The intent_hash does NOT verify semantic fidelity. Two mitigations:

1. The rubric-to-scenario mapping is part of the intent tuple. If the LLM drops a rubric criterion, the variation's intent_hash differs from the parent's, and the variation is auto-flagged stale (without needing expert review). This catches structural drift.
2. Expert review is the only defense against semantic drift within the same rubric_criteria. No automated check can verify "is this customer still angry enough to test empathy." This is why C-7 makes expert review mandatory, not optional.

Recommendation (D-047, REQ-SCEN-04):

• Ship the _pending/ directory + generated_from backref + intent_hash fields in v0.3.
• AI variations are generated offline by scripts/generate_variation.py (a CLI tool, not in the voice path); output goes to _pending/.
• Expert review is mandatory; no auto-promotion. The review is a git PR against the scenarios/ directory — the expert reviews the YAML diff.
• One-level variation chain only (no variations of variations).
• intent_hash catches structural drift (rubric_criteria change); expert review catches semantic drift.
• The ≥6 expert scenarios in D-047 are the floor; AI variations are supplemental and cannot substitute for the expert floor.

Confidence: 0.80 — the workflow is sound and matches industry practice for AI-assisted content authoring (e.g., how Khanmigo, Duolingo's GPT-4 content pipeline handle AI-generated exercises). The intent_hash mechanism is a Praxis-specific design; it's a reasonable heuristic for structural drift but is not a published technique, hence the 0.80 not 0.95.

Q11 — Rubric-to-scenario mapping: YAML field shape, coverage across a path

The YAML field shape. Each scenario declares which rubric criteria it exercises via a rubric_criteria field — a list of criterion IDs that reference the rubric file (rubrics/customer_service.yaml per D-039):

# scenarios/customer_service/cs_refund_ca_v01.yaml
id: cs_refund_ca_v01
path: customer_service
# ... existing v0.1 fields ...
rubric_criteria:
  - criterion_id: empathy
    weight: 1.0           # relative weight within this scenario (default 1.0)
    evidence_required: true   # must be observed to count toward mastery
  - criterion_id: concrete_resolution
    weight: 1.0
    evidence_required: true
  - criterion_id: next_steps
    weight: 0.5
    evidence_required: false

Two design choices in this shape:

1. List of objects, not a list of strings. Each entry carries a criterion_id (referencing the rubric) plus per-scenario metadata about that criterion (weight within this scenario, whether evidence is required). A bare list of strings (rubric_criteria: [empathy, concrete_resolution, next_steps]) is simpler but loses the per-scenario weighting — and weighting matters because a scenario may exercise one criterion as the primary skill and another as secondary.
2. Reference by ID, not inline. The criterion's full definition (5-level anchors, weight-within-skill) lives in rubrics/customer_service.yaml (per D-039). The scenario references it by ID. This keeps the rubric single-source (a criterion's anchors are defined once) and lets the coverage checker work on IDs without parsing every scenario's full content.

Coverage across a path. D-032's mastery gate requires N=3 distinct-scenario successes. For the gate to be meaningful, the N scenarios must collectively exercise all the rubric's criteria — otherwise a learner could pass the gate by succeeding on scenarios that only test a subset of the skill. The coverage requirement is: every rubric criterion for the path is exercised by ≥ M scenarios, where M ≥ 2 (so there's at least one expert scenario and one alternative — an AI variation or a second expert scenario — to prevent single-scenario gaming).

Coverage check (load-time). Build the rubric-criterion-ID set from rubrics/customer_service.yaml, walk scenarios/index.yaml, and count scenarios per criterion (only status: live scenarios count):

# pseudocode for scripts/check_coverage.py
rubric = yaml.safe_load(open("rubrics/customer_service.yaml"))
required_criteria = {c["id"] for c in rubric["criteria"]}
index = yaml.safe_load(open("scenarios/index.yaml"))
scenarios = [s for s in index["path_scenarios"]["customer_service"]
             if s["status"] == "live"]
coverage = {cid: sum(1 for s in scenarios if cid in s["rubric_criteria"])
            for cid in required_criteria}
under_covered = {cid: n for cid, n in coverage.items() if n < MIN_COVERAGE}
if under_covered:
    fail(f"Coverage gap: {under_covered} — each criterion needs ≥ {MIN_COVERAGE} scenarios")

With MIN_COVERAGE = 2 for v0.3. This runs at CI time and as a pre-merge gate on scenarios/ changes.

Interaction with D-047's ≥6 scenarios. Six expert scenarios × 3 rubric criteria per scenario = 18 criterion-exercise slots. If the rubric has 5 criteria, each needs ≥ 2 scenarios = 10 slots minimum — well within the 18 available, so 6 scenarios is comfortably enough for coverage if the scenarios are authored to distribute across criteria (not all 6 testing only empathy + concrete_resolution). The coverage check catches the case where authoring concentrates on a subset of criteria.

Recommendation (D-036, D-039, D-047):

• rubric_criteria on each scenario is a list of objects: {criterion_id, weight, evidence_required}.
• Criterion definitions live in rubrics/<skill>.yaml (per D-039); scenarios reference by ID.
• Coverage check: every criterion in the path's rubric is exercised by ≥ 2 live scenarios (MIN_COVERAGE = 2 for v0.3).
• scripts/check_coverage.py runs in CI; fails the build on coverage gaps.
• Authoring guidance for the ≥6 expert scenarios: distribute across criteria so no criterion is exercised by only one scenario.

Confidence: 0.85 — the ID-reference pattern is standard content-relationship modeling; the coverage check is a straightforward graph invariant. The MIN_COVERAGE = 2 choice is a v0.3 pragmatic floor (it could be raised to 3 in later milestones for more robust anti-gaming, at the cost of more authoring).

---

Cross-Cutting Recommendations for the PLAN Stage

1. Anonymization pipeline is a single suppression function, swappable for DP later. Design aggregate_cohort(dimensions, window) to return rows with a cell_suppressed column. The k-anonymity suppression is one predicate (COUNT(DISTINCT learner_id) >= 10); a future DP mechanism replaces the predicate with a noise-addition step. The UI and the rest of the pipeline are unchanged.

2. IRT θ update and mastery gate are independent. Don't couple them. The mastery gate (D-032) is rule-based on rubric scores + N=3 distinct scenarios. θ (D-035) is for scenario selection, not for mastery certification. A learner can open a mastery gate before θ is "reliable" by the σ² criterion, and that's correct — the gate is the authoritative mastery signal.

3. Scenario library is the linchpin. Three v0.3 subsystems read from it: the IRT selector (reads difficulty, irt_target_p), the coverage checker (reads rubric_criteria), and the mastery gate (reads status, version, generated_from). Design index.yaml first; the rest follows.

4. Expert authoring is the bottleneck. D-047's ≥6 expert CS scenarios is a content-authoring task, not an engineering task. The PLAN stage should identify the persona (learning designer + domain expert) and the schedule for authoring the 6 scenarios, and treat it as a critical-path dependency for the IRT and mastery-gate slices.

5. Three CI gates for the scenario library:

   • scripts/check_coverage.py — every rubric criterion exercised by ≥ 2 live scenarios.
   • scripts/check_index_sync.py — index.yaml is in sync with the per-scenario YAMLs (no missing entries, no stale entries).
   • scripts/check_intent_hash.py — no live scenario has a stale intent_hash (catches parent-reauthoring drift).

---

Open Questions for the PLAN Stage

1. Cohort view dimensions — exact set. Q2 recommends 2-D views only. Which 2-D views does the operator dashboard expose? Candidate set: path × week (progression), path × outcome (mastery), path × failure_pattern (diagnostics). Confirm with the operator persona (training manager) before PLAN.

2. IRT b recalibration cadence. Q5 recommends nightly E-M recalibration of b once per-scenario response counts exceed ~20. At v0.3's scale (~100 responses per scenario), nightly is overkill — weekly is fine. But the trigger ("recalibrate when count > 20") needs to be in the nightly job, not hardcoded.

3. AI variation generation tooling. Q10 specifies scripts/generate_variation.py as an offline CLI. Does it run locally (expert's laptop) or in the praxis container? Locally is simpler (no LLM-in-production-container concern); the output is a YAML file checked into git. Recommend local.

4. Mastery-gate evidence and scenario versioning. Q9 specifies MAJOR bumps invalidate prior gate evidence. Concretely: if cs_refund_ca_v01 is bumped to cs_refund_ca_v02 with a rubric_criteria change, do learners who passed v01 need to re-pass v02? The conservative answer is yes (re-pass required), but this is a UX/policy decision that the PLAN stage should surface to the product owner.

5. Operator dashboard: θ reporting threshold. Q7 recommends reporting θ only when σ² < 0.2. Should the dashboard show "warming up — N sessions until reliable" for learners below the threshold, or suppress entirely? Showing a count is more useful but leaks information about how few sessions the learner has (a re-identification vector if combined with other cells). Recommend: aggregate the "warming up" count across the cohort (k-anonymized), don't show per-learner.