# Plan: Assurance by Design (wire the assurance research into the assessment generators)

Status: PROPOSED (scoping only; no code written yet)
Author: Opus (with Brent)
Date: 2026-07-03

## Problem (verified against the repo)

The `assurance-audit` skill is research-backed: `skills/assurance-audit/references/` holds the AI-Forward framework (D1-D6), the Assurance Model (Two-Lane, FIRE, AIAS 1-5, Substitutability, coverage tiers), and the Authenticity framework (Gulikers 5DF), grounded in Bloom, Gilbert BEM, Mager, TILT, cognitive load, SDT.

That research does NOT reach the assessment generators. Verified:
- `quiz-generator`, `assignment-generator`, `discussion-generator`, `needs-analysis` cite `standards/instructional-design.md` (HPI/ID standard) + `standards/assessment-best-practices.md` (NBME item writing, distractors, Bloom, tier system, rubric design). Solid pedagogy.
- None reference the assurance frameworks. The only "assurance" hit in a generator was `course-build`'s "## Phase 6: Quality Assurance" heading (generic QA, not assessment-assurance).
- `standards/instructional-design.md` and `standards/assessment-best-practices.md` contain ZERO mention of substitutability, AI-forward, AI-resistant, AIAS, or "can students bypass with AI."

Consequence: when a user uploads material and asks "suggest assessments," the output is pedagogically sound but AI-blind. It can be born as exactly the AI-substitutable recall quiz the assurance audit would later flag. The lens runs downstream (audit), never upstream (design).

## Goal

Assessments are designed AI-aware by default. When a generator (or the needs-analysis / course-build strategy step) proposes or builds an assessment, it applies the assurance lens: prefer designs where a real student's work can be told from AI's, that measure real thinking, and that build AI-use judgment. Substitutability is flagged at design time, with the AI-resistant and AI-forward move offered, not discovered in a later audit.

## Non-goals (scope boundaries)

- NOT a full assurance scorecard at generation time. No tier, no coverage %, no `needs_ai_review`. This is prescriptive DESIGN GUIDANCE plus a lightweight self-check, not an embedded audit.
- Does NOT push proctoring. Proctoring / oral / synchronous stay last-resort (weak vs current AI, unscalable, student barriers) per the program stance; the default moves are process evidence, authentic tasks, and judgment-of-AI.
- Does NOT block or gate generation. If the assurance-preferred move does not fit the material or the faculty constraint, the generator proceeds and notes why in one line.
- Does NOT duplicate the four audit references into each generator. Single shared source.

## Reuse: the Assessment Design Playbook is already the content

`scripts/assessment_design_playbook.py` `_CATEGORIES` already encodes, for five categories, the exact design-time guidance the generators need:

| Playbook category | What it carries | Generator it maps to |
|---|---|---|
| Quizzes and Exams | vulnerable / AI-resistant moves / AI-forward moves / technologies / structural anchor | `quiz-generator`, `course-fit-quiz` |
| Papers and Written Assignments | same shape | `assignment-generator` |
| Discussions | same shape | `discussion-generator` |
| Projects | same shape | `assignment-generator`, `interactive-content` |
| Presentations and Oral Assessments | same shape | `assignment-generator` |

Today that content is locked inside a Python script (the playbook is a render-only deliverable). The plan surfaces it as a shared standard both the playbook and the generators read.

## Design

Mirror the existing HPI/ID standard seam (a CLAUDE.md Operating-Standard clause that auto-applies + a per-skill header hook + a `standards/` doc).

### 1. New shared standard: `standards/assessment-assurance-design.md`
Canonical, generation-facing distillation of the assurance research, organized as DESIGN MOVES (not audit criteria). Sections:
- **The two design questions** (plain): can a real student's work be told from AI's? does it measure real thinking and build AI-use judgment?
- **Design levers** (from AI-Forward D1-D6 + Two-Lane + Gulikers, condensed): secure at least one anchor for the grade (Two-Lane); capture process / authorship evidence (D5); grade judgment-of-AI where AI is allowed (D4); declare task-level AI use, AIAS 1-5 (D1); bind to authentic, course-specific evidence (Gulikers 5DF); align Bloom to the outcome, avoid counterfeit demand (D2); keep it equitable and 7.5-week-feasible (D6).
- **TILT pairing rule**: any "verify / critique / improve the AI" move ships with the rubric AND two or three annotated model examples up front (Winkelmes), else it penalizes students who do not already know the target.
- **Per-category design moves**: the five playbook categories (vulnerable / AI-resistant / AI-forward / anchor), authored once here.
- **The by-design self-check** (3 lines a generator runs before returning): (a) Could a student produce this graded artifact with AI and have it look the same? If yes, add a process or authenticity move or say why not. (b) Does it measure the intended thinking, or just recall dressed up? (c) Is task-level AI use stated? House style: APA caps, stop-slop, no em-dashes.

### 2. NO playbook refactor (decided: simple > complexity)
The new standard is a LEAN, generation-facing distillation (design directives + the 3-line self-check), NOT a copy of the playbook. The playbook (`assessment_design_playbook.py`) stays as-is: it is the fuller, faculty-facing deliverable, a different artifact for a different reader. Because the two serve different purposes and both change rarely, we accept the small overlap in the five-category moves rather than build a markdown-parsing refactor for a speculative dedup. Drop a one-line "keep roughly in sync with `standards/assessment-assurance-design.md`" comment near `_CATEGORIES`. Merge the two only if they ever actually drift (YAGNI). This removes PR 2, the markdown parser, and the snapshot-test work.

### 3. CLAUDE.md Operating-Standard clause: "Assurance by Design"
Add a short clause to `CLAUDE.md` (parallel to the HPI/ID practitioner standard) stating that every skill that proposes or builds a graded assessment applies `standards/assessment-assurance-design.md` automatically: prefer AI-resistant / AI-forward designs, run the by-design self-check, and offer (not force) the resistant/forward move. Exempt purely mechanical skills. Update `AGENTS.md` to match (the always-on condensed contract).

### 4. Per-generator hooks
Each targeted generator's `SKILL.md` gets (a) a one-line header hook mirroring the HPI/ID line, pointing at `standards/assessment-assurance-design.md`, and (b) one workflow step: "Assurance by design: apply the moves for this assessment type; run the by-design self-check before returning; when the AI-resistant move does not fit, state why in one line."

## Scope per file

| File | Change | Size |
|---|---|---|
| `standards/assessment-assurance-design.md` | NEW: the shared design-for-assurance standard (levers + 5-category moves + self-check + TILT rule) | new doc |
| `CLAUDE.md` | ADD Operating-Standard clause "Assurance by Design" + note in the routing/standards sections | small |
| `AGENTS.md` | Mirror the clause | small |
| `skills/quiz-generator/SKILL.md` | Header hook + assurance-by-design step (Quizzes/Exams moves; default to application-from-pool, course-bound items; note unproctored auto-graded recall is a weak anchor; offer verify-the-AI variant + TILT) | small |
| `skills/assignment-generator/SKILL.md` | Header hook + step (Papers/Projects moves; default to process evidence + authentic/personal binding; AI-use appendix graded for judgment, paired with TILT models) | small |
| `skills/discussion-generator/SKILL.md` | Header hook + step (Discussions moves; cohort-specific references, audio/video identity option, critique-an-AI-answer variant) | small |
| `skills/rubric-creator/SKILL.md` | Header hook + step (add criteria that reward process/authorship and judgment-of-AI when the assignment is AI-forward; pair verify-the-AI with model-example criteria) | small |
| `skills/interactive-content/SKILL.md` | Light note: interactives are formative, not the graded anchor; if made gradable, apply the standard | tiny |
| `skills/needs-analysis/SKILL.md` | Step in the assessment-strategy output: ensure the proposed architecture secures at least one anchor for the grade (Two-Lane) and plans task-level AI declaration; this is the UPSTREAM insertion, arguably highest-leverage | small |
| `skills/course-build/SKILL.md` | Its assessment-architecture phase applies the same standard | small |
| `scripts/assessment_design_playbook.py` | One-line "keep roughly in sync with `standards/assessment-assurance-design.md`" comment near `_CATEGORIES`. NO refactor (Decision 1). | tiny |
| `tests/` | Structural tests (below) | small |

## Entry points covered

"Suggest assessments for this [material]" has three real entry points; all get the lens:
1. **Strategy** (`needs-analysis`, `course-build` assessment architecture) — designs the mix AI-aware (secure an anchor, plan AI declaration). Upstream, highest leverage.
2. **Instrument** (`quiz-generator`, `assignment-generator`, `discussion-generator`, `rubric-creator`) — builds each item AI-aware.
3. **Formative** (`interactive-content`, `course-fit-quiz`) — light touch (not the graded anchor).

## Acceptance criteria

1. `standards/assessment-assurance-design.md` exists: two design questions, the levers (mapped to D1-D6 / Two-Lane / 5DF), the TILT rule, the five per-category move sets, and the 3-line self-check. No em-dashes; APA caps; stop-slop.
2. `CLAUDE.md` (and `AGENTS.md`) carry the "Assurance by Design" clause, auto-applied to assessment-proposing/building skills, exempting mechanical ones.
3. Each targeted `SKILL.md` references the standard in a header hook and has one assurance-by-design workflow step.
4. Behavioral validation (manual, not a unit test): generate an assessment for a sample upload BEFORE and AFTER the change, run `assurance-audit` on both; the AFTER assessment should score better on AI-forward dimensions (D1/D4/D5) and be less substitutable. Record the delta in the PR.
5. `assessment_design_playbook.py` is unchanged except a one-line sync comment near `_CATEGORIES` (no refactor, Decision 1).

## Tests

Structural (generation is Claude-driven, so tests assert wiring, not output quality):
- `tests/test_assurance_design_standard.py`: the standard file exists; contains the required section anchors; has no em/en dashes; lists all five categories. Same file also asserts each targeted `SKILL.md` contains the header hook string and the standard path (one new test file, not scattered edits).

## Rollout

- ONE PR: the lean standard + CLAUDE.md/AGENTS.md clause + the SKILL.md hooks + structural tests. Independently reviewed (coder != reviewer). Delivers born-AI-aware generation. No PR 2 (playbook refactor dropped, see Design section 2).
- Semver: `feat` (new standard + behavior guidance) -> minor bump.

## Risks and boundaries

- **Guidance bloat**: keep each generator's addition to one hook line + one step. The depth lives in the shared standard, read on demand.
- **Drift** between the playbook and the standard until Stage 2 lands: mitigate by authoring the standard as the source and having the playbook cite it in a comment until refactored.
- **Over-prescription**: the self-check offers the move and accepts a one-line "does not fit because ..." It never blocks generation or forces proctoring.
- **The seam is narrow**: only skills that touch GRADED assessments. Mechanical skills (staging, media-upload, canvas-nav, images, visual-builder) are exempt, same as the HPI/ID standard.

## Decisions

1. Playbook dedupe: **RESOLVED, do not refactor.** Lean standard coexists with the playbook; no PR 2 (see Design section 2). Rationale: simple > complexity.
2. Self-check depth: **recommended = the light 3-line version** (adds negligible length; a fuller checklist taxes every generation for little gain).
3. `needs-analysis` in PR 1: **recommended = yes, include it.** It is the same thin hook + step as the generators (not added machinery) and it is the highest-leverage insertion point (designs the assessment MIX AI-aware, not just each item). Leaving it out is a false economy. So one PR covers the generators + needs-analysis + course-build.

(2 and 3 are recommendations; confirm or override.)
