# Plan: Assessment Assurance Audit (standalone skill, live-API) — v2

Adapts the framework from the external "AI Assessment Evaluator" skill (Two-Lane assurance, AIAS, TVO, Authenticity, Counterfeit Assurance) into ASUCB. No upstream code copied verbatim. v2 incorporates the converged Opus + GPT-5.5 plan review.

## Locked scope decisions

1. **Standalone skill `assurance-audit`, NOT a course-audit mode.** Audit-family but decoupled from `course-audit`/QA (the QA process is considered broken). Do not couple new assurance code to it.
2. **Live Canvas API only. No IMSCC.**
3. **Reuse only generic, pure utilities** (`blooms.py`, the HTML report shell). Do NOT reuse `criterion_evaluator` logic or `extract_assessments` (see Reuse Reality below).
4. **Built for scale, ranked on deterministic signal.** Cross-course ranking uses only deterministic metrics; LLM judgment is within-course annotation, never a ranking axis.
5. **Process:** plan reviewed by Opus + GPT-5.5 (done, v2 reflects it). Sonnet codes. Opus reviews the code (reviewer != coder). PR-only.

## Reuse Reality (corrected after review)

The original plan over-claimed reuse. Verified facts:
- `alignment_validator.extract_assessments` returns `id, title, type, canvas_type, canvas_id, module, points, mlo_ids, clo_ids, has_rubric` — **NOT submission type, NOT grade weight**, and its `points` reads 0 from the stripped `fetch_course_tree` shape. `has_rubric` is also **quiz-blind** (rubric_map keyed off assignment/discussion-topic IDs only). **Do not reuse it for the inventory.**
- **Grade weight (% of final grade) is computed nowhere in the codebase.** It is the headline input to Assurance Coverage and must be built + tested as a first-class deliverable.
- Confirmed reusable: `blooms.classify_blooms/is_measurable`, `canvas_api.{list_assignments,list_quizzes,list_discussions,get_assignment,get_quiz,get_quiz_questions,get_assignment_groups,get_syllabus,get_modules,paginated_get}`, `templates/report-html/report-base.html`, `backlog_manager`, `course-config.json` `course.level/tier`.

## Component 1 — Normalized Assessment Inventory (`scripts/assurance_inventory.py`) — gating deliverable

Build a correct, live, weight-aware inventory. This is the foundation; everything else scores off it.

- **Primary fetch:** `get_assignment_groups(config, include_assignments=True)` — one paginated family returns groups (with `group_weight`), and per-assignment `points_possible`, `submission_types`, `omit_from_final_grade`, `published`/`workflow_state`, `peer_reviews`, `is_quiz_assignment`, `external_tool_tag` in a single call set. Prefer this over N per-item GETs.
- **Grade weight computation (golden-tested):**
  - Read course `apply_assignment_group_weights`.
  - If **true**: per-assessment weight = `group_weight × (points / sum(points of counted items in that group))`.
  - If **false**: per-assessment weight = `points / sum(points of all counted items)`.
  - "Counted" excludes `omit_from_final_grade`, unpublished, and `points_possible == 0` items.
  - Emit `grade_weight_pct` per assessment; weights sum to ~100% (assert + tolerance).
- **Object normalization** (route all graded objects to one record shape):
  - Classic quizzes (`list_quizzes`), New Quizzes (detect via assignment `is_quiz_assignment` + `external_tool` / `submission_types == ["external_tool"]`; if the New Quizzes item bank API is unreachable with a PAT, **flag `engine: new_quizzes_opaque` and degrade gracefully** — score what metadata exists, never crash).
  - Graded discussions: join `list_discussions` topics to their backing assignment (points + weight live on the assignment, not the topic).
  - External-tool / LTI assignments (proctoring, Respondus/Lockdown, publisher tools): capture `external_tool` + any `access_code`/`ip_filter` on quizzes as a Lane 1 / proctoring signal.
  - Skip ungraded/unpublished from weighted scoring (note counts separately).
- **Record shape:** `{id, title, canvas_type, engine, points, grade_weight_pct, submission_types, has_rubric, peer_reviews, omit_from_final_grade, published, module, group, is_external_tool, proctoring_signal}`.
- **Rubric for quizzes:** fetch separately (the alignment rubric_map is quiz-blind).

## Component 2 — Throughput + caching (scale safety) — gating

ASU rate-limits (~700 req/hr/token; `criterion_evaluator` already had a 429-truncation bug). The inventory enrichment is call-heavy across hundreds of courses.
- **Shared token-bucket throttle** + small inter-request delay layered on `paginated_get` (or a thin wrapper); shared across a batch run, not per-course.
- **Persistent cache** keyed by `(course_id, endpoint)` for a run; reuse `course_navigator`'s tree cache; dedupe modules/assignments calls.
- **Prefer `include[]`-batched fetches** over per-item GETs (assignment-groups-with-assignments is the big win; only fall back to per-quiz `get_quiz_questions` for question-type when needed, and only for the top-K weighted items — see Component 4).
- **Per-course call budget** with a logged ceiling; a unit test asserts the inventory build stays under the budget for a representative course.

## Component 3 — Deterministic scoring + tier (the cross-course ranking axis)

All cross-course-comparable output is deterministic.
- **Lane classification (heuristic, deterministic):** proctoring_signal / oral / process-trail keywords + submission_types → Lane 1 candidate; take-home essay / quiz-written with no security property → Lane 2; high `grade_weight_pct` + no security property → **Mismatch**.
- **AIAS presence (deterministic prefilter):** detect whether any AI-use policy exists in syllabus + assignment descriptions (presence only). **AIAS level 1-5 classification is AI-judged** (Component 4), not deterministic. Report `Undeclared` + transparency flag when absent.
- **Assurance Coverage (primary metric):** % of `grade_weight_pct` on Secured Lane 1 anchors. Bands → **named tier enum**, the canonical batch sort key:
  - `Assured` (≥40% on secured anchors, ≥1 anchor) | `Partially Assured` | `Unassured` | `Counterfeit` (high-weight items failing the security test / undeclared at scale).
- **Cognitive demand (deterministic):** `blooms.classify_blooms` per assessment → grade-weighted Bloom distribution + arc + >50%/>70% surface flags.
- **Counterfeit-high-level test:** **deterministic for auto-graded objective items** (MC/TF/matching claiming Analyze+ → counterfeit by type); **AI-judged only for free-response/essay/upload** where type doesn't settle it.

## Component 4 — Bounded AI annotation (within-course only, never ranked)

LLM judgment is diagnostic, capped, and reproducible-as-possible:
- **Scope cap:** only the **top-K assessments by `grade_weight_pct`** (covering ~top 80% of weight) get AI judgment. The long tail is scored deterministically. Record K + the cap in `assumptions`.
- **Judgments:** Gulikers 5DF authenticity (top-K), Substitutability test for Lane 1 / high-weight, FIRE quality for Lane 2, AIAS level classification, evaluative-judgment for AIAS 4-5, counterfeit test for free-response only.
- **Reproducibility:** versioned prompt id + content-hash cache (same assessment text → cached judgment); record model + settings in the JSON. AI scores are labeled **"within-course diagnostic, not cross-course comparable"** in every output.
- **Packaging:** each AI item carries assessment id, name, points, grade_weight, and the specific question, so judgments are auditable (mirrors `criterion_evaluator`'s `needs_ai_review` discipline without importing it).

## Component 5 — Dimensions (final shape)

Three **scored** dimensions + one **pointer**:
- **A — Authenticity Index** (Gulikers 5DF): AI-judged top-K, deterministic tiering. Scored.
- **C — Cognitive Demand**: deterministic Bloom distribution + arc + counterfeit (det for objective, AI for free-response). Scored.
- **D — Assurance & AI-Integration** (headline): deterministic Lane/AIAS-presence/Coverage/tier + bounded AI for Substitutability/FIRE/AIAS-level. Scored. Leads the report.
- **B — Learning Science**: **pointer/deltas-only, NOT a scored dimension.** Defer to `alignment` (Biggs constructive alignment) + the HPI/ID standard; report only assurance-specific deltas (e.g., high-weight summative with no formative practice loop = the terminal-performance test). Avoids duplicating the learning-science corpus and the "uncomputable on bare courses (0 CLOs)" failure.

## Component 6 — Separate batch runner (no QA coupling)

- Own batch path: generic registry fan-out (Agent subagents over `courses/registry.json`) + **own aggregate template** (`courses/reports/{ts}-batch-assurance.html`). Do NOT route through `course-audit`'s batch code / `audit_report.py` aggregator.
- Add `assurance-audit` to §Batch Execution eligible list as a read-only skill, but implement its own runner.
- **Reject** seeding Lane/AIAS heuristics into `criterion_evaluator` (would re-couple to broken QA). If a future course-audit rework wants them, it imports from `assurance-audit`, not the reverse.

## Component 7 — Skill + registration

- `skills/assurance-audit/SKILL.md` + `references/{assurance-model.md, authenticity-framework.md, learning-science-lens.md}` (ASUCB-tuned, no IMSCC; learning-science ref is pointer-heavy per Dimension B).
- Routing in CLAUDE.md + AGENTS.md: "AI-proof / assessment assurance / can students bypass with AI / AIAS / authentic assessment" → `assurance-audit`, explicitly separate from `course-audit`.
- Context defaults from `course-config.json` (level/tier) + course code/name; modality default async; AI policy from syllabus; model assumption default Conservative. Single-course interactive MAY refine; **batch never stops to ask** — defaults recorded in JSON `assumptions`.

## Scale output (the goal: good/bad at scale)
- **Per-course JSON scorecard** (stable, golden-tested schema): tier enum + Assurance Coverage + A/C/D grades + flags + durable-skills + assumptions + AI-prompt-version. Deterministic fields marked comparable; AI fields marked diagnostic-only.
- **Batch aggregate** ranked by the deterministic **tier enum** then Assurance Coverage; surfaces the cohort certifying-on-nothing.

## Tests (`tests/test_assurance_audit.py` + `tests/test_assurance_inventory.py`)
- **Grade-weight golden test:** weighted-groups course vs points-only course produce correctly different per-assessment weights; weights sum ~100%; omit/unpublished/zero-point excluded.
- Object normalization: graded discussion joins to backing assignment; New Quizzes opaque-degrade; external-tool proctoring signal captured.
- Deterministic Lane/AIAS-presence/Coverage/tier math; Bloom distribution + flags; counterfeit deterministic-for-MC.
- Throughput: per-course call-budget assertion; cache dedupes repeat fetches.
- AI packaging: top-K cap honored; each item carries id/points/weight/question; prompt-version + content-hash cache key present.
- Determinism firewall: cross-course ranking fields are deterministic-only (assert no AI field feeds tier/Coverage).
- No IMSCC code path exists (assert no zip/imscc handling).
- Separation: no import of `criterion_evaluator`; routing rule present in CLAUDE.md + AGENTS.md; own batch aggregate template.
- Full `pytest tests/` green.

## Acceptance
- Runs on a live course (id/URL), no IMSCC; produces 4-section report (D leads) + per-course JSON led by tier + Assurance Coverage; every finding cites a named assessment + point value + grade weight.
- Grade-weight math correct for both weighting modes (golden-tested).
- Cross-course ranking deterministic; AI judgment bounded to top-K, cached, versioned, labeled non-comparable.
- Decoupled from `course-audit`/`criterion_evaluator`; own batch runner; stays under the per-course call budget.
- No surveillance framing; recommendations = secure/re-lane/declare/liberate.
- Full suite green; PR-only; Opus reviews Sonnet's code.

## Phase 2 (opt-in, separate PR)
ASU-branded leadership PDF + Tufte exec deck (compose `asu-brand-visual-identity` + `tufte-exec-presentations` + `pptx` + `pdf`). Renders existing JSON; never re-scores.

## Resolved review questions
- Naming → `assurance-audit`.
- Dimension B → pointer/deltas-only (not scored).
- Heuristics location → `assurance_audit.py` only; never `criterion_evaluator`.

The skill stays fully decoupled from course-audit/QA and carries no QA-status commentary or remediation hooks for the QA process itself. Assurance findings are about the course's assessment strategy only.
