# Plan: Assurance audit — stop classifying from titles, read content

## Problem (observed on COM 100 / course 259115, 2026-06-29)

The assurance pipeline classifies its most consequential signals from **titles and
keyword lists**, never from assessment content. When titles are generic (the normal
case), it silently degrades to Unknown / Lane 2 / undeclared and reports that
degradation as a finding about the course.

Concrete failures seen today:
- `objective_extractor._MLO_RE` requires a literal `MLO N.N` prefix → COM 100's
  "Learning Objectives" bullet lists yielded **zero** MLOs → 100% Unknown Bloom.
- `_attach_bloom` (title-verb regex → MLO → Unknown) never opens the assessment →
  37/37 assessments scored Unknown despite richly-objectived modules.
- `_detect_aias_presence` keyword match includes `"prompt"`/`"chatgpt"` → false-positive
  prone; misses policies worded without the magic words.
- `_classify_lane` / `_detect_proctoring` key off title keywords + LTI-URL keywords →
  account-level proctoring LTI (Honorlock/RPNow) is invisible → exams asserted
  "unproctored" with no way to tell unknown from absent.

## Architecture constraint (do NOT violate)

The deterministic scripts produce the scorecard + `needs_ai_review` items; **Claude
does the AI judgment inline** (Step 3 of the skill). So "read the content with AI"
means: **the script fetches and ATTACHES content; Claude reads it inline.** Do NOT
add an LLM API client to the Python scripts. The content is mostly already fetched
(`quiz_detail`, `discussion_topic`, assignment `description`, syllabus) and then
discarded — attach it instead.

## Acceptance criteria

### A. MLO extraction fallback — `objective_extractor.py`
1. Primary path (literal `MLO N.N`) unchanged.
2. When zero `MLO N.N` lines are found on an overview page, fall back to: locate a
   heading matching `/(module )?learning objectives?|module objectives|outcomes/i`
   and harvest the `<li>` items that follow (reuse the CLO heading-window helper).
3. Skip "Self-Check" / "Self Check" sub-lists (logistics, not objectives).
4. Verify on 259115: `module-1-overview`..`module-7-overview-2` each yield 8–14
   objectives carrying Bloom verbs (Explain/Describe/Identify/Define/Formulate/Discuss).
5. Unit test with a fixture page using the `Learning Objectives` + chapter-subheading
   + `<li>` structure, asserting >0 MLOs and a non-zero bloom_index.

### B. Content-grounded Bloom — `assurance_inventory.py`
1. `_attach_bloom` keeps deterministic title→MLO grounding.
2. When the result is `Unknown`, attach the assessment's available content to the
   record under `content_excerpt` (quiz question stems if fetched, else assignment
   `description`, else discussion `message`; strip HTML, cap ~2000 chars).
3. `assurance_audit.score_course` emits a `bloom_content` item in `needs_ai_review`
   for each still-Unknown, non-zero-weight assessment, carrying `content_excerpt`,
   so Claude classifies Bloom inline from real text.
4. `bloom_source` gains a `content_ai` value (set by Claude's inline layer, documented
   in the skill — the script only emits the item; it does not invent the level).
5. Quiz question stems: fetch `/quizzes/{id}/questions` for classic quizzes whose
   title is Bloom-Unknown (bounded; skip if >N questions, take first ~10 stems).
   Record `per_course_api_calls` honestly.

### C. Content-grounded AIAS — `assurance_audit.py`
1. Keyword list stays as a **prefilter to FIND candidate text**, not as the verdict.
2. Remove `"prompt"` and `"chatgpt"` from the standalone presence triggers (too
   false-positive prone); they may still flag a candidate span for Claude to read.
3. The `aias_level` needs_ai_review item carries the matched policy text span(s)
   (`policy_excerpt`), so Claude classifies AIAS 1–5 from the real wording.
4. `declared_presence` stays deterministic but documents that level + task-vs-syllabus
   scope are Claude's inline read.

### D. Content-grounded lane / honest proctoring — both scripts
1. `_detect_proctoring` returns a tri-state: `True` / `False` / `"unknown"`.
   `"unknown"` when an exam/high-weight quiz has no detectable signal AND ASU
   account-level proctoring LTI cannot be confirmed via API. (Keep `True`/`False`
   for the cases it CAN confirm: access_code/ip_filter/LTI-URL keyword.)
2. `_classify_lane` scans the assignment `description` (not just the title) for
   explicit auth mechanisms (record/video/oral/draft history/process log/live).
3. For ambiguous high-weight items, emit a `lane_review` needs_ai_review item with
   `description_excerpt` so Claude can confirm the lane from the body.
4. The report states "proctoring unknown — confirm" for `"unknown"` items rather than
   asserting "unproctored". Scorecard must not count `"unknown"` as a secured anchor.

### E. Dedup the AI fan-out — `assurance_audit.py`
1. Before emitting `needs_ai_review`, cluster structurally-identical assessments
   (same canvas_type + submission_types + has_rubric + same title template after
   stripping a trailing index/chapter token, e.g. "Quiz: Chapter N", "SUBMIT: ChN
   ... Short Paper").
2. Emit one representative item per (cluster × judgment_type); record `cluster_size`
   and `cluster_members` so the inline verdict applies to all members.
3. Net effect on 259115: ~92 items → roughly ≤ 30. Assert the reduction in a test.

## Out of scope
- No LLM API client in Python.
- No change to the deterministic `tier` / `assurance_coverage_pct` math (B–E feed the
  AI layer + reporting, not the headline arithmetic — except D, which only PREVENTS
  an `"unknown"` from being miscounted as secured; it never invents an anchor).

## Verification
- Full `pytest tests/ -q` green on a clean checkout.
- Re-run inventory+score on 259115: Bloom no longer 100% Unknown; proctoring on the
  two exams reads `"unknown"` not `False`; needs_ai_review materially smaller.
- Read the diff by hand (not the coder's summary) before merge. Coder ≠ reviewer.
