# Plan: Draw the deterministic/AI classification boundary in the assurance audit

## Problem

Every classification bug found in the assurance audit (2026-06-29 session) is the same
mistake in a different place: **a regex/keyword match on prose was given authority to set
a headline number.** Regex fails both ways on natural language — it misses (different
wording) and false-matches (substring, or right-word/wrong-meaning).

Evidence:
- `_classify_lane` (assurance_audit.py): substring-matched `'oral'` inside "behavioral"
  and `'presentation'` inside a discussion's "physical presentation" prompt topic, and
  promoted a 200-word text discussion (COM 100 / 259115, id 7385654) to a secured Lane 1
  anchor. That flipped the deterministic tier Unassured → Partially Assured on a single
  false anchor. Its weight (1.71%) was below the `lane_review` threshold, so the false
  positive fed the headline silently.
- `_attach_bloom` (assurance_inventory.py): an auto-graded MCQ quiz (Quiz: Chapter 1/3)
  inherited the **Create** Bloom level from a Module 1 objective ("Formulate your own
  communication ethic"). A multiple-choice item cannot measure Create; the deterministic
  Cognitive-Demand distribution showed 2.3% Create that is an inheritance artifact.
- `_detect_aias_presence` (assurance_audit.py): historically matched substrings like
  `prompt`/`chatgpt` (fixed in #172, but the keyword list still needs word-boundary hygiene).

## The invariant (one principle, applied everywhere)

The audit already separates deterministic facts from AI judgment (`needs_ai_review` +
Claude inline). The bugs come from applying it inconsistently. Fix the boundary with three
rules:

1. **Deterministic logic only over structured Canvas metadata.** `submission_types`,
   proctoring config (access_code / ip_filter / proctoring LTI tool id), points, grade
   weight, `has_rubric`, `peer_review`, engine type. These are API facts. Equality checks
   on them are correct and safe.

2. **Prose keyword hits are advisory, never authoritative.** A keyword found in a title,
   description, or prompt may *nominate* an item for AI review or *lower* confidence. It may
   never, on its own, *promote* an item to a higher-assurance state (Lane 1 anchor,
   "secured", a higher Bloom level than the instrument can measure). Promotion comes only
   from structured metadata or AI confirmation.

3. **Conservative default: uncertainty resolves to the safer state** (Lane 2, Unknown), and
   the deterministic headline counts only structurally-secured weight. An audit that
   under-claims security is recoverable; one that over-claims it gives false comfort about
   AI bypass, which is the worse error for an assurance instrument.

This is a hybrid, not a rewrite. Regex/keywords stay as prefilters and over structured
fields. We remove their authority to over-promote. No embeddings, no LLM client in the
Python scripts — the AI layer is the existing Claude-inline `needs_ai_review` path.

## Acceptance criteria

### A. Lane classification — `assurance_audit.py::_classify_lane` + `score_course`
1. Deterministic **Lane 1 ("secured")** is set ONLY by structured metadata:
   `proctoring_signal is True` OR `peer_reviews is True`. Nothing else promotes to Lane 1
   in the deterministic path.
2. Oral / process-trail KEYWORDS (`_ORAL_KEYWORDS`, `_PROCESS_TRAIL_KEYWORDS`) in title or
   `content_excerpt` no longer return `"lane1"`. Instead they mark the item as a
   **lane1_candidate** and emit a `lane_review` `needs_ai_review` item **regardless of grade
   weight** (remove the `HIGH_WEIGHT_THRESHOLD` gate for keyword-nominated candidates). Until
   AI confirms, the item counts as Lane 2 for the deterministic tier/coverage.
3. All keyword scans use **word-boundary** matching (`re.search(r"\b"+re.escape(kw)+r"\b", text)`),
   not substring `in`. (Hygiene; multi-word keywords still work.)
4. `assurance_coverage_pct` and `tier` count only structurally-secured Lane 1 weight (rule 1).
   AI-confirmed anchors are reported in the narrative as within-course annotations, never folded
   into the deterministic coverage number (preserves cross-course comparability).
5. Regression tests: COM 100 Module 2 Discussion excerpt ("behavioral" / "physical
   presentation") → Lane 2 + a `lane_review` item emitted. A proctored quiz (access_code) →
   Lane 1. A peer-reviewed assignment → Lane 1. A title "Oral Presentation" → Lane 2 deterministic
   + `lane_review` emitted (not auto-secured).

### B. Bloom inheritance cap — `assurance_inventory.py::_attach_bloom`
1. An assessment that is **auto-graded objective** (engine `classic_quiz`/`new_quizzes` with
   objective submission, no rubric) may inherit at most **Understand** from its module MLO.
   If the module MLO level is Analyze+ (index ≥ 4), do NOT stamp that level deterministically:
   cap the inherited level at Understand AND emit a `counterfeit_objective_mlo` (or
   `bloom_content`) `needs_ai_review` item so Claude can confirm whether the item truly reaches
   the higher level.
2. A clear **title verb** still wins deterministically (a title is a deliberate naming signal,
   short and assessment-scoped) — unchanged. Only MLO *inheritance* onto objective items is capped.
3. The deterministic Cognitive-Demand distribution must not show a Bloom level above what the
   instrument can measure purely from MLO inheritance. (COM 100: the 2.3% "Create" on the
   Chapter 1/3 MCQ quizzes disappears from the deterministic Create bucket; those items route to
   review or cap at Understand.)
4. Regression test: an auto-graded quiz in a module whose MLO is Create/Analyze inherits at most
   Understand and emits a review item; a rubric-scored upload in the same module is unaffected.

### C. AIAS keyword hygiene — `assurance_audit.py::_detect_aias_presence`
1. Word-boundary matching on `_AIAS_POLICY_KEYWORDS` (prevents short tokens like `llm`/`aias`
   matching inside other words).
2. `declared_presence` stays deterministic but is documented as "a candidate policy span was
   found"; the policy LEVEL and task-vs-syllabus scope remain AI-judged (already true post-#172).
   `declared_presence` must feed no deterministic headline number (confirm it does not affect
   `tier`/`coverage`).
3. Regression test: a syllabus prose mention that merely contains a keyword inside another word
   does not set `declared_presence`; a real "Use of AI" policy sentence does, and carries its
   `policy_excerpt`.

### Cross-cutting
- A shared `_keyword_hit(text, keywords)` helper doing word-boundary matching, reused by lane and
  AIAS scans (one place to get the matching right).
- No change to tier thresholds, the weighting math, or the `needs_ai_review` schema beyond the
  new `lane_review`/`bloom_content` emissions described above.

## Out of scope
- Embeddings or a trained classifier (the existing Claude-inline layer covers semantics; adding a
  second ML system is unjustified at this volume).
- Any LLM API client inside the Python scripts.
- Changing which fields are cross-course-comparable (deterministic fields only — unchanged).

## Verification
- Full `pytest tests/ -q` green on a clean checkout.
- 259115 re-run: all 8 discussions classify Lane 2; tier = Unassured, coverage = 0%; no false
  anchor; the Chapter 1/3 quizzes no longer contribute deterministic Create weight; AIAS presence
  unchanged (real policy still detected).
- Read the diff by hand before merge. Coder ≠ reviewer.
