# Assessment Best Practices

> **Plugin Reference** — This document is part of the ASU Canvas Builder plugin.
> Evidence-based assessment design standards for higher education courses.
> Referenced by: `quiz-generator`, `assignment-generator`, `rubric-creator`, `course-audit` skills.
> **Note**: For proctored exams, see the `proctoring` section in `templates/course-config.json` and the CRC Category 8 checks in `standards/canvas-standards.md`.

## 1. NBME-STYLE QUESTION WRITING

### Clinical Vignette Structure (Standard Order)
> This format is standard for health sciences courses. For other disciplines, adapt the vignette structure to use discipline-appropriate scenarios (case studies, situational prompts, data sets, etc.).

1. **Context/demographics**: Relevant background (e.g., "A 52-year-old woman" or "A small business owner")
2. **Situation**: Where/when presenting (e.g., "comes to the emergency department" or "contacts your consulting firm")
3. **Primary issue**: Reason for the scenario with relevant timeframe
4. **Supporting details**: Quality, severity, timing, associated factors
5. **Relevant history** (optional): Background information pertinent to the case
6. **Data/observations**: Key findings, measurements, or evidence
7. **Additional data** (optional): Lab values, metrics, survey results, financial data
8. **Context update** (optional): What has been done, how situation has evolved

### Lead-in (Stem) Rules
- MUST end with a question mark
- Should be answerable from the stem alone (before seeing options)
- Avoid negative phrasing ("Which of the following is NOT...")
- Avoid vague phrasing ("Which of the following is true about...")
- Use focused lead-ins:
  - "Which of the following is the most likely explanation?"
  - "Which of the following is the most likely underlying mechanism?"
  - "Which of the following is the most appropriate next step?"
  - "Which of the following best explains these findings?"
  - "Which of the following factors is most likely responsible?"

### Discipline-Specific Lead-in Examples
> Adapt these to your subject area. The examples below use health sciences language but the pattern applies to any discipline.

- "Which of the following changes would be expected in this scenario?"
- "Which of the following best explains the mechanism of this effect?"
- "Which factor is most likely responsible for this finding?"
- "What is the most likely outcome of this intervention?"
- "Which of the following compensatory responses is most likely activated?"

### Answer Option Rules
- **Number of options**: 4-5 options (3 strong distractors preferred over 5 weak ones)
- **Homogeneity**: All options must be the same type (all explanations, all mechanisms, all approaches)
- **Parallel structure**: Same grammatical form and approximate length
- **Alphabetical or logical order**: Don't cluster correct answer in one position
- **No "all of the above" or "none of the above"**
- **No absolutes**: Avoid "always," "never" in options
- **No vague qualifiers**: Avoid "usually," "sometimes," "often"
- Correct answer should NOT be conspicuously longer than distractors

---

## 2. DISTRACTOR DESIGN

### Principles of Good Distractors
- Based on **common student misconceptions** or frequent errors
- **Plausible** to students who haven't mastered the concept
- **Homogeneous** with the correct answer (same category/type)
- Each distractor chosen by >5% of examinees (functional threshold)
- 2-3 strong distractors > 4-5 weak ones

### Common Distractor Strategies
| Strategy | Example |
|---|---|
| Related but different mechanism | If answer is "increased X," distractor could be "increased Y" (related variable) |
| Opposite direction | If answer is "decrease," distractor is "increase" |
| Different system/category | If answer is one subsystem's response, distractor is another's |
| Common student confusion | Mix up closely related concepts or terms |
| Partially correct | Correct reasoning but wrong conclusion or context |
| Associated but not causal | Related finding that is effect rather than cause |

### 14 Item Writing Flaws to Avoid

**Testwiseness Flaws (give clues to correct answer):**
1. **Grammatical cues**: Distractors don't follow grammatically from stem
2. **Logical cues**: Subset of options is collectively exhaustive (A or not-A)
3. **Absolute terms**: "Always" or "never" in some options (usually wrong)
4. **Long correct answer**: Correct option is longer/more specific than others
5. **Word repeats**: Word from stem appears in correct answer only
6. **Convergence**: Correct answer shares most elements with other options

**Irrelevant Difficulty Flaws (make question harder without testing knowledge):**
7. **Complex/double options**: Options are unnecessarily long or compound
8. **Inconsistent numeric data**: Numbers lack consistent formatting
9. **Vague terminology**: Imprecise qualifiers ("rarely," "usually")
10. **Non-parallel language**: Inconsistent grammar across options
11. **Non-logical ordering**: Options lack logical sequence
12. **"None of the above"**: Not recommended
13. **Tricky/misleading stems**: Unnecessarily complex question phrasing
14. **Hinged answers**: Answer depends on other questions

---

## 3. BLOOM'S TAXONOMY FOR QUIZ QUESTIONS

### Cognitive Levels with Examples

#### Level 1: REMEMBER (Recall facts)
- **Verbs**: Define, list, name, identify, recognize, recall, state
- **Target**: ~10-15% of quiz questions
- **When to use**: Foundational vocabulary, key values, basic structures

#### Level 2: UNDERSTAND (Explain concepts)
- **Verbs**: Describe, explain, summarize, classify, compare, interpret, paraphrase
- **Target**: ~20-25% of quiz questions
- **When to use**: Mechanism explanations, concept relationships

#### Level 3: APPLY (Use knowledge in new situations)
- **Verbs**: Calculate, demonstrate, solve, use, implement, predict
- **Target**: ~30-35% of quiz questions (bulk of assessment)
- **When to use**: Scenarios, calculations, predicting outcomes

#### Level 4: ANALYZE (Break down and examine relationships)
- **Verbs**: Compare, differentiate, distinguish, examine, organize, relate
- **Target**: ~20-25% of quiz questions
- **When to use**: Reasoning tasks, relationship analysis, mechanism analysis

#### Level 5: EVALUATE (Justify decisions)
- **Verbs**: Assess, critique, defend, judge, justify, recommend
- **Target**: ~5-10% of quiz questions
- **When to use**: Decision-making, evidence evaluation

#### Level 6: CREATE (Generate new solutions)
- **Verbs**: Design, construct, develop, formulate, propose, synthesize
- **Not typically tested via MCQ** — better for essays, case analyses, projects
- **Target**: 0% of MCQ; use in assignments/discussions instead

### Target Distribution
- Goal: minimum 40% higher-order items (Level 3+)
- Adjust based on course tier — see the Course Tier System section below for formal Bloom's mapping per tier

---

## COURSE TIER SYSTEM

All assessment skills reference this section as the single source of truth for tier-based difficulty calibration.

### Tier Definitions

| Tier | Course Levels | Bloom's Emphasis | Scaffolding Philosophy |
|---|---|---|---|
| **Introductory** | 100–299 | Understand / Apply, touch Analyze | More structure: guided prompts, explicit checklists, step-by-step instructions, lower word counts, encouraging feedback pointing to specific content |
| **Advanced** | 300–499 | Apply / Analyze / Evaluate | Less structure: independent reasoning, complex multi-step scenarios, higher word counts, feedback that prompts deeper thinking |
| **Graduate** | 500+ | Analyze / Evaluate / Create | Minimal structure: scholarly and professional deliverables, seminar-style discourse, peer critique, collegial feedback referencing conceptual debates rather than course content |

### Tier Detection Algorithm

1. Extract numeric portion from `course_code` (e.g., "BIO 101" yields 101, "ENG 320" yields 320, "MGT 560" yields 560)
2. Floor to nearest hundred (101 → 100, 320 → 300, 560 → 500)
3. Map to tier — auto-assign silently:
   - **100–299** = `"introductory"`
   - **300–499** = `"advanced"`
   - **500+** = `"advanced"` — graduate courses default to Advanced calibration. Graduate-specific calibration (scholarly deliverables, seminar discourse, no scaffolding) is opt-in: user says "set this to graduate tier" or sets `course.tier: "graduate"` in `course-config.json`.
4. If course_code has no parseable digits: ask user to select tier explicitly: "I couldn't determine the course level from '[code]'. Is this an introductory (100–299) or advanced (300–499) course?"
5. Store `course.level` (numeric string) and `course.tier` ("introductory", "advanced", or "graduate") in `course-config.json`

### Tier-Specific Bloom's Mapping

| Tier | Quiz (Practice) | Quiz (Graded) | Assignment Default | Discussion Types |
|---|---|---|---|---|
| **Introductory** | Remember → Understand → Apply | Apply → Analyze | L3 (Apply); late modules may reach L4 | Type E, Type A; late modules may reach Type B |
| **Advanced** | Understand → Apply → Analyze | Analyze → Evaluate | L4 (Analyze); late modules reach L5 | Type B, C; late modules reach Type D |
| **Graduate** | Apply → Analyze → Evaluate | Evaluate → Create | L5 (Evaluate) / L6 (Create); research synthesis, professional deliverable, or policy brief | Type D, Type C; peer critique and theoretical debate throughout |

Module position adjusts within the tier's band. Tier sets the baseline; module position fine-tunes.

### Tier-Specific Assessment Parameters

| Parameter | Introductory | Advanced | Graduate |
|---|---|---|---|
| **Quiz question mix (5 Qs, Practice)** | 2 concept MC, 1 T/F, 1 Matching, 1 Fill-in-Blank | 2 scenario MC, 1 Multiple-Answer, 1 Matching, 1 Fill-in-Blank | 2 evidence-appraisal MC, 1 Multiple-Answer, 1 scenario MC, 1 short justification prompt |
| **Quiz question mix (5 Qs, Graded)** | 2 scenario MC (single-step), 1 Multiple-Answer, 1 Matching, 1 T/F | 3 scenario MC (multi-step), 1 Multiple-Answer, 1 Matching | 3 complex scenario MC (competing valid interpretations), 1 Multiple-Answer, 1 evidence-evaluation item |
| **Quiz scenario complexity** | 1–2 sentences, single decision point, direct cause-effect | 2–3 sentences, layered data, competing explanations, multi-step reasoning | 3–4 sentences, ambiguous data, competing scholarly interpretations, requires synthesis across concepts |
| **Distractor design** | Direct misconceptions, single-concept errors | Multi-step reasoning errors, partially correct chains of logic | Competing theoretical frameworks, methodologically defensible alternatives, nuanced disciplinary debates |
| **Feedback style** | Explicit content references, encouraging ("Review Section 2.3 on membrane transport") | Deeper-thinking prompts ("Consider how X relates to Y — what does this imply about Z?") | Collegial, scholarly ("This is where the literature diverges — consider how [framework] would reframe the interpretation") |
| **Assignment word count** | 300–500 words (or 2–4 min video) | 500–800 words (or 4–7 min video) | 800–1,200 words (or 8–12 min presentation); research proposals and policy briefs may exceed this |
| **Assignment scaffolding** | Step-by-step guidance ("First, identify... Then, explain... Finally, connect..."), granular checklists | Goal-and-constraints framing, fewer broader requirements, student determines approach | Goal + deliverable format only. No steps, no checklist. Student determines scope, evidence, and structure. |
| **Assignment deliverable orientation** | Concept application artifact (summary, explanation, comparison) | Analysis artifact (case analysis, annotated argument, comparative evaluation) | Scholarly or professional product (literature synthesis, research proposal, policy brief, professional recommendation memo) |
| **Discussion initial post** | 200–300 words | 300–400 words | 400–500 words |
| **Discussion peer response** | 100–200 words | 150–250 words | 200–300 words; substantive critique expected, not affirmation |
| **Discussion prompt design** | Guiding sub-questions within the prompt, sentence starters or frameworks | Single open-ended question or scenario, student determines analytical structure | Single theoretical or professional question, no sub-questions. Students expected to frame their own analytical lens. |
| **Peer engagement scaffolding** | Scaffolded prompts ("Respond by either agreeing and adding a new example, or respectfully disagreeing with evidence") | Independent ("Engage substantively with your classmates' reasoning") | Seminar-style critique ("Engage with your colleague's methodological assumptions or theoretical framing") |

### Rubric Calibration by Tier

| Level | Introductory | Advanced | Graduate |
|---|---|---|---|
| **Exemplary** | Thorough, accurate application with clear examples. Makes connections to module content. Uses precise terminology. | Sophisticated insight, novel cross-module connections, independent analytical reasoning. Integrates multiple perspectives. | Original scholarly contribution or professional insight. Synthesizes across theoretical frameworks. Demonstrates expert judgment. Connects to current disciplinary debates. |
| **Proficient** | Correctly applies concepts. Addresses all required components. Uses mostly appropriate terminology. | Strong analysis with evidence-based reasoning. Identifies multiple perspectives. Draws on broader conceptual frameworks. | Sound analysis with appropriate scholarly or professional rigor. Engages multiple frameworks or evidence sources. Draws defensible conclusions. |
| **Developing** | Attempts application but has gaps. Some concepts correct but incomplete. | Attempts analysis but reasoning is surface-level or relies on single perspective. | Analysis present but descriptive rather than evaluative. Relies on a single framework without interrogating its assumptions. Limited synthesis. |
| **Beginning** | Minimal attempt. Key concepts missing or described incorrectly. Specific remediation: "Review [content section]." | Fails to engage analytically. Descriptive rather than analytical. Points to foundational concepts that need strengthening. | Fails to engage at the scholarly or professional level expected. Restates course content without analysis. Recommend reviewing core theoretical foundations before resubmitting. |

### Regeneration Bloom's Range

When users request regeneration ("too easy" / "too hard" / "different angle"), adjust within these bounds. These ranges represent the overall tier band across all assessment types — for mode-specific ranges (practice vs graded quiz, assignment vs discussion), see the Tier-Specific Bloom's Mapping table above.

| Tier | Base Range | "Too Easy" — Bump Up (ceiling) | "Too Hard" — Bump Down (floor) |
|---|---|---|---|
| **Introductory** | Remember – Understand – Apply | Understand – Apply – Analyze | Remember – Understand |
| **Advanced** | Apply – Analyze – Evaluate | Analyze – Evaluate | Apply – Analyze |
| **Graduate** | Analyze – Evaluate – Create | Evaluate – Create | Analyze – Evaluate |

- **Bump Up**: Shift Bloom's range up one step. Increase scenario complexity. Subtler distractors. Reduce scaffolding.
- **Bump Down**: Shift Bloom's range down one step. Simplify scenarios. More distinct distractors. Add scaffolding.
- **Different Angle**: Keep same Bloom's range and parameters. Generate entirely new content targeting the same objectives. No reuse.
- **At ceiling/floor**: Inform user: "This is already at the [highest/lowest] difficulty within the [tier] range. To go further, change the course tier."

---

## 4. RUBRIC DESIGN

### Four Essential Components
1. **Task description**: What the student is being assessed on
2. **Scale**: Rating levels (typically 3-5 levels)
3. **Dimensions/Criteria**: Categories being evaluated
4. **Performance descriptions**: What each level looks like for each criterion

### Recommended Rating Scale Options

**4-Level Scale (most common for coursework):**
| Level | Label | Description Pattern |
|---|---|---|
| 4 | Exemplary | Exceeds expectations; demonstrates mastery with integration |
| 3 | Proficient | Meets expectations; demonstrates competence |
| 2 | Developing | Approaches expectations; demonstrates partial understanding |
| 1 | Beginning | Below expectations; demonstrates minimal understanding |

**Dreyfus Model (for clinical/professional competency):**
Novice → Advanced Beginner → Competent → Proficient → Expert

### Rubric Templates by Assignment Type

#### Case Analysis Rubric (4 criteria)
1. **Core Concepts** (30%): Accuracy and depth of explanation
2. **Application** (25%): Connection between theory and case
3. **Critical Analysis** (25%): Evaluation of evidence, consideration of alternatives
4. **Communication** (20%): Clarity, organization, use of appropriate terminology

#### Discussion Board Rubric (4 criteria)
1. **Initial Post Quality** (30%): Depth of analysis, evidence-based reasoning
2. **Peer Engagement** (25%): Substantive responses to peers, building on ideas
3. **Content Accuracy** (25%): Correctness of concepts and claims
4. **Timeliness & Professionalism** (20%): Meeting deadlines, respectful discourse

#### Research/Report Rubric (5 criteria)
1. **Thesis/Position** (15%): Clear, well-defined, arguable
2. **Evidence/Data** (20%): Accurate collection, appropriate presentation
3. **Analysis** (25%): Correct interpretation, sound reasoning
4. **Explanation** (25%): Mechanism-based or theory-based interpretation
5. **Conclusions** (15%): Summary, limitations, implications

### Critical Thinking Criteria (for any rubric)
| Level | Description |
|---|---|
| Exemplary | Synthesizes multiple perspectives; identifies assumptions; evaluates evidence quality; proposes novel connections |
| Proficient | Analyzes from multiple angles; supports claims with evidence; recognizes limitations |
| Developing | Provides surface-level analysis; relies on single perspective; limited evidence use |
| Beginning | Restates information without analysis; no evidence of critical evaluation |

---

## 5. Assignment & Discussion Feedback Design

**Framing**: The patterns below represent recommended defaults. IDs may adapt or override them based on course context, faculty preference, or institutional conventions.

### Feedback vs. Feedforward

Most Canvas feedback defaults to grade justification — explaining why a score was earned. Feedforward reorients the comment toward what the student can do differently next time.

**Grade-justification example** (not recommended as primary pattern):
> "You met the basic requirements but your thesis was underdeveloped. 72/100."

**Feedforward example** (recommended pattern):
> "Your central argument has real potential. For the revision or your next essay, try stating your claim in one sentence before your supporting evidence — that constraint often clarifies the logic. See [resource] for a model."

Both examples can coexist in a single comment; the recommended pattern leads with the actionable forward direction.

### Feedback Structure by Assessment Type

**For assignments (recommended 3-step pattern)**:
1. **Anchor** — acknowledge the strongest element of the submission
2. **Focus** — identify the single most high-leverage area for growth (not a list)
3. **Forward** — give one concrete, actionable next step or strategy

**For discussions (recommended 3-step pattern)**:
1. **Engage** — respond to the student's specific idea, not the prompt in general
2. **Extend** — add a perspective, question, or connection the student didn't surface
3. **Invite** — close with a question that invites continued thinking

These are starting structures, not rigid scripts. IDs should adapt them to fit course tone, faculty voice, and discipline conventions.

### Tier Calibration

Feedforward language should match the cognitive expectations of the course tier:

| Tier | Recommended Feedforward Orientation |
|---|---|
| **Introductory** | Procedural and conceptual clarity — "What to add or clarify next time" |
| **Advanced** | Analytical depth and evidence quality — "What assumption to examine or evidence to find" |
| **Graduate** | Scholarly positioning and synthesis — "How to situate this claim in the literature or make the contribution explicit" |

These are defaults. IDs may adjust calibration based on where the student is in the course arc.

### Timing Guidance

- **Formative assessments** (knowledge checks, low-stakes discussions): recommend returning feedback within 5–7 days
- **Summative assessments** (major assignments, final projects): recommend returning feedback within 10–14 days

These targets are for instructor guide surfacing — they are not student-facing commitments unless the ID chooses to include them in the syllabus. Course-config does not enforce these; they appear as notes in the generated instructor guide.

---

## 6. Due Date Distribution Pattern

**Recommended pattern**: Aim for no more than 30% of total graded items falling in the final 25% of course duration.

This is a guideline, not a gate. When the distribution exceeds this threshold, course-config surfaces a note for ID review. The ID may proceed with any distribution if there is instructional rationale (e.g., capstone-heavy courses, competency-based sequencing, modular electives).

**What course-config displays when threshold is exceeded**:
> "Heads up: [N] of [total] graded items ([X]%) fall in the final quarter of the course. The recommended pattern is ≤30%. If this reflects intentional design (e.g., a capstone or portfolio structure), no action needed — just confirm the syllabus communicates the pacing clearly to students."

**Capstone exception**: Courses with a single summative capstone project in the final module are exempt from this pattern by design. The ID should note this in course-config (`grading.capstoneCourse: true`) to suppress the surface note.

**Rationale**: Back-loaded courses can create student stress and make it harder to incorporate feedback before the course ends. Distributing graded work more evenly supports ongoing learning and reduces assessment crunch.

---

## 7. QUESTION WRITING PROCESS (Step-by-Step)

### Step 1: Start with Learning Objective
- Identify the specific objective being assessed
- Determine the Bloom's level appropriate for that objective
- Choose whether MCQ, essay, or other format best assesses it

### Step 2: Write the Scenario/Stem
- For Apply+ levels: Use a scenario or case
- Include only information needed to answer (plus some context)
- Avoid "red herrings" (deliberately misleading information)
- Make the question answerable from the stem alone

### Step 3: Write the Correct Answer First
- Must be clearly and unambiguously correct
- Should be concise and specific

### Step 4: Write Distractors
- Base each on a specific misconception or common error
- Make all options homogeneous and parallel
- Check that no option is obviously wrong
- Verify correct answer is not conspicuously different

### Step 5: Review for Technical Flaws
- Check all 14 item writing flaws
- Verify grammatical consistency
- Ensure no cueing between questions
- Confirm appropriate Bloom's level
- Test: Can a non-expert eliminate options using test-taking strategy alone?

### Step 6: Write Feedback
- Correct answer feedback: Explain WHY it's correct (reinforce learning)
- Incorrect answer feedback: Explain the specific misconception
- Reference the learning objective and relevant content

---

## 8. DISCIPLINE-SPECIFIC ASSESSMENT TIPS

> The examples below use health sciences scenarios. Adapt the patterns to your discipline by replacing clinical contexts with discipline-appropriate cases, data sets, or scenarios.

### Integrative Question Design
The best questions require integrating knowledge across topics:
- Questions that span multiple modules or concepts
- Scenarios requiring synthesis of several principles
- "What if" variations that test transfer of learning

### High-Yield Question Patterns
| Pattern | Description |
|---|---|
| Mechanism questions | "Which mechanism best explains..." |
| Prediction questions | "What would happen if..." |
| Comparison questions | "How does X differ from Y in this context?" |
| Application questions | "Given this data, which approach is most appropriate?" |
| Integration questions | Combine concepts from multiple modules |
| Error analysis questions | "What is the most likely error in this reasoning?" |
