By Theresa Suart & Eleni Katsoulas
Writing and editing test questions is an ongoing challenge for most instructors. Creating solid multiple choice questions (MCQs) that adequately address learning objectives can be a time-consuming endeavor.
Sometimes you may have existing questions that are pretty good, but not quite where you need them to be. Similar to a house reno versus new construction, sometimes it might be worth investing the time improve what you already have. How do you know which questions need attention and how can you rework them?
Previous exams are analyzed to determine which questions work well and which don’t. This can provide some guidance about questions that can be improved.
To select questions for an MCQ renovation, you can start with checking out the statistics from last year’s exams (available from your curricular coordinator or from Eleni).
Two statistics are useful indicators for selecting individual questions for tweaking, rewriting or other fixes: Item Difficulty and Discrimination Index.
Item difficulty is a check on if questions are too easy or too hard. This statistic measures the proportion of exam takers who answered the question correctly.
Discrimination index differentiates among text takers with high and low levels of knowledge based on their overall performance on the exam. (Did people who scored well on the exam get it right? Did people who scored poorly get it right?)
These two statistics are closely intertwined: If questions are too easy or too hard (see item difficulty), they won’t provide much discrimination amongst examinees.
If questions from previous years’ tests were deemed too easy or too hard, or had a low discrimination index, they’re ripe for a rewrite. Once you have a handful of questions to rewrite, where do you start? Recall that every MCQ has three parts and any of these could be changed:
- The stem (the set-up for the question)
- The lead-in (the question or start of the sentence to be finished with the answer)
- The options (correct answer and three plausible but incorrect distractors*)
The statistics can inform what changes could be necessary to improve the questions. For one-on-one help with this, feel free to contact Eleni, however, here are some general suggestions:
Ways to change the stem:
- Can you change the clinical scenario in the stem to change the question but use the same distractors? (e.g. – a stem for a question that asks students what the most likely diagnosis is based on a patient presenting with confusion with the correct answer being dementia, can be then re-written to change the diagnosis to delirium)
- Ensure the stem includes all information needed to answer the question.
- Is there irrelevant information that needs to be removed?
Ways to change the lead-in:
- Decide if the questions is to test recall, comprehension, or application.
- Recall questions should be used sparingly for mid-terms and finals (but are the focus for RATs)
- Verbs for comprehension questions include: predict, estimate, explain, indicate, distinguish. How can these be used with an MCQ? For example: “Select the best estimate of…” or “Identify the best explanation…”
- You can use the same stem, but change the lead in (and then, of course, the answers) – so if you had a stem where you described a particular rash and asked students to arrive at the correct diagnosis, you can keep the stem, but change the lead-in to be about management (and then re-write your answers/distractors).
Ways to change one or more distractors:
- Avoid grammatical cues such as a/an or singular/plural differences
- Check that the answer and the distractors are homogeneous to each other: all should be diagnoses, tests or treatments, not a mix.
- Make the distractors a similar length to the correct answer
- Ensure the distractors are reasonably plausible, not wildly outrageous responses
- Skip “none of the above” and “all of the above” as distractors
As you dig into question rewriting, remember the Education Team is available to assist. Feel free to get in touch.
Watch for MCQ Writing 2.0 later this spring.
* Yes, there could be more than three distractors, but not at Queen’s UGME. The Student Assessment Committee (SAC) policy limits MCQs to four options.