Improving existing MCQs

By Theresa Suart & Eleni Katsoulas

Writing and editing test questions is an ongoing challenge for most instructors. Creating solid multiple choice questions (MCQs) that adequately address learning objectives can be a time-consuming endeavor.

Sometimes you may have existing questions that are pretty good, but not quite where you need them to be. Similar to a house reno versus new construction, sometimes it might be worth investing the time improve what you already have. How do you know which questions need attention and how can you rework them?

Previous exams are analyzed to determine which questions work well and which don’t. This can provide some guidance about questions that can be improved.

To select questions for an MCQ renovation, you can start with checking out the statistics from last year’s exams (available from your curricular coordinator or from Eleni).

Two statistics are useful indicators for selecting individual questions for tweaking, rewriting or other fixes: Item Difficulty and Discrimination Index.

Item difficulty is a check on if questions are too easy or too hard. This statistic measures the proportion of exam takers who answered the question correctly.

Discrimination index differentiates among text takers with high and low levels of knowledge based on their overall performance on the exam. (Did people who scored well on the exam get it right? Did people who scored poorly get it right?)

These two statistics are closely intertwined: If questions are too easy or too hard (see item difficulty), they won’t provide much discrimination amongst examinees.

If questions from previous years’ tests were deemed too easy or too hard, or had a low discrimination index, they’re ripe for a rewrite. Once you have a handful of questions to rewrite, where do you start? Recall that every MCQ has three parts and any of these could be changed:Exam

  • The stem (the set-up for the question)
  • The lead-in (the question or start of the sentence to be finished with the answer)
  • The options (correct answer and three plausible but incorrect distractors*)

The statistics can inform what changes could be necessary to improve the questions. For one-on-one help with this, feel free to contact Eleni, however, here are some general suggestions:

Ways to change the stem:

  • Can you change the clinical scenario in the stem to change the question but use the same distractors? (e.g. – a stem for a question that asks students what the most likely diagnosis is based on a patient presenting with confusion with the correct answer being dementia, can be then re-written to change the diagnosis to delirium)
  • Ensure the stem includes all information needed to answer the question.
  • Is there irrelevant information that needs to be removed?

Ways to change the lead-in:

  • Decide if the questions is to test recall, comprehension, or application.
  • Recall questions should be used sparingly for mid-terms and finals (but are the focus for RATs)
  • Verbs for comprehension questions include: predict, estimate, explain, indicate, distinguish. How can these be used with an MCQ? For example: “Select the best estimate of…” or “Identify the best explanation…”
  • You can use the same stem, but change the lead in (and then, of course, the answers) – so if you had a stem where you described a particular rash and asked students to arrive at the correct diagnosis, you can keep the stem, but change the lead-in to be about management (and then re-write your answers/distractors).

Ways to change one or more distractors:

  • Avoid grammatical cues such as a/an or singular/plural differences
  • Check that the answer and the distractors are homogeneous to each other: all should be diagnoses, tests or treatments, not a mix.
  • Make the distractors a similar length to the correct answer
  • Ensure the distractors are reasonably plausible, not wildly outrageous responses
  • Skip “none of the above” and “all of the above” as distractors

As you dig into question rewriting, remember the Education Team is available to assist. Feel free to get in touch.

Watch for MCQ Writing 2.0 later this spring.


* Yes, there could be more than three distractors, but not at Queen’s UGME. The Student Assessment Committee (SAC) policy limits MCQs to four options.

Posted on

QMed students cooking up wellness strategies

 by Meghan Bhatia, AS Wellness Officer

and Monica Mullin, Nutritional Wellness Lead

What is wellness? This is a question that proves far more complex than it would appear to be. Although on the surface it may seem easy to define, wellness is an interesting topic to discuss because it can be very personal and take different roles in students’ lives. Buzzwords often surround the wellness curriculum, things like work-life balance, healthy eating, ‘Get Your 150’ and mental or emotional well-being. These categories do indeed contribute to wellness, but with 400 different students and multiple faculty, one size does not fit all.

The idea of taking ownership of one’s own wellness was what piloted Wellness Month at Queen’s University. We may all know the areas of personal wellness, but this month added structure and challenge to these categories, in a hope that people would get new ideas, form habits and lifelong learning would result naturally.

The #keepsmewell challenge was piloted at Queen’s Medicine last year and this year was taken nationally through the CFMS, and run across the country concurrently. At Queen’s we had 160 QMed students participate (including clerks) as well as 18 faculty/staff and 16 QuARMS students.Salad

What was the #keepsmewell challenge? It was a positive habits challenge that had four themed weeks: nutrition week, mental health week, physical week and social academic balance week. Students would receive points for completing tasks on the spreadsheet and were often asked to promote these activities on social media with #keepsmewell.

It was always interesting seeing students stay active and well through their photos with all of the creative paths they took. In particular, the amazing cooking photos from last year were the inspiration behind the QMed cookbook. We decided to compile what students did throughout the challenge so they would have a reference for the rest of the year, of ideas and inspirations; QMED COOKS is available in ibooks or pdf and is free for anyone. It is available here and has been shared nationally and provincially. One of our contributions to the book was adding in nutrition facts and tips that we learnt in school, through resources, or the dietician talk during nutrition week to keep it fun and educational!

Our wellness curriculum is wide and quite diverse, but it is really only a part of QMed students’ wellness. The interest in this month and the positive feedback we have received from this book really does show that students are invested in their own wellness. We both hope that this is just a launching pad for even more nutritional integration into the curriculum, and that many wellness months will continue on, as wellness is difficult to teach, but so essential to learn.

 

 

 

 

Posted on

Why students do (or do not) attend lectures

Imagine it’s just before 8:30 on a Monday morning in the School of Medicine Building. The class is assembling for the first session of the day – a lecture to be delivered by a clinical faculty member who teaches perhaps 4-5 times each academic year. The session has been prepared based on objectives assigned by the Course Director. This material, they’re assured, is consistent with the course plan and important to the overall learning plan for the class. As the 8:30 mark approaches and then passes, a sense of unease comes upon the room as it becomes clear that only about half the seats will be occupied. The lecturer, conscious of time, needs to get underway and does so, but is unsettled both by the poor attendance and apparent distraction of those students present, all of whom appear more attentive to their individual screens than to the dialogue. The students in attendance pick up on this unease and reflect it in their feedback about the session. The faculty member is left somewhat discouraged and perhaps embittered that their efforts appear to have been spurned by students, who they might perceive as poorly motivated and unappreciative. In short, everybody loses.

This is, fortunately, not a common occurrence in our school, but is an issue that comes to attention periodically, often early in the winter term when learning fatigue and the weather seem to combine to reduce student enthusiasm. When it does occur, it can be quite damaging and threatens to compromise the excellent student-faculty interaction that is otherwise a strength and characteristic of our school.

What’s really going on? It’s a topic, I think, worth some thought and exploration.

Even a quick literature survey (see below) makes it clear we are not alone with respect to this issue, and there’s no shortage of perspectives on causes and solutions.

The lecture has a long and venerable place in the history of medical education. Great physicians of the past such as Harvey and Osler are remembered as much for their lectures and oratory as for their scientific discoveries. Indeed, medical schools and universities continue to recognize excellence through named lectureships.

The classical lecture is unadorned with slides, videos or instantaneous audience feedback. It is, simply, an encounter between a learned, skilled orator and eager, attentive learners. An accomplished lecturer is informative, inspirational, provocative and thought provoking, and is somehow able to weave all these elements into a compelling narrative, capturing the attention of listeners at a very personal level, leaving them satisfied and enriched for the experience. Listeners at such sessions are wholly engaged, with no personal or electronic distractions. The attempt to “multi-task” is irrelevant, and in fact, detrimental to the experience.

Modern approaches to medical education quite rightfully emphasize the importance of active, small group and case-based approaches. At our 2007 accreditation review, our school was strongly criticized for being too “lecture heavy”. At that time, 80-90% of our teaching content was delivered in lecture format. The curricular review group tasked with revising our curriculum in the wake of that review decided (I think wisely) not to abandon the lecture format entirely, but to establish a balance with small group and case-based teaching, which it felt bring great and complimentary value to our students. Our School of Medicine Building was, in fact, designed with the clear intention of providing venues where both lecture and small group format teaching could be provided, even within the same session. Transition was difficult but has resulted in our current curriculum providing about 40-50% of teaching content in lecture format.

Lecture Classroom

To get a clearer idea of this issue at our school, I surveyed the second year class, posing three simple, open-ended questions:

  1. What do you find most valuable about lectures?
  2. Why do you attend lectures?
  3. Why do you not attend lectures?

I deliberately avoided providing pre-stated options and instead asked for narrative responses. Sixty students provided about 200 separate commentaries. Eleni Katsoulas, our Assessment Consultant, and I then carried out an analysis of the responses. The major themes that emerged are as follows:

What do you find most valuable about lectures?

Graph: What do you find most valuable about lectures?

It appears that what students find most valuable about lectures (37% of responses) is that attendance provides some educational value above and beyond what they can get from their own review of the available information. Almost as many (35%) expressed this somewhat differently, in terms of “clarification” of the information, which we interpreted as providing greater understanding about specific points or differentiation of more clinically relevant information rather than deeper understanding. About 20% identified the direct interaction with faculty and ability to ask questions as the most valuable feature. Smaller numbers identified the interaction with their own classmates (6%) as the most valuable feature, and a few (2%) noted the ability to bring real patients into the sessions.

Why do you attend lectures?

Graph: Why do you attend lectures?

The most common responses, by far, were comments related to the concept that attendance at lectures enhanced and deepened learning of the material (64%). About 16% attend lectures as a means of keeping track of the curriculum and not falling behind. Thirteen percent attend for purely social reasons, to interact with classmates. A few (3%) attend only when particular assessments or specific learning events are planned. Another 3% indicate they attend only because they’ve paid tuition to do so and essentially wish to “get their money’s worth”.

Why do you not attend lectures?

Graph: Why do you not attend lectures?

The most common reason (44%) students cite for not attending lectures is a belief that the sessions bring not value beyond what they can derive through their own review of the material. Another 29% miss for a variety of reasons that could be termed “personal”, which includes anything from preferring to sleep in, to events or activities that they find difficult to schedule outside lectures times. An additional 20% miss lectures in order to attend other activities they feel are more important to their learning, such as observerships. Three percent note that they have found they simply don’t learn in a lecture format, and another 3% indicate they choose not to attend when they fail to see the “relevance” of the material presented.

Additional comments included several expressions of disappointment on the part of students regarding low attendance, and a desire for video recording of lectures.

So, what are we to make of all this? A couple of key points would seem to emerge from both our results and the literature, perhaps self-evident, but relevant to this issue and probably worth articulating.

The students of today have very different learning needs than those of a generation (or two) ago. Fundamentally, they don’t need to attend lectures to gain pure knowledge or factual information, as did the students of Harvey and Osler or, for that matter, as did many of our current senior faculty. That information is readily available to them. Effective teachers understand this and, in addition to factual information, provide personal insights and novel perspectives borne of their own experience and ideas that enhance and complement the student’s personal learning experience. This “higher level” learning that students speak of can take many forms. It may involve explanation of key and complex concepts, guidance to key sources of reliable information, learning how to “translate” factual information into clinical decision-making, or simply the “real life” picture of how experienced clinicians manage the conditions they’re attempting to learn and understand. Fundamentally, they don’t need us (faculty) to deliver information they can easily and more efficiently obtain in other ways. They’re looking for something more.

The lecture is a very human, and therefore “social” event. This is what gives the lecture its power and potential to be a highly effective learning opportunity, providing something above and beyond what can be attained from any recorded material or electronic format. But this is only true if the format is appropriately utilized. From the perspective of faculty, this brings considerable responsibility, and probably some significant stress. They are the centre of attention. The lecture basically excels or fails on their “performance”. Moreover, their “real time” presence at the event requires them to be personally invested in the event, and allows them to interact with the listeners, sense their receptivity to the material, vary their approach, and respond to individual questions. What they provide, in essence, is something very personal, and much more valuable than simple recitation of facts and information.

Students, for their part, also contribute to the success of the lecture by not simply showing up, but by truly attending and participating actively. They must recognize that they get maximal value (the “higher level learning”) by being actively engaged and listening carefully not simply for the factual information, but for added insights the faculty member is able to provide.

In a greater sense, we might regard all this in the context of the Information-Knowledge-Wisdom paradigm. Information consists of all the factual content and points of understanding that are essential to the practice of medicine. The essentials of anatomy, physiologic processes, pathologic conditions and clinical examination would be examples relevant to the study of medicine. Knowledge can be defined as the accumulation of key information, in a manner that allows it to be used for a specific purpose. Learning how to manage a patient presenting with a particular clinical condition requires such accumulated and integrated knowledge. Wisdom is the ability to make correct judgments and decisions. In medicine, it can be considered the ability to decide whether established approaches are appropriate in a particular patient, or how to approach a specific patient when diagnosis is elusive or established approaches are not available. Wisdom derives from a combination of personal attributes, much accumulated knowledge/expertise, and acquired experience. Albert Einstein once said, “wisdom is not a product of schooling but of the lifelong attempt to acquire it.”

The educational process can be thought of as progressions through those three stages of learning as illustrated below:

Illustration of the educational process

In medical education, they clearly overlap, but the first two years of medical school can be considered as largely devoted to developing the information component, with some development of knowledge-based approaches to clinical illness. Clerkship and residency further develop and refine the knowledge component and, hopefully, begin the process of developing wisdom. The development of wisdom, of course, never ends, and never reaches perfection. In fact the three components are perhaps better illustrated in this way:

Better illustration of the educational process

Getting back to the lecture issue, its true place can be considered as providing a means to impart the knowledge and wisdom components of medical practice to the novice learner. That may be its greatest power, and greatest purpose.

The lecture, I would conclude, has evolved and must continue to evolve with the needs of our learners, but has a unique and valuable role in medical education. It is important that both students and faculty understand and actively engage its purpose if it’s full educational potential is to be realized.

A final, summarizing message to our students on this topic might be to remind them of a well-established adage:

“Knowledge speaks, but wisdom listens”

Anthony J. Sanfilippo, MD, FRCP(C)
Associate Dean,
Undergraduate Medical Education

Many thanks to Eleni Katsoulis, UGME Assessment Consultant, for her valuable assistance in the compilation of information for this article.

Further reading suggestions:

  • Charlton BG. Lectures are an effective teaching method because they exploit human evolved “human nature’ to improve learning. Medical Hypotheses 2006; 67:1261.
  • Dolnicar S. What makes students attend lectures? The shift towards pragmatism in undergraduate lecture attendance. Conference proceedings of the Australian and New Zealand Marketing Academy. 2004. (http://ro.uow.edu.au/commpapers/81)
  • Massingham P, Herrington T. Does attendance matter? An examination of student attitudes, participation, performance and attendance. Journal of Univeristy Teaching and Learning Practice 2006; 3: 82.
  • Harvard Initiative for Learning and Teaching. 2014. Lecture attendance research: Methods and preliminary findings. http://hilt.harvard.edu/files/hilt/files/attendancestudy.pdf

Posted on

3 Key Teaching and Learning Principles: Revisiting RIA in UGME

filing cabinet
Sheila digging around in her filing cabinet

This weekend, I was digging around in my hard drive, and pulling files, as I’m working with Dr. Lindsay Davidson on the concept of integrated threads in our curriculum. (Stay tuned for a future blog.) All of a sudden, out popped a document called “3 key teaching principles,” which Dr. Elaine Van Melle and I worked on in 2008.  It eventually became part of the Teaching and Learning Policy for UGME.

I took a look and it’s one of those ageless documents that I think we can still learn a lot from and perhaps refresh in the light of 2016. Do any of you recall “RIA“?  Come take a journey back and forward with me around the 3 Key Principles of Relevance, Integration and Active learning.

 

All learning experiences should be . . .

RELEVANT

“to have significant and demonstrable bearing on the needs of the learner.”

A student says, Why should I care about this?

A teacher says, Why is this important for a student to know?

Why relevance?

  • Creating relevance fosters interest, motivation and engagement.   It is a key step in facilitating retention and transfer of information.

How can I make teaching/learning relevant?

  • Illustrate clinical applicability in the primary management of patients
  • Ask these key questions about foundational concepts: “What does every physician need to know about this concept?” and “What does a learner entering my sub-specialty need to know?”
  • Link the material to the Medical Council of Canada’s (MCC) objectives as the MCC objectives document forms the basis for the licensing exam.
  • Begin with a clear statement of essential learning objectives reasonable for the time allotted.
  • Explicitly state the relationship between the learning experience and the assessment process

Back to 2016, calendar consider this checklist for relevance in your teaching:

  1. Do I use case studies both of my own, and as activities to let students apply learning to “real life”? relevant 1Do I use lots of examples to clarify concepts?
  2. Have I reviewed the MCC’s for my learning event and made sure that my teaching is aligned to them?
  3. Have I got 2-3 clear statements of learning objectives at the level the learners per 1 hour learning event?
  4. Can I state a key idea or “core message” for this one hour of teaching
  5. Do I describe why this is important for students to know?
Learning is enhanced when it is relevant, particularly to the solution and understanding of real-life problems and practice. (Kaufman and Mann, 2007)

INTEGRATED

“to be connected and interrelated”

A student says, Where does this fit?

A teacher says, How can I connect this with other teaching and learning?

Why integrate?

  • Connecting to the knowledge of the learner facilitates retention & transfer of information from one context to another
  • You’re not the only person in the curriculum teaching about this topic.

How do I integrate?

  • Ensure learning is appropriate to the level of the learner and relates to the learner’s previous experiences.
  • Structure information in a way that demonstrates the relationship between key ideas.
  • Link to other sessions to allow for progressive reinforcement integrate 4of fundamental concepts.
  • Connect with other teachers to minimize unnecessary redundancy.
  • Create horizontal integration by explicitly connecting to sessions that have come before and those that will follow a particular learning experience.
  • Create vertical integration by linking to other types of learning experiences that may be going on at the same time e.g. problem-based learning, clinical skills, basic science teaching, etc.)

Back in 2016, calendar try this checklist for integrated learning:

  1. Have I vetted the level of learning in my teaching with other faculty, my course director and/or an Educational Developer?
  2. Have I checked where else in the curriculum the topics of this learning event are taught? (Tip: Year Director and Educational Developers can help. So can MEdTech: Curriculum: Curriculum Search. TLIC is working on Integrated Threads.)integrated 2
  3. Is my learning event “integrated” and well-organized in itself with sub-topics, links back to the introduction and a summary? Do I provide an outline and refer back to it during the learning event to orient the students?
  4. Do I know where my material fits in with in Clinical Skills, FSGL, and other parts of this course as well as others?
  5. If I’m teaching in C2, or a clerkship seminar, does this topic build on and become more complex than the foundational concepts taught in years 1 or 2 and C1? (Have I looked back at those? Looked forward to C3? Thought about how this applies in clinical clerkship rotations?)
In the hands of the most effective instructors, [this] then becomes a way to clarify and simplify complex material while engaging important and challenging questions…(Bain, 2004)

ACTIVE

“ Students engage with and take responsibility for learning”

A student says, How will I learn this?

A teacher says, How will I engage the students?

Why use active learning?

  • Facilitates retention and transfer through the construction of new ideas and/or ways of thinking.
  • Learning is a process that results in some modification, relatively permanent, of the learner’s way of thinking, feeling or doing.
  • Requires the active construction of new ideas or ways of thinking on the part of the learner.

How do I use active learning strategies?

  • Students are encouraged to take responsibility to achieve new levels of understanding and/or skill development
  • Create learning environments that foster rich interactions among students, between the instructor and students, and between the student and the learning materials.  active 5
  • Students learn well by doing, and participating in “real-world” experiences.

 

Here’s the 2016 checklist calendarfor active learning:

  1. How will I change the students’ ways of thinking, feeling or doing with this learning event
  2. As a way to engage, have I tried using video clips? Illustrations? Demonstrations? Real (live) patients? A poll to take the “temperature” of the class? My own experiences in the clinic or workplace?
  3. How can I get the students to “construct” new ideas? Have I tried asking probing questions in key places in the learning event, or providing a worksheet or algorithm for the session? Have I tried to present an intriguing question, problem or case study and use different points in my lecture to solve the problem? Can I use “real world” artifacts to engage the students?active3.jpg
  4. How can I get the students interacting with each other, or with me and other faculty or residents in the room? Have I tried partner work, or small group work? Have I thought about Group RATs? Have I tried, Think, Pair, Share?
  5. Do I pause at key points and “change up” what is happening in the room?
  6. Have I integrated student activity in the learning event, or partnered with an expanded clinical skills or clinical skills learning event?
  7. Do I give the students a chance to demonstrate what they are learning?
    Learning is not a spectator sport. Students… must talk about what they are learning , write about it, relate it to past experiences, apply it to their daily lives.” (Chickering and Gamson, 1987)

I hope you’re finding the results of my filing cabinet diving helpful.  Do the checklists make sense now in 2016?  Is there anything here you can use?  Please check in and let me know. Or contact one of us in Educational Development at UGME.

Posted on

CARMS Match Day: 2016

What our students are experiencing, and how to help them get through it

For medical students in Canada, there are three days in the course of their career that stand out above all others: the day they receive their letter of acceptance to medical school; convocation (when they officially become graduate physicians); and Match Day. The most emotionally charged by far, is Match Day. For those of you not familiar, Match Day is when all fourth year students learn which postgraduate program they will be entering. The match is the final step in a long process of contemplation, exploration and application. The match and the day itself are full of drama, with all results being released simultaneously at noon.

This year, Match Day is March 2. By approximately 12:00:05 that day, all students will know their fate. As you can imagine, there will be much anxiety leading up to the release. For most (hopefully all), the day will be one of relief and celebration. For a very few (and hopefully none), there may be disappointment and confusion. Many schools release their fourth year clinical clerks from clinical duties on Match Day. At Queen’s we have taken the position that our students take on professional obligations during their training and their personal celebrations should not supervene those obligations. Having said that, I’d like to remind any faculty supervising our fourth year students on March 2nd of the following:

  1. Anticipate that your student will be distracted that morning
  2. Please ensure your student is able to review their results at noon.
  3. Check on your student. If he or she is disappointed, please be advised that the student counselors and myself are standing by that day to help any student deal with their situation and develop a plan.
  4. Be advised that the students will almost certainly be holding some type of celebratory event that evening. Although your students are not excused for personal purposes, I would ask that you give them every reasonable consideration.

Fortunately, we have an excellent Student Affairs team, headed by Renee Fitzpatrick, who are available and very willing to answer any questions you may have and respond to concerns regarding our students. They can be accessed through Jacqueline Findlay at jacqueline.findlay@queensu.ca, or 613-533-2542. The faculty counselors can also be contacted directly at the following:

FitzpatrickDr. Renee Fitzpatrick, MD, MRC Psych, FRCPC
Wellness Advisor

 

 

 

 


 

KellyHowseDr. Kelly Howse, BSc (Hon), MD, CCFP
Career Advisor
kelly.howse@dfm.queensu.ca

 

 

 

 


 

SusanHaleyDr. Susan Haley, MD, FRCPC
Career Advisor
haleys@kgh.kari.net

 

 

 

 


 

LakoffDr. Joshua Lakoff, MD, FRCPC
Career Advisor
lakoffj@kgh.kari.net

 

 

 

 


Thanks for your consideration, and please feel free to get in touch with myself or any of the Student Affairs Team if you have any questions or concerns about Match Day or beyond.

Anthony J. Sanfilippo, MD, FRCP(C)
Associate Dean, Undergraduate Medical Education

 

Posted on