Using the IDEAL banks of questions for your assessments
Obtaining IDEAL Consortium Questions
Queen’s School of Medicine has joined the IDEAL Consortium, an international assessment item-sharing collaboration among Schools of Medicine. The Consortium has 27 member schools from 11 countries. Queen’s and UBC are currently the only Canadian members.
The IDEAL Restricted Question Bank contains over 20,625 assessment items including 17,109 MCQs, 539 short-answer questions and 461 OSCE stations. Collectively, members contribute about 4,000 new questions to the restricted and non-restricted question banks annually.
Restricted Bank: Please contact your Curricular Coordinator to request sets of restricted bank questions in your subject area or questions on particular topics. (Zdenka Ko for Year 1, Tara Hartman for Year 2, Jane Gordon for Clerkship Rotations and Candace Trott for “C” courses in clerkship.) Restricted bank questions need to be kept secure, so they can only be used on final examinations. A Word document containing the questions (as well as their answers and “item numbers”) will be couriered to you, or you can request that a secure MEdTech community be created for you to share restricted questions with other faculty members in your course.
To use restricted questions on final exams, simply provide your Curricular Coordinator with the item number of each question and the order in which you would like the questions to appear on the final exam. If you are sharing restricted questions via a secure MEdTech community, you can copy and paste your question selections into a Word document and upload it to the Curriculum Coordinator’s folder in the secure community. It is important that the IDEAL restricted bank questions not be emailed except in password-protected Word files. The restricted questions must not be viewed by students except during the writing of final exams.
You can specify edits to any of the IDEAL items – including OSCE stations. If you edit the items yourself, please highlight your edits so that your Curriculum Coordinator can transfer the edits to the local copy of the IDEAL bank.
The old LXR bank contained many duplicate and triplicate questions, so please let your Curriculum Coordinator know the origin of each exam question (IDEAL? LXR? Original? From a colleague?) We especially need to know, for copyright and item submission reasons, if any questions did not originate at Queen’s. Questions that did not originate at Queen’s will be marked, “Do not submit to IDEAL”, but can be stored in the local copy of the IDEAL bank and used on Queen’s exams.
Unrestricted Bank: Unrestricted bank items can be used in online quizzes, in clicker sessions, on midterms etc. Students can have full access to all unrestricted bank questions. Currently the MEdTech team is creating an interface for the unrestricted bank so that faculty members will have full access to the questions. At present, requests for emailed sets of unrestricted bank question sets can be sent to Catherine Isaacs (firstname.lastname@example.org).
Great Health Care Requires Great Medical Educators
Education is not an industrial process; it is a human one.
In the Dec. 10 edition of The Atlantic Monthly, Richard Gunderman, MD. PhD., examines different sets of components of excellence in medical education: curriculum, instructional methods, and assessment techniques AND creativity, commitment, and inspiration of medical educators. He focuses on the critical importance of fostering a generation of medical educators through support of medical education. For the article see
What do p and R-values mean anyhow? : Understanding how to interpret multiple-choice test scores using statistics.
Have you ever wondered whether or not your multiple-choice questions (MCQs) are too easy? The answer to this question can be found in the p-values or item difficulty: the percentage of students who answered correctly. The difficulty of a MCQ can range from 0.00 to 1.00; the higher the p-value, the easier the question. What we should be concerned with are high difficulty questions with p-values less than 0.3.
Have you ever wondered which questions tricked students who otherwise performed well on a test overall? The R-value or item discrimination looks at the relationship between how well students performed on a question and their total score. Item discrimination indicates students who know the tested material and those who do not. The higher the R-value, the more discriminating the test question. We should try to remove questions on the test with discrimination values (R-values) near or less than 0.3. This is because students who did poorly on the test did better on this question than students who performed better overall.
Did you Know?
Multiple-choice questions that use words in the stem such as best, most, first, or most correct require higher-level thinking but often confuse students because they are ambiguously worded. Our students have struggled lately with ambiguity in the wording of MCQs on RATs and exams such as “Which is the most likely….”. They assume “most likely” to be “most common”, whereas the most likely answer could be an uncommon situation. It’s important to word the question clearly so that students are not confused. So for example, the question could state, “In light of the clinical information provided above, which diagnosis would you make?
You can also ask students about “most common”, “most concerning”, or “what is the first test you would perform” etc. but it is always good to anchor these stems by referring to the data presented previously. Then the key is to require them to choose, evaluate, interpret, judge, infer from data, solve problems, and apply principles.
Did you Know?
The Student Assessment Committee has posted several articles, checklists and PowerPoint slides to assist you with Multiple Choice Questions.
For more guidance on writing high-quality multiple-choice questions refer to MCQ Guidelines and Writing MCQ’s in School of Medicine Faculty and Staff Resources at:
Queen’s School of Medicine: Faculty and Staff Resources.
Translating students’ comments on course evaluations
Navigating students’ comments could be one of the most challenging aspects of interpreting course evaluations. In an article in Innovative Higher Education, Linda Hodges and Katherine Stanton (2007) suggest using these comments as “windows into the process of student learning and intellectual development” rather than as reviews of “how they have been entertained” by an instructor.
Hodges is Director of the Harold W. McGraw, Jr. Center for Teaching and Learning at Princeton University; Stanton is the center’s assistant director. They point out that sometimes students’ comments stem from “students’ expectations of or prior experiences with college classes” that “entail teachers standing in front of the room ‘telling.’”
For example, is a comment like “I did not learn in this class because the teacher did not teach” evidence of a lack of effective teaching, or evidence that the style of teaching – including lots of team-based work – wasn’t what the student was expecting? Reframing student comments in this light can ultimately help improve teaching, Hodges and Stanton suggest.
“We may see our evaluations less as judgments of our performance and more as insight into our students’ intellectual growth—insight that may engage us in intellectual growth as teachers and scholars.”
Hodges, L.C., and Stanton, K. (2007). “Translating comments on student evaluations into the language of learning” in Innovative Higher Education 31:279-286.
New Faculty Resources Community
You are invited to view the new Faculty Resources Community:
The Faculty Resources Community was created to provide faculty members, Course Directors, Year Directors and Committee Chairs with advance notice of topics to be discussed at Curriculum Committee meetings as well as easy access to Curriculum Committee Highlights, links to faculty development on teaching and assessment and materials distributed at Course Directors’ Retreats. Other resources posted in the new community include the Future of Medical Education in Canada reports and the latest LCME accreditation standards. We welcome suggestions for additional resources.
The School of Medicine would like to provide all of its faculty members with opportunities to comment on draft policies and procedures prior to their finalization and implementation. It is our aim to post Curriculum Committee agendae to the Faculty Resources Community 48 hours in advance of each meeting. Highlights of each Curriculum Committee meeting will be posted as soon as the minutes of the meeting have been approved. All faculty members are welcome to attend Curriculum Committee meetings, although advance notice is appreciated so that appropriate space can be booked. Faculty participation in Curriculum Committee discussions is at the discretion of the Chair Dr. Michelle Gibson. Requests may be communicated in writing prior to a meeting.
Educational Development and Faculty Support
Welcome back to another academic year–and welcome to new faculty and to new students! The Educational Development and Faculty Support Team is available to assist you.
Our role is to assist faculty in development and implementation of courses, sessions, assessments, teaching methods, and generally to assist with any educational concerns you have. We work in partnership with MEdTech and with the Bracken Health Sciences librarians to help you.
- Sheila Pinchin, Manager of the Education Team, email@example.com ext. 78757
- Theresa Suart, Educational Developer, firstname.lastname@example.org ext. 75485
- Eleni Katsoulas, Assessment and Evaluation Consultant, email@example.com
- Alice Rush-Rhodes, Special Curricular Assistant, firstname.lastname@example.org
- Catherine Isaacs, our Coordinator of Accreditation is also a part of the team, and helps us keep accreditation and quality assurance as part of our focus.
Feel free to contact any of us. We look forward to meeting you!