What do p and R-values mean anyhow? : Understanding how to interpret multiple-choice test scores using statistics.

Post Thumbnail

Have you ever wondered whether or not your multiple-choice questions (MCQs) are too easy? The answer to this question can be found in the p-values or item difficulty: the percentage of students who answered correctly. The difficulty of a MCQ can range from 0.00 to 1.00; the higher the p-value, the easier the question. What we should be concerned with are high difficulty questions with p-values less than 0.3.

Have you ever wondered which questions tricked students who otherwise performed well on a test overall? The R-value or item discrimination looks at the relationship between how well students performed on a question and their total score. Item discrimination indicates students who know the tested material and those who do not. The higher the R-value, the more discriminating the test question. We should try to remove questions on the test with discrimination values (R-values) near or less than 0.3. This is because students who did poorly on the test did better on this question than students who performed better overall.

Did you Know?

Multiple-choice questions that use words in the stem such as best, most, first, or most correct require higher-level thinking but often confuse students because they are ambiguously worded. Our students have struggled lately with ambiguity in the wording of MCQs on RATs and exams such as “Which is the most likely….”. They assume “most likely” to be “most common”, whereas the most likely answer could be an uncommon situation. It’s important to word the question clearly so that students are not confused. So for example, the question could state, “In light of the clinical information provided above, which diagnosis would you make?

You can also ask students about “most common”, “most concerning”, or “what is the first test you would perform” etc. but it is always good to anchor these stems by referring to the data presented previously. Then the key is to require them to choose, evaluate, interpret, judge, infer from data, solve problems, and apply principles.

Did you Know?

The Student Assessment Committee has posted several articles, checklists and PowerPoint slides to assist you with Multiple Choice Questions.

For more guidance on writing high-quality multiple-choice questions refer to MCQ Guidelines and Writing MCQ’s in School of Medicine Faculty and Staff Resources at:

http://meds.queensu.ca/home/faculty_staff_resources/assessment_resources

 

References

http://ctl.utexas.edu/programs-and-services/scanning/interpreting-results/

http://www.washington.edu/oea/services/scanning_scoring/scoring/item_analysis.html

Queen’s School of Medicine: Faculty and Staff Resources.
http://meds.queensu.ca/home/faculty_staff_resources/assessment_resources

Leave a Reply

Post Timeline

Curriculum Committee Information – September 28, 2017
Published Wed, November 15, 2017

Faculty and staff interested in attending Curriculum Committee meetings should contact the Committee Secretary, Candace Miller (umecc@queensu.ca), for information relating to agenda items and meeting schedules. A meeting of the Curriculum Committee was held on September 28, 2017.  To review the topics discussed at this meeting, please click HERE to view the agenda. Faculty interested in reviewing the minutes of the September … Continue reading

Grade Inflation – the “dirty little secret” of academia
Published Mon, November 13, 2017

“Would any of us have gotten into medical school today?” This was the tongue-in-cheek question I posed to my classmates at our medical school reunion last year. They were rather amused by it and, being very much aware of the high academic standards required by our current admissions processes, believed the answer was an obvious “no”. I tried to raise … Continue reading

Facebook thinks I’m a doctor…
Published Mon, November 6, 2017

  And other unusual things that happen when you’re an educational developer at a medical school It’s a unique and interesting thing being one of the non-medically-trained employees who work (mostly behind the scenes) to help run the undergraduate medical education program at Queen’s. On the one hand, friends and family can sometimes think I’ve magically completed medical school in … Continue reading

Nominations open for next Exceptional Healer Award
Published Mon, October 30, 2017

Instilling the values of patient-centered care is one of our goals in the UGME program. It’s also what the Kingston Health Sciences Centre Exceptional Healer Award recognizes in physicians from both the Hotel Dieu and KGH sites. Launched earlier this year, the Exceptional Healer Award is sponsored by the KHSC Patient & Family Advisory Council. It honours a physician who … Continue reading

Students striving to make a difference in our community
Published Mon, October 23, 2017

One of the attributes that our Admissions Committee works very hard to identify in applicants is a commitment to service. This has multiple dimensions, involving service to both individual patients and communities. It’s therefore always very gratifying to learn of efforts such as that described below in todays guest article provided by students Lauren Wilson, Katherine Rabicki and Melissa Lorenzo. … Continue reading