What do p and R-values mean anyhow? : Understanding how to interpret multiple-choice test scores using statistics.

Have you ever wondered whether or not your multiple-choice questions (MCQs) are too easy? The answer to this question can be found in the p-values or item difficulty: the percentage of students who answered correctly. The difficulty of a MCQ can range from 0.00 to 1.00; the higher the p-value, the easier the question. What we should be concerned with are high difficulty questions with p-values less than 0.3.

Have you ever wondered which questions tricked students who otherwise performed well on a test overall? The R-value or item discrimination looks at the relationship between how well students performed on a question and their total score. Item discrimination indicates students who know the tested material and those who do not. The higher the R-value, the more discriminating the test question. We should try to remove questions on the test with discrimination values (R-values) near or less than 0.3. This is because students who did poorly on the test did better on this question than students who performed better overall.

Did you Know?

Multiple-choice questions that use words in the stem such as best, most, first, or most correct require higher-level thinking but often confuse students because they are ambiguously worded. Our students have struggled lately with ambiguity in the wording of MCQs on RATs and exams such as “Which is the most likely….”. They assume “most likely” to be “most common”, whereas the most likely answer could be an uncommon situation. It’s important to word the question clearly so that students are not confused. So for example, the question could state, “In light of the clinical information provided above, which diagnosis would you make?

You can also ask students about “most common”, “most concerning”, or “what is the first test you would perform” etc. but it is always good to anchor these stems by referring to the data presented previously. Then the key is to require them to choose, evaluate, interpret, judge, infer from data, solve problems, and apply principles.

Did you Know?

The Student Assessment Committee has posted several articles, checklists and PowerPoint slides to assist you with Multiple Choice Questions.

For more guidance on writing high-quality multiple-choice questions refer to MCQ Guidelines and Writing MCQ’s in School of Medicine Faculty and Staff Resources at:

http://meds.queensu.ca/home/faculty_staff_resources/assessment_resources

 

References

http://ctl.utexas.edu/programs-and-services/scanning/interpreting-results/

http://www.washington.edu/oea/services/scanning_scoring/scoring/item_analysis.html

Queen’s School of Medicine: Faculty and Staff Resources.
http://meds.queensu.ca/home/faculty_staff_resources/assessment_resources

Posted on

Translating students’ comments on course evaluations

Navigating students’ comments could be one of the most challenging aspects of interpreting course evaluations. In an article in Innovative Higher Education, Linda Hodges and Katherine Stanton (2007) suggest using these comments as “windows into the process of student learning and intellectual development” rather than as reviews of “how they have been entertained” by an instructor.

Hodges is Director of the Harold W. McGraw, Jr. Center for Teaching and Learning at Princeton University; Stanton is the center’s assistant director. They point out that sometimes students’ comments stem from “students’ expectations of or prior experiences with college classes” that “entail teachers standing in front of the room ‘telling.’”

For example, is a comment like “I did not learn in this class because the teacher did not teach” evidence of a lack of effective teaching, or evidence that the style of teaching – including lots of team-based work – wasn’t what the student was expecting? Reframing student comments in this light can ultimately help improve teaching, Hodges and Stanton suggest.

“We may see our evaluations less as judgments of our performance and more as insight into our students’ intellectual growth—insight that may engage us in intellectual growth as teachers and scholars.”

Hodges, L.C., and Stanton, K. (2007). “Translating comments on student evaluations into the language of learning” in Innovative Higher Education 31:279-286.

 Permalink: http://resolver.scholarsportal.info/resolve/07425627/v31i0005/279_tcoseitlol

 

Posted on

Boy Scouts, Role Models and the Hidden Curriculum

Last Saturday morning, entering our local Loblaws supermarket, my wife and I were confronted by an adorable and entirely engaging boy of about 8 years of age dressed in a Boy Scout uniform.  He handed us a plastic bag and explained in a most earnest and obviously practiced speech that they were collecting for the Food Bank and we were invited to fill the bag during our shopping.  He was polite, articulate, sincere and clear, both about the process and ultimate destination of the donations.  In short, he was utterly irresistible, and we would have been convinced even if his cause had not been so worthy.

As he was speaking, I hadn’t really noticed the gentleman standing behind him, dressed in a version of the same uniform, who now spoke up and greeted me by name.  I recognized Bill Racz, my former Professor of Pharmacology, who had taught me many years ago about adrenergically active medications and the evils of pharmaceutical advertising.  I’ve continued to encounter Bill around campus over the years in contexts ranging from teaching and committee work to our mutual incompetence at noontime basketball at the gym.  In talking to Bill that morning, I learned for the first time that he’s been involved in the Boy Scouts movement for over 35 years.

On the way home, I couldn’t help reflecting on the tremendous generosity of spirit that motivates an accomplished and highly respected academic to donate time and energy to such a community cause and, more importantly, to modeling those values to young people in the most powerful way possible, by actually living the experience.  It’s easy to imagine that young boy one day taking on the same role and passing those lessons on to another generation.

The powerful influence of role modeling in medical education is well appreciated.  Medical graduates invariably recall particular teacher/mentors as much more influential to their eventual development than any curricular element or teaching methodology.  At a medical leadership symposium I attended recently, panelists were invited to individually list key components of effective leadership.  Common to every list was some variation on “lead by example”.  An extensive body of research is emerging on the “Hidden Curriculum”, a term used to refer to all the factors that influence learner development but are outside planned curriculum, arising as a result of observed behaviors and attitudes expressed unintentionally.  What’s becoming clear in the education world, and has always been clear to good parents, is that what we do is much more powerful that what we profess.  Good teachers and good leaders know this and therefore strive to “walk the walk”.

By “walking the walk” that Saturday morning, Bill Racz was providing an invaluable example and living lesson to a group of young boys.  He continues to teach and inspire me.

Posted on

Auda abu Tayi would have made a great Course Director

The Academy Award winning motion picture “Lawrence of Arabia”, provides surprising and instructive studies in leadership.   The main character, T. E. Lawrence, portrayed brilliantly by Peter O’Toole, is a junior officer in the British Army during the First World War.  He finds himself unable to conform to a model of leadership that demands unquestioning obedience to superiors of dubious competence, and threatens harsh punishment for questioning authority or any act of “insubordination”.  After being assigned to explore the Sahara desert and seek out alliances with the Bedouins, he encounters Auda abu Tayi, a tribal chief portrayed with equal brilliance by Anthony Quinn.  Lawrence is perplexed by the apparent paradox of Auda’s ability to unify usually rebellious and independent individuals, despite their “uncivilized” state and lack of any traditional military regulations or hierarchical command structure.   In a particularly memorable scene, the two are negotiating in Auda’s luxurious tent.  Lawrence presses Auda, challenging his authority and ability to maintain control of his people.   Auda rises in anger, lists the many wounds he has suffered for his people, points dramatically to the throngs of tribesmen waiting outside, and declares, “I AM A RIVER TO MY PEOPLE”.  The entire tribe, on cue, rises and cheers in agreement.  Lawrence gets the message.  He understands, for the first time, that Auda’s authority stems not from inherited right or fear of reprisal, but from the fact that his leadership is earned by understanding and providing for the needs of those he leads.  He will maintain his influence as long as he provides.

History teaches that every prominent and highly successful leader achieves and maintains authority by understanding and providing for the needs of whatever political or social structure they lead.  Leadership and service are intertwined.  Similarly, every great overthrow or rebellion can be traced to a failure to continue to provide for those needs.

Leadership in a medical or academic context would seem a long way from the Arabian Desert or rebellions, but some intriguing presentations I attended recently at the AAMC annual meeting on the topic would suggest there are similarities worth noting.  Dr. Eugene Washington, a highly respected leader in medical education and Dean of UCLA School of Medicine, lists the following elements as essential to leadership at any level, whether it’s direction of a course, department or entire university:

  1. Leaders create a shared vision of what the organization, or group, is striving to achieve.
  2. Leaders affirm the core values of the organization
  3. Leaders motivate individual members, by finding roles that are of value and interest to both the member and the organization
  4. Leaders achieve a “workable unity” within the organization, which is a commonly held understanding of every member’s role in achieving organizational goals
  5. Leaders manage, which means solving problems and ensuring both fairness and openness in the execution of those solutions.  This also means assuring accountability for all members.
  6. Leaders continually communicate and seek input from their members
  7. Leaders serve as a symbol and lead by example.

This might seem rather lofty and a long way from the role of a Course Director in the Undergraduate curriculum, but I believe these elements are operative in any leadership role, and that any effective change requires solid leadership.  We’ll explore this further in upcoming blogs.  In the meantime, I’d suggest dropping by Classic Video to pick up a copy of Lawrence of Arabia.

Posted on

Copyright Update

A letter has been sent to all of us from the Provost’s Office about copyright.  In essence it states:

  1.  Queen’s has decided along with many other universities not to sign a license with Access Copyright.
  2. All course materials must comply with Copyright Act and other university policies/guidelines. Do NOT post scanned copyrighted material to MEdTech (or Moodle) or on a course website without obtaining clearance from the Copyright Advisory Office.
  3. The Copyright Advisory Office’s service will facilitate copyright compliance for course materials.  This involves investigation and purchase of copyright clearance for chapters from a book or other print source.
  4. Course Packs and Library course reserves are other options for you.
  5. The Copyright Advisory Office can help you select specific options for your courses.
  6. A new system for this academic year will assist you in uploading materials and assuring materials comply with both accessibility and copyright materials.  More details will follow in the fall.
  7. Please contact the Copyright Advisory Office, using their website:  which has FAQs etc.  Or contact Copyright Specialist Mark Swartz at (613) 533-6000 ext 78510 Email: mark.swartz@queensu.ca
  8. For the complete letter from the Provost’s Office click here.

Posted on

Curricular Goals and Objectives

In an article entitled “The Case for Core Curriculum”, author James Bradshaw raises concerns regarding university teaching that should cause some unease as we consider our medical courses and curriculum.  “The tide seems to be turning”, he writes, “with business leaders lamenting that, although the new talent arriving at their doorsteps has deep technical knowledge, it lacks the skills needed to put this knowledge to full use”. (http://www.theglobeandmail.com/news/national/time-to-lead/why-university-students-need-a-well-rounded-education/article4610406/)

Rather disturbingly, this observation echoes the growing concern among postgraduate training directors and clinical faculty that our graduates seem adept and comfortable providing factual information, but considerably less so when challenged to assess undifferentiated patient presentations and integrate factual information into cogent and practical management plans.  Lest we dismiss such commentary as isolated rumblings, it’s useful to keep in mind that the results for Queen’s graduates on the MCC Part 2 examination would seem to support the contention that our graduates struggle in the domains of clinical reasoning and comprehensive patient management, in sharp distinction to their well above average performance in knowledge-based components of the Part 1 examination.  Although the effects of our revised curriculum and enhanced assessment practices are not yet influencing examination results, it would seem unwise to simply dismiss these observations.

Bradshaw goes on to point out that, at the university undergraduate level, there exist political, logistic and economic barriers to providing integrated educational experiences that address what we, in the medical education context, would term “competencies” rather than traditional discipline-based content.  Indeed, we are well aware of the challenges of blending traditional disciplines and developing both content and assessment that address what many refer to as the “softer skills” relevant to medical practice, such as critical thinking, communication with disparate populations, collaboration, and the ability to advocate effectively for patients, health system delivery, and oneself.

Our curricular goals and objectives, as well as our Competency Framework (see “Curricular Goals and Competency based Objectives”), was developed in 2007 and has served as the central focus for the restructuring of curriculum and assessment methods that has allowed us to both develop a much more effective learning experience for our students, and achieve compliance with all accreditation standards.  The principles it espouses should remain our central guiding force.  However, the observations noted above should cause us to consider whether a clearer definition of the expectations we have for our graduating students is in order, recognizing that many of the competencies we espouse (professionalism, advocacy, communication, collaboration) are not ends in themselves, but necessary components of a graduates “competence” to assess, diagnose and manage patients with a variety of clinical presentations.

I’ll be encouraging a dialogue on this issue at our major committees and among our Course Directors.  This blog seems an appropriate place to start.  Feel free to provide feedback.

 

Posted on

Can Students Multitask?

You may have noticed an occasional student referring to his Facebook page, or her ipod or ipad while also apparently listening to your lecture, or working with her/his team-mates in small group learning.  They are multitasking, as part of the “M” generation.  But are they really multitasking?  And is it working for them as successful learners?  Dr. MaryEllen Weimer has collected evidence in her article that students compromise their learning by multitasking and suggests we present them with the evidence to help them re-evaluate their approach to learning.  For a synopsis of the research she has collected, go to Faculty Focus.

Posted on

Pager service in the New Medical Building

When the New Medical Building was completed in 2011 it did not take long for physicians to notice that their pagers were not reliably receiving hospital pages while they were in the building. This was a big issue because it meant that if you were on-call neither your pager or your cell phone would work. Once this problem was brought to our attention, Queen’s IT Services worked with PageNet to install an amplifier that would boost the PageNet signal within the building, thus solving the problem… for PageNet customers. We later discovered that some physicians were using Alliance paging services and were still encountering pager reliability issues.

After a brief discussion with Alliance about the issue they came up with a solution for us to distribute, so we thought this blog would be a good way of letting folks know.

If you are an Alliance paging customer and you would like your pager to work while you are physically located in the New Medical Building please contact Rita Peters (Alliance paging specialist) at 613-546-1141. Rita will assist you with upgrading your Alliance pager to one that is compatible with the PageNet signal. There is a small monthly price increase associated with this upgrade; however, your pager will work in the New Medical Building with this plan.

Posted on

Curriculum Matters: Giving Feedback

“Learning without feedback is like learning archery in a darkened room.” (Cross)

The terms feedback and formative assessment have been used frequently in our UG curriculum both in clerkship and pre-clerkship. It’s important that we all recognize a consistent definition of feedback, as well as consider some effective tips and strategies from the educational literature. Sue Fostaty-Young weighs in with some helpful ideas.

Feedback is information regarding students’ performance that is offered with the express purpose of improving their learning and future performance. It is considered one of the most powerful influences on learning and achievement (Hattie & Timperley, 2007).

Tips for Giving Effective Feedback:

  1. Prepare students to receive the feedback you’re about to give
    • Unfortunately, students do not always recognize feedback as such, especially oral feedback during rounds, or in clinic, etc. Consider starting your end of the conversation by saying “I have some feedback for you”.
    • Alert students at the beginning of a rotation or any workplace teaching to the fact that feedback may be oral, and part of ongoing bedside teaching.
  2. Ask a student how he/she thinks they’re doing
    • In asking a student to comment on their own performance you initiate the process of self-reflection and emphasize the learner’s active role in the feedback process.
    • You may ask them to prepare some notes prior to the feedback session or create a structure for this dialogue.
  3. Be specific and objective about what you observed
    • Begin by identifying what students are doing correctly in their performance.
    • Provide examples of what you observed, not what you inferred. For example, rather than saying “You’re shy” (an inference), you might say “I noticed you made very little eye contact with the patient and you spoke very quietly, making it difficult for her to hear your questions” (an observation).
  4. Give feedback in a timely fashion and in time for students to improve
    • Feedback offered almost immediately after an activity has more meaning because the event and the students’ behaviours during it are fresh in their minds.
    • Timely feedback allows for students to alter their performance, to practice, and improve. Set a time to review improvement if possible.
  5. Outline specific targets or goals; articulate expectation
    • Feedback has the most impact when learning-specific information is offered.
    • Offer specific feedback that is within the student’s ability to improve and a rationale for your suggestions. For example, rather than say, “Stop being so shy,” you might offer specific suggestions such as “Try to make eye contact more regularly so patients know you’re engaged and to gauge their reactions. Speak louder so your instructions can be heard.” You may want to suggest that they practice with a peer or videotape their own practice.
  6. Invite the student to respond to the feedback you’ve just given
    • Initiating this dialogue can help establish whether the student has interpreted the feedback as it was intended, and will be able to use the feedback in a meaningful way.
  7. Help the student develop an action plan with timelines for improvement to help them use the feedback in a constructive way.

Adapted from:
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81-112.

Posted on

New Faculty Resources Community

You are invited to view the new Faculty Resources Community:

The Faculty Resources Community was created to provide faculty members, Course Directors, Year Directors and Committee Chairs with advance notice of topics to be discussed at Curriculum Committee meetings as well as easy access to Curriculum Committee Highlights, links to faculty development on teaching and assessment and materials distributed at Course Directors’ Retreats. Other resources posted in the new community include the Future of Medical Education in Canada reports and the latest LCME accreditation standards. We welcome suggestions for additional resources.

The School of Medicine would like to provide all of its faculty members with opportunities to comment on draft policies and procedures prior to their finalization and implementation. It is our aim to post Curriculum Committee agendae to the Faculty Resources Community 48 hours in advance of each meeting. Highlights of each Curriculum Committee meeting will be posted as soon as the minutes of the meeting have been approved. All faculty members are welcome to attend Curriculum Committee meetings, although advance notice is appreciated so that appropriate space can be booked. Faculty participation in Curriculum Committee discussions is at the discretion of the Chair Dr. Michelle Gibson. Requests may be communicated in writing prior to a meeting.

Posted on