On Nov. 6, a band of intrepid medical educators from Queen’s travelled to Philadelphia to attend the Association of American Medical Colleges’ annual meeting.

Attending sessions from Nov. 6 to 10, Dr. Renee Fitzpatrick, Dr. John Drover, Dr. Laura McEwen, (Assessment Specialist in Post Grad), Ulemu Luhanga (a doctoral candidate working as a research assistant with Laura), Eleni Katsoulas (our UG Assessment and Evaluation Consultant) and I learned a great deal from our American counterparts.

But I would venture to say that they learned some from us too.  For example Laura and Ulemu presented their poster Queen’s Multisource Feedback Rubrics: Operationalizing Frames of Reference for Raters and Residents, and judging from the crowd around them during the whole session, their ideas were well-received.

I attended a great morning with the Directors of Clinical Skills group, where milestones were the topic of discussion and I’ve been invited to share our work on the clinical skills milestones we’re developing.  I’ll write more about milestones and the work from the AAMC in a later blog.

And Eleni went to hear the speakers of a great article, Jim Crossley and Brian Jolly, who wrote Making sense of work‐based assessment: Ask the right questions, in the right way, about the right things, of the right people.

Here she writes about their presentation and their article.  Please feel free to share your thoughts

Multiple Perspectives: finding relevance in idiosyncrasies

 

owl or coffee cups 2

http://www.listofimages.com/coffee-owl-coffee-beans-coffee-cup-creative.html

What do you see in the picture above?

Do you see an owl? Or coffee beans and two cups of coffee? Some of you might say you see both things! Each piece is only part of the whole. This illustrates how different people can look at the same thing and see different things. Any one of these perspectives might be useful depending on the context!

How are  subjectivity and reliability related?

Does every context provide equally valid and reliable data for every domain? Crossley & Jolly (2012) argue that not every context provides good data for accessing every domain! So when might standing back and considering the whole performance give you a better picture than the sum of its parts? Or when are other perspectives just as useful?

The literature demonstrates that those who “have the competence to judge an aspect of performance, and have had the opportunity to observe it, appear to provide more reliable ratings” (Crossley & Jolly,p.35). Since clinical competence is so broad, isn’t it rational to say that multiple perspectives might be equally valid in certain contexts?

What does this mean for those of us teaching and assessing in medical education?
The authors argue that while historically, assessments have often
measured the measurable now we are concerned with measuring the important.  Workplace based assessments are often based on subjective judgements.  Sometimes the assessors develop an ” instrumental impressionism” whereby he or she makes a judgement that is global but, nevertheless, is vitally dependent on an overall, somewhat merged,
perception of the details (Crossley & Jolly, p. 33) and this is an equally valid if not more valid perspective.

For us, a few things emerge:  Some of our finely grained assessments are asking too much of assessors to provide meaningful assessment, especially in cases of concepts that are hard if not impossible to observe in the assessment setting.  Our assessors should be those with experience and knowledge in the specific areas of assessment, and their judgements as opposed to objective observations can and should play a large role in assessment. And finally, we should ask the right questions, in the right way, about the right things, of the right people.

References

Crossley, J. & Jolly, B. (2012). Making sense of work-based assessment: ask the right questions, in the right way, about the right things, of the right people. Medical Education, 46, p-28-37.