How much course evaluation feedback is “just right”?

How much feedback is too much feedback? How much is just enough?

That’s a question both the Course and Faculty Evaluation Committee (CFRC) and our students have been exploring.

At present, students are required to complete a 15-question course evaluation for each course as they complete it. As well, they’re required to complete faculty evaluations for each faculty member who taught at least four hours during that course. For our pre-clerkship students, this translates into 24 courses over the first two years of our program. Some courses are divided into units for evaluation, so that further increases the evaluation load.

As noted in a recent CFRC report to the Curriculum Committee: “Response rates have dropped significantly during the previous academic year on all course and faculty evaluations. It is assumed that a major contributing factor to the fall is the number of evaluations students are being asked to complete.”

We won’t ever do away with student course evaluations as these provide valuable feedback for curricular improvements. The CFRC is interested, however, in reducing the evaluation workload for students while still collecting solid feedback.

After consulting with the Aesculapian Society, the CFRC has proposed that only a subset of students will be asked to complete course and faculty evaluations for each course. Remaining students will have the option to complete evaluations. (In other words, students will always be able to comment on any of their courses and faculty if they want to provide additional feedback).

To determine if this will result in greater compliance (and data adequate for evaluation purposes), the CFRC will pilot this procedure on several Term 2 and 4 courses. The pilot project (Reduced number of targeted respondents for course and faculty evaluations), was approved by the Curriculum Committee at its November meeting.

For the pilot, students in both Meds 2019 and 2020 will be divided into randomized groups of 25 students each. One group of 25 students will be assigned to complete evaluations for each of the courses in the pilot.

Courses included in the pilot will be:course-eval-screen-shot

Meds 2020

  • Meds 121 Fundamentals of Therapeutics
  • Meds 125 Blood and Coagulation
  • Meds 127 MSK

Meds 2019

  • Meds 240 Genitourinary and Reproduction
  • Meds 241 Gastroenterology and Surgery
  • Meds 245 Neurosciences
  • Meds 246 Psychiatry

All students will be asked to complete the term 2 and term 4 course and faculty evaluations for those courses not included in the pilot. Also, Course Directors for the targeted pilot courses will be asked to confirm if there are any faculty to be excluded from the reduced pool of respondents and included in a group to be completed by the entire class.

Results of the pilot will be reported to the Curriculum Committee in August 2017.

Posted on

Bringing things into focus: Using focus groups to collect feedback

By Theresa Suart & Eleni Katsoulas

Amongst the plethora of student feedback we solicit about our courses, you may wonder why we sometimes add in focus groups. What could be added to the more than a dozen questions on course evaluation and faculty feedback surveys?

The information we gather in student focus groups doesn’t replace the very valuable narrative feedback from course evaluations, rather, it allows us to ask targeted questions, clarify responses and drill down into the data.

Developed from “focused interviews” around the time of the Second World War, focus groups emerged as a key qualitative research tool in the latter half of the 20th century. Robert K. Merton, a sociologist from Columbia University, is hailed as the “father of the focus group.” (He died in 2003 at age 92.)

Merton used focused interviews to gain insight into groups’ responses to text, radio programs and films. Politicians and marketing companies soon seized upon focus groups to gauge voter and consumer trends. The Queen’s UGME Education Team uses focus groups in a targeted way to augment information gleaned from course evaluation feedback, course director’s meetings with academic reps and other feedback tools.

According to a briefing paper from Carnegie Mellon University, focus groups are “particularly effective” for eliciting suggestions for improvement. “They are also much more flexible than surveys or scales because they allow for question clarification and follow-up questions to probe vague or unexpected responses.” It also helps that faculty rate focus groups as “accurate, useful and believable”.

If you’re asked to participate in a focus group, only agree if you think you have something to contribute to the investigator’s project or purpose. (Sure, it’s fun to come for the free food, but be prepared to contribute in a meaningful way).

groupe-discussion2-1What you can expect when you take part in a focus group:

  • To be informed if the focus group is for research or curricular innovation (or both). Research studies must have approval from the Research Ethics Board and require specific paperwork to document informed consent. Curricular innovation focus groups are less formal, but will still respect confidentiality of participants. These might not have the same paperwork.
  • The facilitator to set the ground rules, and guide the discussion. Savvy facilitators will do this with a minimum of fuss: they will listen more than they speak. (But you can certainly ask for clarification if you’re not sure of a question).
  • A co-facilitator will likely take notes and monitor any recording equipment used. The co-facilitator may summarize after each question and solicit further input as required.
  • You’ll be asked specific questions, and engage in conversation with the other participants.

What you shouldn’t expect:

  • A venting session. This isn’t the time to just complain. A focus group is looking for constructive feedback and suggested solutions.
  • To always have your say: the facilitator may realize they have reached saturation on a particular question and will move on. This is to respect your time. (You’ll have an opportunity to send additional comments electronically afterwards if you felt there is an important point that was missed).

What you can do to prepare:

  • If the questions are provided in advance (this is best practice but not always possible on tight timelines!) you should take some time to think about them.
  • Be sure you know where the meeting room is, and arrive on time.

What you can do during:

  • Contribute, but make sure you don’t end up dominating the conversation. The facilitator will be looking for a balance of views and contributors.
  • Listen attentively to others and avoid interrupting. The facilitator will make sure everyone has a chance to contribute – you’ll get your turn.

What you can expect from data collected at a focus group:

  • It will be confidential. Different strategies are employed. For example, you may be assigned a number during the focus group and participants asked to refer to people by number (“Participant 2 said…”).
  • In a formal research study, you should be offered an opportunity to review the data transcript after it is prepared. (This is sometimes waived on the consent form, so read carefully so you can have realistic expectations of the investigator).
  • The end product is a summary of the conversation, with any emergent themes identified to answer the research questions.

What you can’t expect:

  • A magic bullet solution to a challenge in a course or class.
  • One hundred percent consensus from all participants – you can agree to disagree.
  • For all outlier opinions to be represented in the final report. These may be omitted from summary reports.

We’re always grateful to our students for donating their time to our various focus group requests throughout the year. These contributions are invaluable.

If you think this type of data collection could be useful in your course review and revisions, feel free to get in touch. It’s one of the tools in our qualitative research toolbox and we’re happy to deploy it for you as may be appropriate.

Eleni Katsoulas eleni.katsoulas@queensu.ca

Theresa Suart theresa.suart@queensu.ca

 

Posted on