Measuring Teaching Effectiveness: Correspondence Between Students’ Evaluations of Teaching and Different Measures of Student Learning

Abstract

Relating students’ evaluations of teaching (SETs) to student learning as an approach to validate SETs has produced inconsistent results. The present study tested the hypothesis that the strength of association of SETs and student learning varies with the criteria used to indicate student learning. A multisection validity approach was employed to investigate the association of SETs and two different criteria of student learning, a multiple-choice test and a practical examination. Participants were N = 883 medical students, enrolled in k = 32 sections of the same course. As expected, results showed a strong positive association between SETs and the practical examination but no significant correlation between SETs and multiple-choice test scores. Furthermore, students’ subjective perception of learning significantly correlated with the practical examination score whereas no relation was found for subjective learning and the multiple choice test. It is discussed whether these results might be due to different measures of student learning varying in the degree to which they reflect teaching effectiveness.

Appendices

Appendix A: Exemplary item of the MC test, type A+

A 67-year-old male patient presents with a histologically verified adenocarcinoma of the rectum. Which of the following examinations yields the important cue for defining the therapeutic concept (neoadjuvant radiotherapy versus primary operation)?

• Endosonography

• Proctoscopy

• Colonoscopy

• Sphincter manometry

• Barium enema

Appendix B: Exemplary checklist for assessment of OSCE performance

• For each item, the student can achieve a maximum of 5 points.

• 5 points: all items fulfilled without help

• 3 points: all items fulfilled with defined help of the examiner

• 1 point: items fulfilled incompletely despite defined help of the examiner

Task

Information to the student: Please perform an abdominal examination on this young patient and comment on what you are doing! The patient suffers from pain of the right lower abdominal quadrant since yesterday. He has no diarrhoea, no vomiting and no dysuria. There is no relevant former disease history. You do not need to perform an anamnesis!!!

Information to the examiner: Please instruct the simulated patient to lie on the stretcher with legs bent and arms entwined behind the head! The simulated patient has been trained to react strongly to palpation of the right lower abdominal quadrant, simulate signs of peritonitis there and give an impression of feeling ill.

Item 1: Auscultation

Information to the examiner: Use global rating between 0 and 5 points on this item and take security in performing and commenting into consideration!

The student…

• introduces himself to the patient and explains to him the task he has to perform

• asks the patient to lie relaxed with legs sprawled out or slightly bent and arms next to the body

• starts with the auscultation in order not to mask pathological finding by prior palpation

• comments on t the auscultation in all 4 quadrants

Item 2: Palpation

Information to the examiner: Use global rating between 0 and 5 points on this item and take security in performing and commenting into consideration. If student does not spontaneously examine Lanz’s point, McBurney’s point and the psoas sign, ask him/her to do so and reduce on the rating accordingly!

The student…

• starts palpation in the left lower quadrant distant to the point of maximal pain

• palpates all quadrants first superficially so as not to miss abnormalities of the abdominal wall

• performs a second deep palpation in all 4 quadrants

• palpates the area of maximal pain at last

Item 3: Signs of peritonitis

Information to the examiner: Use global rating between 0 and 5 points on this item and take security in performing and commenting into consideration. If student does not spontaneously examine signs of peritonitis, ask him/her to do so and reduce on the rating accordingly!

The student examines and comments on the following signs of peritonitis…

• abdominal tenderness

• ipsilateral and contralateral rebound tenderness

• and interpretation as local or generalized peritonitis

Item 4: Rectal examination

Information to the examiner: If student does not spontaneously mention the need of a rectal examination in this case, ask him/her whether and why he would perform a rectal exam and ask him to explain what he would specifically pay attention to. Use rating of single answers on this item (see below)!

The student pays attention to the following aspects:

• Stool in the rectal ampulla as a sign of constipation (1 point)

• pathological lump, e.g. tumour, polyp (1 point)

• sphincter tonus size and consistency of the prostate in males (1 point)

• palpation of Douglas’s space, where pain is a sign of Douglas’ abscess (2 points)

Item 5: Interpretation of results and further

Information to the student: You have now gathered clinical findings by examining the abdomen. The rectal examination is painful when palpating the Douglas’ space. How do you interpret these results and what further medical information will you gather?

Information to the examiner: Use rating of single answers on this item (see below)!

The student…

• interprets the results as strong signs of acute appendicitis, possibly already perforated (2 points)

• asks for lab works on signs of inflammation and needed prior to an operation (e.g. blood count for leucocytes, CRP, electrolytes, coagulation tests) (1 point)

• either asks for an additional ultrasound or argues for the decision of an operation on the basis of the clinical findings (2 points)

Reason for rating below 17 points (minimal competence expected on the basis of the standard setting):