Seeing What They Want to See

Just in case you believe that what you think doesn't bias what you "see."

Student Evaluations of Teaching (Mostly) Do Not Measure Teaching Effectiveness

Anne Boring, Kellie Ottoboni, & Philip Stark

2016-01-07

ScienceOpen Research

Abstract: Student evaluations of teaching (SET) are widely used in academic personnel decisions as a measure of teaching effectiveness. We show:

• SET are biased against female instructors by an amount that is large and statistically significant
• the bias affects how students rate even putatively objective aspects of teaching, such as how promptly assignments are graded
• the bias varies by discipline and by student gender, among other things
• it is not possible to adjust for the bias, because it depends on so many factors

• SET are more sensitive to students' gender bias and grade expectations than they are to teaching effectiveness
• gender biases can be large enough to cause more effective instructors to get lower SET than less effective instructors.

These findings are based on nonparametric statistical tests applied to two datasets: 23,001 SET of 379 instructors by 4,423 students in six mandatory first-year courses in a five-year natural experiment at a French university, and 43 SET for four sections of an online course in a randomized, controlled, blind experiment at a US university.

Source: https://www.scienceopen.com/document/vid/818d8ec0-5908-47d8-86b4-5dc38f04b23e

Inside Higher Ed has a lengthy piece about this study: https://www.insidehighered.com/news/2016/01/11/new-analysis-offers-more-evidence-against-student-evaluations-teaching

Heading Down

Inside Higher Ed has a lengthy article about a new AAUP survey of teaching evaluation practices.* (No link is provided to the survey and I couldn't find it after poking around on the web this morning.) However, one statement caught my eye:

Respondents who said their institutions had adopted online evaluations reported much lower student return rates than those who stuck with paper evaluations: 20-40 percent versus 80 percent or higher. “With such a rate of return, all pretensions to ‘validity’ are rendered dubious,” the paper says. “Faculty report that the comments coming in are from the students on either of the extremes: those very happy with their experience and/or their grade, and those very unhappy.”

There is a counterpart to this observation with regard to MOOCs. In principle, they reach everyone by being online but actual completion rates for MOOCs are low. Of course, there are other issues with regard to standard teaching evaluations other than response rates. Nonetheless, in the case of evaluations note that the same computer technology that makes possible the gathering of such evaluations online could produce near universal response rates if completion of an evaluation was a requirement for obtaining credit for the course. Surely a bit of computer programming could handle the link between evaluation completion and course completion. (You could add the ability to choose to abstain from answering any or all of the questions after signing on to evaluate for students who had some philosophical objection to the process or feared that they would be identified.)
---
*https://www.insidehighered.com/news/2015/06/10/aaup-committee-survey-data-raise-questions-effectiveness-student-teaching

Student Evaluations of Teaching: Always a Hot Topic

What are they saying?

Inside Higher Ed today notes complaints from Australian faculty about increased use of student evaluations of teaching for faculty advancement.  The article is at

http://www.insidehighered.com/quicktakes/2012/08/13/australian-professors-object-student-evaluations and it links to an article in the Sydney Morning Herald which says (excerpt):

…Teachers at the Australian National University now need to explain themselves if too many students are not pleased with them and colleagues must argue why courses with student satisfaction rates of less than 50 per cent should be kept. One teacher emailed the union: ''I feel under pressure to lower standards and make the student experience more comfortable so I don't end up before the head of school to explain myself.'' …

Full article at

http://www.smh.com.au/national/postgraduate-education/anu-staff-object-after-getting-an-f-for-teaching-20120811-2415o.html

Of course, there has been much research on this issue, e.g.:

“Hot or Not: Do Professors Perceived as Physically Attractive Receive Higher Student Evaluations?”

The answer, by the way, is “yes.”

See http://www.suu.edu/faculty/robertsw/Evaluation%20Task%20Force/Hot%20or%20not.pdf

So here’s the problem for our colleagues in Australia: