The Wisdom of Crowds of Frat Boys

Over at evolgen, RPM is indignant about being rated by students, citing some pig-ignorant comments from RateMyProfessors. Interestingly, someone brought this up to the Dean Dad a little while ago, and he had an interesting response:

A reader wrote to ask a dean's-eye perspective on ratemyprofessors.com.

The short version: I consider it electronic gossip.

The slightly longer version: I've had enough social science training to know that small, self-selected samples lead to skewed outcomes.

The long version: and yet, from what I've seen, the ratings usually aren't that far off. Which is fascinating, given that the methodology is so bad.

I don't have a lot of ratings on RateMyProfessors (and I don't want a bunch of fake ones, either, so don't bother), but this description fits my experience with student evaluations in general.

I've generally found that, when I can manage to step back and look at them dispassionately, I have the same reaction to the comments I get on teaching evaluations. There's usually one or two outliers-- there's always somebody who thinks it's the Worst Class Ever-- but for the most part, the aggregate comments tend to correlate pretty well with how successful the class really was. The classes where I've been happy with my own performance have generally gotten me good evaluations. The classes where I've gotten mostly negative comments have been classes in which I wasn't at my best-- the first term with a new syllabus, the term when I got in a shouting argument with the guy teaching one of the other sections, etc.

The numerical scores are a little dodgier, in part because the questions on the bubble-sheet form aren't all that well formed, but the written comments taken as a whole have been fairly accurate. The hard part is managing to step back a bit and see that-- it's much too easy to take the negative comments personally, particularly as junior faculty, when your whole career seems to be riding on the poorly-expressed opinions of adolescents.

A couple of important disclaimers, here: I have the obvious advantage of being white and male and teaching in the physical sciences. I have heard from female colleagues and colleagues who teach classes that touch on race and gender issues that they get some astonishingly vile comments on the anonymous forms. I've also had the good fortune to mostly teach in sections for science and engineering majors, rather than general education classes full of economics majors who are just trying to fill a requirement.

That said, my experience with evaluations in general has been fairly similar to the Dean Dad's experience with RateMyProfessors: I think the student evaluation system we use is terrible in many ways, not least the fact that it allows childish sophomores to write hateful massages to faculty who challenge their worldviews, but it can also be remarkably accurate.

I'll also note that we don't rely heavily on the written evaluation forms for the student portion of tenure reviews. During reappointment and tenure reviews, the review committee conducts in-person interviews with at least 20 students chosen at random from the candidate's classes. Those give a much more accurate picture of what's going on, from all reports, and are widely agreed to be vastly superior to the standardized comment forms.

Tags

More like this

If you've been a student or faculty member at an American college or university in the past twenty years or so, you've almost certainly run across student course evaluation surveys. They're different in detail, but the key idea is always the same: toward the end of the term, students in every…
Inside Higher Ed reports on a new study of RateMyProfessors showing that the ratings correlate well with "official" evaluations: What if RateMyProfessors.com -- the site that professors love to hate -- is more accurate than they think? Or what if officially sanctioned student evaluations of faculty…
There's a paper in the Journal of Political Economy that has sparked a bunch of discussion. The article, bearing the snappy title "Does Professor Quality Matter? Evidence from Random Assignment of Students to Professors," looks at the scores of over 10,000 students at the US Air Force Academy over…
One of my least favorite end-of-term rituals for faculty is the dreaded student course evaluations. These have two components: the numerical bubble-sheet evaluations, which provide the pseudo-quantitatvie evaluation used to compare courses, and written responses to a half-dozen very general…

It's even worse for high school teachers.

Too many students and/or parents complaining about a particular teacher, is frequently grounds for that teacher being fired after the school year is finished. (ie. Administrators don't like to hire "high maintenance" teachers). Teachers in this dreadful situation are also subjected to things like physical assaults from their more violent students. (ie. Violent students doing things like: punching and/or kicking the teacher in the face, hitting the teacher over the head or in the knees with a baseball bat, smashing the teacher's head against the wall or pavement, stabbing the teacher with a knife, or even shooting the teacher with a gun).

If the final grades are too low, it's also grounds for a teacher being fired.

Generally school administrators have largely washed their hands of the situation, and will only hire teachers who are relatively "low maintenance". Administrators have all kinds of other stuff to deal with, that they don't want any additional crap from teachers. (ie. They will not deal with student misbehavior and discipline problems, unless it involves the police and/or SWAT team taking the particular students away).

With workplace conditions of this sort in the teaching profession, it's no big surprise as to why there's a high attrition rate.

Web sites like "rate my teacher" or "rate my professor", is largely another form of academic extortion against teachers. Some administrators even covertly watch these sites, to decide who they'll fire at the end of the year and NOT hire back for the next school year.

From what you've posted on this blog, it sounds like you generally have good students, but there's a depressingly amusing and somewhat cynical response to ratemyprofessors in the blog Rate your Students.

Did you know that a company is using public records laws to collect the grade distributions !!! sorted by class and professor from public institutions in some states?

There is no comparison between HS and college concerning the effect on faculty of student teaching evaluations (particularly informal ones to the parents) and grade curves. One thing I've blogged about Orientation is the need to tell freshmen that "Unlike HS, you are not required by law to attend classes in college. Also unlike HS, your college teachers are not required to pass you." It is a reality that HS teachers are expected to pass students at the lower margin of any class, so the kids from the cohort who end up going to college were never at risk of failing in HS. They were given a C. Of course, it is also true that some colleges and universities have adopted that HS approach to keep the "customers" happy.

You and I have the advantage of being male. It is also a fact (see the blog earlier this year by Melancholic Feminista) that Teaching While Female is considered a crime by many science students. We are respected for doing things that they are reproached for doing. Might be worse south of the Mason-Dixon line, so YMMV.

One thing that continues to amaze me is that I get good evaluations (sometimes spectacular ones) despite being known as a hard grader. It helps that, like you, I have goal oriented pre-engineers who are more interested in learning than other students, so they seem to accept rigorous grading as long as it helps them learn and succeed in next year's classes.

Some of the comments on RMP are amusing. When a student says that I am so bad that he did not get it after three attempts, is he saying something about me or something about himself? Followup comments from other students lean towards the second opinion, and I don't even have to know who wrote it to know that he never came to my office for help outside of class.