evaluation

Archaeology Programmes At Swedish Universities Evaluated

aardvarchaeology — Thu, 19 Dec 2013 13:20:40 +0000

Archaeology Programmes At Swedish Universities Evaluated

The Swedish Higher Education Authority (Universitetskanslersämbetet) has evaluated our basic university programmes in a long series of subjects. The results for archaeology were published yesterday, based on the status 2012. There were 21 BA (3 yrs), Mag.Phil. (4 yrs) and MA (5 yrs) programmes at the country's archaeology departments. The median grade they've received is "high quality", which translates to a pass here. Let's look at the eleven programmes that flunked or passed with distinction.

Gothenburg. Mag.phil. in Mediterranean archaeology. Very high quality.
Gothenburg. BA in Mediterranean archaeology. Very high quality.
Gothenburg. MA in Scandinavian Prehistory. Insufficient quality.
Gothenburg. BA in Scandinavian Prehistory. Insufficient quality.
Gothenburg. MA in heritage management. Very high quality.
Lund. MA in Mediterranean archaeology. Very high quality.
Lund. BA in Mediterranean archaeology. Very high quality.
Lund. BA in historical archaeology. Insufficient quality.
Stockholm. MA in various archaeological specialities. Insufficient quality.
Umeå. Mag.phil. in Scandinavian Prehistory. Insufficient quality.
Umeå. Mag.phil. in environmental archaeology. Insufficient quality.

Overall, the places that come out on top here are Gothenburg and Lund, though even they have problems with some of their programmes. Umeå places last, though I hasten to add that they have had at least one incredibly good-looking and keen temp teacher on the Scandy Prehistory programme this academic year, after the one evaluated. One point that makes me sad is that not a single one of the country's programmes in my subject, Scandy Prehistory, passed with distinction. One funny point is that the Mediterranean archaeologists in Gothenburg must now be really smug at the same time as their Scandy prehistorian colleagues are really angry.

Mind you, the evaluation methodology is controversial. A correspondent of mine at one of the evaluated departments writes "It's been a lot of work for an evaluation system that isn't approved by the EU, has no scientific backing and uses evaluation goals that are 30 years out of date".

The last time I looked at results of a similar evaluation in 2009, archaeology at Gotland University College received severe criticism. That entire campus has now become a branch of the University of Uppsala and so hasn't been evaluated separately.

Update 27 December: Ulla Rajala pointed out something important. Formally speaking, the University of Stockholm doesn't offer specialised archaeology programmes like the other universities do. This means that the grade that is differentiated at e.g. Gothenburg is a mashed-up average at Stockholm. When the Stockholm MA programme flunked, there may actually have been an extremely good programme in e.g. osteology hidden behind that grade. It all comes down to the random sample of student papers that the evaluation looked at. It seems to have been proportional to the number of students in each programme.

Thanks to Ing-Marie Back Danielsson for the tip-off.

aardvarchaeology Thu, 12/19/2013 - 08:20

Tags

archaeology

sweden

Academia

evaluation

quality

archaeology

Well, let's at least give some credit to Umeå for realising they have a problem and calling in the incredibly good looking cavalry.

Haha, indeed!

Actually, MA at Stockholm includes all archaeologies (Nordic, Mediterranean, Laborative, Osteology). And my understanding is that the methodology is indeed slightly warped. It sounds like one easily mixes the quality of teaching to the quality of the students and their coursework... Considering some second-hand comments from the evaluation of a series of randomly chosen dissertations.

Thanks Ulla, I'll update the blog entry.

And just suppose you had no grade

rallain — Fri, 27 Aug 2010 07:37:34 +0000

And just suppose you had no grade

It would be nice if I could come up with a good rhyme for grade to fit this title.

One of my brothers is a biochemistry faculty at Appalachian State University (hint - he is the one with the same last name that I have). We were talking (and surprisingly agreeing) that grades were dumb. What would happen if we stopped grading? Wouldn't that be awesome?

So, what would happen if there were no grades? Here are some thoughts.

We would only have one job in the class - help students learn. The second job of evaluating student understanding would only be there to help them learn more.
It would be painfully obvious to the student that learning was for them and not us (the faculty). Why come to class if you are just going to read the newspaper or surf on your fancy iPhone 4?
But don't grades tell employers and graduate schools how good a student is? How about students are responsible for presenting their own evidence for how awesome they are? Maybe a portfolio or something - something like: these are the things I am good at. Oh, but they would lie about what they can do? What is the stop them from doing that now? Isn't that what a job interview is for?
But if you don't grade, they won't do the work! Won't they? How can you have your pudding if you don't eat your meat!? Do we really want students that are learning just because we are holding the grade-stick to them? This is college, these students are adults.
I think there still needs to be some evaluation - evaluation to know where the students are so that you can help them learn. However, this evaluation should not be the Scarlet Letter grade on their chest. Just some type of feedback.
Cats and dogs, living together.

/ul>

Here is my favorite. Would we stop hearing this?

"Excuse me, Dr. Allain? Is this going to be on the test?"

To tell you the truth, I rarely hear students say that. But, I do know that it drives some faculty crazy. But I guess I am already crazy - crazy for thinking of a world without grades. Oh - there could be no degree requirements too!

rallain Fri, 08/27/2010 - 03:37

Tags

Are you familiar with Evergreen State College in Olympia? As I understand it, they mostly don't use letter grades.

"Oh - there could be no degree requirements too!"

Right, so your transcript just says that you registered for a bunch of classes.

If one university tried it, the results would depend strongly on its size and reputation.

If you instituted such a policy nationwide, testing services would spring up to allow people to demonstrate their knowledge to potential employers. Universities would be driven by market forces to structure their curricula around the dominant tests.

Here is my favorite. Would we stop hearing this?
"Excuse me, Dr. Allain? Is this going to be on the test?"

Or the more general variant, "What will the test be on?" I have a stock answer for the latter: "8 1/2 by 11 inch paper."

Alverno College in Milwaukee is a no-grade school. In the mid-90s they were held up by many as having a good working assessment model.

Alverno College is a four-year, liberal arts, independent, Catholic college for women, located in Milwaukee, Wisconsin. Chartered in 1887 by the School Sisters of St. Francis, the college offers weekday undergraduate programs and Alverno on the Weekend undergraduate programs for women, as well as graduate programs for women and men. This nationally renowned institution is heralded as an innovative leader in pedagogy, and has received international acclaim as a pioneer in developing a measurable, outcome-focused curriculum in which the individual student can grow as a learner.

They have the benefit of being a small school, with small classes, so the extra time faculty require to be involved in assessment is available. I attended an assessment conference there many years ago; I think it would be difficult to take there model and use it at a larger school, with larger classes: the per-student time faculty need to have to make it work wouldn't be a reasonable. However, the school is held in reasonably high esteem, and the teachers they graduate do quite well.

New College of Florida uses a contract system with written evaluations instead of grades. Among the advantages of this system is that written feedback from the instructor is much more informative than a letter grade. This system also better accommodates students with varying learning objectives in the same classes. I'm not aware of any students having difficulty getting into graduate programs or finding jobs because of the lack of a GPA. And yes, it was pretty awesome.

I believe Hampshire College (Amherst, Mass.) is still using this model: students complete projects and, over time, assemble portfolios of their work. Another small liberal arts college. I think (as with New College) the professors write evaluations, but it's not a numerical or letter grade.

I don't this system would work for the job market. Employers would have to review each portfolio of every applicant, wich would take way to much time, instead of just dismissing everyone with a GPA lower then 3.0 and then take a more in-depth look.

From a student's perspective (although not in the US system), grades are useful! Sure, learning for learning's sake is great, but without assessment I have no idea of how well I've grasped the material - how much I understand and how well I can express it. So as you say, some form of feedback would certainly be necessary to see where students are, but surely it would take longer for you to give individual written feedback than to assign a grade or percentage to a piece of work?

I don't think most people are very good at evaluating their own understanding of a topic - you don't know what you don't know. Without grades, employers would need to implement some kind of test (facts or skills) to distinguish between job applicants. Many already do, though, in competitive jobs. I guess in real life, grades aren't that useful in job applications, because they just tell you how well a student learnt a particular subject, not how well they can apply that knowledge to the job, but they are often used to narrow down a list, despite this.

I'm lucky enough to be at a university where, aside from lectures, a lot of the teaching is done in small groups, with a supervisor (most often a postdoc), setting weekly essays and giving feedback on them. Then we have one set of exams at the end of the year, which is painful, but overall I think that teaching method works well.

Actually, I like the grades. They motivate you to keep working and studying to make sure you maintain a high GPA, and if you already have a good depth in the subject, they encourage competitions for best designs, best ideas, and best projects.

Have you read 'The Dispossessed' by Ursula Le Guin? One part deals with a professor (... sort of) from an anarchist culture (in this case a planet) which did not use grades, only learning for the sake of learning; this professor comes to teach in a capitalist culture which does use grades. The book is definitely worth a read, and not only for this small part.

This made me think of Bill Cosby, talking about his wife, "She has a childhood psych degree with a B average, which means if you ask her a question about a child's behavior she would give you an 85 answer."

As long as we are thinking about grading's effect on education - what do you think about this lecture on unschooling by Astra Taylor?

http://www.youtube.com/watch?v=LwIyy1Fi-4Q

UC Santa Cruz had a no grade policy for many years, the system broke down for engineering and science and I think is no de facto a grade system for many or all majors, but it lasted a good long while and seemed to mostly work.

Just suppose you had no grade
No way to recognize the efforts you'd made
No one to see an "F" and go on a tirade
It's anarchy, I tell you! Be very afraid!

The school I went to as an undergraduate would allow you to take graduate-level courses on a pass/fail basis if you were really interested in the subject. I thought it was fantastic, because you could get an appreciation for why you really, really need to slog through the applied math to do anything interesting. Even if you did not have the background that was really necessary, you could do well enough to pass the course, and it was great motivation to get the basics down.

@The Other Rob,

I applaud you sir. Very well played.

I went to St. John's College (campuses in Santa Fe, NM and Annapolis, MD). There are no grades at St. John's. Students are evaluated through a process called a "Don Rag" (a term borrowed from Oxford), wherein faculty verbally evaluate the student and the student is given the opportunity to respond. Participation is expected in the form of reading (lots of reading), papers, and above all, in-class discussion. There are some who don't bother to do the work; but doing the work serves one's own education, pure and simple.

Also, dogs and cats live together.

I think there is a big problem with the one approach fits all. Some students need grades and will not do anything otherwise; many students conditioned by the system in fact to respond to grades and only grades... They game the system. Other students hate grades and don't find them necessary and in fact learn less when coerced by grades. Employers want to see grades as not just a measure of what students know, but how consistently they are able to play a coerced game.

There may be a point in making the grading system operate closer to life than the existing system... where you can catch up even if you are behind and still receive an A if you messed up something considering you can fix your mistake... etc. The point being that there needs to be an entire debate and discourse about levels of coercion, their necessity, the difference between student behaviour and needs, etc. The one approach fits all is a big problem that needs to be dealt with.

Ultimately, the schooling system is set up to create employees more than anything else... and their interests come first. Whether those employers are universities or the corporate world. The entire system exists to serve their interests more so than the student's. Should this be fixed? Probably. Will it be? Unlikely.

One more angle I wanted to add was an angle of competition. Grades spur competition and people associate themselves with certain social classes based on the brand of their school and their experience with the games that these schools make them play; i.e. grades.

So for instance if you went to Harvard and were an "A" student, or whatever is a perfect GPA; then you are the cream of the crop, you are special; you are elite. Girls give you more attention; it's easier to get dates... Employers give you more attention; etc. In fact I had a friend that used this to his full advantage... and it works wonders. Doesn't matter whether you actually know anything in fact, but these are social signifier or your position in society.

So society demands to an extent that we provide students with grades so that some feel better about themselves and their lives because of the grades that they earn. If everyone is the same, what make you special? What makes you strive to achieve, to get on top, and feel good about yourself and what you have "accomplished"?

Anyone who claims that 18-20 yr old males are adults either is not serious, is not very observant, or is not really a teacher of such.

Seriously, these kids literally can't think through consequences a week ahead. Grades have evolved to get them to work. Sure, small sell-selected populations will disprove this, but for the general college population, well, good luck.

I can see modifying the grading system...but presumably you'd still need some marker of pass or fail. Whether that's per class, or for an overall placement exam. Why? You have to set some kind of standard. You have to say these people did enough, and these people didn't. Otherwise, if your university doesn't have any standards, what's the point. Might as well be me lecturing out of my garage and handing degrees out to kids who pass by.

Getting rid of degree requirements seems pretty stupid. As it is now for engineering you have an accreditation system that determines a "core" curricula that colleges have to teach students. If I'm trying to hire an engineer, I don't want to have to look up that guy's college and figure out whether he learned fluid dynamics or not, whether he learned basic circuits or not. If you say there's no requirements and he can take whatever he wants to, how do I have any clue he's ready to be an engineer.

Again, unless I have some sort of exam. Like say I require all my applicants have passed the Fundamentals of Engineering exam. Well at that point, schools would want their engineers to be better prepared to pass that exam. So they would want course requirements that encompass the topics on that exam. So they would require a set of courses very similar to what's on the exam. And you are back where you started.

Right now as an employer you can look at somebody's resume, and even if you don't see a GPA you know that person "passed" his engineering courses. You know almost all engineering programs have the same necessary core programs that you need to prepare you for work. I don't want to have to look at a portfolio, or interview that candidate and ASK him what classes he took and whether that meets my minimum standards. I should be able to look at his resume, see the degree, and know it is sufficient.

I can see where you might want to get out of assigning individual grades per class, but I think the responsibility is still on the college/university for having a "standard" for what a curriculum in engineering, or biology, or chemistry even MEANS, and then indicating to what level that student understood and could use and convey that knowledge. If that means there's a portfolio at the end or a series of projects that tests that knowledge rather than a grade per class that's fine, but you still need some way to quickly sum up what that means to an employer.

I attended The Evergreen State College in Olympia, class of '95 BA in Math and Bio.

Classes were conducted on a "learning community" model. I took a class called "Math Systems" that covered what most schools would separate as "Calc III," "Diff EQ," etc. and I took another class called "Molecule to Organism," which other schools would separate into "Organic Chem," "Molecular Biology," "A&P," etc.

A small group of us who wanted to focus our A&P credits on other animals besides humans wrote our own syllabus, found our own text, conducted our own lectures, researched and ordered dissection specimens, wrote and graded our own tests, and evaluated each other - all with faculty oversight of course. Of all of the classes in my college career, this one was the most interesting and from which I learned the most.

In the other classes we did take tests that were graded, but our work from each quarter was evaluated by the faculty and we also wrote our own evaluations.

I have never seen a job requirement for a biotech (where I used my degree) or any other job I've had that asked what my grades were. They only wanted to know what experience and skills I could bring to the work. I have never received a grade in any job I've worked. I have received narrative evaluations, very much like the ones I received and wrote at TESC.

I have seen job postings for jobs in academia that wanted to know grades, however. I think this indicates the difference between the goals of the hiring managers: In companies they want to know what you can do, in academia they want to know what you have done.

Without issuing a single letter grade, The Evergreen State College has been graduating well-educated people who do good work in the world since 1967.

Are you familiar with the work of Alfie Kohn? Many of his articles are available at his website. He makes a solid, well-researched case that giving grades is deeply counterproductive.

I heartily agree!

I have long contended that teachers could be much more effective (at all levels) if students did not see them as "police, judge, and jury" and could just see them as "teacher". The job of a teacher should be to help students learn -- put the teacher squarely on the same side as the student and eliminate the adversarial aspect of the relationship.

This is not to say that student should not be evaluated. Just saying that that should be done outside of the classroom. Standardized tests can evaluate knowledge and mastery of a subject. (and before anyone objects about "teaching just for the test"... standardized tests can be made quite effectively impossible to teach for by having a large (and ever changing, updating) body of available questions, from which the questions on any given student's test can be randomly selected).

For evaluation of accomplishments/achievements, students can submit papers and other work to "review boards" -- normally made up of people other than those directly responsible for teaching the student.

There is no reason this general approach to teaching/evaluation should not begin at the earliest age.

By and large I have to agree with Jimmy @20. I was by no means an academic slouch, but I doubt I'd have pushed myself if I wasn't getting a very concrete indicator of how I was measuring up. Part of that is simply laziness, I'm sure; the rest is probably the lack of competition. I think most people like to have some metrics of how well they're performing, and generally I think a grade is a more objective way of doing this than the professor simply giving me an 'attaboy.' Certainly there are some true intellectuals who will strive their utmost for the sheer pleasure of learning, but the rest of us bums appreciate a little motivation.

I can't believe the thread got this far w/out a mention of William Glasser, who has been calling for gradeless schools for about 20 years now.

The other Rob is pretty cool, but "kool-aid" comes to mind when I think of the way we think about grades.

I actually attended a college that did not have grades. It was a pass/fail system: you either knew your stuff, or you didn't. In many ways it was much harder than a grade system; there could be exams where you only got one aspect of a problem incorrect, so you had to take it over again. However, when retaking tests, you had to have learned or relearned the material. We were also evaluated by outside assessors. These were professional people in various disciplines in the community who would periodically come in and observe us in some sort of situation, and then assess each individual student on their performance. Perhaps that sounds easy, but I assure you, it wasn't. These outside assessors were our future employers not just our professors. Sometimes, it was very tough. There were, sometimes, things I didn't like about it, but I earned my degree. I worked hard for it. I didn't just earn my degree, I Learned it. I am a 1999 graduate of Alverno College, Milwaukee, Wi. Check it out.

I don't agree with/like the school system. It was designed by Rockefeller to provide an entry level working class and has not grown with our expanding knowledge of how people learn nor our technological advancements.
I plan on Un-schooling my children (i.e Experiential learning). Learning should NOT be passive (sitting in a desk for 6-8 hours while mindless facts are presented that have no impact on the child's current world).
It's time to revamp the system instead of dumbing down our children.

Allegory of the Grade

rallain — Fri, 16 Jul 2010 09:56:53 +0000

Allegory of the Grade

I have been thinking about grades lately and I am pretty sure they are dumb. The main problem is that it seems that many many many people (politicians, parents, students, administrators, some other faculty, and zombies) think that the grade is the THE THING to worry about. Really, it is just a pale representation of the real thing.

This brings me to the allegory of the cave. I know you remember this when you read Plato's The Republic, right? Here is a picture that explains the whole thing:

I don't know where this image came from, it was on a boat load of other websites, none looked like the original. But, I salute you allegory-cave-drawing-person.

The basic idea of the cave is that people are in a cave (duh) looking at shadows of puppets of real things. They don't see the real things. They can't see the real things unless they leave the cave.

Back to grades

All these people are seeing the shadow that they call grades. What is the real thing? Learning is the real thing. I think that we (as a society) have fallen into the trap of thinking mostly about grades because it is easy. Measuring real learning is complicated and difficult. You can't easily evaluate the level of understanding of 1000 students with a multiple-choice test.

Some examples

Here are some cases where people might be looking at the shadow called "grades" and thinking it is a real thing.

Official hired to raise math scores. I can raise math scores. It is pretty simple (and I wouldn't even need 160k salary for two years). Just take all the students scores on the math tests and add 20 points. Oh, maybe they meant that this person was hired to raise understanding of math? No one said that.
Parent to child or teacher: WHY AREN'T YOU GETTING BETTER GRADES! Did the parent mean "why aren't you learning more"?
"Louisiana needs to raise college graduation rates". Again, isn't this simple to fix? Just graduate more students. My favorite saying "if you have to have a piece of paper that says you graduated from college, it probably wasn't worth it."

rallain Fri, 07/16/2010 - 05:56

Tags

I'd like to add the opinion that people don't even quite see the shadow of the real thing when it comes to grades...they see the shadow of a slightly different thing.

I guess it depends on whether you consider a student's performance on a test or assignment to be a measure of the student's ability/effort/comprehension OR a measure of the teacher's ability to successfully teach the material.

I've seen you write about this before, and I agree with the foundation of your argument. However, I'd have to ask, what's the alternative? Pass/no-pass? Something else?

Presumably when a university assigns a diploma they need something to determine whether a student has met the requirements for that diploma, and something on a class-by-class basis. What would you suggest?

I suspect that grades are analogous to the drunk person looking for his keys under the light because he can see there. In an environment where there is great pressure to provide quantitative measurements, people naturally tend to measure what can be most readily quantified. That means grades and/or test scores. The discussion which hasn't happened among people who can do anything about it is whether grades and test scores actually measure the things which the people who want the measurements want to measure. Rhett argues that they don't.

I don't have a good alternative to this dilemma. Pass/fail grading or variants thereof are useful when the question is whether the student has mastered the basic concepts expected of him, but they can only give a binary answer, and they are subject to the same kind of gaming that graduation rates are subject to. They won't satisfy anyone who demands a measurement which spans a range; on the contrary they make it easier to hide a gradual slip in standards.

I graduated from a British university, in which the grade of the degree depended entirely on the examinations at the end of the three years. This does have its negative aspects, but one result was that there was an emphasis on questioning, learning and understanding rather than passing a multitude of small tests.

When I was first in a Canadian university I was quite shocked when a student asked how many marks there were for a particular assignment and, on being told that it would not be graded, replied that in that case there was no point in doing it.

We teachers are trapped in a system that depends on grades as a measure of student learning and teacher performance. Schools rank students based on GPA, student learning is assumed to correlate with SAT, SATII, AP scores, and state standardized graduation exams. Some portions of these exams do provide students with an opportunity to demonstrate a higher level of understanding of concepts, but most test for bulk memorization of content. There are lots of learning activities and assessment tools teachers can use to get beyond the shadow of grades. These all take time, skill, and resources. As with so many other things in our lives, this issue boils down to money. Parents and governments spend huge amounts of money paying for our education systems and they understandably demand an accounting of their investment. Grades are the simplest metric that satisfies this demand for demonstrable results. But if a teacher has 30 students in a biology or chemistry lab, the top priorities become safety and crowd control, not learning. Not until citizens and their elected government officials move education out of the ideological and political cesspool it is in now and find a more equitable and sustainable method of financing school systems will we find a resolution to this problem.

Teaching, learning, grades, and student evaluations.

jstemwedel — Thu, 17 Jun 2010 11:33:41 +0000

Teaching, learning, grades, and student evaluations.

Chad has posted an interesting discussion of a study of students' academic performance and how it is correlated to their evaluations of the faculty teaching them. The study in question is Carrell, S., & West, J. (2010). Does Professor Quality Matter? Evidence from Random Assignment of Students to Professors Journal of Political Economy, 118 (3), 409-432 (DOI 10.1086/653808) . Go read Chad's post for a detailed discussion of the methodology of the study, since it will likely answer your questions about my quick overview here. After the overview, I'm going to offer a few more thoughts on the explanations the study authors propose for their findings.

The study, done with data from the U.S. Air Force Academy (where there is a large-ish set of courses all students are required to take, to which students are assigned at random, and which are evaluated on the basis of common exams in which faculty are not necessarily grading their own students, etc.), found that:

There was a small but significantly significant correlation between a faculty member's academic rank, teaching experience, and terminal degree status and the average grades of the students in that faculty member's courses. However, it was a negative correlation -- which is to say, the students of the less experienced instructors did slightly better in the Calculus I sections they had with those instructors than did the students taking the Calculus I from more experienced instructors.
In the courses in the required sequence after Calculus I, there was a small but significantly significant correlation between students' performance and their performance in Calculus I. Again, though, this is a negative correlation -- the students with better grades in Calculus I tended to end up with worse grades in the subsequent course in the sequence.
Student evaluations of their instructors were positively correlated with the grades they earned in the instructor's course -- higher grades correlated with more favorable evaluation of instructor performance and vice versa.

Chad writes:

So, what's the cause of this? Nice as their dataset is, they don't have any way to really work that out, but they offer three possible explanations: First, that less experienced instructors are more likely to adhere strictly to the common curriculum, and are thus effectively "teaching to the test," while their more experienced colleagues teach a broader range of stuff that leads to better understanding and thus better performance in future classes. The second possibility is that students whose introductory course instructors are "teaching to the test" pick up bad study habits, which come back and bite them in later courses. The final explanation, which even the authors characterize as "cynical," is that students who get low-value-added professors in the introductory course put out more effort in the subsequent courses, in order to pick up their GPA.

They are admirably cautious in drawing conclusions based on all this. About the strongest statement they make in the paper is the concluding paragraph:

Regardless of how these effects may operate, our results show that student evaluations reward professors who increase achievement in the contemporaneous course being taught, not those who increase deep learning. Using our various measures of teacher quality to rankâorder teachers leads to profoundly different results. Since many U.S. colleges and universities use student evaluations as a measurement of teaching quality for academic promotion and tenure decisions, this finding draws into question the value and accuracy of this practice.

The explanations the researchers offer don't seem too unreasonable. Indeed, given how much attention is typically paid to preparing Ph.D. candidates to be classroom teachers, what seems to be happening in this study seems almost predictable.

Most of the teaching experiences people get during their graduate training is as teaching assistants -- maybe they lead recitation sections that include lecture-like presentations, or maybe they mostly field questions. Due to the "weed 'em out" structuring of math and science curricula at many of the institutions where graduate students are trained, there are usually many, many more introductory sections that need T.A.s than intermediate and advanced sections (since you've failed or scared away enough of the intro students that you don't need the same number of sections of courses further on in the sequence). And given the emphasis placed on research over teaching in most graduate programs, graduate students typically don't pick up much teaching (even T.A.ing) beyond the bare minimum that is required. In my chemistry graduate program, the requirement was that you T.A. for the equivalent of one academic year (and most people finished that their first year).

This is to say, when we're talking about newish faculty having relatively little teaching experience, it may be a lot less than you expect (especially if you're a parent writing the check for tuition).

Given their experience being students, though, graduate T.A.s are often remarkably focused on helping their own students not to freak out. They work hard to find ways to make the material make sense, if for no other reason than how draining it can be to deal with panicky students in recitation sections week after week. If the job is to get students through the intro material without imploding, they can do that.

As the study suggests, though, getting through the first course in the sequence can maybe be accomplished in a way that is suboptimal when it comes to laying the groundwork for the next course, or the course after that.

Some of this may be due to the focus on the immediate task (getting students through course in progress at the time) rather than pulling back to tell one's panicky students, "Here's how you will use and extend what we're doing here in the next course." In chemistry, there was also the issue that subsequent courses in the sequence moved beyond the simpler models that worked well enough in the earlier courses (so an intro student's grasp of "orbitals" could be pretty confined to atomic orbitals, while in later courses understanding "orbitals" meant understanding hybrid or molecular orbitals). Some students who did really well in their introductory courses were not very good at letting go of the models that got them through.

Especially because abandoning those simple models in favor of the more complicated ones could feel like saying that the T.A. with all the empathy, who made that weeder class both do-able and enjoyable, must have lied to you by pointing you toward those simple models.

To my mind, this is a more complicated situation than students picking up inadequate study skills or teachers just teaching to the tests. Students are often surprised that learning a subject requires learning a sequence of increasingly more sophisticated models, or increasingly more sophisticated analytical techniques or methods of approximation, or what have you. Learning the next chunk of knowledge in the line is not just a matter of adding more on, but also of recognizing the problems with the chunk of knowledge you learned before. This is a surprise to many students, and the fact that it can throw students out of equilibrium is not necessarily something you notice as a newish instructor trying to establish your own equilibrium.

It's certainly not the kind of thing you notice until you're teaching one of those courses that is later in the sequence, where you are helping students cope with the realization that what they thought they knew is a lie (of the useful sort that simple models or methods often are).

Do students who do worse in intro courses already have a more pragmatic view of the simpler models or methods transmitted in those courses? I have no idea. But I have seen students who did well in intro courses flounder in the courses that follow upon them because of an overinflated confidence in the soundness of the knowledge they had mastered at the introductory level. Testing whether something like this is really going on would require more information than is contained in the data about grades, though.

One conclusion of this study, that student evaluations of faculty performance don't indicate that the students have learned all that we want them to, is no surprise at all. This is part of why institutions that care about teaching hardly ever rely on student evaluations of teaching as the only source of data to evaluate faculty teaching. (At my university, for example, there is regular peer reviewing of teaching, and these peer reviews are important in retention, tenure, and promotion decisions.)

But we shouldn't conclude from this that student evaluations of teaching are worthless if for no other reason that students' experiences of their learning in the classroom matter. Keeping them entertained but teaching them nothing would obviously be a problem (and one we ought to be able to track though evaluation tools like grades), but so too would teaching students a great deal while leaving them traumatized in the process.

That would be no way to encourage students to seek out more knowledge.

jstemwedel Thu, 06/17/2010 - 07:33

Tags

Academia

Teaching and Learning

evaluation

teaching

Academia

So far as student evaluations go, some students will rate the instructor the best ever, some will demand that the instructor be fired for incompetence, with most students somewhere in between. I have found student evaluations useful to tell me what students liked and did not like about my teaching.

Even in an introductory course, which I teach from a somewhat historical perspective, models change with time. I make a particular point, when discussing genes, that our understanding, and therefore correct answers on tests, will change as our understanding becomes more modern.

"In the courses in the required sequence after Calculus I, there was a small but significantly significant correlation between students' performance and their performance in Calculus I. Again, though, this is a negative correlation -- the students with better grades in Calculus I tended to end up with worse grades in the subsequent course in the sequence."

Isn't that called regression toward the mean?

What is a grade?

rallain — Thu, 20 May 2010 08:06:02 +0000

What is a grade?

Grades are all over the place, but what are they? Well, I guess there are a few questions. What is a grade? What is the grade supposed to be? Why do we give grades?

I think the grade is supposed to be a measure of a students' understanding of the material. Probably everyone would agree with that description. But, it is still a bit tricky. Who (or what) determines what a student should understand? Who determines what an "A" means? Fortunately, there is not a governing body (yet at least in physics) that says what an "A" grade means. It is left up to the expert evaluations of faculty. I am not saying this is ideal, but what happens when you try to get everyone to formally agree? Standardized tests. These are inherently evil. It is just extremely difficult to make a test that accurately measures the understanding of many students.

Why do we (faculty) give grades? I know some people would say "because we just do - that is the way the university works". These are the people that just do what has been done before. Others will say "if you don't give grades, the students won't work". Again, this leads to a problem. If students are just going to school to get grades, why don't we just give them the grades? If grades are the point - let them have grades.

We are supposed to give grades so that we can share our evaluation of the student with other institutions or jobs or parents or whatever. Of course, there is no universality of grades (and I am not pushing for that because it would be disaster). A grade in intro physics at MIT might mean as well as be interpreted differently than the same grade in the same course (with the same course description) at a 2 year community college. Should they be interpreted differently? They should mean the same thing.

Ok, one last thing. What about giving a grade because a student deserves that grade. You know this student worked hard, should the student get at least a B? With my interpretation of a grade, unfortunately the answer would be "no". I want to be nice, but grades are not about being nice. Grades are about evaluating what a student knows. At the end of the semester, if a student thinks they deserve a higher grade, I am open to that possibility. However, that student would have to clearly demonstrate they understand the material. If the final exam didn't accurately evaluate the student, maybe the student working a problem on the board can do the trick. Really, if I had the time, I would prefer to just meet with students at the end of the semester and talk about physics. After half an hour, I am sure we (me and the student) could agree on a grade that would represent the student's level of understanding.

rallain Thu, 05/20/2010 - 04:06

Tags

Love the general idea of meeting and coming to agreement on a grade. As a painfully shy one-time physics student I have a couple of suggestions. Meet early in the term, too, and make it mandatory so the shy ones can't wiggle out. Talk at the board, focus on the physics and don't probe *at* the student. Share excitement and fun -- I guarantee the shy ones feel it, too, but they might not show it right away. If they're used to the idea by the time of the "talk that matters", it will make a huge difference. I missed out on a lot of my education by being afraid to bother professors, so if you are the one they're not afraid to talk to, you might have a bigger positive impact than you'd think.

Gonna be hard for you, though -- some people really do invest a lot of their self-image into their grades, and it will be tough to damage that face-to-face.

Having gotten a lot of poor grades and ended up with a bad average, I found myself left wondering.

Was that guy who got magna cum laude and all sorts of honours while pursuing his business degree really âbetterâ than me, who had worked hard at a maths degree but only just barely passed?

It's further complicated by the fact that I want to do grad school in linguistics. I want the schools to see that I'm worth the time to let into a Ph.D. program, but I need those âgradeâ things to do it.

I think, when I am a professor issuing grades, I'll do them based on three things with a âpick 2â approach.
1 - How good are you at the subject.
2 - How much did you improve at the subject.
3 - How hard did you try to get 1 & 2.

There is no âA for Effortâ, but there is an âA for Being Good and Making Effortâ, an âA for Improving and Making Effortâ, an âA for Being Good and Improving (even w/o effort)â

But I don't know for sure. Pressures to grade more 'normally' might conspire to mess that up.

I enjoy your blog. Keep up the good work. Seems like you are inclined toward "standards based grading". I have been reading another blog recently (http://101studiostreet.com/wordpress/)that addresses standards based grading in high school for math and physics students. I think it gets directly to the heart of the point you make about interviewing a student at the end of term to assess their understanding. I'd be curious to know your thoughts if decide to take a look.

thanks

@Luke,

yeah - I saw that post on SBG. Really, that seems better than what we have - but ideally, I would be happy with no grades.

I guess I'd be okay if people accepted that grades are rather subjective, but it bothers me that because we assign them numbers, people think about them in terms of absolute accurate measurement. I know someone who was .05 pts shy of the required GPA for grad school. He was accepted on a probationary basis, couldn't receive an assistantship, and the dean of the college *would not budge* despite the fact that several professors and the chair lobbied on behalf of this person. I know most people are more reasonable than that, but there are enough people who aren't, which really gets to me.

@Cherish,

Honestly, I think some people give tests (like multiple-choice) so that they won't have to be subjective. I, for one, welcome our subjective grade giving overlords.

Never mind what an A means---read the syllabus and you'll have some idea. What does a B or a C mean? I can recognize a student who understands, say, derivatives. But how do I quantify partial understanding?

Building a Better Student Evaluation

drorzel — Wed, 14 Jan 2009 09:43:27 +0000

Building a Better Student Evaluation

If you've been a student or faculty member at an American college or university in the past twenty years or so, you've almost certainly run across student course evaluation surveys. They're different in detail, but the key idea is always the same: toward the end of the term, students in every course are asked to fill out a questionnaire, usually a bubble sheet, assigning numerical values to various aspects of the course and the professor's teaching. Most schools also provide some option for free-form written comments as well.

These course surveys, particularly the numerical scores, figure highly in evaluations of faculty for things like merit pay, tenure, and promotion. And yet, almost everybody in academia agrees that they're highly flawed, easily gamed, and totally inadequate to the real task of evaluating faculty performance.

A passing mention of course evaluations in yesterday's links dump (see also Matt's front-line reporting) prompted some suggestions of alternative ways of doing course evaluations. So I thought I'd throw this out there more explicitly:

Suggest some practical ways of improving the standard student course evaluation process.

The constraints are that you have to provide some legitimate avenue for student feedback about the quality of the class, that the scheme has to be legal and ethical, and it has to be something that is not orders of magnitude more difficult or expensive than the current bubble-sheet systems (we do 20+ interviews of randomly selected students for tenure and promotion reviews, which is undoubtedly more accurate, but not remotely feasible for regular evaluations).

Suggestions can be minor tweaks (I personally favor the figure skating system, where you throw out the highest and lowest scores before calculating the average; other people do mid-term evaluations, and use them to adjust the course on the fly), or major overhauls (scrap the whole thing, and base promotion reviews on RateMyProfessors chili-pepper ratings). I'd love to hear some new ideas about how to make the process work better.

drorzel Wed, 01/14/2009 - 04:43

Tags

My department head says that the whole evaluation thing could be replaced with one question:

"Do you like this instructor?"

Other than that, it really depends on what the evaluations are for. If they are used for tenure decisions and stuff like that, not sure what can be done. I suggest just observing those faculty's classes.

The other use for evaluations is so that a faculty member can use the information to improve his/her courses. The form of most evaluations is not very good for this use. There are better mid-course methods for doing this kind of thing (more course specific questions with immediate feedback).

If you're using evaluations to improve your teaching, end of course evaluations are clearly too late, and I'd argue mid term are too late too. You can do a casual feedback exercise in class (it only takes a couple of minutes) and get feedback every two or three weeks. If you respond to this feedback, you'll notice that your official feedback will improve a lot - not just because you improved, but because the students had other opportunities to vent their frustrations with the course.

A lot of the variance in ratings is known to be due to the variance in students' abilities and grades. If the class is hard for you, you'll rate the professor lower. It's probably practically impossible, but correlating ratings with students' aptitude, as measured before the class started, would be very useful. Heck, even just correlating the ratings with the answer to the question "what was your grade in the last class you took in this department?" would be very interesting.

I think that there are two evaluations of teaching â not one. Student evaluation of the instructor and the course materials is, of course important for what it is. But more important is the effectiveness of the evaluated instructor in getting the material over to the student. This more important evaluation cannot be self-evaluated by the recipient, the student. One solution is to assign to the evaluated instructor (for purposes of reappointment, promotion, tenure, constructive year-end criticism and such) the following statistic. Using the institutionâs grading computer system, follow each student of the evaluated instructor, particularly in core and major courses, in future courses which have the instructorâs course(s) as a prerequisite. In subsequent course A, calculate the numerical average of the term grades assigned by course Aâs instructor. For each student being followed, calculate the delta statistic equal to the followed studentâs term grade minus the average grade in course A. Average all the followed studentsâ indices in course A and assign that average to the evaluated instructor. The range of such important average deltas (positive or negative) will give the appropriate dean or committee, important quantities indicating the teaching effectiveness of the evaluated instructor.

If memory serves, the forms that Union uses for the evaluations are quite terrible. It starts off with the standard survey-type questions using a scale of 1-5 (noted as strongly disagree and strongly agree) and then asks questions that are poorly phrased; e.g. "This professor is very effective." and "How much has your knowledge of the subject improved." Logically, these blow the usefulness of the scale and make it hard to get an accurate read of the student (e.g. "I strongly disagree that the professor is *very* effective, but I know that a score of 1 would be read as me thinking that they are completely *in*effective.")

I frequently skipped the survey questions and went straight for the short-answer on the back, but I'm not so sure that the answers I wrote would carry the same weight as the overall scores of the survey section.

A simple 3 question sheet would be more effective, I think:
1. Do you feel this professor teaches effectively.
2. What aspects of this professor's teaching would you encourage or discourage?
3. Any other thoughts?

This is an interesting question, we are going through a university-wide review of the system, so there is a lot to say, I might write something about that later. Let me just say that this is a part of a larger issue, which is how to evaluate teaching. In this context one has to wonder what the student evaluations are for. Clearly the students are not in a position to answer some of the questions they are asked, which is one reason for the randomness. This could make sense only as a (small) part of evaluation of teaching.

If we distance ourselves from the evaluation part, and just concentrate on the feedback part, we can ask students which parts of the course worked for them, and which part needs changing, and how. We now have an online evaluation forms, and the course instructor is invited to provide course-specific questions. In addition we are testing a system of feedback forms to be handed to the students during the semester, so their feedback has some effect on the current course. I think any universal set of question will almost always miss the point.

I'm re-submitting something that seems to have timed out on first try.

I programmed a (on hardcopy) 3-D graphics output of student evaluation in FORTRAN at University of Massachusetts in 1973, which they continued to use for a very long time as legacy software.

All the bells and whistles are pointless if the department Chairman and Deans don't use them. I had the highest evaluation semester after semester in one private university class that I taught. When my contract was non-renewed after 5 semesters, I had a discussion with the Chair, with my advocate present (the then Vice-Chair of the Faculty Senate, now Chair). It seems that the Chairman had literally not known (as he admitted) that I'd been teaching this specific course, although he was vaguely aware of other courses that I'd been teaching.

He said it was out of his hands. It was up to the Dean of the College of Arts & Sciences. Said dean later appointed her spouse to teach the course.

"Those who cast the votes decide nothing. Those who count the votes decide everything." -- Communist Tyrant and mass murderer Josef Stalin (attributed).

Posted by: Jonathan Vos Post | January 14, 2009 12:37 PM

And yet, almost everybody in academia agrees that they're highly flawed, easily gamed, and totally inadequate to the real task of evaluating faculty performance.

This conclusion is based on a huge number of scientific studies.

Let's think about this for a minute. If we know that something is uselessâeven counter-productiveâthen what's the rationale for continuing it?

Any department that relies on student evaluations for tenure and promotion decisions is doing a huge disservice to its faculty. As it turns out, there probably aren't many such departments in research universities.

I think student evaluations should be banned in all classes taught in the first two years of university. By the time they reach the third and fourth years they are in a position to make reasonable responses to the standard questions.

The only acceptable alternative is the one suggested in #1. That's to replace the existing questions with the only real question that's being answered; "Do you like your Professor?"

Harlan, you bring up a very good point. In fact, something similar to this was already being done at my school. There was a question that asked "What grade do you expect to get in this course?"

Here is a manual trackback. Executive summary: scrap the whole evaluation thing, and build a system aimed at providing student feedback instead.

An experience I had that ties in to this and one of your other threads: when I took undergrad QM, the class (and, hence, the professor) was almost universally disliked by the students. The reason, as they would explain it: "The class is too mathematical; we never learn any physics." Those few of us who had a solid background in linear algebra and differential equations didn't see it that way, and liked the class just fine. But as for most of the class, they were woefully underprepared, spent the whole time scrambling to make sense of the mathematics, and concluded that the professor was awful at getting the physics across.

In short, I'm afraid that to some extent getting accurate student evaluations would also require overhauling the entire curriculum.

What are they for: Student evaluations came out of the '60s turmoil. Students felt they and their concerns were ignored by administration (quite true then). So they were given visible feedback on their instructors. The results are often available to students, to use in selecting/avoiding instructors.

Do they measure effective teaching? No.

Do they provide students with a sense of what other students experienced in this or that professor's classes? Yes.

The problem we have is that this sop to students' concerns has been perverted into a tool to measure teaching. The question, "How do we improve it?" is fundamentally misguided. If we want to measure teaching effectiveness (if it's even possible to do that), we need to start from scratch: first define what we mean by that and then work on ways to measure it. But we can't start with the student evaluation: "If I wanted to get there, I wouldn't start from here."

"As it turns out, there probably aren't many such departments in research universities." Even if these universities had a perfect way to evaluate teaching effectiveness, they wouldn't use it much in tenure/promotion decisions. Because teaching doesn't matter to them. So that's not evidence that teaching evaluations are useless (although there may well be some "scientific studies" supporting that; feel free to cite?).

To my mind, the most important thing that can be easily changed about course evaluations is that they are collected at multiple points. It's much better for helping students feel listened if you have that early data point.
(as far as format- I think Danny's proposal is pretty good; but I always liked the short-answer type questions much better than inane bubbles)

@Bill Watson- so professors who recommend that their students take fewer classes at a time in future semesters will be rewarded?
Basically there are many techniques, ranging from legitimate study skills to true "gaming the system" instruction that could affect how well students did in subsequent courses without having anything to do with subject-material teaching efficacy.
I get what you're saying, but any single measure of student achievement is just as fraught with problems as any single measure of teaching achievement. To say nothing of the many complicating variables which alter how the former is related to the later. Not that it's bad to try...

I'm one of the students on a committee that provides student input for retention/tenure decisions for a deparment at a research university (posted anonymously for obvious reasons). The way it basically works here is that the student committee (there is one for graduates and undergraduates) provides input to the department during the decision process, except if the students decide not to recommend there is also a seperate university committee that looks at why the students voted no (and they actually take this pretty seriously here), so compared to alot of places my university is fairly liberal (although its not exactly a liberal university by any standard). This actually works pretty well; the people on the commitee are mostly upper level undergrads, so by then we have a good idea that everyone in gened classes classes tend to complain. On the other hand, we look mostly for trends -- significant overwhelming positive or negative reviews with backing evidence or numerical scores that are several deviations from the norm. Normally we end up saying that someone is a good teacher and make a few recommendations on points they might consider improving. Very rarely there is a serious issue that we write up. This seems to work pretty well, because it does a good job filtering all the noise from the student comments. Furthermore if we have questions we can pretty easily go talk to people (who we know, because we are students) who have been in the classes to ask followup questions that might not be so easy for faculty to ask students. We get training every year on how the tenure process works, so we are keenly aware of the consequences for what we are doing; everyone takes it incredibley seriously. So I'm not convinced that student evaluations have to be worthless (although I agree with the people who think these numerical questions are horrible).

Twenty years ago?

I was peripherally involved in discussions thirty years ago about revising a system that had been in place for at least a decade. I think it started during the campus strife of the late 60s.

Yes, it measures and mis-measures many things. There is a fine line between the students learning more because they like the instructor and engage in the class, and students liking the professor because they learn very little.

One thing that has proven eye opening is that there are quite a few students at my CC who actively complain about the second type. They are here for an education, often spending their own money, not to get free tickets to a football game. I was stunned when I overheard a discussion where someone was telling another student: Don't take Prof SoAndSo. You won't learn enough to do well in Next Class.

I never finished university, but as a student I hated evaluations. Invariably half or more of the questions would be maddeningly vague, non-questions, or not make any sense at all. I'd fill up half the short answer sections complaining about just a few of the worst questions. Then I'd try to think up something that would be both useful to the prof and yet close enough to their biases they'd take it seriously. I had a hard time doing that without prior preparation, so sometimes my comments on the prof were not terribly useful.

I was also greatly bothered by the fact that as far as I knew, no-one was testing the profs in any objective fashion (such as Bill Watson in #4 suggests). Nor, as far as I could determine, was there any effort to measure whether the evaluations themselves were of any use. (I didn't find studies on the usefulness of student evaluations until after I left college. When I did, it seemed they all agreed the bubble question in student evaluations were of little or no value, but sometimes profs felt they learned something from the short answer section. )

Adandon the bubble sheets!!! I don't think they provide useful information.

For improving the class, I think free form or short answer questions would be most useful. Ask about specific things that they found helpful in learning. You can in theory act on them.

I'd suggest peer observation for overall quality feedback. Perhaps across departments to avoid what are really curriculum debates. I think an experienced professor can tell when students are engaged and learning while observing a class. If everyone looks shell shocked it's a bad thing.