Adventures in Ethics and Science

I am, as it happens, done grading. But I need to express my concern (OK, bumfuzzlement) about something I saw quite a lot of on the final exams I was grading.

You may recall that I let my students prepare a single page of notes (8.5″ by 11″, front and back) that they can use to help them on their exam. Sadly, not all uses of such an officially sanctioned cheat-sheet end up being helpful. Imagine the following exam question, which the students are asked to answer in a few sentences:

Give van Fraassen’s definition of “observable”. Then, using this definition, classify each of the following as observable or unobservable: an electron, a dinosaur, and a unicorn. Explain your classification.

Ideally, the exam-takers would read the question, think about what it’s asking for (and what they recall about how van Fraassen draws the line between observable and unobservable), and then consult the cheat-sheet to see if they included van Fraassen’s definition of “observable” on it. (For those playing at home, “observable = detectable with unaided senses”.) Then, they would apply this definition to the classification of an electron, a dinosaur, and a unicorn and explain their reasoning.

What is less ideal is writing down (what I can only hope was) every single word they wrote about van Fraassen on the cheat-sheet, including a whole bunch that have no bearing whatsoever on the exam question that needs answering, and managing neither to give van Fraassen’s definition of “observable” nor to classify an electron, a dinosaur, or a unicorn as observable or unobservable.

Folks, I want to be able to give you partial credit, but you have to meet me halfway and at least read the questions. Thinking while using those cheat-sheets might also help.

If I didn’t have eleventy-dozen cookies to make, I’d have half a mind to sit down and write the next term’s final exam tonight. My hunch is that, on an exam where the questions focus on the same small set of examples from the perspectives of different philosophical views, a few more student might recognize on the fly that just transcribing their cheat-sheet notes on X is a bad strategy — because without reading the questions for anything deeper than the name of the philosopher, or philosophical view, or key example, they’d end up transcribing the same content for multiple questions. After dumping the same cheat-sheet data for the third time or so, you’d start to wonder how it could be that the exam was asking the same question three times, right?

I’m not out of bounds in giving exams that require thinking, am I? I mean, that’s what the coursework is supposed to be fostering here. Having the answer somewhere in your notes is useful, but being able to recognize what it takes to answer a question adequately it even better.

Comments

#1 Dave C-H
December 23, 2009

Wow, you’re surprised by this after all these years? You guys must have good students….
#2 Miki Z
December 23, 2009

As far as I’m concerned, asking students to think on exams is entirely fair, especially when the questions are not multiple-choice or true/false. I try to avoid questions that rely on some ‘trick’ (probably more a danger in mathematics than philosophy). If everyone misses a question, I’ve done something wrong either in the construction of the question or the teaching of the material, but a few people missing them is to be expected.
#3 DNLee
December 23, 2009

that’s sad – for the student. Thinking is so, so, so very important.
#4 MRW
December 23, 2009

I’ve had students do essentially the same thing without the benefit of a cheat sheet to work from. They see something they recognize in the question, then write down everything they can remember about the thing they recognized, without ever saying anything that addresses the actual question.
#5 Leo Martins
December 23, 2009

What makes things even worse is that at some point in the past they had to actually sit down and write the cheat-sheet, which I presumed implies organizing what goes in there. Or did they have the option of photocopying someone’s neat cheat-sheet?
#6 Pteryxx
December 23, 2009

I’ve taken a math class alongside students who staged a revolt, actually cornering the professor and demanding restitution of their scores, because the exam required them to think. The leader claimed that demanding thought instead of regurgitation was “unfair”.

I passed the course, but I am ashamed to this day that I didn’t loudly denounce them instead of shyly asking “But aren’t we supposed to be thinking?”
#7 Anon
December 23, 2009

I used a book once that had a particularly unhelpful definition for a particular term. We spent a great deal of time explaining why it was unhelpful, and coming up with better definitions. I asked the definition of the term on an exam; anyone who quoted the textbook’s definition word for word got no credit. (I told them they could get full credit if they could explain what the definition meant. None could.)

Sometimes it pays to actually attend the class.
#8 Rosie Redfield
December 23, 2009

Part of the problem might be that they haven’t had enough experience with cheat-sheet exams to learn that your questions won’t just be testing memorization. They forget that you knew in advance that they’d have the cheat sheets.

All my exams are open book, but I make sure the students have had open-book quizzes and midterms before they face the open-book final.
#9 Daddyo
December 23, 2009

If you haven’t taught them how to think by the final exam, you’re much too late. Start asking their opinion on the first quiz, and then do it in class! You may be surprised by what you find- oh yeah, and don’t listen to anyone who just wanted to tell you how stupid their students are.
#10 Matt Platte
December 23, 2009

Last time I had a real class – many years ago – it was common knowledge that “open book” meant a much more difficult exam, one that probably would require thinking and producing stuff on the spot. Being a good rememberer, that somewhat nullified my advantage. I hate having to perform in public.
#11 Janet D. Stemwedel
December 23, 2009

The same students had experience using similarly prepared cheat-sheets on their midterm exams (on which none of them fell into the full-transcript trap).

They are allowed to compose these sheets of notes any way they want, which doesn’t preclude getting a friend to make one for them. You’d think, though, that they’d recognize this as a risky call — that they’d want to at least *understand* all the stuff on their cheat-sheets even if they subcontracted the decisions about what went on them.

As for asking them to think, I did that pretty much all semester (including on the midterm exam). But I get the feeling some of the students decided that the thinking ended with the last of the new content, and that the final exam could be completed on the basis of regurgitation instead.
#12 Tony P
December 24, 2009

I always hated cheat sheet or open book exams. I’d never use either method. I either knew the answers or I didn’t.
#13 Paleoprof
December 24, 2009

I’m with MRW in that I have the same problem on non cheat sheet materials. Students see a word and they spew everything they know onto the page. This is a recurring problem on quizzes, tests and homework throughout the semester. I sometimes think that it may be a reading comprehension problem not a thinking problem.
#14 Zeno
December 24, 2009

Although I refuse to call them “cheat-sheets”, I do generally allow my students to use a handwritten notecard during exams and quizzes. I do it because I want to (a) lower the emphasis on rote memorization, (b) lower their test anxiety, and (c) fool them into studying as they go line-by-line through their notes and textbook for things to write on their notecards.

What some of my students fail to realize is that I grade more harshly in the event they get formulas or definitions wrong. (Ask the student who carelessly wrote the quadratic formula on her notecard incorrectly.) And I put more stress on thinking up solutions to problems that are more than just plugging numbers into formulas. I’ve had students ask me if they could have easier exams without the notecards (but not many; most of them love the cards — even if they don’t use them very well).
#15 stripey_cat
December 24, 2009

When I was doing my finals (English, so the final was everything) we were constantly warned against spotting a key-word in the question and regurgitating your pet arguments on that topic, especially on the popular “easy” topics. (This was Classics, specifically ancient history, and some charismatic people or themes were very vulnerable to it.) Taking time out of your precious 45 minutes an essay to actually read the questions was, apparently, not always as popular as it should have been. Nonetheless, every year it turned up in the examiners’ reports.
#16 ET
December 24, 2009

My favorite exam question/answer was in physiology:
How do camels survive in the desert?
The student answered: They survive well. (a bit is lost in translation here as the exam wasn’t in English) All the other students listed desert adaptations. She went on to get a PhD.
#17 Bijan Parsia
December 24, 2009

Not only do I think that having questions requiring thought is in bounds, I think it’s required and, indeed, most of your questions should be thought driven.

In the UK our exams are all moderated (multiple times). In computer science I’ve gotten dinged for having too many “book” or “recall” questions. And I think I’m right to be so dinged (though, to be fair to me, people tend to see conceptual questions as recall…they miss the idea that you have to recall the definition *and* apply it).

I tend to weigh things a bit more toward the recall mostly due to the impression (given by other faculty) that CS students aren’t really trained to think (in this way). But that’s not a problem for your students.

Re: strategies for next exam: That might work, but I wouldn’t work too hard trying to construct an exam that attempts to counter this effect. That’s a treadmill of doom, in my experience. I think people are perfectly capable of repeating the same rote answer multiple times (esp. if what we’re getting is a substitution effect…i.e., instead of the repeated blank answer you’re getting repeated copied answers; heck, it’s even rational: if I don’t know which question my copied answer is right, I might as well spam it).

You might add a bit to your spiel about cheat sheets pointing out that they are not an unmitigated good and here are specific known problems.
#18 Susan
December 24, 2009

Straightforward and succinct answers to questions are appreciated by teachers and attorneys, but out in the ‘real world’ straightforward and succinct are anti-social. Your students may be attempting to apply the more common conversational meaning of questions, in which the question ‘X?’ actually means ‘Tell me a story tangentially related to X.’

This leaves the choice of grading scholastically on the basis of an actual answer to the question, or grading socially on the entertainment value of the story. From your description, most of them will lose either way, though.
#19 Cath W
December 24, 2009

It’s sad, but they may have been taught this strategy in high school (or even earlier). My understanding is that the grading rubric for written portions of standardized tests in subjects other than writing only looks for keyword usage, and so teachers suggest the spaghetti-flinging approach as a good test-taking strategy.

I really hope my understanding is wrong.
#20 Pinko Punko
December 24, 2009

What promotes this I think is the idea that the shotgun approach works for partial credit. I think I would have a rule that if you say a bunch of wrong stuff, you can also get partial negative credit. Also, no partial credit for sentence fragments or lists.
#21 Rev Matt
December 24, 2009

As a History major rote memorization was really the way to go, but I would imagine that particularly in a *philosophy* class that thinking would be a core requirement for the exams.
#22 Heather B
December 24, 2009

Ah, that explains why I got good grades on such tests when I didn’t think I’d done that well. I hadn’t realized that bothering to read the questions for comprehension was unusual.

Re incorrect quadratic formula: I think all algebra students should be taught the “Pop Goes the Weasel” version of the formula, like so:

“x equals negative b
plus or minus the square root
of b squared minus 4 a c
ALL over 2 a.”
#23 Russell
December 24, 2009

I trust the unicorn counts as an observable. Whether any exist, there’s no reason to think they couldn’t be observed if they do. (There is, of course, a can of worms in the definition. “Has been observed” is a matter of history. “Can be observed” contains an implicit counterfactual. And counterfactuals are more tricky that most people realize.)
#24 Rob Knop
December 24, 2009

Yeah, I’ve seen exactly the same thing more than once. I’ve seen evidence that students had memorized long passages out of the book, without understanding them.

I had a question on a test that was something like “ultimately where does the energy come for each type of supernova, a core-collapse supernova and a thermonuclear supernova?”

Many, many students instead answered the question “write down as many details as you can remember that describes the two types of supernovae, without ever explicitly addressing the source of energy.” It’s the standard student approach: when you don’t understand the question, do a tremendous fact dump in hopes that you accidentally answer it.

Meanwhile, I gave very substantial partial credit to one woman who ran out of time, and was only able to write a few words:

* gravitational potential energy
* nuclear fusion

Five words, and she’s more or less answered the entire question… whereas many other people filled up an entire page all about supernovae without answering it.
#25 blf
December 24, 2009

As a student I usually hated exams that had more than a token amount of questions which didn’t require thinking, only memorization. I was and still am poor at remembering precise details (excepting a few areas of intense interest), but prefer to work from general principles. So I generally preferred exams where I could—expected to or not—derive/deduce an answer over those where all I had to do was regurgitate something.

Amusingly/annoyingly, this tendency has caused me problems at work, where some of my colleagues seem to prefer to be told what to do rather than what to accomplish. A trivial simplified example: “Verify the input before processing it (to avoid GIGO)” is what I’ll say, but what it seems they want to hear is “check that the input is an X, not a Y, and it’s Z is Ok; and test by checking the resulting output is always a FOO if the input’s Ok, or else an error.” Not everyone needs such detail, or understands the detail, but I keep being surprised by people who don’t want to derive the details.
#26 Perry
December 24, 2009

You seem to be assuming that the students are not thinking, but in my experience students are ususally thinking pretty well. This is what students do when they have thought but realize they don’t know the answer to the question due to lack of preparation. When you don’t know the definition, writing down everything on the cheat sheet is the best way to get maximum partial credit. That’s why they are doing this. If you hadn’t given them cheat sheets, you would have gotten lengthy B.S. answers because that’s another way to get lucky with partial credit. If you gave the students the definition in the question and then got back this kind of thing, then you could conclude that their weren’t reading the question or weren’t thinking, but that isn’t the situation as you’ve described it. What else do you expect students to do? Leave the questions they don’t know blank? That’s a suboptimal strategy and would suggest they are not thinking even more. It is unrealistic to expect that all students will have done the reading and come to class regularly. I don’t think this situation suggests lack of thought but lack of studying.
#27 lylebot
December 24, 2009

The best (worst?) is when they plod through the exam question-by-question and spend so much time writing info-dumps for early questions that they run out of time for the later ones they could have answered. While info-dumping may be a valid test-taking strategy, you should at least answer everything you do know first. The fact that they don’t suggests there’s really not a lot of thought going on.
#28 ARJ
December 25, 2009

geeee, I guess not much has changed since I was a student a third of a century ago!
#29 MPL
December 25, 2009

It could be worse. I’ve graded a class where the students memorized the answers to the practice test, despite the fact that the numbers have all been changed for the real exam.

Also, you get students who try to copy answers off a student with a different test form, but that’s more a just desserts sort of thing.
#30 Jean
December 26, 2009

Your students might also be doing this because it works well in other classes. I remember in college biology that all students eventually came to the conclusion that the best biology test-taking strategy was to be a “fountain of key words.” TAs and professors seemed to tire while grading so many tests, and skim for whatever word seemed most applicable to the question. Mention endosymbiosis but describe Jessica Simpson’s new hairstyle? Point. Describe endosymbiosis correctly without using the word? No point. Solution? Mention as many key vocab words as possible. Thinking: not required. Obviously, the strategy fails when your professor actually reads what you write and expects reasoning.
#31 Paul Murray
December 27, 2009

@23 – yes, unicorns are observable – with the exception of the Invisible Pink Unicorn BBHHH on account of her invisibility and the daintiness of her tread.
#32 George D Turner
December 28, 2009

It seems to me that you answered your own question @11, which might be posed as why did the students behave differently in the final than on the mid-terms? Of course thinking under the pressure of a mid-term is different from thinking under the pressure of a final if only because it is their last whack.
If the students’ general experience with finals is that they are of the regurgitation type, you have the option of either giving them just that, assuming you have already accomplished the higher goal, at least to an acceptable degree, or making a special point of warning them of the dangers ahead.

In my experience, it is untactful to express to students the idea that you are going to teach them to think. Many believe, and correctly so, that they already know how to think or they wouldn’t be in your class at all. Thinking in the style of philosophy of science under the pressure of a final exam- that is, of course a unicorn of a different color.
#33 R. K. Optericks
December 31, 2009

@30 and others are right on. Students learn how to take tests in large courses and should ideally un-learn that as they complete their majors. Many don’t know what to do when the prof grades the exam rather than a battalion of tired, uninterested, keyword-hunting TAs.

@26 has a different view than I do. The fact-dump is a good evidence that students are not actually thinking, they are panicking, not thinking about the question, and simply racing to get something on paper for partial credit. In the cold light of day, even the student can recognize a “squid-ink” answer.

The cheat-sheet (odious term) is a useful mechanism to reduce pre-exam stress about minor details but all that comes roaring back when they waste a lot of time on a fact-dump answer and get freaked out by the clock. The only thing I’ve found that helps is to limit the length of answers (2 sentences in my classes) but I will concede that doesn’t help much either.

In a perfect world, a tutorial with 2 or 3 students is just about the best way to teach a technical subject. Too bad grad students don’t always realize how valuable those are.