Cognitive Daily

The New York Times recently published sample top-scoring essays from the new written component of the SAT test in order to show the type of work that was likely to score highly. Several bloggers, as well as the Times itself, have noted that the writing isn’t exactly compelling.

In fact, I’ve been carrying on a bit of a debate with Chad Orzel, of ScienceBlogs’ Uncertain Principles on this very subject. Chad argues that it’s unfair to put a microscope to the the highschoolers’ prose, written in just 25 minutes based on a prompt they had never encountered before.

In the comments, I expressed surprise:

I’m sorry, that was just painful to read. These are really the best? I do understand that these writers got only 25 minutes, on a topic they hadn’t prepared for, but still, the best?

Then Chad ~~laid down the gantlet~~ threw down the gauntlet:

Somebody ought to get a bunch of bloggers together, and give them the writing SAT under timed conditions, and see what they come up with.

I think you can figure out where this is headed. Chad and I have set up a test for you to use to find out if you can do any better than a bunch of highschoolers.

We’re especially interested in finding out if bloggers, because of their regular practice in short-form writing, might be able to perform well on the test. On the blogger’s side, they’re used to cranking out pointless rants on a moment’s notice. But highschoolers are well-practiced at responding to their teachers’ inane writing prompts. Bloggers get to choose their topics, so blogging may not transfer well to the SAT’s writing prompt. Who can perform better on the SAT test? There’s only one way to find out.

So, without further ado, we present the Blogger SAT Challenge. We’ve prepared a (relatively) controlled environment where our ~~victims~~ volunteers can respond to a sample SAT question.

Then the test will be scored in up to two ways: (1) ideally, if we get enough volunteers, we’ll grade the test according to SAT guidelines, (2) we’ll create a special blog where our readers will rate the essays on the 6-point SAT scale. We’ll then analyze the results to see how bloggers compare to college-bound highschoolers.

You can take the test at your leisure, but there are a few rules:

Don’t reveal the question until we’ve closed the test (at 11:59 p.m. EDT on Wednesday, September 20). You don’t want to give your competitors an unfair advantage, do you?
You will have just 20 minutes from the time you click on the link below to complete your response (actually we’re giving you 21 minutes so that you have a chance to read the instructions and fill out the identifying information). There won’t be a clock visible on the test page, but you are being timed, and essays that go over the limit will not be scored. Yes, this is less than the 25 minutes than the highschoolers had, but they also had to handwrite their essays, so we think is the fairest compromise.
The text box that our software provides is rather cramped, so you may want to compose your response offline using a text editor, then copy and paste your response. Just make sure you leave the survey window open while you work.
You will have a chance to eliminate your essay from consideration after you complete your response. But please, for science’s sake, use this option sparingly. We promise not to identify the authors of the essays.
However, if you’re pleased with your work, we will provide a way for you to link to it from your own blog after the study is complete. You’ll even get a nifty badge that you can post on your blog to proclaim your madd writing skilz. But don’t reveal your answer until we let you know it’s okay, after scoring is complete — we don’t want to bias any of the judges.

Click here to start. Timing begins immediately.

Oh, and one more thing: we *do* need volunteers, preferably college English teachers, to help score the exams. So, if you’re interested in that, please let us know, either via email or in the comments.

Comments

#1 Stephen
September 15, 2006

I’ve now read the essays published by the NYT. Given the nature of the question, it isn’t much of a surprise that these were the kind of answers submitted. I especially liked the essay by the violinist who wrote on a basically unrelated topic. Classic. Maybe i liked it because i’m studying violin myself, though i doubt it. But the essays were painful? The question was clearly painful. It’s obligatory for any answers.

The surveymonkey question is certainly in that same class. Great work! I’d expect bloggers to do about the same sort of stuff as the SAT takers. More words from those that type fast. Correct spelling from those that bother. Yes, i naturally spell check comments that i submit. Anything that should have been caught by a spell checker can be considered stylistic. No, i omitted the usual proof reading passes that go into a blog entry. Twenty minutes isn’t forever.

The NYT article goes on and on about the perfect scores. These are irrelevant. Less than 1% of test takers achieved perfection. That’s the noise. I’d be more interested in how many did poorly, and why. I’d especially like to see a statistical treatment of how students performed on the essays compared with the rest of the test. Did students who aced the essays always do well with verbal? Did good math students also perform well in other areas?

Anyone can be critical. Now, the SAT’s require critical essay judges.

One lesson from education is that if you want to be good at writing, do alot of reading.
#2 Natalie
September 15, 2006

Can we still play even if we’re not bloggers ourselves? I barely have time to read blogs, let alone write one!
#3 Dave Munger
September 15, 2006

Natalie,

Sure, no blog necessary to play. You can leave those fields blank on the questionnaire.
#4 Katherine Moore
September 15, 2006

I just read the 3 NYT examples. I think the violin one wasn’t good because it was off-topic without getting back to topic much at all. The other two were fine. Of course there were some awkward sentences among other problems, but as it has been pointed out, the kids had just 25 minutes to HAND-write an essay. Most kids these days are used to using computers for writing, and have essentially unlimited time to revise papers.

I wonder about how handwriting affects some of these kids. Back when I took the SAT II writing (which sounds like the same test as the writing section of the new SAT), I received about a 4 out of 6. The examples of “4″ essays were pretty sad, so I wasn’t happy with my score in that area, though I had a good score overall due to the multiple choice grammar section. I always wondered whether the essay score had something to do with my horrible handwriting. Once I took the GRE (2 essays, typed), I was able to pull off a perfect score. I’m not sure, however, whether the difference was due to the computer or to test preparation. The GRE book I had taught me a good “formula” for completing successful essays.
#5 Davis
September 15, 2006

Some additional things to keep in mind regarding the SAT essay portion: using factually incorrect information to support your argument is perfectly okay, provided it actually supports your argument; and the students are space-limited, in addition to having to write the essay by hand — they have only the two pages provided with the exam.

Personally, I think the essay portion of the SAT encourages students to use a writing method that is 100% wrong. No outlining, no drafts, no revision, no fact-checking. Yuck. (Actually, having prepped quite a few high school students for the SAT, I think a similar statement holds for the math portion of the exam.)
#6 Hypercycloid
September 15, 2006

Thanks for tackling this – as an SAT tutor for ten years, I took the SAT twice last year when the test requirements changed, and I was surprised by how much difficulty I, a literate and competent writer, had with the essay section. I think part of my trouble is that I am physically not accustomed to handwriting for 25 minutes straight, and it was tiring and distracting just trying to write legibly! The five minutes taken off the time is not enough to compensate for your readers being able to submit typed passages. My typing speed is at least 160% of my handwriting speed.

Also, students who are coached in taking the exam are told to write predictable, formulaic essays because these conform best to the grading rubric that essay scorers are using.

I really wanted to figure out a way to test the system by writing an essay which is well-written and completely adheres to the SAT’s grading standards, but is offensive in supporting an unpopular political or moral stance. It is hard to believe that the human beings who are scoring these essays are really capable of being as objective as the SAT empire supposes.
#7 Caledonian
September 15, 2006

It might help if you didn’t spell ‘laid’ as ‘layed’.
#8 Caledonian
September 15, 2006

Also, ‘gauntlet’ has a ‘u’ in it.

Is this a subtle test, or an insidious way of driving us mad?
#9 Carolyn
September 15, 2006

Wow – that was so much harder than I imagined! I don’t typically have a hard time formulating my ideas, but I came nowhere close to finishing this essay. I am quite sure whoever reads it will have NO idea what I was trying to say. I had a hard time understanding it, and that’s with the added benefit of knowing what I was thinking! I would recommend an extra ten minutes at least for SAT takers. After all, I typically spend 45 minutes to an hour putting together a blog post – sometimes longer, depending on the depth of what I’m writing.

Great idea, though – I’m interested to see the results!
#10 Classicalclarinet
September 15, 2006

As a HS student myself, I’m very interested in what the results of this challenge would be. The NYT-given essays didn’t surprise me too much- in fact, the quality of the essays matched almost exactly to the sample essays I’ve seen in test-prep books, etc. The SAT empire marches on, I guess.
I also agree w/ Davis above that the fact- and revision- free nature of the essay is exasperating. The NYT article quotes inanity like this :”Memory is often the deciding factor between humans and animals,”. And that essay supposedly got a perfect score.
#11 Classicalclarinet
September 15, 2006

Adding, does anyone know where the rest of the 20 essays the article refers to could be found? I have a nagging suspicion that the reporter might have cherry-picked some of the worse ones.
#12 John Novak
September 15, 2006

Huh.

Interesting, and somewhat harder than I expected, because even with my mad typing skillz, I still think I came in right under the damned wire on submission… and it only took me about a minute to compose the outline of the thing in my head.

Also, potentially unfair because I can type a whole goddam lot faster than I can write, and I’m pretty sure there is no way I could have hand-written that out in 25 minutes. (To my credit, I wrote it semi-linearly– meaning, I wrote a bunch of paragraphs in parallel, but didn’t keep optimizing each paragraph. Anything I kept for more than five seconds stayed there, and I didn’t add stuff in the middle of a paragraph. In short, I tried to emulate paper.)
#13 Davis
September 16, 2006

…the fact- and revision- free nature of the essay is exasperating.

I really want to see someone creative devise a mostly-counterfactual alternate history of the world (beforehand), and then use the manufactured history extensively to support whatever position is being taken in the essay. Theoretically this should not affect one’s ability to achieve a 6 on the essay. However, I think it would be completely remiss of me to suggest this to my SAT students.
#14 Davis
September 16, 2006

I would recommend an extra ten minutes at least for SAT takers.

The main problem with this is that the SAT is already really long — the length increased to 4 hours with the introduction of the (multiple choice) writing section and the essay. Perhaps they should eliminate the writing section entirely, and just have an hour-long essay.

(Personally, I’d like to see the SAT go away completely.)
#15 Dave Munger
September 16, 2006

Caledonian –

You got me on “layed” / “laid”. At least I didn’t substitute “pus” for “plus,” like Chad did in his post. I fixed it now.

I do believe “gantlet” is an acceptable alternate spelling of “gauntlet,” though.
#16 Monimonika
September 17, 2006

Ok, I wrote my “essay”, which turned out to be only four (very) short paragraphs long. I spent the first three explaining the psychology and then the very last one on personal experience, which may or may not have been coherent.

I actually went back during the last three minutes to correct pronoun usage (and other grammatical stuff) and to read over the entire thing. Kept time with my watch, which indicated I turned the essay in with a little less than a minute to spare.

This writing experiment was more stressful than when I took the SATs way back when in the late 1990′s. I’m so glad I graduated while things were still reasonable. I actually took the test several times, and one of the hardest essay questions I faced was along the lines of,

“Write about someone you know (whether personally, from an article, from TV, or yourself) who is following in the footsteps of a great person.”

Wha???

I completely blanked out that time, although I usually can come up with at least something on most occasions. I eventually wrote up some lame, obviously-phony, dreck about a (fake) friend who wanted to become a teacher. All I could think of was, ‘Why the heck am I supposed to know someone who wants to follow in the path of some supposedly great person!? Not everyone necessarily have role models, you know!’

Another stupid one was, “What is silence?” Ugh.
#17 Kate Nepveu
September 17, 2006

Wow. I am not sure I made the time limit. That was a lot harder than I expected.

(In a way, I think that typing slowed me down, because I could do a lot more revision, rather than sticking with whatever awful sentence had just appeared on the page.)
#18 Spencer Hensley
September 18, 2006

“The first draft of anything is shit.”

–Ernest Hemingway
#19 Socratic Gadfly
September 18, 2006

Oops, Dave made a writing mistake himself, early on:

“Then Chad laid down the gantlet:”

WRONG.

(Although the reverse of the error that’s made too often today.)

You RUN the “gantlet.” You THROW DOWN the “gauntlet.” As I said, usually people err with “running the gauntlet.” But, your error’s still an error.

And, Caledonian gets it reversed on her etymology.

“Gauntlet” has become an accepted, though NOT preferred, alternative for “gantlet,” precisely because so many people blow this.

(I caught National Geographic doing that; to its credit, the magazine fessed up.)
#20 Socratic Gadfly
September 18, 2006

Puhleeze, for everybody saying, “Poor students,they only had a 25-minute time limit.”

What are they going to do in college, or has college been dumbed down that much in the 20 years since I was there?

Essay questions in history classes? We were expected to crank out several grafs, handwritten, in 25 minutes. True, it was a subject for which we had, theoretically, been studying, but still…
#21 apantomimehorse
September 18, 2006

There’s a reason forced essay-writing has been used as punishment in totalitarian societies: it is the ultimate act of intrusion upon freedom of thought because it demands the subject produce the doctrine in their own words; the victim must give their “own appreciation” of the dogma and “relate it to their own life”.

Of course, our schools don’t approach being totalitarian gulags, though I think there are definite parralles. In my schooling (in the late 90′s), censorship of opinion was generally not a problem, but I strongly disliked how writing and “critical thinking” were taught. Standardized written tests are merely the extreme example of these problems.

Generic prompts are especially pernicious because they demand that the writer have an opinion but not really have an opinion, thereby rewarding disengenuousness. The problem is not that test takers are significantly persecuted by graders for having improper opinions but rather that the tests are far easier for those with trite opinions. Intelligent, honest people think hard when formulating their opinions and naturally censor themselves until they have something to say which they themselves would find worth reading. A primary value of writing is as a process of discovery, and those who understood this, at least at an intuittive level, may have a hard time detaching themselves to write cynically (note the irony: you may be punished for cynical opinions but rewarded for cynically writing treacle).

The solution, I think, is to first of all not try to mix tests for different subjects/skills together. Are these generic-prompt essays supposed to test students for their creativity? Their original thinking? Their analytical skills? If not, using prompts which “anyone can respond to equally as well as other people” is not a real solution because such prompts do not exist. For reasons I suggested above, sticking to “generic life experience” questions biases against people for whom forming and offering such opinions is not done quickly or lightly; we don’t all read self-help books and watch Oprah.

Second, students should not be forced to come up with any ideas at all. If we’re going to be giving them silly prompts that call for superficial answers, we may as well provide all the information. So rather than writing an essay from scratch, students would rewrite a given essay to improve it. This would not only put the focus on all-important rewriting skills, it would help simplify and standardize grading.
#22 Gray
September 18, 2006

An interesting test, but I gave up after 12 minutes. English isn’t my native language, and writing an essay about a philosophic question, under time pressure, seems to be beyond my abilities. I would have felt much more comfortable with the original topic, that, at least imho, offers more opportunities for different approaches to handle it. I have lots of ideas what sources could be used for an essay about memory, but really no clue what to make of the topic at hand.

Which makes me think that A.P. Horse (above) has a valid point: “The solution, I think, is to first of all not try to mix tests for different subjects/skills together.”

Yes, really, what is going to be tested here? Simply the writing abilities of the participant or his success at being able to grasp abstract ideas and present his thoughts in a compelling matter? If this is ‘only’ about writing, a topic that is closer to issues that the participants already thought about would be better. The writer already would have an opinion and could concentrate on wording it. If the question is of a more philosophical nature, those who happen to have stumbled upon it before have an unfair advantage. To build a coherent opinion about a question that hasn’t come up before, and to craft it into an essay, is a bit much to ask for in 20 minites imho.

Well, maybe the real problem is that I don’t like thinking about the topic of your test, and that others don’t have the same scruples. Then, my refusal to finish the test says more about me than about possible problems of your approach…
#23 jscan
September 18, 2006

Hi Dave, thanks for the opportunity to test my spontaneous writing [thinking?] skills! I’m afraid I’m not a blogger so that I only completed the ‘name’ portion of the survey ( and not the URL).. So my question is; how will I know how to access my essay, that is; how will I know how I scored?
#24 Gray
September 18, 2006

“But, your error’s still an error.”

Nitpickers of the world, unite!
#25 Gray
September 18, 2006

“The first draft of anything is shit.”
–Ernest Hemingway

Good quote, Spencer!
And this quote from Winston Churchill may also apply:
“I lacked the time to make it shorter.”
#26 RT
September 18, 2006

Gray – Winnie may have used that line, but it originates with Blaise Pascal.
#27 Dave Munger
September 18, 2006

Okay, I cry “uncle” on the whole gauntlet / gantlet thing. It’s inevitable that we make writing errors when writing about writing, but I’ve fixed it now, so people can focus on the task at hand.
#28 Lisa S
September 18, 2006

You really think the top scoring SAT answers are bad? Clearly you haven’t marked any first year college student papers lately. Those essays show a thoughtfulness, literacy, ability to have and sustain a point of view (and even a point) that I would be delighted if even 25% of the average first year philosophy student had. Instead, any student with that level of writing ability (and it will be the rare student who can write like that and not understand the material) is practically guaranteed an A. And the writing is legible! In a class of 300, I would say about 5 would perform that well on a written exam (as opposed to a take home essay, which they can revise).
#29 Gray
September 18, 2006

“Winnie may have used that line, but it originates with Blaise Pascal.”

Wow, that’s interesting! Thx RT.
#30 Dave Munger
September 18, 2006

Lisa,

It’s been a few years, but I have graded FYC papers in my time. My students’ work was much worse than the example essays provided, but my school did not attract the top SAT scorers, and even if they did, they would have placed out of comp. I was surprised that the *best* writing was this bad, not that such bad writing exists.

I should say, however, having now taken the test, that it was harder than I expected. It’s tough to write well-constructed sentences when you’re under such a time constraint.
#31 Lisa S
September 18, 2006

Dave,
The students I see as a philosophy prof have typically already either tested out of FYC or have already taken some sort of writing course, and I would be really really pleased if they wrote essays on a timed exam that were as coherent, grammatically correct, and generally thoughtful as the SAT answers quoted by the NYT. They manage to sustain an argument, which in my recent experience is something that only the very best students come into college capable of achieving. (Rumour has it that much HS writing in recent past has involved ‘self-expression’ and journal writing, and has not been argument driven — you see this in the violinist’s piece.)
I should say that I teach at a Canadian university, where admission in no way involves SAT scores, but rather is highly competitive and largely based on HS average (all universities are state institutions, and admission is guaranteed if your average meets a cut off).
#32 Constance Reader
September 18, 2006

I just finished the essay and I am content with the results, however, the subject of Mr. Washington’s quote is an issue that I have given a great deal of thought to over the past few years. Therefore, I think that I started writing with an advantage over high school students who would attempt the same essay. I also have the added advantage of life experience. Perhaps part of the problem with these essays is that the topics are not relevant to the life experience of the students that must write on them.
#33 Dave
September 18, 2006

I finished writing the essay just now: I was surprised at how open-ended it was. I finished just under 15 minutes (I think) with four or five short paragraphs; drawing primarily on personal experience.

Good choice of question, by the way.
#34 Battlepanda
September 18, 2006

I coached the SATs this summer, so if you need a minimally experience judge, drop me a line. I think I did alright on the quiz, by my historical standards. My transitions were not smooth, so points off for that. Still, I think I would have given me a five if I were feeling generous. Certainly a four.

Remember folks: What can make or break you on the essay is examples. Top scoring essays typically utilizes three, one for each body paragraph (plus one intro and one conclusion makes a five para essay). If you’re stuck for an example, remember you can use WWII for every essay prompt ever produced by the college board (that I’ve ever seen anyhow.)

By the way bloggers, we probably oughtn’t be smug if we see nothing but fives and sixes on this test — those kids don’t get an option to abort. Perhaps the percentage of tests started but not submitted should be available in the final results.
#35 Darryl Pearce
September 18, 2006

…well, I’ll give it try. …tonight, after work. …or sometime!

How many subjects to you have? What’s the “judge” to subject ratio?
#36 Battlepanda
September 18, 2006

I just read the NYtimes articles, by the way, and I have to agree with Chad — given the limits, I think they were pretty damned good. The third, written in the first person, is the weakest, but then it also seems the least “coached”.
#37 Dave Munger
September 18, 2006

“Perhaps the percentage of tests started but not submitted should be available in the final results.”

Oh, they will. And based on the early returns, it’s a staggeringly large number.
#38 Dave Munger
September 18, 2006

Darryl: At last count, there were 78 responses which included an actual attempt to write the essay. It’s possible that not all of these will be graded, since I didn’t check to see how many of them agreed to have them scored.

As to the ratio of graders to tests, that depends on how much work our volunteer graders are willing to do. I would hope each grader would have to score no more than 30 or so.

We’re also going to put all the answers into a blog, with a poll that will allow readers to grade each one, for the “popular” method of grading.
#39 Gelf
September 18, 2006

I tried this, but I’m pretty sure I ran out of time (would it have told me?). I’m also pretty sure I wouldn’t have had two coworkers interrupting me in my cubicle and three IMing me near-constantly with completely pointless non-work-related stuff in the background during the SATs.

It’s been a long time since I’ve been under a short-term time crunch like that, and I think I may have learned more about my work environment than my writing skills in the last 20 minutes. Yeesh. IM is like some sort of bizarre techno-schizophrenia. You’re trying to get something accomplished but the voices in your extended headspace just… won’t… STOP.

Regardless, I freely admit I didn’t manage to turn out what I would consider an essay. I could have formulated a thesis on that topic and defended it, but not under those conditions. I had to content myself with more or less merely expounding upon the topic generally.
#40 universalcat
September 18, 2006

Man, did that stink. I completely forgot what it’s like to bullshit at such high speeds. Thank goodness for the SAT’s… training a generation of students to lie effectively.
#41 Shalanna Collins
September 18, 2006

One reason you may not see essays for everyone who started the challenge is that the JavaScript dumped them . . . it dumped me after I spent 17 minutes or so typing my essay and then clicked on “Next.” It then went to “The page cannot be found.” I tried refreshing/resending and so forth, and then the BACK button, but to no avail. I never did get it to take my essay. After the challenge is over, I’m going to post my essay on my own LJ; I hate to waste all that typing. (I had the presence of mind to copy the contents of the textbox into a notepad file, because I’ve had stuff like this happen to me in other online surveys.)
#42 Schaz
September 18, 2006

Not so easy! Thank goodness I had a keyboard. I wonder what instructional value the time limit has, in the actual test — or rather, why they felt a more generous time limit wasn’t appropriate?
#43 Dave Munger
September 19, 2006

Shalanna — I’m sorry to hear that happened to you — it’s the first we’ve heard of it happening. Unfortunately we won’t be able to include your essay in our analysis, but thanks for participating. Be sure to send us the link when you post it.
#44 Darryl Pearce
September 19, 2006

Okay…, I did it. Boy, my life sucks when I don’t have a keyboard and an internet search engine in front of me!
#45 Alison
September 20, 2006

Okay. I took the challenge, THEN went to read the Times article. I’ll be interested to see the results. Twenty minutes really isn’t much time. I crank out blog entries pretty quickly (usually), then spend a bit of time tweaking and editing. The time constraint didn’t allow for much editing…
#46 Wray Cummings
September 20, 2006

It’s been half a lifetime since I’ve had to produce an essay for grading. One word: atrophy. Topic sentences, argument funnels, supporting examples are vague memory; pottery shards by a long abandoned cooking fire.

It was an interesting question, worthy of more time than was available. For me it inspires a (rhetorical) followup: does achievement as measured by the SAT accurately reflect the struggles collectively called Education?

(For comparison purposes, it took more than 20 minutes to compose this comment.)