We received an astonishing number of responses to last week's Casual Fridays study, which claimed to be able to identify what makes a good writer in just a few minutes.
Of course, I wasn't actually very confident that a brief survey could actually identify the factors that make a good writer. But I did have a hunch that there were certain traits that were more likely to be associated with good writing.
Was there a trick to the study?
Some respondents had a hunch that writing wasn't the only thing we were interested in. You were right -- we were also studying a completely unrelated phenomenon -- more on that later.
But we did want to know about your writing as well, so let's start with that. The study asked a few questions about writing ability: how much writing you do for work/study, how easy writing comes to you, whether you've been published, and so on. Then there was a surprise writing test: 3 minutes to write as much as you can on any topic, to be judged for coherence but not content. Finally, a few more questions.
This week's study asked more of our readers than we usually do, so we expected that we wouldn't get as many responses as usual. We were wrong about that: over 1,400 responded to the survey, and over 800 wrote an essay response. The average response length was 133 words -- quite impressive for a three-minute time limit!
Many of the essays were skeptical that any human would actually read them, but I read every single one. I wanted to get a rough sense of the quality of the essays, so I assigned each a "grade." To get an A, you had to be coherent for the entire essay, and not switch topics. Just writing complete sentences and only switching topics once or twice earned a B. A semi-random string of sentences earned a C. Incoherent drivel got a D, or in rare cases, an F. These were converted to a 4-point grade scale (where A=4 and F=0). This graph shows the distribution of grades:
As you can see, B was by far the most common grade, with very few Ds and Fs. There were some great little stories, including several I wish the writers had had time to finish. Lots about babies and cats. But did the questions we asked shed any light on what makes a good writer?
For the most part, it confirmed what you might expect. People who said they were good writers were more likely to write better essays. People who said writing came easy to them tended to write more. People who wrote more for work or study also wrote more in our test:
We asked respondents to estimate how much serious writing they did each week, whether for work or for study. This graph plots the number of pages they said they wrote against what they actually wrote in our test. As you can see, people who write more for work/study also tended to write more here, and the correlation was significant.
If English wasn't your native language, or if you don't keep a blog, or if you weren't participating in NaNoWriMo, you were likely to write less and get a lower grade on your writing. People who had read a novel more recently tended to write more (although there was no correlation between reading recently and the grade received). If you type faster, you were likely to write longer and get a better grade.
These, of course, are all just correlations. We can't say whether any of these things cause you to be a better writer, or whether the reverse is true. You could, for example, have been motivated to learn to type by your interest and writing ability.
So, what's the catch?
As I mentioned, there was one additional twist to this study--a genuine experiment. It was motivated by the fact that my family participated in the Arbitron Radio ratings this past week. It's quite a bit of work to do--every member of the family has to record every single radio station they listen to for an entire week. So how does Arbitron get people to do it? They use several tricks. They send a couple dollars with each family member's diary. They call several times to make sure you received their materials. But one thing I noticed is that at the end of every call, the interviewer was always sure to ask "can I count on you to return your surveys?" I wondered if that language could be used to motivate people to write.
So in our study, we divided respondents into three groups. Before they started their essays, everyone received the same instructions:
The next part of the study is a brief writing test. It will require you to be totally focused on your writing, but it only takes three minutes.
But one group was specifically asked "Can we count on you to devote your full attention to this section?", one was asked "will you do it?", and one group was asked no question at all, and instead proceeded directly to the essay test. Did people write more based on the question they were asked? Here are the results:
Those who were asked specifically about whether they would write were significantly more likely to actually complete the essay, and they wrote significantly longer essays. However, there was no significant difference in the results based on the question asked -- while more people completed the essay after being asked "can we count on you" rather than "will you do it," this difference wasn't statistically significant. What about writing quality? Here are those results:
This time, there was no significant difference in grades between the three groups. But that means that there was also no speed-accuracy trade-off. Even though those who were asked whether they would respond wrote more, their quality wasn't any worse than those who weren't asked. Even though writing is a difficult and painful activity for many, just asking them if they will do it actually motivates people to write more.
I have to say, some of the writing produced for this test was very entertaining. Unfortunately, a lot of it is personal and there may be copyright issues, so I'm not going to share any of it with you. This was a fascinating study, and well-worth the time it took me to grade.
(Just a reminder: All Casual Fridays studies are non-scientific. This doesn't mean we can't use scientific principles to assess what's going on, but we can't make general claims based on the results)
- Log in to post comments
Hmm...babies and cats. Are there any correlations between good writers and topics?
I wrote about Warren Buffett because I had him on my TV before I started.
You also asked at the end if we'd noticed any typos. What was that about?
I'd be interested to find out about the general topics chosen, for instance how many were about babies, cats, personal, about the test itself, or about some really random topic.
Uh. My essay consisted entirely of babies and cats (with sleep playing a somewhat minor role.) I can't imagine that so many others did as well? Seriously?!? If so, then I am now more curious than ever to see what others wrote!
I second the idea to do an overview on the subjects chosen.
Chris,
As I was making the survey I noticed a couple typos. Then I decided to leave them in! I was curious if people who caught more typos would be better (or worse) writers than others. There was no correlation between how many typos writers caught and how much they wrote, or their grade.
Beck/Steve:
I'm not sure I have the wherewithal to go through all those essays again... we'll see. There were definitely quite a few essays about writing -- no doubt, that was number one. But babies and cats were a close second.
I vaguely remember what I wrote about, but could not tell you whether, in my own opinion, I would have judged it to have remained on one topic or split into two. If memory serves, which it often does not, I would call it one topic but would suspect that many people would consider it two.
But that is neither here nor there. My main question at this point (I'd love to know what "grade" I got, but given the above, I'm not that concerned) is: how many words was my essay?
I know this was not a genuine experiment, and cannot thus be expected to have been run by an IRB (was it?) nor to have a formal and personal debriefing... but is there any way for participants to find out where they fell on your distribution? (And, as a completely separate question--what are the ethical constraints of blog experiments? Should we hold these trivial (but fun, and IMHO very worthwhile) experiments to something approaching the same ethical standards to which we hold "real" experiments?)
Hah. I wrote about writer's block. I didn't notice any typos. AND I was one of the "can I count on you?" folk, so I thought the trick to the survey was seeing if you could ignore the minute countdown, or whether it distracted you. Too bad about not being able to share the essays--that would be fun!
Oh professor, if you are going to read and grade all writing assignments, we demand our grade! We also demand individual responses about whether we have what it takes to be a good writer.
I'd say just go ahead and publish the essays. Copyright lawyers need more work. I pledge to contribute to your legal defense fund.
Seriously, great project.
I knew someone would ask for their grade.......Thanks for the results-very intersting and I'm glad I participated
I wish I copied down my essay to see how many words I wrote and to try and determine my grade. Arg, I want to know!
I'm not at all surprised that people want to know their grade. I'm one of them (though based on the criteria listed, I'm pretty sure I got an A). The sad thing is how our education system causes us to be so concerned about the grade (even decades out of school). That's just one of many reasons I homeschool my kids!
Thanks for the (significant) effort that you put into this. Good read.
Percentage graphs should never be zoomed in. It looks like you are trying to manipulate implied results when you exaggerate the graph like that.
Other than that, cool study.
I enjoyed doing this. I did wonder what the extra research would be: never trust a psychologist to be testing you on what s/he says s/he is testing you on!
I think I was one of the no-prompt writers. It didn't occur to me not to write for three minutes having come that far.
I'm a bit disappointed that we know how well we write. A small part of me always hopes that I am actually doing better than I know I am.
I was happy to take part in this study. Thanks for the thorough analysis. I'd say that the most important thing in writing is just... starting to write. Thus a good incentive must be provided to motivate the writer-to-be. (i had the question "can we count on you..?", very good motivator!)
This is why nanowrimo is good, this is why writing with friends is good. Word count, quality writing and being a non-native english writer come after :)
I was curious about the essays, who has written the best and what were the topics chosen?how serious they were really?
I wrote about the TV show Lost because I was watching it on TV at the time. I attempted to explain a particularly tricky piece of non-linear storytelling and found it exceedingly difficult to explain it coherently (if you've ever tried to explain the show to someone who doesn't watch it, I'm sure you know what I mean). I wonder if that affected my grade... :)
re. typos: i do a bit of editing for work (scientists are not on the whole coherent writers) so i try to be alert to typos in those situations. But, in this situation typos were irrelevant; at the time, i couldn't say whether i saw any or not because they were inconsequential to my reading and understanding of the writing - it hs bn shwn tht yu cn ndrstnd txt wth mst vwls rmovd wth lttl dffclty - the mind sees what it needs to see and can fill in what is expected but absent.
for me, anyone who would be looking for typos or who, especially, would write in to point them out in a blog, or other, informal piece is no lover of language, merely self-absorbed and incapable of finding any relevant comment to make.
i would hope i am not alone in this opinion, so any attempt to find a relationship between quality of writing and an eye for error would be meaningless unless you specifically requested it and incorporated a time element into the search for them.
in all, i loved the exercise/experiment but being a 2-finger typist i kept checking the clock, fumbling over the keyboard and trying to chase my thought train.
cheers.
I was not well on that particular Casual Friday. Can I get my grade bumped up? I have a doc's cert.
Another statistic it would be interesting to see has to do with your comment about stories that you wish had been finished. It seems to me that a really competent writer should be able to judge how much writing he can do in 3 minutes and come up with a topic/theme/whatever that he could actually finish in the required time. Alternatively, he could watch the clock and when it was down to the last 10-15 seconds, round things out with an well-chosen sentence. About 20 years ago, I took some time off to write the Great American Novel and what killed me was getting bogged down in planning, outlining, and worrying about how to make everything fit together perfectly. I think that the ability to plan writing quickly and well is a critically important part of good writing, and I wonder if that was reflected in some way in this little study.
That's interesting, i thought participating it for fun.
but yes.. can't help but to return here again to see
if there is my grade ever published.
Great job..
That's pretty cool. I had no idea what to write about, so I just talked about what my roommates and I had been discussing a few minutes earlier.
But the importance of language choice actually reminds me of something I stumbled upon last week. This guy experimented with various degrees of forcefulness in telling readers about his Twitter page, and found a similar result.
http://dustincurtis.com/you_should_follow_me_on_twitter.html
>I'm a bit disappointed that we know how well we write. A small part of me always hopes that I am actually doing better than I know I am.
This comment made me curious. Did you check whether gender mattered here? ie Did women underrate their abilities, and/or men overrate?
I didn't see this last week, or I would have joined in. Sounds fun.
To the folks who wish they could see their 'grade': This is one person's opinion of your writing. He may grade you higher or lower than someone else would. He's already told you that most of you grade yourselves accurately. Relax, and enjoy your writing...
Such a great study! So glad I participated and look forward to reading more from you. Great way to bring me back to your website!
I wrote about the topic I am writing my thesis about - mental health parity law. I have no idea how much I wrote because I paid no attention to the clock and only happened to notice that it had run out.
Pretty cool survey, though. My favorite I have participated in so far. The curious part of me definitely wants to know what other people wrote about and read some stories, but I understand that ethical considerations. Though we did not give any kind of informed consent, I think many of us would consider confidentiality to be implied to some extent.
For what it's worth, I give my permission to share my written response with anybody you want, so long as I am given credit for it and it is not used for profit.
Is it too much to ask what was my grade and etc? Maybe. I was the one who wrote about writing and how I didn't know what to write about. I bet I wasn't the only one to do so. A bit selfish of me to just come here and ask for my grade, I guess. But can I count on you?
I wonder how this fits in with the Hemingway maxim that "the first draft of anything is shit"?