Cognitive Daily

We received an astonishing number of responses to last week’s Casual Fridays study, which claimed to be able to identify what makes a good writer in just a few minutes.

Of course, I wasn’t actually very confident that a brief survey could actually identify the factors that make a good writer. But I did have a hunch that there were certain traits that were more likely to be associated with good writing.

Was there a trick to the study?
Some respondents had a hunch that writing wasn’t the only thing we were interested in. You were right — we were also studying a completely unrelated phenomenon — more on that later.

But we did want to know about your writing as well, so let’s start with that. The study asked a few questions about writing ability: how much writing you do for work/study, how easy writing comes to you, whether you’ve been published, and so on. Then there was a surprise writing test: 3 minutes to write as much as you can on any topic, to be judged for coherence but not content. Finally, a few more questions.

This week’s study asked more of our readers than we usually do, so we expected that we wouldn’t get as many responses as usual. We were wrong about that: over 1,400 responded to the survey, and over 800 wrote an essay response. The average response length was 133 words — quite impressive for a three-minute time limit!

Many of the essays were skeptical that any human would actually read them, but I read every single one. I wanted to get a rough sense of the quality of the essays, so I assigned each a “grade.” To get an A, you had to be coherent for the entire essay, and not switch topics. Just writing complete sentences and only switching topics once or twice earned a B. A semi-random string of sentences earned a C. Incoherent drivel got a D, or in rare cases, an F. These were converted to a 4-point grade scale (where A=4 and F=0). This graph shows the distribution of grades:

i-c553428e0efc72693f42a2ecf4ef7f99-writign1.jpg

As you can see, B was by far the most common grade, with very few Ds and Fs. There were some great little stories, including several I wish the writers had had time to finish. Lots about babies and cats. But did the questions we asked shed any light on what makes a good writer?


For the most part, it confirmed what you might expect. People who said they were good writers were more likely to write better essays. People who said writing came easy to them tended to write more. People who wrote more for work or study also wrote more in our test:

i-c3cf231d5f026b2b2cf247499ae6b64b-writing2.jpg

We asked respondents to estimate how much serious writing they did each week, whether for work or for study. This graph plots the number of pages they said they wrote against what they actually wrote in our test. As you can see, people who write more for work/study also tended to write more here, and the correlation was significant.

If English wasn’t your native language, or if you don’t keep a blog, or if you weren’t participating in NaNoWriMo, you were likely to write less and get a lower grade on your writing. People who had read a novel more recently tended to write more (although there was no correlation between reading recently and the grade received). If you type faster, you were likely to write longer and get a better grade.

These, of course, are all just correlations. We can’t say whether any of these things cause you to be a better writer, or whether the reverse is true. You could, for example, have been motivated to learn to type by your interest and writing ability.

So, what’s the catch?
As I mentioned, there was one additional twist to this study–a genuine experiment. It was motivated by the fact that my family participated in the Arbitron Radio ratings this past week. It’s quite a bit of work to do–every member of the family has to record every single radio station they listen to for an entire week. So how does Arbitron get people to do it? They use several tricks. They send a couple dollars with each family member’s diary. They call several times to make sure you received their materials. But one thing I noticed is that at the end of every call, the interviewer was always sure to ask “can I count on you to return your surveys?” I wondered if that language could be used to motivate people to write.

So in our study, we divided respondents into three groups. Before they started their essays, everyone received the same instructions:

The next part of the study is a brief writing test. It will require you to be totally focused on your writing, but it only takes three minutes.

But one group was specifically asked “Can we count on you to devote your full attention to this section?”, one was asked “will you do it?”, and one group was asked no question at all, and instead proceeded directly to the essay test. Did people write more based on the question they were asked? Here are the results:

i-e07dde39372bb7a36c9c3048e30a6c56-writing3.gif

Those who were asked specifically about whether they would write were significantly more likely to actually complete the essay, and they wrote significantly longer essays. However, there was no significant difference in the results based on the question asked — while more people completed the essay after being asked “can we count on you” rather than “will you do it,” this difference wasn’t statistically significant. What about writing quality? Here are those results:

i-5f827d509940a64de90508676f78ecda-writing4.gif

This time, there was no significant difference in grades between the three groups. But that means that there was also no speed-accuracy trade-off. Even though those who were asked whether they would respond wrote more, their quality wasn’t any worse than those who weren’t asked. Even though writing is a difficult and painful activity for many, just asking them if they will do it actually motivates people to write more.

I have to say, some of the writing produced for this test was very entertaining. Unfortunately, a lot of it is personal and there may be copyright issues, so I’m not going to share any of it with you. This was a fascinating study, and well-worth the time it took me to grade.

(Just a reminder: All Casual Fridays studies are non-scientific. This doesn’t mean we can’t use scientific principles to assess what’s going on, but we can’t make general claims based on the results)

Comments

  1. #1 Mary
    November 20, 2009

    Hmm…babies and cats. Are there any correlations between good writers and topics?

    I wrote about Warren Buffett because I had him on my TV before I started.

  2. #2 Chris Grimes
    November 20, 2009

    You also asked at the end if we’d noticed any typos. What was that about?

  3. #3 Steve Alcorn
    November 20, 2009

    I’d be interested to find out about the general topics chosen, for instance how many were about babies, cats, personal, about the test itself, or about some really random topic.

  4. #4 beck
    November 20, 2009

    Uh. My essay consisted entirely of babies and cats (with sleep playing a somewhat minor role.) I can’t imagine that so many others did as well? Seriously?!? If so, then I am now more curious than ever to see what others wrote!

    I second the idea to do an overview on the subjects chosen.

  5. #5 Dave Munger
    November 20, 2009

    Chris,

    As I was making the survey I noticed a couple typos. Then I decided to leave them in! I was curious if people who caught more typos would be better (or worse) writers than others. There was no correlation between how many typos writers caught and how much they wrote, or their grade.

    Beck/Steve:

    I’m not sure I have the wherewithal to go through all those essays again… we’ll see. There were definitely quite a few essays about writing — no doubt, that was number one. But babies and cats were a close second.

  6. #6 Anon
    November 20, 2009

    I vaguely remember what I wrote about, but could not tell you whether, in my own opinion, I would have judged it to have remained on one topic or split into two. If memory serves, which it often does not, I would call it one topic but would suspect that many people would consider it two.

    But that is neither here nor there. My main question at this point (I’d love to know what “grade” I got, but given the above, I’m not that concerned) is: how many words was my essay?

    I know this was not a genuine experiment, and cannot thus be expected to have been run by an IRB (was it?) nor to have a formal and personal debriefing… but is there any way for participants to find out where they fell on your distribution? (And, as a completely separate question–what are the ethical constraints of blog experiments? Should we hold these trivial (but fun, and IMHO very worthwhile) experiments to something approaching the same ethical standards to which we hold “real” experiments?)

  7. #7 OmegaMom
    November 20, 2009

    Hah. I wrote about writer’s block. I didn’t notice any typos. AND I was one of the “can I count on you?” folk, so I thought the trick to the survey was seeing if you could ignore the minute countdown, or whether it distracted you. Too bad about not being able to share the essays–that would be fun!

  8. #8 Bill
    November 20, 2009

    Oh professor, if you are going to read and grade all writing assignments, we demand our grade! We also demand individual responses about whether we have what it takes to be a good writer.

    I’d say just go ahead and publish the essays. Copyright lawyers need more work. I pledge to contribute to your legal defense fund.

    Seriously, great project.

  9. #9 Heather
    November 20, 2009

    I knew someone would ask for their grade…….Thanks for the results-very intersting and I’m glad I participated

  10. #10 Jen
    November 20, 2009

    I wish I copied down my essay to see how many words I wrote and to try and determine my grade. Arg, I want to know!

  11. #11 Not June Cleaver
    November 20, 2009

    I’m not at all surprised that people want to know their grade. I’m one of them (though based on the criteria listed, I’m pretty sure I got an A). The sad thing is how our education system causes us to be so concerned about the grade (even decades out of school). That’s just one of many reasons I homeschool my kids!

  12. #12 Pala
    November 21, 2009

    Thanks for the (significant) effort that you put into this. Good read.

  13. #13 david craig
    November 21, 2009

    Percentage graphs should never be zoomed in. It looks like you are trying to manipulate implied results when you exaggerate the graph like that.

    Other than that, cool study.

  14. #14 fairyhedgehog
    November 21, 2009

    I enjoyed doing this. I did wonder what the extra research would be: never trust a psychologist to be testing you on what s/he says s/he is testing you on!

    I think I was one of the no-prompt writers. It didn’t occur to me not to write for three minutes having come that far.

    I’m a bit disappointed that we know how well we write. A small part of me always hopes that I am actually doing better than I know I am.

  15. #15 HED
    November 21, 2009

    I was happy to take part in this study. Thanks for the thorough analysis. I’d say that the most important thing in writing is just… starting to write. Thus a good incentive must be provided to motivate the writer-to-be. (i had the question “can we count on you..?”, very good motivator!)

    This is why nanowrimo is good, this is why writing with friends is good. Word count, quality writing and being a non-native english writer come after :)

  16. #16 KGH
    November 21, 2009

    I was curious about the essays, who has written the best and what were the topics chosen?how serious they were really?

  17. #17 lylebot
    November 21, 2009

    I wrote about the TV show Lost because I was watching it on TV at the time. I attempted to explain a particularly tricky piece of non-linear storytelling and found it exceedingly difficult to explain it coherently (if you’ve ever tried to explain the show to someone who doesn’t watch it, I’m sure you know what I mean). I wonder if that affected my grade… :)

  18. #18 jeff m
    November 21, 2009

    re. typos: i do a bit of editing for work (scientists are not on the whole coherent writers) so i try to be alert to typos in those situations. But, in this situation typos were irrelevant; at the time, i couldn’t say whether i saw any or not because they were inconsequential to my reading and understanding of the writing – it hs bn shwn tht yu cn ndrstnd txt wth mst vwls rmovd wth lttl dffclty – the mind sees what it needs to see and can fill in what is expected but absent.
    for me, anyone who would be looking for typos or who, especially, would write in to point them out in a blog, or other, informal piece is no lover of language, merely self-absorbed and incapable of finding any relevant comment to make.
    i would hope i am not alone in this opinion, so any attempt to find a relationship between quality of writing and an eye for error would be meaningless unless you specifically requested it and incorporated a time element into the search for them.
    in all, i loved the exercise/experiment but being a 2-finger typist i kept checking the clock, fumbling over the keyboard and trying to chase my thought train.
    cheers.

  19. #19 CB
    November 21, 2009

    I was not well on that particular Casual Friday. Can I get my grade bumped up? I have a doc’s cert.

  20. #20 Greg Shenaut
    November 22, 2009

    Another statistic it would be interesting to see has to do with your comment about stories that you wish had been finished. It seems to me that a really competent writer should be able to judge how much writing he can do in 3 minutes and come up with a topic/theme/whatever that he could actually finish in the required time. Alternatively, he could watch the clock and when it was down to the last 10-15 seconds, round things out with an well-chosen sentence. About 20 years ago, I took some time off to write the Great American Novel and what killed me was getting bogged down in planning, outlining, and worrying about how to make everything fit together perfectly. I think that the ability to plan writing quickly and well is a critically important part of good writing, and I wonder if that was reflected in some way in this little study.

  21. #21 namran
    November 22, 2009

    That’s interesting, i thought participating it for fun.
    but yes.. can’t help but to return here again to see
    if there is my grade ever published.

    Great job..

  22. #22 curiousgeorgie
    November 23, 2009

    That’s pretty cool. I had no idea what to write about, so I just talked about what my roommates and I had been discussing a few minutes earlier.

    But the importance of language choice actually reminds me of something I stumbled upon last week. This guy experimented with various degrees of forcefulness in telling readers about his Twitter page, and found a similar result.
    http://dustincurtis.com/you_should_follow_me_on_twitter.html

  23. #23 Sue VanHattum
    November 23, 2009

    >I’m a bit disappointed that we know how well we write. A small part of me always hopes that I am actually doing better than I know I am.

    This comment made me curious. Did you check whether gender mattered here? ie Did women underrate their abilities, and/or men overrate?

    I didn’t see this last week, or I would have joined in. Sounds fun.

    To the folks who wish they could see their ‘grade’: This is one person’s opinion of your writing. He may grade you higher or lower than someone else would. He’s already told you that most of you grade yourselves accurately. Relax, and enjoy your writing…

  24. #24 Sarah Merion
    November 23, 2009

    Such a great study! So glad I participated and look forward to reading more from you. Great way to bring me back to your website!

  25. #25 Casey
    November 24, 2009

    I wrote about the topic I am writing my thesis about – mental health parity law. I have no idea how much I wrote because I paid no attention to the clock and only happened to notice that it had run out.

    Pretty cool survey, though. My favorite I have participated in so far. The curious part of me definitely wants to know what other people wrote about and read some stories, but I understand that ethical considerations. Though we did not give any kind of informed consent, I think many of us would consider confidentiality to be implied to some extent.

    For what it’s worth, I give my permission to share my written response with anybody you want, so long as I am given credit for it and it is not used for profit.

  26. #26 Fernando
    November 24, 2009

    Is it too much to ask what was my grade and etc? Maybe. I was the one who wrote about writing and how I didn’t know what to write about. I bet I wasn’t the only one to do so. A bit selfish of me to just come here and ask for my grade, I guess. But can I count on you?

  27. #27 Sorcha
    December 10, 2009

    I wonder how this fits in with the Hemingway maxim that “the first draft of anything is shit”?

The site is undergoing maintenance presently. Commenting has been disabled. Please check back later!