Grey literature considered harmful?

So the United Nations' Intergovernmental Panel on Climate Change is mired in a rapidly heating controversy over a report that apparently let some dubious information slip through the cracks. Here's the money quote:

The discovery of the glaciers mistake has focused attention on the IPCC's use of so-called grey literature: reports that do not appear in conventional scientific journals, and are instead drawn from sources such as campaign groups, companies and student theses. The IPCC's rules allow such grey literature, but many people have been surprised at the scale of its inclusion.

Oh my, oh dear. Let's take that apart a bit at a time, see what's likely going on, and suggest some remedies.

That's a fairish definition of grey literature, but what it leaves out is that the importance and acceptance of certain genres of grey literature varies considerably by discipline (although I think most accept dissertations as quality, citable work, assuming they're recent enough). For example, quite a few social-science disciplines have a flourishing working-papers culture; the expectation is that many (though probably not all) working papers will go through the peer-review wringer at some juncture, but it's important both for authors and readers to circulate the ideas fast.

I don't know of any literature on this specific point (which doesn't mean it doesn't exist, just that I haven't looked), but from my admittedly anecdotal experience, one factor that seems to create a grey-literature–friendly culture is a desire for influence beyond the academy: influence on practitioners, policymakers, non-profits, or the public generally. The IR I run has several grey-literature collections predicated on precisely this, and I don't think it's any coincidence that a strong vein of public-policy discourse runs through the research blogosphere generally and ScienceBlogs specifically.

Why is grey literature a better choice than the peer-reviewed journal literature in this context? Simply because it's much more accessible. Practitioners, policymakers, and particularly nonprofits have limited if any access to toll-access peer-reviewed journals. If you want them to see it and use it, placing it in a toll-access journal is tantamount to shredding it.

Now go back and read the paragraph from the Guardian again. "Surprised at the scale of [grey literature's] inclusion." Well, I'm not. I think a mixture of two factors is an odds-on bet in this clash of the literatures: a grey-literature–friendly discipline working with a grey-literature–unfriendly one, and the simple matter of access I just explained.

Climate science strikes me as grey-literature–unfriendly for cogent reasons. One is that the peer-review process (one hopes!) does yeoman's work eliminating bad science and bad data in a field where plenty of people grinding axes are happy to pollute the discourse with bad science and bad data. Another is that climate scientists need to limit their discourse population somewhat, or they'll be overwhelmed with axe-grinding bizarrerie from outside the field. I would guess it doesn't trouble them much that their literature isn't open-access, and it may even please them, by way of getting on with their work without having to stop every five seconds to deal with some ignorant lout who Googled their latest paper.

But social scientists, they love their grey-lit—and thus the clash of cultures. Neither the social scientists nor the climate scientists realized that they didn't have the same standards for reliable, citable previous work. The social scientists applied their normal quality standards and search techniques, not realizing that the climate scientists had different ones, not even thinking to ask. What the social scientists found on the open Web about climate science had a high probability of being junk, given that peer-reviewed climate science is almost entirely toll-access. (I found only 22 journals in DOAJ, two of which are Bentham Open and thus liable to be junk; on the green-OA side, I certainly haven't heard that climate scientists are heavy self-archivers.) I expect similar clashes play out in quite a few interdisciplinary collaborations.

So what do we take away from this?

• Scientists, if you want people outside your discipline to read your work in preference to whatever they can find on the open Web, make it open access; that may not be sufficient, of course, but it's absolutely necessary. However, all of us need to take into account that the knee-jerk call for everything to be OA removes a shield from a significant set of researchers, those poor hot-spotlighted souls whose professional lives OA would make a living hell.
• If you're troubled by bad science and bad data on the open Web, the answer is to make better science and better data open access. There will always be bad science and bad data, and it will seek the path of least resistance to earn the most eyeballs possible. Locking up the broccoli only increases candy consumption.
• Interdisciplinary collaborations had better start by establishing common ground for citable literature. Disciplinary cultures differ, and for legitimate reasons; practices do not necessarily transfer.
• Post-publication review and commentary don't just offer an alternative to peer review. They may help keep junk science from taking over the wider discourse, to the benefit of all.

I would never recommend that libraries discontinue collecting grey literature, especially in digital form. For one thing, some grey literature gets much more scrutiny than the average peer-reviewed article: consider how often and by how many people dissertations are reviewed, and how many rewrites they get! For another, I see no reason collection practices should be governed by the most restrictive disciplinary cultures out there.

I do acknowledge grey literature for what it is, however, and I hope this culture-clash sparks discussion about how best to manage differing citation practices when collaborating.