Grey literature considered harmful?

So the United Nations' Intergovernmental Panel on Climate Change is mired in a rapidly heating controversy over a report that apparently let some dubious information slip through the cracks. Here's the money quote:

The discovery of the glaciers mistake has focused attention on the IPCC's use of so-called grey literature: reports that do not appear in conventional scientific journals, and are instead drawn from sources such as campaign groups, companies and student theses. The IPCC's rules allow such grey literature, but many people have been surprised at the scale of its inclusion.

Oh my, oh dear. Let's take that apart a bit at a time, see what's likely going on, and suggest some remedies.

That's a fairish definition of grey literature, but what it leaves out is that the importance and acceptance of certain genres of grey literature varies considerably by discipline (although I think most accept dissertations as quality, citable work, assuming they're recent enough). For example, quite a few social-science disciplines have a flourishing working-papers culture; the expectation is that many (though probably not all) working papers will go through the peer-review wringer at some juncture, but it's important both for authors and readers to circulate the ideas fast.

I don't know of any literature on this specific point (which doesn't mean it doesn't exist, just that I haven't looked), but from my admittedly anecdotal experience, one factor that seems to create a grey-literature–friendly culture is a desire for influence beyond the academy: influence on practitioners, policymakers, non-profits, or the public generally. The IR I run has several grey-literature collections predicated on precisely this, and I don't think it's any coincidence that a strong vein of public-policy discourse runs through the research blogosphere generally and ScienceBlogs specifically.

Why is grey literature a better choice than the peer-reviewed journal literature in this context? Simply because it's much more accessible. Practitioners, policymakers, and particularly nonprofits have limited if any access to toll-access peer-reviewed journals. If you want them to see it and use it, placing it in a toll-access journal is tantamount to shredding it.

Now go back and read the paragraph from the Guardian again. "Surprised at the scale of [grey literature's] inclusion." Well, I'm not. I think a mixture of two factors is an odds-on bet in this clash of the literatures: a grey-literature–friendly discipline working with a grey-literature–unfriendly one, and the simple matter of access I just explained.

Climate science strikes me as grey-literature–unfriendly for cogent reasons. One is that the peer-review process (one hopes!) does yeoman's work eliminating bad science and bad data in a field where plenty of people grinding axes are happy to pollute the discourse with bad science and bad data. Another is that climate scientists need to limit their discourse population somewhat, or they'll be overwhelmed with axe-grinding bizarrerie from outside the field. I would guess it doesn't trouble them much that their literature isn't open-access, and it may even please them, by way of getting on with their work without having to stop every five seconds to deal with some ignorant lout who Googled their latest paper.

But social scientists, they love their grey-lit—and thus the clash of cultures. Neither the social scientists nor the climate scientists realized that they didn't have the same standards for reliable, citable previous work. The social scientists applied their normal quality standards and search techniques, not realizing that the climate scientists had different ones, not even thinking to ask. What the social scientists found on the open Web about climate science had a high probability of being junk, given that peer-reviewed climate science is almost entirely toll-access. (I found only 22 journals in DOAJ, two of which are Bentham Open and thus liable to be junk; on the green-OA side, I certainly haven't heard that climate scientists are heavy self-archivers.) I expect similar clashes play out in quite a few interdisciplinary collaborations.

So what do we take away from this?

  • Scientists, if you want people outside your discipline to read your work in preference to whatever they can find on the open Web, make it open access; that may not be sufficient, of course, but it's absolutely necessary. However, all of us need to take into account that the knee-jerk call for everything to be OA removes a shield from a significant set of researchers, those poor hot-spotlighted souls whose professional lives OA would make a living hell.
  • If you're troubled by bad science and bad data on the open Web, the answer is to make better science and better data open access. There will always be bad science and bad data, and it will seek the path of least resistance to earn the most eyeballs possible. Locking up the broccoli only increases candy consumption.
  • Interdisciplinary collaborations had better start by establishing common ground for citable literature. Disciplinary cultures differ, and for legitimate reasons; practices do not necessarily transfer.
  • Post-publication review and commentary don't just offer an alternative to peer review. They may help keep junk science from taking over the wider discourse, to the benefit of all.

I would never recommend that libraries discontinue collecting grey literature, especially in digital form. For one thing, some grey literature gets much more scrutiny than the average peer-reviewed article: consider how often and by how many people dissertations are reviewed, and how many rewrites they get! For another, I see no reason collection practices should be governed by the most restrictive disciplinary cultures out there.

I do acknowledge grey literature for what it is, however, and I hope this culture-clash sparks discussion about how best to manage differing citation practices when collaborating.

More like this

I worked with a bunch of Urban Planning academics once; very heavily into grey literature (as in seeing it as extremely important input into their work). And you're right about economics (WoPEc, etc), Physics (ArXiv & its extensions etc) and so on. I think you're right about the specific risks in climate science.

But some of your comments made me wonder. How grey are blogs? Try for example...

By Chris Rusbridge (not verified) on 01 Mar 2010 #permalink

Blogs are so grey that librarians haven't even managed to agree yet that collecting them is a good idea.

I think the added scrutiny of peer review trumps timeliness and accessiblity - the IPCC reports should reflect what's well-establised, so reliablity is more important. I don't think accessibility is that big of an issue, since the IPCC report itself is supposed to be that accessible summary of the state of the science.

However, the controversy here isn't about the inclusion of grey literature overall, it's about the inclusion of grey literature that shouldn't be used even if grey literature is allowed - the specific citations mentioned in the article are from advocacy groups, which, while they have their place, are inappropriate as a source for the IPCC reports. There wouldn't be such an uproar if the non-peer reviewed journal citations were all theses or working papers from reputable sources. The problem with the broad inclusion of grey literature is that it's allowing in grey literature that is of dubious quality. Since most of the action in climate science is in peer-reviewed journals, I don't think much would be lost by sticking to them.

I understand that and I agree with it. I'm more interested in "how did this happen?" and "how are such misunderstandings to be prevented in future?"

I also think it's important to establish that the report authors were not necessarily acting in bad faith -- just by a different set of assumptions about what sources were all right to refer to.

There is also the flip side of the coin, at lot of the "grey" literature is governmental and other technical reports which have important information which will never be peer-reviewed, or policy impact analysis, which will again not appear in the peer-reviewed literature. A recent example from a different area is the House of Commons report on homoeopathy, not peer-reviewed but a valuable information resource and a key pointer to policy directions.

And I would not trust the Guardians analysis, they seem to be having a problem with... ahh... accuracy (see also links therein)

Absolutely. Truth does lurk outside the peer-reviewed literature!

Though perhaps not in the Guardian... Still, it's an interesting test case.

The assumption that peer review is the gold standard for truth and justice is as critical to its credibility as are the many unstated assumptions that accompany AGW models. The peer review process assumes integrity on the part of editors (who select reviewers based on or not based upon known biases) as well as the reviewers (who are only blind reviewers if they choose to be). It is increasingly aparent that there is more to be gained (grants) by assuming AGW in your models than there is in questioning it. Until recently, only the AGW sceptics have been akin to Galileo in challenging popular beliefs.

This, by no means questions the very legitimate concern that "grey lit" is replete with garbage which (as our friend promoting the demise of Himilayan glaciers found) is useful in getting panic spread through an IPCC report.