A Blog Around The Clock

As the boundaries between formal and informal scientific communication is blurring – think of pre-print sites, Open Notebook Science and blogs, for starters – the issue of what is citable and how it should be cited is becoming more and more prominent.

There is a very interesting discussion on this topic in the comments section at the Sauropod Vertebra Picture of the Week blog, discussing the place of science blogs in the new communication ecosystem and if a blog post can be and should be cited. What counts as a “real publication”? Is the use of the phrase “real publication” in itself misleading?

You may remember that I have touched on this topic several times over the past few years (as two of my blog posts got cited in scientific papers), most recently in the bottom third (after the word “Except…”) of this post, where I wrote:

National Library of Medicine even made some kind of “official rules for citing blogs” which are incredibly misguided and stupid (and were not changed despite some of us, including myself, contacting them and explaining why their rules are stupid – I got a seemingly polite response telling me pretty much that my opinion does not matter). Anyway, how can anyone make such things ‘official’ when each journal has its own reference formatting rules? If you decide to cite a blog post, you can pretty much use your own brain and put together a citation in a format that makes sense.

The thing is, citing blogs is a pretty new thing, and many people are going to be uneasy about it, or ignorant of the ability and appropriateness to cite blogs, or just so unaware of blogs they would not even know that relevant information can be found on them and subsequently cited. So, if you see that a new paper did not cite your paper with relevant information in it, you can get rightfully angry, but if you see that a new paper did not cite your blog post with relevant information in it, you just shrug your shoulders and hope that one day people will learn….

One of the usual reasons given for not citing blog posts is that they are not peer-reviewed. Which is not true. First, if the post contained errors, readers would point them out in the comments. That is the first layer of peer review. Then, the authors of the manuscript found and read a blog post, evaluated its accuracy and relevance and CHOSE to use it as a reference. That is the second layer of peer-review. Then, the people who review the manuscript will also check the references and, if there is a problem with the cited blog post, they will point this out to the editor. This is the third layer of peer-review. How much more peer-review can one ask for?

And all of that ignores that book chapters, books, popular magazine articles and even newspaper articles are regularly cited, not to mention the ubiqutous “personal communication”. But blogs have a bad rep, because dinosaur corporate curmudgeon journalists think that Drudge and Powerline are blogs – the best blogs, actually – and thus write idiotic articles about the bad quality of blogs and other similar nonsense. Well, if you thought Powerline is the best blog (as Time did, quite intentionally, in order to smear all of the blogosphere by equating it with the very worst right-wing blathering idiotic website that happens to use a blogging software), you would have a low of opinion of blogs, too, wouldn’t you?

But what about one’s inability to detect relevant blog posts, as opposed to research papers to cite? Well, Google it. Google loves blogs and puts them high up in searches. If you are doing research, you are likely to regularly search your keywords not just on MedLine or Web Of Science, but also on Google, in which case the relevant blog posts will pop right up. So, there is no excuse there.

But, some will say, still….a blog post is not peer-reviewed!

Remember that the institution of peer review is very recent. It developed gradually in mid-20th century. None of the science published before that was peer reviewed. Yup, only one of Einstein’s papers ever saw peer review.

Much of the science published today is not peer reviewed either as it is done by industry and by the military and, if published at all, is published only internally (or on the other extreme: citizen science which is published on Flickr or Twitter!). But we see and tend to focus only on the academic research that shows up in academic journals – a tip of the iceberg.

If you think that the editorial process is really important, remember that manuscripts, in their final version, need to be submitted by the authors in the form and format ready for publication. It is not the job of editors to rewrite your poorly written manuscript for you. Thus, scientific papers, even those that went through several rounds of peer review on content are, just like blog posts, self-edited on style, grammar and punctuation (and comprehensibility!). What is the difference between peer reviewers and blog commenters? There are more commenters.

Then, think about the way gradual moving away from the Impact Factor erodes the importance of the venue of publication. This kind of GlamorMagz worship is bound to vanish as article-level metrics get more broadly accepted – faster in less competitive areas of research and in bigger countries, slower in biomedical/cancer research and in small countries.

As the form of the scientific paper itself becomes more and more dynamic and individuals get recognized for their contributions regardless of the URL where that contribution happens, why not cite quality blog posts? Or quality comments on papers themselves?

Comments

#1 Coturnix
June 11, 2009

And if you are unsure, if the blog (or the individual blog post) is aggregated on ResearchBlogging.org, or is a part of an exclusive, invitation-only blogging network like scienceblogs.com, then there is some kind of “stamp of approval” for that blog by the rest of the scientific community.
#2 Blake Stacey
June 11, 2009

In physics land, it’s not too uncommon to see a footnote citing a paper which only exists as a preprint on the arXiv. Citing a blog post seems no less respectable than that, or, to put it more accurately, the ranges of respectability seem to overlap. And citing a blog post can certainly provide more useful information than a note saying nothing more than “Smith and Jones, in press”.

Gawddamn, I hate having to find more information on a subject and getting told that the numbers I need are “in press”.
#3 Matt Wedel
June 11, 2009

Hi Bora, thanks for the link, the discussion, and the resources. I had not given this question much thought until it was raised by one of our readers–which is great example of how blogs really can foster an exchange of information (as opposed to a broadcast). We SV-POW!sketeers will be posting our thoughts in the next few days. For now we are just gratified to see the interest and the range of opinions that the post has generated.
#4 Christina Pikas
June 11, 2009

Not to take anything away from your larger point – which I support wholeheartedly – but research done for the US DoD or by the DoD labs is frequently peer reviewed and then published in technical reports, retrievable from DTIC. Some appears in conferences and journals, too.
#5 rob
June 11, 2009

Then, the people who review the manuscript will also check the references and, if there is a problem with the cited blog post, they will point this out to the editor.
I don’t disagree with what you are saying, just thought I would point out that in my experience I doubt many reviewers actually check many references of papers they are reviewing.
#6 Lobster
June 11, 2009

I disagree with the assertion that it should be OK to cite blogs simply because we never used to peer review things. We started doing peer reviews because it’s valuable, it works, and it improves the quality of the output. It’s unscientific to discard something new and useful simply because it’s more convenient and a longer-standing tradition to use the previous model. If you were to reject a newer, better theory in favor of a simpler, older one purely because you found it more comfortable, you’d never be taken seriously (in fact, you’d be a Creationist).

New theories and new techniques are adopted when they’re better and are rejected when they’re not. It’s how science works, and it’s what separates science from doctrine.
#7 Coturnix
June 11, 2009

That is an interesting view. I am kinda surprised that you are seeing this as going back into the past. I see it as a bright and exciting future, an innovative thing that opens up science in ways that was never open before.
#8 Mickey Schafer
June 11, 2009

“I disagree with the assertion that it should be OK to cite blogs simply because we never used to peer review things. We started doing peer reviews because it’s valuable, it works, and it improves the quality of the output.”

Are dissertations considered “peer reviewed”? Certainly they are not anonymously reviewed, and if the grad student cannot yet be considered a peer of his/her committee members, then maybe the term “peer reviewed” isn’t appropriately applied to dissertations either. The point? Dissertations provide some of the best up-to-date discussions, and I encourage students who have the time to use dissertations, especially intros which often have very good reviews. But my guess is that they fall under the rubric “non-published manuscript” and “non-peer-reviewed”, though not usually unavailable. I find Bora has done an excellent job here of pointing out that available sources of information are themselves far more varied than what is implied by a constrained definition of “peer review”, and therefore, holding the latter up as the only standard is frustrating at best. When I was a grad student, we had things called “working papers” (or something like — it was a while ago) where grad students at various institutions would put together a xerox bound copy of the best papers of the last year (sometimes themed, sometimes not) and would exchange them at conferences as a way of learning new stuff, but also being initiated into the “peer review” process. We used these papers as sources for our own work, albeit as “unpublished manuscripts”. The point isn’t that peer review is bad, but that defining it solely as the vetting process favored by journals is inaccurate.
#9 Lobster
June 11, 2009

Coturnix, as I see it, the sheer volume of information on the internet, combined with extremely low barrier for entry, can make it exceedingly difficult to separate truth from misinformation. The way Google works can make it a little easier to separate the wheat from the chaff, but the newer or more obscure the info, the less that mechanism works.

Mickey Schafer, I’m not a scientist, just a science enthusiast, so I’ll do my best to understand where you’re coming from but forgive me if I’m not intimately familiar with the dissertation system. I’ll agree that they do provide up-to-date, interesting /discussions/. There are some gems in there that will later be polished into accepted discoveries. Now with dissertations (and blogs for that matter) there’s a question of credibility in that you can trust that peer review WOULD yield the same results and that the information IS good, but I see that as the same level of evidence as expert testimony. Which is to say it’s interesting, it’s useful, but it’s not quite hard evidence.

I’m sorry if it sounds like I’m being too hard on the academic process; as I said I’m an outsider and don’t have any first-hand experience with it. From my outsider point of view, I feel like ideally, science is a puzzle with concrete, distinct pieces that indisputably fit together. If someone describes to you what one piece looks like, or provides for you a reasonable facsimile, you can get a better image of what the completed puzzle looks like but you can’t definitively complete the puzzle.

Ideally, you understand.
#10 Zach Miller
June 11, 2009

You have to learn to figure out the ethos of a blog. I’d rather cite Darren Naish than myself, for instance. It’s important to know the credentials of the person running the blog and whether they know what they’re talking about or not. This can be difficult for somebody unfamiliar in the field, but is readily apparent to a specialist.

It’s also important to take that one step further: Let’s say you’re reading blogs by Ryoksuke Motani and Jack Horner. They both write posts about ichythosaurs. I’m going to cite Motani’s blog over Horner’s because Motani’s specialty is ichthyosaurs.
#11 PeggyOh
June 11, 2009

I think refining the stamp of approval and citation/impact metrics are critical because some peer review is better than others. For instance, sponsored research in the pharmaceutical industry has been criticized for self-selecting reviewers and I suspect this argument is being made about science blogs too. Sponsorship may include support for publishing/disseminating positive results (via invited conference presentations and paid fee-based journal supplements, technical publications, or continuing education collaterals). External peer reviewers – anonymously selected by an independent third party – are perceived to be more objective. Rob: they are expected to know the literature and therefore provide a “check”. Medical journals editors have worked hard to enlist prestigious, independent reviewers and to establish/enforce rigorous standards (selective reporting of data, conflict of interest, etc.). Lets hope greater transparency, including new metrics and the growing crowd will fulfill the promise in the blogosphere.
#12 Comrade PhysioProf
June 11, 2009

Thus, scientific papers, even those that went through several rounds of peer review on content are, just like blog posts, self-edited on style, grammar and punctuation (and comprehensibility!).

This is not universally the case. Many journals employ copy editors to copy edit accepted manuscripts.
#13 Isis the Scientist
June 11, 2009

Nooooooo! I need my peer review!!!!
#14 Coturnix
June 11, 2009

Matt and Mike wrote the second installment
#15 daedalus2u
June 12, 2009

There are some good things about peer review; there are also some bad things. It is good if it keeps stuff that is wrong out of the literature. Peers are not magic. There is plenty of stuff that is wrong in the peer reviewed literature. Sometimes peer review keeps correct stuff out if it contradicts wrong stuff that is accepted by peers.

A good example I like to use is the photoelectric effect. Millikan (obviously a peer in physics) didn’t accept the photoelectric effect as being real even after it was Millikan’s data that proved it to everyone else. If Millikan had been reviewing Einstein’s publications of the photoelectric effect they never would have been published. It was Millikan’s belief that Einstein’s description of the photoelectric was wrong that motivated him to gather the very precise data that showed it to be correct.

If you do not understand a reference, you should not use it, whether it is peer reviewed or not. If you feel uncomfortable using a particular reference that is not peer-reviewed, what that unease actually means is that you are not sufficiently expert in that field to evaluate the reference as a peer. If you can’t evaluate a paper as a peer, then you should not use it as a reference even if others have reviewed it. Scientific understanding is not something that can be delegated. If you do, then you are not a scientist, you are a believer, someone who takes things on faith because they were printed in a peer-reviewed glamour mag.

This is much of the problem of non-scientists cherry-picking results from peer reviewed articles and misusing them to promote an agenda. The problem isn’t that the original paper wasn’t peer reviewed; it is that the person using it doesn’t understand it, and/or is deliberately distorting what it says.

Non-peers can misuse peer reviewed papers very easily. To me, that is much more of a problem than peers misusing non-peer reviewed papers.

I really dislike the term “peer”. It is an elitist term used to support the Kyriarchy (aka the kabuki dance) of science. Getting a degree does not make one a “peer”. The PhD candidate is expected to be the most expert person in the world on their dissertation subject. They do not magically become a “peer” the day their dissertation is signed off on. That is putting form ahead of substance. Holding out recognition as a peer until after the thesis is signed off on is nonsense. It distorts the scientific process by putting a social pecking order ahead of scientific judgment.
#16 James F
June 13, 2009

Who throws a shoe? Er…Who cites a blog post? Honestly!

In the biological sciences, the overwhelming majority of citations are 1) research papers presenting relevant primary data, 2) literature reviews, and 3) online databases and tools. References to unpublished data haven’t dwindled as much I thought they would in this age of limitless online supplementary material, and I’m still suspicious when I see them. Likewise for personal communications, although I understand that situations arise when they are appropriate. What could be presented in a blog post that isn’t covered by these categories? The best I can come up with is an insightful discussion of a paper or topic that hasn’t appeared elsewhere, cited as one would a review article – but then again, how many letters to the editor or commentaries make it to the bibliography of published papers? In short, not impossible, but the circumstances would be exceedingly rare.
#17 Matt Wedel
June 14, 2009

I’m still suspicious when I see them. Likewise for personal communications

My co-blogger Mike Taylor derogatorily refer to pers comms as the tweets of the academic world.
#18 Nick Anthis
June 14, 2009

I don’t see a major problem citing a blog post per se, at least if it’s done in place of citing a review article. However, though, it’s really a moot point, because ideally most of your references should be to the original experimental work. In this case, then, it really does need to be an original, published, peer-reviewed study–and a blog post can’t really legitimately or reliably substitute for that.
#19 James F
June 14, 2009

Matt Wedel,

Ha! That’s a good one, although “data not shown” gets my vote for the equivalent of a tweet. At least with pers comms you have to secure a written statement for the journal.