Cognitive Daily

Most CogDaily readers are familiar with the little icon we developed to indicate when we were reporting on peer reviewed research. We created it when we began to offer links to news and blog posts, as a way of distinguishing those less “serious” posts from when we were talking about peer-reviewed journal articles.

But Sister Edith Bogue of Monastic Musings recently pointed out that other academic bloggers could also make use of the icon, to distinguish when they’re blogging about news, family, books, etc., from serious scholarship. But our icon isn’t ideal for this purpose since its design is specifically linked to our site. I also think a public icon should come with some guidelines for use.

So Sister Edith and I, along with ScienceBloggers John Wilkins and Mike Dunford, have decided to work together to develop such an icon, along with a web site where we can link to bloggers who’ve pledged to use it following the guidelines we develop. But we don’t represent the blogging community as a whole, so we thought we’d also ask for your input. I’ll start the discussion with a few key questions. You can post answers — or your own questions — in the comments section.

Is “Report on Peer Reviewed Research” a good tagline? Any suggestions for a different wording?
What should we call the organization that sponsors the icon? I was thinking something on the lines of “Bloggers for peer review.” Any other ideas?
What, exactly, should the icon signify? At a minimum, the blogger should have carefully read the original research report. Any other guidelines? (On CogDaily, it means that both Greta and I have read the report, and that we’re attempting to offer a thoughtful summary of the results)
How do we define “peer review?” For example, some conference presentations are technically peer reviewed, but this process seems to me too cursory to qualify — after all, the reviewers haven’t even seen the final product. Some journals with very limited peer review processes also might not qualify. How do we decide what’s in and what’s out? Do we make a list?
Should there be a process for policing abuses of the icon? How would that work?
What about copyright for the icon? Should it be in the public domain? Or would some sort of license like the GPL or Creative Commons be better?
How should we design the icon? A contest? How would results be judged?

That ought to be enough to get the discussion rolling. As I suggested, feel free to offer both answers and additional questions. This is an exciting project!

Comments

#1 Andrew
August 8, 2007

I’d advise you to the copyright and licence the icon. If you put the icon in the public domain then you can’t control its use.
#2 Todd O.
August 8, 2007

Be sure to include us social scientists in this project. Sociologists and Anthropologists would greatly benefit from a way to point to peer-reviewed research in our posts as well. Thanks for taking this on.
#3 Mike fitzGerald
August 8, 2007

You are looking for a symbol that shows that somebody with credentials is commenting on something within their field of expertise. As a first suggestion I would recommend that use of the icon provides a link to a professional biography so that readers can check this. This biography would typically be on a “biography” or “media” page which most bloggers would have anyway. Any more policing or classification than this would be a nightmare to implement and easy to abuse.
#4 Todd O.
August 8, 2007

My university’s online catalogue has a way to look up a journal to check if it’s peer reviewed. It’s not perfect, but it may be a “good enough” solution for this. Also, different disciplines have different processes of peer review, so the wording could include something about it being “discipline specific.”

Perhaps the icon could itself either include a URL or be a link to a one page web site outlining what the icon means and that the user of the icon has pledged to abide by those rules.

Another idea: It might be relatively easy to set up a “Disputed Sources” page with links to the blog post in question for those occasions when someone who uses the icon notices it being misused.

And I definitely think that it would be a good idea to have users “sign up” with a list of criteria, and use their full name, credentials, and/or university affiliation available as a list on the Web page for the Icon.
#5 Zachary Tong
August 8, 2007

Mike: I think you are looking at the intent of the icon a little wrong. I don’t think it is meant to verify that the blog author has valid credentials, but rather to:

—Label blog posts that are about valid research rather than news, opinion or miscellaneous topics
—Identify that the blog author is writing a report on a primary source (the study/journal) and not a secondary source (an article about the study in Discover Magazine, for instance)
—Identify studies that have been peer reviewed by an established journal and is not sensationalist hype or unsubstantiated data

I think. Someone please correct me if I’m wrong Some more thoughts:

—I think that anyone using this icon should link to the original study (even if it is to a for-pay access journal, the link should be provided).
—How do you deal with journals like PLoS? I love PLoS but am not sure how their peer review process works. On the same hand, I would hate to see it excluded because they have some good (and open!) studies.
—The main site for the organization should have an automatic feed based on trackbacks. That way, bloggers who post with the icon automatically get a link sent to the main site. The site could then be used by readers as a central depository for blog posts about peer-reviewed studies.
#6 The Primate Diaries
August 8, 2007

I agree with Zachary, the icon should just be to let people know the discussion to follow is about a serious piece of scholarship rather than a bit of whimsical fluff (not that we ever do anything of the kind).

However, what about PNAS? It’s peer reviewed for some but National Academy members can publish what they like. Would this journal qualify since only the top scientists are admitted (and we just assume they’re up to snuff)?

Also, would you put the icon on the blog when you’re reviewing a peer reviewed article as well as quoting a popular review of the same article?

This may be more detail than is necessary. I think the icon is a great idea. I would go along with a contest idea for design submissions.
#7 John Carter
August 8, 2007

You should use a _much_ bigger font in the icon.

I couldn’t read it until this article highlighted it so I peered closely at it.

To me it just looked like a cheezburger. Perhaps, I thought, you like cheese burgers, or perhaps, like me, you like lolcats
#8 Alun
August 8, 2007

Report on Peer Reviewed Research would be good for the logo, but perhaps Peer-Review Reporting would be shorter for the index website?

A central website aggregating excerpts from the posts would be a useful central point, and would also provide a simple means to police the use of the logo. If your posts aren’t getting indexed then why use the logo? The central website would also provide a reasonable challenge to Googlenews.

I’d expect the logo would be used for blog posts about research the blogger has actually read. The exact definition of peer-reviewed is a problem and there may be no one-size fits all solution if the bloggers are from different fields. Possibly guidance that if in doubt don’t use the logo would be ok. After all it doesn’t stop you blogging about it anyway. But maybe that’s just me getting a bit fed up of publication by press release in archaeology.

A competition for the icon could be good, or possibly a design theme if there is going to be a central site. There’s no need to CC the licence if you’re going to allow use of the logo by bloggers participating in the project.
#9 Dave Munger
August 8, 2007

I tend to agree with Zachary as well — of course, I’m a bit biased since I don’t have a Ph.D. myself. I also think the credentialing would be difficult to enforce since many bloggers blog anonymously.

Todd — can you point us to your school library’s site? Or another similar site? That at least would be a starting point.

Linking to the source sounds like a good idea, except that it’s rather pointless when the article is behind a paywall. That’s the main reason we don’t usually link. At a minimum I think there should be a full citation of the source.
#10 Bora Zivkovic
August 8, 2007

All papers in all PLoS journals (yes, that includes PLoS ONE) are fully peer-reviewed.
#11 coturnix
August 8, 2007

How much “value added” is expected? Just linking to a peer-reviewed paper is not enough – it should be enveloped inside thoughtful commentary as well.
#12 Zachary Tong
August 8, 2007

Dave: Thats a good point. The citation would work great.

Bora: Great news! Sorry, I didn’t mean to insult PLoS with my comment. I just recently found it and wasn’t sure how the process worked

Coturnix: I agree, the post should either fully summarize the paper, or make commentatry and frequent references.
#13 Sandra Kiume
August 8, 2007

Some of these ideas already exist in similar forms; for example there is markup language you can use to get posts on papers (or conferences, or books) picked up by aggregators. From Postgenomic:

“If a post is about one paper in particular then you should mark it as such. To do this, simply add rev=’review’ to the HTML of the link that points to the paper that you are reviewing, like so: Some paper”

There’s a plugin to do it on WordPress.

Postgenomic is one site that could potentially make use of the logos in aggregating.
#14 Hank Roberts
August 8, 2007

>”Report on Peer Reviewed Research”

I agree with the caution to copyright the image, and any copyright from this posting can be assigned only to you for this use.

I suggest a cowboy with a lasso — a roper — like the left image in this set.

http://xroads.virginia.edu/~UG00/3on1/cowboy/images/vaudpost.jpg

Or even see if you can use the recognizable face of Will Rogers, one level of abstraction away.

I just searched quickly for images online; having one drawn up of course is the proper way to go. These are inspiration if anything.

http://www.inquiry.net/…/spin_rope/moving_targets.htm

http://img.tfd.com/dict/CC/62B69-lasso.gif

http://www.texasmonthly.com/ranch/rodeo/images/lasso.right.gif

http://gug.sunsite.dk/docs/Grokking-the-GIMP-v1.0/img245.gif
#15 Greenpa
August 8, 2007

First let me say we’re really on the same “side” here. We’d both like to help readers judge the value of what they’re reading.

My problem, as a scientist, is that I’ve begun to lose faith in the “peer review” process. In many, many cases- it’s a “broken” phenomenon; it does NOT result in “better” science. Ok, more than “begun”. Faith is lost.

We’ve had quite a few discussions on this over at my blog; one just recently; http://littlebloginthebigwoods.blogspot.com/2007/08/pants-on-fire-part-1.html

Here’s the thing; for many readers; a nice “stamp of approval” is FAR too likely to add weight to a paper; making it seem like “truth”; rather than what it is; someone’s research, flaws and all.

Increasingly, I’m in favor of NOT certifying research writings; but rather requiring the readers to (hopefully intelligently) decide for themselves if a researcher is sound, or not.

Boy, if somebody can come up with a “certification” that truly works; I’m all in favor. But at the moment; not only does peer review commonly fail or not bother to reveal flaws- but if you listen inside the scientific community, the problem is getting WORSE, not better. And nobody really knows what to do about it.

Personally – I’d rather have a standard statement of identity for each author- then let the reader decide. “Professor Blip is 29 years old, and received his PhD from Hoodwunk U, Alabama. His research typically focuses on warts in cats. His research is funded primarily by Purina. He is not tenured, is a vegan, and his wife is pregant with their 2nd child.”

You get the idea. Is all that relevant? Oh, yeah.

More cumbersome, to be sure. But perhaps far more useful than “This is CERTIFIED Research”. No, it isn’t.
#16 Kevin Z
August 8, 2007

Alot of what I think has been said. I think its a great idea to standardize peer-reviewed research reporting because:
1) It gets conversation started about the research in a public venue
2) Enables bloggers have a centralized repository for the various research types we are all interested in

This is my own personal 2 cents in how I might approach this.
1) Report on Peer-Reviewed Research is good or maybe an acronym, i.e. PR^3, RPRR. I think a magnifying glass is a good icon because it is simple, cross-disciplinary and universally recognized as a tool that gets to the bottom of something or magnifies detail…
2) Bloggers for Science, Bloggers for Science Reporting or Bloggers for Peer Review as you mentioned are good for an umbrella organization. Some people mentioned a universal website that aggregates trackbacks for the icon. That is a brilliant idea. Make the icon static, bloggers using the icon need to upload the icon using the trackback URL (as opposed to uploading a picture file). The icon gets uploaded to the blog and everytime it gets published, the aggregate feed posts the post on the Bloggers for Science or whatever the name might be website.
3) The icon should symbolize that the blogger has read the entire paper and feels he/she has enough of a background to thoughtfully comment on the paper. To that end, in order to use the icon and be allowed on the aggregate feed on the centralized website. Each blogger should have to register, providing their email address, blog title and URL, a sentence explaining their professional background, maybe some tick boxes for areas of expertise (i.e. Molecular Biology, Psychology, Marine Biology, maybe specific taxa…etc.), and make a declaration of integrity of some sort. This helps you police the appropriate use of the icon, by making it self-policed and being able to kick out anyone who doesn’t abide by a code of conduct (which in reality probably wouldn’t happen).
4) Peer Review should be loosely defined for ease. Anything from an established journal with a degree of peer-review should be allowed. The significant majority of bloggers will be reporting on research from established journals anyways.
5) As I mentioned in #3 self-policing is possible and likely to be easily implemented. It would require someone to oversee registrations, which wouldn’t take much.
6) I think that if you used some kind of system as I’ve described, the icon could licensed only to registrants, to prevent abuse of the icon from non-registrants (and to encourage registrants). All registrants would be allowed on as authors to the centralized aggregation website, and hence be able to copy and use the URL for the icon.
7) If there is a contest, you edih, john and mike should be the judges!

This is just top-of-my-head thinking so some it might not make much sense.

Great idea though! Anything that lends much deserved credibility to science bloggers is in my opinion a good thing.
#17 cfeagans
August 9, 2007

I like the idea. In the past, I’ve commented on peer-reviewed papers that were recently released or (at the time) in publication and the icon would be a nice way to alert the reader that what’s to follow is from a primary rather than a secondary source. The fields I usually comment on are anthropology and (especially) archaeology, so I hope you’ll include these.

Being able to follow the icon back to a citation on a site that aggregates *all* the other commentaries would be a interesting and useful idea, particularly if you can, once you get there, sort by blogger, blog, date, field of science, etc. This might take some sort of php database, but it would be nice if you could login to the site as a blogger, link the url to your post and fill out a citation form.

Okay… I’m probably reaching a bit. But that’s what brainstorming is about, eh?
#18 coturnix
August 9, 2007

Postgenomic already aggregates stuff written by bloggers about peer-reviewed research, but perhaps it could be further refined.
#19 peter murray-rust
August 9, 2007

There is a lot of similar activity in the chemical blogosphere and I have commented on your approach in http://wwmm.ch.cam.ac.uk/blogs/murrayrust/?p=472
(blogging peer-reviewed articles – icons and greasemonkey)

I like the idea of an icon, and also suggest a mechanism (via greasemonkey) where this can appear on table of contents of journals showing which articles have been blogged. It does not require any work non the publishers’ part.
#20 coturnix
August 9, 2007

http://postgenomic.com/wiki/doku.php?id=markup
#21 Zachary Tong
August 9, 2007

Ironically, this post shows up at Postgenomic:

http://postgenomic.com/story.php?cluster_id=12737
#22 Pedro Beltrao
August 11, 2007

I don’t know about the icon. I typically try to write for my own sake of keeping record and to communicate with peers that might have similar interests. I think that it is clear that if I link to the publishers website and not to a some news site that I am talking about primary research. I would just add my vote to the idea of always providing a link to the primary research (or provinding the DOI in metadata). Aggregated blog comments on primary research can serve as another criteria to evaluate the research’s impact.
About the icon and a set of rules of conduct, it would be valuable if people would really stick to it. It might be a bit tricky since different fields have so many different habits and criteria.
#23 Mark Palmer
August 13, 2007

This sounds like the beginnings of a great idea, but beware of the risk of people getting the idea that blog entries reporting in detail on peer-reviewed research are the only ones worth reading and everything else is ‘whimsical fluff’ (to quote one of the comments) – these are two ends of a continuum, not the only two kinds there are.

Looking at your blog, I see that you’re not in danger of making this mistake, but might others make it? We at CABI (a not-for-profit organisation with a publishing division) see our role as drawing people’s attention to information that is likely to be useful or interesting to them. This can include peer-reviewed articles (whether or not we have had the opportunity to read the full text), grey literature, books, etc., or other blogs and news stories. Our blog entries are not intended to be formal reviews, but to discuss information of the kinds mentioned with the help of our extensive bibliographic database of published research. We agree that providing a link to the primary research is important.

So we think that it might be worth having an icon that serves to point out the scholarly nature of the work, but then perhaps it would also help to indicate what sort of material it might be: Peer Reviewed Research, Book, Grey Literature (useful stuff for many areas of research), and there are probably others. So there would be different variants for different types of scholarly work.

Perhaps it would also be worth having a series of defined tags for different types of literature, with peer-reviewed research articles being one such type; these could be associated with the icons to allow people to search for blogs commenting on different types of scholarly work by subject area etc.

Mark Palmer, on behalf of the handpicked… and carefully sorted blogging team at CABI.
#24 Michael Brant
August 13, 2007

I cannot commend you enough! Three cheers for all thoughts of provenanace, attribution and verification on the Web!

I recommend that the icon point to the peer-reviewed research itself. Doh!