Mike the Mad Biologist

There are two very interesting posts about scientific publishing that raise some very good points. The first post by petermr makes a critical point about the publication process–academia has ceded promotion and tenure decisions to professional editorial boards, not experts and colleagues (italics mine):

The academic system (in which I include public funders) has, by default, given away a significant part of its decision-making to the publishing industry. (I use “industry” to include non-profits such as learned societies, and like all industries there are extremes of good and bad practices). This gifting has been done gradually, over about 2 decades, without any conscious decisions by academia, and without – in the beginning – any conscious strategy from the publishers. The gifts have all been oneway – from academia to industry, which has grown in both wealth and power at the expense of academia. In effect academia has unconsciously stood by, dreaming, during the creation of a 10 billion USD industry, almost all of whose revenues come from academia, frequently to their detriment. Like Morbius in Forbidden Planet we have created our own monsters….

The historical purposes of publication did not include bibliometric evaluation of the publication as a means of assessing scientists or institutions. This is the monster we have allowed to be born and which we must now control. I do not believe it should be part of the formal reasons for publication. And if it retreats to informality we should take formal steps to control it.

Which leads to Joe Pickrell’s excellent discussion of what a non-publication scientific communication mechanism could look like and. He concludes:

Before the internet, peer-reviewed journals and researchers had a happy symbiosis: scientists had no way of getting their best scientific results to the largest audience possible, and journals could perform that service while making a bit of profit. Now, that symbiosis has turned into parasitism: peer-reviewed journals actively prevent the best scientific results from being disseminated, siphoning off time and money that would be better spent doing other things. The funny thing is, somehow we’ve been convinced that this parasite is doing us a favor, and that we can’t survive any other way. It’s not, and we can.

While he argues that we’re essentially one killer-app away from leaving publication in the dust, I think petermr’s point is critical: publication, I would argue, primarily serves as an assessment mechanism–if you’re an active researcher (and able to attend meetings), you probably heard about the work one way or another before it was published. But tenure committees and grant panels need some way of differentiating faculty and proposals. If we move to this brave new world of scientific communication, how will assessment work? I don’t think voting things up or down on the intertoobz will be sufficient, but maybe I’m wrong.

Discuss.

Comments

#1 BobC
July 14, 2011

Solution: peer-reviewed conferences. My field (a subfield of computer science) has two high-quality conferences per year (~20 percent accept rate), held roughly 6 months apart. If you want to see the paper faster than that, most authors post preprints of their accepted papers, either on their own web site, or through an aggregator site like sciweavers. We’ve even managed to convince the promotion committees at department and college level that publications in a highly competitive conference count the same as a journal publication. As a community, we have taken steps to avoid the vagaries of the peer review process… superficial reviews, reviews that miss the point, split reviews where someone loves the paper and someone hates it. After reviews are posted, the author gets a chance to submit a short rebuttal, to point out obvious reviewing errors/biases, and the editor and reviewers then have a window of time for online discussion of disparate opinions and to try to reach consensus. It is really a wonderful system. I don’t think it is a good idea to abandon the peer-review process; just make it more nimble and accountable.
#2 pierre
July 14, 2011

Isn’t it about time we start the ball rolling on where the scientific community is going to eventually end up thanks to the internet: a website that publishes all scientific papers without conventional peer-review, but allows entire scientific-community peer-review. In other words, papers get published, then are critiqued, rather than the current process of critique then publish.
#3 Wow
July 14, 2011

It’s a lot easier to have bad ideas than to show those ideas bad.

Look at Monckton for an example.

http://www.altenergyaction.org/Monckton.html

That’s a lot of work to debunk a lot of crap.

Peer review filters most of the crap out so you don’t have to.
#4 pierre
July 14, 2011

Bad ideas get filtered out, but so do good ideas.
How many good ideas is our sake of convenience worth?
#5 Eric Lund
July 14, 2011

Pierre @2: Physics already has something approximating what you propose: the arXiv. It’s not perfect (there have been claims that the sponsoring requirement that is in place as a minimal kook filter has been abused to shut out some legitimate papers), but it’s a step in the right direction, and already it is routine for published articles to cite papers on the arXiv.
#6 Wow
July 14, 2011

“Bad ideas get filtered out, but so do good ideas.”

Assuming facts not in evidence.

Go look at that link I posted again.
#7 Dan
July 14, 2011

…publication, I would argue, primarily serves as an assessment mechanism–if you’re an active researcher (and able to attend meetings), you probably heard about the work one way or another before it was published.

No.

Maybe, if you have very narrow interests and research is all you do for your job, this approximates the truth. But what if you’re an active researcher interested in learning about a new sub-discipline? What if you’re a new graduate student and learning all kinds of new things? What if you have teaching responsibilities and can only go to one conference a year? Don’t the needs of these folks count too?
#8 pierre
July 14, 2011

/sigh

I’m not trying to prove the system I’m suggesting is the best, but if you look at the trend of scientific publication, it is always headed towards more open publication. After all, there’s hundreds of journals today, but how many were there when people first started sharing scientific knowledge? I’m just saying, why put it off when it’s probably gonna happen in the future anyways?

@Eric: Good point. As a physics grad student, I am well aware of arXiv, and consider it a positive addition to the physics community, so is it wrong for me to suggest implementing a like-minded system across all fields?
#9 Wow
July 14, 2011

> if you look at the trend of scientific publication, it is always headed towards more open publication

If by that you mean “the pulication is free to read” then yes.

But you were promoting the idea of throwing out the peer review. Can I remind you?

“all scientific papers without conventional peer-review”

(note: now will you find the paper scientific if it isn’t reviewed by a scientist, e.g. peer)

“Bad ideas get filtered out, but so do good ideas.”

(presumes that peer review is bad and wrong, cf your later statement “I’m not trying to prove the system I’m suggesting is the best”. Either false or you’re trying to prove the current one is worse, which is still assuming facts not in evidence).

“After all, there’s hundreds of journals today, but how many were there when people first started sharing scientific knowledge?”

There is no factual or causal link between them.

They prove nothing (in fact, they *state* nothing).

There is a problem that so many papers are produced now (yet you assert that good stuff is thrown away: surely the same ideas come through in another paper though) and the actual education is drowned in a sea of crude polemic pretending to be scientific.

But apart from that (which goes against your case), it says nothing.

“I’m just saying, why put it off when it’s probably gonna happen in the future anyways?”

You know, most people reckon that the future is hard to predict, but you seem to know the future already.

No. It’s probably NOT going to happen in the future because as is patently obvious, peer review gets rid of the majority of the crap and throwing out anything that turns up with “New Science Paper” on the top line is going to kill science research in the same way as thousands of patents of “Doing X, but on the INTERNET!” has killed people checking the patents for ideas to use.

The OISM (Oregon Petition) was written with a fraudulent header that made it look like a Nature article. Because it wasn’t peer reviewed, people thought it had more weight than it was, thought it was a science paper rather than PR fluff piece.

And that will be necessary in the future too, unless mankind stops allowing selfishness and lies (which means “no” still).
#10 DPSisler
July 14, 2011

@Eric and Peter: Where does the term “arXiv” originate? do you know? does it have a meaning in physics terminology?
#11 pierre
July 14, 2011

So you don’t think my idea is any good. ok. Just don’t sweat it so much. I know I don’t.
#12 pierre
July 14, 2011

The X in arXiv can be interpreted as the greek letter Chi, so arXiv can be pronounced like Archive, i.e. an archive of physics papers that get submitted to various journals
#13 Wow
July 14, 2011

“So you don’t think my idea is any good.”

No, I don’t think we can do without peer review.

Whether the results are posted on the interweb or promulgated via some other method is a different matter.
#14 focahotel
July 14, 2011

So you don’t think my idea is any good.
I’m not trying to prove the system I’m suggesting is the best, but if you look at the trend of scientific publication, it is always headed towards more open publication. After all, there’s hundreds of journals today, but how many were there when people first started sharing scientific knowledge? I’m just saying, why put it off when it’s probably gonna happen in the future anyways?
thank blog science
focahotel
#15 dzdt
July 14, 2011

@DPSisler #10 : back when the crust of the internet was cooling, what is now arxiv.org was xxx.lanl.gov. At some point they succumbed to pressure to get a url that didn’t look like it belonged to a porn site. Possibly this coincided with a funding shift; I’m not sure about that last. I think the ‘x’ in ‘arxiv’ is partly there to commemorate the old ‘xxx’ prefix.
#16 Ian
July 15, 2011

Wikiesque peer reviewed open database.

Anyone can upload a paper in any supported category. However, it is not publicly viewable at first.

You would start with a selected group of reviewers, probably the sorts who are already doing it for journals or conferences or professional organizations. They review papers that are submitted to the database. If it passes, it goes public, and is a peer-reviewed paper.

Anyone who gets a paper published this way becomes part of the second tier of reviewers. They are able to select candidate papers to be viewable by the public, but that does not count as peer review. To prevent abuse, they would only be allowed to promote a certain number of papers per year…or maybe a certain number per peer-reviewed paper of their own.

Members of the second tier would automatically advance to the first tier after having a certain number of papers pass peer review. There could also be other processes for promotion, like elections, or maybe certain awards would automatically grant it.

Stuff that would pass muster now would get the stamp of approval. Stuff that may be interesting but not rigorous enough (or whatever) would be viewable, but not get official approval. Crackpot bullshit would have a tough time getting through.
#17 kalamworld
July 15, 2011

It’s great that you have published such knowledge. I just hope these are all reliable. It would be better if you also include in your page the references that you are using. Thanks!