I have obtained a document that describes the secret, inner workings of the on line publication PLoS ONE. The document also exposes future plans for the enterprise.

The link is below the fold.

ResearchBlogging.orgThe link for the PDF of the document is here. Don’t tell anyone where you got it.

From the Abstract, which I have decoded for you:

PLoS ONE, a peer-reviewed Open Access academic journal published by the Public Library of Science, was founded in 2006 with the intent of reevaluating many of the aspects of the scholarly journal. As a result, PLoS ONE has taken elements of the traditional publishing model for scholarly journals and separated them into those functions that are most effectively carried out before publication (for example, peer review in order to evaluate whether the article deserves to join the scien- tific literature) and those that can most effectively be carried out after publication (for example, how impactful the article was once it joined the literature). With this basic premise in place, and using the online tools that are now available, the journal has grown to the extent that in 2009 it will become one of the largest journals in the world (by publication volume). This article overviews the development of the journal to date – how it differs from most other journals and how it engages with its core audiences. In March 2009, the journal (along with other PLoS titles) began a program to place ‘article-level metrics’ on each publication, and this article out- lines how this has been achieved, as well as plans for further development. In conclusion, this article looks forward to the future developments of this transformational journal.

One of the most important parts of this paper is probably the discussion of PLoS’s secret weapon, which is highly classified, of article level metrics. The essence of article level metrics is devious in its simplicity.

Heretofore, articles have been measured in various ways with respect to their qualities, but one of the most important measures has been the journal of publication itself. This began decades ago when some tenuring bodies made lists of articles that were “high” vs. “medium” vs “low” (or some other metric, sometimes using a rather fine tuned numeric point system) with respect to importance in someone getting tenure or promotion. This eventually evolved into the official “impact factor” which you can read about here.

In the old days, it was more the article itself that mattered, but that ‘metric’ has disappeared for a number of very bad reasons. (And maybe some good ones, but I know of none.) One of those reasons is that people are so often highly specialized within their field that other departmental members who are supposed to decide on tenure cannot possibly evaluate the person’s work. In these cases, having your work published in a blue ribbon journal produces the necessary “wow” factor that even your idiot colleagues can understand how important you are.

This system, however, is highly flawed because there is potentially huge variation in the actual quality of articles across all publications, and it is not at all uncommon for an article that has very little ‘wow’ when it is first published not because it is mediocre but because reviewers don’t understand the importance of it yet. This happens far more often than one might think.

PLoS’s secret weapon is to develop an article based assessment of ‘impact,’ as it were.

PLoS believes that the journal in which an article is published should not be the primary mechanism to determine whether that article will have any worth. Instead, PLoS feels that each article should be evaluated based solely on its own contribution to the literature

This is roughly parallel to the widespread Internet-brooded (if not born) idea that “argument from authority” is wrong.

The other developing contribution from PLoS ONE that is of interest is the post publication messing around people can do with the papers, such as various Web 2.0 thingies which are all top secret projects being discussed at the highest levels in the boardrooms of major publishing companies. Just don’t get found out participating in any of these post publication activities.

The publication process itself is also different than traditional publicatoi, but secret details of this process have already leaked out and have probably been discussed in the open more than any other feature of PLoS generally or PLoS ONE in particular.

Now that you have access to this closely guarded document, I do hope you spend a little time reading it. But don’t tell anyone where you got it.

Please visit this post by Bora to tell him what you think of PLoS ONE and beat up on the trolls.

Binfield, Peter (2009). PLoS ONE: Background, future development, and article-level metrics PLoS ONE

Comments

  1. #1 Jeremy
    June 27, 2009

    It has always been possible to use Science Citation Index to do article-level impact analysis. Just that very few people can be bothered.

  2. #2 sczhang
    June 27, 2009

    It would be better for a journal like PLOS ONE have both SCI IF and article-level metrics.

  3. #3 Greg Laden
    June 27, 2009

    PLoS ONE will never have a high IF value because of the way it operates. The other PLoS journals are totally different in this regard. PLoS ONE is explicitly set up in a way that makes sense for a certain (very worthy and important) mission, but as a result of the way it intentionally works, while there will be great advantages, it simply can’t have a high IF.

    Yet, PLoS ONE will be an important journal and it has already published numerous very important papers.

  4. #4 Lois Wingerson
    June 29, 2009

    Many thanks for pointing out this very important information.

    The Advisory Board of medical specialists at SearchMedica, the custom search engine for health care professionals, have been considering whether to retain PLoS One among the journals on our list of recommended authoritative sources, because searches occasionally retrieve articles from PLos One that are of no medical relevance. For instance, a recent search on the psychiatric term “flight of ideas” retrieved something about a flying dinosaur.

    Understanding the goals and advantages of this journal help to support the Advisory Board’s general opinion that we should retain the journal despite these rogue results. In fact, it is now obvious that it would be a serious mistake to exclude it.

    Yours,

    Lois Wingerson
    Content Manager
    SearchMedica

  5. @Lois – good news. We cover the whole of science, which is why you found one of our dinosaur papers. However, in reality the bulk of our articles are currently in medical/biomedical areas. We have individual RSS feeds for each of 50 different subject areas, so you could look at only specific feeds if that is better for you.

    @Jeremy – true. But Article Level Metrics is about a lot more than citation figures. Plus, the Web of Science (aka SCI) is not very comprehensive (it only covers approx 9,000 out of approx 25,000 journals) and so misses large numbers of citations from its database.

  6. #6 MINLI
    October 18, 2009

    Hi people. I just created a poll for the estimate of the first PloS ONE impact factor.

    please give your guess at
    http://andrew.cmu.edu/user/minlix/poll_plosone.html

    thanks :)

Current ye@r *