Time for computer science to grow up?

Posted by John Dupuis on September 21, 2009
(13)
More »

That’s the question asked by Lance Fortnow in a recent Communications of the ACM Viewpoint article (free fulltext).

Fortnow’s article continues a discussion about scholarly communication patterns in computer science that’s been going on for a while in the “pages” of the CACM. I’ve blogged about it a couple of times here and here.

Fortnow’s main idea is that CS needs to get past the youthful stage of using conferences as the main vehicle for disseminating new ideas and move to a journal-based model, like most of the rest of scientific disciplines. In the end, it’s all about peer review:

Unlike every other academic field, computer science uses conferences rather than journals as the main publication venue. While this made sense for a young discipline, our field has matured and the conference model has fractured the discipline and skewered it toward short-term, deadline-driven research. Computer science should refocus the conference system on its primary purpose of bringing researchers together. We should use archive sites as the main method of quick paper dissemination and the journal system as the vehicle for advancing researchers’ reputations.

*snip*

But even worse, the focus on using conferences to rate papers has led to a great growth in the number of meetings. Most researchers don’t have the time and/or money to travel to conferences where they do not have a paper. This greatly affects the other roles, as conferences no longer bring the community together and thus we are only disseminating, networking, and discussing with a tiny subset of the community. Other academic fields leave rating papers and researchers to academic journals, where one can have a more lengthy and detailed reviews of submissions. This leaves conferences to act as a broad forum and bring their communities together.

*snip*

Our conference system forces researchers to focus too heavily on quick, technical, and safe papers instead of considering broader and newer ideas. Meanwhile, we have devoted much of our time and money to conferences where we can present our research that we can rarely attend conferences and workshops to work and socialize with our colleagues.

Computer science has grown to become a mature field where no major university can survive without a strong CS department. It is time for computer science to grow up and publish in a way that represents the major discipline it has become.

Check out the comments on the original post as well as at Fortnow’s blog. He’s also collected various blog reactions here.

I find it interesting that the CS community, surely among the most “online” of all the scholarly disciplines, is reaffirming the value of strong peer review. More interesting, perhaps, is the way Fortnow distinguishes between speedy initial publications in archives combined with slower, more deliberate peer review before something later appears in a journal. Which is sort of like what’s happening in various physics communities with arxiv and later journal publication.

However, I’m not convinced that you need a “journal” to have peer review, that it can’t be embedded as another layer in whatever kind of fast-track publication system ends up being used.

If the CS community decides to go in this direction, it would be an amazing opportunity to rebuild their corner scholarly publishing from the ground up, to decide on what they truly value. Some ideas:

  • fast and efficient to dissemination of findings

  • combining data, code, narrative and other pieces into one scholarly object
  • a layering of traditional peer review and open commenting and discussion over the collection of scholarly objects
  • versioning of those objects
  • working with the various stakeholders to figure out how it’s all going to be funded and built: departments, societies, libraries, publishers, conference organizers, journal editorial boards.

They could, in essence, lead the way into the post-journal scholarly communications landscape.

(13)
More »

Comments

  1. #1 Daniel Lemire
    September 21, 2009

    Oh. I’m not sure it has anything to do with strong peer review. Or, at least, you’d need to make this statement precise.

    In Computer Science, it is typically much more difficult to get a paper accepted in a top conference than a top journal. The review process is just more brutal within the conference systems than with journals.

    (Editorial: Which is all the more ironic to see Librarians typically not indexing conferences! That is why there is a strong disconnection between Computer Science researchers and librarians. If you only index Computer Science journals, then you index about 20% of what is going on… 80% of the new results will never make it to a journal.)

    The net negative effect is the whole purpose of conferences becomes to quickly publish the top results in a given field. It is no longer a community meeting, it has become primarily a publication outlet.

    There are many problems with conferences as publication outlets. Lance has his own ideas on the topic. As for myself, I just dislike the harsh length limitations (inherited from the days when proceedings were printed on paper), fixed deadlines, artificially low acceptance rates (there are only so many rooms for talks at meetings!) and, most of all, expensive and painful trips (I hate to sit for 6 hours in an overcrowded plane).

    The net conclusion is that if you want to meet people from your research community, why not just do that… instead of insisting that it must also be a major publication outlet and a way to meet people. The two don’t mix very well.

  2. #2 John Dupuis
    September 21, 2009

    Hi Daniel, thanks for the input. In retrospect, I guess strong peer review is probably not the best term. It’s good to see that so many want to change the way conferences work in CS — the challenge is to figure out how to get from point A to point B!

    As for librarians & conferences, I think the issue is a long standing one — back in the day when the indexing services started, it was probably nearly impossible to get a hold of all those conferences! At the same time, I think if you look into the issue a little closer, you’ll see that INSPEC does do a respectable job of indexing CS and other conferences, especially the main ones (IEEE, ACM, LNCS). Last time I checked, Web of Science did index LNCS but not the others — and of course this is the most annoying one since for CS people since it misses so much of the citation information. The math people have the same issue, but with monographs which are somewhat more important to them that to the other science disciplines. Scopus, I’m not sure off the top of my head how much conference coverage they have. Anyone out there know?

  3. #3 Jane
    September 21, 2009

    The proliferation of conferences in CS has another negative effect: as the number of conferences grows, the scope of each conference gets narrower. For those of us (hi!) whose research straddles several different subfields, it becomes more difficult to find a conference that will accept your work. I will sometimes submit the paper to 4 or 5 conferences before the paper is accepted! It’s very frustrating.

    I think journals will become a more viable means of publishing if and when the review cycle times shorten. One of the major advantages of conferences is the shorter time to publication, in general. Journal review cycles—even for the journals that do all of the review, submission, etc online—can sometimes still take years. That’s totally unacceptable in our field!

  4. #4 Ian
    September 23, 2009

    According to their info site (info.scopus.com), Scopus indexes 360 book series and has > 3.6×10^6 conference papers. I usually recommend my Computer Science faculty give Scopus a try – I think it is a better tool for them than WoS.

    I don’t agree with the notion that peer review = journal. To my mind there are three things that really matter to scholarly publishing:

    1. the number of eyeballs that look at the research (can be approximated by the number of times the article is downloaded)

    2. the number of times something has been cited (a rough measure of the value of the research to the community)

    3. some kind of moderation system (this needn’t be traditional peer review – being able to assign a score out of 5 + comments would be probably be sufficient)

    Do any of those things require traditional journals?

  5. #5 John Dupuis
    September 23, 2009

    Jane, thanks for dropping by!

    For me, the really interesting question in all this is how to get from point A to point B, in other words how to actually change the publishing culture. Does a new “journal” culture need to be built before people start to see conferences differently or is it the other way around?

  6. #6 John Dupuis
    September 23, 2009

    Thanks, Ian, for checking on Scopus. It’s good to know. As for your comment on scholarly publishing, I agree completely.

  7. #7 Jane
    September 23, 2009

    John, that’s a really good question. I think it can change in the short term *if* journals, or at least some journals, become places for faster dissemination of research. One of the complaints I often hear from other computer scientists about journal pubs is that the publishing timeline is so slow that the research is obsolete by the time it’s published. So if we could have some journals that operated on conference-like timelines, I think the culture could change. And I think this would be a good time for that change—registration fees for conferences are insanely expensive, especially for the top conferences (at least in my subfield(s)), which means that conference travel ends up being very expensive. If I could get the same results (a pub + another line on the CV in a relatively short amount of time) without the hassle and expense of traveling, that would be ideal!

  8. #8 John Dupuis
    September 24, 2009

    Jane, to play devil’s advocate, what’s the incentive for those journals to change? I’m sure all the editorial board members of those journals are thinking, “hey, we don’t need to do that. That’s what conferences are for.”

    I think at least some of the pressure to change would have to come from new (or at least newer) journals. Perhaps something like PLoS ONE that would focus on computer science — something that would get valid research out there really fast, with a minimum of peer review — basically a journal with conference-like timelines. That could take the pressure off conferences and their organizers to perform that function.

    Another interesting question arises from what seems to me to be a feeling that CS needs fewer, but better, conferences. How to wind down those other ones that aren’t really needed. Maybe if people stopped needing them so much for their CVs, the lower impact ones would gradually just wither away.

  9. #9 Daniel Lemire
    September 24, 2009

    As for Scopus, let us see… My most highly cited paper is

    Slope one predictors for online rating-based collaborative filtering
    (cited nearly 60 times according to Google Scholar)

    This paper is not even listed in Scopus, even though it appeared in a prestigious conference. For various historical reasons, it is a fairly important paper in general (and I am not just saying this because I wrote it).

    My second most highly cited paper is

    Racofi: A rule-applying collaborative filtering system

    While Google Scholar says it has been cited 49 times, it does not exist in Scopus.

    My third most cited paper is

    Tag-cloud drawing: Algorithms for cloud visualization

    (cited 43 times according to Google Scholar)…

    And, yes, you guessed right, it is not listed in Scopus.

    I could continue, but you are getting the idea.

    And it is not just that I care about citations… it is that, frankly, if anyone wants to know what has been written on a topic, then Scopus is really a bad choice. It fails very badly.

  10. #10 Jane
    September 24, 2009

    I think you’re probably right, John, about the changes happening with the newer journals. “Journal creep” is starting to become an issue too, though—on one mailing list I’m on, there’s at least one announcement for a new journal every week! And I also wonder if the field itself is ready for the change—I think we’re approaching the tipping point, but we’re not quite there yet.

  11. #11 John Dupuis
    September 25, 2009

    Jane, thanks for the comment. I am looking forward to seeing what happens in the coming years.

    One wild card is energy prices. If oil prices jump up again like they did a year or two ago, that will make travel significantly more expensive, which may drive changes in the conference landscape. If people end up going to half the conferences that they currently do…

    And I don’t think that it’s an “if” for travel costs, it’s a “when.”

  12. #12 John Dupuis
    September 25, 2009

    Daniel, thanks for the info. Although I’ve never used Scopus much with CS people, I have to admit I wasn’t expecting it to be that bad.

    On the other hand, there have been studies that show that to get a complete citation picture, you need to use Scopus, Web of Science and Google Scholar, as they are all non-overlapping to some extent. As you point out, that is probably somewhat less the case for CS than other disciplines.

  13. #13 Ian
    September 25, 2009

    Daniel, not to pick nits, but I found 22 articles that cite your first example in Scopus (in the more tab), even though as you correctly point out the paper itself had not been indexed by Scopus (though in the web tab there was a link to the paper). I also tried a cited ref search in WoS and it did much worse (6).

    I think John’s statement about getting a complete picture using all three holds regardless of discipline.

eXTReMe Tracker