Thoughts on the Value of Peer Review in CS

This is a bit of a random first post, but it seemed like a blog was the best place to say something like this, so here it is. Stay tuned for future posts about goings on in my little corner of the CS/database research community.


There are lots of problems with the peer-review system as it is currently implemented for computer science conferences, and especially within the database community, where I do most of my publication. Jeff Naughton recently gave a keynote at ICDE [Link to PDF of his slides] describing the issues in detail, but briefly, the central issue seems to revolve around the fact that review quality is low.  It's not uncommon for reviews to be frustratingly short, non-specific, or demonstrate serious misunderstanding on the part of a reviewer.  Most CS researchers (not just in the database community) have review horror stories.

As an example, here's the entire content of the "weak points" and "detailed comments" sections of a review we recently received, from a reviewer I like to call "Dr. Specificity":

Major weak points: Various tradeoffs and heuristic choices are interesting, but the paper lacks novel conceptual ideas.

Detailed comments:  Interesting well-written paper on an important problem. I enjoyed reading it.

This sort of vague, wishy-washy review makes me think the Dr. is trying out his or her new random review generator, or maybe just cutting and pasting the same review into every paper.  (As an aside, I'd like to say that the vast majority of reviews I receive are not like this, especially in the top conferences, and are generally fair and thoughtful.)

When you only receive 3 reviews for a paper, one careless review like this can easily skew the average score for a paper.  This effect is exacerbated by the lack of face-to-face meetings of program committees in the database community, where such reviews could be discounted.  Instead, there's an  "electronic meeting", where reviewers are allowed to read other reviews, and are encouraged to have discussion with each other.  Unfortunately, some reviewers don't participate, and it can be the case that papers are accepted based simply on whether their average score is above or below some arbitrary threshold.

There are lots of ways one might fix this system (add more reviews, have face-to-face meetings, hold reviewers accountable in some way, get better PC chairs, etc.)  Many of these require reviewers to do more work, so the net result is grumbling and minor tweaks to the system that haven't fixed the problem (which people have been complaining about since I published my first paper 10+ years ago.)

In his keynote, one possible solution Jeff proposed was to eliminate peer review altogether.  Several young researchers also came out in favor of this idea on Twitter (@daniel_abadi and @sudiptdas).  The argument goes something as follows:  now that we're unencumbered by actual printed proceedings, there's no limit on the number of papers that can be accepted, so why not just accept them all?  No more crappy reviews, and no more grief and consternation about having your paper rejected!  Put everything up on a website, and let The Interwebs sort it out.

I believe this is a terrible idea, as I'll try to describe.

First something that most people who've never served on a program committee don't know: most submissions really aren't very good (except, of course, my rejected papers, which are all under-appreciated works of groundbreaking Science!).  For a database conference, I typically rate about 1 in 6 or 7 papers a weak accept or higher, even when I'm trying hard to give authors the benefit of the doubt and not be overly critical (I'm generally a more positive than average reviewer.)  It's not unusual for a PC chair from a conference with 400-500 submissions to struggle to fill a program of 60-70 papers.  Weeding out these lower quality papers is the primary role of the committee, and even broken reviewers like Dr. Specificity usually agree about them.  Winnowing down the papers in this way allows the community to pay attention to the ideas that are more complete, well formed, or have the potential for impact.  

Second, conferences only have so many slots for talks.  Someone has to decide who gets to talk, which appears to require the equivalent of a program committee.

Third, even an imperfect filter like the system we have now causes researchers to work extremely hard to write up their ideas in the clearest, most cogent way possible.  Eliminating the review system will discourage people from working hard on their papers, and inhibit the exchange of ideas in the community.  Additionally, often rejected papers are revised and resubmitted in much better shape than they were before.  This is an important part of the process, which would be lost by accepting everything.

Fourth, having publications in top conferences is one of the important quantifiable ways in which students and faculty are evaluated.  It's an imperfect metric -- I'm certainly not a fan of "counting papers" and most reasonable schools don't do this -- but I firmly believed a good researcher should consistently be able to get his or her best ideas published in a top place.   Accepting everything belittles the efforts of those doing the best work.  (Also, though this perhaps shouldn't be a primary consideration, in academia we've fought very hard to educate tenure committees on the value of conference publications in computer systems;  we'll look pretty silly if we suddenly declare conference publications aren't worth anything!)

Fifth, for academics, publication provides a form of validation, and a way to measure progress and success.  I often find that after a student has a couple of papers under the belt, they are more confident and assertive, which makes them better researchers.   If papers have no value, this important psychological benefit will be lost.

In summary:  we need to improve review quality is CS.  It can be frustrating, even devastating, to be rejected at the hands of inadequate reviews. But that doesn't mean we should discard peer review altogether, as it provides a number of important benefits.