A few months back, Bertrand Meyer wrote about the nastiness problem in computer science, questioning whether we as reviewers are “malevolent grumps.” Judging by the user comments on the page, this hit a nerve with readers who were the victims of such grumpiness! Jeannette Wing then followed up on this with some numbers from NSF grant rejections which did indeed indicate that computer scientists are hyper-critical. Much as I enjoy the colourful phrasing, I feel that a field full of malevolent grumps is not something we should simply accept. In fact, even if there are only a few grumps out there, it’s in all our interests to civilize them.
So what can computer scientists do to reduce the nastiness problem when reviewing? Reviewers, authors, programme committee members, conference chairs, and journal editors can all do their bit by simply refusing to tolerate discourtesy. Let’s embrace the “no asshole” rule: We no longer ignore bad behaviour. As reviewers, we can aim to be polite (yet stringent) ourselves but also to point out to co-reviewers if we find their impoliteness unacceptable. As authors, we don’t have to accept a rude review and just lie down to lick our wounds. We can (politely!) raise the issue of rudeness with the program chair or editor so it is less likely to occur in the future. As editors, chairs, and program committee members we can include the issue of courtesy in the reviewing guidelines and be firm about requesting reviewers to moderate their tone if we notice inappropriate remarks.
One of the first steps is to separate intellectual rigour from discourtesy. It is possible to be critical without being rude or dismissive. We can maintain standards in the field without resorting to ill-natured comments. (Believe it or not, it is also possible to ask genuine questions at a conference without seeking to show off one’s own intellectual chops, but that is another matter). The purpose of reviewing, in my view, is to help an author improve their work, not to crush them under the weight of your own cleverness. It’s not the author’s fault that you had a bad day, or that some other reviewer just rejected your own paper.
Of course, there are some pockets of good reviewing practice within the field which we can draw on. I am sure there are many, but I have chosen CHI because I have been writing for it recently. The CHI conference is one of the biggest, well-respected annual human computer interaction conferences. Last year there were 2,000 attendees from 38 countries. This year there were 1,577 paper submissions with a 23% acceptance rate. This was the first year I submitted papers to it, and I have been impressed by the quality of the reviews in terms of their fairness, constructiveness, and level of detail. They contained greater insight and intellectual oomph than the reviews I had from a high-impact journal recently. For one of my CHI submissions, the reviewers did not agree with the paper on some points–it is on a controversial topic–but they still offered suggestions for how to resolve these issues rather than simply rejecting the paper. Was I just lucky in the reviewers I was allocated? Possibly, but the CHI reviewing process has some interesting features built in to maintain review quality*.
In the guidelines for reviewers, courtesy is explicitly mentioned: “please be polite to authors. Even if you rate a paper poorly, you can critique it in a positive voice. As part of polite reviewing practice, you should always state what is good about a paper first, followed by your criticisms. If possible, you should offer suggestions for improvement along with your criticism.”
Ts This is a fairly heavy weight process, but if conference organisers adopted even just one more of the practices from points 1-5, or if journal editors added a courtesy clause to their review instructions the world would be a slightly better place. Make it your New Year's resolution to stop tolerating malevolvent grumps in 2012!
*Thanks to Tom Erickson–the person who runs the CHI author survey–for kindly raising some of these points.
Dear Judy, I embrace your resolution and wish to make an optimistic observation. Bernardo Huberman and I are currently performing meta-reviews and discussing acceptance/rejection decisions in the social networks track of the WWW 2012 conference. At the beginning of the review period we did remind PC members to be polite to authors. We don't know if that reminder had a positive effect, or if our PC is simply made of very nice people. But the fact is that with very rare exceptions, the ~600 reviews are written in a very friendly, supportive, and constructive tone (even though most reviews are negative in such a competitive venue). Let's hope this is the start of the turning point you advocate!
-Fil Menczer, WWW 2012 social networks track co-chair
Displaying 1 comment