Comment on Zeller’s ‘Peer Review Collusion’
Prof. Andreas Zeller posted about peer review collusion circles in software engineering (SE), and what to do about it.
It’s a fascinating article especially for someone, like me who a) has never been to a physical PC meeting, more’s the pity; b) currently a PC chair for a moderately big conference, and soon to be area chair for ICSE 27.
Peer review, briefly
I’ll start by saying I am gravely worried about peer review and the current scientific model. Quite apart from questions about whether citizen-funded science is worthwhile (it emphatically is), I think AI is exploding the peer review model that has been in place for many decades. AI is writing or at least speeding up whole papers—someone apparently seriously submitted AI generated papers in the dozens to an AI conference—and is being used (against most policies) in peer review. I’ve personally received two reviews that seemed AI generated, e.g., extensive and wordy, overuse of Markdown formatting, bullets everywhere.
If science becomes AI talking to AI, we might as well close up shop as human researchers. There is no reason for people to pay us to simply run AI tools. Of course, I also think the idea of AI taking over entirely in the research process is farcical and usually stated by people who have never done the messy side of science.
Back to the article
Back to Dr Zeller’s article. I agree with most of what he suggests, but I will just point out that the big problem underpinning all of this is the scalability of the humans with integrity who run the whole thing (none of whom, it should be repeated, make much from their efforts).
For example, paper assignments. In Dr Zeller’s model, we ought to stop allowing so many bids, or use of systems like TPMS that are gameable. But what is the alternative? If we receive 1000 papers, who is going to assign that to the 200+ peer reviewers to do the work? How to scale this, because manual assignment is impossible at that scale? And if we have area chairs and editors, how do we know they can be trusted? In the old days of in person PC meetings, we could build trust face to face, and the number of papers was manageable by 1 or 2 moderators. Nowadays the top conferences cannot come close to handling this manually.
Verifying conflicts or checking for suspicious patterns: again, great idea, but who will be asked to dedicate their time to do that work? One would think that is precisely what journal publishers and conference sponsors should be doing, but they are among the worst at trying to cut corners and AI-all-the-things. I can tell you emphatically that being a PC chair is a lot of work, and a lot of dealing with edge cases: papers with AI writing, late reviewers, personal disputes, etc. And while I see being a PC chair or editor as part of my job, and a privilege, one has to wonder exactly how many hours should be dedicated to helping other people publish papers. Certainly a number of community members do far less than their share as compared to papers published.
A modest proposal
My modest proposal1 is to dispense entirely with peer review, or perhaps have a first hurdle to get into a peer review phase. If a paper gets three people who agree it should be rejected, or weakly rejected, that paper should not have been sent out for review in the first place. One of the most annoying things about AI generated slop papers is the sheer amount of other people’s time they consume.
The obvious response to being asked to work for free on someone’s paper that has minimal effort behind it is to cut corners or simply stop accepting review requests. This is why I also think Dr Zeller’s last point, about incentives, should be more prominent. Collusion and bad behavior exists because getting a paper accepted at ICSE or ICLR or ICML can be extremely financially lucrative, in terms of improved career prospects and state-funded reward structures. If we removed these incentives, or created better aligned ones, that would mitigate against a lot of the bad behavior. For example, two of my colleagues have been ICSE PC chairs, I’ve served on the PC. From what I can tell my employer gives us no credit for these activities in salary review. There are hidden rewards like good vibes, of course, but also recognition(?) and service awards. However, the real focus is on papers published, grants obtained, students graduated, and courses taught. If that is the incentive model (and I doubt it is unique) how can the community move forward?