Semantic Werks

Clearly I’m somewhat inexperienced, having only 2 papers published to date, and never having served on review committees. However, I find the issues raised by Moshe Vardie in the SIGMOD journal very cogent. He questions the model of publishing currently in use throughout computer science research.

The typical argument is that since this is a fast-changing field (a conjecture hitherto unproven by evidence — is it faster-changing than genetics? Than theoretical physics? Than sociology?), we need to publish at annual conferences, because journal turnaround times are much too slow to keep up. Firstly, if it takes 2 years to publish in a journal, I’d argue that’s a different problem.

Vardie’s central point is that conferences aren’t doing a good job in evaluating important research. At best, he says, the reviews serve to correctly identify 1/3 to 1/4 of the interesting work. The rest is of dubious quality (relative to that rejected). Furthermore, most reviews are hardly peer-review quality, and there is little opportunity to turn things around if a paper has flaws (I expect the model used is, What a lot of submissions – if it has a single flaw, reject it.

Here’s my possible amended model: agile conference development. The call is published as normal, six months or so beforehand. The call is very tightly focused on the conference theme. For example, ICSE is not just about “software engineering”, but software engineering in a particular context, e.g. testing, formal methods, industrial experience reports, etc. Authors submit abstracts 4 months ahead of the actual conference date. The editor and two associate editors  filter aggressively any submissions out of scope. 2 months later, full submissions are due. The editor and associates check the papers for obvious problems like spelling, illegible figures, etc. Then they arrange a publishing day, during which all the program committee get together and collaboratively decide on each paper.With sites like Google Documents and others, this is becoming feasible.

Papers get sorted after a quick 10 minute scan into piles based on suitability for the conference. Those that are ‘maybes’ then get further detailed review by one specialist.

The idea is to quickly filter the papers which won’t make it (e.g., insufficient empirical support). Then, the others remaining get true peer review, fact checking, etc. There may even be room for correspondence with the authors seeking clarification.