Running a “Critical Research Review” at #RE15
Today we conducted our first attempt at “Critical Research Reviews” (CRR) at our workshop on empirical requirements engineering (EmpiRE) at the 2015 Requirements Engineering Conference.
CRR was introduced to me by Mark Guzdial’s post on the same exercise at ICER last year, which was run by Colleen Lewis. The idea (as I understand it) is to have researchers present work in progress, ideally at the research design stage. The purpose of the workshop is to “leverage smart people for an hour” in improving and stress-testing your research idea and methodology.
The cool part about doing this at EmpiRE is that our proposers got to leverage some of the leading empirical researchers in the RE community. These are the people likely reviewing your full paper, so it makes sense to get their critique up front.
We had three accepted “research proposal” papers as a special category for the workshop call. In the afternoon, (2pm-5.30pm) we had the three presenters do a 15 minute plenary presentation to get everyone in the workshop (25 or so) aware of the work. I restricted any questions so this was almost entirely over in 45 minutes. After a coffee break, I introduced the CRR concept and some ground rules, as well as a list of potential questions to consider. Then, for the next 45 minutes or so, the participants were invited to join the presenter that interested them and have a (polite) discussion about the proposed research.
Finally, I had asked each group to bring some wide-ranging thoughts back for the entire group for the last 30 minutes. My intent here was not to go into specifics on the proposals; rather, to get some other lessons that might be useful for the people who were not part of that particular group. This worked pretty well; it did tend to go into more detail than perhaps warranted, but it did stimulate some interesting discussion.
From what I heard, people quite enjoyed this approach to research evaluation. It’s much more fun trying to poke holes in research approaches when the author on the other end can rebut your arguments! Look for another edition next year.
You can find my slides introducing the idea here, and our proceedings, with the presenters’ research proposals, will be posted whenever IEEE gets around to it.
Lessons learned
The room was terrible: large central conference table. One group retreated to the coffee room which had large circular tables.
No one used the flip charts: I think the presenters were writing their own notes down on their laptops anyway.
We mostly had established researchers presenting. In the future we are considering perhaps restricting this to early career or Phd students, who likely need the assistance more. But I think the more senior researchers still benefited. The primary difference, I think, will be that the senior researchers will have considered more of the potential threats.
I was main facilitator: having one group a 2 minute walk away made this harder. No group really needed help, but I can certainly see possibilities where that would be an issue. For instance, if you get too many people going to one presenter, or one person dominating the discussion, or too much negativity (the usual group dynamics, in other words).