ISSTA has always been the premier venue for all things related to software testing and analysis. What’s new this year?
We’re having a number of novelties. We’re bringing back rebuttals, such that the authors have a chance to comment on the reviews. Besides technical papers, we’re explicitly calling for experience reports, because we all love industrial applications and lessons learned. The most important change is that we’re introducing double-blind reviewing, a premiere for a Software Engineering conference with a physical PC meeting.
Why do you introduce double-blind reviewing?
We’re worried about the impact a known name or institution could possibly have on the reviewing. As reviewers, we want to focus on the scientific content only. But even if we consciously try to ignore context information, we may still be unconsciously biased – both in favor of known names, or in their disfavor. Already while reviewers are “bidding” for a paper (that is, express their willingness to review a specific paper), such context information may play a large role. So we want to give the reviewers a chance to stay impartial as long as possible.
Even if a submission is anonymized, wouldn’t reviewers be able to find out who the authors are?
If I get a paper on, say, an empirical study on concolic security testing of Windows mobile devices, it wouldn’t take me much effort to search these particular terms and to find out who has been working in that particular subject in the past. The point is not to completely hide authorship, it is about allowing reviewers to be impartial, and remove any unconscious bias.
One problem with double-blind reviewing is potential conflicts of interest. How do you handle these?
If I do not know the authors, I may inadvertently be assigned a paper to review that has been written by a friend or collaborator of mine. To avoid such conflicts of interests, we ask the authors to declare potential conflicts. We also prepare a list of authors and institutions, sprinkle in a large number of additional past authors, and ask our reviewers to declare conflicts on these. This exercise is done, prior to allocation of papers to reviewers.
Do authors stay anonymous until the decision is made?
There’s a lot of debate on this in the community. Some argue for “full” double-blind reviewing, that is, keeping the authors anonymous until the final decision is made; the goal is to maximize fairness. Others prefer “partial” double-blind reviewing, or revealing authorship before the final decision, claiming that this allows for better assessment of conflicts and increment over previous work. With ISSTA 2016, we are going for this “partial” model, revealing the author’s identity to a reviewer ahead of the Program Committee meeting.
But what if a reviewer changes her assessment as soon as the author is revealed?
As PC chair, I track all changes to reviews. So if a reviewer changes her mind, she will have to justify this. Note that there might be good reasons for that – for instance, by discovering that the submitted work is too close to other published work by the same authors, or that she actually has just submitted a joint proposal with one of the authors. We want to be able to detect such issues as soon as possible, and in particular before the author’s rebuttal.
As an author, if I want to supply data sets as part of my submission, would I have to anonymize these data sets, too?
Upon submission, yes – and you would have to place them on an anonymous server. However, if during reviewing, a reviewer finds that her questions would best be answered by providing an artifact (typically, your data or your program), then you can also provide this artifact as part of the rebuttal phase. Note that the artifacts are evaluated by a separate Artifact Evaluation Committee who give their comments directly to the PC chair.
Some reviewers claim that rebuttals don’t change anything, and are not worth the effort. Why do you have them at ISSTA 2016?
In both my experience as well as the experience of our General Chair, Andreas Zeller, rebuttals frequently do make a difference. The difference may not be significant for the individual reviewer, but it is important for the chairs to detect any flaws in the process as soon as possible; and first and foremost, rebuttals are an essential part of establishing fairness towards authors. In our decision process, we will ensure that rebuttals will be explicitly considered in every decision.
What is the key for a successful submission?
Besides scientific originality and value: Don’t do everything at the last minute! Be sure to have your results and contributions all ready a few days before the deadline, such that you can gather comments and feedback from your friends and co-workers. All reviewers appreciate if you have done your homework.
Any other question you’d like to address?
If there’s something you or any reader would like to know, just send it to me; we’ll be happy to expand this interview as required. Looking forward to your submissions!
Find more interviews behind the link.