Simply Statistics A statistics blog by Rafa Irizarry, Roger Peng, and Jeff Leek

Cooperation between Referees and Authors Increases Peer Review Accuracy

Jeff Leek and colleagues just published an article in PLoS ONE on the differences between anonymous (closed) and non-anonymous (open) peer review of research articles. They developed a “peer review game” as a model system to track authors’ and reviewers’ behavior over time under open and closed systems.

Under the open system, it was possible for authors to see who was reviewing their work. They found that under the open system authors and reviewers tended to cooperate by reviewing each others’ work. Interestingly, they say

It was not immediately clear that cooperation between referees and authors would increase reviewing accuracy. Intuitively, one might expect that players who cooperate would always accept each others solutions - regardless of whether they were correct. However, we observed that when a submitter and reviewer acted cooperatively, reviewing accuracy actually increased by 11%.