Prior and Prejudice

Modern machine learning and computer science conferences are experiencing a surge in the number of submissions that challenges quality peer review as competent reviewers is growing at much slower rate. To curb this trend reduce burden on reviewers, several have started encouraging or even requiring authors to declare previous submission history their papers. Such initiatives been met with skepticism among authors, who raise concern about potential bias reviewers' recommendations induced by information. In work, we investigate whether exhibit caused knowledge under was previously rejected similar venue, focusing population novice constitute large fraction reviewer pool leading conferences. We design conduct randomized controlled trial closely replicating relevant components peer-review pipeline $133$ (master's, junior PhD students, recent graduates top US universities) writing reviews for $19$ The analysis reveals indeed become negatively biased when they receive signal paper being resubmission, giving almost 1 point lower overall score 10-point Likert item (Δ = -0.78, 95% CI [-1.30, -0.24]) than do not such signal. Looking specific criteria scores (originality, quality, clarity significance), observe tend underrate most.