Are scientific editors reliable gatekeepers of the publication process?

Editors are gatekeepers of the scientific publication process and many manuscripts never make it past them. For the journal Biological Conservation, editors reject about half of the manuscripts assigned to them (known as desk reject) before sending the remaining manuscripts out for review. Given the importance of the editor's role in the publication process we ask: how consistent are editors' decisions in Biological Conservation? To answer this question, in 2018 we asked 10 editors of the journal to evaluate 40 manuscripts that had been previously evaluated in 2017 as part of the regular review process. The editors did not know which manuscripts had been sent out for review (20 total) and which had been desk rejected (20 total) in 2017. We found that in 2018, agreement among editors was reasonable in their decisions, even when papers that editors were asked to evaluate fell outside their area of expertise, and decisions did not differ with editorial experience. Of the 40 manuscripts, at least seven editors agreed on the decisions for 73% (29) of them. On average, for the 20 manuscripts sent out for review in 2017, 70% of editors in 2018 agreed with the decisions to review them. For the 20 manuscripts that were desk rejected in 2017, on average 67% of editors in 2018 agreed with the decisions to desk reject them. While variation in editors' decisions on manuscripts exists, most decisions are repeatable and reliable. The journal can potentially improve the reliability of the review process by encouraging discussion among editors and by providing more detailed guidelines to editors. We urge other journals to carry out similar evaluations of this important desk rejection process.