Guidance on Peer Review for a Journal#

The text is inspired by this blog post by Rene Bekkers, released under the CC BY 4.0 licence. There is no consensus among journals and editors on a set of minimum criteria that the peer review process should cover. The extensive set of criteria provided here are general considerations from the perspectives of openness and reproducibility. The bar is set pretty high: a review checking each of the criteria will take you at least three hours, but more likely between five and eight hours.

Before you accept an invitation to peer review#

The following steps will help you make a decision whether to review the paper or not. They need to be taken fast. It is a good practice to let the editor know immediately if you are unable to review a paper, so the editor can start to look for someone else to review the paper. Suggesting other possible reviewers is very helpful. Your non-response means a delay for the authors and the editor.

  1. Check what kind of journal this is. Have you heard of it before? Does it have a reputation in your discipline? Are you planning to submit your own work there? Is this journal an Open Access journal? Beware of predatory journals.

  2. Check if this paper is of interest to review and if you are the right person to review the paper. Reviewers do not need to be experts in all respects. The editor may ask you to focus on a specific aspect of the paper. Alternatively, you may qualify your review with an indication of where you focused your review, if you do not feel you have the necessary expertise to evaluate all aspects of the paper. It is a good practice to describe your disciplinary background and expertise in relation to the manuscript you are invited to review. If you do not have useful expertise to improve the paper, politely decline. If you do not have sufficient time to review the paper, politely decline. Recommending other potential reviewers when declining is helpful, but not necessary - a timely accept/decline response to the review request is more important.

  3. Check whether you know the author(s). If you do, you might describe how you know the author(s) and ask the editor whether it is appropriate for you to review the paper. If you have a conflict of interest, notify the editor and politely decline.

  4. Check whether the authors have made the data and code available - see also Open Access. You can set a requirement for the author to make the data available for peer review or openly available before you accept an invitation to review their work. It needs to be taken into account that some data would be illegal or dangerous to make available, such as datasets that contain identifying information that cannot be removed. In most cases, the authors can provide at least partial access to the data by excluding variables that contain personal information. If the paper does not provide a link to the data and the analysis script, you can ask the editor to ask the authors to provide the data and the code.

You might get requests to review a paper that you have reviewed before, for a conference or another journal. In these cases, you can let the editor know and ask the editor whether she would like to see the previous review. For the editor it will be useful to know whether the current manuscript is the same as the version, or includes revisions.

Writing a review#

How to organize the review work#

There is no established procedure, you need to find what works for you. The journal you are reviewing for may also provide guidance on how to write a review. If you are reviewing for the first time, this workflow might be useful.

  1. It might be a good idea to read a paper twice. First, go over the paper superficially and quickly. This gets you a sense of where the authors are going.

  2. After a quick glance at the paper, you can determine whether you think the paper has the potential to be published, perhaps after revisions. If you think the paper is beyond repair, explain this to the editor.

  3. If the paper is good enough to be revised and resubmitted, you can start providing more detailed comments.

  4. The second time you go over the paper, do a very close reading. Some of the information may be in the supplementary information provided with the paper - read them carefully as well.

The tone of the review There is a ‘golden rule’ for people writing a review: treat others as you would like to be treated. Write the review report that you would have liked to receive if you had been the author. The following principles might help you:

  1. Be honest but constructive. You are not at war. There is no need to burn a paper to the ground.

  2. Avoid addressing the authors personally. Say: “the paper could benefit from…” instead of “the authors need”.

  3. Stay close to the facts. Do not speculate about reasons why the authors have made certain choices beyond the arguments stated in the paper.

  4. Take a developmental approach. Any paper will contain flaws and imperfections. Your job is to improve science by identifying problems and suggesting ways to repair them. Think with the authors about ways they can improve the paper in such a way that it benefits collective scholarship.

  5. Try to see beyond bad writing style and mistakes in spelling. Also be mindful of disciplinary and cultural differences between the authors and yourself.

The substance of the advice

  1. It is a good principle to begin the review report by describing the way you reviewed the paper. If you searched for literature, checked the data and verified the results, ran additional analyses, state this. It will allow the editor to adjudicate the review.

  2. Then give a brief overview of the paper. Include a two- or three-sentence summary with your interpretation of the hypothesis, main results and conclusions. This summary provides the author an indication of the clarity of their paper.

  3. If the invitation asks you to provide a general recommendation, state it early in the review. The recommendation is the first thing the editor wants to know. Typically, you are invited to recommend ‘reject’, ‘revise & resubmit’ – with major or minor revisions, or ‘accept’.

  • Major revisions are needed when the conclusions are jeopardized by concerns or the usefullness of the data is limited. Examples are problems with the applied methods, analyses, or the reasoning. Reviewers should indicate the seriousness of the major concern and how it could be addressed. If the concern cannot be addressed it may be better to ‘reject’ the work. Only suggest additional analyses or testing only when the conclusions of the study are otherwise undermined - otherwise they can be addressed in a future paper.

  • Minor revisions are needed to improve the clarity of the paper. Minor revisions can include additional citations, addition of details regarding the method, change in title or abstract to reflect the paper content, whether figures are clear, necessairy and in correspondence with the text and any presentation issues (spelling errors, wording).

  1. Use ‘reject’ only if absolutely necessary. Most journal editors make use of desk-rejection if the paper seems a poor fit; if the editor sent the paper out for review, she probably thinks it has the potential to be published. Start from the assumption that the authors have invested a great deal of time in the paper and that they want to improve it.

  2. List the strengths and the weaknesses of the paper. To ease the message you can use the sandwich principle: start with the strengths, then discuss the weaknesses, and conclude with an encouragement.

  3. Try to give actionable advice. For the weaknesses in the paper, suggest ways to repair them. Constructive criticism does not have to be in the form of a specific solution. It can be much more useful to be specific about the problem and to let the authors identify a solution, than it is to provide the fix for them. It is convenient for both the editor and the authors to number your suggestions. Distinguish major issues, such as not discussing alternative explanations, from minor issues such as missing references and typos. There could be points that the authors are underselling. In that case, you could identify them as strengths that the authors can emphasize more strongly. It is handy to refer to issues with direct quotes and page numbers or line numbers.

  4. Traditionally, the review process is designed to be anonymous. However, some reviewers sign their reviews. A non-anonymous review can be seen as an accountability device: by exposing who they are to the authors of the paper, the reviewers set higher standards for themselves. This needs to be done carefully: some editors do not appreciate signed reviews. There are voices that express concerns for a graduate student or an early career researcher facing retribution for being critical of a more senior researcher. There is also a process called open peer review; in this case, the review is open for everyone viewing the paper. The peer review might blend these features; for example, one might be able to see the review text but not the name of the reviewer who did it, or the name of the reviewer but not the text itself. The design of the peer review process and its transparency varies from journal to journal. Generally there is a way to communicate just with the editor, for example, via confidential remarks. This communication is not provided to the authors, so you can be candid here about your limitations and concerns regarding the submitted paper.

Second round of reviewing process If the authors have revised their paper, check if your suggestions have been implemented or whether sufficient arguments have been provided to not do so. It is a good principle to avoid making new demands for the second round that you have not made before.