Today, I will discuss the task of reviewing papers in academia. I will discuss why it is important to review papers, and then give tips about how to review papers and also talk about what a reviewer should do and should not do. This topic is important for young researchers who are invited to review research papers for conferences and journals, and want to do this task well.
Why reviewing paper is important?
From the perspective of authors and publishers, the review process in academia is important as it ensures that papers meet some quality standard before they are published. Moreover, the review process is used to filter out papers that do not meet the requirements of the journal in terms of quality or other criteria (e.g. a paper should not contain plagiarism). The review process is also important as reviewers can provide constructive comments to help authors improve their paper, even when the paper is rejected.
From the perspective of reviewers, reviewing papers is also important. There are a few reasons. First, it means that the reviewer is recognized as having enough expertise to review papers. For example, if you are invited to review papers for some famous journals, you can mention it in your CV and on your website, as it shows that some famous journals are trusting you for doing reviews. This kind of experience is valuable in academia, but not so much for the industry. Second, by doing reviews, a reviewer can get a glimpse of the latest research that is unpublished in the field. Of course, a reviewer should always be profesionnal and not take advantage of this information. In fact, a reviewer should only use information about unpublished papers for the purpose of reviewing, and should not share information with other people about reviews. But it can still give a broad overview of what other people are working on to the reviewer, for example to know what topics are popular in general. In that sense, it can be interesting. Third, by reviewing papers, you can also know more about how other people will review your papers. In other words, it helps you to think like a reviewer, and write better papers because you will anticipate problems that the reviewers could raise in your paper. Fourth, by reviewing paper, the reviewer can feel that he is helping the research community.
Is there a drawback to review papers?
Yes, of course, there is. The drawback is that it takes quite a lot of time, and in life, we have a limited amount of time. Personally, I receive a lot of requests to do reviews from journals. At first, I was accepting all of them when I was a Ph.D. student or early in my career. But now, I decline many of them because otherwise it will take too much of my time. So I usually only review the papers that are related to my field and for the top journals and conferences. If I receive some offer to review papers that are unrelated to what I am doing or from journal or conferences that I never heard of, I will decline the invitation. There is of course some exceptions. For example, if a friend ask me to review for his journal, I will usually say yes even if the paper is not too much related to what I am doing. Actually, you can consider the job of a reviewer as free work as the reviewers usually never get paid. In fact, in academia, the publisher typically earns money by selling the papers that are written and reviewed for free by researchers (for typical non open-access and free journals), which is a strange model but it is how it works.
How to review a paper?
Now that I have explained why reviewing papers is important. I will present some criteria that a reviewer should use in general to evaluate a paper. Of course, depending on the research field, some of these criteria may be more or less relevant.
- Is the paper easy to understand and well-written?
- Does the paper follows the format required by the conference or journal?
- Does the title of the paper is appropriate and describe the content of the paper?
- Does the abstract accurately describes the content of the paper? In particular, it should explain why the problem addressed in the paper is important, describe the contributions and briefly talk about results.
- Does the introduction explains why the problem addressed in the paper is important?
- Does the introduction discusses limitations of previous work?
- Does the paper clearly explains what are the new contributions made in the paper with respect to previous work?Are some important citations missing? Are the references too old?
- Does the paper contains plagiarism? Several papers that are submitted to conferences and journals contains plagiarized content. When I review a paper, one of the first thing that I do is check for plagiarism. I copy some sentences from the paper and search for these sentences using a Web search engine to see if the paper has already been published or contains a considerable amount of text from another paper. If there is plagiarism, I directly reject the paper.
- Does the proposed solution is described with enough details? For example, if someone is proposing a new data mining algorithm, does all the details of the algorithm are provided or is some important details missing?
- Is the paper technically sound? In other words, is there some technical errors in the paper (e.g. some lemma or theorem are incorrect). Does some important technical details are missing? Do the authors make some formal proofs (if necessary in his field) that their solution is correct?
- Does the experiments are appropriate to evaluate the proposed solution? A proposed solution should ideally be compared with solutions from other researchers and a fair experiment should be done to evaluate whether the proposed solution is better. Also, the experiments should be designed to evaluate what needs to be evaluated in the paper.
Besides, a reviewer should:
- Give a fair evaluation of what is good and what is bad in the paper. It is important to be fair and review the papers of other people in a fair way, just like we would like other people to review our paper.
- Provide constructive comments. When reviewing a paper, it is important to give constructive comments about how to improve the paper if it needs to be improved. This will help to improve the paper.
And a reviewer should NOT:
- Perform a review if there is a conflict of interest. A reviewer should not write a biased review. The review process should be neutral. The definition of a conflict of interest depends on the journal or conference. But generally, a conflict of interest can occur when a reviewer is invited to review his own paper or the paper of someone closely related to him such as a collaborator during the last few years or family member. Other types of conflict of interests also exist such as reviewing the paper of someone who is in competition with the reviewer in terms of research (for example, to reject papers of competitors in a biased way). As a reviewer, one should avoid conflicts of interests. But it still happen quite often. For example, as a program committee member of an international conference, I have found that one of the reviewer was trying to review his own paper, a few years ago. I informed the organizers of the conference and they then banned that person from the committee of that conference, as it is a very serious academic misconduct.
- Share or use the information about unpublished papers for other purposes than reviewing the paper. A reviewer has access to papers that are unpublished. It is the responsibility of the reviewer to follow the rules of the journal/conference and not share the unpublished papers with unauthorized persons, and also not use information in the unpublished papers for any purposes other than to review the papers. I will tell you a short story about that. A few years ago, I submitted a paper to PKDD 2012. My paper was rejected but then I found using a web search engine that my paper was online on the webserver of one of the reviewer. I sent an e-mail to PKDD to complain about this, as my paper was unpublished and should not be shared online by a reviewer. The, the reviewer removed the paper and sent me an apology. He explained that he just put my paper on his server with other papers because he was travelling and use it as a storage space. He then removed the paper.
The Review Generator
You can also try a website that I have developed to help generate a review draft called the paper review generator:
Conclusion
In this blog post, I have discussed why reviewing papers is important, how to review papers, and talked about what a reviewer should do and not do. I tried to give as much information as possible. If you think that I forgot something important or if you have some interesting story, please post it in the comments below. I will be happy to read it.
==
Philippe Fournier-Viger is a full professor and the founder of the open-source data mining software SPMF, offering more than 140 data mining algorithms. If you like this blog, you can tweet about it and/or subscribe to my twitter account @philfv to get notified about new posts.
