Sometimes physicists can be very talkative when describing their research findings, but when it comes to responding to others’ achievements, they seem to get straight to the point and use words sparingly.
An analysis of peer review reports conducted by Publons in 2018 revealed that physicists and scientists in related fields generally write shorter review reports compared to other scientific disciplines such as biology, psychology, and environmental science. Regardless of the subfield or the journal’s rank, physicists tend to prefer concise and precise review reports.
Although Publons’ data may not represent the entire academic community’s peer review ecosystem, it offers us a rare opportunity to glimpse into the habits of scientists.
Publons is a free website that allows researchers to receive recognition for their peer reviews. Users can log their review report information or have their endorsements from other peers added to increase their “reviewer value” on the website.
Reviewing research manuscripts for peers is usually a form of professional development for researchers. Besides keeping themselves at the forefront of research, there are hardly any other benefits. Therefore, most scholars are reluctant to engage in this arduous task.
Read More
For the academic community, having online access to review records is highly valuable for reviewers. It allows funding agencies to understand how much time the applicants have spent reviewing other researchers’ work when evaluating research proposals.
Since its launch in 2012, Publons has registered over 300,000 researchers. The website’s revenue comes from partnerships with publishers and universities, as well as funding from angel investors in New Zealand.
In June 2017, Clarivate Analytics, an information company, acquired Publons, providing information on journal impact factors.
Publons’ dataset includes over 300,000 reviewers, and on average, a review report consists of approximately 292 English words (see Figure 1). Most review reports in physics and related fields are significantly lower than this value, including applied physics, applied physics engineering, materials science, and general physics. However, Earth and Space Science is an exception, with an average of 494 words per review report.
(Below Figure) Word Count of Review Reports in Different Scientific Fields. According to the review report data provided by Publons users, the word counts of reports vary significantly across different scientific fields. Compared to other fields, physics and related disciplines tend to receive shorter review reports. (Source: Publons)
We take a closer look at general physics (see below figure), we will find that the average word count of review reports in almost all subfields is less than 300.
Andrew Preston, co-founder and managing director of Publons, suspects that this may be because many authors upload their papers to arXiv (Note: a preprint server for papers without peer review) before submitting them to journals. Therefore, they receive a lot of feedback and suggestions even before the submission to journals.
Nuclear physics is an exception, with an average report length of 436 words. However, nuclear physics has the fewest reviewers among all physics subfields in the Publons database. Andrea Taroni, the editor-in-chief of Nature Physics in London, said, “I can’t think of a reasonable explanation for why (nuclear physicists’) comments would be longer.”
(Below Figure) In the field of physics, the length of review reports varies. According to data from publons, the average length of review reports for physics articles is below 300 words. Nuclear physics stands out as noticeably different from other subfields of physics, although it could also be due to statistical error caused by a few reviewers who prefer lengthy discussions. (source: publons)
In recent years, Lester Ingber, the project director of Extreme Science and Engineering Discovery Environment (XSEDE) and the most valuable reviewer in the field of nuclear physics on Publons, has been known for his concise review reports.
He mentioned that his recent review report was approximately 400 words long, which is only about half the usual length. According to him, “I fail to see the purpose of summarizing the article. However, I do know that many reviewers do this to demonstrate that they have indeed read and understood the paper.”
AJ Mitchell, an experimental nuclear physicist at the Australian National University in Canberra, shared that his review reports typically range from 500 to 800 words.
While he recognizes the importance of concise and impactful reviews, he raised concerns about the possibility of skewed statistical results due to a significant portion of researchers providing inadequate information, resulting in a decrease in the average word count.
Based on Publons’ analysis of scientific data, review reports from high-impact factor journals are generally longer compared to those from low-impact factor journals (see below Figure). The Publons team published their analysis results on the website’s blog, suggesting that this phenomenon may be due to reviewers believing that submitting more comprehensive review reports would benefit their future publications in the same journal.
Read about Publons Global State Of Peer Review 2018 Report
(Below figure) Journal impact factor and review report length. Although the average length of review reports in top-tier physics journals exceeds 350 words, there doesn’t appear to be a correlation between the length of physics review reports and journal impact factors overall. However, the trend is clear in the field of science as a whole. Here, the x-axis merely divides the journal impact factors into twenty segments, which are not the actual impact factors. Additionally, the actual impact factor ranges in each interval differ between the left and right charts. (source: publons)
However, physics reviewers don’t believe in this approach; on average, the review reports of some high-impact journals are even shorter than those of low-impact ones. Taroni said, “Based on my own experience, physicists focus on the quality of the manuscripts.”
Preston hopes that Publons’ data can establish a set of tools to enhance the efficiency of peer review. For example, his expectation is that one day there will be a tool that can automatically remind journal editors when a reviewer’s workload has exceeded its limit. In October 2017, Publons activated the Review Distribution Index to calculate the workload of reviewers.