When Didier Raoult published several studies last year purporting to show the promise of the malaria drug hydroxychloroquine as a treatment for COVID-19, critics quickly denounced his methods. Raoult, a microbiologist at Aix-Marseille University, now faces disciplinary action by a French medical regulator, and the drug has largely been discredited as a COVID-19 treatment.
But some researchers had another concern: Raoult’s astonishingly prolific publication in the journal New Microbes and New Infections, where some of Raoult’s collaborators serve as associate editors and editor-in-chief. Since the journal’s creation in 2013, Raoult’s name appeared on one-third of its 728 papers. Florian Naudet, a metascientist at the University of Rennes, wondered how common the pattern was. He and his colleagues teamed up with University of Oxford psychologist Dorothy Bishop, who had developed a method to identify prolific authorship, to explore its extent in the biomedical research literature.
The group extracted data on nearly 5 million papers published between 2015 and 2019 in more than 5000 biomedical journals indexed by the U.S. National Library of Medicine’s Broad Subject Terms, which catalogs journals’ subject focus. This method didn’t capture journals that aren’t registered in the catalog with these subject terms—among them, less established journals such as New Microbes and New Infections, Naudet says. The researchers then counted the number of articles each author had published to identify the most prolific researcher at each journal.
In half of the journals, the most prolific author published less than 3% of the papers. But 206 journals were outliers, with a single author responsible for between 11% and 40% of the papers, the team reports in a preprint posted this month. Although many of these outlier journals are obscure, some are recognizable titles with significant impact factors: the Journal of Enzyme Inhibition and Medicinal Chemistry, Journal of the American Dental Association, and Current Problems in Surgery. If New Microbes and New Infections had been included in the analysis, Raoult’s publication rate would place it in the top 10 outlier journals. Raoult and Michel Drancourt, editor-in-chief of New Microbes and New Infections, did not respond to requests for comment.
The researchers also compared the time from submission to publication and found that prolific authors enjoyed faster peer reviews. And in a random sample of 100 of the outlier journals chosen for closer scrutiny, the researchers found what they consider evidence of favoritism or, as they call it, “nepotism”: For about one-quarter of these journals, the prolific author was the editor-in-chief of the journal, and in 61% of them the author was on the editorial board.
The study, which has not yet been peer reviewed, is “well done,” says Ludo Waltman, a bibliometrician at Leiden University, and raises questions about the integrity of the scientific literature. Study co-author Clara Locher, a pharmacologist at the University of Rennes, notes there is still work to be done because the analysis does not show whether papers authored by prolific researchers at these “nepotistic” journals are lower in quality. Naudet says asking readers blinded to the status of the journals to grade a subset of the papers could shed light on that question.
But Waltman cautions against making simple binary distinctions between “good” and “bad” journals. Many fall in a gray zone, he says, and drawing bright lines risks giving a tacit stamp of approval to journals that don’t exceed an arbitrary cut-off but may still have substantial problems.
What’s needed, he says, is more transparency from journals about their editorial processes. The best way for journals to avoid nepotism, he says, would be to publish comments from each article’s peer reviewers, allowing readers to judge for themselves whether it has been properly reviewed.
The underlying assumption of this preprint is that it analyzed only those papers submitted for peer review and successfully eliminated other regular features, such as “…articles explicitly referenced as editorial, correspondence or news articles…”
There is at least one example in which that is not the case. The Journal of the American Dental Association (JADA) was determined by the authors to be an outlier “nepotistic” journal, in which a single author was responsible for about 33% of the articles published in the journal between 2015 and 2019.
This is absolutely correct. Our analysis of Scopus data shows that Romina Brignardello-Petersen authored 475 of 1475 articles, or roughly 33%, of articles published in JADA between 2015 and 2019.
What is not correct, however, is that these articles were submitted as original research articles subject to peer review. Rather, Brignardello-Petersen was the section editor responsible for producing the “Clinical Scans” feature that appeared in each issue of JADA during that period. Clinical Scans summarized articles in the professional literature other than JADA to keep JADA readers abreast of important advances in oral-health research. Each Clinical Scan provided a review of a single journal article, and JADA typically published about 10 Clinical Scans in each issue, which accounts for all 475 “articles” published by this single author. Since the Clinical Scans were commissioned features rather than original research papers, they were not submitted for peer review and therefore could not have received preferential, “nepotistic” preference in the peer review process. They should not have been included in study set of articles analyzed in the preprint.
The second most prolific author was Michael Glick, the journal’s editor in chief at the time, who wrote 43 editorials, accounting for 2.9% of JADA articles published during that time, well under the 11% threshold the study suggests for labelling a journal “nepotistic.” The third most prolific author published 2.4% of all articles in the journal during the study period. All other authors were even less prolific than that.
In light of the mischaracterization of Clinical Scans, I have asked the study authors to remove JADA from their list of “nepotistic journals.” I also urged them to perform a more rigorous screening process, one that successfully identifies only those articles eligible for preferential treatment in peer review. I suspect they will find a far smaller set of journals practicing nepotistic policies.
I believe there is a lesson in this, not only for the study authors, but also for journalists who report on preprints. While O’Grady accurately notes that the study was not peer-reviewed, she quotes Ludo Waltman, a bibliometrician at Leiden University, who pronounces the study to be “well done,” which serves, I guess, as de facto peer review. And indeed, the study presents a cornucopia of statistical analyses.
However, a statistical analysis on a fundamentally flawed data set will lead to fundamentally flawed conclusions. And one has to hope that the basic error in this study will be discovered and addressed in peer review.
If anything, the study and the ScienceInsider post serve to underscore the value of peer review, as well as the danger of reporting on unvetted data sets.
I suspect that JADA may not be the only journal whose integrity has been wrongly called into question by both this flawed study and the news report.
Michelle Hoffman, VP, Publishing, American Dental Association