Paper mills and AI: how flawed research enters LLM training data
Scientific paper mills and AI-generated content threaten research integrity. Retractions reveal only a fraction of the problem, says TalTech researcher Anton Sokolov.
In early September a Novaator experiment showed how easily co-authorship can be purchased on the black market. With just a few WhatsApp messages, 20 days and $800 it is possible to become an author of a scientific paper ready for publication. Misconduct involving authorship is only one part of a much broader problem concerning scientific literature integrity.
After the experiment published Anton Sokolov, a doctoral student at Tallinn University of Technology and researcher at the Tyche Institute, contacted the outlet. As part of his own research he has taken a closer look at what happens to bogus science once it enters databases. Among the sources he uses is the Retraction Watch database, which compiles retractions and other research integrity cases from around the world.
Retraction numbers tell part of the story
If we look at Retraction Watch's public database there are about 61,000 retracted papers from 2010 to 2025. Currently about two papers per thousand are retracted each year. The number of retractions shows how much is being cleaned up, not how much garbage remains, Sokolov said.
As part of his research Sokolov compiled a dataset containing nearly 15,000 retraction records focusing on computer science literature. Analysis showed that violations directly related to authorship accounted for less than 2 percent of cases. But the small proportion does not mean authorship is rarely sold. The notice shows what the publisher was able to prove, not how much is actually being sold, he explained.
Such transactions usually take place in private conversations and there is nothing in the published text that unequivocally indicates authorship was purchased. For this reason retraction notices much more frequently cite problems with peer review. A single notice usually lists several reasons at once, with peer-review problems cited in about seven out of 10 cases, he added.
Computer-generated text and AI in retractions
Computer-generated content has also become an increasingly serious problem. According to Sokolov, for example the most common reason for retractions in computer science and artificial intelligence is computer-generated text, accounting for about one-third of cases.
In 2023 the number of officially retracted scientific papers exceeded 10,000 for the first time. Sokolov said however that the figure reveals little about the true prevalence of problematic papers because the vast majority of retracted papers came from large-scale cleanup programs by a handful of academic publishers. The number of retractions simultaneously measures the problem itself, its detection and classification and a publisher's ability to see cases through to completion — not the actual prevalence of problematic authorship or the use of artificial intelligence, the researcher said.
Estonian cases in retraction records
Retraction Watch also lists papers co-authored by Estonian researchers. As of September there are 13 retractions linked to Estonian institutions and 11 of them involve co-authors from other countries. About half are older cases involving issues such as duplicate publication or errors in analysis. The rest are more recent, dating from 2023 onward and involve problems with the review of the papers or their data, Sokolov explained.
According to Retraction Watch data one retraction linked to Estonia is also flagged as involving a paper mill. But that does not show whether any of the authors here purchased authorship, Sokolov said.
LLM risk: flawed research in training data
The researcher believes the impact of paper mills could also extend to artificial intelligence systems. Because large language models are trained on material found online and in scientific publications, flawed or manipulated research can also make its way into them. The risk is real and it has already been measured. In a study published last year a language model was asked to assess retracted or questionable papers. In more than 6,500 assessments it did not mention the retraction even once, Sokolov said.
The problem is that artificial intelligence may absorb bogus scientific material even before a paper is officially retracted. The publicly available text-screening tool Problematic Paper Screener has flagged tens of thousands of papers as questionable. According to Sokolov's preliminary pilot calculation more than 20,000 of these papers had not been retracted as of June.
Toward digital authorship verification
One possible solution he said would be to use digital signatures to verify authorship of scientific publications. At the moment an author of a scientific paper is simply a name in the text. In Estonia we use our ID cards to sign contracts and applications; similarly each author could digitally sign off on their contribution, Sokolov suggested.
He said digital identification of authors and other parties involved in scientific publishing would help increase transparency and make it more difficult to manipulate authorship. This has not yet been implemented but we plan to continue this research, he said.