You write a paper, and now you ask yourself three questions: what is the problem you set for this paper? What does this symbol represent? What part of the legal text supports the conclusion in the abstract?
If the paper was really written by you, these questions would be so simple that you would be able to answer them without preparation.
However, in a recent small-scale experiment in the machine learning journal TMLR (Transactions on Machine Learning Research), the authors of three papers failed to answer even these basic questions.
On 16 September, TMLR published an article on its official blog entitled Asking the Author of Their Own Papers by Nihar B. Shah of Carnegie Mellon University, one of the journal's co-editors. During his two-week period as editor-in-chief, he selected 10 posts that should have been directly rejected by desk rejection.
Gautam Kamath, TMLR's director, called it a "heroic experiment" and said it confirmed a long-held conjecture that some of the authors didn't know what they were writing about.
10 rejections, all rejected
According to Shah, the experiment took place between August 14 and August 28, 2026, when he was working as the editor-in-chief. He unofficially extracted 10 papers from papers that were being rejected directly and sent a short message to the authors via the review platform OpenReview: An editor wanted to talk to you before sending the trial to better understand the paper, and if convenient, please email an appointment.
After the news was released, things quickly split up. The author of one paper withdrew the draft directly; one author replied that there was too much on hand and no phone calls; the remaining eight meetings were scheduled, but one of the authors did not show up.
In the end, Shah met with the authors of seven papers. Their identities vary, including undergraduate students, master's students, doctoral students, university professors and independent researchers.
Shah's questions were divided into two categories: one on the basic questions of problem setting, symbols and the results claimed in papers, and the other on the details of specific technical expressions, theoretical results and experimental design choices.
In seven interviews, only the authors of one paper answered all the questions. Three other authors were able to clarify the high-level thinking, but when asked about the technical details, they found it difficult. Worst of all, the other three authors failed to answer the basic questions, and all three were single-author papers.
Shah writes that two authors seem to have little substantive understanding of the content of their papers, while the other can't figure out exactly where the key findings claimed in the abstract are given or supported.
The only complete answer to the question did not pass. Shah found that one of the main conclusions of the paper was a major error during the review, and the author subsequently acknowledged it. TMLR rejected the paper directly, but allowed the authors to re-post it after correcting the errors or narrowing the conclusions. The remaining nine were rejected directly and not re-opened.
The next two episodes.
Shah also recorded two other things in the text, which he read with black humour.
One: The authors of two papers did not answer basic questions at the meeting, but sent him a written reply after the meeting. Shah delivered the two emails to the AI text testing tool Pangram, and the results were judged to be "100% AI".
Second: In another interview, one author attempted to introduce Shah to the analytical method he had used, and the result was that he unintentionally described a complete set of p-hacking processes, i.e. by repeatedly adjusting the analytical method until the data was "significant".
It should be pointed out that Shah himself does not exclude AI. He frankly, in order to read 8 papers in two weeks, he himself used the LLM to help understand and even learn some previously unfamiliar concepts.
Why do you want to do this?
TMLR was founded in 2022 and is operated by the team behind JMLR and is renowned for its review standards "only looking at whether the conclusions are supported by evidence, without novelty and SOTA as a reason for rejecting". Its reviewers, executive editors, and editors are all free volunteers, making it a force to be reckoned with in this year's posting trend.
According to TMLR's June announcement, the number of posts has tripled in the past year, with single-author posts soaring to 13 times, and the editorial department has even seen five papers published a day. Another set of figures given by Shah in the latest article are more intuitive: In 2023, the percentage of direct rejections by TMLR was about 6%, and today it has risen to about 53%, meaning that more than half of the posts do not go to external audits.
In the face of this situation, the TMLR intensively launched a series of measures this summer.
The first is that the annual post quota is set by the author. Unlike the fixed limit of "maximum N posts per person" at many conferences, the TMLR adopts a "moderation" quota rule, where the amount of a paper consumed by each author increases as the number of participants decreases. The specific parameters are: those who only vote for the author's paper have a maximum of 2 votes per year; if all posts are combined by 9 people, a maximum of 9 posts per year; the amount of active reviewers and executive editors is doubled. The rule has been implemented since July 1.
The second was the introduction of an AI review. In July, TMLR announced that each post would be accompanied by an AI-generated review opinion outside of a regular artificial review, evaluating only the "reliability" of the paper, i.e. whether the conclusions were supported by accurate, clear, compelling evidence, without making subjective judgments, nor making admissions recommendations, and the final decision was still made by the executive editor.
On 28 August, TMLR modified the original "public interest" standard, explicitly requiring the paper to communicate its findings clearly to the reader. Announcing bluntly, current AI writing tends to be perplexing, terminologically stacked, and difficult to read; if a paper is almost entirely AI-produced and lacks human involvement, at least at the level of existing AI systems, it is unlikely to meet this standard.
At the end of the article, Shah concluded that this meeting helped the editorial team to have more confidence in the existing direct rejection process, the quota system and the emphasis on clear writing.
It's not just TMLR.
TMLR's encounter is not unique. Since this year, several major publishing channels in the field of AI have been struggling with the same problem.
In early June, the official blog of NeurIPS revealed that out of 971 posts on the NeurIPS 2026 Position Paper, 273 posts (28.2%) were judged by Pangram to have an AI score of 100%.
NeurIPS also noted that the growth in AI writing is comprehensive: in the "Rating with Dataset" track, the Pangram score for papers of at least 90% increased more than tenfold from 2025 to 2026. However, NeurIPS also acknowledged that after testing results for parametrically sensitive, modified medium-sized test windows, the AI score for papers in 90% to 100% ranges will drop from 42.7% to 12.7%.
Earlier in mid-May, Thomas G. Dietterich, head of arXiv's computer science panel, announced a clear penalty on social media: If a post contains "correct evidence" that authors have failed to verify the big model output, such as delusional references and chatbot conversational phrases left in the official text, all authors will be banned from posting on arXiv, and posts must be reviewed by a formal peer-to-peer channel a year later. Dietterich emphasized that signing the paper means that each author is responsible for all content, regardless of how it is generated.
Related data are equally striking. According to a Columbia University Lancet study, the proportion of biomedical papers collected by PubMed Central that contain at least one fictitious citation rose from about 4 million in 2023 to about 57 million in early 2026.
From "looking at text" to "looking at people".
Shah admitted in the text that the experiment took him 20 to 25 hours to process only 8 papers, which is difficult to promote given the size of posts today.
He also mentioned that his team is working on scalable solutions.
Just a few days ago, Shah and CMU's Justin Payan, Bálint Gyevnár, Atoosa Kasirzadeh published a paper proposing an evaluation method called greCAPTCHA. Half of the name was taken from the US graduate entrance exam GRE, half from the CAPTCHA validation code that distinguishes between humans and machines.
Its idea is: no longer to test whether the text is written by AI, but to directly examine the author. The author submits a paper and a personal contribution statement, the system automatically generates a topic, the author answers under supervision, and finally generates an evaluation report, for journals, recruitment or recruitment references. Researchers call the ability to examine a "capacity to verify" - that is, whether the author has the ability to critically evaluate the knowledge and reasoning of their contributions.
The topics are divided into four categories: identifying the actual reporting data of the paper in multiple versions, pre-recognition of errors, design choices for explanatory papers for which no explicit reasons are written, interpreting the context concepts known to the readers by default, and indicating under what specific conditions the method will fail.
The team recruited 31 researchers for the test, each responding to their own papers and a stranger paper. The results showed that the system distinguished between "own papers" and "stranger papers" by an AUC of 0.90, which increased to 0.934 after eliminating the subject.
One subject's experience illustrates the problem: he threw a stranger's paper to AI before the test and let it explain it all over again, but he still couldn't answer the in-depth question, with a score of 1.3, and his own score of 51.
But the system is far from mature. The minimum misjudgment rate of the paper report is about 20%, meaning that one in every five real authors may be misjudged. Respondents complain the most about the scores: not knowing how accurate the answer is, the scoreboard is dead. An author stands out when faced with the criteria for denying his or her answer, which is clearly my own design. Respondents point out that some of the collaborative papers were completed by the co-authors, who would not have answered themselves; and there are concerns that such exams would examine English language expression, typing speed, and unfairness to non-English speakers and researchers with disabilities in the abstract.
What is the meaning of this verse?
Shah lists a few conclusions in the text, both of which are worth looking at:
One is credit allocation. In today's scientific research ecosystem, academic contributions are mainly identified by the signing of papers. If authors are unable to explain or defend their papers, the signal value of a publication as a "research contribution" is greatly reduced.
The second is the cycle of the review system. Many journals and conferences invite authors to review other people's papers. If the authors don't even see their own papers, they naturally ask questions.
Of course, there are obvious limitations to the experiment itself. Ten samples were taken from Shah's "informal" papers, all of which were originally intended to be rejected directly, and could explain "what's in a low-quality post" rather than the overall appearance of a TMLR post. Shah himself wrote that one of the purposes of the experiment was to check whether the direct rejection process was reliable.
But Shah's experiment gives at least one bottom line: A paper can be done with AI's help, but the signer must be able to say clearly what it says.