I have all the information needed. Let me write the article.
IMPORTANT: yes
The Gap Between Using AI Code and Knowing If It Works
When a researcher pastes an error message into ChatGPT and gets back a fix, the immediate question is whether to trust it. The code might run. The error might disappear. But whether the fix actually preserves the scientific intent of the program, whether it quietly changes a coefficient or drops a boundary condition that matters, is a different kind of question entirely. A new study from researchers at the University of Michigan, Sandia National Laboratories, and the University of Tennessee examines how scientists answer that question, and the picture it draws is both detailed and uncomfortable.
The study analyzes 527 free-text responses from a 2025 survey of researchers who write code, most of them at US universities. Each respondent described a single recent task where they used a generative AI tool, how they used it, and what they did to check whether the output was acceptable. The researchers coded each response for the type of task and the evaluation strategy reported, then related both to programming experience, research area, and three confidence ratings: confidence in doing the task alone, confidence in the tool, and confidence in evaluating the output.
Five Tasks Dominate, and They Are the Ones Where Correctness Is Hardest to See
Use was concentrated in five task categories, which together accounted for 76% of coded accounts. Data handling was the most common at 24%, covering operations like merging dataframes, reshaping tables, converting between data structures, and parsing specialized file formats from instruments. Visualization came next at 20%, mostly incremental adjustments to existing plots: changing colors, adding variables, formatting for publication. Debugging accounted for 17%, typically pasting error messages and asking the tool to explain or resolve them. Mathematical and scientific computing made up 12%, translating equations from papers or textbooks into working code. Statistical analysis rounded out the top five at 11%, implementing regression models, running tests, or exploring available packages.
The pattern is revealing. Four of these five tasks share a property that makes verification difficult: the correctness of the output is often not visible in the output itself. A dataframe merge can run without errors while dropping rows in ways that silently corrupt downstream analysis. A statistical model can fit without throwing exceptions while using the wrong likelihood function. A mathematical implementation can produce plausible numbers while solving a different equation than the one intended. Even debugging, where the most tangible signal exists (the error message either appears or does not), can be misleading: a fix might clear the error while altering code that was working correctly.
Visualization is the exception. A plot either shows what you expect or it does not, and respondents treated it that way. One wrote that "it's entirely about the visualization so it's literally visually validated." Another said that for plot generation specifically, "I probably accept it immediately." This self-validating quality makes visualization the easiest task to check, and it is no coincidence that evaluation confidence was highest for visualization use cases.
Running the Code Is the Default, and Almost Everything Else Is Rare
The evaluation strategies respondents described were overwhelmingly informal and individual. Running the generated code was reported in 53% of accounts. Inspecting the output appeared in 23%. Reading the code was in 19%. Inspecting a visualization showed up in 16%. Drawing on domain knowledge or intuition was in 11%. These five strategies, all variations of "I looked at it and it seemed fine," dominated the corpus.
More rigorous checks were rare. Consulting documentation or reference implementations appeared in 9% of accounts. Comparing output against a benchmark or known result was in 8%. Automated testing of any kind, including unit tests, appeared in just 3% of accounts. Review by another person showed up in 2%. Checking the mathematics by hand was in 1%. Using a different AI tool to cross-check was in 0.4% of accounts.
The researchers note that these numbers reflect what respondents chose to report, not necessarily everything they did. Someone who wrote "I ran it" might also have compared the output against expectations without saying so. But even accounting for under-reporting, the pattern is clear: validation rests primarily on the individual who prompted for the code, running it, looking at the result, and making a judgment call. There is no shared infrastructure, no standard process, and rarely a second pair of eyes.
The evaluation strategies did not vary much with programming experience. More experienced programmers did not report more strategies, more diverse strategies, or different strategies than less experienced ones. The one difference that approached significance was debugging, where less experienced programmers were slightly more likely to describe debugging use cases, but even this did not survive correction for multiple comparisons.
Confidence Diverges at Four Years of Experience
The most striking quantitative finding concerns confidence. The study asked three questions on a 1-to-5 scale: how confident are you in doing this task without AI, how confident are you in the tool's ability to do it, and how confident are you in your ability to evaluate the output.
Solo confidence (doing the task alone) rose steadily with programming experience, as expected. GenAI confidence (trusting the tool) was flat across experience levels. The gap between the two, measured as solo confidence minus GenAI confidence, widened with experience and crossed zero at roughly four years of programming. Below four years, researchers trusted the tool more than themselves. Above four years, they trusted themselves more than the tool.
This means less experienced programmers are using AI for tasks they are not confident they could do alone, while experienced programmers are using it to save time on tasks they could do themselves. The use cases and evaluation strategies are similar across both groups, but the underlying relationship to the work is different. A junior researcher asking ChatGPT to implement a regression model they have never run before is in a fundamentally different position than a senior researcher asking it to write a pandas merge they could do in ten minutes.
Evaluation confidence, the third rating, did not vary with the strategies respondents reported. People who ran the code were not more or less confident in their evaluation than people who read the code line by line. The number of distinct strategies reported was also unrelated to evaluation confidence. The strongest correlates of evaluation confidence were, separately, confidence in the tool and confidence in doing the task alone, two ratings that were uncorrelated with each other. This suggests that evaluation confidence is not a product of how thoroughly someone checks, but of how much they trust themselves and their tools in general.
Task-Specific Patterns Reveal Where Evaluation Is Hardest
When the researchers conditioned evaluation strategies on use case, clear task-specific patterns emerged. Debugging was the use case where running code most often appeared as the sole strategy: 31 of 88 debugging accounts described only running the code, with no other check. The typical workflow was paste an error, get a fix, run it, see if the error is gone. Whether the fix changed something unrelated to the error was generally not assessed. One respondent described finding a substantive error in a 2000-line codebase by pasting it into ChatGPT, which "instantly located the exact error," but the account does not describe any independent verification that the fix was correct beyond the error disappearing.
Mathematical and scientific computing showed the opposite pattern: most accounts described multiple strategies. Running the code was still the most common, but respondents also compared against published results, checked derivations step by step, verified hand-computed quantities against the code's output, or consulted supervisors and colleagues. One respondent described implementing a function with "heavy math that I didn't fully understand, and didn't need to fully understand," validating it entirely through a test bench that confirmed the function returned expected results. This is verification by behavior rather than by inspection, and it works only when you know what the expected behavior is.
Statistical analysis showed a similar pattern of combined strategies, with respondents checking output against known values, comparing against alternative implementations, or inspecting for specific failure modes like asymmetry in matrices that should be symmetric. Several used the tool as a recommender system, asking what analysis methods were available for their data, which introduces a different kind of risk: the tool might recommend a method that is inappropriate for the data structure, and the researcher might not know enough to recognize it.
Data handling, the most common use case, showed moderate evaluation diversity. About half of accounts described more than one strategy, typically running the code and inspecting the output. Some created mock datasets with known answers, but many did not specify what they checked beyond "it worked" or "I got my desired result."
The Closed Loop Problem
Nearly every account in the corpus describes a closed loop: the researcher prompts the tool, gets output, checks it themselves, and decides. Review by another person appeared in 2% of accounts. Automated testing appeared in 3%. The person who prompted for the code is the same person who evaluates it, using their own judgment, with no external reference point.
This is not unique to AI-assisted programming. Scientific software development has long had informal verification practices. The researchers cite prior work showing that scientists often validate through expertly curated diagnostic plots rather than test batteries, and that many hold a broader notion of verification that covers the mathematics the code implements and the physical experiments it models. But AI-generated code introduces a new complication: the person evaluating the code may not have written it and may not fully understand it. Storey has argued that code produced without the programmer's full understanding accrues a distinct kind of debt in comprehension and intent.
The study does not claim that scientists are under-evaluating. The appropriate level of evaluation depends on the task, the observability of its output, and the scientist's knowledge of the domain. For a cosmetic change to a plot, visual inspection may be exactly right. For a statistical model applied to clinical data, it is clearly insufficient. The problem is not that people evaluate too little, but that the evaluation strategy is decoupled from the risk profile of the task. The same "I ran it and it worked" approach is applied to plot formatting and to regression implementation, even though the consequences of an error are vastly different.
What Tool Designers Could Do Differently
The researchers propose several directions for tool design. The core idea is that interfaces could support task-appropriate evaluation rather than leaving it entirely to the user.
For data handling, tools could automatically generate a small, easy-to-inspect sample dataset to run through the generated code, so the researcher can observe the transformation directly. Several respondents already did this manually, creating mock data with known properties. Automating it would lower the barrier.
For visualization, tools could expose the transformations applied before plotting, showing what data manipulations occurred and letting the researcher verify that the pipeline is correct even if the final plot looks right. A plot can look correct while being based on silently corrupted data.
For debugging, tools could produce a regression test showing that the fix addresses the original failure without changing unrelated behavior. This is the standard practice in professional software engineering, but almost no one in the survey described doing it.
For mathematical and scientific computing, tools could generate comparison artifacts: a check against a known quantity, a limiting case, or an alternative implementation. The goal is to provide independent evidence that does not require the researcher to devise the verification strategy from scratch.
For statistical analysis, tools could report what assumptions the method makes, whether the data meets those assumptions, and what the output would look like under common misspecifications. This would shift some of the verification burden from the researcher's domain knowledge to the tool's ability to flag its own limitations.
The researchers also suggest that training could focus on making the informal evaluation strategies used by senior scientists more explicit. When an experienced researcher inspects output, what specific qualities do they check? What diagnostic plots do they make? What internal comparison between expected and actual results do they perform? Teaching these practices directly, rather than assuming they will be absorbed through apprenticeship, could help less experienced researchers develop better-calibrated evaluation habits.
What This Means for Research Integrity
The study's findings land in a context where coding errors have already forced retractions of published findings in healthcare and public policy. Scientific code is increasingly generated by tools that may be plausible but wrong, and the people evaluating that code often do so individually, informally, and with high confidence that is not well calibrated to the strategies they actually use.
The most concerning finding is not that people use lightweight checks, but that evaluation confidence does not track evaluation strategy. A researcher who runs the code and looks at the output reports the same evaluation confidence as one who reads the code line by line, compares against a benchmark, and consults a colleague. Confidence appears to come from trust in the tool and trust in oneself, not from the evidence gathered during evaluation.
This matters because confidence influences behavior. If you believe you have evaluated the output thoroughly, you are less likely to seek additional verification. If you trust the tool, you are less likely to scrutinize its output. The study cites prior work showing that knowledge workers with higher confidence in generative AI engaged in less critical evaluation. The data here cannot establish whether highly confident respondents evaluated poorly, but it does suggest that subjective confidence should not be treated as a proxy for thoroughness.
The researchers are careful to note the study's limitations. The data consists of self-reported accounts of single recalled episodes. Respondents may have under-reported checks they performed. Social desirability may have led some to over-report diligence. The sample is concentrated in US higher education and likely overrepresents scientists interested enough in programming to answer a survey about it. The survey asked about one specific episode, so person-level claims are not licensed by the data.
But the core finding stands: validating AI contributions to scientific code rests largely on individual judgment, exercised outside shared infrastructure for testing or review. As agentic tools that execute code and iterate on their own become more common, this dynamic may shift. Scientists may move from designing their own verification strategies to judging whether an agent's strategies are appropriate. But for now, the gap between using AI-generated code and knowing whether it works remains wide, and closing it is left mostly to the person sitting at the keyboard.