Large language models are now routinely used to write code, but their ability to reason about how that code actually executes during runtime remains largely unexamined. A model can generate syntactically correct Python and still fail to predict whether a loop terminates, whether a variable holds the expected value at a given point, or whether an exception propagates past a handler. This gap between code generation and code execution reasoning is the focus of a new study that introduces the largest repository-level benchmark of its kind.
The problem with existing execution-reasoning benchmarks
Existing benchmarks that test whether LLMs understand program behavior fall into two categories, each with significant limitations. The first category consists of repository-level question answering benchmarks such as RepoQA and CodeRepoQA, which evaluate static code understanding. These benchmarks ask models to answer questions about code from source text, documentation, and repository structure. They do not test whether a model can reconstruct what actually happens when code executes, including how control flows through functions, how variables change over time, or how exceptions propagate across file boundaries.
The second category includes execution-reasoning benchmarks such as CRUXEval, REval, CoCoNUT, and TraceEval, which ask models to predict properties of program behavior. These benchmarks provide valuable evidence about model capabilities, but they share two main limitations. First, each covers only a subset of runtime behavior rather than systematically evaluating all the major semantic structures involved in execution. Second, they evaluate self-contained functions, snippets, or extracted fragments rather than executions within real repositories that contain cross-file dependencies, framework behavior, and repository-specific environments. A model that can predict the output of an isolated function may struggle completely when the same behavior depends on class inheritance, dynamic dispatch, or helper functions spread across multiple files.
These gaps motivated a team of researchers from York University and the University of Texas at Dallas to build a benchmark that addresses both requirements simultaneously. Their paper, "Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark," introduces SWE-Flux, a benchmark containing 480 execution-grounded instances across 12 real Python repositories.
What makes SWE-Flux different
SWE-Flux differs from prior benchmarks in three fundamental ways. First, every instance is grounded in a real repository at a pinned commit, with a specific test file that exercises the target code. Second, the gold answers are not written manually by annotators or judged by another LLM. Instead, they are automatically harvested from instrumented test executions run inside reproducible containers. Third, the benchmark covers a comprehensive taxonomy of runtime behaviors that spans from simple intra-procedural control flow to suite-level aggregation across many tests.
The taxonomy has two levels. The first level distinguishes single-test questions, which ask about the behavior of one specific execution, from multiple-test questions, which ask about aggregated behavior across a suite of tests. The second level covers seven categories of runtime behavior: intra-procedural control flow, inter-procedural control flow, loops and temporal behavior, program state, dataflow, exceptions, and program invariants.
To give a concrete sense of what these questions look like, a single-test intra-procedural control flow question might ask for the exact ordered line path executed inside a specific function. A multiple-test dataflow question might ask which runtime def-use pairs are observed across every test in a suite that calls a function with different array inputs. An invariant question might ask whether a variable always holds a value of length 8 or starting with a specific digit across all observations from a suite of tests.
The benchmark draws from 12 repositories spanning diverse domains: scientific computing with SymPy and PyBaMM, machine learning with Keras, visualization with Matplotlib, command-line tools with yt-dlp, web frameworks with python-telegram-bot and Reflex, cloud tools with cfn-lint, and desktop applications with Qtile. The codebases range from 70,000 to 760,000 lines of code, ensuring that questions are grounded in realistic, non-trivial environments. The median instance triggers 603 function calls across 28 distinct functions, covers 192 unique lines over 7 files and 5.5 classes, and reaches a call-stack depth of 8 while executing nearly 4,000 lines of code.
How the benchmark is built
Building SWE-Flux is a carefully multi-stage process that ensures every gold answer is both correct and reproducible. For each repository, the researchers adopt the SWE-bench-Live evaluation image, which freezes the codebase at a specific commit together with its dependencies and runtime environment. All construction and evaluation takes place inside containers started from this image.
Each instance targets a concrete dynamic behavior. The researchers select a category from the taxonomy, choose a subcategory and a code target spanning multiple functions, classes, or files, and then author five artifacts: a test file that exercises the target, a natural-language question, an answer template that fixes the expected response structure, a tracing configuration that declares which runtime events to record, and an instance-specific parser that reconstructs the answer from the execution trace. Authoring is AI-assisted, but every artifact is designed, reviewed, and curated by the authors.
The authors iterate inside live containers, running the harvesting pipeline and debugging the test and parser until the answer is correct. Once an instance converges, its artifacts are copied back to the host and the instance becomes immutable. A separate automated pipeline then re-harvests every oracle answer from a clean container state, ensuring reproducibility. The final validation layer uses an agent powered by Sonnet-4.6 to answer each question independently, checking that the question is well-specified and answerable from the released materials alone.
The results reveal a major capability gap
When five LLMs were evaluated on SWE-Flux, the results were sobering. The best-performing model, GPT-5.4, achieved only 37.71% accuracy across all 480 instances. GPT-5.3-Codex followed at 33.75%. Open-weight models performed substantially lower: GPT-OSS-20B at 20.62%, Gemma-4-31B at 19.17%, and Gemma-4-4B at 17.29%. Even the strongest closed-source model answers fewer than four out of ten execution-reasoning questions correctly.
Performance varies across repositories, with GPT-5.4 ranging from 65% on cfnlint to just 12% on PyBaMM. However, the per-question-category results reveal a clearer pattern than the per-repository results. Models perform best on program invariants, with an average accuracy of 69%, followed by intra-procedural control flow at 53% and exceptions at 42%. These categories involve localized reasoning: identifying which statements execute within a function, recognizing exception patterns, and verifying properties that hold across tests.
The contrast with the hardest categories is stark. Runtime dataflow questions average only 1.7% accuracy across all models, with S4 dataflow at 6.8% and M4 dataflow at 1.8%. Inter-procedural control flow averages just 16% for single-test questions and 8% for multi-test questions. The gap between intra-procedural and inter-procedural performance is dramatic: accuracy drops from 53% on single-test intra-procedural questions to 22% on multi-test intra-procedural coverage, and from 16% to 8% on inter-procedural reasoning. This pattern suggests a fundamental boundary in current models between recognizing local behavior and faithfully reconstructing execution across function boundaries.
Understanding how LLMs fail
Manual inspection of 299 incorrect answers from the strongest model revealed that 93% of failures are substantive semantic errors rather than formatting mistakes or minor numerical deviations. The researchers identified seven distinct failure patterns.
The first pattern is that local behavioral recognition is easier than runtime behavior prediction. Models can reason about paths within a single function but struggle to follow execution across function boundaries. The second pattern is that most failures are semantic rather than formatting errors. The third pattern involves boundary errors, where models either stop their reasoning too early or follow execution paths that do not actually execute.
The fourth and most striking pattern is that runtime dataflow is the hardest form of dynamic reasoning. Dataflow questions require determining not only which statements execute, but which definitions reach which uses under the concrete runtime path. The near-zero accuracy across all models suggests that current LLMs do not reliably model variable lifetime, last definitions, loop-carried dependencies, or scope boundaries during execution.
The fifth pattern involves suite-level reasoning, where models struggle with aggregation rather than extrapolation. A model might correctly predict the behavior of one test case but fail to synthesize the correct answer across an entire suite. The sixth pattern is that correct code paths do not guarantee correct runtime states: a model might identify the right execution path but still get the variable values wrong at specific points along that path.
The seventh pattern is that static priors can override runtime evidence. Models sometimes rely on what they expect the code to do based on its structure, ignoring what actually happens during execution. This is particularly problematic when the static appearance of code misleads the model about its dynamic behavior.
Generating harder benchmark variants
One of the most practical contributions of the paper is the oracle-harvesting pipeline's ability to generate fresh benchmark variants through input perturbation. By modifying the inputs to existing test cases and re-running the instrumented execution, the pipeline can produce new instances with the same structure but different runtime behavior. The pipeline successfully harvests valid variants for 89.7% of selected instances, corresponding to 52 of 58 cases.
The generated variants are substantially more challenging. When evaluated on the selected original instances, GPT-5.4 achieved 100% accuracy, but on the perturbed variants its accuracy drops to 43.1%. Models not involved in the perturbation process, such as Gemma-4 and GPT-OSS, also show dramatic declines, dropping from approximately 47-48% accuracy to 24.1%. This indicates that the perturbation method creates genuinely harder questions that expose execution-reasoning weaknesses across model families, not just memorized patterns.
The practical implication is that SWE-Flux can continuously generate fresh benchmark instances without manual effort. As models improve on existing questions, new perturbed variants can be added to keep the benchmark challenging. This makes SWE-Flux not just a static evaluation tool but a living benchmark that can evolve alongside model capabilities.
Limitations and trade-offs
The study has several limitations worth noting. The evaluation uses a fixed set of five models, and the results may not generalize to all LLMs or to future models that may perform differently on these tasks. The context window of 32,000 tokens for open-source models may constrain their ability to reason about larger repositories, though the same constraint applies to all models in the comparison.
The benchmark focuses exclusively on Python repositories and execution-reasoning questions. While Python is a widely used language, the findings may not transfer directly to other programming languages, particularly those with different execution models, typing systems, or concurrency primitives. The taxonomy covers seven categories of runtime behavior, but there may be other important dimensions of execution reasoning that are not captured.
The oracle-harvesting pipeline depends on the ability to run code in reproducible containers, which requires that all dependencies are available and that the code executes deterministically. Some repositories or code paths may not be amenable to this approach. The perturbation method also assumes that small input changes produce valid but different runtime behaviors, which may not hold for all code patterns.
Finally, the benchmark measures exact-match accuracy against harvested oracles. While the answer templates reduce formatting ambiguity, some questions may admit multiple valid answers that differ in granularity or phrasing. The validation layer mitigates this risk, but it cannot eliminate it entirely.
What this means for coding agents
The practical implications for software engineering tools are significant. Current coding agents can generate code that passes tests, but they cannot reliably reason about what the code does before or after execution. This limits their usefulness for tasks that require understanding runtime behavior without actually running the code, such as debugging without a debugger, performing security analysis on untrusted code, or estimating the impact of a change before deploying it.
The finding that models struggle most with dataflow and inter-procedural execution suggests that the next frontier for coding AI is not generating more code, but understanding the runtime semantics of the code it already has access to. A coding agent that can accurately predict variable values, track dependencies across function boundaries, and reason about exception propagation would be substantially more capable than one that can only generate syntactically correct programs.
The SWE-Flux benchmark provides the first comprehensive, execution-grounded evaluation of this capability at repository scale. By making the benchmark, its construction pipeline, and its code publicly available, the researchers have given the community a tool to measure progress, identify failure modes, and direct future research toward the specific capabilities that current models lack.