...Falsifier outperformed all baselines on 14 of 21 specifications. On six of them, the counterexample was found on the very first simulation in every run. This is a fundamental impossibility for any numerical optimizer, which has zero information before its first evaluation. The LLM reasons from the specification alone: if the spec says speed must stay below 120 and the throttle is currently at 100, pushing throttle higher will likely violate it. The model comparison tested three configurations: GPT-OSS-20B (open-source, low reasoning effort), GPT-5-nano (proprietary, low reasoning), and GPT-5-mini (proprietary, high reasoning). GPT-5-mini consistently achieved the fewest simulations, often by a wide margin. On the challenging AT52 spec, GPT-5-mini needed 1.3 simulations on average versus 6.5 for GPT-5-nano and 3.8 for GPT-OSS-20B. The open-source model remained competitive on specs where semantic reasoning dominated, matching proprietary models at 1.0 average simulations on CC1 through CC3 and NNx. Runtime costs vary with model size. GPT-OSS-20B averages 4.4 seconds per iteration, GPT-5-nano averages 10.9 seconds, and GPT-5-mini averages 81.8 seconds. Total API cost for all experiments was approximately $100. For domains where a single simulation takes minutes or hours, this overhead is negligible.

Where the Method Reaches Its Limits

Not all specifications yield to semantic reasoning. Four specs in the benchmark, AT51, AT54, NN, and SC, could not be falsified by any variant within the 100-simulation budget. The NN spec involves a neural network controller with a discrete gear signal that creates flat robustness plateaus. The semantic enrichment provides no leverage on landscapes where small input changes produce no change in the output signal. Similarly, SC involves a complex temporal specification that no tool in the competition handles well.

The method also inherits the latency of LLM inference. Sample efficiency does not automatically translate to wall-clock efficiency, especially when the LLM itself takes tens of seconds per call. The approach is most attractive when simulation evaluations dominate the runtime, which is common in automotive and aerospace CPS domains.

Explainability remains preliminary. Reasoning traces were analyzed on the open-source GPT-OSS-20B because its full chain-of-thought is accessible. The proprietary models driving the main results do not expose their internal reasoning, so the mechanisms behind their stronger performance are not fully understood.

What This Opens Up for Practitioners

For CPS engineers, the practical lesson is that LLMs can function as falsification engines today, not as research curiosities. The semantic enrichment strategy, combining variable names, output trajectories, and critical-time witnesses, is straightforward to layer onto existing STL monitoring infrastructure. The authors provide open-source code at github.com/aliabigdeli/llm-falsifier.

The broader principle extends beyond falsification. Any optimization problem where the objective can be evaluated through simulation and the search space has natural-language structure, such as named parameters with physical meaning and interpretable specifications, may benefit from an LLM-based search. The key is designing prompts that expose that structure to the model.

Hybrid approaches are the natural next step: using an LLM for high-level exploration guided by semantic reasoning, then switching to a classical optimizer for fine-grained local search once the search is in a promising region. Incorporating graybox model information, such as differential equations or control structure, directly into prompts could further improve efficiency. For now, LLM-Falsifier establishes that language models, given the right contextual information, can navigate the rugged robustness landscapes of cyber-physical systems faster than the specialized numerical tools built over decades.

Read the paper on arXiv