Can a local LLM beat -O3?

Local LLMs Can Beat -O3, But the Results Come With Caveats

A set of experiments presented as a poster at CppCon this week asks whether a small language model running on consumer hardware can outperform GCC and Clang's top optimization flag. The answer is yes, sometimes, with important caveats about correctness and the kind of code being optimized.

The work was inspired by Andrei Alexandrescu's keynote at ACCU on Sea in June, where he predicted LLMs would eventually integrate into compiler pipelines, processing optimization remarks and proposing changes that traditional compilers struggle to discover. The poster tests that prediction against reality using eight local models and five deliberately inefficient C++ benchmark programs.

Five Functions, Five Optimization Problems

The benchmark suite consists of five exported functions, each isolated in its own source file. Fibonacci uses a naive recursive implementation with an obvious algorithmic improvement available. Format_list serializes arrays in a loop with inefficient string concatenation. Repeated_sort gives the model a chance to detect redundant work. Count_matches searches for elements of one array in another, where a different data structure would help. The most complex, top_words_from_file, parses a file and counts word frequencies using a deliberately recursive approach that overflows the stack at -O0.

Each function was designed to offer clear optimization opportunities with large potential gains. The author notes this was likely the most consequential decision in the experiment. The benchmarks reward high-level algorithmic rethinking, the kind of change a human developer would make, rather than the low-level instruction-level tweaks where compilers excel. A follow-up with less obvious targets would test a different and arguably harder question.

Measuring Carefully on CPU Only

The hardware is an AMD Ryzen 9 5950X with 128 GB of RAM and no GPU. Every model runs on CPU alone. This constraint reflects a deliberate choice: compilers are tools you download and run locally, and the experiment asks whether an LLM-assisted compiler could work the same way.

Each test function is optimized in isolation, then compiled into a shared library alongside the other four unoptimized functions. A loader application first runs a correctness suite, then benchmarks throughput in calls per second. Each candidate is measured five times in randomized order, and the median result is used. A 2 percent threshold separates genuine improvements from noise.

The author found three candidates that passed correctness tests but contained undefined behavior, including a signed overflow on INT_MIN and a new[]/free() allocator mismatch. These were flagged as invalid after adding ASan and UBSan validation.

C++ Optimizations Win, LLVM IR Does Not

Four optimization strategies were tested. Naive C++ sends the full source file to the model with a prompt to optimize. Full LLVM IR sends the complete intermediate representation generated at -O1. Extracted IR uses llvm-extract to isolate just the target function. Guided C++ feeds the model compiler optimization remarks and runs up to three iterations, selecting candidates based on measured performance.

The results were clear: all meaningful speedups came from C++ strategies. Full LLVM IR produced only five valid candidates across forty attempts, mostly because the context was too large for the models to handle. Extracted IR produced zero successful modifications. The C++ approaches did far better, particularly on format_list, where the model attempted optimizations similar to what a compiler might do, such as reserving output string capacity and manually converting integers.

Against the fastest LLVM configuration, the best LLM candidate on format_list was within roughly 2 percent. On other benchmarks, the gap was larger, but the LLM still won.

Eight Models, Very Different Results

The models tested include four general coding models in the 12 to 14 billion parameter range (Qwen3 14B, Qwen2.5-Coder 14B, Gemma 4 12B, Ministral 3 14B), two compiler-specialized models from Meta's LLM Compiler work (7B and 13B), and two larger models (gpt-oss-20b and Devstral Small 2 24B).

The compiler-specialized models, pretrained on LLVM IR and assembly, produced no valid optimization candidates in this experiment. The general coding models varied widely. Response truncation was the most common failure, with 71 cases where the model exhausted its token budget without producing output. Context overflow accounted for 33 failures, primarily with LLVM IR where the prompt alone exceeded the window. Twenty cases produced code that would not compile, and fourteen failed correctness validation.

The author notes that larger cloud models perform drastically better and make LLVM IR optimization viable, a finding covered in a separate follow-up post.

What the LLMs Actually Changed

The successful candidates targeted the deliberate inefficiencies in the benchmark code. The most interesting case is format_list, where models attempted string capacity reservations and manual integer conversions. These are the kinds of transformations a compiler might perform, but most still did not match -O3. The other benchmarks saw the LLMs make higher-level algorithmic changes, like replacing recursive approaches with iterative ones or selecting better data structures.

Guided C++, which fed compiler remarks to the model across multiple iterations, made 82 LLM calls over more than 36 hours. Many of the remarks it sent were low-information hints like GVN LoadClobbered or LICM LoadWithLoopInvariantAddressInvalidated. The model mostly ignored these and continued proposing algorithmic changes. The experiment does not isolate whether the compiler hints contributed meaningfully versus the additional optimization attempts and performance-based selection doing the work.

The Core Finding

LLMs in this experiment were far more effective at the C++ source level than at LLVM IR. The intermediate representation is too large, too unfamiliar, or both, for small local models to work with productively. At the C++ level, they can find real improvements, particularly when the code contains obvious algorithmic opportunities.

But LLM-generated code cannot be assumed to preserve functional behavior. It frequently fails to compile, sometimes fails correctness tests, and occasionally contains undefined behavior that passes validation but is still wrong. Any system that uses an LLM as an optimization step needs strong validation behind it. The compiler itself is not enough.

The experiments are open source, with code linked from the author's blog. The poster was presented at CppCon's registration reception on September 14, 2026.