PTXBench: Benchmarking and Adapting LLMs for GPU Kernel Optimization

PTXBench evaluates LLM-generated GPU kernels on real hardware

Genghan Zhang and collaborators introduced PTXBench in August 2026 as a benchmark for evaluating and adapting large language models to use architecture-specific PTX for GPU kernel optimization. The benchmark measures three dimensions: functional correctness, whether selected target instructions execute at runtime, and speedup over frontier libraries. Testing focuses on GEMM and attention workloads across NVIDIA H100 and B200 GPUs. The release came through arXiv as a computer science paper under the Computation and Language category, with three versions published between August and September 2026.

The need for such a benchmark arises from the gap between an LLM's ability to generate syntactically valid PTX assembly and its ability to produce kernels that actually deliver performance improvements. PTX, or Parallel Thread Execution, is the virtual instruction set architecture for NVIDIA GPUs. Models that can output correct PTX instructions may still fail to achieve runtime speedups, or may generate code that crashes or produces incorrect results. PTXBench addresses this by running generated kernels on real hardware and measuring outcomes.

Workloads that separate capable models from competent ones

Evaluation across GEMM (matrix multiplication) and attention workloads revealed that no tested model consistently matches frontier libraries such as cuBLAS or FlashAttention. GEMM workloads, which involve dense linear algebra, showed relatively higher success rates. Attention workloads, particularly the backward pass that computes gradients, proved more challenging. Success rates on complex attention backward tasks fell substantially compared to forward passes or GEMM operations.

The benchmark also measured whether generated instructions actually run on the GPU. In some cases, models produced PTX that was syntactically valid but would fault at runtime due to unsupported instruction combinations or register pressure issues. Even when instructions executed, the reported speedup over established libraries varied widely. Some models achieved modest improvements, while others generated kernels that ran slower than optimized library calls. This disconnect between correct execution and competitive performance indicates that PTX generation requires more than surface-level syntax knowledge.

Adapting Qwen3.6-27B for architecture-specific PTX

The authors further adapted Qwen3.6-27B, a 27-billion-parameter model, using supervised fine-tuning on PTX generation tasks. The fine-tuning setup aimed to teach the model to produce PTX that not only compiles but also runs fast on target GPUs. Repair-conditioned training, a technique where incorrect outputs are corrected and re-fed into the training loop, improved performance on several tasks. However, generalization remained uneven across different kernel types and GPU architectures.

The results suggest that dataset size alone does not determine adaptation quality. The coverage of kernel types in the training data, the balance between GEMM and attention examples, and the quality of the "reasoning teacher"—which provides corrected PTX and explanations—all influence how well the adapted model generalizes. A model trained on a large but imbalanced dataset may excel at GEMM while struggling with attention kernels, or perform well on H100 but fail to transfer to B200.

Repair-conditioned training and the gap between instruction execution and performance

One notable finding is that executing target instructions at runtime does not necessarily translate into competitive performance. A model may generate PTX that compiles and runs without error, yet the resulting kernel may not outpace established libraries. This gap indicates that effective PTX generation requires understanding of GPU microarchitecture, memory coalescing, warp scheduling, and other low-level details that go beyond producing valid assembly.

Repair-conditioned training helped close this gap for some tasks. By repeatedly cycling incorrect outputs through a correction mechanism, the model learned to avoid common pitfalls such as incorrect shared memory usage or suboptimal thread block dimensions. However, the improvement was task-specific. Models still struggled to generalize across diverse kernel patterns, and the benefit diminished when faced with workloads not well-represented in the training set.

PTXBench as a living testbed for GPU-aware language models

PTXBench provides an auditable testbed for measuring and improving LLMs' ability to exploit evolving GPU architectures. The benchmark is designed to remain relevant as new GPU features and instruction sets become available. By tracking functional correctness, runtime execution, and performance speedup in a single framework, PTXBench allows researchers to isolate which aspect of PTX generation is most limiting for a given model.

The benchmark's scope—GEMM and attention on H100 and B200—covers some of the most common and compute-intensive patterns in modern AI workloads. GEMM underpins much of linear algebra used in training and inference, while attention is the core operation in transformer-based models. Optimizing these kernels with architecture-specific PTX could yield significant performance gains, but PTXBench shows that current LLMs still have room for improvement before they can reliably replace hand-tuned libraries.

For developers working on agent systems or GPU-accelerated applications, the key takeaway is that PTX generation is a hard subproblem. LLMs can be trained and fine-tuned to produce more correct and more performant kernels, but the adaptation process requires careful attention to data quality, task balance, and the specific GPU targets. PTXBench offers a way to measure progress on all three dimensions simultaneously, providing a clearer picture of where models stand and what remains to be done.