A developer who has spent enough time tuning CUDA kernels knows the frustration of iterating on performance by hand. The bertaye/agentic-cuda-optimizer project, recently published on GitHub, automates that cycle by pairing an AI agent with a C++ CUDA execution environment that compiles, runs, benchmarks, and refines kernel code without continuous human intervention.
How the Agent Works
The system is built on LangGraph, which orchestrates an agent that explores kernel implementations and GPU launch configurations. The agent receives a workload description and proceeds through an automated loop: it generates CUDA kernel code, compiles it using NVRTC, launches it through the CUDA Driver API, compares the output against a NumPy reference, and measures latency. Results feed back into the next iteration, and invalid candidates are repaired within a configured iteration budget.
What sets this apart from a simple code generator is that the agent can modify both the kernel source and the per-case launch configuration. It queries GPU properties, retrieves optimization guidance from NVIDIA documentation, and can inspect Nsight Compute performance counters to inform its next experiment. Every step is recorded.
The Validation and Ranking Pipeline
Each candidate kernel must pass a correctness check before it is considered. The project validates every case against a reference implementation. Ranking is not based on a single measurement. Instead, the system computes the geometric mean of latency across performance cases, while small correctness-only cases do not influence the score.
Timing defaults to 10 warmup launches followed by 100 measured launches using CUDA events. Compilation time and profiler replay durations are explicitly excluded from the ranking calculation. This prevents the optimizer from being penalized for build overhead and ensures the score reflects actual kernel execution performance.
Setup and Requirements
The project was developed on Windows with an RTX 3060 Laptop GPU. It requires Python 3.12 or later, an NVIDIA GPU with a compatible CUDA Toolkit and driver, CMake 3.24 or later, a C++17 compiler, and an OpenAI API key. The default model is gpt-5-mini with medium reasoning effort, and API usage is billed to the user's account.
Setting up the project involves creating a virtual environment, installing Python dependencies, and building a standalone C++ test harness with CMake. A configuration file specifying the OpenAI key is placed in the repository root. The command-line interface accepts a workload description and optional flags to control the optimization process.
Optional Profiling and Research
Two flags extend the agent's capabilities. The --use-nsight flag enables per-case profiling with Nsight Compute after each valid candidate, exposing performance counter data directly to the model. This requires Nsight Compute and permission to access GPU performance counters. The --nvidia-research flag instructs the agent to retrieve NVIDIA optimization guidance before generating kernels.
When both flags are active, the agent has a richer feedback loop. It can see hardware counters, cross-reference vendor recommendations, and use that information to guide its search through the space of possible implementations and configurations.
Outputs and Session Tracking
Each optimization session produces a directory under results/run-NNN/ containing the generated kernel sources, all requests and responses exchanged with the model, input and output data, a JSON execution history, and a summary file. Successful runs export the best-performing kernel as best.cu, per-case replay requests, and a timing heatmap in both PNG and SVG format.
Sessions can be resumed from an earlier run by supplying the saved input cases, reference kernel, and best kernel from the previous attempt. This allows a developer to extend optimization beyond the initial iteration budget without losing the work already completed.
Limitations to Be Aware Of
The project is explicitly experimental and targets individual kernel optimization. Passing the supplied test cases does not constitute a proof of general correctness, and a generated reference kernel is not an independent correctness oracle. Improvements are workload-dependent, and the project does not currently compare its results against cuBLAS or other vendor-optimized libraries. Generated input scripts execute locally as Python subprocesses without a sandbox, and generated CUDA kernels run directly on the local GPU.
Despite those caveats, the project represents a concrete step toward delegating CUDA kernel tuning to an autonomous agent. For developers working on GPU workloads where the cost of manual optimization is high, the approach of letting an agent explore configurations with hardware-backed feedback is a practical alternative to trial and error.