A new benchmark is testing whether large language models can do more than generate text. StarSkirmish gives each LLM one hour to write a StarCraft: Brood War bot in C++, then evaluates those bots in a full tournament against other LLM-generated bots and established human-written competitors. The results reveal a clear performance gap among frontier models and suggest that writing strategy code under a real-time clock constraint tests capabilities that standard coding benchmarks do not fully capture.
How the Benchmark Works
Each participating LLM is asked to write a Protoss bot in C++ that runs against BWAPI 4.4.0 on OpenBW. Games are Protoss versus Protoss on three ladder maps: Heartbreak Ridge, Benzene, and Destination. A game ends when one side's buildings are destroyed. For games that reach the 60-minute cap of 86,400 frames, the winner is decided by the BASIL ladder scoring system, which prioritizes kills and building demolitions.
The key ingredient is the tool set. Each LLM has access to three tools: a compiler, a batch simulator that plays practice games against opponents at various skill tiers, and a game transcript reader that provides build timings, fight summaries, and economy recaps. This means the model must learn from its own practice games and iteratively improve its strategy across the full hour. It is not a one-shot code generation task. It is a closed loop of coding, testing, and refining.
The Tournament Structure
The field consists of 62 entrants: 50 LLM bots, which are 10 models each running 5 separate instances, plus 9 competitive human-written bots and 3 stock demo bots. Every entrant plays every other entrant six times, twice on each map. Elo ratings are fitted from all games simultaneously, and each model's benchmark score is the average rating across its five bots rather than its best performer.
Practice opponents are organized into tiers. The D tier contains the three demo bots that every LLM can read. C and B tiers consist of mid-ranked human-written bots. The A tier includes BananaBrain and Locutus, and the S tier is anchored by Stardust, the top human-written bot, which serves as the benchmark's ceiling with a score of 100. The weakest demo bot, Four Gate Dragoon, anchors the floor at 0.
Who Came Out on Top
GPT-6 Astra and Claude Opus 5.5 are functionally tied as the highest-scoring LLMs on the StarSkirmish Bench, with GPT-6 Sol forming a clear step above every other tested model at writing StarCraft strategy code in C++. When performance is weighed against the average API cost of a one-hour run, GPT-6 Sol stands apart as offering uniquely strong value.
The ten models tested all ran inside the same harness built on Inspect's ReAct deepagent, with access to bash, a text editor, a memories tool, and research subagents. Models were called through OpenRouter with high reasoning effort, and the harness settings required first-party inference so that open-weights models received their full quality. There is no submit tool; when the hour expires, the harness automatically picks up the completed bot code.
What It Actually Measures
The creators of StarSkirmish emphasize that it is not a novelty benchmark. The correlation analysis shows it tracks closely with four public coding benchmarks, and most strongly with Vals IOI, which tests olympiad-level programming problems solved in C++, and WeirdML, an independent benchmark of unusual machine learning coding tasks that is thought to be less over-optimized by existing methods.
The benchmark is designed to capture long-horizon reasoning and agentic coding. An LLM that can write a StarCraft bot must plan across an hour of runtime, interpret game state from transcripts, adjust its strategy based on losses, and compile working C++ code that survives against opponents it has never seen. Each of those capabilities matters for real-world agentic systems, and the tournament format means partial solutions are exposed immediately by competition.
Looking Forward
One notable finding is that state-of-the-art reasoning models appear to benefit significantly from reasoning periods longer than the one-hour cap used in this benchmark. The team has indicated plans to launch versions of StarSkirmish that can test model capabilities across substantially longer reasoning windows, which would likely widen the gap between frontier models and the rest of the field. For now, the benchmark offers a public, reproducible window into how far LLMs have come at writing functional, competitive strategy code under real constraints.