LLMs have shown impressive reasoning abilities, but whether they can coordinate like a swarm of bees or ants remains an open question. A team from Renmin University of China's Gaoling School of Artificial Intelligence has released SwarmBench, a benchmark that tests exactly this: whether large language models can achieve decentralized coordination when operating under the same tight constraints that govern natural swarms.
Why Swarm Coordination Is Harder Than It Looks
Most existing multi-agent benchmarks let agents share unlimited information. SwarmBench takes the opposite approach. Agents operate on a 2D grid with only a k × k local view and can communicate only with nearby neighbors. There is no global map, no central controller, and no way to see the full state of the world. This mirrors how real swarms work -- ants follow pheromone trails, birds react to immediate neighbors, and collective behavior emerges from strictly local interactions.
The gap this fills matters because real-world decentralized systems -- drone fleets, sensor networks, robot collectives -- rarely have perfect global information. Testing LLMs under these constraints reveals whether they can actually coordinate or whether they just fail when the information gets thin.
Five Tasks That Stress Different Coordination Skills
SwarmBench includes five foundational multi-agent tasks, each targeting a distinct coordination challenge.
- Pursuit requires agents to collaboratively track and corner a faster-moving target. Success depends on flanking and maintaining formation without a leader.
- Synchronization asks agents to align an internal binary state across the entire swarm using only local communication -- essentially building consensus from scratch.
- Foraging combines navigation and task allocation: agents must find a food source, transport it back to a nest, and coordinate who does what.
- Flocking demands that agents move as a cohesive group while maintaining both alignment and separation, balancing two competing spatial pressures.
- Transport requires multiple agents to cooperate physically, pushing a large object toward a designated goal.
Each task runs in a configurable grid environment (default 10 × 10) with adjustable view size and up to 10 agents per run, evaluated over 100 rounds.
How the Benchmark Was Built
The toolkit is open-source and built around a customizable physical simulation system. Researchers can configure environments, define standardized prompts, and run evaluations through a single Python API. The evaluation pipeline accepts model configuration with an API key and base URL, then submits jobs across tasks, models, and seeds for parallel execution.
After running evaluations, the toolkit provides scripts to generate replay videos of simulations, aggregate scores across runs, and export performance plots. An interactive replay tool allows step-by-step inspection of agent behavior at each round.
What the Results Show
The zero-shot evaluation of 14 leading models reveals a stark performance gap. o4-mini led the pack with an average score of 9.60 (standard deviation 0.49 across 5 runs), followed by Gemini-2.0-Flash at 8.80 and GPT-4.1 at 8.40. These models showed the strongest ability to handle the coordination demands across the five tasks.
Performance drops off sharply after that. Claude-3-7-Sonnet-20250219 scored 4.40, DeepSeek-V3 managed 4.20, and o3-mini came in at 3.60. The middle tier -- GPT-4o at 3.40 and QWQ-32B at 2.20 -- struggled considerably. At the bottom, Meta-Llama-3.1-70B-Instruct scored 1.80, GPT-4.1-mini 1.40, Llama-4-Scout 1.20, DeepSeek-R1 1.00, and Claude-3.5-Haiku-20241022 just 0.60.
The spread from 9.60 to 0.60 -- a 16-fold difference -- shows that swarm coordination is not a general capability that comes with language understanding. It is a distinct skill that a small number of models appear to possess.
The Gap Between Language and Coordination
According to the researchers, while some rudimentary coordination was observed across all models, current LLMs significantly struggle with robust long-range planning and adaptive strategy formation under the uncertainty inherent in decentralized scenarios. The models that performed well seemed to find ways to approximate swarm-like behavior, but even they showed clear limitations when tasks demanded sustained coordination over many rounds.
This finding has practical implications. If an organization is considering deploying LLM-based agents in a decentralized setting -- for warehouse robotics, distributed monitoring, or autonomous vehicle coordination -- the benchmark results suggest that only a narrow set of current models can handle the job without extensive fine-tuning.
SwarmRLVR: Training Agents With Reinforcement Learning
Beyond the benchmark, the project includes a companion effort called SwarmRLVR. The team adapted SwarmBench into a synchronous training framework that extracts prompts from the swarm environment, generates model responses, and feeds them back -- a loop compatible with OpenRLHF and other fine-tuning frameworks. This opens a path to train smaller models specifically for swarm coordination, rather than relying on general-purpose LLMs that happen to do well at it.
An Open Toolkit for Reproducible Research
SwarmBench is released as an open-source toolkit with a customizable physical system, standardized environments and prompts, evaluation scripts, and comprehensive experimental datasets. The authors invite community contributions including new tasks, feature implementations, and collaborative research. The academic paper is published on arXiv (2505.04364) and the code is available on GitHub.
The benchmark fills a clear hole in the research landscape. By forcing LLMs to operate under the same informational constraints as natural swarms, it provides a rigorous test of whether these models can genuinely coordinate or whether they are fundamentally dependent on centralized information that real-world systems do not have.