A new open-source project puts language models to the test across five classic board games, pitting them against humans and each other to measure something that benchmarks cannot capture: how well a model actually plays when it has to think strategically, follow rules, and make decisions with consequences.
Five games, eleven models
Chess5.ai supports Chess, Go, Xiangqi, Gomoku, and Othello, and it runs them against eleven language models: GPT, Claude, Gemini, Grok, Muse Spark, Mistral, DeepSeek, Kimi, Qwen, GLM, and MiniMax. You can play against any of them directly, or watch two models play each other. Auto-play mode lets you sit back while the systems battle it out without human intervention.
The model chooses each move entirely on its own. There is no external chess engine guiding the decision, no Monte Carlo tree search, no evaluation function. The model sees the board state, thinks about it in whatever way its architecture allows, and outputs a move. The application parses that output, validates it against the game rules, and updates the board. If the model outputs an illegal move, it gets a warning. Three illegal outputs in a single match count as a loss.
That rule matters because it forces the model to understand the game it is playing, not just produce plausible-looking moves. A model that hallucinates piece movements or ignores the rules of Go will lose quickly, and the illegal-move counter makes that failure visible rather than silently correcting it.
What board games reveal that benchmarks do not
Standard language model benchmarks measure knowledge recall, reasoning on isolated problems, and the ability to follow instructions. Board games measure something different: sustained strategic thinking over many turns, where each decision affects the options available later. A model that aces a multiple-choice logic test might play terrible chess because it cannot maintain a coherent plan across thirty moves.
The five games chosen cover different types of strategic thinking. Chess requires piece coordination and tactical calculation. Go demands pattern recognition and territory estimation. Xiangqi, Chinese chess, introduces different movement rules and piece values. Gomoku is simpler in rules but requires reading ahead in a different way than chess. Othello, with its flipping mechanic, rewards positional awareness and the ability to think about how a single move changes the entire board.
By testing across all five, the project gives a broader picture of each model's strategic capability than any single game could provide. A model that plays strong chess but weak Go is revealing something about its architecture and training that a text-based benchmark would miss.
Adjustable difficulty and the beginner use case
The difficulty is adjustable, which makes the project useful beyond benchmarking. Beginners and children learning board games can play against models at a reduced strength, getting feedback on their moves without facing the full capability of a large language model. The game engine handles rule enforcement, so a beginner does not need to know every rule of Xiangqi or Go to play.
This is a different kind of AI educational tool than the ones that explain strategy or annotate games. It is simply a opponent that plays at the level you set, providing the kind of practice that used to require a human partner or a dedicated chess program. The fact that the opponent is a language model adds an element of novelty, but the core value is the same: structured practice with immediate feedback.
Model vs model spectating
The spectating and auto-play modes turn Chess5.ai into a tool for comparing models head-to-head. You can watch GPT play Claude at chess, or pit Gemini against DeepSeek at Go, and observe how each model approaches the game differently. The differences are often more interesting than the final result. One model might play aggressively, trading pieces for initiative. Another might play conservatively, building position before attacking. These tendencies emerge from the model's training and architecture, and watching them play out across a full game makes them visible in a way that a benchmark score does not.
Auto-play takes this further by letting you run multiple games without watching each one. You can set up a round-robin tournament between models and collect win rates across all five games, building a picture of which models are strongest at which types of strategic thinking.
Open source and what it enables
The project is open source, which means the game parsing logic, the model integration, and the board rendering are all available for inspection. Developers can study how the application validates moves, how it handles model output that does not conform to expected formats, and how it manages game state across turns. For anyone building tools that need to parse and validate structured output from language models, the parsing approach used here is a concrete example.
The source also reveals the practical challenges of getting language models to play board games correctly. Models output moves in different formats, sometimes with explanation, sometimes with coordinates that need mapping to the game's notation system. Handling that variance without breaking the game loop is a non-trivial engineering problem, and the open-source implementation provides a working solution.
For developers interested in evaluating language models on tasks that require sustained reasoning rather than single-turn knowledge retrieval, Chess5.ai offers a ready-made test harness. The games are defined, the rules are enforced, and the infrastructure for running models against each other is already built. Adding a new model or a new game is a matter of implementing the integration and the rule set, not building the platform from scratch.