Modern video games are one of the most demanding testbeds for artificial intelligence. They demand that a model simultaneously perceive visual scenes, decompose natural-language instructions into subgoals, plan across multiple time scales, and emit precise sequences of keyboard and mouse actions. A model that can play a complex AAA game well is one that can arguably reason, plan, and act in the real world. Yet until now, no single benchmark has measured all of these abilities together, across diverse games and model families, with reproducible results.
That is the gap filled by GameHorizon Suite, introduced on September 21, 2026 by Yiran Wang and colleagues at Tencent ARC Lab alongside collaborators from Great Bay University, the National University of Singapore, Huazhong University of Science and Technology, CUHK MMLab, and the University of Macau. The suite comprises three tightly integrated components: an automated annotation pipeline (GameHorizon-Annotator), a large-scale gameplay dataset (GameHorizon-Data), and a benchmark with both offline and online evaluation tracks (GameHorizon-Bench). Together, they form the first standardized yardstick for measuring gameplay capabilities at multiple temporal horizons across diverse model families.
Why existing benchmarks fall short
Prior attempts to evaluate AI on video games have each hit a wall. Some datasets, such as MineDojo, VPT, and STEVE-1, are confined to a single title like Minecraft, making it impossible to know whether a model's performance reflects genuine generalizable ability or narrow overfitting to one game world. Others, like WildWorld, are limited to a single AAA title, Monster Hunter Wilds, and rely on AI-generated action labels rather than recordings from actual human players.
Second, language instructions are largely missing from existing corpora. Datasets such as NitroGen, GameVerse, D2E, and VPT contain no text instructions at all. Open-P2P provides annotations, but they are so sparse that one instruction appears only every few minutes, with uneven temporal coverage. Without dense, temporally aligned text at multiple horizons, it is impossible to test whether a model can take a long-term objective such as "defend the bridge" and figure out which specific keystrokes realize it over the next several minutes.
Third, prior benchmarks have relied almost exclusively on online rollouts with small sample sizes. Lumine reports task success rates based on just three trials per scene. GameVerse conducts rollouts on 33 to 2020 cases. Such small samples produce low-confidence comparisons, and because the results are sensitive to specific game environments and custom agent harnesses, they are difficult to reproduce. Worse, an aggregate success rate conflates distinct failure modes: a model might misidentify what is happening in the current frame, infer the wrong next subgoal, or fail to map that subgoal to the correct sequence of controls. Without isolating where the breakdown occurs, the benchmark tells you little about what to fix.
The GameHorizon-Annotator: building an instruction pyramid automatically
At the heart of the suite is GameHorizon-Annotator, an automated pipeline that constructs a three-level pyramid of natural-language instructions from raw gameplay recordings. The levels are short-horizon operations (L1, lasting 1 to 5 seconds), medium-horizon goals (L2, lasting 10 to 22 minutes), and long-horizon strategies (L3, lasting 50 to 80 minutes).
The pipeline proceeds bottom-up, which is a deliberate architectural choice. Training a vision-language model to produce fine-grained instructions over an entire hour-long gameplay session is unreliable; the model loses track of precise action details. So instead, the annotator first partitions a recording into short clips using an action-aware segmentation technique. Rather than relying on visual similarity alone, which tends to over-segment during rapid camera motion, the pipeline uses the recorded keyboard-mouse traces to identify key action transitions. For example, it maps raw key events to game-specific semantics such as "sprinting" in Cyberpunk 2077. A vision-language model then examines sampled frames to disambiguate action semantics and refine the clip boundaries.
Once the L1 clips and their instructions are produced, the pipeline merges adjacent L1 segments into L2 clips. A vision-language model judges whether neighboring short-horizon operations form a continuous progression toward the same medium-horizon goal. The same process is then applied at the next level, merging L2 goals into L3 strategies. A dynamic programming algorithm enforces the duration ranges at each level. The bottom-up design ensures that higher-level instructions are grounded in actual action sequences rather than invented from scratch.
For L1 clips, the annotator receives sampled video frames alongside the aligned keyboard-mouse actions and produces an action-grounded operation description, including coordinates, object descriptions, and spatial relations where needed. For L2 clips, the inputs include the frames, actions, and constituent L1 instructions; the output is a goal description that strips away local operational detail while preserving the strategic intent. For L3 clips, only video frames and L2 instructions are provided, keeping the focus on high-level strategy without action-level noise.
GameHorizon-Data: 5,000 hours of human gameplay
Applying the annotator to recordings collected from 100 experienced human players yields GameHorizon-Data, the first large-scale AAA gameplay dataset with temporally aligned videos, player actions, and multi-horizon instructions. The corpus spans 5,000 hours of gameplay across 21 titles, including Valorant (617.5 hours), Minecraft (589.8 hours), Grand Theft Auto V (566.1 hours), Palworld (438.7 hours), Delta Force (337.0 hours), Red Dead Redemption 2 (304.4 hours), Cyberpunk 2077 (301.2 hours), Genshin Impact (232.6 hours), and PUBG: Battlegrounds (197.4 hours), among others.
The dataset covers diverse genres: open-world, action role-playing, competitive shooter, sandbox survival, and creature-collecting adventure. All recordings are captured at 2K resolution and 60 frames per second, with synchronized keyboard-mouse actions sampled at 20 Hz. After filtering out low-quality content such as prolonged cutscenes, 4,341 hours (86.8 percent) were retained for instruction annotation.
The numbers are substantial. The dataset contains 411.03 million keyboard-mouse action events, averaging 22.84 events per second. It is annotated with 6,184,036 distinct instructions: 5,947,588 short-horizon operations, 189,158 medium-horizon goals, and 47,290 long-horizon strategies. On average, there is one distinct L1 instruction every 2.63 seconds, and every frame is simultaneously aligned with instructions at all three horizons. This density is a critical differentiator: Open-P2P provides one instruction every few minutes, and STEVE-1 contains only 10,000 short-term text instructions in total.
GameHorizon-Bench: reproducible offline and stepwise online testing
The benchmark component, GameHorizon-Bench, addresses the reproducibility and diagnostic shortcomings of prior evaluation methods through two complementary tracks.
The offline track: thousands of standardized multiple-choice questions
The offline track formulates gameplay evaluation as multiple-choice questions derived from the aligned frames, instructions, and actions in GameHorizon-Data. It contains thousands of questions organized into three primary tasks and ten diagnostic variant tasks. Because the questions use standardized textual action representations (mapping keyboard-mouse controls to action sequences via game-specific keybinds), they enable unified evaluation across games and models without being constrained by heterogeneous action spaces.
The first primary task, single-horizon action (T1), asks the model to determine the correct action sequence given input frames and a short-horizon L1 instruction. This is a rigorous test of action perception: can the model look at a game scene, read the instruction, and output the right keystrokes?
The second task, multi-horizon decomposition (T2), requires the model to take a medium-horizon L2 goal and identify the ordered sequence of short-horizon L1 operations that realize it. The correct option is the actual L1 sequence from the instruction pyramid, while distractors differ in their composition or temporal order. This tests top-down goal decomposition and temporal planning.
The third task, cross-horizon consistency (T3), assesses whether the model can verify the overall alignment across video frames, L3 strategies, L2 goals, L1 operations, and actions. The correct option consists of the aligned instructions across all three horizons plus the corresponding action sequence.
The ten variant tasks probe different capabilities. For T1 star, the paper compares current-action perception versus future-action planning under three input settings: frames only, frames with an L1 instruction, and frames with multi-horizon instructions. For T2 star, top-down decomposition is compared against bottom-up abstraction. For T3 star, instruction-only options are contrasted with options that also include action traces. These variants reveal whether performance bottlenecks arise from goal planning or action decoding.
The online track: stepwise evaluation with failure localization
The online track tests whether offline scores actually reflect real gameplay capabilities. It evaluates long-horizon objectives through collections of verifiable short-horizon subtasks, organized as either causal tasks (subtasks must be completed in a prescribed order due to dependencies) or thematic tasks (subtasks share a theme but can be performed in any order). Each long-horizon task comprises 2 to 6 verifiable subtasks.
The key innovation is the stepwise protocol. When an agent fails at a subtask, the game environment is reset to the corresponding success state so that evaluation can continue. This prevents a single failure from derailing the entire evaluation and, crucially, localizes the failure to a specific step within a long-horizon objective. A long-horizon task is considered passed only when all constituent subtasks succeed.
Results: a meaningful hierarchy of model capabilities
The paper evaluates 47 models through more than one million model invocations. The models span four categories: general-purpose vision-language models (including GPT-6-Astra, Gemini 3.8 Flash, and Claude models), unified multimodal models, coding and GUI agents, and dedicated game agents. The average performance across all 44 evaluated models on the offline primary tasks is 57.3 percent on T1, 65.1 percent on T2, and 71.6 percent on T3, yielding an overall average of 64.7 percent.
The results reveal a clear hierarchy. GPT-6-Astra leads with an overall score of 80.2 percent, achieving 69.4 percent on T1, 79.6 percent on T2, and 91.5 percent on T3. Gemini 3.8 Flash follows at 77.3 overall, and Gemini 3.7 Flash at 76.7. A Tier 2 cluster includes models scoring between roughly 66 and 75 percent, while Tier 3 and Tier 4 models fall below 65 percent, with some dropping into the mid-40s.
Several patterns emerge from the data. First, task difficulty increases meaningfully from T1 to T3: action perception is easier than goal decomposition, which is easier than cross-horizon consistency verification. Second, there are pronounced differences in model capabilities: the best model scores 80.2 overall while the average is 64.7, and the bottom-tier models fall below 50 percent on individual tasks. Third, the offline and online rankings show a clear positive association: models that score well on the offline MCQs tend to perform better on long-horizon online tasks, validating that the offline track is a reasonable proxy for actual gameplay capability.
The variant tasks yield additional insights. Planning future actions and decomposing complex goals prove more challenging than deciding the current action at a single moment. Incorporating medium- and long-horizon instructions improves future-action planning by 7.2 percentage points compared to vision-only input, confirming that the multi-horizon instruction pyramid provides genuine signal rather than noise.
Limitations and trade-offs
The paper acknowledges several boundaries. The dataset, while large, is drawn from 21 games selected for genre diversity rather than attempting to cover every AAA title. The 100 human players, though experienced, represent a specific demographic and skill distribution. The annotator, while automated, relies on a vision-language model that may introduce its own errors in instruction quality. The offline track, despite its rigor, uses multiple-choice questions that cannot capture the full open-ended nature of real gameplay. And while the online track localizes failures, it covers only 20 long-horizon tasks across 62 subtasks, a small fraction of the task space the dataset could theoretically support.
There is also the practical reality that the code, benchmark, and data are scheduled for release around October 25, 2026, meaning the full evaluation infrastructure is not yet publicly available at the time of writing. The leaderboard on the project page already shows results for 44 models on the offline tasks, but the complete data and annotator remain forthcoming.
What this means in practice
For developers and researchers building AI agents for games, GameHorizon provides the first dataset where videos, actions, and multi-horizon text instructions are temporally aligned at scale. This makes it possible to train models that can take a high-level instruction and reason about the sequence of actions needed to fulfill it, rather than simply predicting the next keystroke. For the broader AI community, the benchmark offers a standardized, reproducible evaluation protocol that does not require custom game harnesses or environment-specific code, enabling fair comparisons across model families.
The stepwise online track is perhaps the most practically valuable feature for debugging real agent behavior. Instead of a single pass-fail metric that tells you nothing about where an agent went wrong, the protocol resets the game to the point of failure and continues evaluation, pinpointing exactly which subtask broke down. This transforms evaluation from a score into a diagnostic tool.
The release of the dataset, annotator, and benchmark is expected around October 25, 2026, via the GitHub repository at TencentARC/GameHorizon and the project page at gamehorizon-suite.github.io. With 5,000 hours of AAA gameplay, 6.18 million annotated instructions, and 47 benchmarked models, GameHorizon Suite establishes a new standard for what it means to evaluate AI on the hardest task domain available: playing modern video games.