Benchmarks Are Expensive to Build and Easy to Break
Evaluation drives progress in embodied AI. When a model claims to understand spatial relationships, plan multi-step tasks, or interact with physical environments, the benchmark is the arbiter. But building these benchmarks is a logistical mess. A single embodied benchmark requires coordinated artifacts: images or simulator states, grounded questions, reference answers, task instructions, scoring protocols, executable verifiers. These artifacts depend on each other in chains. A defective evidence record produces a malformed question, which produces a wrong answer, which poisons the entire evaluation. Existing automation covers isolated stages, scene generation or task synthesis, and passes intermediate artifacts downstream without checking them. Defects propagate silently.
A team from QiYuan Lab, Tsinghua, BUPT, UESTC, and several other Chinese institutions, led by Baoyang Jiang with corresponding author Qiang Ma, presents Embodied-BenchForge. The framework treats benchmark construction as a closed-loop process: synthesize forward, verify backward, repair locally. It constructs seven benchmarks covering six offline EQA settings and one interactive track with 220 executable tasks, and the experiments show the resulting benchmarks reliably distinguish model capabilities across both observation-based understanding and closed-loop execution.
Two Problems with Current Benchmark Automation
The first problem is scope. Automated benchmark tools are typically specialized to a single environment, task family, or resource type. ProcTHOR generates scenes. MimicGen expands robot demonstrations. Eureka designs rewards. Each reduces costs within its niche, but none transforms a user's evaluation intent into a complete benchmark package across heterogeneous resources. An embodied benchmark might consume images, simulator states, trajectory logs, and environment configurations, while producing grounded questions, task instructions, scoring protocols, and executable verifiers. The gap between raw resources and finished artifacts still requires substantial manual configuration.
The second problem is process reliability. Benchmark construction is a pipeline of dependent operations: source processing, evidence grounding, state structuring, task synthesis, answer derivation, verifier construction. Without artifact-specific verification at each stage, a defect in an early artifact silently corrupts everything downstream. Final-stage inspection can catch some invalid items, but it wastes the computation that produced them. Complete regeneration discards artifacts that were already verified. Neither approach is satisfactory when the pipeline is long and the artifacts are heterogeneous.
Closed-Loop Benchmark Synthesis
Embodied-BenchForge formulates construction as Closed-Loop Benchmark Synthesis, which couples forward artifact synthesis with backward verification and repair. The framework has two main components.
Skill-Orchestrated Artifact Synthesis represents construction operations as typed, reusable skills. Each artifact is a four-tuple: type, content, metadata with verification state, and provenance. Each skill specifies input and output artifact types, execution preconditions, and a backend that can be a model, program, simulator interface, or verifier compiler. Skills decompose hierarchically into reusable subskills. The framework selects and composes skills into executable workflows based on the evaluation intent, available resources, and target artifact types.
The forward construction path flows through five stages: Intent Blueprinting converts the user's evaluation intent into capability targets, resource requirements, and task specifications. Data Collection accesses heterogeneous resources through unified interfaces. Evidence and State Structuring transforms raw resources into registered evidence records with spatial, semantic, and temporal information. Benchmark Synthesis generates questions, answers, task instructions, action interfaces, goals, and verifiers. Evaluation Reporting summarizes results.
Requirement-Guided Verification and Repair is the backward loop. Each artifact type has a requirement contract specifying structural, grounding, semantic, and execution requirements. When verification fails, the framework traces the failure through the artifact dependency graph and triggers one of several repair operators: local field repair, template re-binding, candidate re-synthesis, skill re-execution, or upstream rollback. Only affected downstream artifacts are reconstructed; unrelated verified artifacts are retained.
The quality gate process is specific to each track. For Offline EQA, it includes invalid item screening and image-dependency blind evaluation (checking whether solving an item actually requires its visual evidence, not just text). For the Interactive track, it adds task validity checks, task completion tests that execute the task and verify the terminal state, and initial-state restoration checks. Both tracks undergo typical model evaluation and Item Response Theory (IRT) diagnosis to assess item difficulty and discrimination.
Seven Benchmarks Across Two Tracks
The framework constructs two evaluation tracks. The Offline EQA Track contains six benchmarks covering diverse embodiments:
- SpatialHome-Task (SHT): Household robot scenarios from ALFRED simulator
- SpatialHome-Nav (SHN): Mobile robot navigation from Habitat simulator
- SpatialDrive (SD): Autonomous vehicle scenarios from CARLA simulator
- SpatialArm (SA): Robotic arm manipulation from LIBERO simulator
- SpatialUAV (SU): Drone scenarios from real aerial images
- SpatialQuadruped (SQ): Quadruped robot scenarios from TartanGround
Each Offline EQA instance combines embodied observations with grounded questions, reference answers, and executable scoring protocols. The six benchmarks span capability dimensions, question types, and expert templates, with SA (robotic arm) containing 203 expert templates and SU (UAV) containing 260.
The Interactive Embodied Track contains 220 instruction-conditioned tasks constructed from AI2-THOR scenes across kitchens, living rooms, bedrooms, and bathrooms. Each task specifies a restorable initial state, an instruction, an action interface, goal conditions, and an executable terminal verifier, while retaining trajectories for trajectory-grounded question answering.
Quality Assessment and Model Evaluation
Artifact quality was assessed by four LLM/VLM judges and ten human annotators across eight dimensions. The six OE-Track benchmarks scored consistently high, with Overall scores ranging from 88.34 to 90.75. The small variation across heterogeneous data sources indicates stable construction quality. The weakest dimension was target-signal dependency (checking whether items rely on intended visual evidence rather than shortcuts), scoring 86-91 across benchmarks.
The benchmarks show clear model discrimination. On Offline EQA, humans achieved a mean score of 85.89, outperforming the strongest model (GPT-5.5 at 57.67) by 28.22 points. GPT-5.5 ranked first among models, followed by Claude Opus 4.7 at 56.61 and Qwen3.6-35B at 49.86 as the strongest local model. The gap between humans and models indicates substantial room for improvement in embodied understanding.
On the Interactive track, the gap between understanding and execution is stark. GPT-5.5 achieved 83.18% task success rate, 73.64% EQA accuracy, and 61.36% combined success-rate-and-EQA metric. Claude Opus 4.7 reached 77.73% success rate. Several smaller models retained moderate EQA accuracy (above 50%) despite near-zero task success, revealing that trajectory understanding does not guarantee executable embodied competence. Humans achieved 97.73% success rate, showing the tasks are feasible but non-trivial.
Cross-track diagnosis shows complementary failure patterns. Offline failures concentrate on structured answer binding, multi-view consistency, and temporal reasoning. Interactive failures concentrate on spatial navigation, repeated execution errors, recovery failures, and action ordering. The two tracks measure different aspects of embodied intelligence.
Construction Efficiency and Skill Reuse
Embodied-BenchForge constructs a 10,000-item OE-Track benchmark in 38 to 160 minutes (86 minutes average). Throughput averages 116.3 items per minute. Constructing and validating 20 executable IE-Track tasks takes 1.5 hours. OE-Track orchestration averages 11.45 million tokens, or 1,145 tokens per accepted item.
Skill reuse is high. The four simulator-based OE benchmarks reuse 97.5% of their invoked modules and require only one source-specific adapter on average. SpatialUAV and SpatialQuadruped introduce additional modules for real-image and dataset-specific processing, but maintain reuse rates above 81%. Overall, 91.6% of invoked modules are reused across all benchmarks, demonstrating that the shared skill backbone supports heterogeneous construction with limited source-specific extensions.
Ablation Studies: What Matters and What Costs
System-level ablation across the six OE-Track benchmarks shows that removing verification and repair causes the largest degradation, dropping human quality scores by 23.25 points and the valid rate from 93.7% to 62.4%. Source grounding and evidence construction are also critical, causing human-score drops of 19.16 and 17.20 points respectively. The reduction in Judge-Human agreement from 0.87 to 0.78 when verification is removed indicates that incomplete pipelines produce ambiguous artifacts.
Fine-grained ablation on SpatialUAV isolates the contribution of each quality control component. Final-stage inspection alone yields only 10.4% valid items and 14.6% high-discrimination items. Removing requirement contracts drops quality by 10.47 points and valid rate by 31.5 percentage points. Without provenance-guided repair, quality remains similar but token cost rises from 2.62M to 3.93M for the same 1,000-item output, because the system cannot localize failures and must re-synthesize more broadly. The full process achieves the best balance across quality, validity, diagnostic value, and construction cost.
These numbers tell a practical story. Process-level verification is not a luxury. Without it, the majority of synthesized items fail basic requirements, and the ones that pass are less diagnostically useful. The investment in verification and repair pays for itself in item quality and construction efficiency.
Limitations and Trade-offs
The framework has several constraints worth noting. The construction workflow uses Qwen3.6-27B for orchestration and GPT-5.5-Pro for artifact synthesis and model-assisted verification. The quality of generated benchmarks depends on these underlying models. The ablation study shows that removing design skills, source grounding, or evidence construction causes significant quality drops, which suggests the pipeline is sensitive to the quality of its intermediate steps.
The benchmark suite covers six embodiment types and one interactive setting, but all environments are simulated. Real-world deployment would introduce noise, sensor failure, and physical constraints that simulators approximate but do not capture. The UAV benchmark uses real aerial images, which is a step toward real-world grounding, but the question generation and verification pipeline still operates in a simulated reasoning framework.
The token costs are non-trivial. At 11.45M tokens per 10,000-item benchmark and 1,145 tokens per accepted item, constructing large-scale benchmarks at scale requires significant API budgets. The efficiency improvements from skill reuse help, but the per-item cost remains higher than purely procedural generation methods, which sacrifice the quality guarantees that verification provides.
The IE-Track contains 220 tasks across four scene types. This is sufficient for evaluation but small for training or fine-tuning. Scaling the interactive track to thousands of tasks would require the construction pipeline to handle more diverse action spaces and goal conditions, which the paper does not demonstrate.
What This Means for Embodied AI Evaluation
Embodied-BenchForge demonstrates that benchmark construction can be automated end-to-end without sacrificing quality, provided the pipeline includes process-level verification and repair. The 91.6% skill reuse rate across heterogeneous resources suggests the typed-skill abstraction generalizes across different simulators, data sources, and task types. The artifact dependency graph with provenance tracking enables localized repair, which is more efficient than regenerating entire benchmarks when defects are found.
The cross-track diagnosis results are perhaps the most practically useful finding. The offline and interactive tracks measure genuinely different capabilities, and models that perform well on one do not necessarily transfer to the other. Smaller models with moderate EQA accuracy but near-zero task success illustrate this gap clearly. Any evaluation of embodied AI should ideally cover both observation-based understanding and closed-loop execution, and the framework makes constructing both types of benchmarks feasible from a single workflow.
For practitioners building embodied AI systems, the implication is that benchmark construction should not be a one-time manual effort. The framework is designed to be continually updatable: new resources, new task types, and new evaluation intents can be processed through the same skill library and verification pipeline. As embodied AI environments and capabilities evolve, the benchmarks can evolve with them.