Language models have grown surprisingly capable at following instructions, yet their ability to reason about three-dimensional spaces remains fragile. Interpreting spatial tasks in 3D scenes requires programs to construct geometric references that natural language leaves implicit. The paper VoxProg: 3D Spatial Reasoning through Voxel Programming addresses this gap by framing 3D object placement as voxel programming.
VoxProg translates task interpretations into executable programs over local voxel views of scene surfaces. These views index surface geometry on regular grids with explicit coordinates and neighborhood relations, allowing an evidence program to isolate task-relevant regions and boundaries directly from scene meshes. A downstream pose program then measures the resulting geometry to compute a target SE (3) pose.
What makes the approach distinctive is the feedback loop. Visual and numerical feedback from execution lets the model inspect intermediate selections and candidate placements, enabling iterative revision of its code. This inspection capability means the model can detect when a placement misses a geometric constraint and adjust its program accordingly, rather than committing to a single flawed interpretation.
VoxProg-Bench: a 71-task benchmark
To evaluate open-ended placement across functional use, geometric fit, and contextual arrangements, the authors introduce VoxProg-Bench, a benchmark comprising 71 tasks. The evaluation suite pairs perceptual Intent Alignment—measuring whether the placed object satisfies the user's intent—with 3D Physical Validity, which checks whether the placement respects collision constraints and physical plausibility. These two metrics together capture both the semantic correctness and the mechanical feasibility of each placement.
Across VoxProg-Bench and external benchmarks, VoxProg consistently outperforms relation-based, keypoint-based, and mesh-based baselines. The voxel representation provides a more structured geometry than free-form relations, while retaining explicit coordinates that keypoint methods often lack. Meanwhile, the programmatic format allows for revision and inspection, unlike opaque mesh-based approaches.
Limitations and practical considerations
The voxel grid resolution imposes a trade-off: finer grids capture more detail but increase computational cost, while coarser grids risk losing thin or delicate geometry. The evidence program's ability to isolate task-relevant regions depends on the quality of the initial scene mesh; poorly reconstructed surfaces limit the downstream pose program's accuracy. Additionally, the current framework focuses on placement tasks; extending the pipeline to more complex manipulation sequences would require hierarchical program synthesis.
Takeaways for developers
For teams working with 3D scene understanding or robotic manipulation, VoxProg offers a concrete framework for grounding language instructions in volumetric geometry. The key insight—that explicit voxel coordinates enable both evidence extraction and pose computation—can be adopted by replacing implicit geometric heuristics with programmatic voxel queries. The iterative feedback loop also provides a pattern for building self-correcting systems that can reason about their own mistakes.