From a Single Image to a Complete 3D Room: How HARMONY Solves the Compositional Reconstruction Puzzle
Reconstructing a complete 3D indoor scene from a single monocular image remains one of the most stubborn problems in computer vision and graphics. The challenge is twofold. A single view captures only partial geometry, typically obscured by heavy inter-object occlusions. Beyond geometry alone, 2D images provide no explicit cues about exact object scales, and the spatial relationships between objects are completely invisible. For years, researchers have attacked this problem from two directions: agentic reasoning that provides rich semantic understanding but lacks precise alignment with the input image, and visual geometry models that predict dense point maps but struggle with reconstruction quality on complex scenes.
HARMONY, introduced by Shufan Sun, Chen Wang, Enxin Song, Jiatao Gu, and Lingjie Liu at the University of Pennsylvania, proposes a fundamentally different approach. Rather than choosing between semantic reasoning and geometric grounding, HARMONY marries them. The framework introduces a hierarchical chain-of-thought pipeline that uses a vision-language model (VLM) to reason about spatial relationships while relying on visual geometry signals to anchor every placement in precise metric space. The result is a system that can reconstruct semantically consistent and perceptually aligned 3D scenes from a single image, extending compositional reconstruction to complex indoor environments.
The Two-Pronged Problem and Why Prior Methods Fall Short
To appreciate HARMONY's contribution, it helps to understand why existing approaches fall short. Pure geometry-based methods decompose the scene by segmenting and reconstructing each object separately, then assembling them into a shared coordinate system using depth and point-cloud estimates. Relying solely on low-level perceptual cues without explicit reasoning over inter-object relationships, they often struggle with occluded and small objects, producing errors in object pose and relative placement. A chair might end up floating above the floor, or a lamp might be placed where a painting should be.
The alternative approach leverages VLMs for spatial reasoning. Text-conditioned methods exploit spatial reasoning to plan a hierarchical placement order from wall-aligned items to furniture, but because they operate purely in language and ground the scene through asset retrieval, they are restricted to scenes composed of simple objects and template-like arrangements. Image-conditioned VLM methods restore visual grounding and achieve strong high-level semantic alignment, but inherit the VLM's well-known weakness: they cannot perform precise visual-geometric reasoning, yielding inaccurate object placements and noticeable visual mismatch with the input. A VLM might correctly reason that a vase sits on a table but place it several centimeters off-center, or confuse two visually similar end tables.
HARMONY's insight is that these two approaches are not competing but complementary. The VLM excels at understanding what an object leans against, sits on, faces, or pairs with. Visual geometry models excel at determining exactly where in metric space an object belongs. The challenge is combining them without one overwhelming the other.
The Architecture of HARMONY: Four Stages of Structured Reasoning
HARMONY's pipeline proceeds through four distinct stages, each building on the output of the previous one. The backbone is a hierarchical chain-of-thought framework based on VLM, designed to decompose the monolithic problem of scene reconstruction into manageable reasoning steps.
Camera Calibration and Room Initialization
The pipeline begins with an empty 3D room box. The VLM first infers approximate room dimensions from semantic cues in the reference image, such as recognizing a bedroom is roughly (3,4,3) meters in scale. It then identifies the deepest visible room corner as a spatial anchor and labels each visible wall relative to this anchor, establishing which surface regions in the canonical mesh correspond to which image walls. This provides a semantically grounded spatial frame of reference.
The geometric refinement stage uses VGGT, a visual geometry foundation model, to estimate a Manhattan frame from the predicted point cloud. Through SVD-based clustering of surface normals, VGGT yields three mutually orthogonal axes corresponding to width, vertical, and depth directions. The room's six bounding planes are located along these axes. Camera calibration then solves in closed form for a single similarity transform that re-expresses the VGGT camera pose into the canonical room frame, using the room's known height as the sole external metric reference. This gives a precise camera projection that aligns with the input view.
Object Segmentation and Mesh Reconstruction
With the room frame established, HARMONY detects and segments objects hierarchically, processing one level at a time: wall-mounted items (paintings, windows), free-standing furniture and ceiling-mounted objects (desks, chandeliers), and finally decorations that rest on furniture. At each level, the VLM parses the reference image to list objects with per-instance counts, which are passed to open-vocabulary detection for bounding boxes and then to a segmentation model for masks.
For each detected object, the VLM produces a detailed description conditioned on the surrounding scene context. This description, together with the cropped object region, is passed to an image-editing model to inpaint occluded regions. A half-occluded table, for example, is described such that the model can generate a complete table compatible with the scene. The VLM then inspects the inpainted result and regenerates if needed with additional material and color hints.
Each object is then reconstructed into a 3D mesh using an image-to-3D generator. Since the inpainted views inherit the perspective of the input image, the resulting meshes are in non-canonical poses. HARMONY canonicalizes each mesh by applying Principal Component Analysis to its vertices and aligning its dominant axis with world-up. The VLM then inspects multi-view renders and labels each object's facing direction, assigning a per-object canonical frame that the placement stage uses to enforce correct relative orientations.
Hierarchical Scene Placement with Depth-First Traversal
The core innovation of HARMONY lies in how objects are placed into the 3D scene. Rather than placing all objects at once, the VLM reasons through placement in three ordered stages: wall-mounted items, free-standing furniture, and decorations. Within each stage, the VLM performs a depth-first traversal over visible objects ordered by proximity to the deepest room corner.
This depth-first strategy has a crucial geometric consequence: at the moment any object is placed, every previously placed object lies behind it from the camera's perspective. Each new candidate therefore appears in a clean, unoccluded view of the partial scene, allowing the VLM to reason about its orientation and pairing without any interference from visual clutter. The VLM assigns an initial size to each object from prior knowledge and reasons about per-object spatial relationships, such as "sofa back against the left wall" or "vase on the desk." Using each object's canonical front, it sets the orientation accordingly.
For decorations, the VLM identifies which previously placed piece of furniture supports each decoration and verifies it rests on the correct one. Because the VLM's semantic reasoning can confuse furniture that shares a category label, HARMONY includes a consistency check: when a vase is flagged as placed on the wrong table, the system triggers a bounding-box check comparing the decoration's box against candidate hosts and reassigns it to the host whose box it overlaps most.
Visual Geometry-Grounded Refinement
The VLM placements capture correct semantic relationships but only approximate scale and location. HARMONY refines each placement through two coupled geometric stages. The image-space silhouette fixes the object's lateral position and its scale-to-depth ratio from the apparent silhouette width, while the point cloud fixes the forward distance, which in turn resolves the metric scale.
For image-space alignment, the rendered object silhouette's horizontal center and width are aligned with its segmentation mask. The horizontal offset back-projects to a lateral translation. For objects not flagged as heavily occluded, the apparent angular size from the silhouette determines only the scale-to-depth ratio, leaving the metric scale to the depth stage. For depth-space alignment, HARMONY leverages the VGGT point cloud segmented with the per-object mask, extracts the robust front-surface means through iterative median with MAD outlier rejection, and corrects the depth by their displacement along the view direction. Combining the silhouette-derived angular size with the corrected depth yields the final metric scale.
The Reflective Feedback Loop
After each placement stage, HARMONY renders the updated 3D scene and sends it back to the VLM together with the input image. The VLM performs a reflective visual check for issues such as incorrect scale, inaccurate orientation, or misplaced items, and applies corrective actions before the pipeline proceeds. It also reasons over groups of visually matched objects to equalize their scale, such as inferring that chairs placed as a pair share a common size. After the decoration stage, the VLM counts the placed objects against the reference and fills any missing instance, either by reusing an existing mesh or regenerating one.
Results: State of the Art Across Every Metric
HARMONY is evaluated on two datasets: Front3D, which contains 100 synthetic images with 3D ground truth, and HARMONY30, which contains 30 real-world, copyright-free in-the-wild images spanning indoor scenes with varying layouts, styles, and lighting. The authors additionally construct HARMONY300, a broader benchmark containing 300 single-image indoor scenes. Evaluation metrics include Negative-CLIP score for semantic fidelity, photometric loss for pixel accuracy, and Chamfer Distance and F-score for geometry quality on Front3D.
On the Front3D dataset, HARMONY achieves the best score on every metric. Its N-CLIP score of 0.092 is the lowest (indicating highest semantic faithfulness), its photometric loss of 0.041 is the lowest, and its Chamfer Distance of 0.049 is the lowest. The F-score at threshold 0.1 reaches 89.82%, compared to 85.70% for the next best method, CAST. The geometry improvements are substantial: the F-score at threshold 0.001 reaches 0.130, far exceeding the 0.093 of CAST or 0.090 of GPT-6 Astra.
On real-world inputs from the HARMONY30 subset, HARMONY again leads on all perceptual metrics: N-CLIP of 0.112 versus 0.127 for GPT-6 Astra, and photometric loss of 0.045 versus 0.047 for GPT-6 Astra. In a user study with 16 participants across 20 scenes, HARMONY is ranked first in 67.0% of choices and in the top two in 86.0%, with a mean rank of 1.54. Every baseline is beaten: SAM3D ranks first in only 13.7% of choices, Gen3DSR in 11.3%, CAST in 3.3%, 3D-ReGen in 4.0%, and VIGA in just 0.7%.
The ablation study reveals how these gains are achieved. Removing camera calibration hurts perceptual similarity the most, driving N-CLIP from 0.1034 to 0.1381, confirming that metric grounding from dense image evidence is essential. Removing the reflective feedback loop drives N-CLIP to 0.1184, showing that error prevention at each stage is critical. Removing depth-first traversal and placement order both degrade performance, isolating the value of structured reasoning. Notably, removing VGGT refinement hurts both N-CLIP and photometric loss, confirming that semantic and geometric grounding are not redundant but complementary.
Limitations and Practical Considerations
The authors identify three common failure modes. The first is rare-type synonym mismatch: for rare object types, a vocabulary mismatch between the VLM and the detector affects detection quality, with 25.9% of detections carrying low grounding confidence. However, genuine placement failures after routing and de-duplication remain rare at only 0.3% of objects, and these issues can be resolved using stronger foundation models within the same framework.
The second failure mode involves heavy occlusion. When an object sits very close to the camera and is largely clipped, the VLM still recovers the correct semantic relationship, but the reconstructed mesh is not well constrained from the render viewpoint. This is an inherent limitation of monocular reconstruction.
The system also depends on the quality of the underlying image-to-3D generator and the VLM's reasoning capabilities. As these foundation models improve, HARMONY's performance is expected to scale accordingly, since the framework is designed to be modular with respect to its component models.
What This Means in Practice
HARMONY's significance extends beyond benchmark numbers. The framework demonstrates that compositional 3D scene reconstruction from a single image can achieve both semantic fidelity and geometric precision through structured hierarchical reasoning, without requiring manual 3D asset curation or text-only descriptions. The depth-first traversal strategy solves a practical problem that had been overlooked in prior work: allowing the VLM to see each object in isolation, free from visual clutter, dramatically improves placement accuracy.
The reflective feedback loop introduces a form of self-correction that is particularly important for complex scenes where errors in early stages propagate catastrophically. By checking after each placement stage, HARMONY catches and corrects errors before they compound, a design principle that could generalize to other multi-stage reasoning systems.
The paper also demonstrates practical downstream applications. HARMONY300, the broader benchmark, enables evaluation of scene reconstruction across diverse room types and layouts. The reconstructed scenes support physical simulations and robot interaction tasks, pointing toward real-world applications in AR/VR content creation, embodied AI planning, and interactive scene editing.
The project page at http://cwchenwang.github.io/harmony provides further details, and the broader HARMONY300 dataset offers a standardized benchmark for future work in single-image compositional 3D reconstruction.
The Bottom Line
HARMONY solves a long-standing tension in compositional 3D reconstruction: the gap between semantic understanding and geometric precision. By structuring the reasoning process hierarchically, using depth-first traversal to give the VLM clean views of each object, grounding every placement in metric image evidence, and inserting reflective feedback loops to prevent error accumulation, the framework achieves state-of-the-art results on both synthetic and real-world inputs. It is the first method to simultaneously lead on semantic fidelity, pixel accuracy, and geometric precision across all evaluated metrics, and the user study confirms that human judges consistently prefer its output over all baselines including GPT-6 Astra. For anyone working on single-image 3D scene reconstruction, HARMONY establishes a new benchmark for what is possible when semantic reasoning and geometric grounding are designed to work together.