Deploying robots in homes, hospitals, and offices requires them to understand what a human actually wants, and human instructions are rarely precise. A person might point vaguely toward a bottle on a table while saying "put this there," but the pointing direction is noisy, the gesture is ambiguous, and the intended target might not lie exactly along the line of the finger. Robots, unlike humans, cannot effortlessly combine a sloppy gesture with surrounding context to infer the right object. The paper MIGU: Multimodal Instruction Grounding under Uncertainty for Manipulation Planning, by Mingke Lu, Anxing Xiao, and David Hsu at the National University of Singapore, tackles this problem head-on by building a system that explicitly models uncertainty from every source, language, gesture, and geometry, and uses that model to decide whether to act or ask for clarification.
The core problem: why pointing alone is unreliable
Human pointing gestures are deceptively imprecise. The paper identifies three distinct sources of uncertainty that compound when a robot tries to interpret a pointing gesture. First, there is viewing-direction uncertainty: the camera sees the hand from a particular angle, and small errors in locating hand landmarks translate into angular errors in the estimated pointing direction. Second, there is depth uncertainty: the distance to the intended target is inferred from depth maps, and depth perception has inherent noise that grows with distance. Third, there is hand-direction estimation error: the system must infer where the finger is pointing from landmark positions, and this estimation introduces its own error independent of the geometric noise.
Humans resolve these ambiguities by combining the gesture with language and visual context. If someone points vaguely toward the kitchen while saying "give me the red one," the combination of pointing direction and the word "red" narrows the candidate objects. A robot that only uses one modality, however, is stuck: a geometric-only system may assign high probability to multiple objects along the pointing cone, while a language-only system may lack the spatial precision to distinguish between two similarly colored bottles on different shelves.
How MIGU models geometric uncertainty
The geometric component of MIGU constructs a principled 3D likelihood over candidate target locations. The framework starts from the observed pointing direction and the detected hand landmarks, then propagates the known sources of uncertainty through the geometry of the eye, hand, and target.
Specifically, the system models the viewing direction and depth as independent Gaussian random variables. The angular uncertainty is set to σ_u = 0.044 radians, based on visual localization standards from prior work, while the relative depth-noise coefficient is α_d = 0.04, meaning the depth uncertainty scales linearly with distance. The system then computes the vector from the fingertip to each candidate 3D point in the scene and projects the combined uncertainty onto the tangent plane of the sphere at the nominal pointing direction.
The result is a closed-form geometric likelihood: for each candidate point, the system computes the probability that the observed pointing direction would be produced given that the target is at that point. This is done using the logarithmic map on the sphere to compute the angular residual between the observed direction and the candidate-induced direction, then evaluating it against the projected covariance. The key insight is that uncertainty grows with distance: a target twice as far away produces a wider cone of possible pointing directions, and the geometric model captures this naturally.
How MIGU models semantic uncertainty
For the semantic component, MIGU uses a vision-language model (VLM) to process the language instruction and the RGB-D image jointly. The authors use GPT-5.6 Sol for its state-of-the-art visual reasoning capabilities. The VLM is prompted to identify N candidate bounding boxes in the scene, each corresponding to a hypothesis about what the instruction might refer to. For each hypothesis, the VLM provides a confidence score and a 2D bounding box.
A practical design choice matters here: rather than trying to calibrate the VLM's confidence scores (which would require a large calibration dataset), the authors use the confidence as an unnormalized semantic prior. Each bounding box is passed to a class-agnostic segmentation model (SAM) to produce a pixel-level mask, which is then converted to a set of 3D scene points via RGB-D correspondence. The semantic belief distributes the VLM's confidence uniformly over all 3D points inside the mask, avoiding premature commitment to a particular point.
Bayes-inspired fusion: combining what the eyes see and what the language says
The fusion step is where MIGU's modular architecture pays off. Under the assumption that the semantic evidence and the observed gesture are conditionally independent given the intended target, the fused score for each candidate point is simply the product of the geometric likelihood and the semantic score. For each hypothesis (candidate object), the system aggregates the pointwise fused scores using max pooling, retaining the point that best agrees with the observed gesture. The resulting hypothesis scores are normalized into a probability distribution over candidates, forming the unified grounding belief.
This belief is not just a winner-take-all prediction. It preserves the relative probabilities across candidates, which is crucial for the next step: deciding whether to act or ask for clarification.
Behavior planning: act now or ask a question
With the grounding belief in hand, MIGU formulates the execute-or-clarify decision as a one-step belief-space problem. The robot considers two options. The first is direct execution of the highest-probability hypothesis. The expected value is simply the probability of that hypothesis multiplied by the reward for successful task completion. The second option is to ask a clarification question, which partitions the candidate set based on a visual attribute such as color, size, shape, or relative position.
The system evaluates every possible attribute-based query, computes the expected value of each after receiving the (binary) user response, and selects the query with the highest expected value. If the best query's expected value exceeds the value of direct execution, the robot asks the question. Otherwise, it proceeds. Importantly, only a single clarification round is permitted, reflecting real-time interaction constraints. The authors acknowledge this as a practical limitation, noting that multi-round dialogue could further refine the belief but would violate latency requirements.
Real-world results: 83% accuracy, improved to 91.7% with clarification
The authors evaluate MIGU on a benchmark of 100 multimodal instruction-grounding scenarios collected in real-world daily-life settings, spanning semantic ambiguity, geometric ambiguity, and their combination. Interaction distances range from short-range tabletop settings (0.2 to 0.5 meters) to medium-range settings (0.5 to 3 meters). The benchmark measures object-level accuracy, top-2 accuracy, scaled 2D grounding deviation, negative log-likelihood, and Brier score.
MIGU outperforms every baseline tested: a pure geometric method, the Touch-Line Transformer, LEGS-POMDP, GIRAF, and three pure VLM variants using different prompting strategies (Set-of-Mark, image grid, and iterative visual proposal). MIGU achieves 83% object-level accuracy and 95% top-2 accuracy. Its scaled 2D grounding deviation is just 0.0592, far lower than the next best methods, which is critical for geometrically sensitive tasks like placing objects precisely or wiping surfaces.
The probabilistic metrics tell an equally compelling story. MIGU achieves the lowest negative log-likelihood (1.17) and Brier score (0.25), outperforming LEGS-POMDP by 22.5% and 60.3% respectively on these metrics. This means MIGU's belief distribution is not only more accurate in its top prediction but also better calibrated overall, providing a more reliable basis for the execute-or-clarify decision.
The ablation studies are particularly revealing. Removing the semantic component drops accuracy to 51%, and removing geometry drops it to 46%, confirming that both modalities are essential. Replacing the proposed 3D geometric model with simpler alternatives like ray-distance scoring (76%) or ray-angle scoring (58%) degrades performance, supporting the value of the principled uncertainty propagation. Among semantic strategies, directly querying the VLM outperforms using an external detector with SoM prompting (77%), Grid prompting (40%), or PIVOT prompting (42%), with the full MIGU system achieving 83%.
From grounding to manipulation: mobile and tabletop tasks
The grounding belief is not an end in itself but an input to downstream manipulation planning. MIGU integrates with two robotic systems to demonstrate this connection.
On an Autolife S2 wheeled humanoid robot equipped with omnidirectional base, a 4-DoF waist, and two 7-DoF arms, MIGU is tested on eight real-world mobile manipulation tasks. The results show a clear progression: semantic-only grounding achieves 45.8% planning success, geometry-only achieves 41.7%, the full MIGU system without clarification achieves 83.3%, and enabling clarification brings the success rate to 91.7%, an improvement of 8.4 percentage points. The clarification mechanism proves especially valuable when the fused belief remains ambiguous: in one example, the robot faces multiple bottles with similar semantic priors and noisy pointing, and the two leading candidates retain probabilities of 0.54 and 0.41. The robot asks a clarification question about color, resolves the ambiguity, and successfully plans a whole-body grasp.
On a Fetch robot in cluttered tabletop scenes, MIGU is integrated with PDDLStream for task-and-motion planning. The system handles both object selection and region specification: in one task, the robot receives an instruction to retrieve a bottle from behind a blocker and plans the intermediate step of moving the obstacle first; in another, it asks a spatial clarification question about which apple to pick up. These examples demonstrate that the grounding belief feeds naturally into symbolic planning, supporting complex multi-step manipulation.
Limitations and trade-offs
The authors are transparent about three practical design compromises. First, the system relies on the VLM's commonsense reasoning, assuming that semantic cues do not severely bias the geometric evidence in a harmful way. If the VLM is strongly but incorrectly confident about a candidate, the fused belief could be skewed. Second, the semantic confidence is treated as a relative prior rather than a calibrated probability, since reliable calibration would require a large in-domain calibration dataset. Third, clarification is restricted to a single binary round, meaning the system cannot engage in iterative dialogue to progressively narrow down the correct interpretation.
The code is listed as "Coming Soon" on the project website, meaning the full implementation has not yet been released at the time of writing. The system also depends on GPT-5.6 Sol for semantic grounding, which introduces a dependency on a proprietary model and may raise latency and cost concerns in deployment.
What this means in practice
MIGU's contribution is a modular, principled framework for the specific problem of grounding human multimodal instructions in physically meaningful ways. For roboticists building assistant robots for homes or hospitals, the system provides a template: construct geometric uncertainty from pointing data, use a VLM for semantic priors, fuse them via Bayes-inspired combination, and let the resulting belief drive both the execute-or-clarify decision and downstream manipulation planning.
The explicit uncertainty modeling is the key differentiator. Prior systems either treated grounding as a classification problem (pick the most likely object) or used ad hoc geometric scoring without principled uncertainty propagation. MIGU's closed-form uncertainty model, combined with its behavior planner, gives the robot not just a guess but a quantified confidence that can be acted upon or queried about. The 8.4 percentage point improvement from clarification shows that the model knows when it does not know, and that knowing when to ask is as important as knowing what to do.
As robots move into environments where instructions are naturally imprecise, the ability to reason about uncertainty, not just classify targets, will separate systems that work in the lab from systems that work in the real world. MIGU makes a compelling case that explicit probabilistic modeling of multimodal uncertainty is the right approach.