Audio-visual navigation asks an embodied agent to find a sound source in a 3D environment using only what it can see and hear. The task sounds simple until you consider what the agent actually perceives. Its visual field is narrow and self-centered, lacking global context. Transparent obstacles like glass doors produce false passability signals. Abnormal lighting obscures geometric details. Traditional approaches that map raw inputs directly to actions place blind trust in this flawed perceptual data, leading to planning failures when the map lies.
The physical world offers a corrective: collisions. When an agent walks into a wall it did not see, that collision is reliable geometric information. But most systems treat collisions as failures to recover from, not as data to learn from. TDGP (Transformer-based Token Fusion and Dynamic Graph Planning) takes a different stance. It uses collisions as an active planning signal, removing edges from the navigation graph in real time and penalizing the agent for selecting paths that lead to physical contact. The result is a system that adapts its own map when vision fails, rather than waiting to撞 into obstacles and hoping to recover.
Decoupling Perception from Control
The architecture splits navigation into two layers. The high-level Tokenization Fusion Mechanism (TFM) identifies a local navigation target on a 3x3 grid surrounding the agent. The low-level Collision-Penalty Path Planner (CPP) translates that target into atomic actions by computing shortest paths on a dynamic navigation graph. This decoupling means the high-level policy focuses on direction selection while the low-level planner handles path execution and obstacle avoidance, a separation that the ablation study confirms is critical: removing CPP collapses success rate from 98.7% to 56.3% on Replica heard scenarios.
The high-level policy operates at every environmental time step, not at fixed intervals. This means the agent updates its local goal based on the latest maps and sounds continuously, rather than committing to a waypoint and following it to completion. The distinction matters in dynamic environments where the acoustic landscape shifts and the agent's own movement reveals new geometric information.
Transformer Token Fusion for Multimodal Alignment
The TFM module fuses visual and auditory features using a patch embedding strategy borrowed from ViT. Visual features from an RGB encoder and acoustic features from binaural audio spectrograms are each projected into token sequences through 2D convolutional layers with kernel size and stride P, producing tokens of dimension 128. A learnable [CLS] token is prepended, and the full sequence is fed through a standard Transformer encoder.
The multi-head attention mechanism enables deep information exchange between tokens of different modalities. This is the key difference from prior work, which typically concatenates visual and audio features and processes them through GRUs or simple attention. The ablation shows that replacing the Transformer fusion with concatenation drops success rate by 13.3% on Replica heard scenarios (from 98.7% to 85.4%) and by 4.7% on MP3D heard scenarios (from 73.7% to 69.0%). The cross-modal attention allows the model to learn which visual regions are relevant to specific audio patterns, rather than treating the two modalities as independent feature streams.
After the Transformer, the [CLS] token serves as the global context vector. A GRU processes this vector alongside the previous frame's hidden state, capturing temporal dependencies, and outputs a probability distribution over the 3x3 local grid. The agent selects the highest-probability cell as its immediate navigation target.
Collision-Penalty Path Planner
The CPP maintains a dynamic navigation graph G_t = (V, E_t), where V is the set of discrete navigable nodes and E_t is the set of valid edges at time t. When the high-level policy selects a target on the 3x3 grid, the planner transforms it from egocentric coordinates to global map coordinates, maps it to the nearest node, and runs Dijkstra to compute the shortest path. The next node along that path determines the atomic action based on relative azimuth with the agent's current orientation.
The critical mechanism is edge removal on collision. When the agent executes an action and receives a collision signal from the environment, the planner immediately removes the edge connecting the current node to the node the agent attempted to reach. The graph is updated before the next decision step: E_{t+1} = E_t minus the colliding edge. This is not a soft penalty or a learned cost. It is a hard removal. The navigation graph physically loses the edge, and Dijkstra will never plan through it again in that episode.
A collision penalty term in the reward function supplements this hard mechanism. The total reward combines a target reward, a proximity reward, and a collision penalty: R_t = r_g + r_s + r_c. The proximity reward incentivizes the agent to move toward the target while the collision penalty discourages paths that lead to physical contact. During training with PPO, the high-level policy learns to anticipate potential dangers and select safer targets, rather than relying solely on post-collision graph updates.
The authors acknowledge a limitation: in dynamic environments, a permanently removed edge may discard paths that become traversable again. The current system has no edge recovery mechanism. Time-decay edge recovery, which gradually reconsiders removed edges, is flagged as future work.
Audio Enhancement for Generalization
The audio augmentation strategy adds complexity to the acoustic environment during training. A noise source can be generated at a random non-target location and mixed with the target sound. A second sound source may also be mixed at the target location. Spectral masking applies temporal and frequency masks to the generated spectrogram. These augmentations are probabilistically activated at the start of each episode.
The effect on generalization is substantial. On MP3D unheard scenarios, adding audio enhancement improves success rate from 42.2% to 46.0% (a 3.8 percentage point gain). The comparison with DB-Nav, which is specifically designed for complex and moving sound sources, reveals an interesting tradeoff. DB-Nav achieves higher success rate on MP3D unheard (72.5% vs. 46.0%) but uses recurrent neural networks and complex acoustic memory banks to fit audio noise. TDGP's approach is architecturally simpler: the Transformer token fusion handles multimodal alignment and the collision-penalty planner handles geometric uncertainty, with augmentation providing acoustic robustness.
The gap between DB-Nav and TDGP on MP3D unheard suggests that the collision-penalty mechanism does not fully compensate for limited acoustic memory in complex noise scenarios. However, TDGP's architectural simplicity and faster convergence (it reaches higher SPL with fewer training steps than the SoundSpaces baseline) make it a practical alternative when acoustic complexity is moderate.
Experimental Results on Replica and Matterport3D
On Replica heard scenarios, TDGP achieves 89.2% SPL, 98.7% success rate, and 73.7% SNA (Success weighted by Normalized Agent-Action). These represent improvements of 14.8%, 7.3%, and 25.6% over the SoundSpaces baseline. On Replica unheard scenarios, the improvements are 7.0%, 9.1%, and 9.5%. On MP3D heard scenarios: 8.3%, 6.0%, and 22.9%. On MP3D unheard: 14.2%, 8.7%, and 15.5%.
The SNA metric, which weights success by the efficiency of the agent's actions rather than just path length, shows the largest gains. This is consistent with the collision-penalty mechanism's design: by penalizing inefficient actions that lead to collisions, the planner encourages the agent to select not just short paths, but paths that avoid wasted movement. The 25.6% SNA improvement on Replica heard is particularly notable, suggesting that the hierarchical decomposition produces qualitatively different navigation behavior, not just quantitatively better scores on the same behavior.
The ablation without TFM (replacing Transformer fusion with concatenation) shows that cross-modal interaction is essential for success rate but less critical for SPL when the agent does succeed. The ablation without CPP shows the opposite: the end-to-end system can achieve reasonable SPL when it succeeds, but its success rate collapses. This confirms that the two layers address complementary failure modes: TFM handles perceptual uncertainty (which direction to go) while CPP handles geometric uncertainty (how to get there without hitting obstacles).
Limitations and What Comes Next
The evaluation is entirely in simulation. Real-world deployment requires handling sensor calibration errors, actuator noise, latency, and dynamic obstacles, none of which the current system addresses. The collision-penalty mechanism assumes a static environment: once an edge is removed, it stays removed for the episode. In a real building where furniture moves or doors open and close, this assumption breaks.
The CPP relies on a projected 2D occupancy grid map, which is susceptible to depth-sensor noise. In large-scale, complex environments like MP3D, this noise can cause the planner to remove edges that are actually traversable, reducing SPL even when the agent eventually reaches the target. The authors note this as a tradeoff: CPP ensures success rate but may sacrifice path efficiency under noisy perception.
The audio augmentation strategy is straightforward (noise mixing and spectral masking) and does not model realistic acoustic phenomena like reverberation, occlusion, or Doppler shifts. More sophisticated acoustic modeling could improve generalization further, particularly in large environments where sound propagation is complex.
Despite these limitations, the core architectural insight holds: decoupling high-level perception from low-level planning, and using physical collisions as a geometric signal rather than just a failure mode, produces measurably better navigation under perceptual uncertainty. The collision-penalty graph update is simple to implement, requires no learned model of obstacle geometry, and adapts in real time to the agent's own discoveries. For developers building embodied agents that must navigate with imperfect vision, this is a practical and provably effective design pattern.