Audio-driven talking head synthesis has a structural mismatch at its core. Speech features extracted by ASR models like DeepSpeech or HuBERT are excellent at capturing temporal acoustic patterns, but they describe nothing about facial geometry. They do not say which parts of the face should move, how much, or in what direction. When you plug these features directly into a 3D deformation model, the result is mouth motion that tracks phonemes reasonably well but produces weak expressions, local artifacts, and unstable geometry in regions where the audio signal provides no spatial guidance.
This paper addresses that mismatch with a two-pronged approach: use predicted facial landmarks to inject spatial structure where it is missing, and build that structure into the 3D Gaussian Splatting representation that has become the efficient alternative to NeRF for real-time rendering.
The 3DGS Foundation and Why It Is Not Enough
NeRF-based talking head methods improved view-consistent rendering and pose control over GAN-based 2D approaches, but their training and rendering costs limit real-time use. 3D Gaussian Splatting (3DGS) replaces volumetric ray-marching with a set of 3D Gaussians projected onto the image plane, achieving comparable visual quality at a fraction of the compute. Recent 3DGS talking-head methods like TalkingGaussian, Gaussian Talker, and DEGSTalk have pushed fidelity further.
But the driving signal remains speech, which is temporally rich and spatially empty. When a 3DGS model deforms Gaussians based on audio features alone, it lacks explicit correspondence between acoustic patterns and facial regions. Rapid lip movements and expression changes produce local artifacts because the model has no spatial anchor to tell it where deformation should concentrate. The authors frame this as an under-constrained problem: audio features provide temporal alignment but no facial layout, so the mapping from sound to 3D point displacement is ambiguous across regions.
Spatial Enhancement: Landmarks as Structural Priors
The first contribution is a facial-keypoint-guided spatial enhancement module. A pretrained audio-to-landmark network predicts 68 3D facial keypoints from the audio sequence for each frame. These landmarks are not used directly to drive the face. Instead, they serve as structural cues for enriching the spatial point representation that the 3DGS model operates on.
The mechanism works through cosine similarity in a shared feature space. Both the spatial 3D points and the 68 predicted keypoints are encoded by a tri-plane hash encoder. For each spatial point, the maximum cosine similarity to any keypoint is computed. If that maximum exceeds a threshold (set to 0.6), the point is selected for enhancement, and the feature of its most similar keypoint is added to the point's own feature vector. Points below the threshold retain their original features unchanged.
The threshold controls a selectivity-coverage tradeoff. At 0.4, nearly half of all spatial points are enhanced, reducing the module's ability to focus on structurally important regions. At 0.8, only a small fraction of points are selected, leaving important facial areas uncovered. At 0.6, approximately 20% of points are enhanced, covering the mouth region and its neighborhood while maintaining selectivity. The ablation confirms this: LMD (landmark distance) drops from 2.41 at theta=0.4 to 2.28 at theta=0.6, then rises to 2.35 at theta=0.8.
The result is that expression-sensitive regions like the mouth, eyes, and surrounding areas receive stronger modeling emphasis, while the rest of the face retains its baseline representation. This is a lightweight operation: it adds keypoint features to selected points without restructuring the 3DGS pipeline.
Landmark Compensation: Global Structural Correction
Local enhancement alone cannot capture coordination between facial regions. When a subject exhibits large expression changes or has a facial shape that differs from the training distribution, point-wise matching may produce inconsistent deformation. The second contribution addresses this with a global landmark compensation mechanism.
The encoded features of all 68 landmarks in a frame are concatenated and passed through an MLP, producing a global conditioning vector. This vector is broadcast to every spatial point and concatenated with its local feature. The fused feature goes to two independent MLPs: one predicts fine-grained offsets for 3D Gaussian attributes (translation, rotation, scale), and the other predicts a point-specific scaling factor that adaptively controls compensation strength. The final offset is the MLP output multiplied by this factor.
This design avoids two failure modes. An unconstrained global compensation could introduce unstable motion. A fixed global coefficient could be too weak for some facial regions and too strong for others. The per-point scaling factor lets the model apply compensation where it helps and suppress it where it does not.
The ablation shows this component contributes more to landmark accuracy than the spatial enhancement module. Removing LC increases LMD from 2.28 to 2.52, a larger jump than removing SE (2.28 to 2.45). This makes sense: local enhancement improves spatial point representation, but global compensation provides the structural coherence that prevents individual regions from drifting independently.
Audio Representation: Replacing DeepSpeech with WavLM
The audio branch replaces DeepSpeech, the standard in prior work, with WavLM, a self-supervised speech model. The raw WavLM features are high-dimensional, so a lightweight encoder compresses them: a sliding window of size 2 captures local acoustic transitions, four stacked 1D convolutional layers reduce dimensionality, and a linear layer projects to a compact representation.
The ablation shows this choice primarily affects lip synchronization rather than visual quality. Replacing the proposed Audio Net with DeepSpeech changes LMD and LPIPS only slightly (2.36 vs. 2.28, 0.0135 vs. 0.0128) but degrades LSE-D and LSE-C more clearly (7.663 vs. 7.150, 7.443 vs. 7.900). Removing the Audio Encoder entirely also weakens synchronization, confirming that WavLM features require temporal compression before use in facial motion prediction.
Geometry Regularizer: Stability Through Depth and Normals
Sparse viewpoints and limited training frames can cause 3DGS to learn unstable geometry in unseen views. A human geometry estimator provides depth and surface-normal maps from real images, and the model is regularized to match these from its generated outputs. The loss combines a scale-invariant depth loss with a normal consistency term, both weighted at 1e-4.
This component does not produce large numerical gains on the metrics. Its contribution is visual: the ablation visualization shows it reduces local facial artifacts and makes generated faces more natural. In practice, this is a regularization term that stabilizes training rather than a feature that changes the model's representational capacity.
Quantitative Results: Self-Driven and Cross-Driven
In the self-reconstruction setting, the method achieves the best PSNR (38), LPIPS (0.0128), and LMD (2.28), while obtaining competitive lip-sync metrics. Wav2Lip achieves a higher LSE-C (8.745 vs. 7.900), but it uses SyncNet as an expert discriminator during training, which biases that metric upward. The LMD improvement over the next best (DEGSTalk at 2.66) is substantial: a 14.3% reduction in landmark distance.
In the cross-driven setting, where a trained model is driven by audio from other speakers, the method achieves the lowest LSE-D and highest LSE-C on both audio clips (LSE-D: 9.091 and 8.981; LSE-C: 5.736 and 5.506). This is the more meaningful evaluation for real applications, since a talking head system must generalize beyond the specific utterances seen during training. The gap over DEGSTalk (LSE-D: 9.650 and 9.482) shows the landmark guidance provides structural information that helps the model generalize to unfamiliar audio.
What This Means for Talking Head Systems
The core insight is that audio-driven facial synthesis benefits from explicit spatial structure injection, even when that structure comes from a separate pretrained model rather than being jointly learned. The landmark prediction network provides facial layout information that audio features lack, and the spatial enhancement and compensation modules translate that layout into 3DGS deformation guidance.
The architecture is lightweight enough for practical deployment. Training takes about one hour on a Tesla T4. The Audio Net is compact. The spatial enhancement and compensation modules add minimal parameters. The frozen 3DGS backbone from TalkingGaussian handles the heavy rendering work.
The limitation is that the quality of spatial guidance depends entirely on the pretrained audio-to-landmark model. If that model produces inaccurate landmarks for a speaker, the enhancement and compensation will propagate that error. The threshold sensitivity analysis suggests this is manageable within a reasonable range, but the dependency on a separate model's accuracy is an architectural constraint worth noting.
The dataset is small: three speakers (two male, one female) from public video datasets. The cross-driven evaluation uses only two audio clips from the Obama dataset. The method has not been tested on diverse ethnicities, ages, or speaking styles at scale. Whether the landmark guidance generalizes to faces with very different proportions or expression patterns remains an open question.