Dance Videos Give Sparse Cues. Music Needs Dense Structure. How Do You Bridge That Gap?
Dance-to-music generation asks a deceptively simple question: given a video of someone dancing, synthesize music that matches. The difficulty is not the modality switch but the information density mismatch. Dance videos provide only sparse signals, rhythm timing and movement style. Music requires dense information: global structure, melodic and harmonic phrases, rhythmic patterns, instrument timbre, and expressive dynamics. Existing systems rely on these sparse dance cues and supervise only the final audio output. The generator must simultaneously learn rhythmic alignment and music composition from reconstruction loss alone. The result is music that often lacks structural coherence, musicality, and expressive richness.
A team from HKUST(GZ) and Tencent, led by Jinting Wang and corresponding author Li Liu, proposes CMA-OT (Curriculum-guided Multi-scale representation Alignment with scale-aware Optimal Transport). The core idea is to bring in an external music expert that provides hierarchical supervision at multiple semantic levels, then progressively transfer that knowledge to the generator through a curriculum, using optimal transport to handle the alignment between heterogeneous feature spaces. Experiments on AIST++ and TikTok datasets show state-of-the-art performance across rhythmic synchronization, perceptual quality, and overall music generation.
Why Existing Methods Struggle
Prior dance-to-music systems follow a common pattern. They extract motion features from dance videos using pre-trained encoders like STGCN or pose-based rhythmic cues, then train a music generator through reconstruction objectives. This works for capturing global motion semantics and basic rhythmic alignment, but it leaves a fundamental gap. The model gets rhythm and style information from the dance but must invent all musical structure from scratch, with no explicit supervision on what good music looks like internally.
The consequence is visible in the output. Generated music tends to be rhythmically approximate but structurally flat, lacking the hierarchical organization that makes real music coherent. Long-range form is weak. Melodic development is thin. Instrumentation lacks variation. These are not failures of the generator architecture but failures of the supervision signal. The dance tells you when to hit a beat. It does not tell you what note to play, how to build a phrase, or when to shift dynamics.
Recent work on representation alignment for generative models, particularly REPA, has shown that aligning a generator's internal representations with those of a pre-trained foundation model can improve both semantic consistency and training efficiency. This insight extends naturally to dance-to-music: if a pre-trained music model already understands musical structure at multiple scales, its representations can guide the generator's latent space toward musically meaningful territory.
The Multi-Scale Music Expert
CMA-OT uses a pre-trained multi-scale VQ-VAE encoder, specifically from the Jukebox model, as its external music expert. This encoder captures hierarchical latent representations at three temporal resolutions through a cascaded architecture:
- Top-level codes encode global musical semantics such as style and genre. These correspond to the macro-level structure of a piece, its overall character and form.
- Middle-level codes capture intermediate structures including melodic and harmonic phrases. These represent the meso-level organization that gives music its thematic development.
- Bottom-level codes represent fine-grained rhythmic textures and timbre. These are the micro-level details that make music feel alive and temporally precise.
The expert operates on clean audio, while the generator processes noised latents during diffusion training. This mismatch means the two feature spaces are not directly comparable. Projection heads (two-layer MLPs with GELU activations) map the generator's internal representations into a shared semantic space with the expert features, following the REPA approach.
The key architectural decision is how these three scales connect to the generator. The generator is a Diffusion Transformer (DiT) built on 12 LLaMA decoder layers with 764 hidden dimensions and 12 self-attention heads, using conditional flow matching rather than traditional denoising diffusion. The alignment maps expert scales to DiT layers as follows: top-level expert features align with deep DiT layers (high-level semantics), middle-level features with middle DiT layers (compositional structure), and bottom-level features with shallow DiT layers (temporal dynamics). This layer assignment was determined through analysis of representation similarity across DiT layers, not arbitrary.
Curriculum Learning: Coarse to Fine
The first challenge is training stability. If you activate alignment at all three scales simultaneously from the start, gradients from different hierarchical levels conflict. The model tries to learn global style, phrase structure, and rhythmic detail all at once, and the competing objectives destabilize optimization.
CMA-OT solves this with a three-phase curriculum using cosine-annealed weights. The progression is:
- Phase 1 (epochs 1-30): Top-level alignment dominates. The model learns global musical style and high-level semantics first, establishing a coarse structural foundation.
- Phase 2 (epochs 30-100): Middle-level alignment weight gradually increases while top-level weight softly decreases. The model learns melodic and harmonic structures on top of the established global style.
- Phase 3 (epochs 100-200): Bottom-level alignment becomes dominant to refine fine-grained temporal and rhythmic precision, while higher-level weights decay smoothly.
Each transition phase lasts 25 epochs to ensure smooth weight adaptation. The alignment loss is computed only at the final denoising step, when DiT representations approximate clean data most closely, making the comparison perceptually meaningful against clean expert features.
The ablation study confirms this matters. Multi-scale alignment without curriculum learning (Base + MA-OT) actually degrades performance compared to single-scale alignment, because the conflicting gradients destabilize training. Curriculum-guided multi-scale alignment (Base + CMA) reduces FADp by 40.81% relative to the base model, demonstrating that the coarse-to-fine progression is what makes hierarchical supervision work.
Four curriculum strategies were compared: fixed weights (all scales active simultaneously), hard switch (abrupt sequential activation), linear schedule (gradual transitions), and the proposed cosine schedule. Cosine consistently achieves the best results, confirming that smooth and continuous emphasis during training improves both optimization stability and output quality.
Scale-Aware Optimal Transport Alignment
The second challenge is alignment quality. Standard cosine similarity or point-wise feature matching cannot handle the temporal and structural differences between expert representations and generator latents. The expert and generator have different temporal resolutions across hierarchical scales, and the relationship between their features is not one-to-one.
CMA-OT uses Fused Gromov-Wasserstein (FGW) distance, which jointly optimizes two objectives: feature correspondence (how similar individual elements are) and structural consistency (how the relationships between elements are preserved). The FGW distance is defined over a transport plan P that maps elements from the source sequence to the target sequence, balancing feature similarity with relational structure preservation through a parameter alpha.
The formula combines a feature cost c (cosine distance between generator and expert features) and a structural cost L (MSE between intra-sequence relation matrices), weighted by alpha. The transport plan is constrained to valid couplings with specified marginals, ensuring the alignment is biologically plausible.
The scale-aware part is the adaptive weighting factor alpha_k. Rather than fixing alpha across all scales, CMA-OT computes it dynamically based on the ratio of expected structural cost to expected semantic cost under the current transport plan. The sigmoid function ensures the factor stays in [0, 1]. This lets the alignment mechanism adapt to each scale's characteristics: scales where structural relationships dominate get higher alpha (more emphasis on structure), while scales where semantic similarity matters more get lower alpha.
The ablation shows this is not just a minor tweak. Replacing cosine similarity with FGW (comparing Base + CMA and Base + CMA-OT*) improves all metrics, particularly fine-grained rhythmic alignment. Adding scale-aware adaptive weighting on top (Base + CMA-OT* to full CMA-OT) yields further gains. Standard optimal transport, which models only semantic similarity without structural constraints, underperforms FGW across all metrics, confirming that structural preservation matters for this task.
Results on AIST++ and TikTok
CMA-OT was evaluated against five baseline methods: D2M-GAN, CDCD, LORIS, Textual-Inv, and MotionComposer. Two datasets were used: AIST++ (1,020 professional studio dance videos across 10 genres) and TikTok (445 social media dance videos covering 85 songs). Training used 5-second music clips sampled at 44.1 kHz, with 2D pose skeletons extracted via DWpose and video features from a pre-trained I3D model.
On AIST++, the numbers tell the story clearly:
- Rhythmic alignment: CMA-OT achieves BCS of 99.14 and BHS of 99.12, versus MotionComposer's 95.84 and 95.09. The F1 score reaches 99.12 against the previous best of 96.45.
- Music quality: FADp drops to 12.50 from MotionComposer's 27.52, and FADc to 0.26 from 0.49. These are the most dramatic improvements in the table, indicating substantially better perceptual and structural music quality.
- Aesthetic quality: CMA-OT scores highest in all four Meta Audiobox-aesthetics dimensions: Production Quality (7.44), Production Complexity (6.71), Content Enjoyment (6.93), and Content Usefulness (7.49).
On TikTok, which is more challenging due to its unconstrained social media origin, CMA-OT maintains consistent superiority. BCS reaches 92.88 versus MotionComposer's 89.09, and FADp drops to 27.53 from 30.61. The gains are smaller on TikTok than AIST++, which is expected given the noisier data, but the direction is consistent.
A user study with 40 volunteers rating 30 test samples confirmed the objective metrics. CMA-OT achieved the highest average scores in both perceptual quality and dance-music relevance, with the gap particularly pronounced on the relevance dimension, validating that hierarchical expert supervision produces music that humans perceive as better matched to the dance.
Training Efficiency and Convergence
Beyond final quality, CMA-OT converges significantly faster than the baseline. The training curves show that the multi-scale representation alignment accelerates learning, with the model reaching comparable quality to the baseline's final performance much earlier in training. This is a practical benefit: the hierarchical supervision provides a stronger learning signal that guides the generator toward good solutions more quickly.
The base model without any external alignment achieves FADp of 21.12 on AIST++. Adding curriculum-guided multi-scale alignment alone (Base + CMA, using cosine similarity) drops this to 13.36, a 36.7% reduction. The full CMA-OT with scale-aware FGW further reduces it to 12.50. Each component contributes measurably to the final result.
Limitations and Practical Considerations
The paper does not extensively discuss limitations, but several trade-offs are visible from the design. The method requires a pre-trained multi-scale music expert, which means it cannot generate music in domains far from the expert's training distribution. The Jukebox encoder was trained on music data, and its hierarchical representations reflect that training. For genres or styles outside its coverage, the expert supervision may not transfer well.
The curriculum schedule introduces hyperparameters (phase boundaries at epochs 30, 100, and the transition durations) that were tuned for this specific setup. The paper compares four curriculum strategies and selects cosine annealing, but the sensitivity of the results to these schedule parameters is not explored. Different models or datasets might require different schedules.
The evaluation uses 5-second music clips, which is standard for this task but limits assessment of long-range structure. Dance-to-music generation for longer sequences, such as full performances, would require the model to maintain coherence over minutes rather than seconds, and it is unclear how well the hierarchical supervision extends to that regime.
Computational cost is not reported in detail. The FGW optimization involves solving an optimal transport problem at each alignment step across three scales, which adds overhead beyond simple cosine similarity. For production deployment, this cost would need to be weighed against the quality gains.
What This Means for Cross-Modal Generation
CMA-OT demonstrates a general principle that applies beyond dance-to-music: when the source modality provides sparse signals and the target modality requires dense structure, external expert supervision at multiple semantic levels can bridge the gap. The curriculum learning strategy ensures stable knowledge transfer, and the optimal transport alignment handles the inherent mismatch between heterogeneous feature spaces.
The approach is architecturally agnostic in the sense that the expert could be any pre-trained model with hierarchical representations, and the generator could be any diffusion-based architecture. The key ingredients are the multi-scale expert, the progressive curriculum, and the structure-aware alignment mechanism. These are composable components that could be adapted to other cross-modal generation tasks where the input modality is information-sparse relative to the output.
The results also suggest that representation alignment is underexplored as a training paradigm for generative models in domains outside vision and language. Music generation, speech synthesis, and other audio tasks may benefit similarly from hierarchical expert supervision, particularly when the conditioning signal does not carry enough information to fully specify the output.