Why Your Robot Hesitates Before Every Move

Factory robots using Vision-Language-Action models spend a large chunk of their time waiting for the model to think. Each time the robot looks at a scene, processes a language instruction, and decides what to do next, it runs a heavy neural network pipeline. The gap between one action chunk and the next is pure dead time, and it directly affects how smoothly and responsively the robot moves. In a factory setting where the same picking-and-placing task repeats hundreds of times a day, that wasted time adds up.

rMuscle, a new inference framework from Shanghai Jiao Tong University, attacks this problem by asking a simple question: if the robot is doing the same task it did five minutes ago, why recompute everything from scratch?

The Muscle Memory Analogy, Made Literal

The paper draws an explicit analogy to human motor learning. When you first learn to type, every keystroke requires conscious thought. After practice, your fingers move without thinking, the motor patterns stored as muscle memory. Your brain does not re-derive the motion each time, it retrieves a learned pattern and adjusts it for the current context.

rMuscle does the same thing for VLA model inference. The system maintains a cache of previous inference states, organized by task and context. When the robot encounters a situation similar to one it has seen before, rMuscle reuses cached intermediate computations rather than recomputing them from scratch. The key insight is that this similarity is not just superficial (similar camera views, similar trajectories) but extends to the internal model states: the FFN inputs and outputs, and the neuron activation patterns in the denoising layers.

Two Caches for Two Bottlenecks

A modern VLA model like π₀.₅ has two distinct inference phases, and each one is slow for a different reason. The first phase, VLM prefill, processes images and language through a vision-language model to produce context features. This phase is compute-bound: the model is crunching large matrices and the GPU is the bottleneck. The second phase, action denoising, takes noisy action samples and iteratively refines them through a diffusion process. This phase is memory-bound: the model loads weight matrices repeatedly for small computations, and memory bandwidth is the bottleneck.

Prior acceleration methods either optimize one phase or the other, or apply generic techniques like step-skipping that degrade quality. rMuscle instead provides a dual-phase cache, with a different mechanism for each bottleneck.

Context Cache: Selective Token Recomputation

The Context Cache accelerates VLM prefill by reusing cached FFN outputs from similar past executions. The trick is deciding which visual tokens to recompute and which to reuse. The authors profile a relationship they call "visual-token coverage": if you rank visual tokens by the magnitude of change in their FFN inputs between a reference execution and the current one, recomputing just the top 40% of tokens captures roughly 80% of the total output change.

At runtime, rMuscle compares the current FFN inputs against a retrieved reference, ranks tokens by normalized RMS input change, recomputes the top fraction, and merges those new outputs into a buffer initialized with the cached reference outputs. Instruction tokens and robot-state tokens are always recomputed since they change frequently. The attention layers still see the full sequence, so no information is lost, only the expensive FFN computations are reduced.

Action Cache: Sparse Neuron Updates

The Action Cache tackles denoising by exploiting two observations about neuron behavior. First, not all neurons contribute equally to the output. Ranking neurons by their contribution score (a product of activation magnitude and weight norm), the top 50% of neurons account for 84.4% of the total contribution. Second, the same task executed repeatedly tends to activate overlapping sets of important neurons. For same-task executions, important-neuron overlap averages 86%. For different tasks, it drops to 71%.

rMuscle uses a reference execution's neuron activation pattern to predict which neurons will be important in the current execution. It loads and computes only those neurons' weights, then adds their contribution to the output of a dense "anchor" step that was computed fully. The anchor establishes a baseline, and the sparse updates refine it. The subtraction of the anchor's contribution before adding the current value avoids double-counting.

To reduce weight-gathering overhead, consecutive denoising steps are grouped, with each group sharing one neuron mask. For the ten-step π₀.₅ configuration, step 0 anchors for steps 1 through 4, and step 5 anchors for steps 6 through 9. This means the system only needs to load and cache two neuron masks per denoising sequence rather than ten.

Cache Management That Stays Out of the Way

The obvious risk with any caching scheme is that cache management itself becomes a bottleneck. rMuscle avoids this through three techniques.

First, asynchronous online reconstruction. Expanded VLM states are too large to store persistently for every reference. For a three-view π₀.₅ setup, each of the 17 FFN layers holds roughly 6 MB of BF16 input-output pairs, totaling about 107 MB per policy call. Storing that for 40 calls would require over 4 GB. Instead, rMuscle stores only compact reference inputs (about 1.7 MB each) and reconstructs the full states during the robot's physical execution window. Since the robot executes actions at 20-30 Hz, there is roughly 500 ms of idle GPU time between inference calls, enough to prepare about 20 references.

Second, sliding-window prefetching. The authors find that 96.6% of reference matches track the current episode's progress within about ten percentage points of aligned progress. This temporal locality means a small sliding window of recently relevant references covers most needs. The GPU working set maintains only these candidates, bounded by a configurable memory budget.

Third, visual similarity retrieval. When the robot encounters a new situation, rMuscle matches the current ViT features against stored reference features using cosine similarity, then reranks the top candidates by joint-angle distance to distinguish visually similar scenes with different arm configurations. The fallback policy switches to dense execution if visual similarity drops below 0.8, ensuring the system never uses a bad reference.

The Numbers: 1.29 to 1.42 Times Faster

On an RTX 4090, rMuscle achieves inference rates of 28.1 Hz for π₀.₅, 51.0 Hz for GR00T N1.6, and 35.7 Hz for X-VLA, corresponding to speedups of 1.29×, 1.20×, and 1.50× over FlashRT (the current SOTA engine). On Jetson Thor (the robot-grade chip with 128 GB unified memory), the rates are 13.4, 17.5, and 19.7 Hz, with speedups of 1.42×, 1.23×, and 1.43×.

The speedup increases with model size. Scaling from a 2B VLM with a 0.3B denoiser up to 7B with 7B, the speedup grows from 1.3× to 1.6× on RTX 4090 and from 1.4× to 1.9× on Thor. This makes sense: larger models have more redundant computation to exploit.

The Context Cache alone provides the larger share of speedup on RTX 4090 (87% of the total latency reduction), while the Action Cache is more impactful on Jetson Thor, consistent with Thor being more memory-bound. Combined, they attack both phases of the pipeline.

Preserving Task Success Rates

The speedup would be worthless if it degraded robot performance. The authors evaluate across 40 LIBERO tasks, 50 RoboTwin tasks, and two physical-robot tasks. On LIBERO, rMuscle matches vanilla success rates across all three VLA models. On RoboTwin, it matches the 35.2% average success rate of vanilla π₀.₅, exceeding DP-Cache by 1.3 points and NIRVANA by 8.3 points. On physical robots, it achieves 76% on dual-arm ALOHA bottle pick-and-place and 84% on DOBOT/Franka conveyor furniture packing, both matching the vanilla baselines.

The comparison with NIRVANA (which reuses reference denoising outputs directly) is telling. NIRVANA drops success rates substantially because VLA models use far fewer denoising steps (4-10) than image-generation models (20-50), making each step too important to skip. rMuscle's approach of reusing internal states while still computing each step preserves quality.

Memory Cost: Small Enough to Ignore

On RTX 4090, GPU memory increases from 8.4 GB for vanilla to 9.4-11.4 GB with rMuscle, raising occupancy by less than 13 percentage points. On Jetson Thor, unified memory rises from 6.6% to at most 12.6% of 128 GB. The CPU cache for compact reference inputs uses 3.7-14.9% of 32 GB host memory. These are manageable overheads for a real deployment.

Where It Falls Short

The system relies on having relevant references in the cache. Unseen tasks start with dense execution and build their reference library over time. On RoboTwin, two tasks (Handover Block and Place Empty Cup) trail vanilla by 2-3 points because observation similarity sits right at the fallback threshold of 0.8. The authors note that raising this threshold or adding more reference episodes resolves these cases, but the general problem of cold-start and reference coverage remains.

The framework also does not support dynamic cache updates during deployment. New task variants must be profiled offline to populate the cache. An online learning component that admits successful episodes from underrepresented observation clusters would make the system more robust to novel situations, but that is left for future work.

The evaluation is limited to three VLA models and two simulation benchmarks, plus two physical tasks. The generality across the full landscape of VLA architectures and real-world deployment scenarios remains to be demonstrated.

What This Means for Robot Deployment

For teams deploying VLA-powered robots in factories, rMuscle offers a practical way to increase throughput without hardware upgrades. The 29-42% speedup translates directly into more action chunks per second, smoother motion, and faster task completion. The system works with existing VLA models and hardware, requiring no model retraining or architectural changes.

The deeper contribution is the characterization of embodied workloads. The finding that internal model states (FFN inputs, neuron activations) exhibit high similarity across repeated executions opens a new axis for optimization beyond the frame-to-frame reuse that prior work focused on. This suggests that future VLA models could be designed from the ground up to exploit cross-execution redundancy, rather than retrofitting caching onto existing architectures.

The muscle-memory framing is apt. The more a robot repeats a task, the less computation it needs to perform it well. That is exactly how biological motor control works, and it may be the right model for scaling industrial robotics.

Read the paper on arXiv