MoE Microscope Replays Mixture-of-Experts Training One Frame at a Time
Mixture-of-Experts models route each token to a subset of available experts during training and inference. Understanding how that routing evolves over time, which experts get more traffic, how load balancing shifts, and where bottlenecks appear, usually requires instrumenting a training run and digging through logs. MoE Microscope tries to make that process visual by replaying recorded training data as a sequence of frames.
The project is a static page, not an interactive computation. Nothing runs in your browser. Every number displayed comes from a JSON fixture stored in the repository. This is an important design choice: the page is reproducible, auditable, and fast, but it does not compute anything live.
Four Fixtures, Two Sources
Three of the four fixtures are recordings from actual training runs. They were captured using the emit_frame function in sw-MLPL, a tool for logging training telemetry, and routed through mlpl-serve. Each fixture is pinned by hash, so the page always displays the same data it was built against.
The fourth fixture was generated differently. An export script trained the same mixture configuration and wrote the data after training completed. The gate, a verification step in the project, regenerates or checks all four fixtures. If the data does not match what the gate expects, the page does not display it.
The distinction between recorded training data and script-generated output matters for trust. The three live fixtures represent real training behavior, including whatever noise, scheduling quirks, and hardware-specific patterns the original runs exhibited. The fourth provides a clean reference point from a controlled retraining.
Stepping Through Frames
The page is organized around lessons. Each lesson presents a sequence of frames, and the user steps through them one at a time. A frame captures the state of the mixture at a specific point in training: which experts received tokens, how the routing decisions looked, and what the load distribution was across the available experts.
This frame-by-frame approach is useful because Mixture-of-Experts training is not a static process. Early in training, routing may be noisy or concentrated on a small number of experts. As training progresses, the model learns to distribute load more evenly, or in some cases, it collapses into a pattern where only a few experts do meaningful work. Watching that evolution unfold across frames makes patterns visible that would be hard to spot in aggregate statistics.
Why a Playback Matters
Live training instrumentation is expensive and fragile. Connecting a visualization tool to a running training job introduces overhead and potential failure points. MoE Microscope sidesteps that entirely by recording the data once and replaying it as many times as needed. The tradeoff is that you cannot interact with a live training run through this tool. The tradeoff in return is reliability, speed, and the ability to share a fixed dataset that anyone can examine.
The project lives at sw-ml-study.github.io/moe-microscope, with the source and fixtures available in the repository. For anyone working on Mixture-of-Experts architectures, the frame-by-frame replay offers a concrete way to study routing behavior without instrumenting their own training runs.