MLX on Mac with Apple silicon lets AI researchers and developers experiment locally with the latest models and methods in a private environment. The open-source array framework leverages the unified memory architecture of Apple silicon, enabling anyone to run new inference and fine-tuning methods on their own hardware and explore other AI techniques. In a previous post, the team highlighted the advantages of the M5 architecture and its GPU Neural Accelerators. In this post, the team shows how to run the latest frontier-class model — Muse Spark Open, with nearly a trillion parameters — right from a desk, using the latest version of MLX released on the same day. On a cluster of four Mac Studios, Muse Spark Open generates over 36 tokens per second, well above average human reading speed.

Beginning on a single Mac Studio with up to 512GB of unified memory, then scaling out across multiple Macs with the RDMA over Thunderbolt driver, the post shares an overview of the range of models that can now run on Mac Studio. The team benchmarks several popular open-weight LLMs on Mac Studio using the MLX LM package. They compare two generations of Mac Studio: the recently released M5 Ultra and the previous generation, M3 Ultra. The M5 generation features the GPU Neural Accelerator, which supports dedicated matrix multiplication operations, essential for many machine learning workloads.

Frontier-class models on Mac Studio

The team evaluates three frontier-class models spanning a wide range of sizes in billions of parameters: the newly released Muse Spark Open (816B), Mistral Medium 3.5 (128B), and Qwen 3.8 (27B). Muse Spark Open is a mixture of experts model, which keeps only a fraction of its experts — and thus parameters — active, reducing compute at inference, and it is evaluated in NVFP4 precision. Mistral Medium 3.5 and Qwen 3.8 are dense architectures, where all parameters are active at inference, and they are evaluated in their native floating-point precision (FP8 and BF16, respectively). Qwen 3.8 27B and Mistral Medium 3.5 128B fit comfortably on a single Mac Studio with 512GB, taking up to 68GB and 158GB of RAM in the large context size (65,536 tokens) setup. Muse Spark Open would require more than 1.6TB of RAM in its native precision (BF16) for the model parameters alone. In a 4-bit quantized (NVFP4) format, the model memory footprint is lowered to about 493GB. Add the memory needed for intermediate computations during inference, and the model cannot fit on a single machine. It requires parallelization across multiple Mac Studio computers.

The evaluation measures three metrics across a range of context sizes: time to first token in seconds, generation throughput in tokens per second, and peak power usage in watts. All the performance numbers in the post were obtained with mlx_lm.benchmark. For example, running Qwen 3.8 with 8,192 tokens of context and generating 128 tokens is performed with the command line:

mlx_lm.benchmark --model Qwen/Qwen3.8-27B -p 8192 -g 128 -b 1

Inference performance on a single Mac Studio

TTFT is essentially compute bound, and that is where the Neural Accelerators in the M5 Ultra GPU shine: the specialized hardware support for fast matrix-multiplication of the GPU Neural Accelerator directly translates into significant speedups for TTFT, up to 4.6× faster compared to an M3 Ultra. That makes the Mac Studio with M5 Ultra the desktop of choice when it comes to handling tasks involving very large contexts. Generation speed in tokens per second is memory-bandwidth bound. The M5 Ultra chip delivers 1.2TB/s of unified memory bandwidth, up from 819GB/s for the M3 Ultra. Here again, that translates directly into generation speed, with the Mac Studio with M5 Ultra reporting 1.6× faster tokens/s. The Mac Studio is also energy-efficient for LLM inference: peak power consumption is about 200W for the Mac Studio with M3 Ultra, and 300W for the Mac Studio with M5 Ultra — a fraction of the power used by a common toaster. More importantly, that modest amount of power consumption means a Mac Studio can be deployed for AI work in any home or office environment.

Pipelining inference across two Mac Studio computers

Very large frontier-class models like Muse Spark Open will not fit on a single machine. It is, however, possible to store half of the model on one Mac Studio and the other half on another. The team can then pipeline the model’s inference across two computers: the first Mac Studio executes the first half of the model’s layers, then sends the output to the second Mac Studio, which executes the second half. This parallelization scheme is called pipeline parallelization and is extremely simple to implement, requiring only minimal code changes.

MLX supports different parallelization schemes. These rely on fast inter-machine communication: macOS 26.2 introduced a low-latency RDMA driver over Thunderbolt 5, and MLX v0.30.1 and later builds on top of this driver and provides new communication collectives (jaccl, pronounced "jackal") for different parallelization schemes. In the team's setup, the two Mac Studios are connected with two Thunderbolt cables (to double the link bandwidth), providing the RDMA-over-Thunderbolt link that jaccl uses. Once the hosts are set up, MLX provides mlx.launch, a helper command to launch jobs across multiple nodes:

mlx.launch --hostfile hostfile.json --backend jaccl --env MLX_METAL_FAST_SYNCH=1 -- \
mlx_lm.benchmark --model mistralai/Mistral-Medium-3.5-128B -p 8192 -g 128 -b 1 --prefill-step-size 8192 --pipeline

In this command, --backend jaccl specifies the RDMA Thunderbolt driver backend. The --env MLX_METAL_FAST_SYNCH=1 flag enables an experimental MLX feature for fast CPU-GPU synchronization, reducing communication overhead between the CPU, which initiates the Thunderbolt communication, and the GPU, which does the compute. In general, decode inference speed with pipeline parallelization is very similar to that of a single machine, since the layers are still evaluated sequentially. Communication overhead is also very low, as only the intermediate outputs between the two sets of layers need to be exchanged.

It is however notable that for very large contexts (65,536 tokens in the benchmark), pipeline parallelization ends up faster than a single node for prefill — up to 1.8× faster. This is because MLX segments large contexts into smaller chunks (here, 8,192 tokens, as set by --prefill-step-size 8192) and pipelines the prefill across the machines: while the second machine computes the later layers of a chunk, the first machine computes the earlier layers of a chunk, so consecutive chunks overlap in time. For even better compute parallelization, the team next turns to tensor parallelization.

Tensor parallelization for faster inference

Tensor parallelization distributes the LLM's width across several computers, in contrast to pipeline parallelization, which distributes the model across its depth. Each machine now holds a fraction of each of the layers of the model, such that per-layer computation is parallelized. When efficiently implemented, tensor parallelization improves inference speed. However, tensor parallelization is more complex to implement than pipeline parallelization: each layer must be parallelized, which increases code complexity. The scheme also involves many small communication operations: for each input token and for each layer, one must broadcast inputs and aggregate outputs. As a result, a very low-latency communication technology like Thunderbolt RDMA, supported by MLX jaccl, is critical for this approach to yield any speedup.

With MLX's high-level mlx.launch helper, it is easy to switch to tensor parallelization. The Mac Studio computers are connected in a Thunderbolt mesh, with each pair of machines linked peer-to-peer by two Thunderbolt cables. Assuming the hosts are set up, the team simply launches the benchmark across multiple machines with:

mlx.launch --hostfile hostfile.json --backend jaccl --env MLX_METAL_FAST_SYNCH=1 -- \
mlx_lm.benchmark --model meta-models/Muse-Spark-Open-816B-A42B-NVFP4 -p 8192 -g 128 -b 1

Figure 8 and Figure 9 show the speedups obtained with two or four Mac Studio computers with M5 Ultra, relative to a single-machine baseline. Notably, dense architectures scale better across multiple computers. With a small number of machines, the communication overhead of tensor parallelization can offset some of its compute benefits at very large contexts, though this applies to prefill only. This shows up with Muse Spark Open, where separate normalization of the experts' outputs adds some prefill communication, enough that at very large contexts (65,536 tokens), two-machine tensor parallelization lags behind two-machine pipeline parallelization. The approach however pays off as machines are added, as seen with four M5 Ultras. Decode is unaffected: Muse Spark Open sustains a significant decode speedup under tensor parallelization even at large contexts, comparable to dense models like Qwen 3.8 and Mistral Medium 3.5. The two-cable mesh also pays off during prefill, where the extra link bandwidth is felt. For Muse Spark Open on four M5 Ultras at a 65,536-token context, two cables per link cut TTFT from about 50.5s (one cable) to 40.6s.

Local agentic AI on Mac

Recently, agentic AI on Mac has gained considerable traction, with agentic coding tools seeing widespread adoption among developers. With MLX, powering these tools locally is as simple as launching a server with the mlx_lm.server tool. For example, a server running Muse Spark Open on port 8080 is started with:

mlx.launch --hostfile hostfile.json --backend jaccl --env MLX_METAL_FAST_SYNCH=1 -- \
mlx_lm.server --model meta-models/Muse-Spark-Open-816B-A42B-NVFP4 --port 8080

As with mlx_lm.benchmark, the mlx_lm.server helper can be prefixed with mlx.launch to parallelize computation across multiple Mac Studios. From there, agentic AI can be enabled with tools like OpenCode, which can be easily configured to use that local LLM server:

{
  "$schema": "https://opencode.ai/config.json",
  "model": "meta-models/Muse-Spark-Open-816B-A42B-NVFP4",
  "provider": {
    "mlx": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Local MLX",
      "options": {
        "baseURL": "http://127.0.0.1:8080/v1"
      },
      "models": {
        "meta-models/Muse-Spark-Open-816B-A42B-NVFP4": {
          "name": "Muse Spark Open 816B (MLX)"
        }
      }
    }
  }
}

Many other local agentic AI experiences can be enabled through a similar approach.

Conclusion

Over the past year, the team has worked on several key innovations in MLX to make it easier and more efficient to run state-of-the-art LLMs locally on recent Apple silicon. These include software support for the Neural Accelerators in the GPU of the M5 family of chips, low-latency communication across several Macs using RDMA over Thunderbolt, and efficient implementations of recent popular architectures such as mixture of experts. Each of these helps on its own, but it is their combination with the latest Apple silicon — Mac Studio with M5 Ultra, Neural Accelerators in the M5 Ultra GPU, 512GB of unified memory, and RDMA over Thunderbolt 5 — that now lets the team run models like Muse Spark Open with more than 800B parameters locally, at up to 4.6× the start-up (time-to-first-token) speed of the previous generation of Mac Studio.

Get Started with MLX. Related readings and updates. Apple is presenting new research at the annual International Conference on Learning Representations, which takes place in person in Rio de Janeiro, Brazil, from April 23 to 27. The team is proud to again sponsor the conference, which brings together the scientific and industrial research communities focused on deep learning. Below is an overview of Apple's participation at ICLR 2026. Exploring LLMs with MLX and the Neural Accelerators in the M5 GPU, November 19, 2025. Mac with Apple silicon is increasingly popular among AI developers and researchers interested in using their Mac to experiment with the latest models and techniques. With MLX, users can explore and run LLMs efficiently on Mac. It allows researchers to experiment with new inference or fine-tuning techniques, or investigate AI techniques in a private environment, on their own hardware. MLX works with all Apple silicon systems, and with the latest...