Apple ML Research Announces Distributed Inference for MLX

Apple ML Research Announces Distributed Inference for MLX

At a Glance

Item Value
Published 2026-10-16
Source type Primary source (the publisher itself)

Values determined by this site’s code when the information was collected. Dates are JST.

Overview

On October 15, 2026, Apple ML Research announced a new method for distributed inference of large language models on Mac Studio. By leveraging the latest MLX (v0.30.1 or later) and the low-latency RDMA driver via Thunderbolt 5 introduced in macOS 26.2, it becomes possible to run ultra-large models that cannot run on a single Mac.

Announcement Details

Apple published benchmark results running “Muse Spark Open", a state-of-the-art model with approximately 1 trillion parameters, utilizing a cluster of Mac Studios equipped with M5 Ultra chips. The main technical details are as follows:

Distributed Inference Methods

MLX supports two parallelization schemes for splitting and running large models across multiple Macs:

  • Pipeline Parallelization: Splits the model layers and assigns them to each machine. For example, a configuration using two Mac Studios to run the first and second halves of the model respectively. It is characterized by very simple implementation.
  • Tensor Parallelization: Splits the “width" of the model and performs the calculation of each layer in parallel across multiple machines. Implementation is more complex than pipeline parallelization, and low-latency communication technology is essential as frequent communication occurs between machines.

Communication Technology and Software

To realize these parallelizations, MLX provides a new communication collective called jaccl. This utilizes the RDMA (Remote Direct Memory Access) driver via Thunderbolt 5 introduced in macOS 26.2. Additionally, as an experimental feature, the MLX_METAL_FAST_SYNCH=1 flag is provided to reduce communication overhead between the CPU and GPU.

Benchmark Results

When configuring four M5 Ultra-equipped Mac Studios with tensor parallelization, Muse Spark Open (816B parameters, NVFP4 precision) achieved a generation speed of 36.73 tokens/s or more at a context size of 8,192 tokens. This is a figure that greatly exceeds the average human reading speed. Furthermore, for Muse Spark Open which cannot run on a single M5 Ultra, it has been demonstrated that by combining multiple Macs, the context size can be expanded to 65,536 tokens for execution.

Background

Apple has been developing “MLX", an open-source array framework utilizing Apple silicon’s unified memory architecture, to enable AI researchers and developers to experiment locally with the latest models and methods in a private environment. Previous posts have also touched upon the benefits of the M5 architecture and GPU Neural Accelerators.

The purpose of this announcement is to enable ultra-large frontier-class models, which lack sufficient memory capacity to run on a single Mac Studio (up to 512GB of unified memory), to run in a desktop environment through a cluster configuration connecting multiple Macs.

Impact on Local LLM Users

For engineers using open-weight models, this announcement has significant implications in the following areas:

Local Execution Possibility of Ultra-Large Models

Even for models like Muse Spark Open that result in out-of-memory errors on a single machine (requiring 1.6TB+ of RAM at BF16 precision), local execution becomes possible by connecting multiple Mac Studios via Thunderbolt and performing distributed inference using MLX. This expands the options for handling huge models in a private environment without relying on cloud APIs.

Execution Environment Flexibility and Scalability

MLX supports two methods: “pipeline parallelization", which splits model layers, and “tensor parallelization", which splits computations within layers. Users can choose the parallelization scheme depending on whether they prioritize implementation simplicity or commit multiple machines to improve inference speed. In particular, with the M5 Ultra generation chips, high performance is expected in both single-node and distributed environments due to faster matrix operations by the GPU Neural Accelerator and increased memory bandwidth (from 819 GB/s in M3 Ultra to 1.2 TB/s).

Application to Agent Applications

MLX provides the mlx_lm.server tool, making it possible to launch it as a local server while maintaining the distributed inference configuration. This allows for the construction and use of agent-based coding applications in private environments backed by large-scale LLMs running locally, using tools like OpenCode.

Sources