Apple Shows Trillion-Parameter-Class LLMs Running Locally on Mac Studio Clusters with MLX
In short: Apple's MLX team published benchmarks showing large open-weight models — including a near-trillion-parameter MoE model called Muse Spark Open — running on Mac Studio hardware, both on a single machine and distributed across up to four Mac Studios using a new RDMA-over-Thunderbolt driver. The new M5 Ultra chip, with dedicated GPU Neural Accelerators and 1.2 TB/s memory bandwidth, delivers up to 4.6x faster time-to-first-token and 1.6x faster generation speed versus the previous M3 Ultra. For models too large to fit on one machine, MLX now supports pipeline and tensor parallelization across multiple Macs connected via Thunderbolt RDMA, letting a four-Mac-Studio cluster generate over 36 tokens per second on an 816B-parameter model.
This summary was generated automatically by AI from Apple Machine Learning's publication. It is our own text, not a copy of the original — facts, figures and quotes belong to the source, linked above and below.
What changed?
- 1New MLX version released with jaccl, RDMA-over-Thunderbolt communication collectives for distributed inference
- 2M5 Ultra Mac Studio delivers up to 4.6x faster time-to-first-token and 1.6x faster decode speed vs M3 Ultra
- 3Pipeline parallelization splits a model across depth between two Macs; tensor parallelization splits across width across two or four Macs
- 4816B-parameter MoE model (42B active, NVFP4 quantized to ~493GB) runs across four Mac Studios at over 36 tokens/second
- 5Two-cable Thunderbolt mesh cuts time-to-first-token for the largest model from ~50.5s to ~40.6s at 65,536-token context
- 6Mac Studio peak power draw for inference is about 200W (M3 Ultra) to 300W (M5 Ultra)
Why it matters
This demonstrates that very large, frontier-class open-weight models can now be run privately on consumer-grade desktop hardware instead of requiring cloud GPU clusters, which matters for teams wanting local, offline, or low-power LLM deployments with large context windows (useful for agentic workloads).
What it means for AI agents and contact centers
This is mainly relevant to teams exploring self-hosted or private LLM inference rather than cloud APIs; for voice AI and contact-center pipelines, the practical takeaway is that local Mac Studio clusters could in principle host large reasoning or agent-orchestration models with large contexts, but real-time voice latency, STT/TTS integration, and Lithuanian language support are not addressed in this research.
Sources
- Apple Machine LearningOfficialPrimary sourceOriginal article →„Local and Distributed Large LLM Inference with MLX and RDMA on Mac Studio“15 Oct 2026, 03:00
- Published by source
- 15 Oct 2026, 03:00
- Found by our system
- 15 Oct 2026, 20:11
- Summary generated
- 15 Oct 2026, 20:12
This article was written by AI from the original source. Facts, numbers and prices come from the source; missing values are marked “Not specified”. Legal notice, copyright and privacy