Can you run an LLM across multiple Macs?

Yes, but there are two different approaches and they solve different problems. Routing sends each request to whichever Mac can answer it fastest, which raises throughput and reliability but does not let you run a model larger than one machine. Sharding splits one model's layers across several Macs so they jointly serve a model too large for any single one, at the cost of speed and setup complexity. Ollama Herd does routing. exo and Apple's distributed MLX do sharding.

Do I need a Mac mini cluster to run local LLMs?

Usually no. A single Mac with enough unified memory runs most local models well, and one machine is far simpler to operate. A cluster earns its keep when you have several Macs already, when you want requests served while you are using a machine for other work, or when you need more total throughput than one box provides.

What is the difference between Ollama Herd and exo?

exo shards one model across devices so a fleet can run a model too large for any single machine. Ollama Herd routes whole requests to whole machines using device-aware scoring, so it cannot run a model larger than your biggest Mac. If your goal is running a 400B model on four Macs, use exo. If your goal is making several Macs serve requests reliably without stepping on your work, use Herd.

Does Ollama Herd split a model across Macs?

Not today. Herd routes many requests across many machines; it does not shard one model across them. Each request runs entirely on one node, so the largest model you can serve is bounded by your largest single Mac.

Do I need Thunderbolt 5 to cluster Macs for AI?

Only for the sharding approach at high speed. Apple's JACCL backend uses RDMA over Thunderbolt 5 for tensor-parallel work, which requires recent hardware and macOS. Routing needs nothing special: plain Wi-Fi or Ethernet is fine, because only the request and response cross the network, not model activations.

Mac Mini Cluster for LLMs: Route Across Every Mac

Two different jobs, one confusing phrase

"Run an LLM across multiple Macs" describes two approaches that share almost nothing. Getting this wrong is the most expensive mistake in this space, so start here:

	Routing (many requests)	Sharding (one model)
What it does	Sends each request to the best Mac	Splits one model's layers across Macs
Solves	Throughput, reliability, using idle machines	Running a model too big for any one Mac
Max model size	Bounded by your largest single Mac	Roughly the sum of the fleet's memory
Speed effect	Neutral to faster (parallel requests)	Slower per token than one big machine
Network	Wi-Fi or Ethernet is fine	Wants Thunderbolt 5 to be quick
Tools	Ollama Herd	exo, Apple distributed MLX, OLOL

Conceptual: two different architectures

Routing: whole requests, separate machines

Buys throughput. Each machine holds its own full copy of the model, and nothing crosses the network mid-token.

Request 1

Request 2

Request 3

one endpoint

RouterOllama Herd

each request goes to one node

Mac Afull model

Mac Bfull model

Mac Cfull model

Sharding: one model, split across machines

Buys capacity. Every token passes through each machine's share of the layers, so activations cross the network on every step.

One request

the cluster's own endpoint

Sharding runtimeexo, MLX distributed, llama.cpp RPC

every token, through every machine in turn

Mac Afirst third of the layers

Mac Bsecond third

Mac Clast third

Routing sends each whole request to one machine and adds throughput; sharding splits one model across machines and adds capacity, not speed. Ollama Herd routes: it does not shard, and a sharded cluster runs as its own endpoint next to a herd rather than as a node in it.

Only sharding lets you run a model bigger than your biggest Mac. If that is your goal, Ollama Herd is the wrong tool and the rest of this page will not help. Skip to sharding.

First: do you actually need a cluster?

Often not, and it is worth saying plainly before you buy hardware. One Mac with enough unified memory runs most local models well and is dramatically simpler to operate. A second machine adds a network hop, a discovery problem, and a failure mode.

A fleet earns its place when one of these is true:

You already own the Macs. The MacBook that sits closed on a shelf and the old Mini under the desk are free capacity. This is the honest common case.
You want to use your Mac while it serves. One machine doing inference and your actual work does both badly.
You need concurrency. Several agents, or a team, hitting one Ollama means queueing behind each other.
You want a model to stay warm. Dedicating one machine to holding a model resident removes the 15 to 30 second cold-load stall.

None of those are "run a bigger model." That is the other approach.

Routing: make several Macs act like one endpoint

This is what Ollama Herd does. Every Mac runs Ollama (or MLX) as normal; Herd sits in front as one endpoint and decides per request which machine answers.

The decision is the interesting part. Round-robin across Macs is worse than useless, because Macs are not interchangeable servers: one has the model already loaded, one has a long queue, one is in a video call, one has 8GB free and one has 400GB. Herd scores on eight signals, including model residency, memory fit, queue depth, latency history, and session affinity, and routes accordingly. With adaptive capacity turned on, it also pauses a Mac that is in a video call.

# On each Mac you want to contribute:
herd-node

# On the machine that will be the endpoint:
herd

# Point any Ollama-compatible client at the herd:
#   http://localhost:11435

Nodes find each other over mDNS, so there is no config file listing IPs. Full setup in the Quickstart, and the scoring detail in Routing Engine.

What this does not give you: a bigger model. Four 64GB Macs under Herd still cap out at what one 64GB Mac can hold, because each request runs entirely on one node.

Sharding: run one model too big for any single Mac

If you want a 400B-class model on a pile of Macs, you need the model split across them. Ollama Herd does not do this today. The real options:

exo, the best-known option for chaining Macs into one logical machine. Most of the community's Mac-cluster mindshare lives here.
Apple's distributed MLX, first-party and the fastest-moving. Its JACCL backend runs collectives over RDMA on Thunderbolt 5 for tensor-parallel work; a plain LAN ring backend pools memory without special hardware.
OLOL, gRPC-based Ollama clustering that also does layer partitioning.

Expect a real tradeoff. Sharding is slower per token than the same model on one machine that could hold it, because activations cross the network on every forward pass. Tensor parallelism is also bottlenecked by your smallest node, so a mixed fleet of one big Mac and one small one can end up worse than the big Mac alone. Mixed fleets should pool memory (pipeline) rather than chase tensor-parallel speedups.

They compose

These approaches are not rivals. A sharded cluster runs as its own endpoint next to the herd; it cannot join as a Herd node today, because Herd nodes need Ollama's API. You can run a big sharded model for the hard cases and let Herd route everyday requests to whichever machine is free: clients call the cluster for the giant model and Herd for everything else.

Running LLMs Across Multiple Macs

Two different jobs, one confusing phrase

First: do you actually need a cluster?

Routing: make several Macs act like one endpoint

Sharding: run one model too big for any single Mac

They compose

Related Reading