Running LLMs Across Multiple Macs

Two different jobs, one confusing phrase

"Run an LLM across multiple Macs" describes two approaches that share almost nothing. Getting this wrong is the most expensive mistake in this space, so start here:

Only sharding lets you run a model bigger than your biggest Mac. If that is your goal, Ollama Herd is the wrong tool and the rest of this page will not help. Skip to sharding.

First: do you actually need a cluster?

Often not, and it is worth saying plainly before you buy hardware. One Mac with enough unified memory runs most local models well and is dramatically simpler to operate. A second machine adds a network hop, a discovery problem, and a failure mode.

A fleet earns its place when one of these is true:

• You already own the Macs. The MacBook that sits closed on a shelf and the old Mini under the desk are free capacity. This is the honest common case.
• You want to use your Mac while it serves. One machine doing inference and your actual work does both badly.
• You need concurrency. Several agents, or a team, hitting one Ollama means queueing behind each other.
• You want a model to stay warm. Dedicating one machine to holding a model resident removes the 15 to 30 second cold-load stall.

None of those are "run a bigger model." That is the other approach.

Routing: make several Macs act like one endpoint

This is what Ollama Herd does. Every Mac runs Ollama (or MLX) as normal; Herd sits in front as one endpoint and decides per request which machine answers.

The decision is the interesting part. Round-robin across Macs is worse than useless, because Macs are not interchangeable servers: one has the model already loaded, one is thermally throttling, one is in a video call, one has 8GB free and one has 400GB. Herd scores on eight signals, including model residency, memory fit, queue depth, latency history, and thermal state, and routes accordingly.

# On each Mac you want to contribute:
herd-node

# On the machine that will be the endpoint:
herd

# Point any Ollama-compatible client at the herd:
#   http://localhost:11435

Nodes find each other over mDNS, so there is no config file listing IPs. Full setup in the Quickstart, and the scoring detail in Routing Engine.

What this does not give you: a bigger model. Four 64GB Macs under Herd still cap out at what one 64GB Mac can hold, because each request runs entirely on one node.

Sharding: run one model too big for any single Mac

If you want a 400B-class model on a pile of Macs, you need the model split across them. Ollama Herd does not do this today. The real options:

• exo, the best-known option for chaining Macs into one logical machine. Most of the community's Mac-cluster mindshare lives here.
• Apple's distributed MLX, first-party and the fastest-moving. Its JACCL backend runs collectives over RDMA on Thunderbolt 5 for tensor-parallel work; a plain LAN ring backend pools memory without special hardware.
• OLOL, gRPC-based Ollama clustering that also does layer partitioning.

Expect a real tradeoff. Sharding is slower per token than the same model on one machine that could hold it, because activations cross the network on every forward pass. Tensor parallelism is also bottlenecked by your smallest node, so a mixed fleet of one big Mac and one small one can end up worse than the big Mac alone. Mixed fleets should pool memory (pipeline) rather than chase tensor-parallel speedups.

They compose

These approaches are not rivals. A sharded cluster can register with Herd as a single node, so the herd sees one endpoint whether one Mac or four sit behind it. You can run a big sharded model for the hard cases and let Herd route everyday requests to whichever machine is free, from the same endpoint.

Related Reading

• Load Balancing Ollama, the HAProxy and nginx options, and why they fall short for Macs
• Routing Engine, the eight signals behind each decision
• Adaptive Capacity, how Herd avoids the Mac you are using
• Ollama Herd vs exo, routing vs sharding compared in full