What is the difference between routing and sharding for local AI?

Sharding splits one model across several machines so you can run a model too big for any single one. It buys capacity. Routing sends each independent request to whichever whole machine is best suited, so more machines means more concurrent work. It buys throughput. Sharding couples the machines together on every token; routing keeps them independent.

Should I split a model across my Macs or route between them?

If the model fits on one machine with headroom, route. Sharding a model that already fits makes it slower, because every token now crosses a cable. If the model does not fit at any quantization you find acceptable, sharding is your only option, and you should expect it to buy the ability to run the model rather than speed.

How do I know if a model fits on one Mac?

Estimate weights at roughly 0.55 to 0.6 GB per billion parameters at 4-bit (about 1.05 to 1.1 at 8-bit), add the KV cache for your context length, and aim to land under about 70 percent of the machine's unified memory. macOS reserves memory for the system and caps how much the GPU can wire, and loading peaks higher than the resident size, so a file that fits on disk can still fail to load.

Can I use both routing and sharding together?

Yes, and at fleet scale they are different layers rather than competing choices. A sharded cluster exposes a single OpenAI-compatible endpoint, so a router can treat the whole cluster as one backend: the big model runs on the cluster, everything else routes to individual machines. Note that a sharded cluster has very different latency, so it should be chosen by model rather than by load.

Is exo still the best way to cluster Macs?

exo is the most-starred project in this space, but its public repository has had no maintainer commits since June 2026, with the last release in April 2026 and community pull requests sitting unmerged. MLX distributed is the actively developed path (Apple ships it, and the repository is updated continuously). Check both yourself before depending on either.

Routing vs Sharding: Which Does Your Mac Fleet Need?

Q: Does splitting a model across two Macs make it faster?

Usually not. Published measurements show a small model that fits on one machine running about 2.7 times slower when sharded across two. Apple states directly that pipeline parallelism does not speed up inference. Tensor parallelism can speed things up on a Thunderbolt 5 RDMA mesh, but that mainly helps for models that already fit, which is the case where you did not need to shard.

"Running AI across multiple Macs" describes two completely different architectures, and people routinely pick the wrong one. The distinction is worth five minutes because getting it backwards makes your setup slower than the single machine you started with.

	Sharding	Routing
What it does	Splits one model across machines	Sends each request to one machine
Buys you	Capacity: run models that do not fit	Throughput: more concurrent work
Machines are	Coupled. Every token crosses the network	Independent. No cross-talk at all
One node dies	The model is down	You lose capacity, not service
Needs	Thunderbolt mesh, matched macOS, setup	A network
Tools	MLX distributed, exo, llama.cpp RPC	Ollama Herd, Olla, a load balancer

The one question that decides it

Does your model fit on one machine, with headroom?

Yes and you want it faster or serving more people: route. Sharding something that already fits will slow it down.
No, at any quantization you find acceptable: shard. It is your only option, and it buys the ability to run the model, not speed.
Both problems (one giant model plus everyday work): run the shard as a cluster and put a router in front of everything. They are different layers, not competing choices.

Working out whether it fits

Three numbers, and the third is the one people forget.

1. Weights. A reliable rule of thumb, cross-checked against several published model sizes:

Quantization	Approx. size	Example
4-bit	~0.55 to 0.6 GB per billion params	480B coder ≈ 276 GB
8-bit	~1.05 to 1.1 GB per billion params	1T model ≈ 1 TB

It is above the naive 0.5 GB per billion because quantization also stores scales and zero-points, and some tensors stay at higher precision.

2. KV cache, which grows with context length. Here is the part almost nobody mentions: KV cost is driven by layer count and attention-head geometry, not total parameter count. A 744B mixture-of-experts model can have a smaller KV cache than a 70B dense model. So for giant MoE models the weights dominate, while for long-context work on mid-size dense models the KV cache can be the binding constraint. (Some newer architectures compress the cache substantially, so treat generic formulas as an upper bound.)

3. Headroom. Aim to land under roughly 70% of unified memory. macOS keeps a system reserve, caps how much memory the GPU can wire down (near 66 to 75% by default), and peaks higher during loading than at rest.

That third point is not theoretical. We pulled a 418 GB 4-bit quant onto a 512 GB Mac Studio, which leaves 94 GB free on paper, and it would not load. The load transient crossed physical memory and macOS killed it. A file that fits on disk can still fail to load, which is exactly the situation where people start looking at sharding.

What sharding actually costs

The marketing for distributed inference emphasizes the speed-ups. The measurements are more sobering, and you should go in with the right expectation.

If the model already fits, sharding makes it slower. Published benchmarks show a small model going from 5.0 ms per token on one Mac to 13.4 ms across two, roughly 2.7 times worse. Every token now waits on a cable that used to be a memory bus.

If the model does not fit, sharding wins enormously, because the alternative is streaming weights off an SSD. The same source measured 4.9 tok/s streaming versus 20.5 tok/s sharded, about 4.2 times better.

Adding nodes past that scales poorly. A community benchmark on a five-node Thunderbolt 5 mesh running a 1T-parameter model measured 13.15 tok/s on two nodes and 14.45 tok/s on five. That is roughly 10% more throughput for 2.5 times the hardware. The same benchmark found only about 2% between pipeline and tensor parallelism at that scale.

Two things follow. First, Apple states plainly that pipeline parallelism does not speed up inference: each token still passes through the layer groups in sequence. It buys capacity only. Tensor parallelism can speed things up (Apple demonstrated roughly 3x on four machines with a model that fits), but it needs a full Thunderbolt 5 RDMA mesh, and it mostly helps in the case where you did not need to shard.

Second, sharding is a capacity technology, not a speed technology. Judge it on "can I run this at all," not on tokens per second.

The setup burden is real

Worth knowing before you commit a weekend. A current MLX distributed setup needs:

A full Thunderbolt mesh. A direct cable between every pair of machines: 1 cable for 2 nodes, 3 for 3, 6 for 4, 10 for 5. Thunderbolt port count becomes the real limit.
macOS 26.2 or later, and RDMA enabled on each machine, which involves a reboot and cannot be done remotely. You need physical access to every node.
Passwordless SSH between all nodes, plus the Python binary and your script at identical absolute paths on every machine.
A hostfile describing the topology, generated by a config tool that probes the cabling.

Routing, by contrast, needs the machines to be on a network. That asymmetry is most of the practical difference.

How routing scales, and what we cannot tell you

Routing scales differently because independent requests have no coupling: each one executes entirely on a single machine, and the router only chooses a destination. Nothing crosses the network mid-token. Aggregate throughput should therefore grow with machine count until the router itself becomes the bottleneck.

We are stating that as a structural argument, not a measured one. We could not find a credible published benchmark of aggregate throughput across several Macs for independent requests. Numbers circulating for this ("linear up to N backends," specific latency reductions) trace back to sites that appear to be automatically generated, and we are not going to repeat them. If you see a precise figure for multi-Mac routing throughput, ask where it was measured.

What is well established is that a single instance saturates: independent benchmarking shows Ollama flattening out under concurrency well below what dedicated serving engines reach, and on-device batching hitting diminishing returns as memory bandwidth saturates. Adding machines is how you get past that wall, which is the routing case.

Using both

These are not really competitors at fleet scale. A sharded cluster exposes one OpenAI-compatible endpoint, the same interface a single machine presents. A router cannot tell the difference and does not need to: the cluster is simply one unusually large, unusually slow backend in the pool.

So the sensible architecture for someone with both problems is one sharded cluster for the model that does not fit, plus individual machines for everything else, with one endpoint in front. One caveat worth respecting: a sharded cluster has very different latency characteristics (notably much slower time-to-first-token), so it should be selected by model rather than treated as interchangeable with whole machines under load.

A note on the tools

If you decide you need sharding, check the current state of the projects yourself before depending on one. As of August 2026:

MLX distributed is the actively developed path. Apple ships it, presented it at WWDC26, and the repositories are updated continuously.
exo is the most-starred project in this space by a wide margin, and its approach is genuinely impressive. But its public repository has had no maintainer commits since June 2026, the last release was April 2026, and community pull requests are sitting unmerged. That may well be temporary, and we hope it is. Check the repository before you build on it.
llama.cpp RPC still carries a proof-of-concept warning in its own README, and independent benchmarking has measured it getting slower as nodes are added on Mac hardware.

Common situations

Situation	Answer
2 to 3 Macs, want a faster coding agent	Route. Coding models fit on one machine easily. Sharding would slow them down, and what actually limits agent work is concurrency and time-to-first-token, both of which sharding makes worse.
Several people sharing a fleet	Route. Multiple users means independent requests, the workload with zero coupling. Sharding would slow every request and couple all the failure domains: one node down takes the model down.
Want to run a 400B model at 4-bit	That is roughly 230 to 290 GB, so it fits on one 512 GB machine. Do not shard. Two 256 GB machines, though, would have to.
Want to run a 700B+ model at 4-bit	Shard, or find a smaller quantization, or try a different runtime. Check the cheaper options first: a lower-bit quant can bring a huge model under a single machine's ceiling, and load paths differ between runtimes.
One giant model plus everyday work	Both. Cluster for the big model, individual machines for the rest, one endpoint in front.

The short version

Does it fit on one machine with headroom? Route. Does it not fit at any quantization you will accept? Shard, and judge it on capability rather than speed. Have both problems? Put the shard behind the router.

If you landed here because a model would not load, start with the load-memory ceiling, which covers why disk size misleads. If you landed here because your fleet feels slow, it is more often cold-node routing or single-machine concurrency limits than anything to do with sharding.

Routing vs Sharding: Which Do You Actually Need?