Why Your Ollama Fleet Keeps Hitting a Cold Node

Here is a pattern that catches almost everyone who adds a second machine to a local AI setup: you doubled the hardware and the experience got worse. Long conversations feel sluggish. The first reply is fine, the second one stalls. Nothing in the logs looks broken.

The usual cause is not your models and not your Macs. It is that most multi-instance setups pick a backend at random, and random selection throws away the single most valuable thing a warm machine has: the cache of the conversation you are already having.

What a follow-up message actually costs

When you send a message, the model has to read everything before it can write anything. That reading step is prefill, and it scales with how much context you send. A short question on a fresh chat is cheap. A follow-up in a 30,000-token conversation is not: the model has to process all 30,000 tokens again before producing a single new one.

Unless it doesn't have to. If the same machine handled the previous turn, it can still hold that conversation's prefix cache, the computed state for everything you already said. The follow-up then costs only the handful of new tokens, not the whole history. That is the difference between a reply that starts immediately and one that appears to hang.

So on a fleet, the routing decision quietly determines whether each turn is cheap or expensive.

Why random selection is worse than it looks

The most common way to use several Ollama servers is to register them all and let the client choose. Open WebUI, for example, supports multiple Ollama connections and picks among them randomly. Their own source carries a note that something smarter (weighted round-robin, least connections, or least response time) would be an improvement, and there is an open discussion asking for cache-aware routing specifically.

That is a reasonable default for identical, idle servers. On a real fleet it produces three problems:

• Cold-cache penalty. With two machines, a random pick sends roughly half your follow-ups to a node with no memory of the conversation, paying a full prefill each time. With three machines it is two-thirds.
• Cold-model penalty. If the requested model isn't resident on the chosen node, it loads first. That is seconds to minutes on a large model, and it can evict something else that was usefully warm.
• Hardware blindness. Macs are not interchangeable. A random pick will happily send a heavy request to the laptop that is thermally throttling and on battery, while the Studio sits idle.

Round-robin has the same flaw with more even distribution. Neither knows anything about the state of the machine it just chose.

What cache-aware routing does instead

Ollama Herd treats "which node already has this conversation warm" as a first-class input to the routing decision. It ships as session affinity, one of eight scoring signals: a multi-turn conversation is pinned to the node holding its warm prefix, so turn N+1 re-uses the cache instead of re-encoding the history.

Two design details matter here, and they are worth stating honestly:

• The bonus is deliberately not decisive. A warm cache is worth a lot, but a node about to thermally throttle still loses to a cool one. Pinning conversations to a machine that is about to slow down would trade one problem for another.
• It composes with the other seven signals (thermal state, memory fit, queue depth, latency history, role affinity, availability, context fit), so affinity is weighed against real conditions rather than applied blindly.

See the routing engine guide for exactly how the signals are scored and combined.

How to check what your setup is doing

Don't take anyone's word for it, including ours. Send the same conversation twice and look at which machine answered:

curl -i http://localhost:11435/api/chat \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3.6:27b","messages":[{"role":"user","content":"Hello"}],"stream":false}' \
  | grep -i x-fleet

Every response carries X-Fleet-Node (which machine answered) and X-Fleet-Score (its winning score). Run a first turn, then a follow-up in the same conversation, and compare:

• Same node both times means the follow-up re-used a warm cache.
• Different node means you just paid a full prefill for the second turn.

That single comparison tells you more about your fleet's real behavior than any throughput benchmark.

What this does not fix

Cache-aware routing removes the repeat cost, not the first one. The initial prefill of a long prompt still takes what it takes, and on Apple Silicon prompt processing is a genuine hardware and model constraint that no router can route around. If your very first message on a fresh conversation is slow, routing is not your problem; model size, quantization, and context length are (the memory guide covers that sizing).

What routing fixes is the waste: paying that same cost again on every turn because the request landed somewhere with no memory of the conversation. On a multi-machine setup, that waste is usually the larger number.

Related Reading

• Open WebUI with multiple Ollama servers, the full setup walkthrough
• Routing engine, how the 8 signals are scored
• Concurrency tuning, when the bottleneck is one machine instead of routing