Why Your Ollama Fleet Keeps Hitting a Cold Node

Here is a pattern that catches almost everyone who adds a second machine to a local AI setup: you doubled the hardware and the experience got worse. Long conversations feel sluggish. The first reply is fine, the second one stalls. Nothing in the logs looks broken.

The usual cause is not your models and not your hardware. It is that most multi-instance setups pick a backend at random, and random selection throws away the single most valuable thing a warm machine has: the cache of the conversation you are already having.

What a follow-up message actually costs

When you send a message, the model has to read everything before it can write anything. That reading step is prefill, and it scales with how much context you send. A short question on a fresh chat is cheap. A follow-up in a 30,000-token conversation is not: the model has to process all 30,000 tokens again before producing a single new one.

Unless it doesn't have to. If the same machine handled the previous turn, it can still hold that conversation's prefix cache, the computed state for everything you already said. The follow-up then costs only the handful of new tokens, not the whole history. That is the difference between a reply that starts immediately and one that appears to hang.

So on a fleet, the routing decision quietly determines whether each turn is cheap or expensive.

Why random selection is worse than it looks

The most common way to use several Ollama servers is to register them all and let the client choose. Open WebUI, for example, supports multiple Ollama connections and picks among them randomly. Their own source carries a note that something smarter (weighted round-robin, least connections, or least response time) would be an improvement, and there is an open discussion asking for cache-aware routing specifically.

That is a reasonable default for identical, idle servers. On a real fleet it produces three problems:

• Cold-cache penalty. With two machines, a random pick sends roughly half your follow-ups to a node with no memory of the conversation, paying a full prefill each time. With three machines it is two-thirds.
• Cold-model penalty. If the requested model isn't resident on the chosen node, it loads first. That is seconds to minutes on a large model, and it can evict something else that was usefully warm.
• Hardware blindness. Machines are not interchangeable. A random pick will happily send a request for a large model to the laptop that barely fits it, while the desktop with memory to spare sits idle.

Round-robin has the same flaw with more even distribution. Neither knows anything about the state of the machine it just chose.

What cache-aware routing does instead

Ollama Herd treats "which node already has this conversation warm" as a first-class input to the routing decision. It ships as session affinity, one of eight scoring signals: a multi-turn conversation is pinned to the node holding its warm prefix, so turn N+1 re-uses the cache instead of re-encoding the history.

Two design details matter here, and they are worth stating honestly:

• The bonus is deliberately not decisive. A warm cache is worth a lot, but the bonus is capped at +20, below the +50 for having the model loaded, and it shrinks as the pinned node's queue grows, so a saturated node loses to an idle peer. Pinning conversations to a machine that is already backed up would trade one problem for another.
• It composes with the other seven signals (model already loaded, memory fit, queue depth, latency history, role affinity, availability, context fit), so affinity is weighed against real conditions rather than applied blindly.

Here is what "shrinks as the queue grows" means in numbers. The bonus is 20 / (1 + depth / 2), where depth counts the requests already running or waiting for that model on the pinned node, and the queue-depth signal takes 6 points per request. Against an otherwise identical idle node, the pinned node leads by 20 with an empty queue and by 7.3 with one request, then trails by 2 at a depth of two.

See the routing engine guide for exactly how the signals are scored and combined.

How to check what your setup is doing

Don't take anyone's word for it, including ours. Send the same conversation twice and look at which machine answered:

curl -i http://localhost:11435/api/chat \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3.6:27b","messages":[{"role":"user","content":"Hello"}],"stream":false}' \
  | grep -i x-fleet

Every response carries X-Fleet-Node (which machine answered) and X-Fleet-Score (its winning score). Run a first turn, then a follow-up in the same conversation, and compare:

• Same node both times means the router kept the follow-up where its prefix cache can be re-used. The header reports the routing decision, not a cache hit: whether Ollama still held that prefix depends on its parallel slots and memory.
• Different node means you just paid a full prefill for the second turn.

That single comparison tells you more about your fleet's real behavior than any throughput benchmark.

What this does not fix

Cache-aware routing removes the repeat cost, not the first one. The initial prefill of a long prompt still takes what it takes, and prompt processing is a genuine hardware and model constraint (most noticeably on Apple Silicon) that no router can route around. If your very first message on a fresh conversation is slow, routing is not your problem; model size, quantization, and context length are (the memory guide covers that sizing).

What routing fixes is the waste: paying that same cost again on every turn because the request landed somewhere with no memory of the conversation. On a multi-machine setup, that waste is usually the larger number.

Related Reading

• Open WebUI with multiple Ollama servers, the full setup walkthrough
• Routing engine, how the 8 signals are scored
• Concurrency tuning, when the bottleneck is one machine instead of routing