Routing Engine
How the router makes smart decisions about where to send every request.
The Five-Stage Pipeline
Every incoming request passes through five stages. The first three decide where it runs; the last two run in the background, keeping the next requests fast.
node:model queue and runs there, wholeThe goal of every decision is to minimize total response time:
total_time = queue_wait_time + cold_load_time + inference_time
A request routed to a node with the model already loaded and an empty queue beats a request routed to an idle node that needs to load the model first.
Stage 1: Elimination
Before any scoring, the router removes nodes that can't serve the request. This is binary, pass or out.
| Condition | Outcome If False |
|---|---|
| Node is online (heard from within the last 30 seconds) | Eliminated |
| Model is on disk on this node | Eliminated |
| Node has enough memory to load the model (free RAM plus GPU memory the OS does not count as RAM, capped by the capacity ceiling) | Eliminated |
| Not under critical memory pressure, unless the model is already loaded there | Eliminated |
| Node is not in hard-pause mode | Eliminated |
Critical memory pressure (reported on macOS and Linux) withholds cold loads, not service: since v0.10.0 a node under critical pressure keeps answering for the models it already has loaded, which allocate nothing new, and is eliminated only for a model it would have to load. Refusing a request frees no memory, because the memory is held by the resident weights, and eliminating the node outright left a one-machine fleet unable to answer anything.
Hard-pause triggers, with capacity learning turned on (opt-in): camera or microphone active (meeting in progress, macOS only), CPU above 85% for 2 minutes or more, availability score below 0.20, or the first 7 days of observation.
If nothing survives: The request enters a holding queue. The router rescores it every 2 seconds, for up to 30 seconds, as node states change. The request waits rather than errors.
Stage 2: Scoring
Each surviving node gets scored across 8 weighted signals, and from v0.12.10 charged for any model a cold load there would push out of memory (the eviction charge). Higher total wins.
Signal 1: Model Warmth (up to +50)
The most important signal. Cold-loading a 40GB model takes 15–30 seconds.
| State | Points |
|---|---|
| Model currently loaded in memory (hot) | +50 |
| Model loaded but partly on the CPU (Ollama could not fit it on the GPU; v0.12.12) | +10, the same as on disk |
| Model on disk, loaded within last 30 min (likely OS-cached) | +30 |
| Model on disk, not recently used | +10 |
| Model not on disk (requires download) | +0 |
The partly-on-the-CPU rule applies only when Ollama reports how much of the model sits in GPU memory (size_vram) and that share is above zero but under 90% of the model’s size. Nodes that report nothing (older agents, machines with no GPU) score a loaded model +50 as before, and Apple Silicon is never affected, because the whole model is in unified memory. The effect: a fully resident copy on another machine wins, instead of the request landing on a card where the model crawls.
The "recently loaded" tier exists because the operating system keeps recently read model files in its page cache (macOS is especially aggressive about this), so a model unloaded 20 minutes ago often reloads much faster. From v0.12.10 it applies only when loading the model back evicts nothing; otherwise the node scores +10.
Eviction Charge (down to -50)
New in v0.12.10. A node that can load the model into memory that is already free, with a model slot to spare, pays nothing. Otherwise Ollama would push resident models out to make room, so the router charges the node for each one it would lose, cheapest first, by how much that model is in use:
requests running or queued for it -40 (hot minus cold)
used recently -20 (warm minus cold)
idle -10
memory in use that no loaded model accounts for
(a load in progress, another program) charged as busy
a model no other online node can hold -50 (v0.12.11)
total capped at -50
A node with room therefore beats one that must evict a busy model by more than the warm-to-cold gap, so two models that cannot share one GPU settle on different machines instead of swapping on every request. A loaded model costs nothing here. The charge appears as eviction in each trace’s score breakdown. When every node would have to evict, the request may be served by a loaded model of the same category instead (see the fallback chain).
From v0.12.11 two more rules keep a mixed fleet from thrashing or starving. A node whose load would push out a model that no other online node can hold is skipped while any other node can serve, so a 35 GB model on the only 48 GB card stays there while smaller models swap on the smaller cards. And when a load has to evict a busy model, the router stops pinning keep_alive for that node’s other models and, after at most 30 seconds, holds their new requests until the load has started, because Ollama only evicts a model once its requests have finished. Requests for a model already loading on a node join that load. The benchmark shows the effect.
Signal 2: Memory Fit (up to +20)
How comfortably the model fits given current utilization and the node's dynamic memory ceiling.
available = free memory a load can use without evicting a model:
free GPU memory on a node with a GPU, free RAM on unified memory,
capped by the capacity ceiling if learning is on
fit_ratio = available / model_memory_gb
fit_ratio > 2.0 +20 comfortable headroom
fit_ratio 1.5-2.0 +15
fit_ratio 1.2-1.5 +8
fit_ratio 1.0-1.2 +3 tight -- risk of memory pressure
fit_ratio < 1.0 eliminated in Stage 1
A model that is already loaded gets the full +20. model_memory_gb is the model's measured resident size when the fleet has seen it loaded (weights plus KV cache), otherwise its weights. On a node with capacity learning turned on, the adaptive ceiling from the capacity learner also caps available: a 128GB laptop in learned-low mode has a 16GB ceiling.
Signal 3: Queue Depth Penalty (up to -30)
A hot model on a saturated node is less attractive than a warm model on an empty node. From v0.12.12 embeddings count in the depth while they run, so a backlog of them on one node sends the next ones to another.
depth = in_flight + pending
penalty = min(30, depth x 6)
A queue of 5 subtracts the maximum 30 points. This naturally spreads load when multiple nodes can serve the same model.
Signal 4: Estimated Wait Time Penalty (up to -25)
Queue depth alone is misleading. A queue of 3 on a fast 7B model completes in seconds. A queue of 3 on a slow 70B model takes minutes.
The router uses p75 historical latency per node per model:
est_wait = (in_flight + pending) x p75_latency_ms
penalty = min(25, est_wait_seconds / 10)
During the first 7 days before enough data is collected, the router uses a heuristic based on memory bandwidth and model size.
Signal 5: Role Affinity (up to +25)
Large models belong on fast machines. Small models should run on lighter hardware to preserve the fast machine's capacity. The router scores this from each node's memory bandwidth when it's known (the real bottleneck for prompt processing), and falls back to total memory when it isn't.
bandwidth bonus = 5 + bandwidth_GBps / 40, capped at 25
100 GB/s ~+7.5 400 GB/s ~+15 800 GB/s +25
Model over 20 GB: full bandwidth bonus (fast nodes win)
Model under 8 GB: 18 - 0.6 x bonus, at least +3 (slower nodes win)
In between: 0.6 x bonus
Bandwidth unknown (memory-tier fallback):
Model over 20 GB: 128GB+ node +15, 32GB+ node +5, smaller +0
Model under 8 GB: 32GB-or-less node +15, up to 128GB +8, larger +3
Machine brand never enters into it: a Linux server or Windows PC is placed by the same numbers as a Mac. Size thresholds are configurable (FLEET_SCORE_ROLE_LARGE_THRESHOLD_GB, FLEET_SCORE_ROLE_SMALL_THRESHOLD_GB).
Without affinity, every small-model request would drift to the most powerful machine, crowding out the large models it's uniquely suited for.
Signal 6: Availability Trend (up to +10)
Only nodes with capacity learning turned on (it is opt-in) get this signal; every other node scores 0 here. The bonus is the node's availability score, 0.0 to 1.0, times 10:
bonus = availability_score x 10
The availability score blends the learned baseline for this hour of the week, the current CPU and memory load, and whether CPU use has been rising or falling over the last 5 minutes (see Adaptive Capacity). Rising use lowers the score, which keeps a long inference request off a machine whose owner just sat down to start working.
Signal 7: Context Fit (+15 to -15)
Rewards nodes whose loaded context window comfortably handles the estimated request size. Penalizes nodes where the request might trigger a context resize (which causes a model reload).
Signal 8: Session Affinity (up to +20, or +10 for a shared prompt)
Pins a multi-turn conversation to the node that already holds its warm prefix cache. On turn N+1 that node re-uses the cached prefix, so it costs a few hundred prefill tokens instead of re-encoding the whole ~30K-token history, and it avoids the decode interference a fresh prefill inflicts on other streams sharing the node. The bonus is deliberately not decisive: a warm cache is worth a lot, but the bonus stays below Signal 1's +50 and shrinks as the pinned node's queue grows, so a saturated node loses to an idle peer. Added in v0.9.1.
session bonus = 20 / (1 + depth / 2)
depth = requests running or waiting for this model on that node
Prefix affinity (since v0.10.0) is the same idea for a conversation’s first turn, which has no session to pin yet. When its system prompt and tool list come to at least 2,000 tokens and another conversation already sent the same ones to a node, that node gets up to +10, decayed by queue depth the same way. Five agents sharing one large system prompt start where it is already cached, without all piling onto one machine. It applies only when the session bonus is 0, and the router keeps only a hash of the prompt, never its text.
Signal 3 takes 6 points per queued request at the same time, so between two otherwise identical nodes the session's node leads by 20 with an empty queue, by 7.3 with one request, and trails by 2 once two requests are running or waiting there.
Stage 3: Queue and Execute
The highest-scoring node wins. The request enters that node's dedicated queue for that model. Each node+model pair has its own queue with dynamic concurrency calculated from available memory.
Stage 4: Pre-Warm
Every 5 seconds, a background loop checks each queue: is it getting deep?
If a queue holds 3 or more requests, running plus waiting (the pre-warm threshold), the router loads the same model on the runner-up node by sending an empty generate request. By the time overflow requests arrive, the model is already hot.
A lock prevents duplicate pre-warm requests for the same model on the same node. From v0.12.10 a pre-warm never evicts: if loading on the runner-up would push out a resident model, it is skipped, since a speculative load that evicts a busy model only moves the cold load to the next request.
Stage 5: Rebalance
A background process runs every 5 seconds:
- Scans all queues for ones with 4 or more requests waiting (the rebalance threshold)
- For each overloaded queue, checks if another node has the same model hot with spare capacity
- Moves pending requests (not in-flight) to the better node
- Caps movement at 3 requests per cycle to prevent oscillation
The rebalancer only moves requests to nodes where the model is already loaded, it never triggers cold loads.
Fallback Chain
When a request lists fallback models, the router scores the primary and every fallback in the same pass, each through the full elimination and scoring pipeline:
- If any of them is already loaded on a surviving node, the first such model in list order wins. A loaded fallback is used rather than cold-loading the primary.
- Otherwise the first model in the list that some node can load wins. If its best node can load it without evicting anything, that node loads it.
- If every node would have to evict a resident model to load it, the request is served by a model of the same category that is already loaded, when there is one (v0.12.10; a model of another category is never substituted). The response’s
X-Fleet-Fallbackheader and the trace say so. Strict mode,X-Fleet-No-Fallback: trueor"fallback": falsein the request, turns this off. Failing that, the best node evicts and loads the requested model. - If no node can serve any of them, the request waits in the holding queue, rescoring the whole list every 2 seconds for up to 30 seconds.
- If nothing frees up (and auto-pull cannot fetch the model), the router returns 503.
Configuration
All scoring weights are tunable via environment variables:
| Variable | Default | Effect |
|---|---|---|
FLEET_SCORE_MODEL_HOT | 50 | Increase to prefer hot models more aggressively |
FLEET_SCORE_QUEUE_DEPTH_PENALTY_PER | 6 | Decrease to tolerate deeper queues |
FLEET_PRE_WARM_THRESHOLD | 3 | Lower to pre-warm earlier |
FLEET_REBALANCE_THRESHOLD | 4 | Lower to rebalance more aggressively |
See the full configuration reference for all 44+ variables.
Next Steps
- Adaptive Capacity, How the capacity learner feeds into Signal 2 and Signal 6
- Deployment, Monitoring and tuning in production
- API Reference, Response headers that show routing decisions