Multi-machine Ollama benchmark: 4x throughput at saturation on four GPUs
More machines add capacity for concurrent requests, not speed for one request. Measured on 1 to 5 NVIDIA machines, with the raw data and the scripts to rerun it.
We rented NVIDIA machines on Lambda Cloud twice, five each time, and measured what a fleet is supposed to fix: running out of throughput on one machine, a machine going down, two models fighting over one GPU, unequal hardware, and coding, embeddings and image questions sharing one box. We also put Herd next to a plain load balancer on the same machines. Most results come from released Ollama Herd 0.12.10 and 0.12.11, and every chart names the build it measured. Herd is free and open source, and the scripts and raw results are published, so you can run every test here on your own machines.
Herd sends each whole request to one machine; it does not split a model across machines. More machines add capacity for more requests at once, not speed for one.
Results at a glance
Each result with the conditions it was measured under and what it does not show.
| Result | Measured with | What it does not show | Data |
|---|---|---|---|
| 4.0x aggregate throughput on four machines: 480 tokens/s against 120 | Four identical 24 GB GPUs, one model, enough concurrent requests to fill every machine; released 0.12.10 | One request is not faster: 78 to 80 tokens/s on any number of machines | JSON |
| 4, 8 and 16 concurrent requests on 1, 2 and 4 machines with 95% of first tokens within 2 seconds | Same runs, Ollama set to run 4 requests at once on each machine; held in all three | The limit is Ollama's 4 slots per machine; Herd's part is filling every slot | JSON |
| 14 of 1,174 requests failed while two of four machines went down, in three runs | Four A10 24 GB, 8 concurrent requests; one machine's Ollama stopped, another machine vanished | The failures were replies already streaming on the dying machines; Herd retries only before the first token | JSON |
| 45 of 45 requests answered on mixed hardware in each of three runs, against 35 to 41 through HAProxy, a conventional, model-unaware load balancer | Two A10 24 GB and an A100 40 GB, four models, one request every 4 seconds; the 30 GB model fits only the big card | HAProxy's misses were that model sent to cards that don't have it; past what a fleet can serve, every front door queues | JSON |
| 63 ms to the first token instead of 7.8 seconds when two models can't share one GPU | Two 15.7 GB models alternating on 24 GB cards: two machines behind Herd against one machine with Ollama directly | Leaves out each model's first, cold request (36 and 8 seconds through Herd) | JSON |
| 1,629 image questions answered alongside coding and embeddings, where one machine answered 14 | Three machines (two A10 24 GB, one A100 40 GB) against one A10 24 GB holding all three models; median of three runs | Not a controlled comparison: the three-machine arm also has more GPU memory; in one run embeddings stayed on a busy card and only 50 were answered (0.12.12 counts embeddings as load to stop that) | JSON |
| 0.38 s instead of 2.9 s to the first token after 6 idle minutes | One Mac, qwen3-coder:30b, Ollama's default 5-minute keep-alive, through Herd against straight to Ollama | Herd keeps the model's memory held until another model needs it; three runs per arm | JSON |
| 6.3 ms added per request by the router, at the median | Router and node on separate machines on a private network, 200 alternating pairs | One router took about 175 requests a second on a 14-vCPU cloud machine before requests queued | JSON |
What should I do?
What the results mean for a purchase or a deployment, each row linked to the test behind it.
| If you have | Do this | Measured in |
|---|---|---|
| One user, one model, one request at a time | The fastest single GPU you can afford: more machines do not make one reply faster | Scaling, big or small |
| Several users or agents at once | Add machines: here each one added 4 more requests at once within a 2 s wait | Scaling |
| Models that cannot share one GPU | Spread them across machines so each stays loaded | Two models, one GPU |
| Mixed GPU sizes and several models | Model-aware routing: a conventional load balancer sends models to cards that cannot hold them | Against HAProxy |
| A model bigger than any one machine | A bigger GPU: Herd sends each request to one machine and does not combine memory across machines | How Herd routes |
| Mostly idle machines | Herd keeps models loaded, so the first request after a quiet spell skips the reload; that memory stays held | Idle test |
| Long multi-turn agent sessions | Test them before relying on them: the first turn of a session is currently slower through Herd | Limitations |
Throughput as you add machines
One model, qwen3-coder:30b, loaded on 1, 2 or 4 identical machines, a fixed number of requests in flight for two minutes per measurement, on Ollama Herd 0.12.10.
| Concurrent requests | 1 machine | 2 machines | 4 machines |
|---|---|---|---|
| 1 | 80 tok/s 562 / 595 ms | 80 tok/s 571 / 607 ms | 80 tok/s 549 / 587 ms |
| 2 | 90 tok/s 756 / 1,107 ms | 170 tok/s 274 / 576 ms | 170 tok/s 532 / 572 ms |
| 4 | 120 tok/s 1,296 / 1,596 ms | 195 tok/s 763 / 1,112 ms | 338.3 tok/s 438 / 578 ms |
| 8 | 120 tok/s 8,478 / 8,718 ms | 228.3 tok/s 1,309 / 1,586 ms | 350 tok/s 762 / 1,117 ms |
| 16 | not run | 240 tok/s 8,504 / 8,765 ms | 463.3 tok/s 1,309 / 1,589 ms |
| 32 | not run | not run | 480 tok/s 8,495 / 8,824 ms |
Aggregate completion tokens per second, then the time to the first token at the median and at the 95th percentile. Bold: the most concurrent requests each fleet served with 95% of first tokens within 2 seconds. The multiples quoted in the text (2.00x, 4.00x) are calculated from these measurements; the chart shows the range across runs. Raw data.
- Requests per machine is what counts. One machine stops gaining at 4 requests, its four Ollama slots; at 8, half the requests wait and the first token takes 8.5 seconds at the median. Four machines with 8 requests have only two each in flight, so they deliver 350 tokens/s, not four times one machine.
- Placement was even once every machine had work. With at least as many requests as machines, each machine served within a few requests of its share. With fewer, the spare machines sat idle: one request runs on one machine.
- The runs agree. No failed request in 45 runs, and saturated throughput was the same in all three. The widest spread was two machines at 4 requests (163 to 203 tokens/s).
Capacity at a wait you would accept. With 95% of requests getting their first token within 2 seconds, one machine served 4 concurrent requests, two served 8, four served 16, in every run. Saturated, that is 120, 240 and 480 tokens per second: 4.0x one machine on four. The limit comes from Ollama: each machine runs 4 requests at once (OLLAMA_NUM_PARALLEL=4) and the next one waits for a free slot. Herd’s part is spreading requests so that every slot is in use. Doubling the requests past that point gives a little more throughput and an 8.7 second wait.
Getting OLLAMA_NUM_PARALLEL right matters more than any routing decision here. Before 0.12.10, a node whose Ollama ran as a systemd service reported no parallelism, so Herd sent it one request at a time. How Ollama handles concurrent requests covers the setting.
Losing a machine under load
Four A10 24 GB machines, 8 requests in flight for four minutes. A minute in, Ollama stopped on one machine; at two and a half minutes a second machine vanished: its herd-node killed without a goodbye and its Ollama stopped. Ollama Herd 0.12.11.
14 of 1,174 requests failed across the three runs (4, 5 and 5). Each was a reply already streaming on a machine as it died: once tokens have reached the client, Herd cannot move the reply to another machine. No request sent to a dead machine failed. Herd keeps routing to a machine for up to 30 seconds after its last status update, and each request that landed there was retried on another machine before its first token. Throughput stepped down as capacity left, from about 450 tokens/s on four machines to about 250 on two. An earlier run on four Quadro RTX 6000 cards with 0.12.10 looked the same: 4 of 342 failed.
Against a plain load balancer
The same machines behind HAProxy, a conventional, model-unaware load balancer, instead of Herd. On four identical A10 machines with one model: 440 tokens/s through Herd, 400 through HAProxy round robin. Round robin sends each request to the next machine in turn whether or not it is free, and Ollama’s logs show some machines holding three requests while another held one; Herd kept two on each. On four Quadro RTX 6000 machines in an earlier run the two tied, 361.7 against 363.3, so how much the uneven spread costs depends on the card. Routing decisions matter more when the machines and models differ, so the second test used two A10 24 GB cards and an A100 40 GB with four models chosen at random: qwen3-coder:30b, qwen2.5:14b, phi4:14b and qwen2.5:32b at q6_K, a 30 GB model only the A100 can hold. Four models on three cards means some model swapping whoever routes.
Ollama’s own logs show the difference on identical machines: Herd held each one at two requests 97% of the time against 76% for round robin.
| Front door | Machines | Throughput | Answered | First token, median | p95 | 30 GB model served | Model loads |
|---|---|---|---|---|---|---|---|
| Ollama Herd | 4 identical | 440 tok/s (415 tok/s to 440 tok/s) | 0.6 s | 0.9 s | |||
| HAProxy round robin | 4 identical | 400 tok/s (396.7 tok/s to 401.7 tok/s) | 0.7 s | 0.7 s | |||
| Ollama Herd | mixed | 45 of 45 | 1.4 s | 15.0 s | 29 of 29 | 11 | |
| HAProxy least connections | mixed | 38 of 45 (35 to 41) | 5.6 s | 8.3 s | 8 of 29 | 23 | |
| HAProxy round robin | mixed | 40 of 45 (36 to 41) | 5.7 s | 10.8 s | 11 of 29 | 27 |
The range across runs in brackets. Identical machines: four A10 24 GB, one model, 8 concurrent requests, 120 s. Mixed: two A10 24 GB and one A100 40 GB, four models chosen at random, one request every 4 seconds for 180 s, seeds 7, 11 and 13, the same start for each front door; only the A100 40 GB holds the 30 GB model, whose requests are counted across the three runs. Ollama Herd 0.12.11; HAProxy 2.4, a conventional, model-unaware load balancer. Raw data.
At one request every 4 seconds Herd answered 45 of 45 in each of the three runs, with fewer than half the model loads and the shortest median wait for a first token (its 95th percentile was longer, 15.0 s against 8.3 s for least connections). HAProxy least connections answered 35 to 41: its misses were the 30 GB model, sent to A10s that don’t have it, and it served 8 of 29 requests for that model across the runs. Run 4, on Quadro RTX 6000 and RTX A6000 cards with the 35 GB q8 model, found the same: 45 of 45 against 41 of 45. At twice that rate (run 4 only) the fleet is past what it can serve without constant swapping, and every front door queued for tens of seconds. Herd answered 76 and 90 of 90 in two runs; HAProxy least connections 76 and 78, round robin 80.
This test found two placement bugs, both fixed in 0.12.11 before these numbers. Herd evicted the 35 GB model, which nothing else could hold, to load a 15 GB one the small cards could take. And when every card held a busy model, Herd’s own keep_alive on each request cancelled Ollama’s eviction, so one model starved until its requests timed out. With 0.12.10 Herd answered 57 of 90 at the higher rate.
Two models that can’t share one GPU
qwen2.5:14b and phi4:14b at 8-bit take about 15.7 GB each, so a 24 GB card holds one at a time. A client alternates between them for 20 requests. On one machine Ollama has to evict one model to load the other, every time. Behind Herd, with a second machine, each model can stay loaded somewhere. Measured on the main branch shortly before 0.12.10.
The first request for each model is a cold load either way (36 and 8 seconds through Herd). After that the single machine reloads on every request, 7.6 to 8.7 seconds to the first token, and the two machines never do: 59 to 92 ms, so the 20 requests took 82 seconds instead of 248. Every answer came from the model asked for: the test sends Herd’s strict-mode header, which turns a substitution into an error.
One big card or two small ones
Case study, not a controlled test: the cards differ in class as well as size.
The buyer’s question: one A100 40 GB, or two A10 24 GB behind Herd? Both are Ampere cards; the A100 is a data-centre card with faster memory, the A10 an inference card. Same router for both, Ollama Herd 0.12.11.
| Workload | One A100 40 GB | Two A10 24 GB |
|---|---|---|
| One request | 115 tok/s, first token 0.3 s | 93.3 tok/s, first token 0.4 s |
| 8 at once | 206.7 tok/s, first token 5.0 s | 273.3 tok/s, first token 0.8 s |
| Mixed: coding requests | 104, p95 1.8 s | 118, p95 1.0 s |
| Mixed: embeddings | 8,334 | 10,023 |
| Price | $1.99 an hour | $2.58 an hour |
| Tokens per dollar, 8 at once | 373,930 | 381,349 |
qwen3-coder:30b; the mixed run adds qwen3.5:9b and nomic-embed-text, four workers each, 180 s. Same router for both, Ollama Herd 0.12.11. No failures in any run. Raw data.
The big card wins one request at a time: 115 tokens/s against 93.3. As soon as requests overlap, two small cards win: 273.3 tokens/s against 206.7 at 8 at once, with a much shorter wait for the first token, because they have twice the slots. At $1.29 an hour each against the A100’s ($1.99), that came to about the same work per dollar. An earlier run with one RTX A6000 48 GB and two Quadro RTX 6000 24 GB showed the same pattern, and there the two small cards did about a fifth more work per dollar.
Coding, embeddings and image questions at once
A coding model (qwen3-coder:30b), an embedding model (nomic-embed-text) and a small vision model (moondream, one image question per request), four clients each for three minutes. Once on one A10 24 GB machine with all three loaded, once on three: two A10 cards and the A100 40 GB. Same router for both, Ollama Herd 0.12.11.
This compares two realistic deployments, not machine count alone: the three-machine arm also has more GPU memory and a larger card. It shows what a fleet does for a mixed workload; it is not per-GPU scaling, which the first test isolates.
- On one machine the image model starved. It answered 14 questions in three minutes, waiting tens of seconds for each first token, while the coding model and embeddings kept the card busy.
- On three machines Herd separated the work: image questions on the A100, 1,629 answered, and the coding model answered more requests, faster. In two of the three runs embeddings went to the A100 too. In the other, the first embedding requests reached an A10 before the router knew the A100 had the model loaded, every later one followed them there, and only 50 were answered: a routing bug in 0.12.11. In 0.12.12 embeddings count as load, so they move off a backed-up card.
0.12.12 goes after both: it counts embeddings as load and scores a model loaded partly on the CPU like a cold load. Run again on the same fleet, alternating with 0.12.11, chat throughput went from 401.9 to 503.1 tokens per second and the 95th-percentile first token from 576.9 ms to 71.1 ms.
With the first embeddings steered onto a busy A10 on purpose, 0.12.11 sent the idle A100 almost none of the rest. The first version of the fix sent it more than half, and twice as many embeddings were answered.
After a machine sits idle
Most herds in the community stats are one or two machines with a light load, so the request people wait on most is the first one after a quiet spell. Ollama unloads a model after 5 minutes without requests (its default keep_alive), and the next request has to load it again. Herd sends keep_alive: -1 with the requests it routes (except while another model is waiting to load on that machine), so the model stays loaded. We measured both on one Mac.
| Model | Straight to Ollama | Through Herd |
|---|---|---|
qwen3-coder:30b | 2,869 ms (reloaded, 2,868 to 2,876) | 382 ms (still loaded, 379 to 382) |
llama3.2:1b | 1,094 ms (reloaded) | 120 ms (still loaded) |
Time to the first token of the first request after 6 minutes with no traffic, the median where a model ran more than once. One Apple Silicon Mac with 128 GB, Ollama 0.40.15 with default settings. qwen3-coder:30b: three runs per arm on Ollama Herd 0.12.11. llama3.2:1b: one run per arm on Ollama Herd 0.12.10. Raw data.
With qwen3-coder:30b, the first request after six idle minutes took 2.9 seconds to its first token straight to Ollama, with the model file still in the operating system’s cache (10.6 seconds the first time, from disk), and 0.38 seconds through Herd, the same in all three runs of each. The cost is memory: Herd keeps the model loaded until another model needs the room. Requests sent in Ollama’s own format can still set their own keep_alive.
How it was measured
| Value | |
|---|---|
| Machines | Lambda Cloud, one private network per run. November 20: 4 x NVIDIA Quadro RTX 6000 24 GB ($0.69 an hour each), 1 x RTX A6000 48 GB ($1.09), for scaling and model swapping. November 21: 4 x A10 24 GB ($1.29 an hour each), 1 x A100 40 GB ($1.99), for losing a machine, the load balancer, big or small and vision (the RTX 6000 and A6000 were out of capacity) |
| Operating system | Ubuntu 22.04 |
| Ollama | 0.40.16 (the idle test 0.40.15, the 0.12.12 checks 0.40.17), OLLAMA_NUM_PARALLEL=4 in its systemd unit (Ollama’s documented Linux method). On November 21 also OLLAMA_CONTEXT_LENGTH=4096 on every machine: Ollama sizes its default context from GPU memory, which gave the A100 eight times the A10’s and put the 32B model partly on the CPU |
| Ollama Herd | 0.12.10 from PyPI for scaling; 0.12.11 from PyPI for losing a machine, the load balancer, big or small and vision; 0.12.10 and 0.12.11 for the idle test; the code released as 0.12.12 for the 0.12.12 check. A few tests ran a commit between releases, and their charts name it. Default settings, one router, telemetry off |
| Load balancer | HAProxy 2.4, HTTP mode, round robin and least connections, in front of the same Ollamas |
| Requests | About 1,250 prompt and 200 completion tokens each (throughput counts the completion tokens of requests that finished), the average request size reported on ollamaherd.com/stats; a different prompt per request so no answer comes from a cache |
| Warm-up | Every machine loaded its models before each measurement and was checked to hold exactly them, so cold loads are not counted unless the test is about them |
| Repeats | Fleet tests ran three times, two to four minutes a run, and the page quotes the median. Once: model swapping, router overhead and its ceiling, the 256-machine test, the A6000 point and the llama3.2:1b idle row. Twice: the stranding test. The load balancer test used three random seeds |
What the router adds
The same short request, 200 times straight to the node’s Ollama and 200 times through Herd 0.12.10, alternating so that drift lands on both. Through Herd it took 6.3 ms longer at the median (the 95th percentile was 7.1 ms higher), and Herd’s own X-Fleet-Overhead-Ms response header reported 6.0 ms. On 0.12.9 the same test measured 23.8 ms.
How many requests one router can take, measured the way API gateways publish it: a stand-in for Ollama that answers every request after a fixed 20 ms delay, requests sent at a fixed rate whether or not earlier ones have finished, direct and then through Herd.
| Requests per second | Added at the median | p99 through Herd | Failed |
|---|---|---|---|
| 100 | 7.1 ms | 37 ms | 0 |
| 125 | 5.8 ms | 49 ms | 0 |
| 150 | 11.5 ms | 83 ms | 0 |
| 175 | 11.2 ms | 104 ms | 0 |
| 200 | 204.1 ms | 798 ms | 0 |
| 300 | overloaded | 1,707 of 6,000 | |
One router process on a 14-vCPU AMD EPYC cloud machine, 20 seconds per rate; the stand-in held 24 ms on its own at every rate up to 800 per second. Raw data.
One router took about 175 requests a second here, adding about 11 ms at the median; at 200 a second requests queued and it added 204 ms. A chat request keeps a GPU busy for seconds, so LLM traffic comes nowhere near that: the four-machine run above peaked at about 2 requests a second. Embedding ingestion is where the limit can matter; send texts in batches (an array per request) rather than one request each.
How many machines one router can manage is a separate question from how many requests it can take. We ran the released router on one Mac against 256 simulated machines (each sending the status updates a real herd-node sends every 5 seconds) while it routed 50 requests a second: no update failed, choosing a machine took 1.5 ms per request at the median, and the router still added about 3 ms. The largest herd in the community stats has 14 machines. Past its request ceiling the router is slow to answer those updates too, which is open work. Raw data.
What these results do not show
How strong each result is: how often it was run and whether the setups it compares differ in more than one way.
| Experiment | Runs | Hardware controlled | Latency reported | Evidence |
|---|---|---|---|---|
| Scaling | 3 | yes | p95 | replicated |
| Losing a machine | 3 | yes | completions | replicated |
| Against HAProxy, identical machines | 3 | yes | p95 | replicated |
| Against HAProxy, mixed hardware | 3 | yes | p95 | replicated |
| Two models, one GPU | 1 | yes | per request | one run |
| One big card or two small | 3 | no | p95 | case study |
| Coding, embeddings and images | 3 | no | p95 | case study |
| After a machine sits idle | 3 | yes | first token | replicated |
| Router overhead | 1 | yes | p95 | one run |
| Router with 256 machines | 1 | yes | p99 status updates | simulation |
Replicated: three or more runs on controlled hardware. One run: controlled, not yet repeated. Case study: the compared setups differ in more than one way. Simulation: simulated machines.
- Multi-turn agent sessions. We measured them, and the first turn of a session through Herd was about twice as slow as spreading requests evenly. That is open work, not a result.
- Mac fleets. The fleet tests ran on rented NVIDIA machines; only the idle test ran on a Mac. Herd runs the same routing code on Macs, where memory bandwidth and the MLX backend also come into play. Have a Mac fleet? Run the scripts below and send us the numbers.
- A fleet past capacity. When several models compete for one card, Herd still swaps them request by request rather than batching each model’s requests. That is open work.
- Per-GPU scaling from the vision or big-or-small tests. Their arms differ in memory and card type as well as machine count.
Run it yourself
Ollama Herd is free and open source: pip install ollama-herd, run herd on one machine and herd-node on each of the others (every install option). Then measure your own fleet and share what you find, especially on hardware we haven’t tested.
Set OLLAMA_NUM_PARALLEL the same on every machine, load each model once so cold loads don’t count, then run the commands below and compare your JSON with ours. The scripts are in Ollama Herd 0.12.11; every figure and table above links its raw results.
Throughput, from any machine that can reach the router, with scripts/benchmark.py (--output keeps every request):
python scripts/benchmark.py --url http://<router>:11435 --models qwen3-coder:30b \
--concurrency 8 --duration 120 --prompt-tokens 1250 --max-tokens 200 --output run.json
Model alternation, once against one machine’s Ollama (add --fleet-url none) and once against the router:
python scripts/benchmark.py --url http://<router>:11435 --api openai \
--models qwen2.5:14b-instruct-q8_0 phi4:14b-q8_0 --sequence alternate \
--concurrency 1 --requests 20 --max-tokens 128 --skip-warmup --output alternation.json
Mixed hardware at a fixed rate, through Herd or, with --url pointing at a load balancer, observed through Herd with --fleet-url:
python scripts/benchmark.py --url http://<router>:11435 \
--models qwen3-coder:30b qwen2.5:14b-instruct-q8_0 phi4:14b-q8_0 qwen2.5:32b-instruct-q8_0 \
--sequence random --seed 7 --rate 0.25 --duration 180 --skip-warmup --timeout 300 --output mixed.json
Router overhead, with the router on one machine and Ollama on another, with scripts/router_overhead.py:
python scripts/router_overhead.py --ollama http://<node>:11434 \
--router http://<router>:11435 --model qwen3-coder:30b --requests 200
For the per-second table, run scripts/mock_ollama.py --workers 8 on the node machine, start that machine’s herd-node pointed at it with OLLAMA_NUM_PARALLEL=64 (otherwise Herd’s own per-model queue limit is what you measure), and send load with router-ceiling-loadgen.py.
Record your Ollama version with the results. A run with telemetry on reports a test herd to the public stats: set FLEET_NODE_TELEMETRY=false on the router while benchmarking.
FAQ
Does Ollama run faster on two machines?
One request does not: a single chat request runs on one machine, so it streams at that machine's speed (78 to 80 tokens per second for qwen3-coder:30b on an RTX 6000, however many machines there are). Total throughput does. With enough concurrent requests, two machines delivered 240 tokens per second against one machine's 120, and four delivered 480.
How much faster are four machines than one?
4.0x in aggregate throughput, once there are enough requests to keep every machine busy: 480 tokens per second on four NVIDIA Quadro RTX 6000 cards against 120 on one, the same in each of three runs. With 8 concurrent requests, four machines delivered 350, because each had only two requests in flight.
Is Ollama Herd better than a conventional load balancer like HAProxy?
On mixed hardware with several models, yes in our tests: at one request every 4 seconds Herd answered 45 of 45 in each of three runs, HAProxy least connections 35 to 41, failing the model only the A100 could hold. On identical machines serving one model the gap is small and depends on the card: Herd delivered 440 tokens per second against round robin’s 400 on four A10s, because round robin left some machines with three requests and others with one, and they tied on four Quadro RTX 6000s (361.7 against 363.3).
What happens when a machine goes down?
Requests move to the machines that are left. In our test two of four machines went down under load, three times: 14 of 1,174 requests failed, all replies that were already streaming on a dying machine, which Herd cannot move once tokens have reached the client. Requests that landed on a dead machine were retried elsewhere before their first token.
Should I buy one big GPU or two smaller ones?
For one request at a time, one big card: an A100 40 GB streamed qwen3-coder:30b at 115 tokens per second against 93.3 on an A10 24 GB. For several requests at once, two small cards: 273.3 tokens per second against 206.7 at 8 concurrent requests, with a much shorter wait for the first token, for about the same cost per token at Lambda’s prices. An earlier run with an RTX A6000 and two Quadro RTX 6000 cards showed the same pattern.
What happens when two models do not fit on one GPU?
On one machine Ollama evicts one model to load the other, so alternating between them costs a reload on every request: about 7.8 seconds to the first token on a 24 GB card. Behind Ollama Herd on two machines each model stayed loaded on its own machine, and the first token came in about 63 ms.
How much latency does Ollama Herd add?
About 6 ms per request: 6.3 ms at the median, with the 95th percentile 7.1 ms higher, over a private network against the same request sent straight to Ollama (200 alternating pairs, Ollama Herd 0.12.10). One router process on a 14-vCPU cloud machine took about 175 requests a second before requests started to queue.
Does Ollama Herd keep models loaded between requests?
Yes. Ollama unloads a model after 5 idle minutes by default; Herd sends keep_alive -1 with the requests it routes (except while another model is waiting to load on that machine), so the model stays loaded until another model needs the memory. On one Mac, the first request after six idle minutes took 0.38 seconds to its first token through Herd against 2.9 seconds straight to Ollama, which had to reload qwen3-coder:30b, in each of three runs. Requests sent in Ollama’s own format can still set their own keep_alive.
Can I run these benchmarks myself?
Yes, and we'd love to see your numbers. The scripts and raw results are in the Ollama Herd repository on GitHub: scripts/benchmark.py runs the fleet tests, router_overhead.py and router-ceiling-loadgen.py measure the router, idle-keep-alive.py the idle test, and docs/experiments holds the results. The commands are on this page.
Related Reading
- Load balancing Ollama, why round-robin is the wrong baseline for this
- Routing vs sharding, why more machines add capacity, not speed for one request
- Cold starts, what a model load costs and how to avoid it