Ollama has no built-in load balancing. Here are the real options, from a hand-written HAProxy config to a zero-config router that knows which machine already has your model loaded.
Ollama runs on a single machine and serves requests from that machine's memory and GPU. It has no built-in way to distribute requests across multiple instances. By default it also processes requests largely sequentially, so the moment you have more than one caller (a coding agent, a RAG pipeline, a few teammates), you hit a throughput ceiling: requests queue behind each other on one box while your other machines sit idle.
Load balancing fixes this by putting a dispatcher in front of several Ollama instances so requests spread across all of them. Teams that move from a single instance to a balanced multi-instance setup routinely cut median latency under concurrent load and gain failover when one machine goes down.
There are four practical ways to do it. This guide covers each honestly, then explains when a purpose-built router beats a generic load balancer.
Whichever method you pick, the prerequisite is the same. Install Ollama on each machine and make it listen on the network, not just localhost:
# On each machine, expose Ollama on the LAN
export OLLAMA_HOST=0.0.0.0:11434
ollama serve
# Pull the models you want that machine to serve
ollama pull llama3.3:70bNow each machine answers at http://<its-ip>:11434. The job of a load balancer is to sit in front of all of them and present one endpoint.
HAProxy is the most common answer you'll find. It's a battle-tested TCP/HTTP load balancer. A minimal haproxy.cfg that round-robins across three Ollama machines with health checks:
defaults
mode http
timeout connect 5s
timeout client 300s
timeout server 300s
frontend ollama_front
bind *:11434
default_backend ollama_nodes
backend ollama_nodes
balance roundrobin
option httpchk GET /api/version
server mac-studio 192.168.1.10:11434 check
server macbook-pro 192.168.1.11:11434 check
server mac-mini 192.168.1.12:11434 checkPoint your client at the HAProxy host on port 11434 and it fans requests out across the three machines, dropping any that fail the /api/version health check.
Good for: dedicated, identical inference servers where every node has the same models loaded and you just want to spread load. The catch: you maintain the config by hand (every new machine is a config edit and reload), and round-robin is blind, which we'll come back to.
nginx does the same job with an upstream block:
upstream ollama {
server 192.168.1.10:11434;
server 192.168.1.11:11434;
server 192.168.1.12:11434;
}
server {
listen 11434;
location / {
proxy_pass http://ollama;
proxy_read_timeout 300s;
}
}Functionally similar to HAProxy for this use case. Same trade-off: manual config, blind distribution.
A few open-source tools are built specifically for Ollama and add a little more intelligence than raw round-robin:
These are a step up from a plain proxy, but most still treat your machines as interchangeable and require you to list every endpoint.
Round-robin and least-connections share a blind spot: they don't know what's happening inside each Ollama. That matters a lot for local LLMs:
Ollama Herd is purpose-built for exactly this. Instead of round-robin, it scores every node on 8 signals (is the model already loaded, does it fit in memory, how deep is the queue, thermal state, and more) and sends each request to the machine that will actually answer fastest. And it discovers nodes automatically over mDNS, so there's no endpoint list to maintain.
# On your most powerful machine (the router)
pip install ollama-herd
herd
# On every other machine running Ollama
herd-nodeThat's it. Nodes announce themselves, the router builds a live picture of the fleet, and clients point at one endpoint (http://<router-ip>:11435) that speaks the OpenAI, Ollama, and Anthropic Messages APIs. Add a machine later and it just appears, no config edit. It also routes embeddings, image generation, speech-to-text, and vision, not just chat.
In short: HAProxy spreads load blindly across identical servers. Herd routes intelligently across the real, mixed machines you actually have.
| Your situation | Best fit |
|---|---|
| Identical dedicated servers, same models on each, you like editing configs | HAProxy / nginx |
| A few Ollama boxes, want response-time-aware dispatch, minimal setup | ollama_load_balancer / OLOL |
| Mixed machines (different sizes, laptops that are also used for work) | Ollama Herd |
| You want zero config and automatic discovery of new machines | Ollama Herd |
| You route more than chat (embeddings, image gen, speech-to-text) | Ollama Herd |
| You need model-residency-aware routing to avoid cold-load stalls | Ollama Herd |
If your fleet is anything other than a rack of identical servers, the two-command router is the fastest path to a load-balanced Ollama setup:
pip install ollama-herd # or: brew install ollama-herd
herd # start the router
herd-node # on each deviceSee the Quickstart for the full walkthrough, or the Routing Engine guide for how the 8-signal scoring works under the hood.
No. Ollama runs on one instance with no native way to distribute across machines. You need an external load balancer (HAProxy, nginx), a purpose-built tool, or a router like Ollama Herd.
For identical servers, HAProxy round-robin is quick. For a zero-config setup that also avoids cold-load stalls and handles mixed machines, Ollama Herd auto-discovers nodes and routes with a scoring engine, no config file.
Round-robin ignores which machine has the model loaded, how busy each node is, and whether a node is throttling. For local LLMs, where a cold load costs 15 to 30 seconds, sending a request to the wrong node is a real latency hit. Scoring-based routing avoids it.
With a blind balancer, mixing a 512GB Studio and a 16GB laptop is risky (big requests can land on the small machine). Ollama Herd's memory-fit scoring routes large models to capable machines automatically.