How to Load Balance Ollama Across Multiple Machines

Why Ollama needs a load balancer

Ollama runs on a single machine and serves requests from that machine's memory and GPU. It has no built-in way to distribute requests across multiple instances. By default it also processes requests largely sequentially, so the moment you have more than one caller (a coding agent, a RAG pipeline, a few teammates), you hit a throughput ceiling: requests queue behind each other on one box while your other machines sit idle.

Load balancing fixes this by putting a dispatcher in front of several Ollama instances so requests spread across all of them. Teams that move from a single instance to a balanced multi-instance setup routinely cut median latency under concurrent load and gain failover when one machine goes down.

There are four practical ways to do it. This guide covers each honestly, then explains when a purpose-built router beats a generic load balancer.

Before you start: run Ollama on every machine

Whichever method you pick, the prerequisite is the same. Install Ollama on each machine and make it listen on the network, not just localhost:

# On each machine, expose Ollama on the LAN
export OLLAMA_HOST=0.0.0.0:11434
ollama serve

# Pull the models you want that machine to serve
ollama pull llama3.3:70b

Now each machine answers at http://<its-ip>:11434. The job of a load balancer is to sit in front of all of them and present one endpoint.

Option 1: HAProxy (the common DIY path)

HAProxy is the most common answer you'll find. It's a battle-tested TCP/HTTP load balancer. A minimal haproxy.cfg that round-robins across three Ollama machines with health checks:

defaults
    mode http
    timeout connect 5s
    timeout client  300s
    timeout server  300s

frontend ollama_front
    bind *:11434
    default_backend ollama_nodes

backend ollama_nodes
    balance roundrobin
    option httpchk GET /api/version
    server mac-studio  192.168.1.10:11434 check
    server macbook-pro 192.168.1.11:11434 check
    server mac-mini    192.168.1.12:11434 check

Point your client at the HAProxy host on port 11434 and it fans requests out across the three machines, dropping any that fail the /api/version health check.

Good for: dedicated, identical inference servers where every node has the same models loaded and you just want to spread load. The catch: you maintain the config by hand (every new machine is a config edit and reload), and round-robin is blind, which we'll come back to.

Option 2: nginx

nginx does the same job with an upstream block:

upstream ollama {
    server 192.168.1.10:11434;
    server 192.168.1.11:11434;
    server 192.168.1.12:11434;
}
server {
    listen 11434;
    location / {
        proxy_pass http://ollama;
        proxy_read_timeout 300s;
    }
}

Functionally similar to HAProxy for this use case. Same trade-off: manual config, blind distribution.

Option 3: purpose-built Ollama load balancers

A few open-source tools are built specifically for Ollama and add a little more intelligence than raw round-robin:

• ollama_load_balancer, a Rust utility that dispatches to the most responsive server based on a health-value system.
• OLOL, presents a unified API endpoint and transparently clusters requests across instances while staying Ollama-API-compatible.
• Open WebUI, if you already run it, its multi-backend support can distribute across Ollama connections with cache-aware and least-connections strategies.

These are a step up from a plain proxy, but most still treat your machines as interchangeable and require you to list every endpoint.

The problem with blind distribution

Round-robin and least-connections share a blind spot: they don't know what's happening inside each Ollama. That matters a lot for local LLMs:

• Cold loads are expensive. If the model isn't already loaded on the machine a request lands on, Ollama has to load it, which can take 15 to 30 seconds for a large model. A blind balancer will happily send a request to a cold node while a machine with the model already hot sits one hop away.
• Machines aren't identical. A 512GB Mac Studio and a 16GB MacBook Air are not interchangeable. Sending a 70B request to the laptop fails or crawls.
• Real machines get busy. A laptop that's thermal-throttling or running a video call should drop out of rotation. A generic balancer keeps sending to it.
• Every new machine is a config edit. Add a node, edit the config, reload. Forget one and traffic silently skips it.

Option 4: Ollama Herd (a router, not just a balancer)

Ollama Herd is purpose-built for exactly this. Instead of round-robin, it scores every node on 8 signals (is the model already loaded, does it fit in memory, how deep is the queue, thermal state, and more) and sends each request to the machine that will actually answer fastest. And it discovers nodes automatically over mDNS, so there's no endpoint list to maintain.

# On your most powerful machine (the router)
pip install ollama-herd
herd

# On every other machine running Ollama
herd-node

That's it. Nodes announce themselves, the router builds a live picture of the fleet, and clients point at one endpoint (http://<router-ip>:11435) that speaks the OpenAI, Ollama, and Anthropic Messages APIs. Add a machine later and it just appears, no config edit. It also routes embeddings, image generation, speech-to-text, and vision, not just chat.

In short: HAProxy spreads load blindly across identical servers. Herd routes intelligently across the real, mixed machines you actually have.

Which should you use?

Get started

If your fleet is anything other than a rack of identical servers, the two-command router is the fastest path to a load-balanced Ollama setup:

pip install ollama-herd    # or: brew install ollama-herd
herd                       # start the router
herd-node                  # on each device

See the Quickstart for the full walkthrough, or the Routing Engine guide for how the 8-signal scoring works under the hood.

FAQ

Does Ollama have built-in load balancing?

No. Ollama runs on one instance with no native way to distribute across machines. You need an external load balancer (HAProxy, nginx), a purpose-built tool, or a router like Ollama Herd.

What's the simplest way to load balance Ollama?

For identical servers, HAProxy round-robin is quick. For a zero-config setup that also avoids cold-load stalls and handles mixed machines, Ollama Herd auto-discovers nodes and routes with a scoring engine, no config file.

Why isn't round-robin ideal for Ollama?

Round-robin ignores which machine has the model loaded, how busy each node is, and whether a node is throttling. For local LLMs, where a cold load costs 15 to 30 seconds, sending a request to the wrong node is a real latency hit. Scoring-based routing avoids it.

Can I mix machine sizes?

With a blind balancer, mixing a 512GB Studio and a 16GB laptop is risky (big requests can land on the small machine). Ollama Herd's memory-fit scoring routes large models to capable machines automatically.

Related Reading

• Ollama Herd vs Ollama proxy tools, the broader ecosystem of routing utilities
• Ollama Herd vs single Ollama, when to move from one machine to a fleet
• Routing Engine, the 5-stage scoring pipeline in detail