Ollama Herd vs HAProxy

TL;DR

HAProxy is the right tool for identical dedicated servers. Ollama Herd is the right tool for a fleet of real, mixed machines. HAProxy is one of the best load balancers ever built, but it treats each Ollama as an opaque HTTP endpoint. It has no idea which machine already has your model loaded, how much memory each has, or how deep each machine's queue is. For local LLMs, where a cold model load costs 15 to 30 seconds, that blindness is a real latency cost. Herd is built to see inside the fleet.

What is HAProxy?

HAProxy is a mature, open-source, high-performance TCP and HTTP load balancer used across the internet. It's not Ollama-specific; you point it at any set of backends. To load balance Ollama, you write an haproxy.cfg listing each machine's Ollama endpoint, pick a balancing algorithm (round-robin, least-connections, source-hash), and add a health check. It's rock-solid, extremely fast, and does far more than routing: TLS termination, rate limiting, sticky sessions, ACLs, observability.

What is Ollama Herd?

Ollama Herd is an open-source smart AI router that turns the machines you already own (Apple Silicon Macs, Linux servers, and Windows PCs, with or without NVIDIA GPUs) into one endpoint. It routes LLMs, embeddings, image generation, speech-to-text, and vision with 8-signal scoring, mDNS auto-discovery, an 8-tab real-time dashboard, and OpenAI + Ollama + Anthropic Messages API compatibility. Apple Silicon Macs get extras: an MLX backend, native image generation, and speech-to-text. Two commands to set up, zero config files. pip install ollama-herd or brew install ollama-herd.

The core difference

HAProxy load balances. Herd routes. The distinction matters for local LLMs:

• HAProxy is model-blind. Round-robin will happily send a request to a machine that has to cold-load the model (15 to 30 seconds) while a machine with the model already hot sits idle one hop away. Herd's scoring gives a hot model a large bonus, so it routes to the warm machine.
• HAProxy treats machines as identical. It has no concept of memory fit, so a 70B request can land on a 16GB laptop. Herd scores memory fit and sends big models to capable machines.
• HAProxy needs a config file. Every new machine is an haproxy.cfg edit and reload. Herd discovers nodes over mDNS; a new node just appears.
• HAProxy speaks HTTP, not LLM. Herd exposes OpenAI, Ollama, and Anthropic Messages APIs and routes embeddings, image generation, and speech-to-text as distinct service types.

Conceptual: two machines, illustrative

HAProxy: round-robin

Fast and hardened, and fine when every backend is identical. Least-connections and source-hash are just as model-blind.

Chat request

to HAProxy

HAProxynext backend in the rotation

each request goes to one backend

Mac mininext in line: must cold-load the model, 15 to 30 s

Mac Studiomodel already loaded, idle this turn

Ollama Herd: model-aware

Sees which machine has the model loaded and whether it fits in memory.

Chat request

to Herd :11435

Herd routermodel warmth, memory fit, queue depth

each request goes to one node

Mac miniwould cold-load: skipped

Mac Studiochosen: model already loaded

Feature comparison

Where HAProxy wins

1. Production pedigree. HAProxy runs some of the largest sites on the internet. If you need bulletproof TCP/HTTP load balancing with decades of hardening, it's the standard.
2. It does far more than routing. TLS termination, rate limiting, ACLs, sticky sessions, DDoS mitigation, deep observability. Herd is a router, not a gateway.
3. Backend-agnostic. HAProxy balances anything. If your fleet is a mix of Ollama, vLLM, and custom services behind one HTTP contract, HAProxy front-doors all of it.
4. Raw performance. A finely tuned HAProxy handles enormous request volumes with minimal overhead.
5. Explicit control. If you want every routing rule spelled out in a config you own, HAProxy gives you exactly that.

Where Ollama Herd wins

1. Model-residency-aware routing. The single biggest win for local LLMs. Herd avoids cold-load stalls by scoring which machine already has the model hot. HAProxy can't see this.
2. Zero-config discovery. No haproxy.cfg to write or maintain. Start herd-node on a machine and it joins the fleet in about 60 seconds.
3. Mixed-machine awareness. Memory-fit scoring keeps big models off small laptops; opt-in workstation awareness routes around a Mac that's in a meeting.
4. Multimodal, multi-API. Routes embeddings, image generation, speech-to-text, and vision, and speaks OpenAI, Ollama, and Anthropic Messages so Claude Code points straight at it.
5. Appliance simplicity. Two commands, no config file, a real fleet dashboard out of the box.

They can work together

This isn't strictly either/or. HAProxy is a great front door and Herd is a great brain. A common pattern: HAProxy handles TLS termination and rate limiting at the edge, then forwards to the Herd router, which does the model-aware routing across your fleet. You get HAProxy's hardened edge and Herd's LLM intelligence.

Conceptual: where each tool sits (a tool can span layers)

Clientschat UIs, coding agents, apps

Open WebUI Claude Code Codex Your apps and agents

Gateway / edgeauth, rate limits, cloud providers

LiteLLM Bifrost Envoy AI Gateway HAProxy

Fleet routerpicks one machine per request

Ollama Herd Olla NVIDIA PAIR LoLLMs Hub ollama_load_balancer OLOL LocalAI (distributed mode) GPUStack (cluster manager) llama-swap (peers) LiteLLM (Ollama load balancing)

Inference serverruns the model on one machine

Ollama mlx_lm.server vLLM LocalAI llama-swap Docker Model Runner LM Studio llama.cpp server NVIDIA PAIR (manages engines)

One model, many machinessplits a model too big for one

exo MLX distributed LocalAI (worker mode) OLOL (sharding) vLLM (multi-node)

Hardwarethe machines themselves

Macs Linux servers Windows PCs NVIDIA GPU servers

When to choose

Bottom line

HAProxy is one of the best generic load balancers in existence, and for a rack of identical servers it's a fine way to spread Ollama traffic. But a load balancer that can't see which machine has your model loaded is leaving the biggest local-LLM optimization on the table. Herd is built for exactly that: it routes on what's happening inside each Ollama, discovers your fleet automatically, and speaks the APIs your tools already use.

Getting started

pip install ollama-herd    # or: brew install ollama-herd
herd                       # start the router
herd-node                  # on each device

See the load-balancing guide for the full picture, including a working HAProxy config.

FAQ

Can HAProxy load balance Ollama?

Yes, with round-robin across your Ollama backends and a health check. It works for identical servers, but it can't tell which machine has your model loaded or how busy each one is.

What does Herd do that HAProxy doesn't?

Herd discovers nodes automatically over mDNS (no config file), and scores each node on model residency, memory fit, queue depth, and latency history, so it avoids cold-load stalls and sends big models to capable machines. It also speaks the OpenAI and Anthropic Messages APIs and routes embeddings, image generation, and speech-to-text, not just chat.

Is Ollama Herd free?

Yes, MIT licensed. HAProxy is also open source (with a commercial edition available).

See Also

• How to load balance Ollama, the full guide with configs
• Ollama Herd vs ollama_load_balancer, a purpose-built Ollama LB
• Ollama Herd vs OLOL, an Ollama clustering system