Ollama Herd vs HAProxy

TL;DR

HAProxy is the right tool for identical dedicated servers. Ollama Herd is the right tool for a fleet of real, mixed machines. HAProxy is one of the best load balancers ever built, but it treats each Ollama as an opaque HTTP endpoint. It has no idea which machine already has your model loaded, how much memory each has, or whether a laptop is thermal-throttling. For local LLMs, where a cold model load costs 15 to 30 seconds, that blindness is a real latency cost. Herd is built to see inside the fleet.

What is HAProxy?

HAProxy is a mature, open-source, high-performance TCP and HTTP load balancer used across the internet. It's not Ollama-specific; you point it at any set of backends. To load balance Ollama, you write an haproxy.cfg listing each machine's Ollama endpoint, pick a balancing algorithm (round-robin, least-connections, source-hash), and add a health check. It's rock-solid, extremely fast, and does far more than routing: TLS termination, rate limiting, sticky sessions, ACLs, observability.

What is Ollama Herd?

Ollama Herd is an open-source smart multimodal AI router that turns multiple inference nodes across Apple Silicon and mixed hardware into one intelligent endpoint. It routes LLMs, embeddings, image generation, speech-to-text, and vision with a 8-signal scoring engine, mDNS auto-discovery, an 8-tab real-time dashboard, and OpenAI + Ollama + Anthropic Messages API compatibility. Two commands to set up, zero config files. pip install ollama-herd or brew install ollama-herd.

The core difference

HAProxy load balances. Herd routes. The distinction matters for local LLMs:

• HAProxy is model-blind. Round-robin will happily send a request to a machine that has to cold-load the model (15 to 30 seconds) while a machine with the model already hot sits idle one hop away. Herd's scoring gives a hot model a large bonus, so it routes to the warm machine.
• HAProxy treats machines as identical. It has no concept of memory fit, so a 70B request can land on a 16GB laptop. Herd scores memory fit and sends big models to capable machines.
• HAProxy needs a config file. Every new machine is an haproxy.cfg edit and reload. Herd discovers nodes over mDNS; a new node just appears.
• HAProxy speaks HTTP, not LLM. Herd exposes OpenAI, Ollama, and Anthropic Messages APIs and routes embeddings, image generation, and speech-to-text as distinct service types.

Feature comparison

Where HAProxy wins

1. Production pedigree. HAProxy runs some of the largest sites on the internet. If you need bulletproof TCP/HTTP load balancing with decades of hardening, it's the standard.
2. It does far more than routing. TLS termination, rate limiting, ACLs, sticky sessions, DDoS mitigation, deep observability. Herd is a router, not a gateway.
3. Backend-agnostic. HAProxy balances anything. If your fleet is a mix of Ollama, vLLM, and custom services behind one HTTP contract, HAProxy front-doors all of it.
4. Raw performance. A finely tuned HAProxy handles enormous request volumes with minimal overhead.
5. Explicit control. If you want every routing rule spelled out in a config you own, HAProxy gives you exactly that.

Where Ollama Herd wins

1. Model-residency-aware routing. The single biggest win for local LLMs. Herd avoids cold-load stalls by scoring which machine already has the model hot. HAProxy can't see this.
2. Zero-config discovery. No haproxy.cfg to write or maintain. Start herd-node on a machine and it joins the fleet in about 60 seconds.
3. Mixed-machine awareness. Memory-fit scoring keeps big models off small laptops; workstation awareness routes around a Mac that's in a meeting or throttling.
4. Multimodal, multi-API. Routes embeddings, image generation, speech-to-text, and vision, and speaks OpenAI, Ollama, and Anthropic Messages so Claude Code points straight at it.
5. Appliance simplicity. Two commands, no config file, a real fleet dashboard out of the box.

They can work together

This isn't strictly either/or. HAProxy is a great front door and Herd is a great brain. A common pattern: HAProxy handles TLS termination and rate limiting at the edge, then forwards to the Herd router, which does the model-aware routing across your fleet. You get HAProxy's hardened edge and Herd's LLM intelligence.

When to choose

Bottom line

HAProxy is one of the best generic load balancers in existence, and for a rack of identical servers it's a fine way to spread Ollama traffic. But a load balancer that can't see which machine has your model loaded is leaving the biggest local-LLM optimization on the table. Herd is built for exactly that: it routes on what's happening inside each Ollama, discovers your fleet automatically, and speaks the APIs your tools already use.

Getting started

pip install ollama-herd    # or: brew install ollama-herd
herd                       # start the router
herd-node                  # on each device

See the load-balancing guide for the full picture, including a working HAProxy config.

FAQ

Can HAProxy load balance Ollama?

Yes, with round-robin across your Ollama backends and a health check. It works for identical servers, but it can't tell which machine has your model loaded or how busy each one is.

What does Herd do that HAProxy doesn't?

Model-residency-aware routing, mDNS auto-discovery, memory-fit and workstation awareness, multimodal routing, and native OpenAI/Anthropic APIs.

Is Ollama Herd free?

Yes, MIT licensed. HAProxy is also open source (with a commercial edition available).

See Also

• How to load balance Ollama, the full guide with configs
• Ollama Herd vs ollama_load_balancer, a purpose-built Ollama LB
• Ollama Herd vs OLOL, an Ollama clustering system