Ollama Herd vs ollama_load_balancer
ollama_load_balancer is a lean Rust binary that races the same request across several Ollama backends and returns the fastest. Ollama Herd routes each request to one best-scored node. Different philosophies: minimize tail latency by duplicating work, or place each request precisely.
TL;DR
ollama_load_balancer is a great little tool for a couple of identical Ollama boxes. Ollama Herd is a full router for a real, mixed fleet. The Rust balancer's signature trick is clever: for chat, it fires the same request at several backends at once and streams back whichever answers first. That minimizes tail latency, but it does so by running your prompt on multiple machines and discarding all but the fastest. Herd takes the other path: score every node and place each request on the single best one, so nothing is wasted.
What is ollama_load_balancer?
ollama_load_balancer is a lightweight, single-binary Rust load balancer for Ollama (26 stars, MIT). You give it a list of Ollama server URLs on the command line or in a file. It aggregates /api/tags across backends, filters to servers that actually have the requested model, and for /api/chat it sends parallel requests to several health-scored backends and returns the fastest stream. A health-value system nudges each server's score up on success and down on error. It's lean and reliable for what it covers. Its last release was in April 2025, and several documented endpoints (/status, /add_server) are marked not yet implemented.
What is Ollama Herd?
Ollama Herd is an open-source smart multimodal AI router that turns multiple inference nodes across Apple Silicon and mixed hardware into one intelligent endpoint. It routes LLMs, embeddings, image generation, speech-to-text, and vision with a 8-signal scoring engine, mDNS auto-discovery, an 8-tab real-time dashboard, and OpenAI + Ollama + Anthropic Messages API compatibility. Two commands to set up, zero config files. pip install ollama-herd or brew install ollama-herd.
The core difference: race vs place
Both tools want your request answered fast. They disagree on how.
- ollama_load_balancer races. It sends the chat request to multiple backends and keeps the fastest response. Best-case tail latency is excellent, because a slow or busy node can't hold you up. The cost: the same prompt is generated on several machines, so your effective compute load multiplies, and a fleet that's already busy gets busier.
- Ollama Herd places. It scores every node on 8 signals (is the model hot, does it fit in memory, how deep is the queue, thermal state, and more) and sends the request to the one node most likely to answer fastest. No duplicate compute; the fleet's total capacity goes further.
Racing wins when you have spare capacity and only care about latency. Placing wins when you want the fleet to serve more total work, or when machines are shared and you can't afford to run every prompt three times.
Feature comparison
| Feature | ollama_load_balancer | Ollama Herd |
|---|---|---|
| Core approach | Race parallel backends, keep fastest | Score nodes, place on best one |
| Duplicate compute | Yes (by design, for chat) | No |
| Node discovery | CLI args or server file | mDNS auto-discovery, zero config |
| Model awareness | Filters servers that have the model | Scores hot vs cold, plus memory fit |
| Workstation awareness | No | Thermal, meeting, foreground-app |
| API surface | Ollama-native (chat fully, generate partial) | OpenAI + Ollama + Anthropic Messages |
| Multimodal routing | No (chat focus) | LLM, embeddings, image gen, STT, vision |
| Dashboard | No (status endpoint planned) | 8-tab live fleet dashboard |
| Footprint | Single Rust binary, very lean | Python, heavier per node |
| Maintenance | Last release April 2025 | Actively developed |
| License | MIT | MIT |
Where ollama_load_balancer wins
- Simplicity and footprint. One small Rust binary, a list of servers, done. Nothing to
pip install, no Python. - Best-case chat latency. The race-and-keep-fastest approach genuinely minimizes tail latency when you have spare capacity to burn.
- Fault tolerance for free. If a backend in the race dies mid-request, another is already answering. High availability falls out of the design.
- Lean and dependency-free. A compiled binary with a tiny memory footprint is easy to drop onto any box.
Where Ollama Herd wins
- It doesn't waste compute. Scoring places each request on one node, so your fleet serves more total requests instead of running each prompt several times.
- Zero-config discovery. No server list to maintain. Start
herd-nodeand it joins over mDNS. - Deeper awareness. Hot-vs-cold model scoring, memory fit, thermal state, and meeting detection, not just a health value.
- Multimodal and multi-API. Routes embeddings, image generation, speech-to-text, and vision, and speaks OpenAI and Anthropic Messages so Claude Code points straight at it.
- Operational visibility and active development. A real 8-tab dashboard, ongoing releases, and 30+ automated health checks.
When to choose
| Scenario | Choose |
|---|---|
| Two or three identical Ollama boxes, you want the leanest binary | ollama_load_balancer |
| Spare capacity, latency is all that matters, don't mind duplicate compute | ollama_load_balancer |
| Mixed machines you want the fleet to serve efficiently | Ollama Herd |
| Shared machines where running every prompt several times is a problem | Ollama Herd |
| You route more than chat, or need OpenAI/Anthropic APIs | Ollama Herd |
| You want zero config, a dashboard, and active maintenance | Ollama Herd |
Bottom line
ollama_load_balancer is a neat, honest little tool with a genuinely clever latency trick, and for a small set of identical boxes with spare capacity, it does its job well. Herd is for when you want the fleet to go further: place each request precisely instead of running it everywhere, discover machines automatically, route every modality, and see the whole thing in a dashboard.
Getting started
pip install ollama-herd # or: brew install ollama-herd
herd # start the router
herd-node # on each deviceFAQ
Does racing parallel requests waste compute?
Yes, for chat it runs the same prompt on several machines and keeps only the fastest. Great for tail latency, but it multiplies effective load. Herd sends each request to one scored node.
Is ollama_load_balancer still maintained?
Its last release was April 2025 and some endpoints remain unimplemented. Herd is under active development.
Is Ollama Herd free?
Yes. Both projects are MIT licensed.
See Also
- How to load balance Ollama, the full guide
- Ollama Herd vs HAProxy, the generic load-balancer path
- Ollama Herd vs OLOL, an Ollama clustering system