Ollama Herd vs DIY Scripts & nginx Round-Robin

What are DIY Scripts and nginx Round-Robin?

DIY routing is the most common first attempt at distributing Ollama requests across multiple machines. It typically starts with an nginx upstream block that round-robins requests across backend servers, then evolves into health-check scripts, model-aware routing in Python, and eventually a homegrown dashboard. Each stage adds maintenance burden and still lacks the fleet intelligence that purpose-built routers provide.

What is Ollama Herd?

Ollama Herd is an open-source smart multimodal AI router that turns multiple Ollama instances across Apple Silicon devices into one intelligent endpoint. It routes LLMs, embeddings, image generation, speech-to-text, and vision with a 8-signal scoring engine, mDNS auto-discovery, and an 8-tab real-time dashboard. Two commands to set up, zero config files. pip install ollama-herd or brew install ollama-herd.

Overview

The most common "competitor" to Ollama Herd isn't another product, it's the impulse to build something yourself. Every engineer with multiple Macs running Ollama has the same thought: "I'll just throw nginx in front of these and round-robin requests." It works on day one. It breaks on day thirty.

What DIY Typically Looks Like

Most DIY setups follow a predictable evolution:

Stage 1, nginx upstream block (30 minutes)

upstream ollama {
    server mac-mini-1:11434;
    server mac-mini-2:11434;
    server macbook:11434;
}

Round-robin distribution. No health checks. No awareness of which models live where. Works fine until one machine is asleep or running a different model.

Stage 2, Health check scripts (1–3 days)

A cron job or bash script that pings /api/tags on each node, updates an nginx config, and reloads. Now you handle nodes going offline, but you still don't know which node is best for a given request.

Stage 3, Model-aware routing (1–2 weeks)

A Python Flask/FastAPI app that queries each node for available models and routes by model name. You now have a basic model registry, but it gets stale, doesn't handle concurrent requests well, and has no retry logic.

Stage 4, The "I need a dashboard" moment (ongoing)

You realize you have no visibility. You start logging to a file. Then you want graphs. Then you want to see queue depth. Then you want tracing. Each addition is another weekend project that you maintain forever.

Feature Comparison

What DIY Gets You

• Full control. You own every line. No black box decisions. If the routing logic does something weird, you wrote it and you can fix it.
• No dependencies. No pip install, no new process to manage, no upstream project that might change direction.
• Learning experience. Building a load balancer teaches you about distributed systems, health checking, retry strategies, and failure modes. Genuinely valuable.
• Custom requirements. If you need routing logic that is deeply specific to your workflow, like "always send coding requests to the Mac Studio and creative writing to the MacBook Pro", DIY lets you hardcode exactly what you want.
• Minimal footprint. An nginx config is a few lines. No Python runtime, no additional process, no dashboard consuming resources.

What DIY Misses

The gap between "requests go to different machines" and "requests go to the right machine" is enormous:

• Scoring intelligence. Herd evaluates 8 signals per request: model availability, GPU memory, system memory, thermal state, queue depth, historical performance, and node health. DIY round-robin treats a thermally throttled M1 MacBook Air the same as an idle M4 Max Mac Studio.
• Thermal awareness. macOS thermal throttling can cut inference speed by 50–70%. Herd reads thermal state and routes away from hot nodes. DIY has no mechanism for this without writing custom macOS integrations.
• Capacity learning. Herd's adaptive capacity engine observes actual inference times per model per node and adjusts scoring weights over time. Your M2 Pro is 2x faster than your M1 for codellama? Herd learns that. DIY treats them equally forever unless you manually tune weights.
• Meeting detection. When someone is on a Zoom call, Herd reduces load on their machine. DIY doesn't know or care.
• Queue management. Herd tracks how many requests are queued on each node and applies backpressure. DIY nginx sends the next request to a node that already has 5 requests waiting.
• Auto-retry with intelligence. When a node fails mid-request, Herd retries on the next-best node automatically. DIY nginx can retry, but it picks randomly, possibly sending the retry to an even worse node.
• Model fallbacks. If a node doesn't have the requested model, Herd can fall back to a compatible alternative. DIY returns a 404.
• Multimodal routing. Herd routes LLMs, embeddings, image generation, speech-to-text, and vision through one unified engine with model-type-aware scoring. DIY means building four separate routing configs.
• Observability. Herd ships with an 8-tab dashboard, request tracing, and health monitoring. DIY means grepping log files.

The Maintenance Burden

DIY routing has a specific failure mode: it works perfectly on day 1 and degrades as your fleet changes.

• Day 1: Two Mac Minis, same model, round-robin. Everything works.
• Day 15: You add a MacBook Pro. Edit nginx config. Reload. Still fine.
• Day 30: You pull a new model on one machine but forget the others. Requests for that model fail 66% of the time. Debug for an hour before realizing the issue.
• Day 45: One Mac Mini starts thermal throttling. Half your requests are now slow. No way to tell from nginx which node is struggling.
• Day 60: You need embeddings and image generation too. Now you need separate upstream blocks, separate health checks, separate model registries. Triple the config.
• Day 90: Someone asks "what's the average latency per model?" You have no idea without building a logging pipeline.

Every fleet change, adding a node, removing a node, pulling a new model, changing hardware, requires manual config updates. Herd handles all of these automatically via mDNS discovery and real-time model polling.

Time-to-Build Comparison

When DIY Makes Sense

• You're learning. Building a load balancer is a great exercise. Do it, learn from it, then evaluate whether to maintain it long-term.
• You have exactly 2 identical nodes. Round-robin is genuinely fine when all nodes are identical and running the same models. The gap grows with fleet heterogeneity.
• You need deeply custom routing logic that is specific to your business and can't be expressed through Herd's configuration. Example: routing based on request content analysis, user identity, or integration with an internal scheduling system.
• You're embedding routing into a larger custom system where adding Herd as a dependency creates more complexity than building the routing logic inline.
• You only use one model type. If you only run LLM chat completions and never need embeddings, image gen, or STT, the multimodal routing advantage doesn't apply.

When to Just Use Herd

• You have 3+ nodes, heterogeneous fleets are where scoring intelligence matters most.
• You run multiple model types, unified routing beats maintaining 4 separate configs.
• You value your weekends, 2 minutes to install vs. months to build equivalent functionality.
• You want observability, the dashboard and tracing alone save hours of debugging.
• Your fleet changes over time, mDNS discovery means zero config changes when nodes come and go.
• You're building agent applications, OpenAI API compatibility means your agents work without modification.
• You use your Macs for other things, thermal awareness and meeting detection keep your daily work smooth.

Bottom Line

DIY routing is a valid choice for simple, static, single-model-type setups, or as a learning exercise. But the engineering effort to match Herd's capabilities is measured in months, not hours. The question isn't "can I build this myself?", of course you can. The question is "is building and maintaining a custom AI routing engine the best use of my engineering time?"

For most people running local AI on Apple Silicon, the answer is: install Herd in 2 minutes and spend those engineering weekends on the things you're actually building with AI.

Getting Started

pip install ollama-herd    # or: brew install ollama-herd
herd                       # start the router
herd-node                  # on each device

Replacing Your DIY Setup with Ollama Herd

If you've been running nginx round-robin or custom scripts:

1. Install Ollama Herd, pip install ollama-herd on the machine currently running your proxy/scripts.
2. Start the router, herd. It listens on port 11435. Update your clients to point here instead of your nginx/script endpoint.
3. Start node agents, run herd-node on each Ollama machine. Herd discovers them via mDNS, no need to maintain your list of backend IPs.

You can run Herd alongside your existing setup during transition. Once you've verified it works, remove the old scripts. Your health checks, retry logic, and load balancing are now handled automatically, and they're better than what you built, because Herd has 8-signal scoring and 1,200+ tests behind it.

Frequently Asked Questions

How long does it take to build a DIY Ollama load balancer?

Basic round-robin takes 30 minutes. Adding health checks takes 1–3 days. Model-aware routing takes 1–2 weeks. Reaching feature parity with Herd (scoring, thermal awareness, multimodal routing, dashboard, retries) takes 3–6 months of weekends, and then you maintain it forever.

Can I start with DIY and switch to Herd later?

Yes. Herd uses the standard Ollama and OpenAI APIs, so any client pointing at your DIY proxy can switch to Herd by changing the endpoint URL. Your Ollama installations and models stay exactly the same.

Does Herd handle everything nginx does?

For Ollama routing, yes and more. Herd handles load balancing, health checks, retries, and model-aware routing, plus scoring intelligence, thermal awareness, and multimodal support that nginx cannot provide. For general web serving or reverse proxy needs unrelated to Ollama, nginx remains the right tool.

What if I need custom routing logic?

Herd covers the vast majority of routing scenarios through its 8-signal scoring and configuration options. If you need deeply custom logic, like routing based on request content analysis or integration with an internal scheduling system, DIY may still make sense. But evaluate whether Herd's configuration handles your needs before building from scratch.

Is the DIY approach viable for a learning exercise?

Absolutely. Building a load balancer teaches valuable lessons about distributed systems, health checking, and failure modes. Build it, learn from it, then decide whether maintaining it long-term is worth the ongoing effort compared to a 2-minute Herd install.

See Also

• Ollama Herd vs Single Ollama, The starting point before you need any routing at all
• Ollama Herd vs Ollama Proxies, How Herd compares to existing open-source proxies
• Ollama Herd vs Open WebUI, Chat interface vs routing engine, and how they complement each other