Quickstart

From zero to routed inference in 60 seconds.

Prerequisites

  • One machine to start (macOS, Linux, or Windows). Add more on the same local network any time; see running on one machine.
  • Ollama installed on each machine
  • At least one model pulled (e.g., ollama pull llama3.2:3b)
  • Python 3.11+ on the router machine

Step 1: Install Ollama Herd

On the machine you want as the router (typically your most powerful device):

pip install ollama-herd

Or with Homebrew (macOS/Linux):

brew tap geeks-accelerator/ollama-herd
brew trust geeks-accelerator/ollama-herd   # Homebrew 6.x requires this for third-party taps
brew install ollama-herd

Step 2: Start the Router

herd

The router starts on port 11435. You'll see:

Ollama Herd ready on port 11435

Step 3: Start Node Agents

On each device running Ollama (including the router machine if it also runs Ollama):

pip install ollama-herd
herd-node

The node discovers the router automatically via mDNS:

Discovered router at 10.0.0.100:11435
Heartbeat sent: 2 models loaded, 128GB available

The same herd and herd-node commands work on Windows from PowerShell or Command Prompt.

Can't use mDNS? (common on corporate networks) Connect directly: herd-node --router-url http://10.0.0.100:11435. To find the router's LAN IP, run ipconfig getifaddr en0 on macOS, hostname -I on Linux, or ipconfig on Windows (the IPv4 Address line).

Running on one machine

Start with the computer you have. Run both commands on it, each in its own terminal:

herd        # terminal 1: the router
herd-node   # terminal 2: the node agent for this machine's Ollama

Your tools get one endpoint that speaks the OpenAI, Ollama and Anthropic APIs, plus the dashboard, request traces and health checks. Adding a second machine later is one command, herd-node on that machine, and nothing changes for your tools: they keep the same URL. If one machine and one tool is all you will ever run, plain Ollama is enough; Herd earns its place when a second machine or a second busy client shows up.

To keep the router off your network (on a laptop, say), bind it to loopback and point the node at it directly, because discovery hands out the machine's network address, which a loopback-only router doesn't listen on:

herd --host 127.0.0.1
herd-node --router-url http://127.0.0.1:11435

On macOS, nothing restarts herd after a reboot; ready-made launchd agents are in the repository's docs/examples/launchd.

Tested 2026-10-26 on one Mac: Ollama Herd 0.11.4 from PyPI in a clean environment, router and node on the same machine, in front of Ollama 0.40.3.

Step 4: Verify the Fleet

Check that nodes are online:

curl -s http://localhost:11435/fleet/status | python3 -m json.tool

You should see your nodes listed with their models, memory, and status.

Or open the dashboard in your browser:

http://localhost:11435/dashboard

Step 5: Send Your First Request

OpenAI format:

curl http://localhost:11435/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.2:3b",
    "messages": [{"role": "user", "content": "Hello from the fleet!"}],
    "stream": false
  }'

Ollama format:

curl http://localhost:11435/api/chat -d '{
  "model": "llama3.2:3b",
  "messages": [{"role": "user", "content": "Hello from the fleet!"}],
  "stream": false
}'

The router scores all available nodes and routes the request to the best one. Check the response headers to see which node handled it:

curl -v http://localhost:11435/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "llama3.2:3b", "messages": [{"role": "user", "content": "Which node am I on?"}], "stream": false}' \
  2>&1 | grep X-Fleet
< X-Fleet-Node: mac-studio-ultra
< X-Fleet-Score: 85

Step 6: Use with Your Tools

Point any OpenAI-compatible tool at the router, no code changes needed:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11435/v1", api_key="not-needed")
response = client.chat.completions.create(
    model="llama3.2:3b",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
)
for chunk in response:
    print(chunk.choices[0].delta.content or "", end="")

Replace localhost with the router's LAN IP if connecting from another machine.

What Just Happened

  1. Your node agents discovered the router via mDNS
  2. Each node sends heartbeats every 5 seconds with system state (CPU, memory, thermal, loaded models)
  3. Your request hit the router, which scored all nodes on 8 signals
  4. The highest-scoring node received the request through its dedicated queue
  5. The response streamed back through the router to your client

Next Steps

  • Core Concepts, Understand the mental model behind scoring, queues, and capacity
  • Integrations, Connect Open WebUI, LangChain, CrewAI, and other tools
  • Deployment, Production setup, monitoring, and tuning
  • Dashboard, Open http://localhost:11435/dashboard to see your fleet in real time

Upgrading

pip install --upgrade ollama-herd
# or: brew upgrade ollama-herd

Restart the router and node agents after upgrading. See CHANGELOG for what's new.

Coming From Another Tool?

See how Ollama Herd compares: