Demo

See it route in 60 seconds

Four machines, one endpoint. Your code names a model, and Herd sends each request to the machine that has it ready.

A home office with two Mac Studios and a tower PC by a window, captioned: You already own an AI cluster.
60 seconds, narrated, with captions on screen. The video plays from YouTube when you press play; nothing loads from YouTube before that. Watch on YouTube.

What you're watching

  1. 0:00 You already own an AI cluster. Two Mac Studios, a Linux tower and a laptop: the kind of machines people already have on their desks, each one working alone.
  2. 0:05 One command per machine. herd on one machine, herd-node on each of the others. The nodes find the router over mDNS and appear on the dashboard. No config files.
  3. 0:15 Your code names a model. Herd picks the machine. A Python script sends 24 jobs across six models to one URL. Every reply carries an X-Fleet-Node header, so the output shows which machine answered each job.
  4. 0:26 Most days, the Mac Studios carry the load. Requests go to the machine that already has the model loaded and has room for it, so the big models land on the big machines.
  5. 0:32 The laptop joins in. Start herd-node on a laptop and it joins the herd and starts taking requests, with nothing to change in any client.
  6. 0:40 Claude Code runs on it too. Claude Code points at the same endpoint with one environment variable, and its requests are routed like everything else.
  7. 0:44 A machine leaves. Nothing breaks. A review bot is working through pull requests on the Linux box when it goes offline. The rest go to a Mac Studio and the bot finishes with zero errors.

The script from the video

This is the whole program behind the routing scene. It uses the standard OpenAI Python SDK, and the only Herd-specific line is the base URL. It never says which machine should run what: it names a model, and the router decides.

from concurrent.futures import ThreadPoolExecutor
from openai import OpenAI

# The only config: one URL. No machines, no ports, no per-device setup.
herd = OpenAI(base_url="http://127.0.0.1:11435/v1", api_key="local")

JOBS = [
    ("qwen3:8b",         "Tag this support ticket"),
    ("gemma3:27b",       "Summarise this pull request"),
    ("gpt-oss:120b",     "Draft the release notes"),
    ("qwen3-coder:30b",  "Write a test for parse()"),
    ("nomic-embed-text", "Embed a docs page for search"),
    ("qwen2.5:7b",       "Classify this commit message"),
] * 4

def run(job):
    model, task = job
    if "embed" in model:
        r = herd.embeddings.with_raw_response.create(model=model, input=[task])
    else:
        r = herd.chat.completions.with_raw_response.create(
            model=model, max_tokens=80, messages=[{"role": "user", "content": task}])
    print(f"{task:30} {model:17} → {r.headers['x-fleet-node']}")

with ThreadPoolExecutor(max_workers=8) as pool:
    list(pool.map(run, JOBS))

Every response carries X-Fleet-Node, the machine that served it, alongside the other X-Fleet-* headers listed in the API reference. The same endpoint speaks the Ollama and Anthropic APIs too, so Open WebUI, Claude Code and Codex work without a model map.

Try it on your own machines

Install on every machine that runs Ollama (macOS, Linux or Windows):

pip install ollama-herd

Start the router on one machine, then a node on each of the others:

herd        # on one machine
herd-node   # on each of the others

Point any client at http://<router>:11435. For Claude Code:

export ANTHROPIC_BASE_URL=http://<router>:11435
export ANTHROPIC_AUTH_TOKEN=dummy   # any non-empty value
claude

Open the dashboard at http://<router>:11435/dashboard to watch requests land, as in the video.

Get started in 60 seconds → Star on GitHub

Read next