Deployment

Production setup, monitoring, log analysis, health checks, and operational tips.

Architecture Decisions

Before deploying, decide:

Which machine runs the router? Pick a machine that's always on. The router is lightweight (minimal CPU/memory), it doesn't run inference, just coordinates. Your most powerful machine is usually the best choice since it's likely always on and also runs Ollama.

Which devices enable capacity learning? Dedicated servers (Mac Studio, Linux boxes) should leave it disabled, they always run at full capacity. Laptops and shared devices should enable it so routing adapts to usage patterns.

Which devices run which models? The router handles this dynamically, but you control which models are pulled to each node. Large models go on high-RAM machines. Small fast models go on everything else. Embedding models go on one or two nodes.

Starting the Fleet

Router:

pip install ollama-herd
herd

Each node:

pip install ollama-herd
herd-node

Running Linux servers in containers? See Running Herd in a container.

For nodes that double as workstations:

FLEET_NODE_ENABLE_CAPACITY_LEARNING=true herd-node

Running as Background Services

macOS:

# Router
nohup herd &>/dev/null & disown

# Node
nohup herd-node &>/dev/null & disown

For automatic startup, the OSS repo ships example launchd plists for the router and the node.

Linux (systemd):

# /etc/systemd/system/ollama-herd.service
[Unit]
Description=Ollama Herd Router
After=network.target

[Service]
ExecStart=/usr/local/bin/herd
Restart=always
User=your-user

[Install]
WantedBy=multi-user.target
sudo systemctl enable --now ollama-herd

Windows: the same herd and herd-node commands run from PowerShell or Command Prompt. Herd does not ship a Windows service definition, so keep them running with whatever you already use to start long-lived processes.

Monitoring

Dashboard

Open http://router-ip:11435/dashboard for real-time fleet monitoring, in light or dark, with 5 tabs:

  • Overview, Fleet verdict, live figures with sparklines, activity chart, node cards
  • Models, Per-model performance and trends (24h–7d)
  • Usage, Per-tag analytics (requires request tagging)
  • Health, 45+ automated health checks, with model mix recommendations
  • Settings, Runtime toggles, node versions, appearance

Benchmarks (capacity growth over time) has its own page, linked from Models and Settings.

Health API

curl -s http://localhost:11435/dashboard/api/health | python3 -m json.tool

Returns 45+ automated checks covering fleet liveness, routing quality, backend reliability, and observability. Each check carries a severity (INFO / WARNING / CRITICAL) and an actionable recommendation. Highlights:

  • Fleet liveness, offline nodes, degraded nodes, memory pressure (OS-reported on macOS and Linux), underutilized nodes
  • Routing quality, VRAM fallbacks (requests served by a loaded model of the same category), model thrashing, request timeouts, retry rates, context waste detection
  • MLX backend (Apple Silicon only), server down (CRITICAL), server quarantined (after 5 crashes in 5 minutes), memory-blocked (skipped start due to memory gate)
  • Text embedding backend, embed error rate, text-embedding backend missing (fastembed not installed), text-embedding Ollama bypass (native server not running, embed requests still contend with LLMs), nomic still loaded in Ollama despite native server traffic (VRAM waste)
  • Vision backend, backend missing (weights cached but onnxruntime not loadable)
  • Observability, trace-store write failures (closes a silent SQLite-contention black hole), version mismatch, KV-cache bloat, zombie reaper activity
  • Stream + client integrity, client disconnects, incomplete streams, context protection events

The dashboard Health tab renders every check with current status and recommended remediation. For per-model context window utilization, the /dashboard/api/context-usage endpoint shows allocated vs. actual usage and recommends right-sized values to reclaim wasted VRAM.

Fleet Status

curl -s http://localhost:11435/fleet/status | python3 -m json.tool

Returns per-node details: status, hardware, memory, CPU, loaded models, queue depths.

Queue Depths (Lightweight)

curl -s http://localhost:11435/fleet/queue | python3 -m json.tool

Returns just queue depths, designed for client-side backoff logic.

Log Analysis

Structured Logs (JSONL)

All events are written to ~/.fleet-manager/logs/herd.jsonl, one JSON object per line, daily rotation, 30-day retention.

# Tail the live log
tail -f ~/.fleet-manager/logs/herd.jsonl | python3 -m json.tool

# Find errors
grep '"level":"ERROR"' ~/.fleet-manager/logs/herd.jsonl

# Find events for a specific model
grep '"model":"llama3.3:70b"' ~/.fleet-manager/logs/herd.jsonl

# Count errors by component
grep '"level":"ERROR"' ~/.fleet-manager/logs/herd.jsonl | \
  python3 -c "import sys,json; from collections import Counter; \
  c=Counter(json.loads(l)['logger'] for l in sys.stdin); \
  print('\n'.join(f'{v:4d} {k}' for k,v in c.most_common()))"

Log Levels

VariableDefaultControls
FLEET_LOG_LEVELDEBUGWhat's written to JSONL
FLEET_CONSOLE_LOG_LEVELINFOWhat's printed to terminal

Set FLEET_LOG_LEVEL=INFO in production to reduce file size.

Request Traces (SQLite)

Every routing decision is recorded in ~/.fleet-manager/latency.db:

# Recent requests
sqlite3 ~/.fleet-manager/latency.db \
  "SELECT model, node_id, latency_ms, status FROM request_traces ORDER BY timestamp DESC LIMIT 10"

# Failures
sqlite3 ~/.fleet-manager/latency.db \
  "SELECT model, node_id, error_message FROM request_traces WHERE status='failed' ORDER BY timestamp DESC LIMIT 10"

# Average latency per model
sqlite3 ~/.fleet-manager/latency.db \
  "SELECT model, ROUND(AVG(latency_ms)/1000.0, 1) as avg_secs, COUNT(*) as requests \
   FROM request_traces WHERE status='completed' GROUP BY model ORDER BY requests DESC"

# Error rate over the last hour
sqlite3 ~/.fleet-manager/latency.db \
  "SELECT status, COUNT(*) FROM request_traces \
   WHERE timestamp > strftime('%s','now') - 3600 GROUP BY status"

Resilience Features

Auto-Retry

If a node fails before the first chunk, the router re-scores and retries on the next-best node. Up to 2 retries (configurable via FLEET_MAX_RETRIES).

Model Fallbacks

Clients specify backup models: "fallback_models": ["qwen2.5:32b", "qwen2.5:7b"]. The router tries each in order through the full scoring pipeline.

Auto-Pull

Missing models are automatically pulled to the best available node. Configurable via FLEET_AUTO_PULL (default: true).

Context Protection

Strips unnecessary num_ctx from requests to prevent model reload hangs. Auto-upgrades to a larger loaded model when possible. Configurable via FLEET_CONTEXT_PROTECTION (default: strip).

Graceful Drain

Send SIGTERM to a node agent:

  1. Capacity learner state saves to disk
  2. Drain heartbeat sent to router
  3. Router stops routing new requests to this node
  4. In-flight requests complete normally
  5. Pending requests rebalance to other nodes
  6. Agent shuts down cleanly

Zombie Reaper

Background task detects in-flight requests that never completed (connection drops, Ollama crashes) and cleans them up so queues stay accurate.

Configuration

All settings via environment variables. No config files.

Key Server Variables

VariableDefaultWhat
FLEET_PORT11435Router listen port
FLEET_HEARTBEAT_TIMEOUT15.0Seconds before node is degraded
FLEET_HEARTBEAT_OFFLINE30.0Seconds before node is offline
FLEET_MAX_RETRIES2Max retry attempts per request
FLEET_AUTO_PULLtrueAuto-pull missing models
FLEET_CONTEXT_PROTECTIONstripContext size protection mode
FLEET_DYNAMIC_NUM_CTXfalseEnable automatic context window optimization
FLEET_LOG_LEVELDEBUGJSONL log level

Key Node Variables

VariableDefaultWhat
FLEET_NODE_ENABLE_CAPACITY_LEARNINGfalseEnable adaptive capacity
FLEET_NODE_DATA_DIR~/.fleet-managerState file directory

See the full configuration reference for all 44+ variables with tuning guidance.

Ollama Settings

For best results with a fleet, set these Ollama environment variables on each node. A shell export only reaches an ollama serve started from that shell, so set them where your platform starts Ollama:

# macOS (Ollama app): set, then quit and reopen the Ollama app
launchctl setenv OLLAMA_NUM_PARALLEL 2
launchctl setenv OLLAMA_KEEP_ALIVE -1
launchctl setenv OLLAMA_MAX_LOADED_MODELS 10

# Linux (systemd): add under [Service], then restart
sudo systemctl edit ollama
#   Environment="OLLAMA_NUM_PARALLEL=2"
#   Environment="OLLAMA_KEEP_ALIVE=-1"
#   Environment="OLLAMA_MAX_LOADED_MODELS=10"
sudo systemctl restart ollama

# Windows (PowerShell): set, then restart Ollama from the system tray
[System.Environment]::SetEnvironmentVariable("OLLAMA_NUM_PARALLEL", "2", "User")
[System.Environment]::SetEnvironmentVariable("OLLAMA_KEEP_ALIVE", "-1", "User")
[System.Environment]::SetEnvironmentVariable("OLLAMA_MAX_LOADED_MODELS", "10", "User")

KEEP_ALIVE=-1 prevents model thrashing. Set MAX_LOADED_MODELS to a positive integer: -1 is not "unlimited" there, it fails to parse and Ollama silently falls back to a default of 3 loaded models. On macOS, a value set in ~/.zshrc overrides launchctl setenv, so update both. Afterwards, confirm Ollama is back with curl -s localhost:11434/api/version.

Running Herd in a container

No Docker is required: pip install ollama-herd is the main install, and the only one for Mac and Windows nodes. If you run Linux servers in containers anyway, there is an official image, ghcr.io/geeks-accelerator/ollama-herd, for linux/amd64 and linux/arm64, since v0.12.7. One image runs both programs: herd by default, herd-node when given as the command. Ollama is not in the image and stays on the host. Tags are X.Y.Z per release, X.Y and latest; pin X.Y.Z to choose when you upgrade. The download page has the two-line quick start; this section is the full set.

RoleIn a container?
Router on a Linux server or NASYes: one port and a data volume
Router under Docker Desktop (Mac, Windows)Works, but nodes cannot discover it over mDNS, so give each node the router’s URL
Node on LinuxYes, with --network host (discovery, the node’s address and its Ollama proxy all need the host network)
Node on a Mac or Windows PCNo. Docker there is a Linux VM, so the node would report the VM’s memory and cores instead of the machine’s. Install natively

The router

docker run -d --name herd --network host --restart unless-stopped \
  -v herd-data:/home/herd/.fleet-manager \
  ghcr.io/geeks-accelerator/ollama-herd

Always mount the volume. /home/herd/.fleet-manager holds the trace store, logs, learned latencies and the install id, so without it every recreated container starts with no history (Herd logs a warning when that happens). For a bind mount, give the directory to the image’s user first: sudo chown 10001:10001 /path/to/dir. With -p 11435:11435 instead of the host network the router works, but advertises the container’s own address over mDNS, so start every node with the router’s URL.

A node on Linux

docker run -d --name herd-node --network host --restart unless-stopped \
  -v herd-node-data:/home/herd/.fleet-manager \
  ghcr.io/geeks-accelerator/ollama-herd herd-node

With the host network the node finds Ollama at http://localhost:11434, native or in its own container with the port published; set FLEET_NODE_OLLAMA_HOST if it is elsewhere. Useful extras, each as -e NAME=value:

  • FLEET_NODE_ROUTER_URL=http://192.168.1.10:11435 skips discovery. Needed when the router runs with a published port or under Docker Desktop.
  • OLLAMA_NUM_PARALLEL=4, if you raised it for Ollama. Outside a container the node reads it from Ollama’s systemd unit; inside one it cannot, and without it the router sends that node one request at a time.
  • FLEET_NODE_NODE_ID picks a node name instead of the host’s.

GPUs

NVIDIA: add --gpus all (with the NVIDIA Container Toolkit installed) so the node can run nvidia-smi. The node then counts the card’s free memory when the router checks whether a model fits; without it the card’s memory is invisible to Herd. AMD: nothing extra, since GPU memory is read from the amdgpu driver’s files in /sys. If the dashboard’s node card shows no GPU line or a wrong figure (an AMD APU with a large BIOS carve-out, say), declare it with -e FLEET_NODE_GPU_MEMORY_GB=160. Either way Herd reads GPU memory only; it does not read GPU load. The mixed-GPU guide covers this in depth.

Rootless Docker and Podman

Rootless Podman’s --network host uses the real host network, so the commands above work with podman in place of docker. Rootless Docker needs Engine 29.5 or later for that. A host’s own avahi-daemon can keep running; they share mDNS port 5353. The project tested the node container on Ubuntu 26.04 under rootful Docker, rootless Docker and rootless Podman 5.7, discovering a router on another machine and serving requests.

What a containerized node does not do

  • Start or restart Ollama: start it on the host first. The router’s "restart Ollama" action does not reach the host either.
  • Serve native text embeddings: the image has the base dependencies, so embedding models, nomic-embed-text included, are served through Ollama.
  • Run the Apple Silicon extras (MLX, image generation, speech-to-text), which need a Mac, natively.

Upgrading is docker pull ghcr.io/geeks-accelerator/ollama-herd:latest, then the same docker run with the same volume. Run into something on your distribution? Open an issue with what you saw.

A node against a remote Ollama

By default a node tells the router where to find its Ollama by reporting its own LAN address. That is right on a machine sitting on your desk and wrong in a container on a bridge network, where the address it sees is the Docker bridge (172.17.0.x) and the router cannot reach it. Set the address explicitly:

# On the node, before starting herd-node
export FLEET_NODE_OLLAMA_HOST=http://192.168.1.42:11434   # reachable from the ROUTER

The router resolves a node's Ollama address in this order, most trusted first:

  1. An explicitly configured non-loopback FLEET_NODE_OLLAMA_HOST. If you set it, you know your topology better than any inference we could make.
  2. The address the heartbeat arrived from, which is reachable by construction because a packet just came from it.
  3. The node's self-reported LAN IP, as a last resort.

A node co-located with the router is a special case and always uses localhost, which sidesteps firewall rules and IP changes.

Before v0.9.3 the configured value was discarded, so containerized nodes registered an address the router could not reach: every request failed while the dashboard showed the node as healthy. If you run nodes in containers, upgrade rather than working around it.

Data Storage

All persistent data lives in ~/.fleet-manager/ (configurable via FLEET_DATA_DIR):

~/.fleet-manager/
  latency.db                           # SQLite: traces, latency, usage, benchmarks
  logs/
    herd.jsonl                         # Structured logs (daily rotation)
  capacity-learner-{node-id}.json      # Learned behavioral data (per node)

SQLite uses WAL mode for concurrent read/write. All files are human-readable and can be backed up, queried, or deleted at will.

Next Steps