Guides

Start Here

• Quickstart, Install to first routed request in 60 seconds. Create a fleet with two commands, send a request, and see it land on the right machine.
• Core Concepts, The mental model behind Ollama Herd. Nodes, heartbeats, scoring signals, queues, capacity modes, and how they fit together.

Learn the System

• Routing Engine, How the 5-stage scoring pipeline eliminates bad candidates, scores survivors across 8 signals, and picks a winner for every request.
• Adaptive Capacity, How your fleet learns when each device has spare compute. Weekly behavioral models, meeting detection, app fingerprinting, and memory ceilings.

Put It to Work

• Claude Code CLI, Point Claude Code CLI at your hardware. Native Anthropic Messages API, three-layer context management, per-tier model routing, tool-schema fixup. Fixes the "breaks at 30K tokens" failure mode on local Qwen3-Coder models.
• Integrations, Connect Ollama Herd to Open WebUI, LangChain, CrewAI, OpenClaw, Aider, Continue.dev, LlamaIndex, and any OpenAI-compatible client.
• Deployment, Multi-node setup, monitoring, log analysis, health checks, graceful drain, and production tips.
• Load Balancing Ollama, How to distribute requests across multiple Ollama instances, from HAProxy configs to zero-config routing that avoids cold-load stalls.
• Codex CLI, Run OpenAI Codex and the Codex desktop app against local models on your Macs. One config block, no model map, verified end to end on both surfaces.
• OpenClaw, Run OpenClaw agents on your own Macs. Install the Herd skill from ClawHub, point OpenClaw at the fleet, and give long agent sessions the context headroom they need.
• Mac Cluster, The two different things people mean by running LLMs across multiple Macs, routing vs sharding, which one you actually need, and where Herd is the wrong tool.
• Local Models, Which open LLMs, vision, and coding models actually run on Apple Silicon, tested first-hand on a real Mac fleet, including the ones that do not run locally yet.
• Mac Memory, How much unified memory each model needs, the sizing math, and which Mac runs a 30B, 70B, 120B, or 480B model.
• Load Ceiling, Why a 418 GB model fits on a 512 GB Mac's disk but won't load through MLX, the iogpu.wired_limit_mb footgun, and why your setup may differ.
• Routing vs Sharding, Splitting one model across Macs and routing requests between them solve opposite problems. The memory math and how to tell which you need.
• MLX vs Ollama, We benchmarked both backends head to head on an M3 Ultra. They tie on speed, so the real choice is operational. Here is how to pick.
• Multimodal, Herd routes five model types from one endpoint: LLMs, embeddings, image generation, speech-to-text, and vision. The pattern behind all of them.
• Image Generation, Route Flux and Stable Diffusion image generation across a fleet of Macs: mflux, DiffusionKit, and Ollama-native, through one endpoint.
• Troubleshooting, Why Claude Code breaks around 30K tokens, why models loop on tool calls, return empty, or hit Metal out-of-memory, and how to fix each.
• Local AI Cost, When running AI locally on Apple Silicon actually beats cloud API pricing, the break-even math, and why fleet scale flips the answer.
• Open WebUI, Connect Open WebUI to several Ollama machines through one Herd endpoint with model-aware routing instead of random backend selection.
• Concurrency, Tune OLLAMA_NUM_PARALLEL, loaded models, and queue depth on one machine without OOM errors, and when to scale across a fleet.
• Cold Starts, Why random load balancing across Ollama servers throws away warm KV caches, and what cache-aware routing does instead.
• RAG at Scale, Keep document-ingestion embeddings from blocking interactive chat by separating embedding and generation across a fleet.
• Remote Access, Reach your Ollama fleet from anywhere over Tailscale without exposing a single inference node to the public internet.
• n8n, Run n8n automations, agents, and RAG against a fleet of local machines through one endpoint, with concurrency that spreads across nodes.
• API Reference, Every endpoint with request/response schemas, headers, error codes, and curl examples.