Run Claude Code with Local Models

TL;DR

If you already have a herd running, two env vars:

export ANTHROPIC_BASE_URL=http://localhost:11435
export ANTHROPIC_AUTH_TOKEN=dummy   # any non-empty value
claude

That's it. Claude Code now talks to your local model with full tool use, streaming, and the standard agentic loop, routed through Ollama Herd's scoring engine, queue pipeline, and context management layers.

This is the Claude Code client driving an open model you run, not Claude. Anthropic does not support pointing Claude Code at non-Claude models, and a local model is less reliable at long tool-using tasks than Claude. It works because Herd speaks the Anthropic Messages API; see the troubleshooting section and the note on logs before relying on it.

Want the full setup from scratch? Start with the Quickstart to get herd + herd-node running, then come back here for Claude Code specifics.

Ollama already does the single-machine case. When do you need Herd?

Ollama speaks the Anthropic Messages API natively, so ollama launch claude can point Claude Code at a single Ollama instance on your machine. If that is your whole setup, use it: it is the simplest path and it works well.

Herd is for the cases the single-node path does not cover:

Ollama column per Ollama's Anthropic compatibility docs, checked 2026-09-29.

Short version: one machine and short sessions, use Ollama's built-in path. Several machines, or long tool-heavy sessions, use Herd. Other tools run Claude Code on a single local machine, or route it out to cloud providers. Herd is the one that routes across the machines you already own, Macs, Linux servers and Windows PCs alike, picking the one that is idle and already has your model loaded. The same endpoint also serves Codex, so both agent CLIs share one fleet. The rest of this guide covers the Herd specifics.

Local models break agentic loops. Herd keeps them alive.

The hard part of running Claude Code on a local model is not raw speed, it is reliability. Local models drop out of JSON mid tool call, omit required parameters, and degrade at long context in ways that quietly break the agentic loop. A demo works; a 2,000 message session falls apart. Herd ships the machinery that keeps the loop running:

• Tool-call repair recovers malformed tool-call JSON on MLX-served models, so a bad call is fixed instead of failing the turn.
• Tool-schema fixup injects the optional-parameter defaults that Qwen3-Coder starts silently omitting at around 30K tokens (root cause traced to llama.cpp#20164).
• Three-layer context management keeps long sessions under the model's real limits instead of wedging on a multi-minute prefill.

This is the difference between a coding agent that works in a screenshot and one that survives a real day of use. Each layer is detailed below.

Verified end to end. Claude Code 2.1.68 was driven headless against the fleet on a multi-file task: a repo whose test suite would not collect, needing a module written from scratch, a bug fixed, and a guard added. It reached 6 passed on its own across 14 turns, 12 of them tool-calling, exercising Bash, Edit, Glob, Read, Write, and TodoWrite. No translation defects surfaced.

What Ollama Herd does for Claude Code

Ollama Herd exposes a native Anthropic Messages API (/v1/messages and /v1/messages/count_tokens), translates requests to Ollama's wire format, runs them through the same scoring + queue + trace pipeline as every other route, and translates the response back to Anthropic SSE event sequences.

No LiteLLM sidecar. No OpenAI-format proxy. Claude Code's ANTHROPIC_BASE_URL points straight at the herd router.

Set the context length first. It silently breaks Claude Code.

Claude Code sends a large system prompt and more than a dozen tool definitions on every turn, so a small context window fails fast. Ollama sizes its default context by GPU memory (4K under 24 GB, 32K from 24 to 48 GB, 256K above that), which means a 16 GB laptop and a 24 GB RTX card start from very different places. Ollama recommends at least 64000 tokens for coding agents.

Set OLLAMA_CONTEXT_LENGTH on every node to the largest per-request context the fleet needs, for example 65536 or 131072. Setting it lower is the expensive mistake: on our reference fleet, dropping it from 131072 to 32768 cut prefix-cache hits from 5,772 to 770 and took time to first token from about 1.0 s to about 6.3 s, while decode speed did not move at all. Every throughput graph stayed green while latency doubled.

# macOS (Ollama app): set it, then quit and reopen the app
launchctl setenv OLLAMA_CONTEXT_LENGTH 131072

# Linux (systemd)
sudo systemctl edit ollama   # add: Environment="OLLAMA_CONTEXT_LENGTH=131072"

# Windows: set it as a system environment variable, then restart Ollama

Ollama multiplies this by OLLAMA_NUM_PARALLEL, and ollama ps does not show the per-request value. Read it from the launch arguments instead:

ps -Ao args | grep llama-server | grep -oE '\-c [0-9]+ \-np [0-9]+'
# per-request context = -c divided by -np

Then tell Claude Code how big the window really is, so it compacts before the model degrades instead of assuming a hosted Claude window. Claude Code's CLAUDE_CODE_MAX_CONTEXT_TOKENS sets the window it assumes; see Claude Code settings that matter below.

Ollama defaults and the 64000 recommendation per Ollama's context length docs, checked 2026-09-29. Fleet measurements from Ollama Herd v0.9.5's configuration reference.

Setup

Step 1: Pull a coding model

Claude Code makes many tool calls, so you want a coding-tuned model that follows a schema without getting derailed. One model per tier:

# Main coding loop (sonnet tier)
ollama pull qwen3-coder:30b

# Heavier tier for hard problems (opus tier)
ollama pull qwen3:32b

# Fast tier for cheap turns (haiku tier)
ollama pull qwen3:14b

These are the models behind the defaults below. Newer families are available too (Qwen 3.6, Gemma 4, GLM-5, Kimi K2 Code, DeepSeek V4), so substitute freely. The tier shape is what matters, not these exact names.

Tool-use quality matters more than chat quality here, because Claude Code calls tools constantly. How the models we have run hold up:

Size for the KV cache, not just the weights: a model's resident memory is weights plus KV cache, the KV cache grows with context, and Ollama allocates it once per parallel slot (OLLAMA_NUM_PARALLEL). qwen3-coder:30b is 18.6 GB of weights, but with 4 parallel slots at its default 262K context it measured 122.9 GB resident, which is why the context length you set above matters as much as the model you pick.

Step 2: Point Claude Code at your router

export ANTHROPIC_BASE_URL=http://localhost:11435
export ANTHROPIC_AUTH_TOKEN=dummy   # any non-empty string
claude

Step 3: (Optional) Pin a model

You can skip this. Claude Code sends model IDs like claude-sonnet-4-5, and as of v0.9.0 Herd resolves them itself. There is no default map to configure and nothing to keep in sync with what you have pulled.

Resolution order, first match wins:

Resolution order in v0.9.5 (first match wins)

1. Does FLEET_ANTHROPIC_MODEL_MAP have an entry for this exact ID?
   Yes: use it. Your pins always win. Logged as explicit-map.
2. Is the ID a model your fleet actually has?
   Yes: pass it through unchanged (passthrough). This is why claude --model qwen3-coder:30b works.
3. Is a suitable model already loaded somewhere on the fleet?
   Yes: use the best loaded one for the tier, no cold load (auto-loaded). Coding models rank first; haiku leans toward smaller models and opus toward larger ones; a request with images considers only vision models. Embedding and image-generation models are never picked.
4. Is a suitable model on disk?
   Yes: use the best on-disk one for the tier, accepting a cold load (auto-ondisk).
5. Does the map have a "default" key?
   Yes: use it. Otherwise the request returns a 404 that tells you to pull a model.

This matters because model IDs change. When Anthropic ships a new version string, auto-routing picks it up; a hand written map would have silently sent it to the wrong tier.

To pin a specific ID anyway:

export FLEET_ANTHROPIC_MODEL_MAP='{
  "claude-opus-4-7": "mlx:Qwen3-Coder-Next-4bit",
  "claude-haiku-4-5": "gpt-oss:120b"
}'

Entries you set override auto-routing; unmapped IDs still auto-route. Values can reference Ollama models (qwen3-coder:30b) or MLX models (mlx:Qwen3-Coder-Next-4bit), see MLX Backend below. To disable auto-routing and require an explicit map, set FLEET_ANTHROPIC_AUTO_ROUTE=false. Two optional overrides apply after resolution, and both are off unless you set them: FLEET_ANTHROPIC_VISION_MODEL for requests that contain images, and size-based escalation for long prompts.

Step 4: Verify it is working

Before opening Claude Code, check the router answers in Anthropic format, and that tools round-trip:

# Plain request
curl -s http://localhost:11435/v1/messages \
  -H "Content-Type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -d '{"model":"claude-sonnet-4-5","max_tokens":80,
       "messages":[{"role":"user","content":"say hi"}]}'
# Expect: "type": "message" and "stop_reason": "end_turn"

# Tool round-trip
curl -s http://localhost:11435/v1/messages \
  -H "Content-Type: application/json" \
  -d '{"model":"claude-sonnet-4-5","max_tokens":300,
       "messages":[{"role":"user","content":"What is the weather in Paris? Use the get_weather tool."}],
       "tools":[{"name":"get_weather","description":"Get current weather for a city",
                 "input_schema":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}]}'
# Expect: a tool_use block for get_weather and "stop_reason": "tool_use"

Every response carries X-Fleet-Node (which machine answered) and, when the model you asked for was not available and a fallback served it, X-Fleet-Fallback. Add -si to either command to see them.

Running on Linux, Windows or NVIDIA

Nothing on this page needs a Mac except the MLX backend. Herd routes Claude Code's requests to Ollama over its HTTP API, so any machine that runs Ollama can serve them: a Linux box with an RTX card, a Windows desktop, or a Mac, mixed freely in one fleet. Start herd-node on each one and it finds the router over mDNS.

pip install ollama-herd
herd                                            # on the router machine
herd-node                                       # on every machine running Ollama
herd-node --router-url http://ROUTER_IP:11435   # if mDNS is blocked

Running the node in Docker? Set FLEET_NODE_OLLAMA_HOST to an address the router can reach. See running a node in a container.

Apple Silicon only: mlx: models, FLEET_NODE_MLX_SERVERS, and meeting detection. Tool-schema fixup and context management run in the router, so they work the same whatever serves the model; tool-call JSON repair applies to MLX-served models only. Our end-to-end Claude Code runs were on Macs. Memory-fit scoring reads system RAM, not GPU VRAM.

Router on another machine

export ANTHROPIC_BASE_URL=http://ROUTER_IP:11435   # no /v1 suffix
export ANTHROPIC_AUTH_TOKEN=dummy

/v1/messages is open by default, like the rest of the router. On a shared network, require a key on the router and send the same value from Claude Code:

# on the router
export FLEET_ANTHROPIC_REQUIRE_KEY=true
export FLEET_ANTHROPIC_API_KEY=sk-local-something-long

# on the machine running Claude Code
export ANTHROPIC_AUTH_TOKEN=sk-local-something-long

For access from outside your LAN, see secure remote access with Tailscale.

Per-Tier Model Routing

Claude Code's --model flag lets users trade speed for quality per-invocation:

claude --model claude-haiku-4-5      # fast turns, cheaper model
claude --model claude-sonnet-4-5     # default balance
claude --model claude-opus-4-7       # highest quality

A good production mapping splits speed from quality and diversifies failure modes:

• claude-haiku-* → a smaller hot Ollama model (fast, already pinned)
• claude-sonnet-* → a solid coding model for the main loop
• claude-opus-* → an 80B-class MoE via MLX for the tough cases

Different model families on different tiers also means if Qwen3 has a bad day, your haiku fallback still works.

Claude Code settings that matter with local models

These are Claude Code's own variables, documented by Anthropic in its model configuration and gateway compatibility pages. Herd does not read them, and we have not tested each one against Herd.

One more client-side limit to know: through a custom base URL, Claude Code aborts a stream that sends nothing for 300 seconds. A very large model loading cold can take minutes before its first token, so pre-warm big models before a session rather than letting the first request pay for the load (see Troubleshooting).

The /compact command works out of the box

Claude Code's /compact slash command is client-side orchestration over the standard /v1/messages endpoint, it sends a normal request with a trailing user message asking the model to summarise the conversation, then locally replaces the in-memory history with the response. No special beta header, endpoint, or body field.

That means:

• /compact works against Ollama Herd with no special support required. Same as against hosted Claude.
• We augment it with hosted-Claude-parity context management layers that run before the model sees the request (see next section).

In practice, a 2,700-message session that would have timed out or produced garbage on raw local inference can hit /compact successfully on our fleet. In the session we measured, Layer 1 alone cut the prompt from 206K to 125K tokens (39 percent) before it reached the model.

Three-Layer Context Management

The biggest structural difference between hosted Claude and raw local inference is context hygiene. Hosted Claude silently strips stale tool results, summarises long sessions, and refuses oversized requests. Raw local Ollama doesn't. Ollama Herd ships three layers that close that gap.

Pipeline with v0.9.5 defaults; token counts from one measured session

Claude Code request206K tokens in the measured session

over 100K tokens?

Layer 1: tool-result clearingOn by default. Older tool_result bodies become a placeholder; the 3 most recent stay verbatim. No LLM call.206K to 125K tokens, 39% smaller

then

Layer 2: LLM compactorOff by default. When enabled: summarizes bulky older tool results above 20K tokens, and every older tool result above 150K.

still over 180K tokens?

Layer 3: HTTP 413Refused before the model sees it. Run /compact and resubmit.

Otherwise: sent to the model125K tokens in the measured session

Layer 1, Mechanical tool-result clearing

When the Anthropic request exceeds FLEET_ANTHROPIC_AUTO_CLEAR_TOOL_USES_TRIGGER_TOKENS (default 100K), older tool_result blocks are replaced with a short placeholder before the request reaches the model. No LLM call, microsecond-scale. Matches hosted Claude's Context Editing API.

Configurable:

FLEET_ANTHROPIC_AUTO_CLEAR_TOOL_USES_TRIGGER_TOKENS=100000
FLEET_ANTHROPIC_AUTO_CLEAR_TOOL_USES_KEEP_RECENT=3   # keep 3 most-recent tool_result blocks verbatim

tool_use blocks (the model's own output) are never cleared, conversation structure stays intact, only stale bodies are dropped. Per-request log line shows tokens_before → tokens_after and cleared count for observability.

Layer 2, LLM-based compactor with dynamic curator selection

Off by default; enable it with FLEET_CONTEXT_COMPACTION_ENABLED=true (config below). Once enabled, it runs after clearing on any prompt over FLEET_CONTEXT_COMPACTION_BUDGET_TOKENS (default 20K): an LLM summarizes bulky older tool results and leaves the most recent turns verbatim. If the prompt still exceeds FLEET_CONTEXT_COMPACTION_FORCE_TRIGGER_TOKENS (default 150K) after clearing, it summarizes every older tool result regardless of size. Summary work goes to whatever capable model is already hot and idle rather than cold-loading a configured default.

Ranking: hot + eligible + idle (pinned models preferred when idle, penalized when busy, quality tiebreaks by params_b); falls back to the configured default when nothing suitable is hot; fails-open (no compaction) when even the default is saturated. The configured default must be an Ollama model served by the Ollama on the router's own machine (localhost:11434); an mlx: model will not work there.

FLEET_CONTEXT_COMPACTION_ENABLED=true
FLEET_CONTEXT_COMPACTION_MODEL=gpt-oss:120b
FLEET_CONTEXT_COMPACTION_IDLE_WINDOW_S=120
FLEET_CONTEXT_COMPACTION_CURATOR_MIN_PARAMS_B=7.0

Layer 3, Pre-inference 413 cap

If the prompt is still oversized after clearing + compaction, the route returns HTTP 413 with a "run /compact and resubmit" message before the request ever reaches the model, no 5-minute MLX prefill wedge, no silent retry.

FLEET_ANTHROPIC_MAX_PROMPT_TOKENS=180000   # default

No silent server-side retry, client owns the decision of whether to resubmit, because correctness of agentic tool-use workflows depends on not altering context mid-turn.

Tool-Schema Fixup (Qwen3-Coder)

Claude Code's 27-tool schema has heavy optional-param usage, Grep alone has 13 optional params. llama.cpp#20164 documents that Qwen3-Coder starts silently dropping optional params at ~30K tokens and loops tool calls with a field consistently missing.

Ollama Herd fixes this by promoting optional params with known-safe defaults (Bash.timeout=120000, Grep.head_limit=250, Read.offset=0, etc.) to required-with-default in the outbound schema.

FLEET_ANTHROPIC_TOOL_SCHEMA_FIXUP=inject   # default
# Other modes:
# FLEET_ANTHROPIC_TOOL_SCHEMA_FIXUP=promote   # only existing defaults
# FLEET_ANTHROPIC_TOOL_SCHEMA_FIXUP=off       # pass-through

Backed by a CLAUDE_CODE_TOOL_DEFAULTS table keyed by (tool, param). Unknown tools pass through unchanged.

Tool-Call JSON Repair

Local coding models occasionally emit tool_use.input with minor syntax errors (trailing commas, unescaped quotes, missing brackets). On MLX-served models (Apple Silicon), Herd runs a repair cascade; models served by Ollama use Ollama's own tool-call parsing. The cascade:

1. Strict parse
2. json-repair library
3. 4-pattern regex catalog (adapted from nicedreamzapp/claude-code-local): parameter=key>value, <parameter_key>value, malformed "arguments" objects, single-arg tool inference for Bash/Read/Write/Glob/Grep/WebFetch/WebSearch/TodoWrite
4. Pass-through original

Schema-gated, no repair substitutes unless it passes structural validation against the tool's input_schema. Per-model repair counters exposed on /fleet/queue so operators can see if a model's repair rate is climbing (>1% sustained = signal to reconsider the model).

Token-Saving Knobs

Drop unused tool definitions

FLEET_ANTHROPIC_TOOLS_DENY strips specified Claude Code tools from every /v1/messages request before translation. Saves 200–600 prompt tokens per turn.

export FLEET_ANTHROPIC_TOOLS_DENY="WebSearch,WebFetch,NotebookEdit"

Pairs with client-side permissions.deny in .claude/settings.json, client-side only blocks execution; this removes the definitions from the wire entirely. Names matched exactly (case-sensitive).

Size-based model escalation

FLEET_ANTHROPIC_SIZE_ESCALATION_TOKENS + FLEET_ANTHROPIC_SIZE_ESCALATION_MODEL auto-route prompts over N tokens to a different (larger) model:

export FLEET_ANTHROPIC_SIZE_ESCALATION_TOKENS=50000
export FLEET_ANTHROPIC_SIZE_ESCALATION_MODEL=mlx:Qwen3-Coder-Next-4bit

Sonnet maps to qwen3-coder:30b for fast turns; escalates to Qwen3-Coder-Next (an 80B MoE) above 50K tokens. Trades small-request throughput for large-request quality where it matters.

MLX Backend (Apple Silicon)

Apple Silicon nodes can run mlx_lm.server alongside Ollama. Useful for MLX-specific models (Qwen3-Coder-Next MoE, Qwen3-Coder-30B-A3B-Instruct) and for running a dedicated compactor model side-by-side with the main coding model without Ollama eviction risk.

# on the node
export FLEET_NODE_MLX_ENABLED=true
export FLEET_NODE_MLX_SERVERS='[
  {"model":"mlx-community/Qwen3-Coder-Next-4bit","port":11440,"kv_bits":8},
  {"model":"mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit","port":11441,"kv_bits":8}
]'

# on the router
export FLEET_MLX_ENABLED=true

Each server runs in its own process with independent logs at ~/.fleet-manager/logs/mlx-server-<port>.log. Memory-pressure startup gate refuses to spawn when total (model + FLEET_NODE_MLX_MEMORY_HEADROOM_GB) won't fit. Requires the Ollama Herd MLX patch: run ./scripts/setup-mlx.sh from a clone of the GitHub repository (the script isn't included in the pip package).

Once configured, reference MLX models in your model map with the mlx: prefix:

FLEET_ANTHROPIC_MODEL_MAP='{"claude-opus-4-7": "mlx:Qwen3-Coder-Next-4bit"}'

Stability Techniques for Long Sessions

MLX wall-clock timeout

FLEET_MLX_WALL_CLOCK_TIMEOUT_S (default 300s) catches wedged-request syndrome where mlx_lm.server keeps emitting tokens slowly but never stops. On timeout, the slot is released and the route returns 413 with the /compact hint.

Default 300s is reasonable for most workloads. Long Claude Code sessions (2000+ messages) on Qwen3-Coder-Next-4bit routinely run 200–245s and need 600 to avoid edge-case 300.5s-type timeouts.

export FLEET_MLX_WALL_CLOCK_TIMEOUT_S=600

Warm-prompt preload

After mlx_lm.server passes its health check, a fire-and-forget 1-token request primes the prompt cache with the system-prompt prefix. Measured 1.3–2.25× TTFT improvement on the first real request. Non-fatal on failure. Enabled by default.

Ollama tuning for 128GB Mac Studios and below

Claude Code on qwen3-coder:30b-agent at 131K ctx on 128GB MacBooks triggered Jetsam OOM kills under real load. The combination that makes it reliable:

OLLAMA_NUM_PARALLEL=1
OLLAMA_KV_CACHE_TYPE=q8_0
OLLAMA_FLASH_ATTENTION=1
OLLAMA_KEEP_ALIVE=-1

Observed result on an M4 Max 128GB: 0% → 100% success on the big_agentic stress pattern (55 msgs, 27 tools).

OLLAMA_NUM_PARALLEL=1 is a memory-constrained exception for one long session on a laptop. On a machine with headroom, the general recommendation is 2 to 4, which lets several requests share one loaded model. Remember it multiplies the context allocation (see context length).

Configuration questions people actually ask

Which model does Claude Code use by default?

Two different defaults are in play, and mixing them up is the most common source of confusion.

• Claude Code's default is a hosted Claude model (a sonnet tier unless you pass --model or configure otherwise). That choice happens entirely in the CLI, before any request leaves your machine. Ollama Herd has no say in it.
• Herd has no fixed default. Since v0.9.0 it routes each claude-* ID to the best model you actually have loaded for that tier, falling back to the best one on disk. Whatever you pulled is what runs.

So "the default model" is really: Claude Code picks a claude-* ID, and Herd routes it to your best available local model. To force a specific one, pin it in FLEET_ANTHROPIC_MODEL_MAP or pass claude --model qwen3-coder:30b directly.

Which model names work?

Herd accepts three kinds of value in a request, resolved in this order:

1. An exact key in your map (claude-sonnet-4-5), which returns the mapped local model.
2. A real local model name, passed straight through untouched. Anything containing a colon (qwen3-coder:30b, mlx:Qwen3-Coder-Next-4bit) or not starting with claude is treated as a literal model name. This is why claude --model qwen3-coder:30b works.
3. Anything else starting with claude, which auto-routes to the best model loaded for that tier, then the best on disk.

The practical consequence: a typo'd or newly-released claude-* ID does not error, it resolves to your best available model. That is usually what you want, but if a tier seems to be ignoring a pin you set, check the exact ID string first.

How do I select a model per session with claude --model?

Use Claude Code's own flag. Herd maps whatever arrives:

claude --model claude-haiku-4-5       # -> your fast tier
claude --model claude-opus-4-7        # -> your quality tier
claude --model qwen3-coder:30b        # -> passthrough, exact local model

That last form bypasses the map entirely, which is the quickest way to test one specific model without touching config. See Per-Tier Model Routing for how to design the tiers.

How do I set max output tokens?

Another two-sided setting, and the two do not do the same job:

• CLAUDE_CODE_MAX_OUTPUT_TOKENS is Claude Code's environment variable. It controls what max_tokens the client puts in the request. This is not a Herd setting and Herd never reads it.
• FLEET_ANTHROPIC_DEFAULT_MAX_TOKENS (default 4096) is Herd's setting, and it applies only when the client omits max_tokens entirely. If Claude Code sends a value, the client wins and this is ignored.

So if responses are cut short, the value to raise is usually the client's, not Herd's. Raise FLEET_ANTHROPIC_DEFAULT_MAX_TOKENS only for clients that send no max_tokens at all. Note that reasoning models need a large budget or they never finish thinking.

What happens when every node is busy?

Herd holds the request in a per-node queue rather than failing it, and the request goes to whichever node frees up first. If the queue itself is saturated you get an explicit 503 instead of an ever-growing wait, which is deliberate: a saturated node that keeps accepting work just gets slower for everyone. You can inspect live queue depth at /fleet/queue.

This is separate from Claude Code's own client-side behavior when you type while it is working. That queueing happens in the CLI and never reaches the router.

What is the difference between Ollama Herd and claude-code-router?

claude-code-router is a proxy that rewrites Claude Code requests toward other providers, configured with a JSON file describing providers and routing rules. It is a good fit if you want to fan out to several cloud providers.

The difference is what each one knows about. claude-code-router routes by rule: you tell it which provider handles which case. Herd routes by condition: it knows which of your machines has the model already loaded, which has queue depth to spare, and which has the memory to fit it (and, on Macs with adaptive capacity turned on, which is in a video call), and it picks per request. And Herd is a fleet router, so the answer is not one backend but whichever of your machines is best right now.

If you route to cloud providers, use claude-code-router. If you route across your own Macs, Linux boxes and GPU servers, that is what Herd is for. Setup is one env var rather than a config file:

export ANTHROPIC_BASE_URL=http://localhost:11435

Does Claude Code work with local models on Linux or NVIDIA?

Yes. Herd translates Claude Code's requests to Ollama's API, so any machine that runs Ollama can serve them: a Mac, a Linux box with an NVIDIA GPU, or a Windows PC. Start herd-node on each machine and point ANTHROPIC_BASE_URL at the router. Our end-to-end Claude Code runs were on Macs, but the routing path is the same on every platform. The MLX backend is the one piece that is Apple Silicon only.

What context length should I use for Claude Code with Ollama?

At least 64K. Ollama's own default depends on GPU memory (4K under 24 GB, 32K from 24 to 48 GB, 256K above that), and Ollama recommends at least 64000 tokens for coding agents. Set OLLAMA_CONTEXT_LENGTH on every node to the largest context the fleet needs, not lower: on our reference fleet a too-low value took time to first token from about 1.0 s to about 6.3 s with decode speed unchanged. Then tell Claude Code the real window with its CLAUDE_CODE_MAX_CONTEXT_TOKENS variable so it compacts on time.

Why don't my local models show up in /model?

Claude Code's gateway model discovery is off by default, and when it is on it keeps only model IDs that contain claude or anthropic. You do not need it with Herd: pick a Claude tier and Herd routes it to your best loaded model, or pass a local name directly with claude --model qwen3-coder:30b, which Herd passes through unchanged.

Does Claude Code work offline with local models?

Inference does. With ANTHROPIC_BASE_URL pointed at Herd, every model request goes to your own machines. Claude Code itself still makes a few calls of its own to Anthropic, such as its WebFetch domain safety check, so a fully air-gapped setup also needs Claude Code's own settings for turning off traffic outside the gateway path.

Does Anthropic support running Claude Code on non-Claude models?

No. Anthropic's gateway documentation says it does not support routing Claude Code to non-Claude models through any gateway. It works because Herd implements the Anthropic Messages API that Claude Code already speaks, and we verify it against real Claude Code releases: Claude Code 2.1.68 completed a multi-file coding task end to end on a local fleet.

Troubleshooting

"Claude Code is truncating responses mid-stream"

Qwen3-Coder models can emit <|im_start|> or <|endoftext|> at ~30K tokens when attention to role separators weakens. Ollama Herd adds these to the MLX stop[] list and defensively strips them from any text that leaks through before the stop fires. If you still see literal <|im_start|> in your output, your OSS version is out of date, upgrade to v0.6.0+.

"The model keeps calling tools with missing params"

This is the llama.cpp#20164 bug. Confirm FLEET_ANTHROPIC_TOOL_SCHEMA_FIXUP=inject is active. Check /fleet/queue for tool_repair counters. If repair rate exceeds 1% sustained, consider escalating to a larger model or using a different family (e.g. gpt-oss:120b for haiku tier).

"HTTP 413 responses mid-session"

Expected behavior when a prompt exceeds FLEET_ANTHROPIC_MAX_PROMPT_TOKENS after clearing + compaction. Claude Code users should run /compact to trim history, then resubmit. This protects against multi-minute MLX prefill wedges.

"My fleet dashboard shows tool_repair counters climbing"

Signal that the model is struggling with structured output. Options: (1) switch to a larger model for this tier, (2) reduce tool count via FLEET_ANTHROPIC_TOOLS_DENY, (3) enable size escalation to route long prompts to a heavier model.

"Claude Code was working, then started returning text instead of tool calls"

Usually the model you expected is no longer loaded and a weaker fallback served the request. Check for the fallback header, then pre-warm the model you want (run the second command against the node's Ollama):

curl -sS -D - -o /dev/null http://localhost:11435/v1/messages \
  -H "Content-Type: application/json" -H "anthropic-version: 2023-06-01" \
  -d '{"model":"claude-sonnet-4-5","max_tokens":10,"messages":[{"role":"user","content":"hi"}]}' \
  | grep -i x-fleet-fallback

curl http://localhost:11434/api/generate \
  -d '{"model":"qwen3-coder:30b","prompt":"hi","keep_alive":-1,"stream":false}'

"The first request after a restart is very slow, or aborts"

That is a cold load: the model has to be read from disk into memory first, and a large one can take minutes. Pre-warm it with the /api/generate call above (or ollama run MODEL once) before starting a session. Claude Code gives up on a stream that stays silent for 300 seconds, so do not let the first real request pay for loading a very large model.

One thing to know about logs

FLEET_LOG_LEVEL defaults to DEBUG, and the Anthropic route writes a truncated request-body preview per request. That means the head of your system prompt and your messages lands in ~/.fleet-manager/logs/herd.jsonl.

It stays on your machine and it is bounded (around 1.3% of log volume in a real session). But if you are running Claude Code locally because prompts should not go anywhere, you should know the file exists. FLEET_LOG_LEVEL=INFO turns it off.

Prompt caching, honestly

Ollama 0.32.1 has working prefix caching, so a long session is no longer a full re-encode every turn. What you do not get is Anthropic's cache_control, so there are no explicit cache breakpoints. Keep models resident with OLLAMA_KEEP_ALIVE=-1 so the prefix cache survives between turns.

What Ollama Herd does not implement

• Anthropic Compaction API (anthropic-beta: compact-2026-01-12 + context_management.edits body field), Ant-only beta, external Claude Code users don't send it.
• Microcompact (cache_edits content blocks, cache-editing-20250919), also Ant-only today. We log first-occurrence of any unknown block type so if microcompact ever starts firing we notice without spam.

Full three-mechanism analysis in the research doc: why-claude-code-degrades-at-30k.md.

Related Reading

• Quickstart, get herd + herd-node running
• Deployment, running nodes on Linux, in containers, and against a remote Ollama
• Routing Engine, 8-signal scoring, including device-aware bandwidth routing
• API Reference, full endpoint list including /v1/messages
• Codex CLI and Desktop, the same fleet for OpenAI's agent CLI
• Integrations, other clients (OpenWebUI, LangChain, LiteLLM, etc.)
• Feature summary, the full Claude Code reliability layer on one page