Claude Code CLI Integration

TL;DR

If you already have a herd running, three env vars:

export ANTHROPIC_BASE_URL=http://localhost:11435
export ANTHROPIC_AUTH_TOKEN=dummy   # any non-empty value
claude

That's it. Claude Code now talks to your local model with full tool use, streaming, and the standard agentic loop, routed through Ollama Herd's scoring engine, queue pipeline, and context management layers.

Want the full setup from scratch? Start with the Quickstart to get herd + herd-node running, then come back here for Claude Code specifics.

Ollama already does the single-Mac case. When do you need Herd?

Ollama speaks the Anthropic Messages API natively, so ollama launch claude can point Claude Code at a single Ollama instance on your machine. If that is your whole setup, use it: it is the simplest path and it works well.

Herd is for the cases the single-node path does not cover:

Short version: one Mac and short sessions, use Ollama's built-in path. A fleet, or long tool-heavy sessions, use Herd. Other tools run Claude Code on a single local machine, or route it out to cloud providers. Herd is the one that routes across the Macs you already own, picking the machine that is idle, not thermally throttling, and already has your model loaded. The same endpoint also serves Codex, so both agent CLIs share one fleet. The rest of this guide covers the Herd specifics.

Local models break agentic loops. Herd keeps them alive.

The hard part of running Claude Code on a local model is not raw speed, it is reliability. Local models drop out of JSON mid tool call, omit required parameters, and degrade at long context in ways that quietly break the agentic loop. A demo works; a 2,000 message session falls apart. Herd ships the machinery that keeps the loop running:

• Tool-call repair recovers malformed tool calls, dropped JSON, missing parameters, so a bad call is fixed instead of failing the turn.
• Tool-schema fixup injects the optional-parameter defaults that Qwen3-Coder starts silently omitting at around 30K tokens (root cause traced to llama.cpp#20164).
• Three-layer context management keeps long sessions under the model's real limits instead of wedging on a multi-minute prefill.

This is the difference between a coding agent that works in a screenshot and one that survives a real day of use. Each layer is detailed below.

Verified end to end. Claude Code 2.1.68 was driven headless against the fleet on a multi-file task: a repo whose test suite would not collect, needing a module written from scratch, a bug fixed, and a guard added. It reached 6 passed on its own across 14 turns, 12 of them tool-calling, exercising Bash, Edit, Glob, Read, Write, and TodoWrite. No translation defects surfaced.

What Ollama Herd does for Claude Code

Ollama Herd exposes a native Anthropic Messages API (/v1/messages and /v1/messages/count_tokens), translates requests to Ollama's wire format, runs them through the same scoring + queue + trace pipeline as every other route, and translates the response back to Anthropic SSE event sequences.

No LiteLLM sidecar. No OpenAI-format proxy. Claude Code's ANTHROPIC_BASE_URL points straight at the herd router.

Setup

Step 1: Pull a coding model

Claude Code makes many tool calls, so you want a coding-tuned model that follows a schema without getting derailed. One model per tier:

# Main coding loop (sonnet tier)
ollama pull qwen3-coder:30b

# Heavier tier for hard problems (opus tier)
ollama pull qwen3:32b

# Fast tier for cheap turns (haiku tier)
ollama pull qwen3:14b

These are the models behind the defaults below. Newer families are available too (Qwen 3.6, Gemma 4, GLM-5, Kimi K2 Code, DeepSeek V4), so substitute freely. The tier shape is what matters, not these exact names.

Step 2: Point Claude Code at your router

export ANTHROPIC_BASE_URL=http://localhost:11435
export ANTHROPIC_AUTH_TOKEN=dummy   # any non-empty string
claude

Step 3: (Optional) Pin a model

You can skip this. Claude Code sends model IDs like claude-sonnet-4-5, and as of v0.9.0 Herd resolves them itself. There is no default map to configure and nothing to keep in sync with what you have pulled.

Resolution order, first match wins:

1. An explicit FLEET_ANTHROPIC_MODEL_MAP entry for that exact ID.
2. A real local model name, passed through unchanged. This is why claude --model qwen3-coder:30b works.
3. The best model already loaded on the fleet for that tier. No cold load, it is already resident.
4. The best on-disk model for that tier, accepting a cold load.

This matters because model IDs change. When Anthropic ships a new version string, auto-routing picks it up; a hand written map would have silently sent it to the wrong tier.

To pin a specific ID anyway:

export FLEET_ANTHROPIC_MODEL_MAP='{
  "claude-opus-4-7": "mlx:Qwen3-Coder-Next-4bit",
  "claude-haiku-4-5": "gpt-oss:120b"
}'

Entries you set override auto-routing; unmapped IDs still auto-route. Values can reference Ollama models (qwen3-coder:30b) or MLX models (mlx:Qwen3-Coder-Next-4bit), see MLX Backend below. To disable auto-routing and require an explicit map, set FLEET_ANTHROPIC_AUTO_ROUTE=false.

Per-Tier Model Routing

Claude Code's --model flag lets users trade speed for quality per-invocation:

claude --model claude-haiku-4-5      # fast turns, cheaper model
claude --model claude-sonnet-4-5     # default balance
claude --model claude-opus-4-7       # highest quality

A good production mapping splits speed from quality and diversifies failure modes:

• claude-haiku-* → a smaller hot Ollama model (fast, already pinned)
• claude-sonnet-* → a solid coding model for the main loop
• claude-opus-* → an 80B-class MoE via MLX for the tough cases

Different model families on different tiers also means if Qwen3 has a bad day, your haiku fallback still works.

The /compact command works out of the box

Claude Code's /compact slash command is client-side orchestration over the standard /v1/messages endpoint, it sends a normal request with a trailing user message asking the model to summarise the conversation, then locally replaces the in-memory history with the response. No special beta header, endpoint, or body field.

That means:

• /compact works against Ollama Herd with no special support required. Same as against hosted Claude.
• We augment it with hosted-Claude-parity context management layers that run before the model sees the request (see next section).

In practice, a 2,700-message session that would have timed out or produced garbage on raw local inference can hit /compact successfully on our fleet, Layer 1 alone typically shrinks the prompt by 60%+ before it hits the model.

Three-Layer Context Management

The biggest structural difference between hosted Claude and raw local inference is context hygiene. Hosted Claude silently strips stale tool results, summarises long sessions, and refuses oversized requests. Raw local Ollama doesn't. Ollama Herd ships three layers that close that gap.

Layer 1, Mechanical tool-result clearing

When the Anthropic request exceeds FLEET_ANTHROPIC_AUTO_CLEAR_TOOL_USES_TRIGGER_TOKENS (default 100K), older tool_result blocks are replaced with a short placeholder before the request reaches the model. No LLM call, microsecond-scale. Matches hosted Claude's Context Editing API.

Configurable:

FLEET_ANTHROPIC_AUTO_CLEAR_TOOL_USES_TRIGGER_TOKENS=100000
FLEET_ANTHROPIC_AUTO_CLEAR_TOOL_USES_KEEP_RECENT=3   # keep 3 most-recent tool_result blocks verbatim

tool_use blocks (the model's own output) are never cleared, conversation structure stays intact, only stale bodies are dropped. Per-request log line shows tokens_before → tokens_after and cleared count for observability.

Layer 2, LLM-based compactor with dynamic curator selection

After clearing, if the prompt still exceeds FLEET_CONTEXT_COMPACTION_FORCE_TRIGGER_TOKENS (default 150K), an LLM-based summarizer runs on remaining content. Summary work goes to whatever capable model is already hot and idle rather than cold-loading a configured default.

Ranking: hot + eligible + idle (pinned models preferred when idle, penalized when busy, quality tiebreaks by params_b); falls back to the configured default when nothing suitable is hot; fails-open (no compaction) when even the default is saturated.

FLEET_CONTEXT_COMPACTION_ENABLED=true
FLEET_CONTEXT_COMPACTION_MODEL=mlx:mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit
FLEET_CONTEXT_COMPACTION_IDLE_WINDOW_S=120
FLEET_CONTEXT_COMPACTION_CURATOR_MIN_PARAMS_B=7.0

Layer 3, Pre-inference 413 cap

If the prompt is still oversized after clearing + compaction, the route returns HTTP 413 with a "run /compact and resubmit" message before the request ever reaches the model, no 5-minute MLX prefill wedge, no silent retry.

FLEET_ANTHROPIC_MAX_PROMPT_TOKENS=180000   # default

No silent server-side retry, client owns the decision of whether to resubmit, because correctness of agentic tool-use workflows depends on not altering context mid-turn.

Tool-Schema Fixup (Qwen3-Coder)

Claude Code's 27-tool schema has heavy optional-param usage, Grep alone has 13 optional params. llama.cpp#20164 documents that Qwen3-Coder starts silently dropping optional params at ~30K tokens and loops tool calls with a field consistently missing.

Ollama Herd fixes this by promoting optional params with known-safe defaults (Bash.timeout=120000, Grep.head_limit=250, Read.offset=0, etc.) to required-with-default in the outbound schema.

FLEET_ANTHROPIC_TOOL_SCHEMA_FIXUP=inject   # default
# Other modes:
# FLEET_ANTHROPIC_TOOL_SCHEMA_FIXUP=promote   # only existing defaults
# FLEET_ANTHROPIC_TOOL_SCHEMA_FIXUP=off       # pass-through

Backed by a CLAUDE_CODE_TOOL_DEFAULTS table keyed by (tool, param). Unknown tools pass through unchanged.

Tool-Call JSON Repair

Local coding models occasionally emit tool_use.input with minor syntax errors (trailing commas, unescaped quotes, missing brackets). The repair cascade:

1. Strict parse
2. json-repair library
3. 4-pattern regex catalog (adapted from nicedreamzapp/claude-code-local): parameter=key>value, <parameter_key>value, malformed "arguments" objects, single-arg tool inference for Bash/Read/Write/Glob/Grep/WebFetch/WebSearch/TodoWrite
4. Pass-through original

Schema-gated, no repair substitutes unless it passes structural validation against the tool's input_schema. Per-model repair counters exposed on /fleet/queue so operators can see if a model's repair rate is climbing (>1% sustained = signal to reconsider the model).

Token-Saving Knobs

Drop unused tool definitions

FLEET_ANTHROPIC_TOOLS_DENY strips specified Claude Code tools from every /v1/messages request before translation. Saves 200–600 prompt tokens per turn.

export FLEET_ANTHROPIC_TOOLS_DENY="WebSearch,WebFetch,NotebookEdit"

Pairs with client-side permissions.deny in .claude/settings.json, client-side only blocks execution; this removes the definitions from the wire entirely. Names matched exactly (case-sensitive).

Size-based model escalation

FLEET_ANTHROPIC_SIZE_ESCALATION_TOKENS + FLEET_ANTHROPIC_SIZE_ESCALATION_MODEL auto-route prompts over N tokens to a different (larger) model:

export FLEET_ANTHROPIC_SIZE_ESCALATION_TOKENS=50000
export FLEET_ANTHROPIC_SIZE_ESCALATION_MODEL=mlx:Qwen3-Coder-Next-4bit

Sonnet maps to qwen3-coder:30b for fast turns; escalates to the 480B MoE above 50K tokens. Trades small-request throughput for large-request quality where it matters.

MLX Backend

Apple Silicon nodes can run mlx_lm.server alongside Ollama. Useful for MLX-specific models (Qwen3-Coder-Next MoE, Qwen3-Coder-30B-A3B-Instruct) and for running a dedicated compactor model side-by-side with the main coding model without Ollama eviction risk.

export FLEET_NODE_MLX_SERVERS='[
  {"model":"mlx-community/Qwen3-Coder-Next-4bit","port":11440,"kv_bits":8},
  {"model":"mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit","port":11441,"kv_bits":8}
]'

Each server runs in its own process with independent logs at ~/.fleet-manager/logs/mlx-server-<port>.log. Memory-pressure startup gate refuses to spawn when total (model + FLEET_NODE_MLX_MEMORY_HEADROOM_GB) won't fit. Requires the Ollama Herd MLX patch, run scripts/setup-mlx.sh.

Once configured, reference MLX models in your model map with the mlx: prefix:

FLEET_ANTHROPIC_MODEL_MAP='{"claude-opus-4-7": "mlx:Qwen3-Coder-Next-4bit"}'

Stability Techniques for Long Sessions

MLX wall-clock timeout

FLEET_MLX_WALL_CLOCK_TIMEOUT_S (default 300s) catches wedged-request syndrome where mlx_lm.server keeps emitting tokens slowly but never stops. On timeout, the slot is released and the route returns 413 with the /compact hint.

Default 300s is reasonable for most workloads. Long Claude Code sessions (2000+ messages) on Qwen3-Coder-Next-4bit routinely run 200–245s and need 600 to avoid edge-case 300.5s-type timeouts.

export FLEET_MLX_WALL_CLOCK_TIMEOUT_S=600

Warm-prompt preload

After mlx_lm.server passes its health check, a fire-and-forget 1-token request primes the prompt cache with the system-prompt prefix. Measured 1.3–2.25× TTFT improvement on the first real request. Non-fatal on failure. Enabled by default.

Ollama tuning for 128GB Mac Studios and below

Claude Code on qwen3-coder:30b-agent at 131K ctx on 128GB MacBooks triggered Jetsam OOM kills under real load. The combination that makes it reliable:

OLLAMA_NUM_PARALLEL=1
OLLAMA_KV_CACHE_TYPE=q8_0
OLLAMA_FLASH_ATTENTION=1
OLLAMA_KEEP_ALIVE=-1

Observed result on an M4 Max 128GB: 0% → 100% success on the big_agentic stress pattern (55 msgs, 27 tools).

Configuration questions people actually ask

Which model does Claude Code use by default?

Two different defaults are in play, and mixing them up is the most common source of confusion.

• Claude Code's default is a hosted Claude model (a sonnet tier unless you pass --model or configure otherwise). That choice happens entirely in the CLI, before any request leaves your machine. Ollama Herd has no say in it.
• Herd has no fixed default. Since v0.9.0 it routes each claude-* ID to the best model you actually have loaded for that tier, falling back to the best one on disk. Whatever you pulled is what runs.

So "the default model" is really: Claude Code picks a claude-* ID, and Herd routes it to your best available local model. To force a specific one, pin it in FLEET_ANTHROPIC_MODEL_MAP or pass claude --model qwen3-coder:30b directly.

Which model names work?

Herd accepts three kinds of value in a request, resolved in this order:

1. An exact key in your map (claude-sonnet-4-5), which returns the mapped local model.
2. A real local model name, passed straight through untouched. Anything containing a colon (qwen3-coder:30b, mlx:Qwen3-Coder-Next-4bit) or not starting with claude is treated as a literal model name. This is why claude --model qwen3-coder:30b works.
3. Anything else starting with claude, which auto-routes to the best model loaded for that tier, then the best on disk.

The practical consequence: a typo'd or newly-released claude-* ID does not error, it resolves to your best available model. That is usually what you want, but if a tier seems to be ignoring a pin you set, check the exact ID string first.

How do I select a model per session?

Use Claude Code's own flag. Herd maps whatever arrives:

claude --model claude-haiku-4-5       # -> your fast tier
claude --model claude-opus-4-7        # -> your quality tier
claude --model qwen3-coder:30b        # -> passthrough, exact local model

That last form bypasses the map entirely, which is the quickest way to test one specific model without touching config. See Per-Tier Model Routing for how to design the tiers.

How do I set max output tokens?

Another two-sided setting, and the two do not do the same job:

• CLAUDE_CODE_MAX_OUTPUT_TOKENS is Claude Code's environment variable. It controls what max_tokens the client puts in the request. This is not a Herd setting and Herd never reads it.
• FLEET_ANTHROPIC_DEFAULT_MAX_TOKENS (default 4096) is Herd's setting, and it applies only when the client omits max_tokens entirely. If Claude Code sends a value, the client wins and this is ignored.

So if responses are cut short, the value to raise is usually the client's, not Herd's. Raise FLEET_ANTHROPIC_DEFAULT_MAX_TOKENS only for clients that send no max_tokens at all. Note that reasoning models need a large budget or they never finish thinking.

What happens when every node is busy?

Herd holds the request in a per-node queue rather than failing it, and the request goes to whichever node frees up first. If the queue itself is saturated you get an explicit 503 instead of an ever-growing wait, which is deliberate: a saturated node that keeps accepting work just gets slower for everyone. You can inspect live queue depth at /fleet/queue.

This is separate from Claude Code's own client-side behavior when you type while it is working. That queueing happens in the CLI and never reaches the router.

What is the difference between Ollama Herd and claude-code-router?

claude-code-router is a proxy that rewrites Claude Code requests toward other providers, configured with a JSON file describing providers and routing rules. It is a good fit if you want to fan out to several cloud providers.

The difference is what each one knows about. claude-code-router routes by rule: you tell it which provider handles which case. Herd routes by hardware condition: it knows which of your Macs has the model already resident, which is thermally throttling, which is in a video call, and which has queue depth to spare, and it picks per request. And Herd is a fleet router, so the answer is not one backend but whichever of your machines is best right now.

If you route to cloud providers, use claude-code-router. If you route across your own Macs, that is what Herd is for. Setup is one env var rather than a config file:

export ANTHROPIC_BASE_URL=http://localhost:11435

Troubleshooting

"Claude Code is truncating responses mid-stream"

Qwen3-Coder models can emit <|im_start|> or <|endoftext|> at ~30K tokens when attention to role separators weakens. Ollama Herd adds these to the MLX stop[] list and defensively strips them from any text that leaks through before the stop fires. If you still see literal <|im_start|> in your output, your OSS version is out of date, upgrade to v0.6.0+.

"The model keeps calling tools with missing params"

This is the llama.cpp#20164 bug. Confirm FLEET_ANTHROPIC_TOOL_SCHEMA_FIXUP=inject is active. Check /fleet/queue for tool_repair counters. If repair rate exceeds 1% sustained, consider escalating to a larger model or using a different family (e.g. gpt-oss:120b for haiku tier).

"HTTP 413 responses mid-session"

Expected behavior when a prompt exceeds FLEET_ANTHROPIC_MAX_PROMPT_TOKENS after clearing + compaction. Claude Code users should run /compact to trim history, then resubmit. This protects against multi-minute MLX prefill wedges.

"My fleet dashboard shows tool_repair counters climbing"

Signal that the model is struggling with structured output. Options: (1) switch to a larger model for this tier, (2) reduce tool count via FLEET_ANTHROPIC_TOOLS_DENY, (3) enable size escalation to route long prompts to a heavier model.

One thing to know about logs

FLEET_LOG_LEVEL defaults to DEBUG, and the Anthropic route writes a truncated request-body preview per request. That means the head of your system prompt and your messages lands in ~/.fleet-manager/logs/herd.jsonl.

It stays on your machine and it is bounded (around 1.3% of log volume in a real session). But if you are running Claude Code locally because prompts should not go anywhere, you should know the file exists. FLEET_LOG_LEVEL=INFO turns it off.

Prompt caching, honestly

Ollama 0.32.1 has working prefix caching, so a long session is no longer a full re-encode every turn. What you do not get is Anthropic's cache_control, so there are no explicit cache breakpoints. Keep models resident with OLLAMA_KEEP_ALIVE=-1 so the prefix cache survives between turns.

What Ollama Herd does not implement

• Anthropic Compaction API (anthropic-beta: compact-2026-01-12 + context_management.edits body field), Ant-only beta, external Claude Code users don't send it.
• Microcompact (cache_edits content blocks, cache-editing-20250919), also Ant-only today. We log first-occurrence of any unknown block type so if microcompact ever starts firing we notice without spam.

Full three-mechanism analysis in the research doc: why-claude-code-degrades-at-30k.md.

Related Reading

• Quickstart, get herd + herd-node running
• Routing Engine, 8-signal scoring, including device-aware bandwidth routing
• API Reference, full endpoint list including /v1/messages
• Codex CLI and Desktop, the same fleet for OpenAI's agent CLI
• Integrations, other clients (OpenWebUI, LangChain, LiteLLM, etc.)
• Feature summary, the full Claude Code reliability layer on one page