Why does Claude Code break around 30K tokens on local models?

It is usually not a generic long-context failure. It is a documented Qwen3-Coder tool-calling bug (llama.cpp issue #20164): once context passes roughly 20 percent of the model's window, the parser starts dropping optional tool parameters, then loops or abandons the tool. Claude Code's ~27 tools have many optional parameters (Grep alone has 13), so it triggers the bug early and often. The fix is to reshape tool schemas so optional parameters become required-with-default, which Ollama Herd does automatically.

Why does my local model return an empty or very short response?

Most often it is a thinking model that reasoned internally and ran out of its token budget before emitting a visible answer. Models like Muse Glimmer and gpt-oss reason before they respond, so a tight max_tokens comes back empty. Raise the token budget. Ollama Herd auto-inflates the budget for models it knows are thinking models so this does not happen.

Why does my model crash with a Metal out-of-memory error?

Either the model plus its KV cache exceeds your unified memory, or you are hitting an immature runtime for a very new model. Check the sizing math first. If memory is fine, the model's architecture may not be properly supported yet: some 2026 models have loaders that leak Metal memory and crash mid-generation. Verify the model actually runs before relying on it.

Why is the first response from a local model so slow?

The model is loading into memory (a cold start), which can take tens of seconds for a large model, even though every response after is fast. Keep the model resident instead of letting it unload between requests. On a fleet, Ollama Herd keeps models warm and uses session affinity to route a conversation back to the node that already has its context cached.

Fixing Common Local LLM Errors on Apple Silicon

Local models on Apple Silicon fail in a handful of predictable ways, and the obvious diagnosis is usually wrong. "It slows down at long context" is rarely a memory problem. "It returned nothing" is rarely a broken model. Here are the failures that actually happen, the real cause of each, and the fix, grounded in request traces from a fleet that runs these models daily.

Claude Code loops or drops tool calls around 30K tokens

Symptom: short sessions work fine, then past ~30K tokens the model starts calling the same tool repeatedly with a parameter silently missing each time, loops without converging, or drifts off task. Memory is not the bottleneck: it happens with hundreds of GB free and no thermal throttling.

Real cause: this is a specific Qwen3-Coder tool-calling bug, documented in llama.cpp issue #20164, not a generic long-context collapse. Once context passes about 20 percent of the model's window, grammar-constrained decoding starts dropping optional tool parameters. The reporter's smoking gun: making one optional parameter required eliminated the problem entirely. Claude Code hits this hard because its ~27 tools are full of optional parameters, Grep alone has 13, so every tool call past the threshold is rolling a die.

Fix: reshape the tool schema so optional parameters with sane defaults become required-with-default before the local model sees them. This turns "many holes the parser can fall into" into "fewer required fields, pre-filled." Ollama Herd does this automatically and pairs it with server-side context management, which together push reliable Claude Code sessions well past 60K tokens on the same weights.

The model "forgets" earlier context long before its advertised limit

Symptom: a model advertised at 128K or 1M tokens starts losing the thread at a fraction of that.

Real cause: advertised context is 2 to 4 times larger than effective context for almost every open-weight model. NVIDIA's RULER benchmark, the standard for real long-context capability, found Llama-3.1-8B's 128K claim holds to about 32K in practice, and one model's 1M-token claim degraded to a 64K effective window. The number on the model card is a structural capability (it can ingest that many tokens without crashing), not a promise it can reason over them.

Fix: size for the context you will really use, and lean on context management to keep the working set inside the effective window. Every frontier lab does aggressive server-side context management on hosted models; doing it locally is a correction, not a workaround. See the memory guide for why over-allocating KV cache for a context you cannot use is wasted RAM.

The model returns an empty or truncated response

Symptom: you get back nothing, or a response that cuts off before it says anything useful.

Real cause: it is usually a thinking model that reasoned internally and exhausted its token budget before emitting the visible answer. Models like Muse Glimmer and gpt-oss produce reasoning tokens first; a tight max_tokens gets consumed by the thinking and never reaches the reply.

Fix: raise the token budget for thinking models. Ollama Herd auto-inflates the budget for models it recognizes as thinking models, so a conservative client setting does not starve the answer. See the tested-models matrix for which models are thinking models.

Metal out-of-memory, or the model will not load at all

Symptom: the model fails to load, or crashes partway through a long generation with a Metal memory error.

Real cause: two possibilities. Either the weights plus KV cache genuinely exceed your unified memory (do the sizing math first), or you are running a very new model whose runtime support is immature. Runtime maturity, not hardware, gates the newest models: some 2026 releases have loaders that leak Metal memory and hard-crash mid-generation even on a 512 GB Mac. DeepSeek-V4-Flash is a current example, its loader is not yet in a maintained mlx-lm release.

Fix: if memory is the issue, drop to a smaller quant or a smaller-active-param MoE, or spread models across more machines. If it is runtime maturity, do not fight it; pick a model that is verified to actually run, and wait for the loader to land upstream.

The first response is slow, then everything is fast

Symptom: the first request after a pause takes tens of seconds; subsequent ones are quick.

Real cause: a cold start. The model is being loaded into memory, which is slow for large weights, and then it is unloaded again when idle, so the next request pays the cost all over.

Fix: keep the model resident rather than letting it unload between requests. On a single machine that means pinning it hot. On a fleet, Ollama Herd keeps models warm across nodes and uses session affinity to route a conversation back to the node that already holds its cached context, so you pay the load cost once, not every turn.