Ollama Parallel Requests and OLLAMA_NUM_PARALLEL
How to make one Ollama server answer several requests at once, what it costs in memory, why some models ignore the setting, and when to add a second machine instead.
Ollama runs one request at a time per model by default. To process several at once, set OLLAMA_NUM_PARALLEL on the Ollama server (default 1 since Ollama 0.10.0, July 2025) and restart it. Requests beyond the parallel slots wait in a queue of up to OLLAMA_MAX_QUEUE requests (default 512); once that is full, Ollama answers 503.
Checked October 26, 2026 against Ollama 0.40.11 (October 24, 2026): its FAQ, envconfig/config.go, and server/sched.go. Ollama ships several releases a week; recheck a default before relying on it.
The short answer
- Where to set it: on the server process, not in your client. A terminal
exportdoes not reach the macOS app or a systemd service; use the commands below. - What it costs: every slot gets its own full context, so context memory is context length times slots.
qwen3-coder:30bat 32K context measured 31.0 GB with 4 slots, against about 21.8 GB with one. - What it buys: more total throughput, not N times as much. On one busy M3 Ultra,
gemma3:27bdecoded 22.5 tokens/s for a single request and 27.1 tokens/s in total with four slots busy (1.2x). - When it does nothing: embedding models, models Ollama runs on its MLX engine (the Apple Silicon default since 0.40), and a few architectures always run one request at a time. See models that ignore the setting.
How to set OLLAMA_NUM_PARALLEL
macOS (Ollama app): set it for launchd, then quit Ollama from the menu bar and open it again. A launchctl setenv value does not survive a reboot.
launchctl setenv OLLAMA_NUM_PARALLEL 2
Linux (systemd service): add an override, then reload and restart.
sudo systemctl edit ollama.service
# in the editor, under [Service], add:
# Environment="OLLAMA_NUM_PARALLEL=2"
sudo systemctl daemon-reload
sudo systemctl restart ollama
Windows: quit Ollama from the system tray, set a user environment variable (in Settings, search for "environment variables", or run the command below), then start Ollama from the Start menu.
setx OLLAMA_NUM_PARALLEL 2
Docker: pass it to the container.
docker run -d -e OLLAMA_NUM_PARALLEL=2 -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
Check that it took: Ollama logs its settings at startup in a server config line that includes OLLAMA_NUM_PARALLEL.
grep "server config" ~/.ollama/logs/server.log | tail -1 # macOS
journalctl -u ollama --no-pager | grep "server config" | tail -1 # Linux
docker logs ollama 2>&1 | grep "server config" | tail -1 # Docker
On Windows the log is server.log in %LOCALAPPDATA%\Ollama. Set OLLAMA_MAX_QUEUE, OLLAMA_MAX_LOADED_MODELS, and OLLAMA_CONTEXT_LENGTH the same way.
More parallelism is not automatically faster. Every parallel sequence needs its own context state, so memory use grows with both parallel request count and context length. The right method is to benchmark one node at a time, increase parallelism gradually, and add fleet-level routing when one machine is no longer the right bottleneck.
The three layers of concurrency
Application concurrency
many users, agents, workflows, or batch jobs
|
v
Fleet concurrency
Ollama Herd routes independent requests across machines
|
v
Node concurrency
one Ollama server loads models and runs parallel sequences
A slow application can be blocked at any layer. Increasing a limit at the wrong layer can make performance worse.
How Ollama handles concurrent requests
Ollama's current documentation describes three important behaviors:
- Multiple models can stay loaded when the available system memory or GPU memory is sufficient.
- A loaded model can process more than one request in parallel when enough memory was available at load time.
- Requests queue when a required model or parallel slot cannot be admitted. Once the queue is full, Ollama returns an overloaded
503response.
For a single model, Ollama states that required memory scales with:
OLLAMA_NUM_PARALLEL × context length
That is why four short requests may fit while four 64K agent sessions do not.
The three main settings
| Setting | What it controls | Current documented default | Main risk when increased |
|---|---|---|---|
OLLAMA_NUM_PARALLEL |
Parallel requests per loaded model | 1 (since 0.10.0) |
Context/KV-cache memory multiplication |
OLLAMA_MAX_LOADED_MODELS |
Number of models that may remain loaded when they fit | 3 × GPU count, or 3 for CPU inference |
Model weights consume memory and trigger eviction or offload |
OLLAMA_MAX_QUEUE |
Waiting requests before new work is rejected | 512 |
Very long waits and hidden overload |
Defaults can change. Confirm them against the Ollama version you deploy.
Two related controls matter:
OLLAMA_KEEP_ALIVEdecides how long idle models stay loaded. Ollama's documented default is five minutes.OLLAMA_CONTEXT_LENGTHsets the server's context allocation and has a direct memory cost.
Models that ignore OLLAMA_NUM_PARALLEL
If you raised the setting, confirmed it in the log, and requests for a model still run one after another, the model is one that Ollama serves serially whatever the setting says. In Ollama 0.40.11:
| Case | What Ollama does | How to tell |
|---|---|---|
| Models on the MLX engine (Apple Silicon) | The MLX runner processes one request at a time. Since 0.40, Macs run every architecture MLX supports on MLX automatically | ollama ps shows mlx in the RUNNER column |
| Embedding models | Always loaded with one slot | The model only produces embeddings |
Architectures qwen35, qwen35moe, qwen3next, qwen3vl, qwen3vlmoe, mllama, lfm2, lfm2moe, nemotron_h, nemotron_h_moe, nemotron_h_omni |
Forced to one slot on every platform | The server log warns "model architecture does not currently support parallel requests" |
Parallel requests on the MLX engine are being worked on upstream (ollama#17280, with open pull requests #17317 and #19197) but are not in a release. On a Mac today you can run the model's llama.cpp variant where it has one (ollama run --runner llamacpp MODEL, or "runner": "llamacpp" in an /api/chat request), use an MLX server with continuous batching (see oMLX vs Ollama), or spread requests over a second machine.
Start with a baseline, not a copied “best” value
A safe tuning cycle is:
- Choose one model and one representative prompt.
- Run one request at a time.
- Record latency, tokens per second, memory, and whether the model is fully accelerated.
- Test concurrency
2with the same context and prompts. - Test
4only when memory remains comfortable. - Compare aggregate throughput, p50 latency, p95 latency, and errors.
- Repeat for long-context and short-context workloads separately.
Do not benchmark only a one-sentence prompt. Agent, coding, and RAG requests spend much more time in prompt evaluation and can allocate far more context.
Example configuration profiles
These are starting points for measurement, not universal recommendations.
Stability-first: one large model
export OLLAMA_NUM_PARALLEL=1
export OLLAMA_MAX_LOADED_MODELS=1
export OLLAMA_MAX_QUEUE=64
ollama serve
Use when one model consumes most available memory or long contexts dominate.
Balanced interactive use
export OLLAMA_NUM_PARALLEL=2
export OLLAMA_MAX_LOADED_MODELS=2
export OLLAMA_MAX_QUEUE=64
ollama serve
Use only when both loaded-model weights and two context slots fit with headroom.
Throughput experiment
export OLLAMA_NUM_PARALLEL=4
export OLLAMA_MAX_LOADED_MODELS=1
export OLLAMA_MAX_QUEUE=128
ollama serve
This isolates same-model parallelism. It is useful for a test, not a production default. Four long contexts can consume far more memory than expected.
These export lines are for a foreground ollama serve test. For the macOS app, a systemd service, Windows, or Docker, set the same variables with the platform commands above.
Verify what is actually loaded
Use:
ollama ps
Inspect:
- model size;
- processor placement;
- allocated context;
- how long the model will stay loaded.
A model partially offloaded to CPU may remain functional but behave very differently under concurrent load. Record this state with benchmark results.
A small reproducible load test
Install httpx:
python3 -m pip install httpx
Save as ollama_load_test.py:
from __future__ import annotations
import argparse
import asyncio
import math
import statistics
import time
from dataclasses import dataclass
import httpx
@dataclass
class Result:
seconds: float
status: int
error: str | None = None
async def run_one(
client: httpx.AsyncClient,
url: str,
model: str,
prompt: str,
) -> Result:
started = time.perf_counter()
try:
response = await client.post(
url,
json={
"model": model,
"messages": [{"role": "user", "content": prompt}],
"stream": False,
},
)
elapsed = time.perf_counter() - started
response.raise_for_status()
return Result(seconds=elapsed, status=response.status_code)
except Exception as exc: # Keep failed requests in the report.
return Result(
seconds=time.perf_counter() - started,
status=getattr(getattr(exc, "response", None), "status_code", 0),
error=str(exc),
)
async def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--base-url", default="http://localhost:11434")
parser.add_argument("--model", required=True)
parser.add_argument("--requests", type=int, default=12)
parser.add_argument("--concurrency", type=int, default=2)
parser.add_argument(
"--prompt",
default="Explain in four concise bullets why context length affects memory.",
)
args = parser.parse_args()
semaphore = asyncio.Semaphore(args.concurrency)
async with httpx.AsyncClient(timeout=600.0) as client:
async def bounded() -> Result:
async with semaphore:
return await run_one(
client,
f"{args.base_url.rstrip('/')}/api/chat",
args.model,
args.prompt,
)
wall_started = time.perf_counter()
results = await asyncio.gather(
*(bounded() for _ in range(args.requests))
)
wall_seconds = time.perf_counter() - wall_started
successes = [r.seconds for r in results if r.error is None]
failures = [r for r in results if r.error is not None]
print(f"requests: {len(results)}")
print(f"concurrency: {args.concurrency}")
print(f"wall seconds: {wall_seconds:.2f}")
print(f"successes: {len(successes)}")
print(f"failures: {len(failures)}")
if successes:
ordered = sorted(successes)
p95_index = max(0, math.ceil(0.95 * len(ordered)) - 1)
print(f"mean latency: {statistics.mean(successes):.2f}s")
print(f"median latency: {statistics.median(successes):.2f}s")
print(f"p95 latency: {ordered[p95_index]:.2f}s")
print(f"throughput: {len(successes) / wall_seconds:.3f} req/s")
for failure in failures[:5]:
print(f"failure status={failure.status}: {failure.error}")
if __name__ == "__main__":
asyncio.run(main())
Run a baseline:
python3 ollama_load_test.py \
--model llama3.2:3b \
--requests 12 \
--concurrency 1
Then compare:
python3 ollama_load_test.py --model llama3.2:3b --requests 12 --concurrency 2
python3 ollama_load_test.py --model llama3.2:3b --requests 12 --concurrency 4
For Herd, change the URL:
python3 ollama_load_test.py \
--base-url http://router-ip:11435 \
--model llama3.2:3b \
--requests 24 \
--concurrency 6
Do not compare runs while model state changes. Warm the model first or label cold-start runs separately.
What to measure
At minimum, record:
- total wall time;
- requests per second;
- p50 and p95 latency;
- time to first token for streaming workloads;
- prompt and generation token counts;
- error and timeout rate;
- peak memory;
- model processor placement;
- queue depth;
- cold versus hot model state.
A higher requests-per-second number can coexist with worse per-user latency. Choose the metric that matches the workload.
Context length is often the real concurrency limit
Ollama's default context depends on the VRAM it detects: 4K below 24 GiB, 32K from 24 to 48 GiB, and 256K (262,144 tokens) at 48 GiB or more. It recommends at least 64K for agent and coding tasks. On a large machine, then, four slots at the default context are four 256K slots, which is the 122.9 GB row in the chart below.
This creates a direct tradeoff. Ollama allocates the KV cache for context per request times OLLAMA_NUM_PARALLEL, so four parallel 64K slots cost the same memory as one 262K slot. On qwen3-coder:30b with an f16 KV cache, that is about 0.1 GB per 1,000 tokens of total context on top of 18.6 GB of weights (the Ollama memory calculator works it out for other models):
Examples of bad tuning:
- setting 64K context for a classifier that uses 1K;
- allowing four parallel 128K coding sessions on a machine sized for one;
- sending a different
num_ctxon each request and forcing model reloads; - benchmarking short prompts, then deploying long document requests.
Use real trace data to size context. Herd can report context usage and protect Ollama-format requests from unnecessary context changes.
Flash Attention and KV-cache quantization
Ollama can use Flash Attention when the backend and hardware support it. It also supports a global OLLAMA_KV_CACHE_TYPE setting when Flash Attention is enabled.
These features can reduce context-memory pressure, but they do not remove the need to benchmark. A global KV-cache setting affects every model, and quality or compatibility can vary by workload.
Change one variable at a time and preserve a baseline.
When fleet routing is better than more parallelism
Increase single-node parallelism when:
- one model fits with several context slots;
- aggregate throughput improves without unacceptable latency;
- the node remains fully accelerated;
- memory pressure and error rate stay low.
Add or use fleet capacity when:
- more than one capable machine is available;
- several independent users or agents create bursts;
- different models compete for one machine's memory;
- long contexts make same-model parallelism expensive;
- failover matters;
- embedding or batch work should not block interactive chat.
With Herd, place the same popular model on multiple nodes for horizontal concurrency. Keep specialized large models on machines that can hold them, and let smaller nodes handle lightweight work.
Herd reads each node's OLLAMA_NUM_PARALLEL and knows which models Ollama will serve one at a time (MLX-run models, embedders, and the architectures above), so it sends a node only as many requests as that node's Ollama will actually decode and keeps the rest in its own queue, visible at /fleet/queue, instead of piling them up inside Ollama.
A practical fleet tuning sequence
- Tune each Ollama node separately with concurrency
1. - Record the comfortable context and model set for each node.
- Test
OLLAMA_NUM_PARALLEL=2on nodes with headroom. - Start Herd and verify
/fleet/statusand/fleet/queue. - Duplicate the most frequently requested model on a second node.
- Run the same load test through port
11435. - Inspect
X-Fleet-Nodeheaders and request traces. - Compare distribution, latency, errors, and memory.
- Pin or prewarm only the models that earn their memory cost.
- Set client-side backoff for
503and other transient overload responses.
Troubleshooting by symptom
| Symptom | Likely cause | First change to test |
|---|---|---|
| Second request is much slower | Parallel sequences compete for memory bandwidth | Compare NUM_PARALLEL=1 and 2; measure aggregate throughput |
| OOM after increasing parallelism | Context/KV-cache multiplication | Reduce parallelism or context length |
| Frequent model reloads | Too many loaded models or changing context settings | Reduce model count; stabilize context; keep hot models deliberate |
503 server overloaded |
Ollama queue is full | Reduce burst rate, add backoff, or add eligible capacity |
| Long waits but few errors | Queue limit hides overload | Shorten queues and surface backpressure sooner |
| Model partly on CPU | Insufficient accelerator memory | Use a smaller quant/model or reduce context/model count |
| One fleet node receives everything | Only that node has the model hot or others are ineligible | Replicate the model and inspect node health/memory |
| First request is slow | Cold model load | Preload or pin the model; measure hot and cold separately |
Frequently asked questions
Does Ollama support concurrent requests?
Yes, at two levels. It can keep several models loaded when memory allows (OLLAMA_MAX_LOADED_MODELS), and it can run several requests for one model at once (OLLAMA_NUM_PARALLEL). By default each model runs one request at a time, and the rest wait in a queue until a slot is free.
What is the default value of OLLAMA_NUM_PARALLEL?
1, since Ollama 0.10.0 (July 2025). Earlier versions chose automatically: 4 or 1 depending on available memory through 0.6, then 2 or 1 through 0.9. That is why many guides still say 4.
Why does OLLAMA_NUM_PARALLEL have no effect?
Usually one of two reasons. The variable never reached the server (a terminal export does not configure the macOS app or a systemd service; check the server config line in Ollama's log), or the model is one Ollama always runs one request at a time: embedding models, models on the MLX engine on Apple Silicon, and a few architectures such as Qwen 3.5 and Qwen3-VL.
What should I set OLLAMA_NUM_PARALLEL to?
Start with 1, benchmark 2, and test 4 only when memory and context size permit. There is no safe universal value because model size, context length, quantization, backend, and hardware all matter.
Why did parallel requests make each response slower?
Several sequences share memory bandwidth and compute. Aggregate throughput may improve even while individual latency rises. Measure both.
Why does context length affect parallelism so much?
Each active sequence needs context state. Ollama documents that memory scales with parallelism multiplied by context length, so long agent sessions are much more expensive than short prompts.
Should I raise OLLAMA_MAX_QUEUE to stop 503 errors?
A larger queue reduces immediate rejections but can create very long waits. Fix or expose the capacity problem rather than hiding it indefinitely.
Is Herd a replacement for OLLAMA_NUM_PARALLEL?
No. Ollama parallelism controls one node. Herd routes across nodes. Use both at conservative, measured settings.
How do I support several users?
Replicate popular models across capable nodes, route through Herd, set reasonable per-node parallelism, limit queues, and use client backoff. Add application authentication and quotas separately when users are not all inside one trusted boundary.
Related Reading
- Ollama memory calculator, what each parallel slot and context length costs on your machine
- llama-swap vs Ollama, three ways to run more models than fit in memory
- Load balancing across Ollama servers
- Deploy Ollama Herd for ongoing operations
- Troubleshoot local LLM failures
- Plan Mac memory for local AI
- Build RAG and embedding workloads across a fleet
- Ollama Herd API reference
- How the Ollama Herd routing engine works