Ollama Parallel Requests and OLLAMA_NUM_PARALLEL

How to make one Ollama server answer several requests at once, what it costs in memory, why some models ignore the setting, and when to add a second machine instead.

Ollama runs one request at a time per model by default. To process several at once, set OLLAMA_NUM_PARALLEL on the Ollama server (default 1 since Ollama 0.10.0, July 2025) and restart it. Requests beyond the parallel slots wait in a queue of up to OLLAMA_MAX_QUEUE requests (default 512); once that is full, Ollama answers 503.

Checked October 26, 2026 against Ollama 0.40.11 (October 24, 2026): its FAQ, envconfig/config.go, and server/sched.go. Ollama ships several releases a week; recheck a default before relying on it.

The short answer

  • Where to set it: on the server process, not in your client. A terminal export does not reach the macOS app or a systemd service; use the commands below.
  • What it costs: every slot gets its own full context, so context memory is context length times slots. qwen3-coder:30b at 32K context measured 31.0 GB with 4 slots, against about 21.8 GB with one.
  • What it buys: more total throughput, not N times as much. On one busy M3 Ultra, gemma3:27b decoded 22.5 tokens/s for a single request and 27.1 tokens/s in total with four slots busy (1.2x).
  • When it does nothing: embedding models, models Ollama runs on its MLX engine (the Apple Silicon default since 0.40), and a few architectures always run one request at a time. See models that ignore the setting.

How to set OLLAMA_NUM_PARALLEL

macOS (Ollama app): set it for launchd, then quit Ollama from the menu bar and open it again. A launchctl setenv value does not survive a reboot.

launchctl setenv OLLAMA_NUM_PARALLEL 2

Linux (systemd service): add an override, then reload and restart.

sudo systemctl edit ollama.service
# in the editor, under [Service], add:
#   Environment="OLLAMA_NUM_PARALLEL=2"
sudo systemctl daemon-reload
sudo systemctl restart ollama

Windows: quit Ollama from the system tray, set a user environment variable (in Settings, search for "environment variables", or run the command below), then start Ollama from the Start menu.

setx OLLAMA_NUM_PARALLEL 2

Docker: pass it to the container.

docker run -d -e OLLAMA_NUM_PARALLEL=2 -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

Check that it took: Ollama logs its settings at startup in a server config line that includes OLLAMA_NUM_PARALLEL.

grep "server config" ~/.ollama/logs/server.log | tail -1             # macOS
journalctl -u ollama --no-pager | grep "server config" | tail -1    # Linux
docker logs ollama 2>&1 | grep "server config" | tail -1           # Docker

On Windows the log is server.log in %LOCALAPPDATA%\Ollama. Set OLLAMA_MAX_QUEUE, OLLAMA_MAX_LOADED_MODELS, and OLLAMA_CONTEXT_LENGTH the same way.

More parallelism is not automatically faster. Every parallel sequence needs its own context state, so memory use grows with both parallel request count and context length. The right method is to benchmark one node at a time, increase parallelism gradually, and add fleet-level routing when one machine is no longer the right bottleneck.

The three layers of concurrency

Application concurrency
  many users, agents, workflows, or batch jobs
                    |
                    v
Fleet concurrency
  Ollama Herd routes independent requests across machines
                    |
                    v
Node concurrency
  one Ollama server loads models and runs parallel sequences

A slow application can be blocked at any layer. Increasing a limit at the wrong layer can make performance worse.

How Ollama handles concurrent requests

Ollama's current documentation describes three important behaviors:

  1. Multiple models can stay loaded when the available system memory or GPU memory is sufficient.
  2. A loaded model can process more than one request in parallel when enough memory was available at load time.
  3. Requests queue when a required model or parallel slot cannot be admitted. Once the queue is full, Ollama returns an overloaded 503 response.

For a single model, Ollama states that required memory scales with:

OLLAMA_NUM_PARALLEL × context length

That is why four short requests may fit while four 64K agent sessions do not.

The three main settings

Setting What it controls Current documented default Main risk when increased
OLLAMA_NUM_PARALLEL Parallel requests per loaded model 1 (since 0.10.0) Context/KV-cache memory multiplication
OLLAMA_MAX_LOADED_MODELS Number of models that may remain loaded when they fit 3 × GPU count, or 3 for CPU inference Model weights consume memory and trigger eviction or offload
OLLAMA_MAX_QUEUE Waiting requests before new work is rejected 512 Very long waits and hidden overload

Defaults can change. Confirm them against the Ollama version you deploy.

Two related controls matter:

  • OLLAMA_KEEP_ALIVE decides how long idle models stay loaded. Ollama's documented default is five minutes.
  • OLLAMA_CONTEXT_LENGTH sets the server's context allocation and has a direct memory cost.

Models that ignore OLLAMA_NUM_PARALLEL

If you raised the setting, confirmed it in the log, and requests for a model still run one after another, the model is one that Ollama serves serially whatever the setting says. In Ollama 0.40.11:

Case What Ollama does How to tell
Models on the MLX engine (Apple Silicon) The MLX runner processes one request at a time. Since 0.40, Macs run every architecture MLX supports on MLX automatically ollama ps shows mlx in the RUNNER column
Embedding models Always loaded with one slot The model only produces embeddings
Architectures qwen35, qwen35moe, qwen3next, qwen3vl, qwen3vlmoe, mllama, lfm2, lfm2moe, nemotron_h, nemotron_h_moe, nemotron_h_omni Forced to one slot on every platform The server log warns "model architecture does not currently support parallel requests"

Parallel requests on the MLX engine are being worked on upstream (ollama#17280, with open pull requests #17317 and #19197) but are not in a release. On a Mac today you can run the model's llama.cpp variant where it has one (ollama run --runner llamacpp MODEL, or "runner": "llamacpp" in an /api/chat request), use an MLX server with continuous batching (see oMLX vs Ollama), or spread requests over a second machine.

Start with a baseline, not a copied “best” value

A safe tuning cycle is:

  1. Choose one model and one representative prompt.
  2. Run one request at a time.
  3. Record latency, tokens per second, memory, and whether the model is fully accelerated.
  4. Test concurrency 2 with the same context and prompts.
  5. Test 4 only when memory remains comfortable.
  6. Compare aggregate throughput, p50 latency, p95 latency, and errors.
  7. Repeat for long-context and short-context workloads separately.

Do not benchmark only a one-sentence prompt. Agent, coding, and RAG requests spend much more time in prompt evaluation and can allocate far more context.

Example configuration profiles

These are starting points for measurement, not universal recommendations.

Stability-first: one large model

export OLLAMA_NUM_PARALLEL=1
export OLLAMA_MAX_LOADED_MODELS=1
export OLLAMA_MAX_QUEUE=64
ollama serve

Use when one model consumes most available memory or long contexts dominate.

Balanced interactive use

export OLLAMA_NUM_PARALLEL=2
export OLLAMA_MAX_LOADED_MODELS=2
export OLLAMA_MAX_QUEUE=64
ollama serve

Use only when both loaded-model weights and two context slots fit with headroom.

Throughput experiment

export OLLAMA_NUM_PARALLEL=4
export OLLAMA_MAX_LOADED_MODELS=1
export OLLAMA_MAX_QUEUE=128
ollama serve

This isolates same-model parallelism. It is useful for a test, not a production default. Four long contexts can consume far more memory than expected.

These export lines are for a foreground ollama serve test. For the macOS app, a systemd service, Windows, or Docker, set the same variables with the platform commands above.

Verify what is actually loaded

Use:

ollama ps

Inspect:

  • model size;
  • processor placement;
  • allocated context;
  • how long the model will stay loaded.

A model partially offloaded to CPU may remain functional but behave very differently under concurrent load. Record this state with benchmark results.

A small reproducible load test

Install httpx:

python3 -m pip install httpx

Save as ollama_load_test.py:

from __future__ import annotations

import argparse
import asyncio
import math
import statistics
import time
from dataclasses import dataclass

import httpx


@dataclass
class Result:
    seconds: float
    status: int
    error: str | None = None


async def run_one(
    client: httpx.AsyncClient,
    url: str,
    model: str,
    prompt: str,
) -> Result:
    started = time.perf_counter()
    try:
        response = await client.post(
            url,
            json={
                "model": model,
                "messages": [{"role": "user", "content": prompt}],
                "stream": False,
            },
        )
        elapsed = time.perf_counter() - started
        response.raise_for_status()
        return Result(seconds=elapsed, status=response.status_code)
    except Exception as exc:  # Keep failed requests in the report.
        return Result(
            seconds=time.perf_counter() - started,
            status=getattr(getattr(exc, "response", None), "status_code", 0),
            error=str(exc),
        )


async def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--base-url", default="http://localhost:11434")
    parser.add_argument("--model", required=True)
    parser.add_argument("--requests", type=int, default=12)
    parser.add_argument("--concurrency", type=int, default=2)
    parser.add_argument(
        "--prompt",
        default="Explain in four concise bullets why context length affects memory.",
    )
    args = parser.parse_args()

    semaphore = asyncio.Semaphore(args.concurrency)

    async with httpx.AsyncClient(timeout=600.0) as client:
        async def bounded() -> Result:
            async with semaphore:
                return await run_one(
                    client,
                    f"{args.base_url.rstrip('/')}/api/chat",
                    args.model,
                    args.prompt,
                )

        wall_started = time.perf_counter()
        results = await asyncio.gather(
            *(bounded() for _ in range(args.requests))
        )
        wall_seconds = time.perf_counter() - wall_started

    successes = [r.seconds for r in results if r.error is None]
    failures = [r for r in results if r.error is not None]

    print(f"requests:       {len(results)}")
    print(f"concurrency:    {args.concurrency}")
    print(f"wall seconds:   {wall_seconds:.2f}")
    print(f"successes:      {len(successes)}")
    print(f"failures:       {len(failures)}")

    if successes:
        ordered = sorted(successes)
        p95_index = max(0, math.ceil(0.95 * len(ordered)) - 1)
        print(f"mean latency:   {statistics.mean(successes):.2f}s")
        print(f"median latency: {statistics.median(successes):.2f}s")
        print(f"p95 latency:    {ordered[p95_index]:.2f}s")
        print(f"throughput:     {len(successes) / wall_seconds:.3f} req/s")

    for failure in failures[:5]:
        print(f"failure status={failure.status}: {failure.error}")


if __name__ == "__main__":
    asyncio.run(main())

Run a baseline:

python3 ollama_load_test.py \
  --model llama3.2:3b \
  --requests 12 \
  --concurrency 1

Then compare:

python3 ollama_load_test.py --model llama3.2:3b --requests 12 --concurrency 2
python3 ollama_load_test.py --model llama3.2:3b --requests 12 --concurrency 4

For Herd, change the URL:

python3 ollama_load_test.py \
  --base-url http://router-ip:11435 \
  --model llama3.2:3b \
  --requests 24 \
  --concurrency 6

Do not compare runs while model state changes. Warm the model first or label cold-start runs separately.

What to measure

At minimum, record:

  • total wall time;
  • requests per second;
  • p50 and p95 latency;
  • time to first token for streaming workloads;
  • prompt and generation token counts;
  • error and timeout rate;
  • peak memory;
  • model processor placement;
  • queue depth;
  • cold versus hot model state.

A higher requests-per-second number can coexist with worse per-user latency. Choose the metric that matches the workload.

Context length is often the real concurrency limit

Ollama's default context depends on the VRAM it detects: 4K below 24 GiB, 32K from 24 to 48 GiB, and 256K (262,144 tokens) at 48 GiB or more. It recommends at least 64K for agent and coding tasks. On a large machine, then, four slots at the default context are four 256K slots, which is the 122.9 GB row in the chart below.

This creates a direct tradeoff. Ollama allocates the KV cache for context per request times OLLAMA_NUM_PARALLEL, so four parallel 64K slots cost the same memory as one 262K slot. On qwen3-coder:30b with an f16 KV cache, that is about 0.1 GB per 1,000 tokens of total context on top of 18.6 GB of weights (the Ollama memory calculator works it out for other models):

Stacked bars of estimated resident memory for qwen3-coder:30b, 18.6 GB of weights in every case. 32K context times 1 slot: about 21.8 GB. 32K times 4 slots: 31.0 GB, measured. 262K times 1 slot: about 44.0 GB. 64K times 4 slots: about 44.0 GB. 128K times 4 slots: about 69.5 GB. 262K times 4 slots: 122.9 GB, measured.
Context per request times parallel slots sets the KV cache: the same model runs in about 22 GB with one 32K slot and needed 122.9 GB with four 262K slots. Modeled from two measurements, one model, GGUF build on llama.cpp, f16 KV cache (the 32K x 4 and 262K x 4 rows are measured). Source: Ollama Herd v0.9.5 issue notes on model sizing and the KV cache.

Examples of bad tuning:

  • setting 64K context for a classifier that uses 1K;
  • allowing four parallel 128K coding sessions on a machine sized for one;
  • sending a different num_ctx on each request and forcing model reloads;
  • benchmarking short prompts, then deploying long document requests.

Use real trace data to size context. Herd can report context usage and protect Ollama-format requests from unnecessary context changes.

Flash Attention and KV-cache quantization

Ollama can use Flash Attention when the backend and hardware support it. It also supports a global OLLAMA_KV_CACHE_TYPE setting when Flash Attention is enabled.

These features can reduce context-memory pressure, but they do not remove the need to benchmark. A global KV-cache setting affects every model, and quality or compatibility can vary by workload.

Change one variable at a time and preserve a baseline.

When fleet routing is better than more parallelism

Increase single-node parallelism when:

  • one model fits with several context slots;
  • aggregate throughput improves without unacceptable latency;
  • the node remains fully accelerated;
  • memory pressure and error rate stay low.

Add or use fleet capacity when:

  • more than one capable machine is available;
  • several independent users or agents create bursts;
  • different models compete for one machine's memory;
  • long contexts make same-model parallelism expensive;
  • failover matters;
  • embedding or batch work should not block interactive chat.

With Herd, place the same popular model on multiple nodes for horizontal concurrency. Keep specialized large models on machines that can hold them, and let smaller nodes handle lightweight work.

Herd reads each node's OLLAMA_NUM_PARALLEL and knows which models Ollama will serve one at a time (MLX-run models, embedders, and the architectures above), so it sends a node only as many requests as that node's Ollama will actually decode and keeps the rest in its own queue, visible at /fleet/queue, instead of piling them up inside Ollama.

A practical fleet tuning sequence

  1. Tune each Ollama node separately with concurrency 1.
  2. Record the comfortable context and model set for each node.
  3. Test OLLAMA_NUM_PARALLEL=2 on nodes with headroom.
  4. Start Herd and verify /fleet/status and /fleet/queue.
  5. Duplicate the most frequently requested model on a second node.
  6. Run the same load test through port 11435.
  7. Inspect X-Fleet-Node headers and request traces.
  8. Compare distribution, latency, errors, and memory.
  9. Pin or prewarm only the models that earn their memory cost.
  10. Set client-side backoff for 503 and other transient overload responses.

Troubleshooting by symptom

Symptom Likely cause First change to test
Second request is much slower Parallel sequences compete for memory bandwidth Compare NUM_PARALLEL=1 and 2; measure aggregate throughput
OOM after increasing parallelism Context/KV-cache multiplication Reduce parallelism or context length
Frequent model reloads Too many loaded models or changing context settings Reduce model count; stabilize context; keep hot models deliberate
503 server overloaded Ollama queue is full Reduce burst rate, add backoff, or add eligible capacity
Long waits but few errors Queue limit hides overload Shorten queues and surface backpressure sooner
Model partly on CPU Insufficient accelerator memory Use a smaller quant/model or reduce context/model count
One fleet node receives everything Only that node has the model hot or others are ineligible Replicate the model and inspect node health/memory
First request is slow Cold model load Preload or pin the model; measure hot and cold separately

Frequently asked questions

Does Ollama support concurrent requests?

Yes, at two levels. It can keep several models loaded when memory allows (OLLAMA_MAX_LOADED_MODELS), and it can run several requests for one model at once (OLLAMA_NUM_PARALLEL). By default each model runs one request at a time, and the rest wait in a queue until a slot is free.

What is the default value of OLLAMA_NUM_PARALLEL?

1, since Ollama 0.10.0 (July 2025). Earlier versions chose automatically: 4 or 1 depending on available memory through 0.6, then 2 or 1 through 0.9. That is why many guides still say 4.

Why does OLLAMA_NUM_PARALLEL have no effect?

Usually one of two reasons. The variable never reached the server (a terminal export does not configure the macOS app or a systemd service; check the server config line in Ollama's log), or the model is one Ollama always runs one request at a time: embedding models, models on the MLX engine on Apple Silicon, and a few architectures such as Qwen 3.5 and Qwen3-VL.

What should I set OLLAMA_NUM_PARALLEL to?

Start with 1, benchmark 2, and test 4 only when memory and context size permit. There is no safe universal value because model size, context length, quantization, backend, and hardware all matter.

Why did parallel requests make each response slower?

Several sequences share memory bandwidth and compute. Aggregate throughput may improve even while individual latency rises. Measure both.

Why does context length affect parallelism so much?

Each active sequence needs context state. Ollama documents that memory scales with parallelism multiplied by context length, so long agent sessions are much more expensive than short prompts.

Should I raise OLLAMA_MAX_QUEUE to stop 503 errors?

A larger queue reduces immediate rejections but can create very long waits. Fix or expose the capacity problem rather than hiding it indefinitely.

Is Herd a replacement for OLLAMA_NUM_PARALLEL?

No. Ollama parallelism controls one node. Herd routes across nodes. Use both at conservative, measured settings.

How do I support several users?

Replicate popular models across capable nodes, route through Herd, set reasonable per-node parallelism, limit queues, and use client backoff. Add application authentication and quotas separately when users are not all inside one trusted boundary.