Ollama Concurrent Requests and Parallelism

Ollama can process concurrent work at two levels: it can keep several models loaded when memory allows, and it can process parallel requests for one loaded model. The key controls are OLLAMA_MAX_LOADED_MODELS, OLLAMA_NUM_PARALLEL, and OLLAMA_MAX_QUEUE.

More parallelism is not automatically faster. Every parallel sequence needs its own context state, so memory use grows with both parallel request count and context length. The right method is to benchmark one node at a time, increase parallelism gradually, and add fleet-level routing when one machine is no longer the right bottleneck.

The three layers of concurrency

Application concurrency
  many users, agents, workflows, or batch jobs
                    |
                    v
Fleet concurrency
  Ollama Herd routes independent requests across machines
                    |
                    v
Node concurrency
  one Ollama server loads models and runs parallel sequences

A slow application can be blocked at any layer. Increasing a limit at the wrong layer can make performance worse.

How Ollama handles concurrent requests

Ollama's current documentation describes three important behaviors:

1. Multiple models can stay loaded when the available system memory or GPU memory is sufficient.
2. A loaded model can process more than one request in parallel when enough memory was available at load time.
3. Requests queue when a required model or parallel slot cannot be admitted. Once the queue is full, Ollama returns an overloaded 503 response.

For a single model, Ollama states that required memory scales with:

OLLAMA_NUM_PARALLEL × context length

That is why four short requests may fit while four 64K agent sessions do not.

The three main settings

Defaults can change. Confirm them against the Ollama version you deploy.

Two related controls matter:

• OLLAMA_KEEP_ALIVE decides how long idle models stay loaded. Ollama's documented default is five minutes.
• OLLAMA_CONTEXT_LENGTH sets the server's context allocation and has a direct memory cost.

Start with a baseline, not a copied “best” value

A safe tuning cycle is:

1. Choose one model and one representative prompt.
2. Run one request at a time.
3. Record latency, tokens per second, memory, and whether the model is fully accelerated.
4. Test concurrency 2 with the same context and prompts.
5. Test 4 only when memory remains comfortable.
6. Compare aggregate throughput, p50 latency, p95 latency, and errors.
7. Repeat for long-context and short-context workloads separately.

Do not benchmark only a one-sentence prompt. Agent, coding, and RAG requests spend much more time in prompt evaluation and can allocate far more context.

Example configuration profiles

These are starting points for measurement, not universal recommendations.

Stability-first: one large model

export OLLAMA_NUM_PARALLEL=1
export OLLAMA_MAX_LOADED_MODELS=1
export OLLAMA_MAX_QUEUE=64
ollama serve

Use when one model consumes most available memory or long contexts dominate.

Balanced interactive use

export OLLAMA_NUM_PARALLEL=2
export OLLAMA_MAX_LOADED_MODELS=2
export OLLAMA_MAX_QUEUE=64
ollama serve

Use only when both loaded-model weights and two context slots fit with headroom.

Throughput experiment

export OLLAMA_NUM_PARALLEL=4
export OLLAMA_MAX_LOADED_MODELS=1
export OLLAMA_MAX_QUEUE=128
ollama serve

This isolates same-model parallelism. It is useful for a test, not a production default. Four long contexts can consume far more memory than expected.

Configure environment variables through the platform's supported service mechanism for a persistent installation rather than relying on a temporary terminal export.

Verify what is actually loaded

Use:

ollama ps

Inspect:

• model size;
• processor placement;
• allocated context;
• how long the model will stay loaded.

A model partially offloaded to CPU may remain functional but behave very differently under concurrent load. Record this state with benchmark results.

A small reproducible load test

Install httpx:

python3 -m pip install httpx

Save as ollama_load_test.py:

from __future__ import annotations

import argparse
import asyncio
import math
import statistics
import time
from dataclasses import dataclass

import httpx


@dataclass
class Result:
    seconds: float
    status: int
    error: str | None = None


async def run_one(
    client: httpx.AsyncClient,
    url: str,
    model: str,
    prompt: str,
) -> Result:
    started = time.perf_counter()
    try:
        response = await client.post(
            url,
            json={
                "model": model,
                "messages": [{"role": "user", "content": prompt}],
                "stream": False,
            },
        )
        elapsed = time.perf_counter() - started
        response.raise_for_status()
        return Result(seconds=elapsed, status=response.status_code)
    except Exception as exc:  # Keep failed requests in the report.
        return Result(
            seconds=time.perf_counter() - started,
            status=getattr(getattr(exc, "response", None), "status_code", 0),
            error=str(exc),
        )


async def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--base-url", default="http://localhost:11434")
    parser.add_argument("--model", required=True)
    parser.add_argument("--requests", type=int, default=12)
    parser.add_argument("--concurrency", type=int, default=2)
    parser.add_argument(
        "--prompt",
        default="Explain in four concise bullets why context length affects memory.",
    )
    args = parser.parse_args()

    semaphore = asyncio.Semaphore(args.concurrency)

    async with httpx.AsyncClient(timeout=600.0) as client:
        async def bounded() -> Result:
            async with semaphore:
                return await run_one(
                    client,
                    f"{args.base_url.rstrip('/')}/api/chat",
                    args.model,
                    args.prompt,
                )

        wall_started = time.perf_counter()
        results = await asyncio.gather(
            *(bounded() for _ in range(args.requests))
        )
        wall_seconds = time.perf_counter() - wall_started

    successes = [r.seconds for r in results if r.error is None]
    failures = [r for r in results if r.error is not None]

    print(f"requests:       {len(results)}")
    print(f"concurrency:    {args.concurrency}")
    print(f"wall seconds:   {wall_seconds:.2f}")
    print(f"successes:      {len(successes)}")
    print(f"failures:       {len(failures)}")

    if successes:
        ordered = sorted(successes)
        p95_index = max(0, math.ceil(0.95 * len(ordered)) - 1)
        print(f"mean latency:   {statistics.mean(successes):.2f}s")
        print(f"median latency: {statistics.median(successes):.2f}s")
        print(f"p95 latency:    {ordered[p95_index]:.2f}s")
        print(f"throughput:     {len(successes) / wall_seconds:.3f} req/s")

    for failure in failures[:5]:
        print(f"failure status={failure.status}: {failure.error}")


if __name__ == "__main__":
    asyncio.run(main())

Run a baseline:

python3 ollama_load_test.py \
  --model llama3.2:3b \
  --requests 12 \
  --concurrency 1

Then compare:

python3 ollama_load_test.py --model llama3.2:3b --requests 12 --concurrency 2
python3 ollama_load_test.py --model llama3.2:3b --requests 12 --concurrency 4

For Herd, change the URL:

python3 ollama_load_test.py \
  --base-url http://router-ip:11435 \
  --model llama3.2:3b \
  --requests 24 \
  --concurrency 6

Do not compare runs while model state changes. Warm the model first or label cold-start runs separately.

What to measure

At minimum, record:

• total wall time;
• requests per second;
• p50 and p95 latency;
• time to first token for streaming workloads;
• prompt and generation token counts;
• error and timeout rate;
• peak memory;
• model processor placement;
• queue depth;
• cold versus hot model state.

A higher requests-per-second number can coexist with worse per-user latency. Choose the metric that matches the workload.

Context length is often the real concurrency limit

Ollama's context-length guidance currently varies the default allocation by available VRAM, and it recommends large contexts for agent and coding tasks. Larger context also requires more memory.

This creates a direct tradeoff:

more context per request
          ×
more parallel requests
          =
more context memory

Examples of bad tuning:

• setting 64K context for a classifier that uses 1K;
• allowing four parallel 128K coding sessions on a machine sized for one;
• sending a different num_ctx on each request and forcing model reloads;
• benchmarking short prompts, then deploying long document requests.

Use real trace data to size context. Herd can report context usage and protect Ollama-format requests from unnecessary context changes.

Flash Attention and KV-cache quantization

Ollama can use Flash Attention when the backend and hardware support it. It also supports a global OLLAMA_KV_CACHE_TYPE setting when Flash Attention is enabled.

These features can reduce context-memory pressure, but they do not remove the need to benchmark. A global KV-cache setting affects every model, and quality or compatibility can vary by workload.

Change one variable at a time and preserve a baseline.

When fleet routing is better than more parallelism

Increase single-node parallelism when:

• one model fits with several context slots;
• aggregate throughput improves without unacceptable latency;
• the node remains fully accelerated;
• memory pressure and error rate stay low.

Add or use fleet capacity when:

• more than one capable machine is available;
• several independent users or agents create bursts;
• different models compete for one machine's memory;
• long contexts make same-model parallelism expensive;
• failover matters;
• embedding or batch work should not block interactive chat.

With Herd, place the same popular model on multiple nodes for horizontal concurrency. Keep specialized large models on machines that can hold them, and let smaller nodes handle lightweight work.

A practical fleet tuning sequence

1. Tune each Ollama node separately with concurrency 1.
2. Record the comfortable context and model set for each node.
3. Test OLLAMA_NUM_PARALLEL=2 on nodes with headroom.
4. Start Herd and verify /fleet/status and /fleet/queue.
5. Duplicate the most frequently requested model on a second node.
6. Run the same load test through port 11435.
7. Inspect X-Fleet-Node headers and request traces.
8. Compare distribution, latency, errors, and memory.
9. Pin or prewarm only the models that earn their memory cost.
10. Set client-side backoff for 503 and other transient overload responses.

Troubleshooting by symptom

Frequently asked questions

Does Ollama support concurrent requests?

Yes. It can load several models when memory allows and can process parallel requests for one model. Requests that cannot be admitted wait in a queue until capacity is available.

What should I set OLLAMA_NUM_PARALLEL to?

Start with 1, benchmark 2, and test 4 only when memory and context size permit. There is no safe universal value because model size, context length, quantization, backend, and hardware all matter.

Why did parallel requests make each response slower?

Several sequences share memory bandwidth and compute. Aggregate throughput may improve even while individual latency rises. Measure both.

Why does context length affect parallelism so much?

Each active sequence needs context state. Ollama documents that memory scales with parallelism multiplied by context length, so long agent sessions are much more expensive than short prompts.

Should I raise OLLAMA_MAX_QUEUE to stop 503 errors?

A larger queue reduces immediate rejections but can create very long waits. Fix or expose the capacity problem rather than hiding it indefinitely.

Is Herd a replacement for OLLAMA_NUM_PARALLEL?

No. Ollama parallelism controls one node. Herd routes across nodes. Use both at conservative, measured settings.

How do I support several users?

Replicate popular models across capable nodes, route through Herd, set reasonable per-node parallelism, limit queues, and use client backoff. Add application authentication and quotas separately when users are not all inside one trusted boundary.

Related Reading

• Load balancing across Ollama servers
• Deploy Ollama Herd for ongoing operations
• Troubleshoot local LLM failures
• Plan Mac memory for local AI
• Build RAG and embedding workloads across a fleet
• Ollama Herd API reference
• How the Ollama Herd routing engine works