Open WebUI with Multiple Ollama Servers

Open WebUI can connect to more than one Ollama instance, but its native multi-instance mode uses random selection for basic load distribution. Pointing Open WebUI at one Ollama Herd URL keeps the chat interface, accounts, conversations, and RAG features in Open WebUI while Herd chooses the best available inference node for each request.

This guide shows the complete setup, including Docker networking, model discovery, verification, context settings, and common failures.

The result: Open WebUI talks to http://router-ip:11435 as though it were one Ollama server. Behind that URL, Ollama Herd presents models from the fleet and routes requests according to model availability, memory fit, queue state, node health, and other live signals.

What you are building

Browser
   |
   v
Open WebUI
   |
   | Ollama API
   v
Ollama Herd :11435
   |--------------------|--------------------|
   v                    v                    v
Ollama node A       Ollama node B       Ollama node C
large models        fast models         overflow / backup

Open WebUI remains the user-facing application. It provides chat, authentication, conversation history, document management, and other interface features. Herd remains the routing layer. It provides one API endpoint for a changing set of inference machines.

Native Open WebUI balancing versus Herd routing

Open WebUI supports adding several Ollama connections. Its documentation describes a random selection strategy for basic load balancing, with matching model IDs required when models should merge into one visible entry.

That can be enough when every server is identical. It is less useful when:

• one machine has much more memory than another;
• only some machines hold a requested model;
• a model is already hot on one node and cold on another;
• a laptop is busy or thermally constrained;
• one backend has a deeper queue;
• a node disappears during a request burst.

Herd moves that decision into a model-aware router. Open WebUI only needs one connection, and the fleet can change without editing the UI every time a machine joins or leaves.

Prerequisites

You need:

• Open WebUI installed and reachable;
• Ollama running on two or more machines;
• Ollama Herd installed on the router and node machines;
• at least one model available in the fleet;
• network access from Open WebUI to the Herd router on port 11435.

Start the router:

pip install --upgrade ollama-herd
herd

Start a node agent on every machine that contributes an Ollama instance, including the router machine when it also runs models:

pip install --upgrade ollama-herd
herd-node

When mDNS discovery is unavailable, point the node directly at the router:

herd-node --router-url http://192.168.1.20:11435

Replace 192.168.1.20 with the router's address.

Step 1: Verify the fleet before opening Open WebUI

Check node status from the router machine:

curl -s http://localhost:11435/fleet/status | python3 -m json.tool

Then verify the Ollama-compatible model list:

curl -s http://localhost:11435/api/tags | python3 -m json.tool

The response should contain models available across the fleet. Herd's /api/tags response can also include fleet-node information for each model.

Do not continue until both commands work. This separates a Herd or node-discovery problem from an Open WebUI connection problem.

Step 2: Add Herd as the Ollama connection

In Open WebUI:

1. Open Admin Settings.
2. Go to Connections → Ollama.
3. Open the connection manager.
4. Add the Herd router URL:

http://192.168.1.20:11435

5. Save the connection.
6. Disable or remove direct Ollama URLs when you want every chat request to pass through Herd.

Leaving direct Ollama connections enabled is valid, but it makes routing behavior harder to understand. A model may be reachable both directly and through Herd, and a user can accidentally bypass fleet routing by choosing the direct connection.

Step 3: Handle Docker networking correctly

The most common failure is using localhost from the wrong network namespace.

For Docker on Linux, a typical host mapping is:

services:
  open-webui:
    extra_hosts:
      - "host.docker.internal:host-gateway"

Then configure Open WebUI with:

http://host.docker.internal:11435

Test from inside the Open WebUI container when the UI still reports a connection error:

docker exec -it open-webui sh
wget -qO- http://host.docker.internal:11435/api/tags

Use the actual container name if it is not open-webui.

Step 4: Send a verification request outside the UI

Before testing a long conversation, send one small request directly to Herd:

curl -i http://192.168.1.20:11435/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.2:3b",
    "messages": [{"role": "user", "content": "Reply with the word ready."}],
    "stream": false
  }'

Replace the model with one returned by /api/tags.

Look for routing headers such as:

X-Fleet-Node: mac-studio
X-Fleet-Score: 84

Those headers prove that the request reached Herd and identify the selected node. Open WebUI does not need to display the headers for routing to work.

Step 5: Test from Open WebUI

Refresh the model list, choose a fleet model, and start a new chat. A healthy setup should have these properties:

• models returned by Herd are available in Open WebUI;
• a normal chat streams successfully;
• the Herd dashboard records the request;
• the selected model and node appear in the fleet trace;
• taking one eligible node offline does not break model discovery when another node can serve the model.

Open the Herd dashboard at:

http://192.168.1.20:11435/dashboard

Use it to distinguish UI latency from inference latency, inspect queues, and confirm which node actually ran the request.

Context length: avoid a silent Open WebUI override

Open WebUI has a per-model and per-chat num_ctx setting. When it is set, Open WebUI sends that value with each Ollama request, overriding the Ollama server's default context setting.

That matters because:

• a small value can truncate a conversation or tool schema;
• a large value consumes more memory;
• changing context settings can force a model reload;
• different Open WebUI presets can make the same model behave differently.

A good default is to leave num_ctx unset in Open WebUI and configure a stable context size on the inference side. Set it per model only when you have measured the memory cost and know the workflow requires it.

Herd's context-protection mode can strip unnecessary num_ctx values from Ollama-format requests when the requested value does not exceed the already-loaded context. That reduces accidental reloads, but it is still better to keep the client configuration deliberate.

Model management: use the right interface

Open WebUI can list, pull, and unload models when its backend implements the expected Ollama management endpoints. Herd exposes fleet-wide model listing and model pulling, but node placement adds a decision that a single-server UI does not express well.

Use this division of responsibility:

• Open WebUI: choose models, chat, manage users, conversations, and knowledge bases;
• Herd dashboard or API: inspect which nodes hold a model, pull to a chosen node, delete node-specific copies, pin hot models, and view fleet health;
• Ollama CLI: perform direct node maintenance when troubleshooting one machine.

This keeps a fleet-wide action from being mistaken for a single-server action.

Using Open WebUI RAG through Herd

Open WebUI owns the document pipeline: uploads, chunking, vector storage, retrieval, and source display. It sends embedding and generation requests to its configured providers.

When the Ollama connection points to Herd:

• embedding calls can use Herd's /api/embed or /api/embeddings compatibility;
• generation calls use the same router URL;
• embedding and chat work can be placed on different machines;
• document-ingestion bursts do not have to share one inference queue with interactive chat.

For a full workload-separation design, link to the dedicated Ollama RAG and Embeddings Across a Fleet guide.

Recommended production layout

For a small team:

Open WebUI host
  └─ one Ollama connection: Herd router

Herd router
  ├─ always-on machine
  ├─ dashboard restricted to trusted users
  └─ request logs retained according to your privacy policy

Fleet nodes
  ├─ large-model node
  ├─ fast interactive node
  ├─ embedding node
  └─ optional laptop capacity

Keep Open WebUI authentication in front of users. Keep Herd on a trusted LAN or private VPN. Do not treat an unauthenticated local inference API as a public internet service.

Troubleshooting

Open WebUI says the connection failed

Run this from the Open WebUI host or container:

curl -v http://router-ip:11435/api/tags

• Connection refused means the router is not listening at that address or a firewall blocks the port.
• A timeout usually means the address is unreachable from that container or network.
• A successful JSON response means the network path works; recheck the saved Open WebUI URL.

The model list is empty

Check the fleet first:

curl -s http://router-ip:11435/fleet/status | python3 -m json.tool
curl -s http://router-ip:11435/api/tags | python3 -m json.tool

Confirm that node agents are online and that at least one model is present. Model tags must be exact: model:tag and model:other-tag are different identifiers.

A chat works directly but not through Docker

localhost inside a container points back to that container. Use host.docker.internal, a Docker service name, or the router's reachable LAN/Tailnet address.

Long conversations become blank or lose tools

Check whether Open WebUI is sending a small num_ctx. A preset can override the server even after the Ollama context setting was increased. Remove the override or set a realistic value that fits the node's memory.

The first response is much slower than the next one

The model was probably cold and had to load. Keep important models resident, place popular models on more than one node when concurrency justifies it, and use the Herd dashboard to confirm whether requests hit hot or cold copies.

Pulling or deleting from Open WebUI does not target the expected node

Use Herd's model-management controls or API for node-specific placement. A generic Ollama UI cannot infer your intended fleet topology.

Frequently asked questions

Does Open WebUI already support multiple Ollama servers?

Yes. Its native connection manager can add multiple Ollama instances and distribute requests with a random selection strategy. Herd is useful when the backends are heterogeneous or when routing should consider model residency, memory, queue state, health, and failover.

Should I add every Ollama node to Open WebUI and also add Herd?

Usually no. Add only Herd when you want one understandable routing path. Keep a direct node connection only for deliberate administration or debugging.

Does Herd replace Open WebUI?

No. Open WebUI is the user interface and conversation application. Herd is the inference-routing backend. They solve different layers of the stack.

Can Open WebUI users share the same fleet?

Yes. Open WebUI handles users and sessions; all of their model calls can flow through one Herd endpoint. Capacity planning and access policy still need to match the number of users.

Can I use Open WebUI RAG with Herd?

Yes. Open WebUI keeps responsibility for documents and retrieval, while Herd can route the embedding and generation requests. Use a stable embedding model for both indexing and querying.

Is a single Ollama server simpler?

Yes. When one machine serves every required model with acceptable latency, connect Open WebUI directly. Herd becomes useful when you have several machines, several workload types, meaningful concurrency, or a need for failover and fleet visibility.

Related Reading

• Install and verify Ollama Herd
• Ollama Herd versus Open WebUI
• Load balancing across Ollama servers
• Build RAG and embedding workloads across a fleet
• Ollama Herd API reference
• Deploy Ollama Herd for ongoing operations