Ollama RAG and Embeddings Across a Fleet
Stop document-ingestion embeddings from blocking interactive chat. Separate embedding and generation across a fleet so retrieval and inference scale on their own.
A local RAG system does not run one workload. It runs at least three: embedding, vector search, and generation. On a single Ollama server, a large document import can compete with interactive chat for model slots and memory. With Ollama Herd, the application keeps one API URL while embedding and generation requests can use different machines or backends.
The most important design rules are:
- use the same embedding model for indexing and querying;
- batch document embeddings during ingestion;
- keep vector-store collections tied to one embedding model and dimension;
- separate embedding capacity from interactive LLM capacity when ingestion is frequent;
- measure retrieval quality independently from answer quality.
Reference architecture
INGESTION PATH
Documents -> parse -> chunk -> /api/embed -> embedding node
|
v
vector store
QUERY PATH
Question -> /api/embed -> query vector -> vector search
|
v
relevant chunks
|
v
/api/chat -> generation node
|
v
answer + sources
All model calls use one Ollama Herd router URL.
Herd does not replace the document loader or vector database. It routes the model work those components create.
Why embeddings can block chat
A document import may create hundreds or thousands of chunks. Each chunk needs an embedding before it can be stored. When the embedding model and chat model share one constrained Ollama process, ingestion can cause:
- long interactive waits;
- model loading and unloading;
- queue saturation;
- memory pressure from several hot models;
- unpredictable first-token latency;
- failed imports when clients time out.
Separating workloads lets the chat model stay hot while embedding capacity processes the ingestion batch.
Prerequisites
- a working Ollama Herd router;
- at least one chat or generation model in the fleet;
- an embedding model in the fleet;
- a vector store for production use;
- a document parser and chunking strategy.
Verify the fleet:
curl -s http://router-ip:11435/api/tags | python3 -m json.tool
Step 1: choose and freeze the embedding contract
Ollama's current embeddings documentation recommends models such as embeddinggemma, qwen3-embedding, and all-minilm. Ollama Herd can route other compatible embedding models exposed by the fleet as well.
Choose based on:
- retrieval quality on your actual documents and questions;
- language coverage;
- vector dimension;
- maximum useful input length;
- inference speed and memory;
- licensing and deployment constraints.
Then record the contract:
embedding_model: embeddinggemma
embedding_dimension: measured_from_first_response
chunking_version: v1
collection: product_docs_embeddinggemma_v1
distance: cosine
Do not silently switch the embedding model while reusing the old collection. Vectors from different models are not interchangeable.
Step 2: place embedding capacity deliberately
Three common layouts work.
Layout A: one shared Ollama node
Use for a small knowledge base and infrequent imports.
node A: embedding model + chat model
It is simple, but ingestion competes with chat.
Layout B: dedicated embedding node
Use when documents are imported regularly.
node A: chat / reasoning models
node B: embedding model
The embedding node can be a smaller machine because embedding workloads often need less memory than a large generative model.
Layout C: replicated embedding model
Use for high ingestion throughput or availability.
node A: chat models
node B: embedding model
node C: embedding model
Herd can route independent embedding requests across eligible nodes. Measure whether the client sends enough concurrent work to benefit from replication.
Step 3: test one embedding
Ollama-compatible request:
curl -s http://router-ip:11435/api/embed \
-H "Content-Type: application/json" \
-d '{
"model": "embeddinggemma",
"input": "Ollama Herd routes local AI requests across a fleet."
}' | python3 -m json.tool
OpenAI-compatible request:
curl -s http://router-ip:11435/v1/embeddings \
-H "Content-Type: application/json" \
-d '{
"model": "embeddinggemma",
"input": "Ollama Herd routes local AI requests across a fleet."
}' | python3 -m json.tool
Record the returned vector length. Configure the vector-store collection to match it.
Step 4: batch ingestion inputs
Ollama's /api/embed endpoint accepts an array of strings. Batching reduces HTTP overhead and gives the backend more opportunity to process inputs efficiently.
curl -s http://router-ip:11435/api/embed \
-H "Content-Type: application/json" \
-d '{
"model": "embeddinggemma",
"input": [
"First document chunk.",
"Second document chunk.",
"Third document chunk."
]
}' | python3 -m json.tool
Do not create enormous batches without testing. Bound batch size by request-body size, model input limits, latency, and retry cost. A failed batch of 32 chunks is easier to retry than a failed batch of 10,000.
Step 5: build a minimal RAG example
This example uses only the OpenAI Python client and an in-memory list so the routing path is easy to see. Replace the list with a durable vector database in production.
Install:
python3 -m pip install openai
Save as minimal_fleet_rag.py:
from __future__ import annotations
import math
from dataclasses import dataclass
from openai import OpenAI
ROUTER = "http://router-ip:11435/v1"
EMBED_MODEL = "embeddinggemma"
CHAT_MODEL = "llama3.2:3b"
client = OpenAI(base_url=ROUTER, api_key="not-needed")
@dataclass
class Chunk:
source: str
text: str
vector: list[float]
def embed(texts: list[str]) -> list[list[float]]:
response = client.embeddings.create(
model=EMBED_MODEL,
input=texts,
extra_headers={"X-Herd-Tags": "rag,rag-embedding"},
)
return [item.embedding for item in response.data]
def cosine_similarity(a: list[float], b: list[float]) -> float:
dot = sum(x * y for x, y in zip(a, b, strict=True))
norm_a = math.sqrt(sum(x * x for x in a))
norm_b = math.sqrt(sum(y * y for y in b))
if norm_a == 0.0 or norm_b == 0.0:
return 0.0
return dot / (norm_a * norm_b)
def build_index(documents: list[tuple[str, str]]) -> list[Chunk]:
vectors = embed([text for _, text in documents])
return [
Chunk(source=source, text=text, vector=vector)
for (source, text), vector in zip(documents, vectors, strict=True)
]
def search(index: list[Chunk], question: str, limit: int = 3) -> list[Chunk]:
query_vector = embed([question])[0]
ranked = sorted(
index,
key=lambda chunk: cosine_similarity(chunk.vector, query_vector),
reverse=True,
)
return ranked[:limit]
def answer(index: list[Chunk], question: str) -> str:
hits = search(index, question)
context = "\n\n".join(
f"SOURCE: {chunk.source}\n{chunk.text}" for chunk in hits
)
response = client.chat.completions.create(
model=CHAT_MODEL,
messages=[
{
"role": "system",
"content": (
"Answer only from the supplied context. "
"Cite source names in brackets. If the context is insufficient, say so."
),
},
{
"role": "user",
"content": f"CONTEXT:\n{context}\n\nQUESTION:\n{question}",
},
],
extra_headers={"X-Herd-Tags": "rag,rag-generation"},
)
return response.choices[0].message.content or ""
documents = [
(
"routing.md",
"Ollama Herd presents one endpoint and routes requests across eligible local nodes.",
),
(
"queues.md",
"Queue-aware routing helps avoid sending every request to a busy inference node.",
),
(
"security.md",
"The core router should stay on a trusted LAN or private VPN rather than the public internet.",
),
]
index = build_index(documents)
print(answer(index, "Where should the router be exposed?"))
Run it:
python3 minimal_fleet_rag.py
This example intentionally omits production concerns such as durable storage, document updates, chunk IDs, access filters, reranking, and evaluation.
Step 6: add a production vector store
A production collection should store at least:
- vector;
- chunk text or a stable text reference;
- document and chunk IDs;
- source path or URL;
- title and section;
- tenant or access-control metadata;
- content hash;
- embedding model and version;
- chunking version;
- created and updated timestamps.
Use metadata filters before generation so a user never retrieves a document they are not authorized to read.
Popular local options include Qdrant, pgvector, Chroma, and other vector databases. Choose based on durability, filtering, backup, operations, and ecosystem, not just a one-line demo.
Step 7: separate ingestion and query traffic
Tag requests so the dashboard can distinguish the two paths:
rag,rag-ingest
rag,rag-query-embed
rag,rag-generation
Ingestion is usually throughput-oriented. Query embedding is latency-oriented. They can use the same model while still needing different queue and batching policies.
A useful operating policy is:
- ingestion may use larger batches and controlled background concurrency;
- query embeddings stay small and receive interactive priority;
- generation requests use separate model capacity;
- retries preserve stable chunk IDs so duplicate vectors are not inserted.
Optional: native text-embedding backend in source deployments
Current Ollama Herd source includes an optional FastEmbed/ONNX text-embedding service for nomic-embed-text. It runs separately from Ollama, allowing those embedding requests to avoid Ollama's LLM inference slots.
From a source checkout on an embedding node:
git clone https://github.com/geeks-accelerator/ollama-herd.git
cd ollama-herd
uv sync --extra embedding
uv run herd-node
The node starts the sidecar when the optional dependencies are available. Confirm the current installation instructions, model mapping, port, and health checks against the release you publish. Package extras and supported embedding models can change.
This option is most valuable when nomic-embed-text ingestion repeatedly contends with generative Ollama workloads.
Chunking strategy
There is no universal best chunk size. Start from document structure rather than an arbitrary token count.
Good chunks:
- preserve one coherent idea;
- retain useful headings and source metadata;
- are large enough to answer a question;
- are small enough to retrieve precisely;
- overlap only when the boundary would otherwise remove essential context.
Evaluate several strategies with real questions. Retrieval quality should be measured before the retrieved text reaches the LLM.
Evaluate retrieval separately from generation
A polished answer can hide poor retrieval. Build a small test set containing:
- question;
- expected source document;
- relevant passage;
- whether the answer is present;
- access-control expectation.
Measure:
- recall at
k; - precision at
k; - mean reciprocal rank;
- retrieval latency;
- answer faithfulness;
- citation correctness;
- “insufficient context” behavior.
Use the same test set when changing embedding models, chunking, metadata filters, or reranking.
Capacity planning
Embedding nodes
Optimize for:
- batch throughput;
- predictable latency;
- enough memory for the embedding model;
- availability during ingestion windows.
Generation nodes
Optimize for:
- model quality;
- context length;
- time to first token;
- tool or structured-output capability;
- interactive queue latency.
Vector store
Optimize for:
- durable storage and backups;
- metadata-filter performance;
- collection migration;
- tenant isolation;
- observability.
Do not assume that the machine best suited to a large chat model is also the best place for the vector database or embedding service.
Common failure modes
Indexing and querying use different embedding models
Symptoms include irrelevant retrieval or dimension errors. Store the embedding model in collection metadata and enforce it in code.
The embedding dimension changed
Create a new collection or migrate explicitly. Do not write vectors with a new dimension into an old schema.
Document ingestion makes chat unusable
Move embeddings to a dedicated node or sidecar, batch inputs, cap ingestion concurrency, and monitor queues separately.
Retrieval is good but answers hallucinate
Use a stricter system prompt, require source citations, lower generation randomness, and evaluate whether the selected model follows grounding instructions reliably.
Retrieval is bad despite a strong chat model
Fix chunking, metadata, embedding choice, filters, or query formulation. A larger generative model does not repair missing evidence.
Duplicate chunks fill the vector store
Use deterministic document and chunk IDs based on source identity, chunking version, and content hash. Upsert rather than blindly insert.
A model is missing during an import
Check /api/tags before launching a large job and fail fast when the required embedding model is unavailable. Do not let a long import discover the problem after thousands of documents are parsed.
Using the architecture with Open WebUI
Open WebUI handles document uploads, chunking, vector storage, retrieval, and source presentation. Point its Ollama connection to Herd so embedding and chat calls share one fleet URL, then place the embedding model and generation model on appropriate nodes.
Using the architecture with n8n
Use Embeddings Ollama for ingestion and query vectors, a vector-store node such as Qdrant for storage and retrieval, and Ollama Chat Model for the final answer. Configure every Ollama credential with the Herd router URL.
Frequently asked questions
Can Ollama create embeddings for RAG?
Yes. Ollama exposes /api/embed, supports single and batched inputs, and documents that the returned vectors are L2-normalized.
Can one Herd endpoint handle both embeddings and chat?
Yes. Use /api/embed or /v1/embeddings for vectors and /api/chat or /v1/chat/completions for generation. Herd routes each request according to the requested model and eligible capacity.
Should embeddings and chat use the same machine?
They can for small workloads. Separate them when ingestion causes interactive latency, model eviction, or queue contention.
Must I use the same embedding model for documents and questions?
Yes. Index and query vectors must come from the same embedding space. Changing the model normally requires rebuilding or migrating the collection.
Should I use one huge batch?
No. Batch enough to reduce overhead, but keep retries and memory manageable. Measure the model and network rather than copying a fixed batch size.
Does Herd include a vector database?
No. Herd routes model requests. Use a vector store such as Qdrant, pgvector, Chroma, or another system suited to your persistence and filtering needs.
Is RAG automatically private because it is local?
Local processing reduces data sent to external model providers, but privacy still depends on network exposure, logs, backups, vector-store permissions, user authorization, and tool behavior.