LiteLLM vs Ollama, and Where Ollama Herd Fits
Ollama runs models. LiteLLM is a gateway that sends requests to models, Ollama included. They are different layers, so the usual setup is LiteLLM in front of Ollama. Ollama Herd sits between them once you have more than one machine.
Checked October 26, 2026 against LiteLLM v1.106.4, Ollama v0.40.11, and Ollama Herd 0.11.4, using each project's docs and source code.
LiteLLM vs Ollama: what's the difference
Ollama runs models. LiteLLM routes requests to models. Ollama downloads a model and serves it from the machine it is installed on (port 11434), through its own API plus OpenAI-compatible and Anthropic-compatible endpoints. LiteLLM runs no models at all. It is a Python SDK and proxy (port 4000) that puts one OpenAI-style API in front of 100+ providers, Ollama among them, and adds what a shared endpoint needs: virtual keys, budgets, spend tracking, retries, fallbacks, and logging.
| Question | Ollama | LiteLLM |
|---|---|---|
| What it is | Model runtime (inference server) | API gateway (SDK and proxy server) |
| Runs models | Yes, on the machine it is installed on | No. It forwards each request to a backend that does |
| APIs | Ollama API, OpenAI-compatible (/v1/chat/completions, /v1/embeddings, /v1/responses), Anthropic-compatible (/v1/messages) | OpenAI format, whatever the backend |
| Backends | Models it has pulled, plus Ollama's cloud models | 100+ providers: OpenAI, Anthropic, Bedrock, Azure, Vertex, Ollama, vLLM, and more |
| Keys, budgets, spend | No | Yes (virtual keys and spend tracking need a Postgres database) |
| More than one server | No, one server per machine | Yes, it load balances the servers you list in its config |
So the real question is usually not which one, but whether you need the gateway on top. One app calling local models needs only Ollama. Add LiteLLM when several people or apps share the endpoint and need their own keys and budgets, or when you want cloud and local models behind one API.
How to use LiteLLM with Ollama
Give each model a LiteLLM provider prefix and an api_base. Use ollama_chat/, which calls Ollama's /api/chat and is the prefix LiteLLM recommends. The plain ollama/ prefix calls /api/generate; you still need it for embeddings and fill-in-the-middle completions.
# config.yaml
model_list:
- model_name: qwen3
litellm_params:
model: ollama_chat/qwen3:8b
api_base: http://localhost:11434
keep_alive: "8m" # optional, passed through to Ollama
model_info:
supports_function_calling: true
- model_name: embed
litellm_params:
model: ollama/nomic-embed-text
api_base: http://localhost:11434
uv tool install 'litellm[proxy]'
export LITELLM_MASTER_KEY="sk-$(openssl rand -hex 32)"
litellm --config config.yaml # serves on :4000
curl http://localhost:4000/v1/chat/completions \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "qwen3", "messages": [{"role": "user", "content": "Hello"}]}'
Clients now call qwen3 on port 4000 in OpenAI format with that key, and LiteLLM forwards to Ollama. Current LiteLLM proxy releases refuse to start without a master key (LITELLM_MASTER_KEY or general_settings.master_key), so set one even on a laptop; it is also what you issue virtual keys from.
Several Ollama servers behind LiteLLM
LiteLLM load balances any entries that share a model_name. List one entry per machine:
model_list:
- model_name: qwen3
litellm_params:
model: ollama_chat/qwen3:8b
api_base: http://mac-studio.local:11434
- model_name: qwen3
litellm_params:
model: ollama_chat/qwen3:8b
api_base: http://linux-box.local:11434
router_settings:
routing_strategy: least-busy # default is simple-shuffle
num_retries: 2
timeout: 120
The strategies are simple-shuffle (default), least-busy, latency-based-routing, usage-based-routing, and cost-based-routing. A server that keeps failing is put in a short cooldown and skipped. This works well for a few servers that all hold the same models. The limits: every machine and model is a line you maintain by hand, and none of the strategies sees which machine already has the model loaded, how much memory each has free, or how long each queue is.
Where Ollama Herd fits
Ollama Herd does the multi-machine part of that job with machine awareness: it discovers Ollama nodes on your network by mDNS and scores each request against 8 signals (model warmth, memory fit, queue depth, wait time, role affinity, availability, context fit, session affinity). It is not a gateway. It has no virtual keys, budgets, or cloud providers. So the three setups are:
| Setup | Good for | Trade-off |
|---|---|---|
| LiteLLM → one Ollama | A shared endpoint with keys and budgets on one machine | One machine's capacity |
| LiteLLM → several Ollama servers | A few identical servers, models you rarely change | Static list; routing ignores loaded models and free memory |
| LiteLLM → Ollama Herd → your machines | Mixed Macs, Linux servers, and Windows PCs with different models and memory | One more service to run (the Herd router) |
To put Herd behind LiteLLM, give LiteLLM a single entry pointing at the Herd router (port 11435) instead of one per machine. Herd speaks the Ollama API, so the same prefix works:
model_list:
- model_name: local
litellm_params:
model: ollama_chat/qwen3:8b
api_base: http://router-ip:11435
Herd also serves the OpenAI format at /v1/chat/completions, so model: openai/qwen3:8b with api_base: http://router-ip:11435/v1 works too. Herd does not check keys on that route, but LiteLLM's OpenAI client requires a value, so set api_key: none. New machines join the herd without a LiteLLM config change, and Herd returns a 503 when no machine can take the request, which LiteLLM's retries and fallbacks act on.
Tested 2026-10-26: both entries above, through a LiteLLM 1.106.4 proxy, to a Herd 0.11.2 router and node in front of Ollama 0.40.3 (qwen3:8b). The first entry reaches Herd's Ollama API (/api/chat), the second its OpenAI API (/v1/chat/completions).
What is LiteLLM?
LiteLLM (~62K GitHub stars as of October 2026) is an open-source Python SDK and proxy server built by BerriAI. It lets you call 100+ LLM providers (OpenAI, Anthropic, Bedrock, Azure, Vertex, Cohere, and more) through a single OpenAI-compatible interface. LiteLLM handles provider abstraction, API key management, rate limiting, spend tracking, and team governance, making it the de facto standard for cloud LLM API routing.
What is Ollama Herd?
Ollama Herd is an open-source smart AI router that turns the machines you already own (Apple Silicon Macs, Linux servers, and Windows PCs, with or without NVIDIA GPUs) into one endpoint. It routes LLMs, embeddings, image generation, speech-to-text, and vision with 8-signal scoring, mDNS auto-discovery, and a live dashboard in light and dark. Apple Silicon Macs get extras: an MLX backend, native image generation, and speech-to-text. Two commands to set up, zero config files. pip install ollama-herd or brew install ollama-herd.
Ollama Herd vs LiteLLM: feature comparison
| Feature | LiteLLM | Ollama Herd |
|---|---|---|
| Primary function | LLM API gateway (cloud and self-hosted) | Local device fleet router |
| Supported providers | 100+ providers, cloud and self-hosted (including Ollama and vLLM) | Ollama instances on local network |
| Model types | LLMs, embeddings, images, audio (whatever the provider serves) | LLMs, embeddings, image gen, STT |
| API compatibility | OpenAI format | OpenAI, Ollama, Anthropic Messages, and OpenAI Responses formats |
| Discovery | Manual provider config | mDNS auto-discovery (zero config) |
| Routing intelligence | Shuffle, least-busy, latency, usage, or cost routing; retries, cooldowns, fallbacks | 8-signal scoring (model warmth, memory fit, queue depth, wait time, role affinity, availability, context fit, session affinity) |
| Hardware awareness | None (cloud abstraction) | Memory fit, loaded models, queue depth, meeting detection (macOS) |
| Cost tracking | Per-token spend tracking across providers | Free (local inference, no API costs) |
| API key management | Virtual keys, budgets, rotation | Not applicable (no API keys needed) |
| Team management | SSO, RBAC, per-team budgets | Single-user / small-team fleet |
| Guardrails | Content filtering, PII masking | None (local inference, you own the data) |
| Logging/observability | Request logging, Prometheus, custom callbacks | Live dashboard, real-time fleet metrics |
| Caching | Semantic caching | Dynamic context optimization |
| Failover | Automatic provider fallback | Automatic device fallback with re-scoring |
| Deployment | Docker, pip, hosted proxy | pip, Homebrew; runs on macOS, Linux, and Windows |
| Language | Python | Python |
| Cloud dependency | None required (can front only local servers) | None (fully local) |
| Data sovereignty | Stays on your network if every backend is local; cloud backends send it out | Data never leaves your network |
| Test suite | Community tested | 1900+ tests, 45+ health checks |
Where LiteLLM Wins
- Provider breadth. 100+ cloud providers through one API. If you need GPT-4, Claude, Gemini, Mistral, and Bedrock all behind one endpoint, LiteLLM is unmatched.
- Cost management. Per-token spend tracking, budget limits, and cost-based routing. Essential for teams burning through cloud API credits.
- Team governance. Virtual API keys with per-user/per-team budgets, SSO, RBAC. Built for enterprise deployment with dozens of developers.
- API key rotation. Automatic rotation and load balancing across multiple API keys for the same provider, avoiding rate limits.
- Ecosystem maturity. About 62K GitHub stars as of October 2026, battle-tested in production at scale, extensive documentation, large community.
- Guardrails. Content filtering, PII detection, prompt injection defense at the gateway level.
- Hosted option. BerriAI offers a managed proxy so you don't have to self-host.
Where Ollama Herd Wins
- Local device routing. Herd knows what hardware you have, what models are loaded, how much memory each machine has free, and how deep each queue is. LiteLLM has no concept of physical devices.
- 8-signal scoring. Routes based on model warmth (loaded, recently loaded, or cold), memory fit, queue depth, estimated wait, role affinity, availability trend, context fit, and session affinity. LiteLLM routes based on cost, latency, and availability, no hardware signals.
- Zero configuration. mDNS auto-discovery finds every Ollama instance on your network. No config files, no API keys, no provider setup. LiteLLM requires explicit provider configuration.
- Multimodal routing. Natively routes 5 model types (LLMs, embeddings, image gen, STT, vision) to the right hardware. A Mac running mflux gets the image gen work; a Linux box or a MacBook gets the text queries.
- Complete data sovereignty. Nothing leaves your local network. For privacy-sensitive work (legal, medical, financial), this is non-negotiable.
- No ongoing costs. Local inference has zero marginal cost. LiteLLM is free but the cloud APIs it routes to are not.
- Capacity learning. Herd learns actual device throughput over time and improves routing decisions. LiteLLM's latency-based routing tracks response time per deployment, but not what each machine has loaded or how much memory is free.
- Meeting detection (macOS). Automatically de-prioritizes Macs in active video calls. LiteLLM has no awareness of user context.
- Dashboard. A real-time dashboard showing fleet health, routing decisions, model distribution, and device metrics.
The Complementary Story
LiteLLM and Ollama Herd are not competitors, they operate at different layers:
- LiteLLM = API gateway (one OpenAI-format endpoint, keys and budgets, in front of cloud providers and self-hosted servers alike)
- Ollama Herd = local fleet router (routes between your Mac Studio, Linux server, Windows gaming PC, etc.)
They can work together in two ways:
- Herd as a LiteLLM backend. One LiteLLM entry pointing at the Herd router (
ollama_chat/prefix, port 11435) instead of one per machine. Your team gets one gateway for cloud APIs and your own machines. Config above. - Cloud as overflow. Give a cloud deployment and the Herd deployment the same
model_name, withorder: 1on Herd andorder: 2on the cloud model (insidelitellm_params). LiteLLM tries the next order level when Herd returns an error, such as a 503 when no machine can take the request.
When to Choose Each
| Scenario | Choose |
|---|---|
| Need GPT-4, Claude, Gemini behind one API | LiteLLM |
| Need to track cloud API spend across teams | LiteLLM |
| Have several machines (Mac, Linux, Windows) and want to use them all | Ollama Herd |
| Data cannot leave your network | Ollama Herd |
| Want zero inference costs | Ollama Herd |
| Enterprise team with budget governance needs | LiteLLM |
| Personal/small-team local AI setup | Ollama Herd |
| Need multimodal routing (image gen, STT) on local hardware | Ollama Herd |
| Want both cloud and local behind one endpoint | LiteLLM + Ollama Herd together |
Bottom Line
The comparison between LiteLLM and Ollama Herd is mostly a category error. LiteLLM is an API gateway; Herd is a local fleet router. They overlap only in the narrow sense that both route AI requests to backends.
The real question is not "which one?" but "do I need cloud routing, local routing, or both?" For teams with hardware of their own (Macs, Linux boxes, Windows PCs) that want private, free, hardware-aware inference routing, Herd does something LiteLLM fundamentally cannot. For teams that need 100+ cloud providers behind one endpoint, LiteLLM does something Herd has no interest in doing.
The best setup for many teams is both: Herd for local, LiteLLM for cloud, with Herd registered as a LiteLLM backend for seamless hybrid routing.
Getting Started
You can try Ollama Herd alongside LiteLLM without changing your existing cloud setup. Install Herd, point your local apps at it for private inference, and add it to LiteLLM as one ollama_chat/ entry on port 11435 (config above).
pip install ollama-herd
herd # start router
herd-node # on each device
Frequently Asked Questions
What is the difference between LiteLLM and Ollama?
Ollama runs models on the machine it is installed on. LiteLLM runs no models: it is a gateway that gives 100+ providers, Ollama included, one OpenAI-style API and adds virtual keys, budgets, spend tracking, retries, and fallbacks. They are different layers, and LiteLLM is commonly placed in front of Ollama.
How do I use LiteLLM with Ollama?
In LiteLLM's config.yaml, add a model_list entry whose model is ollama_chat/ followed by the Ollama model name, with api_base set to your Ollama server (http://localhost:11434). Start the proxy with litellm --config config.yaml and call it on port 4000. Use the ollama/ prefix for embedding models.
Can LiteLLM load balance several Ollama servers?
Yes. Give several model_list entries the same model_name, one per server, and LiteLLM spreads requests across them using simple-shuffle (default), least-busy, latency-based, usage-based, or cost-based routing. It does not see which server has the model loaded or how much memory is free. Ollama Herd adds that and discovers servers automatically, and can sit behind LiteLLM as one entry.
Is Ollama Herd a good alternative to LiteLLM?
They are complementary rather than competitive. LiteLLM excels at routing between cloud API providers with cost tracking and team governance. Ollama Herd excels at routing between local devices with hardware-aware scoring. If your goal is private local inference across the machines you own (Macs, Linux boxes, Windows PCs) with no per-token billing, Herd is the right tool.
Can I use Ollama Herd with LiteLLM?
Yes. Add the Herd router to LiteLLM as one model entry with the ollama_chat/ prefix and api_base set to the router on port 11435. LiteLLM keeps the keys, budgets, and cloud providers; Herd picks which of your machines serves each local request.
How does Ollama Herd compare to LiteLLM for local inference?
Herd is purpose-built for routing across your own machines: 8-signal scoring that includes model warmth, memory fit, and queue depth, plus mDNS auto-discovery and multimodal routing. LiteLLM can load balance Ollama servers you list, but its routing does not see loaded models or free memory. For several machines with different models and memory, Herd makes better-informed choices; LiteLLM adds the keys, budgets, and cloud providers Herd does not have.
Does Ollama Herd require API keys?
No. Ollama Herd routes to local Ollama instances on your network. There are no API keys, no provider configuration, and no cloud accounts needed. Everything runs on hardware you own.
Is Ollama Herd free?
Yes. Ollama Herd is open-source under the MIT license. No paid tiers, no API keys, no subscriptions.
See Also
- Ollama Herd vs Bifrost, High-performance Go-based LLM API gateway
- Ollama Herd vs Envoy AI Gateway, Envoy-based AI traffic management
- Ollama Herd vs cloud APIs, Local fleet routing vs pay-per-token cloud inference