Beyond LLMs: Routing Multimodal AI
LLM inference is one piece of what agents do. They also transcribe audio, embed documents, generate images, and read pictures. Ollama Herd routes all five model types across your fleet from a single endpoint.
An AI agent does more than chat. It transcribes meeting audio, embeds documents for retrieval, generates images for content, and reads screenshots. Every one of those workloads has the same shape as LLM inference: it is compute-heavy, it competes for memory, it runs on specific machines, and it benefits from smart routing across a fleet. So Ollama Herd is not just an LLM router. It routes five model types through one endpoint.
The five model types Herd routes
| Modality | Endpoint | What runs |
|---|---|---|
| LLMs | /api/chat, /v1/chat/completions, /v1/messages, /v1/responses | Chat and completions, plus native Claude Code and Codex protocols |
| Embeddings | /api/embed, /v1/embeddings, /api/embed-image | Text embeddings (fastembed) and image embeddings (DINOv2, SigLIP, CLIP) |
| Image generation | /api/generate-image, /v1/images/generations | Flux via mflux and Stable Diffusion 3 via DiffusionKit (Apple Silicon), plus Ollama native image models wherever Ollama runs them |
| Speech-to-text | /api/transcribe | Audio transcription on Apple Silicon nodes running an MLX ASR model |
| Vision | /api/chat with images | Image understanding, OCR, and scene analysis via vision-capable models |
Every one of these is a real, shipped endpoint. Send an image to a chat request and it auto-routes to a vision-capable node; send audio to /api/transcribe and it routes to a node with a speech model. Your client hits localhost:11435 for all of it.
LLMs, embeddings, and vision run on any node, Mac, Linux, or Windows. Speech-to-text and the mflux and DiffusionKit image backends need Apple Silicon, so in a mixed fleet those requests go to the Macs that report them.
The same routing pattern for every modality
What makes this work is that each workload uses the identical four-part pattern Herd built for LLMs:
- Heartbeat capability detection. Each node reports what it can do (which image, ASR, or embedding models it has), so the router knows the fleet's capabilities without any configuration.
- A node-side wrapper that turns a CLI tool (mflux, an ASR binary) into an HTTP endpoint.
- A router endpoint that accepts the request, scores candidates, and proxies to the best node.
- Scoring logic that fits the workload: penalize nodes already generating an image or transcribing, prefer nodes with the memory headroom.
Because the pattern is uniform, you get the same fleet benefits for every modality: auto-discovery instead of hardcoded IPs, score-based load balancing, automatic failover, per-node queuing, and one dashboard that shows every workload.
Cascaded pipelines: chaining modalities across nodes
The payoff is that an agent can chain modalities, and each step routes independently to the node with capacity:
localhost:11435, three times in sequencePOST /api/transcribe, routed to one MacPOST /api/chat, routed to one nodePOST /api/generate-image, routed to one nodeA single machine cannot run all of these at once without resource contention. A fleet of three to five machines with Herd routing can, because each step lands wherever there is headroom, and the agent never knows or cares which machine handled which stage.
Why cascaded beats one big multimodal model, locally
You could instead use a single unified multimodal model that does everything internally (the hosted GPT and Gemini approach). For a local fleet, cascaded specialized models win:
- Swap independently. Upgrade your transcription model without touching your image model.
- Route each stage optimally. Send transcription to the node with the ASR model, image generation to the node with mflux.
- Per-stage visibility. You get real performance data for each modality instead of one opaque number.
On the roadmap
The four-part pattern is the template for workloads not yet shipped. Text-to-speech, OCR as a dedicated endpoint, and video processing follow the same heartbeat-detect, node-wrapper, route-and-score design. They are not in the released build yet; when they land, they route through the same one endpoint as everything else. (For the economic case behind consolidating all of this onto hardware you already own, see the cost breakdown.)
Frequently asked questions
Does Ollama Herd only route LLMs?
No. It routes five model types through one endpoint: LLMs (/api/chat, /v1/chat/completions, /v1/messages for Claude Code, /v1/responses for Codex), text and image embeddings (/api/embed, /v1/embeddings, /api/embed-image), image generation (/api/generate-image, /v1/images/generations), speech-to-text (/api/transcribe), and vision through chat requests with images. Each request is scored and proxied to the node best suited to it.
Which model types need a Mac in a mixed fleet?
Speech-to-text and the mflux and DiffusionKit image backends need Apple Silicon, so in a mixed fleet those requests go to the Macs that report them. LLMs, embeddings and vision run on any node, Mac, Linux or Windows, and Ollama's native image models run wherever Ollama runs them.
How do I send an image to a vision model through Ollama Herd?
Send it in an ordinary /api/chat request. Herd sees the image and auto-routes the request to a node with a vision-capable model, which handles image understanding, OCR and scene analysis.
Can I transcribe audio across a fleet of Macs?
Yes. POST an audio file to /api/transcribe and Herd routes it to an Apple Silicon node running an MLX speech-to-text model. Scoring penalizes nodes that are already transcribing, so batch jobs spread across the fleet instead of queueing on one machine.
What is a cascaded multimodal pipeline?
It chains separate specialized models, one per modality, instead of using one all-in-one model. An agent might transcribe a voice note, reason over it with an LLM, then generate an image, with each step routed on its own to the node with capacity. For a local fleet this wins because you can swap each model independently, route each stage to the right node, and see performance per stage.
Does Ollama Herd route text-to-speech or video?
Not yet. Text-to-speech, OCR as a dedicated endpoint, and video processing are on the roadmap and not in the released build. When they land they will follow the same pattern and route through the same endpoint as the other model types.
Related Reading
- Local image generation, the image side in depth
- Routing engine, how the scoring works
- Local AI cost, the economics of running it all yourself