Beyond LLMs: Routing Multimodal AI

LLM inference is one piece of what agents do. They also transcribe audio, embed documents, generate images, and read pictures. Ollama Herd routes all five model types across your fleet from a single endpoint.

An AI agent does more than chat. It transcribes meeting audio, embeds documents for retrieval, generates images for content, and reads screenshots. Every one of those workloads has the same shape as LLM inference: it is compute-heavy, it competes for memory, it runs on specific machines, and it benefits from smart routing across a fleet. So Ollama Herd is not just an LLM router. It routes five model types through one endpoint.

The five model types Herd routes

ModalityEndpointWhat runs
LLMs/api/chat, /v1/chat/completions, /v1/messages, /v1/responsesChat and completions, plus native Claude Code and Codex protocols
Embeddings/api/embed, /v1/embeddings, /api/embed-imageText embeddings (fastembed) and image embeddings (DINOv2, SigLIP, CLIP)
Image generation/api/generate-image, /v1/images/generationsFlux via mflux and Stable Diffusion 3 via DiffusionKit (Apple Silicon), plus Ollama native image models wherever Ollama runs them
Speech-to-text/api/transcribeAudio transcription on Apple Silicon nodes running an MLX ASR model
Vision/api/chat with imagesImage understanding, OCR, and scene analysis via vision-capable models

Every one of these is a real, shipped endpoint. Send an image to a chat request and it auto-routes to a vision-capable node; send audio to /api/transcribe and it routes to a node with a speech model. Your client hits localhost:11435 for all of it.

LLMs, embeddings, and vision run on any node, Mac, Linux, or Windows. Speech-to-text and the mflux and DiffusionKit image backends need Apple Silicon, so in a mixed fleet those requests go to the Macs that report them.

The same routing pattern for every modality

What makes this work is that each workload uses the identical four-part pattern Herd built for LLMs:

  • Heartbeat capability detection. Each node reports what it can do (which image, ASR, or embedding models it has), so the router knows the fleet's capabilities without any configuration.
  • A node-side wrapper that turns a CLI tool (mflux, an ASR binary) into an HTTP endpoint.
  • A router endpoint that accepts the request, scores candidates, and proxies to the best node.
  • Scoring logic that fits the workload: penalize nodes already generating an image or transcribing, prefer nodes with the memory headroom.

Because the pattern is uniform, you get the same fleet benefits for every modality: auto-discovery instead of hardcoded IPs, score-based load balancing, automatic failover, per-node queuing, and one dashboard that shows every workload.

Cascaded pipelines: chaining modalities across nodes

The payoff is that an agent can chain modalities, and each step routes independently to the node with capacity:

Example fleet: one agent turn, three routed calls
Agent receives a voice messageand calls one endpoint, localhost:11435, three times in sequence
1. POST /api/transcribe, routed to one Mac
Mac minispeech-to-textMacMLX speech model
transcript back to the agent, then 2. POST /api/chat, routed to one node
Linux PCLLM reasoning, any node with the modelOllama
reply back to the agent, then 3. POST /api/generate-image, routed to one node
Mac Studioimage generationMacmflux
image back to the agent
Agent replies with text + imageit never needed to know which machine ran which step
Each step of a cascaded pipeline is a separate request that Herd routes on its own: speech-to-text and mflux image generation go to Macs that report them, while the LLM step can land on any node with the model.

A single machine cannot run all of these at once without resource contention. A fleet of three to five machines with Herd routing can, because each step lands wherever there is headroom, and the agent never knows or cares which machine handled which stage.

Why cascaded beats one big multimodal model, locally

You could instead use a single unified multimodal model that does everything internally (the hosted GPT and Gemini approach). For a local fleet, cascaded specialized models win:

  • Swap independently. Upgrade your transcription model without touching your image model.
  • Route each stage optimally. Send transcription to the node with the ASR model, image generation to the node with mflux.
  • Per-stage visibility. You get real performance data for each modality instead of one opaque number.

On the roadmap

The four-part pattern is the template for workloads not yet shipped. Text-to-speech, OCR as a dedicated endpoint, and video processing follow the same heartbeat-detect, node-wrapper, route-and-score design. They are not in the released build yet; when they land, they route through the same one endpoint as everything else. (For the economic case behind consolidating all of this onto hardware you already own, see the cost breakdown.)

Frequently asked questions

Does Ollama Herd only route LLMs?

No. It routes five model types through one endpoint: LLMs (/api/chat, /v1/chat/completions, /v1/messages for Claude Code, /v1/responses for Codex), text and image embeddings (/api/embed, /v1/embeddings, /api/embed-image), image generation (/api/generate-image, /v1/images/generations), speech-to-text (/api/transcribe), and vision through chat requests with images. Each request is scored and proxied to the node best suited to it.

Which model types need a Mac in a mixed fleet?

Speech-to-text and the mflux and DiffusionKit image backends need Apple Silicon, so in a mixed fleet those requests go to the Macs that report them. LLMs, embeddings and vision run on any node, Mac, Linux or Windows, and Ollama's native image models run wherever Ollama runs them.

How do I send an image to a vision model through Ollama Herd?

Send it in an ordinary /api/chat request. Herd sees the image and auto-routes the request to a node with a vision-capable model, which handles image understanding, OCR and scene analysis.

Can I transcribe audio across a fleet of Macs?

Yes. POST an audio file to /api/transcribe and Herd routes it to an Apple Silicon node running an MLX speech-to-text model. Scoring penalizes nodes that are already transcribing, so batch jobs spread across the fleet instead of queueing on one machine.

What is a cascaded multimodal pipeline?

It chains separate specialized models, one per modality, instead of using one all-in-one model. An agent might transcribe a voice note, reason over it with an LLM, then generate an image, with each step routed on its own to the node with capacity. For a local fleet this wins because you can swap each model independently, route each stage to the right node, and see performance per stage.

Does Ollama Herd route text-to-speech or video?

Not yet. Text-to-speech, OCR as a dedicated endpoint, and video processing are on the roadmap and not in the released build. When they land they will follow the same pattern and route through the same endpoint as the other model types.