Beyond LLMs: Routing Multimodal AI

An AI agent does more than chat. It transcribes meeting audio, embeds documents for retrieval, generates images for content, and reads screenshots. Every one of those workloads has the same shape as LLM inference: it is compute-heavy, it competes for memory, it runs on specific machines, and it benefits from smart routing across a fleet. So Ollama Herd is not just an LLM router. It routes five model types through one endpoint.

The five model types Herd routes

Every one of these is a real, shipped endpoint. Send an image to a chat request and it auto-routes to a vision-capable node; send audio to /api/transcribe and it routes to a node with a speech model. Your client hits localhost:11435 for all of it.

LLMs, embeddings, and vision run on any node, Mac, Linux, or Windows. Speech-to-text and the mflux and DiffusionKit image backends need Apple Silicon, so in a mixed fleet those requests go to the Macs that report them.

The same routing pattern for every modality

What makes this work is that each workload uses the identical four-part pattern Herd built for LLMs:

• Heartbeat capability detection. Each node reports what it can do (which image, ASR, or embedding models it has), so the router knows the fleet's capabilities without any configuration.
• A node-side wrapper that turns a CLI tool (mflux, an ASR binary) into an HTTP endpoint.
• A router endpoint that accepts the request, scores candidates, and proxies to the best node.
• Scoring logic that fits the workload: penalize nodes already generating an image or transcribing, prefer nodes with the memory headroom.

Because the pattern is uniform, you get the same fleet benefits for every modality: auto-discovery instead of hardcoded IPs, score-based load balancing, automatic failover, per-node queuing, and one dashboard that shows every workload.

Cascaded pipelines: chaining modalities across nodes

The payoff is that an agent can chain modalities, and each step routes independently to the node with capacity:

Example fleet: one agent turn, three routed calls

Agent receives a voice messageand calls one endpoint, localhost:11435, three times in sequence

1. POST /api/transcribe, routed to one Mac

Mac minispeech-to-textMacMLX speech model

transcript back to the agent, then 2. POST /api/chat, routed to one node

Linux PCLLM reasoning, any node with the modelOllama

reply back to the agent, then 3. POST /api/generate-image, routed to one node

Mac Studioimage generationMacmflux

image back to the agent

Agent replies with text + imageit never needed to know which machine ran which step

A single machine cannot run all of these at once without resource contention. A fleet of three to five machines with Herd routing can, because each step lands wherever there is headroom, and the agent never knows or cares which machine handled which stage.

Why cascaded beats one big multimodal model, locally

You could instead use a single unified multimodal model that does everything internally (the hosted GPT and Gemini approach). For a local fleet, cascaded specialized models win:

• Swap independently. Upgrade your transcription model without touching your image model.
• Route each stage optimally. Send transcription to the node with the ASR model, image generation to the node with mflux.
• Per-stage visibility. You get real performance data for each modality instead of one opaque number.

On the roadmap

The four-part pattern is the template for workloads not yet shipped. Text-to-speech, OCR as a dedicated endpoint, and video processing follow the same heartbeat-detect, node-wrapper, route-and-score design. They are not in the released build yet; when they land, they route through the same one endpoint as everything else. (For the economic case behind consolidating all of this onto hardware you already own, see the cost breakdown.)

Related Reading

• Local image generation, the image side in depth
• Routing engine, how the scoring works
• Local AI cost, the economics of running it all yourself