Ollama Across Mixed GPUs

NVIDIA cards, AMD cards and APUs, and Macs behind one endpoint, with models placed by the GPU memory each machine actually has.

The short answer

Yes, one Ollama endpoint can span NVIDIA cards, AMD cards and APUs, and Macs at once. Ollama Herd is free and open source: install it on each machine, run herd on one and herd-node on the rest, and every model on every machine answers at http://router:11435. Since v0.12.9 a Linux node reads its GPU memory by itself (NVIDIA through nvidia-smi, AMD from the amdgpu driver), and the router uses it to decide where each model fits and which machine should serve it. No model map, no per-card config.

pip install ollama-herd
herd            # on one machine: the router
herd-node       # on every machine running Ollama

What Herd reads on a GPU machine

Routing decisions on a mixed fleet come down to memory: whether a model fits on a card, and how much room it leaves. That is what Herd reads.

Herd readsUsed for
Free RAM plus free GPU memory the OS does not count as RAMWhether a model can load on this machine at all (v0.12.9)
Free GPU memory alone, on a machine with a GPUMemory-fit scoring: an empty card beats a full one (v0.12.10)
How much of each loaded model Ollama put in GPU memory (size_vram)What a load would evict, and whether a model is running partly on the CPU (v0.12.12)
The card’s name, for a handful of NVIDIA cards in its bandwidth tableSteering large models to faster hardware

Herd does not read GPU utilization or GPU load, and does not route by them. Load comes from its own queues: how many requests each machine is serving and how long that model usually takes there. A machine with two cards counts their memory together.

NVIDIA on Linux

Install the NVIDIA driver so nvidia-smi works, run Ollama as usual, then herd-node. The node’s card on the dashboard shows "GPU … free of …" once it has found the memory. Detection uses each card’s measured use, so other programs on the GPU count too. The project checked this on an NVIDIA Quadro RTX 6000 with 54 GB of RAM: a 57 GB model that v0.12.8 refused was routed and ran split across RAM and the card, and the detected use matched nvidia-smi and Ollama.

  • Parallel requests: if you raised OLLAMA_NUM_PARALLEL in Ollama’s systemd unit (Ollama’s documented Linux method), the node reads it from there since v0.12.10. See Ollama concurrent requests for choosing a value.
  • In a container: run the node with --network host and --gpus all, or the card’s memory is invisible to it. The deployment guide has the commands.
  • DGX Spark and other unified-memory parts report no separate GPU memory, which is correct: there, RAM is the GPU’s memory.

AMD cards and APUs

On AMD, nothing extra is needed: the node reads the amdgpu driver’s mem_info_vram_* files under /sys, which works inside a container too. On a discrete card that is its VRAM; on an APU it is the BIOS carve-out, the "UMA frame buffer" the firmware withholds from Linux. AMD’s GTT pool is never counted, because it is ordinary RAM the GPU can map and is already in Linux’s figures.

This path reads the kernel’s documented files and is covered by unit tests, but as of v0.12.12 the project has not yet watched it on a real AMD machine. If you run Herd on an AMD box, check the dashboard’s GPU line against grep . /sys/class/drm/card*/device/mem_info_vram_total and tell us what you see, matching or not. If it reads wrong, the next section’s setting corrects it.

Strix Halo (Ryzen AI Max) and other large carve-outs

Ryzen AI Max machines (GMKtec EVO-X2 and EVO-X5 Pro, Framework Desktop) can give most of their memory to the GPU in the BIOS. An EVO-X5 Pro with 160 GB of its 192 GB given to the GPU shows Linux about 30 GB, and up to v0.12.8 Herd judged that machine on the 30 GB and refused bigger models that Ollama runs fine there.

Since v0.12.9 the node detects the carve-out itself. When it can’t, or reads it wrong, declare it in GB and restart the node:

free -g
grep . /sys/class/drm/card*/device/mem_info_vram_total   # bytes

FLEET_NODE_GPU_MEMORY_GB=160 herd-node

What the setting does, exactly: the router adds that memory, less what loaded models already hold there, to the machine’s free RAM when it decides whether a model fits. RAM figures and usage stats still report RAM. Don’t set it where GPU memory is already part of RAM (Apple Silicon, AMD GTT, DGX Spark), or it is counted twice; 0 turns detection off. The original report of full memory not being visible on these machines (issue #9) is still open, so if you have one, a report on that issue of what Herd sees is the most useful thing you can send.

How Herd places models on mixed cards

Different card sizes are where a plain load balancer goes wrong: it sends work to a card that can’t hold the model, or swaps two models on one card while another sits empty. These rules, all in v0.12.12, are what keep a mixed fleet settled:

  • Room first. A model that isn’t loaded goes where it fits in free GPU memory without evicting anything, so two models that can’t share a card settle on different machines (v0.12.10).
  • Evictions cost. A load that would push out a resident model is charged for it, more for a busy one, and a model that only one machine can hold is not evicted for one the others could take (v0.12.10, v0.12.11).
  • Partly on the CPU counts as cold. When Ollama could only partly fit a model on a card, it runs partly on the CPU, so it scores like a cold load and a fully resident copy elsewhere wins (v0.12.12).
  • Context per machine. Ollama sizes a model’s default context by GPU memory, so the same model can take more memory on a 48 GB card than a 24 GB one; Herd learns that per node (v0.12.10).

The routing engine guide has every signal and its points.

What we measured

We ran these rules on rented NVIDIA machines with cards of different sizes, against a plain load balancer. The fleet benchmark has the numbers and the raw data: against a plain load balancer, two models that can’t share one GPU, one big card or two small ones, and coding, embeddings and image questions at once. Running AMD, or a mix we haven’t tried? The same script runs on your machines; run it yourself and send us the results.

Macs and Windows PCs in the same fleet

Macs join the same router. Apple Silicon has unified memory, so free RAM is the GPU’s memory and nothing extra is read or needed; Macs also bring the extras that need Apple Silicon (MLX, image generation, speech-to-text), and requests for those go to a Mac. A Windows node runs the same herd-node (v0.11.3 or later), but its GPU memory is not detected: set FLEET_NODE_GPU_MEMORY_GB on it so the router counts the card when it checks a model fits.

Try it on your own machines

Ollama Herd is free and open source. Two commands get you a fleet, and the dashboard at http://router:11435/dashboard shows each machine’s GPU memory as soon as its node connects. Start with the quickstart, or download it now.

Frequently asked questions

Can Ollama Herd route across NVIDIA and AMD GPUs at once?

Yes. Each Linux machine runs herd-node next to its Ollama, and the router treats them as one fleet. NVIDIA memory is read through nvidia-smi and AMD memory from the amdgpu driver, both since v0.12.9, so the router knows where each model fits. Macs and Windows PCs can join the same router.

Does Ollama Herd use GPU utilization to route requests?

No. Herd reads GPU memory, never GPU utilization or load. It uses free RAM plus free GPU memory to decide whether a model can load on a machine, and free GPU memory to score how comfortably it fits. Load comes from Herd’s own queues and each model’s measured latency on each machine.

Why does Herd refuse a model on my Ryzen AI Max machine?

Probably because the BIOS gives most of the memory to the GPU and Linux sees only the rest. Since v0.12.9 the node detects that carve-out itself; if the dashboard shows no GPU line or a wrong figure, set FLEET_NODE_GPU_MEMORY_GB on the node to the carve-out in GB and restart herd-node. Issue #9 on GitHub tracks these machines.

Do I need Docker to run Herd on a GPU server?

No Docker is required: pip install ollama-herd works on Linux, macOS and Windows. If you prefer containers on Linux, the official image runs the router and Linux nodes with --network host, plus --gpus all on NVIDIA.