Home / Multi-GPU

llama.cpp · homogeneous 2–4 GPUs · not a PC builder

Can this model run across multiple GPUs?

Phase 8 answers memory feasibility for identical cards under llama.cpp. Default split is layer (pipeline parallel: each GPU owns a slice of layers; KV stays with those layers). This is not 2 × VRAM = pool, and it is not a complete PC build.

2× NVIDIA GTX 1080 Ti 11 GB

Fits fully across the GPUs

Model
Mistral Small 3.2 24B Instruct · Q4 · 8K
Runtime / split
llama.cpp · layer · calculator 1.0.0
Single-GPU status
CPU_RAM_OFFLOAD_REQUIRED · required 17.2 GB vs usable 9.9 GB
Per card advertised / usable
11 GB / 9.9 GB
Aggregate advertised
22.0 GB — capacity, not effective model capacity
Nominal aggregate usable
19.8 GB (N × advertised × 0.90)
Effective per-GPU peak
9.0 GB vs usable 9.9 GB
Per-GPU split (approx.)
weights 7.0 GB · KV 0.6 GB · runtime 0.9 GB · safety 0.5 GB
Why
Each homogeneous GPU holds about 1/2 of weights and 1/2 of KV under llama.cpp layer split, plus a full per-device runtime/safety reserve. Peak per GPU is within usable VRAM.
Topology
Requires space for 2 discrete GPUs. Exact cooler slot width is not in the RigForAI graph. layer split is pipeline parallel and can run over PCIe. KV stays with the layers on each GPU. NVLink is not treated as a 1× aggregate VRAM pool.
Performance
Phase 5 predictions are single-GPU. Multi-GPU tok/s is not published.
GPU-only current cost
Cost unavailable — no fresh matched Amazon offer for this canonical GPU.
Amazon
Formula
per_gpu_peak = ceil(weight_bytes/N) + ceil(kv_bytes/N) + runtime_base + safety_base + weight_fractions×ceil(weight_bytes/N). Compare to usable = advertised×0.90. Not (single_gpu_required × N) and not advertised_vram × N as effective capacity.

NVIDIA GTX 1080 Ti 11 GB · Single-GPU can-run · Find a single GPU · Build a machine for this GPU setup

Query combinations are not indexed. Source: llama.cpp multi-GPU documentation.