Home / Multi-GPU
llama.cpp · homogeneous 2–4 GPUs · not a PC builder
Can this model run across multiple GPUs?
Phase 8 answers memory feasibility for identical cards under llama.cpp. Default split is layer (pipeline parallel: each GPU owns a slice of layers; KV stays with those layers). This is not 2 × VRAM = pool, and it is not a complete PC build.
2× NVIDIA RTX 4090 24 GB
Fits fully across the GPUs
- Model
- Mixtral 8x7B Instruct · Q4 · 8K
- Runtime / split
- llama.cpp ·
layer· calculator 1.0.0 - Single-GPU status
- CPU_RAM_OFFLOAD_REQUIRED · required 31.3 GB vs usable 21.6 GB
- Per card advertised / usable
- 24 GB / 21.6 GB
- Aggregate advertised
- 48.0 GB — capacity, not effective model capacity
- Nominal aggregate usable
- 43.2 GB (N × advertised × 0.90)
- Effective per-GPU peak
- 16.0 GB vs usable 21.6 GB
- Per-GPU split (approx.)
- weights 13.7 GB · KV 0.5 GB · runtime 1.2 GB · safety 0.7 GB
- Why
- Each homogeneous GPU holds about 1/2 of weights and 1/2 of KV under llama.cpp layer split, plus a full per-device runtime/safety reserve. Peak per GPU is within usable VRAM.
- Topology
- Requires space for 2 discrete GPUs. Exact cooler slot width is not in the RigForAI graph. layer split is pipeline parallel and can run over PCIe. KV stays with the layers on each GPU. NVLink is not treated as a 1× aggregate VRAM pool.
- Performance
- Phase 5 predictions are single-GPU. Multi-GPU tok/s is not published.
- GPU-only current cost
- Cost unavailable — no fresh matched Amazon offer for this canonical GPU.
- Formula
- per_gpu_peak = ceil(weight_bytes/N) + ceil(kv_bytes/N) + runtime_base + safety_base + weight_fractions×ceil(weight_bytes/N). Compare to usable = advertised×0.90. Not (single_gpu_required × N) and not advertised_vram × N as effective capacity.
NVIDIA RTX 4090 24 GB · Single-GPU can-run · Find a single GPU · Build a machine for this GPU setup
Query combinations are not indexed. Source: llama.cpp multi-GPU documentation.