Home / Methodology / Performance
How performance numbers are produced
Memory fit answers whether a model fits in usable VRAM. Performance answers how fast llama.cpp is likely to run a short-context, batch-1 decode on that GPU. The two facts are independent. A larger VRAM card is not assumed to be faster.
Runtime
Phase 5 publishes llama.cpp with the CUDA backend only. Ollama, vLLM, ExLlamaV2, MLX and LM Studio are not assigned tok/s values in this version. Runtime changes the number; we do not invent a universal GPU speed.
Provenance states
- MEASURED
- A llama-bench observation. No formula produced the displayed value. The v1 scoreboard is Llama 2 7B Q4_0, pp512 (prefill) and tg128 (decode), full GPU offload, from the official llama.cpp CUDA discussion.
- CALIBRATED
- A deterministic prediction whose GPU efficiency was taken from a measured llama-bench row on the same GPU, for a dense model whose parameter count is within 2× of Llama 2 7B.
- ESTIMATED
- The same formula, but the model is farther from the measured reference (parameter count, MoE, or GPU efficiency taken from an architecture median rather than that exact GPU).
We never relabel an estimate as measured because it looks realistic.
Decode formula
At batch 1, decode is treated as memory-bandwidth bound: each generated token loads the quantized weights.
decode tok/s ≈ GPU_efficiency × memory_bandwidth / weight_bytes
GPU_efficiency is measured_tg128 × Llama-2-7B-Q4_0 weight / that GPU’s advertised peak bandwidth. Weight bytes come from the existing compatibility calculator (not a second VRAM model). For MoE models the active parameter count is used; total parameter count is not treated as the per-token load.
Prefill and time to first token
Prefill uses llama-bench pp512. When we only have a Llama 2 7B measurement, prefill is scaled by parameter count (compute-bound assumption). Time to first token is shown only as 512 / prefill tok/s. We do not fabricate 8K or 128K TTFT.
Confidence
HIGH: measured, or calibrated with a close parameter count on the same GPU. MEDIUM: calibrated otherwise, or estimated with a same-GPU efficiency and moderate size gap. LOW: architecture-median efficiency or a large size gap. LOW estimates are usually not published.
What this version cannot claim
- Speed at 8K–128K context
- AMD ROCm / Vulkan numbers (the scoreboard is CUDA)
- vLLM, Ollama, or other runtimes
- Multi-GPU or offloaded layers
- “Fastest GPU” or value rankings