How Much VRAM Do I Need for Local AI?

The short answer: VRAM needs depend on more than the model’s parameter count. You need enough memory for the model weights, the KV cache created by your context, and the runtime overhead required by your inference software.

A rough first estimate is:

Required VRAM ≈ quantized model weights
               + KV cache
               + runtime overhead
               + headroom

For a small single-user language model, the weights are usually the largest component. With long contexts, large batches, multimodal inputs, or fine-tuning, the KV cache and runtime memory can become just as important—or larger.

The four VRAM buckets that matter

1. Model weights

Model weights are the learned parameters loaded into memory for inference. A model described as “7B” has approximately seven billion parameters; “70B” has approximately 70 billion.

The basic estimate is:

Weight memory ≈ number of parameters × bytes per parameter

Common weight-storage assumptions are:

Weight format	Approximate bytes per parameter	Typical use
FP32	4 bytes	High-precision workloads; rarely practical for large local models
FP16 or BF16	2 bytes	Higher-quality inference when memory allows
8-bit	1 byte	A compromise between memory use and quality
4-bit	0.5 bytes	Common for fitting larger language models on consumer GPUs

These are arithmetic estimates, not exact model-file sizes. Quantized formats include scales, metadata, and format-specific overhead. The model’s architecture and vocabulary also affect the final size.

For example, a 7B model has an approximate raw weight footprint of:

FP16:  7 billion × 2 bytes  ≈ 14 GB
8-bit: 7 billion × 1 byte   ≈ 7 GB
4-bit: 7 billion × 0.5 byte ≈ 3.5 GB

A real 4-bit model file will generally not be exactly 3.5 GB, and loading it also requires memory beyond the file itself.

2. KV cache

The KV cache stores key and value tensors from the current conversation or sequence. It allows the model to reuse previous attention calculations instead of recomputing the entire context for every new token.

KV-cache memory grows with:

• Context length
• Number of layers
• Number of key/value heads
• Head dimension
• KV-cache data type
• Batch size or number of simultaneous sequences

A simplified estimate for a decoder-only transformer is:

KV cache ≈ 2 × layers × KV heads × head dimension
           × tokens × bytes per KV value

The factor of two represents the key and value tensors.

Grouped-query attention can substantially reduce KV memory because the number of KV heads is lower than the number of attention heads. This is why two models with similar parameter counts can have different context-memory requirements.

Worked KV-cache example

Suppose a model uses:

• 32 layers
• 8 KV heads
• 128 dimensions per head
• A 4,096-token context
• FP16 KV values, or 2 bytes per value

Then:

2 × 32 × 8 × 128 × 4,096 × 2 bytes
≈ 537 million bytes
≈ 0.5 GiB

At 8,192 tokens, the same illustrative KV cache would be approximately 1 GiB. Running two sequences at once would roughly double that component.

This is an example of the calculation, not a claim about every 32-layer model. To estimate a specific model accurately, you need its architecture and the settings used by your inference engine.

3. Runtime overhead

Inference software needs memory in addition to the weights and KV cache. Runtime allocations can include:

• Temporary activations
• CUDA or GPU kernels
• Attention and matrix-multiplication workspaces
• Memory used by the model loader
• Buffers for prompt processing and token generation
• CUDA graphs or other execution optimizations
• Allocator fragmentation

Runtime overhead varies by framework, backend, quantization format, context length, batch size, and enabled features. It is not safe to treat a model file’s size as the amount of VRAM required.

4. Headroom

A GPU should not normally be planned around its exact advertised VRAM capacity. The operating system, display server, other applications, and the inference runtime may already consume part of the available memory.

Headroom also helps prevent failures when:

• A prompt is longer than expected
• The context window expands during a conversation
• You increase batch size
• You load an adapter or multimodal projector
• The software reserves memory differently than expected

A useful planning rule is to leave several gigabytes free for smaller models and a larger percentage of capacity for borderline fits. This is a rule of thumb, not a universal requirement. The closer your estimate is to the GPU’s limit, the more likely you are to need lower context, a smaller quantization, CPU offload, or a different model.

A practical VRAM estimation workflow

Use this process before choosing a GPU.

Step 1: Identify the exact model and parameter count

“A 7B model” is only a starting point. Record:

• Exact model family and variant
• Parameter count
• Whether it is dense or a mixture-of-experts model
• Whether you need text-only or multimodal inference
• Intended context length
• Number of concurrent users or sequences

For mixture-of-experts models, the active parameters per token and total stored parameters are different. VRAM generally needs to account for the weights that must be loaded, not only the parameters active in one token calculation.

Step 2: Choose the weight format

Estimate the weights using:

Weight memory ≈ parameter count × quantization bytes per parameter

Start with the format you actually intend to run. A 4-bit build and an FP16 build of the same model can have very different hardware requirements.

Quantization can reduce memory substantially, but it may introduce some quality loss and can affect supported operations or performance. The lowest-bit option is not automatically the best choice if you have enough memory for a higher-quality format.

Step 3: Estimate the KV cache for your target context

Do not use the model’s maximum advertised context automatically. Decide how much context you really need.

A 4K-token context, an 8K-token context, and a very long context can have meaningfully different memory requirements. If you need multiple simultaneous conversations, multiply the active KV-cache estimate by the number of sequences, subject to how the runtime manages shared or paged caches.

Step 4: Add runtime overhead and headroom

Add a practical allowance for the inference engine, temporary buffers, and memory variation. The exact amount depends on the software stack, so treat this as an estimate rather than a fixed specification.

A simple planning version is:

Planning VRAM =
    estimated weight memory
  + estimated KV-cache memory
  + runtime allowance
  + safety headroom

Step 5: Compare the result with usable VRAM

Compare your estimate with the memory actually available to the inference workload, not just the GPU’s advertised capacity.

If the estimate is close to the limit, consider:

• A smaller quantized model
• A shorter context
• Fewer simultaneous sequences
• KV-cache quantization, if supported by your software
• CPU or system-memory offload
• A GPU with more VRAM

Offloading can make a model load when it would not fit entirely in VRAM, but it usually changes the performance trade-off. Data moving between system memory and the GPU is slower than keeping the working set on the GPU.

Example weight estimates by model size

The table below shows raw arithmetic for common parameter counts. It does not include KV cache, runtime memory, or headroom.

Model size	FP16 weights	8-bit weights	4-bit weights
7B	≈14 GB	≈7 GB	≈3.5 GB
13B	≈26 GB	≈13 GB	≈6.5 GB
34B	≈68 GB	≈34 GB	≈17 GB
70B	≈140 GB	≈70 GB	≈35 GB

These figures are useful for rejecting obviously unsuitable GPUs, but they are not purchase recommendations by themselves.

For example, a 13B model in a 4-bit format may have an arithmetic weight estimate around 6.5 GB. That does not mean every GPU with 6.5 GB of VRAM will run it comfortably. You still need memory for the quantization metadata, KV cache, runtime, and your desired context.

Conversely, a model may run with partial CPU offload on a GPU with less VRAM than its total weight footprint. That can be useful for experimentation, but it is a different experience from a fully GPU-resident model.

How context length changes the answer

Context length is one of the most frequently overlooked variables in local AI hardware planning.

The weights are loaded once, but the KV cache grows as the active sequence grows. If you use a long document, a large conversation history, retrieval results, or multiple parallel requests, the memory requirement rises.

This creates two different questions:

1. Can the model weights fit?
2. Can the model run at the desired context and concurrency?

A GPU may answer “yes” to the first and “no” to the second. Always specify the context length and number of simultaneous sequences when comparing systems.

What if the model does not fit?

You have several options, each with a trade-off.

Use a lower-bit quantization

Moving from FP16 to 8-bit or 4-bit weights reduces memory substantially. The trade-off is potential quality loss, different software support, and sometimes different performance.

Reduce the context length

A shorter context reduces KV-cache usage. This is often the simplest way to make a borderline model fit, but it limits how much conversation or source material the model can consider at once.

Reduce concurrency or batch size

Multiple active sequences consume more cache and working memory. A configuration designed for one interactive user may not fit the same way when serving several requests.

Use CPU offload

Offloading some weights or data to system RAM can let you run a larger model with limited VRAM. Expect lower or less predictable performance compared with a fully GPU-resident configuration.

Choose a smaller model

A smaller model with a higher-quality quantization can be a better practical choice than a larger model that barely fits. Usable speed, context capacity, and reliability matter more than model size alone.

VRAM versus system RAM

VRAM is the main constraint for GPU-resident inference, but system RAM still matters.

You may need system RAM for:

• CPU-offloaded weights
• Model loading and conversion
• Tokenizers and application processes
• Retrieval databases and document processing
• Operating-system and desktop use

A machine can have enough total system memory to hold a model while still lacking enough VRAM for fast GPU inference. Treat VRAM and RAM as separate resources rather than interchangeable specifications.

Use the model-to-GPU calculator before buying

Once you know the model, quantization, context length, and concurrency you want, use the RigForAI Find a GPU tool to narrow hardware options.

The calculator is most useful when you enter a realistic workload instead of asking whether a model “fits” in the abstract. Check both the estimated fit and the remaining headroom. If your workload is borderline, compare the result with a lower context, a smaller quantization, or a GPU with more VRAM.

A compact checklist

Before buying hardware for local AI, confirm:

• Exact model and parameter count
• Weight format: FP16, 8-bit, 4-bit, or another quantization
• Target context length
• Number of concurrent sequences
• KV-cache data type and whether cache quantization is supported
• Runtime and framework overhead
• Whether multimodal components or adapters are required
• Available VRAM after display and other applications
• Whether CPU offload is acceptable
• Enough headroom for prompts longer than the minimum case

The most reliable purchase decision is not based on parameter count alone. Estimate the weights, account for the KV cache, add runtime memory and headroom, then choose a GPU that fits the workload you actually plan to run.