What AI Models Can You Run on a 8GB GPU?

An 8GB GPU is practical for many small and medium local AI models, but it is not enough to judge compatibility by parameter count alone.

As a realistic starting point:

• 1B–4B language models: usually comfortable, including higher-precision formats.
• 7B–8B models: commonly usable in 4-bit quantization, with moderate context lengths.
• 9B–13B models: sometimes possible in 4-bit, but often require reduced context, careful settings, or CPU/RAM offload.
• Larger models: generally require substantial offload, multiple GPUs, or CPU-only inference.

These are planning ranges, not guarantees. The exact result depends on the model architecture, quantization format, context length, inference software, GPU allocation, and whether the model uses part of its weights outside VRAM.

Check a specific model and GPU with RigForAI's What can I run? tool before downloading large files.

What determines whether a model fits?

A local model needs VRAM for more than its downloaded weight file. The main requirements are:

1. Model weights — the learned parameters stored in formats such as FP16, INT8, or 4-bit.
2. Runtime overhead — memory used to load and execute the model.
3. KV cache — memory used to retain the conversation or prompt context.
4. Temporary workspace — memory required by the inference engine for computation.
5. Other GPU allocations — desktop rendering, browser windows, drivers, or another application.

A model file that appears to be 7.5GB may therefore fail to load on an 8GB card. The file size is only part of the requirement, and an 8GB GPU does not necessarily provide 8GB of usable VRAM.

A useful sizing formula

A rough first estimate for model weights is:

Weight memory ≈ parameter count × bytes per parameter

Typical approximate storage values are:

Format	Approximate weight storage
FP32	4 bytes per parameter
FP16 or BF16	2 bytes per parameter
INT8	1 byte per parameter
4-bit	0.5 bytes per parameter, plus quantization metadata

These figures are estimates. Quantized files include scales and other metadata, and different formats can have different overhead. Runtime memory can also vary substantially between inference engines.

For example, a 7B model in idealized 4-bit storage would need roughly:

7 billion × 0.5 bytes ≈ 3.5GB for the weights

That does not mean it needs only 3.5GB of VRAM. The runtime, KV cache, temporary buffers, and desktop overhead must be added.

Realistic model ranges for an 8GB GPU

The following table assumes local text-generation workloads and a modern inference engine that supports the model's format. It describes practical starting points rather than hard compatibility limits.

Model size	FP16/BF16	INT8	4-bit	Practical expectation on 8GB
1B–3B	Usually easy	Easy	Easy	Good headroom for longer context and multitasking
4B–6B	Often possible	Usually comfortable	Easy	A strong fit for local use
7B–8B	Usually too large for full GPU loading	May be tight	Commonly practical	Good target at moderate context lengths
9B–13B	Generally too large	Usually impractical for full loading	Borderline to practical with compromises	May need short context or partial offload
14B and above	Not realistic entirely in 8GB	Not realistic entirely in 8GB	Usually requires offload	Expect lower speed and more configuration

“Easy” does not mean every model in that range will work with every application. Architectures with different attention designs, larger vocabularies, or unusual runtime requirements can change the result.

1B–4B models: the comfortable range

Models in this range are the least demanding choice for an 8GB card. Depending on the architecture and format, they can often run in FP16, BF16, or INT8 while leaving useful VRAM for context and other applications.

This range is appropriate when you want:

• Longer prompts or conversations.
• Faster response generation.
• Lower risk of memory errors.
• Room for a GPU-accelerated embedding or reranking model.
• More flexibility with image or audio tools running at the same time.

If quality from a larger model is not essential, a smaller model with more context headroom may provide a better experience than a barely fitting 8B model.

4B–6B models: a strong balance

Models around 4B to 6B are often a good compromise for an 8GB GPU. They can provide more capability than the smallest models without consuming as much memory as a 7B or 8B model.

4-bit versions should generally leave more room for context and runtime overhead. Higher-precision versions may fit depending on the model and software, but you should verify the actual memory requirement rather than assuming that parameter count alone is sufficient.

7B–8B models: the common 4-bit target

A 7B or 8B model in a good 4-bit format is the most common serious target for an 8GB GPU.

In practice, you should expect to:

• Use a 4-bit quantization such as a supported GPTQ, AWQ, or other runtime-compatible format.
• Keep some VRAM free instead of targeting 100% allocation.
• Start with a moderate context length.
• Avoid running multiple GPU-heavy applications at the same time.
• Check whether your chosen application places all layers on the GPU or silently offloads some to system RAM.

A 4-bit model can be very usable, but quantization quality matters. Two 4-bit versions of the same model may differ in calibration, supported kernels, file size, and output quality. The smallest file is not automatically the best choice.

9B–13B models: possible, but no longer straightforward

A 9B–13B model may fit partially or, in some cases, fully in an 8GB GPU using aggressive quantization. However, this is where trial-and-error becomes expensive.

Common compromises include:

• Lowering the context length.
• Offloading some layers to system RAM.
• Accepting slower generation.
• Using a smaller quantization.
• Closing other GPU applications.
• Selecting a model format supported by your inference engine.

If you specifically want a 13B-class model, an 8GB GPU should be viewed as an acceleration device rather than a guaranteed full-model host. A model can technically run with offload while still being too slow for your intended workflow.

Quantization: what should you use?

Quantization reduces the number of bits used to store model weights. It is the main reason an 8GB card can run models that would not fit in FP16.

FP16 and BF16

FP16 and BF16 use approximately 2 bytes per parameter for the weights. They preserve more numerical precision than heavily quantized formats, but their memory requirements grow quickly.

On an 8GB GPU, these formats are most realistic for smaller models, generally in the 1B–4B range. A 7B model in FP16 needs roughly 14GB just for idealized weights before runtime memory is considered, so it is not a full-GPU target for an 8GB card.

BF16 support also depends on the GPU and software stack. Do not assume that a GPU supporting FP16 will handle BF16 in the same way.

INT8

INT8 uses approximately 1 byte per parameter for the weights, plus format-specific overhead. It can be a useful middle ground when supported by the model and inference stack.

A 7B model would require roughly 7GB of idealized INT8 weight storage before overhead, leaving little room on an 8GB card. As a result, INT8 is usually more comfortable for smaller models on this class of GPU.

4-bit quantization

4-bit formats use approximately 0.5 bytes per parameter for the raw weights, plus metadata and runtime overhead. This is why 7B–8B models are commonly practical on an 8GB GPU.

Trade-offs include:

• Lower memory use.
• Potentially lower output quality than FP16 or BF16.
• Different quality between quantization methods and files.
• Dependence on supported GPU kernels.
• Possible changes in speed depending on the inference engine.

For many users, a well-made 4-bit 7B or 8B model is a better fit than a larger model that needs extensive CPU offload.

Context length can make a working model fail

Model weights are mostly a fixed cost. The KV cache grows as you process more tokens, so memory usage can increase significantly when you raise the context length.

This means the same model may:

• Load successfully at 4K context.
• Become unstable or fail at 8K context.
• Use substantially more memory at 16K context.
• Run more slowly as the prompt grows, even if it still fits.

The exact KV-cache cost depends on the model architecture, number of layers, attention dimensions, key/value head configuration, cache data type, and inference software. There is no single context-to-VRAM conversion that applies to every model.

Practical context guidance

On an 8GB GPU:

• Start with the model's default or a moderate context length.
• Increase context gradually while monitoring VRAM.
• Leave headroom for the cache rather than allocating every available megabyte to weights.
• Do not assume the model's advertised maximum context is practical on your hardware.
• Treat long-context support as a separate hardware requirement from model loading.

If you need long documents, retrieval-augmented generation can sometimes be a better design than keeping the entire document in the active context. It reduces the number of tokens that must remain available at once, although the overall quality depends on the retrieval system.

How much VRAM headroom should you leave?

There is no universal safe percentage because desktop use, drivers, and inference software differ. However, planning to consume all 8GB is risky.

A safer approach is:

Usable VRAM = total VRAM − current GPU allocations − safety margin

Check the memory reported by your operating system and inference tool while the GPU is idle. A display-connected GPU may already have memory reserved for the desktop. Browsers, recording software, games, and image-generation applications can consume additional VRAM.

For a model that nearly fills the card, expect more sensitivity to:

• Context length.
• Batch size.
• Prompt processing.
• Multiple simultaneous requests.
• CUDA or other backend workspaces.
• Other processes using the GPU.

A model that leaves several hundred megabytes free may work for a short prompt but fail once the conversation grows. More headroom generally makes the system easier to use.

When CPU or RAM offload becomes necessary

CPU offload places some model layers, the KV cache, or other data in system RAM instead of GPU VRAM. It can make a model load when full GPU execution is impossible.

This is useful when:

• The model is only slightly larger than available VRAM.
• You value the ability to run the model more than maximum speed.
• Your system has sufficient RAM.
• Your CPU-to-GPU connection is fast enough for the workload.
• You are testing a model before upgrading hardware.

The main drawback is speed. If inference repeatedly moves data between system RAM and VRAM, token generation can become much slower and less consistent. CPU offload is not a way to turn an 8GB GPU into a high-VRAM card; it is a compromise that trades performance for capacity.

You should also budget system RAM for:

• The offloaded model weights.
• The operating system and applications.
• The inference runtime.
• File loading and caching.
• The KV cache if your configuration places it in system memory.

As a rule of thumb, an offloaded model should not consume nearly all available system RAM. Exact RAM requirements depend on the model format and runtime, so check the application's allocation report rather than relying only on the downloaded file size.

What about image-generation models?

If “AI models” includes image generation, the answer changes because image models use a different memory profile from language models.

An 8GB GPU can run some image-generation workflows, but resolution, batch size, model family, precision, VAE behavior, ControlNet modules, upscalers, and other extensions all affect VRAM. A base model that fits alone may exceed 8GB when additional components are loaded.

To improve the odds of fitting:

• Use the supported low-memory or memory-efficient mode.
• Start with a single image and a moderate resolution.
• Avoid loading multiple adapters or ControlNet models initially.
• Enable attention or VAE optimizations when supported.
• Monitor VRAM during both model loading and image generation.

Image-generation compatibility should be checked separately from LLM compatibility. A GPU that is comfortable for a 7B 4-bit language model may still need careful settings for a complex image workflow.

How to choose a model for an 8GB GPU

Use this decision process before downloading:

1. Identify the workload. Text generation, embeddings, image generation, speech, and video have different memory behavior.
2. Check the parameter count. Use it as an initial filter, not a final answer.
3. Choose the quantization. For 7B–8B language models, 4-bit is usually the starting point on an 8GB card.
4. Check the actual file size and format. Make sure your inference application supports it.
5. Account for context length. A model that fits at a short context may not fit at the length you need.
6. Reserve headroom. Do not plan around the full advertised VRAM capacity.
7. Decide whether offload is acceptable. If not, filter out models that cannot fit entirely in VRAM.
8. Verify the combination. Model, quantization, GPU, context, and runtime must be considered together.

For a quick compatibility check, use RigForAI's What can I run? results. It is especially useful when comparing several model sizes or quantizations before committing to a large download.

Bottom line

An 8GB GPU is a capable entry point for local AI, particularly for smaller models and 4-bit 7B–8B language models. Models around 1B–6B provide more headroom, while 9B–13B models increasingly depend on aggressive quantization, short contexts, or CPU offload.

The safest choice is not the largest model whose weight file appears to fit. Choose a model and quantization that leave room for the KV cache, runtime overhead, and the context length you actually plan to use.