Complete Guide to LLM VRAM Requirements
What is VRAM and Why Does It Matter for LLMs?
VRAM (Video Random Access Memory) is the dedicated memory on your graphics card used to store data that the GPU needs to process quickly. When running Large Language Models (LLMs) locally, VRAM is the most critical resource because the entire model must fit into this memory for fast inference. Unlike system RAM, VRAM offers significantly higher bandwidth (up to 1 TB/s on modern GPUs), which is essential for the massive matrix multiplications that LLMs perform.
If a model doesn't fit in VRAM, the system will fall back to using system RAM, which is 10-50x slower. This results in extremely slow token generation (often less than 1 token per second) or complete failure to load the model.
What is LLM Quantization?
Quantization reduces the precision of model weights to save memory. GGUF K-quant formats add block metadata (scale/min values per 256-weight block), so the effective bits-per-weight (bpw) is always higher than the nominal bit depth. AWQ and GPTQ use group-based quantization with ~3% overhead from scale factors.
| Format | Effective bpw | Quality | 7B Weights |
|---|---|---|---|
| FP16 / BF16 | 16.00 bpw | Original quality | ~14.0 GB |
| Q8_0 (GGUF) | 8.50 bpw | Near-lossless | ~7.4 GB |
| Q6_K (GGUF) | 6.57 bpw | Excellent | ~5.7 GB |
| Q5_K_M (GGUF) | 5.67 bpw | Very good | ~5.0 GB |
| Q4_K_M (Recommended) | 4.83 bpw | Good | ~4.2 GB |
| AWQ / GPTQ (4-bit) | 4.00 bpw | Good | ~3.5 GB |
| Q3_K_M (GGUF) | 3.89 bpw | Acceptable | ~3.4 GB |
| IQ2_XS (GGUF) | 2.05 bpw | Low / constrained HW | ~1.8 GB |
Q4 quantization (4-bit) is the sweet spot for most users, offering a good balance between model quality and memory usage. Modern quantization methods like GGUF (used by llama.cpp) and AWQ (used by vLLM) preserve model quality remarkably well even at 4-bit precision.
Understanding Context Window and KV Cache
The context window determines how many tokens the model can "remember" at once. A larger context window allows the model to process longer documents and maintain coherence over extended conversations. However, the KV cache (Key-Value cache) that stores attention states grows linearly with context length.
For a 7B model with 32 layers, each additional 1,000 tokens of context requires approximately 0.06 GB of additional VRAM. At 128K context, this alone can add 7-8 GB of VRAM usage on top of the model weights.
GPU Recommendations for LLM Inference
Budget (8-12 GB VRAM)
RTX 3060 12GB, RTX 4060, RTX 4070. Can run 7B models at Q4 with moderate context. Good for experimentation and small-scale use.
Mid-Range (16 GB VRAM)
RTX 4070 Ti Super, RTX 4080. Handles 7B-13B models comfortably at Q4 with large context windows. Ideal for most developers.
High-End (24 GB VRAM)
RTX 3090, RTX 4090. Can run 30B-34B models at Q4. The best consumer option for serious LLM work. RTX 3090 offers the best value.
Enterprise (48+ GB VRAM)
RTX 6000 Ada, A100, H100. Required for 70B+ models or training. Consider cloud instances (AWS, GCP, Lambda Labs) for cost efficiency.
Tips to Reduce VRAM Usage
- Use Q4 quantization instead of FP16 to cut memory usage by 75%
- Reduce context window to the minimum needed for your task
- Use CPU offloading (llama.cpp) to split the model between GPU and system RAM
- Enable paged attention (vLLM) for more efficient KV cache management
- Use speculative decoding to reduce the number of forward passes needed
- Consider smaller models like Llama-3.2 3B or Phi-3 mini for resource-constrained environments
Popular LLM Models and Their VRAM Requirements
Here's a quick reference for popular models at Q4 quantization with 4K context:
| Model | Parameters | VRAM (Q4) | Min GPU |
|---|---|---|---|
| Llama-3.2 3B | 3B | ~2.3 GB | RTX 3060 |
| Phi-3 Mini | 3.8B | ~2.8 GB | RTX 3060 |
| Mistral 7B | 7B | ~5.1 GB | RTX 3060 |
| Llama-3 8B | 8B | ~5.6 GB | RTX 3060 |
| Qwen2.5-32B | 32B | ~20 GB | RTX 3090/4090 |
| Llama-3 70B | 70B | ~46 GB | 2x RTX 4090 |
| DeepSeek V3/R1 | 671B (MoE) | ~370 GB | 8x H100 / Cloud |