Can I Run This Model? — LLM VRAM Calculator

Find out whether your GPU can run a given large language model. Pick a model (Llama, Mistral, Qwen, Gemma, Phi, Mixtral, DeepSeek and more, or a custom parameter count) and your hardware (NVIDIA, Apple Silicon, CPU, or a custom VRAM and bandwidth), and it works out the VRAM needed at every quantization level from FP16 down to IQ2, whether it fits with your context length, the longest context that fits and a rough tokens-per-second figure, then recommends the best quant. Real weight, KV-cache and memory-bandwidth arithmetic, shown step by step; nothing is uploaded.

Calculator Web & Dev Updated Oct 3, 2026
Learn how this works
How to Use
  1. Pick the model you want to run. Llama, Mistral, Qwen, Gemma, Phi, Mixtral, DeepSeek and others are built in, or choose Custom and type a parameter count in billions. A Preset loads a common pairing, such as Llama 3.1 8B on an RTX 4060, in one click.
  2. Pick your hardware: an NVIDIA GeForce or data-centre GPU, an Apple Silicon configuration, a CPU with system RAM, or Custom to enter your own VRAM and memory bandwidth.
  3. Set the context length with the slider (512 to 128K tokens; longer context needs more memory for the KV cache) and, if you like, store the KV cache at 8 or 4 bits to save memory.
  4. Read the verdict and the four readouts: total VRAM, best quantization, estimated tokens per second and the longest context that fits.
  5. Check the quantization table for every level at once, then scroll to <a href="#showwork-heading">Show Work</a> to see the weight, KV-cache and overhead arithmetic for your selection.
Input
4K
Presets
Verdict
Total VRAM
16.2 GB
Best quant
FP16
Speed (est.)
49.7 tok/s
Max context
69,164 tok

Worked Example

Llama 3.1 8B on an RTX 4090, 4K context. The weights at FP16 are 8.03 × 10⁹ × 16 ÷ 8 = 16,060,000,000 bytes = 14.96 GB. The model has 32 layers and 8 KV heads of size 128, so the KV cache is 2 × 32 × 4,096 × 8 × 128 × 2 bytes = 536,870,912 bytes = 0.50 GB. Overhead is 0.6 + 4,096 ÷ 8,192 × 0.3 = 0.75 GB. The total is 16.21 GB against 24 GB, so it runs at full precision with 7.79 GB to spare, at about 1,008 ÷ 15.21 × 0.75 = 49.7 tokens a second.

A 70B model on the same card. Llama 3.1 70B at Q4_K_M needs 39.70 GB of weights and 41.70 GB in total, so it does not fit. Even Q2_K (29.53 GB) is too big; only IQ2_XXS fits, at 18.93 GB, and with heavy quality loss, so the verdict is “Barely”. An 80 GB A100 runs it at Q8_0 (71.86 GB).

The common mistake: sizing only the weights. Llama 3.1 8B at Q4_K_M is 4.52 GB of weights, which looks safe on an 8 GB RTX 4060, and at 4K context it is (5.77 GB in total). At 32K context the KV cache grows to 2 × 32 × 32,768 × 8 × 128 × 2 bytes = 4.00 GB and overhead to 1.80 GB, so the total is 10.32 GB and it no longer fits. Storing the KV cache at 8 bits halves it to 2.00 GB, but even then only Q3_K_M (7.46 GB) fits.

Show Work

Pick a model and your hardware to see the working.

Formulas

Model weights
params × bits per weight ÷ 8 bytes
KV cache
2 × layers × context × KV heads × head size × bytes
Overhead
0.6 + context ÷ 8,192 × 0.3 GB
Fits?
weights + KV + overhead ≤ VRAM (× 0.72 for unified memory)
Speed (est.)
bandwidth ÷ (active weights + ½ KV) × 0.75
Max context
(usable − weights − 0.6 GB) ÷ KV bytes per token

A GB here is 2³⁰ = 1,073,741,824 bytes (strictly a GiB), the unit graphics cards are labelled in, so 24 GB on an RTX 4090 means 24 × 2³⁰ bytes.

The Quantization Ladder

Quantization is how a big model fits on a small card: each weight is stored in fewer bits, trading a little accuracy for a lot less memory. These are the common GGUF levels, from full precision down to the aggressive formats that squeeze a 70B model onto a 24 GB card.

FP16 · 16 bit
reference quality · full size
Q8_0 · 8.5 bit
near-lossless · 53% of FP16
Q6_K · 6.56 bit
effectively lossless · 41%
Q4_K_M · 4.83 bit
the sweet spot, recommended · 30%
Q3_K_M · 3.91 bit
noticeable degradation · 24%
Q2_K / IQ2_XXS · 3.35 / 2.06 bit
big models on small cards, real quality loss · 21% / 13%

Why Memory, Not Compute, Decides

Running large language models on home hardware took off in March 2023, when Georgi Gerganov released llama.cpp, a plain C/C++ inference engine that ran Meta’s LLaMA weights on a laptop CPU by storing them in 4 bits. The “k-quant” formats such as Q4_K_M followed in June 2023, and in August 2023 the project replaced its GGML files with GGUF, the single-file format most local models are now shared in.

The KV cache became the other half of the memory bill as context windows grew. Noam Shazeer’s 2019 paper on multi-query attention showed that the keys and values could be shared between attention heads, and grouped-query attention (Ainslie and colleagues at Google, 2023) settled on a middle ground of a few KV heads per layer. Llama 2 7B still uses one KV head per query head (32), so its cache at 4K context is 2.00 GB; Llama 3.1 8B uses 8, and its cache is 0.50 GB, a quarter of the size.

Generation speed is set by memory bandwidth because each new token has to stream every active weight through the GPU once. That is also why Mixture-of-Experts models such as Mixtral 8x7B (Mistral AI, December 2023) run faster than their size suggests: only two of the eight experts in each layer are used per token.

About This Tool

This calculator answers “will it fit, and how fast will it run?” for 28 open-weight models, 23 GPU, Apple Silicon and CPU configurations, nine GGUF quantization levels, contexts from 512 to 131,072 tokens and three KV-cache precisions, or for any custom model size and custom card. It uses each model’s real layer count, KV heads and head size, so models with grouped-query attention are not overcharged for their KV cache, and Mixture-of-Experts models are sized on all their parameters but timed on the active ones.

The specifications and the arithmetic are all in the page, and it runs entirely in your browser; nothing you pick is sent anywhere. The figures are estimates for a single-GPU, GGUF-style load: real usage moves with the backend, batch size and drivers.

Related tools: Minifloat / Custom Float Converter, IEEE 754 Float Converter, and File Size Converter.

Frequently Asked Questions

How does it calculate the VRAM?

It adds three parts. Weights = parameters × bits per weight ÷ 8: Llama 3.1 8B at FP16 is 8.03 × 10⁹ × 16 ÷ 8 = 16,060,000,000 bytes, or 14.96 GB. KV cache = 2 × layers × context × KV heads × head size × bytes: 2 × 32 × 4,096 × 8 × 128 × 2 = 536,870,912 bytes, or 0.50 GB. Overhead for the compute context = 0.6 + 4,096 ÷ 8,192 × 0.3 = 0.75 GB. The total, 16.21 GB, is compared with the usable memory: 24 GB on an RTX 4090, so it fits with 7.79 GB to spare.

What do the quantization levels mean?

Quantization stores each weight in fewer bits. The table uses the effective bits per weight of the common GGUF formats: FP16 16, Q8_0 8.5, Q6_K 6.56, Q5_K_M 5.67, Q4_K_M 4.83, Q4_0 4.55, Q3_K_M 3.91, Q2_K 3.35 and IQ2_XXS 2.06. Q4_K_M keeps 4.83 ÷ 16 = 30% of the FP16 size (Llama 3.1 8B drops from 14.96 GB to 4.52 GB) with only a small loss of quality, which is why it is the recommended level. Below Q3 the loss becomes obvious.

How accurate is the tokens-per-second estimate?

It is a ballpark, not a benchmark. Generating each token reads every active weight once, so speed is limited by memory bandwidth: bandwidth ÷ bytes read per token × 0.75 for real-world efficiency. For Llama 3.1 8B at FP16 on an RTX 4090 that is 1,008 ÷ (14.96 + 0.25) × 0.75 = 49.7 tokens a second. The backend (llama.cpp, vLLM, Transformers), flash attention, batch size and drivers all move the real figure, and reading the prompt is compute-bound, not bandwidth-bound.

Does it handle Mixture-of-Experts models like Mixtral?

Yes. Every expert has to sit in memory, so Mixtral 8x7B is sized on all 46.7B parameters: at Q8_0 that is 47.46 GB with a 4K context, which just fits the 48 GB of an RTX A6000. Only about 12.9B parameters are active per token, so speed is worked out on those: about 44.3 tokens a second, where a dense 46.7B model would manage far fewer.

What about Apple Silicon or running on the CPU?

Apple Silicon shares one pool of unified memory between CPU and GPU, and macOS only lets the GPU wire part of it by default; the calculator allows 72%, so an M4 Max with 128 GB is treated as 128 × 0.72 = 92.16 GB. That runs Llama 3.1 70B at Q8_0 (71.86 GB) at roughly 5.8 tokens a second, limited by its 546 GB/s bandwidth. The CPU options assume dual-channel DDR5 at about 80 GB/s, so expect single-digit speeds on large models.

How do I use the Can I Run This Model? — LLM VRAM Calculator?

Just type your numbers. The answer shows up right away — there is no button to press. Change anything and it updates by itself.

Do I need to install or sign up for anything?

Not at all — it runs in the browser with nothing to install and no account. After it loads once, it even works without an internet connection.

Is my information private?

Yes. Everything happens in your browser. Nothing you type is sent to a server or saved anywhere.

Common Use Cases

Before you download

Llama 3.1 70B at Q4_K_M needs 41.70 GB with a 4K context, more than a 32 GB RTX 5090, so check before pulling a 40 GB file.

Pick the right quant

An 8 GB RTX 4060 holds Llama 3.1 8B at Q6_K (7.38 GB) but not at Q8_0 (9.20 GB).

Plan a GPU purchase

Llama 3.1 70B at Q8_0 needs 71.86 GB, which points to an 80 GB A100 or H100.

Size the context

Llama 3.1 8B at FP16 on an RTX 4090 has room for up to 69,164 tokens of context.

Estimate speed

Mixtral 8x7B at Q4_K_M on an RTX A6000 comes out at about 76.8 tokens a second.

Spec a Mac

An M4 Max with 128 GB gives the GPU about 92.16 GB, enough for a 70B model at Q8_0.

Last updated: