VRAM, the GPU’s own fast memory, decides which AI models can load on your card.
By Tony Yu / tkal · November 9, 2026 · One-minute visual guide
Does lunch fit the lunchbox?~
24 GB card, the weights must fit ♡
↓
27B model16.5 GB of weights: fits, about 6 GB spare
vs
70B model42.5 GB before context: short by almost half
Whether the cache fits can swing the speed ten times on the same card.
A simplified picture: the lunchbox is Mochi’s analogy; the sizes are ComputingForGeeks’ Q4_K_M examples on a 24 GB RTX 4090.
Keep this bitVRAM means video random access memory: the high-speed memory available directly to the GPU, and a model’s weights have to fit inside it.
Why care?
Only if you run AI locally. When you chat with ChatGPT or Claude in a browser, the model runs on someone else’s hardware. Download an open model to your own card, though, and VRAM picks which sizes load. HackerNoon’s warning is blunt: a GPU with too little memory may fail to load the model.
The lunchbox test
Think of VRAM as a lunchbox. A 27B model at Q4_K_M needs 16.5 GB for its weights, so a 24 GB RTX 4090 holds it with about 6 GB spare. A 70B at Q4_K_M needs 42.5 GB before context, and the same card is short by almost half. The weights aren’t the only thing packed in: during inference VRAM also holds input data, intermediate calculations and caches.
27B model
70B model