VRAM Optimization for Local AI: Offload, Quantization and Context

Where the memory actually goes

Three consumers compete for the same pool, and knowing their relative sizes tells you which lever to pull first.

Most guidance to VRAM optimization for local AI addresses only the first row. That is a mistake, because the second row is the one that causes the failure people actually experience: a model that loaded fine and then ran out of memory twenty minutes into a long document.

Context is the free lever

Runtimes let you set a maximum context length, and the default is frequently the model’s theoretical maximum rather than what you need. Reserving space for a very long context costs memory whether or not you ever fill it.

Set it to what your work actually requires. If your longest realistic prompt is four thousand tokens, allocating for a hundred thousand wastes memory that could hold more model. This is genuinely free capacity for most users, and it is the first change to make.

Two related habits help. Trim the prompt itself — pasting an entire document when three paragraphs matter costs cache on every request. And clear conversation history between unrelated tasks, because a long running chat is a long context whether you meant it or not.

Quantizing the cache, not just the weights

Everyone quantizes weights. Fewer people know the context cache can be quantized too, and on long-context work it is the larger saving.

The cache stores intermediate state for every token in the context across every layer. Storing that state at reduced precision typically halves or quarters its footprint, with a quality cost that is small for most tasks and concentrated in long-range recall. Where a runtime exposes the setting, it is the highest-value option in the whole of VRAM optimization for local AI on long documents.

Test it rather than assuming. Run a task that depends on information early in a long input, once with the cache at full precision and once quantized, and compare. If the answers match, keep the saving.

Tuning layer offload deliberately

When a model does not fit, runtimes place some layers in system memory. Left automatic, this is a cliff: throughput can fall by an order of magnitude because those layers travel across the PCIe bus for every token.

Take control of it. Most runtimes expose the number of layers to place on the GPU, and the right approach is to raise it until you are close to full and then stop one step short, leaving headroom for the cache. A configuration where every layer is resident and the context is modest beats one where all layers technically fit and the cache then forces a spill mid-session.

The honest conclusion when nothing fits is to change model rather than tune around it. A 13B at 4-bit fully resident will outperform a 32B partially offloaded on almost any single-user workload.

Housekeeping that reclaims gigabytes

• Unload the previous model. Runtimes that keep models cached in memory will hold two at once and fail on the third. Set an idle timeout.
• Keep batch size at one for single-user work. Larger batches raise throughput for concurrent requests and cost memory you are not using.
• Close the browser before a large job. Hardware-accelerated browsers with many tabs can hold well over a gigabyte of graphics memory.
• Run the display on integrated graphics where the machine supports it, freeing the whole discrete card.
• Do not run two inference servers. Each reserves memory independently, and the second one usually fails in a way that looks like a bug.
• Enable memory-efficient attention where offered; it reduces activation memory at no quality cost.

Measure, do not estimate

Every recommendation above should be verified rather than trusted, and the tooling is built in. Watch actual memory use with the vendor’s monitoring utility while a long request runs, not while the model sits idle — idle numbers hide the cache growth that causes failures.

Record three figures for each configuration you try: peak memory during a long request, tokens per second, and whether output quality held. That table is the entire discipline of VRAM optimization for local AI, and it takes an afternoon to build once and then answers every future question in seconds.

One last note on expectations. The goal is not to fill the card. It is to leave enough spare that a long prompt does not push you over, because the failure mode is not a warning — it is a tenfold slowdown or a crash, arriving at the moment you fed it something important.