How to Run Local LLMs on an RTX 5080: Setup and VRAM Limits

The 16GB ceiling

NVIDIA’s published specification for the GeForce RTX 5080 lists “16 GB GDDR7” on a “256-bit” interface, with “10752” CUDA cores and a total graphics power of “360” watts.

If you plan to run local LLMs on RTX 5080 hardware, that memory figure is the whole specification. For gaming, the core count is the headline. For local language models it is almost irrelevant, because inference on a single request is memory-bound rather than compute-bound: the machine spends its time moving weights, not multiplying them. The practical ceiling is therefore how many gigabytes of weights you can hold on the card, and that number is 16 minus whatever your display and operating system already occupy.

Budget realistically. A desktop running a browser and a couple of monitors will typically leave 13 to 14GB usable for a model. Then leave headroom on top of the weights for the context cache, which grows with how much text you feed in.

What actually fits

Weight memory is parameters multiplied by bits per parameter, divided by eight. Applied against roughly 14GB of usable VRAM:

Read that table before buying anything. It is the shortest honest answer to what you can run local LLMs on RTX 5080 silicon at usable speed. The interesting consequence is that the best configuration for this card is usually a 13B-class model at 4-bit rather than a 7B at 8-bit: same rough memory footprint, more parameters, and the quality difference between 4-bit and 8-bit is smaller than the difference between 7B and 13B for most tasks.

Why compute is not the constraint

It is worth understanding why a card with over ten thousand shader cores is limited by memory, because it explains several otherwise confusing observations.

Generating one token requires reading essentially every weight in the model once. With a 7GB model, that is 7GB of memory traffic per token. The arithmetic performed on those weights is trivial by comparison, so the graphics processor spends most of each token waiting for data. This is why memory bandwidth predicts generation speed far better than core count does, and why two cards with similar bandwidth generate at similar rates despite very different compute specifications.

Two things follow. Prompt processing behaves differently — it reads the model once for many tokens at a time, so it is compute-heavy and the core count helps there. And batching multiple requests amortizes the weight reads, which is why server deployments get far more total throughput than a single-user desktop from identical hardware.

Setup, in order

1. Install the current driver before anything else. Local inference runtimes depend on recent CUDA support, and a stale driver is the most common cause of a runtime silently falling back to the processor.
2. Install one runtime and learn it before adding a second. Running two model servers that both try to claim the card causes memory conflicts that look like bugs.
3. Pull a 13B-class model at 4-bit as your baseline, not the largest one available.
4. Confirm every layer is on the GPU. Runtimes report how many layers were offloaded; if it is not all of them, you are paying the penalty described below.
5. Measure tokens per second once, on a fixed prompt, and write it down. It is your reference for every later change.
6. Only then increase context length or model size, one variable at a time, watching VRAM as you go.

The offload cliff

When a model does not fit, runtimes will usually still load it by keeping some layers in system RAM. This works, and it is dramatically slower — not by a few percent but often by an order of magnitude, because those layers are read across the PCIe bus instead of from on-card memory.

The behavior is a cliff rather than a slope. A model that fits with one gigabyte spare runs at full speed; the same model one gigabyte too large crawls. If output speed collapses after you change models or raise the context window, this is almost always why, and the fix is to reduce one of the three variables: model size, quantization level, or context length.

A useful discipline when you run local LLMs on RTX 5080 hardware: treat 14GB as a hard budget and pick a combination that fits inside it with room to spare, rather than the largest configuration that technically loads.

If you genuinely need 70B

A 70B model at 4-bit needs roughly 35GB before context. Three honest routes exist, and buying a second consumer card is the least appealing of them for most people.

• Unified-memory systems. Machines that share one large memory pool between processor and graphics can hold models far beyond any consumer card’s VRAM, at lower bandwidth. Slower per token, but it runs at all.
• Two cards. Splitting a model across two 16GB cards gives 32GB of weight capacity, which is still short of a 70B at 4-bit with context. It works for 32B-class models.
• Rent the large model, keep the small one local. Route routine work to your local 13B and send the genuinely hard requests to a hosted model. This is usually the cheapest answer and it keeps most data on your machine.

None of that diminishes the card. A 13B model at 4-bit running entirely in 16GB of GDDR7 generates faster than most people read, which is the actual goal. The mistake people make when they set out to run local LLMs on RTX 5080 cards is assuming a premium graphics card removes a memory constraint that is set by the model, not by the silicon.