Hardware for 70B Models: Memory, Bandwidth and Real Throughput

The weight budget

Start with the arithmetic, because it eliminates most configurations immediately. Weight memory is parameters multiplied by bits per parameter, divided by eight.

Four-bit is the practical target, so 40GB of addressable memory is the real requirement once the context cache is accounted for. That single figure rules out every consumer graphics card sold today as a solo solution, which is the fact most buyers discover after purchasing rather than before.

When sizing hardware for 70B models, dropping below 4-bit is tempting and is where degradation becomes noticeable — specifically in the capabilities that justify choosing a 70B model at all: multi-step reasoning, instruction adherence and arithmetic. Compressing the model until it fits can remove the reason you wanted it.

Bandwidth sets the ceiling

Fitting the model tells you whether it runs. Bandwidth tells you how fast, and for single-user inference the relationship is unusually simple.

Every generated token requires reading the full set of weights once, so the theoretical maximum generation rate is memory bandwidth divided by model size. For a 35GB model:

• 200 GB/s ÷ 35GB ≈ 5.7 tokens per second — slower than comfortable reading
• 400 GB/s ÷ 35GB ≈ 11.4 tokens per second — usable
• 800 GB/s ÷ 35GB ≈ 22.9 tokens per second — comfortable

Anyone shopping for hardware for 70B models should run that division first. Real throughput lands below these ceilings because of overhead, but the ratio holds and it explains a great deal. It is why a machine with ample capacity but modest bandwidth runs a large model slowly rather than not at all, and why quantizing further speeds generation up as well as shrinking the footprint — a 26GB model reads faster on the same bus.

When comparing hardware for 70B models, bandwidth is therefore the second specification to check and the one retail listings most often omit entirely.

Four routes to 35GB

The last row deserves emphasis because it is what most people accidentally end up doing. It genuinely runs, and at a few tokens per second it is unusable for interactive work while remaining perfectly serviceable for overnight batch jobs. Decide which of those two you need before spending anything.

On multi-card setups, note that splitting a model across cards adds communication between them for every token. Aggregate bandwidth is not simply additive, and the interconnect becomes a factor that a single-card configuration never has to think about.

Prompt processing is a different problem

Generation is memory-bound; reading your prompt is not. Processing an input reads the weights once for many tokens simultaneously, so it is compute-bound and scales with arithmetic throughput rather than bandwidth.

The practical consequence is that a long prompt produces a pause before the first token appears, and that pause is governed by different hardware than the generation speed that follows. A configuration which generates acceptably can still feel sluggish if you routinely paste long documents. Measure both figures separately: time to first token, and tokens per second afterwards.

Whether 70B is worth it

Before committing to this class of hardware, test whether the size is what you actually need. Two checks settle it cheaply and both are worth doing first.

Run your real task set against a 13B and a 32B model at 4-bit. If the 32B already passes, a 70B buys very little at several times the hardware cost. Then run the same tasks against a hosted flagship: if a 70B open-weight model still does not reach the quality you need, local hardware was never the answer to your problem — the model class was.

Where this size genuinely earns its keep is a narrow band: work that needs strong reasoning, must remain on your own infrastructure, and runs at enough volume that per-token pricing is uncomfortable. That is a real set of circumstances, and it is considerably smaller than the amount of hardware sold to serve it.

The checklist

• Budget 40GB of addressable memory, not 35 — the context cache is not optional.
• Find the memory bandwidth and divide it by model size for your speed ceiling.
• Decide interactive or batch before considering system-RAM offload.
• Test a 32B first. If it passes your tasks, stop there and keep the money.
• Measure time to first token separately from tokens per second.
• Do not drop below 4-bit to force a fit; it removes the reasoning you were paying for.

Approached that way, choosing hardware for 70B models becomes two divisions and a test rather than a shopping expedition — and it quite often ends with the conclusion that a smaller model, fully resident and fast, is the better machine for the work in front of you.