Local AI Cost Modeling: When Self-Hosting Beats Per-Token Pricing

What local actually costs

Self-hosting feels free after purchase, which is why local AI cost modeling is so often skipped. It is not free; the costs are simply front-loaded and then recurring in electricity.

Take a concrete configuration. A 16GB consumer card at around $1,000, amortized over three years, is about $333 a year. Under sustained inference load the card draws a few hundred watts; at 300W and $0.15 per kilowatt-hour, that is $0.045 an hour, or about $131 a year at eight hours of daily use. Total: roughly $464 a year.

Adjust the inputs to your situation — your electricity price, your hardware cost, your hours — but keep the structure. The point is that the annual figure is a few hundred dollars, not zero, and it does not scale with usage until you saturate the card.

That last property is the real advantage of local inference and the reason to model it. Beyond the fixed cost, additional requests are nearly free until you run out of hours in the day.

What hosted actually costs

Hosted pricing is per token, published, and easy to model. Current list rates give a spread across capability tiers — for one vendor’s family, roughly $1 per million input tokens and $6 per million output tokens at the cost-efficient tier, rising to $5 and $30 at the flagship.

Convert that into a per-request figure, because tokens are not an intuitive unit. A request with 1,000 tokens of input and 500 tokens of output at the cheap tier costs:

• Input: 1,000 ÷ 1,000,000 × $1 = $0.001
• Output: 500 ÷ 1,000,000 × $6 = $0.003
• Total: $0.004 per request

Four tenths of a cent. That number is the reason so many teams never bother with local inference, and any honest local AI cost modeling has to start by acknowledging how cheap the alternative is at low volume.

The break-even calculation

Divide the annual local cost by the per-request hosted cost:

$464 ÷ $0.004 = 116,000 requests per year, which is about 318 requests per day.

The table changes the conclusion depending on which tier you would otherwise use, and that is the single most important output of the exercise. If your work genuinely needs a flagship hosted model, self-hosting pays back at only around 64 requests a day. If a cheap hosted tier would do the job, you need five times that volume.

But note the asymmetry buried in that comparison: a 13B model you can run locally is not equivalent to a hosted flagship. Comparing their prices is only valid if both actually complete your task.

The costs neither side advertises

Local, understated: your time. Installing, updating, debugging a runtime and swapping models is real labour, and at any professional hourly rate a few hours a month exceeds the electricity. Add cooling and noise in a small room, and the opportunity cost of a machine tied up serving inference.

Hosted, understated: retries and failures still bill. Long system prompts bill on every request, which is why prompt caching matters commercially. Reasoning modes that produce internal tokens bill for those. And the tier you start on is rarely the tier you end on.

Both: the human review cost of a wrong answer usually dwarfs the inference cost of producing it. A cheaper model that needs checking twice is not cheaper.

Why the answer is usually both

The two cost structures are complementary rather than competing. Local inference has a high fixed cost and near-zero marginal cost; hosted has zero fixed cost and a linear marginal cost. Any workload with a large predictable base and an occasional hard tail is cheapest split across both.

Route high-volume, low-difficulty work — classification, extraction, tagging, first-draft summarisation — to the local model, where volume is free. Route the genuinely hard requests to a hosted flagship, where you pay only for the few that need it. Sound local AI cost modeling almost always produces this shape rather than a single winner.

The engineering that makes it work is the router, and the number to watch is its misclassification rate. Sending hard work down to a small model converts a saving into rework.

Your own worksheet

• Annual local cost = (hardware ÷ years) + (watts ÷ 1000 × hours per year × price per kWh).
• Per-request hosted cost = (input tokens ÷ 1M × input rate) + (output tokens ÷ 1M × output rate).
• Break-even requests per year = annual local cost ÷ per-request hosted cost.
• Sanity check: can the local model you can actually load complete the task at all?
• Add your labour at a real hourly rate, then re-run it.
• Model the hybrid split before choosing either extreme.

Measure your own token counts rather than estimating them — most people are wrong by a factor of two about how long their prompts are, and that error moves the break-even point more than any other input in the model.