Choosing a Local Coding Model: Benchmarks, Context and Licenses

Why coding benchmarks travel badly

Published coding scores are produced on curated problem sets with a scaffold around the model, and both parts move the number. The scaffold — how many attempts are allowed, whether tests are run and errors fed back, how long the model is permitted to think — frequently matters more than the model.

Three specific reasons a score does not predict your experience. Benchmark problems are self-contained functions, while real work is a change inside an existing codebase with conventions and dependencies. Benchmark languages skew heavily toward Python, so a strong score says little about your TypeScript or Rust project. And contamination is a persistent concern: widely published problems may appear in training data, which inflates results in a way no leaderboard can detect.

The consequence for a local coding model comparison is that scores are useful only for excluding the clearly unsuitable. Beyond that, the ranking has to come from your own repository.

Context is the binding constraint

Coding is the task where context length stops being a specification and becomes the limit on usefulness, because a real change touches several files at once.

Do the arithmetic in tokens. A rough working rule is that source code runs around three to four characters per token, so a 500-line file of typical density is somewhere in the low thousands of tokens. A change that needs the file you are editing, two files it imports, a test file and a type definition can therefore consume ten thousand tokens before the model has produced anything.

That has two implications. A model with a short context cannot see enough of your project to be useful for anything beyond single-function work, regardless of its benchmark score. And a long advertised context is not the same as usable recall across that whole window — models frequently degrade at retrieving detail from the middle of a very long input, so test retrieval at the length you actually intend to use.

Fit the model to your memory

Local means memory-bound. Weight memory is parameters times bits divided by eight, so the tiers available on common hardware are narrow:

Because coding needs long context, the trade is sharper here than for chat. Reserve memory for the context before choosing the model size, not after — a 24B model that leaves no room for four files open is worse at your job than a 13B that does.

Prefer a code-specialized model at a given size over a general model of the same size. The specialisation is worth more than a step up in parameters for this task, which is the clearest finding in any local coding model comparison.

The license question for code

Code generation raises license questions that chat does not, and they need answering before the model touches a commercial repository.

• Does the model license permit commercial use without a threshold or a separate agreement?
• Are there restrictions on the outputs — some licenses constrain what generated content may be used for.
• Can you use outputs to train another model? Frequently prohibited.
• Is it a standard license such as Apache 2.0, or bespoke? Bespoke needs legal reading.
• What does the model card say about training data? This is what you will be asked first if provenance is ever questioned.

The practical mitigation, independent of license: review generated code as you would a contribution from an unfamiliar developer. That is the correct standard regardless of which model produced it.

Test with your own harness

Build a small fixed evaluation from your own repository and it will outrank every published comparison. Ten tasks is enough to be decisive.

1. Pick ten real changes you have already made, ideally with tests that already pass.
2. Revert each one and record the files a developer would need open to redo it.
3. Give each model the same files and the same instruction, with the same number of attempts.
4. Score by whether the tests pass, not by whether the code looks plausible.
5. Record tokens, time and how often you had to intervene.
6. Run it twice per model, because output varies between attempts.

Remember the resolution limit: with ten tasks, one task is ten percentage points. Treat small gaps as ties and decide those on context length, memory footprint and license instead.

The checklist

• Measure your typical working context in tokens before shortlisting anything.
• Reserve memory for context first, then pick the largest model that still fits.
• Prefer a code-specialized model over a larger general one.
• Read the license for commercial use, output restrictions and training clauses.
• Test retrieval at your real context length, not just at the advertised maximum.
• Score on passing tests, twice per model, on your own repository.

Run that and a local coding model comparison usually resolves to a 13B-class code model at 4-bit with a generous context allowance — a conclusion no leaderboard would have handed you, because the leaderboard does not know how much memory you have or which language you write.