Your Code Doesn't Have to Leave the Building

Every developer I know has done the thing. Deadline pressure, a function that won't behave, and you paste it into a chat box because it's four in the afternoon and you want to go home.

I've done it. You've done it. And most of us have never read the retention policy of the service we did it to.

That's the actual situation. Not a hypothetical about corporate espionage — just a quiet, cumulative transfer of proprietary code out of the building, one frustrated paste at a time, into systems whose data-handling terms change by press release. You will find out what happened to it the same way you find out about anything: later, and from someone else.

The good news is that the alternative stopped being a compromise. In 2026 you can run a model on your own hardware that outscores GPT-4o on the coding benchmarks that still mean something. It costs an afternoon and about forty gigabytes of disk. Here's the whole stack, and enough of the internals that you can fix it yourself when it misbehaves.

Three Layers, and You Can Replace Any of Them

Before an install command, get the stack straight. It's three pieces and troubleshooting depends entirely on knowing which one broke.

The weights. A large binary — usually GGUF format — holding billions of numerical parameters. That's the model. A 7B at 4-bit quantization is roughly 4.6GB on disk; a 70B at the same quantization is about 43GB. Those numbers aren't rules of thumb, they're arithmetic, and the section on quantization below shows how to compute them for any model you're considering. These files are inert. They don't do anything.

The runtime. Software that loads those weights into memory and runs inference against them. We're using Ollama, which wraps llama.cpp behind a CLI and a REST API, handles GPU allocation, and manages model storage. It listens on localhost:11434. Think Docker: you pull a thing, you run it.

The interface. Anything that speaks HTTP to that port. curl. A Python script. A VS Code extension. A web UI. The runtime does not care.

Weights on disk. Runtime loads them and exposes an API. Interface talks to the API. Swap any layer without touching the others — and when something breaks, you now know which of three places to look instead of staring at the whole thing.

VRAM Is the Whole Question

Everything about local inference is a memory problem. The weights have to be somewhere fast, and how fast that somewhere is determines your tokens per second.

Discrete GPU: VRAM, and the model needs to fit entirely inside it. Spill into system RAM and you don't degrade gracefully — you fall off a cliff. How steep that cliff is comes out of the arithmetic below, and it is much worse than most guides admit.

Apple Silicon: unified memory. CPU and GPU share the same pool at hardware level, so all your system RAM is effectively VRAM. This is why a MacBook with 36GB competes with a desktop running a 3090, and why the Apple path scales past where consumer GPUs stop.

CPU only: system RAM. Works for 3B–7B models at conversational speed. Anything bigger gets painful — single-digit tokens per second, which is fine for a question you'll wait on and useless for autocomplete.

Where Those Speeds Come From

Most guides hand you a table of tokens per second with no explanation, which makes it impossible to reason about hardware they didn't list. So here's the model.

For a dense transformer, generating one token requires reading every weight from memory once. That makes generation memory-bandwidth-bound, not compute-bound, and gives you a hard ceiling:

tokens/sec ceiling = memory bandwidth ÷ model size in memory

Published bandwidth figures, run against common model sizes at Q4:

                                    7B       14B       32B       70B
RTX 4090 (GDDR6X, 1008 GB/s)       214       112        50        25
RTX 3090 (GDDR6X,  936 GB/s)       199       104        47        23
M3 Ultra (unified, 800 GB/s)       170        89        40        20
M3 Max 40-core   (400 GB/s)         85        44        20        10
M3 Max 14-core   (300 GB/s)         64        33        15         8
DDR5-6000 dual channel (96 GB/s)    20        11         5         2

Those are ceilings, not measurements — real throughput lands somewhere under them depending on the runtime, the quantization, batch handling and how much of the bandwidth you actually get. But the ratios hold, and the ratios are what you're buying.

Check it against published benchmarks and the model behaves: a 70B at Q4 on an M3 Ultra has a ceiling of 20 tok/s, and reported real-world figures for 70–72B on high-end Apple Silicon sit around 9–14. Roughly half the ceiling, consistently, which is what you'd expect.

Now the important number. GDDR6X on a 4090 moves about 1,008 GB/s. Dual-channel DDR5 moves about 96. That's a 10.5× gap, which means any layer that doesn't fit in VRAM runs about ten times slower than the same layer would have in VRAM.

That is the cliff. Not a 30% penalty, not "somewhat slower" — an order of magnitude on every offloaded layer, which is why published tests of a 70B at Q4 on a single 24GB card report throughput collapsing into single digits. The model didn't get worse. Most of it is being read across a bus a tenth as fast.

It also explains the two things about this table that look strange at first glance. Apple Silicon competes with discrete GPUs despite lower peak bandwidth because the model fits — 400 GB/s that holds all the weights beats 1,008 GB/s that holds two thirds of them. And CPU-only inference isn't slow because CPUs are weak at math; it's slow because dual-channel DDR5 is a tenth the pipe.

Capacity first. Bandwidth second. In that order, always.

The practical floor for real work is an 8GB GPU running a 7B coding model. Twenty to forty tokens per second is faster than you read. That's enough.

If you're buying: a used RTX 3090 is still the value pick in 2026. Same 24GB as a 4090, runs the same models, costs meaningfully less. The 4090 is faster; it is not more capable. Above that, Apple Silicon with 64GB+ is the only consumer route to 70B-class models without stacking cards.

On AMD, be careful, because this is where the previous version of this guide contradicted itself. AMD acceleration on Linux via ROCm is genuinely competitive — a 7900 XTX trades blows with a 3090. On Windows, Ollama does not officially support AMD GPU acceleration; you're on WSL2 or CPU. And check Ollama's current ROCm version requirement before you install anything, because it has moved more than once and a guide written six months ago will send you to the wrong package.

Storage: 100GB of NVMe if you want a few models around. Capacity is the obvious reason; load time is the one people underestimate. Model load is a straight sequential read, so it's pure bandwidth:

                            4.7GB (7B)  20GB (32B)  43GB (70B)
NVMe Gen4 (~7 GB/s)               1 s         3 s         6 s
NVMe Gen3 (~3.5 GB/s)             1 s         6 s        12 s
SATA SSD (~550 MB/s)              9 s        37 s        79 s
7200rpm HDD (~150 MB/s)          32 s      2.3 min     4.8 min

A 32B off NVMe is a three-second pause. Off a spinning disk it's two and a half minutes, every cold start, every time the keep-alive window expires. That's not a minor inconvenience, it's the difference between a tool you reach for and one you avoid because you don't want to wait.

Installing the Runtime

Linux:

curl -fsSL https://ollama.com/install.sh | sh

That sets up a systemd service. Verify it's alive:

ollama --version
systemctl status ollama

macOS and Windows get installers from ollama.com/download. One macOS gotcha worth stating because it wastes an evening: if you install the app, do not also run ollama serve. The app already runs the server. Both means a port conflict on 11434 and a confusing error.

Then the only verification that actually matters:

ollama run llama3.1:8b "say hello"
ollama ps

ollama ps shows you where the model actually landed:

NAME              ID              SIZE    PROCESSOR    UNTIL
llama3.1:8b       365c0bd3109b    6.7 GB  100% GPU     4 minutes from now

100% GPU is the goal. If you see 100% CPU on a machine with a supported card, stop and fix the driver before going further. The difference is roughly 35 tokens per second against 5, and every other decision you make downstream is meaningless until this line is right.

Which Models, and Why the Benchmark Everyone Quotes Is Useless

For coding, the Qwen2.5-Coder family is the answer on consumer hardware, and it isn't close.

ollama pull qwen2.5-coder:7b     # fits 8GB VRAM
ollama pull qwen2.5-coder:14b    # 16GB, noticeably better on hard tasks
ollama pull qwen2.5-coder:32b    # 24GB, where local stops apologising

Now, the numbers — and this is the part the previous version of this guide got wrong, so let's do it properly.

You will see HumanEval scores quoted everywhere. Ignore them. The benchmark is saturated: nearly every current model scores above 90%, which means it has stopped distinguishing between them. Qwen2.5-Coder-7B scores in the high 80s. The 32B is around 92%. GPT-4o is about 90%. Those numbers are all the same number wearing different hats, and a two-point gap on a saturated benchmark tells you nothing about whether the thing can refactor your service layer.

The benchmarks that still separate models are the ones that haven't been trained into oblivion:

• LiveCodeBench — Qwen2.5-Coder-32B scores 37.2%, against GPT-4o's 29.2%. The 7B lands around 28.6%.
• Aider (code repair, not generation) — the 32B scores 73.7, roughly comparable to GPT-4o.
  Read that first line again. A 32B model you can run on a single 24GB card outscores GPT-4o on LiveCodeBench. Not "approaches." Outscores. That's the actual state of local coding models in 2026, and it's a much stronger claim than anything HumanEval will tell you.

For general chat and research, Llama 3.1 8B is the workhorse and 3.3 70B is the ceiling if you have the memory. For step-by-step debugging, DeepSeek R1 thinks out loud before answering — you'll see <think> tags in the output, which is the point, not a bug. On minimal hardware, Phi-4 Mini and Gemma 3 1B are surprisingly capable for their size.

Quantization: Fewer Bits, More Parameters

Weights are trained at FP16 or FP32. Quantization drops that precision to save memory, and the key insight is that inference doesn't need training precision. Going FP16 → 4-bit cuts the footprint roughly 75% for modest quality loss.

The naming is misleading, though, and it's worth knowing why. A "4-bit" k-quant is not 4 bits per weight — the K-quant formats store block scales and minimums alongside the weights, so the real figure is higher. Effective bits per weight, which is what actually determines file size:

Which makes model size a one-line calculation:

size in GB = parameters (billions) x bits_per_weight / 8

Check it against reality — computed against what Ollama actually ships:

  qwen2.5-coder:7b     computed  4.6 GB | published  4.7 GB | delta 2.1%
  llama3.1:8b          computed  4.8 GB | published  4.9 GB | delta 1.1%
  qwen2.5-coder:32b    computed 19.8 GB | published 20.0 GB | delta 1.0%

Within two percent across the range. So you can size any model on any hardware before downloading a single byte, including ones nobody has written a guide about.

Note also that Q4_K_M lands at 70% smaller than F16, not the 75% everyone repeats — that gap is the k-quant metadata, and on a 32B model it's a couple of gigabytes you need to budget for.

The _K_M suffix means mixed quantization: more precision preserved in the layers that affect output quality most. It beats naive Q4 at nearly identical size, which is why Ollama pulls it by default.

Now the rule worth internalising: more parameters at lower precision beats fewer parameters at higher precision. Run the arithmetic on a 12GB budget:

  7B  @ Q8_0   =  8.1 GB
  14B @ Q4_K_M =  8.9 GB   <- more params, same budget

Nearly identical footprint, twice the parameters. Take the bigger model and the cheaper bits.

The Context Setting That Silently Eats Your Data

This is the single most expensive thing to not know, and it's why "the model forgot what we were talking about" is the most common complaint about local setups.

When your conversation exceeds the context window, Ollama does not error. It silently drops the oldest tokens and keeps going. The model answers confidently using half your input. Nothing in the output tells you this happened.

Worse, the default is smaller than people assume and it is no longer one number. Ollama now picks a default based on detected VRAM: roughly 4K under 24GiB, 32K for 24–48GiB, and 256K above that. So the same command on your workstation and your laptop gives you two very different context windows, and only one of them will quietly truncate the file you pasted.

There are three ways to change it, and they are not equivalent.

In the REPL, temporarily — gone when the model unloads:

/set parameter num_ctx 32768

For the whole server, via environment variable:

OLLAMA_CONTEXT_LENGTH=64000 ollama serve

Permanently, per model, with a Modelfile — the one that actually survives a reboot:

FROM qwen2.5-coder:14b
PARAMETER num_ctx 32768

ollama create qwen-coder-32k -f Modelfile

That rebuild doesn't re-download anything; it stacks a config layer on the existing weights. Note the underscore in num_ctx — a hyphenated num-ctx inside a Modelfile is silently invalid, which is a wonderful thing to discover three hours later.

And via the API, per request:

curl http://localhost:11434/api/generate -d '{
  "model": "qwen2.5-coder:14b",
  "prompt": "...",
  "options": { "num_ctx": 32768 }
}'

Context costs VRAM. A 32K window uses substantially more than 4K on the same model — budget for it, or you'll push layers out of GPU and wonder why everything got slow. For agent and coding-tool workloads, Ollama's own guidance is to set at least 64,000 tokens explicitly rather than trusting the auto-default.

If you take one thing from this article, take this: check your context length before you blame the model.

What Context Actually Costs in VRAM

"Larger context uses more VRAM" is where every guide stops. That's not good enough to plan with, because the number is computable and it's bigger than people expect.

The KV cache holds the key and value tensors for every token in the window, on every layer. Its size is:

bytes = 2 (K and V) x layers x kv_heads x head_dim x context x bytes_per_element

Qwen2.5-7B's published architecture: 28 layers, hidden size 3584, 28 query heads, 4 key-value heads. Head dimension is 3584 ÷ 28 = 128. At fp16, two bytes per element:

2 x 28 x 4 x 128 x 2 = 57,344 bytes per token

Fifty-seven kilobytes of VRAM per token of context. Which scales exactly how you'd fear:

   context     KV cache    total w/ 4.7GB Q4 weights
     4,096       0.22 GB                     4.92 GB
     8,192       0.44 GB                     5.14 GB
    16,384       0.88 GB                     5.58 GB
    32,768       1.75 GB                     6.45 GB
    65,536       3.50 GB                     8.20 GB
   131,072       7.00 GB                    11.70 GB

Read the right column against your card. A 7B at Q4 with a 4K window fits an 8GB GPU with room to spare. Push it to 64K and you need more than 8GB — the weights didn't change, the cache ate the difference. That's the actual mechanism behind "I raised my context and everything got slow." You didn't hit a software limit. You pushed layers out of VRAM and into system RAM, and now you're paying the spill penalty on every token.

There's a second thing hiding in that formula worth understanding, because it's why any of this is affordable at all. Qwen uses grouped-query attention: 28 query heads share just 4 key-value heads. The KV cache scales with kv_heads, not q_heads. Run the same arithmetic with 28 KV heads, as older architectures had:

without GQA (28 KV heads) at 32k: 12.25 GB   <- 7x larger

Seven times the cache. A 32K context on a 7B model would need over twelve gigabytes for the cache alone, before the weights. GQA is the single architectural change that made long context viable on consumer hardware, and it's invisible in every spec sheet you'll read.

Plug your own model's numbers into that formula before you set num_ctx. ollama show <model> gives you the architecture. Then you're sizing a context window on arithmetic instead of raising it until something breaks.

Halving the Cache Without Touching the Context

The KV cache doesn't have to be fp16. Ollama can quantize it, and this is the single biggest lever for running long context on a small card.

OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve

Per Ollama's own FAQ, q8_0 uses about half the memory of f16 "with a very small loss in precision, this usually has no noticeable impact on the model's quality." q4_0 gets you to roughly a quarter with a real, measurable trade-off. Applied to the numbers above:

  context       f16      q8_0      q4_0
    8,192    0.44 GB   0.22 GB   0.11 GB
   32,768    1.75 GB   0.88 GB   0.44 GB
   65,536    3.50 GB   1.75 GB   0.88 GB
  131,072    7.00 GB   3.50 GB   1.75 GB

Now make it concrete. An 8GB card holding a 7B at Q4 has roughly 3GB left after weights and overhead:

    65,536 @ f16   =  3.50 GB  spills
    65,536 @ q8_0  =  1.75 GB  FITS
   131,072 @ f16   =  7.00 GB  spills
   131,072 @ q4_0  =  1.75 GB  FITS

A 64K context that does not fit at f16 fits comfortably at q8_0, on the same hardware, for a precision loss the Ollama docs describe as unnoticeable. That's a genuinely free upgrade and almost nobody sets it.

Three caveats, and the third one will cost you an afternoon if you don't know it.

It requires flash attention to be active. Older guides tell you to set OLLAMA_FLASH_ATTENTION=1; on current builds it's on by default for supported hardware and the variable now acts as a three-state override.

Flash attention does not shrink the cache. This gets repeated wrong constantly. It computes attention in blocks that fit in GPU SRAM, which makes the operation faster and lowers peak transient memory. The K and V vectors are still stored in full. It is a speed optimisation that happens to be a prerequisite for the memory optimisation.

On unsupported architectures it silently falls back to f16. Not a warning. Not an error. Ollama's own issue tracker documents developers setting q8_0, calculating their VRAM budget against half-size cache, and then hitting unexpected OOMs because the server quietly kept using f16. So verify rather than assume:

grep "KV self size" ~/.ollama/logs/server.log

That line reports the actual cache size and type the runtime allocated. If it says f16 after you asked for q8_0, your architecture isn't on the allowlist and every capacity calculation you just did is wrong.

One more detail worth knowing if you push to q4_0: the K cache is measurably more sensitive to quantization than the V cache. Ollama exposes a single global setting, but the underlying llama.cpp allows them to be set separately, and the quality-per-byte sweet spot in community testing has been q8_0 keys with q4_0 values. If you're running llama.cpp directly, that's the configuration to try before accepting q4_0 across the board.

A Web Interface, in One Command

Terminal is fine for smoke tests. For real use you want history, file uploads, and model switching. Open WebUI, entirely local:

docker run -d \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui \
  --restart always \
  ghcr.io/open-webui/open-webui:main

The flag that people drop and then can't debug is --add-host=host.docker.internal:host-gateway. Without it the container has no route to Ollama on the host, and the failure looks like Open WebUI simply not finding any models.

It lands at http://localhost:3000 and asks you to create an admin account — stored locally, no registration, nothing leaves. If auto-detection fails, point it at http://host.docker.internal:11434 on macOS and Windows, or http://172.17.0.1:11434 on Linux.

Wiring It Into VS Code

This is where local stops being a privacy argument and starts being a tool you actually use. The Continue extension gives you inline completion and chat against your local runtime.

Configure two models, not one, and configure them differently:

• Autocomplete fires on every keystroke. Put the 7B there. Latency is the entire product.
• Chat is something you deliberately invoked and are already waiting on. Put the 14B or 32B there.
  Continue has changed its configuration schema more than once — check its current docs for the exact file and format rather than copying a config from any guide, including this one. What matters conceptually is that each entry points at provider: ollama and apiBase: http://localhost:11434, and that the autocomplete model is the small fast one.

The feature that justifies the whole exercise is codebase context. Asking "find everywhere we don't handle the error from the database call" means indexing your repository. With a cloud model that means shipping your repository to somebody else — which is exactly the thing most employers prohibit and most developers do anyway. Locally it's just a disk read.

A review loop that works:

1. Make changes on a branch.
2. In chat: @diff Summarize these changes and flag any bugs or security issues.
3. Read the response as a first-pass reviewer, not a verdict.
   Nothing leaves. The diff stays on the machine that made it.

When It Breaks

Five failures cover almost everything, and each one has a diagnostic that takes ten seconds.

ollama ps says 100% CPU on a machine with a GPU.

The runtime never found your card. Check the driver first, then the logs:

nvidia-smi                       # NVIDIA: driver present and version?
rocm-smi                         # AMD on Linux: ROCm present?
journalctl -u ollama -n 50       # what did detection actually say?

The Ollama log records the detection attempt and the specific reason it failed. Usual causes: driver older than Ollama's minimum, missing ROCm runtime on AMD Linux, or a card that isn't on the supported list. Fix this before anything else — you are otherwise tuning a system running seven times slower than it should.

Output is garbled, repetitive, or incoherent.

Check ollama ps for a split load. If the processor column says something like 60% GPU / 40% CPU, the model didn't fit and part of it is in system RAM. Pull a smaller quantization, pull a smaller model, or reduce context so the whole thing fits. Coherent-but-slow is a spill; incoherent usually means something worse — try re-pulling the model in case the download corrupted.

The model "forgot" the beginning of the conversation.

Context truncation, silently. Covered above — check your num_ctx against what you actually fed it. This is the single most common complaint about local setups and it is almost never a model quality problem.

Open WebUI can't see any models.

Test the runtime directly first, from the host:

curl http://localhost:11434/api/tags

JSON list of models means Ollama is fine and the problem is Docker networking — confirm --add-host=host.docker.internal:host-gateway is present, and on Linux try pointing the UI at http://172.17.0.1:11434. A connection error means Ollama itself isn't running and the UI is innocent.

Everything is slow despite a working GPU.

Something else is holding VRAM. Watch it live:

nvidia-smi --loop=1

Another model still loaded, a browser doing GPU compositing, a game you forgot about. Unload what you're not using:

ollama stop <model>

Models stay resident for a keep-alive window after use, which is a feature until you're running two and wondering why the second one crawls.

What to Check Before You Ask Anyone for Help

In order, because this sequence answers most questions before they get asked:

ollama ps                                    # is it on the GPU at all?
ollama show <model>                          # architecture, context, quantization
grep "KV self size" ~/.ollama/logs/server.log  # what cache did it actually allocate?
journalctl -u ollama -n 50                   # what did it say when it loaded?

Four commands. Between them they tell you where the model is running, how big its window is, what the cache cost, and what the runtime thought about all of it. Almost every "why is my local LLM bad" question is answered somewhere in that output.

When the Cloud Is Still the Right Call

I'd rather be useful than partisan, so: local is not always the answer.

Frontier reasoning still favours the cloud. Decompiling obfuscated binaries, architecting a distributed system, chasing a subtle race condition across a large codebase — the biggest hosted models remain better at the genuinely hard end. Local has closed most of the gap on everyday work and not all of it on the hard stuff.

Wanting a 70B without the memory for one is not a plan. Running a 70B at Q4 on an 8GB card means spilling to system RAM and living at single-digit tokens per second. You will hate it and stop using it.

Serving customers is a different problem. A model on your workstation is a tool. A model serving production traffic is inference infrastructure — batching, queueing, autoscaling, GPU scheduling. Don't confuse the two because the ollama run command looked easy.

And you'll lag the frontier by days to weeks. New models reach hosted APIs before they reach the Ollama library.

For everything else — daily coding, review, research, documentation, the endless small questions you'd never open a ticket for — local wins on cost, on latency, on being there at 3AM when the hosted API is having an incident.

And on the one thing you can't buy back. Every other cost here is recoverable: hardware depreciates, models get better, a bad quantization choice is a re-pull. Code that left the building did that permanently, and no amount of hardware brings it home.

That's the argument. Everything above is just the implementation.