Why is my local LLM so slow compared to online models?

Because a hosted model runs on hardware with an order of magnitude more memory bandwidth than a laptop, often across several accelerators. The gap is architectural rather than a configuration problem. What a laptop offers instead is privacy and offline use, which is the actual reason to run models locally.

Does more VRAM make AI models faster?

It decides whether the model fits, which is the difference between usable and unusable, but it does not itself set the speed once it fits. Bandwidth does that. Capacity and bandwidth usually rise together on better hardware, which is why they are easy to confuse.

What is the best quantisation for a laptop?

4-bit quantisation is the common sweet spot: it roughly halves the memory a model needs against 8-bit, reads faster, and costs surprisingly little quality for most uses. If a model only fits at a heavier quantisation, a smaller model at a lighter one is often the better trade.

Problems · Under load

Why is my local AI model so slow on my laptop?

Because generating tokens is mostly a memory bandwidth problem rather than a compute problem. The model's weights have to be read for every token produced, so how fast your machine can move data sets the speed, and a faster processor with the same bandwidth changes very little.

That explains most of what people find confusing here: why quantisation helps so much, why a model that barely fits runs disproportionately badly, and why a machine with unified memory can outperform one with a nominally stronger graphics chip.

First, notice when it happens

From the first minute

Whether the model loads and how quickly the first token appears is capacity and prompt processing, and it depends on whether the model fits in fast memory at all.

Only after minutes of use

Long generations are sustained loads and behave like any other, settling as the machine reaches its thermal limit, which is why a long answer slows toward the end on thin laptops.

The causes, ranked

1. The model not fitting in fast memory

most common

How to confirm it

Generation that is not merely slow but dramatically slow, often with heavy disk activity, means the model has spilled out of graphics or unified memory.

The fix

Use a smaller model or a heavier quantisation so it fits. The difference between just fitting and just not fitting is far larger than any other variable here.

2. Memory bandwidth, which sets the ceiling

most common

How to confirm it

Compare your machine's bandwidth figure against others running the same model. Token rates track bandwidth closely across very different machines.

The fix

Nothing to change on the machine you own, but everything to know before buying the next one. Bandwidth predicts token speed better than any processor specification.

3. Quantisation level

common

How to confirm it

Running a model at full precision when a quantised version would fit comfortably and read faster.

The fix

Use a quantised build. The quality cost is usually small and the speed and capacity gains are large, which is why nearly everyone running models locally does this.

4. Long context lengths

common

How to confirm it

Speed that degrades as a conversation grows, because the context has to be processed and held alongside the model.

The fix

Trim the context, start fresh conversations for new topics, and be aware that a large context window consumes memory that the model needed.

5. Running on the processor instead of the accelerator

common

How to confirm it

Check which device the runtime chose. Falling back to processor-only inference is much slower and happens silently when a model does not fit.

The fix

Set the device explicitly and check the layer offload settings. A partial offload can be far slower than a smaller model that fits entirely.

6. Thermal settling during long generations

occasional

How to confirm it

Sustained inference over minutes brings the machine to its thermal limit like any other load.

The fix

Mains power, performance profile, hard surface. Worth doing, though far smaller in effect than the memory questions above.

Measure it instead of guessing

Run the deep test to see what your machine sustains, since a long generation sits in exactly that territory. It will not measure tokens per second, but it will tell you whether the machine holds its power through a long job or fades, which is the difference between a generation that finishes evenly and one that crawls at the end.

Run the free test

What software cannot tell you here

We cannot measure token rates, memory bandwidth or model fit, which the runtime itself reports.
We cannot see your quantisation or offload settings, which are the levers that matter most here.

Asked, in people’s own words

Why is my local LLM so slow compared to online models?: Because a hosted model runs on hardware with an order of magnitude more memory bandwidth than a laptop, often across several accelerators. The gap is architectural rather than a configuration problem. What a laptop offers instead is privacy and offline use, which is the actual reason to run models locally.
Does more VRAM make AI models faster?: It decides whether the model fits, which is the difference between usable and unusable, but it does not itself set the speed once it fits. Bandwidth does that. Capacity and bandwidth usually rise together on better hardware, which is why they are easy to confuse.
What is the best quantisation for a laptop?: 4-bit quantisation is the common sweet spot: it roughly halves the memory a model needs against 8-bit, reads faster, and costs surprisingly little quality for most uses. If a model only fits at a heavier quantisation, a smaller model at a lighter one is often the better trade.

Name the cause. Three and a half minutes, in the tab you have open.

Run the test