Quantisation (LLM)

A model's weights take parameters times bytes per weight. Bytes per weight: 2 at 16-bit, 1 at 8-bit, 0.5 at 4-bit. So a 7 billion parameter model's weights are 14 GB at 16-bit, 7 GB at 8-bit, 3.5 GB at 4-bit. Quantisation, storing the weights at fewer bits, is usually what moves a model from not fitting in a laptop's graphics memory to fitting.

Halving the bits halves the weights' memory and the data read for every generated token, which is why quantised models also tend to generate faster on memory-bound hardware. It can cost some quality, and by how much depends on the model and the method, a question for the model's own documentation. It does not shrink the context: the memory a runtime keeps for the conversation grows with its length on top of the weights, and real quantised files are somewhat larger than the bare arithmetic because they store scales alongside the weights.

The local AI readiness check sets the weights of each common size, at each precision, against the graphics memory the browser let it allocate and verify, and reports the largest that fits. It does not predict tokens per second, because the runtime, the model's design and the context length all change that.