What is Quantization?
Also called model compression, low-precision inference.
Quantization stores a model's parameters at lower numeric precision, for example using eight-bit or four-bit integers instead of sixteen-bit floating point numbers. This shrinks memory use roughly in proportion to the bit reduction and usually speeds up inference. Accuracy declines with it, typically slightly at eight bits and more noticeably at four bits and below.
Memory, not raw arithmetic, is the binding constraint when serving models, so halving the bits per weight roughly halves the memory needed and often lets a model fit onto cheaper or more widely available hardware. Modern accelerators also execute low-precision arithmetic faster than full precision, so the change usually improves throughput as well as footprint.
Methods differ mainly in when they are applied. Post-training quantization converts an existing model directly and is fast but loses more accuracy. Quantization-aware training simulates low precision during the training run so the model adapts to it, giving better results at a higher cost. Mixed schemes keep the most sensitive layers at higher precision throughout.
Degradation is uneven, and that unevenness is the trap. A quantized model can score almost identically on common public evaluations while getting measurably worse at long-context recall, multi-step arithmetic, code generation, or lower-resource languages. Testing on the actual workload, rather than on a general leaderboard, is the only reliable way to check.
Quantization is distinct from distillation and pruning, though all three shrink models. Distillation trains a smaller model to imitate a larger one, pruning removes weights or whole structures entirely, and quantization keeps every weight but describes each one with fewer bits. The three are frequently combined in the same deployment pipeline.
Key points
- Stores weights at lower bit precision to cut memory and increase speed.
- Eight-bit is usually near-lossless; four-bit and below degrade more.
- Losses concentrate in long context, math, code, and rarer languages.
- Different from distillation and pruning, and often combined with them.
In practice
A model whose weights need sixteen gigabytes in half precision will not fit a consumer graphics card. Quantized to four bits, it needs roughly four gigabytes and runs locally. Everyday summarizing and drafting feel unchanged. Ask it to trace a subtle bug across three files and the errors that were rare in the full-precision version become common.