Quantization
Quantization is a compression technique that rewrites a model's weights from high-precision decimals (32-bit floats) to smaller numbers (8-bit or 4-bit integers), so the model takes up less memory and runs faster on cheaper hardware, at a small, measurable cost in accuracy.
No jargon, no math
In plain words, what is it?
A neural network is a pile of numbers, the weights. Trained, they are stored as decimals with many digits (32-bit floats). Quantization rounds every weight to a much coarser grid: instead of 0.673891 it stores 0.7, or an integer code that stands for it. Two wins follow. First, memory: an 8-bit weight takes a quarter of the space, so a 14 GB model fits in under 4 GB. Second, speed: integer math is what CPUs and phone chips do natively, so tokens come out faster even on hardware without a big GPU. The trade-off is accuracy. Rounding blurs the weights a little, and the model can get slightly worse at exact tasks: arithmetic, rare words, precise formats. Push to 4 bits and the blur grows, and some models collapse. So you quantize, then test on your real task, and you keep the smallest size that does not hurt. That test is not optional.
Terms that live nearby.
Inference
Inference is using a trained model: you send input in, the model applies what training encoded in its parameters, and you get an answer out, one token at a time.
ReadParameter
A parameter is one of the internal numbers a trained model stores: billions of them, each tuned during training, and together they encode everything the model learned.
ReadVRAM
VRAM is the dedicated memory on a graphics card or AI accelerator that holds the model's weights and the intermediate numbers of each calculation, so the processors doing the AI work never wait on slower system RAM.
Read§ 03
Want this running in your stack?
We design and ship AI systems with permissions, logs and human approval. It starts with a 30-minute scoping call.