LLM Quantization: How to Make Language Models Smaller and Faster

Quantization stores model weights, and optionally activations and the KV cache, in fewer bits so models need less memory and bandwidth and often run faster. BF16 or FP16 is the usual baseline; FP8 and INT8 typically cost little quality on supported hardware; 4-bit formats such as GPTQ, AWQ, GGUF variants and newer 4-bit floating-point formats save more but need careful evaluation. Choose formats your hardware and serving engine accelerate, calibrate with representative data and compare quality, latency and cost against the unquantized model on your own tasks.

Quantization is one lever in AI inference optimization. Hardware considerations are in GPU optimization for AI, local and device deployment in AI edge deployment and running open-weight models in LLM self-hosting.

A model with 8 billion parameters needs roughly 16 GB just for weights in 16-bit precision, about 8 GB in 8-bit and about 4 to 5 GB in 4-bit formats including overhead. Smaller weights fit on smaller or fewer GPUs, leave more memory for the KV cache (and therefore more concurrent users or longer contexts) and reduce the bytes read per generated token, which speeds up memory-bound decoding.

Post-training quantization converts a trained model without retraining, using a small calibration dataset. GPTQ quantizes weights layer by layer to minimize output error; AWQ scales weights to protect those most important to activations. Quantization-aware training simulates low precision during training or fine-tuning for better quality at low bit widths, at higher cost. QLoRA fine-tunes adapters on top of a 4-bit base model, reducing fine-tuning memory.

Serving engines support many formats; vLLM, for example, lists FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF and others. Check that your engine accelerates, rather than merely loads, the format you choose.

Quality loss is uneven. Quantized models may perform nearly identically on common tasks but degrade on reasoning, long contexts, code, maths, less common languages or strict output formats. Smaller models tend to be more sensitive than larger ones. Calibration data matters: calibrate on text resembling your workload. Never rely on a single benchmark score.

For long contexts and high concurrency, the KV cache can use more memory than the weights. Some engines support storing it in FP8 or other reduced formats, increasing the number of tokens that fit. Evaluate long-context tasks specifically, since errors can accumulate over long sequences.

Speed gains require hardware and kernel support. FP8 compute is available on recent data-centre GPU generations; 4-bit floating-point formats need the newest architectures; integer formats have broad support through optimized kernels. On CPUs and Apple silicon, llama.cpp and GGUF are common. On phones and embedded devices, mobile runtimes have their own supported formats; see AI edge deployment.

Quantization often makes self-hosting and edge deployment practical and can reduce serving costs considerably. It can quietly reduce quality on specific tasks, gains depend on hardware support and quantized community models vary in reliability. Treat it like a model change: evaluate, roll out gradually and monitor.

Quantization also changes fine-tuning economics. QLoRA, described in Dettmers et al., fine-tunes low-rank adapters on top of a 4-bit quantized base model, making it possible to adapt larger models on a single GPU. Quality of the resulting model should be evaluated against your tasks like any fine-tune. When deploying, you can serve the adapter on a quantized or full-precision base, and results can differ slightly, so evaluate the exact serving configuration.

On phones, laptops and embedded hardware, quantization is often required rather than optional, because memory and power are tight. Mobile and edge runtimes support specific integer and floating-point formats, and neural processing units may only accelerate certain operations and bit widths. Test the exact format on target devices for accuracy, latency, memory and battery or thermal behaviour under sustained use. See AI edge deployment.

An illustrative scenario, not a client case: a team wants to self-host a mid-sized open-weight model on GPUs that cannot fit it in 16-bit at the required concurrency. An FP8 version meets quality targets on their evaluation set and fits with room for KV cache. A 4-bit version uses even less memory but drops on structured extraction tasks, so they deploy FP8 for production and use the 4-bit version only for internal experimentation.

Quantization trades bits for efficiency. Choose formats your hardware accelerates, calibrate on realistic data and let evaluation on your own tasks decide how far to go.