LLM quantisation explained: what 4-bit and 8-bit models trade away

You've found an open-weight model that does the job, but the full-precision version won't fit on your hardware. The download page offers a dozen smaller variants labelled 8-bit, 4-bit, Q4_K_M, AWQ, GPTQ. Each one is smaller and some are faster. The question nobody on the download page answers is what you lose.

The short answer from the published research: at 8 bits, very little. At 4 bits, a little, and it shows up in places that headline benchmark scores tend to hide. Below 4 bits, quite a lot. The useful part is knowing where the damage hides and how to check for it on your own work.

If you want the memory side of this (how many gigabytes each format needs), we covered that in how much VRAM you need to run an LLM locally. This article is about quality.

What quantisation actually does to a model#

A model's weights are normally stored as 16-bit floating-point numbers. Quantisation stores them with fewer bits, usually 8 or 4, so each weight can only take one of a small set of values. A 4-bit weight has 16 possible values. To make that workable, weights are grouped into small blocks, and each block gets its own scale so the 16 values cover the range that block actually uses. The finer the groups, the less precision you lose, at the cost of storing more scales.

Two details decide how much this hurts.

The first is that not all weights matter equally. The LLM.int8() paper from Tim Dettmers and colleagues found systematic outlier features in transformer models that dominate attention and prediction quality. Their fix was to pull those outlier dimensions out and keep them in 16-bit, while more than 99.9% of the values were still multiplied in 8-bit. With that split, they ran models up to 175B parameters without measurable loss. Round those outliers naively along with everything else and quality drops.

The second is how the rounding is chosen. Simple methods round each weight to the nearest available value. Smarter ones use a small calibration dataset to decide how to round so the layer's output changes as little as possible. GPTQ does this in one pass using approximate second-order information, and its authors report 3 to 4 bits per weight with negligible accuracy loss, quantising a 175B model in about four GPU hours. AWQ starts from the observation that not all weights in an LLM are equally important, and picks out the important ones using activation statistics rather than the weights themselves. Because it doesn't rely on backpropagation or reconstruction, its authors say it avoids overfitting the calibration data.

That's why two "4-bit" files of the same model can behave differently. The bit width is only part of the story. The method, the group size and the calibration data matter too.

What the big studies found at 8 and 4 bits#

One of the largest public evaluations is the Red Hat AI and ISTA study by Eldar Kurtic and colleagues. They ran over 500,000 evaluations across the Llama 3.1 family, comparing FP8, INT8 and 4-bit weight-only (W4A16) formats against the 16-bit originals. Their 4-bit models used GPTQ with 128-element groups.

The headline numbers are reassuring. On the older Open LLM Leaderboard tasks, 8-bit models recovered 99.75% of the baseline score on average and 4-bit models 99.36%. On coding benchmarks, 8-bit recovered 99.9% and 4-bit 98.9%. On the RULER long-context benchmark, every format recovered at least 98% on average. FP8 was effectively lossless across all model sizes, and well-tuned INT8 lost 1 to 3%.

The caveats sit in the details. INT8 with quantised activations worked well for the 8B and 405B models but caused noticeable drops at 70B, so one recipe doesn't suit every size. On reasoning tasks with distilled DeepSeek-R1 models, recovery was above 99% except for the smallest models at 4-bit, which dropped more. On the newer Open LLM Leaderboard tasks, the smaller models also showed more variance.

Hugging Face's guide to choosing a method summarises the field in much the same way: 8-bit is very close to the bf16 baseline for roughly half the memory, 4-bit keeps relatively high accuracy at about a quarter, and below 4 bits there's a noticeable drop, especially at 2-bit. The empirical study of LLaMA3 quantisation by Wei Huang and colleagues, which tested ten methods from 1 to 8 bits, reached the same conclusion about the low end. The degradation was substantial and most pronounced at ultra-low bit widths, on both language and vision tasks.

Why "99% of the score" can still mean different answers#

Here's the part that matters most if you're automating something. A benchmark average can stay almost flat while individual answers change a lot.

Microsoft Research's paper Accuracy is Not All You Need measured this directly. They counted "flips": questions where the full-precision model was right and the compressed one wrong, or the other way round. Across the quantisation schemes they tested, accuracy stayed within 0 to 2% of the 16-bit baseline. Flips went as high as 13.6%. On the GSM8K maths benchmark, two of the bitsandbytes schemes flipped 12 to 30% of answers. The gains and losses roughly cancelled out in the average, which is why the score looked fine.

There was one clear exception. GPTQ at 8-bit weights kept flips negligible.

Flips also predicted quality on open-ended tasks. On MT-Bench, a free-form conversation benchmark, models with many flips scored significantly worse than the baseline, and the authors' manual review found the 4-bit model often ignored instructions and rambled more. That's the failure you'll notice in practice: a summariser that stops following your format, or an extraction step that starts adding commentary.

For a chatbot answering general questions, a few percent of different answers may not matter. For an automation that classifies invoices, routes tickets or extracts fields, it means the quantised model will handle some of your cases differently from the one you tested. The overall error rate can look the same while the specific errors move.

Choosing a format#

Some practical rules follow from the evidence.

If it fits, use 8-bit. The studies above find it close to lossless, and the Red Hat paper found FP8 effectively lossless at every size it tested. This is the safe default when you have the memory.

If you need 4-bit, prefer a calibrated method such as GPTQ or AWQ, or a well-regarded pre-made quant, over the simplest on-the-fly option. Hugging Face notes that AWQ and GPTQ often lead in accuracy but need a calibration step, while on-the-fly options like bitsandbytes need no calibration data and are easier to use. Many published GPTQ and AWQ models have already done the calibration for you.

Be more cautious with small models. Both the Red Hat results and the variance they report point the same way: the smallest models lose more at 4-bit. Whether a bigger model at 4-bit beats a smaller one at 8-bit for your job is something to test rather than assume.

Treat anything below 4 bits as experimental for production work. It can be fine for a personal assistant, but the evidence says quality drops noticeably.

Speed is a separate question. The Red Hat authors found 4-bit weight-only was the most cost-efficient for synchronous setups (roughly, one request at a time), while 8-bit with quantised activations did better under continuous batching with many concurrent requests. Your serving pattern can change which format wins.

How to test a quantised model on your own work#

Hugging Face's guide puts the main advice plainly: always benchmark the quantised model on your specific task and hardware. Here's a cheap way to do that.

Collect 50 to 200 real inputs from the job you're automating, with the answers you'd accept. If you've built an evaluation set already (we wrote about how to evaluate an AI automation), reuse it. Run both the full-precision model and the quantised one with the same prompts and temperature set to zero, or as low as your tool allows.

Then compare more than the score. Count how many individual answers changed between the two models, not just how many were right. That's the flip rate from the Microsoft paper, and it's easy to compute in a spreadsheet. Read the changed answers. Check format compliance separately: did the JSON still parse, did it still stick to the allowed labels, did summaries stay within the length you asked for? These are the instruction-following slips that the MT-Bench result suggests get worse at 4-bit.

If you can't run the full-precision model at all, rent a cloud GPU for an hour to produce the reference answers. That costs far less than discovering the drift in production.

The Microsoft authors also suggest KL-divergence between the two models' output probabilities as a measure, and found it tracks flips closely (a Spearman correlation of 0.981 on MMLU). A flip count on your own examples is easier to explain to whoever signs off on the automation, though.

At TheAILAB we treat the quantisation level as part of the model choice, so a switch from 8-bit to 4-bit gets the same re-test as a switch to a different model. If the flip rate on your own examples is low and the changed answers are ones you'd accept, take the memory savings. If it isn't, step up a format or a model size before you go looking for prompt fixes.

Sources

1. "Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization (v3) — Eldar Kurtic, Alexandre Marques, Shubhra Pandit, Mark Kurtz and Dan Alistarh, Red Hat AI and ISTA (arXiv 2411.02355) (checked 2026-10-15)
2. Accuracy is Not All You Need — Abhinav Dutta, Sanjeev Krishnan, Nipun Kwatra and Ramachandran Ramjee, Microsoft Research (July 2024) (checked 2026-10-15)
3. Selecting a quantization method — Hugging Face Transformers documentation (checked 2026-10-15)
4. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — Elias Frantar, Saleh Ashkboos, Torsten Hoefler and Dan Alistarh (arXiv, March 2023 version) (checked 2026-10-15)
5. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration — Ji Lin, Jiaming Tang, Haotian Tang et al., MIT Han Lab (arXiv, July 2024 version) (checked 2026-10-15)
6. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — Tim Dettmers, Mike Lewis, Younes Belkada and Luke Zettlemoyer (arXiv, November 2022 version) (checked 2026-10-15)
7. An empirical study of LLaMA3 quantization: from LLMs to MLLMs — Wei Huang, Xingyu Zheng, Xudong Ma et al. (arXiv, April 2024) (checked 2026-10-15)