LLM Quantization Explained: Model Size Trade-offs
On this page
Learning Objective
Explain what quantization does to a model's weights and why it improves inference speed as well as memory, reason about the systems-level consequences of model size selection, and decide between a hosted API and self-hosted inference on the grounds that actually determine the answer.
Concept Explanation
Quantization reduces the numeric precision used to store and compute with a model's weights, typically from 16-bit floating point to 8-bit or 4-bit integers. The memory saving is the obvious benefit and it is proportional: a 70-billion-parameter model needs roughly 140 gigabytes at 16-bit, about 70 at 8-bit, and about 35 at 4-bit, which is the difference between several high-end accelerators and one. The less obvious benefit is speed, and it follows directly from the decode bottleneck established earlier in this phase: since generation is bound by the bandwidth of streaming weights out of memory, halving the bytes per weight roughly halves the data that must move per token, so throughput improves even when the arithmetic is still performed at higher precision. This is why quantization helps latency and not only capacity.
The quality cost is real but smaller than intuition suggests, because quantization methods are not naive rounding. Post-training quantization methods such as GPTQ and AWQ analyze the model against a calibration dataset to decide how to allocate the reduced precision, protecting the weights that matter most and accepting more error where it matters least. Mixed-precision approaches keep sensitive components, often including outlier channels in activations, at higher precision while quantizing the bulk. Empirically, 8-bit quantization is close to lossless for most purposes, 4-bit is usable with a modest and task-dependent degradation, and below 4-bit the degradation becomes pronounced. The important caveat is that averaged benchmark scores hide where the loss lands: quantization tends to degrade the hardest capabilities first, so long-chain reasoning, precise instruction following, and rare-language performance suffer before conversational fluency does, which means a quantized model can look fine in casual testing and fail on your hardest ten percent of requests.
Model size selection is a systems decision before it is a quality one. A larger model is better at hard reasoning, long instruction chains, and nuanced judgment. A smaller model is cheaper, faster in both time to first token and tokens per second, fits on less hardware, and permits far higher batch sizes at the same memory budget, which compounds into much better throughput per dollar. The mistake is choosing one model for the whole system. Most production traffic is a mixture in which classification, extraction, routing, formatting, and short grounded answers dominate by volume while genuinely hard requests are a minority, which is precisely the shape that rewards routing. Distillation is worth knowing as the adjacent technique: training a small model on the outputs of a large one to capture much of its behavior on a specific task, which is how a small specialized model can beat a large general one on the task it was distilled for.
The hosted-versus-self-hosted decision is usually argued on cost and usually decided by something else. Cost crossover does exist and it favors self-hosting only at high, sustained utilization, because a reserved accelerator is billed whether or not it is busy while an API bills per token, so bursty or low-volume workloads are almost always cheaper on an API. The considerations that more often decide it are data residency and regulatory constraints that prohibit sending content to a third party, the need for a stable model version that will not be deprecated or silently updated underneath you, latency requirements that demand co-location with your own infrastructure, and the ability to serve a fine-tuned or specialized model that no provider offers. Against those sit the real costs of self-hosting: accelerator supply, serving-stack expertise, capacity planning for a workload whose memory demand varies with sequence length, and an on-call rotation for a component that was previously someone else's problem.
The pragmatic architecture for most teams is a hybrid that the provider abstraction from the fifth lesson of this course makes cheap to build. Route the bulk of traffic to a small, quantized, self-hosted or low-cost model; escalate the hard minority to a frontier hosted model; keep a second provider configured as a fallback for outages; and ensure the eval suite runs against every route so that a routing change is a measured decision rather than a hopeful one. This gives most of the cost advantage of self-hosting, most of the capability of a frontier model, and resilience against any single provider, without committing the whole system to one bet.
| Precision | Memory for a 70B Model | Relative Decode Speed | Typical Quality Impact | Use For |
|---|---|---|---|---|
| FP16 / BF16 | ~140 GB | Baseline | None; the reference | Quality-critical work with hardware to spare |
| INT8 | ~70 GB | Noticeably faster | Close to lossless for most tasks | The usual default for self-hosted serving |
| INT4 (GPTQ, AWQ) | ~35 GB | Substantially faster | Modest, task-dependent; hits hard reasoning first | High-throughput or constrained hardware, after evaluation |
| Below 4-bit | Smaller still | Faster still | Pronounced degradation | Experimental and edge cases only |
Worked Example: Evaluating a Quantized Model on the Requests That Matter
The essential discipline is to evaluate on a stratified set rather than an average, because quantization degrades the hardest requests first and an aggregate score will hide exactly the regression you care about.
# WRONG: one aggregate number hides where the loss actually landed.
# fp16 model: 0.891 on the eval set
# int4 model: 0.874 on the eval set
# -> "1.7% drop, ship it" <- this conclusion is not supported
# RIGHT: stratify by difficulty, because degradation is not uniform.
STRATA = ['simple_extraction', 'grounded_qa', 'multi_step_reasoning',
'strict_format', 'non_english']
for stratum in STRATA:
subset = eval_set.filter(category=stratum)
report(stratum, fp16=score(fp16_model, subset), int4=score(int4_model, subset))
# Typical shape of the real result:
# simple_extraction fp16 0.96 int4 0.96 no change
# grounded_qa fp16 0.93 int4 0.92 negligible
# strict_format fp16 0.95 int4 0.89 NOTICEABLE
# multi_step_reasoning fp16 0.78 int4 0.61 SEVERE
# non_english fp16 0.84 int4 0.71 SEVERE
#
# The aggregate moved 1.7%. Two strata moved 13-17 points. If reasoning and
# non-English traffic are 10% of volume, the average is arithmetically
# correct and operationally useless.
# The architecture this result implies is routing, not a single choice:
def select_model(request):
if request.language != 'en' or request.needs_reasoning:
return 'frontier-hosted' # the 10% that needs it
return 'llama-70b-int4-selfhosted' # the 90% that does not
# And the cost crossover that decides self-hosting at all:
# reserved accelerator: billed 24/7 whether busy or idle
# hosted API: billed per token, zero at idle
# -> self-hosting wins only at high SUSTAINED utilization. A bursty
# workload on reserved hardware pays for the peak all day.Real-World Examples
- The llama.cpp project and its GGUF quantized model format made it routine to run multi-billion-parameter models on consumer laptops and single consumer GPUs, with 4-bit quantized builds of models that would otherwise require datacenter accelerators. It is the clearest demonstration that quantization changed which hardware class an entire category of models could run on, not merely how efficiently they ran.
- Meta released Llama 3.1 405B in 2024 alongside an FP8 quantized variant explicitly intended to make serving feasible on a single server node rather than a multi-node deployment, and DeepSeek-V3 was trained using FP8 precision. Both illustrate that reduced precision moved from a post-hoc compression trick applied by users to a first-class part of how frontier-scale models are released and even trained.
Review Questions & Answers
Because decoding is bound by memory bandwidth. Producing each output token requires the model's weights to be streamed out of high-bandwidth memory, and that transfer dominates the time per step. Halving the bytes used to store each weight halves the volume of data that must move for the same computation, so tokens per second improves roughly in proportion even if the arithmetic itself is still performed at higher precision after dequantization. The memory saving and the speed improvement are therefore two consequences of the same change rather than independent benefits.
Because the quality loss is not distributed evenly across request types. Quantization tends to degrade the most demanding capabilities first, so multi-step reasoning, precise adherence to an output format, and performance on languages that were less represented in training fall noticeably while conversational fluency and simple extraction barely move. An averaged score dominated by easy cases therefore shows a small decline that conceals a severe one in the minority of requests that are hardest and often most valuable. Only a stratified evaluation, scored separately per category, exposes that.
Cost only favors self-hosting at high sustained utilization, because reserved accelerators are billed continuously while an API bills per token and costs nothing at idle, so bursty or modest workloads are almost always cheaper hosted. The considerations that more often decide it are regulatory or contractual constraints preventing content from leaving your infrastructure, the need for a model version that will not be deprecated or silently updated, latency requirements that demand co-location, and the need to serve a fine-tuned or specialized model no provider offers. Those are weighed against accelerator availability, serving-stack expertise, capacity planning for memory that varies with sequence length, and taking on-call responsibility for a component that used to be a vendor's problem.
Because production traffic is a mixture in which easy requests dominate by volume and hard requests dominate by value, and no single model is the right answer for both. Routing the majority to a small quantized model captures most of the available cost and latency benefit, while escalating the difficult minority to a frontier model preserves capability where it actually matters. Keeping a second provider configured additionally removes single-vendor outage risk. The requirement that makes this safe rather than hopeful is running the eval suite against every route, so that a change in routing policy is a measured tradeoff rather than an assumption.