Quantization, in plain words

A model is a pile of numbers

A model is a file full of numbers. Those numbers are called weights, and a model has billions of them.

Each weight is written down using a set number of bits. A bit is a single one or zero, the smallest piece of computer storage.

llama.cpp's documentation says models are usually shared in a 16-bit format. Llama 3.1 8B stored that way is a file of nearly 15 GiB.

That is more memory than many laptops can spare. This is the problem quantization was made to solve.

Backed by llama.cpp quantize tool: quantization methods, sizes and bits per weight.

What quantization actually does

It is not a zip file. A zip can be unpacked to give back every original byte. This cannot.

Instead each weight is rewritten with fewer bits. Every weight is rounded to the nearest value the new format can hold.

llama.cpp is the engine under most local AI tools. Its documentation says quantization reduces the precision of the weights, which shrinks the file and can speed the model up.

The rounding is where the loss lives. Once a number has been rounded, the old value is gone for good.

Backed by llama.cpp quantize tool: quantization methods, sizes and bits per weight.

Why the file shrinks, and by how much

Going from 16 bits to 4 bits should make the file a quarter of the size. That is the right first idea, and it is not quite what happens.

Two things get in the way.

• Weights are stored in small groups. Each group carries an extra number saying how to scale it back up, and that bookkeeping costs bits.
• Not every part is treated the same. The common 4-bit setting keeps some parts of the model at 6 bits, and keeping them sharper measurably improves the result.

Backed by k-quants (pull request 1684), with perplexity, file size and timing tables, llama.cpp quantize tool: quantization methods, sizes and bits per weight.

The measured sizes

So llama.cpp's common 4-bit setting, called Q4_K_M, really uses about 4.9 bits per weight rather than 4.

Measured on Llama 3.1 8B, the 16-bit file is 14.96 GiB and the Q4_K_M file is 4.58 GiB.

That is roughly a third, not a quarter. It is still the difference between a model you cannot run and one you can.

Bigger models do not escape their size, though. llama.cpp has a second table that starts from the 32-bit file rather than the 16-bit one. There, Llama 3.1 70B is 280.9 GB before quantizing and 43.1 GB at Q4_K_M. That is an enormous saving, and it is still far more memory than a laptop has.

Backed by llama.cpp quantize tool: quantization methods, sizes and bits per weight.

Reading the ending on the file name

A download page usually lists several files with the same model name and different endings. That ending is the setting the file was made with.

Take it apart from the left. Q means quantized. The number after it is the target bits per weight.

K means a k-quant. That is the family that stores weights in groups and mixes formats inside one file.

S, M and L pick how much of the model stays sharper. Q4_K_S uses the 4-bit format for every part. Q4_K_M keeps some parts at 6 bits, which is why it is the larger file.

A plain 0, as in Q4_0 or Q8_0, is an older and simpler format from before this family. F16 or BF16 means the file has not been quantized at all.

Endings that start with IQ are a newer family. Judge one the same way as the rest: by its bits per weight and its size on disk.

Here is llama.cpp's own table for Llama 3.1 8B, so you can see the shape of the ladder.

• Q2_K: 3.16 bits per weight, 2.95 GiB
• Q4_K_S: 4.67 bits per weight, 4.36 GiB
• Q4_K_M: 4.89 bits per weight, 4.58 GiB
• Q5_K_M: 5.70 bits per weight, 5.33 GiB
• Q6_K: 6.56 bits per weight, 6.14 GiB
• Q8_0: 8.50 bits per weight, 7.95 GiB
• F16: 16.00 bits per weight, 14.96 GiB

Backed by llama.cpp quantize tool: quantization methods, sizes and bits per weight, k-quants (pull request 1684), with perplexity, file size and timing tables.

What it costs, and how far to push

llama.cpp's own documentation says the process may introduce some accuracy loss. This lesson will not pretend otherwise.

The loss is usually measured with perplexity, a score for how well a model predicts real text. With perplexity a lower number is the better one. The unrounded model sets the best score, and every rounded version comes out higher.

The author of these methods measured them on LLaMA 7B. The 16-bit file scored 5.9066 and the 4-bit file scored 5.9601, which is about one percent worse for a file under a third the size.

Push further and the cost climbs fast. The 2-bit file scored 6.7764, about fifteen percent worse, to save one more gigabyte.

Speed moves the other way. On an Apple M2 Max using four threads, that 7B model took 116 milliseconds per token at 16 bits and 55 at 4 bits.

• 6-bit: measured within 0.1 percent of the original on that test. Use it when the file fits.
• 4-bit: the usual choice. About one percent worse there, at roughly a third of the size.
• 2-bit and below: a real drop. Worth it only when nothing larger will fit at all.

Backed by k-quants (pull request 1684), with perplexity, file size and timing tables, llama.cpp quantize tool: quantization methods, sizes and bits per weight, Perplexity of fixed-length models.

Choosing between them

Three rules cover almost every real choice.

The first is that it has to fit. A file your memory cannot hold is not a slower option. It is no option at all.

Why is fitting first? Because running out of memory is a cliff, not a slope. The author of these methods found his 2-bit 30B model fitted a 16 GB graphics card when the others did not, and the speed difference was large.

The second rule is a surprise. At the same file size, a larger model at a rough setting can beat a smaller one at a fine setting. On that 2023 test, 13B at 2 bits scored 5.8545 while 7B at 6 bits scored 5.9110. The two files were within about 20 MB of each other.

The third is that smaller is not always faster. On the M2 Max at four threads, Q4_K_S took 50 milliseconds per token. The smaller Q2_K took 56, and Q3_K_S took 81.

So the move is this. Take the largest model that fits, at the highest setting that still fits, then measure it on your own machine.

Backed by k-quants (pull request 1684), with perplexity, file size and timing tables, llama.cpp quantize tool: quantization methods, sizes and bits per weight.