Measuring it honestly

A number with no machine is noise

Somebody on a forum posts a speed for a model. You try the same model and get nothing like it. One of you must be wrong.

Neither of you is. The number was never a fact about the model on its own.

llama.cpp's own guide shows how much else is in it. The test machine had an A6000 graphics card with 48 GB, seven processor cores and 32 GB of memory.

A 30B model, squeezed to 4 bits, ran there at 1.7 tokens a second. Turning on the setting that hands layers to the graphics card took it to 8.7.

One change, about five times the speed, on one machine with one model. That is why a rate with no machine attached cannot help you plan.

Backed by Token generation performance troubleshooting, with the test machine named.

One run is not a measurement

Run the same test twice and the two numbers differ. That is normal, and it is the thing that fools people.

llama.cpp comes with a tool for this called llama-bench. It runs each test five times by default, averages them, and reports the spread.

The spread is how far the five runs sat from their average.

Its documentation shows one such test in full. The machine was an AMD Ryzen 7 7800X3D with an RTX 4080 graphics card, running Qwen2.5 7B squeezed to 4 bits.

That test was reading a 512-token prompt. The five runs came out between 6,863 and 7,237 tokens a second, with nothing changed between them.

That is a gap of about five percent, out of thin air, on a machine doing nothing different.

• Measure first, several times, before you touch anything.
• Change one thing.
• Measure again, the same number of times.
• Believe the change only if it is larger than the spread you already saw.

Backed by llama-bench: repetitions, warm-up runs, output columns and sample results.

One measurement, from the top

Here is a whole measurement, start to finish, with the tool llama.cpp ships for it.

The command is llama-bench, pointed at a model file. Nothing else is required, because it has defaults for the rest.

Those defaults do two tests, because there are two jobs. It reads a prompt of 512 tokens. It writes 128 tokens.

Each test runs five times, and then it prints a small table. These are the two rows from the project's own example:

That is the whole output. Seven columns, two rows.

• llama 7B mostly Q4_0 | 3.56 GiB | 6.74 B | CUDA | -1 | pp 512 | 2368.80 ± 93.24
• llama 7B mostly Q4_0 | 3.56 GiB | 6.74 B | CUDA | -1 | tg 128 | 131.42 ± 0.59

Backed by llama-bench: repetitions, warm-up runs, output columns and sample results.

What each column is telling you

Read left to right and the row stops looking like noise.

The first four columns name the model, its exact form, its size and the kind of chip that did the work. That is already more than a forum post usually gives you.

What no column here names is the machine. This printed table cannot be quoted as a speed for anybody, and the last section of this lesson is how to get one that can.

• model: which model, and which form of it. Q4_0 is one of the compact ones.
• size: 3.56 GiB, the room the file takes up.
• params: 6.74 B, how many numbers the model holds.
• backend: CUDA, meaning the work went to an NVIDIA graphics card.
• ngl: how many layers went to the graphics chip. Minus one is its default, meaning the tool chose.
• test: pp 512 is reading a 512-token prompt. tg 128 is writing 128 tokens.
• t/s: the average of the five runs. After the plus-or-minus sign is their spread.

Backed by llama-bench: repetitions, warm-up runs, output columns and sample results.

Real, or just noise?

That last column is the one that answers the question you came with.

Look at the two spreads in those rows. Reading wobbled by 93.24 out of 2368.80, which is close to four percent. Writing wobbled by 0.59 out of 131.42, which is under one percent.

Same machine, same minute, two very different amounts of noise. There is no single figure for how much a run wobbles.

So the same gain means opposite things on the two rows. Three percent on the reading row is inside the noise. Three percent on the writing row is well outside it.

The rule is short. Compare your change to the spread on the row you changed, and not to a number you remember from last week.

Measure at the depth you work at, too. llama-bench can begin a test with a conversation already in the cache. A number taken from an empty conversation is not the one you live with.

Backed by llama-bench: repetitions, warm-up runs, output columns and sample results.

The first run is not like the rest

The first answer after you start up is slower than the ones after it. The model has to be read off your disk into memory.

Ollama hands you that number. Every response it returns carries a load duration, which is the time it spent loading the model.

It then keeps the model in memory for five minutes. Ask again inside that window and the loading is already done, so you pay nothing.

Leave it longer and the model is dropped, and the next question pays again.

llama-bench deals with this by running warm-ups before it measures. There is an option to skip them, so you know they are on by default.

Both numbers are real. A cold first answer and a warm tenth answer are not the same measurement, and neither one is the truth.

Backed by Ollama API: generate a response, including the load and evaluation timings, Ollama FAQ: how long a model is kept loaded in memory, llama-bench: repetitions, warm-up runs, output columns and sample results.

Heat, length, and whatever else is open

Your computer has no single fixed speed. It runs faster while it has room to get hotter.

Intel says so about its own chips. A processor climbs above its base speed only while it stays inside safe power and temperature limits.

How far it climbs depends on the work and on the thermal headroom, which is how much hotter the chip can safely get.

So a ten-second test and a ten-minute test are not comparable. By the tenth minute the machine is warm and has eased off.

Intel even gives collecting consistent measurements as a reason people turn that boost off. A chip maker is telling you its own speed is not a constant.

Crowding the processor costs you too. On that A6000 machine, llama.cpp's guide measured 9.1 tokens a second with four threads and 8.7 with seven.

• Do the same amount of work both times. llama-bench asks for a fixed number of tokens, 128 by default.
• Leave a gap between runs. llama-bench can wait a set number of seconds between tests.
• Close the same things both times, or close nothing both times.
• Do not weigh a run from a cool room against one from a hot afternoon.

Backed by What Is Intel Turbo Boost Technology?, llama-bench: repetitions, warm-up runs, output columns and sample results, Token generation performance troubleshooting, with the test machine named.

The honest way to do it

Two free tools give you a number you can stand behind. You have read one of them already.

Ask llama-bench for its results as data and it records your processor, your graphics card, the model, the build and the date beside each result.

Ollama is the other. Add the verbose option to ollama run and it prints its timings once the answer finishes.

Those lines split the job in two, as lesson 22 described. Prompt eval count and rate are the reading. Eval count and rate are the writing. Load duration is the model arriving in memory.

One more line is worth having. Prompt eval cached says how many tokens it did not have to read again. That is lesson 23's reuse, shown as a number.

Then write down what you measured with. The machine, the model, the settings, the date. Without those you have made another forum number.

Backed by llama-bench: repetitions, warm-up runs, output columns and sample results, Ollama command line source: run's verbose flag, "Show timings for response", Ollama API source: the timing lines printed after a verbose run, Ollama API: generate a response, including the load and evaluation timings.