Where the time actually goes

Every answer is two jobs

When you ask a model something, it does two separate jobs.

First it reads everything you sent. NVIDIA's guide to measuring these systems calls this prefill, and describes it as the model working through the full input prompt.

Then it writes. The same guide calls that decode, and describes the model giving out the answer one token at a time.

Reading happens once. Writing happens over and over, once for every piece of text that appears.

The two jobs run at very different speeds. Nearly every odd thing about local AI speed comes out of that gap.

Backed by LLM inference benchmarking: the prefill and decode stages.

Reading is fast, because it happens together

The model can look at your whole question at once. Every token in it can be worked on side by side.

llama.cpp measures the two jobs apart for exactly this reason. Its Apple speed test says prompt processing is run in batches of 512 tokens, while text generation is run one token at a time.

The gap that opens up is wide. Take one M2 Max with 38 graphics cores, running a 7 billion parameter Llama 2 model in the most compact of the three forms tested.

llama.cpp measured 671.31 tokens a second reading, and 65.95 writing.

Same machine. Same model. Same run. Reading was about ten times faster than writing.

Backed by Performance of llama.cpp on Apple Silicon M-series.

Writing is slow, because it is one at a time

To write a single token, the model has to pull its own numbers out of memory. Then it does the whole thing again for the next one.

So writing is held back by how fast your machine can read from its own memory. That speed has a name: memory bandwidth.

llama.cpp's own summary draws this out. It plots prompt processing against the number of graphics cores, and text generation against memory bandwidth. Two jobs, two limits.

The measurements show it. Two M3 Ultra machines running that same model both have 800 GB a second of memory bandwidth. One has 60 graphics cores and the other has 80.

With a third more cores, reading went from 1073.09 to 1471.24 tokens a second. Writing went from 88.40 to 92.14. Reading gained about a third. Writing gained about four percent.

Backed by Performance of llama.cpp on Apple Silicon M-series.

Your work, first pile: mostly reading

Look at your own week and most of it sorts into two piles.

The first pile is short answers about long input. You paste a contract and ask what the notice period is. You hand over a thread of forty emails and ask who agreed to what. You point a coding tool at a file and ask what it does.

In every one of those the model reads a great deal and writes very little.

What it feels like: a pause with nothing on the screen, then a short answer that appears quickly and is over almost as soon as it starts.

Nearly all of your wait happened before the first word. NVIDIA says as much directly. Longer prompts make that first wait longer, because the model works through the whole input before it writes anything.

Backed by LLM inference benchmarking: the prefill and decode stages, LLM inference benchmarking: time to first token, and the cost of a growing sequence.

Your work, second pile: mostly writing

The second pile is long answers from short input.

Draft this page. Turn these four bullets into an email. Translate this paragraph. Write the code that does this one thing.

The model reads a line or two, then writes for a long time.

What it feels like: words start almost at once, then arrive at a steady pace and keep on arriving. There is no pause at the front, and no burst at the end.

That pace is your token rate. Ask for twice as much text and you wait about twice as long. It also eases off a little as the answer grows, because NVIDIA says the cost of each new token rises with everything said so far.

So the two piles fail in opposite ways. The first keeps you staring at nothing. The second gives you something to read immediately and then makes you watch it crawl.

Backed by LLM inference benchmarking: time to first token, and the cost of a growing sequence.

Why the first answer is the slow one

You have probably noticed this. The first reply in a chat takes longer than the ones after it. Two things cause that.

The first is loading. The model file has to be brought into memory before anything can happen. Ollama keeps a model in memory for five minutes after you use it, so the next question does not pay that cost again.

The second is that most of the reading can be skipped. llama.cpp's server keeps the work it already did on the earlier part of a chat.

Its documentation puts it plainly. The common start does not have to be processed again, and only the part that differs is read. This is on by default.

So turn two of a conversation skips nearly all of the reading. That is why it feels quicker.

Backed by Ollama FAQ: keeping a model loaded in memory, llama.cpp server: prompt caching and the timings it reports.

Which machine helps which pile

Now those two M3 Ultra machines pay for themselves, because they answer a question about money.

Same memory bandwidth. A third more graphics cores. Reading got 37 percent faster and writing got about 4 percent faster.

If your week is the first pile, graphics cores are what you are buying, and they deliver. If your week is the second pile, they are close to wasted money.

The second pile is bought with memory bandwidth instead, and bandwidth is not something you can turn up. In llama.cpp's table it is a fixed property of each chip, listed beside it.

Be careful even with the 4 percent. On those same two machines, running the largest and least compressed form of that model, the 80-core one actually wrote slightly slower: 39.78 tokens a second against 42.24.

More cores is a strong claim about reading. It is almost no claim at all about writing.

Backed by Performance of llama.cpp on Apple Silicon M-series.

What this explains

Hold on to one idea. Reading is fast and happens all together. Writing is slow and happens one piece at a time.

Five familiar things fall out of it.

• A pasted document sits there doing nothing for a while. The model is reading it, and nothing appears until that is done.
• A long answer takes a long time, however short your question was. Each piece of it is its own trip to memory.
• A long conversation gets slower at both jobs, because NVIDIA says the cost of each new token grows with everything said so far.
• Most real work is a mix. A chat about a document is one big read, then many small writes.
• Starting a fresh chat throws away reading that was already done. There is a real cost to it, on top of losing the thread.

Backed by Performance of llama.cpp on Apple Silicon M-series, llama.cpp server: prompt caching and the timings it reports, LLM inference benchmarking: time to first token, and the cost of a growing sequence.