The four numbers that matter

Four numbers, over and over

Read any forum thread about local AI and the same four numbers come up. People pass them around as if everyone already knows them.

They are tokens a second, the wait before the first word, context, and memory.

Each one measures a different thing. Only one of them decides whether a model will run on your computer at all.

This lesson gives you all four in plain terms. Then it gives you the words that sit around them, because those are what really stop a new reader.

Tokens a second

This is the number people quote most. A token is a small piece of text, as lesson 13 explained. Tokens a second is how many of those come out.

It is written t/s. You will see it in every speed table.

On its own it is noise. It only means something with the machine attached to it.

Here is one with its machine. The llama.cpp project ran a 7 billion parameter Llama 2 model, in the most compact of the three forms tested, on an M2 Max with 38 graphics cores. It wrote 65.95 tokens a second.

Change the machine and that number changes. Change the model and it changes again.

Backed by Performance of llama.cpp on Apple Silicon M-series.

The wait before the first word

You send a question. Nothing happens for a moment. Then words start to appear.

That gap has a name. NVIDIA's guide to measuring these systems calls it time to first token. It defines the measure as the time from sending your question to getting the first token back.

This is not the same as tokens a second. A model can write quickly and still keep you waiting a long time before it starts.

The same guide says longer prompts make that wait longer, because the model works through your whole input before it writes anything.

The next lesson is about why that is.

Backed by LLM inference benchmarking: metrics, including time to first token.

Context

Context is how much text the model can hold at once. Ollama's documentation calls it the largest number of tokens the model has access to in memory.

Lesson 13 covered what that means for your work. Here it matters as a number, because you pay for it.

You pay in memory. Ollama picks its own default from how much graphics memory you have. Under 24 GiB it gives you a 4k context, which is roughly four thousand tokens. From 24 to 48 GiB it gives 32k. At 48 GiB and above it gives 256k.

That ladder is the whole point. More context is not free, and your machine decides how much of it you get.

Backed by Ollama context length, its defaults and its memory cost.

Memory, and why it beats the rest

Memory is the fourth number, and the first one to check.

The other three describe how well a model runs. Memory decides whether it runs.

Ollama shows this plainly. The ollama ps command lists each loaded model with its size, and their own example shows a 70 billion parameter model taking 42 GB.

If that does not fit in your fast memory, part of it goes somewhere slower. Lesson 14 covered what that feels like.

So here is the order worth using.

• Memory first. It decides whether the model runs at all.
• The wait before the first word next. It is the part you notice.
• Tokens a second after that. It matters most when answers are long.
• Context last, and only when your work involves long documents.

Backed by Ollama FAQ: ollama ps, model size and loaded models, LLM inference benchmarking: metrics, including time to first token.

Words for what the model is doing

Those four numbers carry you through most of a thread. The words below are the ones that stop people.

Each gets one line. None of them is a setting you have to touch.

• Prefill: reading everything you sent. NVIDIA lists it as the stage where the model works through the full input prompt.
• Decode: writing the reply, one token at a time. The same guide also calls this generation.
• Batch: a group of tokens handled together. llama.cpp's Apple speed test reads prompts in groups of 512 and writes in groups of 1.
• KV cache: the working notes the model keeps about text it has already read. Hugging Face says the cache stores those sums so they can be reused instead of done again.
• Time per output token: the average gap between one word and the next. NVIDIA also calls this inter-token latency, and threads shorten it to TPOT.

Backed by LLM inference benchmarking: the prefill and decode stages, LLM inference benchmarking: metrics, including time to first token, Performance of llama.cpp on Apple Silicon M-series, Transformers: cache strategies, and what a KV cache stores.

Words for where the model sits

A model does not sit in one place. These three words are how people say which place.

The third one trips almost everybody, because it points in every direction.

llama.cpp's own guide says it offloads layers to the graphics chip. Ollama's guide tells you to avoid offloading the model to your processor. Ollama's cloud models are offloaded to Ollama's own service.

One word, three destinations. Read which way the writer meant, rather than the way you expect.

• Layer: one slice of the model. A model is a stack of them, and each slice can sit in a different place.
• VRAM: the memory that belongs to your graphics chip. llama.cpp describes its graphics setting as the largest number of layers to store in VRAM.
• Offload: moving part of the work somewhere else.

Backed by llama.cpp server: the gpu-layers setting and VRAM, Token generation performance troubleshooting: offloading layers to the GPU, Ollama context length, its defaults and its memory cost, Ollama: cloud models, offloaded to Ollama's service.

The letters on the file name

On a download page the same model comes as several files. Their names end in things like Q4_K_M, Q8_0 and F16.

The number after the Q is roughly how many bits each of the model's own numbers gets. Fewer bits, smaller file.

llama.cpp lists them all for Llama 3.1 8B. Q4_K_M gives each number about 4.9 bits and makes a 4.58 GiB file. Q8_0 gives about 8.5 bits and 7.95 GiB. F16 keeps 16 bits and 14.96 GiB.

The letters after the number sort them further, and they matter. Q4_K_S of that same model is 4.36 GiB, while Q4_K_M is 4.58 GiB.

What it costs: llama.cpp says the shrinking may bring some accuracy loss. Lesson 8 covered how that feels.

None of the words in this lesson is a job for you to do. They are vocabulary. The next thread you open can be about your problem instead of its own language.

Backed by llama.cpp quantize tool: bits per weight, file sizes and accuracy loss, Ollama context length, its defaults and its memory cost.