The settings that move the needle

Most people should change nothing

This is the honest opening. The settings below are real, and for most readers the right number of them to touch is none.

The tools have already done the work. llama.cpp's setting for how much of the model goes on your graphics chip defaults to auto. Its thread count is worked out from your system.

It goes further. llama.cpp has a fit setting, on by default, that adjusts the settings you did not set so the model fits your device memory.

The largest win open to you is not on this page at all. It is choosing a model that fits your machine, which lesson 11 covered.

Read the rest of this as a map of what the words mean. Change something only when you have a reason and a way to check the result.

Backed by llama.cpp server: gpu-layers, threads, ctx-size, cache types, fit and prompt caching, llama-bench: thread defaults and the different-numbers-of-threads example.

How much sits on the graphics chip

A model is built in layers. Each one can sit on your graphics chip or in ordinary system memory.

llama.cpp calls this setting gpu-layers. Its documentation describes it as the largest number of layers to store in graphics memory, and the default is auto.

Ollama does not hand you a number for this. It shows you where the model landed instead, which the next section is about.

What it costs: graphics memory. Every layer you move there takes room your context also needs.

Backed by llama.cpp server: gpu-layers, threads, ctx-size, cache types, fit and prompt caching, Ollama context length, its memory cost and checking the split.

What you see when you change one

You do not have to guess whether the layers moved. Both tools tell you, in different ways.

llama.cpp tells you as it starts up. Its troubleshooting guide gives the lines to look for. One reads offloading 60 layers to GPU. The next reads total VRAM used: 17223 MB.

Ollama tells you afterwards. Run ollama ps and you get one row per loaded model. Their example row reads 9.6 GB, then 100% GPU, then a context of 131072, then two minutes before it unloads.

That Processor column is the one to read. Ollama says 100% GPU means all of the model went to the graphics chip, 100% CPU means none of it did, and 48%/52% means it was split.

Then the number that moves. llama.cpp's guide names its test machine: an A6000 card with 48 GB, seven processor cores and 32 GB of memory. A 30 billion parameter model at 4 bits ran there at 1.7 tokens a second, and 8.7 once its layers went to the card.

One machine, about five times faster. On yours the change could be bigger, smaller, or nothing at all, because your model may already be sitting there.

Backed by Token generation performance troubleshooting: the startup lines, and one machine's numbers, Ollama FAQ: ollama ps, concurrent requests, Flash Attention and K/V cache types, Ollama context length, its memory cost and checking the split.

How many threads

A thread is one stream of work your processor can carry. More threads means more of the job done at once.

llama.cpp has two of these settings, not one. There is a thread count for generation, and a separate one for reading the prompt, which starts out the same as the first.

Two settings for two jobs. That is lesson 22 showing up in a command line.

The project's own guide shows an example run at 1, 2, 4, 8, 16 and 32 threads. Reading the prompt was fastest at 32. Writing was no faster at 32 than it had been at 8.

What it costs: threads belong to your whole computer, not to the model. Hand them all over and everything else you are doing slows down.

Backed by llama.cpp server: gpu-layers, threads, ctx-size, cache types, fit and prompt caching, llama-bench: thread defaults and the different-numbers-of-threads example.

How much context

This is the setting most likely to break something, in either direction.

In Ollama you change it with a slider in the app, or by setting a context length when you start the server. Their documentation says a larger context needs more memory to run a model.

In llama.cpp it is called ctx-size, and it starts at 0, which means use the size the model was built with.

What it costs: memory, out of the same pool the model is sitting in. Ask for a large context on a small machine and you push the model off the graphics chip, which is the slow path from lesson 14.

It multiplies, too. Ollama warns that handling several requests at once raises the context by that number. Their example is a 2K context with four at a time becoming an 8K one.

Ollama's guidance is to use the largest context a model allows, and to keep the model off the processor. On a small machine those two pull against each other, and that tension is the setting.

Backed by Ollama context length, its memory cost and checking the split, Ollama FAQ: ollama ps, concurrent requests, Flash Attention and K/V cache types, llama.cpp server: gpu-layers, threads, ctx-size, cache types, fit and prompt caching.

How big the conversation's notes are

Lesson 21 named the KV cache: the notes the model keeps about text it has already read. They sit in memory next to the model, and they can be made smaller.

Ollama calls the setting OLLAMA_KV_CACHE_TYPE, and it starts at f16. llama.cpp calls the same idea cache-type-k and cache-type-v, also f16.

Ollama prices both alternatives for you. Setting q8_0 uses about half the memory of f16, with what they call a very small loss in precision that usually has no noticeable effect.

Setting q4_0 uses about a quarter of the memory, with a small to medium loss they say may be more noticeable at higher context sizes.

What it costs: quality, by an amount nobody can tell you in advance. Ollama says the effect depends on the model and the task. In Ollama the setting is also global, so every model you run gets it.

Backed by Ollama FAQ: ollama ps, concurrent requests, Flash Attention and K/V cache types, llama.cpp server: gpu-layers, threads, ctx-size, cache types, fit and prompt caching.

Reusing a conversation, and where the knobs live

The last one is not a dial you turn up. It is work you avoid, and you lose it by opening a fresh chat for every question.

llama.cpp's server keeps the reading it already did, and reads only the part of your text that changed. Its documentation says this is on by default.

It has a cost, and llama.cpp names it. Reusing that work can make answers less repeatable. The same question may not come back word for word twice.

Pip publishes a measurement of the same idea. On a 14-core Intel i9 with 32 GB of memory and no separate graphics card, reusing a conversation let the next answer begin in 1.6 seconds instead of 33.2.

Pip's page is careful with that. It says the app must label every conversation consistently before every user gets the benefit, and that Pip was not tested beside llama.cpp or Ollama.

• llama.cpp: all of it, as flags on a command line. Nothing is hidden and nothing is arranged.
• Ollama: a context slider, and ollama ps to see where the model landed. Defaults come from your machine.
• Pip: fewer choices again. It aims to recommend a tested model so you do not compare settings at all.

Backed by llama.cpp server: gpu-layers, threads, ctx-size, cache types, fit and prompt caching, How Pip is tested: results, comfort, method and limits, Pip.cpp: use AI freely, keep control.

What to expect on your own machine

Every number on this page came off somebody else's computer. Not one of them is a forecast for yours.

The thread example is the clearest warning. On that same A6000 machine, seven threads gave 8.7 tokens a second and four threads gave 9.1. Fewer was faster.

The big jump, 1.7 to 8.7, is a warning too. It came from a machine not using its graphics card at all. If yours already is, that win is not waiting for you.

So the honest answer to how much any of this helps you is that nobody knows in advance. You have to measure it, and the next lesson is how.

Many readers should stop here instead. If ollama ps already says 100% GPU, and words arrive at a pace you can comfortably read, this page has nothing for you to do.

Backed by Token generation performance troubleshooting: the startup lines, and one machine's numbers, Ollama context length, its memory cost and checking the split.