When it is slow or wrong

Slow and wrong are different faults

Your local model has let you down. Before you change anything, name the fault.

Slow means the words crawl out. Or nothing happens for a long time. The answer may be fine when it finally arrives.

Wrong means the answer came at a normal speed and was poor. It missed the point, made something up, or ignored half of what you asked.

These two have different causes. Fixing one will not fix the other. So decide which you have, then work down the list below in order.

The order below is by how cheap each check is. The first takes seconds, the last takes a download.

The order to check in, and what each step costs

Here is the whole list on one screen. Each step gets its own section below.

Do them in this order even when you are sure you know the answer. Stop as soon as the fault goes away, and note which step fixed it. Next time you will start there.

• Name the fault. Slow, or wrong. Ten seconds, and it halves the list.
• Run ollama ps. One command. It shows every model in memory, its size, where it is running, and its window.
• Unload anything you have finished with. One more command, and it can free gigabytes before you touch the four causes.
• Read the Processor column for a split, and the window number beside it. Free, and it explains most slow sessions.
• Rewrite the ask once. Two minutes, and it repairs more poor answers than anything under it.
• Change the model, or how hard it was squeezed. A download, and the only step with a real cost.

Before the four: is something else holding the memory?

This is the cheapest check there is, and it is missing from most advice.

Ollama keeps a model in memory for five minutes after your last message, so it can answer the next one quickly. That is a sensible default and it is easy to forget.

It will also hold more than one at a time. The limit is three models on a machine without a graphics card, and three per card on a machine with them.

So the model you tried an hour ago may still be sitting there. Run ollama ps and you will see every one of them, with its size.

The fix is one command. Run ollama stop with the name of the model you are finished with, and the memory comes back immediately.

Two seven-gigabyte models loaded at once on a sixteen-gigabyte machine explains a slow session entirely, and neither of them is at fault.

Backed by Ollama FAQ: ollama ps, loaded models, keep-alive and concurrency, Ollama CLI Reference: ps and stop.

One: the model does not fit

This is the most common cause of slowness, and the quickest to rule out.

A model has to sit in fast memory while it runs. If it is too big for that memory, your computer keeps part of it somewhere slower. The model still works. It crawls.

Ollama will show you this. Run the command ollama ps and read the Processor column. It says 100% GPU when the whole model is in the fast memory. It says 100% CPU when the model is in ordinary system memory. It can also show a split, such as 48%/52%, when the model is spread over both.

A split is the finding. It means part of the work is being done on the slow side, for every word, all session.

Ollama's own guidance is to avoid running on the CPU where you can. If you see a split, the fix is a smaller model, or a more heavily squeezed copy of the same one.

One more thing worth ruling out. Ollama's troubleshooting notes describe an out-of-date graphics driver causing it to fall back to the CPU. If the split appeared after a system update, that is where to look.

Backed by Ollama FAQ: ollama ps, loaded models, keep-alive and concurrency, Ollama: Context length, its memory cost and the processor split, Ollama: Troubleshooting, including falling back to the CPU.

Two: a long chat costs you memory

This one is usually told in one sentence, and the one sentence is misleading. It has four links.

One. The chat gets long, and the model reads all of it again on every turn.

Two. It stops fitting, so you raise the context setting. That is the correct move, and it is where the cost enters.

Three. A bigger window costs memory. Ollama says a larger context length increases the memory needed to run a model.

Four. That memory comes out of the same pool the model is sitting in. Once the total stops fitting, part of the work moves to the slow path, which is the split from cause one.

So a long chat does not slow the model down by itself. It leads you to a setting, and the setting is what pushes the work onto the slow side.

Backed by Ollama: Context length, its memory cost and the processor split, Ollama FAQ: ollama ps, loaded models, keep-alive and concurrency.

Two, continued: long text also hurts the answer

That was the speed half of cause two. Long input costs you accuracy as well, and it does it quietly.

Researchers tested how well models use long inputs. Answers were best when the needed fact sat near the start or the end. Facts buried in the middle were missed far more often, even by models built for long inputs.

So the same twenty pages that slowed the session down are also the reason the model skipped the thing on page eleven.

The fix is the same for both faults. Send less. Start a fresh chat. Paste the three pages that matter instead of the whole report.

One free action, two different faults. That is rare enough to do first.

Backed by Lost in the Middle: How Language Models Use Long Contexts.

Three: the ask was vague

Now for answers that arrive at a normal speed and are still poor.

A small model follows instructions less well than a large one. It fills gaps with guesses. A loose question that a big cloud model would read kindly can leave a small local model with far too much room.

Before you blame the model, rewrite the ask once. Say what you want. Say how long it should be. Say what to leave out. Give one example of a good answer if you have one.

Lesson 12 covers this in full. It is worth the second try. A rewritten question fixes poor answers more often than a bigger model does.

Run the new version in a fresh chat, so the failed answer is not still sitting in the window shaping the next one.

Four: the model is too small or too squeezed

This is last because it costs the most to fix.

The copies people download for home use are usually shrunk first. The numbers inside are rounded off so the file takes less room. llama.cpp says plainly that this may bring some loss of accuracy.

That loss has been measured. In llama.cpp's own 2023 tables, a 7B model at 2 bits predicted text noticeably worse than the same model at 16 bits. The same page reports the 6-bit copy landing within about a tenth of a percent of the original.

So try the cheaper half of this step first. Move up one level of squeeze rather than up a whole model size. For Llama 3.1 8B that is 4.58 GiB to 6.14 GiB, rather than a forty-gigabyte download.

If a less squeezed copy fixes it, you have your answer. If it does not, the job may need a bigger model than your machine can hold. Track 4 is about what to do then.

Backed by llama.cpp quantize tool: accuracy loss and quantized file sizes, k-quants (pull request 1684), with perplexity by quantization level.