Context: the model's short memory

A window, not a memory

A model does not remember your last conversation. Each time you ask, the whole text is handed to it again.

How much text it can be handed at once is called the context window.

Anthropic's documentation describes that window as a working memory, and is careful to keep it separate from what the model was trained on.

Everything shares it. Your question, the document you pasted, the earlier turns of the chat, and the answer being written now.

Backed by Glossary: tokens and the context window, llama.cpp server: n_keep, context size, the truncated flag and the context-shift default.

What a token is, roughly

Window sizes are quoted in tokens, not words. A token is the small piece of text a model actually reads.

A token can be a whole word, part of a word, a single character, or even a single byte.

Anthropic say that for Claude a token stands for about 3.5 English characters, and that the exact figure varies with the language.

At that rate a page of text is several hundred tokens. Every model splits text its own way, so treat any count you work out as an estimate.

Backed by Glossary: tokens and the context window.

What happens when it fills up

The window has an end, and a long conversation runs into it.

llama.cpp's server has a setting for how much of your text to keep when the context size is exceeded and tokens have to be discarded.

Discarded is the word to hold on to. The oldest part of the conversation is dropped so the newest part can fit.

That is why a long chat starts to contradict itself. The model is not being careless. The part it needs is no longer in front of it.

Different tools handle the moment differently, and the next section is what each one looks like from your chair.

Backed by llama.cpp server: n_keep, context size, the truncated flag and the context-shift default.

What it looks like on screen

There are three shapes, and only one of them tells you what happened.

The first is a refusal you can read. llama.cpp's server sends an error instead of an answer. The wording is close to this, with your own numbers in the brackets: input (18422 tokens) is larger than the max context size (4096 tokens). skipping.

That is the kind one. It named the problem and the two numbers you need.

The second is an answer that simply stops. Mid-sentence, no error, nothing on screen to explain it. The server marked that reply truncated, but you never see the flag.

This happens because llama.cpp's server ships with context shift turned off. Rather than throw away your earlier text, it stops writing.

The third is the one that fools people. Ollama drops whole messages from the front of the chat until the rest fits, keeping your latest message and any system message.

Nothing appears on screen at all. The model answers your new question perfectly well, and has no idea the first half of the conversation ever existed.

Backed by llama.cpp server: n_keep, context size, the truncated flag and the context-shift default, llama.cpp server source: the message sent when input exceeds the context size, Ollama server source: chatPrompt drops earlier messages to fit the window.

Find out what window you actually have

You can stop guessing in about ten seconds.

Run ollama ps. It prints a line for each loaded model, and one of the columns is CONTEXT. Ollama's own example line reads 9.6 GB, then 100% GPU, then 131072. That third number is the window in tokens.

Compare it with the model card. If the card says 262,144 and the column says 4096, the tool is the limit, not the model.

Ollama's default is 4,096 tokens. It also picks by graphics memory, and a machine with under 24 GiB of it gets a 4k window.

You can raise it. There is a slider in the Ollama app, a /set parameter num_ctx command inside a chat, and an OLLAMA_CONTEXT_LENGTH setting when starting the server. In llama.cpp the flag is -c.

Every one of those costs memory. Ollama says so in the same paragraph as the instructions, which is the right place for it.

Backed by Ollama context length, its defaults and its memory cost, Ollama FAQ: the default context window and how to change it, llama.cpp server: n_keep, context size, the truncated flag and the context-shift default, Qwen3.5-0.8B model card, model overview section.

A big number is not a promise

Model cards quote large windows. Qwen3.5-0.8B lists 262,144 tokens.

Being able to hold that much text is not the same as using it well, and the gap has been measured.

One study found performance was highest when the useful information sat at the start or the end of the input. It dropped when the same information sat in the middle.

A benchmark called RULER tested seventeen models that all claimed 32,000 tokens or more. Only half of them held up at 32,000.

The authors also looked closely at one model built for 200,000 tokens, and found plenty of room for improvement as the input grew.

This is an active area rather than a settled one. The numbers keep improving, and the gap between the advertised window and the usable one has not closed.

Backed by Qwen3.5-0.8B model card, model overview section, Lost in the Middle: How Language Models Use Long Contexts, RULER: What's the Real Context Size of Your Long-Context Language Models?.

What to do about it

None of the above needs a bigger machine. It needs you to send less, and to send it in a better order.

The first two habits below are the ones that pay immediately. The rest are worth building over a few weeks.

• Check the window your tool is actually using, not the one on the card.
• Cut the document down. Paste the two pages that matter, not the whole report.
• Put your question after the text, so the thing you care about is not stranded in the middle.
• Ask one thing per message, and start a fresh chat between jobs.
• For long documents, work in parts. Summarise each part, then work from the summaries.
• Paste anything you must not lose again in your latest message, since that is the message every tool keeps.

Backed by Ollama context length, its defaults and its memory cost, Lost in the Middle: How Language Models Use Long Contexts, Ollama server source: chatPrompt drops earlier messages to fit the window.