Asking a local model well

Why a small model needs more from you

A model that fits on your desk is smaller than one in a data centre. Smaller models know less and hold less together.

You can see the gap in the makers' own numbers. On MMLU-Pro, a broad knowledge test in Qwen's comparison table, their 2B model scored 55.3 and their 0.8B model scored 29.7.

There is a second gap, and it matters more here. IFEval tests whether a model obeys instructions you can check afterwards. Its authors give examples such as write more than 400 words, or mention this keyword three times.

On that test, in the same Qwen table, the 0.8B model scored 52.1 and the 2B model 61.2. A 4B model from the previous generation scored 83.4.

Read that as a warning about your own instructions. The smaller the model, the more of them go past it.

That gap is not a fault. It is the price of fitting on your machine. So the difference between a poor answer and a good one is often the question, not the model.

Backed by Qwen3.5-0.8B model card: benchmark table, best practices, thinking mode, Instruction-Following Eval (IFEval) for Large Language Models.

Louder is not clearer

People reach for emphasis when an answer disappoints. Capital letters. The words VERY IMPORTANT. The same request written three times.

That mostly adds words. Underneath, the model is continuing a sequence of text one small piece at a time, rather than weighing how strongly you feel.

Now the other half of the truth, because it would be easy to overstate this. Wording is not powerless at all.

One study changed only small things in how prompts were laid out. Spacing, separators, capital letters. Nothing that changed the meaning.

Scores moved by about 10 points on average, and by as much as 76 points on one model. The authors' own word for the effect was unpredictable.

So changing your words does change the answer. What it does not do is change it in the direction you meant. That paper's advice was to report a range of results across formats, because no single wording can be trusted in advance.

That is the honest position. Emphasis adds words, and those words move things unpredictably. Clarity adds information, and that moves things the way you chose.

Backed by Chat templates: how chat models continue a sequence of tokens, Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design (full text).

Say what the answer should look like

Model makers use this themselves. Qwen's card tells people to add instructions that fix the shape of the output.

For maths, their example prompt asks the model to reason step by step and to put the final answer inside a marker they can find afterwards.

For multiple choice, they ask for the chosen letter on its own, in a named field.

You can do the same in ordinary words.

• Answer in three bullet points.
• Reply with only the corrected sentence, and nothing else.
• Give me a table with two columns, name and reason.
• If the text does not say, reply that it does not say.

Backed by Qwen3.5-0.8B model card: benchmark table, best practices, thinking mode.

Hand over the facts

A local model has read a great deal, and it has not read your work. Its view of your world is whatever you put in the message.

Anthropic's documentation calls that view a working memory, and separates it clearly from what the model was trained on.

So paste the paragraph. Paste the email. Paste the two rows of the table you are asking about.

The model can only work with what is in front of it. Handing it the text takes the guessing out of the job.

Backed by Glossary: tokens, the context window, and temperature.

The same job, asked two ways

Here is a page of meeting notes and one job: get the actions out of it.

The weak version is the one most people type first. Summarise this. VERY IMPORTANT, get the actions right. Then the notes.

Back comes a paragraph about the meeting. Some actions are in it, some are not, and none of it can be pasted anywhere useful.

Look at what you actually asked for. You asked for a summary. The word actions arrived as emphasis, not as an instruction, and the model had nothing to check its draft against.

The strong version asks for the thing you want. List every action from the notes below. One line each. Start each line with the person's name, then a dash, then what they agreed to do. If a line has no owner, write unassigned. If there are no actions, reply none. Then the notes.

Back comes a list you can paste into your task tracker, or the single word none.

Backed by Qwen3.5-0.8B model card: benchmark table, best practices, thinking mode.

What changed, and why it worked

Three things changed, and not one of them was emphasis.

You named the output. A list, one line each, instead of the word summarise, which can mean almost anything.

You gave the line a shape. Name, dash, task. Now the model can look at its own draft and see whether each line has those parts.

You said what to do when the answer is not there. Without that, a model with nothing to report will often produce something anyway, because producing text is what it does.

That last one is the instruction people leave out most, and it is the one that catches invented answers before you paste them into an email.

"Think step by step" is not a magic phrase

You may have been told to tell a model to think step by step. The idea is called chain of thought, and it came from a real result with a real limit.

The 2022 paper that introduced chain-of-thought prompting found it helped very large models. On the small models of the day it did not.

Its authors wrote that models of smaller scale produced fluent but illogical chains of thought, which made the answers worse rather than better.

Small models have changed since 2022. Qwen's 0.8B model, for one, has a thinking mode of its own. So try it, and check whether it helped. That is the honest state of it.

Backed by Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (full text), Qwen3.5-0.8B model card: benchmark table, best practices, thinking mode.

How to repair a prompt without guessing

When an answer comes back wrong, the temptation is to rewrite the whole thing. Then it works, and you have no idea why.

Change one thing. Run it again in a fresh chat, so the failed answer is not still sitting in front of the model.

Run it twice before you believe it. The randomness setting is rarely at zero, and Qwen's own advice for their model does not put it there. Anthropic's documentation adds that even at zero, identical inputs can give different outputs when a model is served over an API.

That rules out a lot of false conclusions. Two runs of the old prompt and two of the new one is a small price for knowing which is better.

• Change one thing per try, in this order: the facts, the named output, the checkable shape, the line about what to do when the answer is not there.
• Run the old version and the new one twice each, in a fresh chat every time.
• Write down what you changed and what came back. Two lines is enough.
• Keep the version that won both of its runs, not the one that won once.
• Still wrong after four honest tries? Now it is worth a bigger model.

Backed by Glossary: tokens, the context window, and temperature, Qwen3.5-0.8B model card: benchmark table, best practices, thinking mode.