Choosing your first model

Memory is the ceiling

llama.cpp's documentation puts the rule bluntly. Models are loaded fully into memory, so you need enough disk to store the file and enough memory to load it.

The same page adds a line worth keeping. At the moment, it says, memory and disk requirements are the same. So the number on the download page answers both questions at once.

The first question is therefore not which model is best. It is how much memory your computer has going spare.

LM Studio's own requirements give a plain guide. It recommends 16 GB of memory or more. On an 8 GB Mac you can still run models, but they say to stick to small ones and modest amounts of text.

Leave headroom. Your operating system, your browser and the conversation itself all want memory too. Ollama's documentation is direct about that last one: a larger context window needs more memory to run the same model.

Backed by llama.cpp quantize tool: memory and disk requirements, quantized sizes, LM Studio system requirements, Ollama context length and its memory cost.

What actually fits

Use the file size on the download page, not the number in the model's name. They are different things.

One model can come in half a dozen sizes. llama.cpp lists Llama 3.1 8B at 14.96 GiB when it is not squeezed at all, and at 2.95 GiB at its 2-bit setting. Same model, same name, five times the file. The same table goes smaller still, down to 1.87 GiB.

• Llama 3.1 8B at the common 4-bit setting is a 4.9 GB file.
• Llama 3.1 at 70B is 43.1 GB. At 405B it is 249.1 GB.
• A small model such as Qwen3.5 0.8B is 563 MB at 4 bits, and 1.56 GB at full size.

Backed by llama.cpp quantize tool: memory and disk requirements, quantized sizes, Qwen3.5-0.8B-GGUF file listing, llama.cpp: LLM inference in C/C++.

One machine, worked all the way through

Take a real machine. Apple lists the MacBook Air as starting with 16 GB of unified memory and a 512 GB drive. Say you have 40 GB free on that drive.

Now line up the candidates. All four are Llama 3.1 8B. Only the squeeze is different.

The 16-bit file is about the size of the whole machine's memory. It is out before you start, because your operating system and browser have to live somewhere.

The 8-bit file loads and leaves about half the memory free. That is workable, and it is tight once you paste in a long document.

The 4-bit file leaves you roughly eleven gigabytes for everything else. Comfortably more free memory than the file itself takes.

The 70B model, even squeezed to 4 bits, is 43.1 GB. That is more than the free disk, and more than two and a half times the memory. It is not a near miss. It is a different class of machine.

So the answer here is the 4-bit 8B file. Not because it scored best anywhere, but because it is the largest one that leaves the machine room to work.

• 16 bits: 14.96 GiB. Fills the machine.
• 8 bits: 7.95 GiB. Fits, with about half the memory left.
• 4 bits: 4.58 GiB. Fits, with plenty left.
• 2 bits: 2.95 GiB. Fits easily, and rounds the numbers hardest.

Backed by llama.cpp quantize tool: memory and disk requirements, quantized sizes, MacBook Air technical specifications: memory and storage.

Start one size below your ceiling

The usual mistake is to download the largest model that will technically load.

A model that barely fits leaves nothing for the conversation. It will also be slow enough that you quietly stop using it.

Start one size down. If it is quick and good enough, you are done. If it is not, you now know exactly what you are short of.

Quantization gives you a second dial, and it is a real one. In llama.cpp's 2023 measurements on LLaMA, the 13B model at 2 bits predicted text better than the 7B model at 16 bits, in a file under half the size.

The same tests showed the squeezed copy was also faster. On an Apple M2 Max using four threads, the 7B model took 116 milliseconds per token at 16 bits and 55 at the common 4-bit setting.

That was one measure on one pair of models, and an old pair. Take it as a hint about where to look, not as a law.

Backed by k-quants (pull request 1684), with perplexity, file size and timing tables.

Judge it on your own work

Benchmark tables are real measurements. They are not measurements of your job.

They are also usually published by the model's own makers. Qwen's card asks people to use prompts that standardise the model's answers before benchmarking it.

So keep five prompts that look like the work you actually do. Run them on any model you are weighing up.

Judge on those five. A model that comes second on a leaderboard and first on your prompts is the better model for you.

Backed by Qwen3.5-0.8B model card, best practices section.

What those five prompts should be

Vague tests give vague answers. Pick five jobs you already know the right answer to, so you can tell straight away whether the model got it.

Here is a set that covers the usual ground. Swap any of them for something closer to your week.

Save them in a plain text file. Paste the same five into every model you try, in the same order, and note what came back.

Then judge. Two of five is a model you will fight with. Four of five is a model you will use.

• A piece of your own writing to shorten, where you know what has to survive.
• A paragraph of someone else's text with a question whose answer is in it.
• A messy list to turn into a table with columns you name.
• A thing you know it should refuse to guess at, to see whether it says so.
• The most boring job you do weekly, exactly as you would hand it to a person.

A first choice that usually works

Here is a starting point, not a rule. It is meant to get you running today.

LM Studio gives the same advice on the download itself: choose a 4-bit option or higher, if your machine can run it.

• Pick an instruct or chat version, not a base one. Base models continue text rather than answer you.
• Pick the 4-bit file unless you have memory to spare.
• Leave at least as much memory free as the file itself takes, for the conversation and everything else.
• Run your five prompts. Keep it, or step down one size and run them again.

Backed by Chat templates: base models and chat fine-tunes, Ollama context length and its memory cost, LM Studio: Download an LLM, and which download option to choose.