When one machine is not enough

The first thing that breaks is waiting

You have been running a model on your own machine and it has been fine. Then a second person starts using it.

The machine does not fall over. It makes somebody wait.

Ollama handles one request per model at a time unless you tell it otherwise. That is the setting it ships with.

The second request joins a queue. Ollama holds up to 512 waiting requests, and past that it turns new ones away with an error saying the server is overloaded.

llama.cpp's server works in slots, and one request sits in one slot. It counts the requests it had to put off, so you can watch that number climb.

The first sign of growth is not a slow answer. It is somebody staring at a blank screen while your answer finishes.

Backed by Ollama FAQ: concurrent requests, the queue, and the memory they cost, llama.cpp HTTP server: parallel decoding, continuous batching, slots and deferred requests.

What the second person actually sees

Nothing on the screen says you are in a line. That is the whole trouble with it.

They just see a longer pause before the first word. NVIDIA is direct about this. The wait before the first token usually includes queuing time, alongside the reading and the network.

So the queue hides inside a number you already knew about. It does not arrive with a label on it.

Pip's public test run shows what a machine at its limit looks like over time. On a 14-core Intel i9 with 32 GB of memory and no separate graphics card, 10,981 requests arrived across 45 days and 97.93 percent of them finished.

The other 2.07 percent arrived when the machine had no room left, and were asked to retry. Pip's page says that computer had less memory than the 48 GB it recommends for that model, and was asked to do more at once than it could hold.

That is the honest picture of one machine serving more than one person. Not a crash. A small, steady share of people told to come back.

Backed by LLM inference benchmarking: queuing time, and throughput against per-user speed, Ollama FAQ: concurrent requests, the queue, and the memory they cost, How Pip is tested: 45 days, 10,981 requests, and what happened when it was full.

Several at once can beat one at a time

Two jobs done together are not always twice the work. For a model they are often much less.

llama.cpp says so plainly. Working out several tokens in one batch is cheaper than working them out one after another.

Its server does what it calls continuous batching, and it is on by default. Answers being written for different people are worked on in the same pass.

So the machine gets more done per hour. Each person is sharing it now, so nobody's answer is the fastest it could be.

Batching buys throughput, not speed. Throughput is how much a machine finishes in an hour. Speed is how long one person waits. llama.cpp's own guide treats those as two goals you trade between.

Backed by llama.cpp speculative decoding: why a batch is cheaper than one at a time, llama.cpp HTTP server: parallel decoding, continuous batching, slots and deferred requests, Using multiple GPUs with llama.cpp: when you need it, and when it is slower.

What sharing looks like in numbers

That trade has a shape, and both sides of it can be measured.

NVIDIA states the rule without any numbers. As more requests run at once, the tokens a second across the whole system go up. The tokens a second any one person sees go down, because their wait is longer.

llama.cpp ships a tool for measuring exactly this, and publishes a sample table with it. That table does not name the machine it ran on. So the speeds in it are not yours and are not anybody's, and this lesson uses only the ratio between them.

Going from one request at a time to eight, the machine finished more than three times as much work in total. Each person waited more than twice as long for their own answer.

The memory grew alongside it. The room the conversations needed went from 256 tokens' worth to 2,048.

Backed by LLM inference benchmarking: queuing time, and throughput against per-user speed, llama-batched-bench: sample results for one, two, four, eight and more at once.

What serving several at once costs

The model is shared between them. The conversations are not.

Every request in flight needs its own room to hold its own conversation. That room is memory, and memory was already your ceiling.

Ollama is blunt about the price. Four requests at once with a 2K context give you an 8K context and more memory used.

llama.cpp says the same in other words. Each slot holds its own store, and its guide lists cutting the number of slots as a fix when a machine runs out of memory.

So the setting that lets more people in is the setting that eats the memory. There is no version of this that is free.

Backed by Ollama FAQ: concurrent requests, the queue, and the memory they cost, Using multiple GPUs with llama.cpp: when you need it, and when it is slower.

Three options, and only three

When one machine is no longer enough, you have three moves. Everything else is a mix of them.

Try them in the order they are given, because that is the order of what they cost.

• Make the work smaller. A smaller model, or a shorter context, which Ollama says needs less memory. It costs quality: llama.cpp says squeezing a model may bring some accuracy loss.
• Buy a bigger machine. What you are buying is memory. llama.cpp says a model too large for one graphics card runs its leftovers from slower system memory.
• Send some of it away. Ollama's cloud models run on its service rather than yours, so models too big for a personal computer still work. The work has left your machine.

Backed by llama.cpp quantize tool: shrinking a model may cost accuracy, Ollama: context length and the memory it needs, Using multiple GPUs with llama.cpp: when you need it, and when it is slower, Ollama: Cloud models.

What each one costs you in effort

Money is the easy part of that list. Effort is the part people find out about afterwards.

Making the work smaller costs an afternoon and can be undone. Fetch a smaller model, or turn the context down. The bill arrives as quality, and you are the person who notices it.

A bigger machine costs money once and attention from then on. Somebody has to set it up, keep it updated, and be the person it is broken for.

llama.cpp's guide to running two graphics cards is a fair preview of that job. One of its modes is marked experimental. One option has to be switched on or the thing refuses to start. Some model designs are not supported at all.

Sending work away costs the least effort and the most of what Track 4 asked about first. Ollama's cloud models need an account with Ollama and a sign-in, and the work runs on their computers.

The cheapest of the three is the first one, and it is the one people skip.

Backed by Using multiple GPUs with llama.cpp: when you need it, and when it is slower, Ollama: Cloud models, Ollama: context length and the memory it needs.

When to stop scaling the laptop

Sometimes none of the three is the answer. Sometimes the honest move is to stop putting this on a laptop at all.

A laptop is one person's machine. It closes, it sleeps, it goes home in a bag. A thing other people depend on cannot do that.

Here are the markers. Any two of them and you have gone past the point.

Lesson 17 gave you the order of the questions for what comes next. Lesson 18 gave you the rule for routing the work.

You have not failed at local AI. You have outgrown one seat of it, and that is a better problem than the one you started with.

• More than one person needs an answer at the same time, most days.
• The machine is also the machine somebody has to work on.
• It has to answer while its owner is asleep, or away, or on a train.
• You are thinking about a second laptop to take the overflow.