Ollama vs llama.cpp vs vLLM: which local LLM tool fits the job

You've decided to run a language model on your own hardware. Then you hit the next question: which software actually runs it? Three names come up again and again (Ollama, llama.cpp and vLLM), and the forum advice on them is mostly people defending whatever they installed first. They solve different problems. Once you know what each one is built for, the choice usually makes itself.

Here's the short version. llama.cpp is the engine. Ollama wraps an engine like it in a friendly tool for one person or a small team. vLLM is a serving system for GPUs that have to answer many requests at once. The rest of this post explains why, using what each project's own documentation says, as of October 2026.

llama.cpp: the engine that runs almost anywhere#

llama.cpp states its main goal as enabling LLM inference "with minimal setup and state-of-the-art performance". It's a plain C/C++ implementation with no dependencies. Apple silicon is treated as a first-class platform through Metal, there are CUDA kernels for NVIDIA cards, AMD support via HIP, and Vulkan and SYCL backends as well.

Two features matter most for small setups. First, it supports integer quantisation from 1.5-bit up to 8-bit, which is how a model that would need a data-centre GPU at full precision ends up fitting on a desktop card. Second, it can do CPU+GPU hybrid inference, splitting a model so that part runs on the GPU and the rest on the CPU. That lets you run something larger than your VRAM, slowly, instead of not at all.

It also ships its own HTTP server, llama-server. The README lists OpenAI-compatible chat completions, responses and embeddings routes, parallel decoding with multi-user support, continuous batching, monitoring endpoints, schema-constrained JSON output and a built-in web UI. The number of parallel slots defaults to automatic, and the server listens on 127.0.0.1 unless you tell it otherwise.

The cost is that you're closer to the metal. You choose and download the GGUF model file yourself, pick the quantisation, and set options such as context size and the number of server slots. That's ideal if you want control or you're building llama.cpp into something else. It's more work if you just want a model answering questions this afternoon.

Ollama: the easiest path for one person or a small team#

Ollama is the tool most people meet first, and for good reason. The Ollama repository offers installers for macOS, Windows and Linux plus an official Docker image, and it lists llama.cpp among its supported backends. Its quickstart is a single command, ollama run followed by a model name, after which Ollama downloads the model and starts a chat.

A few defaults from the Ollama FAQ explain how it behaves in practice:

• It binds to 127.0.0.1 on port 11434, so nothing else on your network can reach it until you change OLLAMA_HOST.
• Models stay in memory for 5 minutes after the last request, then unload. The first request after that waits while the model loads again.
• Each model processes 1 request at a time by default (OLLAMA_NUM_PARALLEL). When it's busy, Ollama queues up to 512 requests before rejecting more.
• It can keep several models loaded at once (by default three per GPU, or three for CPU inference), provided they fit in memory.

That parallel default is the one that surprises people. One developer chatting to a model won't notice. Point five staff, or an automation that fires off a batch of documents, at the same Ollama instance and requests will queue behind each other. You can raise OLLAMA_NUM_PARALLEL, but the FAQ also says required RAM scales with the number of parallel requests multiplied by the context length, so check that the extra slots still fit.

Ollama speaks a subset of the OpenAI API at http://localhost:11434/v1/, covering chat completions, completions, models, embeddings and a non-stateful version of responses. In practice, many tools written for OpenAI can be pointed at a local Ollama by changing the base URL. "Subset" is the important word: test the specific features your code relies on.

You can also bring your own model files. Ollama's import guide shows a Modelfile with a FROM line pointing at a GGUF file, then ollama create. It also says Ollama doesn't quantise GGUF models during import and suggests preparing them first with llama.cpp's llama-quantize. That's a neat illustration of how the two projects relate.

vLLM: built to serve many requests on GPUs#

vLLM comes from a different starting point. Its documentation describes it as "a fast and easy-to-use library for LLM inference and serving", and its feature list reads like a throughput checklist: continuous batching of incoming requests, chunked prefill, prefix caching, speculative decoding, and tensor, pipeline, data, expert and context parallelism for spreading a model over several GPUs.

The core idea is in the original PagedAttention paper by the vLLM authors. Serving many users at once is limited by the key-value (KV) cache each request needs, and that memory gets wasted through fragmentation and duplication when it's managed naively. PagedAttention manages it in blocks, much as an operating system pages memory. In the paper's evaluations, vLLM improved throughput of popular models by 2–4× at the same latency compared with FasterTransformer and Orca. Those were the authors' benchmarks from 2023 against the systems of that time, so read them as the reason the design exists, not a figure you'll reproduce on your own server.

The current docs list support for NVIDIA GPUs, AMD GPUs and x86, ARM and PowerPC CPUs, with plugins for other accelerators. Quantisation is covered too, including FP8, INT8, INT4, GPTQ/AWQ and GGUF, and it can produce structured outputs and handle tool calling. Like the others, it exposes an OpenAI-compatible API server.

The trade-offs show up in the quickstart. It lists Linux as the operating system (with a separate vLLM-Metal route mentioned for Apple Silicon), and it says the server currently hosts one model at a time. You start it with vllm serve <model> and it listens on port 8000. So vLLM suits a dedicated GPU box serving one model to lots of callers. It's an awkward fit for a laptop where you want to flip between three models during the day.

Choosing between them#

Most decisions come down to who is sending requests and on what hardware.

Because all three offer OpenAI-compatible endpoints, the choice isn't permanent. If you write your application against that interface and keep the base URL and model name in configuration, you can prototype on Ollama and move to vLLM later without rewriting the code that calls it. That's how we approach client builds at TheAILAB: start simple, and only take on the operational weight of a serving stack once real usage shows you need it.

Things that catch people out#

Network exposure first. Both Ollama and llama-server listen only on 127.0.0.1 by default, which is the safe choice. The moment you change that so colleagues can connect, you've got an API on your network that anyone who can reach the port can use. Put it behind a reverse proxy with authentication, or a VPN, and treat it like any other internal service. Running a model locally keeps your data off a third party's servers. It doesn't make the setup secure on its own.

Second, measure with your real workload. A model that feels quick in a single chat can slow to a crawl when ten requests arrive together on a tool with one slot. Before you commit, send a realistic burst of requests, at your real prompt lengths, to each candidate and time it.

Third, compare like with like. The same model at Q4 in llama.cpp and at FP8 in vLLM are different files with different quality and memory use. Fix the model and quantisation first, then compare the tools.

If you only take one step today: install Ollama, run the model you're interested in, and write down how many people or processes will call it at once. If that number is one or two, you may never need anything else. If it's dozens, that's when vLLM earns its setup time.

Sources

1. FAQ — Ollama documentation (checked 2026-10-10)
2. OpenAI compatibility — Ollama documentation (checked 2026-10-10)
3. Importing a Model — Ollama documentation (checked 2026-10-10)
4. Quickstart — Ollama documentation (checked 2026-10-10)
5. ollama/ollama repository README — Ollama (GitHub) (checked 2026-10-10)
6. llama.cpp: LLM inference in C/C++ — ggml-org (GitHub) (checked 2026-10-10)
7. llama.cpp HTTP server README — ggml-org/llama.cpp (GitHub) (checked 2026-10-10)
8. vLLM documentation — vLLM project (latest docs) (checked 2026-10-10)
9. Quickstart — vLLM documentation (checked 2026-10-10)
10. Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon, Li, Zhuang, Sheng, Zheng, Yu, Gonzalez, Zhang and Stoica (arXiv, 2023) (checked 2026-10-10)