Ollama vs llama.cpp: which to use and when
The simplest way to run open models on your own machine. Portable C/C++ inference that runs open models almost anywhere.
At a glance
| Capability | Ollama | llama.cpp |
|---|---|---|
| Licence | MIT | MIT |
| Implementation | Go (wrapping llama.cpp) | C/C++ |
| Runs on | CPU or GPU | CPU or GPU |
| Continuous batching | No | Yes |
| Paged attention | No | No |
| Tensor parallelism | No | No |
| Quantisation | GGUF (2–8 bit) | GGUF (1.5–8 bit), K-quants, imatrix |
| OpenAI-compatible API | Yes | Yes |
How to choose
Ollama — Developers who want a model running locally in one command, and teams prototyping against open weights before committing to a serving stack.
llama.cpp — CPU inference, Apple Silicon, edge and embedded targets, and anywhere a Python runtime and a CUDA driver are not welcome.
Both expose an OpenAI-compatible HTTP API, so this is not a one-way door: switching is a base-URL change, and running one locally while serving on the other is a common and sensible split.
Frequently asked
- Should I use Ollama or llama.cpp?
- Developers who want a model running locally in one command, and teams prototyping against open weights before committing to a serving stack. By contrast, llama.cpp is the better answer when: cPU inference, Apple Silicon, edge and embedded targets, and anywhere a Python runtime and a CUDA driver are not welcome.
- Can I use both?
- Yes, and most teams do. Because both expose an OpenAI-compatible HTTP API, moving a workload between them is a base-URL change. A common pattern is developing against the lighter runtime locally and serving production traffic on the higher-throughput one.
Project home: https://ollama.com