llama.cpp: what it is, what it is for, and when to use something else
Portable C/C++ inference that runs open models almost anywhere.
At a glance
| Capability | llama.cpp |
|---|---|
| Licence | MIT |
| Implementation | C/C++ |
| Runs on | CPU or GPU |
| Continuous batching | Yes |
| Paged attention | No |
| Tensor parallelism | No |
| Quantisation | GGUF (1.5–8 bit), K-quants, imatrix |
| OpenAI-compatible API | Yes |
llama.cpp is a dependency-light C/C++ implementation of transformer inference, and it is the substrate a large share of the local-model ecosystem is built on — Ollama and LM Studio both rely on it. It runs with no Python runtime, compiles for x86, ARM and Apple Silicon, and offloads to CUDA, Metal, ROCm or Vulkan when hardware is available.
Its second contribution is GGUF, the quantisation format that made small-machine inference practical. K-quants and importance-matrix quantisation preserve quality at aggressive bit widths far better than naive rounding, which is what allows a mid-size model to run usefully in single-digit gigabytes of memory.
For production GPU serving at concurrency, vLLM or SGLang will extract more from the same card. llama.cpp wins wherever portability, CPU execution, Apple Silicon, or a minimal deployment footprint matter more than peak aggregate throughput.
Where it stops being the right tool
Throughput under heavy concurrent load trails purpose-built GPU servers; it optimises for portability and single-stream latency rather than fleet throughput.
Frequently asked
- What is llama.cpp used for?
- CPU inference, Apple Silicon, edge and embedded targets, and anywhere a Python runtime and a CUDA driver are not welcome.
- What are the limitations of llama.cpp?
- Throughput under heavy concurrent load trails purpose-built GPU servers; it optimises for portability and single-stream latency rather than fleet throughput.
- Does llama.cpp need a GPU?
- No. llama.cpp runs on CPU, and offloads to a GPU when one is available. CPU execution is much slower but works for evaluation and small models.
Compare with
- Ollama — The simplest way to run open models on your own machine.
- LM Studio — A desktop GUI for discovering, running and chatting with local models.
- vLLM — High-throughput inference server built around PagedAttention.
- SGLang — Inference server optimised for structured generation and prefix reuse.
Project home: https://github.com/ggml-org/llama.cpp