Ollama: what it is, what it is for, and when to use something else
The simplest way to run open models on your own machine.
At a glance
| Capability | Ollama |
|---|---|
| Licence | MIT |
| Implementation | Go (wrapping llama.cpp) |
| Runs on | CPU or GPU |
| Continuous batching | No |
| Paged attention | No |
| Tensor parallelism | No |
| Quantisation | GGUF (2–8 bit) |
| OpenAI-compatible API | Yes |
Ollama is a local model runner built on top of llama.cpp. It wraps model download, quantisation selection, prompt templating and an HTTP server behind a single command, which is why it has become the default entry point to running open weights. A Modelfile gives you a Docker-like way to pin a base model with a system prompt and parameters.
The reason it matters operationally is the API surface: Ollama exposes an OpenAI-compatible endpoint on localhost, so the same client code that talks to a hosted provider talks to a local model with a base-URL change. That makes it useful as a development target even for teams whose production traffic never touches it.
Where it runs out of road is concurrency. Ollama serves requests without continuous batching or paged attention, so throughput under parallel load falls well short of a dedicated inference server. It also runs on CPU when no GPU is present, which is excellent for accessibility and poor for latency. Teams typically develop against Ollama and serve with vLLM or SGLang.
Where it stops being the right tool
Single-node by design. There is no tensor parallelism and no built-in multi-GPU sharding for models larger than one card, so it is not the tool for production serving at concurrency.
Frequently asked
- What is Ollama used for?
- Developers who want a model running locally in one command, and teams prototyping against open weights before committing to a serving stack.
- What are the limitations of Ollama?
- Single-node by design. There is no tensor parallelism and no built-in multi-GPU sharding for models larger than one card, so it is not the tool for production serving at concurrency.
- Does Ollama need a GPU?
- No. Ollama runs on CPU, and offloads to a GPU when one is available. CPU execution is much slower but works for evaluation and small models.
Compare with
- LM Studio — A desktop GUI for discovering, running and chatting with local models.
- vLLM — High-throughput inference server built around PagedAttention.
- llama.cpp — Portable C/C++ inference that runs open models almost anywhere.
- SGLang — Inference server optimised for structured generation and prefix reuse.
Project home: https://ollama.com