Text Generation Inference (TGI): what it is, what it is for, and when to use something else
Hugging Face's production inference server.
At a glance
| Capability | Text Generation Inference (TGI) |
|---|---|
| Licence | Apache 2.0 |
| Implementation | Rust + Python |
| Runs on | GPU only |
| Continuous batching | Yes |
| Paged attention | Yes |
| Tensor parallelism | Yes |
| Quantisation | AWQ, GPTQ, EETQ, bitsandbytes |
| OpenAI-compatible API | Yes |
TGI is Hugging Face's inference server, with a Rust HTTP layer over Python model code. It implements continuous batching, paged attention and tensor parallelism, and pulls weights directly from the Hub, which removes a step for teams whose model governance already lives there.
Its differentiator is integration rather than a novel algorithm: token streaming, safetensors loading, and Hub-native model resolution work without glue code. For organisations whose model registry is the Hub, that is a real operational saving.
Where vLLM and SGLang compete on serving throughput and novel cache management, TGI competes on being the path of least resistance inside an existing Hugging Face workflow.
Where it stops being the right tool
Smaller community momentum than vLLM, and historically a more restrictive licence period that still colours some procurement reviews.
Frequently asked
- What is Text Generation Inference (TGI) used for?
- Teams already standardised on the Hugging Face stack who want a supported serving path with tight Hub integration.
- What are the limitations of Text Generation Inference (TGI)?
- Smaller community momentum than vLLM, and historically a more restrictive licence period that still colours some procurement reviews.
- Does Text Generation Inference (TGI) need a GPU?
- Yes. Text Generation Inference (TGI) targets GPU execution and is not a practical CPU inference path.
Compare with
- Ollama — The simplest way to run open models on your own machine.
- LM Studio — A desktop GUI for discovering, running and chatting with local models.
- vLLM — High-throughput inference server built around PagedAttention.
- llama.cpp — Portable C/C++ inference that runs open models almost anywhere.
Project home: https://huggingface.co/docs/text-generation-inference