How to set up an LLM gateway

Start with a model you can already reach so you can tell a gateway fault from a model fault. Two good choices: a hosted provider where you have an API key, or a local OpenAI-compatible server. llama.cpp's llama-server is one: llama-server -m model.gguf --port 8080 serves POST /v1/chat/completions on that port.

Write down three things: the provider or server address, the model name that backend expects, and the key (or "none" for a local server with no --api-key). You will put these in the config file in step 3. Do not put real keys in the file itself; LiteLLM reads them from environment variables, as you will see.

If you want to test with no paid account at all, run the local server and use that. Our guide to running LLMs locally covers getting a model onto your machine.

LiteLLM documents two starts. The quickstart script deploys the gateway and a Postgres database with Docker Compose, writes generated security keys to a .env file, and serves the gateway and an admin UI on http://localhost:4000. It is the fastest way to see the product. Read the script before you run it, as you should with anything you pipe into a shell.

The second start is a YAML config file and one docker run, with no database. The documentation describes it as the minimal setup for API-only use and notes that budget enforcement needs a database. This guide uses the config file, because a file you keep in version control is repeatable and reviewable, then adds Postgres in step 5.

If you take the quickstart route, you can still follow steps 6 to 9, and you manage models and keys in the admin UI instead of the file.

The config file has three parts that matter now. model_list is the list of models clients can ask for. For each entry, model_name is the name clients use, and litellm_params.model is the provider-specific string LiteLLM passes on. general_settings.master_key is the admin credential, and the documentation shows it read from the environment as os.environ/LITELLM_MASTER_KEY.

Model strings use a provider/model form, such as anthropic/<model> or openai/<model>. For a custom OpenAI-compatible server, such as llama.cpp, use the openai/ prefix plus an api_base. The docs show exactly this pattern for a vLLM endpoint, with api_key: none. For a hosted provider, put the key in an environment variable and refer to it as os.environ/NAME.

Clients never see the provider key. They send the gateway key, and the gateway adds the provider key on the way out. That is most of the value of the pattern.

Run the official image with the config file mounted and the environment variables passed through. The documentation shows the mount form docker run -v /path/to/config.yaml:/app/config.yaml docker.litellm.ai/berriai/litellm:latest --config /app/config.yaml. Replace latest with a version tag you have tested, and publish port 4000.

Then send a chat request with curl. The endpoint is /chat/completions on port 4000, in the OpenAI format. Because you set a master key, send it as a bearer token. The model value is your model_name from the config, not the provider string.

Any OpenAI SDK now works by changing two values: the base URL to http://localhost:4000 and the API key to a gateway key. Application code does not otherwise change. That is how you migrate applications onto a gateway one at a time.

Do not give applications the master key. Create a virtual key for each application or team. LiteLLM generates them with POST /key/generate, authorised by the master key, and each key can carry a max_budget, a budget_duration, a list of allowed models, and rpm_limit and tpm_limit for requests and tokens per minute.

Budgets and virtual keys need a database. The documentation is explicit that budgets require one, and it takes a PostgreSQL connection string in the DATABASE_URL environment variable. Start a Postgres instance (the quickstart Compose file includes one), add -e DATABASE_URL to the docker run command, and restart the proxy.

When a key goes over its budget the gateway rejects the request with an error such as ExceededTokenBudget, which your application must handle like any other API failure. Set budgets slightly above expected use at first and tighten them once you have a month of data. Rate limits per model can also be set in model_list, for example rpm: 6 on an entry.

Fallbacks are the feature that makes a gateway earn its place. In litellm_settings you set num_retries, request_timeout, allowed_fails and cooldown_time, and a fallbacks list that maps a model name to the models to try next. The documentation says fallbacks run in order, and that a model which fails more than allowed_fails times in a minute is put on cooldown for cooldown_time seconds.

There are specialised fallback lists too: context_window_fallbacks for context-length errors, content_policy_fallbacks for content policy errors and default_fallbacks for anything else. A common pattern is a small, cheap model as the primary, and a larger one as the fallback for context overflows.

Test it. Stop the backend for the primary model, send a request, and check the answer arrives from the fallback. A fallback you have not triggered on purpose is a fallback you do not know works. Check the data protection position of every model in the chain: a fallback to a different provider changes where your data goes.

LiteLLM exposes three health endpoints. GET /health/liveliness says the process is up and checks no dependencies. GET /health/readiness says the worker is ready for traffic, and reports the database state. GET /health runs a real test request against every configured model, so it costs a few tokens per model: do not point a once-a-second probe at it. Use the first two for container probes and the third for an occasional manual check.

The production guide lists the settings to change: export LITELLM_MODE=PRODUCTION to stop automatic .env loading, set LITELLM_SALT_KEY to a random value to encrypt stored credentials (and never change it after deployment), write spend updates to the database every 60 seconds with proxy_batch_write_at: 60, keep error logs out of the database with disable_error_logs: True, and set json_logs: true for structured logs. For several instances it recommends Redis 7.0 or later to share rate limits and router state.

Put TLS in front of the gateway with your reverse proxy or load balancer, and do not expose the admin UI to the internet. Keep the gateway on a network the applications can reach and the public cannot.

A gateway sees every prompt and every answer, which is its value and its risk. Decide what you keep. At minimum you want who called, which model, token counts, cost and latency. Whether you also keep prompt and response text depends on your data protection position; keeping them makes debugging easier and makes the log a store of personal data you must protect, retain and delete properly.

LiteLLM supports guardrails, including a Presidio-based PII masking guardrail with a pre_call mode that masks or blocks entities such as credit card numbers before the request reaches the model. The configuration is a guardrails block with a guardrail_name and litellm_params. Read the documentation page for the services Presidio needs before you rely on it, and test it on realistic text, because pattern-based masking misses things.

Redaction in the gateway does not replace deciding what data should go to a model at all. Combine it with an allow-list of models per key and, for sensitive work, a self-hosted model. For the wider practice, see how to monitor AI agents in production.

Pin versions. Run an image tag you have tested, not latest, and promote a new tag through staging. On 24 March 2026 two releases of the litellm Python package on PyPI, 1.82.7 and 1.82.8, were malicious. LiteLLM's own incident write-up says they were live for about 40 minutes, that the official Docker proxy image was not affected because it pins its dependencies, and that users of the bad versions should rotate every secret, look for litellm_init.pth, and pin to 1.82.6 or earlier until safe releases were confirmed. It reports releases from 1.78.0 to 1.82.6 audited, and a new CI pipeline for 1.83.0.

The lesson applies to any gateway, not only this one. The gateway holds your provider keys, so it is a high-value target. Install from a pinned image or a lockfile with hashes, keep provider keys in a secrets manager, give each application a virtual key with the narrowest model list and budget, and keep the admin interface off the public internet. Plan the rotation: if you had to replace every provider key tonight, how long would it take?

Finally, decide who is paged when the gateway is down. A gateway moves a failure from many applications to one service, which is easier to watch and more damaging when it breaks. Run at least two instances behind a load balancer once real traffic depends on it.