How to reduce LLM costs

You cannot cut a bill you cannot attribute. Provider dashboards show spend per key, which is rarely the unit you care about. Wrap your model calls in one small function that writes a log line per call with a feature name, a model name, a user or tenant id and the token counts the provider returns.

The field names differ by provider. Anthropic returns input_tokens, output_tokens, cache_creation_input_tokens and cache_read_input_tokens in the usage object. OpenAI Responses returns input_tokens, output_tokens, input_tokens_details.cached_tokens and output_tokens_details.reasoning_tokens. Log all of them. Reasoning tokens and cache writes are billed, and they are the usual reason a bill does not match a naive count.

Do not log prompt text here. Token counts and ids are enough for cost work, and keeping content out of the log avoids a second data-protection problem.

After a day or a week of logs, add the tokens up by feature and by model. Multiply by your current per-million-token prices to get money. Most teams find that two or three features cause most of the cost, and that within those features a few long prompts or very long outputs matter more than the average request.

Look for four patterns. A large static prefix repeated on every call (caching will help). Output far longer than the task needs (a cap will help). Work nobody is waiting for, such as nightly enrichment or evaluation runs (a batch API will help). And an expensive model doing simple work such as classification or extraction (routing will help). Write down which pattern each of your top three matches before you touch any code.

Output tokens cost more than input tokens at every major provider, so unbounded output is the cheapest place to save. Set an explicit limit on every call. Anthropic calls it max_tokens, the maximum number of tokens to generate before stopping. OpenAI Responses calls it max_output_tokens, an upper bound that includes visible output and reasoning tokens.

Set the cap from your log, not from a guess: take the 99th percentile of output tokens for the feature and add a margin. Then check the stop_reason field on Anthropic responses. A value of max_tokens means the answer was cut off, so the cap is too low for that feature. Ask for the format you need in the prompt itself ("answer in two sentences", "return only the JSON object") rather than relying on the cap to truncate.

Trim the input as well. Remove instructions the model no longer needs, drop examples that never change the answer, and stop re-sending a whole conversation when a short summary of it would do. Keep an eye on quality while you do this, because a shorter prompt that lowers accuracy is not a saving.

Put everything that does not change first (system prompt, tool definitions, reference documents) and everything that changes last (the user message). Caching works on the unchanged beginning of the prompt, so one timestamp or request id near the top defeats it.

On Anthropic, add cache_control with type ephemeral to the system block, or at the top level of the request for automatic caching. The default lifetime is 5 minutes, and "ttl": "1h" selects a 1-hour cache. As of 2026-10-06 the docs list a 5-minute cache write at 1.25 times the base input price, a 1-hour write at 2 times, and a cache read at 0.1 times for most models, with lower read multipliers on a few newer ones. Prompts shorter than the model minimum (512, 1,024, 2,048 or 4,096 tokens depending on the model) are not cached.

On OpenAI, caching is on by default for supported models, so there is nothing to switch on. The docs give a 1,024-token minimum on the newest models, a 30-minute default lifetime there, and a read price of 0.1 times the uncached input price. Earlier models use prompt_cache_retention with in_memory or 24h. A stable prompt_cache_key helps requests with a shared prefix land together. Check the hit rate in the field you logged: cached tokens divided by total input tokens.

Evaluation runs, nightly enrichment, bulk classification and back-fills do not need an answer in two seconds. Both Anthropic and OpenAI document a 50% discount for batch processing. On Anthropic, all batch usage is charged at 50% of standard prices. On OpenAI, the batch guide states a 50% discount and a completion window that can currently only be set to 24h.

The limits matter when you plan the job. Anthropic allows up to 100,000 requests or 256 MB per batch, most batches finish within an hour, they expire after 24 hours, and results stay available for 29 days. OpenAI allows up to 50,000 requests and a 200 MB input file per batch. Each request carries a custom_id so you can match results to inputs.

The example below creates an Anthropic batch with two requests and then polls and reads the results. Streaming is not supported inside a batch. Check the provider docs for whether the batch discount combines with caching on your model, because Anthropic suggests the 1-hour cache for batches that share context.

Model routing sends each request to the cheapest model that meets your quality bar. The simplest version needs no machine learning: pick the cheap model for whole features that are easy (classification, extraction, reformatting) and keep the stronger model for planning and open-ended reasoning. A second version tries the cheap model first and escalates when a check fails, for example when the output does not parse against a schema or when a confidence field is low.

Do not switch on feel. Take the 30 to 50 real requests from your prerequisites, run them through both models and compare the outputs with whatever scoring you trust: exact match for labels, schema validity for extraction, a rubric for free text. Move a feature to the cheaper model only when the score holds. Then keep the gate: re-run the set whenever you change a prompt or a model.

Count the escalations. If the cheap model fails a third of the time and every failure triggers a second call to the strong model, you pay for two calls on those requests and may spend more than before. Log which model answered, and watch the escalation rate next to cost.

A response cache stores the final answer for an identical request. It suits deterministic features such as classification of a repeated input, FAQ-style questions and generated summaries of documents that have not changed. Key the cache on the model id, the full prompt and the settings, and give entries a lifetime so stale answers expire. Do not cache anything that depends on the user, or that contains personal data, unless the key includes the user.

If you retrieve passages from a knowledge base, check how many you send. Many pipelines pass the top ten chunks when the top three contain the answer. Test the smaller number against your request set. Fewer, better chunks cost less and often answer better. Our cost of RAG page breaks down where retrieval spend goes, and how to build a RAG system covers choosing chunk size and count.

Exact-match caching is the safe starting point. Semantic caching, which returns an earlier answer for a similar question, saves more calls but can return a wrong answer for a question that only looks similar. Add it only with a measured similarity threshold and a sample review.

Put a budget on each feature, and alert when daily spend passes a multiple of its trailing average. Runaway cost is almost always a loop, a retry storm or a prompt that grew. A gateway can enforce caps for you; how to set up an LLM gateway shows the steps.

Self-hosting turns a variable cost into a fixed one, so it only wins at steady, high volume. The break-even is the monthly fixed cost divided by the saving per token. Fixed cost is the GPU rental (or purchase, spread over its life), power, and the engineering time to run and patch it. Use the throughput you measure in your own load test, at your latency target, with your prompts, because published benchmark numbers rarely match your workload.

The worked calculation below uses hypothetical prices to show the method. Replace every number with your own before you decide anything. If the break-even volume is far above what you send today, stay on the API and revisit when volume grows. If you want to try it anyway, how to self-host an LLM is the next guide.