Why do I get context_length_exceeded when my prompt is under the limit?

Because `max_tokens` reserves completion space inside the same window. The provider adds your prompt tokens to the requested output ceiling before generating, so a 126K prompt with a 4096-token ceiling overflows a 128K window even though the answer would have been short. Lower the ceiling or shrink the prompt.

How do I count tokens before sending a request?

For OpenAI models use `tiktoken` with the encoding for that model, and add a few tokens per message for the chat formatting overhead. For Claude, use Anthropic's `count_tokens` endpoint, which accounts for the system prompt, tools, and messages together: `tiktoken` is a different tokenizer and undercounts Claude prompts significantly.

Should I truncate or summarise conversation history?

Truncate when only recent turns matter. It is free and predictable. Summarise when the agent must remember decisions from far earlier, accepting one extra model call per compaction. Most production systems combine both: a pinned running summary plus the last N turns kept verbatim.

Does a larger context window solve context_length_exceeded?

It removes the error but not the cost. Input tokens are billed per token and time-to-first-token grows with prompt size, so a ten-times-larger prompt costs roughly ten times more on every call and responds more slowly. Use a large window for genuinely large documents, not as a substitute for trimming and retrieval.

Do tool and function definitions count toward the context window?

Yes. Tool names, descriptions, and full JSON schemas are serialised into the prompt on every request, and a large tool catalogue can consume thousands of tokens before the user says anything. Trim descriptions, flatten deep schemas, or load tools on demand rather than registering all of them every call.

Error reference

context_length_exceeded: Why Your Prompt Overflows and How to Fix It

Updated July 27, 2026

In short

context_length_exceeded ("This model's maximum context length is N tokens") means the prompt plus the requested completion is larger than the model's window. Input and output share one budget, so the most common cause is not a huge prompt but an oversized max_tokens, which reserves output space inside the same window. Count tokens before you send (tiktoken for OpenAI models, the count_tokens endpoint for Claude), then trim history, summarise, or retrieve less.

start here

python -c "import tiktoken,sys; print(len(tiktoken.get_encoding('o200k_base').encode(sys.stdin.read())))" < prompt.txt

Diagnosis

Causes at a glance

Cause	How to recognize it	Fix
max_tokens reserves output inside the same window	The error fires before any output, and the prompt alone fits comfortably	Compute max_tokens as window minus prompt tokens minus a safety margin
Unbounded conversation history	The first turns work, then every turn after roughly turn N fails	Trim the oldest turns or fold them into a running summary before each call
RAG retrieving too many or too large chunks	Failure tracks top_k or correlates with one long source document	Cap chunk size, rerank, and pass only the top few chunks
System prompt and tool schemas counted too	A short user message still overflows; many tools are registered	Trim tool descriptions and schemas, or load tools on demand
A different model than you assumed	The message quotes a smaller window than the model you configured	Log the resolved model ID: a fallback or proxy alias may have swapped it

Every provider enforces a single context budget covering the system prompt, tool and function schemas, the full message history, any retrieved documents, and the tokens the model is allowed to generate. When people size a prompt against the advertised window they usually forget the last item, and that is what tips the request over.

The reservation happens before generation, which is why the error arrives instantly rather than mid-response. A 128K-window model handed a 126K prompt and max_tokens: 4096 fails immediately even though the answer would have been two hundred tokens long: the request asked to reserve 4096 it did not have.

# OpenAI
{"error":{"message":"This model's maximum context length is 128000 tokens. However, you requested 130512 tokens (126416 in the messages, 4096 in the completion).","type":"invalid_request_error","code":"context_length_exceeded"}}

# Anthropic
{"type":"error","error":{"type":"invalid_request_error","message":"prompt is too long: 213418 tokens > 200000 maximum"}}

Step 1

Step 1: count tokens before you send

Guessing from character counts is unreliable: the ratio shifts sharply for code, JSON, and non-English text. Use the tokenizer that matches the model. For OpenAI models, tiktoken with the encoding for that model gives an exact count of the content, though you should add a small per-message overhead for the chat formatting itself.

For Claude, do not use tiktoken. It is a different tokenizer and undercounts. Anthropic exposes a count_tokens endpoint that accounts for the system prompt, tools, and messages together, which is the only accurate way to measure a Claude request before sending it.

# OpenAI models
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
prompt_tokens = sum(len(enc.encode(m["content"])) for m in messages)
prompt_tokens += 4 * len(messages) + 3   # per-message + priming overhead

# Claude models — tiktoken is the wrong tokenizer here
count = client.messages.count_tokens(
    model="claude-sonnet-4-5", system=system, tools=tools, messages=messages
)
print(count.input_tokens)

Step 2

Step 2: size max_tokens against what is left

Stop hard-coding the completion ceiling. Derive it from the measured prompt size and the model window, keep a margin for formatting overhead and count drift, and floor it so a nearly full window fails loudly instead of returning a truncated one-word answer.

The same calculation gives you a cheap guard: if the remaining budget falls below your floor, you know before spending a request that the prompt itself has to shrink.

WINDOW = 128_000
RESERVE = 512            # formatting overhead + counting margin
MIN_OUTPUT = 256

available = WINDOW - prompt_tokens - RESERVE
if available < MIN_OUTPUT:
    raise ValueError(f"prompt too large: {prompt_tokens} of {WINDOW} tokens used")

resp = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    max_completion_tokens=min(available, 4096),
)

Step 3

Step 3, choose a history strategy: truncate, summarise, or slide

Truncation drops the oldest turns and keeps the system prompt. It is cheap, predictable, and lossy in an obvious way: fine for support chats and task assistants where old turns stop mattering.

Summarisation compacts older turns into a running summary with an extra model call. It preserves long-range facts at the cost of latency and a second request, and it is the right default for agents that must remember decisions made an hour ago.

A sliding window keeps the system prompt, a pinned summary, and the last N turns verbatim: the common production shape, because it bounds cost per turn while keeping recent context exact. Whichever you pick, apply it before the request rather than reacting to the error afterwards.

def fit(system, history, budget, count):
    """Keep the system prompt plus the newest turns that fit in budget."""
    kept, used = [], count([system])
    for msg in reversed(history):            # newest first
        cost = count([msg])
        if used + cost > budget:
            break
        kept.append(msg)
        used += cost
    return [system] + list(reversed(kept))   # chronological order restored

Step 4

Step 4: fix RAG chunking and retrieval depth

Retrieval pipelines overflow for a structural reason: top_k multiplies chunk size, and neither is bounded by the context window. Ten 2,000-token chunks is 20,000 tokens of input before the conversation even starts, and one oversized source document can blow the budget on its own.

Cap chunk size at ingestion, use a modest overlap, then rerank the candidates and pass only the few that survive. Retrieving twenty chunks and reranking to four is both cheaper and more accurate than sending twenty: relevant content buried in a wall of near-misses measurably degrades answers.

CHUNK_TOKENS = 512
OVERLAP = 64
RETRIEVE_K = 20     # wide recall from the vector store
KEEP_K = 4          # narrow precision after reranking

candidates = store.search(query, k=RETRIEVE_K)
chunks = rerank(query, candidates)[:KEEP_K]
context_tokens = sum(count(c) for c in chunks)   # bounded and predictable

Long-context models are not a free fix

Moving to a 200K or 1M window removes the error but not the underlying cost. Input tokens are billed per token, so a prompt that grew ten times costs roughly ten times as much on every single call, and time-to-first-token rises with prompt size because the whole prefix must be processed before generation starts.

Retrieval quality also degrades when everything is stuffed in. Prompt caching helps when a large prefix is genuinely stable and reused, but it does not rescue a design where the prompt grows without bound. Treat a big window as headroom for legitimately large single documents, not as a substitute for trimming, summarising, and retrieving well.

Beyond the one-off fix

Context length is where cost is decided: every token you leave in the prompt is billed on every turn, and agent loops resend the same bloated history dozens of times an hour. Nexus measures baseline-versus-compressed token usage per agent action, so trimming history, tightening RAG chunking, or caching a prefix produces a number rather than a hunch, and the attribution is per user, repo, and terminal, which is how you find the one agent whose context has been growing unbounded since a prompt change three days ago.

Explore Swfte Nexus

Common questions

Why do I get context_length_exceeded when my prompt is under the limit?: Because `max_tokens` reserves completion space inside the same window. The provider adds your prompt tokens to the requested output ceiling before generating, so a 126K prompt with a 4096-token ceiling overflows a 128K window even though the answer would have been short. Lower the ceiling or shrink the prompt.
How do I count tokens before sending a request?: For OpenAI models use `tiktoken` with the encoding for that model, and add a few tokens per message for the chat formatting overhead. For Claude, use Anthropic's `count_tokens` endpoint, which accounts for the system prompt, tools, and messages together: `tiktoken` is a different tokenizer and undercounts Claude prompts significantly.
Should I truncate or summarise conversation history?: Truncate when only recent turns matter. It is free and predictable. Summarise when the agent must remember decisions from far earlier, accepting one extra model call per compaction. Most production systems combine both: a pinned running summary plus the last N turns kept verbatim.
Does a larger context window solve context_length_exceeded?: It removes the error but not the cost. Input tokens are billed per token and time-to-first-token grows with prompt size, so a ten-times-larger prompt costs roughly ten times more on every call and responds more slowly. Use a large window for genuinely large documents, not as a substitute for trimming and retrieval.
Do tool and function definitions count toward the context window?: Yes. Tool names, descriptions, and full JSON schemas are serialised into the prompt on every request, and a large tool catalogue can consume thousands of tokens before the user says anything. Trim descriptions, flatten deep schemas, or load tools on demand rather than registering all of them every call.