LLM Observability: What to Measure, Which Tools to Use, and What They Miss
LLM observability is the practice of capturing, tracing, and analyzing everything your application sends to and receives from large language models (prompts, completions, tokens, latency, errors, and cost) so you can explain any output after the fact and catch regressions before users do. It differs from classic APM because LLM failures are semantic: a hallucinated answer returns HTTP 200. And in 2026 the two things most tooling still under-covers are cost accounting and agent behavior: what the model did, not just what it said.
Every team that ships an LLM feature eventually hits the same wall: something went wrong, nobody can say why, and the logs show a wall of 200s. LLM observability (sometimes called LLM monitoring) exists because language models fail differently from ordinary software. This guide covers what to capture, how it differs from APM, the honest state of the tooling market, and where agent-level platforms like Swfte Nexus pick up where call-level tracing stops.
What LLM observability covers: the five pillars
A complete LLM observability practice captures five kinds of signal. Most teams start with traces, add latency dashboards, and only get serious about the other three after an incident or an invoice.
| Pillar | What you capture | Why it matters |
|---|---|---|
| Tokens & cost | Input, output, and cached tokens per request; cost per request, per feature, per user, per model | The only pillar that shows up on an invoice. Runaway loops and bloated prompts burn budgets silently. |
| Latency | Time to first token, total generation time, queue time, retry counts | LLM latency is variable and provider-dependent; p50 tells you almost nothing, p95/p99 tell the story. |
| Quality | Eval scores, user feedback, hallucination and refusal rates, output drift over time | The failure mode users notice. A model or prompt change can degrade quality with zero errors logged. |
| Traces | End-to-end request traces: prompt → retrieval → model call(s) → tool calls → final output | Multi-step chains fail in the middle. Without traces, debugging is guessing. |
| Errors & limits | Rate limits (429s), context-length overflows, timeouts, provider outages, malformed tool calls | LLM APIs fail in LLM-specific ways that generic HTTP monitoring lumps into one bucket. |
The error pillar deserves specific instrumentation: LLM APIs fail in LLM-specific ways. See our troubleshooting guides for the two most common, OpenAI 429 rate limits and context length exceeded.
How LLM observability differs from APM
The instinct to point Datadog or New Relic at an LLM feature is reasonable, and insufficient. APM was designed for deterministic software where failure means an exception. LLM systems break every one of those assumptions.
| Dimension | Classic APM | LLM observability |
|---|---|---|
| Unit of work | Request / transaction | Prompt → completion, often a multi-step chain or agent loop |
| Failure mode | Errors, timeouts, exceptions | Semantic failures: hallucination, drift, silent quality loss, usually HTTP 200 |
| Cost model | Infrastructure cost, roughly fixed per request | Per-token, variable by 100x depending on prompt size, model, and caching |
| Determinism | Same input → same output; reproducible | Non-deterministic; the same prompt can succeed today and fail tomorrow |
| Debugging | Stack traces and logs | Full prompt/completion traces plus evals: the "stack trace" is the conversation |
The LLM observability tooling market
The honest version: the call-level tracing market is mature and well served. These four cover the vast majority of deployments.
Langfuse
Open-source LLM engineering platform: tracing, evals, prompt management. Strong self-host story; the default OSS choice.
LangSmith
LangChain's observability product. Deepest if you are already on LangChain/LangGraph; tightly coupled to that ecosystem.
Helicone
Proxy-based logging and cost tracking: one base-URL change to integrate. Great for visibility fast; lighter on evals.
Datadog LLM Observability
LLM traces inside the APM suite you may already pay for. Convenient consolidation; priced and designed for existing Datadog shops.
Comparing options? See our alternatives breakdowns for Langfuse, LangSmith, and Helicone.
Why cost is the forgotten pillar
Traces get built first because they help engineers debug. Latency gets dashboarded because it is visible to users. Cost gets discovered, usually by whoever opens the provider invoice at the end of the month.
The problem is structural. Token spend is invisible at the point where it is incurred: a prompt that grew from 2K to 20K tokens behaves identically in every test, a retry loop hitting rate limits doubles spend without a single error surfacing to the user, and an agent that re-reads the same files every turn multiplies context silently. Per-request cost can vary by two orders of magnitude with no functional difference in output. Nothing about the developer experience pushes back.
Useful cost observability is therefore attribution, not a total: cost per feature, per user, per repo, per session, per model: with a baseline to compare against, so you can see what context compression or caching actually saved. A single monthly number tells you that you have a problem; attribution tells you where it lives and whether your fix worked.
Where Nexus fits: from observing calls to governing agents
Nexus is an AI-agent governance and observability platform by Swfte. It occupies a layer the call-level tools do not: the tools above observe what a model said: prompt in, completion out. But the fastest-growing LLM workload in 2026 is agents, and agents act. Claude Code runs shell commands, edits files, and installs dependencies. An API-call trace shows none of that.
Nexus combines three things that otherwise require three products: agent-level observability (every session, tool action, file change, dependency install, and per-turn token usage captured via runtime hooks), governance (an inventory of agents and non-human identities, an identity graph with blast-radius analysis, shadow-AI detection), and enforcement (policy evaluated before a tool call executes, so a violating action is denied rather than logged). Prompts are captured as fingerprint and length only in the default privacy tier: no prompt text stored. The capture layer is open-core: one command, nexus wrap claude, instruments a terminal.
If your LLM usage is a product feature calling an API, start with a call-level tracer from the list above. If your organization runs coding agents and autonomous agents, read our companion guide to agent observability : the discipline of observing behavior, not just completions.
LLM observability FAQ
What is LLM observability?
LLM observability is the practice of capturing, tracing, and analyzing everything your application sends to and receives from large language models (prompts, completions, token counts, latency, errors, and cost) so you can explain any output after the fact, debug multi-step failures, and catch quality or cost regressions before users do.
What is the difference between LLM observability and LLM monitoring?
LLM monitoring tracks known metrics against thresholds (error rate, latency, spend) and alerts when they drift. LLM observability is broader: it captures enough raw signal (full traces, prompts, evals) to answer questions you did not know to ask in advance. Monitoring tells you something is wrong; observability lets you find out why. Most teams need both, and most tools ship both under one roof.
What metrics should I track for LLM applications?
Five pillars: tokens and cost (per request, per feature, per user), latency (time to first token, p95/p99 total), quality (eval scores, feedback, hallucination rate), traces (end-to-end chain visibility), and errors (429 rate limits, context-length overflows, timeouts). Cost is the most commonly skipped pillar and the one that shows up on an invoice.
How is LLM observability different from APM?
APM assumes deterministic software where failures are exceptions and timeouts. LLM systems fail semantically (a hallucinated answer returns HTTP 200) and cost varies per token rather than per request. LLM observability adds prompt/completion traces, eval pipelines, and token-level cost accounting that APM tools were never built to model, which is why APM vendors bolted on separate LLM observability products.
What are the best LLM observability tools?
The main options in 2026: Langfuse (open-source tracing and evals, strong self-hosting), LangSmith (deepest for LangChain/LangGraph stacks), Helicone (proxy-based logging and cost tracking, fastest to integrate), and Datadog LLM Observability (for teams already on Datadog). For AI agents (systems that run commands and edit files, not just call APIs) an agent-level layer like Swfte Nexus captures behavior these call-level tools do not see.
How is Nexus different from LLM observability tools like Langfuse or LangSmith?
Langfuse and LangSmith observe API calls: prompt in, completion out. Nexus observes agents: the session, every tool action, every file change, every dependency install, and token usage per turn, and it adds governance, enforcing policy before an action executes rather than only logging it afterward. If your LLM usage is a chat feature, call-level tracing fits; if it is Claude Code and autonomous agents acting on real systems, you need agent-level observability plus enforcement.
Observability that includes the invoice, and the agent
Nexus captures every agent session, tool action, and token spent, attributes cost per terminal, repo, and user, and enforces policy before actions execute. The capture layer is open source.
Open-core CLI · Metadata-only privacy tier by default · nexus wrap claude