Agent Observability: Monitoring AI Agents That Act, Not Just Answer

LLM observability answers questions about model calls: what was the prompt, what came back, how long did it take, what did it cost, was the answer good. That framing fits chat features and RAG pipelines, where the model's output is the product.

Agents break the framing because agents act. A Claude Code session does not end at a completion. It runs shell commands, edits thirty files, installs a dependency, connects to MCP servers, and commits. Every one of those is an action on a real system, performed with the developer's own credentials, and none of them appears in an API-call trace. The worst LLM failure is a wrong answer; the worst agent failure is a wrong action: a deleted directory, a leaked secret, a poisoned dependency.

So the unit of observation shifts. Call-level tools observe the conversation. Agent-level observability observes behavior: the session as the trace, the tool call as the span, and the file system, package registry, and credential set as the blast zone. If you monitor agents with an LLM-call tracer, you see the thinking and miss the doing.

Here is the uncomfortable part of the agentic-observability wave: a dashboard cannot stop anything. Agents act at machine speed, unattended. By the time a trace shows a write to .env or a curl | sh, the action already ran. For agents, purely retrospective observability is forensics with better charts.

The step beyond is enforcement: evaluating policy before the tool call executes. Nexus performs this check in-flight, the same hook that records a tool_action event first evaluates it against policy, and a violating action (a destructive command, a write to a protected path, an install from an unapproved registry) is denied with a stated reason before it runs. The denial itself is captured with a blocked flag, so the audit trail includes what was attempted and why it was stopped. Policy checks are local and synchronous (a network hiccup cannot bypass them) while capture is fail-open, so a down collector never blocks legitimate work.

This is the line between agent observability and agent governance: observability tells you what happened; governance also decides what is allowed to happen. For the concrete case of securing Claude Code specifically, see Claude Code security.

Every agent is a non-human identity: it authenticates, holds credentials, and takes actions, usually with no owner, no lifecycle, and no offboarding trigger. Observing individual sessions is necessary but not sufficient; you also need the aggregate view of who these actors are and what they can reach.

The Nexus console maintains an inventory of agents and non-human identities and connects them into an identity graph. The graph exists to answer the one question that matters in an incident: if this agent's credential were compromised right now, what could an attacker reach through it? That blast-radius analysis turns a vague worry about "AI agents with access" into a scoped list of systems, repos, and connections: the difference between an afternoon of incident response and a week of it. The same connection data powers the MCP picture: which agents talk to which servers, from which terminals, the visibility gap covered in MCP security best practices.

Shadow AI used to mean personal ChatGPT accounts. In developer organizations it now means shadow agents: a coding agent installed in a terminal, an MCP server added to a local config, an unsanctioned model provider: none of which touch SaaS logs, OAuth grants, or SSO, and all of which are invisible to SaaS-side security tooling by construction.

Because Nexus captures at the endpoint (the terminal where agents actually run) unmanaged agents, unsanctioned providers, and unknown non-human identities surface as findings in the console rather than remaining unknowable. Detection is the entry point; the paved path is governance attached to sanctioned tools, so the governed route is also the easiest one.

Agents consume tokens differently from chat: long sessions, repeated context, tool output flowing back into the window, retries. A single runaway session can spend more than a month of a chat feature, and a provider invoice attributes none of it.

Nexus meters token usage per turn (input, output, baseline versus compressed) and attributes cost per terminal, per repo, per user. Because it records a baseline alongside actuals, it can show what context compression and output filtering saved rather than asserting it. Teams use the same events to find runaway sessions, enforce budgets, and answer the question finance actually asks: not "what did AI cost" but "which team, which repo, which workflow."