Etiket: Langfuse

Agent Observability Is the Next Observability


Why logs, metrics, and traces are no longer enough for production AI.

TL;DR: Modern AI agents are not just making API calls—they are making decisions. Traditional observability tells us whether a system is healthy. Agent observability tells us why an agent behaved the way it did. That distinction will define the next generation of production AI.

We Solved Infrastructure Observability. AI Introduces a New Problem.

Over the last decade, software engineering has become incredibly good at observing distributed systems.

We have mature tools for:

  • Logs
  • Metrics
  • Distributed traces
  • Profiling
  • Alerting
  • Service maps

Platforms such as OpenTelemetry, Prometheus, Grafana, Datadog, and Jaeger have made production debugging dramatically easier.

When an API slows down, we can usually answer questions like:

  • Which service caused the latency?
  • Which database query was expensive?
  • Which deployment introduced the regression?
  • Which dependency failed?

For cloud-native applications, these questions are routine.

But AI agents change the nature of the problem.

Infrastructure can be perfectly healthy while the agent produces a completely incorrect result.

That is a fundamentally different debugging challenge.

AI Agents Behave More Like Employees Than APIs

Traditional software follows deterministic code paths.

AI agents don’t.

Instead, they continuously make decisions.

For example:

Receive task
        │
        ▼
Plan execution
        │
        ▼
Search documentation
        │
        ▼
Read memory
        │
        ▼
Choose tools
        │
        ▼
Call GitHub
        │
        ▼
Call MCP Server
        │
        ▼
Query database
        │
        ▼
Generate answer
        │
        ▼
Self-evaluate
        │
        ▼
Return response

Every arrow represents a decision.

Every decision can influence the final outcome.

This means debugging an AI agent is much closer to investigating a workflow than diagnosing a REST API.

Traditional Observability Doesn’t Explain Behavior

Imagine this production log:

POST /chat

Status: 200
Latency: 7.2 seconds

Everything appears normal.

But what actually happened?

Perhaps the agent:

  • Retrieved outdated documentation.
  • Selected the wrong MCP tool.
  • Called GitHub four times.
  • Ignored the highest-confidence search result.
  • Consumed 250,000 tokens.
  • Produced an incorrect answer anyway.

None of this appears in traditional monitoring.

The infrastructure succeeded.

The agent failed.

The Questions Production Engineers Actually Ask

When debugging production AI, engineers rarely ask:

“Was the CPU overloaded?”

Instead, they ask:

  • Why was this tool selected?
  • Why was this tool selected three times?
  • Which retrieved document influenced the answer?
  • Which prompt version was used?
  • Which memory entry changed the decision?
  • Why did identical requests produce different outputs?
  • Which context caused hallucination?
  • Why did token cost suddenly double?

Notice something interesting.

These are not infrastructure questions.

They are behavioral questions.

The Four Pillars of Agent Observability

I believe future observability platforms will revolve around four primary dimensions.

1. Execution Observability

This is the closest equivalent to distributed tracing.

Instead of following network calls, we follow decisions.

Example:

User
 │
 ▼
Planner
 │
 ▼
Retriever
 │
 ▼
Memory
 │
 ▼
GitHub
 │
 ▼
Filesystem
 │
 ▼
MCP
 │
 ▼
LLM
 │
 ▼
Verifier
 │
 ▼
Response

Each node represents a reasoning step rather than simply a service.

2. Context Observability

Context is rapidly becoming one of the most valuable assets inside an AI application.

Questions include:

  • Which documents entered the context window?
  • Which documents were discarded?
  • Which memories were loaded?
  • Which MCP resources were exposed?
  • Which retrieval strategy was used?

Poor context often explains poor answers.

3. Decision Observability

Understanding what happened is useful.

Understanding why it happened is even more valuable.

Useful telemetry might include:

  • Tool selection
  • Confidence scores
  • Retry decisions
  • Permission checks
  • Planning steps
  • Verification outcomes

This transforms debugging from reading logs into understanding behavior.

4. Cost Observability

AI systems introduce an entirely new production metric.

Cost.

Every execution now has measurable resource consumption.

Useful metrics include:

  • Prompt tokens
  • Completion tokens
  • Cost per request
  • Cost per tool
  • Cost per workflow
  • Cost per successful outcome

Organizations will increasingly optimize agents not only for accuracy but also for efficiency.

MCP Makes Observability Even More Important

The Model Context Protocol (MCP) expands what agents can access by standardizing connections to tools, data sources, and external services.

Instead of isolated LLM calls, agents can interact with GitHub, databases, documentation systems, cloud services, and internal business platforms through a common interface.

That flexibility is powerful—but it also increases operational complexity.

An agent may execute multiple MCP tool calls before producing a response.

Questions quickly emerge:

  • Which MCP server was used?
  • Which tool was selected?
  • Which parameters were passed?
  • How long did each tool take?
  • Which call failed?
  • Which permissions were granted?
  • Was the returned data actually used?

As MCP adoption grows, execution traces will need to include tool interactions as first-class citizens rather than treating them as opaque external calls.

Existing Tools Are Building the Foundation

The ecosystem is already moving toward this future.

OpenTelemetry

OpenTelemetry standardized telemetry for cloud-native software.

It gave us a common language for logs, metrics, and distributed traces.

Many emerging AI observability platforms are extending these concepts rather than replacing them.

The opportunity is to enrich traces with AI-specific events such as tool invocations, prompt versions, token usage, and retrieval metadata.

Langfuse

Langfuse focuses on LLM tracing, prompt management, evaluations, and production monitoring.

It provides visibility into:

  • Prompt execution
  • Token usage
  • Costs
  • Latency
  • User feedback
  • Evaluations

For many teams, it serves as an operational dashboard for AI applications.

LangSmith

LangSmith emphasizes debugging and evaluating LLM-powered applications.

It helps developers inspect execution flows, compare runs, evaluate prompts, and identify regressions.

This is especially valuable during iterative development and testing.

OpenLIT

OpenLIT brings OpenTelemetry concepts into AI workloads.

Rather than creating an entirely separate observability stack, it extends familiar telemetry patterns to LLMs, vector databases, and AI frameworks.

This approach is attractive for organizations already invested in OpenTelemetry.

Arize Phoenix

Arize Phoenix focuses on evaluating and diagnosing AI systems.

Its strengths include:

  • Retrieval analysis
  • Hallucination investigation
  • Embedding visualization
  • Prompt inspection
  • LLM evaluations

Instead of monitoring infrastructure health, it helps teams understand model behavior and output quality.

The Next Evolution: Agent Replay

Perhaps the most valuable capability isn’t another dashboard.

It’s replay.

Imagine opening a production incident and replaying an entire agent execution:

  • Every retrieved document
  • Every MCP tool invocation
  • Every prompt
  • Every model response
  • Every retry
  • Every permission decision
  • Every context update

Instead of reading thousands of log lines, engineers could simply watch the execution unfold.

Distributed tracing transformed backend debugging.

Execution replay could do the same for AI agents.

What an Ideal Agent Trace Might Look Like

User Request
        │
        ▼
Planning
        │
        ▼
Memory Lookup
        │
        ▼
Vector Search
        │
        ▼
MCP GitHub Tool
        │
        ▼
Repository Analysis
        │
        ▼
LLM Reasoning
        │
        ▼
Verification
        │
        ▼
Response

Clicking any node would reveal:

  • Execution time
  • Tokens
  • Cost
  • Context
  • Prompt version
  • Tool parameters
  • Output
  • Evaluation results

This level of visibility would make AI systems significantly easier to operate and improve.

Why This Matters for Software Engineering

The software industry has spent years building reliable systems through observability.

AI agents introduce a new abstraction layer where the primary source of failure is no longer infrastructure—it is behavior.

Healthy services can still produce poor outcomes if the agent chooses the wrong tool, retrieves irrelevant context, or follows an ineffective execution path.

That means the next generation of engineering practices must evolve beyond monitoring systems to understanding autonomous workflows.

Observability is no longer just about uptime.

It is about explainability, accountability, and operational confidence.

Final Thoughts

Cloud-native engineering gave us logs, metrics, and traces.

Production AI will require something more.

As AI agents become responsible for software development, customer support, security investigations, business workflows, and autonomous operations, understanding how they reached a decision will become just as important as knowing what they returned.

The teams that succeed won’t simply build more capable agents.

They will build agents that can be observed, replayed, evaluated, and trusted.

Because in production, intelligence without observability is just another black box.

Key Takeaways

  • Traditional observability explains system health; agent observability explains agent behavior.
  • AI agents require visibility into decisions, context, tool usage, and execution paths.
  • MCP increases the need for rich tracing because agents interact with many external tools and services.
  • Emerging platforms such as OpenTelemetry, Langfuse, LangSmith, OpenLIT, and Arize Phoenix are laying the groundwork for AI-native observability.
  • The next major evolution is likely to be agent replay, enabling engineers to reconstruct and analyze complete autonomous workflows rather than relying solely on logs.