Capturing structured telemetry for LLM apps — prompts, completions, token counts, latency, cost, tool calls, and the full nested trace of an agent's steps — so you can debug, monitor, and evaluate real behavior.
How it works
When you make a call to a language model, each step in that process is captured as a Span in a distributed trace. This means that for every prompt sent and every completion received, as well as every interaction with external tools, a span is created. The OpenTelemetry GenAI semantic conventions standardize what attributes get logged, such as gen_ai.request.model, token counts, and operation names. This standardization ensures that any backend that supports OpenTelemetry, like Langfuse or Datadog, can interpret and visualize these spans correctly. For example, if a model call takes longer than expected, you can trace back through the spans to identify where the slowdown occurred, whether it was in the model processing or in an external tool call.
These spans are linked together in a way that forms a complete picture of the execution flow, allowing you to see not just individual calls but also how they relate to one another. For instance, if an agent makes multiple tool calls as part of a single user request, each of those calls will be represented as a span that branches off from the main request span. This hierarchical structure is crucial for understanding complex interactions in your application. You can filter these traces based on different criteria, such as low scoring or high latency runs, which helps you focus on the most problematic areas for further investigation.
By capturing structured telemetry data, you can monitor the performance of your language model applications in real time. You can track metrics like latency and cost alongside the actual prompts and completions. This combination allows for a comprehensive evaluation of your model's behavior and efficiency. For example, if you notice that a particular prompt consistently leads to high costs, you can investigate the associated spans to find out why, whether it's due to excessive token usage or inefficient tool calls.
- Capture user prompts as spans.
- Log model and tool calls as spans.
- Collect telemetry data for analysis.
- Send data to monitoring tools.
- Standardize on OpenTelemetry conventions.
The software-engineering analogue
In traditional software engineering, you often use Application Performance Monitoring (APM) tools and distributed tracing systems like Jaeger or Datadog to monitor microservices. These tools track requests as they flow through various services, capturing metrics like response times and error rates. In this context, you can think of each service call as a span in a distributed trace, providing insights into performance bottlenecks.
However, the analogy breaks down when you consider the nature of the payloads in GenAI applications. Instead of standard HTTP requests and responses, you are dealing with prompts and model outputs. The key metrics shift from simple response times to more complex measures like token counts and quality scores. A single user request can trigger a cascade of model and tool calls, creating a deep tree of interactions that you need to reconstruct for effective debugging.
Why not the obvious alternative
Many newcomers might first think of using plain logs to track their model interactions. However, this approach quickly falls short when you need to diagnose complex issues. For instance, if a model produces a poor response, plain logs won't tell you which version of the prompt was used, which documents were retrieved, or which tool calls were made during that interaction. Without this information, you can't pinpoint the root cause of the failure.
Structured GenAI tracing addresses these shortcomings by providing a detailed view of the entire execution flow. It captures the relationships between prompts, completions, and tool calls, making it possible to diagnose non-deterministic failures that occur across multiple hops. This level of detail is essential for understanding why a particular response was generated and what factors contributed to its cost and quality.
When to use which
Whenever you have an agent, a retrieval-augmented generation (RAG) system, or any multi-step chain in production, implementing tracing is essential. It's not just a nice-to-have; it's a fundamental requirement for effective monitoring and debugging. By standardizing on OpenTelemetry GenAI conventions, you ensure that your telemetry data is portable across different vendors, avoiding lock-in with a single tool's SDK. It's crucial to redact any personally identifiable information (PII) in the captured payloads to maintain privacy and compliance.
In production: when it breaks
Why it happens. One common reason for increased latency in LLM applications is the lack of proper tracing and observability, which can lead to bottlenecks in processing. If certain segments of the processing pipeline are not instrumented, you may not be able to identify where delays are occurring, resulting in slow response times.