Catalog My learning Blog
Build AI Agents with MCP: 30-Day Course · Phase 6 · Lesson 28 · 5 min read

AI Agent Observability and Cost: OpenTelemetry

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: Tracing and Costing Each Model Call
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Instrument an agent so you can answer what it did, why, how long it took and what it cost: trace model and tool calls with OpenTelemetry, account for tokens including cache reads, attribute cost per task, and apply the main levers for reducing cost and latency.

Concept Explanation

An agent in production is a distributed system whose control flow is decided at run time by a model. When a user reports a bad answer, you need the full trajectory: every model call with its inputs and outputs, every tool call with arguments, results and errors, and the timing of each. Logs scattered across services are not enough; you need a trace that ties one task together.

Distributed tracing fits naturally. A task is a trace; each model call and each tool call is a span within it, with attributes such as model name, token counts, tool name, error status and latency. OpenTelemetry's generative AI semantic conventions define standard attribute names such as gen_ai.request.model and gen_ai.usage.input_tokens, so traces from different frameworks look alike in tools such as Jaeger, Honeycomb, Datadog or LLM-specific platforms such as Langfuse and LangSmith. MCP reserves the traceparent key in request metadata, so a trace can continue from the agent into an MCP server and on into the databases it queries.

Cost needs its own accounting because agents resend the conversation on every step. The Claude API reports usage per response: input tokens, output tokens, and tokens written to and read from the prompt cache. Cache reads are billed at about a tenth of the normal input price, and caching is the single largest cost lever for agents, because the system prompt, tool definitions and earlier turns form a stable prefix that is reused every step. Keep that prefix stable: no timestamps in the system prompt, a deterministic tool order, and append-only history. Watch cache_read_input_tokens; if it stays at zero, something is invalidating the cache.

Measure cost per completed task, not per request. A cheaper model that needs twice the steps or fails more often is not cheaper. Other levers, roughly in order of how often they help, are trimming tool results, lowering the effort setting for simple routes, routing easy requests to smaller models, batching non-interactive work at half price, and capping steps and budgets so runaway loops stop early.

Latency has its own levers. Run independent tool calls in parallel. Stream output so users see progress. Use fast models for classification and routing steps. Report progress for long tool calls. And watch the tail: the slowest five percent of tasks often come from loops or slow tools that a trace makes obvious.

SignalSourceUse
Trace per task, span per model or tool callOpenTelemetry instrumentationDebug a specific bad run end to end
input, output, cache write and cache read tokensusage on each API responseCost per step and cache health
Cost per completed taskToken counts times prices, divided by successesCompare designs honestly
Steps per taskCount of model callsDetect loops and inefficiency
Tool latency and error rateTool spansFind slow or failing tools
Cache hit ratiocache reads over total inputSpot prompt prefix instability

Worked Example: Tracing and Costing Each Model Call

This wrapper creates an OpenTelemetry span for every model call with standard generative AI attributes, and accumulates cost from the usage block. Prices are per million tokens for Claude Opus 5.5 in October 2026; keep them in configuration, not code.

Python
from opentelemetry import trace

tracer = trace.get_tracer("support-agent")
PRICE = {"input": 4.00, "output": 20.00, "cache_write": 5.00, "cache_read": 0.20}   # USD per 1M

class CostMeter:
    def __init__(self):
        self.usd = 0.0
        self.cache_read = self.input_total = 0

    def add(self, usage) -> float:
        cw = usage.cache_creation_input_tokens or 0
        cr = usage.cache_read_input_tokens or 0
        cost = (usage.input_tokens * PRICE["input"] + usage.output_tokens * PRICE["output"]
                + cw * PRICE["cache_write"] + cr * PRICE["cache_read"]) / 1_000_000
        self.usd += cost
        self.cache_read += cr
        self.input_total += usage.input_tokens + cw + cr
        return cost

def traced_call(client, meter: CostMeter, **request):
    with tracer.start_as_current_span("chat " + request["model"]) as span:
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.request.model", request["model"])
        response = client.messages.create(**request)
        u = response.usage
        span.set_attribute("gen_ai.usage.input_tokens", u.input_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", u.output_tokens)
        span.set_attribute("gen_ai.response.finish_reasons", [response.stop_reason])
        span.set_attribute("app.cost_usd", meter.add(u))
        return response

# After a task: log meter.usd and meter.cache_read / meter.input_total (the cache hit ratio).
# A ratio near zero on a 20-step task means the prompt prefix is changing between calls.

Wrap tool calls the same way with an execute_tool span carrying the tool name. In a trace viewer, one task then reads as a timeline: model call, three parallel tool spans, model call, final answer, with cost on every model span.

Each task is a trace and each model or tool call is a span with model, token, cost and error attributes. Parallel tool spans overlap, cache reads show whether the prompt prefix is stable, and the total gives cost per task.
Each task is a trace and each model or tool call is a span with model, token, cost and error attributes. Parallel tool spans overlap, cache reads show whether the prompt prefix is stable, and the total gives cost per task.

Real-World Examples

  • OpenTelemetry's generative AI semantic conventions define attributes such as gen_ai.request.model and gen_ai.usage.input_tokens, and are supported by vendors including Datadog, Honeycomb and Grafana as well as LLM observability tools such as Langfuse.
  • Anthropic's prompt caching documentation prices cache reads at a small fraction of base input cost, which is why long-running coding agents such as Claude Code are designed to keep a stable, append-only prompt prefix and achieve high cache hit rates.
Key TakeawaysTrace every task with a span per model and tool call using OpenTelemetry's generative AI conventions, and propagate trace context into MCP servers. Account for input, output, cache write and cache read tokens, and judge designs by cost per completed task. Protect the prompt cache with a stable prefix, then trim results, tune effort, route easy work to smaller models and cap runaway loops. Parallelize tools and stream output for latency.

Review Questions & Answers

A trace ties every model call and tool call of one task together in order, with timing, inputs, outputs and errors, so you can follow the exact trajectory that led to a bad result.

Agents resend the system prompt, tools and history on every step. If that prefix is stable it is served from cache at a fraction of the input price, which cuts cost and latency across many steps.

Something is changing the prompt prefix between calls, such as a timestamp in the system prompt, a tool list in a different order, or history being rewritten, so the cache is invalidated every time.

A design with cheaper requests may need more steps or fail more often. Only cost per successful task shows which design is actually cheaper for the outcome you need.