Tracing agent runs with OpenTelemetry: the GenAI span names, attributes and privacy defaults

When an agent run goes wrong, the question is almost never "did the model API return 200?" It is "which of the eleven model calls decided to call the refund tool, what did it pass, and why did the third tool call take nine seconds?" Plain HTTP logs cannot answer that. You need a trace that mirrors the shape of the run itself: one parent for the agent invocation, a child for every model call, a child for every tool call, and a link across the wire when a tool lives in another process.

OpenTelemetry, the vendor-neutral standard most tracing backends already accept, has a set of semantic conventions for exactly this. They name the spans, the attributes and the metrics so that a trace from a LangChain agent, a hand-rolled Python loop and an MCP server all line up in the same dashboard. This article walks through what those conventions say today, how to apply them to a loop you wrote yourself, and the one default you should understand before you turn anything on: prompts and tool arguments are not recorded unless you opt in.

Where the conventions live now, and how stable they are

If you bookmarked the GenAI pages on the OpenTelemetry site, you will find a notice on the OpenTelemetry semantic conventions site that they have moved. The conventions are now maintained in their own repository, open-telemetry/semantic-conventions-genai, which covers client inference, embeddings, events, exceptions, metrics, model spans and agent spans, plus provider-specific pages for Anthropic, OpenAI, AWS Bedrock and Azure AI Inference, and separate pages for MCP and the Agent2Agent protocol.

Every one of those documents is marked with the status Development. Under the OpenTelemetry document status rules, that means names can still change between releases. In practice this is a reason to centralize your attribute names in one module rather than sprinkle string literals through your codebase, not a reason to wait. The point of using the conventions at all is that your backend, your teammates' tools and any library instrumentation you add later will agree on what gen_ai.tool.name means.

The span tree for one agent run

The conventions define a small set of span types, and the useful mental model is a tree.

At the top sits the agent invocation. The agent spans document defines two flavors. An agent you call over a remote service, with the OpenAI Assistants API and AWS Bedrock Agents given as examples, gets a span of kind CLIENT. An agent that runs in your own process, with LangChain and CrewAI agents given as examples, gets a span of kind INTERNAL. Either way the operation name is invoke_agent and the span name should be invoke_agent {gen_ai.agent.name}, falling back to plain invoke_agent when the agent has no name. The same document defines invoke_workflow for multi-agent workflows, create_agent for provisioning one, and a plan span for a distinct planning step.

Under the agent span you get one span per model call. The client inference document names it {gen_ai.operation.name} {gen_ai.request.model}, so a chat completion against a model shows up as something like chat gpt-4 (the spec's own example model) in your trace view. Its kind should be CLIENT when the model runs in another process or over HTTP, which is nearly always the case for hosted APIs. The required attribute is gen_ai.provider.name, with examples such as openai and gcp.vertex_ai, and the recommended ones include gen_ai.request.model, gen_ai.response.model, gen_ai.response.finish_reasons, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. There are also recommended attributes for cache reads, cache writes and reasoning tokens, each to be set when the provider reports a non-zero value.

Next to the model calls sit the tool calls. The model spans document defines the execute tool span with operation name execute_tool, span name execute_tool {gen_ai.tool.name}, and kind INTERNAL. Two attributes are required, the operation name and the tool name. gen_ai.tool.call.id is required if and only if the model supplied one, which ties the span back to the specific tool call the model emitted. gen_ai.tool.type is recommended and distinguishes function tools that your client executes, extension tools the agent runs to call external APIs, and datastore tools used for retrieval.

The document says something that is easy to miss: tools are often executed directly by application code, and developers are encouraged to instrument the tool calls that automatic instrumentation does not cover. If your agent framework traces model calls for you but your tools are plain functions, nobody is going to create those execute_tool spans except you.

Instrumenting a hand-written loop

Here is what that looks like in a Python loop using the standard OpenTelemetry Python API. The model call itself is left abstract, since the shape of the spans is the point.

from opentelemetry import trace
from opentelemetry.trace import SpanKind, Status, StatusCode

tracer = trace.get_tracer("support-agent")

def run_agent(user_message):
    with tracer.start_as_current_span(
        "invoke_agent support-agent",
        kind=SpanKind.INTERNAL,
        attributes={"gen_ai.operation.name": "invoke_agent",
                    "gen_ai.agent.name": "support-agent"},
    ):
        for turn in range(MAX_TURNS):
            with tracer.start_as_current_span(
                "chat example-model", kind=SpanKind.CLIENT,
                attributes={"gen_ai.operation.name": "chat",
                            "gen_ai.provider.name": "anthropic",
                            "gen_ai.request.model": "example-model"},
            ) as span:
                reply = call_model(history)
                span.set_attribute("gen_ai.usage.input_tokens", reply.input_tokens)
                span.set_attribute("gen_ai.usage.output_tokens", reply.output_tokens)
            if not reply.tool_calls:
                return reply.text
            for call in reply.tool_calls:
                run_tool(call)

def run_tool(call):
    with tracer.start_as_current_span(
        f"execute_tool {call.name}", kind=SpanKind.INTERNAL,
        attributes={"gen_ai.operation.name": "execute_tool",
                    "gen_ai.tool.name": call.name,
                    "gen_ai.tool.call.id": call.id,
                    "gen_ai.tool.type": "function"},
    ) as span:
        try:
            return TOOLS[call.name](**call.arguments)
        except Exception as exc:
            span.set_attribute("error.type", type(exc).__name__)
            span.set_status(Status(StatusCode.ERROR))
            raise

Notice that every naming attribute is passed when the span starts rather than set afterward. The spans document lists gen_ai.agent.name, gen_ai.operation.name, gen_ai.tool.name and gen_ai.tool.type as attributes that matter for sampling decisions and says they should be provided at span creation time. A sampler cannot keep "all execute_tool issue_refund spans" if it never sees the tool name.

The error handling follows the same document, which makes error.type conditionally required when the operation ended in an error and asks for a low-cardinality value such as an exception class name or a provider error code. That pairs well with the retry and stop rules from our piece on handling tool errors in agent loops: once each failed tool call is a span with an error type, you can count which tools fail and how often a retry actually rescues the run.

The privacy default: no prompts, no arguments, no results

The most important section of the spec is about what not to record. The spans document states that model instructions, user messages and model outputs are sensitive and often large, and that instrumentations should not capture them by default but should offer an opt-in. The opt-in covers more than chat messages. It governs gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions, gen_ai.tool.call.arguments, gen_ai.tool.call.result, retrieval queries, prompt template variables and memory records.

When an instrumentation library exposes that switch through an environment variable, the spec requires it to be named OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT, with the values NO_CONTENT (the default), SPAN_ONLY, EVENT_ONLY and SPAN_AND_EVENT. An unrecognized value should fall back to NO_CONTENT. Tool definitions are treated differently: gen_ai.tool.definitions describes schemas rather than user content, so it is not governed by this switch.

The spec lays out three usage patterns. The default is to record no content at all. The second is to record content directly on spans, which it describes as best suited to situations where telemetry volume is manageable and privacy rules either do not apply or are met by the storage, such as pre-production. The third, recommended for production, is to upload content to separate storage and record only references on the span, using attributes like gen_ai.input.messages_ref. Instrumentations that support this are asked to read the hook from OTEL_INSTRUMENTATION_GENAI_CONTENT_HOOK and a destination from OTEL_INSTRUMENTATION_GENAI_UPLOAD_BASE_PATH, which may be a local directory or a cloud storage URI such as s3://my-bucket/prompts.

For an agent builder the practical rule is simple. Leave content capture off in production unless you have a reason and a storage plan. Turn on SPAN_ONLY in a staging environment where you are debugging prompts. If you need prompts in production to investigate incidents, use the reference pattern so the traces stay small and the sensitive text sits behind its own access controls. Tool results deserve the same caution as user messages, because, as we covered in our article on prompt injection in tool results, they are often untrusted text from the outside world.

Following a call into an MCP server

Traces are most useful when they cross process boundaries, and agent tools increasingly live in MCP servers. The MCP conventions make a point that surprises people: HTTP trace propagation only covers the HTTP request, not the individual MCP messages inside it. One MCP request can span several HTTP requests because of retries, and one Streamable HTTP request can carry more than one MCP message.

So the conventions ask instrumentations to inject the trace context into the request's params._meta field, using the unprefixed W3C Trace Context keys traceparent and tracestate, plus baggage if you use it. The server extracts that context and uses it as the parent of its own span. A tools/call request then carries something like "_meta": {"traceparent": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"}, and the server-side work appears under the client's tool call in the same trace.

MCP spans are named {mcp.method.name} {target}, with kind CLIENT on the caller and SERVER on the server. When the method is a tool call, gen_ai.operation.name should be execute_tool and gen_ai.tool.name should be set, which the document describes as compatible with the GenAI execute tool span. If you run your own MCP server, adding this extraction step is a small change that turns a black box into a visible branch of every agent trace that touches it.

Metrics that answer the budget questions

Spans tell you about one run. The GenAI metrics document defines histograms for the fleet view. gen_ai.invoke_agent.duration measures whole agent invocations. gen_ai.invoke_agent.inference_calls and gen_ai.invoke_agent.tool_calls count how many model calls and tool calls a single invocation made, with suggested bucket boundaries of 1, 2, 4, 8, 16, 32, 64 and 128. gen_ai.execute_tool.duration measures tool latency. The document notes that inference calls made by sub-agents are counted against those sub-agents' own invocations, so each call is counted exactly once across the tree.

Those two count histograms are the cheapest early warning you can get for a runaway loop. If the median run makes four model calls and the 99th percentile suddenly makes sixty-four, something changed, whether a prompt, a tool's error behavior or a model version. Pair the histogram with the turn limits and token budgets from our post on budget caps for agent runs, and you can both stop a bad run and see that the pattern is spreading.

A sensible order of work

If you are starting from nothing, add the agent span and the model call spans first, with token usage attributes, because that alone shows you where time and tokens go. Add execute_tool spans next, with the tool call ID so each span matches the model's request. Then propagate context into any MCP servers you own. Leave content capture off until you have decided where prompts are allowed to live, and keep your attribute names in one place so the next revision of a Development-status spec is a one-file change rather than a search through your codebase.

Observability for agent fleets is the same production discipline as any ML system, only with more chatty spans. Designing Machine Learning Systems by Chip Huyen is still the clearest book-length framing of monitoring, data contracts and operational feedback loops that apply when agents call tools in production.

If you run models on your own machine, we keep a short list of local LLM gear — GPUs, Macs, RAM and storage — with plain-text Amazon links.