Agent Observability / Tracing Sprint

What It Is

Agent observability is the habit of treating an agent run as a workflow you can inspect, not just a blob of prompt text and final output. In this Scout Loop, you will create one tiny Python agent with a tool, run it inside a named trace, and inspect whether the trace shows the workflow shape you expected.

The concrete thing you will touch is the OpenAI Agents SDK tracing path: a local Python script, an agent run, a function-tool span, and the exported trace. If you already have a Logfire account ready, you can add a quick comparison at the end; if not, keep the sprint focused on the default trace first.

Why It Matters

For real agent work, traces answer practical questions faster than reading logs: Which model call happened? Did the tool run? Was the failure in orchestration, the model response, or an external dependency? That matters when you are evaluating whether an agent workflow is demo-ready, client-safe, or worth turning into reusable consulting infrastructure.

The useful decision in this loop is not whether tracing exists. It is whether the default trace gives you enough operational signal with little setup, and whether a second backend such as Logfire or a future OpenTelemetry pipeline is worth your next block of time.

Guided First Play

You are going to build the smallest traceable agent workflow: one agent, one tool, one named trace, one privacy setting, and one verification pass. If you do not have an OpenAI API key available, treat this as an access check: confirm Python and package installation, then stop before the run step.

Create a Clean Scratch Space

In a terminal, make a temporary folder and install only the Agents SDK. The SDK requires Python 3.10 or newer, so check that first instead of debugging package errors later.

python3 --version
mkdir -p agent-trace-scout
cd agent-trace-scout
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip openai-agents

Set your API key for this shell. Also turn off sensitive trace payload capture for the first run. The tracing docs note that generation inputs and outputs and function inputs and outputs can be captured by default, so start with the privacy-safe posture and relax it only when you need payload-level debugging.

export OPENAI_API_KEY="paste-your-key-here"
export OPENAI_AGENTS_TRACE_INCLUDE_SENSITIVE_DATA=false

Write One Traceable Agent

Create trace_scout.py. The tool is intentionally boring: it makes the trace easy to read because there should be one agent span, one generation span, and one function-tool span.

from agents import Agent, Runner, function_tool, trace

try:
    from agents.tracing import flush_traces
except Exception:
    flush_traces = None


@function_tool
def observability_note(workflow: str) -> str:
    return (
        f"For {workflow}, check tool latency, model retries, "
        "sensitive fields, and whether failures are tied to a specific span."
    )


agent = Agent(
    name="trace-scout",
    instructions=(
        "You are a concise observability reviewer. Use the tool once, then return "
        "three bullets: expected spans, likely sensitive fields, and one useful metric."
    ),
    tools=[observability_note],
)

with trace("agent-observability-scout", group_id="scout-loop"):
    result = Runner.run_sync(
        agent,
        "Review tracing needs for a two-step customer support triage agent.",
    )

print(result.final_output)

if flush_traces:
    flush_traces()

Run it once. If the script exits cleanly but no trace appears immediately, wait briefly and run it again; short scripts can finish before buffered traces are visible, which is why the example tries to flush traces when that helper is available.

python trace_scout.py

Read the Trace Like an Operator

Inspect the trace with three questions in mind. First, does the nesting match the mental model of the workflow? Second, can you see enough timing and error context to debug a slow or failing run? Third, did the privacy setting hide the payload details you would not want casually shared in a client demo?

• Expected shape: one trace for agent-observability-scout, with agent execution, generation, and function spans nested under it.
• Useful fields to notice: agent name, operation names, model call timing, tool span timing, and any error type if something failed.
• Privacy check: with OPENAI_AGENTS_TRACE_INCLUDE_SENSITIVE_DATA=false, confirm you are not relying on full prompt or function payload capture for basic debugging.
• Standards check: map what you see to the OpenTelemetry GenAI vocabulary: provider, operation, agent name, model, and error type are the kinds of fields that make traces portable across tools.

Try One Focused Variation

Change only the user request and run the script again. Keep the same trace name and group id so you can compare related runs as a tiny workflow family rather than isolated calls.

result = Runner.run_sync(
    agent,
    "Review tracing needs for a retrieval agent that calls a vector database and a billing API.",
)

This comparison is the real scout evidence. If two small runs already show you where an agent spent time, whether a tool ran, and what you would redact before sharing, default tracing is useful enough to keep in your agent template.

Optional Backend Taste Test

If you already have Logfire access ready, add it as a quick second view. The documented Agents setup is logfire.configure() plus logfire.instrument_openai_agents(). Do not add an OpenTelemetry Collector in this Scout Loop; the Logfire docs frame the Collector as useful for centralized configuration, multiple backends, transformation, enrichment, or existing telemetry sources, which is production-extension territory.

python -m pip install logfire

import logfire

logfire.configure()
logfire.instrument_openai_agents()

# Put those lines near the top of trace_scout.py, after the imports.

End the loop by writing one sentence for yourself: For my next agent demo, tracing is worth keeping because... If you cannot finish that sentence from the evidence you saw, the topic may be interesting but not immediately useful.

Your Next Move

Choose based on the evidence, not the buzzword. Did the trace help you understand an agent run faster than terminal logs would have?

Go Deeper would turn this into a two-hour comparison: default OpenAI traces, Logfire instrumentation, one failing tool, one latency measurement, and a short OpenTelemetry field checklist. Project would turn it into a reusable agent-observability starter kit for demos or client prototypes, with privacy defaults and trace review prompts baked in.

Sources

This loop is grounded in official OpenAI Agents SDK tracing documentation and repository notes, Pydantic Logfire integration docs, and OpenTelemetry GenAI semantic convention docs. The main implementation claims come from the Agents SDK tracing behavior, the Logfire Agents instrumentation setup, and the GenAI span vocabulary for agent and model-call observability.