Distributed tracing was developed for microservices: systems where a single user request might touch ten different services, each with its own logs, and the challenge is stitching together a coherent picture of what happened across all of them. The span model -- a named, time-bounded unit of work with a parent reference -- was the solution.
LLM pipelines are a different kind of distributed system, but the core problem is the same: a single user request touches multiple components (query parsing, retrieval, context assembly, generation, validation), each with its own latency, cost, and failure mode, and the challenge is understanding the whole without losing sight of the parts.
Adapting the OTel span model to LLM pipelines was the central engineering challenge in building Spanloom. This is what we learned.
The naming problem
In service tracing, span names are usually service names or method names: user-service.get_profile, orders-db.query. In LLM pipelines, the natural naming is by operation type, but the variety of operations is wider and less standardized than traditional service calls.
We converged on a two-level naming scheme: {operation_type}.{model_or_resource}. For example: llm.gpt-4o, embed.text-embedding-3-small, retrieve.pinecone, rerank.cohere-rerank. For custom application-level steps without a model: pipeline.query_rewrite, pipeline.context_assemble, pipeline.validate_output.
The two-level naming gives you useful aggregation options. You can group all llm.* spans to get total LLM cost and latency, or filter to llm.gpt-4o specifically to see that model's performance. This is the same kind of cardinality management that makes distributed tracing dashboards useful in service architectures.
What attributes an LLM span must carry
An HTTP span in traditional service tracing carries a small set of well-known attributes: URL, method, status code, latency. An LLM span needs a richer attribute set to be useful. Through iteration, we arrived at the following as the minimum required set for a call span:
gen_ai.system-- the provider (openai, anthropic, aws.bedrock, cohere)gen_ai.request.model-- the specific model identifiergen_ai.usage.input_tokens-- prompt tokens consumedgen_ai.usage.output_tokens-- completion tokens generatedspanloom.cost.usd-- computed cost in USD at time of callspanloom.prompt.version-- a hash or tag identifying the prompt template versiongen_ai.operation.name-- chat, completion, embed, rerank
For retrieval spans specifically:
spanloom.retrieval.query-- the query sent to the retrieval systemspanloom.retrieval.chunk_count-- number of chunks returnedspanloom.retrieval.top_score-- highest relevance score in the result setspanloom.retrieval.empty-- boolean, true if no results were returned
The context propagation challenge in async workflows
Synchronous LLM pipelines -- request in, response out, all in one thread -- are straightforward to instrument. The trace context flows naturally through the call stack, and standard OTel SDK context managers handle propagation automatically.
Async workflows are harder. If a pipeline step enqueues work to be processed later by a different worker, the trace context needs to be serialized into the message and deserialized by the receiving worker. OTel provides baggage propagation APIs for this, but it requires explicit engineering: the message producer must inject the context, and the consumer must extract it and restore it as the active context before creating spans.
We use the W3C TraceContext propagation format for this: serialize the traceparent header value into a metadata field on the message, and extract it on the consumer side. This gives us continuous traces even across async boundaries -- you can see the full trace as a single tree, even if the work happened across multiple processes and with arbitrary delay between steps.
Sampling at scale
Recording every span for every request is not feasible at high volume. A pipeline with 8 spans per request at 5M requests per day produces 40M spans per day. At typical span storage costs, this is significant. Sampling is necessary.
We support three sampling modes, reflecting the tradeoffs teams need to make:
Head-based sampling makes the sampling decision at the start of a trace, before any spans are recorded. It's computationally cheap but blindly samples -- you may under-sample rare error cases.
Tail-based sampling buffers span data and makes the sampling decision after the trace completes, based on its properties (error status, cost, latency). It ensures 100% sampling of interesting traces. It requires buffering, which adds memory overhead.
Priority sampling applies different rates to different span types: always record spans with error status or fallback: true; sample 5% of normal spans. This gives you complete coverage of failure cases with low overhead on the common path.
For most teams at the early stages of operating an LLM application, priority sampling with a 10-20% sample rate for normal spans is the right starting point. As volume grows, the rate can be lowered; the sampling decisions can be refined based on what you've learned about which trace properties are actually interesting to investigate.