Why OpenTelemetry Is the Right Foundation for LLM Observability

When we designed Spanloom's instrumentation layer, we had a choice: build a proprietary event format optimized for LLM-specific fields, or build on top of OpenTelemetry's existing span model. We chose OTel, and this post explains why that decision was right even though OTel was not designed for LLMs.

What OpenTelemetry actually gives you

OpenTelemetry is a vendor-neutral observability framework originally designed for distributed systems: microservices, databases, message queues. Its core concept is the span -- a named, time-bounded unit of work with a set of attributes, events, and a parent/child relationship to other spans. A collection of related spans forms a trace.

The model maps to LLM applications more cleanly than it first appears. An LLM call is a span. The retrieval step that precedes it is a sibling span. The context assembly that happens between them is another span. A multi-step workflow -- query rewrite, retrieve, rerank, generate, parse -- is a trace with a tree of spans below a root. The nesting is natural once you see it.

What OTel does not give you out of the box is LLM-specific attribute conventions: where to record token counts, how to represent a prompt version, what the canonical field name for "model name" is, how to record the cost of a call. That's where the semantic conventions work being done by the community (specifically the GenAI semantic conventions effort) becomes important. We tracked that work closely and aligned our attribute naming with the emerging conventions where they existed.

Why proprietary instrumentation is a trap

The alternative we considered was building a proprietary event model: a custom JSON schema with LLM-first fields, our own SDK, and our own trace storage format. This would have let us optimize the data model for our specific use case. We decided against it for three reasons.

First, proprietary instrumentation means vendor lock-in for your users. If you instrument your application with a proprietary SDK, switching observability tools means reinstrumenting your entire codebase. If you instrument with OTel, switching tools means pointing your exporter at a different endpoint. We did not want to impose switching costs on teams that chose Spanloom early and might want to add other tools later.

Second, OTel has a large ecosystem of collectors, processors, and exporters already built and maintained. A team using OTel for their broader infrastructure can route LLM traces through the same pipeline as their service traces. They don't need a separate sidecar or agent for LLM observability specifically.

Third, the OTel model has already solved a set of hard problems around trace context propagation, baggage, sampling, and SDK lifecycle management. Building these from scratch would have been a significant engineering distraction from the actual product work.

What we had to extend

OTel spans have a fixed set of built-in fields: name, trace ID, span ID, parent span ID, start/end timestamp, status, attributes (key-value pairs), and events. All the LLM-specific fields live in attributes. This works, but attribute namespacing requires discipline. We follow the emerging GenAI convention for well-known fields:

• gen_ai.system -- the model provider (openai, anthropic, bedrock)
• gen_ai.request.model -- the specific model name
• gen_ai.usage.input_tokens and gen_ai.usage.output_tokens
• gen_ai.operation.name -- chat, completion, embed, rerank

For fields not yet covered by the GenAI convention, we use a spanloom.* namespace: spanloom.prompt.version, spanloom.retrieval.chunk_count, spanloom.cost.usd. This avoids collisions with future standard fields while keeping our extensions clearly namespaced.

The sampling question

OTel has a built-in sampling model. For high-volume LLM applications, you may not want to record every span -- at 5M requests per day, storing full traces for all of them is expensive. The OTel sampling API lets you define rules: sample 100% of traces where any span has an error, sample 10% of normal traces, always sample traces that exceed a cost threshold.

This is one of the most useful things the OTel foundation gives you. Building a statistically sound sampling system from scratch is hard. The OTel tail-based sampling model, while complex to configure, solves real problems that any high-volume instrumentation layer has to face.

Should you instrument with OTel directly?

For most teams, the answer is: use an SDK that wraps OTel rather than instrumenting with the OTel API directly. The raw OTel API is verbose. The SDK handles span creation, attribute serialization, context propagation, and export plumbing. What you write in your application code is 5-10 lines rather than 40.

The underlying point stands regardless of how you instrument: choosing OTel as the foundation means your data is portable, your instrumentation code has a long support lifetime, and you get the ecosystem for free. For an AI application that you expect to run for years and that you may want to observe with multiple tools, that foundation is worth more than any short-term convenience a proprietary format might offer.