Latency is the first signal every team measures because it's the most obvious and the easiest to instrument. You wrap a call in a timer and you have a number. It's also, in isolation, one of the least useful signals for understanding what your LLM application is actually doing in production.
High latency is a symptom. The four signals below are root causes -- or at minimum, they point much more directly at root causes than a latency number can.
Signal 1: Input token count distribution per flow
The distribution of input token counts for a named LLM workflow tells you two things that latency cannot: whether your context is growing unexpectedly, and whether different query types are consuming materially different resources.
Track this as a time series. If the median input token count for your main generation endpoint grows from 800 to 1,400 tokens over four weeks without a deliberate product change, something in your context assembly is accumulating. This could be a prompt that has grown through incremental edits, a retrieval step returning more chunks, a conversation history window that expanded, or a template change that wasn't audited for length impact.
Track it also as a distribution. A bimodal distribution -- most requests around 600 tokens, a tail around 3,200 tokens -- often reveals a query class you didn't design for explicitly. That tail is expensive and may be producing lower-quality output due to context overload.
Signal 2: Retrieval efficiency ratio
In RAG applications, the retrieval step consumes tokens that go into the context window but may not contribute to the quality of the answer. The retrieval efficiency ratio is a proxy for this: chunks_used_in_context / chunks_retrieved. If you retrieve 8 chunks and only 3 of them make it into the final context (because the rest didn't pass a relevance threshold, or because the context budget was exhausted), your retrieval is working at 37.5% efficiency.
A low retrieval efficiency ratio suggests one of several problems: your retrieval is returning topically related but not semantically relevant content, your index quality is low, or your chunk boundaries are too coarse to produce high-precision results. None of these are visible from latency alone. Retrieval efficiency is its own operational signal.
Track it per query type or intent class. Retrieval efficiency is often high for factual lookups and low for comparison or synthesis queries, because the latter require content from multiple parts of the knowledge base that may not cluster well in embedding space.
Signal 3: Output token variability
Output token count variability for a named workflow tells you whether the model's generation behavior is consistent. A workflow where average output tokens are 380 but standard deviation is 280 is generating highly variable responses. This often correlates with output quality issues: the model is not following the format or length constraints in the prompt reliably.
High variability in output tokens is typically a prompt instruction problem. The model doesn't have clear constraints, or the constraints are ambiguous, or there is a distribution of input types that the prompt doesn't handle uniformly. Measuring output token variability gives you a diagnostic handle on instruction-following consistency that reading individual responses cannot.
Low output variability is usually good. A workflow where outputs are consistently 350-420 tokens is one where the model has clear format guidance and is following it. When output variability increases without a prompt change, something about the input distribution changed.
Signal 4: Span error and fallback rate
As discussed in other contexts, LLM applications fail silently. An HTTP 200 is returned even when the application has fallen back to a default, skipped a step, or produced an output that failed internal validation. Standard error rates won't catch these.
Span-level fallback rate is the signal that catches silent failures. Each span in your trace that has a fallback path should emit a fallback: true attribute (or equivalent) when that path is taken. Aggregate this across your traffic to produce a fallback rate per span type.
A fallback rate above your baseline for a specific span type is a high-priority signal. If your response validation step has a fallback rate that jumps from 2% to 15% without a code change, something about the incoming distribution has changed: a new query type, a change in the data the retrieval is pulling, a model behavior change on a prompt version boundary.
Putting these signals together
Latency is not useless. But it's a lagging indicator -- it tells you after the fact that something is wrong, without telling you what. The four signals above are more diagnostic: they have clearer relationships to specific root causes and can surface problems before they manifest as user-visible quality issues.
The instrumentation cost for all four is low. Input and output token counts come directly from the model API response. Retrieval efficiency requires recording chunk counts in your retrieval span. Output variability comes from computing the standard deviation of output token count over a time window. Fallback rate requires adding a flag to your catch blocks. None of these requires a new data pipeline or a new monitoring system -- they're attributes on spans you should already be recording.