API billing for LLM providers works on a token basis: you pay for what you send and what you receive. The math is simple. The problem is that most teams don't see the per-request breakdown until the invoice period closes.
An invoice tells you how much you spent. It doesn't tell you which workflow spent the most, which user segment drove the spike, or whether cost is growing because you have more users or because your average context window got larger. Per-request cost tracking is what bridges that gap -- and it's straightforward to instrument if you capture the right data at the right point.
What to measure at the request level
Every LLM API response contains token usage metadata. For OpenAI, it's in usage.prompt_tokens and usage.completion_tokens. For Anthropic, it's in usage.input_tokens and usage.output_tokens. For Bedrock, it varies by model but is accessible in the response metadata. Whatever provider you use, the token counts are in the response object -- you just have to record them.
The cost calculation is: cost_usd = (input_tokens / 1_000_000) * input_rate_per_million + (output_tokens / 1_000_000) * output_rate_per_million. Model pricing tables are public. Build a simple lookup by model identifier and compute the cost in your instrumentation layer at call time.
Record both the token counts and the computed cost as span attributes. Don't rely on reconstructing the cost later from token counts -- pricing changes, models get added and removed, and a computed cost at call time is more reliable than a reconstruction at analysis time.
Breaking down cost by component
If you have a single LLM call per user request, per-request cost tracking is trivial. Most production LLM applications have more than one: a query rewrite call, a main generation call, sometimes a validation call or a secondary model call for a specific capability. Per-request cost in these cases is the sum of individual call costs.
More useful than the aggregate is the breakdown: which call accounts for what fraction of the total cost? In a RAG pipeline with query rewriting, the generation call typically dominates, but the query rewrite call may be consuming 15-25% of total cost for a small marginal quality improvement. That's a tradeoff decision that should be explicit, not invisible.
Track cost per span, not just per trace. A span-level cost breakdown -- visible in the trace timeline alongside latency -- is what lets you answer "is this optimization worth it?" with actual numbers rather than estimates.
Cost trending: the signal that matters most
The most actionable cost signal is not today's cost but cost trend per workflow over time. If your average cost per request for a named endpoint was $0.0018 three months ago and is $0.0027 today, something changed. Identifying what changed is the diagnostic work that prevents a problem from becoming a bill.
Common causes of cost trend increases:
- Prompt growth through incremental iteration without auditing (the most frequent cause)
- Retrieval returning more chunks -- a config change during an experiment that never got reverted
- Conversation history window expanding as users have longer sessions
- A new workflow path being activated for more users than initially intended
- A migration to a more capable (and more expensive) model without a corresponding review of token efficiency
You cannot find these without a time series of per-flow cost data. The invoice will eventually tell you the total has increased; it won't tell you why or where.
Setting cost thresholds that actually fire
Most teams that set up LLM cost alerts set a threshold on total monthly spend -- something like "alert me if monthly spend exceeds $X." This alert fires after the problem has already accumulated for weeks.
Per-request cost thresholds are more useful. An alert that fires when the trailing 1-hour average cost per request for a specific workflow is more than 30% above the 7-day moving average is actionable in minutes, not weeks. The signal is: something changed, and you should find out what before it compounds.
Set these thresholds per flow, not globally. Your main generation endpoint and your embedding endpoint will have different cost profiles and different baselines. A global threshold that makes sense for one will either be too tight or too loose for the other. Per-flow baselines with percentage-deviation alerts are the model that's actually useful in practice.
What you get by doing this
Per-request token cost tracking with per-flow trending and threshold alerts gives you three capabilities that invoice-level monitoring does not: early detection of cost increases before they accumulate, component-level visibility into which step in a workflow is expensive, and a factual basis for optimization decisions. That last point matters -- when you can say "query rewriting costs $X per request and our data shows it improves retrieval quality by Y%," you can make an informed decision. Without the instrumentation, you're guessing.