When we first built an LLM health dashboard for internal use, we made the same mistakes that most teams make. We're writing this down because we still see the same anti-patterns in almost every early-access conversation we have.
This isn't a post about which charting library to use. It's about what to actually measure, what not to measure, and why the natural first instincts lead you somewhere less useful than you'd expect.
Mistake 1: Starting with latency because it's easy to measure
Our first dashboard had one chart: p50/p95/p99 request latency, updated every 60 seconds. We thought this was a reasonable start. It wasn't useless -- spikes were at least visible. But it could not tell us anything actionable. A latency spike could mean the model was slow, the retrieval was slow, our context assembly was bloated, there was a retry loop, or there was a network issue. The chart told us something was wrong; it couldn't tell us what or why.
The fix wasn't to add more latency charts. It was to stop treating the full request as an atomic unit and start decomposing it into spans. Once we could see latency per span type -- retrieval, context assembly, LLM call, response parse -- a latency spike immediately identified itself. The first time we did this, we found that 80% of our latency variance was in the retrieval step, not the model. We had been hunting in the wrong place for weeks.
Lesson: Your first monitoring primitive should be span decomposition, not aggregate latency. If you can only add one thing, add per-span timing before you add any other signal.
Mistake 2: Monitoring outputs instead of flow structure
Our second instinct was to track LLM output quality. We built a simple manual rating flow where team members could flag responses as good or bad. This was expensive (human time), slow (ratings lagged production by hours), and sparse (only rated 2% of requests).
The insight that changed our approach: most quality problems in LLM applications are structural, not random. They happen because the inputs to the model changed, not because the model randomly produced bad output. Context that grew too large, a prompt version that reverted, a retrieval step returning fewer relevant chunks -- these are the root causes. If you can monitor the inputs to each step, you often don't need to rate the outputs to find problems.
We shifted from output rating to structural monitoring: input token counts, retrieved chunk counts, context-to-output token ratios, prompt version distribution. When any of these drifted from baseline, we investigated. This caught 90% of what we had been catching with manual rating, at zero human cost per request.
Mistake 3: Building dashboards for the wrong audience
We built our first dashboards for management: high-level cost trends, total requests, daily active usage. These were fine for weekly reports but useless for debugging. When something went wrong in production, we had beautiful charts that told us volumes and costs were increasing but nothing that helped us find the source of a quality problem.
The dashboards that turned out to be genuinely useful were engineer-facing: per-flow span timelines, per-flow token count distributions, prompt version breakdowns per endpoint. If an engineer couldn't use a chart to narrow down the cause of a production incident within five minutes, the chart wasn't earning its place on the screen.
Rebuild principle: For each chart on your dashboard, write down the specific debugging question it answers. "What is our current token cost per flow?" is a question. "Overall usage trend" is not. Delete anything that doesn't have a clear debugging question attached to it.
Mistake 4: Alerting on the wrong things
Our first alert was on error rate above 5%. It never fired because, as noted above, LLM applications don't fail with errors -- they fail with silent fallbacks. We had alerts on signal we didn't need to be alerted on (HTTP errors, which we already watched separately) and no alerts on signal we did (token cost per flow above baseline, retrieval empty-return rate, context window utilization above 85%).
The alerts that proved their worth in practice:
- Input token count for a named flow exceeds its 7-day moving average by more than 25%
- Hourly average cost per request for a named flow crosses a fixed threshold
- Retrieval empty-return rate above 10% in a 15-minute window
- Prompt version distribution for a flow shows a version you didn't intentionally deploy
These alerts fire rarely. When they fire, something real is happening. That's the bar for an alert worth having.
What we shipped in the end
The dashboard we run today has four main views: span timeline for debugging individual requests, per-flow cost and token trends, prompt version distribution, and a retrieval health section. The management summary is generated from the same data but is not our primary working view.
If we were starting over, we would instrument spans from day one, skip the aggregate latency chart entirely until we had per-span breakdown, and not build any output quality metrics until we had exhausted what structural monitoring could tell us. The structural signals are cheaper to collect, faster to update, and far more actionable. That's the lesson that took us the longest to learn.