Complete LLM observability
in one dashboard
Four focused modules that cover everything from a single request trace to fleet-wide cost trends. No extra infrastructure, no separate logging pipeline.
Trace Waterfall
Every LLM request becomes a structured trace with child spans for retrieval, embedding, model calls, and output parsing. The waterfall view shows duration, tokens, and cost at each step.
- OpenTelemetry-compatible span model
- Nested spans for chained and multi-agent workflows
- Per-span token counts and latency attribution
- Filter by trace ID, model, or custom metadata
Token Cost Monitoring
Attribute token spend to every user, endpoint, model, and request path. Know your cost per feature before the monthly invoice arrives.
- Per-request cost breakdown across input and output tokens
- Model-level spend comparison in a single view
- Daily and monthly trend charts with anomaly flags
- Budget alert thresholds via webhook or email
| Endpoint | Model | Requests | Cost |
|---|---|---|---|
| /chat/answer | gpt-4o | 4,821 | $48.21 |
| /summarize | claude-3.5 | 1,203 | $12.47 |
| /classify | gpt-4o-mini | 9,112 | $4.08 |
| /embed | text-embed-3 | 22,400 | $0.90 |
| Total | $65.66 | ||
Prompt Drift Detection
Prompt text changes silently through iterations, A/B tests, and rushed hotfixes. Spanloom tracks every prompt version hash and alerts when the live version diverges from your declared baseline.
- Automatic prompt fingerprinting per request
- Version timeline with diff view between versions
- Drift alert triggers: hash change, token count change, semantic distance
- Rollback annotations to mark which version is canonical
RAG Retrieval Auditing
When a RAG pipeline fails, the problem is almost always in retrieval. Spanloom shows which chunks were fetched, their relevance scores, and whether they ended up influencing the answer.
- Per-request chunk inspection with relevance scores
- Context utilization rate (how much of the context window was used)
- Dead-chunk detection: retrieved but not cited in output
- k-value and threshold tuning recommendations
| Chunk | Score | Tokens | Used |
|---|---|---|---|
| doc_42 p.3 | 0.91 | 312 | yes |
| doc_17 p.1 | 0.87 | 280 | yes |
| doc_88 p.5 | 0.72 | 195 | yes |
| doc_55 p.2 | 0.58 | 401 | no |
| doc_03 p.7 | 0.51 | 388 | no |