An LLM application can be fully available and consistently wrong. Observability needs to cover quality and cost, not just uptime.
Signals
- Latency — time to first token and total, by step.
- Cost — tokens and spend per request, per feature and per customer.
- Quality — sampled human review, automated checks and user feedback.
- Retrieval — which sources were used, and whether they were relevant.
- Errors — timeouts, refusals, malformed outputs, fallback use.
Add trace IDs so you can follow one request through retrieval, model calls and tools. Debugging without them is guesswork.