Caching
- Prompt caching — reuse long, stable prefixes such as instructions and reference text where your provider supports it.
- Result caching — store answers to repeated, identical questions, with expiry tied to how fast the facts change.
- Retrieval caching — cache embeddings and search results for popular queries.
Routing
Classify each request by difficulty and send routine ones to a smaller, faster model; escalate only when the first answer fails a check or confidence is low. Measure the share of traffic each tier handles and the quality on a shared test set.
Caching must respect permissions. A cached answer for one user must never be served to someone who isn't allowed to see it.