Interactive chat needs speed. Back-office tasks such as summarizing yesterday's tickets or classifying documents usually don't.
Patterns
- Queue non-urgent work and process it in batches.
- Use provider batch options where offered, which are typically discounted, with slower turnaround.
- Precompute common outputs overnight.
- Apply rate limits and backpressure so spikes don't cascade.
- Set per-job budgets and timeouts.
Classify each AI task by how quickly someone needs the answer. The slowest acceptable option is usually the cheapest.