Teams choose models by capability and discover in testing that the most capable option is also the slowest. For a back-office batch job that is fine. For a chat widget or a voice agent, where a person is waiting, delay changes how the whole product feels: a slightly less capable system that answers in two seconds is often judged better than a brilliant one that takes ten.
Set a budget first
Decide how long the user should wait from their action to the first useful response, and treat it like any other requirement. For text, the first visible words matter most; for voice, the pause before the reply starts is what people notice. Then divide that budget across the steps.
Where the time goes
- Input processing — speech recognition, document parsing, network upload.
- Retrieval — searching your knowledge base, which should be a small fraction of the total.
- Model time to first token — driven by model size, prompt length and whether a reasoning step happens first.
- Generation — the rest of the answer, which can overlap with everything after it if you stream.
- Output processing — speech synthesis, formatting, tool calls.
The biggest levers
- Stream the answer — show or speak the first sentence while the rest is still being written. This changes perceived speed far more than shaving milliseconds.
- Overlap the steps — start synthesizing speech for the first sentence while the model writes the second; run independent lookups in parallel.
- Shorten the prompt — less context means a faster first token, and often better answers.
- Match the model to the task — a smaller model for routine questions, a larger one only when needed.
- Limit hidden thinking — reasoning models can spend seconds before the first word; cap their effort for interactive uses.
Measure the delay the user experiences, from the moment they stop speaking or press send to the moment they get something useful, and track its slowest cases as well as the average. A good average with a painful tail still feels broken.
A useful rule
Spend latency only where it buys quality the user can notice. For interactive features, pick the fastest option that clears your accuracy bar on your own test set, and stream the result. Save the slow, careful models for work where nobody is watching the clock.