Language-model spend scales with usage, so a feature that cost almost nothing in testing can become a line item in production. The instinct is to switch to a cheaper model. That works sometimes, but it is the blunt option. Five less obvious changes usually deliver more, with less risk to answer quality — and each one is measurable.
1. Measure cost per task, not per token
Tokens are an input; a completed task is the output you pay for. Log cost against each business outcome — a resolved ticket, a processed document — so you can see which workflows are expensive and whether the expense buys value. Without this you are optimizing blind.
2. Route by difficulty
Most requests are easy. Send them to a smaller, cheaper model and reserve the large model for the cases that need it, using a lightweight classifier or a confidence signal to decide. Validate the split on your own test set, because difficulty is specific to your data.
3. Cache what repeats
- Prompt caching — long, stable instructions and reference text can often be cached by the provider at a lower rate, so you stop paying full price for the same preamble on every call.
- Answer caching — for questions that repeat verbatim or nearly so, store verified answers and serve them directly, with an expiry tied to how often the underlying facts change.
4. Send less context
Many systems stuff entire documents into every call. Better retrieval and tighter chunking often cut the context by a large margin and improve accuracy at the same time, because the model sees less irrelevant text. Trim conversation history as well: summarize old turns instead of resending them.
5. Batch what isn't urgent
Back-office jobs — classification, extraction, nightly summaries — rarely need an instant answer. Providers commonly offer cheaper asynchronous batch processing for exactly this. Separate the work that needs speed from the work that doesn't, and price them differently.
Whatever you change, put the eval gate in front of it. A cost cut that quietly degrades answers is not a saving; it is a deferred support bill.
A sensible order
Start with measurement, then context size, then caching, then routing, and consider model changes last. That order puts the cheapest, safest wins first and leaves the model choice as a decision made with data instead of a reaction to an invoice.