Every AI feature demo runs on a handful of test queries and costs almost nothing. Production runs on every user, every session, every retry — and the bill scales with usage in ways that traditional software economics don't prepare teams for. A feature that felt free in the prototype can become the largest line item in your cloud spend within a quarter if nobody modeled it first.
The variables that actually drive cost
- Input tokens — every word of context you feed the model, including retrieved documents, chat history, and system prompts, is billed. A RAG system that stuffs ten documents into every query pays for ten documents every query.
- Output length — longer, more elaborate responses cost more to generate. A model instructed to 'be thorough' is a model instructed to cost more.
- Model tier — frontier models are dramatically more expensive per token than smaller or older ones. Not every task needs the newest model; some need the cheapest one that clears the accuracy bar.
- Retry and fallback logic — a naive retry-on-failure loop can silently multiply cost during an outage or a rate-limit event, exactly when nobody's watching the dashboard.
Three engineering levers, in order of impact
- Model tiering — route easy queries to a small, cheap model and reserve the frontier model for genuinely hard cases. Most query distributions are lopsided toward easy.
- Semantic caching — if the same or similar question gets asked repeatedly, cache the answer instead of recomputing it. This alone often cuts cost more than any prompt optimization.
- Context trimming — retrieve and pass only the passages that matter, not entire documents. Better retrieval ranking is a cost lever, not just a quality one.
The question to ask before launch isn't 'what does this cost per query' — it's 'what does this cost per active user per month at ten times current traffic.' Most nasty surprises live in that second number.
What we build into every deployment
A cost dashboard broken down per feature and per model, before launch, not after the first alarming invoice. Budget alerts tied to actual usage, not guesswork. And a documented model-swap procedure, so when a cheaper or better model appears — and one always does — switching is an afternoon's work, not a re-architecture.