Smaller language models, including ones that run on a laptop, a phone or a private server, are among the most discussed shifts in applied AI. The argument for them is rarely raw capability. It's the fit between a narrow task and a model that is faster, cheaper and easier to keep inside your own boundary.
Where small models tend to win
- High-volume, narrow tasks — classification, extraction, routing and tagging, where the output space is small and the cost per call matters.
- Latency-sensitive features — anything interactive where waiting for a large remote model breaks the experience.
- Sensitive data — keeping text on your own infrastructure or the user's device removes a class of transfer and vendor questions.
- Offline or poor-connectivity settings — field work, plants, vehicles and branches with unreliable networks.
Where large models still earn their cost
- Open-ended reasoning across long, messy context.
- Rare, high-stakes requests where a failure costs more than the model.
- Tasks where quality varies widely and you can't yet write a good test set.
The pattern that works: route most requests to a small model, escalate the hard ones to a larger model, and measure both on the same test set.
How to decide
Build a test set from real examples, run candidate models of different sizes against it, and compare accuracy, latency and cost per task. Let the numbers choose. Re-run the comparison when new models appear, because the ranking changes faster than most procurement cycles.