A smaller LLM becomes the better product decision when it meets the task’s quality and deployment requirements at a lower total cost. The qualification matters. A cheap answer that somebody has to repair is an unfinished job with an attractive invoice.
People forget the problem they wanted to solve. The conversation drifts towards which model looks most impressive, and eventually the product becomes a place to demonstrate the model. I would start somewhere less glamorous: what must the person using this feature be able to finish?
Describe the job before choosing the engine
Consider an illustrative support workflow. It reads a message, identifies the product and routes it to the right queue. That is a different job from interpreting a disputed refund policy and deciding what the company should do. Putting both under “customer support AI” hides the distinction that should drive model selection.
A narrow classifier might be enough for the first task. The second needs a much harder evaluation and may still require human judgement. Neither decision follows from a model’s position on a general leaderboard.
Google’s Flash-Lite documentation explicitly associates that model with high-volume classification and simple extraction. That is useful evidence about its intended workload, not proof that it will handle yours. The page also restricts access for new projects, a reminder to check availability as well as capability.
For closed models, a “small” or “lite” label does not give you a reliable parameter count. Compare actual deployment options. Write down the quality floor, acceptable delay and data constraints before looking at the price column. These are the quiet product decisions that make the later comparison meaningful.
The unit is accepted work
Token prices are easy to compare. The bill for getting something useful done is messier.
For a fixed evaluation batch, add inference, retries, tool calls, infrastructure and human correction costs. Divide that total by the number of tasks accepted against your rubric. Keep serious failures visible separately; an average cost must never make an unacceptable outcome look tolerable.
This is a decision metric, not a universal accounting standard. State what you included. A hosted API and a self-hosted model have different operating costs, and idle hardware does not disappear because the model weights were free to download.
A smaller model can win comfortably. It can also lose if it needs longer prompts, repeated attempts or enough review to erase the saving. Measure that before declaring the cheaper endpoint the cheaper product.
Test the wait, and the mistakes
Use the same representative cases and acceptance rules for each candidate. Include incomplete requests, unusual vocabulary and the mistakes that would be expensive to miss. Give each model reasonable prompt tuning, then evaluate on held-out cases rather than the examples used to tune it.
Repeat cases where variability matters. Report results by task type, alongside the overall score. Otherwise a large pile of easy routing decisions can conceal poor handling of the ambiguous messages.
Anthropic’s evaluation guidance discusses tracking latency, token usage, cost per task and errors against a stable task bank. That is why evals belong inside the product decision, rather than arriving after model selection as a ceremonial check.
Measure time to a usable result, including retrieval, validation and retries. Look at the slow tail as well as the median. A quick first token is pleasant; it is not the same as finishing the customer’s request quickly.
Local is a deployment choice, not a privacy certificate
A model running locally may help satisfy a requirement to keep inference inside a particular environment. You still need to inspect where prompts are logged, who can read those logs, what retrieval sends elsewhere and how long records remain.
Smallness does not answer any of those questions. Nor does a larger hosted model automatically fail them. Check the actual service configuration and data flow rather than assigning privacy properties to model size.
Make the larger model earn its place
Routing can reserve a more capable model for harder requests. Anthropic describes this pattern in Building effective agents. But the router introduces another decision to test, plus its own cost and delay. If it misses the difficult cases, the fallback never gets a chance.
I would start with one candidate that clears the requirements and add routing only when measured gains justify the extra machinery. Keep the stronger model when its improvement matters to the user. Choose the smaller one when the whole workflow works better.
The person waiting for their support request to reach the right team has no reason to applaud your model choice. Getting it right should be uneventful.
