Choosing a language model and keeping costs under control

The first question is usually "which model is best?". Understandable, and a dead end. There is no best model. There is the cheapest model that still does your job.

Why benchmarks will not help

Tables of percentages look authoritative, and deciding by them is a mistake.

First, they test something other than what you need. That a model solves maths olympiad problems tells you nothing about whether it will pull the right registration number off a blurry invoice.

Second, the differences at the top are small and they move every couple of months. The model that led in March is third in May and half price in July. Basing a decision on today's leaderboard means redoing it every quarter.

The only benchmark worth anything is your own: take fifty real cases from your business, run them through three models, compare. It costs one afternoon and tells you more than every table combined.

What actually decides it

In our order of importance:

• Can it do the job at all? Test on your data. If a small model can, the discussion is over — take the small one.
• How fast? If a user is waiting, two seconds is not twenty. If it is an overnight batch, who cares.
• What will it cost at real volume? Not at demo volume. Multiply by the actual monthly call count.
• Where does the data go? With sensitive material this can decide it before anything else does.
• Can you swap it? The most important item on the list.

Replaceability beats the choice itself

The market moves too fast to commit to anyone.

In practice that means one thing: keep a thin layer in your code that every call goes through. Not one provider's SDK scattered across twenty files. When a model arrives in six months at half the price, you want to change one file, not twenty.

That same layer, incidentally, gives you retries, timeouts, cost accounting and a fallback provider for when the first one goes down. Worth writing while it is still an hour of work.

Where the cost comes from

You pay for tokens in and tokens out, and output tokens are typically much more expensive. That has direct consequences:

Long conversations get expensive fast. The model remembers nothing, so every call resends the entire history. The tenth question in a chat costs several times the first — not because it is harder, but because nine earlier ones travel with it.

The system prompt is resent every time. Those two pages of instructions you are proud of are billed on every single call.

"Explain your reasoning" is not free. Let the model think out loud and you pay for every word of the thinking.

What genuinely works

Ordered by effect per unit of effort:

1. Cache responses. Boring and the most effective. The same questions repeat more than you think. An answer served from cache costs nothing.
2. Route to a smaller model. Classification and extraction do not need the flagship. The gap is often an order of magnitude.
3. Use prompt caching. Providers can discount the repeated part of a prompt — put stable instructions first and changing data last.
4. Send less. Do not send the whole document when three paragraphs will do. This is, incidentally, exactly the problem RAG solves.
5. Trim history. Summarise older turns instead of resending them verbatim.
6. Batch what can wait. Providers offer a cheaper tier for work that is not urgent.

Measure from day one

If you do only one thing from this article, do this one — the rest you will work out yourself.

Log on every call: which model, tokens in, tokens out, how long it took, which feature it belonged to. Without that you cannot tell whether one feature is bleeding you or a thousand small ones are, and you will optimise blind.

And set a ceiling. A runaway agent loop or a badly written cycle can spend a monthly budget overnight. This is not hypothetical; it has happened to enough companies that providers now ship their own limits.