What running an AI application costs, and why nobody knows up front

The cost of an AI application is set by how much text flows through the model, not by how many people use it. That is why it cannot be estimated from a licence count — and why it can only be managed by measuring.

Conventional software has a cost you know in advance: the server is so much a month. An AI feature does not work that way. You pay per call to the model, and a call costs more the more text goes into it and the more comes back out.

That sounds like a detail. It is not.

What the bill is made of

The model charges for input and output alike. Input is everything you send it — the user's question, but also the instructions, examples and source material you added to the prompt so it would answer well. Output is what it writes back.

• Context size. Attach ten pages of documentation to a question and you pay for those ten pages on every single call, even when the answer concerns one sentence.
• Choice of model. A larger model costs more for the same work. Plenty of tasks — sorting, extracting fields, a short summary — a smaller one handles just as well.
• Retries. When the model returns something that fails validation and the system tries again, you paid for it twice.
• Traffic nobody sees. Tests, developer experiments and scheduled jobs run on the same account as your customers.

Those last two are usually the surprise. The application works, users are happy, and the bill is double the estimate.

Why the up-front estimate misses

A reliable estimate would mean knowing how many questions people will ask and how long they will be. That is unknowable before launch — and after launch it shifts with how the application is used. An estimate is worth having as an order of magnitude, not as a budget.

A better approach is different: release the feature to a small group, measure real consumption per user, and do the arithmetic from that. Choosing a model and what it costs is a separate piece.

What actually brings the cost down

Not haggling with the provider. These four things:

1. Caching repeated questions. In customer support a large share of questions repeat. An answer the model has already produced does not need producing again.
2. A smaller context. Instead of the whole manual, send only the passages the question touches. That is precisely the job of a retrieval layer (RAG).
3. The right model per step. Break the task up and save the expensive model for the part that genuinely needs it.
4. Caps. A limit per request and per period. Not because you expect trouble, but because a bug in a loop can burn a month's budget in an hour.

Measure per feature, not per invoice

The provider's invoice gives you one number. That is too little to decide anything on. What helps is knowing which feature spends what — then you know whether to optimise it or switch it off, because nobody uses it.

An alert on unusual consumption belongs with that. The difference between "we knew within two hours" and "we found out from the statement" is the difference between a nuisance and an unpleasant month.

All of this is part of how we run AI applications — and if you are weighing whether to build it in-house, see cloud compared with a server of your own.