Orchestrating language models: from one prompt to a system
One prompt is a demo. An application is something else. On splitting work, routing between models, and where to leave ordinary code alone.
The first AI feature is usually one big prompt. You put everything in it, the model somehow copes, and it feels easy.
The second version of that prompt is three hundred lines, contains eight instances of "IMPORTANT:", and still returns nonsense now and then. That is usually when someone asks whether a better model would help. Normally it would not. The problem is not the model. The problem is one task doing five things at once.
Step one: break the task apart
Take processing an incoming invoice. As one prompt it reads: "read the invoice, verify the supplier, check the amounts, assign the right category and write a summary".
Split into steps it looks different:
- extract the fields from the document (supplier, registration number, total, date, line items),
- look the supplier up in our database by registration number — a plain SQL query, no model,
- check the line items add up to the total — arithmetic, no model,
- suggest a category based on this supplier's history,
- if anything is uncertain, hand it to a person.
Suddenly that is two model calls instead of one, but each has a single clear job. Steps two and three need no AI at all — and that is the more important half. A model "verifying" sums is the most expensive way ever devised to get addition wrong.
If you take one sentence from this article: use AI where you cannot write the rules. Everywhere else, write the rules.
Chaining, and what it costs
Put the steps in sequence and one step's output feeds the next. It works, with two catches.
First: errors do not add up, they multiply. Nine steps at ninety percent reliability does not give you ninety percent. It gives you about thirty-eight. So keep the chain short and check after each step that the output makes sense at all.
Second: latency. Every step is a network call measured in seconds. Five in a row means a user is waiting. Run independent steps in parallel, and anything that can happen in the background should happen in the background.
Routing: not every task needs the strongest model
Models differ in price and speed, often by an order of magnitude. Sending "is this a complaint or an enquiry?" to your most expensive model is taking a lorry to the corner shop.
A sensible split looks roughly like this: simple, high-volume work (classification, extraction, short summaries) goes to a small fast model. Work that needs reasoning, joining several things together, or writing code goes to a strong one. Plus one rule for safety: when the small model is unsure, let it escalate.
In practice this means having one place in the code that decides which model gets a task. Not ten places with a model name hard-coded next to a prompt. The second one catches up with you the first time pricing changes, or the first time a new model ships.
Have the model return data, not prose
This is the cheapest improvement available. Ask for structured output — JSON against a schema — instead of a sentence you then fish a number out of with a regex.
The reason is not tidiness. Structured output can be validated. Field missing? Rejected. Amount not a number? Rejected. Category not in the allowed list? Rejected. You now have a gate between the model and your database that stops a hallucination before it breaks something.
And when output fails validation, try again. Models are non-deterministic — the second attempt is often fine.
Where it actually falls over
A few things that always show up in production and never in a demo:
- External APIs go down. Set timeouts and have a fallback. An AI feature that takes checkout with it because a provider had an outage is worse than no AI feature.
- Costs grow quietly. Three cents a call is nothing until it is a hundred thousand calls a month. Measure from day one.
- A prompt is input, not code. Once you interpolate text from a user or an email into a prompt, assume somebody will try to talk you into ignoring your previous instructions. Keep instructions separate from data, and never give a model authority you would not give a stranger.
- Cache. The same question repeats more often than you would think.
When not to do any of this
If a condition in code solves the task, solve it with a condition in code. It is cheaper, faster, testable, and it does not get things wrong.
That sounds obvious, and yet we have seen enough projects where a language model was deciding things an if could have handled. Orchestration is not about how many models you wire in. It is about how many you left out because they were not needed.
Are you solving something similar in your company?
I want a free consultation