System integrations: why data does not move between programs on its own

An integration is not "connecting two systems". It is an agreement about what happens when one of them is down. Everything else is detail.

In most companies that approach us the integration already exists — a person is doing it. Somebody pulls the morning orders out of the shop, retypes them into the warehouse system, moves invoices into accounting in the evening and refreshes stock levels once a week. It works as long as that person is at their desk and does not slip.

Why it is not just "export and import"

Moving data once is trivial. The problem starts on the second run. The system has to know what it already sent, what changed in the meantime, and what became of the message it sent and got no answer to.

This is exactly where quickly written connectors fail. When the network drops halfway through, there are two outcomes: the data is lost, or a retry writes it twice. Nobody notices the first straight away. The second shows up as a duplicated order or stock written off twice.

Four things an integration has to do

• Repeat without harm. The same message sent twice must end in the same state as one sent once. You get there with a unique operation identifier the receiving side remembers.
• Survive the other side going down. When the target does not answer, the transfer must not be dropped. The message waits in a queue and tries again — with a growing pause, so it does not flood a system that is just recovering.
• Know what the source of truth is. If a product price can be changed in the shop and in the ERP, sooner or later the two disagree. Every field needs one system that owns it and others that only display it.
• Say when something broke. A silent integration is the worst kind. When a feed has not run for three days, somebody has to learn it from an alert rather than from a customer whose order never arrived.

Two patterns to choose between

With direct connections the systems call each other. It is the fastest thing to build and makes sense for two or three systems. The drawback appears at the fourth: the number of connections grows faster than the number of systems, and one change touches several places at once.

With a broker in the middle — a queue or an integration layer — each system only talks to that. It costs more to build, but adding a fifth system means one new connection rather than four. It also decouples their speeds: slow accounting no longer slows the shop down, because the shop has already handed its message over.

This is not a matter of fashion. With two systems a queue is pure overhead; with six, direct connections are a mine that goes off the first time a supplier is replaced.

What is worth settling before you start

The most expensive integrations come from an unsettled brief, not from bad code. Four questions are worth answering before the first line:

• Which fields move, and which system is the source of truth for each of them?
• How often — continuously on every change, or in batches? Continuous costs more and is unnecessary for most data.
• What should happen on a conflict, when the same record changed on both sides?
• Who finds out that a transfer failed, and what do they do about it?

The fourth is the one briefs leave out most often — and the only one that decides whether a fault is found in an hour or in a month.

How we build one

We start by mapping what is moved by hand today, because that is the most accurate description of the real process a company has. From it comes a contract between the systems — what goes where and in what shape — and only then the code. The transfer gets a queue, retries and a record of what went through, so "did that order make it?" can be answered without guessing.

More on how this fits with the rest is on our services page. If you have a specific pair of systems to join up, get in touch — from a description of what you do by hand today we can say what is worth automating and what is not.