What happens when an automation fails?
The normal case is quickly built. What makes an automation operable is how it behaves when something does not work — and the question of who notices.
What you will learn
Two kinds of error, two responses
Temporary disruption
- Target system unreachable, timeout
- Response: try again, with increasing intervals
- After the last retry: into the rework queue
Business rejection
- Required field missing, value unknown, rule violated
- Response: do not retry — nothing will change
- Straight into the rework queue, with the reason
Anyone who treats both the same ends up with one of two problems: either a business rejection is retried endlessly and burdens the target system — or a brief disruption throws away transactions that would have gone through on the second attempt.
The rework queue is not an error log
A log records what happened. A rework queue is a work list: each entry is a transaction that needs a decision, with the reason beside it and a way to trigger it again after the correction.
Without this list, failed transactions end up in the log that nobody reads — and come to light at the monthly reconciliation, when the correction is more expensive.
In practice this includes: an overview with the number and age of open cases, the ability to resubmit after the correction, and a person responsible. A list without an owner keeps growing.
Retries need an upper limit
For a temporary disruption, retrying is right — but not indefinitely and not every second. The usual approach is increasing intervals and a fixed upper limit, after which the transaction goes into the rework queue.
A second property goes with this: processing must be repeatable without duplicate effects. If an order was created on the first attempt and only the response was lost, the second attempt must not create a second order. For this, each transaction carries a unique identifier that the target system recognises.
An example from practice
The outage nobody noticed
A nightly transfer ran reliably for weeks. Then an access key changed on the other side. The transfer failed — silently, every night. It was noticed after eleven days, when someone missed figures.
The lesson is not that the transfer had a fault. It had no watchdog: nobody checked whether it had run at all. Monitoring that reports when a run FAILS TO HAPPEN is more important than monitoring that reports when an error occurs — a run that does not happen never reports itself.
Who notices that nothing is running any more?
This is the question that most often remains open. Three things answer it:
With these three details, an automation becomes something you can operate. Without them it is a piece of technology that works as long as nobody is looking.
Common misconceptions
About the author
Fabian Lücker is the managing partner of PHP-Schmiede KG in Rheine and has worked in software development since 2007. He is responsible for development and concept design, and here he writes about decisions that arise in real projects.