Any automation that calls other systems will sometimes time out, crash or run twice. The design question is what happens then.
Principles
- Idempotent actions — repeating the same request with the same key has the same effect as doing it once.
- Unique request IDs stored with the result, so retries are recognized.
- Bounded retries with increasing delays, then escalation to a person.
- Checkpointing — record progress so a restart resumes rather than repeats.
- Compensation — a defined way to undo a partially completed sequence.
Test by deliberately interrupting workflows halfway and re-running them. Do it before production teaches you.