principle-make-operations-idempotent

Make an operation idempotent by ensuring it converges to the same end state regardless of how many times it runs or where it starts. Scan for existing state at startup, clean stale artifacts, and add a reconciliation step wherever leftover state could change the outcome.

Ask three questions: what happens if it runs twice in a row, what happens if the previous run crashed at every possible point, and does re-execution converge to the same end state. If any answer depends on leftover state, add a reconciliation step.

Use PID-based stale lock detection. When acquiring a lock, check whether the process ID recorded in the lock file is still alive; if not, the lock is stale and can be safely reclaimed instead of blocking forever.

Compare by content equivalence whenever artifacts may be recreated in a different order across runs, such as regenerated files or rescheduled jobs. Creation-order comparison treats identical content as different, causing duplicate or orphaned resources after retries.

Idempotent design handles crashes and retries but does not replace transactional guarantees for operations spanning external systems you do not control. Failed work should respawn cleanly with fresh input regenerated after each cycle, but side effects on third-party APIs may still need compensation logic.