principle-make-operations-idempotent

Make an operation idempotent by ensuring it converges to the same end state no matter how many times it runs. Scan for existing state at startup, clean stale artifacts, adopt live sessions, and add a reconciliation step wherever partial state could change the next run's outcome.

Ask three questions: what happens if it runs twice in a row, what happens if the previous run crashed at every possible point, and does re-execution converge to the same end state. If any answer depends on leftover state, the operation needs a reconciliation step.

A stale lock is a lock file left behind by a process that crashed before releasing it. Use PID-based stale lock detection: record the owning process ID in the lock and check whether that process is still alive before treating the lock as valid.

Retries cause bugs when partial state from a failed run changes the outcome of the next run, such as duplicate records or skipped work. Idempotent scheduling fixes this by respawning failed work cleanly and regenerating fresh input after each cycle.

Use content-based cleanup whenever artifacts may be created in varying order across runs, such as after crashes or parallel execution. Comparing by content equivalence avoids deleting or keeping the wrong items when creation order is unreliable.