nvidia-resiliency-ext

To simulate GPU failures for resiliency testing, use fault injection tools to trigger rank failures or hangs after a set delay. This verifies that your distributed training framework can detect the error and gracefully recover.

Asynchronous checkpointing saves training states in the background without blocking execution. It prevents data loss in distributed systems by enabling jobs to resume from the last checkpoint after a failure.

Monitor process health and capture execution traces using built-in diagnostics tools. These utilities log process status and analyze execution paths to help debug faults in distributed training frameworks.

These fault tolerance tools leverage standard Python libraries for process management and distributed communication. They are designed to integrate with distributed training frameworks running in NVIDIA computing environments.

The best way to debug rank failures is combining fault injection with monitoring diagnostics. Simulate the failure scenario, capture the execution traces, and verify the checkpointing mechanism resumes the job correctly.

A distributed training job hangs instead of failing when the fault tolerance mechanism lacks proper process health monitoring. Testing fault injection scenarios helps identify these hang conditions and improve recovery logic.