multi-node-jax-ffi-debug

To debug multi-node JAX FFI kernel failures on a SLURM cluster, follow a structured six-step sequence covering environment propagation, explicit GPU allocation, distributed initialization, and all-rank synchronization to prevent deadlocks.

JAX distributed initialization deadlocks during collective operations when not all ranks participate in the collectives or when coordinator discovery fails. Ensuring all ranks are enabled in distributed initialization and verifying coordinator resolution with scontrol prevents these synchronization failures.

Porting a single-GPU JAX FFI kernel to multi-node deployment requires addressing device visibility, local_device_count, and rank participation in collectives. Use wrapper scripts for environment setup and ensure reliable per-node configurations to stabilize the distributed run.

The best way to resolve the coordinator node for JAX distributed runs on SLURM is using scontrol commands within robust wrapper scripts. This ensures correct per-node setups and reliable coordinator resolution, preventing initialization failures across the multi-node environment.

shard_map works with multi-node JAX CUDA FFI kernels on SLURM when you explicitly request GPUs, propagate the environment correctly, and ensure all ranks participate in collective synchronization. Properly configuring local_device_count and device visibility is required for stable execution.

local_device_count returns zero in multi-node JAX FFI runs due to incorrect device visibility or missing explicit GPU allocation in the SLURM wrapper script. Requesting GPUs explicitly and ensuring proper environment propagation remediate this device visibility issue.