palmetto-apptainer-libcuda-fix

To fix missing libcuda.so errors in Apptainer containers on Slurm, patch the sbatch launcher to create a scratch-local CUDA compatibility directory and symlink the host libcuda. You then bind this compatibility path into the container and adjust LD_LIBRARY_PATH for the worker.

vLLM fails to find libcuda inside Palmetto Apptainer jobs because the container image lacks the CUDA driver library, even though the host provides /lib64/libcuda.so.1. The container path used by the worker does not expose the host driver, requiring a bind mount repair.

To bind host CUDA libraries into an Apptainer container for Slurm jobs, patch the sbatch launcher to symlink the host libcuda into a scratch-local CUDA compatibility directory. Bind this compatibility path into the container and adjust LD_LIBRARY_PATH for the worker runs.

You can verify CUDA driver visibility before resubmitting a Slurm job by running a lightweight preflight and a short smoke test. These steps confirm that Python can load libcuda.so and libcuda.so.1 via ctypes.CDLL inside the container.

The metadata does not specify whether patching the sbatch launcher requires root access on Palmetto. It indicates the patch creates a scratch-local CUDA compatibility directory and symlinks the host libcuda, which typically operates within user-level scratch boundaries.

When Apptainer workers cannot load libcuda.so.1 despite host availability, create a CUDA_COMPAT_DIR with a symlink to the host library. Bind this directory into the container, adjust LD_LIBRARY_PATH, and run a smoke validation to confirm driver visibility.