palmetto-apptainer-libcuda-fix

Name: palmetto-apptainer-libcuda-fix
Availability: InStock
Author: KwongFuk

Patch sbatch launchers to bind host CUDA libraries into Apptainer containers.

Updated Apr 10, 2026

Search Tags:#gpu #container #cuda #slurm #apptainer #palmetto #libcuda

Ready to run

npx skills add https://github.com/KwongFuk/codex-skills --skill palmetto-apptainer-libcuda-fix

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: palmetto-apptainer-libcuda-fix
Source: https://github.com/KwongFuk/codex-skills/tree/main/global/palmetto-apptainer-libcuda-fix
Command: npx skills add https://github.com/KwongFuk/codex-skills --skill palmetto-apptainer-libcuda-fix

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This skill fixes container-side CUDA driver visibility for Palmetto Slurm jobs when Apptainer or vLLM workers fail with a missing libcuda error, even though the host provides /lib64/libcuda.so.1. It enables a safe repair plus a smoke validation step before resubmission.

Core Features & Use Cases

Patch the sbatch launcher to create a scratch-local CUDA compatibility directory and symlink the host libcuda into the container path used by the worker.
Bind the compatibility path into the container and adjust LD_LIBRARY_PATH for the preflight and real runs.
Add a lightweight preflight and a short smoke test to verify that libcuda.so and libcuda.so.1 can be loaded by a Python process inside the container before resubmitting the real job.

Quick Start

Apply the patch to the sbatch launcher and run the included smoke test before resubmitting the original job.

Frequently Asked Questions about palmetto-apptainer-libcuda-fix

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I fix missing libcuda.so errors in Apptainer containers on Slurm?▼

To fix missing libcuda.so errors in Apptainer containers on Slurm, patch the sbatch launcher to create a scratch-local CUDA compatibility directory and symlink the host libcuda. You then bind this compatibility path into the container and adjust LD_LIBRARY_PATH for the worker.

Why does vLLM fail to find libcuda inside my Palmetto Apptainer job?▼

vLLM fails to find libcuda inside Palmetto Apptainer jobs because the container image lacks the CUDA driver library, even though the host provides /lib64/libcuda.so.1. The container path used by the worker does not expose the host driver, requiring a bind mount repair.

How to bind host CUDA libraries into an Apptainer container for Slurm jobs?▼

To bind host CUDA libraries into an Apptainer container for Slurm jobs, patch the sbatch launcher to symlink the host libcuda into a scratch-local CUDA compatibility directory. Bind this compatibility path into the container and adjust LD_LIBRARY_PATH for the worker runs.

Can I verify CUDA driver visibility inside a container before resubmitting a Slurm job?▼

You can verify CUDA driver visibility before resubmitting a Slurm job by running a lightweight preflight and a short smoke test. These steps confirm that Python can load libcuda.so and libcuda.so.1 via ctypes.CDLL inside the container.

Does patching the sbatch launcher require root access on Palmetto?▼

The metadata does not specify whether patching the sbatch launcher requires root access on Palmetto. It indicates the patch creates a scratch-local CUDA compatibility directory and symlinks the host libcuda, which typically operates within user-level scratch boundaries.

What to do when Apptainer workers cannot load libcuda.so.1 despite host availability?▼

When Apptainer workers cannot load libcuda.so.1 despite host availability, create a CUDA_COMPAT_DIR with a symlink to the host library. Bind this directory into the container, adjust LD_LIBRARY_PATH, and run a smoke validation to confirm driver visibility.