skills.rest
NVIDIA-NeMo
NVIDIA-NeMoOFFICIAL
Explore all skills by this author →
WebsiteGitHub

nemo-automodel-launcher-config

Configure NeMo AutoModel job launches for interactive, Slurm, and SkyPilot execution.

960|316|Updated May 21, 2025
Search Tags:#distributed training#slurm#skypilot#yaml configuration#nemo-automodel#nsys profiling#job launcher
One-click install
npx skills add https://github.com/NVIDIA-NeMo/Automodel --skill nemo-automodel-launcher-config-nvidia-nemo
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-automodel-launcher-config
Source: https://github.com/NVIDIA-NeMo/Automodel/tree/main/skills/nemo-automodel-launcher-config
Command: npx skills add https://github.com/NVIDIA-NeMo/Automodel --skill nemo-automodel-launcher-config-nvidia-nemo

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Configuring distributed training job launches across interactive torchrun sessions, Slurm HPC clusters, and SkyPilot cloud environments requires knowing the correct YAML fields, rendezvous settings, and profiling options, which are easy to get wrong. ## Core Features & Use Cases - Slurm Configuration: Generate slurm: YAML blocks with nodes, ntasks_per_node, time, account/partition, container_image, mounts, and env_vars, with automatic WORLD_SIZE, MASTER_ADDR, and MASTER_PORT handling. - SkyPilot Cloud Launches: Configure cloud-agnostic jobs with accelerators, spot instances, disk size, region, and setup commands, including spot preemption checkpointing guidance. - Nsight Systems Profiling: Enable nsys_enabled: true on Slurm jobs to produce .nsys-rep reports for diagnostic performance analysis. - Use Case: A user needs to launch a two-node finetuning job with 8 GPUs per node on Slurm; the skill provides the exact YAML section and explains distributed environment setup. ## Quick Start Ask how to configure a Slurm or SkyPilot launch for a NeMo AutoModel training job and receive the correct YAML configuration. ## Quick Start Ask how to configure a two-node Slurm finetuning job with 8 GPUs per node and get the complete slurm YAML section.

Frequently Asked Questions about nemo-automodel-launcher-config

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I configure a multi-node Slurm job for NeMo AutoModel?▼

Add a slurm YAML section with job_name, nodes, ntasks_per_node, time, account or partition, container_image, hf_home, and optional extra_mounts and env_vars. The launcher derives WORLD_SIZE as nodes times ntasks_per_node and sets MASTER_ADDR and MASTER_PORT automatically.

How do I launch NeMo AutoModel on SkyPilot with spot instances?▼

Add a skypilot YAML section with cloud, accelerators such as H100:8, num_nodes, use_spot: true, disk_size, region, setup, and env_vars. Set a short step_scheduler.checkpoint_interval and resume with restore_from.path since spot instances can be preempted.

How do I enable Nsight Systems profiling on a Slurm job?▼

Set nsys_enabled: true in the slurm YAML section alongside normal Slurm fields. The launcher wraps the training command with nsys profile and writes a .nsys-rep report file in the job working or output directory.

Does interactive mode need a YAML launcher section?▼

No. Interactive mode runs torchrun on the current node with automodel config.yaml --nproc-per-node N, and the CLI routes to torchrun automatically when no slurm or skypilot section is present in the config.

Why does my Slurm job fail with port or mount errors?▼

Port collisions occur when the default master_port 13742 is already in use, so change it. Mount failures happen when an extra_mounts source path does not exist on all allocated nodes, causing container startup errors.

When should I not use Nsight Systems profiling?▼

Profiling is diagnostic-only and adds overhead plus large artifacts. Use it for short investigation runs and disable it for normal production training.

Related Skills

Explore More Software Engineering Skills→
sayalinvidiasayalinvidia

nemo-automodel-launcher-config

Configure NeMo AutoModel launches across interactive, Slurm, and SkyPilot environments.

Community
Advanced
NVIDIANVIDIA
3.2k

nemo-automodel-launcher-config

Configure NeMo AutoModel job launches for interactive, Slurm, and SkyPilot execution.

Official
Intermediate
NVIDIA-NeMoNVIDIA-NeMo
896

nemo-mbridge-perf-nsys-analysis

Diagnose Megatron Bridge training bottlenecks from Nsight Systems traces using critical-path analysis.

Official
Advanced
monoesmonoes
15

pair-programming

Coordinate role-based pair programming sessions with continuous code review and testing.

Official
Advanced