skills.rest
FutureMLS-Lab
FutureMLS-LabOFFICIAL
Explore all skills by this author →
GitHub

debug-cuda-crash

Diagnose SGLang CUDA crashes by recording kernel API calls and tensor metadata.

556|81|Updated May 19, 2026
Search Tags:#sglang#compute sanitizer#cuda debugging#kernel logging#gpu crashes#tensor diagnostics#cuda gdb
One-click install
npx skills add https://github.com/FutureMLS-Lab/OSCAR --skill debug-cuda-crash-futuremls-lab
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: debug-cuda-crash
Source: https://github.com/FutureMLS-Lab/OSCAR/tree/main/sglang-research/.claude/skills/debug-cuda-crash
Command: npx skills add https://github.com/FutureMLS-Lab/OSCAR --skill debug-cuda-crash-futuremls-lab

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This skill helps diagnose difficult SGLang CUDA failures such as illegal memory access, device-side assertions, out-of-bounds errors, NaNs, Infs, and out-of-memory crashes by capturing kernel inputs and execution context before the process fails.

Core Features & Use Cases

  • Kernel Boundary Logging: Enable API logging for custom operations, attention, linear, quantization, rotary, and diffusion entry points.
  • Progressive Diagnostics: Select logging levels that expose function names, tensor metadata, numerical statistics, or crash-safe tensor dumps.
  • Crash Reproduction: Build minimal LLM and diffusion reproducers for validating CUDA failure behavior.
  • Distributed Debugging: Separate logs and dump directories across multiple GPU worker processes.
  • Advanced Investigation: Combine captured inputs with compute-sanitizer, cuda-gdb, CUDA synchronization, and device-side printf debugging.
  • Use Case: When an SGLang model encounters a device-side assertion, use level-3 or level-10 logging to identify the last failing API boundary and inspect tensor shapes, devices, dtypes, and values.

Quick Start

Ask the skill to configure SGLang kernel API logging for a reproducible CUDA crash and identify the likely failing tensor or kernel boundary.

Frequently Asked Questions about debug-cuda-crash

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I diagnose SGLang CUDA crashes like illegal memory access or out-of-bounds errors?▼

Diagnose SGLang CUDA crashes by recording kernel API calls, tensor metadata, numerical statistics, and crash-safe inputs before failure. This captures execution context for custom operations, attention, linear, and quantization kernels to pinpoint the failing boundary.

What is the best way to capture tensor metadata when an LLM model hits a device-side assertion?▼

Capture tensor metadata during a device-side assertion by configuring SGLang kernel logging to level-3 or level-10. These levels expose function names, tensor shapes, devices, dtypes, and values to identify the last failing API boundary before the crash.

Can I use compute-sanitizer and cuda-gdb with SGLang kernel logging?▼

You can combine captured SGLang kernel logging evidence with compute-sanitizer, cuda-gdb, CUDA synchronization, and device-side printf debugging. This advanced investigation validates CUDA failure behavior using pre-captured inputs and external sanitizer tools.

Does SGLang kernel logging support multi-GPU execution and distributed debugging?▼

SGLang kernel logging supports distributed debugging across multi-GPU execution by separating logs and dump directories across multiple GPU worker processes. This isolation ensures accurate crash reproduction and evidence collection for each worker.

How do I build a minimal reproducer for SGLang diffusion or LLM CUDA failures?▼

Build minimal LLM and diffusion reproducers for validating CUDA failure behavior by using captured crash-safe tensor dumps and execution context. This isolates the specific custom operations, rotary, or attention kernels causing the crash.

Related Skills

Explore More Software Engineering Skills→
sqjiansqjian

debug-cuda-crash

Log CUDA kernel API calls and inputs to diagnose SGLang crashes.

Community
Advanced
hey-konghey-kong

debug-cuda-crash

Diagnoses CUDA crashes in SGLang via level-based kernel API logging and crash dumps.

Community
Advanced
fw-aifw-ai
2

debug-cuda-crash

Log FlashInfer API inputs to diagnose CUDA crashes.

Official
Advanced
scottgl9scottgl9
12

debug-cuda-crash

Record CUDA kernel calls and tensor states to diagnose GPU errors.

Community
Intermediate