debug-cuda-crash

Debug SGLang CUDA crashes by enabling kernel API logging to capture call boundaries, tensor metadata, and crash-safe inputs. Incremental logging levels isolate failures deep in kernel execution where standard logs are sparse.

Use Level 3 kernel API logging to inspect tensor shapes, dtypes, devices, and call wiring. This level captures function boundaries and tensor metadata to diagnose SGLang kernel failures effectively.

Capture crash-safe inputs and dumps for reproduction by enabling Level 10 kernel API logging. This records tensor inputs and metadata into dump directories for post-mortem analysis of CUDA failures.

Yes, kernel API logging detects NaN or Inf numerical issues using Level 5. This level inspects tensor statistics and numerical values to identify anomalies causing CUDA kernel crashes.

Yes, you can redirect kernel API logs per rank in multi-process workflows. Use include and exclude patterns to filter API calls and manage logs efficiently for post-mortem analysis.

The best way to diagnose sparse CUDA kernel failures is structured kernel API logging. It captures incremental levels of function boundaries, tensor metadata, and crash dumps for faster root-cause analysis.