Write CUDA LayerNorm Kernel

What problem does it solve?

This Skill provides a correct, high-performance CUDA implementation of LayerNorm (and RMSNorm) kernels with numerically stable mean/variance computation, supporting forward and backward passes, and non-standard normalization axes.

Core Features & Use Cases

• Welford online accumulation: single-pass mean/variance computation for numerical stability.
• RMSNorm variant: optional variant that normalizes by root mean square without mean subtraction.
• Backward support: optional backward pass with saved intermediates for gradient computation.
• Non-standard shapes: handles arbitrary hidden dimensions including non-power-of-two sizes and 3D inputs (N x S x D) with proper per-row normalization.
• FP32 accumulation: all variance and accumulation in FP32 even if inputs are FP16/BF16.

Quick Start

Run the CUDA LayerNorm kernel on a tensor of shape [N, D] or [N, S, D] and verify the output is properly normalized with optional gamma/beta or RMSNorm variant.