Write a Triton RMSNorm Kernel

Implement an RMSNorm kernel in Triton by computing y = x * rsqrt(mean(x^2) + eps) * weight. Achieve numerical stability through one-pass sum-of-squares with fp32 accumulation, ensuring accurate normalization for large transformer blocks.

Yes, you can fuse optional residual connections directly into the Triton RMSNorm forward kernel. This fusion streamlines transformer block execution by combining normalization and residual addition within a single GPU kernel pass.

Write a Triton backward kernel for RMSNorm and expose it through a Python wrapper acting as a PyTorch autograd function. This provides forward and backward capability necessary for LLM training and fine-tuning pipelines.

Triton RMSNorm handles large hidden dimensions up to 8192 using masked tail handling. Masking ensures correct computation for hidden dimensions that are not divisible by the GPU kernel block size.

Precision loss in a Triton RMSNorm kernel often stems from missing fp32 accumulation. Summing squares in fp32 inside the kernel prevents overflow and maintains numerical stability across large hidden dimensions.

Yes, RMSNorm requires per-feature weight broadcasting with a shape matching the hidden dimension. The Triton kernel applies this weight vector across features during the normalization pass.