optimize-pa-decode-gluon

Paged attention decode performance on AMD GPUs is optimized by analyzing Triton/Gluon kernels to improve memory access patterns, compute efficiency, and instruction selection for CDNA3 and CDNA4 architectures. This Skill rectifies API-level inefficiencies in tensor management and kernel dispatch to reduce inference latency.

Your paged attention kernel may be slow due to suboptimal memory access patterns, inefficient loop structures, or improper data type usage in the Triton/Gluon implementation. This Skill identifies and rectifies these compute inefficiencies alongside Python API wrapper overhead to increase throughput.

Tuning Triton kernels for AMD CDNA involves optimizing memory access, compute efficiency, and instruction selection within the Gluon implementation. This Skill targets the pa_decode_gluon.py script to fine-tune these parameters, achieving faster response times for large language models on MI300X and MI350 hardware.

Yes, this paged attention decode optimization specifically targets AMD GPU architectures including CDNA3 and CDNA4. It analyzes data type optimizations and loop structures tailored to these hardware platforms to maximize compute efficiency and increase inference throughput.

Yes, this Skill identifies and rectifies Python API wrapper inefficiencies related to kernel dispatch and tensor management. By streamlining the API layer alongside the underlying Triton/Gluon kernel optimizations, it ensures faster dispatch and reduced latency during paged attention decode.

You should optimize the pa_decode_gluon implementation when running large language models on AMD MI300X or MI350 hardware and experiencing high latency or low throughput. This Skill fine-tunes paged attention kernels to address memory access and compute bottlenecks for demanding inference workloads.