What problem does it solve?
This Skill provides a compact, reproducible standard operating procedure for profiling and optimizing matrix-related operators (GEMV, GEMM, Grouped GEMM) to reach high bandwidth or TFLOPS utilization on NVIDIA Ampere/Hopper GPUs by identifying memory vs compute bottlenecks, enforcing warp-level coalescing, and selecting hardware-aligned tile and pipeline parameters.
Core Features & Use Cases
- Bottleneck classification using arithmetic intensity to decide memory-bound versus compute-bound strategies.
- Memory-bound guidance: warp coalescing rules, shared memory reuse patterns, reduce-thread sizing, and Hopper-specific tips (TMA, cp.async, L2 behavior).
- Compute-bound guidance: tensor-core alignment constraints, shared-memory budgeting, and double-buffered pipelining recommendations.
- Autotune strategy and workflow: recommended search spaces, best practices for warmup/reps, logging, validation, and comparison to cuBLAS/torch baselines.
- Case studies demonstrating real-world gains (e.g., high H200 utilization on large Llama-3 shapes) and pitfalls to avoid.
- Use case example: tune kernels for a production inference workload (large sequence GEMVs) to recover bandwidth and reduce latency.
Quick Start
Use the tune-multiplication guide to compute arithmetic intensity, verify warp-level memory coalescing, design a hardware-aligned autotune search space, and run the autotuner to record the best GEMV/GEMM configuration.
