Write an INT8 Quantized Kernel

What problem does it solve?

This Skill guides the design and implementation of INT8 quantized GEMM or linear-layer kernels for inference, ensuring correctness and high throughput while avoiding common quantization and accumulation mistakes.

Core Features & Use Cases

• Quantization scheme guidance: explicit symmetric and asymmetric formulas, granularity choices (per-tensor, per-channel, per-token), and scale/zero-point computation rules.
• Correct accumulation and epilogue design: enforce INT32 accumulation, dp4a inner loop usage, zero-point correction, and dequantization into FP32/FP16.
• Implementation and evaluation checklist: kernel structure, handling K tails, accuracy tests vs FP32, and an assessment of cuBLAS/CUTLASS vs custom dp4a kernels for production use.
• Use case: implement an INT8 inference GEMM on Turing/Ampere-class GPUs where weights and activations are quantized and per-channel dequantization is required.

Quick Start

Implement an INT8 GEMM using dp4a with INT32 accumulation, apply correct per-channel or per-tensor scales in the epilogue, and validate against an FP32 reference for MAE and max error.