skills.rest
KrxGu
KrxGuCOMMUNITY
Explore all skills by this author →
Website𝕏@krishgupta72GitHub

Write an INT8 Quantized Kernel

Design INT8 quantized GEMM kernels with dp4a and INT32 accumulation for NVIDIA GPUs.

54|7|Updated Apr 10, 2026
Search Tags:#quantization#cuda#gemm#int8#cutlass#dp4a#cublas
One-click install
npx skills add https://github.com/KrxGu/kernel-skills --skill write-an-int8-quantized-kernel
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: Write an INT8 Quantized Kernel
Source: https://github.com/KrxGu/kernel-skills/tree/main/skills/quantization/write-int8-quantized-kernel
Command: npx skills add https://github.com/KrxGu/kernel-skills --skill write-an-int8-quantized-kernel

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill guides the design and implementation of INT8 quantized GEMM or linear-layer kernels for inference, ensuring correctness and high throughput while avoiding common quantization and accumulation mistakes.

Core Features & Use Cases

  • Quantization scheme guidance: explicit symmetric and asymmetric formulas, granularity choices (per-tensor, per-channel, per-token), and scale/zero-point computation rules.
  • Correct accumulation and epilogue design: enforce INT32 accumulation, dp4a inner loop usage, zero-point correction, and dequantization into FP32/FP16.
  • Implementation and evaluation checklist: kernel structure, handling K tails, accuracy tests vs FP32, and an assessment of cuBLAS/CUTLASS vs custom dp4a kernels for production use.
  • Use case: implement an INT8 inference GEMM on Turing/Ampere-class GPUs where weights and activations are quantized and per-channel dequantization is required.

Quick Start

Implement an INT8 GEMM using dp4a with INT32 accumulation, apply correct per-channel or per-tensor scales in the epilogue, and validate against an FP32 reference for MAE and max error.

Frequently Asked Questions about Write an INT8 Quantized Kernel

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I implement an INT8 quantized GEMM kernel using dp4a on Turing GPUs?▼

To implement an INT8 quantized GEMM kernel using dp4a, you must use INT32 accumulation, handle K dimensions as multiples of 4, apply zero-point corrections, and validate against an FP32 reference.

What is the correct way to handle zero-point correction and dequantization in INT8 inference?▼

Correct INT8 inference zero-point correction requires applying per-channel or per-tensor scales during the dequantization epilogue, converting INT32 accumulations into FP32 or FP16 outputs.

Should I use cuBLAS, CUTLASS, or a custom dp4a kernel for INT8 inference?▼

Evaluating cuBLAS or CUTLASS versus a custom dp4a kernel depends on your specific throughput needs, with custom kernels offering tailored zero-point correction and epilogue design for production.

How do I manage K tails and accuracy when writing INT8 quantized CUDA kernels?▼

Managing K tails in INT8 quantized CUDA kernels requires structuring the dp4a inner loop carefully and running accuracy tests against FP32 references to measure MAE and max error.

Does INT8 quantized GEMM support per-channel and per-token granularity on Ampere GPUs?▼

INT8 quantized GEMM supports per-tensor, per-channel, and per-token granularity on Ampere GPUs, allowing flexible scale and zero-point computation for optimized inference layers.

Related Skills

Explore More Software Engineering Skills→
spacecat2002spacecat2002

debug-cuda-crash

Diagnose CUDA crashes in SGLang by logging tensors at kernel API boundaries.

Community
Advanced
richardnguyen0715richardnguyen0715

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for high throughput and low latency.

Community
Advanced
KrxGuKrxGu
54

Write CUDA GEMM Kernel

Design and implement a CUDA GEMM kernel with FP32 accumulation and tiling.

Community
Advanced
mattmirelesmattmireles
57

coreml-validate

Validate CoreML model numerical accuracy against PyTorch reference models.

Community
Advanced