benchmark-kernel

You can benchmark FlashInfer kernels by applying CUPTI Python to measure hardware-level GPU execution times across routines like Attention, GEMM, and MOE. The tool generates reproducible CSV outputs and example command lines for comparison.

Measuring accurate GPU execution times for FlashInfer kernels requires CUPTI-based timing to capture hardware-level performance. This approach compares multiple kernel backends across workloads like BatchDecodeWithPagedKVCacheWrapper and bmm_fp8 with high precision.

CUPTI Python is required for hardware-level timing accuracy with CUDA toolkit support, but the benchmarking process automatically falls back to CUDA events if CUPTI is unavailable on your system.

Comparing multiple FlashInfer backends for workloads like Attention, GEMM, and MOE is supported by running the benchmark suite across specific test cases such as trtllm_fp8_block_scale_moe. It produces reproducible results to evaluate runtimes and throughput.

CUPTI timing works by capturing hardware-level GPU execution times during kernel benchmarking, providing higher accuracy than software timers. It enables direct comparison of runtimes and throughput across different FlashInfer backends.

The CUDA events fallback activates when CUPTI is unavailable, allowing FlashInfer benchmarking to proceed but without the same hardware-level timing accuracy. Use CUPTI Python with the CUDA toolkit for the most precise GPU execution measurements.