add-cuda-kernel

Adding a CUDA kernel to FlashInfer requires implementing the kernel, creating a launcher, writing a TVM-FFI binding, generating a JIT module, exposing a Python API, and completing AOT registration. The workflow assumes a standard FlashInfer project layout and requires CUDA plus the FlashInfer toolchain.

Creating a TVM-FFI binding for a custom CUDA kernel involves writing the C++ binding code to bridge the launcher with the Python runtime. This step is part of the standard FlashInfer integration workflow, enabling the custom operation to be callable via the exposed Python API.

No, you do not need external packages beyond CUDA and the FlashInfer toolchain to integrate a custom CUDA op. The workflow operates entirely within a standard FlashInfer project layout, utilizing built-in mechanisms for JIT generation and AOT registration.

FlashInfer JIT generation for newly added CUDA kernels produces the necessary runtime code to compile the kernel on-demand. Following the creation of the TVM-FFI binding and launcher, the JIT generator ensures the custom op integrates cleanly with the FlashInfer build and deployment tooling.

AOT registration for FlashInfer CUDA kernels should be used to pre-compile custom operations into the deployment binary. This step, which follows JIT generation and Python API exposure, ensures the custom op is immediately available without runtime compilation overhead.