add-cuda-kernel

Name: add-cuda-kernel
Availability: InStock
Author: FlashInfer

Add CUDA kernels to FlashInfer with launchers, TVM-FFI bindings, and JIT modules.

6.1k|1.3k|Updated Jul 22, 2023

Search Tags:#benchmark #kernel #bindings #cuda #jit #flashinfer #tvm-ffi

Ready to run

npx skills add https://github.com/flashinfer-ai/flashinfer --skill add-cuda-kernel-flashinfer-ai

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: add-cuda-kernel
Source: https://github.com/flashinfer-ai/flashinfer/tree/main/.claude/skills/add-cuda-kernel
Command: npx skills add https://github.com/flashinfer-ai/flashinfer --skill add-cuda-kernel-flashinfer-ai

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This instructional Skill provides a complete, end-to-end workflow for adding new CUDA kernels to FlashInfer, from kernel definition to deployment-ready integration.

Core Features & Use Cases

Kernel definition: implement a CUDA kernel in include/flashinfer for a new operation.
Launcher & binding: create a host launcher in csrc and a TVM-FFI binding to expose the kernel to Python and TVM.
JIT generation & Python API: generate JIT modules and provide a high-level Python API to users.
Testing & benchmarking: add unit tests and benchmarks to validate correctness and performance across CUDA architectures.
AOT registration: register the module for ahead-of-time compilation to improve startup latency and reuse.

Quick Start

Follow this end-to-end workflow to add a CUDA kernel: implement the kernel in include/flashinfer, create a launcher and TVM-FFI binding, build a JIT module, expose a Python API, write tests and benchmarks, and register the module for AOT.

Frequently Asked Questions about add-cuda-kernel

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I add a new CUDA kernel to FlashInfer?▼

Adding a CUDA kernel to FlashInfer involves an end-to-end workflow: implementing the kernel, creating a host launcher, setting up TVM-FFI bindings, generating JIT modules, exposing a Python API, and registering for ahead-of-time compilation.

How does TVM-FFI binding expose CUDA kernels to Python?▼

TVM-FFI binding exposes CUDA kernels to Python by creating a host launcher in the csrc directory that bridges the C++ implementation with the TVM runtime, enabling direct invocation and testing from Python APIs.

What is the process for generating JIT modules in FlashInfer?▼

Generating JIT modules in FlashInfer involves building runtime compilation artifacts after defining the CUDA kernel and TVM-FFI bindings, allowing dynamic compilation and seamless Python API exposure for usage.

Do I need to write tests and benchmarks when integrating a CUDA kernel?▼

Yes, you need to write tests and benchmarks when integrating a CUDA kernel to validate correctness and measure performance across various CUDA architectures before achieving production readiness.

Why register FlashInfer modules for ahead-of-time compilation?▼

Registering FlashInfer modules for ahead-of-time compilation improves startup latency and promotes module reuse, ensuring the integrated CUDA kernel is production-ready and avoids runtime compilation overhead.