add-reference-tests

Generate and validate pytest tests for flashinfer_trace reference implementations against FlashInfer or SGLang ground truth.

264|47|Updated Jun 30, 2025
One-click install
npx skills add https://github.com/flashinfer-ai/flashinfer-bench --skill add-reference-tests
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: add-reference-tests
Source: https://github.com/flashinfer-ai/flashinfer-bench/tree/main/.claude/skills/add-reference-tests
Command: npx skills add https://github.com/flashinfer-ai/flashinfer-bench --skill add-reference-tests

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill automates the creation and validation of pytest-based tests that compare flashinfer_trace reference implementations against FlashInfer or SGLang ground truth, helping maintain kernel correctness as new op_types are added or existing ones are updated.

Core Features & Use Cases

  • Automated test scaffolding: generates tests under flashinfer_trace/tests/references/ for new definitions and op_types.
  • Ground-truth validation: ensures reference implementations match the chosen ground truth (FlashInfer by default, with SGLang as fallback when needed).
  • Reproducible testing: provides deterministic inputs and standardized test templates to support consistent results across environments.

Quick Start

/add-reference-tests --op-type mla_paged /add-reference-tests --op-type moe /add-reference-tests --op-type gqa_paged /add-reference-tests --op-type rmsnorm

Frequently Asked Questions about add-reference-tests

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I generate pytest tests to validate FlashInfer kernel reference implementations?

To validate FlashInfer kernel reference implementations, you can generate pytest tests automatically under flashinfer_trace/tests/references/ by specifying the target op_type. The generated tests compare your reference implementations against FlashInfer ground truth.

How does ground truth selection work when validating flashinfer_trace reference implementations?

Ground truth validation for flashinfer_trace reference implementations follows a strict priority: FlashInfer is used as the primary ground truth, with SGLang as a fallback when FlashInfer definitions are unavailable.

Can I use this automated test generation for SGLang op_types like mla_paged and moe?

Yes, you can validate SGLang op_types like mla_paged, moe, gqa_paged, and rmsnorm. The test scaffolding generates deterministic inputs and standardized templates to ensure reproducible results across environments.

What is the best way to automate reference test scaffolding for new op_types in kernel validation?

Automating reference test scaffolding for new op_types involves running a command with the specific op_type argument. This generates standardized pytest files that verify reference definitions against the chosen framework ground truth.

Do I need existing test scaffolding in the repository to generate reference tests for kernel validation?

Yes, generating reference tests for kernel validation requires access to the repository's existing test scaffolding. The automated generation relies on this structure to place tests correctly under flashinfer_trace/tests/references/.