build-eval-dataset

What problem does it solve?

This Skill addresses the underestimation of creating effective evaluation datasets for LLM apps, providing a structured approach to ensure accurate performance metrics.

Core Features & Use Cases

• Dataset Creation: Build datasets that represent real usage and cover hard cases, essential for reliable LLM app evaluations.
• Data Acquisition: Guide on sourcing data from real production traffic, beta usage, expert input, and LLM-generated cases.
• Layered Approach: Develop datasets in layers (smoke, core, regression) to cover different aspects of LLM app performance.
• Label Quality: Focus on creating clear, versioned, and representative datasets for consistent evaluation.
• Verification: Ensure the dataset catches known regressions and provides a baseline for continuous improvement.

Quick Start

Run the skill with the command "build-eval-dataset create --prompt 'What should I test my LLM on?'"