distill-test-set

You distill a noisy dataset by running the skill with your source dataset identifier, which applies deduplication, paraphrase clustering, and failure-weighted sampling to generate a compact, high-signal test suite.

Failure-weighted sampling for AI evaluation is a selection method that prioritizes known agent failures, ensuring the most critical scenarios are included when you distill large datasets into a refined test suite.

You deduplicate conversation logs by removing exact duplicates and clustering paraphrased scenarios while preserving distinct intents, utilizing Python-based heuristics or OpenAI embeddings to ensure representative coverage.

Yes, you can reduce thousands of raw conversation logs into a compact 20-50 case test suite by automatically capping generic fallback responses and selecting a representative, failure-weighted subset for efficient agent evaluation.

No, you do not need OpenAI embeddings to deduplicate test sets; the skill can alternatively utilize Python-based heuristics to ensure representative coverage across intent labels and languages.