skills.rest
elastic
elasticOFFICIAL
Explore all skills by this author →
WebsiteGitHub

evals-write-spec

Generate LLM evaluation spec files with datasets, tasks, and evaluators.

21.2k|8.6k|Updated Jan 26, 2013
Search Tags:#testing#llm#evaluation#spec#playwright#datasets#evaluators
One-click install
npx skills add https://github.com/elastic/kibana --skill evals-write-spec
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: evals-write-spec
Source: https://github.com/elastic/kibana/tree/main/.agents/skills/evals-write-spec
Command: npx skills add https://github.com/elastic/kibana --skill evals-write-spec

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill streamlines the creation and maintenance of evaluation specifications for Large Language Models (LLMs), ensuring robust and reproducible testing.

Core Features & Use Cases

  • Spec File Generation: Provides a structured template for defining evaluation suites, including datasets, tasks, and evaluators.
  • Dataset & Evaluator Management: Facilitates the definition and integration of custom datasets and various evaluator types (CODE, LLM-as-Judge, RAG, Trace-Based).
  • Use Case: When developing a new LLM feature, use this Skill to quickly scaffold the necessary evaluation spec, define test cases with expected outputs, and integrate relevant metrics for performance measurement.

Quick Start

Use the evals-write-spec skill to create a new evaluation spec file for testing LLM responses.

Frequently Asked Questions about evals-write-spec

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create an LLM evaluation spec for testing model responses?▼

To create an LLM evaluation spec, you generate a spec file using the @kbn/evals Playwright fixture that defines datasets with typed examples, processing tasks, and various evaluator types to measure performance.

What types of evaluators can I define for LLM testing in a spec file?▼

You can define several evaluator types for LLM testing, including CODE, LLM-as-Judge criteria, correctness, groundedness, RAG, and trace-based metrics within your evaluation spec.

How do I manage datasets and test cases for LLM evaluation?▼

You manage datasets by defining custom test cases with typed examples and expected outputs directly in the evaluation spec file, which then processes inputs through defined tasks for robust and reproducible testing.

Can I use Playwright to set up and tear down LLM evaluation environments?▼

Yes, the evaluation spec uses a Playwright fixture that facilitates both setup and teardown for your LLM evaluation environments, ensuring the testing context is properly initialized and cleaned up.

What is the best way to scaffold an LLM evaluation suite?▼

The best way to scaffold an LLM evaluation suite is to generate a structured spec file template that integrates your custom datasets, processing tasks, and relevant metrics for performance measurement.

Related Skills

Explore More Software Engineering Skills→
elasticelastic
21.2k

evals-create-suite

Scaffold LLM evaluation suite packages with Playwright configuration files.

Official
Intermediate
HemantSudarshanHemantSudarshan

agent-evaluation

Automate evaluation and benchmarking of AI agents and LLM-powered systems.

Community
Advanced
copilot-ldcopilot-ld
1

libeval

Evaluate RAG agent performance with LLM-as-judge criteria, recall, and trace analyses.

Official
Advanced
plurigridplurigrid
60

code-review

Automate code review for security, quality, and performance issues in pull requests.

Official
Intermediate