skills.rest
phuonghx
phuonghxCOMMUNITY
Explore all skills by this author →
GitHub

llm-evaluation

Build golden datasets and evaluate LLM features with exact match and rubric scoring.

1|Updated Jun 23, 2026
Search Tags:#metrics#NLP#LLM evaluation#regression analysis#golden dataset
One-click install
npx skills add https://github.com/phuonghx/aim-cli --skill llm-evaluation-phuonghx
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-evaluation
Source: https://github.com/phuonghx/aim-cli/tree/main/aim/templates/aim-agents/skills/llm-evaluation
Command: npx skills add https://github.com/phuonghx/aim-cli --skill llm-evaluation-phuonghx

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill helps ensure the quality of LLM features and agents by building evaluation systems with a focus on accuracy, consistency, and regression prevention.

Core Features & Use Cases

  • Golden Dataset Creation: Establish a baseline with a curated collection of input-output pairs for consistent evaluation.
  • Offline & Online Evaluation: Test LLM features using both static datasets and real-world production traffic.
  • Metrics and Analysis: Utilize a range of metrics such as exact match, rubric scoring, and regression analysis to assess performance.
  • Use Case: Before deploying a new LLM feature or model, use this Skill to measure its performance against a golden dataset to ensure it meets quality standards.

Quick Start

Create a golden dataset for your LLM feature and use the skill to evaluate it against the dataset.

Frequently Asked Questions about llm-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate LLM feature quality before deployment?▼

A golden dataset provides a curated collection of input-output pairs that establishes a stable baseline, ensuring consistent measurement and accurate validation for LLM feature evaluations.

How do I prevent regressions when updating NLP agents?▼

Offline evaluation tests LLM features using static datasets, while online evaluation tests them using real-world production traffic, ensuring both baseline consistency and real-world reliability.

What metrics are used for LLM evaluation?▼

Exact match verifies precise string alignment, while rubric scoring assesses subjective quality against defined criteria, together providing comprehensive coverage for NLP evaluation metrics.

Can I test LLM features with real-world production traffic?▼

Yes, online evaluation allows you to test LLM features using real-world production traffic, complementing offline static dataset testing to ensure features perform reliably in live environments.

Do I need a stable measurement system for NLP feature validation?▼

Yes, a stable and accurate measurement system is required for NLP feature validation, providing the consistent baseline needed to apply metrics and handle both offline and online evaluations successfully.

Related Skills

Explore More Software Engineering Skills→
Accelerated-InnovationAccelerated-Innovation
20

eval-suite-planning

Plan LLM evaluation test suites from feature specs and architecture guidance.

Official
Advanced
DhumitechDhumitech

agent-evaluation

Automate AI agent evaluation and benchmarking with repeatable test pipelines.

Community
Advanced
DevelopersGlobalDevelopersGlobal
65

ci-cd-pipelines

Automate CI/CD pipelines with quality gates and deployment processes.

Official
Advanced
yo-stevenyo-steven

byob

Convert JSON, CSV, or Hugging Face datasets into BYOB evaluation benchmarks with custom scorers.

Community
Advanced