llm-evaluation-metrics

Select task-specific metrics and design human evaluation protocols for LLM performance.

14|3|Updated Oct 28, 2025

Search Tags:#summarization #A/B testing #metrics #RAG #classification #LLM evaluation #human evaluation

One-click install

npx skills add https://github.com/tachyon-beep/skillpacks --skill llm-evaluation-metrics

Or copy as Structured Prompt for Agent▼

Please help me install this Agent Skill.
Skill: llm-evaluation-metrics
Source: https://github.com/tachyon-beep/skillpacks/tree/main/plugins/yzmir-llm-specialist/skills/llm-evaluation-metrics
Command: npx skills add https://github.com/tachyon-beep/skillpacks --skill llm-evaluation-metrics

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires scikit-learn, numpy, nltk, rouge, bert-score, torch, transformers, scipy.

What problem does it solve?

This Skill provides a comprehensive framework for evaluating LLM performance across various tasks (classification, generation, RAG, summarization, chat). It ensures you use the right metrics, combine automated and human evaluation, and conduct rigorous A/B testing, preventing you from shipping underperforming or unsafe LLM applications.

Core Features & Use Cases

Task-Specific Metric Selection: Choose appropriate metrics like F1, BLEU, ROUGE, BERTScore, MRR, and Faithfulness based on your LLM's function.
Human Evaluation Protocol: Design robust human evaluation studies to assess fluency, relevance, helpfulness, and safety, capturing nuances automated metrics miss.
Use Case: You've fine-tuned an LLM for customer support and need to prove its effectiveness. This skill guides you to set up an A/B test, define key business metrics (CSAT, completion rate), and perform statistical significance testing before full deployment.

Quick Start

I need to evaluate my LLM's performance for a summarization task. What metrics should I use?