llm-evaluation

Creates customizable evaluation pipelines for LLM outputs with quantitative and qualitative metrics.

1|Updated Apr 14, 2026

Search Tags:#llm #benchmarking #metrics #evaluation #quality-assurance #ab-testing #human-evaluation

Ready to run

npx skills add https://github.com/Sumeet138/qwen-code-agents --skill llm-evaluation-sumeet138

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: llm-evaluation
Source: https://github.com/Sumeet138/qwen-code-agents/tree/main/plugins/llm-application-dev/skills/llm-evaluation
Command: npx skills add https://github.com/Sumeet138/qwen-code-agents --skill llm-evaluation-sumeet138

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Automates comprehensive evaluation of LLM applications by combining automated metrics, human feedback, and benchmarking.

Core Features & Use Cases

Automated Metrics: text generation metrics (BLEU, ROUGE, METEOR, BERTScore), classification metrics (Accuracy, Precision/Recall/F1, AUC-ROC), retrieval metrics (MRR, NDCG, Precision@K, Recall@K).
Human Evaluation: dimensions such as accuracy, coherence, relevance, fluency, safety, usefulness; supports rubric creation.
LLM-as-Judge & Benchmarking: structured evaluation patterns (single-output, pairwise, reference-based); AB testing and regression checks.

Quick Start

Create an evaluation suite with your models, datasets, and metrics, then run it to generate metrics, comparisons, and reports.

Frequently Asked Questions about llm-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I evaluate LLM applications using automated metrics and human feedback?▼

To evaluate LLM applications, combine automated metrics like BLEU, ROUGE, and BERTScore with human evaluation dimensions such as accuracy, fluency, and safety to generate structured comparisons and error analysis reports.

What is the best way to benchmark LLM performance for prompt optimization and model selection?▼

Benchmarking LLM performance for model selection involves running structured evaluation patterns like single-output, pairwise, and reference-based comparisons, alongside A/B testing and regression checks to identify optimal configurations.

Can I assess retrieval-augmented generation pipelines using MRR and NDCG metrics?▼

Yes, you can assess retrieval pipelines by calculating retrieval metrics including MRR, NDCG, Precision@K, and Recall@K to measure the accuracy and ranking quality of retrieved contexts.

How do I set up an LLM evaluation suite for text generation and classification tasks?▼

Set up an LLM evaluation suite by defining your models, datasets, and target metrics, then execute the pipeline to generate automated text generation and classification scores, human evaluation rubrics, and comprehensive reports.

Does LLM-as-Judge work for production-quality QA and regression testing?▼

LLM-as-Judge applies to production-quality QA by automating regression checks and structured evaluations, providing consistent scoring across text generation, classification, and retrieval pipelines without requiring manual review.