skills.rest
EngineerWithAI
EngineerWithAIOFFICIAL
Explore all skills by this author →
GitHub

llm-evaluation

Evaluate LLM applications with automated metrics, human feedback, and benchmarking.

4|Updated Jan 10, 2026
Search Tags:#testing#quality assurance#llm-as-judge#llm#benchmarking#metrics#evaluation
One-click install
npx skills add https://github.com/EngineerWithAI/engineerwith-agents --skill llm-evaluation-engineerwithai
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-evaluation
Source: https://github.com/EngineerWithAI/engineerwith-agents/tree/main/plugins/llm-application-dev/skills/llm-evaluation
Command: npx skills add https://github.com/EngineerWithAI/engineerwith-agents --skill llm-evaluation-engineerwithai

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nltk, rouge-score, bert-score, transformers, detoxify, scipy, numpy, openai, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill addresses the critical need for robust evaluation of Large Language Model (LLM) applications, ensuring their quality, reliability, and performance before and after deployment.

Core Features & Use Cases

  • Automated Metrics: Implements standard metrics like BLEU, ROUGE, BERTScore, Accuracy, Precision, Recall, F1, MRR, NDCG, and Precision@K for quantitative assessment.
  • Human Evaluation: Provides frameworks for structured human feedback on dimensions like accuracy, coherence, relevance, fluency, safety, and helpfulness.
  • LLM-as-Judge: Enables using powerful LLMs to evaluate outputs from other models, both pointwise and pairwise.
  • A/B Testing & Regression: Facilitates statistical analysis for comparing model variants and detecting performance regressions.
  • Use Case: You've developed a new RAG system and want to ensure its retrieval and generation quality is better than the previous version. Use this Skill to set up an A/B test comparing the two systems using metrics like NDCG and human ratings, and statistically validate the improvement.

Quick Start

Use the llm-evaluation skill to evaluate your model's accuracy and BLEU score against a set of test cases.

Frequently Asked Questions about llm-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate LLM application performance using automated metrics?▼

Evaluate LLM performance using automated metrics like BLEU, ROUGE, BERTScore, Accuracy, Precision, Recall, F1, MRR, NDCG, and Precision@K for quantitative assessment against test cases.

What is the best way to compare two LLM model variants and detect regressions?▼

Compare model variants and detect regressions by running A/B tests with statistical analysis, validating performance improvements using quantitative metrics and human ratings.

Can I use an LLM-as-judge approach to evaluate outputs from other models?▼

Yes, you can use the LLM-as-judge approach to evaluate outputs from other models, applying both pointwise and pairwise evaluations to measure accuracy, coherence, and safety.

Does this LLM evaluation framework support structured human feedback?▼

Yes, the framework supports structured human feedback, providing evaluation dimensions for accuracy, coherence, relevance, fluency, safety, and helpfulness to complement automated metrics.

How do I test retrieval and generation quality for a new RAG system?▼

Test RAG system retrieval and generation quality by setting up an A/B test comparing systems using metrics like NDCG and human ratings to statistically validate improvements.

Related Skills

Explore More Software Engineering Skills→
bugrabilgebugrabilge
10

llm-evaluation

Evaluate LLM applications with automated metrics and human feedback.

Community
Advanced
w95w95
145

statistical-analysis

Apply descriptive statistics, trend analysis, outlier detection, and hypothesis testing to datasets.

Community
Intermediate
daymadedaymade
1.3k

promptfoo-evaluation

Configure and run Promptfoo LLM evaluation workflows with YAML configs and CLI commands.

Community
Advanced
hamelsmuhamelsmu
1.6k

build-review-interface

Generate a browser-based annotation interface for reviewing LLM traces with Pass/Fail feedback.

Community
Intermediate