skills.rest
jvsandhu
jvsandhuCOMMUNITY
Explore all skills by this author →
GitHub

llm_evaluation

Implements evaluation frameworks for LLM applications using automated metrics, human feedback, and A/B testing.

Updated Jan 14, 2026
Search Tags:#llm-as-judge#rag#benchmarking#llm evaluation#regression testing#a/b testing#bertscore
One-click install
npx skills add https://github.com/jvsandhu/agentic-skills --skill llm-evaluation-jvsandhu
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm_evaluation
Source: https://github.com/jvsandhu/agentic-skills/tree/main/skills/llm_evaluation
Command: npx skills add https://github.com/jvsandhu/agentic-skills --skill llm-evaluation-jvsandhu

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nltk, rouge-score, bert-score, transformers, detoxify, scikit-learn, scipy, numpy, openai, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve? Measuring the quality of LLM applications is difficult because traditional software tests cannot capture fuzzy qualities like accuracy, coherence, and groundedness. This Skill provides a systematic framework for evaluating LLM outputs so you can detect regressions, compare models, and validate prompt changes before deployment. ## Core Features & Use Cases - Automated Metrics: Compute BLEU, ROUGE, BERTScore, perplexity, and retrieval metrics like MRR and NDCG for RAG pipelines. - LLM-as-Judge: Use stronger models to score responses pointwise, pairwise, or against reference answers. - Human Evaluation & A/B Testing: Structure annotation tasks, measure inter-rater agreement with Cohen's kappa, and run statistically rigorous A/B tests with t-tests and effect sizes. - Use Case: Before shipping a prompt change to your RAG chatbot, run the evaluation suite against a golden dataset of 50 question-answer-context triples, detect any regression in faithfulness, and route low-scoring answers to human reviewers. ## Quick Start Ask the AI to evaluate your LLM application by running automated metrics like BLEU, BERTScore, and groundedness against your test dataset and report any regressions compared to the baseline.

Frequently Asked Questions about llm_evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate LLM application performance automatically?▼

Use automated metrics like BLEU and ROUGE for text overlap, BERTScore for semantic similarity, and custom metrics like groundedness via NLI models. Combine multiple metrics in an evaluation suite run against a representative test dataset for a comprehensive quality view.

What metrics should I use for RAG pipeline evaluation?▼

For retrieval, use MRR, NDCG, Precision@K, and Recall@K to measure ranking quality. For generation, measure faithfulness and groundedness using NLI-based entailment checks against the retrieved context, plus BERTScore for semantic similarity.

How does LLM-as-judge evaluation work?▼

LLM-as-judge uses a stronger model like GPT-5 to score outputs on dimensions such as accuracy, helpfulness, and clarity. It supports pointwise scoring of single responses, pairwise comparison of two responses, and reference-based or reference-free judging with structured JSON output.

How do I detect LLM performance regressions before deployment?▼

Build a regression detector that compares new metric scores against a stored baseline and flags relative decreases beyond a threshold such as 5 percent. Integrate this check into your CI/CD pipeline so every pull request runs the evaluation suite automatically.

How do I measure inter-rater agreement in human evaluation?▼

Use Cohen's kappa from scikit-learn to quantify agreement between two annotators scoring the same responses. Values below 0.2 indicate slight agreement, 0.4 to 0.6 moderate, and above 0.8 almost perfect agreement, helping validate annotation guideline quality.

What are common pitfalls when evaluating LLM outputs?▼

Common pitfalls include optimizing a single metric while ignoring others, drawing conclusions from small sample sizes, testing on training data, and ignoring statistical variance. Always use diverse metrics, representative data, baselines, and proper statistical tests.

Related Skills

Explore More Software Engineering Skills→
marchattonmarchatton
5

evaluate-rag

Evaluate RAG pipelines with Recall@k, Precision@k, MRR, and NDCG@k metrics.

Community
Advanced
take566take566

llmops-operations

Manages the end-to-end lifecycle of LLM applications, including prompt engineering, RAG, evaluation, and deployment.

Community
Advanced
bipinksbipinks
3

ai-evaluation

Benchmark and validate AI models across accuracy, safety, cost, and latency.

Community
Advanced
ancolemanancoleman
503

evaluating-llms

Evaluate LLM systems with automated metrics, LLM-as-judge, and benchmarks.

Community
Advanced