skills.rest
malinovskiy-makar
malinovskiy-makarCOMMUNITY
Explore all skills by this author →
GitHub

llm-evaluation

Implement evaluation frameworks for LLM applications using automated metrics, LLM-as-judge, and A/B testing.

Updated Jun 13, 2026
Search Tags:#llm-as-judge#benchmarking#llm evaluation#regression testing#a/b testing#bertscore#rag metrics
One-click install
npx skills add https://github.com/malinovskiy-makar/qls --skill llm-evaluation-malinovskiy-makar
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-evaluation
Source: https://github.com/malinovskiy-makar/qls/tree/main/.claude/skills/llm-evaluation
Command: npx skills add https://github.com/malinovskiy-makar/qls --skill llm-evaluation-malinovskiy-makar

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Measuring whether an LLM application actually works well is hard: manual spot-checks don't scale, regressions slip into production unnoticed, and comparing models or prompts lacks rigor. This Skill provides a systematic evaluation framework covering automated metrics, human evaluation, LLM-as-judge, and statistical A/B testing. ## Core Features & Use Cases - Automated Metrics: Compute BLEU, ROUGE, BERTScore, perplexity, classification metrics, and RAG retrieval metrics (MRR, NDCG, Precision@K) with ready-to-use Python implementations. - LLM-as-Judge: Score responses with pointwise, pairwise, and reference-based judging using structured Pydantic outputs from Claude. - Human Evaluation & A/B Testing: Build annotation frameworks with inter-rater agreement (Cohen's kappa), run statistically valid A/B tests with t-tests and Cohen's d, and detect regressions against baselines. - Use Case: Before deploying a prompt change, run your test dataset through the EvaluationSuite, compare variants with the ABTest analyzer, and block the release if the RegressionDetector flags a metric drop beyond your threshold. ## Quick Start Ask the AI to build an evaluation suite that scores your model's answers on a test dataset using accuracy, BERTScore, and an LLM judge, then report aggregate results.

Frequently Asked Questions about llm-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate LLM outputs automatically in Python?▼

Build an EvaluationSuite with Metric objects wrapping functions like BLEU, ROUGE, or BERTScore, then run it over test cases containing inputs and expected references. The suite aggregates per-metric means across all predictions.

What is LLM-as-judge and when should I use it?▼

LLM-as-judge uses a stronger model like Claude to score weaker model outputs on accuracy, helpfulness, and clarity. Use it when automated n-gram metrics miss semantic quality, choosing pointwise, pairwise, or reference-based judging depending on whether a gold standard exists.

Which metrics should I use for RAG evaluation?▼

For retrieval quality use MRR, NDCG, Precision@K, and Recall@K. For generated answers, combine groundedness checks using NLI entailment models with reference-based LLM judging for factual accuracy and completeness.

How do I detect LLM performance regressions before deployment?▼

Store baseline metric results, then run new results through a RegressionDetector that computes relative change per metric. Any metric dropping beyond your threshold (default 5%) is flagged as a regression.

How do I know if an A/B test result is statistically significant?▼

Collect scores for both variants and run an independent t-test to get a p-value, plus Cohen's d for effect size. A result is significant when the p-value falls below your alpha (typically 0.05), with Cohen's d indicating whether the effect is small, medium, or large.

What are the limitations of BLEU and ROUGE for LLM evaluation?▼

BLEU and ROUGE measure n-gram overlap, so they penalize correct answers phrased differently from the reference. For open-ended generation, complement them with embedding-based metrics like BERTScore or LLM-as-judge scoring.

Related Skills

Explore More Software Engineering Skills→
Arize-aiArize-ai
11.3k

datasets

Reason about Phoenix dataset examples, outputs, splits, and their role in evaluators and experiments.

Official
Intermediate
pgm1980pgm1980

kaizen:plan-do-check-act

Guide iterative process improvement through Plan-Do-Check-Act experiment cycles.

Community
Basic
supercent-iosupercent-io

langsmith

Instrument LLM pipelines with tracing and run evaluations via LangSmith.

Official
Advanced
tadod12tadod12

evaluating-llms-harness

Benchmark LLMs across standard tasks using lm-evaluation-harness.

Community
Advanced