skills.rest
scoots31
scoots31COMMUNITY
Explore all skills by this author →
GitHub

llm-evaluation

Implements evaluation frameworks for LLM applications using automated metrics, LLM-as-judge, and A/B testing.

Updated Apr 13, 2026
Search Tags:#llm-as-judge#benchmarking#llm evaluation#regression testing#a/b testing#bertscore#bleu
One-click install
npx skills add https://github.com/scoots31/engineering-playbook --skill llm-evaluation-scoots31
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-evaluation
Source: https://github.com/scoots31/engineering-playbook/tree/main/references/llm-evaluation
Command: npx skills add https://github.com/scoots31/engineering-playbook --skill llm-evaluation-scoots31

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nltk, rouge-score, bert-score, transformers, detoxify, scikit-learn, scipy, numpy, anthropic, pydantic, langsmith.

What problem does it solve? Measuring whether an LLM application actually works well is hard: manual spot-checks don't scale, regressions slip into production unnoticed, and prompt or model changes lack objective evidence of improvement. ## Core Features & Use Cases - Automated Metrics: Compute BLEU, ROUGE, BERTScore, perplexity, retrieval metrics (MRR, NDCG, Precision@K), and custom checks like groundedness, toxicity, and factuality. - LLM-as-Judge & Human Evaluation: Score outputs with pointwise, pairwise, and reference-based judge prompts, plus annotation frameworks with inter-rater agreement (Cohen's kappa). - A/B Testing & Regression Detection: Run statistical comparisons between model variants (t-tests, Cohen's d effect size) and flag metric regressions against baselines before deployment. - Use Case: Before shipping a prompt change, run your test dataset through the evaluation suite, compare scores against the baseline, and block the release if any metric regresses beyond your threshold. ## Quick Start Evaluate my LLM's answers on this test dataset using accuracy, BERTScore, and an LLM judge, then tell me if the new prompt version regressed compared to the baseline.

Frequently Asked Questions about llm-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate LLM outputs automatically in Python?▼

Build an evaluation suite that runs each test case through your model and scores predictions with metrics like BLEU, ROUGE, and BERTScore. Aggregate the scores across the dataset to get mean performance per metric.

What is LLM-as-judge and when should I use it?▼

LLM-as-judge uses a stronger model to score or compare outputs on dimensions like accuracy, helpfulness, and clarity. Use it when automated n-gram metrics miss semantic quality, such as open-ended question answering or summarization.

BLEU vs ROUGE vs BERTScore: which metric should I use?▼

BLEU measures n-gram precision and suits translation, ROUGE measures recall and suits summarization, and BERTScore compares embedding similarity for semantic equivalence. BERTScore correlates better with human judgment on paraphrased answers.

How do I detect LLM performance regressions before deployment?▼

Store baseline metric scores, then re-run the same test dataset after any prompt or model change. Flag a regression when a metric drops beyond a relative threshold, such as five percent, compared to the baseline.

How do I know if an A/B test between two prompts is significant?▼

Collect scores for both variants, then run an independent t-test to get a p-value and compute Cohen's d for effect size. A result is statistically significant when the p-value falls below your alpha, typically 0.05.

What are the limitations of automated LLM evaluation metrics?▼

N-gram metrics like BLEU penalize valid paraphrases and miss factual errors, while LLM judges can carry biases toward longer or familiar-style responses. Combine automated metrics with human evaluation and measure inter-rater agreement using Cohen's kappa.

Related Skills

Explore More Software Engineering Skills→
mberto10mberto10

langfuse-dataset-management

Create Langfuse datasets and add traces for regression testing.

Community
Intermediate
reaatechreaatech
2

Regression Gates Skill

Detect model metric regressions and block CI/CD deployments.

Community
Advanced
OpenLAIROpenLAIR
1.0k

phoenix-observability

Trace, evaluate, and monitor LLM applications with OpenTelemetry and self-hosted storage.

Official
Advanced
wpsadiwpsadi

langsmith-evaluator

Build LangSmith evaluation pipelines with LLM-as-Judge and custom code.

Community
Advanced