skills.rest
SanketAdlak
SanketAdlakCOMMUNITY
Explore all skills by this author →
Website𝕏@devsanket14GitHub

llm-evaluation

Implement evaluation frameworks for LLM applications using automated metrics, LLM judges, and A/B testing.

Updated Apr 23, 2026
Search Tags:#llm-as-judge#benchmarking#llm evaluation#regression testing#a/b testing#bertscore#langsmith
One-click install
npx skills add https://github.com/SanketAdlak/PDMProjectDesign --skill llm-evaluation-sanketadlak
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-evaluation
Source: https://github.com/SanketAdlak/PDMProjectDesign/tree/main/.agents/skills/llm-evaluation
Command: npx skills add https://github.com/SanketAdlak/PDMProjectDesign --skill llm-evaluation-sanketadlak

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nltk, rouge-score, bert-score, transformers, detoxify, anthropic, pydantic, scipy, scikit-learn, numpy, langsmith.

What problem does it solve? Measuring whether an LLM application actually works well is hard: manual spot-checks don't scale, regressions slip into production, and prompt or model changes lack objective validation. This Skill provides a complete evaluation toolkit covering automated metrics, human annotation, LLM-as-judge scoring, and statistical A/B testing. ## Core Features & Use Cases - Automated Metrics: Compute BLEU, ROUGE, BERTScore, perplexity, retrieval metrics (MRR, NDCG, Precision@K), and custom checks like groundedness, toxicity, and factuality. - LLM-as-Judge: Use Claude for pointwise quality ratings, pairwise response comparisons, and reference-based evaluation with structured Pydantic outputs. - Human Evaluation & A/B Testing: Build annotation forms, measure inter-rater agreement with Cohen's kappa, and run t-tests with Cohen's d effect sizes to detect statistically significant differences. - Regression Detection & Benchmarking: Compare new results against baselines with configurable thresholds and integrate with LangSmith for dataset-driven experiments. - Use Case: Before deploying a prompt change, run your test suite through the EvaluationSuite, compare variant scores with the ABTest analyzer, and block the release if the RegressionDetector flags a metric drop beyond your threshold. ## Quick Start Evaluate my LLM's answers on these test cases using accuracy, BERTScore, and an LLM judge, then tell me if the new prompt version is a statistically significant improvement over the baseline.

Frequently Asked Questions about llm-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate LLM outputs with automated metrics?▼

Use the EvaluationSuite class with Metric objects for accuracy, BLEU, ROUGE, or BERTScore. Pass test cases containing input, expected output, and optional context, and the suite returns mean scores plus raw per-example results.

What is LLM-as-judge and when should I use it?▼

LLM-as-judge uses a stronger model like Claude to score outputs on accuracy, helpfulness, and clarity, either pointwise, pairwise, or against a reference. Use it when automated n-gram metrics cannot capture semantic quality or factual correctness.

How do I detect LLM performance regressions before deployment?▼

Store baseline metric results and run new outputs through the RegressionDetector with a relative-change threshold such as 0.05. It flags any metric whose score dropped beyond the threshold and reports the baseline, current value, and change.

BLEU vs BERTScore for evaluating text generation?▼

BLEU measures n-gram overlap and suits translation-style tasks, while BERTScore compares contextual embeddings and captures semantic similarity better. The Skill implements both, with BERTScore using the deberta-xlarge-mnli model.

How do I know if an A/B test result is statistically significant?▼

The ABTest class runs an independent t-test on the two variants' scores and reports the p-value, significance at your alpha level, and Cohen's d effect size. A result is significant when the p-value falls below alpha, typically 0.05.

Can I integrate LLM evaluation with LangSmith?▼

Yes. The Skill shows how to create a LangSmith dataset, attach evaluators like qa and context_qa, and run experiments with the evaluate function. Results include aggregate metrics tagged with model and version metadata.

Related Skills

Explore More Software Engineering Skills→
davicqueirozdavicqueiroz

RAG Evaluation Skill

Evaluate RAG retrieval, generation, and latency metrics against defined targets.

Community
Advanced
proffesor-for-testingproffesor-for-testing
436

quality-metrics

Compute DORA and defect metrics to configure quality dashboards and gates.

Community
Intermediate
Arize-aiArize-ai
11.3k

experiments

Run, read, and compare dataset-backed experiments to evaluate prompt and pipeline changes.

Official
Advanced
wpsadiwpsadi

langsmith-dataset

Manage LangSmith datasets via CLI for AI model evaluation.

Community
Advanced