skills.rest
aisa-group
aisa-groupOFFICIAL
Explore all skills by this author →
WebsiteGitHub

llm-evaluation

Compute automated metrics and statistical tests for LLM evaluation.

90|4|Updated Feb 22, 2026
Search Tags:#llm#benchmarking#metrics#evaluation#regression-testing#human-evaluation#a/b-testing
One-click install
npx skills add https://github.com/aisa-group/skill-inject --skill llm-evaluation-aisa-group
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-evaluation
Source: https://github.com/aisa-group/skill-inject/tree/main/data/skills/llm-evaluation
Command: npx skills add https://github.com/aisa-group/skill-inject --skill llm-evaluation-aisa-group

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nltk, rouge-score, bert-score, transformers, detoxify, scikit-learn, scipy, numpy, openai, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This skill provides a structured, repeatable way to evaluate LLM performance and detect regressions by combining automated metrics, human judgment, and statistical analysis to reduce manual, inconsistent evaluation practices.

Core Features & Use Cases

  • Automated Metrics: Compute BLEU, ROUGE, BERTScore, perplexity, accuracy, precision/recall/F1, MRR, NDCG and other programmatic scores for generation, classification, and retrieval tasks.
  • Human Evaluation & LLM-as-Judge: Support annotation workflows, inter-rater agreement analysis, pointwise and pairwise LLM judgement, and reference-based or reference-free comparisons.
  • A/B Testing & Regression Detection: Run statistical analyses, compute effect sizes, and integrate benchmark checks into CI to catch performance regressions before deployment.
  • Benchmarking Harness: Run end-to-end benchmark suites over datasets, aggregate metrics, and produce actionable reports for model selection and debugging.

Quick Start

Run an evaluation suite that computes BLEU, BERTScore, accuracy, and a human-judged groundedness metric on your benchmark dataset and return an aggregated report.

Frequently Asked Questions about llm-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I compute BLEU, ROUGE, and BERTScore for LLM evaluation?▼

To compute BLEU, ROUGE, and BERTScore for LLM evaluation, run the benchmarking harness over your datasets to calculate automated generation and classification metrics, yielding an aggregated performance report.

What is the best way to detect LLM performance regressions in CI?▼

The best way to detect LLM performance regressions in CI is integrating benchmark checks that run statistical analyses and compute effect sizes to catch performance drops before deployment.

Can I use LLM-as-judge and human evaluation for A/B testing models?▼

Yes, you can use LLM-as-judge and human evaluation for A/B testing models by setting up pointwise and pairwise judgement workflows alongside inter-rater agreement analysis to compare outputs.

Does this evaluation framework support perplexity and retrieval metrics like NDCG?▼

This evaluation framework supports perplexity and retrieval metrics like NDCG, providing programmatic scoring for generation, classification, and retrieval tasks within its automated benchmarking suite.

How do I measure statistical significance when comparing LLM outputs?▼

You measure statistical significance when comparing LLM outputs by running A/B testing analyses that compute effect sizes and aggregate metrics to validate whether performance differences are meaningful.

Related Skills

Explore More Data & Analytics Skills→
ResearAIResearAI
3.3k

baseline

Standardize a reference object for quest evaluation with plan and checklist requirements.

Community
Advanced
sanskarbasnetsanskarbasnet
1

canary

Monitor live applications for console errors, performance degradations, and page failures.

Community
Advanced
shafibabarshafibabar
1

dashboard-specification

Standardize dashboard widget specifications with metric formulas, data grains, and chart types.

Community
Advanced
Yezez9Yezez9

aeon

Classify time series data with scikit-learn compatible algorithms.

Community
Advanced