skills.rest
kalilurrahman
kalilurrahmanCOMMUNITY
Explore all skills by this author →
Website𝕏@krahmanGitHub

llm-evaluation

Evaluate LLM outputs with automated metrics, human rubrics, and adversarial testing.

1|Updated Mar 21, 2026
Search Tags:#llm-as-judge#llm evaluation#regression detection#adversarial testing#bertscore#rouge#hallucination testing
One-click install
npx skills add https://github.com/kalilurrahman/kr-claudiator-skills-original-prompts --skill llm-evaluation-kalilurrahman
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-evaluation
Source: https://github.com/kalilurrahman/kr-claudiator-skills-original-prompts/tree/main/04-ai-ml/llm-evaluation
Command: npx skills add https://github.com/kalilurrahman/kr-claudiator-skills-original-prompts --skill llm-evaluation-kalilurrahman

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires openai, nltk, rouge-score, bert-score, numpy.

What problem does it solve? Measuring LLM quality by intuition leads to unreliable deployments and undetected regressions. This Skill provides a systematic evaluation framework combining automated metrics, human evaluation rubrics, and adversarial testing so you can quantify model quality and catch failures before users do. ## Core Features & Use Cases - Automated Metrics: Compute exact match, F1, BLEU, ROUGE-L, and BERTScore against ground-truth test sets, plus LLM-as-judge scoring with GPT-4 for nuanced criteria like helpfulness. - Human Evaluation: Run pairwise A/B comparisons and Likert-scale ratings across accuracy, completeness, clarity, and conciseness dimensions. - Adversarial & Factuality Testing: Probe jailbreak resistance, hallucination rates on unanswerable questions, bias patterns, and claim-level factual verification. - Use Case: Before swapping GPT-3.5 for GPT-4 in a customer support bot, run a 500-example test set through the eval harness, compare category-level F1 scores, and use the regression detector to confirm no capability dropped more than 5%. ## Quick Start Build an evaluation harness for my customer support chatbot with a test set covering common queries, edge cases, and adversarial prompts, then report automated metrics and regression risks.

Frequently Asked Questions about llm-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate LLM output quality automatically?▼

Build a test set with ground-truth answers, then compute metrics like exact match, token-overlap F1, ROUGE-L, and BERTScore for semantic similarity. For subjective qualities like helpfulness, use LLM-as-judge prompting with a scored rubric.

What is the difference between BLEU, ROUGE, and BERTScore?▼

BLEU measures n-gram precision against references, common in translation. ROUGE-L measures longest common subsequence overlap. BERTScore compares embedding similarity, catching semantic equivalence even when wording differs significantly.

How do I test if an LLM hallucinates answers?▼

Ask unanswerable questions, such as private facts or unknowable data, and check whether the model admits uncertainty with phrases like "I don't know." Counting fabricated responses gives a hallucination rate.

When should I use human evaluation instead of automated metrics?▼

Use human evaluation for nuanced criteria like helpfulness, clarity, and tone that string-overlap metrics cannot capture. Pairwise A/B comparison is more reliable than absolute ratings, and at least 100 examples are needed for statistical significance.

How do I detect regressions when upgrading LLM models?▼

Store baseline per-category metric results, then run the same test set against the new model. Flag any category where F1 drops beyond a threshold such as 0.05, since aggregate scores can hide category-level degradation.

Related Skills

Explore More Software Engineering Skills→
DigiStrique-SolutionsDigiStrique-Solutions
1

eval-harness

Automate AI prompt and output evaluation against ground-truth fixtures.

Official
Advanced
agenticgogolagenticgogol
1

production-monitor-setup

Generates a production monitoring configuration for deployed agents using tracing tools and calibrated judges.

Community
Intermediate
coolbeevipcoolbeevip
8

team-skill-evolve

Analyze feedback and execution logs to propose skill library improvements.

Community
Advanced
DoanNgocCuongDoanNgocCuong

evaluating-llms-harness

Evaluate LLMs against over 60 academic benchmarks like MMLU and GSM8K.

Community
Advanced