skills.rest
Jhabbig
JhabbigCOMMUNITY
Explore all skills by this author →
GitHub

llm-evaluation

Design evaluation frameworks for LLM applications with automated and human scoring.

Updated Apr 5, 2026
Search Tags:#benchmarking#llm evaluation#metrics#regression testing#langsmith#ab testing#human review
One-click install
npx skills add https://github.com/Jhabbig/Habbig --skill llm-evaluation-jhabbig
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-evaluation
Source: https://github.com/Jhabbig/Habbig/tree/main/.claude/plugins/wshobson/llm-application-dev/skills/llm-evaluation
Command: npx skills add https://github.com/Jhabbig/Habbig --skill llm-evaluation-jhabbig

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps you evaluate LLM applications systematically so you can measure quality, compare changes, and catch regressions before they reach production.

Core Features & Use Cases

  • Automated metrics: Score outputs with BLEU, ROUGE, BERTScore, perplexity, classification metrics, and retrieval metrics.
  • Human evaluation: Build annotation workflows for accuracy, coherence, relevance, safety, and helpfulness.
  • LLM-as-judge: Use stronger models for pointwise, pairwise, and reference-based judgment.
  • A/B testing and regression checks: Compare variants statistically, estimate effect size, and flag performance drops.
  • Benchmarking and tracking: Run repeatable test suites, analyze aggregate results, and integrate with LangSmith for experiment management.

Quick Start

Ask for an evaluation plan for your LLM system that includes metrics, human review criteria, benchmark tests, and regression detection.

Frequently Asked Questions about llm-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up LLM evaluation metrics for generation and retrieval tasks?▼

To set up LLM evaluation metrics, use automated scoring like BLEU, ROUGE, BERTScore, and retrieval metrics to measure generation and classification task quality systematically.

What is LLM-as-judge and how does it evaluate model outputs?▼

LLM-as-judge is an evaluation method using stronger models to perform pointwise, pairwise, and reference-based judgment of LLM outputs for automated quality assessment.

Can I run statistical A/B testing to compare LLM prompt variants?▼

Yes, you can run statistical A/B testing to compare LLM prompt variants, estimate effect size, and flag performance drops or regressions before deployment.

Does this evaluation framework integrate with LangSmith for experiment tracking?▼

Yes, the evaluation framework integrates with LangSmith to run repeatable benchmark test suites, analyze aggregate results, and manage LLM experiment tracking.

How do I design human evaluation workflows for LLM safety and coherence?▼

Design human evaluation workflows by building annotation tasks that rate LLM outputs on specific criteria including accuracy, coherence, relevance, safety, and helpfulness.

What's the best way to detect regressions in LLM applications before production?▼

The best way to detect LLM regressions is running repeatable benchmark test suites with automated metrics and statistical A/B testing to flag performance drops before production.

Related Skills

Explore More Data & Analytics Skills→
apassuelloapassuello

llm-evaluation

Evaluate LLM outputs with automated metrics, human judgments, and benchmarking.

Community
Advanced
whd4whd4

canary

Monitor live deployments for console errors, performance regressions, and page failures.

Community
Advanced
EurecaMomentEurecaMoment

benchclaw-stage3-evidence-compiler

Compile, clean, and generate ground truth for benchmark data using Python scripts.

Official
Advanced
honysyanghonysyang
18

llm-evaluation

Evaluate Large Language Models with automated metrics and human evaluation frameworks.

Community
Advanced