skills.rest
haxlys
haxlysCOMMUNITY
Explore all skills by this author →
GitHub

llm-evaluation

Quantify LLM performance using automated metrics and human feedback.

1|Updated Apr 27, 2026
Search Tags:#automation#benchmarking#metrics#nlp#llm-evaluation#human-evaluation#model-assessment
One-click install
npx skills add https://github.com/haxlys/skills --skill llm-evaluation-haxlys
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-evaluation
Source: https://github.com/haxlys/skills/tree/main/vendored/wshobson-agents/plugins/llm-application-dev/skills/llm-evaluation
Command: npx skills add https://github.com/haxlys/skills --skill llm-evaluation-haxlys

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Identify and quantify the performance and reliability of LLM-driven systems by combining automated metrics with human judgment to establish trustworthy benchmarks.

Core Features & Use Cases

  • Automated metrics: BLEU, ROUGE, METEOR, BERTScore, perplexity, and retrieval metrics for evaluation.
  • Human evaluation: multidimensional scoring (accuracy, relevance, coherence, safety) and error analysis for real-world prompts.
  • LLM-as-Judge and benchmarking: compare models and prompts using structured evaluation pipelines and reference-based scoring.
  • Use Case: benchmark a new model against a baseline across a suite of tasks and iterate based on results.

Quick Start

Run a baseline evaluation by feeding your model's predictions for a predefined test set and review the aggregated metrics.

Frequently Asked Questions about llm-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate LLM performance using automated metrics and human evaluation?▼

To evaluate LLM performance, you feed your model's predictions into a predefined test set to calculate automated metrics like BLEU and BERTScore, then apply multidimensional human scoring for accuracy and relevance.

What automated metrics are used for benchmarking LLM applications?▼

Benchmarking LLM applications utilizes automated metrics including BLEU, ROUGE, METEOR, BERTScore, perplexity, and retrieval metrics to quantify model performance and enable reproducible comparisons across prompts.

How does LLM-as-Judge work for model comparisons?▼

LLM-as-Judge works by using structured evaluation pipelines and reference-based scoring to compare different models and prompts, establishing trustworthy benchmarks through automated assessments combined with human judgment.

Can I monitor LLM quality over time using human evaluation frameworks?▼

Yes, you can monitor LLM quality over time by repeatedly applying human evaluation frameworks that perform multidimensional scoring and error analysis on real-world prompts to track reliability.

What is the best way to benchmark a new model against a baseline?▼

The best way to benchmark a new model against a baseline is running its predictions through structured evaluation pipelines to compute automated metrics, followed by human evaluation for comprehensive error analysis.

Related Skills

Explore More Data & Analytics Skills→
mvillmowmvillmow
18

evaluate-model

Evaluate ML model performance on test datasets with classification and regression metrics.

Community
Basic
RedWoodOGRedWoodOG
174

evaluating-llms-harness

Evaluate LLMs across 60+ academic benchmarks using lm-evaluation-harness.

Community
Advanced
ollieb89ollieb89

llm-evaluation

Evaluate LLM outputs with automated metrics and LLM-as-Judge workflows.

Community
Advanced
shalhevetvardishalhevetvardi

reels-analytics

Analyze Instagram Reels performance metrics from CSV exports and generate structured insights reports.

Community
Intermediate