skills.rest
ekremmkasap
ekremmkasapCOMMUNITY
Explore all skills by this author →
GitHub

llm-evaluation

Measure LLM performance with automated metrics, human feedback, and benchmarking.

Updated Mar 18, 2026
Search Tags:#benchmarking#llm-evaluation#model-comparison#human-evaluation#automated-metrics#prompt-quality
One-click install
npx skills add https://github.com/ekremmkasap/jarvis --skill llm-evaluation-ekremmkasap
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-evaluation
Source: https://github.com/ekremmkasap/jarvis/tree/main/server/agent_prompts/wshobson/plugins/llm-application-dev/skills/llm-evaluation
Command: npx skills add https://github.com/ekremmkasap/jarvis --skill llm-evaluation-ekremmkasap

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps teams systematically evaluate LLM-based applications using automated metrics, human feedback, and benchmarking to ensure quality and reliability.

Core Features & Use Cases

  • Automated metrics: BLEU, ROUGE, BERTScore, F1, accuracy, and AUC-ROC to quantify model performance.
  • Human evaluation: structured rubrics for accuracy, relevance, coherence, and safety to capture nuanced quality.
  • Benchmarking workflows: end-to-end evaluation pipelines to compare models, prompts, and data slices in production-like scenarios.
  • Use Case: validate a new chat model by running a cross-model comparison across multiple tasks and reporting a single score.

Quick Start

Run an end-to-end evaluation across models with automated metrics and human review to produce a benchmark report.

Frequently Asked Questions about llm-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate LLM performance using automated metrics and human feedback?▼

Evaluate LLM performance by applying automated metrics like BLEU and BERTScore alongside structured human rubrics for relevance and safety. This process captures both quantitative accuracy and nuanced qualitative quality across model variants.

What is the best way to benchmark LLM models and compare prompt quality?▼

Benchmark LLM models and compare prompt quality by running reproducible evaluation pipelines across multiple tasks and data slices. This approach detects regressions and quantifies differences in production-like scenarios.

Can I detect model regressions across different data slices using LLM evaluation?▼

Detect model regressions across data slices by applying consistent automated metrics and human evaluation rubrics across variants. This systematic benchmarking identifies specific areas where performance degrades over iterations.

Does human evaluation work with automated metrics for comprehensive LLM benchmarking?▼

Human evaluation integrates with automated metrics for comprehensive LLM benchmarking by combining structured rubrics for coherence and safety with quantitative scores. This dual approach provides a complete quality assessment.

What automated metrics are needed to measure LLM accuracy and relevance?▼

Automated metrics needed to measure LLM accuracy and relevance include BLEU, ROUGE, BERTScore, F1, accuracy, and AUC-ROC. These metrics quantify model performance systematically across various evaluation tasks.

Related Skills

Explore More Data & Analytics Skills→
rohunvorarohunvora
5

advanced-evaluation

Automate LLM-as-a-judge scoring, pairwise comparison, and rubric generation with bias mitigation.

Community
Advanced
paulinghampaulingham
1

internal-eval-score

Score per-case results against oracle baselines and generate regression diffs as JSON and Markdown.

Community
Advanced
teradata-labsteradata-labs
7

td-data-profiling

Profile Teradata tables using ClearScape Analytics statistical functions.

Official
Advanced
ionmidoriionmidori

evaluating-llms

Evaluate LLM outputs using automated NLP metrics and human annotation frameworks.

Community
Advanced