skills.rest
duanbiao2000
duanbiao2000COMMUNITY
Explore all skills by this author →
WebsiteGitHub

llm-evaluation

Evaluate LLM applications with automated metrics, human judgments, and benchmarking.

3|1|Updated Feb 3, 2026
Search Tags:#llm#benchmarking#metrics#evaluate#human-evaluation#nlp-evaluation
One-click install
npx skills add https://github.com/duanbiao2000/obsidianDoc26 --skill llm-evaluation-duanbiao2000
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-evaluation
Source: https://github.com/duanbiao2000/obsidianDoc26/tree/main/agents-main/plugins/llm-application-dev/skills/llm-evaluation
Command: npx skills add https://github.com/duanbiao2000/obsidianDoc26 --skill llm-evaluation-duanbiao2000

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Automates comprehensive evaluation of LLM applications using automated metrics, human feedback, and benchmarking.

Core Features & Use Cases

Automated Metrics

Fast, repeatable, scalable evaluation using computed scores for text generation, classification, and retrieval.

  • Text Generation: BLEU, ROUGE, METEOR, BERTScore, Perplexity
  • Classification: Accuracy, Precision/Recall/F1, Confusion Matrix, AUC-ROC
  • Retrieval (RAG): MRR, NDCG, Precision@K, Recall@K

Human Evaluation

Manual assessment for quality aspects difficult to automate.

  • Dimensions: Accuracy, Coherence, Relevance, Fluency, Safety, Helpfulness

LLM-as-Judge

Use stronger LLMs to evaluate outputs with pointwise, pairwise, and reference-based approaches.

Quick Start

Run the EvaluationSuite with your model and a test dataset to generate automated metrics and human-evaluation scores.

Frequently Asked Questions about llm-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate LLM performance using automated metrics and human judgments?▼

You can evaluate LLM performance by running the EvaluationSuite with your model and test dataset to generate computed scores for text generation, classification, and retrieval alongside manual human evaluation dimensions.

What automated metrics can I use to benchmark text generation and retrieval tasks?▼

For benchmarking text generation, you can use BLEU, ROUGE, METEOR, BERTScore, and Perplexity, while retrieval tasks can be evaluated using MRR, NDCG, Precision@K, and Recall@K.

How does LLM-as-judge work for evaluating model outputs?▼

LLM-as-judge evaluates model outputs by using stronger LLMs to assess performance through pointwise, pairwise, and reference-based approaches, providing automated quality judgments for research and production environments.

Can I assess LLM quality dimensions like coherence and safety without full automation?▼

Yes, you can assess LLM quality dimensions such as accuracy, coherence, relevance, fluency, safety, and helpfulness using manual human evaluation to capture quality aspects that are difficult to automate.

What is the best way to quantify LLM performance gaps during prompt iteration?▼

The best way to quantify LLM performance gaps during prompt iteration is by combining automated computed scores, human feedback judgments, and benchmarking to identify and measure specific model deficiencies.

Does this evaluation approach support classification metrics like accuracy and AUC-ROC?▼

Yes, this evaluation approach supports classification metrics including Accuracy, Precision, Recall, F1, Confusion Matrix, and AUC-ROC to provide fast, repeatable, and scalable model performance measurements.

Related Skills

Explore More Data & Analytics Skills→
agentic-inagentic-in
577

Inference

Select inference paths, serving stacks, and benchmarking plans for ML models.

Official
Intermediate
nimadorostkarnimadorostkar
25

prompt-engineering

Design, test, and refine prompts for large language models.

Community
Advanced
ionmidoriionmidori

evaluating-llms

Evaluate LLM outputs using automated NLP metrics and human annotation frameworks.

Community
Advanced
ccfccf

llm-evaluation

Automate LLM evaluation with automated metrics, human feedback, and benchmarking.

Community
Advanced