skills.rest
hung-phan
hung-phanCOMMUNITY
Explore all skills by this author →
GitHub

llm-evaluation

Evaluate foundation models with BERTScore, ROUGE, and COMET metrics.

1|Updated Jun 4, 2026
Search Tags:#llm#evaluation#benchmark#hallucination detection#ai assessment
One-click install
npx skills add https://github.com/hung-phan/ml-skills --skill llm-evaluation-hung-phan
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-evaluation
Source: https://github.com/hung-phan/ml-skills/tree/main/skills/ml-review/references/ml-training/llm-evaluation
Command: npx skills add https://github.com/hung-phan/ml-skills --skill llm-evaluation-hung-phan

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires bertscore, rouge-score, comet, lm-evaluation-harness, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill addresses the challenge of evaluating and diagnosing foundation models, providing a comprehensive toolkit for ensuring quality and reliability in AI applications.

Core Features & Use Cases

  • Exact Evaluation: Validates the functional correctness of models through executable tasks.
  • Reference-Based Evaluation: Scores models against reference outputs using lexical and semantic metrics.
  • AI-as-Judge: Utilizes human-like evaluation to assess model performance.
  • Hallucination Detection: Identifies and mitigates hallucinations in model outputs.
  • Public Benchmarks Analysis: Provides insights into the reliability of public benchmarks.
  • Custom Evaluation Pipeline: Builds and customizes evaluation pipelines for specific applications.
  • Use Case: When selecting between LLMs for a real-world task, or when diagnosing hallucinations in a RAG system.

Quick Start

Evaluate the performance of the LLM model using the provided benchmarks and metrics.

Frequently Asked Questions about llm-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I detect hallucinations in my foundation model outputs?▼

Hallucination detection identifies factual inconsistencies in model outputs by evaluating factual consistency against reference data. This Skill diagnoses hallucinations to ensure AI application reliability.

What metrics are used for LLM evaluation and benchmark analysis?▼

LLM evaluation uses lexical and semantic metrics including BERTScore, ROUGE, and COMET to score models against reference outputs, measuring semantic correctness and lexical quality for benchmark analysis.

How can I evaluate the functional correctness of foundation models?▼

Functional correctness is evaluated through executable tasks that validate model outputs. This Skill provides exact evaluation to confirm whether foundation models perform tasks correctly and accurately.

Does this AI assessment approach support custom evaluation pipelines?▼

Yes, custom evaluation pipelines can be built for specific applications. This AI assessment Skill allows you to configure benchmarks and metrics tailored to your specific model validation needs.

Can I use lm-evaluation-harness for public benchmark analysis?▼

Yes, lm-evaluation-harness is supported as a framework for public benchmark analysis. It provides insights into the reliability of public benchmarks for foundation model evaluation.

What is the best way to diagnose a RAG system for hallucinations?▼

Diagnosing RAG systems involves evaluating factual consistency and reference-based outputs. This Skill identifies hallucinations and scores semantic quality to diagnose RAG system reliability.

Related Skills

Explore More Software Engineering Skills→
ArogyaReddyArogyaReddy

llm-evaluation

Automates LLM output evaluation with metrics, human feedback, and benchmarking.

Community
Advanced
fabioc-alohafabioc-aloha

LLM Model Selection Skill

Select LLM model tiers by balancing capability, cost, and latency.

Community
Advanced
yanochka11yanochka11

evaluating-llms-harness

Benchmark large language models with lm-evaluation-harness across standardized metrics.

Community
Advanced
mrsorbatemrsorbate

AgentDB Vector Search

Implement semantic vector search and retrieval-augmented generation with AgentDB.

Community
Advanced