eval-engine

Build and execute LLM evaluation pipelines with LLM-as-Judge and customizable metrics.

Updated Mar 1, 2026

Search Tags:#testing #llm #metrics #evaluation #pipeline #nlp

Ready to run

npx skills add https://github.com/mqzkim/trading --skill eval-engine

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-engine
Source: https://github.com/mqzkim/trading/tree/main/.agents/skills/eval-engine
Command: npx skills add https://github.com/mqzkim/trading --skill eval-engine

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill automates the process of building and running evaluation pipelines for Large Language Models (LLMs), enabling systematic quality assessment and comparison.

Core Features & Use Cases

Dataset Management: Load and manage datasets for evaluation.
LLM-as-Judge Evaluation: Utilize LLMs to evaluate responses based on various criteria.
Customizable Metrics: Supports accuracy, relevance, hallucination, harmfulness, and Korean-specific quality.
A/B Testing: Facilitates comparison between different model runs or configurations.
Use Case: Evaluate a new chatbot's responses against a benchmark dataset using multiple metrics, identifying areas for improvement.

Quick Start

Use the eval-engine skill to run an evaluation on the dataset named 'dataset_abc' using accuracy and hallucination evaluators with the Codex-haiku judge model.

Frequently Asked Questions about eval-engine

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I build an LLM evaluation pipeline for hallucination and relevance?▼

You can build an LLM evaluation pipeline using LLM-as-Judge to systematically assess hallucination and relevance. The process manages datasets and employs parallel processing for model calls to generate structured evaluation reports.

Can I run A/B testing to compare different LLM model runs?▼

Yes, you can run A/B testing to compare different LLM model runs or configurations. The evaluation pipeline systematically evaluates and contrasts response quality across customizable metrics like accuracy and harmfulness.

What metrics are supported for LLM evaluation besides accuracy?▼

Supported LLM evaluation metrics include accuracy, relevance, hallucination, harmfulness, and Korean language quality. These customizable metrics are applied through an LLM-as-Judge mechanism for systematic quality assessment.

How does LLM-as-Judge work for evaluating chatbot responses?▼

LLM-as-Judge evaluates chatbot responses by utilizing a judge model to assess quality based on customizable criteria. It requires robust data handling and parallel processing to execute model calls and generate structured outputs.

Does the evaluation pipeline support Korean language quality metrics?▼

Yes, the evaluation pipeline supports Korean language quality metrics. It includes customizable evaluation criteria designed to assess language-specific quality alongside general metrics like accuracy, relevance, and hallucination.

What is the best way to evaluate a chatbot against a benchmark dataset?▼

The best way to evaluate a chatbot against a benchmark dataset is using an automated LLM evaluation pipeline. It manages datasets, employs LLM-as-Judge for parallel processing, and identifies areas for improvement through structured reporting.