Build and execute LLM evaluation pipelines with LLM-as-Judge and customizable metrics.
npx skills add https://github.com/mqzkim/trading --skill eval-engine
Please help me install this Agent Skill. Skill: eval-engine Source: https://github.com/mqzkim/trading/tree/main/.agents/skills/eval-engine Command: npx skills add https://github.com/mqzkim/trading --skill eval-engine
This Skill automates the process of building and running evaluation pipelines for Large Language Models (LLMs), enabling systematic quality assessment and comparison.
Use the eval-engine skill to run an evaluation on the dataset named 'dataset_abc' using accuracy and hallucination evaluators with the Codex-haiku judge model.
High-intent search queries and answers about installing and using this skill.
You can build an LLM evaluation pipeline using LLM-as-Judge to systematically assess hallucination and relevance. The process manages datasets and employs parallel processing for model calls to generate structured evaluation reports.
Yes, you can run A/B testing to compare different LLM model runs or configurations. The evaluation pipeline systematically evaluates and contrasts response quality across customizable metrics like accuracy and harmfulness.
Supported LLM evaluation metrics include accuracy, relevance, hallucination, harmfulness, and Korean language quality. These customizable metrics are applied through an LLM-as-Judge mechanism for systematic quality assessment.
LLM-as-Judge evaluates chatbot responses by utilizing a judge model to assess quality based on customizable criteria. It requires robust data handling and parallel processing to execute model calls and generate structured outputs.
Yes, the evaluation pipeline supports Korean language quality metrics. It includes customizable evaluation criteria designed to assess language-specific quality alongside general metrics like accuracy, relevance, and hallucination.
The best way to evaluate a chatbot against a benchmark dataset is using an automated LLM evaluation pipeline. It manages datasets, employs LLM-as-Judge for parallel processing, and identifies areas for improvement through structured reporting.
wenermeTranslate natural language requests into EvalScope CLI commands for benchmark evaluations and stress tests.
ThrookieAnalyze draft prompts and map them to ECC components for optimized output.
Productfculty-aipmAutomate evaluation of AI-generated outputs using an LLM judge with PASS/FAIL rubrics.
HermeticOrmusEvaluates LLM outputs using automated metrics, human feedback, and LLM-as-Judge techniques.
