skills.rest
surfingalien
surfingalienCOMMUNITY
Explore all skills by this author →
GitHub

llm-evaluation

Implement evaluation frameworks for LLM applications using automated metrics, LLM-as-judge, and RAGAS.

1|Updated May 6, 2026
Search Tags:#llm-as-judge#rag#benchmarking#llm evaluation#regression testing#a/b testing#ragas
One-click install
npx skills add https://github.com/surfingalien/FinSurfing --skill llm-evaluation-surfingalien
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-evaluation
Source: https://github.com/surfingalien/FinSurfing/tree/main/.claude/skills/llm-evaluation
Command: npx skills add https://github.com/surfingalien/FinSurfing --skill llm-evaluation-surfingalien

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires ragas, datasets, nltk, rouge-score, bert-score, openai, scipy, numpy, scikit-learn, pydantic, transformers, detoxify, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve? Measuring whether an LLM application actually works well is hard: single metrics mislead, regressions slip into production, and open-ended outputs resist simple scoring. This Skill provides a complete evaluation methodology covering automated metrics, LLM-as-judge patterns, human annotation, RAG pipeline scoring, and CI/CD regression detection. ## Core Features & Use Cases - Automated Metrics: Implementations of BLEU, ROUGE, BERTScore, perplexity, and custom metrics like groundedness and toxicity. - LLM-as-Judge: Pointwise, pairwise, and rubric-based judging with position-bias mitigation via order swapping and structured JSON output. - RAG Evaluation: RAGAS metrics (faithfulness, answer relevancy, context precision/recall) to diagnose retrieval and hallucination issues. - Agentic & Structured Output Evaluation: Task success rates, tool-use accuracy, step efficiency, and Pydantic schema compliance checks. - Use Case: Before deploying a prompt change, run your eval suite in CI, compare against the baseline commit, and automatically fail the build if any metric regresses by more than 5%. ## Quick Start Ask the AI to set up an evaluation suite with RAGAS metrics and an LLM-as-judge pairwise comparison for your RAG chatbot's test cases.

Frequently Asked Questions about llm-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate LLM application performance systematically?▼

Combine automated metrics (BLEU, ROUGE, BERTScore) with LLM-as-judge scoring for open-ended tasks and human evaluation for quality dimensions like accuracy and coherence. Run evaluations against a representative test set and track results over time to detect regressions.

How to evaluate a RAG pipeline with RAGAS?▼

RAGAS evaluates RAG pipelines using four metrics: faithfulness (hallucination detection), answer relevancy, context precision, and context recall. Pass a dataset with questions, answers, retrieved contexts, and ground truths to the ragas evaluate function to get scores from 0 to 1.

What is LLM-as-judge and how do I reduce its bias?▼

LLM-as-judge uses a strong model like GPT-4.1 to score or compare outputs, and it is the dominant approach for open-ended tasks. Mitigate position bias by swapping response order and averaging, reduce verbosity bias with structured rubrics, and use third-party judges to avoid self-preference.

Can I run LLM evaluations in a CI/CD pipeline?▼

Yes, treat evals as code by running your evaluation suite on every pull request that touches prompts or LLM logic. Compare results against a baseline commit and fail the build when any metric regresses beyond a threshold such as 5 percent.

Why are BLEU and ROUGE poor metrics for modern LLM outputs?▼

BLEU and ROUGE measure n-gram overlap, which correlates poorly with quality for open-ended generation where many valid answers exist. For modern LLM tasks, prefer BERTScore, LLM-as-judge rubrics, or task-specific metrics like faithfulness and schema compliance.

How do I evaluate structured JSON output from an LLM?▼

Validate the response against a Pydantic schema to check structural compliance, then compare parsed field values against expected outputs to compute field-level accuracy. This separates format failures from content errors so you can diagnose each independently.

Related Skills

Explore More Software Engineering Skills→
trueratruera
3.5k

trulens-dataset-curation

Create and curate ground-truth evaluation datasets for TruLens.

Official
Intermediate
databricks-solutionsdatabricks-solutions
5

deployment-automation

Automate GenAI agent deployment with MLflow evaluation-then-promote workflows.

Official
Advanced
ContextJet-aiContextJet-ai
29

add-llm-evals

Add offline and online evaluation scoring to LLM and agent applications.

Official
Advanced
photonics-dhlphotonics-dhl

evaluating-llms-harness

Benchmark language models across 60+ academic benchmarks using a unified evaluation harness.

Community
Advanced