skills.rest
ehadziabdic
ehadziabdicCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

llm-evaluation

Evaluate LLM quality with automated metrics, LLM-as-judge, RAGAS, and safety test suites.

Updated Sep 23, 2026
Search Tags:#llm-as-judge#llm evaluation#hallucination detection#prompt injection#benchmarks#ragas#safety evaluation
One-click install
npx skills add https://github.com/ehadziabdic/WAgents --skill llm-evaluation-ehadziabdic
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-evaluation
Source: https://github.com/ehadziabdic/WAgents/tree/main/opencode/skills/llm-evaluation
Command: npx skills add https://github.com/ehadziabdic/WAgents --skill llm-evaluation-ehadziabdic

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires openai, anthropic, ragas, langchain-openai, nltk, rouge-score, bert-score, and includes scripts (resource) and references (resource) components.

What problem does it solve? Measuring LLM output quality is hard because responses are open-ended and subjective, and a single metric or one-off benchmark cannot catch regressions, hallucinations, or safety failures before deployment. ## Core Features & Use Cases - Multi-Metric Evaluation Pipeline: Run BLEU, ROUGE, BERTScore, exact-match, RAGAS, and LLM-as-judge scoring over JSONL datasets via scripts/evaluate_llm.py, producing per-sample and aggregate JSON reports. - Safety & Bias Testing: Probe models for toxicity, demographic bias, prompt injection resistance, PII leakage, and hallucination with scripts/safety_eval.py, generating JSON and Markdown scorecards. - Scorer Validation with Negative Controls: Apply the documented negative-control methodology (trivial identity, wrong-task code, non-compiling code) to verify that verifiable-reward scorers reject defective outputs instead of silently inflating scores. - Use Case: After fine-tuning a model, run the evaluation pipeline against a baseline on your domain dataset, then run the safety suite and compare scorecards to decide whether the candidate is safe to deploy. ## Quick Start Evaluate my fine-tuned model against the baseline using the eval dataset in eval.jsonl with RAGAS and LLM-as-judge metrics, then run the full safety test suite and summarize the scorecard.

Frequently Asked Questions about llm-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate LLM responses with RAGAS in Python?▼

Build an EvaluationDataset with user_input, response, retrieved_contexts, and reference fields, then call ragas.evaluate with metrics like Faithfulness, AnswerRelevancy, ContextPrecision, and ContextRecall. Pass an explicit judge LLM such as a LangchainLLMWrapper around ChatOpenAI.

What metrics should I use to evaluate an LLM?▼

No single metric captures LLM quality, so combine types: ROUGE and BERTScore for summarization, exact match and F1 for QA, pass@k for code, and RAGAS faithfulness for RAG. Add LLM-as-judge scoring and human evaluation for nuanced quality assessment.

RAGAS vs DeepEval vs Promptfoo for LLM evaluation?▼

RAGAS focuses on RAG-specific retrieval and generation metrics, DeepEval offers pytest-style general LLM testing in Python, and Promptfoo provides configuration-driven prompt testing with strong CI/CD integration. LangSmith adds tracing and observability alongside evaluation.

How do I test an LLM for prompt injection vulnerabilities?▼

Send known injection payloads such as instruction-override and system-prompt-extraction attempts, then check responses for canary strings and leaked prompt text. The safety_eval.py script automates this and reports an injection resistance score.

Why does my evaluation scorer accept malformed model outputs?▼

Scorers that validate format with regex can accept truncated outputs that still match the signature. Compile or parse the extracted artifact inside the scorer and run negative controls, such as non-compiling code and trivial identity solutions, to confirm defective inputs score zero.

How many examples do I need for statistically significant LLM evaluation?▼

Use at least 200 examples to detect a 5% difference with 80% power, and 500 or more for production evaluation suites. Apply McNemar's test for accuracy comparisons, paired t-tests for mean scores, and bootstrap confidence intervals for multi-model comparisons.

Related Skills

Explore More Software Engineering Skills→
rocklambrosrocklambros
2

scaffolding-llm-eval-harness

Scaffold a reproducible LLM evaluation harness with versioned prompts and JSONL results.

Community
Advanced
agenticgogolagenticgogol
1

judge-builder-new

Write binary PASS/FAIL LLM judge prompts grounded in eval specs for routed failure modes.

Community
Advanced
Orchestra-ResearchOrchestra-Research
13.0k

nemo-guardrails

Implements programmable runtime safety rails for LLM applications using NVIDIA NeMo Guardrails.

Official
Intermediate
HuuBarHuuBar

promptfoo-evaluation

Automates LLM evaluation configuration and execution using Promptfoo framework.

Community
Intermediate