skills.rest
roanbrasil
roanbrasilCOMMUNITY
Explore all skills by this author →
Website𝕏@roanbrasilGitHub

llm-evals

Implement LLM evaluation patterns with LangSmith, RAGAS, and PromptFoo.

5|1|Updated Jun 17, 2026
Search Tags:#llm-as-judge#llm-evaluation#prompt-testing#regression-detection#ci-integration#golden-dataset#rag-testing
One-click install
npx skills add https://github.com/roanbrasil/engineer-grade-agent-skills --skill llm-evals
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-evals
Source: https://github.com/roanbrasil/engineer-grade-agent-skills/tree/main/skills/llm-evals
Command: npx skills add https://github.com/roanbrasil/engineer-grade-agent-skills --skill llm-evals

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

LLMs produce non-deterministic outputs, so prompt tweaks or model updates can silently break existing functionality without any visible warning signs. Manual review of LLM responses is slow, subjective, and impossible to scale across frequent changes, leaving teams with no reliable way to catch quality regressions before they impact users.

Core Features & Use Cases

  • Comprehensive Eval Coverage: Implements patterns for unit, model, agent, and system-level evaluations tailored to different LLM workflow speeds and determinism requirements.
  • Tool Integration: Includes ready-to-use code samples and configurations for leading eval tools including LangSmith, RAGAS, PromptFoo, DeepEval, and Braintrust.
  • Production Guardrails: Provides golden dataset management best practices, LLM-as-judge bias mitigation, CI integration templates, and regression threshold enforcement to prevent bad deployments.
  • Real-World Use Case: A team building a RAG customer support chatbot can use this Skill to track faithfulness and context recall metrics, run adversarial tests for prompt injection, and automatically fail CI builds if quality drops below defined thresholds.

Quick Start

Use the llm-evals skill to configure automated evaluation checks that catch LLM output regressions in your CI pipeline before they reach production.

Frequently Asked Questions about llm-evals

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I catch silent LLM regressions caused by prompt tweaks or model updates?▼

Automated LLM evaluation frameworks catch silent regressions by applying unit, model, agent, and system-level tests against golden datasets. They enforce quality thresholds and mitigate LLM-as-judge bias to prevent bad prompt or model deployments from reaching production.

What is the best way to integrate LLM evaluation into a CI pipeline?▼

Integrate LLM evaluation into CI pipelines using templates for tools like LangSmith, PromptFoo, and DeepEval. These frameworks run automated eval checks against golden datasets and automatically fail CI builds if output quality drops below defined regression thresholds.

How do I evaluate RAG pipelines to ensure faithfulness and prevent prompt injection?▼

RAG pipeline evaluation tracks faithfulness and context recall metrics using frameworks like RAGAS. You can run adversarial tests for prompt injection and enforce regression thresholds to ensure your RAG customer support chatbot maintains consistent quality across updates.

Does this LLM evaluation approach work with LangSmith, RAGAS, and PromptFoo?▼

Yes, the evaluation patterns include ready-to-use code samples and configurations for LangSmith, RAGAS, PromptFoo, DeepEval, and Braintrust. This tool integration supports comprehensive eval coverage across different LLM workflow speeds and determinism requirements.

Why does manual review fail to catch non-deterministic LLM output regressions?▼

Manual review fails to catch non-deterministic LLM output regressions because it is slow, subjective, and impossible to scale across frequent changes. Automated evaluation frameworks provide reliable, production-grade quality assurance that manual review cannot achieve.

When do I need a golden dataset for LLM evaluation?▼

You need a golden dataset for LLM evaluation when managing production guardrails for non-deterministic outputs. Golden datasets provide the baseline reference required to enforce regression thresholds, mitigate LLM-as-judge bias, and track quality across prompt and model versions.

Related Skills

Explore More Software Engineering Skills→
vecearvecear

evaluation-harness

Automate LLM evaluation with golden datasets, rubrics, thresholds, and regression reports.

Community
Advanced
mahg-esmahg-es
3

cicd-quality

Design and enforce CI/CD quality gates for code, tests, security, build, E2E, and performance.

Official
Advanced
Accelerated-InnovationAccelerated-Innovation
20

eval-suite-planning

Plan LLM evaluation test suites from feature specs and architecture guidance.

Official
Advanced
MuvonMuvon
3

ai-evals-design

Designs statistically grounded LLM evaluation methodology with paired significance testing and effect sizes.

Official
Advanced