add-llm-evals

Add offline and online evaluation scoring to LLM and agent applications.

29|8|Updated Jul 5, 2026

Search Tags:#quality assurance #LLM evaluation #CI integration #online scoring #RAG metrics

Ready to run

npx skills add https://github.com/ContextJet-ai/awesome-llm-observability --skill add-llm-evals

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: add-llm-evals
Source: https://github.com/ContextJet-ai/awesome-llm-observability/tree/main/skills/add-llm-evals
Command: npx skills add https://github.com/ContextJet-ai/awesome-llm-observability --skill add-llm-evals

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires promptfoo, deepeval, ragas, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill adds evaluation capabilities to LLM/agent apps, measuring output quality (correctness, faithfulness, relevance, safety) beyond just trace observation.

Core Features & Use Cases

Offline Suite: Automates evaluation on a fixed dataset in CI.
Online Scoring: Optionally grades production traffic for quality.
Reference-based and Reference-free Evaluations: Supports both exact/fuzzy match, semantic similarity, and LLM-as-a-judge rubrics.
RAG Metrics: Implements faithfulness, answer relevance, context precision, and context recall for RAG.

Quick Start

Run the offline evaluation suite on your LLM/agent app's PRs to ensure prompt and model quality.

Frequently Asked Questions about add-llm-evals

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I evaluate LLM output quality in CI pipelines?▼

To evaluate LLM output quality in CI, you can automate an offline evaluation suite on a fixed dataset to measure correctness, faithfulness, and relevance. This ensures prompt and model quality on your pull requests.

What metrics are used for RAG evaluation?▼

RAG evaluation implements faithfulness, answer relevance, context precision, and context recall. These metrics measure how accurately the retrieval-augmented generation output aligns with the provided context.

Do I need promptfoo, DeepEval, or Ragas for offline LLM evaluation?▼

Yes, offline LLM evaluation requires promptfoo, DeepEval, or Ragas. These dependencies provide the testing frameworks needed to measure output quality against fixed datasets.

Can I score production LLM traffic online?▼

Yes, online scoring optionally grades production LLM traffic for output quality. This integrates with observability platforms to measure correctness, safety, and relevance in real-time.

What is the difference between reference-based and reference-free LLM evaluation?▼

Reference-based LLM evaluation uses exact or fuzzy matching against ground truth, while reference-free evaluation uses semantic similarity and LLM-as-a-judge rubrics to score output quality without predefined correct answers.