skills.rest
yonatangross
yonatangrossCOMMUNITY
Explore all skills by this author →
𝕏@yonyonizGitHub

testing-llm

Mock LLM responses and evaluate outputs with DeepEval and RAGAS.

217|20|Updated Dec 31, 2025
Search Tags:#testing#ai#llm#evaluation#mocking#ragas#deepeval
One-click install
npx skills add https://github.com/yonatangross/orchestkit --skill testing-llm
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: testing-llm
Source: https://github.com/yonatangross/orchestkit/tree/main/plugins/ork/skills/testing-llm
Command: npx skills add https://github.com/yonatangross/orchestkit --skill testing-llm

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) and checklists (resource) components.

What problem does it solve?

This Skill addresses the challenge of reliably testing AI and LLM-generated content, ensuring quality, accuracy, and deterministic behavior in your applications.

Core Features & Use Cases

  • LLM Mocking: Create deterministic unit tests by mocking LLM API responses.
  • Quality Evaluation: Validate LLM outputs using frameworks like DeepEval and RAGAS for metrics like relevancy, faithfulness, and hallucination detection.
  • Structured Output Validation: Ensure LLM responses adhere to predefined schemas using Pydantic.
  • Agentic Test Workflows: Implement advanced testing patterns with planner, generator, and healer agents.
  • Use Case: When developing a chatbot that relies on an LLM for responses, use this Skill to write tests that verify the chatbot's answers are relevant, factually correct based on provided context, and adhere to a specific JSON structure.

Quick Start

Use the testing-llm skill to validate the quality of an LLM response against a set of DeepEval metrics.

Frequently Asked Questions about testing-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I test LLM outputs for relevancy and hallucinations?▼

Test LLM outputs for hallucinations by evaluating response quality using frameworks like DeepEval and RAGAS, which validate metrics such as relevancy, faithfulness, and factual accuracy against provided context.

How do I write deterministic unit tests for AI features?▼

Write deterministic unit tests for AI features by mocking LLM API responses, ensuring consistent behavior during testing without relying on live model generation.

Can I validate structured LLM responses against a schema?▼

Validate structured LLM responses using Pydantic schemas to ensure AI outputs adhere to predefined JSON structures and specific data types.

What is an agentic test workflow for LLM evaluation?▼

Agentic test workflows implement advanced testing patterns for LLM evaluation using planner, generator, and healer agents to automate and maintain test coverage.

How do I build an evaluation pipeline for a chatbot?▼

Build a chatbot evaluation pipeline by testing that responses are relevant, factually correct based on context, and adhere to expected structures using provided testing patterns and tools.

Related Skills

Explore More Software Engineering Skills→
LynxLabVNLynxLabVN

instructor

Extract and validate structured data from LLM responses using Pydantic.

Official
Advanced
DevelopersGlobalDevelopersGlobal
65

ci-cd-pipelines

Automate CI/CD pipelines with quality gates and deployment processes.

Official
Advanced
svssdevasvssdeva
1

rag-implementation

Build Retrieval-Augmented Generation systems with semantic search and vector storage.

Community
Advanced
fdshg693fdshg693

copilot-test-skill

Validate Copilot integration with a minimal YAML frontmatter Skill unit.

Community
Basic