promptfoo-evaluation

Name: promptfoo-evaluation
Availability: InStock
Author: huubar

Automates LLM evaluation configuration and execution using Promptfoo framework.

Updated Apr 21, 2026

Search Tags:#llm-evaluation #model-comparison #prompt-testing #promptfoo #python-assertions #evaluation-config

Ready to run

npx skills add https://github.com/HuuBar/skill-routing-experiment --skill promptfoo-evaluation-huubar

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: promptfoo-evaluation
Source: https://github.com/HuuBar/skill-routing-experiment/tree/main/unified_skills/daymade/promptfoo-evaluation
Command: npx skills add https://github.com/HuuBar/skill-routing-experiment --skill promptfoo-evaluation-huubar

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

Automates the configuration and execution of LLM evaluations using the Promptfoo framework to standardize benchmarking and reduce setup time.

Core Features & Use Cases

Configure end-to-end evaluation projects via promptfooconfig.yaml
Use Python-based custom assertions and llm-rubric grading for nuanced scoring
Manage prompts, providers, and tests to compare model performance across scenarios
Leverage few-shot prompts and optional references/scripts to fine-tune evaluation workflows

Quick Start

Create a Promptfoo project, configure promptfooconfig.yaml with your evaluation setup, and run npx promptfoo@latest eval to start testing.

Frequently Asked Questions about promptfoo-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I set up an LLM evaluation project with promptfooconfig.yaml?▼

To set up an LLM evaluation project, create a Promptfoo project structure and define your prompts, providers, and tests in a promptfooconfig.yaml file. You can then execute the evaluation by running the npx promptfoo@latest eval command.

Can I use Python assertions for custom LLM evaluation testing?▼

Yes, you can use Python-based custom assertions to evaluate LLM outputs. The framework supports writing Python assertions to apply nuanced scoring logic tailored to your specific testing scenarios.

What is llm-rubric and how does it work for LLM-as-judge grading?▼

LLM-rubric is an evaluation technique for implementing LLM-as-judge grading. It allows you to use a separate LLM to score and evaluate the outputs of your target models based on defined grading criteria within your test configuration.

How do I compare model performance across different LLM providers?▼

You can compare model performance by configuring multiple providers within your promptfooconfig.yaml file. The framework manages prompts and tests across these diverse models to benchmark their performance side-by-side.

Do I need an existing Promptfoo project structure to run LLM evaluations?▼

Yes, an existing Promptfoo project structure is required. The evaluation workflow depends on this structure to properly organize your prompt configurations, test cases, providers, and optional reference scripts.

How do I use few-shot prompts to fine-tune LLM evaluation workflows?▼

You can leverage few-shot prompts within your configuration to guide model behavior during evaluations. This technique helps fine-tune the evaluation workflow by providing example inputs and outputs alongside your test cases.