skills.rest
iDako7
iDako7COMMUNITY
Explore all skills by this author →
GitHub

promptfoo-evals

Create and update promptfoo evaluation suites for prompts, providers, tests, and assertions.

1|Updated Apr 6, 2026
Search Tags:#testing#evaluation#prompts#providers#assertions#ci#promptfoo
One-click install
npx skills add https://github.com/iDako7/Smart_Grocery_Agent --skill promptfoo-evals-idako7
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: promptfoo-evals
Source: https://github.com/iDako7/Smart_Grocery_Agent/tree/main/.claude/skills/promptfoo-evals
Command: npx skills add https://github.com/iDako7/Smart_Grocery_Agent --skill promptfoo-evals-idako7

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill removes the guesswork and manual effort of building, organizing, and validating promptfoo evaluation suites so teams can quickly add regression checks, format validations, and model comparisons.

Core Features & Use Cases

  • Eval scaffolding: Create promptfooconfig.yaml, prompts/, and tests/ layouts for new suites and extend existing configs.
  • Assertion & provider guidance: Prefer deterministic assertions, show patterns for model-graded checks, JSON output handling, transforms, and grader pinning.
  • CI and debugging: Validate and run suites reliably with --no-cache, recommend report outputs and env var handling for CI pipelines.

Quick Start

Create a new promptfoo suite named "checkout" by adding evals/checkout/promptfooconfig.yaml with prompts/main.txt and tests/example.yaml then validate and run it using the promptfoo CLI with --no-cache.

Frequently Asked Questions about promptfoo-evals

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create a promptfoo evaluation suite for my LLM prompts?▼

Create a promptfoo evaluation suite by scaffolding a promptfooconfig.yaml file alongside prompts/ and tests/ directories to validate prompts, providers, tests, and assertions.

How do I run promptfoo evals reliably in CI pipelines?▼

Run promptfoo evals reliably in CI pipelines by executing the validation and eval commands with the --no-cache flag to ensure stable, deterministic test runs without stale cached results.

What is the best way to structure assertions in promptfoo tests?▼

Structure assertions by enforcing deterministic checks first, then applying model-graded checks, JSON response handling, and grader pinning to validate prompt outputs effectively.

Can I use promptfoo to test HTTP APIs and local provider integrations?▼

Yes, you can use promptfoo to test HTTP APIs and validate local provider integrations by configuring provider settings and organizing tests within your promptfooconfig.yaml file.

How do I debug prompt regressions when comparing different models?▼

Debug prompt regressions by adding evaluation coverage to compare models, applying deterministic assertions, and running promptfoo evals with --no-cache to isolate performance changes.

Why use --no-cache when running promptfoo evaluation commands?▼

Use the --no-cache flag when running promptfoo evaluation commands to bypass stored responses, forcing fresh API calls and guaranteeing stable, reproducible results for CI workflows.

Related Skills

Explore More Software Engineering Skills→
gerald-icagerald-ica

promptfoo-evals

Automate creation and maintenance of promptfoo evaluation suites, prompts, tests, assertions, and provider configurations.

Community
Advanced
fredericvahrenhorstfredericvahrenhorst

eval-harness

Run capability and regression evals for Claude Code sessions with pass@k metrics.

Community
Advanced
NagyViktNagyVikt
2

bump-go-dependencies

Update Go module dependencies sequentially with linting and test validation.

Community
Intermediate
FlexNetOSFlexNetOS

llm-evaluation

Automates LLM evaluation across prompts, outputs, and safety checks with promptfoo, TruLens, and OpenAI evals.

Community
Advanced