promptfoo-ci-lab

Name: promptfoo-ci-lab
Availability: InStock
Author: Shichiyou

Set up and troubleshoot promptfoo evaluation pipelines for LLM prompts.

Updated Aug 27, 2026

Search Tags:#yaml #ci-cd #devcontainer #ollama #promptfoo #lab #llm-eval

Ready to run

npx skills add https://github.com/shichiyou/hermes-agent-001 --skill promptfoo-ci-lab

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: promptfoo-ci-lab
Source: https://github.com/shichiyou/hermes-agent-001/tree/main/.devcontainer/hermes-backup/skills/.archive/promptfoo-ci-lab
Command: npx skills add https://github.com/shichiyou/hermes-agent-001 --skill promptfoo-ci-lab

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Sets up and troubleshoots a promptfoo evaluation pipeline for LLM prompts, covering local execution, Ollama provider configuration, YAML config structure, template variable injection, the prompt×test Cartesian product pitfall, and result inspection.

Core Features & Use Cases

Local evaluation workflow for promptfoo CI/CD labs, including provider configuration, YAML structures, and result inspection.
Guidance to prevent the Cartesian product pitfall by organizing tests and prompts, and by validating variable injection.
Clear troubleshooting guidance for bootstrapping and debugging a promptfoo lab in a dev container or monorepo.

Quick Start

Clone this repository, install dependencies, and open the project in a Dev Container to validate the evaluation workflow.

Frequently Asked Questions about promptfoo-ci-lab

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I set up a promptfoo evaluation pipeline locally?▼

To set up a promptfoo evaluation pipeline locally, clone the repository, install dependencies, and open the project in a Dev Container to validate the YAML config and evaluation workflow.

Can I use Ollama and OpenAI providers in the same promptfoo YAML config?▼

Yes, promptfoo supports provider switches between OpenAI and Ollama within the YAML configuration, allowing you to orchestrate LLM prompt evaluations across both local and provider-backed runtimes.

Why does promptfoo run more test combinations than expected?▼

Unexpected test volume in promptfoo usually stems from the prompt×test Cartesian product pitfall. You can prevent this by carefully organizing tests, validating variable injection, and structuring YAML configurations properly.

What is the best way to debug LLM prompt evaluation failures in a dev container?▼

The best way to debug LLM prompt evaluation failures in a dev container is to inspect the collected promptfoo results output, which helps identify test failures and verify template variable injection.

How does template variable injection work in promptfoo YAML configurations?▼

Template variable injection in promptfoo YAML configurations works by mapping test variables into prompt templates, ensuring deterministic evaluation workflows and reliable prompt testing across different providers.