skills.rest
shichiyou
shichiyouCOMMUNITY
Explore all skills by this author →
GitHub

promptfoo-ci-lab

Set up and troubleshoot promptfoo evaluation pipelines for LLM prompts.

Updated Aug 27, 2026
Search Tags:#yaml#ci-cd#devcontainer#ollama#promptfoo#lab#llm-eval
One-click install
npx skills add https://github.com/shichiyou/hermes-agent-001 --skill promptfoo-ci-lab
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: promptfoo-ci-lab
Source: https://github.com/shichiyou/hermes-agent-001/tree/main/.devcontainer/hermes-backup/skills/.archive/promptfoo-ci-lab
Command: npx skills add https://github.com/shichiyou/hermes-agent-001 --skill promptfoo-ci-lab

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Sets up and troubleshoots a promptfoo evaluation pipeline for LLM prompts, covering local execution, Ollama provider configuration, YAML config structure, template variable injection, the prompt×test Cartesian product pitfall, and result inspection.

Core Features & Use Cases

  • Local evaluation workflow for promptfoo CI/CD labs, including provider configuration, YAML structures, and result inspection.
  • Guidance to prevent the Cartesian product pitfall by organizing tests and prompts, and by validating variable injection.
  • Clear troubleshooting guidance for bootstrapping and debugging a promptfoo lab in a dev container or monorepo.

Quick Start

Clone this repository, install dependencies, and open the project in a Dev Container to validate the evaluation workflow.

Frequently Asked Questions about promptfoo-ci-lab

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up a promptfoo evaluation pipeline locally?▼

To set up a promptfoo evaluation pipeline locally, clone the repository, install dependencies, and open the project in a Dev Container to validate the YAML config and evaluation workflow.

Can I use Ollama and OpenAI providers in the same promptfoo YAML config?▼

Yes, promptfoo supports provider switches between OpenAI and Ollama within the YAML configuration, allowing you to orchestrate LLM prompt evaluations across both local and provider-backed runtimes.

Why does promptfoo run more test combinations than expected?▼

Unexpected test volume in promptfoo usually stems from the prompt×test Cartesian product pitfall. You can prevent this by carefully organizing tests, validating variable injection, and structuring YAML configurations properly.

What is the best way to debug LLM prompt evaluation failures in a dev container?▼

The best way to debug LLM prompt evaluation failures in a dev container is to inspect the collected promptfoo results output, which helps identify test failures and verify template variable injection.

How does template variable injection work in promptfoo YAML configurations?▼

Template variable injection in promptfoo YAML configurations works by mapping test variables into prompt templates, ensuring deterministic evaluation workflows and reliable prompt testing across different providers.

Related Skills

Explore More Software Engineering Skills→
jeffreykthomasjeffreykthomas

add-ollama-tool

Expose on-host Ollama models to NanoClaw agents via an MCP server.

Community
Intermediate
daymadedaymade
1.3k

promptfoo-evaluation

Configure and run Promptfoo LLM evaluation workflows with YAML configs and CLI commands.

Community
Advanced
pinion05pinion05
3

llm-cli

Automate multi-provider LLM usage via the llm CLI with file and stdin inputs.

Community
Advanced
sumik5sumik5
1

evaluating-with-promptfoo

Evaluate LLM outputs and run red-team tests with promptfoo.

Community
Advanced