promptfoo Eval Suite Author

You can build a promptfoo eval suite by organizing configs, prompts, and tests into a maintainable structure. The suite uses file-based or dataset-driven test cases with deterministic assertions to verify LLM features like chatbots and RAG pipelines.

The best way to write assertions for promptfoo eval suites is prioritizing deterministic checks like JSON shape validation, string matching, and regex before relying on model-graded assertions. This approach keeps test suites maintainable and results repeatable across CI runs.

Yes, promptfoo supports regression testing for RAG pipelines, HTTP APIs, chatbots, and prompt templates. It uses a low-provider setup with structured prompts and dataset-driven tests, making it suitable for running repeatable evaluation checks in CI environments.

Promptfoo works with local providers, HTTP APIs, and model comparisons for LLM evaluation. It selects the simplest provider pattern for your setup while keeping provider count low to ensure tests remain focused, structured, and repeatable.

YAML configs help organize promptfoo evaluation suites into a maintainable structure separating prompts, tests, and assertions. This file-based approach enables repeatable test case management and deterministic validation across prompt template changes and CI runs.