promptfoo-evaluation

LLM evaluations with Promptfoo involve initializing a project with npx promptfoo@latest init, configuring promptfooconfig.yaml with your prompts and test cases, then running npx promptfoo@latest eval to benchmark models and view results with npx promptfoo@latest view.

llm-rubric scoring uses an LLM as a judge to evaluate model outputs against defined criteria and thresholds. Promptfoo applies rubric-based grading to automatically assess prompt quality and consistency across test cases without manual scoring.

Yes. Promptfoo evaluations support comparing different models and prompt formats across varied test cases simultaneously, producing structured results that benchmark performance differences and help identify the best configuration.

Promptfoo supports Python custom assertions as metrics. Write Python scripts that validate outputs against your criteria, then reference them in your evaluation configuration to extend built-in grading with domain-specific logic.

Promptfoo uses YAML-based configuration (promptfooconfig.yaml) to define evaluation structure, file-based prompts and tests, and supports Python assertion scripts. This approach keeps evaluation projects version-controllable and reproducible.

Promptfoo handles evaluation logistics, but understanding your prompts and test objectives helps you structure better configurations. No advanced programming is required—YAML configuration and optional Python assertions support teams of varying technical depth.