promptfoo-evals

To create a promptfoo evaluation suite, you place a configuration file in the evals/ directory and add corresponding prompts, tests, and assertions. This automates eval coverage and streamlines debugging across different models.

Debug prompt regressions by updating your promptfoo tests and assertions within the existing eval suite configuration. This allows you to isolate failing scenarios and refine prompts to restore deterministic outputs.

Yes, you can scaffold a new evaluation matrix across multiple models by configuring providers and extending test coverage. This generates diverse scenarios and configs within the evals/ directory structure.

Configuring promptfoo providers and assertions requires a well-defined file layout including evals/, prompts/, tests/, and provider configuration files. This structure ensures deterministic prompts and organized test coverage.

You need to add promptfoo assertions to eval tests when you require deterministic validation of prompt outputs. Assertions help define specific expected results to prevent regressions and maintain output quality.

The best way to organize promptfoo prompts and tests is maintaining a structured evals/ directory with separate prompts, tests, and provider configurations. This ensures maintainable evaluation suites and scalable coverage.