promptfoo-evaluation

To set up an LLM evaluation project, create a Promptfoo project structure and define your prompts, providers, and tests in a promptfooconfig.yaml file. You can then execute the evaluation by running the npx promptfoo@latest eval command.

Yes, you can use Python-based custom assertions to evaluate LLM outputs. The framework supports writing Python assertions to apply nuanced scoring logic tailored to your specific testing scenarios.

LLM-rubric is an evaluation technique for implementing LLM-as-judge grading. It allows you to use a separate LLM to score and evaluate the outputs of your target models based on defined grading criteria within your test configuration.

You can compare model performance by configuring multiple providers within your promptfooconfig.yaml file. The framework manages prompts and tests across these diverse models to benchmark their performance side-by-side.

Yes, an existing Promptfoo project structure is required. The evaluation workflow depends on this structure to properly organize your prompt configurations, test cases, providers, and optional reference scripts.

You can leverage few-shot prompts within your configuration to guide model behavior during evaluations. This technique helps fine-tune the evaluation workflow by providing example inputs and outputs alongside your test cases.