eval-harness

Build reproducible evals for LLM pipelines by defining test cases, scoring rubrics, and expected outputs. A runner executes the pipeline against all cases, while a scorer applies rubrics to record pass/fail results and track evaluation history.

An evaluation harness for LLM quality assurance is a structured framework that measures LLM pipelines using test cases and rubrics. It replaces ad-hoc QA by providing a repeatable runner, scorer, and historical tracker to reduce regression risk.

Yes, you can use an eval harness for prompt-injection resilience testing by configuring optional adversarial tests. These tests run alongside standard evaluation cases to measure and track your pipeline's resistance against prompt-injection attacks.

Track LLM regression testing results across multiple runs using a tracker component backed by a SQL session DB. It maintains a persistent pass/fail history, allowing you to compare current evaluation outcomes against past baseline results.

Yes, the evaluation framework supports testing agent workflows. It applies across various evaluation tasks including prompts, complex pipelines, and agent workflows, using a consistent configuration system for all test cases and rubrics.