evalite

You can automate LLM evaluation by running test suites against LLM tasks using Vitest-based runners to systematically score outputs and trace token usage. This ensures model reliability and performance during development.

AI-as-judge evaluation for LLM applications is a testing mechanism that uses AI models to score output quality and factuality. It is supported alongside deterministic tests to compare model variants or prompt versions against datasets.

Yes, you can compare different LLM model variants and prompt versions by running automated evaluation suites against a dataset of queries. This process measures factuality, quality, and reasoning accuracy across configurations.

Yes, integration with Vitest and the AI SDK is required to run the LLM evaluation suites and perform both deterministic and AI-as-judge evaluations. These frameworks provide the necessary test runners and model interfaces.

You can implement custom scorers or use pre-built ones to measure LLM output factuality and quality. These scorers integrate into the Vitest-based evaluation runners to quantify model performance during testing.

You can trace token usage and latency during LLM testing by executing evaluation suites that capture detailed performance metrics. This tracing functionality is built into the Vitest-based testing and scoring workflow.