eval-driven-dev

Automate LLM app QA by instrumenting code with `enable_storage()` and `@observe`, building golden datasets, writing eval tests, running them via `pixie test`, and inspecting traces to investigate failures.

Catch regressions after prompt changes by building a golden dataset from real runs and writing eval tests using `assert_dataset_pass` to verify LLM-dependent behavior remains consistent.

Build evaluation datasets from real runs by instrumenting your Python application and using `pixie dataset save` to capture LLM inputs and outputs as golden data for regression testing.

Investigate LLM test failures by running evals via `pixie test` and inspecting the generated traces to debug agent behavior and identify where the LLM-dependent logic diverges from expectations.

Yes, this Skill guides agents through the full QA loop using the pixie-qa Python package to instrument code, generate datasets, write evaluations, and debug agent behavior through trace inspection.

The best way to set up an evaluation pipeline is automating the full QA loop: instrumenting with tracing decorators, generating datasets, creating eval tests, and running them to ensure output quality before shipping.