skills.rest
ultraviollettnympho
ultraviollettnymphoCOMMUNITY
Explore all skills by this author →
GitHub

eval-driven-dev

Instruments Python LLM applications and builds trace-based evaluation test pipelines.

1|1|Updated Jan 29, 2026
Search Tags:#testing#python#llm evaluation#instrumentation#regression testing#golden dataset#pixie-qa
One-click install
npx skills add https://github.com/ultraviollettnympho/transit-ticket --skill eval-driven-dev-ultraviollettnympho
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-driven-dev
Source: https://github.com/ultraviollettnympho/transit-ticket/tree/main/.github/skills/eval-driven-dev
Command: npx skills add https://github.com/ultraviollettnympho/transit-ticket --skill eval-driven-dev-ultraviollettnympho

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pixie-qa, and includes references (resource) components.

What problem does it solve? Python LLM applications are hard to test because their outputs are non-deterministic, so prompt changes and regressions slip through unnoticed. This Skill sets up a complete evaluation pipeline that captures real traces from your app, builds golden datasets from actual executions, and runs eval-based tests to catch quality regressions. ## Core Features & Use Cases - Instrumentation and Trace Capture: Adds pixie-qa instrumentation to existing production code paths and stores traces in a local SQLite database. - Golden Dataset Building: Runs the real application through a harness to capture genuine inputs and outputs, never fabricated data. - Eval-Based Testing: Writes tests using evaluators like FactualityEval, ValidJSONEval, and RAG evaluators, run via the pixie test runner with configurable pass thresholds. - Use Case: After changing a system prompt in your RAG chatbot, run the eval suite to verify that factual accuracy and answer relevancy scores still meet your thresholds before shipping. ## Quick Start Ask the AI to set up eval-driven testing for your Python LLM application using the pixie-qa package.

Frequently Asked Questions about eval-driven-dev

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I test a Python LLM application for regressions?▼

Instrument the production code with pixie-qa, capture real traces by running the app through a harness, build a golden dataset from actual outputs, and write eval tests using evaluators like FactualityEval. Run them with the pixie test command after any prompt or code change.

What is an eval-boundary function in LLM testing?▼

The eval-boundary function is the production function that takes structured input, calls the LLM, and returns the response. Everything upstream like webhooks and request parsing is mocked, while everything at and below the function is the real code being evaluated.

Which evaluators should I use for RAG pipeline testing?▼

For RAG pipelines use ContextRelevancyEval to check retrieved context relevance, FaithfulnessEval to verify answers match the context, and AnswerRelevancyEval or AnswerCorrectnessEval for final output quality. These are LLM-as-judge evaluators requiring an OpenAI API key.

Why does my eval test run fail with missing API key errors?▼

LLM-as-judge evaluators like FactualityEval need OPENAI_API_KEY set in the environment. Variables in a .env file are not visible to shell commands or the pixie CLI unless exported, so export the key directly or confirm the app calls load_dotenv before running.

Can I write expected outputs into the dataset JSON manually?▼

You can add expected_output reference answers, but never fabricate eval_output values by hand. Every eval_output must come from a real execution of the app, otherwise the dataset tests a fiction and gives false confidence in quality.

Related Skills

Explore More Software Engineering Skills→
Aradhya0510Aradhya0510
3

instrumenting-with-mlflow-tracing

Instrument Python and TypeScript code with MLflow Tracing for observability.

Community
Intermediate
JoeyJoziahJoeyJoziah

Verification & Quality Assurance

Detect quality regressions and unreliable outputs in codebases and agent tasks.

Community
Advanced
nguyenpv1980-wqnguyenpv1980-wq
2

vitest-unit-component-engineer

Write Vitest unit and component tests with environment selection and deterministic mocks.

Community
Advanced
OpenLAIROpenLAIR
1.0k

phoenix-observability

Trace, evaluate, and monitor LLM applications with OpenTelemetry and self-hosted storage.

Official
Advanced