arize-prompt-optimization

To optimize LLM prompts, this skill extracts prompt templates from trace spans, analyzes evaluation scores and annotations, and applies a data-driven meta-prompting strategy to iteratively refine instructions based on identified failure patterns.

Data-driven prompt optimization works by gathering performance signals like human annotations, error rates, and evaluation scores from your existing traces, then using a meta-prompting approach to generate improved prompt versions targeting those specific failure modes.

You can extract a prompt template from an LLM trace span by using the skill's prompt reconstruction capabilities, which support structured chat messages, prompt templates, and fallback input values to accurately identify and pull the prompts used.

Yes, you need the ax CLI installed to use this skill, as it is required for retrieving trace data, extracting performance signals, and running the analysis needed for the iterative prompt refinement process.

Yes, you can debug factually incorrect chatbot responses by analyzing user interaction traces to identify common errors, then using the skill's meta-prompting strategy to refine the system instructions for better accuracy.

The effectiveness of meta-prompting for prompt refinement is limited by the quality of your trace data, requiring sufficient evaluation scores, human annotations, and error rate signals to accurately identify failure patterns and generate effective improvements.