What problem does it solve? LLM prompts often underperform in production, but teams lack a systematic way to find failures and improve them. This Skill extracts prompts from OpenInference trace spans, gathers evaluation and annotation feedback, and runs a structured optimization loop so prompt changes are driven by real performance data instead of guesswork. ## Core Features & Use Cases - Prompt Extraction from Traces: Locate and reconstruct prompts from LLM spans, including structured chat messages, prompt templates, and template variables, using the ax CLI and jq. - Performance Signal Gathering: Pull human annotations, LLM-as-judge eval scores and explanations, error spans, and experiment results to identify exactly where and why a prompt fails. - Guided Optimization Loop: Apply a meta-prompt that generates revised prompts while preserving template variables and avoiding overfitting, then measure improvement via A/B experiment comparison. - Use Case: A RAG chatbot hallucinates answers. Export unfaithful traces, inspect retriever and LLM spans, add grounding instructions to the system prompt, and verify the fix with a new experiment run. ## Quick Start Ask the AI to find failing LLM traces in your Arize project and optimize the prompt based on the evaluation feedback.
