phoenix-observability

LLM tracing with OpenTelemetry captures detailed traces and spans for LLM calls across frameworks to pinpoint failures and latency hotspots. You need to register OpenTelemetry with the correct traces endpoint in a running Phoenix server to collect this data.

LLM-as-judge evaluators score model outputs for hallucination, relevance, toxicity, and custom evaluation logic. These evaluation flows can log results back into Phoenix to monitor and assess output quality against defined criteria.

Regression testing on LLM prompts uses versioned datasets and repeatable experiments to compare prompts and models. You can run batch workflows across notebooks and production services to compare aggregate metrics before releasing changes.

Phoenix observability integrates with LangChain and LlamaIndex using openinference-instrumentation packages. It also supports OpenAI and Anthropic frameworks, requiring proper instrumentor integration to capture traces from these target LLM frameworks.

A Phoenix server setup is required to debug trace gaps in your LLM application. You must install arize-phoenix, register OpenTelemetry with the correct traces endpoint, and integrate instrumentors for your target LLM frameworks to view results in the Phoenix UI.

Production monitoring and feedback logging queries traces and spans to debug missing or incorrect chatbot responses. You can log user feedback and annotations, then export results for reporting to maintain visibility into production traffic.