phoenix-observability

LLM observability uses OpenTelemetry-based tracing to collect end-to-end visibility into your application, capturing real-time insights to diagnose failures and monitor production deployments. You instrument your app to begin collecting traces after starting the server.

Evaluating LLM outputs involves running experiments to compare prompts, models, and configurations. You can use LLM-as-judge evaluators for quality assessment and leverage versioned datasets for regression testing to ensure reliable deployments.

Yes, you can self-host this open-source observability layer using either PostgreSQL or SQLite. This allows you to maintain full control over your tracing data, evaluation datasets, and experiments within your own infrastructure.

OpenTelemetry-based tracing works by collecting traces from LLM frameworks. This mechanism provides a clear visibility layer to diagnose failures, compare outputs, and monitor production deployments with real-time insights.

You can test multiple models interactively using a built-in playground. This allows you to perform interactive prompt testing and compare outputs before moving to production deployments or running formal experiments.

Versioned datasets are needed for LLM regression testing to compare model outputs consistently across different prompts and configurations. They provide a stable baseline to evaluate quality and ensure reliable deployments over time.