aiconfig-online-evals

Automate LLM evaluation for AI Config variations by attaching custom judges to enable LLM-as-a-judge scoring. This provides structured quality signals and automated output evaluation across your variations and environments.

Attaching judges to AI Configs requires a LaunchDarkly API access token with ai-configs:write permission. You also need compatible AI SDK versions to ensure automatic metric recording functions correctly during evaluation.

You can monitor LLM-as-a-judge scores across environments by viewing judge results in the LaunchDarkly dashboard. This allows you to track quality metrics, apply guardrails, and run experiments on your AI Config outputs.

Collect evaluation results from AI Config judges using configurable sampling to manage evaluation volume. This automated scoring mechanism gathers quality data across variations without requiring manual inspection of every single output.

Use custom judges for AI Config evaluation when managing multiple variations and requiring targeted, automated quality scoring. This approach provides structured evaluation signals necessary for monitoring governance and running experiments.

Automatic judge metrics may not record if your AI SDK versions are incompatible or if the API access token lacks ai-configs:write permission. Ensure both prerequisites are met to successfully attach judges and capture scores.