aiconfig-online-evals

Automate scoring of AI Config responses with LLM-as-a-judge.

Updated Feb 4, 2026

Search Tags:#monitoring #accuracy #relevance #evaluate #LLM-judge #AI-configs #toxicity

Ready to run

npx skills add https://github.com/launchdarkly-labs/aiconfigs-skills --skill aiconfig-online-evals

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: aiconfig-online-evals
Source: https://github.com/launchdarkly-labs/aiconfigs-skills/tree/main/aiconfig-online-evals
Command: npx skills add https://github.com/launchdarkly-labs/aiconfigs-skills --skill aiconfig-online-evals

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Automatically score AI Config responses using LLM-as-a-judge methodology to ensure accuracy, relevance, and safety across variations.

Core Features & Use Cases

Built-in judges: accuracy, relevance, toxicity with 0.0-1.0 scoring.
Async evaluation: results appear in the Monitoring tab after a brief delay.
Works with AI Configs in completion mode; integrates via the LaunchDarkly UI for judge configuration.

Quick Start

Enable judges for your AI Config in the LaunchDarkly UI (AI Configs -> your config -> Variations -> Attach judges). Then run a Python-based completion request using the aiconfig-sdk and inspect the Monitoring tab for judge scores.

Frequently Asked Questions about aiconfig-online-evals

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I automatically evaluate AI Config responses for accuracy and relevance?▼

Use LLM-as-a-judge evaluation to automatically score AI Config responses for accuracy, relevance, and toxicity. Enable judges in the LaunchDarkly UI under your AI Config's Variations, then run completion requests via aiconfig-sdk to generate asynchronous judge scores visible in the Monitoring tab.

What judges are available for evaluating AI Configs?▼

Three built-in judges score AI Config responses: accuracy, relevance, and toxicity. Each produces a 0.0–1.0 score and requires configuration in the LaunchDarkly UI before evaluation begins.

Can I use LLM-based judges to monitor AI Config variations?▼

Yes. Attach judges to AI Config variations in the LaunchDarkly UI, then run completion requests using aiconfig-sdk. Judge scores appear asynchronously in the Monitoring tab, enabling automated quality assessment across variations.

What setup do I need to start evaluating AI Configs with judges?▼

Configure judges for your AI Config in the LaunchDarkly UI, ensure aiconfig-sdk is installed, and run completion-mode requests. Results populate the Monitoring tab after evaluation completes.

Does LLM-as-a-judge evaluation work with all AI Config types?▼

Judges work with AI Configs in completion mode only. Configure judges per variation in the LaunchDarkly UI, then trigger evaluation via aiconfig-sdk completion requests.