skills.rest
launchdarkly-labs
launchdarkly-labsOFFICIAL
Explore all skills by this author →
WebsiteGitHub

aiconfig-online-evals

Automate scoring of AI Config responses with LLM-as-a-judge.

Updated Feb 4, 2026
Search Tags:#monitoring#accuracy#relevance#evaluate#LLM-judge#AI-configs#toxicity
One-click install
npx skills add https://github.com/launchdarkly-labs/aiconfigs-skills --skill aiconfig-online-evals
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: aiconfig-online-evals
Source: https://github.com/launchdarkly-labs/aiconfigs-skills/tree/main/aiconfig-online-evals
Command: npx skills add https://github.com/launchdarkly-labs/aiconfigs-skills --skill aiconfig-online-evals

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Automatically score AI Config responses using LLM-as-a-judge methodology to ensure accuracy, relevance, and safety across variations.

Core Features & Use Cases

  • Built-in judges: accuracy, relevance, toxicity with 0.0-1.0 scoring.
  • Async evaluation: results appear in the Monitoring tab after a brief delay.
  • Works with AI Configs in completion mode; integrates via the LaunchDarkly UI for judge configuration.

Quick Start

Enable judges for your AI Config in the LaunchDarkly UI (AI Configs -> your config -> Variations -> Attach judges). Then run a Python-based completion request using the aiconfig-sdk and inspect the Monitoring tab for judge scores.

Frequently Asked Questions about aiconfig-online-evals

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automatically evaluate AI Config responses for accuracy and relevance?▼

Use LLM-as-a-judge evaluation to automatically score AI Config responses for accuracy, relevance, and toxicity. Enable judges in the LaunchDarkly UI under your AI Config's Variations, then run completion requests via aiconfig-sdk to generate asynchronous judge scores visible in the Monitoring tab.

What judges are available for evaluating AI Configs?▼

Three built-in judges score AI Config responses: accuracy, relevance, and toxicity. Each produces a 0.0–1.0 score and requires configuration in the LaunchDarkly UI before evaluation begins.

Can I use LLM-based judges to monitor AI Config variations?▼

Yes. Attach judges to AI Config variations in the LaunchDarkly UI, then run completion requests using aiconfig-sdk. Judge scores appear asynchronously in the Monitoring tab, enabling automated quality assessment across variations.

What setup do I need to start evaluating AI Configs with judges?▼

Configure judges for your AI Config in the LaunchDarkly UI, ensure aiconfig-sdk is installed, and run completion-mode requests. Results populate the Monitoring tab after evaluation completes.

Does LLM-as-a-judge evaluation work with all AI Config types?▼

Judges work with AI Configs in completion mode only. Configure judges per variation in the LaunchDarkly UI, then trigger evaluation via aiconfig-sdk completion requests.

Related Skills

Explore More Software Engineering Skills→
launchdarklylaunchdarkly
25

aiconfig-online-evals

Attach judges to AI Config variations for automated LLM-based evaluation.

Official
Advanced
ajayatwal1105-emersonajayatwal1105-emerson

quality-report

Analyze brand evaluation logs to generate quality trend reports and regression alerts.

Community
Advanced
launchdarklylaunchdarkly
25

aiconfig-custom-metrics

Manage custom metrics for LaunchDarkly AI Configs via API and SDK.

Official
Advanced
abdullah1854abdullah1854
1

agent-evaluation

Develop evaluation frameworks for non-deterministic agent systems.

Community
Advanced