llm-obs-eval

Name: llm-obs-eval
Availability: InStock
Author: speqqai

Run Datadog LLM Observability Experiments to evaluate agent-tool interactions.

Updated Apr 27, 2026

Search Tags:#evaluation #datadog #tool-selection #experiments #token-cost #llm-observability

Ready to run

npx skills add https://github.com/speqqai/claude --skill llm-obs-eval

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: llm-obs-eval
Source: https://github.com/speqqai/claude/tree/main/skills/llm-obs-eval
Command: npx skills add https://github.com/speqqai/claude --skill llm-obs-eval

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This skill enables automated, instrumented evaluation of agent behavior and tool usage using Datadog LLM Observability Experiments.

Core Features & Use Cases

End-to-end experiment orchestration across multiple scenarios (greeting, product_question, feature_question).
Data collection of per-turn prompts, responses, tool calls, and token usage with Datadog spans and export data.
Programmatic result querying and export via Datadog's Export API to analyze costs, tokens, and tool performance.

Quick Start

Run the Python experiment runner to launch all scenarios and view the resulting Datadog Experiment URL.

Frequently Asked Questions about llm-obs-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I run LLM evaluation experiments with Datadog observability?▼

You can run LLM evaluation experiments by executing the Python experiment runner to orchestrate multiple scenarios and capture Datadog spans for costs, tokens, and tool calls. The runner outputs a Datadog Experiment URL for viewing results.

What data does Datadog LLM observability capture during agent tool interactions?▼

Datadog LLM observability captures per-turn prompts, responses, tool calls, and token usage data. It records this data as spans and export data to help you analyze agent behavior and tool performance across scenarios.

Can I evaluate agent tool selection across multiple scenarios using Datadog experiments?▼

Yes, you can evaluate agent tool selection across predefined scenarios including greeting, product_question, and feature_question. The experiment runner orchestrates these end-to-end scenarios to assess agent behavior programmatically.

Do I need specific environment setup to capture token costs in Datadog LLM experiments?▼

You need correctly loaded Datadog API keys from your container environment to capture token costs and trace data. The Python-based tooling relies on these keys to interact with the agent turns API and Datadog Export API.

How do I query and export LLM evaluation results from Datadog?▼

You query and export LLM evaluation results programmatically using Datadog's Export API. This allows you to extract and analyze collected costs, tokens, and tool performance metrics after running your experiments.

What is the best way to automate repeatable LLM agent evaluations?▼

The best way to automate repeatable LLM agent evaluations is using a Python-based experiment runner that orchestrates scenarios and collects per-record evaluations. This ensures consistent data collection of costs and traces via Datadog observability.