llm-obs-eval

You can run LLM evaluation experiments by executing the Python experiment runner to orchestrate multiple scenarios and capture Datadog spans for costs, tokens, and tool calls. The runner outputs a Datadog Experiment URL for viewing results.

Datadog LLM observability captures per-turn prompts, responses, tool calls, and token usage data. It records this data as spans and export data to help you analyze agent behavior and tool performance across scenarios.

Yes, you can evaluate agent tool selection across predefined scenarios including greeting, product_question, and feature_question. The experiment runner orchestrates these end-to-end scenarios to assess agent behavior programmatically.

You need correctly loaded Datadog API keys from your container environment to capture token costs and trace data. The Python-based tooling relies on these keys to interact with the agent turns API and Datadog Export API.

You query and export LLM evaluation results programmatically using Datadog's Export API. This allows you to extract and analyze collected costs, tokens, and tool performance metrics after running your experiments.

The best way to automate repeatable LLM agent evaluations is using a Python-based experiment runner that orchestrates scenarios and collects per-record evaluations. This ensures consistent data collection of costs and traces via Datadog observability.