using-braintrust

Name: using-braintrust
Availability: InStock
Author: Braintrust

Run Braintrust SQL log queries, batch logging, and automated LLM evaluations.

20|11|Updated Dec 20, 2025

Search Tags:#logging #llm evaluation #observability #ai monitoring #sql querying #braintrust

Ready to run

npx skills add https://github.com/braintrustdata/braintrust-claude-plugin --skill using-braintrust

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: using-braintrust
Source: https://github.com/braintrustdata/braintrust-claude-plugin/tree/main/skills/using-braintrust
Command: npx skills add https://github.com/braintrustdata/braintrust-claude-plugin --skill using-braintrust

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires braintrust, autoevals, requests, python-dotenv, and includes scripts (resource) components.

What problem does it solve?

This Skill eliminates the guesswork from LLM development by providing automated evaluation, logging, and SQL querying capabilities, saving you from manual testing and debugging.

Core Features & Use Cases

SQL Querying: Run complex analytics on your LLM logs with familiar SQL syntax.
Automated Evaluation: Test your models against ground truth data automatically.
Use Case: Imagine you've deployed a customer support chatbot. Use this Skill to automatically query conversation logs, calculate response quality metrics, and identify patterns in user queries.

Quick Start

Query the logs from the last 24 hours to see how your LLM is performing in production.

Frequently Asked Questions about using-braintrust

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I set up LLM evaluation and logging with SQL queries?▼

LLM evaluation with SQL queries enables you to automatically test models against ground truth and analyze logs using familiar SQL syntax. Braintrust handles this by accepting an API key from your environment, then running project-scoped query scripts (query_logs.py, log_data.py, run_eval.py) from your project directory with a .env file to retrieve and evaluate results.

Can I automate evaluation of my language models in production?▼

Automated evaluation tests your models against ground truth data without manual intervention. Braintrust supports custom or predefined scorers to evaluate LLM outputs, batch log metadata alongside predictions, and query results with SQL to identify performance patterns across your deployment.

What's the best way to monitor LLM performance and observability?▼

LLM observability combines logging, querying, and evaluation to track production behavior. Braintrust provides SQL-backed log queries to analyze conversation patterns, batch logging with metadata for context, and automated scoring to measure response quality, giving you real-time insight into model performance and user interactions.

Do I need to write custom evaluation logic, or are scorers included?▼

Braintrust supports both predefined scorers for common evaluation tasks and custom scorers you define. This flexibility lets you start with built-in metrics for standard quality checks or implement domain-specific logic tailored to your chatbot, classification, or generation use case.

What environment setup is required before running evaluations?▼

You need a Braintrust API key (stored in .env or as an environment variable), Python with dependencies installed (braintrust, autoevals, requests, python-dotenv), and a project directory where scripts access logs and run evaluations. This minimal setup lets you query and evaluate from any machine with credentials.

Can I query conversation logs from deployed chatbots to find patterns?▼

SQL querying on conversation logs lets you retrieve interactions, filter by time range or metadata, and calculate aggregate metrics like response quality or user satisfaction. This supports analyzing support chatbots, identifying common failure modes, and extracting ground truth for retraining.