using-braintrust

LLM evaluation with SQL queries enables you to automatically test models against ground truth and analyze logs using familiar SQL syntax. Braintrust handles this by accepting an API key from your environment, then running project-scoped query scripts (query_logs.py, log_data.py, run_eval.py) from your project directory with a .env file to retrieve and evaluate results.

Automated evaluation tests your models against ground truth data without manual intervention. Braintrust supports custom or predefined scorers to evaluate LLM outputs, batch log metadata alongside predictions, and query results with SQL to identify performance patterns across your deployment.

LLM observability combines logging, querying, and evaluation to track production behavior. Braintrust provides SQL-backed log queries to analyze conversation patterns, batch logging with metadata for context, and automated scoring to measure response quality, giving you real-time insight into model performance and user interactions.

Braintrust supports both predefined scorers for common evaluation tasks and custom scorers you define. This flexibility lets you start with built-in metrics for standard quality checks or implement domain-specific logic tailored to your chatbot, classification, or generation use case.

You need a Braintrust API key (stored in .env or as an environment variable), Python with dependencies installed (braintrust, autoevals, requests, python-dotenv), and a project directory where scripts access logs and run evaluations. This minimal setup lets you query and evaluate from any machine with credentials.

SQL querying on conversation logs lets you retrieve interactions, filter by time range or metadata, and calculate aggregate metrics like response quality or user satisfaction. This supports analyzing support chatbots, identifying common failure modes, and extracting ground truth for retraining.