crud-otagent-supabase

Query, aggregate, and safely write OT-Agent eval data in Supabase.

276|37|Updated Dec 5, 2025

Search Tags:#database #supabase #aggregation #data-integrity #benchmarks #registration #model-ids

Ready to run

npx skills add https://github.com/open-thoughts/OpenThoughts-Agent --skill crud-otagent-supabase

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: crud-otagent-supabase
Source: https://github.com/open-thoughts/OpenThoughts-Agent/tree/main/.claude/skills/crud-otagent-supabase
Command: npx skills add https://github.com/open-thoughts/OpenThoughts-Agent --skill crud-otagent-supabase

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires supabase.

What problem does it solve?

The OT-Agent Supabase is the source of truth for model eval results (the numbers behind every ablation/paper table). The scores live in sandbox_jobs; this skill teaches how to query them correctly, aggregate ID/OOD means (and their binomial SE) the way the tables do, and write rows safely.

Core Features & Use Cases

Query and reconcile model IDs, benchmarks, and evaluation results from sandbox_jobs, models, and benchmarks.
Aggregate across multiple entries per model/benchmark, enforce identical-setting averaging, and handle sibling model rows.
Register new models and evaluations, repair failed uploads, and maintain FK safety with cross-user checks.

Quick Start

Connect locally using your secret environment variables and run the provided scripts to register models and evaluations.

Frequently Asked Questions about crud-otagent-supabase

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I query and aggregate OT-Agent evaluation data in Supabase?▼

To query and aggregate OT-Agent evaluation data in Supabase, you access the sandbox_jobs, models, and benchmarks schemas. You can aggregate ID/OOD means and binomial SE using canonical rules for multiple entries per model.

What is the best way to register new models and evaluation jobs in a Supabase registry?▼

The best way to register new models and eval jobs is by running provided scripts connected via local secrets. This safely writes rows to the registry and maintains cross-user foreign key safety.

How does Supabase handle data integrity for cross-user model evaluations?▼

Supabase handles data integrity for cross-user model evaluations by enforcing cross-user foreign key safety checks. This ensures relationships between models, benchmarks, and sandbox_jobs remain consistent across different users.

Can I repair failed uploads for model benchmark results in Supabase?▼

Yes, you can repair failed uploads for model benchmark results in Supabase. The skill provides scripts to reconcile and safely write rows, fixing failed uploads while maintaining database integrity.

How do I identify ID and OOD benchmarks when querying model evaluation results?▼

You identify ID and OOD benchmarks by querying the benchmarks schema within Supabase. The skill supports identifying these benchmark types to correctly aggregate their means and binomial standard errors.

Do I need local secrets to access the Supabase evaluation database?▼

Yes, you need local secrets to access the Supabase evaluation database. Connecting locally using your secret environment variables is required to run the scripts for querying and registering data.