skills.rest
FrekiManagarm
FrekiManagarmCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

exploring-llm-evaluations

Diagnose failing LLM evaluations by analyzing pass/fail/N/A patterns across scored events.

Updated Feb 24, 2026
Search Tags:#posthog#trace analysis#llm judge#llm evaluations#quality diagnostics#hog evaluator#clickhouse sql
One-click install
npx skills add https://github.com/FrekiManagarm/dunlo --skill exploring-llm-evaluations-frekimanagarm
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: exploring-llm-evaluations
Source: https://github.com/FrekiManagarm/dunlo/tree/main/.agents/skills/exploring-llm-evaluations
Command: npx skills add https://github.com/FrekiManagarm/dunlo --skill exploring-llm-evaluations-frekimanagarm

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps you understand why LLM evaluations are failing by finding evaluation definitions, running tests, and summarizing pass/fail patterns across many generations.

Core Features & Use Cases

  • Investigate evaluation results across run types: Use one workflow for both deterministic hog checks and prompt-based llm_judge scoring by querying the shared $ai_evaluation event schema.
  • Inspect and manage evaluation configurations: List, fetch, create, update, enable/disable, run, test (Hog dry-run), and delete evaluation configs to pinpoint what changed or why a verdict differs.
  • Generate AI summaries and verify with raw data: Produce grouped pass/fail/N/A pattern summaries and then validate findings via ClickHouse SQL against evaluation events and reasoning fields.
  • Drill into representative failures: Use trace/generation lookup to inspect concrete examples behind each detected pattern.

Quick Start

Use the exploring-llm-evaluations skill to summarize why evaluation <evaluation_uuid> is failing by running the evaluation summary tool with filter set to fail.

Frequently Asked Questions about exploring-llm-evaluations

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I debug failing LLM evaluations in PostHog?▼

Debug failing LLM evaluations by locating evaluation configs and analyzing pass/fail/N/A patterns across scored generation events. You can run Hog dry-tests or evaluate specific generations to pinpoint configuration issues.

How does LLM judge evaluation differ from deterministic Hog evaluators?▼

LLM judge evaluations use prompt-based scoring, while Hog evaluators apply deterministic checks. This Skill analyzes both by querying the shared $ai_evaluation event schema to compare failure modes across run types.

How do I summarize pass/fail patterns for AI evaluation events using ClickHouse SQL?▼

Summarize pass/fail patterns by generating an LLM-powered evaluation summary, then verify the grouped findings using ClickHouse SQL queries against $ai_evaluation events and their reasoning fields.

Can I test an evaluation against a specific generation before enabling it?▼

Yes, you can test evaluations against specific generations and run Hog dry-runs to inspect outcomes. The workflow also lets you fetch, update, enable, disable, or delete evaluation configurations.

Why does my LLM evaluation return unexpected N/A results?▼

Unexpected N/A results often stem from configuration issues or mismatched generation inputs. Use trace and generation lookup to inspect concrete examples behind each detected pattern and validate findings via ClickHouse SQL.

Related Skills

Explore More Data & Analytics Skills→
tomzxtomzx
5

onboard-repository

Bootstrap repository onboarding with SDLC structure and automated issue triage workflows.

Community
Advanced
adamatdevopsadamatdevops
1

error-analysis

Analyze LLM pipeline traces to prioritize failure modes by root cause.

Community
Intermediate
gitlabhqgitlabhq
34

debug-clickhouse-queries

Debug ClickHouse query failures by inspecting generated SQL and table data.

Official
Advanced
PostHogPostHog
58

exploring-llm-evaluations

Inspect and manage Hog and llm_judge evaluation configurations on PostHog.

Official
Advanced