skills.rest
PostHog
PostHogOFFICIAL
Explore all skills by this author →
Website𝕏@PostHogGitHub

exploring-llm-evaluations

Inspect and manage Hog and llm_judge evaluation configurations on PostHog.

58|5|Updated Jan 19, 2026
Search Tags:#analytics#posthog#llm_judge#llm-evaluations#hog#evaluation-management
One-click install
npx skills add https://github.com/PostHog/skills --skill exploring-llm-evaluations
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: exploring-llm-evaluations
Source: https://github.com/PostHog/skills/tree/main/skills/omnibus/exploring-llm-evaluations
Command: npx skills add https://github.com/PostHog/skills --skill exploring-llm-evaluations

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Inspect, compare, and manage Hog (deterministic) and llm_judge (LLM-based) evaluations across configurations, runs, and summaries to enable debugging, optimization, and lifecycle governance.

Core Features & Use Cases

  • List, inspect, and manage evaluation configs for both Hog and LLM judge types.
  • Run evaluations against specific $ai_generation events, query results, and generate AI-powered summaries of pass/fail patterns.
  • Drill into failing examples and manage the evaluation lifecycle (create, update, enable/disable, delete).

Quick Start

Create a sample Hog or llm_judge evaluation, enable it, and run it against a representative generation to observe the results.

Frequently Asked Questions about exploring-llm-evaluations

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
What is the difference between Hog and LLM judge evaluations in PostHog?▼

LLM evaluations assess AI output quality by running Hog or llm_judge configs against $ai_generation events. This Skill coordinates running these evaluations, querying individual results, and generating AI-powered summaries of pass/fail patterns to diagnose issues.

How do I run LLM evaluations on specific AI generations and check the results?▼

Run LLM evaluations by selecting target $ai_generation events and executing the configured Hog or llm_judge evaluation. This Skill queries individual results and generates AI-powered summaries of pass/fail patterns to diagnose issues.

How do I manage the lifecycle of LLM evaluation configurations?▼

Manage LLM evaluation configurations by using lifecycle tools to create, update, enable, disable, and delete evaluations. This Skill supports both Hog and llm_judge types to ensure proper evaluation governance across the PostHog platform.

Can I use LLM evaluations to debug failing AI generation outputs?▼

You can debug failing AI generation outputs by running evaluations and drilling into failing examples. This Skill generates AI-powered summaries of pass/fail patterns to diagnose issues and optimize your LLM behavior.

When should I use deterministic Hog evaluations versus LLM judge evaluations?▼

Use deterministic Hog evaluations for rule-based checks and LLM judge evaluations for subjective quality assessment. This Skill provides safe guidance on when to use each type to optimize debugging and evaluation governance.

Related Skills

Explore More Data & Analytics Skills→
dallasbpetersdallasbpeters

llm-analytics-setup

Instrument LLM calls with PostHog analytics across major providers.

Community
Advanced
FrekiManagarmFrekiManagarm

exploring-llm-evaluations

Diagnose failing LLM evaluations by analyzing pass/fail/N/A patterns across scored events.

Community
Advanced
FrekiManagarmFrekiManagarm

exploring-llm-clusters

Analyze PostHog LLM clustering runs to compute cost, latency, and error metrics.

Community
Intermediate
maltemdmaltemd

designing-evaluations-for-agents

Design evaluation frameworks with metrics and pass/fail thresholds for LLM-based agents.

Community
Advanced