skills.rest
truera
trueraOFFICIAL
Explore all skills by this author →
Website𝕏@truera_aiGitHub

trulens-diagnosis

Diagnose low LLM evaluation scores and generate targeted improvement recommendations.

3.5k|335|Updated Nov 2, 2020
Search Tags:#diagnosis#LLM evaluation#Root Cause Analysis#trulens#rAG#OTel tracing#feedback functions
One-click install
npx skills add https://github.com/truera/trulens --skill trulens-diagnosis
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: trulens-diagnosis
Source: https://github.com/truera/trulens/tree/main/src/core/trulens/.agents/skills/trulens-diagnosis
Command: npx skills add https://github.com/truera/trulens --skill trulens-diagnosis

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires trulens-core, trulens-dashboard, pandas.

What problem does it solve?

After running LLM evaluations, teams often see low scores but lack a systematic way to find the root cause and decide what to fix. This Skill closes the loop from evaluation results to concrete code changes by tracing failing metrics back to specific spans and prescribing targeted fixes.

Core Features & Use Cases

  • Failure Triage: Pull records with feedback scores via TruSession, filter low-scoring records, and identify which feedback functions fail most often.
  • Root Cause Analysis: Map failing metrics (Context Relevance, Groundedness, Answer Relevance, Tool Selection, Plan Adherence, Logical Consistency) to specific OTEL span types and inspect trace JSON to locate the problematic step.
  • Targeted Fix Recipes: Apply concrete fixes per failure mode, such as tuning retrieval k and chunking, strengthening grounding prompts, improving tool descriptions, or lowering temperature.
  • Regression Verification: Re-run evaluations under a new app version, compare on the leaderboard, and flag any metric that regressed after a fix.
  • Use Case: A RAG application scores 0.4 on Context Relevance. Use this Skill to trace the failure to the RETRIEVAL span, discover the chunk size is too large, apply a smaller chunking strategy, and verify the score improves in v2 without hurting Groundedness.

Quick Start

Diagnose why my RAG app's Context Relevance scores are below 0.7 and recommend fixes using my TruLens session records.

Frequently Asked Questions about trulens-diagnosis

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I diagnose low evaluation scores in TruLens?▼

Use TruSession.get_records_and_feedback() to pull records with feedback columns, filter records below a threshold such as 0.7, and group failures by feedback function. Then inspect the OTEL trace JSON of failing records to locate the problematic span.

How do I fix low Context Relevance in a RAG application?▼

Low Context Relevance usually traces to the RETRIEVAL span where retrieved contexts do not match the query. Fixes include increasing retrieval k, reducing chunk size with overlap, switching embedding models, or adding query rewriting before retrieval.

Which span type corresponds to each failing feedback metric?▼

Context Relevance maps to RETRIEVAL spans, Groundedness to GENERATION spans, Answer Relevance to RECORD_ROOT, Tool Selection and Tool Calling to TOOL or MCP spans, and Plan Adherence to AGENT spans. Logical Consistency requires reviewing the full trace.

Why is my TruLens trace JSON empty when diagnosing failures?▼

OTEL tracing is enabled by default, so check for a leftover TRULENS_OTEL_TRACING=0 environment variable disabling it and confirm spans are being captured. You can also enable debug logging with logging.basicConfig(level=logging.DEBUG).

How do I verify a fix did not regress other metrics?▼

Re-run the same test set under a new app_version, then compare per-metric means between versions using get_records_and_feedback. Flag any metric that dropped by more than 0.05, since a fix for one metric can degrade another.

Related Skills

Explore More Software Engineering Skills→
trueratruera
3.5k

trulens-evaluation-setup

Configure TruLens feedback functions and selectors for LLM output evaluation.

Official
Advanced
trueratruera
3.5k

trulens-evaluation-workflow

Orchestrates end-to-end TruLens evaluation workflows for LLM apps.

Official
Advanced
Supporter09Supporter09

langsmith-observability

Trace, evaluate, and monitor LLM application runs with LangSmith.

Community
Advanced
badhopebadhope

llm-evaluation

Evaluate LLM performance, quality, and safety using Python frameworks.

Community
Advanced