skills.rest
nsuberi
nsuberiCOMMUNITY
Explore all skills by this author →
GitHub

log-analyst

Analyze JSONL logs to detect retrieval mismatches and latency anomalies.

Updated Jul 5, 2025
Search Tags:#log analysis#confidence scoring#agent debugging#jsonl#product analytics#retrieval mismatch#latency forensics
One-click install
npx skills add https://github.com/nsuberi/ai-prototype-hub --skill log-analyst
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: log-analyst
Source: https://github.com/nsuberi/ai-prototype-hub/tree/main/apps/ai-builders-challenge/.claude/skills/log-analyst
Command: npx skills add https://github.com/nsuberi/ai-prototype-hub --skill log-analyst

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

It helps you interpret JSONL logs from an agentic system and quickly identify the high-impact mismatches and performance issues that explain why the agent seems wrong.

Core Features & Use Cases

  • Retrieval mismatch detection: Verifies whether retrieved property IDs match the requested property IDs within each session.
  • Confidence and scoring pattern analysis: Analyzes top_score distributions to flag low-confidence retrievals or suspiciously high scores tied to wrong outputs.
  • Latency and completeness auditing: Flags slow LLM calls (p95 thresholds) and missing response events that indicate broken execution traces.
  • Cross-session anomaly spotting: Detects repeated retrieved IDs across multiple borrowers that may indicate a filter or isolation bug.

Quick Start

Paste the JSONL logs from a borrower-agent run into the skill input and ask for the sessions, events, and fields that indicate retrieval mismatches, latency spikes, or hallucination-related patterns.

Frequently Asked Questions about log-analyst

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I analyze JSONL logs to find borrower-agent retrieval mismatches?▼

To analyze JSONL logs for retrieval mismatches, group events by session_id and verify retrieved_ids against the requested property_id to surface incorrect property matches within each session.

What is the best way to detect high latency in LLM call logs?▼

Detect high latency in LLM call logs by computing execution timing across sessions and flagging traces where the llm_call p95 latency exceeds the 3000ms threshold to identify slow responses.

How do I check confidence scores in agent logs for hallucination patterns?▼

Check confidence scores in agent logs by analyzing top_score distributions to flag low-confidence retrievals or suspiciously high scores tied to wrong outputs, indicating potential hallucination patterns.

Why does my agent return repeated retrieved IDs across different borrower sessions?▼

Repeated retrieved IDs across different borrower sessions may indicate a filter or isolation bug, which you can spot by reporting cross-session retrieval repeats to detect improper data scoping.

How do I identify broken execution traces in structured agent logs?▼

Identify broken execution traces in structured agent logs by auditing for missing response events and slow LLM calls that indicate incomplete or failed execution paths across borrower sessions.

Related Skills

Explore More Data & Analytics Skills→
penkzhoupenkzhou
2

workflow-logging

Automate structured JSONL and text logging for Swiss Army Knife plugin workflows.

Community
Intermediate
hung-phanhung-phan

grpc

Identifies RPC design and debugging issues in gRPC/Protobuf services using HTTP/2.

Community
Advanced
daffy0208daffy0208
34

product-analytics

Implement product analytics event tracking, funnels, cohorts, and metrics.

Community
Intermediate
harshaldhadukharshaldhaduk
1

worker-benchmarks

Benchmark agentic-flow worker operations against configurable latency thresholds.

Community
Intermediate