llmeval-tracking

Aggregate LLM evaluation results and generate per-test trend reports.

2|Updated Aug 16, 2025

Search Tags:#testing #reports #analysis #ci #jsonl #flakiness #llmeval

Ready to run

npx skills add https://github.com/C-Ross/LlamaOfFate --skill llmeval-tracking

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: llmeval-tracking
Source: https://github.com/C-Ross/LlamaOfFate/tree/main/.github/skills/llmeval-tracking
Command: npx skills add https://github.com/C-Ross/LlamaOfFate --skill llmeval-tracking

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

Track and diagnose the stability of LLM evaluation tests across CI runs, surfacing flaky tests and trends to speed up debugging and reliability.

Core Features & Use Cases

Result aggregation: collects per-run evaluation outcomes and stores them as JSONL for easy analysis.
Flake detection: identifies flaky tests and provides trend insights over time.
CI integration: works with daily CI workflows to fetch and report results from multiple runs.
Guided analysis: helps engineers pinpoint failing tests and compare against historical baselines.

Quick Start

Run the llmeval-tracker CLI to fetch the latest results and generate a stability report.

Frequently Asked Questions about llmeval-tracking

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I track LLM evaluation flakiness across multiple CI runs?▼

To track LLM evaluation flakiness across CI runs, fetch per-run outcomes, aggregate the data, and generate stability trend reports. Results are stored as JSONL to pinpoint flaky tests and speed up debugging.

How do I identify flaky tests in my LLM evaluation results?▼

Identify flaky tests by aggregating per-run evaluation outcomes and comparing them against historical baselines. The tracker analyzes these trends to surface flakiness and pinpoint failing tests.

Can I store LLM evaluation results from CI in JSONL format for analysis?▼

Yes, you can store LLM evaluation results in JSONL format. The result aggregation feature collects per-run evaluation outcomes and stores them as JSONL for easy analysis and stability tracking.

What is the best way to generate stability reports for LLM evaluations?▼

The best way to generate stability reports is to run the llmeval-tracker CLI. It fetches the latest results, generates per-test trend reports, and helps pinpoint flaky tests across CI runs.

Does the llmeval-tracker script work with daily CI workflows?▼

Yes, the llmeval-tracker works with daily CI workflows to fetch and report LLM evaluation results from multiple runs. This CI integration helps surface flaky tests and stability trends.