Aggregate LLM evaluation results and generate per-test trend reports.
npx skills add https://github.com/C-Ross/LlamaOfFate --skill llmeval-tracking
Please help me install this Agent Skill. Skill: llmeval-tracking Source: https://github.com/C-Ross/LlamaOfFate/tree/main/.github/skills/llmeval-tracking Command: npx skills add https://github.com/C-Ross/LlamaOfFate --skill llmeval-tracking
Track and diagnose the stability of LLM evaluation tests across CI runs, surfacing flaky tests and trends to speed up debugging and reliability.
Run the llmeval-tracker CLI to fetch the latest results and generate a stability report.
High-intent search queries and answers about installing and using this skill.
To track LLM evaluation flakiness across CI runs, fetch per-run outcomes, aggregate the data, and generate stability trend reports. Results are stored as JSONL to pinpoint flaky tests and speed up debugging.
Identify flaky tests by aggregating per-run evaluation outcomes and comparing them against historical baselines. The tracker analyzes these trends to surface flakiness and pinpoint failing tests.
Yes, you can store LLM evaluation results in JSONL format. The result aggregation feature collects per-run evaluation outcomes and stores them as JSONL for easy analysis and stability tracking.
The best way to generate stability reports is to run the llmeval-tracker CLI. It fetches the latest results, generates per-test trend reports, and helps pinpoint flaky tests across CI runs.
Yes, the llmeval-tracker works with daily CI workflows to fetch and report LLM evaluation results from multiple runs. This CI integration helps surface flaky tests and stability trends.
FrekiManagarmDiagnose failing LLM evaluations by analyzing pass/fail/N/A patterns across scored events.
DevelopersGlobalImplement structured logging, distributed tracing, and alerting with OpenTelemetry.
Dev-GOMRun Unity Test Framework tests and generate failure reports.
jmaglyAnalyze CI history and test execution patterns to identify flaky tests.
