LongMemEval Judge
CommunityLLM-as-judge for LongMemEval, auto-fallback to OpenAI
Authortmuskal
Version1.0.0
Installs0
System Documentation
What problem does it solve?
The LongMemEval Judge Skill provides an interface for LLM-based evaluation of questions in the LongMemEval benchmark, with automatic fallback to OpenAI in case of API key or service issues.
Core Features & Use Cases
- Automatic Evaluation: Quickly evaluate answers to questions within the LongMemEval framework using an LLM.
- Fallback Mechanism: Ensures evaluation can proceed even if the primary Anthropic API key is not available or if service issues occur.
- Single-item and Batch Processing: Support for both individual evaluations and batch processing of results.
Quick Start
Use the /arc-agi-benchmarker:run-benchmark command to evaluate a new item in LongMemEval.
Dependency Matrix
Required Modules
anthropicopenai
Components
scripts
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: LongMemEval Judge Download link: https://github.com/tmuskal/arc-agi-benchmarker/archive/main.zip#longmemeval-judge Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 620,000+ vetted skills library on demand.