LongMemEval Judge

Evaluate QA answers in the LongMemEval benchmark using Anthropic or OpenAI services.

3|1|Updated Apr 11, 2026

Search Tags:#benchmarking #QA #evaluation #LLM #Anthropic

Ready to run

npx skills add https://github.com/tmuskal/arc-agi-benchmarker --skill longmemeval-judge

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: LongMemEval Judge
Source: https://github.com/tmuskal/arc-agi-benchmarker/tree/main/plugins/longmemeval-benchmarker/skills/judge
Command: npx skills add https://github.com/tmuskal/arc-agi-benchmarker --skill longmemeval-judge

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires anthropic, openai, and includes scripts (resource) components.

What problem does it solve?

The LongMemEval Judge Skill provides an interface for LLM-based evaluation of questions in the LongMemEval benchmark, with automatic fallback to OpenAI in case of API key or service issues.

Core Features & Use Cases

Automatic Evaluation: Quickly evaluate answers to questions within the LongMemEval framework using an LLM.
Fallback Mechanism: Ensures evaluation can proceed even if the primary Anthropic API key is not available or if service issues occur.
Single-item and Batch Processing: Support for both individual evaluations and batch processing of results.

Quick Start

Use the /arc-agi-benchmarker:run-benchmark command to evaluate a new item in LongMemEval.

Frequently Asked Questions about LongMemEval Judge

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I evaluate QA answers in the LongMemEval benchmark?▼

To evaluate QA answers in the LongMemEval benchmark, you can use this LLM-as-judge Skill to automatically assess responses via Anthropic or OpenAI services, supporting both single-item and batch processing within your testing framework.

Can I use OpenAI as a fallback for LLM evaluation if Anthropic API access fails?▼

Yes, this LLM evaluation tool provides an automatic fallback to OpenAI, ensuring your benchmarking evaluations proceed uninterrupted if your primary Anthropic API key is unavailable or encounters service issues.

What do I need to run automated LLM-based evaluation for LongMemEval?▼

To run automated LLM-based evaluation for LongMemEval, you need Python installed and valid API access for either Anthropic or OpenAI services to process and judge the QA dataset items.

How does batch processing work for LongMemEval benchmark evaluations?▼

Batch processing for LongMemEval benchmark evaluations allows you to assess multiple QA result items simultaneously using the LLM-as-judge, streamlining automated testing workflows within your benchmarking frameworks.

Anthropic API key is not working for my QA evaluation, how can I proceed?▼

If your Anthropic API key is not working for QA evaluation, this tool automatically falls back to the OpenAI API to ensure your LongMemEval benchmark assessment completes successfully without manual intervention.