skills.rest
tmuskal
tmuskalCOMMUNITY
Explore all skills by this author →
𝕏@tmuskalGitHub

LongMemEval Judge

Evaluate QA answers in the LongMemEval benchmark using Anthropic or OpenAI services.

3|1|Updated Apr 11, 2026
Search Tags:#benchmarking#QA#evaluation#LLM#Anthropic
One-click install
npx skills add https://github.com/tmuskal/arc-agi-benchmarker --skill longmemeval-judge
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: LongMemEval Judge
Source: https://github.com/tmuskal/arc-agi-benchmarker/tree/main/plugins/longmemeval-benchmarker/skills/judge
Command: npx skills add https://github.com/tmuskal/arc-agi-benchmarker --skill longmemeval-judge

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires anthropic, openai, and includes scripts (resource) components.

What problem does it solve?

The LongMemEval Judge Skill provides an interface for LLM-based evaluation of questions in the LongMemEval benchmark, with automatic fallback to OpenAI in case of API key or service issues.

Core Features & Use Cases

  • Automatic Evaluation: Quickly evaluate answers to questions within the LongMemEval framework using an LLM.
  • Fallback Mechanism: Ensures evaluation can proceed even if the primary Anthropic API key is not available or if service issues occur.
  • Single-item and Batch Processing: Support for both individual evaluations and batch processing of results.

Quick Start

Use the /arc-agi-benchmarker:run-benchmark command to evaluate a new item in LongMemEval.

Frequently Asked Questions about LongMemEval Judge

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate QA answers in the LongMemEval benchmark?▼

To evaluate QA answers in the LongMemEval benchmark, you can use this LLM-as-judge Skill to automatically assess responses via Anthropic or OpenAI services, supporting both single-item and batch processing within your testing framework.

Can I use OpenAI as a fallback for LLM evaluation if Anthropic API access fails?▼

Yes, this LLM evaluation tool provides an automatic fallback to OpenAI, ensuring your benchmarking evaluations proceed uninterrupted if your primary Anthropic API key is unavailable or encounters service issues.

What do I need to run automated LLM-based evaluation for LongMemEval?▼

To run automated LLM-based evaluation for LongMemEval, you need Python installed and valid API access for either Anthropic or OpenAI services to process and judge the QA dataset items.

How does batch processing work for LongMemEval benchmark evaluations?▼

Batch processing for LongMemEval benchmark evaluations allows you to assess multiple QA result items simultaneously using the LLM-as-judge, streamlining automated testing workflows within your benchmarking frameworks.

Anthropic API key is not working for my QA evaluation, how can I proceed?▼

If your Anthropic API key is not working for QA evaluation, this tool automatically falls back to the OpenAI API to ensure your LongMemEval benchmark assessment completes successfully without manual intervention.

Related Skills

Explore More Software Engineering Skills→
layeddielayeddie
1

ai-orchestration

Manage and rotate LLM API keys across multiple providers.

Community
Advanced
tmuskaltmuskal
3

cross-harness

Automate LongMemEval benchmarking across Codex, Gemini, and OpenCode harnesses.

Community
Intermediate
DrLuggelsDrLuggels
3

llm-evaluation

Evaluate LLM applications with automated metrics, human feedback, and benchmarking.

Community
Advanced
tawf-labstawf-labs
1

langchain-rag

Build retrieval-augmented generation workflows that ground LLM answers in external documents.

Official
Intermediate