gsm8k-eval

Extract final numeric answers from GSM8K model outputs using official ground-truth formats.

3|Updated Dec 22, 2025

Search Tags:#evaluation #math #dataset #ground-truth #gsm8k #answer-extraction #model-scoring

Ready to run

npx skills add https://github.com/JoaquinCampo/Skills --skill gsm8k-eval

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: gsm8k-eval
Source: https://github.com/JoaquinCampo/Skills/tree/main/gsm8k-eval
Command: npx skills add https://github.com/JoaquinCampo/Skills --skill gsm8k-eval

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

GSM8K evaluation protocol and answer extraction standardizes how to score model reasoning on math problems by focusing on the final numeric answer and consistent ground-truth comparison.

Core Features & Use Cases

Defines extraction rules for final answers across common GSM8K formats (####, The answer is, Answer:, and last-number patterns) and robust normalization to ensure fair comparisons.
Provides guidance for dataset loading, ground-truth extraction, and accuracy scoring to enable reproducible research and benchmarking.
Useful for research teams evaluating language models on math-word problems, verifying prompts, and comparing decoding strategies.

Quick Start

Run the GSM8K evaluation workflow using the official ground-truth and the multi-pattern extractor to score model outputs.

Frequently Asked Questions about gsm8k-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I extract the final numeric answer for GSM8K evaluation?▼

GSM8K evaluation extracts final numeric answers from model outputs by matching common formats like ####, The answer is, Answer:, and boxed patterns. It normalizes extracted values before comparing them to ground-truth data.

What is the standard ground-truth format used for scoring the GSM8K dataset?▼

The standard ground-truth format for scoring the GSM8K dataset uses the official openai/gsm8k main config test split. Evaluation applies normalization and exact-match comparison against this data to compute robust accuracy.

How do I score model reasoning on math word problems consistently?▼

Score model reasoning on math word problems consistently by standardizing final answer extraction and applying robust normalization. Exact-match comparison against official ground-truth ensures reproducible benchmarking across different models and decoding strategies.

Does GSM8K evaluation support multiple answer formats during scoring?▼

Yes, GSM8K evaluation supports multiple answer formats during scoring. The multi-pattern extractor identifies final numeric values from ####, The answer is, Answer:, and last-number patterns before applying normalization and exact-match comparison.

What are the limitations of exact-match comparison for math evaluation?▼

Exact-match comparison for math evaluation requires precise normalization to succeed. If model outputs contain formatting anomalies not covered by the ####, Answer:, or boxed extraction patterns, scoring may fail to recognize correct numeric answers.