skills.rest
woodfishhhh
woodfishhhhCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

paper-autoraters

Score citation faithfulness and literature-review quality in paper drafts.

32|1|Updated May 19, 2026
Search Tags:#llm-as-judge#literature review#semantic scholar#side-by-side comparison#paper scoring#citation f1
One-click install
npx skills add https://github.com/woodfishhhh/EZ_math_model --skill paper-autoraters-woodfishhhh
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: paper-autoraters
Source: https://github.com/woodfishhhh/EZ_math_model/tree/main/skills/ez-math-model/external/paper-orchestra/skills/paper-autoraters
Command: npx skills add https://github.com/woodfishhhh/EZ_math_model --skill paper-autoraters-woodfishhhh

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) and scripts (resource) components.

What problem does it solve?

It helps you rigorously evaluate a paper draft’s academic quality by running PaperOrchestra-inspired LLM-as-judge autoraters for citation faithfulness and literature-review strength.

Core Features & Use Cases

  • Citation F1 (P0/P1 partition): Categorizes references into must-cite (P0) and good-to-cite (P1) and computes Precision/Recall/F1 against a ground-truth paper using Semantic Scholar ID matching.
  • Literature Review Quality (6-axis 0-100): Scores only the Introduction + Related Work function with conservative, anti-inflation rules and produces an overall literature-review quality score.
  • Side-by-side (SxS) comparisons: Runs holistic paper quality and literature-review-only preference judgments between two papers, using a two-pass swapped-order protocol to reduce positional bias.
  • Use Case: When you generate a paper or paper iteration, you can score it against a benchmark/ground-truth draft and also compare two versions side-by-side to decide which draft is stronger.

Quick Start

Ask your agent to run paper-autoraters on your paper draft and (optionally) a reference paper draft to produce citation F1 plus literature-review quality scores.

Frequently Asked Questions about paper-autoraters

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I score citation faithfulness for a paper draft automatically?▼

Citation faithfulness scoring categorizes references into must-cite and good-to-cite sets, then computes Precision, Recall, and F1 against a ground-truth paper using Semantic Scholar ID matching.

Can I compare two paper drafts side-by-side to decide which is stronger?▼

Side-by-side comparison runs holistic and literature-review-only preference judgments between two drafts using a two-pass swapped-order protocol to reduce positional bias.

What is literature review quality evaluation using LLM-as-judge?▼

Literature review quality evaluation scores the Introduction and Related Work sections across six axes on a 0-100 scale, applying anti-inflation rules to produce a conservative overall quality score.

Do I need to extract references and full text before running paper scoring?▼

Yes, paper scoring requires extracting full text or PDF inputs and reference lists to compute citation F1, with optional resolution to Semantic Scholar IDs for deterministic matching.

How does the two-pass swapped-order protocol work for paper comparisons?▼

The two-pass swapped-order protocol evaluates two paper drafts in alternating sequence positions to mitigate positional bias and produce a more reliable preference judgment.

What are the limitations of LLM-as-judge for literature review assessment?▼

LLM-as-judge literature review assessment only scores Introduction and Related Work sections, requires full text extraction, and applies conservative anti-inflation rules that may deliberately lower scores compared to lenient evaluators.

Related Skills

Explore More Education & Research Skills→
Ar9avAr9av
635

paper-autoraters

Compute quality metrics and side-by-side judgments for research paper drafts.

Community
Advanced
raja21068raja21068
2

paper-autoraters

Score paper drafts with citation F1 and literature-review quality JSON outputs.

Community
Intermediate
PthahnixPthahnix
393

Scholar Reference

Retrieve reference papers for a PaperMeta via the Semantic Scholar API.

Community
Intermediate
Victory-HugoVictory-Hugo
1

scientific-critical-thinking

Evaluate scientific literature methodology, biases, and evidence quality.

Community
Advanced