paper-autoraters
CommunityAutomated quality scoring for research papers
Education & Research#benchmarking#literature-review#academic-review#semantic-scholar#paper-evaluation#citation-f1#sxs-comparison
AuthorAr9av
Version1.0.0
Installs0
System Documentation
What problem does it solve?
This Skill automates rigorous, reproducible scoring of research paper drafts so authors and evaluation pipelines can quantify literature-review quality, citation fidelity, and holistic paper quality without manual rubric application.
Core Features & Use Cases
- Citation F1 Partitioning: Partition references into P0/P1 priority buckets and deterministically compute Precision/Recall/F1 against ground-truth via Semantic Scholar ID resolution.
- Literature Review Quality (6-axis): Produce conservative, evidence-based 6-axis JSON scores with penalties and an overall score for Intro + Related Work.
- Side-by-Side (SxS) Comparisons: Run dual-order SxS judgments for full-paper and literature-review-only comparisons to mitigate positional bias.
- Use Case: Validate a paper-generating pipeline by scoring generated drafts against a ground-truth paper, compare two pipeline outputs, or audit literature-review improvements across iterations.
Quick Start
Run the autoraters to score the draft 'submission_v1.pdf' against the ground-truth 'paper_gt.pdf' and produce JSON reports.
Dependency Matrix
Required Modules
None requiredComponents
scriptsreferences
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: paper-autoraters Download link: https://github.com/Ar9av/PaperOrchestra/archive/main.zip#paper-autoraters Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 471,000+ vetted skills library on demand.