Compute quality metrics and side-by-side judgments for research paper drafts.
npx skills add https://github.com/Ar9av/PaperOrchestra --skill paper-autoraters
Please help me install this Agent Skill. Skill: paper-autoraters Source: https://github.com/Ar9av/PaperOrchestra/tree/main/skills/paper-autoraters Command: npx skills add https://github.com/Ar9av/PaperOrchestra --skill paper-autoraters
This Skill automates rigorous, reproducible scoring of research paper drafts so authors and evaluation pipelines can quantify literature-review quality, citation fidelity, and holistic paper quality without manual rubric application.
Run the autoraters to score the draft 'submission_v1.pdf' against the ground-truth 'paper_gt.pdf' and produce JSON reports.
High-intent search queries and answers about installing and using this skill.
Automated research paper evaluation computes structured quality metrics and side-by-side judgments using four autoraters. It produces conservative 6-axis JSON scores with penalties for Intro and Related Work sections, alongside an overall score for literature review quality.
Citation F1 score calculation partitions references into P0 and P1 priority buckets and deterministically computes Precision, Recall, and F1 against ground-truth. This process requires a host agent to resolve references to Semantic Scholar IDs before running the deterministic scoring scripts.
Side-by-side SxS comparison for research benchmarking runs dual-order judgments for full-paper and literature-review-only comparisons to mitigate positional bias. It validates paper-generating pipelines by scoring generated drafts against a ground-truth paper or comparing two pipeline outputs.
Scoring PDF research paper drafts without manual rubric application is supported, but requires a host agent to extract Introduction and Related Work sections. The host agent must also resolve references to Semantic Scholar IDs before the autoraters can compute the structured quality metrics.
Paper evaluation works with Semantic Scholar ID resolution to quantify citation fidelity accurately. The deterministic scripts use these resolved IDs to partition references into priority buckets and compute Precision, Recall, and F1 against the provided ground-truth data.
woodfishhhhScore citation faithfulness and literature-review quality in paper drafts.
notqueExecute pipeline skills against real targets and generate per-subdomain pass/fail reports.
Himanshu040604Evaluate LLM applications using automated metrics, human feedback, and benchmarking.
prathamchopra001Extract structured research findings with claims, confidence scores, and source details from academic paper excerpts.
