paper-autoraters

To evaluate a research paper draft against a ground-truth paper, you can use citation F1 scoring by partitioning references into must-cite (P0) and good-to-cite (P1) sets to compute precision, recall, and F1 metrics.

Literature review quality scoring evaluates the Introduction and Related Work sections across six axes on a 0–100 scale. It applies anti-inflation caps and penalty logic to discourage overconfident grading and ensure objective assessment.

Yes, you can compare two paper versions side by side to generate winners for overall paper quality and literature-review quality. The process uses a positional-bias mitigation protocol to ensure fair evaluation.

No, resolving references to Semantic Scholar paper IDs is optional. The core evaluation requires partitioning references into P0/P1 sets using the provided prompt and emitting structured JSON outputs.

You trigger automated benchmark-style judging by prompting your AI host to run the PaperOrchestra-style citation-grounded autoraters, which then scores your drafts and produces the expected JSON outputs using the included scoring script.