paper-writing-bench

Reverse-engineering a research paper into a benchmark involves extracting core concepts and experiments to produce idea_sparse.md, idea_dense.md, and experimental_log.md artifacts. These files enable end-to-end evaluation of paper-writing pipelines and support autorater comparisons.

The PaperWritingBench construction process is a structured method to reverse-engineer high-quality I/E materials from an existing AI research paper. It applies anonymization and deterministic scripts to create repeatable benchmark cases for evaluating automated paper-writing pipelines.

You can create anonymized experimental logs by processing the input paper through the benchmark construction pipeline. The Skill operates on any paper to produce self-contained experimental_log.md outputs with no external citations or URLs attached.

Yes, you can use this approach to evaluate any LLM pipeline for academic paper writing. The Skill operates on any input paper to produce standardized benchmark artifacts, allowing you to measure pipeline performance and compare different autoraters consistently.

In benchmark artifacts, idea_sparse.md and idea_dense.md represent two levels of extracted research concepts from the paper. They serve as distinct input variables for the paper-writing pipeline to test how varying information density affects the generated output quality.

A limitation when generating benchmark artifacts is that the outputs are strictly self-contained with no external citations or URLs. While it processes any input paper deterministically, highly complex or deeply nested experimental data may require manual verification for accurate benchmarking.