paper-writing-bench

To build a paper-writing benchmark, reverse-engineer target AI research papers by extracting sparse and dense materials. This process strips narrative flow to generate self-contained idea files and experimental logs for pipeline evaluation.

Sparse idea variants are high-level concept memos without math or experiments, while dense variants are rigorous technical proposals with LaTeX math and defined variables. Both formats exclude experimental details to prevent data leakage during benchmark construction.

Extract experimental logs by pulling exhaustive setup details, raw numeric table data, and figure-derived observations from the source PDF. Ensure exact numeric accuracy while omitting citations, URLs, and figure or table references to maintain ground-truth validity.

Experimental log extraction requires exact numeric accuracy for all table data while omitting citations, URLs, and figure or table references. Logs must remain self-contained to prevent experimental leakage when evaluating research writing pipelines.

Yes, you can evaluate a paper-writing pipeline by reverse-engineering an existing AI research paper into standardized (idea, experimental_log) benchmark cases. This process tests whether your pipeline accurately replicates the target paper outputs.