skills.rest
WILLOSCAR
WILLOSCARCOMMUNITY
Explore all skills by this author →
GitHub

dedupe-rank

Deduplicate and rank research papers into a curated core set.

497|38|Updated Jan 7, 2026
Search Tags:#information retrieval#research papers#literature review#deduplication#ranking#core set
One-click install
npx skills add https://github.com/WILLOSCAR/research-units-pipeline-skills --skill dedupe-rank
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: dedupe-rank
Source: https://github.com/WILLOSCAR/research-units-pipeline-skills/tree/main/.codex/skills/dedupe-rank
Command: npx skills add https://github.com/WILLOSCAR/research-units-pipeline-skills --skill dedupe-rank

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill streamlines the process of refining large sets of retrieved research papers into a manageable, high-quality core set, essential for focused literature reviews and analysis.

Core Features & Use Cases

  • Deduplication: Identifies and merges duplicate entries based on normalized titles and years, retaining the most complete metadata.
  • Relevance Ranking: Scores papers based on query relevance and recency, prioritizing the most pertinent research.
  • Core Set Generation: Produces a core_set.csv file containing a curated selection of papers, ideal for building taxonomies, outlines, or mappings.
  • Use Case: After retrieving hundreds of papers on LLM agents, use this Skill to automatically deduplicate them and select the top 50 most relevant and recent papers for your literature review.

Quick Start

Use the dedupe-rank skill to create a core set of 50 papers from the raw paper data.

Frequently Asked Questions about dedupe-rank

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deduplicate research papers for a literature review?▼

To deduplicate research papers, this Skill normalizes titles and years from your raw JSONL data to identify duplicates, merging them while retaining the most complete metadata. It outputs a refined JSONL file containing only unique entries.

What is the best way to rank research papers by relevance and recency?▼

Ranking research papers by relevance involves scoring them against query tokens and incorporating domain-specific signals. This Skill evaluates your retrieved papers and generates a curated CSV core set containing stable IDs and essential metadata for analysis.

How do I select a core set of papers from a large raw dataset?▼

Selecting a core set from a large raw dataset requires scoring relevance and recency to filter results. This Skill processes your raw JSONL file to produce a curated core set CSV, ideal for building taxonomies or outlines.

Can I use this for literature review tasks involving hundreds of retrieved papers?▼

Yes, you can use this for literature review tasks involving hundreds of retrieved papers. It streamlines refining large sets of retrieved papers into a manageable, high-quality core set by automatically deduplicating and selecting the most relevant recent research.

What format do I need to provide to automatically deduplicate research papers?▼

To automatically deduplicate research papers, you need to provide your raw paper data in JSONL format, specifically located at `papers/papers_raw.jsonl`. The Skill then outputs a refined JSONL file and a curated CSV core set.

Related Skills

Explore More Education & Research Skills→
tommy-fcytommy-fcy
1

add

Fetch source URLs and save content into raw/ subdirectories with YAML frontmatter.

Community
Intermediate
kangning-huangkangning-huang
2

review-landscape

Map research domains and rank high-fit gaps against user-defined pillars.

Community
Advanced
lucasfthlucasfth

arxiv

Search arXiv REST API by keyword, author, category, or ID and parse Atom XML metadata.

Community
Intermediate
mwolff328-stackmwolff328-stack
1

sp-friday-sprint-review-rename

Rename completed Claude Code sessions to date-based titles using scheduledTaskId verification.

Community
Intermediate