skills.rest
velamints2
velamints2COMMUNITY
Explore all skills by this author →
GitHub

arxiv-search-collector

Orchestrate model-guided arXiv queries and merge deduplicated papers by ID.

Updated Mar 4, 2026
Search Tags:#metadata#deduplication#arxiv#query planning#model-driven#arxiv-api
One-click install
npx skills add https://github.com/velamints2/clawbot-lab --skill arxiv-search-collector
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: arxiv-search-collector
Source: https://github.com/velamints2/clawbot-lab/tree/main/skills/arxiv-search-collector
Command: npx skills add https://github.com/velamints2/clawbot-lab --skill arxiv-search-collector

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This skill orchestrates model-guided arXiv queries to assemble a deduplicated paper set, storing per-paper metadata in structured directories.

Core Features & Use Cases

  • Model-driven query planning across multiple terms using OR within groups and AND across groups to control recall and relevance.
  • Per-query retrieval with arXiv API and rate-limiting, followed by a merge step to deduplicate by arXiv ID and produce a papers_index.
  • End-to-end pipeline with retry loops, incremental merges, and per-run task logs to enable traceability and reproducibility.
  • Real-world use case: generate a focused paper set on "Lean 4 formalization" and export to papers_index for downstream processing.

Quick Start

Initialize a run with a topic and target range, then run the batch fetch and merge steps to build your paper set.

Frequently Asked Questions about arxiv-search-collector

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I collect and deduplicate arXiv search results for a specific research topic?▼

To collect and deduplicate arXiv search results, use model-guided query planning with synonym groups and logical operators to fetch papers, then merge them by arXiv ID into structured metadata directories.

Can I use logical operators and synonym groups to plan arXiv API queries?▼

Yes, you can plan arXiv API queries using OR within synonym groups and AND across groups to control recall and relevance during the paper retrieval process.

How does the arXiv search pipeline handle rate limiting and incremental merges?▼

The arXiv search pipeline handles rate limiting through configurable delays and ensures robust operation via retry loops, incremental merges, and per-run task logs for traceability.

What is the best way to avoid duplicate papers when fetching from the arXiv API?▼

The best way to avoid duplicate arXiv papers is to execute multiple planned queries and apply a merge step that deduplicates all fetched results by their unique arXiv ID into a papers_index.

Does arxiv-search-collector require external dependencies to run?▼

No, arxiv-search-collector operates without external dependencies, using a configurable I/O contract via task_meta.json to orchestrate the end-to-end paper collection pipeline.

Are there limitations when doing batch fetches and merges of arXiv metadata?▼

Batch fetches and merges of arXiv metadata are constrained by API rate limits, requiring retry loops and incremental processing to ensure robust operation without losing traceability.

Related Skills

Explore More Data & Analytics Skills→
HeXiao-55HeXiao-55
1

arxiv

Search arXiv by topic or ID, download PDFs, and summarize metadata.

Community
Intermediate
I-OnlabsI-Onlabs
3

extract-from-pdfs

Extract structured data from scientific PDFs into analysis-ready formats.

Community
Advanced
WILLOSCARWILLOSCAR
497

arxiv-search

Retrieve arXiv paper metadata via keyword queries and save results as JSONL.

Community
Intermediate
HyunjunJeonHyunjunJeon
53

academic-search

Search arXiv and Semantic Scholar for scholarly articles by keywords, author, category, or boolean syntax.

Community
Advanced