skills.rest
neverprepared
neverpreparedOFFICIAL
Explore all skills by this author →
GitHub

site-crawler

Crawl websites and extract structured content for RAG indexing.

Updated Aug 23, 2026
Search Tags:#documentation#sitemap#web#robots.txt#crawl#content-extraction#rag-ingestion
One-click install
npx skills add https://github.com/neverprepared/ink-bunny --skill site-crawler-neverprepared
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: site-crawler
Source: https://github.com/neverprepared/ink-bunny/tree/main/reflex/plugins/reflex/skills/site-crawler
Command: npx skills add https://github.com/neverprepared/ink-bunny --skill site-crawler-neverprepared

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Documentation sites, blogs, and knowledge bases are often scattered and difficult to harvest at scale. This Skill provides respectful crawling, structure-preserving extraction, and incremental updates to support efficient RAG workflows.

Core Features & Use Cases

  • Respect robots.txt and rate limits while crawling
  • Use sitemaps and in-page links to discover pages and preserve structure
  • Extract meaningful content and convert to Markdown or plain text for ingestion
  • Incremental updates to fetch only changed content

Quick Start

Initiate a full-site crawl from the target base URL and ingest extracted content into your RAG pipeline.

Frequently Asked Questions about site-crawler

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I crawl a documentation site for RAG ingestion?▼

Crawling a documentation site for RAG ingestion involves respecting robots.txt, discovering pages via sitemaps and in-page links, and extracting content into Markdown or plain text for indexing.

What is the best way to extract web content for RAG indexing while preserving site structure?▼

The best way to extract web content for RAG indexing while preserving structure is to use a crawler that follows sitemaps and in-page links, then converts meaningful content to Markdown or plain text.

Does crawling for content extraction respect robots.txt and rate limits?▼

Yes, respectful crawling for content extraction respects robots.txt and rate limits while discovering pages via sitemaps and in-page links to preserve site structure for RAG workflows.

Can I fetch only changed content during incremental site crawling?▼

Yes, you can fetch only changed content during incremental site crawling, which supports efficient RAG workflows by updating the index solely with new or modified pages instead of recrawling the entire site.

Do I need Python libraries like httpx and beautifulsoup4 for web crawling and content extraction?▼

Yes, you need Python libraries such as httpx, beautifulsoup4, lxml, and trafilatura for web crawling and content extraction to harvest structured content for downstream RAG indexing.

Why does my site crawl not preserve the structure of scattered knowledge base pages?▼

A site crawl fails to preserve structure when it does not use sitemaps and in-page links to discover pages, which is required to maintain the relationship between scattered knowledge base content for RAG ingestion.

Related Skills

Explore More Data & Analytics Skills→
roycolumbia-coderoycolumbia-code

seo

Analyzes websites' SEO health through audits of technical, content, schema aspects.

Community
Advanced
MSamiulHasnatMSamiulHasnat
1

AgentDB Vector Search

Index embeddings and perform context-aware similarity queries for document retrieval.

Community
Advanced
zeronx798zeronx798

chroma

Manage vector collections and perform similarity searches with metadata filtering.

Community
Intermediate
stephanjstephanj
9

web-to-markdown

Convert JavaScript-rendered web pages into structured Markdown files.

Community
Intermediate