site-crawler

Name: site-crawler
Availability: InStock
Author: neverprepared

Crawl websites and extract structured content for RAG indexing.

Updated Aug 23, 2026

Search Tags:#documentation #sitemap #web #robots.txt #crawl #content-extraction #rag-ingestion

Ready to run

npx skills add https://github.com/neverprepared/ink-bunny --skill site-crawler-neverprepared

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: site-crawler
Source: https://github.com/neverprepared/ink-bunny/tree/main/reflex/plugins/reflex/skills/site-crawler
Command: npx skills add https://github.com/neverprepared/ink-bunny --skill site-crawler-neverprepared

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Documentation sites, blogs, and knowledge bases are often scattered and difficult to harvest at scale. This Skill provides respectful crawling, structure-preserving extraction, and incremental updates to support efficient RAG workflows.

Core Features & Use Cases

Respect robots.txt and rate limits while crawling
Use sitemaps and in-page links to discover pages and preserve structure
Extract meaningful content and convert to Markdown or plain text for ingestion
Incremental updates to fetch only changed content

Quick Start

Initiate a full-site crawl from the target base URL and ingest extracted content into your RAG pipeline.

Frequently Asked Questions about site-crawler

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I crawl a documentation site for RAG ingestion?▼

Crawling a documentation site for RAG ingestion involves respecting robots.txt, discovering pages via sitemaps and in-page links, and extracting content into Markdown or plain text for indexing.

What is the best way to extract web content for RAG indexing while preserving site structure?▼

The best way to extract web content for RAG indexing while preserving structure is to use a crawler that follows sitemaps and in-page links, then converts meaningful content to Markdown or plain text.

Does crawling for content extraction respect robots.txt and rate limits?▼

Yes, respectful crawling for content extraction respects robots.txt and rate limits while discovering pages via sitemaps and in-page links to preserve site structure for RAG workflows.

Can I fetch only changed content during incremental site crawling?▼

Yes, you can fetch only changed content during incremental site crawling, which supports efficient RAG workflows by updating the index solely with new or modified pages instead of recrawling the entire site.

Do I need Python libraries like httpx and beautifulsoup4 for web crawling and content extraction?▼

Yes, you need Python libraries such as httpx, beautifulsoup4, lxml, and trafilatura for web crawling and content extraction to harvest structured content for downstream RAG indexing.

Why does my site crawl not preserve the structure of scattered knowledge base pages?▼

A site crawl fails to preserve structure when it does not use sitemaps and in-page links to discover pages, which is required to maintain the relationship between scattered knowledge base content for RAG ingestion.