crawl

Crawl websites and save pages as local markdown files.

3|1|Updated Jun 14, 2021

Search Tags:#automation #documentation #markdown #offline #web #knowledge-base #crawl

Ready to run

npx skills add https://github.com/xdanger/dotfiles --skill crawl-xdanger

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: crawl
Source: https://github.com/xdanger/dotfiles/tree/main/agents/skills/crawl
Command: npx skills add https://github.com/xdanger/dotfiles --skill crawl-xdanger

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires jq, curl, npx, mcp-remote, and includes scripts (resource) components.

What problem does it solve?

Crawl websites to save pages as local markdown files for offline access and archival documentation.

Core Features & Use Cases

Crawl websites and save each visited page as a markdown file for offline reading and knowledge capture.
Supports depth control, optional path filtering, and an output directory to organize results.
Use cases include collecting documentation, knowledge bases, and API references for offline review.

Quick Start

Provide a URL to crawl and run the script to produce markdown files in a local directory.

Frequently Asked Questions about crawl

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I crawl a website and save pages as markdown for offline access?▼

You can crawl a website and save pages as markdown by providing a URL to the script, which visits pages and outputs local markdown files for offline reading and knowledge capture.

Can I control crawl depth and filter paths when extracting site-wide content?▼

Yes, you can control crawl depth and filter paths when extracting site-wide content. The script supports depth limits, breadth control, and optional path filtering to target specific documentation sections.

Do I need Tavily MCP to run the web crawl script?▼

Yes, you need Tavily MCP to run the web crawl script. It integrates with Tavily MCP for authentication using either OAuth or an API key to access and extract site content.

What is the best way to archive documentation sites locally as markdown files?▼

The best way to archive documentation sites locally as markdown files is to run a site-wide crawl with depth control and path filtering, saving extracted content directly into an organized local directory.

What dependencies are required to crawl sites and extract knowledge base content?▼

To crawl sites and extract knowledge base content, the required dependencies include jq, curl, npx, and mcp-remote, which support the script execution and remote MCP authentication processes.