crawl

Name: crawl
Availability: InStock
Author: stevessr

Crawl websites and save pages as markdown files using the Tavily API.

Updated Dec 1, 2025

Search Tags:#documentation #markdown #offline #web #knowledge-base #crawl

Ready to run

npx skills add https://github.com/stevessr/dotsfiles --skill crawl-stevessr

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: crawl
Source: https://github.com/stevessr/dotsfiles/tree/main/dot_agents/skills/crawl
Command: npx skills add https://github.com/stevessr/dotsfiles --skill crawl-stevessr

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

Crawl websites to collect content for offline access or analysis without manual copying.

Core Features & Use Cases

Automated site crawling to fetch pages and save them as markdown files.
Depth control, breadth limits, and path filtering to focus on documents, knowledge bases, or sections of a site.
Seamless integration with the Tavily API (requires an API key) for authenticated crawling and predictable outputs.

Quick Start

Run the crawl script with a target URL and optional parameters to save each crawled page as a separate markdown file.

Frequently Asked Questions about crawl

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I crawl a website and save pages as markdown for offline access?▼

To crawl a website and save pages as markdown for offline access, provide a root URL and optional crawling controls like max_depth and max_breadth to fetch content and store each page locally as a separate markdown file.

Do I need a Tavily API key to extract documentation content from a knowledge base?▼

Yes, a Tavily API key is required to extract documentation content from a knowledge base, enabling authenticated crawling that returns results predictably in markdown or JSON formats.

Can I limit site-wide crawling to specific sections using path filtering?▼

You can limit site-wide crawling to specific sections using path filtering by setting select_paths and exclude_paths parameters, restricting content extraction to targeted documentation areas.

What is the best way to archive an entire documentation site locally?▼

The best way to archive an entire documentation site locally is automated web crawling with depth and breadth limits, extracting site content into individual markdown files for offline research.

Does web crawling for offline research work without manual page copying?▼

Web crawling for offline research works without manual page copying by automating site content extraction, fetching multiple pages simultaneously and saving them as local markdown files.

What are the limitations of depth and breadth controls when crawling sites?▼

Depth and breadth controls limit how many layers and links the crawler follows, constraining crawl scope to prevent excessive requests while targeting specific documentation or knowledge base sections.