tavily-crawl

Crawl websites and save pages as local Markdown files.

1|Updated Mar 15, 2026

Search Tags:#automation #documentation #markdown #offline #web #knowledge-base #crawl

Ready to run

npx skills add https://github.com/family3253/skill --skill tavily-crawl-family3253

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: tavily-crawl
Source: https://github.com/family3253/skill/tree/main/skills/tavily-crawl
Command: npx skills add https://github.com/family3253/skill --skill tavily-crawl-family3253

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires jq, curl, node, mcp-remote, and includes scripts (resource) components.

What problem does it solve?

Crawling websites to capture content and save pages locally as Markdown files for offline access and analysis.

Core Features & Use Cases

Save visited pages as Markdown locally for archival and offline reading.
Build offline knowledge bases, documentation dumps, or QA datasets from site content.
Use cases include capturing API docs, product docs, or help center content for offline reference.

Quick Start

Provide a URL to begin crawling and the script will save pages as local Markdown files.

Frequently Asked Questions about tavily-crawl

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I crawl a website and save pages as local Markdown files?▼

To crawl a website and save pages as local Markdown, provide a target URL and the script extracts content, writing output as Markdown files for offline access. It leverages Tavily MCP to handle secure crawling and content extraction.

Can I build an offline knowledge base from web documentation?▼

Yes, you can build an offline knowledge base from web documentation by crawling target sites and saving extracted content as local Markdown files. This allows you to archive product docs, API references, or help centers for offline reading and analysis.

Does website crawling for offline access require authentication or API keys?▼

Website crawling supports authentication flows via Tavily MCP and optional API keys, enabling secure token caching. This allows scripted crawls to access and document protected pages or restricted knowledge bases locally.

What dependencies are needed to automate web crawling and Markdown conversion?▼

Automating web crawling and Markdown conversion requires jq, curl, node, and mcp-remote installed in your environment. These dependencies support script execution, remote MCP communication, and content processing needed to save pages locally.

What is the best way to create a QA dataset from help center content?▼

The best way to create a QA dataset from help center content is to crawl the website and save pages as local Markdown files. This captures text content in a structured format suitable for offline analysis and dataset building.