skills.rest
family3253
family3253COMMUNITY
Explore all skills by this author →
WebsiteGitHub

tavily-crawl

Crawl websites and save pages as local Markdown files.

1|Updated Mar 15, 2026
Search Tags:#automation#documentation#markdown#offline#web#knowledge-base#crawl
One-click install
npx skills add https://github.com/family3253/skill --skill tavily-crawl-family3253
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tavily-crawl
Source: https://github.com/family3253/skill/tree/main/skills/tavily-crawl
Command: npx skills add https://github.com/family3253/skill --skill tavily-crawl-family3253

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires jq, curl, node, mcp-remote, and includes scripts (resource) components.

What problem does it solve?

Crawling websites to capture content and save pages locally as Markdown files for offline access and analysis.

Core Features & Use Cases

  • Save visited pages as Markdown locally for archival and offline reading.
  • Build offline knowledge bases, documentation dumps, or QA datasets from site content.
  • Use cases include capturing API docs, product docs, or help center content for offline reference.

Quick Start

Provide a URL to begin crawling and the script will save pages as local Markdown files.

Frequently Asked Questions about tavily-crawl

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I crawl a website and save pages as local Markdown files?▼

To crawl a website and save pages as local Markdown, provide a target URL and the script extracts content, writing output as Markdown files for offline access. It leverages Tavily MCP to handle secure crawling and content extraction.

Can I build an offline knowledge base from web documentation?▼

Yes, you can build an offline knowledge base from web documentation by crawling target sites and saving extracted content as local Markdown files. This allows you to archive product docs, API references, or help centers for offline reading and analysis.

Does website crawling for offline access require authentication or API keys?▼

Website crawling supports authentication flows via Tavily MCP and optional API keys, enabling secure token caching. This allows scripted crawls to access and document protected pages or restricted knowledge bases locally.

What dependencies are needed to automate web crawling and Markdown conversion?▼

Automating web crawling and Markdown conversion requires jq, curl, node, and mcp-remote installed in your environment. These dependencies support script execution, remote MCP communication, and content processing needed to save pages locally.

What is the best way to create a QA dataset from help center content?▼

The best way to create a QA dataset from help center content is to crawl the website and save pages as local Markdown files. This captures text content in a structured format suitable for offline analysis and dataset building.

Related Skills

Explore More Data & Analytics Skills→
moongci38-ossmoongci38-oss

site-deep-analyze

Analyze website structure, components, and APIs to generate a reimplementation guide.

Community
Advanced
rangerrick337rangerrick337
10

pdf-conversion

Convert PDFs to Markdown with preserved formatting and embedded images.

Community
Intermediate
AradotsoAradotso
5

snowflake-dbt-airbnb-analytics

Automate ETL workflows with Snowflake, dbt, and Streamlit.

Official
Advanced
KhaledaunKhaledaun

tavily-web

A tool for searching the web and aggregating sports news and information feeds.

Community
Intermediate