skills.rest
mitchellvandusen-cell
mitchellvandusen-cellCOMMUNITY
Explore all skills by this author →
GitHub

tavily-crawl

Crawl websites and extract content to local markdown files using the Tavily CLI.

Updated Dec 12, 2025
Search Tags:#markdown#content extraction#data retrieval#website crawling#Tavily CLI
One-click install
npx skills add https://github.com/mitchellvandusen-cell/Flask-Webhook --skill tavily-crawl-mitchellvandusen-cell
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tavily-crawl
Source: https://github.com/mitchellvandusen-cell/Flask-Webhook/tree/main/.claude/skills/tavily-crawl
Command: npx skills add https://github.com/mitchellvandusen-cell/Flask-Webhook --skill tavily-crawl-mitchellvandusen-cell

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires Bash, tvly, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the need for efficient website crawling and content extraction, making it easy to gather information from multiple pages and save it in various formats.

Core Features & Use Cases

  • Website Crawl: Scrape content from websites with customizable depth and breadth.
  • Content Extraction: Download documentation, extract specific sections, or bulk-extract pages.
  • Markdown Output: Save each page as a local markdown file for easy reference.
  • Use Case: For researchers or content curators who need to quickly gather and organize information from a large number of web pages.

Quick Start

To start crawling a website, use the command: tvly crawl "https://example.com" --output-dir ./docs/

Frequently Asked Questions about tavily-crawl

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I crawl a website and save multiple pages as markdown files?▼

You can crawl a website and save pages as markdown files by using the Tavily CLI with a Bash environment. Execute a command like `tvly crawl "https://example.com" --output-dir ./docs/` to extract content and store it locally.

Can I control the depth and breadth when extracting content from web pages?▼

Yes, website crawling supports customizable depth and breadth control. This allows you to specify how extensively the crawler navigates and extracts content from multiple pages within the target domain.

Do I need Bash and the Tavily CLI to extract website content?▼

Yes, you need both Bash and the Tavily CLI installed to perform website crawling and content extraction. These dependencies are required to execute the commands that retrieve and save web pages as markdown.

How do I filter URLs and use semantic instructions for website crawling?▼

Website crawling supports path filtering and semantic instructions to refine content extraction. This allows you to target specific sections or types of pages within a domain, ensuring you retrieve only the relevant information.

What is the best way to bulk-extract documentation from a website?▼

The best way to bulk-extract documentation is using a CLI tool with depth and breadth control to crawl multiple pages. This approach efficiently gathers information and saves each page locally as a markdown file for easy reference.

Related Skills

Explore More Data & Analytics Skills→
npc-livenpc-live
168

tavily-search

Searches web and extracts content via Tavily CLI with automatic fallback.

Official
Intermediate
89jobrien89jobrien
4

file-converter

Generate Python code to convert files between document, data, and image formats.

Community
Advanced
ironkid90ironkid90

firecrawl-crawl

Crawl websites with depth limits and path filtering to extract content.

Community
Advanced
a2ajinkyaa2ajinkya
1

tavily-crawl

Crawl websites and export each page as a markdown file using the Tavily CLI.

Community
Intermediate