tavily-crawl

To crawl a website and save pages as markdown, you can extract content from multiple pages and store each page locally as a markdown file for offline documentation access.

Yes, you can control crawl depth and breadth, apply path filtering, and use semantic instructions to target specific sections when extracting web content for bulk collection.

Downloading all pages under a specific directory involves crawling targeted site sections using path filtering, then extracting content to save each page as a local markdown file.

Yes, the Tavily CLI supports saving extracted web content to individual local markdown files per page, enabling offline access and integration with downstream workflows.

Semantic instructions can guide website content extraction by directing the crawl process, allowing targeted collection of pages saved as markdown files for offline access.