tavily-crawl

To extract content from multiple web pages for bulk research, you can crawl a domain with configurable depth and limits to automatically gather information across multiple URLs. The extracted content can then be saved as structured JSON or local markdown files for offline reference.

Downloading documentation sites as markdown for LLM context is best handled by bulk crawling a website and saving each extracted page to a local markdown file. You can also use semantic targeting to return only relevant chunks per page for more focused context.

Yes, you can filter which specific URLs get crawled during site extraction by applying regex-based path and domain selection. This allows you to include or exclude specific pages to target exactly the content you need from the website.

Yes, you need the tvly tool available to crawl websites and extract site content. The Skill operates by executing web crawling and content extraction through the Tavily CLI, which requires this specific tool dependency to function.

Semantic-focused extraction works for web crawling by using natural-language instructions to return only relevant chunks per page. This provides LLM-friendly context instead of saving full pages, making the extracted content more targeted for downstream processing needs.