What problem does it solve? Gathering content from many pages on the same website—such as an entire documentation section—normally requires visiting and copying each page manually. This Skill automates bulk website crawling and content extraction through the Tavily CLI, saving results as structured JSON or local markdown files. ## Core Features & Use Cases - Bulk Page Crawling: Crawl a site with configurable depth, breadth, and page limits, with path and domain filtering via regex patterns. - Offline Documentation Archives: Save every crawled page as a local markdown file using the --output-dir option. - Semantic Extraction: Use --instructions and --chunks-per-source to return only relevant content chunks, preventing context overload when feeding results to an LLM. - Use Case: You need the entire /docs section of a vendor site for offline reference. Run a crawl with --select-paths "/docs/.*" and --output-dir ./docs/ to download every page as markdown. ## Quick Start Ask the agent to crawl https://docs.example.com with a depth of 2 and save every page as markdown files into a local docs folder.
