Crawl websites and save each page as a local markdown file.
npx skills add https://github.com/tavily-ai/skills --skill tavily-crawl-tavily-ai
Please help me install this Agent Skill. Skill: tavily-crawl Source: https://github.com/tavily-ai/skills/tree/main/skills/tavily-crawl Command: npx skills add https://github.com/tavily-ai/skills --skill tavily-crawl-tavily-ai
Crawl a website and extract content from multiple pages, saving each page as a local markdown file for offline access and aggregated documentation.
Crawl https://docs.example.com and save pages under /docs as separate markdown files.
High-intent search queries and answers about installing and using this skill.
To crawl a website and save pages as markdown, configure the depth and breadth controls to extract content from multiple pages, then export each page locally as a markdown file. This allows offline access and aggregated documentation from the target domain.
Downloading documentation for offline access is best done by crawling the site with configurable depth and path filtering. This extracts content from relevant pages and saves each one as a separate local markdown file for easy reference.
Yes, you can filter which pages get crawled by applying path filters and semantic instructions. This focuses the crawl on relevant sections of the domain, ensuring only the desired pages are extracted and saved as markdown.
You control crawl depth when scraping multiple web pages by adjusting the configurable depth and breadth settings. These controls limit how many links the crawler follows, allowing you to archive specific sections of a site efficiently.
Web scraping with markdown export does support semantic instructions to guide content extraction. By providing these instructions alongside path filtering, you can focus the crawl on specific content across many pages on the same domain.
Archiving sections of a site into local markdown is limited to pages accessible on the same domain. You must configure depth, breadth, and path filtering carefully to avoid crawling irrelevant pages and to ensure the extracted content meets your needs.
BEIRUXDiscover all URLs on a website using the tvly CLI with filter options.
intelligenttools-aiCaptures web pages, transcripts, and files into structured, searchable knowledge articles with provenance tags.
KhaledMKhaledSearch the web and fetch URLs to extract markdown content.
daanteijema-beepCrawl reachable URLs from a starting page and extract content in bulk.
