Crawl websites to download content as local markdown files.
npx skills add https://github.com/tavily-ai/tavily-cursor-plugin --skill crawl-tavily-ai
Please help me install this Agent Skill. Skill: crawl Source: https://github.com/tavily-ai/tavily-cursor-plugin/tree/main/skills/crawl Command: npx skills add https://github.com/tavily-ai/tavily-cursor-plugin --skill crawl-tavily-ai
This Skill automates the process of downloading entire websites or specific sections of them, converting the content into local markdown files for offline access, analysis, or archival.
Crawl the website at https://docs.example.com and save the results to a local directory named ./docs.
High-intent search queries and answers about installing and using this skill.
To download websites as local markdown files, you can use a web scraping skill that crawls a specified root URL and converts the extracted content into markdown. This allows you to archive web content for offline access or build a local knowledge base.
Yes, you can extract specific web content by applying path filters and setting depth limits during the crawl. This focused crawling approach targets designated sections of a website, ensuring you only download the relevant pages instead of the entire site.
The best way to archive documentation for offline analysis is to crawl the documentation site and save the content as local markdown files. This converts the web pages into a clean, readable format suitable for offline access and local knowledge base construction.
Agentic chunking optimizes web content extraction by breaking down the downloaded markdown into relevant content chunks. This process specifically prepares the extracted website data so it can be efficiently fed into LLM contexts for further analysis.
Yes, web scraping requires specifying a root URL to define the starting point for the crawl. From this root URL, the process navigates through linked pages, applying your configured breadth and depth parameters to download content efficiently.
When using a website downloader for large knowledge bases, limitations may arise if the crawl depth and breadth are not properly configured. Unfiltered crawling can lead to downloading irrelevant pages, making it essential to use path filters to constrain the extraction scope.
nishchaydevCrawl full websites and map pages for SEO audits using Firecrawl MCP tools.
drillanCrawl websites and save pages as local markdown files.
eykdConvert web pages to Markdown using the url-to-markdown-cli-tool.
iAziz786Crawl websites with depth limits and path filters via self-hosted Firecrawl.
