Crawl websites and save pages as local markdown files.
npx skills add https://github.com/xdanger/dotfiles --skill crawl-xdanger
Please help me install this Agent Skill. Skill: crawl Source: https://github.com/xdanger/dotfiles/tree/main/agents/skills/crawl Command: npx skills add https://github.com/xdanger/dotfiles --skill crawl-xdanger
Crawl websites to save pages as local markdown files for offline access and archival documentation.
Provide a URL to crawl and run the script to produce markdown files in a local directory.
High-intent search queries and answers about installing and using this skill.
You can crawl a website and save pages as markdown by providing a URL to the script, which visits pages and outputs local markdown files for offline reading and knowledge capture.
Yes, you can control crawl depth and filter paths when extracting site-wide content. The script supports depth limits, breadth control, and optional path filtering to target specific documentation sections.
Yes, you need Tavily MCP to run the web crawl script. It integrates with Tavily MCP for authentication using either OAuth or an API key to access and extract site content.
The best way to archive documentation sites locally as markdown files is to run a site-wide crawl with depth control and path filtering, saving extracted content directly into an organized local directory.
To crawl sites and extract knowledge base content, the required dependencies include jq, curl, npx, and mcp-remote, which support the script execution and remote MCP authentication processes.
tony2015116Crawl websites and save pages as markdown files using Python scripts.
zzafergokConvert Markdown documents to HTML using marked.js, pandoc, gomarkdown, or static site generators.
LuisElPotiPerform LLM-optimized web search, URL mapping, and content extraction with JSON/file output.
kevinnftExtract structured data from static HTML, JS-rendered pages, and hybrid sites.
