crawl-site

What problem does it solve? Fetching one page at a time with WebFetch is too slow when research requires many pages from a single site, such as full documentation, a competitor's site, or a reference library. This Skill walks an entire site and saves every page as a readable local markdown file you can search and load on demand. ## Core Features & Use Cases - Keyless BFS crawler: Built-in same-origin link walking with plain fetch and HTML-to-text extraction, requiring no API key and producing one .md file per page plus an index.json manifest. - Firecrawl backend: When FIRECRAWL_API_KEY is set, the same command routes to the Firecrawl v1 crawl API for JS rendering and clean markdown output. - Bounded, targeted crawls: --max-pages caps the crawl size and --include limits the walk to a path prefix such as /docs, skipping irrelevant marketing pages. - Use Case: Run a crawl of a competitor's documentation with --include /docs, read index.json to see titles and page sizes, then load only the specific pages relevant to your feature comparison. ## Quick Start Ask the agent to crawl a documentation site into local markdown files, for example: crawl https://docs.expo.dev with a max of 30 pages limited to the /docs path and save the output to /tmp/crawl.