Crawl websites and save pages as local markdown files.
npx skills add https://github.com/drillan/marketschema --skill crawl-drillan
Please help me install this Agent Skill. Skill: crawl Source: https://github.com/drillan/marketschema/tree/main/.claude/skills/crawl Command: npx skills add https://github.com/drillan/marketschema --skill crawl-drillan
Crawling websites and saving pages locally enables offline access, documentation capture, and knowledge-base archiving without manual copying.
Use the crawl script to fetch content and save pages locally: ./scripts/crawl.sh '{"url": "https://docs.example.com"}' ./docs Additional parameters can be supplied to adjust depth, limits, and filtering.
High-intent search queries and answers about installing and using this skill.
You can crawl a website and save pages as local markdown files by executing the crawl script with a target URL and output directory, using HTTP requests to retrieve content and generate markdown.
Archiving web knowledge base content for offline access is done by performing site-wide retrieval with depth and breadth controls to capture multiple pages, saving the extracted content locally as markdown files for offline reading or analysis.
Yes, the crawl script relies on a Tavily API key to perform HTTP requests and content retrieval, extracting site-wide pages before generating local markdown files.
Yes, you can control the depth and breadth when scraping a website by supplying additional parameters in the JSON payload to the crawl script, allowing you to retrieve multiple pages and adjust limits or filtering for targeted analysis.
To run the website crawl script for content extraction, you need the command-line tools jq and curl installed, which handle the HTTP requests and JSON parameter parsing required by the local script.
minorun365Check remaining Tavily API credits across keys in a .env file.
stevessrCrawl websites and save pages as markdown files using the Tavily API.
shuyu-labsConvert Word, Excel, PowerPoint, and PDF documents to Markdown with preserved headings, tables, and images.
ZJU-REALSearch, scrape, and monitor web content using the Firecrawl CLI.
