Crawl a company domain and save filtered pages as Markdown files.
npx skills add https://github.com/theworkflowco/gtm-agent-example --skill crawl-website
Please help me install this Agent Skill. Skill: crawl-website Source: https://github.com/theworkflowco/gtm-agent-example/tree/main/skills/crawl-website Command: npx skills add https://github.com/theworkflowco/gtm-agent-example --skill crawl-website
Discover and scrape relevant pages from a company website to build context for research briefs and knowledge bases.
Run the crawl-website skill on a target domain and save discovered pages as markdown under docs/research/website/.
High-intent search queries and answers about installing and using this skill.
Web scraping for research briefs involves crawling a domain to discover pages, filtering for high-value URLs like homepage and pricing, and saving the scraped content as markdown files to build a reusable knowledge base.
Yes, web scraping workflows can filter discovered domain pages to capture high-value content sections like homepage, about, product, pricing, FAQ, team, and customers for focused research documentation.
Yes, you need a SPIDERCLOUD_API_KEY to execute domain-wide web scraping. The workflow supports a fallback to a template repository if local configuration is missing.
The best way to build a knowledge base from web scraping is to crawl a target domain, filter for high-value pages, and save the compiled context-rich results as markdown files under a structured research directory.
Use domain-wide web crawling instead of manual web scraping when you need to automatically discover candidate URLs across a company website, filter for high-value pages, and efficiently compile context-rich research documentation.
s-hiraokuScrape websites, crawl pages, search, and extract structured data via Firecrawl tools.
GitnappMaintain an interconnected markdown knowledge base with schemas, cross-references, and provenance tracking.
0xharryriddleRewrite external URLs through markdown.new to fetch clean Markdown content.
SpaceZephyrFetch URLs and convert HTML page content into readable markdown.
