crawl-website

Name: crawl-website
Availability: InStock
Author: theworkflowco

Crawl a company domain and save filtered pages as Markdown files.

Updated Aug 23, 2026

Search Tags:#markdown #research #websearch #web-scraping #crawl #spidercloud

Ready to run

npx skills add https://github.com/theworkflowco/gtm-agent-example --skill crawl-website

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: crawl-website
Source: https://github.com/theworkflowco/gtm-agent-example/tree/main/skills/crawl-website
Command: npx skills add https://github.com/theworkflowco/gtm-agent-example --skill crawl-website

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Discover and scrape relevant pages from a company website to build context for research briefs and knowledge bases.

Core Features & Use Cases

Discover pages via domain-wide search to identify candidate URLs.
Filter for high-value pages (homepage, about, product, pricing, FAQ, team, customers) to focus scraping efforts.
Save scraped content as Markdown files under docs/research/website/ for easy documentation and reuse.

Quick Start

Run the crawl-website skill on a target domain and save discovered pages as markdown under docs/research/website/.

Frequently Asked Questions about crawl-website

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I scrape a company website for research and save it as markdown?▼

Web scraping for research briefs involves crawling a domain to discover pages, filtering for high-value URLs like homepage and pricing, and saving the scraped content as markdown files to build a reusable knowledge base.

Can I filter web scraping results to only capture high-value pages like pricing and team?▼

Yes, web scraping workflows can filter discovered domain pages to capture high-value content sections like homepage, about, product, pricing, FAQ, team, and customers for focused research documentation.

Do I need a Spidercloud API key to crawl a domain?▼

Yes, you need a SPIDERCLOUD_API_KEY to execute domain-wide web scraping. The workflow supports a fallback to a template repository if local configuration is missing.

What is the best way to build a knowledge base from web scraping a domain?▼

The best way to build a knowledge base from web scraping is to crawl a target domain, filter for high-value pages, and save the compiled context-rich results as markdown files under a structured research directory.

When should I use domain-wide web crawling instead of manual web scraping?▼

Use domain-wide web crawling instead of manual web scraping when you need to automatically discover candidate URLs across a company website, filter for high-value pages, and efficiently compile context-rich research documentation.