crawl

Crawl entire websites by following links from a starting URL.

Updated Feb 19, 2026

Search Tags:#documentation #knowledge base #web scraping #data ingestion #content collection #website crawl

Ready to run

npx skills add https://github.com/olostep/olostep-cursor-plugin --skill crawl-olostep

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: crawl
Source: https://github.com/olostep/olostep-cursor-plugin/tree/main/skills/crawl
Command: npx skills add https://github.com/olostep/olostep-cursor-plugin --skill crawl-olostep

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill automates the tedious and incomplete process of manually collecting pages from multi-page websites, providing comprehensive context for AI tasks.

Core Features & Use Cases

Full Website Ingestion: Renders and scrapes entire websites, including JavaScript-heavy pages and bot-protected sites.
Automated Link Following: Discovers and processes pages by following internal links.
Use Case: Ingest an entire documentation site to build a knowledge base, generate a cheat sheet for a library, or prepare context for a code refactoring task.

Quick Start

Crawl the website starting at https://example.com and process up to 10 pages.

Frequently Asked Questions about crawl

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I ingest an entire website for AI context?▼

You can automate website ingestion by using a crawling tool that follows internal links from a starting URL, rendering pages in a full browser to collect documentation and knowledge bases for AI context.

What is the best way to scrape JavaScript-heavy or bot-protected documentation sites?▼

The best way is using a full browser rendering approach with anti-bot bypass, which successfully scrapes JavaScript-heavy and bot-protected documentation sites by following internal links automatically.

Can I export scraped website content as markdown or JSON?▼

Yes, you can export scraped website content as markdown, HTML, JSON, or text. The ingestion process supports these various output formats to prepare data for your specific AI context requirements.

Does web scraping work with geo-targeting for localized content?▼

Yes, web scraping supports geo-targeting options to render and collect localized website content, ensuring the ingested pages reflect specific geographic locations during the crawl.

How do I collect multiple pages from a knowledge base automatically?▼

You can collect multiple pages from a knowledge base automatically by using a tool with automated link following, which discovers and processes internal pages from a starting URL up to your specified limit.

When do I need full browser rendering for web scraping?▼

You need full browser rendering for web scraping when targeting JavaScript-heavy pages or bot-protected sites, ensuring dynamically loaded content is fully rendered and captured during the website ingestion process.