deep-crawl

Crawl JavaScript-rendered websites with headless Chrome via CDP and output structured JSON reports.

1.0k|75|Updated Mar 15, 2026

Search Tags:#cdp #crawl #headless-chrome #ssrf-protection #web-crawler #javascript-rendering

Ready to run

npx skills add https://github.com/octos-org/octos --skill deep-crawl

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: deep-crawl
Source: https://github.com/octos-org/octos/tree/main/crates/app-skills/deep-crawl
Command: npx skills add https://github.com/octos-org/octos --skill deep-crawl

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Crawling JS-rendered sites to collect rendered text and discovered links, automating content gathering that would be tedious to do manually.

Core Features & Use Cases

Renders JavaScript content using headless Chrome via CDP to extract text from each page.
Follows same-origin links with BFS and saves per-page results to disk for offline analysis.
Ideal for indexing documentation portals, SPAs, and dynamic sites where server-side HTML is insufficient for understanding content.

Quick Start

Provide a seed URL and optional depth and page limits to crawl a website.

Frequently Asked Questions about deep-crawl

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I crawl JavaScript-rendered single page applications for text extraction?▼

Crawl JavaScript-rendered SPAs using a headless Chrome browser via CDP to render content and extract the fully rendered text from each page. This approach automates content gathering that would be tedious to do manually.

Can I limit the crawl depth and number of pages when scraping a documentation portal?▼

Yes, you can configure the crawl depth and page limits when scraping a documentation portal. The crawler follows same-origin links using BFS and saves per-page results to disk for offline analysis.

Does this web crawler include SSRF protection for scraping dynamic sites?▼

Yes, the web crawler includes built-in SSRF protections to ensure safe scraping of dynamic sites. It uses path_prefix filtering and renders JavaScript via headless Chrome to securely collect discovered links and rendered text.

What is the best way to extract content from JS-heavy sites where server-side HTML is insufficient?▼

The best way to extract content from JS-heavy sites is using a headless Chrome browser via CDP to render JavaScript. This ensures you collect the fully rendered text and discovered links, bypassing insufficient server-side HTML.

How does recursive crawling with headless Chrome handle same-origin links?▼

Recursive crawling with headless Chrome handles same-origin links by following them using BFS traversal. It applies configurable depth and page limits, path_prefix filtering, and outputs a structured JSON report of the discovered content.

Is path_prefix filtering supported when indexing dynamic websites?▼

Yes, path_prefix filtering is supported when indexing dynamic websites. This allows you to restrict the recursive headless Chrome crawl to specific URL paths, ensuring the structured JSON report only contains relevant rendered text.