skills.rest
octos-org
octos-orgOFFICIAL
Explore all skills by this author →
GitHub

deep-crawl

Crawl JavaScript-rendered websites with headless Chrome via CDP and output structured JSON reports.

1.0k|75|Updated Mar 15, 2026
Search Tags:#cdp#crawl#headless-chrome#ssrf-protection#web-crawler#javascript-rendering
One-click install
npx skills add https://github.com/octos-org/octos --skill deep-crawl
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: deep-crawl
Source: https://github.com/octos-org/octos/tree/main/crates/app-skills/deep-crawl
Command: npx skills add https://github.com/octos-org/octos --skill deep-crawl

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Crawling JS-rendered sites to collect rendered text and discovered links, automating content gathering that would be tedious to do manually.

Core Features & Use Cases

  • Renders JavaScript content using headless Chrome via CDP to extract text from each page.
  • Follows same-origin links with BFS and saves per-page results to disk for offline analysis.
  • Ideal for indexing documentation portals, SPAs, and dynamic sites where server-side HTML is insufficient for understanding content.

Quick Start

Provide a seed URL and optional depth and page limits to crawl a website.

Frequently Asked Questions about deep-crawl

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I crawl JavaScript-rendered single page applications for text extraction?▼

Crawl JavaScript-rendered SPAs using a headless Chrome browser via CDP to render content and extract the fully rendered text from each page. This approach automates content gathering that would be tedious to do manually.

Can I limit the crawl depth and number of pages when scraping a documentation portal?▼

Yes, you can configure the crawl depth and page limits when scraping a documentation portal. The crawler follows same-origin links using BFS and saves per-page results to disk for offline analysis.

Does this web crawler include SSRF protection for scraping dynamic sites?▼

Yes, the web crawler includes built-in SSRF protections to ensure safe scraping of dynamic sites. It uses path_prefix filtering and renders JavaScript via headless Chrome to securely collect discovered links and rendered text.

What is the best way to extract content from JS-heavy sites where server-side HTML is insufficient?▼

The best way to extract content from JS-heavy sites is using a headless Chrome browser via CDP to render JavaScript. This ensures you collect the fully rendered text and discovered links, bypassing insufficient server-side HTML.

How does recursive crawling with headless Chrome handle same-origin links?▼

Recursive crawling with headless Chrome handles same-origin links by following them using BFS traversal. It applies configurable depth and page limits, path_prefix filtering, and outputs a structured JSON report of the discovered content.

Is path_prefix filtering supported when indexing dynamic websites?▼

Yes, path_prefix filtering is supported when indexing dynamic websites. This allows you to restrict the recursive headless Chrome crawl to specific URL paths, ensuring the structured JSON report only contains relevant rendered text.

Related Skills

Explore More Software Engineering Skills→
shalevaminshalevamin

url-to-markdown

Fetch URLs via Chrome CDP and convert rendered HTML to Markdown.

Community
Intermediate
cas-bigdatalabcas-bigdatalab
541

docx_text_extract

Extract text from DOCX files to plain text using Python's zipfile and xml libraries.

Official
Basic
RxFitRxFit

setup-browser-cookies

Import browser cookies into Playwright headless sessions for authenticated page testing.

Community
Intermediate
quanngynxquanngynx

client-side-rendering

Render React applications in the browser with client-side JavaScript.

Community
Intermediate