web-scraper

Scrape web pages with crawl4ai and extract sanitized structured content.

Updated Apr 10, 2026

Search Tags:#data extraction #web scraping #url validation #crawl4ai #secure #output sanitization #ssrf protection

Ready to run

npx skills add https://github.com/Exia-thd/Digital-Nervous --skill web-scraper-exia-thd

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: web-scraper
Source: https://github.com/Exia-thd/Digital-Nervous/tree/main/skills/web-scraper
Command: npx skills add https://github.com/Exia-thd/Digital-Nervous --skill web-scraper-exia-thd

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

It provides a production‑grade, security‑first solution for extracting structured, LLM‑ready data from websites while defending against SSRF, LFI, prompt injection, and other web‑based attacks.

Core Features & Use Cases

Library‑only mode: Uses crawl4ai directly without Docker or external services, eliminating surface‑area vulnerabilities.
Strict URL validation: Blocks unsafe schemes, private IPs, and resolves DNS rebinding to prevent SSRF and LFI.
Output sanitization: Strips HTML comments, zero‑width characters, hidden text, and limits content size to stop prompt‑injection attacks.
Rate limiting & robots.txt compliance: Enforces 5 requests‑per‑second and respects site policies.
Extraction hierarchy: CSS‑first, Markdown‑second, LLM‑last, ensuring the safest method is used whenever possible.
Integration patterns: Ready for RAG pipelines, Polymath research, NotebookLM, and downstream data stores.

Quick Start

Use the web‑scraper skill to retrieve and sanitize clean markdown content from https://example.com.

Frequently Asked Questions about web-scraper

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I securely scrape web pages and extract clean structured content for RAG pipelines?▼

The crawl4ai library operates in a library-only mode without Docker or external services, blocking unsafe schemes and private IPs to prevent SSRF. It strips hidden text and zero-width characters from extracted content, stopping prompt injection before data reaches your pipeline.

How do I prevent SSRF and prompt injection when extracting data from websites?▼

Web scraping with crawl4ai requires a Python environment and operates in library-only mode without Docker or external services. You provide a target URL, and the skill validates it, enforces rate limiting at 5 requests per second, and returns sanitized markdown content.

Do I need Docker to use crawl4ai for web scraping and data extraction?▼

Yes, web scraping output is formatted as clean markdown content ready for integration into RAG pipelines, Polymath research, NotebookLM, and downstream data stores. The sanitized markdown ensures compatibility with downstream LLM processing and analysis workflows.

What is the best way to extract LLM-ready clean markdown from websites?▼

You should not use web scraping approaches that lack URL validation when targeting untrusted domains, as they expose you to SSRF and LFI attacks. Strict URL validation that blocks unsafe schemes and resolves DNS rebinding is required to safely extract content.

Why does my web scraper trigger SSRF vulnerabilities when crawling URLs?▼

Web scraping for RAG pipelines returns sanitized markdown content that integrates directly into downstream data stores, Polymath research, and NotebookLM. The clean structured output is specifically formatted for LLM-ready data processing and analysis.