web-scraping-pro

Scrape Cloudflare-protected sites using smart fallback methods like Jina Reader and Crawl4AI, which bypass bot protections to return usable page data instead of errors. The skill automatically attempts alternative extraction methods when standard fetching fails.

JavaScript rendering is supported to capture content from dynamic single-page apps. Tools like Crawl4AI render pages so you retrieve fully populated HTML or markdown instead of empty shells.

Structured extraction targets specific page elements using selectors to pull clean markdown, HTML, text, tables, or lists. This enables automated research workflows by returning organized fields rather than raw markup.

Batch scraping is supported, allowing you to extract structured data across multiple URLs. The skill applies retries and selector-based targeting to ensure consistent extraction across large sets of pages.

Web scraping fails on protected sites when bot detection blocks requests or JavaScript rendering is required. Fallback methods using Jina Reader and Crawl4AI are needed when normal fetching returns empty HTML or access errors.

Crawl4AI is a required dependency to use this skill's advanced scraping capabilities. It provides the JavaScript rendering and fallback extraction logic needed to bypass protections and retrieve dynamic content.