web-scraper

To extract structured data from dynamic websites, this tool uses Crawl4AI with Playwright to render JavaScript before extracting content. It escalates from simple httpx fetches to full browser rendering, ensuring dynamic content is captured.

Yes, you can use CSS selectors to scrape specific fields into JSON format by defining a JSON CSS schema. This schema specifies field-level types like text, attribute, html, or existence checks for targeted structured data extraction.

The best way to batch process crawling multiple URLs is by providing a file containing the target URLs. The scraper reads the file and executes batch processing, enabling repeatable pipelines across many web pages automatically.

LLM-guided extraction works with crawled HTML and markdown by applying an optional LLM extraction strategy over the fetched content. This handles complex extraction tasks where CSS selectors alone are insufficient.

You can crawl multiple pages by following links using a deep BFS crawling mode. This link-following approach navigates connected pages systematically, allowing comprehensive site extraction beyond a single-page scrape.

You should use httpx and BeautifulSoup instead of Playwright for simple static web scraping cases. This tiered approach starts with fast simple fetches, reserving Playwright for dynamic content requiring JavaScript rendering.