crawl4ai

Generate a CSS/JSON schema once using an LLM via scripts/extraction_pipeline.py --generate-schema, then reuse it with JsonCssExtractionStrategy for fast extraction without further LLM calls. This is 10-100x more efficient than per-request LLM extraction.

Use AsyncWebCrawler's arun_many() method with a list of URLs and a max_concurrent parameter to control parallelism. The scripts/batch_crawler.py script wraps this pattern and saves results to JSON and individual markdown files.

Yes, Crawl4AI runs a real browser via Playwright, so it executes JavaScript by default. Use the wait_for parameter with a CSS selector or JS condition, the js_code parameter to run custom scripts, and increase page_timeout for slow-loading content.

Use the session_id parameter in CrawlerRunConfig to persist a browser session across crawls. Perform the login once with js_code that fills and submits the form, then reuse the same session_id for subsequent requests to protected pages.

Sites like those behind Cloudflare check TLS fingerprints, navigator properties, and request patterns. Reduce detection by running headed instead of headless, setting a realistic user agent, rotating residential proxies, adding delays between requests, and reusing session cookies.

Use LLM extraction only for complex, irregular, or one-off content where no stable HTML structure exists. For repetitive page layouts like product listings or articles, schema-based JsonCssExtractionStrategy is faster and avoids per-request LLM costs.