crawl4ai

Generate a CSS extraction schema once using an LLM with extraction_pipeline.py --generate-schema, then reuse it with JsonCssExtractionStrategy for fast extraction without further LLM calls. This is 10-100x more efficient than per-request LLM extraction.

Use AsyncWebCrawler's arun_many method with a list of URLs and a max_concurrent parameter to control parallelism. The batch_crawler.py script accepts a file of URLs and saves results as JSON plus individual markdown files.

Yes, Crawl4AI runs a real browser via Playwright and renders JavaScript by default. Use the wait_for parameter with a CSS selector or JS condition, the js_code parameter to execute custom scripts, and increase page_timeout for slow-loading pages.

Yes, use the session_id parameter in CrawlerRunConfig to persist a browser session across crawls. Perform the login with js_code in the first crawl, then reuse the same session_id for subsequent requests to protected pages.

Bot detection triggers on headless browsers and rapid requests. Try headless=False, set a realistic user_agent, configure proxy_config in BrowserConfig, and add delays of 2-5 seconds between requests with asyncio.sleep.

Use LLM extraction only for complex, irregular content or one-time extractions where no repeating structure exists. For repetitive page patterns like product listings, schema-based CSS extraction is faster and avoids per-request LLM costs.