crwl

Web crawling and markdown extraction converts URLs into filtered, LLM-friendly markdown by fetching content and pruning unnecessary elements. This produces clean output recommended for reading and Q&A tasks over web page content.

Structured data extraction uses CSS selectors and a schema to target specific page elements and extract entities into JSON. This CSS-driven strategy isolates exact fields from the HTML structure as an alternative to LLM-based extraction.

Question answering over page content retrieves and summarizes what a page says about a specific topic using a single command. This function relies on the underlying web page crawling and content extraction capabilities.

Converting web pages to markdown for Q&A is best achieved using default pruning flags to generate md-fit markdown. This filters raw content into clean text optimized for language model consumption and accurate responses.

LLM extraction does not require CSS selectors to pull site data into JSON. You can extract entities and fields using either LLM-based instructions or CSS selector targeting plus a schema, offering two distinct extraction strategies.

Configuring browser options is required to scrape site content effectively. The extraction process depends on running the CLI with specific browser and crawler configurations to properly fetch and prune web pages.