webrobot-pipeline

Discover stages with list_stages or the public catalog endpoint, read each stage's arg_schema with describe_stage, then write args as a positional YAML list matching that order. Validate with validate_manifest before deploying via apply_manifest or run_pipeline.

Use scrapy for pure scraping, analytics for small in-RAM relational or Python transforms via DuckDB and Polars, hybrid for scrape-plus-analyze in one job, and spark for big-data scale, ML, or LLM-heavy stages. The same YAML runs on any engine.

Use the wget stage to fetch the PDF URL; the engine auto-detects content type and Tika-parses it into XHTML. Then apply the standard extract stage on the body selector. Avoid browser-based visit stages, which render the pdf.js viewer instead of the document text.

This usually happens when iextract lacks a field prefix and explicit 'as <col>' aliases in its prompt, so columns get named after the whole prompt. Always pass a prefix at position 2 and reference downstream columns as <prefix><snake_case_key>.

Use the visitPaginate stage with the seed URL, the 'next' button selector, and an optional page limit. It loops snapshot-plus-click in one browser session. Do not use explore stages with a bare click action, which run on an unloaded page and time out.

Yes. Define sources as a YAML map keyed by source name, each with its own mini-pipeline; the engine unions the rows and tags them with a source column. Add a match step with a normalized key to group records across sources.