webcrawl

What problem does it solve?

It helps you gather relevant web information and benchmarks when a math modeling problem depends on real-world policies, industries, geographies, or economic reference data.

Core Features & Use Cases

• Conditional web sourcing: Automatically decides when to crawl based on problem background signals (policy/industry/geo/economy) and when not to use web data for pure math/physics mechanism problems.
• Multi-engine retrieval pipeline: Supports single-URL reading via Jina Reader, scraping via Firecrawl, and search-then-read via Tavily, Exa, and SerpAPI with a fallback strategy.
• Deterministic storage & traceability: Saves fetched pages into a structured external web directory with standardized header metadata (source, fetched time, and URL).
• Robust failure handling: Covers missing keys, timeouts, HTTP errors, and pages behind login/verification with clear diagnostics instead of silently continuing.

Quick Start

Use webcrawl to retrieve industry or policy benchmark information for your current math modeling task from the relevant web pages.