What problem does it solve? Extracting content from websites fails unpredictably due to paywalls, CAPTCHAs, rate limits, and JavaScript-rendered pages, while naive scrapers risk SSRF attacks, prompt injection from untrusted content, and legal exposure from bypassing access controls. ## Core Features & Use Cases - Scraping cascade with fallback: Automatically tries Trafilatura, then Requests + BeautifulSoup, then Playwright (sync and async for Jupyter) until content extraction succeeds. - Trust-boundary controls: Validates every URL and redirect hop against private-network destinations, delimits retrieved content as untrusted EXTERNAL_DATA, and blocks credential leakage. - Poison-pill detection: Identifies paywalls, CAPTCHAs, rate limits, Cloudflare checks, and login walls, then stops rather than escalating evasion. - Social media archiving: Extracts YouTube metadata and transcripts via yt-dlp, Instagram posts via instaloader, and TikTok profiles, plus observed-API analysis from browser dev tools. - Use Case: A researcher needs article text from 50 public URLs. The cascade fetches each with polite per-domain delays, detects a paywall on three, and returns delimited, provenance-tagged content for the rest. ## Quick Start Scrape the public page at this URL using the web-scraping cascade and return the extracted content wrapped in EXTERNAL_DATA tags with its source.
