web-scraping

Use a cascading approach: try trafilatura first for plain articles, then requests with rotating user agents, then Playwright with playwright-stealth for JavaScript-heavy protected sites. For TLS fingerprinting defenses, use curl_cffi; for Turnstile challenges, consider SeleniumBase UC Mode or Camoufox.

Check HTTP status codes first (429 for rate limits, 403 for blocks, 404 for missing pages), then match content against regex patterns like 'subscribe to continue' or 'verify you are human'. Known paywall domains with suspiciously short content under 500 characters are also flagged.

Use requests or trafilatura for static HTML pages since they are fast and lightweight. Use Playwright with stealth mode only when the site requires JavaScript rendering or employs anti-bot detection, since browser automation is significantly slower and heavier.

Sync Playwright fails in Jupyter because notebooks run their own event loop. Use the async API with async_playwright and await the fetch calls directly in notebook cells instead.

Use yt-dlp with skip_download enabled to extract title, description, duration, and view counts without downloading the video. For transcripts, enable writesubtitles and writeautomaticsub options to access manual or auto-generated captions.

Scraping public non-credentialed pages is generally not a CFAA violation after hiQ v. LinkedIn, but breaching terms of service creates civil liability. Honor robots.txt, respect rate limits, avoid scraping behind authentication, and stop after a cease-and-desist.