web-scraping

To extract web content from sites with anti-bot protection, use Playwright stealth mode to render pages while avoiding detection, with automated fallbacks to requests and Trafilatura for static articles.

Scraping social media pages and downloading media files requires specialized tools like instaloader and yt_dlp rather than standard requests, enabling reliable metadata extraction from media pages.

Handling paywalls and extraction failures during web scraping is best achieved using a multi-method approach that detects obstacles and applies automated fallbacks across different extraction strategies.

Trafilatura works for extracting article text while enforcing safe, respectful scraping practices through optional caching and rate limiting, yielding deterministic and reproducible results.

Web scrapers fail on dynamic media pages when relying solely on requests because they cannot execute JavaScript; using Playwright with stealth or yt_dlp provides robust fallback extraction.

You need Playwright with stealth mode to scrape websites that block standard requests, as it bypasses anti-bot defenses by rendering dynamic content while maintaining safe scraping practices.