crawl

Web scraping automates extracting article data like titles, authors, and text from websites by visiting pages programmatically and parsing content. This Skill crawls The Atlantic's latest articles, extracts structured data, and saves results to disk, eliminating manual copying and reducing setup time.

Yes. This Skill provides a ready-to-run crawler that fetches multiple articles, extracts titles, subtitles, body text, and author names in a single execution, then saves structured outputs locally. It handles browser automation, error recovery, and UTF-8 logging internally.

Run an automated crawler that fetches the latest article listings, visits each page, extracts full text and metadata, and saves results to local files. This Skill does exactly that for The Atlantic, making it ideal for building daily content summaries or reference libraries.

This Skill requires Python with Pydantic (≥2.12.4) for data validation and python-dotenv (≥1.0.0) for environment configuration. These handle structured output and credential management during long-running crawl operations.

Web scraping depends on stable HTML structure; layout changes can break extraction. This Skill includes robust error handling and UTF-8 logging, but requires periodic review if target pages are redesigned. Rate limits and robots.txt policies should be respected.

Yes. This Skill supports optional translation of five selected articles during the crawl, converting extracted text before saving to disk. You control which pieces receive translation within the execution.