crawl

Name: crawl
Availability: InStock
Author: flyingtimes

Crawl The Atlantic articles and extract titles, authors, and body text to disk.

Updated Nov 17, 2025

Search Tags:#automation #web scraping #article extraction #crawl #The Atlantic #text saving

Ready to run

npx skills add https://github.com/flyingtimes/clark-skill-collection --skill crawl

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: crawl
Source: https://github.com/flyingtimes/clark-skill-collection/tree/main/.claude/skills/crawl
Command: npx skills add https://github.com/flyingtimes/clark-skill-collection --skill crawl

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pydantic>=2.12.4, python-dotenv>=1.0.0, and includes scripts (resource) components.

What problem does it solve?

这个技能接收用户需求，自动抓取 The Atlantic 网站的最新文章，提取标题、作者和正文，并保存为文本，显著降低网页爬取的难度与时间成本。

Core Features & Use Cases

自动化抓取最新文章：从 The Atlantic 最新页面提取文章链接。
逐篇文本提取与保存：保存标题、作者、正文到本地文本文件。
Use Case: 想建立每天的 The Atlantic 摘要库时，运行此技能即可获得新的文章文本。

Quick Start

运行脚本 main.py，程序将爬取 The Atlantic 的最新文章并输出到 output/ 目录。

Frequently Asked Questions about crawl

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I automate web scraping of news articles from a website?▼

Web scraping automates extracting article data like titles, authors, and text from websites by visiting pages programmatically and parsing content. This Skill crawls The Atlantic's latest articles, extracts structured data, and saves results to disk, eliminating manual copying and reducing setup time.

Can I extract article text and metadata at scale without writing a crawler from scratch?▼

Yes. This Skill provides a ready-to-run crawler that fetches multiple articles, extracts titles, subtitles, body text, and author names in a single execution, then saves structured outputs locally. It handles browser automation, error recovery, and UTF-8 logging internally.

What's the best way to build a daily archive of news content from a specific publication?▼

Run an automated crawler that fetches the latest article listings, visits each page, extracts full text and metadata, and saves results to local files. This Skill does exactly that for The Atlantic, making it ideal for building daily content summaries or reference libraries.

Do I need Python and Pydantic to run web scraping automation?▼

This Skill requires Python with Pydantic (≥2.12.4) for data validation and python-dotenv (≥1.0.0) for environment configuration. These handle structured output and credential management during long-running crawl operations.

What are the limitations of automated article extraction from news websites?▼

Web scraping depends on stable HTML structure; layout changes can break extraction. This Skill includes robust error handling and UTF-8 logging, but requires periodic review if target pages are redesigned. Rate limits and robots.txt policies should be respected.

Can I translate extracted articles automatically as part of the crawl?▼

Yes. This Skill supports optional translation of five selected articles during the crawl, converting extracted text before saving to disk. You control which pieces receive translation within the execution.