skills.rest
inclusionAI
inclusionAIOFFICIAL
Explore all skills by this author →
WebsiteGitHub

x-scraper

Scrapes X (Twitter) user timelines and home feeds via CDP browser automation.

1.2k|126|Updated Mar 14, 2025
Search Tags:#scraping#Browser Automation#CDP#Twitter#rss#X#Social Media
One-click install
npx skills add https://github.com/inclusionAI/AWorld --skill x-scraper
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: x-scraper
Source: https://github.com/inclusionAI/AWorld/tree/main/examples/skill_agent/skills/x-scraper
Command: npx skills add https://github.com/inclusionAI/AWorld --skill x-scraper

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires agent-browser, python3.

What problem does it solve?

Collecting posts from X (Twitter) manually is slow and unstructured. This Skill automates scraping of user timelines and home recommendation feeds through a CDP-connected browser, producing clean Markdown, RSS, or JSON output.

Core Features & Use Cases

  • User Timeline Scraping: Fetch the latest posts from any X user, with optional keyword filtering via search queries.
  • Home Feed Scraping: Capture the For You or Following recommendation tabs of the logged-in account, with ad filtering and deduplication.
  • Multi-Format Output: Export results as Markdown reports, RSS feeds, or structured JSON with author, timestamp, text, link, and media flags.
  • Use Case: Monitor a competitor's X account by running the user scraper with a keyword filter and exporting an RSS feed for your feed reader.

Quick Start

Ask the agent to scrape the latest posts from a specific X user and output the results as a Markdown file.

Frequently Asked Questions about x-scraper

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scrape tweets from a specific X user?▼

Run scrape_x_user.sh with the -u flag followed by the username, for example ./scrape_x_user.sh -u chenchengpro. Add -k to filter by keyword, and -f to choose md, rss, or json output.

How to export X home feed as an RSS feed?▼

Run scrape_x_home.sh with -f rss and -o to write the feed to a file, such as ./scrape_x_home.sh -f rss -o home.xml. Use -t following to capture the Following tab instead of For You.

Does X scraping require a logged-in browser session?▼

Yes, the scripts depend on agent-browser connected via CDP to a running browser that is already logged into X. The default CDP port is 9222, configurable with the -p flag.

What output formats does the X scraper support?▼

Both scripts support Markdown, RSS XML, and JSON via the -f flag. JSON output includes structured metadata such as author name, handle, ISO timestamp, link, and media indicators.

Why does X scraping stop before reaching max scrolls?▼

The scripts stop scrolling when a round yields no new posts, indicating the feed has no more content to load. Increase the -n scroll count or check that the page loaded correctly if results seem truncated.

Related Skills

Explore More Data & Analytics Skills→
hammercuihammercui

agent-browser

Automates browser tasks via a command-line interface with CDP-first connections and persistent sessions.

Community
Advanced
lordmaculordmacu
6

RSS / Atom

Fetches and parses RSS, Atom, and JSON feeds via feed-rs.

Community
Basic
marmikcfcmarmikcfc

twitter-profile-lookup

Fetch Twitter/X profile data and engagement metrics for a given handle.

Community
Intermediate
quiznatquiznat

x-non-api-reading

Read X/Twitter posts via browser automation and search-index fallbacks.

Community
Intermediate