skills.rest
inclusionAI
inclusionAIOFFICIAL
Explore all skills by this author →
WebsiteGitHub

xhs-scraper

Scrapes Xiaohongshu search results via CDP browser into Markdown, RSS, or JSON.

1.2k|126|Updated Mar 14, 2025
Search Tags:#Browser Automation#CDP#Json#rss#XiaoHongShu#Social Media#Web scraping
One-click install
npx skills add https://github.com/inclusionAI/AWorld --skill xhs-scraper
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: xhs-scraper
Source: https://github.com/inclusionAI/AWorld/tree/main/examples/skill_agent/skills/xhs-scraper
Command: npx skills add https://github.com/inclusionAI/AWorld --skill xhs-scraper

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires agent-browser, python3, and includes scripts (resource) components.

What problem does it solve?

Manually collecting Xiaohongshu (RED) posts for keyword research, market monitoring, or content analysis is slow and repetitive. This Skill automates the process by driving a CDP-connected browser to scroll search result pages, capture post cards, and optionally open detail pages to extract full note text.

Core Features & Use Cases

  • Keyword Search Scraping: Searches Xiaohongshu by keyword, scrolls the results page, and deduplicates collected post cards (title, author, likes, link, cover).
  • Detail Extraction: Opens the top posts sorted by likes and extracts full note body text and dates via multiple selector fallbacks.
  • Multi-Format Output: Exports results as Markdown, RSS 2.0 feed, or structured JSON with metadata.
  • Use Case: A recruiter wants to track posts about "AI Agent jobs" on Xiaohongshu. Run the scraper with the keyword, output an RSS feed, and subscribe to it in a feed reader for ongoing monitoring.

Quick Start

Ask the agent to scrape Xiaohongshu search results for the keyword "Agent开发工程师" and output the top 10 posts with details as a Markdown file.

Frequently Asked Questions about xhs-scraper

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scrape Xiaohongshu search results by keyword?▼

Run the scrape_xhs.sh script with the -k flag followed by your keyword. It navigates a CDP-connected browser to the Xiaohongshu search page, scrolls to collect post cards, and outputs titles, authors, likes, and links.

How to export Xiaohongshu posts as an RSS feed?▼

Pass -f rss and -o feed.xml to the scraper script. It generates a valid RSS 2.0 XML feed where each post becomes an item with title, link, author, and description containing the note text and like count.

What dependencies are required to run this Xiaohongshu scraper?▼

The scraper requires the agent-browser CLI already connected to a running browser via CDP (default port 9222), plus python3 for JSON processing and output formatting. No additional Python packages are needed beyond the standard library.

Why does the scraper sometimes fail to get post detail text?▼

Detail extraction can fail for image-only posts without text content, or when the note popup selectors change. The script falls back across multiple DOM selectors and skips posts it cannot locate in the page after scrolling.

Can I limit how many posts get full detail extraction?▼

Yes, use the -d flag to set how many top posts (sorted by like count) get their detail pages opened. Setting -d 0 scrapes only the list view without opening any post details, which is faster.

Related Skills

Explore More Data & Analytics Skills→
selfmailselfmail
21

electron

Automate Electron desktop apps via Chrome DevTools Protocol to control UI and extract state.

Official
Advanced
humanerd-drewhumanerd-drew
2

seo-article-harvester

Monitor RSS feeds for SEO articles and generate weekly trend reports.

Official
Advanced
gooseworks-aigooseworks-ai
1.1k

competitor-content-tracker

Scrape competitor blogs, LinkedIn, and Twitter to generate weekly content digests.

Official
Advanced
slavochek2slavochek2
1

tos-review

Audits Terms of Service alignment with newly shipped features and data flows.

Community
Intermediate