xiaohongshu-scraper

What problem does it solve?

It automates the manual, error-prone process of collecting Xiaohongshu search results and post details from the browser into structured data for analysis, saving time for competitor research and content planning.

Core Features & Use Cases

• Browser-based extraction: Uses a Chrome extension context and injected JavaScript to read Xiaohongshu's SSR state and DOM to reliably collect noteIds, titles, engagement counts, tags, images and top comments.
• Scripted workflow with fallbacks: Provides modular scripts for setting filters, extracting search results, opening posts, extracting post details, closing overlays, and a Python post-processing step to merge JSONL into CSV and per-post Markdown files; includes retry and skip strategies for common UI race conditions.
• Outputs and use cases: Produces search_results.jsonl and post_details.jsonl, a combined CSV, and per-post .md files for competitor analysis, content inspiration, trend monitoring, and dataset creation for social research.

Quick Start

Open Chrome while logged into Xiaohongshu and run the provided scripts in sequence to collect search results and post details into JSONL, then run the Python processor to generate CSV and Markdown outputs.