skills.rest
reason-machines
reason-machinesOFFICIAL
Explore all skills by this author →
Website𝕏@reasonmachinesGitHub

autocli-web-scraping

Fetch data from 55+ websites via a Rust CLI with browser session reuse.

4|Updated May 16, 2026
Search Tags:#data extraction#social media#cli#web scraping#twitter#rust#reddit
One-click install
npx skills add https://github.com/reason-machines/devtools-skills --skill autocli-web-scraping-reason-machines
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: autocli-web-scraping
Source: https://github.com/reason-machines/devtools-skills/tree/main/skills/autocli-web-scraping
Command: npx skills add https://github.com/reason-machines/devtools-skills --skill autocli-web-scraping-reason-machines

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Gathering data from social media and content platforms like Twitter, Reddit, YouTube, and Bilibili normally requires writing custom scrapers, managing API tokens, and handling authentication for each site. This Skill provides a single command-line interface to query 55+ websites with 333 built-in commands, reusing logged-in browser sessions instead of managing credentials. ## Core Features & Use Cases - Multi-site data fetching: Query trending topics, search results, timelines, and hot posts across Twitter/X, Reddit, YouTube, HackerNews, Bilibili, Zhihu, Xiaohongshu, Weibo, and more. - Structured output formats: Export results as JSON, YAML, CSV, Markdown, or tables for piping into jq and other tools. - Custom adapter generation: Create declarative YAML pipeline adapters for new sites, or search community-shared adapters via autocli.ai. - Use Case: Monitor a topic like "rust" across Hacker News, Reddit, Twitter, and Dev.to in one script, exporting each platform's results to JSON and merging them with jq. ## Quick Start Use the autocli skill to fetch the top 10 Hacker News stories and output them as JSON.

Frequently Asked Questions about autocli-web-scraping

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scrape Twitter trending topics from the command line?▼

Run autocli twitter trending after installing the autocli binary and Chrome extension. The command reuses your logged-in browser session, so no API token management is needed, and results can be exported with --format json.

What websites does autocli support for data extraction?▼

autocli supports 55+ sites with 333 built-in commands, including Twitter/X, Reddit, YouTube, HackerNews, Bilibili, Zhihu, Xiaohongshu, Weibo, Dev.to, Lobsters, Stack Overflow, arXiv, and Wikipedia. Run autocli list to see all sites and command counts.

Does autocli work without the Chrome extension?▼

Public API commands such as hackernews, devto, lobsters, stackoverflow, arxiv, and wikipedia work without the extension. Browser-based commands for Twitter, Reddit, Bilibili, and similar sites require the extension plus an active logged-in session.

How do I create a custom web scraper adapter with autocli?▼

Define a declarative YAML pipeline in ~/.autocli/adapters/ with fetch, extract, transform, and output steps. Alternatively, use the Chrome extension's selector tool to pick elements visually and let autocli.ai generate the adapter.

Why is autocli not connecting to my browser?▼

Run autocli doctor to diagnose the connection. Verify the Chrome extension is loaded and enabled, and restart Chrome with --remote-debugging-port=9222 so the CLI can reach the browser at ws://localhost:9222.

Can autocli output data in JSON for scripting?▼

Yes, every command supports --format json, yaml, csv, markdown, or table output. JSON output pipes directly into jq for filtering, field extraction, and combining results from multiple sites.

Related Skills

Explore More Software Engineering Skills→
AradotsoAradotso
4

autocli-web-scraping

Fetch web data from 55+ sites via a single CLI workflow.

Official
Intermediate
adiytharpansaadiytharpansa

data-analysis

Analyze structured datasets to identify patterns, trends, and actionable insights.

Community
Intermediate
edwifiguyedwifiguy
1

autocli

Browse and fetch content from 55+ platforms using Chrome login sessions.

Community
Advanced
nguynbon03nguynbon03

web-scraper

Crawl websites and extract structured data to JSON or CSV.

Community
Intermediate