skills.rest
vrcms
vrcmsCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

data-scraper-agent

Scrape public web data and enrich it with Gemini API analysis.

1|Updated Apr 6, 2026
Search Tags:#automation#notion#gemini#scraping#github-actions#data-collection
One-click install
npx skills add https://github.com/vrcms/everything-qwen-code --skill data-scraper-agent-vrcms
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-scraper-agent
Source: https://github.com/vrcms/everything-qwen-code/tree/main/.qwen/skills/data-scraper-agent
Command: npx skills add https://github.com/vrcms/everything-qwen-code --skill data-scraper-agent-vrcms

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires requests, beautifulsoup4, lxml, python-dotenv, pyyaml, notion-client, and includes scripts (resource) components.

What problem does it solve?

This skill solves the challenge of monitoring and collecting data from public sources by automating the entire lifecycle of scraping, AI-driven enrichment, and database synchronization.

Core Features & Use Cases

  • Automated Collection: Scrapes websites, APIs, and RSS feeds on a recurring schedule using GitHub Actions.
  • AI Enrichment: Uses Gemini Flash to score, summarize, and classify collected data based on user-defined priorities.
  • Seamless Storage: Automatically syncs enriched data to Notion, Google Sheets, or Supabase with built-in deduplication.
  • Use Case: Automatically monitor job boards for specific roles, track product price drops, or aggregate news feeds into a centralized dashboard without manual effort.

Quick Start

Use the data-scraper-agent to build a bot that monitors a specific news website for AI-related articles and saves them to my Notion database.

Frequently Asked Questions about data-scraper-agent

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate web scraping and sync the results directly to Notion?▼

Automated web scraping and Notion synchronization are handled by configuring Python scripts to extract public web data and automatically sync the enriched records to your Notion database.

Can I use GitHub Actions to run a recurring data collection schedule?▼

Yes, GitHub Actions can run your data collection scripts on a recurring schedule, automating the entire lifecycle of scraping websites, APIs, and RSS feeds without manual intervention.

How does Gemini API enrich collected data during the scraping process?▼

The Gemini API enriches collected data by using LLM-based analysis to score, summarize, and classify the extracted information based on your user-defined priorities.

What do I need to set up an AI-powered scraper for market monitoring?▼

You need Python dependencies like requests and beautifulsoup4 for extraction, plus API integrations with Gemini for AI enrichment, GitHub Actions for scheduling, and a database provider for storage.

Does this data collection approach handle deduplication when aggregating RSS feeds?▼

Yes, the automated data collection approach includes built-in deduplication to ensure aggregated RSS feeds and scraped content are synced without creating duplicate records.

Are there limitations when scraping public web data with Python and beautifulsoup4?▼

Scraping public web data with Python is limited to accessible HTML structures parsed by beautifulsoup4, requiring external LLM and database integrations to achieve autonomous end-to-end automation.

Related Skills

Explore More Data & Analytics Skills→
rlagycksrlagycks
3

data-scraper-agent

Scrape, process, and store public data from websites, APIs, and RSS feeds.

Community
Advanced
siddsacharsiddsachar
1.4k

data_analyst

Analyze CSV and tabular datasets to generate statistical summaries, narratives, and charts.

Community
Advanced
aayushsoamaayushsoam
1

data-scraper-agent

Automate scheduled collection and enrichment of public data sources.

Community
Advanced
MunreaderMunreader

web-reader

Retrieve and extract titles, HTML, and publication dates from web pages.

Community
Intermediate