skills.rest
llmh333
llmh333COMMUNITY
Explore all skills by this author →
GitHub

data-scraper-agent

Automate scheduled collection, AI enrichment, and deduplicated storage of public data.

Updated Sep 13, 2025
Search Tags:#github actions#gemini#deduplication#data scraping#notion sync#ai enrichment#web monitoring
One-click install
npx skills add https://github.com/llmh333/employee_management_spring --skill data-scraper-agent-llmh333
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-scraper-agent
Source: https://github.com/llmh333/employee_management_spring/tree/main/.gemini/skills/data-scraper-agent
Command: npx skills add https://github.com/llmh333/employee_management_spring --skill data-scraper-agent-llmh333

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill eliminates the manual effort of repeatedly scraping, tracking, and organizing public information by automating collection, AI enrichment, and storage.

Core Features & Use Cases

  • COLLECT → ENRICH → STORE pipeline: Scrapes from public sources on a schedule, enriches results with Gemini Flash, and stores structured outputs for review and action.
  • Batch AI enrichment: Scores, summarizes, and classifies items in batches to stay within free-tier limits while producing consistent JSON-shaped results.
  • Scheduled GitHub Actions + deduplication: Runs via cron on GitHub Actions, deduplicates items by URL, and syncs new entries to a storage provider (e.g., Notion).

Quick Start

Tell your AI to generate a scraper that monitors your chosen public source on a 3-hour schedule, enriches each new item with Gemini Flash, and saves the deduplicated results to Notion using the provided architecture.

Frequently Asked Questions about data-scraper-agent

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate web monitoring and sync scraped data to Notion?▼

Automated web monitoring syncs scraped data to Notion by running a COLLECT→ENRICH→STORE pipeline on a GitHub Actions cron schedule. It scrapes public sources, deduplicates new items by URL, and syncs the structured outputs directly to your Notion database.

Can I use Gemini Flash to enrich and summarize scraped web data automatically?▼

Gemini Flash enriches scraped web data automatically by scoring, summarizing, and classifying items in batches. This batch processing ensures consistent JSON-shaped results while staying within free-tier limits, with a model fallback mechanism for reliability.

What is the best way to scrape JS-rendered pages and RSS feeds on a schedule?▼

Scraping JS-rendered pages and RSS feeds on a schedule is best handled via GitHub Actions cron jobs. This approach automates the recurring collection of public data items from various sources, keeping updates deduplicated by URL for clean tracking.

Does GitHub Actions support deduplication for recurring data scraping tasks?▼

GitHub Actions supports deduplication for recurring data scraping tasks by running scheduled crons that process new items. As the scraper collects public data, it checks existing records and deduplicates entries by URL before syncing to storage providers.

How do I store and sync deduplicated scraped items to Supabase or Google Sheets?▼

Storing and syncing deduplicated scraped items to Supabase or Google Sheets is handled by the storage synchronization phase. After collection and AI enrichment, the pipeline syncs only new, deduplicated entries to your chosen provider for review and action.

Related Skills

Explore More Data & Analytics Skills→
EdanStarfireEdanStarfire
15

github-authenticator

Authenticate GitHub CLI credentials and troubleshoot authentication failures via gh commands.

Community
Intermediate
bendruckerbendrucker
16

json

Process JSON data with jq filters for parsing, filtering, and transforming.

Community
Basic
riftzen-bitriftzen-bit

data-scraper-agent

Collect and monitor public web and API data on a schedule.

Community
Advanced
multiplex-aimultiplex-ai
2

data-scraper-agent

Scrape, enrich, and store public data on a schedule.

Official
Advanced