skills.rest
gganbukim1
gganbukim1COMMUNITY
Explore all skills by this author →
GitHub

data-scraper-agent

Scrape public web sources on a schedule and enrich items with Gemini summaries.

Updated May 4, 2026
Search Tags:#web scraping#GitHub Actions#deduplication#data enrichment#scheduled monitoring#Notion sync#LLM batch processing
One-click install
npx skills add https://github.com/gganbukim1/myskills --skill data-scraper-agent-gganbukim1
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-scraper-agent
Source: https://github.com/gganbukim1/myskills/tree/main/data-scraper-agent
Command: npx skills add https://github.com/gganbukim1/myskills --skill data-scraper-agent-gganbukim1

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Manual web scraping and copy-pasting public updates is slow, brittle, and hard to keep consistent over time, so this Skill automates collecting public data on a schedule, enriching it with AI, and storing results for review and action.

Core Features & Use Cases

  • Scheduled collection: Run scrapers periodically (e.g., via GitHub Actions cron) to monitor public sources like job boards, prices, news, GitHub, and sports.
  • AI enrichment with free Gemini: Batch items and enrich each record with LLM-generated scoring, summaries, and notes using Gemini Flash (with model fallback).
  • Deduped storage workflows: Deduplicate by URL and sync results into a destination like Notion (with room to extend to Sheets or Supabase).
  • Continuous learning from feedback: Record positive/negative user decisions in a JSON file and use it to bias future scoring.

Quick Start

Ask the agent to build a scheduled public-data scraper that collects items from a chosen URL or API, enriches them with Gemini using batch scoring, and writes the deduplicated results into your Notion database.

Frequently Asked Questions about data-scraper-agent

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate scheduled web scraping and sync the results to Notion?▼

Scheduled web scraping can be automated by running scrapers periodically via GitHub Actions cron, enriching the collected items with LLM, deduplicating by URL, and syncing the results directly into a Notion database.

Can I use LLM batch processing to enrich scraped data with summaries and scores?▼

LLM batch processing enriches scraped data by feeding collected items to Gemini Flash to generate AI-driven summaries, scores, and notes. It includes model fallback and rate limiting to ensure consistent batch processing.

What is URL-based deduplication and how does it work for recurring data collection?▼

URL-based deduplication prevents duplicate records during recurring data collection by checking the web address of each scraped item, ensuring that only new, unique results are synced into your target storage system.

Does GitHub Actions support running web scrapers for continuous monitoring of public data?▼

GitHub Actions supports continuous monitoring of public data by utilizing cron schedules to trigger scrapers periodically. This enables automated collection of job boards, prices, news, and GitHub activity without manual intervention.

How do I set up continuous learning from feedback for my AI data enrichment workflow?▼

Continuous learning from feedback is set up by recording positive or negative user decisions in a JSON file, which the system then uses to bias and adjust future LLM-generated scoring for subsequent data enrichment workflows.

What are the limitations of using free Gemini for batch enrichment in data scraping?▼

Using free Gemini for batch enrichment requires managing rate limiting constraints and potential model unavailability, which is handled through model fallback mechanisms to ensure the data scraping workflow completes successfully.

Related Skills

Explore More Data & Analytics Skills→
samymitysamymity
1

data-scraper-agent

Scrape public websites, APIs, and RSS feeds on a schedule.

Community
Advanced
Greenmamba29Greenmamba29

ai-enricher

Enrich data records with LLM-generated insights, classifications, and summaries.

Community
Intermediate
agentmattersagentmatters
3

data-scraper-agent

Automate public data collection and enrichment from websites and APIs.

Official
Advanced
riftzen-bitriftzen-bit

data-scraper-agent

Collect and monitor public web and API data on a schedule.

Community
Advanced