skills.rest
luongldptit
luongldptitCOMMUNITY
Explore all skills by this author →
GitHub

data-scraper-agent

Automate scheduled collection of public data from websites, APIs, and RSS feeds.

Updated Mar 26, 2026
Search Tags:#github actions#web scraping#data automation#notion integration#gemini flash#data monitoring
One-click install
npx skills add https://github.com/luongldptit/move-ticket --skill data-scraper-agent-luongldptit
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-scraper-agent
Source: https://github.com/luongldptit/move-ticket/tree/main/.agent/skills/data-scraper-agent
Command: npx skills add https://github.com/luongldptit/move-ticket --skill data-scraper-agent-luongldptit

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Manually tracking and collecting public data from websites, APIs, job boards, price trackers, news sites, and other sources is time-consuming, error-prone, and impossible to maintain on a consistent schedule without dedicated hosting and engineering work.

Core Features & Use Cases

  • Automated Scheduled Scraping: Collect data from any public website, REST API, RSS feed, or JS-rendered page on a custom schedule, with built-in rate limiting and deduplication to avoid bans and duplicate entries.
  • AI-Powered Enrichment: Use free Gemini Flash to score, summarize, classify, and match scraped items to your custom priorities and context, with automatic model fallback to handle quota limits.
  • Flexible Storage & Learning: Push results directly to Notion, Google Sheets, or Supabase, and improve scoring over time by learning from your saved, skipped, or rejected items. Common use cases include monitoring job listings for matches to your resume, tracking product prices for drops, aggregating news on specific topics, and surveilling GitHub repos for new releases.

Quick Start

Use the data-scraper-agent skill to build an automated agent that scrapes Hacker News for AI startup funding news, scores each post with Gemini Flash, and stores relevant results in your Notion database to run for free on GitHub Actions every 3 hours.

Frequently Asked Questions about data-scraper-agent

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate web scraping on a schedule without paying for hosting?▼

You can automate web scraping for free by hosting the agent on GitHub Actions, which runs your data collection tasks on custom schedules with zero infrastructure costs. This eliminates the need for dedicated servers or paid hosting environments.

How does AI enrichment work for scraped data?▼

AI enrichment uses Gemini Flash to score, summarize, classify, and match scraped items against your custom priorities. It includes automatic model fallback to handle quota limits, ensuring your data processing continues without interruption.

Can I use automated scraping to monitor job boards and track price drops?▼

Yes, automated scraping supports monitoring job boards for resume matches, tracking product price drops, aggregating news, and surveilling GitHub repos. It collects data from websites, APIs, and RSS feeds with built-in deduplication to avoid duplicate entries.

Does this scraping agent integrate with Notion and Google Sheets for storage?▼

Yes, the scraping agent integrates directly with Notion, Google Sheets, and Supabase to push scraped results automatically. This multi-destination storage allows you to save, analyze, and organize collected data within your preferred workspace.

What is the best way to filter aggregated news using AI?▼

The best way to filter aggregated news is using batch Gemini Flash AI enrichment, which scores and classifies scraped posts based on your custom context. It improves over time by learning from your saved, skipped, or rejected items to refine relevance.

How do I prevent bans when scraping public websites and APIs?▼

To prevent bans when scraping public websites and APIs, the agent uses built-in rate limiting and URL deduplication. These features help you avoid triggering anti-bot protections and ensure consistent data collection without IP blocks.

Related Skills

Explore More Data & Analytics Skills→
emiled16emiled16

data-scraper-agent

Build automated agents that collect, enrich, and store public data from websites, APIs, and RSS feeds.

Community
Advanced
sachio222sachio222

microinteractions

Audit and improve web and mobile microinteractions using the Trigger, Rules, Feedback, Loops & Modes framework.

Community
Advanced
Michae2xlMichae2xl

data-scraper-agent

Automate public data collection and AI enrichment with Python and Gemini Flash.

Community
Advanced
AradotsoAradotso
5

llm-public-opinion-analytics

Collect and analyze real-time social media public opinion with LLMs.

Official
Advanced