skills.rest
aayushsoam
aayushsoamCOMMUNITY
Explore all skills by this author →
GitHub

data-scraper-agent

Automate scheduled collection and enrichment of public data sources.

1|Updated Mar 31, 2026
Search Tags:#automation#workflow#notion#github-actions#data-scraping#data-storage#ai-enrichment
One-click install
npx skills add https://github.com/aayushsoam/clawbot-plus --skill data-scraper-agent-aayushsoam
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-scraper-agent
Source: https://github.com/aayushsoam/clawbot-plus/tree/main/skills/data-scraper-agent
Command: npx skills add https://github.com/aayushsoam/clawbot-plus --skill data-scraper-agent-aayushsoam

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Build a production-ready, AI-powered data collection agent for any public data source. Runs on a schedule, enriches results with a free LLM, stores results in Notion/Sheets/Supabase, and learns from user feedback. Runs 100% free on GitHub Actions. Use when the user wants to monitor, collect, or track any public data automatically.

Core Features & Use Cases

The Three Layers

Three-layer architecture: Collect → Enrich → Store, where Scraper, AI (LLM), and Storage orchestrate end-to-end data flows.

  • Free Stack: Scraping with requests/BeautifulSoup, JS-rendering with Playwright, AI enrichment with Gemini Flash, storage in Notion/Sheets/Supabase, and scheduling via GitHub Actions.
  • AI-driven enrichment: Score, summarize, classify, and persist insights; learn from user feedback to improve over time.
  • Use Cases: Monitor job boards, prices, news, GitHub repos, sports results, events, and other public data sources.

Quick Start

Configure your environment, add a data source, and run the agent to start scraping on a schedule.

Frequently Asked Questions about data-scraper-agent

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate web scraping and store the results in Notion on a schedule?▼

Automate web scraping and store results in Notion by running a Python scraper on a GitHub Actions schedule, enriching the data with AI, and syncing it directly to your Notion database for free.

Can I use GitHub Actions to run a Python web scraper for free?▼

Yes, you can run a Python web scraper for free using GitHub Actions. The agent schedules scraping tasks, uses requests or Playwright for data collection, and orchestrates the entire process without server costs.

What is AI enrichment in data scraping and how does it work with Gemini?▼

AI enrichment in data scraping processes collected text to score, summarize, and classify information. This agent uses Gemini Flash API to batch process scraped data and extract actionable insights automatically.

Does this data scraping agent support JavaScript-rendered websites with Playwright?▼

Yes, the data scraping agent supports JavaScript-rendered websites using Playwright. It combines Playwright for dynamic content with requests and BeautifulSoup for standard static page parsing.

How do I monitor job boards and prices automatically without duplicates?▼

Monitor job boards and prices automatically by configuring a scheduled GitHub Actions workflow. The agent scrapes public sources, enriches results with AI, and applies deduplication by URL to prevent repeated entries.

What are the limitations of using free LLMs for data scraping enrichment?▼

Limitations of using free LLMs like Gemini Flash include rate limits on batch AI calls and the need to manage API access. The agent mitigates this with a lightweight feedback loop to improve enrichment quality over time.

Related Skills

Explore More Data & Analytics Skills→
Yog-SothoYog-Sotho
4

data-scraper-agent

Schedule public data crawls with AI enrichment and storage to Notion, Sheets, or Supabase.

Community
Advanced
abhishekdutta18abhishekdutta18

data-scraper-agent

Scrape public data sources with Python and store results in Notion, Sheets, or Supabase.

Community
Intermediate
StarrySerendipityStarrySerendipity

webapp-testing

Automate end-to-end testing of local web apps with Playwright in Python.

Community
Intermediate
XSpoonAiXSpoonAi
17

spoonos-deployment-guide

Automate SpoonOS agent deployment to Docker, AWS Lambda, Cloud Run, Vercel, and VPS.

Official
Advanced