skills.rest
zero3041
zero3041COMMUNITY
Explore all skills by this author →
GitHub

data-scraper-agent

Automate public data collection, AI enrichment, and database storage.

1|Updated Apr 6, 2026
Search Tags:#monitoring#database#data collection#scraper#AI enrichment
One-click install
npx skills add https://github.com/zero3041/PREP --skill data-scraper-agent-zero3041
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-scraper-agent
Source: https://github.com/zero3041/PREP/tree/main/.claude/skills/skills/data-scraper-agent
Command: npx skills add https://github.com/zero3041/PREP --skill data-scraper-agent-zero3041

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires requests, beautifulsoup4, lxml, python-dotenv, pyyaml, notion-client, playwright, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill automates the collection of data from public sources, enriching it with AI insights, and storing it in databases like Notion, Sheets, or Supabase.

Core Features & Use Cases

  • Automated Data Collection: Monitor and collect data from job boards, prices, news, GitHub, sports, and more.
  • AI Enrichment: Use Gemini Flash to score, summarize, classify, and match each item.
  • Storage: Store results in Notion, Sheets, or Supabase.
  • Use Case: Create an agent to track job listings, monitor product prices, or collect news articles.

Quick Start

Use the data-scraper-agent skill to create a scraper for job listings on [URL].

Frequently Asked Questions about data-scraper-agent

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate data collection from public sources and store it in Notion?▼

You can automate data collection from public sources using Python scripts with Playwright, enrich scraped items with Gemini Flash, and store the results in Notion via its database APIs.

Can I use GitHub Actions to schedule a web scraper for monitoring prices?▼

Yes, GitHub Actions can schedule web scrapers for monitoring prices. The scraper uses Python and Playwright to collect public data, applies AI enrichment via Gemini Flash, and stores results in databases like Notion or Supabase.

What is AI enrichment and how does it work with scraped data?▼

AI enrichment processes collected data using Gemini Flash to score, summarize, classify, and match each item. It adds analytical insights to raw data scraped from public sources before storing it in databases.

Does this data collection method work with dynamic JavaScript websites?▼

Yes, scraping dynamic JavaScript websites is supported through the Playwright dependency. It enables browser automation to extract job listings, news articles, and other public data that requires rendering.

Do I need a Python environment to run the data scraper agent?▼

Yes, a Python environment is required. The data scraper agent relies on Python libraries including requests, beautifulsoup4, lxml, and playwright to automate data collection and execute scraping scripts.

What are the limitations of using BeautifulSoup for web scraping?▼

BeautifulSoup parses static HTML using lxml and cannot render dynamic JavaScript content. For dynamic public sources, Playwright is required to automate browser interactions and collect data effectively.

Related Skills

Explore More Data & Analytics Skills→
IhsanDanish25IhsanDanish25
2

theme-detector

Detect and analyze trending market themes across sectors with confidence scores.

Community
Advanced
KanakMalpaniKanakMalpani

data-scraper-agent

Automates scheduled collection, enrichment, and storage of public data into Notion, Sheets, or Supabase.

Community
Advanced
aayushsoamaayushsoam
1

data-scraper-agent

Automate scheduled collection and enrichment of public data sources.

Community
Advanced
ROYCE-8425ROYCE-8425
1

data-scraper-agent

Automate public data collection, AI enrichment, and database storage.

Community
Advanced