skills.rest
klu-dev
klu-devCOMMUNITY
Explore all skills by this author →
GitHub

data-scraper-agent

Automate data collection from public websites and APIs with AI enrichment.

2|Updated Apr 14, 2026
Search Tags:#data collection#GitHub Actions#data scraping#database storage#AI enrichment
One-click install
npx skills add https://github.com/klu-dev/porting-ecc-to-vscode --skill data-scraper-agent-klu-dev
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-scraper-agent
Source: https://github.com/klu-dev/porting-ecc-to-vscode/tree/main/.github/skills/data-scraper-agent
Command: npx skills add https://github.com/klu-dev/porting-ecc-to-vscode --skill data-scraper-agent-klu-dev

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires requests, BeautifulSoup, playwright, notion-client, python-dotenv, pyyaml, json, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill addresses the challenge of manually collecting and processing data from public sources, streamlining the process with AI capabilities.

Core Features & Use Cases

  • Automated Data Scraping: Gather data from job boards, price lists, news, GitHub, sports, and more with scheduling and automated scraping.
  • AI Data Enrichment: Use Gemini Flash (free) to analyze and enrich the scraped data with scores, summaries, and classifications.
  • Storage Integration: Store results in Notion, Google Sheets, or Supabase, leveraging APIs for seamless data management.
  • Use Case: A user might use this Skill to monitor stock prices, gather job listings, or track sports scores automatically.

Quick Start

Set up the data-scraper-agent with the target URL, select the desired data fields, and choose the storage location. Then, configure the schedule and start scraping.

Frequently Asked Questions about data-scraper-agent

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate web scraping and store results directly in Notion?▼

Automated web scraping to Notion is handled by configuring a target URL, selecting data fields, and using the notion-client API to store scraped results directly in your database.

Does data scraping with Playwright work on GitHub Actions?▼

Yes, data scraping with Playwright works on GitHub Actions by scheduling automated workflows to gather public website data without needing a dedicated local server environment.

Can I use AI to enrich scraped data for analysis?▼

AI data enrichment for analysis uses Gemini Flash to automatically process scraped data, adding scores, classifications, and summaries to your collected datasets.

What do I need to set up before scraping public websites and APIs?▼

Scraping public websites and APIs requires internet access, Python libraries like requests and BeautifulSoup, plus valid API keys for your target data sources and storage databases.

What is the best way to track stock prices or job listings automatically?▼

Tracking stock prices or job listings automatically is achieved by scheduling the scraper on GitHub Actions to periodically collect data, enrich it with AI, and update your storage.

Are there limitations when scraping data from public sources?▼

Limitations of scraping data from public sources include requiring internet access, depending on target site structure for BeautifulSoup parsing, and needing valid API keys for database storage.

Related Skills

Explore More Data & Analytics Skills→
Sake-TeamSake-Team

data-scraper-agent

Automate Python-based scraping, AI enrichment, and storage to Notion, Sheets, or Supabase.

Official
Advanced
neuron-oneneuron-one
25

data-analysis

Analyze Google Sheets, Notion and HubSpot data via Composio to generate charts and insights.

Community
Intermediate
GGEduGGEdu

data-scraper-agent

Scrape public sources, enrich with Gemini Flash, and store results.

Community
Advanced
DecodoDecodo
151

decodo-scraper

Retrieve structured web data from Google, Amazon, YouTube, and Reddit via the Decodo API.

Official
Intermediate