skills.rest
freedom909
freedom909COMMUNITY
Explore all skills by this author →
GitHub

data-scraper-agent

Builds scheduled Python scraping agents that enrich data with Gemini and store results in Notion.

Updated Mar 18, 2026
Search Tags:#automation#python#Notion#github actions#data collection#web scraping#Gemini
One-click install
npx skills add https://github.com/freedom909/real-estate-saas --skill data-scraper-agent-freedom909
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-scraper-agent
Source: https://github.com/freedom909/real-estate-saas/tree/main/.trae/skills/data-scraper-agent
Command: npx skills add https://github.com/freedom909/real-estate-saas --skill data-scraper-agent-freedom909

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires requests, beautifulsoup4, lxml, python-dotenv, pyyaml, notion-client, playwright.

What problem does it solve? Manually monitoring websites for new jobs, prices, news, or listings is repetitive and easy to forget. This Skill generates a complete automated data collection agent that scrapes public sources on a schedule, enriches each item with an LLM, stores results in a database, and learns from your feedback — all running free on GitHub Actions. ## Core Features & Use Cases - Three-Layer Agent Architecture: Generates a COLLECT → ENRICH → STORE pipeline with pluggable scraper sources (REST API, HTML, RSS, Playwright), batched Gemini analysis, and Notion/Sheets/Supabase storage. - Free-Tier Engineering: Includes a Gemini model fallback chain, batch API calls (5 items per request), rate limiting, and URL deduplication to stay within free quotas. - Feedback Learning: Converts your accept/reject decisions into prompt bias so scoring improves over time, persisted in a JSON file committed by GitHub Actions. - Use Case: Ask for an agent that monitors Hacker News for AI startup funding news — you get a full repo with config.yaml, scraper sources, AI pipeline, Notion sync, and a cron workflow. ## Quick Start Build me an agent that scrapes new GitHub repos tagged with 'llm', summarizes each one with AI, and saves them to Notion every three hours.

Frequently Asked Questions about data-scraper-agent

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build an automated web scraper that runs on a schedule for free?▼

Combine a Python scraper using requests and BeautifulSoup with a GitHub Actions cron workflow, which is free for public repositories. The generated agent runs on your chosen interval, enriches items with Gemini Flash, and pushes results to Notion without any paid hosting.

How to scrape data from websites using Python and BeautifulSoup?▼

Fetch the page with requests, parse the HTML with BeautifulSoup using the lxml parser, and select elements with CSS selectors like soup.select('.listing-card'). Normalize each result into a consistent schema with name, url, and date_found fields before storage.

What should I use when requests and BeautifulSoup return empty pages?▼

Empty responses usually mean the site renders content with JavaScript. Use Playwright to launch a headless Chromium browser, wait for the target selector, and extract the rendered HTML, or inspect network traffic to find the underlying JSON API.

How do I avoid Gemini API rate limits when enriching scraped data?▼

Batch multiple items per API call (around 5) instead of one call per item, and add a delay between requests. The agent also implements a fallback chain across gemini-2.0-flash-lite, gemini-2.0-flash, and gemini-2.5-flash when a model returns 429.

Can I store scraped results in Google Sheets or Supabase instead of Notion?▼

Yes, the storage layer is pluggable via the provider setting in config.yaml, supporting notion, sheets, supabase, or sqlite. You swap the sync module import in main.py and set the corresponding environment variable for your chosen backend.

Why does my scraper get blocked or banned by websites?▼

Blocks happen when requests are sent too fast or without a proper User-Agent header. Add time.sleep between requests, set a descriptive User-Agent, respect robots.txt, and prefer official public APIs when they are available.

Related Skills

Explore More Software Engineering Skills→
sakamoto-family-smilesakamoto-family-smile

data-scraper-agent

Scrape public sources via REST, HTML, or RSS and store enriched data.

Community
Advanced
cipriantitirecipriantitire

data-scraper-agent

Automate scheduled scraping of public sources with AI enrichment and storage.

Community
Advanced
RichJacobs69RichJacobs69

gha-analyzer

Analyze GitHub Actions workflow runs to debug failures and optimize batch timing.

Community
Advanced
LogicPyLogicPy
8

aminer-free-academic

Search AMiner free APIs and normalize academic entity names.

Community
Intermediate