skills.rest
Femad-6
Femad-6COMMUNITY
Explore all skills by this author →
WebsiteGitHub

data-scraper-agent

Builds scheduled Python scraping agents that enrich data with Gemini and store results in Notion.

Updated Mar 25, 2026
Search Tags:#automation#python#github actions#data collection#web scraping#notion#gemini api
One-click install
npx skills add https://github.com/Femad-6/my-skills --skill data-scraper-agent-femad-6
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-scraper-agent
Source: https://github.com/Femad-6/my-skills/tree/main/.github/skills/data-scraper-agent
Command: npx skills add https://github.com/Femad-6/my-skills --skill data-scraper-agent-femad-6

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires requests, beautifulsoup4, lxml, python-dotenv, pyyaml, notion-client, playwright.

What problem does it solve? Manually monitoring job boards, prices, news feeds, or GitHub repos is repetitive and easy to forget. This Skill generates a complete automated data collection agent that scrapes public sources on a schedule, scores and summarizes items with a free LLM, and stores results in a database you can review. ## Core Features & Use Cases - Three-Layer Agent Architecture: Generates a COLLECT → ENRICH → STORE pipeline with pluggable scraper sources, a Gemini AI client with model fallback, and Notion/Sheets/Supabase storage sync. - Batch AI Enrichment with Learning: Batches items into groups of 5 per API call to stay within free-tier rate limits, and biases future scoring using a feedback.json file built from your past accept/reject decisions. - Zero-Cost Scheduling: Ships a GitHub Actions cron workflow that runs the agent for free on public repos and commits updated feedback history automatically. - Use Case: Ask for an agent that monitors Hacker News for AI startup funding news — you get a config-driven Python project that scrapes on a schedule, scores relevance against your profile, and pushes new items to Notion. ## Quick Start Ask the assistant to build a data scraper agent that monitors your chosen source and stores scored results in Notion using this skill.

Frequently Asked Questions about data-scraper-agent

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build an automated web scraping agent in Python?▼

Use requests with BeautifulSoup for HTML pages, REST APIs, or RSS feeds, then normalize each item into a consistent schema with name, url, and date_found. Orchestrate scraping, AI enrichment, and storage sync in a main.py entry point driven by a config.yaml file.

How to scrape websites for free on a schedule?▼

GitHub Actions cron runs Python scrapers free on public repositories using a schedule trigger like cron: "0 */3 * * *". Store API keys as GitHub Secrets and commit output files such as feedback.json back to the repo after each run.

How do I avoid Gemini API rate limits when enriching scraped data?▼

Batch 5 items per API call instead of one call per item, and add a delay of about 7 seconds between calls. Implement a fallback chain across gemini-2.0-flash-lite, gemini-2.0-flash, and gemini-2.5-flash that retries on HTTP 429 responses.

When should I use Playwright instead of requests for scraping?▼

Use Playwright when pages are JavaScript-rendered and requests returns empty or incomplete HTML. Playwright launches a headless Chromium browser, waits for selectors to load, and returns the fully rendered page content for BeautifulSoup parsing.

How do I prevent duplicate entries when syncing scraped data to Notion?▼

Query the Notion database for all existing URLs before pushing, then skip any item whose URL is already stored. Maintain the seen set during the sync loop so duplicates within the same batch are also filtered out.

Can a scraping agent learn from my past decisions?▼

Yes, by storing positive and negative item examples in a feedback.json file and injecting them into the AI prompt as preference signals. A separate sync script queries your storage for items marked Saved or Rejected and updates the feedback file after each run.

Related Skills

Explore More Software Engineering Skills→
sakamoto-family-smilesakamoto-family-smile

data-scraper-agent

Scrape public sources via REST, HTML, or RSS and store enriched data.

Community
Advanced
genesis-agentsgenesis-agents
2

Relevance Judge

Score candidate entries against a topic with 0-100 relevance ratings.

Community
Intermediate
JhabbigJhabbig

secrets-management

Manage CI/CD secrets with rotation, masking, and audit logging.

Community
Advanced
KuaaMUKuaaMU
4

web-scraping

Extract web content using multi-strategy scraping with anti-bot bypass.

Community
Advanced