data-scraper-agent

Build scheduled web scraping agents that enrich collected data with Gemini and store results in Notion.

2|Updated Mar 29, 2015
One-click install
npx skills add https://github.com/ovisan/dotfiles --skill data-scraper-agent-ovisan
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: data-scraper-agent
Source: https://github.com/ovisan/dotfiles/tree/main/.agents/skills/data-scraper-agent
Command: npx skills add https://github.com/ovisan/dotfiles --skill data-scraper-agent-ovisan

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires requests, beautifulsoup4, lxml, python-dotenv, pyyaml, notion-client, playwright.

What problem does it solve? Manually monitoring job boards, prices, news feeds, or repositories is repetitive and easy to abandon. This Skill guides the creation of a scheduled data collection agent that scrapes public sources, scores and summarizes items with a free LLM, stores results in a database, and learns from user feedback — all running free on GitHub Actions. ## Core Features & Use Cases - Three-Layer Architecture: Implements a Collect → Enrich → Store pipeline with pluggable source connectors (REST API, HTML scraping, RSS, Playwright for JS-rendered pages). - Batched AI Enrichment: Uses Gemini Flash with a four-model fallback chain and batching (5 items per call) to stay within free-tier rate limits. - Feedback Learning: Converts user accept/reject decisions stored in Notion into prompt bias so scoring improves over time. - Use Case: Build an agent that checks job boards every 3 hours, scores each listing against your resume, pushes matches to a Notion database, and refines its scoring as you mark items Applied or Rejected. ## Quick Start Build me an agent that monitors Hacker News for AI startup funding news and stores scored summaries in Notion on a daily schedule.

Frequently Asked Questions about data-scraper-agent

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a web scraper that runs on a schedule for free?

Combine a Python scraper using requests and BeautifulSoup with a GitHub Actions cron workflow. GitHub Actions is free for public repositories and supports scheduled triggers like every 3 hours, with secrets stored in repository settings.

How to scrape JavaScript-rendered pages with Python?

Use Playwright when requests and BeautifulSoup return empty content because the page renders via JavaScript. Launch a headless Chromium browser, wait for the target selector, then parse the rendered HTML with BeautifulSoup.

What is the free tier limit for the Gemini API?

Gemini 2.0 Flash Lite allows 30 requests per minute and 1500 per day, Gemini 2.0 Flash allows 15 RPM, and Gemini 2.5 Flash allows 10 RPM with 500 requests per day. Batch multiple items per call and use a model fallback chain to stay within limits.

Why does my LLM scraping pipeline hit rate limits?

Calling the LLM once per scraped item exhausts rate limits quickly. Batch 5 items per API call, add a delay of about 7 seconds between calls, and fall back across multiple Gemini models when a 429 response occurs.

Can I store scraped data in Google Sheets instead of Notion?

Yes, the architecture supports Notion, Google Sheets, or Supabase as storage providers selected via config.yaml. Swap the notion_sync module for a sheets_sync or supabase_sync implementation and update the corresponding environment variables.