data-scraper-agent

You can automate scheduled web scraping by running this agent on GitHub Actions to collect, deduplicate, and store public data to Notion, Google Sheets, or Supabase without custom infrastructure.

Yes, scraped data can be enriched with LLM summaries and relevance scores using a batch processing model fallback chain with Gemini to add classifications and scoring automatically.

Yes, the agent supports flexible storage targets including Notion, Google Sheets, and Supabase, allowing you to persist scraped and enriched data directly into your preferred database or workspace.

You need to configure your storage target and set the GEMINI_API_KEY environment variable to start scraping, enriching, and persisting scheduled data collection items via GitHub Actions.

The agent applies built-in deduplication logic to scraped results, ensuring that recurring monitoring jobs for prices, job boards, or news feeds do not store duplicate entries in your database.

The best way to track GitHub repositories and RSS feeds on a recurring cadence is using a scheduled agent that scrapes public sources, applies rate-limiting to avoid bans, and batches LLM enrichment for summaries.