data-scraper-agent

Combine a Python scraper using requests and BeautifulSoup with a GitHub Actions cron workflow, which is free for public repositories. The workflow installs dependencies, runs the scraper with secrets from GitHub Secrets, and commits feedback data back to the repo.

Batch multiple items into a single API call instead of one call per item — for example, 33 items in batches of 5 need only 7 calls. Add a fallback chain across Gemini models (flash-lite, flash, 2.5-flash) so requests retry on 429 quota errors.

Empty responses usually mean the site renders content with JavaScript. Use Playwright to launch a headless Chromium browser, wait for the target selector, and extract the rendered HTML, or look for the site's underlying JSON API instead.

Yes, the architecture supports Notion, Google Sheets, Supabase, or SQLite via a storage.provider setting in config.yaml. Swap the notion_sync module for a sheets or supabase sync module and update the corresponding environment variables.

Parsing fails when maxOutputTokens is too low and truncates the JSON response, or when the model wraps output in markdown code fences. Set maxOutputTokens to at least 2048 for batch responses and strip leading code fences before calling json.loads.

Query the Notion database for all existing URLs before pushing new items, then skip any item whose URL is already stored. Maintain the seen set during the sync loop so duplicates within the same batch are also filtered.