skills.rest
discreteds
discretedsCOMMUNITY
Explore all skills by this author →
𝕏@hiivmindGitHub

comp-scout-scrape

Extract competition data from aggregator sites and persist it to GitHub issues.

Updated Dec 9, 2025
Search Tags:#automation#structured data#data extraction#web scraping#playwright#competitions#LLM interpretation
One-click install
npx skills add https://github.com/discreteds/competition-scout --skill comp-scout-scrape
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: comp-scout-scrape
Source: https://github.com/discreteds/competition-scout/tree/main/skills/comp-scout-scrape
Command: npx skills add https://github.com/discreteds/competition-scout --skill comp-scout-scrape

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires playwright, and includes scripts (resource) components.

What problem does it solve?

Manual web scraping for competition details is time-consuming, error-prone, and struggles with varied website layouts and date formats. This Skill automates the process using AI to interpret page content, saving significant manual effort and ensuring you capture accurate data.

Core Features & Use Cases

  • LLM-Enhanced Scraping: Intelligently extracts structured data (dates, prompts, brands, prize values) from complex web pages, outperforming traditional regex-based methods.
  • Two-Phase Workflow: Combines Playwright for robust, JavaScript-rendered page fetching with Claude for nuanced natural language data interpretation.
  • Structured Output: Delivers clean, standardized JSON data for each competition, ready for easy integration into other systems or further analysis.
  • Use Case: Automatically gather comprehensive details for dozens of "25 words or less" competitions from various aggregator sites, ensuring you never miss an opportunity and always have up-to-date information.

Quick Start

Scrape all new "25 words or less" competitions from competitions.com.au and netrewards.com.au, then summarize the findings.

Frequently Asked Questions about comp-scout-scrape

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate web scraping for competition data across multiple sites?▼

Web scraping automation combines Playwright for robust page fetching with LLM interpretation to extract structured competition data like titles, prize values, and closing dates. This approach handles complex layouts and varied formats that traditional regex methods struggle with, delivering clean JSON output ready for integration or analysis.

Can I use Playwright and AI to extract structured data from JavaScript-rendered pages?▼

Yes. Playwright renders JavaScript-heavy pages reliably, then Claude interprets the content to extract structured fields like dates, prize summaries, and URLs with high accuracy. This two-phase workflow outperforms regex-based extraction on pages with dynamic content.

How do I detect and handle duplicate competition listings when scraping multiple sources?▼

Duplicate detection compares new scraped data against existing records by checking URL presence in stored issue bodies and calculating normalized title similarity thresholds. Matches trigger comments on existing issues rather than creating duplicates, maintaining data integrity.

What's the best way to persist scraped competition data to a persistent store?▼

Scraped competition records persist as GitHub issues with structured fields (url, title, prize_value, closing_date, scrape_date) and automatic labeling and milestone assignment. This approach provides searchable, versioned storage and enables workflow automation through issue tracking.

Can I scrape competitions from sites like competitions.com.au and netrewards.com.au automatically?▼

Yes. This Skill identifies and extracts new competition listings from aggregator sites, performing duplicate checks and batch-detail retrieval, then auto-persists results as GitHub issues with standardized fields and metadata.

Why use LLM interpretation instead of regex patterns for competition data extraction?▼

LLMs intelligently interpret varied page layouts, date formats, and phrasing without brittle pattern-matching rules. This handles edge cases and format variations across different competition sites, ensuring accurate extraction where regex would fail or require constant maintenance.

Related Skills

Explore More Data & Analytics Skills→
oiiiwjhoiiiwjh
4

skill-installer

Install AI skills from GitHub repositories into the local Codex environment.

Community
Intermediate
athina-aiathina-ai
1.1k

get-qualified-leads-from-luma

Extract Luma event attendees, qualify leads, and export results to Google Sheets.

Official
Advanced
oogDZoogDZ

issue-create

Create GitHub Issues with labels, milestones, and templates from natural language.

Community
Intermediate
darblexdarblex

web-scraping

Fetch and extract web page data using Node.js fetch and regex parsing.

Community
Intermediate