Analyze unknown websites and generate JSON plus Markdown investigation reports.
npx skills add https://github.com/YH-05/quants --skill site-investigator
Please help me install this Agent Skill. Skill: site-investigator Source: https://github.com/YH-05/quants/tree/main/.claude/skills/site-investigator Command: npx skills add https://github.com/YH-05/quants --skill site-investigator
New developers and data teams often struggle to quickly understand the structure of unknown websites before implementing scraping pipelines. The Site Investigator automates a pre-scraping survey using Playwright MCP to map page structure, selectors, pagination, RSS or sitemap availability, and dynamic behavior, and it outputs structured results in JSON and human-friendly Markdown.
Inspect a target URL and generate a JSON + Markdown site investigation report.
High-intent search queries and answers about installing and using this skill.
To analyze website structure before scraping, you can automate a pre-scraping site investigation to map page types, selectors, pagination, and dynamic behavior. This process outputs a structured JSON and Markdown report detailing the findings for reliable scraping.
Yes, Playwright can be used to detect RSS feeds, sitemaps, and analyze robots.txt during site exploration. It probes unknown websites to identify navigation patterns and basic accessibility cues to inform your crawl policy before scraping.
A pre-scraping site investigation report is a structured JSON and Markdown document detailing a target website's selectors, pagination, and dynamic behavior. It provides a quick handoff to downstream scrapers for decision-making and implementation.
You identify CSS selectors for web scraping automation by systematically probing target pages with Playwright. This site exploration identifies stable selectors and navigation patterns, outputting the results in a ready-to-use JSON report.
Yes, web scraping with Playwright handles dynamic website behavior by mapping it during the initial site investigation. It applies at the start of new scraping projects to analyze dynamic behavior alongside site structure and pagination.
You should run a site investigation for scraping at the start of any new project targeting an unknown website. This ensures you understand the site structure, selectors, and crawl policies before implementing your scraping pipeline.
gtpgg1013Clone a live website's visuals and core interactions from a URL.
antonyfmunozScrape websites into clean markdown and structured JSON via Firecrawl API.
mbastakisCrawl web pages with Python and extract structured data as Markdown or JSON.
kollaboraiAutomate local web app testing with Playwright and Python scripts.
