site-investigator

Analyze unknown websites and generate JSON plus Markdown investigation reports.

1|Updated Jan 11, 2026

Search Tags:#automation #reports #playwright #web-scraping #website-analysis #site-investigation

Ready to run

npx skills add https://github.com/YH-05/quants --skill site-investigator

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: site-investigator
Source: https://github.com/YH-05/quants/tree/main/.claude/skills/site-investigator
Command: npx skills add https://github.com/YH-05/quants --skill site-investigator

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

New developers and data teams often struggle to quickly understand the structure of unknown websites before implementing scraping pipelines. The Site Investigator automates a pre-scraping survey using Playwright MCP to map page structure, selectors, pagination, RSS or sitemap availability, and dynamic behavior, and it outputs structured results in JSON and human-friendly Markdown.

Core Features & Use Cases

Systematic site probing to identify page types, selectors, and navigation patterns for reliable scraping.
RSS/Sitemap detection, robots.txt analysis, and basic accessibility cues to inform crawl policy.
Quick handoff to downstream scrapers with a ready-to-use report for decision-making and implementation.

Quick Start

Inspect a target URL and generate a JSON + Markdown site investigation report.

Frequently Asked Questions about site-investigator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I analyze website structure before scraping?▼

To analyze website structure before scraping, you can automate a pre-scraping site investigation to map page types, selectors, pagination, and dynamic behavior. This process outputs a structured JSON and Markdown report detailing the findings for reliable scraping.

Can Playwright detect RSS feeds and sitemaps for web scraping?▼

Yes, Playwright can be used to detect RSS feeds, sitemaps, and analyze robots.txt during site exploration. It probes unknown websites to identify navigation patterns and basic accessibility cues to inform your crawl policy before scraping.

What is a pre-scraping site investigation report?▼

A pre-scraping site investigation report is a structured JSON and Markdown document detailing a target website's selectors, pagination, and dynamic behavior. It provides a quick handoff to downstream scrapers for decision-making and implementation.

How do I identify CSS selectors for web scraping automation?▼

You identify CSS selectors for web scraping automation by systematically probing target pages with Playwright. This site exploration identifies stable selectors and navigation patterns, outputting the results in a ready-to-use JSON report.

Does web scraping with Playwright handle dynamic website behavior?▼

Yes, web scraping with Playwright handles dynamic website behavior by mapping it during the initial site investigation. It applies at the start of new scraping projects to analyze dynamic behavior alongside site structure and pagination.

When should I run a site investigation for scraping?▼

You should run a site investigation for scraping at the start of any new project targeting an unknown website. This ensures you understand the site structure, selectors, and crawl policies before implementing your scraping pipeline.