spider

Crawls websites with HTTP fetch, link extraction, and HTML parsing for JSON output.

24|6|Updated Dec 19, 2025

Search Tags:#data extraction #security analysis #web scraping #link extraction #crawling

Ready to run

npx skills add https://github.com/johnlindquist/claude --skill spider

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: spider
Source: https://github.com/johnlindquist/claude/tree/main/skills/spider
Command: npx skills add https://github.com/johnlindquist/claude --skill spider

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Web crawling and scraping with analysis. Use for crawling websites, security scanning, and extracting information from web pages.

Core Features & Use Cases

Fetch pages with curl and inspect headers
Extract links and domain-filtered URLs
Quick site scan for headers, scripts, and meta tags
AI-powered analysis for page content and security insights
Simple extraction of structured data from pages

Quick Start

Use the spider skill to fetch a page, extract its links, and perform a quick meta-tag scan on the target URL.

Frequently Asked Questions about spider

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I crawl a website and extract links?▼

Web crawling fetches pages using HTTP requests and extracts all links found on them. The spider skill crawls websites by fetching pages with configurable headers, parsing HTML to identify links, and filtering results by domain, returning structured data for further analysis or pipeline processing.

Can I use web scraping for security scanning?▼

Yes, security scanning via web scraping analyzes page headers, scripts, and metadata to identify potential vulnerabilities. The spider skill extracts scripts, meta tags, and headers from websites, enabling AI-powered security insights and endpoint discovery across single-page and multi-page sites.

What data can I extract from web pages?▼

Web page extraction retrieves text content, metadata, structured data, and HTML elements. The spider skill parses pages to extract text, meta tags, structured data, and scripts, outputting JSON-compatible results suitable for RAG pipelines, content analysis, and downstream processing.

How do I crawl multi-page websites?▼

Multi-page crawling systematically traverses linked pages across a site. The spider skill implements multi-page crawling with sitemap processing, link extraction, and domain filtering to map entire websites and extract data consistently across all discovered pages.

What's the best way to extract structured data from websites?▼

Structured data extraction parses HTML and metadata into machine-readable formats. The spider skill extracts structured data, metadata, and content from pages, formatting results as JSON compatible with downstream analysis tools and RAG pipelines for automation.