skills.rest
johnlindquist
johnlindquistCOMMUNITY
Explore all skills by this author →
Website𝕏@johnlindquistGitHub

spider

Crawls websites with HTTP fetch, link extraction, and HTML parsing for JSON output.

24|6|Updated Dec 19, 2025
Search Tags:#data extraction#security analysis#web scraping#link extraction#crawling
One-click install
npx skills add https://github.com/johnlindquist/claude --skill spider
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: spider
Source: https://github.com/johnlindquist/claude/tree/main/skills/spider
Command: npx skills add https://github.com/johnlindquist/claude --skill spider

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Web crawling and scraping with analysis. Use for crawling websites, security scanning, and extracting information from web pages.

Core Features & Use Cases

  • Fetch pages with curl and inspect headers
  • Extract links and domain-filtered URLs
  • Quick site scan for headers, scripts, and meta tags
  • AI-powered analysis for page content and security insights
  • Simple extraction of structured data from pages

Quick Start

Use the spider skill to fetch a page, extract its links, and perform a quick meta-tag scan on the target URL.

Frequently Asked Questions about spider

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I crawl a website and extract links?▼

Web crawling fetches pages using HTTP requests and extracts all links found on them. The spider skill crawls websites by fetching pages with configurable headers, parsing HTML to identify links, and filtering results by domain, returning structured data for further analysis or pipeline processing.

Can I use web scraping for security scanning?▼

Yes, security scanning via web scraping analyzes page headers, scripts, and metadata to identify potential vulnerabilities. The spider skill extracts scripts, meta tags, and headers from websites, enabling AI-powered security insights and endpoint discovery across single-page and multi-page sites.

What data can I extract from web pages?▼

Web page extraction retrieves text content, metadata, structured data, and HTML elements. The spider skill parses pages to extract text, meta tags, structured data, and scripts, outputting JSON-compatible results suitable for RAG pipelines, content analysis, and downstream processing.

How do I crawl multi-page websites?▼

Multi-page crawling systematically traverses linked pages across a site. The spider skill implements multi-page crawling with sitemap processing, link extraction, and domain filtering to map entire websites and extract data consistently across all discovered pages.

What's the best way to extract structured data from websites?▼

Structured data extraction parses HTML and metadata into machine-readable formats. The spider skill extracts structured data, metadata, and content from pages, formatting results as JSON compatible with downstream analysis tools and RAG pipelines for automation.

Related Skills

Explore More Data & Analytics Skills→
TwuanMinnTwuanMinn

tavily-web

Automate web research and content extraction using the Tavily API.

Community
Advanced
shynlee04shynlee04

web-search

Search the web and extract content via Tavily and Exa using the inference.sh CLI.

Community
Intermediate
Mark393295827Mark393295827
1

regex-vs-llm-structured-text

Parse structured text with regex extraction and selective LLM validation.

Community
Intermediate
gongzeqgongzeq

katana-crawl-web

Crawls authorized web targets to generate deduplicated candidate URLs for vulnerability scanning.

Community
Intermediate