crawl-cli

Crawl websites and extract content into Markdown, JSON, or text.

1|Updated Dec 22, 2025

Search Tags:#scraping #web #crawl

Ready to run

npx skills add https://github.com/abdullah1854/ClaudeSuperSkills --skill crawl-cli

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: crawl-cli
Source: https://github.com/abdullah1854/ClaudeSuperSkills/tree/main/crawl-cli
Command: npx skills add https://github.com/abdullah1854/ClaudeSuperSkills --skill crawl-cli

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

Provides a CLI to crawl websites and output clean Markdown or JSON.

Core Features & Use Cases

Single Page & Pagination: Extract content from single pages and traverse sites.
Output Formats: Markdown, JSON, or plain text.

Quick Start

Crawl a documentation page and convert to Markdown.

Frequently Asked Questions about crawl-cli

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I crawl a website and convert it to Markdown?▼

Web crawling extracts content from websites and converts it to Markdown format. This Skill handles single pages and multi-page traversal, respecting robots.txt, parsing URLs, and outputting clean Markdown with metadata like title, source URL, and timestamp.

Can I crawl multiple pages on a site with pagination?▼

Yes, this Skill supports multi-page traversal with configurable depth. It crawls paginated sites, extracts content across multiple pages, and outputs structured results with pagination support in Markdown, JSON, or text format.

What output formats does web crawling support?▼

Web crawling with this Skill produces three output formats: Markdown, JSON, or plain text. Each format includes structured metadata like title, source URL, timestamp, and extracted links for documentation sites, blogs, and knowledge bases.

Can I use web crawling on documentation sites and knowledge bases?▼

Yes, this Skill is designed for documentation sites, blogs, and knowledge bases. It extracts clean content, handles both single-page and multi-page structures, and respects robots.txt to ensure compliant crawling.

Does web crawling work with single pages and multi-page sites?▼

Web crawling handles both single pages and multi-page traversal. You can configure crawl depth, extract content from individual pages, or traverse paginated structures to produce structured Markdown or JSON output with all discovered links.

What happens to links and metadata when crawling a website?▼

Web crawling extracts and preserves links, titles, source URLs, and timestamps in the output. Metadata is included in structured results, enabling you to reference the crawl source and navigate discovered pages programmatically.