skills.rest
ricable
ricableCOMMUNITY
Explore all skills by this author →
GitHub

crawl

Crawl websites and save pages as local markdown files.

Updated Feb 15, 2026
Search Tags:#documentation#markdown#knowledge base#web crawling#crawl#tavily#oath
One-click install
npx skills add https://github.com/ricable/mcai --skill crawl-ricable
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: crawl
Source: https://github.com/ricable/mcai/tree/main/.agents/skills/crawl
Command: npx skills add https://github.com/ricable/mcai --skill crawl-ricable

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires jq, curl, mcp-remote, and includes scripts (resource) components.

What problem does it solve?

This Skill removes the manual work of visiting pages one by one when you need to archive or analyze website content, documentation, or knowledge bases offline.

Core Features & Use Cases

  • Website Crawling: Collect pages from a target site and convert them into local markdown files.
  • Focused Extraction: Use depth limits, page caps, path filters, and natural-language instructions to narrow the crawl to the most relevant content.
  • Practical Use Cases: Capture a product documentation site for offline review, mirror a knowledge base for research, or save selected articles from a public website for later analysis.

Quick Start

Use the crawl skill to collect pages from the target website and save them as markdown files in an output folder.

Frequently Asked Questions about crawl

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I crawl a website and save pages as local markdown files?▼

You can crawl a website into markdown by using a tool that fetches web content and converts it into local markdown files for offline analysis. This process collects pages from a target site and stores them as markdown in an output folder.

Do I need a Tavily API key to crawl documentation sites offline?▼

Yes, Tavily crawl API access is required to crawl documentation sites and save them as markdown files. The tool relies on this external API to fetch and extract website content into a local knowledge base.

Can I use path filters to narrow down a web crawl to specific content?▼

Yes, you can use path filters, depth limits, page caps, and natural-language instructions to narrow a web crawl. This focused extraction targets only the most relevant content from documentation sites or knowledge bases.

What is the best way to archive a knowledge base for offline research?▼

The best way to archive a knowledge base for offline research is to crawl the website and convert its pages into local markdown files. This allows you to mirror the content for later analysis without requiring an internet connection.

Does web crawling into markdown work with JSON input handling?▼

Yes, web crawling into markdown supports JSON input handling to configure the crawl parameters. This allows you to programmatically define target URLs, path filters, and page caps for extracting documentation content.

Related Skills

Explore More Education & Research Skills→
antgroupantgroup
16

firecrawl

Access Firecrawl API for web search, scraping, and site crawling.

Official
Intermediate
venhdevvenhdev

url-to-markdown

Convert public web page URLs into clean Markdown files.

Community
Intermediate
RheasilviaRheasilvia

here.now

Publish static web content and manage private cloud file storage via API.

Community
Intermediate
limzivectorlimzivector

scrape

Extract text, tables, and images from web pages via browser automation.

Community
Intermediate