skills.rest
Uzay-G
Uzay-GCOMMUNITY
Explore all skills by this author →
Website𝕏@uzpg_GitHub

website-crawler

Crawl websites and ingest content into a Whorl knowledge base as Markdown.

2|Updated Dec 22, 2025
Search Tags:#markdown#website#trafilatura#web content#crawler#whorl
One-click install
npx skills add https://github.com/Uzay-G/whorl --skill website-crawler
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: website-crawler
Source: https://github.com/Uzay-G/whorl/tree/main/skills/website-crawler
Command: npx skills add https://github.com/Uzay-G/whorl --skill website-crawler

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill crawls websites and ingests content into your Whorl knowledge base, helping you build a centralized, searchable archive of your web content without manual copying.

Core Features & Use Cases

  • Web crawling & ingestion: Crawl individual pages or entire sites and store content in Whorl as Markdown with optional frontmatter.
  • Content extraction & formatting: Extract titles and metadata to enrich your docs for quick reference.
  • Use Case: Archive your personal blog or collect web sources for a research knowledge base to reference later.

Quick Start

Use the website-crawler to crawl https://example.com and save the results to your Whorl docs, then run whorl sync to process the new content.