skills.rest
sfc-gh-dflippo
sfc-gh-dflippoCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

doc-scraper

Scrape Snowflake documentation sections and convert them to Markdown with caching.

34|13|Updated Feb 1, 2022
Search Tags:#documentation#markdown#caching#data-extraction#web-scraping#snowflake
One-click install
npx skills add https://github.com/sfc-gh-dflippo/snowflake-dbt-demo --skill doc-scraper
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: doc-scraper
Source: https://github.com/sfc-gh-dflippo/snowflake-dbt-demo/tree/main/.claude/skills/doc-scraper
Command: npx skills add https://github.com/sfc-gh-dflippo/snowflake-dbt-demo --skill doc-scraper

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires requests, beautifulsoup4, lxml, markdownify, pyyaml, python-frontmatter, click, tqdm, ratelimit, and includes scripts (resource) components.

What problem does it solve?

This Skill automates the extraction and organization of Snowflake documentation into structured Markdown with intelligent caching, eliminating manual copying and ensuring content freshness.

Core Features & Use Cases

  • Intelligent Caching: Automatically skips recently scraped pages with 7-day expiration to save time and bandwidth.
  • Configurable Scraping: Target specific documentation sections with base-path filtering and adjustable spider depth.
  • Use Case: Imagine you need the latest Snowflake SQL reference documentation. Use this Skill to scrape the entire SQL reference section, convert it to Markdown with metadata, and generate a comprehensive index for AI access.

Quick Start

Use the doc-scraper skill to extract all migration documentation from docs.snowflake.com into the project's snowflake-docs directory.

Frequently Asked Questions about doc-scraper

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I convert Snowflake documentation to Markdown for AI training?▼

Doc-scraper automates Snowflake documentation conversion to Markdown with intelligent caching and metadata. It scrapes docs.snowflake.com sections, generates offline assets, and produces a structured SKILL.md index ready for AI skill generation without manual copying.

Can I scrape specific sections of Snowflake docs instead of everything?▼

Yes, doc-scraper supports configurable base-path filtering to target specific documentation sections and adjustable spider depth. You control which areas to scrape, reducing extraction time and focusing on relevant content for your use case.

Does doc-scraper cache pages to avoid re-scraping?▼

Doc-scraper includes intelligent SQLite-backed caching with 7-day expiration to automatically skip recently scraped pages. This saves bandwidth and processing time while ensuring content freshness through configurable cache management.

How do I use doc-scraper to build AI-ready documentation datasets?▼

Run doc-scraper with --base-path to target Snowflake doc sections, configure spider depth for link traversal, and specify --output-dir for results. It generates Markdown files, a cache directory, and a SKILL.md index structured for downstream AI model training and skill extraction.

What's the best way to extract and organize large documentation for offline use?▼

Doc-scraper extracts Snowflake documentation at scale with configurable depth, outputs structured Markdown with metadata, and maintains a reproducible cache and configuration. This approach enables offline access, version control, and rapid regeneration of documentation datasets.

Related Skills

Explore More Data & Analytics Skills→
dchukdchuk
3

sm-scraper-adapter

Create and register custom scraper adapters inheriting from SourceMonitor::Scrapers::Base.

Community
Advanced
an8079an8079
4

embedding-generation

Train and optimize embedding models for text, images, and multimodal data.

Community
Advanced
Gimpy666Gimpy666
1

scraped_docs

Convert Tribot JavaDoc HTML pages into per-class Markdown files and a consolidated reference.

Community
Intermediate
Snowflake-LabsSnowflake-Labs
12

read-ors-configuration

A concise summary of your roleplay character: name,</name> personality, and/or behavior, formatted as a single string sentence. Typically不超过50字，此处省略

Official
Intermediate