skills.rest
blueraai
blueraaiOFFICIAL
Explore all skills by this author →
GitHub

crawl

Crawl web pages with natural language instructions for link selection and content extraction.

Updated Aug 27, 2026
Search Tags:#automation#data extraction#web scraping#web crawling#natural language#content retrieval
One-click install
npx skills add https://github.com/blueraai/bluera-knowledge --skill crawl-blueraai
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: crawl
Source: https://github.com/blueraai/bluera-knowledge/tree/main/skills/crawl
Command: npx skills add https://github.com/blueraai/bluera-knowledge --skill crawl-blueraai

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

This Skill automates the process of crawling web pages and extracting specific information, making it easy to gather data from websites without manual browsing and copying.

Core Features & Use Cases

  • Intelligent Crawling: Uses natural language instructions to guide the crawling process, focusing on relevant pages.
  • Content Extraction: Extracts specific data points or sections from crawled pages based on natural language instructions.
  • Fast Mode: Offers a quicker crawling option for sites that don't rely heavily on JavaScript rendering.
  • Use Case: Gather all pricing information from a competitor's website or collect all API documentation links from a developer portal.

Quick Start

Crawl the documentation site at https://code.claude.com/docs/en/ and extract all 'Getting Started' pages into a store named 'claude-docs'.

Frequently Asked Questions about crawl

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract specific content from web pages using natural language instructions?▼

Web content extraction with natural language instructions allows you to target specific data points on a page. This Skill intelligently selects links and extracts relevant sections based on your text prompts, automating data retrieval without manual copying.

What is the best way to crawl websites that require JavaScript rendering?▼

Crawling JavaScript-heavy websites is handled using a headless browser mode. This renders dynamic content fully before extraction, ensuring you acquire complete data from modern web applications that rely on client-side scripts.

Can I use a fast mode for web scraping static sites that don't rely on JavaScript?▼

Fast mode uses an axios-only approach for web scraping static sites. It bypasses heavy browser rendering to acquire data quickly, making it ideal for retrieving content from pages that do not require JavaScript execution.

How do I automate gathering all API documentation links from a developer portal?▼

Automating the gathering of API documentation links is done by providing a starting URL and natural language instructions. The crawler intelligently navigates the developer portal, selecting relevant pages and extracting the targeted links.

Does this web crawling tool support populating a knowledge base with extracted data?▼

Web crawling for knowledge base population is a core supported use case. You can extract targeted data from specified websites and store it directly into a named store for analysis and retrieval.

Related Skills

Explore More Software Engineering Skills→
workromancerworkromancer

web-load2

Fetch web URLs and extract all text content.

Community
Basic
ccfccf

vector-index-tuning

Tune HNSW parameters and quantization for large-scale vector search indexes.

Community
Advanced
delta-and-betadelta-and-beta

engineer-platform-web

Integrate the Braze Web SDK in Smart TVs, browser extensions, and strict CSP contexts.

Community
Intermediate
TIAN0517TIAN0517

web-search

Return structured web search results with URLs, titles, and metadata.

Community
Basic