skills.rest
valtterimelkko
valtterimelkkoCOMMUNITY
Explore all skills by this author →
GitHub

web-crawling

Extract text content and internal links from websites via depth-bounded crawling.

1|1|Updated Jun 11, 2026
Search Tags:#automation#scraping#web-crawling#content-extraction#browserless#data-acquisition
One-click install
npx skills add https://github.com/valtterimelkko/agent-workflow-skills --skill web-crawling-valtterimelkko
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-crawling
Source: https://github.com/valtterimelkko/agent-workflow-skills/tree/main/skills/web-crawling
Command: npx skills add https://github.com/valtterimelkko/agent-workflow-skills --skill web-crawling-valtterimelkko

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires requests, beautifulsoup4, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This skill solves the challenge of gathering content from multiple web pages in a structured, bounded manner, preventing the agent from getting lost in infinite loops or overwhelming the chat context.

Core Features & Use Cases

  • Multi-page crawling: Automatically follows internal links to gather content across a site section.
  • Escalation paths: Provides built-in logic to switch from native fetching to advanced tools like Browserless or residential proxies when encountering bot protection.
  • Use Case: Use this when you need to archive documentation from a site, perform a deep content audit, or gather data from a multi-page report that is not easily accessible via a single fetch.

Quick Start

Use the web-crawling skill to crawl the documentation section of the target website up to a depth of two pages and save the output to a file.

Frequently Asked Questions about web-crawling

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract text content from multiple web pages without getting stuck in infinite loops?▼

Multi-page web crawling extracts text content through a structured, depth-bounded process that limits how deep the crawler follows internal links. This prevents infinite loops and keeps the gathered data within manageable chat context limits.

What's the best way to scrape a documentation site that has bot detection and JavaScript rendering?▼

Web scraping with escalation paths switches from native HTTP fetching to browser automation and residential proxies when encountering bot protection. This handles JavaScript-heavy rendering and sophisticated blocks that standard requests fail to bypass.

Can I use BeautifulSoup for web crawling across multiple linked pages?▼

Yes, BeautifulSoup is integrated for parsing fetched pages during web crawling. It works alongside the structured crawling logic to extract text and internal links, supporting bounded traversal across multiple linked pages.

How does depth-bounded web crawling work for content aggregation?▼

Depth-bounded web crawling limits the traversal depth of internal links, extracting text content page by page up to a specified level. This structured approach aggregates content from a site section without overwhelming the context.

When should I use proxy escalation for web scraping instead of native HTTP requests?▼

Proxy escalation is needed when native HTTP requests fail due to bot detection. The crawling logic automatically escalates to advanced browser automation or residential proxies to reliably acquire data from protected sites.

Does this web scraping approach work for archiving documentation from an entire site section?▼

Yes, multi-page web crawling is designed for archiving documentation and performing deep content audits. It follows internal links to gather data across a site section up to a bounded depth and saves the output to a file.

Related Skills

Explore More Data & Analytics Skills→
orthogonal-shorthogonal-sh
12

olostep

Scrape and crawl web pages into structured data via configurable endpoints.

Official
Advanced
shynlee04shynlee04

web-search

Search the web and extract content via Tavily and Exa using the inference.sh CLI.

Community
Intermediate
valtterimelkkovaltterimelkko
1

youtube-caption

Fetch YouTube transcripts and metadata using the youtube-transcript-api with residential proxy support.

Community
Intermediate
joyce820309joyce820309

scrape

Automate HTML data extraction using BeautifulSoup and requests.

Community
Intermediate