skills.rest
agentbay-ai
agentbay-aiOFFICIAL
Explore all skills by this author →
GitHub

web-scraper

Fetch web pages and save them as HTML or Markdown with optional images.

47|4|Updated Feb 6, 2026
Search Tags:#markdown#html#offline#images#requests#web-scraping#beautifulsoup
One-click install
npx skills add https://github.com/agentbay-ai/agentbay-skills --skill web-scraper-agentbay-ai
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-scraper
Source: https://github.com/agentbay-ai/agentbay-skills/tree/main/web-scraper
Command: npx skills add https://github.com/agentbay-ai/agentbay-skills --skill web-scraper-agentbay-ai

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires requests, beautifulsoup4, and includes scripts (resource) components.

What problem does it solve?

Automates saving and archiving web content locally by fetching pages and downloading embedded images for offline access and data collection.

Core Features & Use Cases

  • Single-page scraping to HTML or Markdown with optional image downloads.
  • Recursive crawling within the same domain to build offline archives and datasets.
  • Simple conversion to Markdown with embedded image references for documentation or research.

Quick Start

Provide a URL and desired output format, and the tool will fetch the page and save it locally.

Frequently Asked Questions about web-scraper

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scrape a web page to Markdown with images for offline access?▼

You can scrape a web page to Markdown with images by fetching the content and saving it locally. This tool downloads embedded images and stores outputs in a structured directory for offline access.

Can I recursively crawl pages within the same domain using BeautifulSoup?▼

Yes, you can recursively crawl pages within the same domain using BeautifulSoup. The tool supports recursive scraping to build offline archives and datasets while respecting robots.txt by default.

Does requests support saving HTML content locally with embedded images?▼

Yes, requests supports fetching web page content to save HTML locally. The tool identifies and fetches pages, optionally downloading embedded images and storing them in a structured directory.

What is the best way to archive web content locally as HTML or Markdown?▼

The best way to archive web content locally as HTML or Markdown is automating the fetch process. This tool fetches pages and downloads embedded images, creating structured directories for data collection.

Does web scraping with requests respect robots.txt by default?▼

Yes, web scraping with requests respects robots.txt by default. The tool fetches web page content and saves it locally in HTML or Markdown format while adhering to standard crawling restrictions.

Related Skills

Explore More Data & Analytics Skills→
olostepolostep

map

Discover and list all URLs on a website with keyword and pattern filtering.

Community
Intermediate
42atom42atom

twitter-media

Extract tweet text, author, and media from Twitter/X URLs via fxTwitter API.

Community
Intermediate
dalehurleydalehurley
2

image-tools

Resize, compress, convert, crop, and inspect images using Pillow.

Community
Intermediate
heysuhasheysuhas
2

scrapling

Scrape static, dynamic, and protected websites using the scrapling Python framework.

Community
Advanced