skills.rest
MadAppGang
MadAppGangOFFICIAL
Explore all skills by this author →
WebsiteGitHub

web-scraping

Extract structured data from web pages with pagination and session handling.

7|4|Updated Feb 26, 2026
Search Tags:#data extraction#authentication#rate limiting#web scraping#spa#pagination
One-click install
npx skills add https://github.com/MadAppGang/magus --skill web-scraping-madappgang
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-scraping
Source: https://github.com/MadAppGang/magus/tree/main/plugins/browser-use/skills/web-scraping
Command: npx skills add https://github.com/MadAppGang/magus --skill web-scraping-madappgang

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill automates the process of extracting specific data from web pages, overcoming challenges like dynamic content, pagination, and login requirements.

Core Features & Use Cases

  • Data Extraction: Pulls structured information (product details, prices, articles) from web pages.
  • Pagination Handling: Navigates through multiple pages using next buttons or numbered links.
  • Dynamic Content: Works with Single Page Applications (SPAs) that load content via JavaScript.
  • Authenticated Scraping: Handles sites requiring login by managing sessions.
  • Rate Limiting & Anti-Bot: Implements delays and strategies to avoid detection.
  • Use Case: Automatically collect all product names and prices from an e-commerce site, even if the data loads dynamically or requires scrolling.

Quick Start

Use the web-scraping skill to extract product names and prices from the URL 'https://shop.example.com/products'.

Frequently Asked Questions about web-scraping

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract structured data from a single page application that loads content dynamically?▼

To extract structured data from a single page application, you need a web scraping approach that renders or captures JavaScript-loaded content. This skill handles dynamic SPAs to pull product details and article content as it loads.

Can I scrape data from websites that require user login and authentication?▼

Yes, you can scrape data from websites requiring authentication. This skill manages authenticated sessions to access and extract protected content like user-specific product listings or gated articles.

What is the best way to handle pagination when collecting product listings from an e-commerce site?▼

The best way to handle pagination during web scraping is to automate navigation through next buttons or numbered links. This skill extracts structured data across multiple pages to gather complete product lists.

How do I avoid bot detection and rate limiting when extracting data from web pages?▼

To avoid bot detection during web scraping, you must implement rate limiting and anti-bot bypass strategies. This skill enforces request delays and manages sessions to prevent blocking while extracting data.

Does web scraping work for collecting article content and tabular data from list-based web pages?▼

Web scraping works effectively for collecting article content and tabular data. This skill automates structured data extraction from list-based pages, handling dynamic loads, logins, and pagination.

Why does my data extraction fail on sites with anti-bot measures and how can I fix it?▼

Data extraction fails on sites with anti-bot measures when scraping lacks request delays or session management. This skill implements rate limiting and robust strategies to bypass detection and ensure successful extraction.

Related Skills

Explore More Software Engineering Skills→
LuminairPrimeLuminairPrime

mcp-firecrawl

Scrape web pages and extract structured content using Firecrawl.

Community
Advanced
Pans0020Pans0020

markdown-exporter

Convert Markdown content into DOCX, PPTX, XLSX, PDF, HTML, IPYNB, MD, CSV, JSON/JSONL, XML, and extracted code files.

Community
Advanced
ajianazajianaz
4

browser

Automate Chrome browser control for site exploration and scraping via CDP.

Community
Advanced
udecodeudecode
434

convex-http

Automate Hono-based routing, streaming, and rate limiting for HTTP services.

Official
Advanced