web-scraping

Name: web-scraping
Availability: InStock
Author: yannsenecheau

Scrape and extract web page content using Python requests and BeautifulSoup.

Updated Aug 23, 2026

Search Tags:#automation #data extraction #python #web scraping #requests #beautifulsoup #html parsing

Ready to run

npx skills add https://github.com/yannsenecheau/skills-cursor --skill web-scraping-yannsenecheau

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: web-scraping
Source: https://github.com/yannsenecheau/skills-cursor/tree/main/web-scraping
Command: npx skills add https://github.com/yannsenecheau/skills-cursor --skill web-scraping-yannsenecheau

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill automates the process of extracting information from websites, saving users time and effort compared to manual browsing and data collection.

Core Features & Use Cases

Web Content Extraction: Fetches and parses HTML content from specified URLs.
Data Scraping: Extracts specific data points using CSS selectors or fallback patterns.
Robustness: Includes retry logic, rate limiting, and caching for reliable scraping.
Use Case: Extract all product names and prices from an e-commerce category page.

Quick Start

Use the web-scraping skill to extract the main content from the URL 'https://example.com'.

Frequently Asked Questions about web-scraping

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I extract content from web pages using Python without getting blocked?▼

To extract web content reliably, use Python scripts that send HTTP requests with custom headers, implement retry mechanisms for transient errors, and enforce rate limiting. This approach automates data scraping while mitigating block risks.

What is the best way to parse HTML and remove noise using BeautifulSoup?▼

The best way to parse HTML is using BeautifulSoup with the lxml parser, applying targeted strategies for content selection, noise removal, and text/link extraction to isolate the specific data points needed from the page structure.

How do I scrape product names and prices from an e-commerce page?▼

You can scrape product names and prices by fetching the category page HTML and extracting specific data points using CSS selectors or fallback patterns to isolate the desired e-commerce elements.

Does this web scraping approach handle transient HTTP errors?▼

Yes, this web scraping approach handles transient HTTP errors by implementing automated retry mechanisms, enforcing rate limiting between requests, and caching responses to optimize performance and reliability.

Can I use custom headers for HTTP requests when scraping websites?▼

Yes, you can use custom headers for HTTP requests when scraping websites. The extraction scripts configure custom headers to fetch HTML content and parse the desired data points effectively.