skills.rest
LeandroBenjaminL
LeandroBenjaminLCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

web-scraping

Scrape HTML and JavaScript-heavy web pages with requests, BeautifulSoup, Selenium, and Playwright.

Updated May 9, 2026
Search Tags:#data extraction#Playwright#web scraping#Selenium#BeautifulSoup
One-click install
npx skills add https://github.com/LeandroBenjaminL/lend-ai --skill web-scraping-leandrobenjaminl
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-scraping
Source: https://github.com/LeandroBenjaminL/lend-ai/tree/main/skills/web-scraping
Command: npx skills add https://github.com/LeandroBenjaminL/lend-ai --skill web-scraping-leandrobenjaminl

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires requests, beautifulsoup4, selenium, playwright, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the challenge of extracting data from websites without violating terms or damaging the site. It automates data retrieval from HTML sources and navigates JavaScript-heavy pages while respecting robots.txt and rate limiting.

Core Features & Use Cases

  • Data Extraction: Automatically scrape data from multiple web pages.
  • JavaScript Navigation: Handle dynamic content by using Selenium or Playwright.
  • Respectful Scraping: Adhere to robots.txt rules and implement rate limiting to avoid overloading servers.
  • Use Case: When you need to gather information from a website without an API, this Skill is ideal for extracting HTML data or automating web navigation.

Quick Start

Use the web-scraping skill to extract all data from the 'example.com' website.

Frequently Asked Questions about web-scraping

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract data from JavaScript-heavy websites that require dynamic content rendering?▼

Web scraping JavaScript-heavy websites requires browser automation tools like Selenium or Playwright to render dynamic content. These frameworks execute client-side scripts, allowing you to extract fully populated HTML data that static HTTP requests cannot retrieve.

What is the best way to scrape web pages without overloading the target server?▼

Ethical web scraping involves respecting robots.txt directives and implementing rate limiting to control request frequency. This prevents server overload and ensures compliant data extraction without violating website terms.

When do I need to use Playwright or Selenium instead of BeautifulSoup for data extraction?▼

You need Playwright or Selenium instead of BeautifulSoup when extracting data from JavaScript-rendered pages. BeautifulSoup parses static HTML, while browser automation tools execute scripts to render dynamic content before extraction.

How to automate web scraping tasks for multiple web pages using Python?▼

Automate web scraping across multiple pages using Python libraries like requests and BeautifulSoup for static content, or Playwright and Selenium for dynamic pages. Implement rate limiting and respect robots.txt for ethical extraction.

Can I use the requests library to scrape data from a website that requires JavaScript execution?▼

No, the requests library only fetches static HTML and cannot execute JavaScript. For websites requiring JavaScript execution to render content, use Playwright or Selenium to automate browser interactions and extract the dynamically generated data.

Does ethical web scraping require checking robots.txt before extracting data?▼

Yes, ethical web scraping requires checking robots.txt before extracting data to identify restricted areas. Respecting these rules and implementing rate limiting ensures compliant data extraction that adheres to website guidelines.

Related Skills

Explore More Data & Analytics Skills→
EvezArtEvezArt
3

web-scraping

Extract data from static and dynamic web pages using Python scraping tools.

Community
Advanced
Wesley1600Wesley1600

data-analysis

Analyze CSV, JSON, and Parquet datasets with exploratory analysis and visualizations.

Community
Advanced
luis2404123luis2404123

proxy-rotation

Rotate IP addresses per request or sticky session for web scraping.

Community
Intermediate
nimadorostkarnimadorostkar
25

web-data-extraction

Extract structured web data via APIs, HTML parsing, or headless browsers.

Community
Advanced