skills.rest
yfe404
yfe404COMMUNITY
Explore all skills by this author →
GitHub

web-scraping

Automate web-scraping strategy discovery and iterative implementation toward Apify Actors.

80|18|Updated Oct 22, 2025
Search Tags:#automation#API#Playwright#web-scraping#Apify#strategy-discovery
One-click install
npx skills add https://github.com/yfe404/web-scraper --skill web-scraping-yfe404
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-scraping
Source: https://github.com/yfe404/web-scraper/tree/main
Command: npx skills add https://github.com/yfe404/web-scraper --skill web-scraping-yfe404

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill automates the process of choosing and implementing the most effective web scraping strategy for a given site, eliminating guesswork and reducing development time.

Core Features & Use Cases

  • Interactive reconnaissance: Open sites in a real browser to observe rendering, interactions, and blockers before coding.
  • Proactive strategy discovery: Detects sitemaps and APIs early to determine the fastest, most reliable extraction path.
  • Strategic recommendations: Presents 2-3 optimized approaches with estimates of time and complexity, plus trade-offs.
  • Iterative implementation: Starts simple and adds robustness only as needed, guiding toward a production-ready flow.
  • Production-ready guidance: Includes TypeScript Actor deployment patterns and productionization steps for Apify.

Real-world example: Given a site with both sitemap and API, this Skill suggests a hybrid approach (sitemap + API) and outlines the minimal code to implement, then scales to a full Actor.

Quick Start

Start by requesting a target site URL and your desired output; the Skill will respond with an actionable plan, recommended strategy, and a path to a working prototype.

Frequently Asked Questions about web-scraping

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I choose the best web scraping strategy for a target website?▼

Choosing the best web scraping strategy requires probing a site to detect sitemaps or APIs early. This approach analyzes rendering and blockers to recommend 2-3 optimized extraction paths with time and complexity trade-offs.

How do I automate web scraping using Playwright when an API is unavailable?▼

Automating web scraping with Playwright involves opening sites in a real browser to observe interactions and blockers before coding. This enables iterative script implementation starting simple and adding robustness as needed.

How do I deploy a web scraper to Apify Actors for production?▼

Deploying a web scraper to Apify Actors for production requires following TypeScript Actor deployment patterns. This guides phased productionization scaling from a minimal working prototype to a full Actor.

Does this web scraping approach work with sites that have both sitemaps and APIs?▼

Yes, this web scraping approach works with sites having both sitemaps and APIs by recommending a hybrid extraction strategy. It outlines minimal code to implement this combined path for faster, more reliable data extraction.

What is the fastest way to discover an API or sitemap for web scraping?▼

The fastest way to discover an API or sitemap for web scraping is through interactive reconnaissance. This technique probes target sites early to determine the most effective extraction path before writing code.

Related Skills

Explore More Software Engineering Skills→
adryanmoldokkr32-pixeladryanmoldokkr32-pixel

browse

Drive a Playwright-based headless browser to navigate, interact, and verify page states.

Community
Intermediate
rawvegrawveg
40

apify-js-sdk

Build and operate Apify Actors and web scrapers using the JavaScript SDK.

Community
Advanced
vikingokftvikingokft

web-scraping

Automate web data extraction using traffic interception, sitemaps, APIs, and DOM scraping.

Official
Advanced
Snowflake-LabsSnowflake-Labs
12

read-ors-configuration

A concise summary of your roleplay character: name,</name> personality, and/or behavior, formatted as a single string sentence. Typically不超过50字，此处省略

Official
Intermediate