skills.rest
zhongjis
zhongjisCOMMUNITY
Explore all skills by this author →
GitHub

scraping

Fetch web pages, parse HTML with CSS selectors, and query REST APIs.

Updated Feb 21, 2024
Search Tags:#data extraction#api#web scraping#web automation#dynamic content#html parsing
One-click install
npx skills add https://github.com/zhongjis/nix-config --skill scraping-zhongjis
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: scraping
Source: https://github.com/zhongjis/nix-config/tree/main/modules/home-manager/features/ai-tools/common/skills/general/scraping
Command: npx skills add https://github.com/zhongjis/nix-config --skill scraping-zhongjis

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill automates the process of extracting data from websites, querying APIs, and handling dynamic web content, saving you time and effort in data collection.

Core Features & Use Cases

  • Web Page Fetching: Retrieve HTML content from specified URLs.
  • HTML Parsing: Extract specific data using CSS selectors.
  • Dynamic Content Handling: Interact with JavaScript-heavy websites using browser automation.
  • API Interaction: Query REST APIs and parse JSON responses.
  • Use Case: Automatically scrape product prices and availability from an e-commerce website or fetch daily stock data from a financial API.

Quick Start

Use the scraping skill to extract all h1 titles from https://example.com.

Frequently Asked Questions about scraping

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract data from a website using CSS selectors?▼

To extract data from a website, this Skill fetches web pages and parses HTML content using CSS selectors. It retrieves specific elements like titles or product details directly from the provided URLs.

Can I scrape dynamic content from JavaScript-heavy websites?▼

Yes, you can scrape dynamic content from JavaScript-heavy websites. This Skill handles dynamic content by utilizing browser automation to interact with and extract data from rendered pages.

Do I need nu-shell to query REST APIs and parse JSON responses?▼

Yes, you need nu-shell to query REST APIs and parse JSON responses. The Skill relies on nu-shell, the 'query web' plugin, and a browser extension to automate API interaction and data extraction.

What is the best way to automate fetching product prices from an e-commerce site?▼

The best way to automate fetching product prices is using this Skill's web page fetching and HTML parsing capabilities. It automates extracting specific data like prices and availability from e-commerce websites.

Does web scraping require a browser extension for dynamic web content?▼

Yes, web scraping dynamic web content requires a browser extension. Along with nu-shell and the 'query web' plugin, the extension is necessary to automate interactions with JavaScript-heavy websites.

How do I query JSON APIs to fetch daily stock data?▼

To query JSON APIs for daily stock data, this Skill calls REST APIs and parses the JSON responses. It automates the process of querying financial APIs to retrieve and structure the extracted data.

Related Skills

Explore More Data & Analytics Skills→
knoopxknoopx
68

scraping

Fetch web pages, parse HTML with CSS selectors, and query REST APIs.

Community
Intermediate
delta-and-betadelta-and-beta

analyst-data-export

Export Braze data from dashboard CSVs and Export API responses into analyzable datasets.

Community
Advanced
CrumbgrabberCrumbgrabber
3

webapp-testing

Automate local web application testing with Python Playwright scripts.

Community
Intermediate
appautomatonappautomaton
18

html-to-markdown

Converts HTML or live web pages into structured Markdown with metadata and link inventories.

Official
Intermediate