skills.rest
VenTheZone
VenTheZoneCOMMUNITY
Explore all skills by this author →
GitHub

using-web-scraping

Search DuckDuckGo and scrape page data with Playwright.

1|1|Updated Feb 21, 2026
Search Tags:#automation#data extraction#web scraping#playwright#content retrieval#duckduckgo
One-click install
npx skills add https://github.com/VenTheZone/favorite-opencode-setup --skill using-web-scraping-venthezone
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: using-web-scraping
Source: https://github.com/VenTheZone/favorite-opencode-setup/tree/main/skills/using-web-scraping
Command: npx skills add https://github.com/VenTheZone/favorite-opencode-setup --skill using-web-scraping-venthezone

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill automates the process of gathering information from public web pages, making it easier to collect data for analysis or summarization without manual browsing.

Core Features & Use Cases

  • Privacy-Focused Search: Utilizes DuckDuckGo for search queries, respecting user privacy.
  • Content Extraction: Extracts key information like title, meta description, main text, and canonical URL from visited web pages.
  • Safe Scraping Practices: Adheres to robots.txt, implements rate-limiting, and uses polite User-Agent strings.
  • Use Case: Gather product details from multiple e-commerce sites for a price comparison report.

Quick Start

Use the web scraping skill to search for 'best open-source AI tools' and extract the title, description, and main text from the top three results.

Frequently Asked Questions about using-web-scraping

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scrape web content safely without violating robots.txt rules?▼

To scrape web content safely, this Skill queries DuckDuckGo and extracts page data using headless Chrome via Playwright. It enforces safe scraping practices by adhering to robots.txt, applying rate limiting, and sending polite User-Agent strings during automated data extraction.

How do I extract titles and meta descriptions from search results using Playwright?▼

You can extract titles, meta descriptions, main text, and canonical URLs by running headless Chrome via Playwright. The Skill navigates to pages found through DuckDuckGo search queries and retrieves structured content from the visited web pages.

Does this web scraping approach respect user privacy during data extraction?▼

Yes, the web scraping approach respects user privacy by utilizing DuckDuckGo for search queries. This ensures that your automated information retrieval and content aggregation tasks do not track search behavior through mainstream search engines.

What is the best way to automate market research data extraction from public web pages?▼

The best way to automate market research data extraction is using a Skill that combines DuckDuckGo search with Playwright headless Chrome. It retrieves structured data like product details from multiple e-commerce sites while maintaining rate limiting and robots.txt compliance.

Can I retrieve content from dynamic web pages using headless Chrome and Playwright?▼

Yes, you can retrieve content from dynamic web pages because the Skill uses headless Chrome via Playwright. This allows it to render JavaScript-driven content and extract the main text, title, meta description, and canonical URL from target pages.

Related Skills

Explore More Content & Communication Skills→
khapn-2933khapn-2933

tkm:automate-browser

Automate browser navigation, interaction, and data extraction with Puppeteer.

Community
Advanced
amychina12amychina12

paper-standardizer

Standardize raw academic paper data into analysis-ready datasets with Python codebooks.

Community
Advanced
kivo360kivo360
1

moai-essentials-refactor

Transform domain knowledge into vector embeddings for semantic retrieval.

Community
Intermediate
sky770825sky770825

web-fetch

Fetch web content via HTTP requests and convert it to Markdown or plain text.

Community
Basic