Crawl websites to fetch HTML content and extract text, links, and structured data.
npx skills add https://github.com/datalayer/agent-runtimes --skill crawl-datalayer
Please help me install this Agent Skill. Skill: crawl Source: https://github.com/datalayer/agent-runtimes/tree/main/skills/crawl Command: npx skills add https://github.com/datalayer/agent-runtimes --skill crawl-datalayer
Web developers and researchers waste time manually visiting pages to collect text, links, and structured data from HTML. This Skill automates web data collection by fetching content and extracting meaningful data.
Crawl a page and extract data: python scripts/fetch_page.py https://example.com Crawl a site with depth limit: python scripts/crawl_site.py https://example.com --max-pages 5 --max-depth 2
High-intent search queries and answers about installing and using this skill.
To extract text and links from HTML pages, this Skill fetches content using httpx and parses HTML with BeautifulSoup to retrieve page titles, metadata, and structured table data automatically.
Crawling a website with depth control involves following internal links up to a specified limit to build datasets across pages. You can configure max-pages and max-depth parameters to scope site-wide data aggregation.
Web scraping JavaScript-rendered pages is supported through optional Playwright integration, allowing you to fetch dynamically loaded HTML content that standard static HTTP requests cannot retrieve.
You can monitor competitor pages by crawling category pages to aggregate product titles and prices for analysis. This automates web data collection across multiple pages for ongoing content monitoring.
To start web crawling with httpx and BeautifulSoup, you need Python tooling and the specified libraries installed. The Skill provides scripts to fetch single pages or crawl entire sites with configurable depth.
You should not use httpx for web scraping when target pages rely heavily on JavaScript to render content. In these cases, switching to Playwright is necessary to execute scripts and fetch the fully rendered HTML.
felipemlanna1Run a Python Playwright script to scrape and filter Google Maps data into CSV.
0xBigBossFetches web pages and converts main content into clean markdown using CSS selectors.
Jignesh-PonamwarExtract structured data from websites using Python scraping tools.
dennislee928Scrapes and crawls websites into LLM-ready markdown or JSON.
