skills.rest
Lingtai-AI
Lingtai-AIOFFICIAL
Explore all skills by this author →
GitHub

web-browsing

Fetch and extract web content using a seven-tier extraction pipeline.

621|58|Updated Mar 15, 2026
Search Tags:#automation#playwright#scraping#trafilatura#extraction#web-browsing#tiered-pipeline
One-click install
npx skills add https://github.com/Lingtai-AI/lingtai --skill web-browsing-lingtai-ai
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-browsing
Source: https://github.com/Lingtai-AI/lingtai/tree/main/tui/internal/preset/skills/web-browsing
Command: npx skills add https://github.com/Lingtai-AI/lingtai --skill web-browsing-lingtai-ai

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires requests, beautifulsoup4, lxml, pymupdf, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

Fetch and extract content from the web using a seven-tier extraction pipeline.

Core Features & Use Cases

  • Multi-tier extraction pipeline (0-5 + 1.5) with deterministic fallbacks for PDFs, APIs, HTML, and JS-rendered pages.
  • Optional repository resources (scripts/, references/, assets/) that enrich site-specific extraction and data templates.
  • Real-time data and deep-dives via reference guides and asset catalogs for rapid agent reasoning.

Quick Start

Provide a URL or query and the system will fetch and extract content using the built-in auto-tier pipeline.

Frequently Asked Questions about web-browsing

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract content from web pages and PDFs using Python?▼

To extract web content and PDFs, this Skill uses a seven-tier extraction pipeline with requests, BeautifulSoup, lxml, and PyMuPDF. It handles APIs, static HTML, JS-rendered pages, and PDFs through deterministic fallbacks.

What is the best way to scrape JS-rendered pages and bypass paywalls?▼

The best way to scrape JS-rendered pages and handle paywalls is via the auto-tier extraction pipeline. It uses targeted tiers and fallback mechanisms to fetch real-time data endpoints and restricted content.

Does this web scraping pipeline support API endpoints and real-time data extraction?▼

Yes, this web scraping pipeline supports API endpoints and real-time data extraction. It includes special handling for real-time data endpoints within its multi-tier extraction architecture.

How do I parse PDF documents for text extraction in an automated workflow?▼

You can parse PDF documents for text extraction in an automated workflow using the bundled PyMuPDF library. The extract_page.py orchestrator automatically targets PDFs with fallback support.

What are the limitations of using BeautifulSoup and lxml for web scraping?▼

BeautifulSoup and lxml are limited to static HTML parsing. The extraction pipeline overcomes this by escalating to higher tiers for JS-rendered pages, APIs, and paywalled content when static extraction fails.

Related Skills

Explore More Data & Analytics Skills→
Gimpy666Gimpy666
1

scraped_docs

Convert Tribot JavaDoc HTML pages into per-class Markdown files and a consolidated reference.

Community
Intermediate
aws-samplesaws-samples
90

file-processing

Process and analyze CSV, JSON, and text data files with automated cleaning and transformation.

Official
Advanced
CrumbgrabberCrumbgrabber
3

webapp-testing

Automate local web application testing with Python Playwright scripts.

Community
Intermediate
0xBigBoss0xBigBoss
53

web-fetch

Fetches web pages and converts main content into clean markdown using CSS selectors.

Community
Intermediate