skills.rest
aws-samples
aws-samplesOFFICIAL
Explore all skills by this author →
WebsiteGitHub

url-fetcher

Fetch web page URLs and extract cleaned text content as structured JSON.

186|56|Updated Aug 13, 2025
Search Tags:#html#url#web#web-scraping#text-extraction#content-extraction
One-click install
npx skills add https://github.com/aws-samples/sample-strands-agent-with-agentcore --skill url-fetcher
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: url-fetcher
Source: https://github.com/aws-samples/sample-strands-agent-with-agentcore/tree/main/chatbot-app/agentcore/skills/url-fetcher
Command: npx skills add https://github.com/aws-samples/sample-strands-agent-with-agentcore --skill url-fetcher

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill automates the tedious process of retrieving and cleaning text from arbitrary web pages so you can quickly read, summarize, analyze, or cite online content without manually stripping navigation, scripts, and boilerplate HTML.

Core Features & Use Cases

  • Fetch and Extract Clean Text: Retrieves a URL and returns the page title and cleaned textual content suitable for NLP tasks and summaries.
  • Flexible Output Options: Optionally include raw HTML, and limit the returned text size with max_length for fast previews or full extracts.
  • Robust Error Feedback: Times out after 30 seconds and returns structured JSON with a success flag and error details for unreachable or non-200 responses.
  • Use Case: Ideal for researchers, content teams, or agents that need canonical text for citation, summarization, or downstream analysis of articles, docs, or postings.

Quick Start

Use the url-fetcher to fetch and extract the main text and title from https://example.com/article and return cleaned content with citations when relevant.

Frequently Asked Questions about url-fetcher

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract clean text from a web page URL for analysis?▼

To extract clean text from a web page, fetch the URL to strip boilerplate HTML, scripts, and navigation. This returns the page title and structured JSON with cleaned textual content suitable for downstream summarization and NLP tasks.

Can I get raw HTML instead of plain text when scraping a URL?▼

Yes, you can get raw HTML when scraping a URL by using the include_html option. This allows you to retrieve the original HTML alongside the cleaned text, providing flexibility for both reading and downstream web scraping tasks.

What is the best way to limit text extraction size for fast web page previews?▼

The best way to limit text extraction size for fast web page previews is to use the max_length option. This constrains the returned text size, delivering quick content previews instead of full article extracts.

Does web content extraction work with non-http protocols or unreachable URLs?▼

Web content extraction does not work with non-http protocols, as it strictly enforces http and https URLs. For unreachable URLs, it times out after 30 seconds and returns structured JSON with a success flag and error details.

Why does my URL text extraction return an error instead of article content?▼

Your URL text extraction returns an error if the target web page provides a non-200 response or is unreachable. The system enforces a 30-second timeout and returns structured JSON detailing the failure rather than article content.

Related Skills

Explore More Content & Communication Skills→
gerald-icagerald-ica

Network 101

Configure and test HTTP, HTTPS, SNMP, and SMB network services.

Community
Intermediate
yangliu2060yangliu2060
39

youtube-video-analyzer

Extract YouTube transcripts and captions into structured summaries with key points.

Community
Basic
wyhlovecppwyhlovecpp

defuddle

Extract readable web page content as markdown or JSON using defuddle-cli.

Community
Intermediate
otrebuotrebu
4

web-to-markdown

Batch-convert JavaScript-rendered web pages to a timestamped Markdown file.

Community
Advanced