skills.rest
TylerSimons1127
TylerSimons1127COMMUNITY
Explore all skills by this author →
GitHub

web-search-setup

Configures free web search and URL-to-markdown extraction fallbacks when Firecrawl credits are unavailable.

Updated Aug 21, 2026
Search Tags:#web search#web scraping#trafilatura#Brave API#markdown extraction#jina reader#firecrawl fallback
One-click install
npx skills add https://github.com/TylerSimons1127/vibe --skill web-search-setup-tylersimons1127
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-search-setup
Source: https://github.com/TylerSimons1127/vibe/tree/main/skills/research/web-search-setup
Command: npx skills add https://github.com/TylerSimons1127/vibe --skill web-search-setup-tylersimons1127

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires trafilatura, and includes scripts (resource) components.

What problem does it solve? Hermes' built-in web_search and web_extract tools route through Firecrawl cloud, which fails when paid credits run out. This Skill provides free alternatives for both finding URLs and converting pages to clean markdown, so research workflows never stall. ## Core Features & Use Cases - Free Search via Brave API: Uses the Brave Search API free tier (2,000 queries/month) through the included brave_search.py CLI, avoiding bot-walled anonymous search engine scraping. - Layered Extraction Fallback: The webget wrapper tries trafilatura locally first, then falls back to the Jina Reader API for JS-rendered or Cloudflare-protected pages, with Playwright MCP as a last resort. - Use Case: When web_extract returns "Web tools are not configured" mid-research, run webget on the target URL to get clean markdown instantly, or use brave_search.py to find sources without any paid credits. ## Quick Start Set the BRAVE_API_KEY and JINA_API_KEY environment variables, install trafilatura with uv tool install, then ask the agent to search for a topic and extract the top result as markdown using the free fallback tools.

Frequently Asked Questions about web-search-setup

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I search the web for free without Firecrawl credits?▼

Use the Brave Search API free tier, which offers 2,000 queries per month with clean JSON and no anti-bot blocking. Get a key from brave.com/search/api, set BRAVE_API_KEY, and run the brave_search.py script with your query.

How to extract markdown from a URL without an API key?▼

Install trafilatura locally with uv tool install trafilatura, then pipe any URL through it: curl -sL <url> | trafilatura --output-format markdown. It runs entirely in-process with no rate limits or background services.

Why does anonymous curl to DuckDuckGo or Bing return empty results?▼

Search engines bot-wall non-browser clients, returning 0-byte bodies or CAPTCHA verification pages. The DDG Instant Answer API is also deprecated. Use the Brave Search API or a self-hosted SearXNG instance instead.

Does Jina Reader work without an API key?▼

Anonymous Jina requests are Cloudflare-blocked from some residential IPs, returning HTTP 403 challenges. An authenticated request with a free JINA_API_KEY from jina.ai bypasses this and also handles JS-rendered pages.

What fallback works for JavaScript-heavy sites that trafilatura cannot parse?▼

Use the Jina Reader API with an API key, which renders JS and bypasses Cloudflare protection. If that fails, Playwright MCP with browser_navigate and browser_snapshot handles SPAs and heavily protected pages as the last resort.

Related Skills

Explore More Education & Research Skills→
drshailesh88drshailesh88

brave-search

Query the Brave Search API and extract readable page content via Node scripts.

Community
Intermediate
BEIRUXBEIRUX

tavily-cli

Perform LLM-optimized web search and content extraction with JSON output.

Official
Intermediate
taracodlabstaracodlabs
779

web_research

Fetch full web pages, verify facts across multiple sources, and cite URLs.

Official
Intermediate
Ocean326Ocean326

searxng-search

Search the web locally via a SearXNG stack without API keys.

Community
Intermediate