insane-search

Bypass blocked websites using yt-dlp, Jina Reader, and public APIs.

Updated Nov 5, 2018
One-click install
npx skills add https://github.com/gyuha/gyuha.github.io --skill insane-search-gyuha
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: insane-search
Source: https://github.com/gyuha/gyuha.github.io/tree/main/.agents/skills/insane-search
Command: npx skills add https://github.com/gyuha/gyuha.github.io --skill insane-search-gyuha

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires curl_cffi, yt-dlp, jina, playwright, beautifulsoup4, pyyaml, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill automates the bypassing of blocked websites and accessing restricted content, making previously inaccessible information readily available.

Core Features & Use Cases

  • Auto-bypass: Automatically tries various methods to access blocked websites until one succeeds.
  • Leverages Multiple Technologies: Utilizes yt-dlp, Jina Reader, public APIs, and a WAF-profile-driven fetch chain.
  • Use Case: When a website like Twitter, Reddit, YouTube, or GitHub is blocked and WebFetch returns an error, this Skill will attempt to access the site using its built-in methods.

Quick Start

Use the 'insane-search' skill to access the blocked website 'example.com'.

Frequently Asked Questions about insane-search

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I bypass WAF restrictions to access blocked website content?

To bypass WAF restrictions and access blocked website content, this Skill applies a WAF-profile-driven fetch chain using curl_cffi and Playwright, automatically attempting multiple access methods until one succeeds.

What's the best way to scrape web pages when WebFetch returns an error?

When WebFetch returns an error, the best way to scrape web pages is using an automated fallback chain that tries Jina Reader, public APIs, and yt-dlp to retrieve the restricted content.

Can I use Playwright and curl_cffi to access restricted content from sites like Reddit or Twitter?

Yes, you can access restricted content from sites like Reddit or Twitter using this Skill, which leverages Playwright and curl_cffi within a multi-method fetch chain to bypass blocks.

How does automated web automation handle sites that block standard scraping tools?

Automated web automation handles blocked sites by sequentially deploying multiple technologies, including yt-dlp and Jina Reader, rotating through a WAF-profile-driven fetch chain until access is granted.

Does this web scraping approach work with YouTube and GitHub when access is blocked?

This web scraping approach works with YouTube and GitHub when access is blocked by utilizing yt-dlp and public APIs as fallback methods alongside its WAF-profile-driven fetch chain.

What are the limitations of using a WAF-profile-driven fetch chain for web automation?

The limitations of a WAF-profile-driven fetch chain for web automation include dependency on multiple libraries like curl_cffi and Playwright, and success depends on whether public APIs or Jina Reader can access the target.