skills.rest
stagsz
stagszCOMMUNITY
Explore all skills by this author →
GitHub

geo-crawlers

Analyze website accessibility for AI crawlers via robots.txt, meta tags, and HTTP headers.

Updated Mar 11, 2026
Search Tags:#content discovery#seo#chatgpt#perplexity#robots.txt#ai visibility#crawler access
One-click install
npx skills add https://github.com/stagsz/GeoRank --skill geo-crawlers-stagsz
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: geo-crawlers
Source: https://github.com/stagsz/GeoRank/tree/main/skills/geo-crawlers
Command: npx skills add https://github.com/stagsz/GeoRank --skill geo-crawlers-stagsz

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps you understand and configure how AI crawlers access your website, ensuring your content is discoverable by AI search engines and assistants.

Core Features & Use Cases

  • AI Crawler Audit: Analyzes robots.txt, meta tags, and HTTP headers to identify which AI crawlers (like GPTBot, ClaudeBot, PerplexityBot) can access your site.
  • Access Map Generation: Provides a clear overview of crawler permissions and their impact on AI visibility.
  • Recommendation Engine: Offers specific guidance on configuring robots.txt for maximum AI visibility while maintaining control.
  • Use Case: A marketing team wants to ensure their new blog content is picked up by ChatGPT and Perplexity. This Skill helps them verify that their site isn't inadvertently blocking these crucial AI crawlers.

Quick Start

Analyze the AI crawler access for the domain example.com.

Frequently Asked Questions about geo-crawlers

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I check if AI crawlers like GPTBot and PerplexityBot can access my website?▼

To check AI crawler access, analyze your website's robots.txt, meta tags, and HTTP headers. This identifies which AI bots like GPTBot or PerplexityBot can access content, providing recommendations for optimizing AI search visibility and control.

What is robots.txt configuration for AI search visibility?▼

Robots.txt configuration for AI search visibility involves setting specific rules to allow or block AI crawlers. Analyzing these rules identifies which AI bots can access your content, enabling recommendations for configuring permissions to maximize visibility while maintaining control.

How do I audit my website for AI crawler access and meta tags?▼

You audit website AI crawler access by examining robots.txt, meta tags, and HTTP headers. This comprehensive site crawling and analysis identifies which AI bots can access your content and provides specific guidance on configuring access for AI search engines.

Does WebFetch work with Bash for analyzing AI crawler permissions?▼

Yes, WebFetch integrates with Bash for comprehensive site crawling and analysis. This combination examines robots.txt, meta tags, and HTTP headers to generate a clear overview of crawler permissions and their impact on AI visibility.

Why does my site inadvertently block ChatGPT and Perplexity crawlers?▼

Your site might inadvertently block ChatGPT and Perplexity crawlers due to restrictive rules in robots.txt or HTTP headers. Analyzing these files identifies which AI bots are blocked and offers specific guidance on configuring permissions for visibility.

Related Skills

Explore More Marketing & Sales Skills→
wredziowredzio

seo

Generate meta tags, canonical URLs, JSON-LD structured data, and robots.txt recommendations.

Community
Advanced
engineai-nzengineai-nz

geo-crawlers

Analyze robots.txt, meta robots, and X-Robots-Tag headers for AI crawler access.

Official
Advanced
REVREBELREVREBEL

seo-firecrawl

Crawl, scrape, and map full websites for SEO analysis via Firecrawl MCP.

Official
Advanced
vrcmsvrcms
1

seo

Audit and implement technical SEO, on-page optimization, and structured data.

Community
Advanced