geo-crawlers

Name: geo-crawlers
Availability: InStock
Author: rajains234-netizen

Analyze website accessibility for AI crawlers via robots.txt, meta tags, and HTTP headers.

Updated Feb 27, 2026

Search Tags:#content strategy #web crawling #seo #visibility #robots.txt #ai crawler

Ready to run

npx skills add https://github.com/rajains234-netizen/GEO --skill geo-crawlers

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: geo-crawlers
Source: https://github.com/rajains234-netizen/GEO/tree/main/skills/geo-crawlers
Command: npx skills add https://github.com/rajains234-netizen/GEO --skill geo-crawlers

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill ensures your website content is discoverable by AI crawlers, preventing accidental blocking that makes your content invisible in AI-generated responses and search results.

Core Features & Use Cases

AI Crawler Audit: Analyzes robots.txt, meta tags, and HTTP headers for major AI crawlers (GPTBot, ClaudeBot, PerplexityBot, etc.).
Access Mapping: Provides a clear map of which AI crawlers can access your site and which are blocked.
Recommendation Engine: Offers specific, actionable recommendations for configuring robots.txt and other directives to maximize AI visibility.
Use Case: A content publisher uses this Skill to identify that their robots.txt is inadvertently blocking GPTBot, leading to their content not appearing in ChatGPT's AI search results. The Skill provides the exact Allow: / directive needed to fix this.

Quick Start

Analyze the AI crawler access for the domain example.com.

Frequently Asked Questions about geo-crawlers

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I check if AI crawlers like GPTBot or ClaudeBot can access my website?▼

To check if AI crawlers can access your website, you must analyze your robots.txt file, meta robots tags, and HTTP headers. This audit identifies which specific AI bots are allowed or blocked from fetching your content for AI-generated search results.

Why does my website content not appear in AI-generated search results?▼

Your website content might not appear in AI-generated search results due to accidental blocking in your robots.txt file or HTTP headers. Analyzing your crawler directives identifies and resolves these access restrictions for major AI bots.

How to optimize robots.txt for AI crawler visibility?▼

To optimize robots.txt for AI crawler visibility, you audit your current directives and apply specific Allow rules for target bots. Generating a precise access map ensures your content is discoverable while maintaining desired control over crawler behavior.

What is an AI crawler audit and how does it work?▼

An AI crawler audit examines your website's robots.txt, meta tags, and HTTP headers to map AI bot accessibility. It identifies which AI crawlers can access your content and provides actionable recommendations to optimize your site's AI visibility.

Can I control which AI bots scrape my site content without blocking search engines?▼

Yes, you can control AI bots by implementing specific User-agent directives in your robots.txt file. Analyzing your HTTP headers and meta robots tags allows you to selectively block or allow AI crawlers while maintaining standard search engine visibility.