geo-crawlers

Analyze robots.txt, meta robots tags, and HTTP headers to map AI crawler access.

25|15|Updated Jun 1, 2026

Search Tags:#robots.txt #ai-visibility #site-audit #web-access #ai-crawlers #seo-access

Ready to run

npx skills add https://github.com/joanby/geo-seo-claude --skill geo-crawlers-joanby

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: geo-crawlers
Source: https://github.com/joanby/geo-seo-claude/tree/main/skills/geo-crawlers
Command: npx skills add https://github.com/joanby/geo-seo-claude --skill geo-crawlers-joanby

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Analyze a website's accessibility to AI crawlers by inspecting robots.txt, meta robots tags, and HTTP headers to determine which AI crawlers can access content, enabling informed optimization for AI-powered search and retrieval.

Core Features & Use Cases

Inspects robots.txt rules, meta robots, and X-Robots-Tag headers to map crawl access for major AI crawlers.
Produces a comprehensive access map and practical recommendations to maximize AI visibility while maintaining control.
Use Case: An SEO or web ops team audits a site to ensure critical AI crawlers can index important pages without exposing sensitive content.

Quick Start

Analyze a domain to generate an AI-crawler access report and recommendations.

Frequently Asked Questions about geo-crawlers

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I check if AI crawlers can access my website?▼

To check if AI crawlers can access your website, you need to inspect your robots.txt file, meta robots tags, and X-Robots-Tag HTTP headers. This analysis identifies which specific AI crawlers are allowed or blocked from indexing your site content.

What is robots.txt parsing for AI crawler visibility?▼

Robots.txt parsing for AI crawler visibility is the process of reading website access rules to determine if AI bots can crawl content. It maps out explicitly allowed and blocked AI crawlers, enabling informed optimization for AI-powered search and retrieval.

How do I audit my site for AI indexing access?▼

You can audit your site for AI indexing access by analyzing robots.txt rules, meta robots tags, and HTTP headers. This generates a structured report listing allowed and blocked AI crawlers, providing actionable recommendations to maximize visibility while maintaining control.

Does blocking AI crawlers in robots.txt affect my site's AI search visibility?▼

Blocking AI crawlers in robots.txt directly prevents those bots from indexing your site, reducing AI search visibility. Analyzing your access rules helps identify these blocks so you can strategically manage which AI crawlers reach your important pages.

Can I use this site audit approach for any domain?▼

Yes, you can use this site audit approach for any domain requiring AI indexing awareness. The analysis examines standard web access protocols like robots.txt and HTTP headers, making it applicable to any website regardless of its underlying platform.

What is the best way to map crawler access without exposing sensitive content?▼

The best way to map crawler access without exposing sensitive content is to inspect robots.txt, meta robots, and X-Robots-Tag headers together. This produces an access map and practical recommendations to maximize AI visibility while keeping sensitive content protected.