skills.rest
rajains234-netizen
rajains234-netizenCOMMUNITY
Explore all skills by this author →
GitHub

geo-crawlers

Analyze website accessibility for AI crawlers via robots.txt, meta tags, and HTTP headers.

Updated Feb 27, 2026
Search Tags:#content strategy#web crawling#seo#visibility#robots.txt#ai crawler
One-click install
npx skills add https://github.com/rajains234-netizen/GEO --skill geo-crawlers
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: geo-crawlers
Source: https://github.com/rajains234-netizen/GEO/tree/main/skills/geo-crawlers
Command: npx skills add https://github.com/rajains234-netizen/GEO --skill geo-crawlers

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill ensures your website content is discoverable by AI crawlers, preventing accidental blocking that makes your content invisible in AI-generated responses and search results.

Core Features & Use Cases

  • AI Crawler Audit: Analyzes robots.txt, meta tags, and HTTP headers for major AI crawlers (GPTBot, ClaudeBot, PerplexityBot, etc.).
  • Access Mapping: Provides a clear map of which AI crawlers can access your site and which are blocked.
  • Recommendation Engine: Offers specific, actionable recommendations for configuring robots.txt and other directives to maximize AI visibility.
  • Use Case: A content publisher uses this Skill to identify that their robots.txt is inadvertently blocking GPTBot, leading to their content not appearing in ChatGPT's AI search results. The Skill provides the exact Allow: / directive needed to fix this.

Quick Start

Analyze the AI crawler access for the domain example.com.

Frequently Asked Questions about geo-crawlers

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I check if AI crawlers like GPTBot or ClaudeBot can access my website?▼

To check if AI crawlers can access your website, you must analyze your robots.txt file, meta robots tags, and HTTP headers. This audit identifies which specific AI bots are allowed or blocked from fetching your content for AI-generated search results.

Why does my website content not appear in AI-generated search results?▼

Your website content might not appear in AI-generated search results due to accidental blocking in your robots.txt file or HTTP headers. Analyzing your crawler directives identifies and resolves these access restrictions for major AI bots.

How to optimize robots.txt for AI crawler visibility?▼

To optimize robots.txt for AI crawler visibility, you audit your current directives and apply specific Allow rules for target bots. Generating a precise access map ensures your content is discoverable while maintaining desired control over crawler behavior.

What is an AI crawler audit and how does it work?▼

An AI crawler audit examines your website's robots.txt, meta tags, and HTTP headers to map AI bot accessibility. It identifies which AI crawlers can access your content and provides actionable recommendations to optimize your site's AI visibility.

Can I control which AI bots scrape my site content without blocking search engines?▼

Yes, you can control AI bots by implementing specific User-agent directives in your robots.txt file. Analyzing your HTTP headers and meta robots tags allows you to selectively block or allow AI crawlers while maintaining standard search engine visibility.

Related Skills

Explore More Marketing & Sales Skills→
joanbyjoanby
25

geo-crawlers

Analyze robots.txt, meta robots tags, and HTTP headers to map AI crawler access.

Community
Intermediate
fabricioctellesfabricioctelles
47

agent-ready-robots-txt

Implement and validate a robots.txt file with crawler access directives.

Community
Basic
ihmissutiihmissuti

ai-search-technical-auditor

Audit HTML, meta tags, schema markup, robots.txt, and llms.txt for AI crawler indexing.

Community
Basic
honysyanghonysyang
18

permission-overreach

Detect permission overreach by comparing declared permissions against actual code behavior.

Community
Intermediate