skills.rest
liuaibin001
liuaibin001COMMUNITY
Explore all skills by this author →
GitHub

geo-crawlers

Analyze website accessibility for AI crawlers via robots.txt, meta tags, and HTTP headers.

Updated Mar 16, 2026
Search Tags:#ai#seo#visibility#robots.txt#geo#crawlers
One-click install
npx skills add https://github.com/liuaibin001/dev-notes-skills --skill geo-crawlers-liuaibin001
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: geo-crawlers
Source: https://github.com/liuaibin001/dev-notes-skills/tree/main/geo-crawlers
Command: npx skills add https://github.com/liuaibin001/dev-notes-skills --skill geo-crawlers-liuaibin001

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires WebFetch, Bash, Grep, Glob, Write, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill helps you understand and control how AI crawlers access your website's content, ensuring your data is discoverable by AI search engines and preventing unintended data usage.

Core Features & Use Cases

  • AI Crawler Audit: Analyzes robots.txt, meta tags, and HTTP headers for major AI crawlers (GPTBot, ClaudeBot, PerplexityBot, etc.).
  • Access Mapping: Determines if each AI crawler is Allowed, Blocked, or Not Mentioned.
  • GEO Optimization: Provides an AI Visibility Score and actionable recommendations to improve your site's discoverability in AI search results.
  • Use Case: Diagnose why your content isn't appearing in AI search results, or proactively configure your site for optimal AI visibility.

Quick Start

Analyze AI crawler access for the domain example.com.

Frequently Asked Questions about geo-crawlers

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I check if AI crawlers like GPTBot and ClaudeBot are blocked by my robots.txt?▼

AI crawler audits parse your robots.txt, meta tags, and HTTP headers to determine if bots like GPTBot and ClaudeBot are Allowed, Blocked, or Not Mentioned. This reveals exactly why your content might be missing from AI search results.

What is Generative Engine Optimization and how does it affect website visibility?▼

Generative Engine Optimization (GEO) improves your site's discoverability in AI search results. It works by analyzing crawler access and providing actionable recommendations to ensure AI bots can index your content.

Why does my website content not appear in AI search results?▼

Your website might not appear in AI search results because robots.txt or HTTP headers are blocking AI crawlers. An audit identifies the exact access status for bots like PerplexityBot to diagnose visibility issues.

How do I audit my website for AI crawler access using HTTP headers and meta tags?▼

You audit AI crawler access by fetching and parsing your website's robots.txt, HTML meta tags, and HTTP headers. This maps the access status of major AI bots and generates an AI Visibility Score for GEO optimization.

Can I analyze AI crawler access for any domain without installing extra software?▼

You can analyze AI crawler access for any domain using built-in web fetching and parsing tools. The process requires no manual software installation as it operates through automated scripts that read your site's HTTP headers.

What is the best way to improve my site's AI Visibility Score for Generative Engine Optimization?▼

The best way to improve your AI Visibility Score is to audit your site's robots.txt and meta tags, then apply actionable GEO recommendations. This ensures AI crawlers are explicitly allowed to access your content.

Related Skills

Explore More Software Engineering Skills→
joanbyjoanby
25

geo-crawlers

Analyze robots.txt, meta robots tags, and HTTP headers to map AI crawler access.

Community
Intermediate
KSfakKSfak
20

geo-crawlers

Evaluate robots.txt, meta robots, and HTTP headers for AI crawler accessibility.

Community
Intermediate
rajains234-netizenrajains234-netizen

geo-crawlers

Analyze website accessibility for AI crawlers via robots.txt, meta tags, and HTTP headers.

Community
Advanced
kennyolofsson23-netizenkennyolofsson23-netizen

geo-technical

Audit website crawlability, indexing, security, performance, and SSR issues.

Community
Advanced