skills.rest
thefear078
thefear078COMMUNITY
Explore all skills by this author →
𝕏@thefear078GitHub

geo-crawlers

Analyzes AI crawler access via robots.txt, meta robots, and HTTP headers to score GEO readiness.

15|3|Updated Jul 9, 2026
Search Tags:#waf#robots.txt#geo#ai crawlers#gptbot#ai search visibility#crawler audit
One-click install
npx skills add https://github.com/thefear078/cursor-kit-for-ai --skill geo-crawlers-thefear078
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: geo-crawlers
Source: https://github.com/thefear078/cursor-kit-for-ai/tree/main/plugins/geo-seo/skills/geo-crawlers
Command: npx skills add https://github.com/thefear078/cursor-kit-for-ai --skill geo-crawlers-thefear078

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? AI crawlers like GPTBot, ClaudeBot, and PerplexityBot are often blocked by legacy robots.txt rules or WAF settings, making a site invisible to AI search engines despite strong traditional SEO. This Skill maps exactly which bots can fetch your content and produces copy-paste robots.txt fixes aligned with your training-vs-search policy. ## Core Features & Use Cases - Crawler Access Matrix: Resolves allow/block/partial status for Tier 1 and Tier 2 AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Bingbot) from robots.txt, meta robots, and X-Robots-Tag headers. - GEO Crawler Scoring: Applies a weighted rubric with severity deductions (Googlebot blocked = fatal, catch-all Disallow, WAF overrides) scaled to a 0-100 score. - Policy Templates: Provides ready-to-paste robots.txt configurations for allow-all, block-training-only, and path-level block patterns, plus Cloudflare-specific verification commands. - Use Case: After enabling Cloudflare's "Block AI Scrapers" toggle, run this audit to discover PerplexityBot is challenged at the edge despite a permissive robots.txt, then adjust WAF rules and re-verify with curl user-agent tests. ## Quick Start Audit example.com for AI crawler access and generate a crawler access report with recommended robots.txt changes.

Frequently Asked Questions about geo-crawlers

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I check if AI crawlers can access my website?▼

Fetch your robots.txt with curl and grep for rules targeting GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot, then check homepage and inner pages for meta robots noindex tags and X-Robots-Tag headers. Bots without explicit rules are allowed by default per the robots.txt specification.

Which AI crawler user-agents should I allow in robots.txt?▼

Allow Tier 1 crawlers: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, and PerplexityBot, since blocking them removes you from ChatGPT, Claude, and Perplexity answers. Never block Googlebot or Bingbot, as they power Google AI Overviews and Microsoft Copilot.

Can I block AI training but still appear in AI search results?▼

Yes, disallow training-only crawlers like GPTBot, Google-Extended, and CCBot while allowing search crawlers such as OAI-SearchBot, ChatGPT-User, PerplexityBot, and ClaudeBot. Google-Extended controls Gemini training data without affecting standard Google ranking.

Does Cloudflare Block AI Scrapers override my robots.txt rules?▼

Yes, Cloudflare's Block AI Scrapers toggle, Bot Fight Mode, and custom WAF user-agent rules can block or challenge AI crawlers at the edge regardless of permissive robots.txt rules. Verify by sending curl requests with crawler user-agents like GPTBot and PerplexityBot after any Cloudflare change.

Why is my site invisible in ChatGPT and Perplexity despite good SEO?▼

The most common cause is a legacy robots.txt with a catch-all Disallow or explicit blocks on AI bots, often copied from old SEO templates. Other causes include homepage noindex directives, WAF user-agent blocking, or staging subdomain rules leaking to production via wildcard DNS.

What happens if I block Googlebot to reduce crawl budget?▼

Blocking Googlebot is fatal for both SEO and GEO, scoring 0 on the crawler rubric, because it removes your site from Google Search and AI Overviews entirely. Crawl budget concerns should be handled with path-level Disallow rules for admin or staging directories instead.

Related Skills

Explore More Marketing & Sales Skills→
kiurakkukiurakku
15

geo-crawlers

Analyzes robots.txt, meta robots, and HTTP headers to map AI crawler access and generate robots.txt recommendations.

Community
Intermediate
kiurakkukiurakku
15

geo-technical

Audits technical SEO and GEO readiness including SSR, crawlability, Core Web Vitals, and AI crawler access.

Community
Advanced
Marshal-NguyenMarshal-Nguyen
1

geo-crawlers

Analyzes robots.txt, meta tags, and HTTP headers to map AI crawler access for a website.

Community
Intermediate
pipixia-labspipixia-labs
38

email-marketing

Generates a single HTML email from a design system.

Official
Intermediate