skills.rest
eprouveze
eprouvezeCOMMUNITY
Explore all skills by this author →
GitHub

geo-crawlers

Inspect robots.txt, meta robots tags, and HTTP headers to map AI crawler access.

Updated Dec 17, 2025
Search Tags:#visibility#robots.txt#meta-tags#http-headers#ai-crawlers#crawler-access
One-click install
npx skills add https://github.com/eprouveze/HealthPulse --skill geo-crawlers-eprouveze
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: geo-crawlers
Source: https://github.com/eprouveze/HealthPulse/tree/main/.agent/skills/geo-crawlers
Command: npx skills add https://github.com/eprouveze/HealthPulse --skill geo-crawlers-eprouveze

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

AI crawler access analysis determines which crawlers can reach a website by inspecting robots.txt, meta robots tags, and HTTP headers, enabling precise access control and visibility planning.

Core Features & Use Cases

  • Analyze robots.txt, meta tags, and HTTP headers to determine per-crawler access.
  • Generate a complete access map and actionable recommendations to maximize AI visibility while preserving control.
  • Use cases include auditing site crawlability for AI models, ensuring privacy, and planning for integration with AI tools.

Quick Start

Fetch robots.txt and sample pages, analyze headers and meta tags, and generate an AI crawler access map.

Frequently Asked Questions about geo-crawlers

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I check which AI crawlers can access my website?▼

AI crawler access is determined by inspecting robots.txt, meta robots tags, and HTTP headers. This Skill fetches these elements from your domain to create a per-crawler access map showing allow/disallow rules and wildcard configurations.

What is the best way to audit robots.txt rules for AI crawler access?▼

Auditing robots.txt for AI crawler access involves detecting per-crawler user-agent allowances, wildcard rules, and potential sitemap directives. This Skill automates that inspection across multiple pages to generate an actionable access map.

Does this Skill analyze HTTP headers and meta tags for crawler access control?▼

Yes, HTTP headers and meta robots tags are analyzed alongside robots.txt to determine crawler access. The Skill detects header overrides and meta tag rules to provide a comprehensive map of AI crawler visibility.

How do I map AI crawler visibility and training exposure for my domain?▼

Mapping AI crawler visibility requires balancing access control and training exposure. The Skill inspects crawler allowances across your site's pages and sitemaps, generating recommendations to maximize AI visibility while preserving control.

Can I detect wildcard rules and per-crawler user-agent coverage across my site?▼

Yes, per-crawler user-agent coverage and wildcard rules are detected during the access analysis. The Skill fetches sample pages and robots.txt files to identify specific allow/disallow rules applied to various AI crawlers.

What are the limitations of automated crawler auditing for AI visibility?▼

Automated crawler auditing relies on fetching robots.txt and sample pages, meaning access maps are based on discovered directives at the time of inspection. It checks existing rules but does not modify server configurations or enforce access controls.

Related Skills

Explore More Software Engineering Skills→
engineai-nzengineai-nz

geo-technical

Audit websites for crawlability, indexability, security, and AI crawler access.

Official
Advanced
engineai-nzengineai-nz

geo-crawlers

Analyze robots.txt, meta robots, and X-Robots-Tag headers for AI crawler access.

Official
Advanced
fabricioctellesfabricioctelles
47

agent-ready-robots-txt

Implement and validate a robots.txt file with crawler access directives.

Community
Basic
mnishanth02mnishanth02

clerk-tanstack-patterns

Add Clerk authentication guards to TanStack Router routes with server-side verify and redirect.

Community
Intermediate