aeo

Fetch your robots.txt and grep for AI user agents like OAI-SearchBot, Claude-SearchBot, and PerplexityBot, then fetch a page with a crawler user-agent string to confirm a 200 response. Robots.txt permission alone is not proof — WAF and CDN bot rules can silently 403 crawlers that robots.txt invites.

Run eight ordered checks: crawler access, raw-HTML visibility, answer-first heading structure, citations and statistics, freshness signals, entity/schema markup, technical hygiene, and llms.txt. Audit the raw HTML fetched without JavaScript execution, since most AI crawlers never render pages.

No — GPTBot controls training data only, while OAI-SearchBot controls ChatGPT Search citations and ChatGPT-User handles live fetches. Each operator maintains separate user agents per class, so blocking one class does not block the others; verify each class individually.

No — GPTBot, ClaudeBot, and PerplexityBot read only the initial HTML response and do not execute JavaScript; only Gemini, riding Googlebot's infrastructure, renders JS. Client-rendered content inside an empty root div is invisible to every other AI engine.

Evidence is weak — server-log studies show roughly 0.1% of AI-bot traffic touches llms.txt, and no major engine commits to consuming it for search. It earns a place mainly for docs-heavy or agent-facing developer sites, and should never be flagged as a critical finding.

The audit fixes nothing — it ends in findings and a plan doc, leaving all edits to the user. Repo-only mode cannot prove live reachability, and sampled pages may not represent the whole site, so unverified areas are declared explicitly in the report.