skills.rest
kostja94
kostja94COMMUNITY
Explore all skills by this author →
WebsiteGitHub

site-crawlability

Diagnose redirect chains, broken links, orphan pages, and pagination issues.

872|122|Updated Feb 26, 2026
Search Tags:#seo#broken links#technical seo#crawlability#crawl budget#site structure#ai crawler
One-click install
npx skills add https://github.com/kostja94/marketing-skills --skill site-crawlability
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: site-crawlability
Source: https://github.com/kostja94/marketing-skills/tree/main/skills/seo/technical/crawlability
Command: npx skills add https://github.com/kostja94/marketing-skills --skill site-crawlability

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill addresses issues that prevent search engines and AI crawlers from effectively discovering and indexing your website's content, such as broken links, poor site structure, and inefficient crawl budget usage.

Core Features & Use Cases

  • Fixes Crawlability Issues: Identifies and provides solutions for redirect chains, broken links (4xx errors), and orphan pages.
  • Optimizes Site Structure: Ensures important pages are easily accessible and recommends pagination over infinite scroll for better discoverability.
  • Manages Crawl Budget: Helps reduce wasted crawl attempts on duplicate or low-value URLs.
  • AI Crawler Optimization: Provides guidance for server-side rendering and URL management to improve visibility for AI bots like GPTBot and ClaudeBot.
  • Use Case: A website owner notices that some of their product pages are not appearing in search results. This Skill can help diagnose if the issue is due to broken internal links, a deep site hierarchy, or problems with how AI crawlers are accessing the content.

Quick Start

Analyze the crawlability of the website and identify any issues with broken links or site structure.

Frequently Asked Questions about site-crawlability

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I fix broken links and redirect chains affecting my site crawlability?▼

Fix site crawlability by identifying 4xx broken link errors and resolving redirect chains that waste crawl budget. Diagnosing these issues ensures search engine crawlers can efficiently discover and index your website content without hitting dead ends.

Why are my product pages not showing up in search results due to site structure?▼

Product pages often remain undiscovered due to deep site hierarchies or orphan pages lacking internal links. Optimizing site architecture ensures important pages are easily accessible, replacing infinite scroll with pagination for better search engine discoverability.

What is the best way to optimize crawl budget for duplicate content URLs?▼

Optimize crawl budget by reducing wasted crawl attempts on duplicate or low-value URLs. Managing crawl budget waste ensures search engine crawlers spend their allocated time indexing high-priority pages rather than repeatedly hitting redundant content.

How do I make my website accessible to AI crawlers like GPTBot and ClaudeBot?▼

Make your website accessible to AI crawlers by implementing server-side rendering and efficient URL management. This ensures AI bots like GPTBot and ClaudeBot can properly access and parse your content for visibility in AI search results.

When should I use pagination instead of infinite scroll for SEO?▼

Use pagination instead of infinite scroll when search engine crawlers need to discover deep content links. Pagination provides crawlable URLs that bots can follow, whereas infinite scroll often hides valuable pages from search engine indexing.

How do I find and fix orphan pages on my website?▼

Find and fix orphan pages by analyzing your site structure to locate URLs with no internal links pointing to them. Resolving orphan page issues ensures all valuable content is connected to your main site architecture for proper crawlability.

Related Skills

Explore More Marketing & Sales Skills→
aaron-he-zhuaaron-he-zhu
142

internal-linking-optimizer

Analyze internal link structures and generate actionable linking plans.

Community
Intermediate
kostja94kostja94
872

xml-sitemap

Generate, audit, and optimize XML sitemaps with hreflang annotations.

Community
Advanced
JDMAXiliusJDMAXilius

site-architecture

Audit website information architecture for crawlability and link equity.

Community
Advanced
meta-pytorchmeta-pytorch
2.5k

pre-submit-pr

Automate pre-submit PR validation with linting, tests, and RFC analysis.

Official
Intermediate