robots

Create and manage robots.txt files with allow and disallow rules.

13|2|Updated Feb 24, 2026

Search Tags:#indexing #sitemap #seo #robots.txt #crawl-budget #web-crawlers #disallow-allow

Ready to run

npx skills add https://github.com/decocms/storefront-skills --skill robots

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: robots
Source: https://github.com/decocms/storefront-skills/tree/main/.claude-seo/skills/robots
Command: npx skills add https://github.com/decocms/storefront-skills --skill robots

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Robots.txt at the root of a website instructs crawlers which pages or sections they are allowed or disallowed to crawl. It helps control indexing and preserves crawl budget by guiding search engines.

Core Features & Use Cases

Define allow and disallow rules for different user-agents to protect sensitive areas (e.g., staging, internal search results) while permitting essential pages.
Ensure search engines can discover your sitemap by including a Sitemap directive.
Validate and test rules with search console tools before publishing changes.

Quick Start

Create or update the robots.txt at your site root to specify which URLs crawlers can access.

Frequently Asked Questions about robots

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I use robots.txt to protect my crawl budget and control site indexing?▼

Robots.txt controls site indexing by specifying User-agent, Allow, and Disallow directives to guide web crawlers. This protects your crawl budget by restricting access to non-essential pages like staging environments and internal search results.

What is the best way to disallow web crawlers from accessing staging environments and internal search results?▼

To disallow web crawlers from staging environments and internal search results, apply Disallow directives in your robots.txt file. This prevents search engines from indexing sensitive or low-value areas while preserving your crawl budget for essential pages.

How do I add a sitemap reference to my robots.txt file?▼

You add a sitemap reference to robots.txt by including a Sitemap directive. This ensures search engines can discover your sitemap, which helps them efficiently crawl and index your permitted storefront website pages.

Can I set different allow and disallow rules for specific user-agents in robots.txt?▼

Yes, you can set different allow and disallow rules for specific user-agents in robots.txt. By grouping directives under specific User-agent names, you grant or restrict crawler access to distinct sections of your storefront website as needed.

How do I validate and test robots.txt rules before publishing changes to my site?▼

You validate and test robots.txt rules using search console tools before publishing changes. This testing process ensures your allow and disallow directives function correctly, preventing accidental blocking of essential product pages from web crawlers.

When do I need to manage robots.txt directives for a storefront website?▼

You need to manage robots.txt directives for a storefront website when you require precise disallow and allow rules for product pages. Proper management ensures web crawlers focus on indexing products instead of wasting crawl budget on internal search results.