robots
OfficialProtect crawl budget and site indexing.
Authordecocms
Version1.0.0
Installs0
System Documentation
What problem does it solve?
Robots.txt at the root of a website instructs crawlers which pages or sections they are allowed or disallowed to crawl. It helps control indexing and preserves crawl budget by guiding search engines.
Core Features & Use Cases
- Define allow and disallow rules for different user-agents to protect sensitive areas (e.g., staging, internal search results) while permitting essential pages.
- Ensure search engines can discover your sitemap by including a Sitemap directive.
- Validate and test rules with search console tools before publishing changes.
Quick Start
Create or update the robots.txt at your site root to specify which URLs crawlers can access.
Dependency Matrix
Required Modules
None requiredComponents
Standard package💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: robots Download link: https://github.com/decocms/storefront-skills/archive/main.zip#robots Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 510,000+ vetted skills library on demand.