Create and manage robots.txt files with allow and disallow rules.
npx skills add https://github.com/decocms/storefront-skills --skill robots
Please help me install this Agent Skill. Skill: robots Source: https://github.com/decocms/storefront-skills/tree/main/.claude-seo/skills/robots Command: npx skills add https://github.com/decocms/storefront-skills --skill robots
Robots.txt at the root of a website instructs crawlers which pages or sections they are allowed or disallowed to crawl. It helps control indexing and preserves crawl budget by guiding search engines.
Create or update the robots.txt at your site root to specify which URLs crawlers can access.
High-intent search queries and answers about installing and using this skill.
Robots.txt controls site indexing by specifying User-agent, Allow, and Disallow directives to guide web crawlers. This protects your crawl budget by restricting access to non-essential pages like staging environments and internal search results.
To disallow web crawlers from staging environments and internal search results, apply Disallow directives in your robots.txt file. This prevents search engines from indexing sensitive or low-value areas while preserving your crawl budget for essential pages.
You add a sitemap reference to robots.txt by including a Sitemap directive. This ensures search engines can discover your sitemap, which helps them efficiently crawl and index your permitted storefront website pages.
Yes, you can set different allow and disallow rules for specific user-agents in robots.txt. By grouping directives under specific User-agent names, you grant or restrict crawler access to distinct sections of your storefront website as needed.
You validate and test robots.txt rules using search console tools before publishing changes. This testing process ensures your allow and disallow directives function correctly, preventing accidental blocking of essential product pages from web crawlers.
You need to manage robots.txt directives for a storefront website when you require precise disallow and allow rules for product pages. Proper management ensures web crawlers focus on indexing products instead of wasting crawl budget on internal search results.
meta-pytorchAutomate pre-submit PR validation with linting, tests, and RFC analysis.
rampstackcoAudit technical SEO issues across crawlability, indexability, rendering, and structured data.
calm-northDiagnose technical SEO issues across crawlability, indexability, renderability, and on-page signals.
eprouvezeInspect robots.txt, meta robots tags, and HTTP headers to map AI crawler access.
