agent-ready-content-signals

To set AI training policies in robots.txt, you add Content-Signal directives for ai-train, search, and ai-input. This clearly communicates to AI crawlers whether they can train on, search, or use your content for inputs.

Content-Signal directives are robots.txt rules that declare website preferences for AI crawlers. They specify permissions for ai-train, search, and ai-input, reducing ambiguity in how automated systems interact with your content.

Yes, you can configure AI content usage preferences through Cloudflare AI Crawl Control. This works alongside robots.txt Content-Signal directives to manage how AI systems train on, search, and input your website data.

You validate robots.txt AI crawler rules by scanning your website with the Is It Agent Ready API. This verifies that your Content-Signal directives for ai-train, search, and ai-input pass the agent-readiness checks.

To allow AI search discovery but block model training in robots.txt, you configure Content-Signal directives to permit search while disallowing ai-train and ai-input. This publisher use case controls AI behavior precisely.