Cloudflare And GoDaddy Partner To Control AI Bot Crawling On Websites

Cloudflare and GoDaddy have launched a partnership to give website owners more visibility and control over how AI agents crawl their websites.

“If you want to shape media strategy—not just optimize it—upgrade your expertise”

ADWEEK

GoDaddy will integrate Cloudflare's AI Crawl Control into its web hosting platform.

ADWEEK

The partnership expands Cloudflare's Web Bot Auth system, which uses cryptographic verification to allow bots to prove their identity.

Automated crawlers run by Anthropic, OpenAI, Google, and others scan vast portions of the web but return far less traffic than traditional search engines.

Cloudflare manages about 20% of the web, while GoDaddy is the world's largest domain name registrar.

The broader goal is to create a permission-based system for the AI-driven web.

Creators would have clearer control and visibility—and a way to participate in the value created by AI systems.

Business Insider

This builds on Cloudflare's earlier moves to block AI crawlers by default.

The stakes are high: if AI systems continue to extract value without sending traffic back, the incentive to create original content could erode.

The partnership aims to rebuild a sustainable economic model for the next phase of the internet.