What is robots.txt (and Why It Matters for AI)?
robots.txt is a plain text file at yoursite.com/robots.txt that tells crawlers which parts of the site they may fetch. It has been a search engine convention for decades, and it now doubles as the control panel for AI: rules for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended decide whether AI systems can learn about your brand and cite your pages.
What it can and cannot do
robots.txt is a request, not an enforcement mechanism. Reputable crawlers honor it; bad actors ignore it. It also does not remove content from an index or a model that already has it. Think of it as a published policy that well-behaved bots follow.
AI-era practice
- Name AI crawlers explicitly rather than relying on the wildcard
- Allow retrieval bots (OAI-SearchBot, PerplexityBot) if you want AI citations
- Decide training bots (GPTBot, Google-Extended, CCBot) as a policy question
- Keep sitemap and llms.txt referenced and current
For brands that want AI visibility, the common mistake is a blanket block that quietly removes the site from AI search retrieval. Audit which user agents your rules actually match.