What is an AI Crawler?
An AI crawler is a bot that fetches your web pages for an AI system. Some collect content for model training (GPTBot, ClaudeBot, CCBot), others retrieve pages live so an assistant can answer with current information (OAI-SearchBot, PerplexityBot). Whether you allow them in robots.txt shapes whether AI systems can learn about and cite your site.
Training crawlers vs retrieval crawlers
The distinction matters for policy. Blocking a training crawler keeps your content out of future model weights. Blocking a retrieval crawler keeps your pages out of live AI answers, which usually costs you citations today. Many sites allow retrieval bots and decide case by case on training bots.
Common AI crawlers
- GPTBot and OAI-SearchBot (OpenAI: training and search retrieval)
- ClaudeBot (Anthropic)
- PerplexityBot (Perplexity)
- Google-Extended (controls Gemini training use of your content)
- CCBot (Common Crawl, a dataset many models train on)
- Bytespider, Amazonbot, meta-externalagent
Naming these bots explicitly in robots.txt signals intent and survives changes to their default behavior. A wildcard allow works today, but explicit rules are the durable version.