Which major companies block AI search crawlers? An S&P 500 audit
We inspected robots.txt on 500 corporate domains. Only 302 files could be classified without uncertainty; search and training agents often had different rules.
- Classifiable302
- Ambiguous content or syntax84
- Inaccessible95
- Missing file (404/410)19
366 readable files include 64 syntax-ambiguous files. Another 19 responses contained ambiguous content and one exceeded the size cap.Data behind this figure ↗
- companies checked
- 500
- agents inventoried
- 8
- classifiable files
- 302
September 26 retrieval / fixed July 26 cohort
Explore the evidenceSearch access and training access are separate decisions
Garmin and Simon Property Group publish blanket Disallow rules for PerplexityBot. Airbnb and Northrop Grumman disallow GPTBot and ClaudeBot while giving search agents different rules. A single “AI blocked” label would erase that distinction.
On September 26, 2026, we revisited the corporate domains from our earlier 500-company llms.txt audit. We inventoried eight agents across search, training and user-directed retrieval, keeping uncertain files outside the confirmed counts.
Three jobs, eight agents
The search group contains OAI-SearchBot, Claude-SearchBot and PerplexityBot. The training group contains GPTBot and ClaudeBot. ChatGPT-User, Claude-User and Perplexity-User represent user-directed retrieval. This is a selected inventory, not every AI-related agent.
500-company cohort; 302 classifiable files. Zero confirmed cases does not mean every company permits the crawler.Data behind this figure ↗
User-directed requests may follow different robots policies from automatic crawling. A published directive aimed at a user fetcher is evidence of the file’s contents, not proof that the fetcher obeys it or that the server denies access.
Which companies had confirmed blanket rules?
| Agent | Companies |
|---|---|
| PerplexityBot | Garmin; Simon Property Group |
| GPTBot | Airbnb; Northrop Grumman; PG&E; Prudential Financial; Simon Property Group |
| ClaudeBot | Airbnb; Fortinet; Northrop Grumman; Simon Property Group |
Narrow path restrictions and rules with Allow exceptions are excluded. Fortinet’s ClaudeBot result uses wildcard fallback.
Inspect the evidence ↗Fortinet’s ClaudeBot and Perplexity-User classifications come from wildcard fallback, rather than an explicit rule naming those agents. The company-level evidence records that distinction.
We found no confirmed blanket Disallow for OAI-SearchBot or Claude-SearchBot in the 302 files our conservative parser could classify. The other 198 companies are not counted as permitting access.
The ambiguous cases change what we can say
Cadence’s file put User-agent and Allow on the same line. Reading its likely intention as permission would require a guess. Falling back to its wildcard block would also produce an overconfident result. We marked it ambiguous.
Our strict inventory also flags paths without a leading slash, full URLs used as paths, orphan directives and other malformed lines. A flagged line makes the whole file ambiguous for this analysis, even if a separate agent group looks clear. Crawler-specific tolerance may differ. These are conservative confirmed counts, not an estimate of blocking prevalence.
How we collected and classified the files
The fixed cohort is the 500 companies represented by 503 securities in our July 26, 2026 S&P 500 snapshot. We did not recertify index membership for September 26. We fetched /robots.txt from the saved corporate origins, not every subsidiary, country site or product domain.
Requests identified CitationWorks, followed normal redirects, used a 15-second timeout and capped bodies at 512,000 bytes. We did not impersonate vendor crawlers. Each observation preserves its timestamp, HTTP status, final URL, content hash and the lines behind each policy.
Specific agent groups take precedence over wildcard fallback; repeated matching groups are merged. Blanket Disallow rules are separated from rules with Allow exceptions and narrower path restrictions. Missing files (404/410), network or HTTP failures, and ambiguous content stay separate. Review was AI-assisted, not independent human double-coding.
What a site owner can use this for
Decide separately how you want training, search discovery and user-requested retrieval to work. Then inspect the relevant agent groups and your server or CDN settings. Changing robots.txt does not guarantee a recommendation.
This audit did not measure whether the vendors actually accessed these sites, indexed their content, recommended their brands or sent them traffic. Published restrictions cannot establish lost visibility or revenue.