# AI crawler rules audit

Scope frozen before inspecting results on 2026-09-26. Reuse the 500-company / 503-security S&P 500 cohort captured 2026-07-26 in ../sp500-constituents-snapshot.json and the resolved origins in ../sp500-llms-txt-audit-data.json. This is a follow-up of that historical cohort, not a freshly certified index membership list. Corporate origins only, not every subsidiary or product domain.

Fetch /robots.txt with an honest CitationWorks research user agent, eight concurrent requests, 15-second timeout, bounded 512,000-byte body, normal redirects. No impersonation, proxy recovery or paid API. Retain retrieval timestamps, response status, final URL, body hash and raw files. Resume by company CIK; do not silently overwrite observations. Initial sandbox DNS failures are preserved separately and excluded from findings.

Eight documented agents, grouped by purpose: search (OAI-SearchBot, Claude-SearchBot, PerplexityBot), training (GPTBot, ClaudeBot), user-directed retrieval (ChatGPT-User, Claude-User, Perplexity-User). This is deliberately not an exhaustive list. Google-Extended is a separate product-use control and is outside this eight-agent crawler inventory.

Use exact case-insensitive product-token groups, merge repeated groups, prefer specific groups over wildcard fallback. Preserve directive text and line numbers. Classify blanket Disallow: / or /* without Allow exceptions separately from blanket rules with exceptions and narrower path restrictions. Empty Disallow means no restriction. Broad conflicting Allow/Disallow rules and malformed lines are conservatively ambiguous. This is a policy inventory, not a full RFC URL evaluator: no claim that every path is blocked, and no extrapolation from root-only /$ to a whole website.

Keep HTML/challenges, oversized or invalid content ambiguous. 404/410 are missing files, not observed explicit permission. Other non-2xx and network failures are inaccessible, not blocks. Report all 500 denominator and readable-file denominator. Do not silently drop unknowns. No robots rule establishes actual server access, compliance by a user-directed fetcher, index inclusion, recommendations, traffic or revenue effects.

Sources checked 2026-09-26:
- https://developers.openai.com/api/docs/bots
- https://privacy.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- https://docs.perplexity.ai/docs/resources/perplexity-crawlers
- https://www.rfc-editor.org/rfc/rfc9309.html

Before publication: offline parser tests, inspect every blanket-disallow case against raw evidence, inspect ambiguous cases and a sample of other outcomes. Publish company-level rules with provenance; distinguish AI-assisted research from independent human review.