Who in the S&P 500 publishes an llms.txt?

Execution plan

1. 1
   Freeze the population CompleteSave a timestamped constituent snapshot and distinguish 503 securities from 500 companies.
2. 2
   Resolve corporate roots CompleteMap company articles to Wikidata official websites; document every manual correction.
3. 3
   Collect factual HTTP evidence CompleteGET /llms.txt, follow redirects, identify soft 404s, and retain status metadata.
4. 4
   Verify exceptions CompleteBrowser retries and randomized control paths resolved fallbacks; repeat access failures are disclosed as inaccessible rather than forced into a binary result.
5. 5
   Publish the adoption story PendingRelease the dated dataset, methodology, corrections channel, and careful provider context.

Editorial framing

Do not call absence a discoverability failure. Google says it ignores llms.txt for Search, and the proposal itself describes an inference-time content map rather than a guaranteed ranking signal.

Next research layer: AI crawler access in robots.txt, sitemaps, crawlable text, and structured data. Keep that separate from this adoption census.

Methodology

Population: the current constituent table from Wikipedia’s S&P 500 list, fetched at the start of this run.

Domains: each company article was resolved to Wikidata and its P856 official-website property. Multiple share classes sharing one company website were combined.

Request: one HTTP GET per unique corporate root, 15-second timeout, 1 retry, redirects followed, maximum 1000 KB retained. Ambiguous results received a Chrome retry; successful HTML responses were compared with a randomized nonexistent path on the same origin.

Interpretation limits

A missing file only means no qualifying response was observed at the checked corporate root URL and time. Repeat blocks, network failures, and server errors are labeled inaccessible and are not counted as confirmed absence.

A proposal-shaped file has a Markdown H1 and at least one Markdown link. This is a structural check, not an endorsement of the contents.

Generated by scripts/run-sp500-llms-audit.mjs. Raw JSON and CSV are stored beside this HTML tracker.