500 websites. 54 llms.txt files with a title and links.

We checked the corporate root website of every company in the S&P 500. The index contained 503 securities in our snapshot, but 500 distinct companies after combining multiple share classes.

Seventy-one companies served a non-empty text response at /llms.txt. Of those, 54 had both a Markdown H1 and at least one Markdown link: 10.8% of the 500 companies.

Of the 71 detected files, 17 lacked an H1 or Markdown links: 23.9% of the files found.

The llms.txt proposal says the H1 is the only required section. Links are optional when no file-list section is used. We therefore call our H1-plus-link test “proposal-shaped,” not strict standards compliance. It is a practical quality threshold for a file whose purpose is to guide a reader to useful content.

We joined the results against the iShares S&P 100 ETF holdings dated July 23, 2026. The holdings contain 101 securities but 100 companies because Alphabet has two share classes.

Sixteen of those 100 blue-chip companies had a detected llms.txt. Eleven met our H1-plus-link threshold. Sixty-three were confirmed absent and 21 were inaccessible from both of our collection methods.

The 16 detected S&P 100 companies were Adobe, American Express, AT&T, Booking Holdings, Cisco, Colgate-Palmolive, CVS Health, GE Aerospace, McDonald’s, Nvidia, Salesforce, Simon Property Group, T-Mobile, Thermo Fisher Scientific, U.S. Bancorp, and Verizon.

Our first HTTP-only pass found 67 files, including 15 S&P 100 companies. Browser recovery added four confirmed files, including T-Mobile, bringing the final counts to 71 and 16.

Checking /llms.txt sounds binary until real websites get involved. Some missing paths redirect to a homepage. Some return a branded HTML error page with HTTP 200. Some serve the same fallback page for every unknown URL. Some security layers block an automated request before it reaches the application.

Our final classification found 330 companies with confirmed absence, including controlled HTML fallbacks and soft 404s. Another 94 were inaccessible after both a direct HTTP request and a browser pass: 64 access denials, 27 network failures, and three repeated server errors. We did not count those 94 as absent.

For successful HTML responses, we requested a randomized nonexistent .txt path on the same origin. When /llms.txt behaved like that control path, we treated it as a fallback rather than a real file.

110 companies returned a 2xx status at /llms.txt and only 71 served a real file. We broke down all 39 false positives in a companion piece, including the redirects, HTML fallbacks, soft 404s, and the one empty file.

• The smallest detected response was one byte. Two files reached our one-megabyte capture limit.
• Energy had zero detected files in this snapshot; information technology and industrials had the highest observed adoption rates at 18.9% and 18.5%.

The files ranged from a one-byte response to documents that exceeded our capture limit. Checking structure, links, and redirects gives a fuller account of an implementation than recording its presence alone.

This audit cannot answer that question. We measured public adoption and implementation, not ranking impact.

The original llms.txt proposal describes a curated Markdown map intended to help LLMs use a website at inference time. It does not promise that any particular assistant reads the file. Google’s current Search guidance is more direct: Google Search ignores llms.txt, including for its generative AI features.

A curated file may help tools that explicitly consume it. Publishing one does not, by itself, show that an assistant can find or cite your company.

OpenAI documents separate user agents for different purposes. OAI-SearchBot is used for ChatGPT search, GPTBot relates to potential model training, and ChatGPT-User may fetch pages in response to a user action. Those are different access decisions. Anthropic likewise documents separate bots for training, search, and user-directed retrieval.

Check robots.txt rules for each agent separately, then inspect the site’s CDN restrictions and crawlable content. An llms.txt response alone cannot show whether the intended crawler can reach those pages.

The audit exposed several cases a file checker needs to keep apart: a Markdown file and a soft 404; a missing file and an access denial; a reachable page and a page cited in an answer. We use those distinctions in our crawl-hygiene checks.

• Measure file existence, structure, redirects, MIME type, size, and link health separately.
• Label blocked and timed-out sites as inaccessible, never as missing.
• Use a browser or proxy fallback when a direct fetch is denied.
• Compare suspicious HTML responses with a randomized missing-path control.
• Track AI mentions and citation sources over time so technical changes can be compared with real outcomes.

A follow-up study could compare answers to the same buyer questions for companies with and without llms.txt, recording mentions, citations, and factual accuracy across assistants. Differences would be observational; they would not establish that the file caused them.

1. Check the exact /llms.txt response, including redirects, content type, and the body—not only the HTTP status.
2. If you publish one, start with a clear H1, add a short brand summary, and curate a small set of important Markdown links with useful descriptions.
3. Keep the linked pages public, canonical, current, and understandable without requiring a browser-side application to reveal the essential facts.
4. Review robots.txt separately for search, training, and user-directed AI agents. Do not assume one rule controls every use.
5. Monitor the business outcome: whether assistants mention the brand, describe it accurately, and cite first-party pages when buyers ask relevant questions.

The audit snapshot was completed on July 26, 2026. We resolved the current S&P 500 constituent securities to 500 distinct companies and official corporate root websites. We requested /llms.txt with redirects followed and retried ambiguous cases in headless Chrome. Successful HTML responses were compared with randomized nonexistent control paths. A detected file required a non-empty, non-HTML 2xx response that did not resemble a soft 404.

Our “proposal-shaped” quality threshold required a Markdown H1 and at least one Markdown link. It is deliberately stricter than the proposal’s minimum requirement, which is an H1. The S&P 100 comparison used the iShares OEF holdings file dated July 23, 2026 and combined Alphabet’s two share classes into one company.

This is a point-in-time technical census. Files, websites, index membership, and provider behavior can change. Corrections are welcome.