We Checked Every S&P 500 Website. Only 54 Had an llms.txt With a Title and Links

We checked the corporate root website of every company in the S&P 500. The index contained 503 securities in our snapshot, but 500 distinct companies after combining multiple share classes.

Seventy-one companies served a non-empty text response at /llms.txt. Only 54—10.8% of the 500 companies—had both a Markdown H1 and at least one Markdown link.

That second number matters. A file can exist without being a useful map. Seventeen of the 71 detected files missed our basic structure threshold. That is 23.9%, or nearly one in four adopters.

One important technical caveat: the llms.txt proposal says the H1 is the only required section. Links are optional when no file-list section is used. We therefore call our H1-plus-link test “proposal-shaped,” not strict standards compliance. It is a practical quality threshold for a file whose purpose is to guide a reader to useful content.

We joined the results against the iShares S&P 100 ETF holdings dated July 23, 2026. The holdings contain 101 securities but 100 companies because Alphabet has two share classes.

Sixteen of those 100 blue-chip companies had a detected llms.txt. Eleven met our H1-plus-link threshold. Sixty-three were confirmed absent and 21 were inaccessible from both of our collection methods.

The 16 detected S&P 100 companies were Adobe, American Express, AT&T, Booking Holdings, Cisco, Colgate-Palmolive, CVS Health, GE Aerospace, McDonald’s, Nvidia, Salesforce, Simon Property Group, T-Mobile, Thermo Fisher Scientific, U.S. Bancorp, and Verizon.

Our first HTTP-only pass found 67 files, including 15 S&P 100 companies. Browser recovery added four confirmed files, including T-Mobile, bringing the final counts to 71 and 16. That difference became one of the most useful lessons in the project.

Checking /llms.txt sounds binary until real websites get involved. Some missing paths redirect to a homepage. Some return a branded HTML error page with HTTP 200. Some serve the same fallback page for every unknown URL. Some security layers block an automated request before it reaches the application.

Our final classification found 330 companies with confirmed absence, including controlled HTML fallbacks and soft 404s. Another 94 were inaccessible after both a direct HTTP request and a browser pass: 64 access denials, 27 network failures, and three repeated server errors. We did not count those 94 as absent.

For successful HTML responses, we requested a randomized nonexistent .txt path on the same origin. When /llms.txt behaved like that control path, we treated it as a fallback rather than a real file. This negative control is the difference between checking a status code and testing the claim.

• 64 detected files were served as text/plain; only seven used text/markdown.
• 23 of 71 detected files redirected before reaching the final file URL.
• The median observed file size was 8,144 bytes, while the 90th percentile was 116,645 bytes.
• The smallest detected response was one byte. Two files reached our one-megabyte capture limit.
• Energy had zero detected files in this snapshot; information technology and industrials had the highest observed adoption rates at 18.9% and 18.5%.

The point of llms.txt is curation. A one-byte response cannot guide anything. A megabyte-scale dump may recreate the context problem the proposal was designed to reduce. Presence is a useful first check, but size, structure, links, redirects, and change history are the operational story.

This audit cannot answer that question. We measured public adoption and implementation, not ranking impact.

The original llms.txt proposal describes a curated Markdown map intended to help LLMs use a website at inference time. It does not promise that any particular assistant reads the file. Google’s current Search guidance is more direct: Google Search ignores llms.txt, including for its generative AI features.

That does not make the file useless. A concise public map can be helpful to tools, agents, documentation workflows, and any system that chooses to consume it. It makes llms.txt a low-cost experiment—not a ranking strategy and not evidence of visibility by itself.

OpenAI documents separate user agents for different purposes. OAI-SearchBot is used for ChatGPT search, GPTBot relates to potential model training, and ChatGPT-User may fetch pages in response to a user action. Those are different access decisions. Anthropic likewise documents separate bots for training, search, and user-directed retrieval.

A serious AI-readiness review therefore cannot stop at llms.txt. It must check robots.txt by user agent, sitemap discovery, canonical URLs, crawlable page content, JavaScript dependence, structured facts, and whether security infrastructure blocks the visitors the brand intended to allow.

The cleanest product lesson is simple: llms.txt is plumbing. Mentions, citations, and factual accuracy are the outcome.

CitationWorks already checks crawl hygiene, generates an llms.txt draft, monitors brand mentions, and records which sources AI answers cite. This audit showed why those pieces belong together. A green check beside a file is not enough. We need to distinguish a real Markdown file from a soft 404, an access denial from confirmed absence, and a technically reachable page from a brand that is actually cited.

• Measure file existence, structure, redirects, MIME type, size, and link health separately.
• Label blocked and timed-out sites as inaccessible, never as missing.
• Use a browser or proxy fallback when a direct fetch is denied.
• Compare suspicious HTML responses with a randomized missing-path control.
• Track AI mentions and citation sources over time so technical changes can be compared with real outcomes.

The next useful study is not another adoption count. It is a matched test: companies with and without llms.txt, the same buyer questions across multiple assistants, and observed differences in mentions, citations, accuracy, and source selection. That still would not prove causality, but it would move the conversation from belief toward evidence.

1. Check the exact /llms.txt response, including redirects, content type, and the body—not only the HTTP status.
2. If you publish one, start with a clear H1, add a short brand summary, and curate a small set of important Markdown links with useful descriptions.
3. Keep the linked pages public, canonical, current, and understandable without requiring a browser-side application to reveal the essential facts.
4. Review robots.txt separately for search, training, and user-directed AI agents. Do not assume one rule controls every use.
5. Monitor the business outcome: whether assistants mention the brand, describe it accurately, and cite first-party pages when buyers ask relevant questions.

The audit snapshot was completed on July 26, 2026. We resolved the current S&P 500 constituent securities to 500 distinct companies and official corporate root websites. We requested /llms.txt with redirects followed and retried ambiguous cases in headless Chrome. Successful HTML responses were compared with randomized nonexistent control paths. A detected file required a non-empty, non-HTML 2xx response that did not resemble a soft 404.

Our “proposal-shaped” quality threshold required a Markdown H1 and at least one Markdown link. It is deliberately stricter than the proposal’s minimum requirement, which is an H1. The S&P 100 comparison used the iShares OEF holdings file dated July 23, 2026 and combined Alphabet’s two share classes into one company.

This is a point-in-time technical census. Files, websites, index membership, and provider behavior can change. Corrections are welcome.