A 200 Status Does Not Mean Your llms.txt Exists

We requested /llms.txt on the corporate root website of all 500 companies in the S&P 500.

110 of them answered with a 2xx status code. Every one of those responses would pass a checker that stops at the status line.

Only 71 of the 110 were an actual text file. The other 39 were something else wearing a 200.

If you have ever pasted your domain into a free AI-readiness tool and seen a green check, this is the number to sit with.

HTTP 200 means the server produced a response. It says nothing about whether the response is the thing you asked for.

The 39 false positives sorted into five patterns.

• 24 redirected away from the path. The request for /llms.txt ended somewhere else entirely, with a 200 on the page it landed on. Procter & Gamble ended at /404/. Palantir ended at /404. Exelon ended at an errors path. Textron ended at /access-denied. UnitedHealth Group ended at page-not-found.html. Hasbro ended at a Japanese-locale where-to-buy page. All of them returned 200.
• 7 served a generic HTML fallback at the exact path. The URL never moved. The body was simply the page this site hands to every unknown URL, as we confirmed with a control request. FedEx, Charles Schwab, Fifth Third Bancorp, EOG Resources, Church & Dwight, W. W. Grainger, and Ulta Beauty landed here. Ulta's response was 15 bytes long and said, in full, "Be Right Back."
• 4 returned HTML at the exact path that was not the generic fallback: Ametek, Evergy, Flex, and Live Nation Entertainment. Different page, still not a text file.
• 3 returned an HTML page that said in its own words that nothing was found: Carnival, Lumentum, and Sysco. A soft 404, dressed as a success.
• 1 returned a real text file with nothing in it. Sherwin-Williams served text/plain, 200, zero bytes.

Notice what these have in common. Nobody did anything wrong on purpose. This is ordinary web infrastructure doing what it was configured to do years before llms.txt existed.

Which is exactly why a checker cannot trust the status code. The status code was never about your file.

It would be tidy to declare every redirect a failure. The data does not allow it.

24 of the 71 real files also ended at a URL different from the one we requested. Most of that was the boring and correct kind: http to https, or a bare domain to www. Archer Daniels Midland, Nucor, Paychex, and Leidos all redirected and all served a genuine file.

Arthur J. Gallagher is the case worth remembering. The real file lives at /llms.txt with a trailing slash, and it is 51,165 bytes of legitimate Markdown. Our first pass called it a soft 404. It was not.

So the question is never "did it redirect." The question is whether the redirect ends at your file or at your marketing site.

A false positive is embarrassing. A false negative is worse, because it tells a company something untrue about its own website.

94 of the 500 sites never gave us an answer we could trust: 64 access denials, 27 network failures, and 3 repeated server errors. We did not count a single one of them as missing. We counted them as inaccessible, which is a statement about our measurement, not about their site.

That distinction earned its keep. Our direct HTTP pass found 67 files. A second pass through headless Chrome found four more that the first pass had written off.

• T-Mobile blocked the direct request with a 403. The browser found a 22,368-byte file with a title and links.
• Essex Property Trust rate-limited us with a 429. The browser found an 8,144-byte file with a title and links.
• Arthur J. Gallagher looked like a soft 404 from one vantage point and was a 51,165-byte file from another.
• Norfolk Southern returned an empty response to the scraper. The browser got the file. It is one byte long.

Four out of 71 files, or 5.6% of everything we found, were invisible to a competent scraper on the first try. If your monitoring tool reports "missing" the moment a WAF says no, it is guessing and calling it data.

Here is the trick that resolved the ambiguous cases, and it costs one extra request.

When /llms.txt comes back as HTML with a success status, ask the same origin for a path that cannot possibly exist. Something random, on the same host, with the same extension. Then compare the two responses.

Same final URL, same normalized body, or the same title at nearly the same size? You are looking at the site's universal fallback, and your file is not there. Different response? Then something specific is being served at that path and it deserves a closer look.

This is the difference between reading a status code and testing a claim. A negative control turns "the server said 200" into "this path behaves differently from a path we know is fake."

One extra request settled all 11 of the cases where the path returned HTML, had not moved, and did not announce itself as an error. Seven turned out to be the site's universal fallback. Four were something else at that path, and still not a file.

1. Request your file and follow redirects while printing the headers. With curl that is curl -sSL -D - -o /dev/null https://yourdomain.com/llms.txt. Read the final URL, not just the last status.
2. Look at the content type. In our audit, 64 real files were served as text/plain and 7 as text/markdown. If yours says text/html, you are serving a web page, not a file.
3. Look at the first 200 bytes of the body. If you see a doctype, a nav bar, or the words "page not found," you have your answer.
4. Request a fake path on the same host, like /this-does-not-exist-9f2a.txt. If it looks like your llms.txt response, your llms.txt response is not real.
5. Check the size. Ours ranged from 1 byte to over a megabyte. A one-byte file passes an existence check and helps nobody. A megabyte-scale dump recreates the problem the file was meant to solve.
6. Repeat the check from outside your office network and outside your CDN's home region. 64 of the 500 sites denied us access, and most of those companies almost certainly believe their pages are publicly reachable.

None of this requires a tool. All of it is the work most tools skip.

We rebuilt our own thinking around this audit. A file check should produce evidence, not a verdict.

• The requested URL and the final URL, kept separate.
• The content type and whether the body looks like HTML.
• The response size, and whether it was truncated.
• Whether a randomized control path on the same origin returned the same thing.
• The measurement method, because a direct fetch and a browser fetch are different experiments.
• The state, in plain words: real file, HTML page at that path, redirected away, confirmed absent, or could not be measured. Five states, not two.

And one more piece of honesty about the file itself. The llms.txt proposal requires an H1. It does not require Markdown links, because file-list sections are optional. When we call a file "nonstandard," we mean it missed our own stricter quality bar, not that it violates a specification. 17 of the 71 detected files fell short of that bar, and 14 of them missed it for exactly one reason: an H1 with no links at all.

Two more are the reverse. Salesforce and U.S. Bancorp published files close to a megabyte, full of links, with no H1 at the top. Size is not curation. The seventeenth is Norfolk Southern's one-byte file, which is technically present and helps nobody.

Everything above is about measuring a file correctly. It is not an argument that the file changes your business.

We have no evidence that publishing llms.txt causes an AI assistant to mention you, cite you, or describe you accurately. Google says Search ignores it, including for generative features. The proposal itself promises a curated map, not a ranking effect.

Publish a clean file if it is cheap and useful. Keep your important pages crawlable and understandable. Then go measure the thing that actually pays: whether assistants mention your brand, get your facts right, and cite your own pages when buyers ask.

A green check on a status code is the easiest number in this entire category to fake. Do not build your AI visibility program on it.

The snapshot was completed on July 26, 2026 against 500 distinct companies, represented by 503 S&P 500 securities after combining multiple share classes. We requested /llms.txt at each company's corporate root website with redirects followed, then retried ambiguous and failed cases in headless Chrome.

Of the 500 checked websites, 110 returned a 2xx status, 295 returned 404, 1 returned 410, 64 returned 403, 3 returned a repeated 5xx, and 27 returned no response at all. A file counted as detected only if the response was a successful, non-empty, non-HTML text body that did not behave like the site's randomized control path. That produced 71 detected files, 330 confirmed absent, 94 inaccessible, 4 HTML responses at the exact path, and 1 empty file.

This is a point-in-time technical census of public files. Websites, security configurations, and index membership all change. If we got your company wrong, tell us and we will correct it.