When AI gets the plan wrong: a 50-product software audit
18 of 200 pricing and feature claims contradicted official sources. We checked the answers against named plans, billing terms and documented limits.
Unsupported and unanswered claims are not errors. Outdated requires dated official evidence that the answer used to be true.Data behind this figure ↗
- products
- 50
- answers
- 100
- targeted claims
- 200
API snapshot, September 28, 2026. Not a benchmark of consumer apps.
Inspect every answerA familiar product, an unfamiliar limit
Ask how many CI/CD minutes come with GitHub Team, and both API configurations in our study gave the same answer: 2,000 per month. The official page we checked listed 3,000. They named the right product and plan, then supplied a different entitlement.
We asked about one price and one feature for each of 50 software plans. Across 100 answers and 200 targeted claims, 153 matched the official reference and 18 contradicted it. We left 22 unsupported claims and 7 unanswered questions outside the error count.
The mistakes concerned details a buyer would use to compare plans: included minutes, storage limits and staff seats. A correct product name did not guarantee a correct description of the plan.
What we tested
We selected 50 software products across work management, development, marketing, accounting and other categories. This was a deliberate sample, not a random survey of the software market. For each product, we recorded a named paid plan, a USD price and billing basis, and one concrete feature or limit from official pricing or documentation.
We sent the same English question once to Gemini 2.5 Flash Lite and once to GPT-4.1 mini through OpenRouter. Both configurations used Parallel turbo web search, up to five search results, temperature 0.2 and a 1,600-token output limit. Each request started fresh. Neither model received our expected answers.
These are two specific API configurations. They do not reproduce the consumer ChatGPT or Gemini apps. The Japanese edition reports this same English-language experiment.
Prices and features need separate denominators
Of the 100 price claims, 78 could be scored and 8 contradicted the reference. Of the 100 feature claims, 93 could be scored and 10 contradicted it.
50 selected products, two configurations, one answer per product/configuration. These are sample observations, not an estimate of all AI answers.Data behind this figure ↗
18 of the 100 answers contained at least one contradiction. That is a different measure from the claim count: each answer supplied two facts we wanted to check.
Three differences a buyer would notice
| Plan / feature | API answers | Official reference |
|---|---|---|
| GitHub Team · CI/CD | 2,000 / month (both configurations) | 3,000 / month |
| Box Business · storage | 5,000 GB / 100 GB | Unlimited |
| Shopify Basic · extra staff | 0 / 2 | 0 |
Reference collected for September 28, 2026. The answer column shows the two configurations; the evidence includes each response and coding reason.
Inspect the evidence ↗We chose these examples because the disagreement is easy to inspect. The full evidence table includes the correct answers too, along with every question, response and coding reason.
An uncertain reference is not an AI error
Official pricing is not always easy to pin down. Geography, tax, billing toggles, promotions and dynamic pages can leave several plausible numbers in an extract. “Unsupported” means our collected official evidence could not settle a claim. It does not mean the answer was false.
Before collection, our frozen reference marked 40 prices and 49 features as verified. A later review found that the Dropbox extract showed both billing-cycle selectors without establishing the selected cycle. We excluded that price from scoring and recorded the amendment, preserving the original reference and questions.
The effective reference covers 39 prices and 49 features. Even where a reference exists, an answer that declines to give a definite fact stays unanswered. We also required dated official historical evidence before calling a contradiction outdated. No claim met that additional requirement in this coding.
What software teams can check
If your product appears in AI answers, read the details around the mention. Start with the plan name, billing cycle, price unit and the limit your buyer cares about. Those checks catch a different problem from simply being absent from the shortlist.
On your own site, distinguish monthly billing from the monthly equivalent of an annual commitment. Put a feature limit beside the plan it belongs to. That makes the offer easier for a buyer to verify. This experiment did not test whether those changes improve AI answers or revenue.
This research is not a demonstration of automated factual verification by the CitationWorks product. It is a reason to choose useful monitoring questions, inspect the actual answers and compare material claims with current official information.
Data, cost and limits
We collected the answers on September 28, 2026. All 100 requests completed. The provider reported about $0.16 in collection costs; that is API spend, not the total cost of research, writing and review.
Source inspection and claim coding were AI-assisted, followed by a second AI review. There was no independent human double-coding. One answer per product/configuration cannot establish a model ranking, measure changes over time or estimate an error rate for the entire software market.