# Software pricing and feature accuracy audit — protocol

Prepared 2026-09-28, before answer collection.

## Question and sample

How reliably do two low-cost, web-grounded API configurations answer concrete software buying questions? Purposive sample of 50 SaaS products. This is not a random sample of software or a benchmark of consumer ChatGPT/Gemini interfaces. The primary evidence is dated official pricing and documentation, frozen before provider collection. Inaccessible or ambiguous official facts stay unknown; a product is not replaced because its answer is inconvenient.

Each product has one named paid plan, an explicit currency/billing basis/unit, and one specific plan feature or limit. One English question asks both facts. No expected answer or excerpt is given to the model. The same question goes to Gemini 2.5 Flash Lite and GPT-4.1 mini through OpenRouter with Parallel turbo web search (up to five results), temperature 0.2, maximum 1,600 output tokens, no conversation history. Gemini reasoning is disabled. One answer per product/configuration, interleaved by product. This tests a snapshot, not repeatability. No model ranking is inferred from one observation per product.

The final frozen manifest records the exact questions and configuration. Ground-truth snapshots include observation date, source URL, and reviewer notes. Official pages can change, vary by location or expose both annual and monthly prices; unresolved conflicts stay outside the scored denominator.

## Coding

Code the two requested facts separately (price and feature), using the full answer's qualifications. Do not grade extra unrequested claims. Five mutually exclusive outcomes:

- **Correct:** explicit answer agrees with the frozen official fact, including material billing and plan conditions. Equivalent annual totals are accepted.
- **Incorrect:** explicit answer contradicts the official fact. A monthly-billed price presented as the annual rate is a contradiction; a correctly qualified alternative is not.
- **Outdated:** contradiction with current evidence AND dated official historical evidence showing the answer was previously true. Never infer outdated merely because a number looks familiar.
- **Unsupported:** a claim is made but collected official evidence cannot resolve it, including conflicting or inaccessible reference material. This is not a factual error.
- **Unanswered:** the requested fact is omitted, the model abstains, or the answer is unusable. This is not a factual error.

Scored claims = correct + incorrect + outdated. Contradiction rate = (incorrect + outdated) / scored claims, with raw counts and unscored outcomes adjacent. Report pricing and features separately, and counts by API configuration. Never divide by all claims while silently treating missing evidence as correct or false. A product/answer with a contradiction is a separate unit from a claim; label denominators explicitly. No significance test or population error rate is warranted.

AI-assisted claim coding is reviewed against saved answers and primary evidence. The authoring agent checks disputed examples and numeric summaries. This is not independent human double-coding. Publish the coding sheet, questions, sanitized answers and source links; keep full third-party page archives out of public downloads. Short excerpts and factual paraphrases suffice.

## Budget and durability

User allowance: about US$5 for this study. Hard collector cap $5, with $0.25 reserved before each sequential request and provider-reported costs logged for every attempt. Persist pending requests before dispatch. No automatic retries. Unknown costs block further collection; they are reserved, never zero. Prior runs, failed attempts and any recovery collection must share this study's allowance. Codex subscription/agent usage is separate and not metered by this provider ledger.

## Publication and interpretation

Publish whichever result the evidence supports. A small error count is still a finding. Avoid claims about lost revenue, ranking penalties, deliberate vendor deception or a measured benefit of CitationWorks. Explain the practical check: compare the buyer-facing answer with your current price, billing basis and plan entitlement. Link to the broader research and product only after the findings. Both English and Japanese articles share the same numeric evidence and limitations.