The same buying questions, one week apart: 180 AI answers

For our project-management question, Teamwork.com fell from 28 of 30 answers to 2 of 30 a week later. Hive went from zero to 28. The buyer, question, requested model and search settings stayed the same.

After the 90-answer September 21 experiment, we collected another 30 answers to each of the same three questions on September 28. That gives us 180 usable answers across two waves. We kept Gemini 2.5 Flash, Exa search, temperature 0.2 and the 1,500-token limit, with a fresh request every time.

Mean overlap between the first and second waves was 64.2% for CRM, 62.6% for email marketing and 24.5% for project management. Jaccard overlap divides the shared brands by the distinct brands in both answers together. It measures agreement between sets, not recommendation quality or ranking.

The CRM answers shifted too: Insightly rose from 5/30 to 23/30, while Copper CRM fell from 24/30 to 8/30. Mailchimp went from 17/30 to 8/30 in the email question. In project management, monday.com rose from 2/30 to 27/30 and Wrike fell from 25/30 to zero. These are observations for these buyer questions, not ratings or market shares.

Within the follow-up wave itself, mean overlap was 66.4% for CRM, 64.9% for email and 62.9% for project management. The project-management answers could look fairly consistent on either collection day while still naming substantially different shortlists across the week.

We cannot isolate the cause. Search results, website content, model behavior and sampling can all contribute; this experiment did not separate them. Two collection dates cannot establish a long-term trend or tell every business how often it should monitor.

The follow-up made 93 requests and retained 90 complete answers. Two timed out and one reached the output limit. Recovery collection increased the timeout from 45 to 90 seconds and filled the sample with fresh requests. Collection dates use Japan time; exact UTC windows are published with the data.

We also preserved source URL metadata and measured its overlap separately from brand sets. A linked page is not proof of causal influence or support for a claim. Name coding was AI-assisted and reviewed against saved answers, without independent human double-coding.

Known API spend for the follow-up was about $0.92. Including the original collection and pilot, known spend is about $1.87. We retained a further $1 reserve for the two unknown-cost timeouts, keeping the study within its $5 allowance.

The sections and figures below preserve the first 90-answer wave. They describe variation within that original collection day; the follow-up above adds the comparison across dates.

Pipedrive appeared in 30 of our 30 CRM answers. GoHighLevel appeared in 1. A screenshot could show either brand on a shortlist. It would not tell you how often that happened.

We sent three buying questions to Gemini 2.5 Flash through the OpenRouter API, with Exa web search enabled. Each question asked for five software options. We repeated it 30 times, keeping the wording and settings fixed and starting a fresh request each time.

The 30 CRM answers produced 8 different sets of brands. The email question produced 10, and the project-management question produced 8. Those totals ignore order: moving the same five names around does not create a new set.

This was a same-day API experiment. The original wave did not test consumer apps or change over time. The September 28 follow-up adds a second collection date. The Japanese edition reports the same English-language experiment.

We chose three small-business software purchases. Each question named a US buyer and a concrete team or audience size, but no brand. These were questions we wrote for the study, not a representative sample of real search queries.

1. Which CRM tools would you recommend for a US consulting firm with 10 employees? Recommend five options and briefly explain each choice.
2. Which email marketing platforms would you recommend for a US online retailer with 5,000 subscribers? Recommend five options and briefly explain each choice.
3. Which project management tools would you recommend for a US design agency with 15 employees? Recommend five options and briefly explain each choice.

We rotated through the three questions once per round, then repeated the round 30 times. Each request contained only its own question. No previous answer was sent back to the model.

Thirty repeats kept the study affordable and gave us more to inspect than a single answer. It is not a threshold that makes an AI visibility score reliable in every setting.

The bars count recommendations in the numbered shortlist. A brand appearing only in an explanation, a citation URL, or a comparison with another product does not count.

Pipedrive and Zoho CRM appeared in every CRM answer; HubSpot CRM missed one. The less frequent names shared the remaining places. HoneyBook and Insightly each appeared five times, while GoHighLevel appeared once.

Klaviyo and Omnisend occupied two places in every email shortlist. The remaining three places varied more: MailerLite appeared 18 times, Mailchimp 17, and Brevo six. All of those answers addressed the same retailer with 5,000 subscribers.

ClickUp was in every design-agency answer, but appeared first in only 12. Six different brands appeared first across the 30 responses. A brand can be a regular recommendation without holding a fixed position in the list.

Exact matches are a demanding test. Replacing one name makes a five-brand set different, even when four names stay. Reordering an unchanged set makes the ordered list different again. We measured those separately.

Among the 435 possible answer pairs for each question, the unordered brand sets matched completely in 162 CRM pairs (37.2%), 83 email pairs (19.1%), and 85 project-management pairs (19.5%).

Requiring the names to appear in the same order reduced those counts to 35, 70, and 9 pairs, respectively. We call this first-mentioned order, not a verified preference ranking: the prompt did not ask the model to rank the options.

We also calculated Jaccard overlap: the number of brands shared by two answers divided by the number of distinct brands across both. Two five-brand lists with four names in common score 4/6, or 66.7%. Identical sets score 100%.

Average overlap was 76.2% for CRM, 63.4% for email marketing, and 69.3% for project management. That captures partial agreement which an exact-match count discards.

We also compared the source URLs in OpenRouter’s citation metadata, separately from the recommendation lists. We removed URL fragments and trailing slashes before comparing sets, but kept different pages on the same domain separate.

Mean source-URL overlap was 77.0% for CRM, 70.8% for email marketing, and 84.9% for project management. Every question had metadata for all 30 answers. The downloadable answer file includes the URLs.

These are provider-reported sources. They do not prove that every linked page supports the product claims in an answer. We did not count a brand as recommended merely because it appeared in one of these URLs, and we cannot separate search caching from repeated retrieval.

For each buying question, I would show the brand’s recommendation count, the number of usable answers, the collection date, and the model settings. For GoHighLevel, “appeared in 1 of 30 answers to this CRM question” tells a reader more than “AI recommends us.”

I would keep different buying situations separate. A brand can appear regularly for a small consulting firm and less often for a retailer. Blending the two hides the question a marketer can work on.

I would also keep the original answers. If a count changes, the next step is to inspect what changed in the responses and cited sources. This study cannot tell us whether a website edit caused that change.

Repeated-query audits already have a research literature. The Dice Roll Method is a recent preprint on how to design and interpret them. Our contribution here is smaller: three explicit questions, fixed settings, and the answers behind every count.

We collected the main sample on 2026-09-21, from 05:55:25 to 06:06:21 UTC. It contains 90 completed answers: 30 per question. Three earlier pilot answers checked the request format and are excluded from every chart and frequency above.

The request used google/gemini-2.5-flash through OpenRouter, temperature 0.2, a 1,500-token output limit, and reasoning disabled. There was no system prompt or conversation history. The legacy web plugin used Exa with up to five search results per request. Requests ran sequentially. The exact request manifest is included with the data.

We used AI-assisted extraction and reviewed every numbered recommendation heading against its source answer. We normalized product-name variants to the same label, preserved the order in which they appeared, and excluded incidental mentions. The coding file records the original headings so those decisions can be checked.

OpenRouter reported $0.9116 for the main sample and $0.0304 for the three completed pilots. Charges cover the API experiment, not the separate work of preparing or writing the article.

This is a small, deliberately chosen sample of English-language software questions about US businesses. We did not vary phrasing, language, model, or temperature. Search results and provider caching may also create dependence between responses. Fresh requests do not prove statistically independent observations, and the 435 pairs for a question reuse the same 30 answers.

We measured which names appeared, not whether the recommendations were good or the product descriptions were accurate. The original wave did not isolate model sampling from search variation or test an SEO intervention. The results do not establish a universal frequency for any brand.

To investigate your own brand, start with the buying questions you care about. CitationWorks tracks brand mentions and citations so you can inspect the answers alongside the counts.