Different languages, different shortlists: 71 answers about Japanese businesses

The English and Japanese shortlists barely overlapped in this follow-up, even after we aligned the buying conditions. Some of that difference concerns how specifically an answer names a venue, so we report name overlap rather than treating every different label as a different business.

On September 28, we reran six scenarios covering coworking, wedding venues and web agencies in Tokyo and Osaka. We corrected wording differences from the pilot, then asked each English and Japanese question three times using Gemini 2.5 Flash Lite and GPT-4.1 mini. Both API configurations used Parallel turbo search through OpenRouter.

71 of 72 requests completed. One Gemini Japanese answer for the Tokyo wedding scenario timed out and was not retried. Of the 12 scenario/configuration groups, eight had all six answers and sufficiently resolved names for the overlap analysis.

Across those eight groups, mean English/Japanese name overlap ranged from 0% to 11.1%. Mean overlap between repeats in the same language ranged from 42.9% to 100%. In these observations, the cross-language gap was larger than the variation between same-language repeats. Each language had only three answers per group; comparisons reuse those answers and are not independent experiments.

Only two Gemini groups qualified: Osaka weddings and Tokyo web agencies. Their cross-language overlap was 7.4% and 0%, respectively. We omitted incomplete or ambiguously coded groups instead of assigning them zero.

We merged clear aliases such as “fabbit京橋” and “fabbit 京橋.” We did not merge a generic chain with a particular branch, or automatically collapse a named ballroom into its hotel. The measure describes named recommendations at the recorded venue granularity. It is not the percentage of different businesses or owners.

We kept URLs visible in the answer separate from URL metadata supplied by the provider. Those fields describe links accompanying the response. They do not establish which source caused a recommendation or whether it supports every claim.

For a business selling to customers in both languages, these results give a practical reason to test equivalent buying questions in both. An English shortlist was a poor stand-in for the Japanese shortlist in this sample. We did not test whether translating more pages would change the recommendations.

Question alignment and name coding received AI-assisted bilingual review, not independent native-speaker review. Subtle language differences may remain. Known provider spend for this extension was about $0.19; the timed-out request remains an unknown cost with a separate reserve.

The sections and figures below preserve the original pilot. Its wording and API setup differ from the follow-up, so we do not pool all 107 answers as one comparable cohort.

Our English answers about Tokyo wedding venues named The Okura Tokyo, Hotel Chinzanso Tokyo and Tokyo Kaikan. Japanese answers included THE Tender HOUSE and International House of Japan. Some names, including The Ritz-Carlton, Tokyo, appeared in both languages.

We wrote six buying or booking scenarios in Tokyo and Osaka, asked each in English and Japanese, and repeated each question three times. The result is 36 answers from one web-grounded Gemini 2.5 Flash Lite API configuration. We did not compare multiple assistants or their consumer apps.

Four scenarios, covering 24 answers, qualified for the name-overlap comparison. Tokyo coworking and Tokyo web agencies scored 0%; Tokyo wedding venues scored 7.4%, and Osaka wedding venues 11.1%. Two scenarios were excluded because their English shortlists contained duplicates or unresolved entries.

The scenarios cover coworking spaces, wedding venues and web agencies, with one Tokyo and one Osaka scenario in each industry. Each question asks for five named options and source links where available. Geography and buyer intent were matched as closely as the wording allowed.

Where an English question required English-language service, its Japanese counterpart did too. There are still wording differences: “central Tokyo” became “the central part of Tokyo’s 23 wards,” and “host about 80 people” became an 80-person wedding reception. This is an exploratory comparison, not proof of a pure language effect.

A changed answer is not, by itself, evidence of a language effect. We also compared the three English repeats with one another and did the same for Japanese.

A 100% score means the coded name sets match; 0% means they share no names. Answer order does not matter. Each answer appears in several comparisons, so the pair counts are not independent experiments.

English names, Japanese names, capitalization and parenthetical descriptions can create false differences. We recorded the recommendations in each answer and assigned explicit canonical identifiers, with the coding decisions available for inspection.

Some answers repeated a candidate or supplied a generic description instead of a named business. Requesting five options did not guarantee five distinct businesses. The evidence includes those exclusions and the resulting comparison counts. We did not independently verify every business’s existence, location or suitability.

We separated URLs visible in the answer from URLs supplied in provider metadata. Hostname overlap describes the linked sites. It does not establish that those pages caused the recommendations or support each explanation.

For a business serving customers who search in both languages, the observed differences are a reason to check both. This experiment cannot show that translating a website will cause the model to recommend it more often.

Collection ran on September 26, 2026 through OpenRouter using google/gemini-2.5-flash-lite, the Parallel web plugin in turbo mode with up to five results, temperature 0.2, a 1,200-token output limit and no conversation history. The frozen questions, settings, coded names and answer text accompany the report.

Japanese wording received AI-assisted bilingual review, not native human review. These six researcher-written scenarios are a convenience sample rather than measured search demand. One configuration on one day cannot establish how every assistant recommends Japanese businesses.