Summary: four search relevance tests
Gondo's test bench, from 22 September to 9 October 2026. The detailed protocol and the reports of each test are available on request and will be published with the module.
What we measured
For a given query, does the engine find the right products, and does it place them at the top of the first 10 results?
- The score (nDCG@10) ranges from 0 to 1. It is 1 when the best products come first.
- The gap is the average difference in score between Gondo's prototype and its opponent, query by query.
- The range brackets this gap at 95%. If it does not contain 0, the gap cannot be explained by chance: we then call it a "clear" gap.
- Won / tied / lost: the number of queries where Gondo gets a better score than the opponent, the same score, or a worse one.
The data come from WANDS, a public dataset from Wayfair: real customer queries, on a catalogue of furniture and home goods, with relevance ratings given by people. Each test covers queries the prototype had never seen during its tuning, and it is measured only once.
The four tests
| Test | Opponent | Queries | Opponent | Gondo | Gap and range | Won / tied / lost | Verdict |
|---|---|---|---|---|---|---|---|
| 1 | Shopify's native search, store in English | 100 | 0.819 | 0.731 | −0.088 [−0.138; −0.041] | 31 / 26 / 43 | less relevant |
| 2 | Shopify's native search, store in French | 62 | 0.647 | 0.558 | −0.088 [−0.157; −0.019] | 20 / 6 / 36 | less relevant |
| 3 | PrestaShop 9's native search | 65 | 0.257 | 0.605 | +0.348 [+0.275; +0.421] | 55 / 4 / 6 | more relevant |
| 4 | Doofinder on PrestaShop, as installed | 46 | 0.398 | 0.491 | +0.093 [+0.026; +0.162] | 31 / 2 / 13 | more relevant |
The "Gondo" column gives the hybrid version of the prototype: it combines keyword search and search by meaning. The keywords-only version gives:
- test 1: 0.662, gap −0.157, 24 / 20 / 56, less relevant;
- test 2: 0.507, gap −0.139, 18 / 6 / 38, less relevant;
- test 3: 0.511, gap +0.254, 46 / 13 / 6, more relevant;
- test 4: 0.413, gap +0.016 [−0.059; +0.090], 24 / 4 / 18, no clear difference with Doofinder.
What we take from it
- On Shopify, the native search is already better than the prototype, in English as in French. In French, it also matches words by meaning ("divan" finds sofas there).
- On PrestaShop, the native search finds nothing for almost one query in two: 40% and 48% of the sealed queries get no results, because it requires every word to be present. The prototype answers all of them.
- Doofinder fixes this flaw: no query without results. The hybrid prototype is still more relevant than Doofinder as installed. It wins 31 queries, loses 13, and 2 are tied. Without its "meaning" part, it is on a par with Doofinder.
- Prototype response time, measured on 9 October 2026 on a 4-vCPU machine (OVH, Gravelines), corrector included, one search at a time: 23.0 ms at the 95th percentile and 70.5 ms at the 99th at the gateway; 45.4 ms and 96.7 ms end to end, from a computer in France. With five simultaneous searches, the 95th percentile rises to 104 ms at the gateway. Limits: a single machine, a single moment, a single client computer, a measurement gateway with no rate limit or token, and a single index on the machine.
The limits, to keep in mind
- Machine-translated catalogue (tests 2 to 4). The translation is automatic and done locally. A real French catalogue is less clean and less consistent.
- Competitors in their default settings. Doofinder was measured with no synonyms, no rules and no boosting. A merchant who tunes it can get better results. PrestaShop's native search kept its installation settings: fuzzy search enabled, words of 3 letters or more, original weights, no alias added.
- AI judge for part of the ratings. The results WANDS had not rated were rated by an AI, blind, without knowing which engine had found them. A second AI re-checked 25% of these ratings: 90 to 98% identical ratings, and the verdict does not change.
- Few queries, from 46 to 100, hence wide ranges. The test 4 queries had already been used on 22 September for another measurement, with no tuning on them.
- A single domain (furniture and home), and relevance only: these tests measure neither sales nor revenue.
- Typos. Without a corrector, a query with a typo keeps about 0.80 of the score of the same query without the typo (0.85 to 0.87 on a second set). With the corrector placed before the engine, adopted for version 1, it keeps 0.97 to 1.00. These figures come from two sets of 50 manufactured typos (two letters swapped, or one letter missing). They still need to be confirmed on real customer searches, with a pilot merchant.
In one sentence
On this bench, Gondo's search is more relevant than PrestaShop's native search, and more relevant than Doofinder as installed. It is not more relevant than Shopify's.