Let's debunk another myth about Google : quality and consistency of results.

First, a diversion. If you read Google's Mayer comment about Google speed on providing 30-result pages instead of 10 :

extrait : "In a survey on search, Google asked people how many results they would want by default; they responded that more is better, Mayer said. So the company conducted an experiment, providing some searchers with 30 default results. But it took, on average, a half-second longer to get those results than when the default was 10 results, she said. Out of frustration, people conducted fewer searches."

I can only wonder whether anybody would trade speed for quality results. If Google spends twice more time but comes up with much better results, I am absolutely ready to take that because it means the stuff I am looking for are more likely to be not just in the first results page, but also among the first top 5 (excluding ads on the top of course).

The question that Mayer obviously skips, intentionally IMHO, is whether this potential speed versus quality trade is making them paranoid or not, and whether they think people out there would switch overnight to any search tool that competes on quality first.

It's also interesting for me that she makes not even a mention of different ways to conduct search, such as clustered search. This, despite the fact it's mandatory to disambiguate results sometimes. This is fairly documented out there, except by Google apparently...

Let's dig a little further in Google's current result quality. For the purpose of the little study (disclaimer : this is just ONE search, so don't be an ass like comScore and extrapolate the behavior to the billions of searches conducted a day), I wanted to check the results depending on the country I am logging in (the IP address can be cracked to tell I am coming from France, especially anytime I don't use a redirecting proxy located outside France), and the language of the operating system I am using.

I would expect that the results don't change with the IP address, except for China of course, because if that is the case, Google would be actively cloaking results, a practice they themselves call bad and dirty. Neither with the language of my operating system, because it does not matter which language I am using. Note that, I am not saying that Google should not take either of these into account in order to provide a localized user interface, or sponsored links from local advertisers. But if Google goes as far as returning different results, it's a whole another thing. It would be bogus. After all, what does my language change to the target website I am looking for? What does it change to the Google indexes obtained by Google bots?

Well, I am sorry to report that it's the case though. Results are different based on the two variables. This is an ugly story, which could end up with blood. You can reproduce it very easily, so you don't have to take my words.

Let's conduct a search for Robert Scoble (without quotes), here are the results from Google.fr and from Google.com respectively,


results from Google.fr, French OS


results from Google.com, French OS

Before you dismiss the case saying that it's unfair to test Google.fr, let me tell you that unless you make a specific sequence of actions, Google.com will redirect to Google.fr based on your IP address. So that meets my case.

Notice above the following differences :

• The top 3 results are not the same. On Google.fr, the top result is for Robert Scoble scobleizer, which is the best answer. Whereas for Google.com, the top result is linking to scoble.weblogs.com, which hasn't been updated for no less than 12 months. Huhh? Huh? huh! WTF?
• The number of results are also different.
• Also note that, while it's only anecdoctical, that Google.fr provides a link to Google groups among others.

If you try and reproduce the results for Google.com, whose top result is bogus, you may actually find another set of results too. Remember that, since I connect using a French IP address, I can only use Google.com from a special feature of the Google website (I had to click on a link called Google.com in English, it's not a direct link). So, potentially another set of results.

But wait there is more.

I also have an English version of Windows. And I have conducted the same search. Here are the pictures,


results from Google.fr, English OS


results from Google.com, English OS

The results are, at least on the first page, the same than with the French OS. But there is a difference, take a look at the number of results. They are different. The language of the OS affects the number of results. WTF again?

Note I haven't tried using a French version of Firefox, and combining this new variable. Neither with a non-French IP address.

Potentially there are 4 degrees of freedom here, 16 combinations, for a search as simple and straight forward as that.

I think it's fair to see that the total number of results as not entirely meaningful anyway. What matters more, at least to me, is the order of magnitude, more than the actual figure. Except if there is just a couple results, in which case, I am eager to know exactly how many there are. But it's interesting that Google still does not care much about that. You would think that for most users, this is a actually a bug. A bug that Google probably may want to fix before they can claim consistency of results.

What do we have seen? An interesting finding. Does it make me feel safe about using Google to conduct searches? Obviously not.

Pass the word...