5 replies to this topic

[bwelford]

Like many, given the importance of Google to web site traffic, I have a mild interest in trying to understand what G is up to and how it may be changing.

Over the past 30 days I have been tracking how many web sites are indexed by Google. One indication of this is the number of web pages indexed in a search for the word "the". I do this by checking out such a search at the individual datacenters such as www-fi.google.com My understanding was that 7 of the 9 datacenters are independent and 2 are redirected to any one of the other 7.

In early July, 6 of the 9 datacenters (fi, dc, va, ab, cw, and in) were all about the same count and the average count was 3.65 billion web pages. Datacenter ex showed 3.35 billion web pages.

Now early in August the average count for the 6 is 3.48 billion web pages. This represents a drop of 170 million web pages over the last 30 days. This presumably is a net figure of web pages added less web pages removed from the databases.

Rather surprisingly, the original standout ex is now showing 3.42 billion web pages. Two new datacenters have been added recently, gv and gv2 and these show 3.43 and 3.44 billion web pages. So all datacenters are showing web page counts in the same ballpark.

Coincidently in the other forum where the Google watchers congregate, for the last few days people are noting a new stability with the Google results. So are we now seeing a Google that has slimmed down to what it regards (currently) as a reputable group of web pages?

Barry Welford

[Bernard]

Thanks for the post Barry. Interesting. I've got a bunch of new pages added (as I'm sure many in here do), so that probably means that there are more pages "dropped" than the simple differential you calculated. Perhaps the Google's rumored new spam filters are working?

[Jill]

Perhaps the Google's rumored new spam filters are working?

Not as far as I can tell. The spammy SERP I check every now and then is still as spammy as ever.

Jill

[bwelford]

Bernard, I believe what you are pointing out is very intriguing. Unfortunately Google says very little and GoogleGuy in the other forum speaks only in riddles. The following is back-of-the-envelope speculation. Google has said that it wishes to be indexing 10 billion web pages within the next few years. Let us assume that they currently have say 5 billion web pages indexed and wish to add another 5 billion web pages say in the next 2 years. Then on average they must add over 200 million web pages every 30 days. Instead in 30 days we see a drop of 170 million web pages for only a slice of their databases (all web pages in the English language containing the word "the"). Taken together, this implies they cut out a total of more than 370 million web pages during this 30 day period. That's more than 7% of all the web pages they have in their databases. To me, that's pretty impressive performance for their spam filters.

However as Jill pointed out they don't catch them all. However some spamming methods would seem to be very tough to catch automatically, which is the only way they can work. For example I can't think how they would spot hidden text in a font colour which is the same as the background colour in the cell of a table. The only hope, I suggest, is that they have sufficient policemen that respond to spam reports and manually remove web sites from their databases. Of course the spiders may find them again. However this may take a month or two, which hopefully is a big enough deterrent, to get the spam merchants to clean up their acts.

Google encourages reports on web sites that are in violation of their Guidelines. At the least, they can try to improve their spam filters to catch more infringers. The link for spam reports to Google is
http://www.google.co...spamreport.html
Jill, have you reported the web sites that concern you so much? What do folks think about this process?

Barry Welford

[Jill]

Jill, have you reported the web sites that concern you so much? What do folks think about this process?

Yep, a number of times. I did it again yesterday when I saw it was still spammy. I've mentioned it in my newsletter (the query is email marketing consultant) and I've mentioned it in other forums.

It's not a SERP I personally care about, nor do I have a client with that keyword phrase. I just happened to type it in once and was amazed at what I found. All different kinds of spam. Nothing to do with hidden text, however. I haven't looked closely enough at what the sites are doing, but there are redirects involved in some of the pages.

Many of the pages that come up seem to be affiliate sites that sell ebooks on the topic.

Jill

[Jill]

Split off a Google cache copywriting thread to here:
http://www.highranki...st=0

Jill