Outdated Google Cache, How can I make Google update its cache Options

[Jupiter]

We launched a site in August 2005. The site was indexed by the main search engines fairly quickly, and started showing good results in MSN and Yahoo. Google, however, placed the site in the supplemental index. I enquired about this and it was brought to my attention that there was probably a duplicate content issue; hence the site was placed in the supplemental index by Google.

The site is built in OS commerce and contains less than 250 products. Initially the site worked for both www dot siteurl dot co dot uk and siteurl dot co dot uk. Once the duplicate content issue was brought to my attention the oscommerce engine was modified and now all pages on the site show up with the www (the homepage still comes up without the www IF the url without the www is typed directly into the browser. Even then once the user clicks anywhere the pages load with the www in the url).

Six months have passed and the site still won’t appear in Google SERPs. When I search for site:www.url Google shows just the homepage, and that is cached on the 15th of August. When I search for site:url (without the www), Google throws up 1200 pages, all cached between 12-15 of August (by comparison MSN sees about 60 pages and Yahoo 15).

I can see in the statistics that Googlebot is visiting the site regularly (60 hits so far this month), but does not update the cached copies. A Google sitemap was submitted sometime in September, and that is also working properly and being indexed by Google regularly, but the cache is still not being updated.

Is there anything I can do to make Google drop the current cache of the site and re-index it afresh? I am thinking of placing a robots.txt file blocking access to Googlebot and removing the sitemap. Will that make Google drop the cache? If so will removing the robots.txt file after Google has dropped the cache enable Google to re-index the site?

Any help on the matter will be greatly appreciated.

P.S. I noticed yesterday that the site was dropped completely from MSN, but I have read in forums that since January a lot of sites have been dropped, and then re-included as there is an update going on atm. Is there anything I should be worried about? Yahoo is going strong as it has been all this time.

[Randy]

There's not much you can do to force them to update anything Jupiter. That said, a couple of things to look into...

• Do you have some good links pointing to your site? The more Important you can make your site in their eyes, the better.
• Have you double checked to make sure you don't have any technical issues?
• Have you double checked to make sure the spiders are able to cache the page?

Other than those things, remember also that Google is in the middle of an update. So it's going to be a bit tough to get a gauge on exactly what they're doing at the moment since the datacenters may all contain different information.

[Bri]

How old is the site? 6 months?

[mpikounis]

Could this be a sandbox issue? Google wont rank new sites immediately - it may take up to a year.

[Jupiter]

Hi all, and thanks for the replies.

Could this be a sandbox issue? Google wont rank new sites immediately - it may take up to a year.

I do not know

How old is the site? 6 months?

Just over 6 months, it was launched in August 05 (early)

    *  Do you have some good links pointing to your site? The more Important you can make your site in their eyes, the better.

Only a couple atm, I have submitted several requests but it takes ages for links to appear (dmoz still haven't listed the site after 5-6 requests over the last 6 months - I recently noticed that the relevant category hasn't got an editor! - all I can think of next is paid links)

* Have you double checked to make sure you don't have any technical issues?

Like? could you give me an example? The site works fine, loads quickly etc. It comes up in Yahoo for the targeted keywords (and MSN until recently)

* Have you double checked to make sure the spiders are able to cache the page?

How can I do that? I can see that spiders are visiting the site, but how can I check caching?

[Jill]

Welcome Jupiter! (IMG:http://www.highrankings.com/forum/style_emoticons/default/bye1.gif)


See the [url=http://www.highrankings.com/forum/index.php?showtopic=12535]aging delay thread[/url].

[Dave]

It could easily be an ageing delay as others have said.

However, I had this issue with a site a while ago. Almost identical, but it followed a site rebuild on an established domain so it couldn't have been an ageing delay. After reading loads of advice and banging my head against a wall for a bit, I came to the conclusion it was a duplicate content thing.

Out of desperation I did a few things at once.

Unfortunately this means I don't know which thing fixed the issue.

Anyway, you might want to tick these things off the list before you sit back and wait for Google to finish it's update. Hey - at least you'll be doing something

1. Check that the same product is not getting spidered in multiple categories, causing a duplicate issue. This may also apply to category pages.

2. Do a Google (xml) sitemap and make sure each product is only listed once.

3. Search for a few bits of your unique content to make sure no-one's nabbed it for their own site.

4. (I woudn't do this one right now as it means changing all of your links throughout your site, plus changing all of your urls mid update = danger)

But if you're still having the same trouble in a month or two, check for url parameters that contain "id" (e.g. "productid, "cat_id", etc) and url/ mod rewrite em out of there.

Hope this helps. Feel your pain.

[Jill]

The things Dave mentioned are always good ones to check. However, it doesn't sound like there's an indexing problem but a ranking one, correct?

In which case, most of those things wouldn't help.

[Jupiter]

Hi jill and Dave.

However, it doesn't sound like there's an indexing problem but a ranking one, correct?

I'm not quite sure to be honest. The site was fully indexed soon after it went live, however, that index has not been updated since (august 05). How often does Google update its index pages?

I would be happy if the site was ranked even at the very bottom, I could then work on improving the rankings, but it does not appear at all in the SERPs (the Google ones). Only yesterday I managed to see one page in the SERPs, but that was for an obscure querry for which Google returned just a handful of pages, and my page was drawn from the supplemental index.

I am aware of the aging issue, but I dont want to find out in a few months that it was not that and there is another issue causing the problem.

A sitemap was posted in September containing all products (once each), but that did not have any effect. According to Google the sitemap is scanned every few hours, but the index has not been updated.

That is why I thought of trying to force Google to drop its current cache of the site and start afresh. Can I not achieve that by blocking Googlebot with a robots.txt disalow command? Wouldn't Google drop the cache in say a month after access has been denied?

Of course I would not do that in the middle of the update which you brought to my attention.

Thank you both for the support and Jill, glad to be here (IMG:http://www.highrankings.com/forum/style_emoticons/default/hi.gif)

I have been visiting these forums for quite some time now and though it was about time I joined (IMG:http://www.highrankings.com/forum/style_emoticons/default/smile.gif)

Dave, I will go down your checklist, although I doubt any of this is happening.

[Jill]

It sounds like it's probably a links problem where Google just doesn't feel the site is worth it's trouble. You said:

Only a couple atm, I have submitted several requests but it takes ages for links to appear (dmoz still haven't listed the site after 5-6 requests over the last 6 months - I recently noticed that the relevant category hasn't got an editor! - all I can think of next is paid links)

I think you need to start thinking about how you can get some real links. Have you read our [url=http://www.highrankings.com/forum/index.php?showtopic=5941]Link Building Articles[/url] yet?

[davidfromraleigh]

Welcome, Jupiter (IMG:http://www.highrankings.com/forum/style_emoticons/default/thumbup1.gif)

You'll get a lot out of HR Forum.

David
(IMG:http://www.highrankings.com/forum/style_emoticons/default/biggrin.gif)

[Jupiter]

It sounds like it's probably a links problem where Google just doesn't feel the site is worth it's trouble.  You said:
I think you need to start thinking about how you can get some real links.  Have you read our [url=http://www.highrankings.com/forum/index.php?showtopic=5941]Link Building Articles[/url] yet?

I have, I think I will start with a few reciprocal links and maybe a few paid ones from niche directories and a couple of the major ones.

I was looking at another site and noticed that out of the 650 links Google can see for the site, roughly 70-80% come from pages within the site itself! How do they do that? I dont suppose that Google counts the internal navigation as external links.



Thanks David, I'm sure I will. As a matter of fact I have already!

[Jill]

I have, I think I will start with a few reciprocal links and maybe a few paid ones from niche directories and a couple of the major ones.

I'm pretty sure those articles weren't really talking about reciprocal links and directories...

[torka]

I dont suppose that Google counts the internal navigation as external links.

Actually, Google (and all the other SEs) rank pages, not sites. And they're no better than humans at figuring out all the nuances of what, exactly, constitutes a "web site." Actually, as things stand right now, they're probably somewhat worse than humans at figuring it out. (IMG:http://www.highrankings.com/forum/style_emoticons/default/smile.gif)

So if the links would "count" if they came from a page that's not on your site, they will also "count" pretty much the same if they come from a page you do consider part of your site. In the context of SEO, "external links" are links that come from other pages, not necessarily from other sites.

That's why good internal navigation is so important from an SEO standpoint. (Of course, it's also good from usability and accessibility standpoints, so it's pretty much a good idea all the way around (IMG:http://www.highrankings.com/forum/style_emoticons/default/thumbup1.gif) )

--Torka (IMG:http://www.highrankings.com/forum/style_emoticons/default/mf_prop.gif)

[Jupiter]

I'm pretty sure those articles weren't really talking about reciprocal links and directories...

I know, I know... but this is a small ecommerce site selling women clothes and I have real trouble in seeing how I can build quality content for it. Apart from the product descriptions and the standard disclaimers and other info pages there isn't much else (there is a facility for product reviews, but I can not say it has been overly popular); hence I am not expecting to see links start developing on their own any time soon (IMG:http://www.highrankings.com/forum/style_emoticons/default/sad.gif) .

That leaves us with directories and press releases.

I am considering a press release and looking into sites such as pr newswire, but I am being a bit sceptical as to the newsworthingness of a PR about such a site.

I have tried free directories with very limited success and have concluded that it is not worth pursuing these any more (extremely low effort/result factor, and since time is money I see it as either pay with money and get more or less guaranteed results, or pay with time and...). Hence all that is left (As far as I can see), is directories that require a link back (reciprocal), or paid listings; unless I have missed something...

That's why good internal navigation is so important from an SEO standpoint. (Of course, it's also good from usability and accessibility standpoints, so it's pretty much a good idea all the way around thumbup1.gif)

1000000000% with you Torka, but...

Shouldn't in that case Google see/report for every site at least as many links as the pages of the site (assuming that the global and section navigation links are visible to Google - as is the case with this site - text links - lynx tested)? This is certainly not the case with most sites I have come across. I understand what you are saying, but I am a bit intrigued as to why some sites seem to perform better in this area than others (always assuming accessible/visible/consistent links throughout).