16 replies to this topic

[jpmuofu]

Hello All,

Hopefully this is the right category for this type of a question, if not I apologize in advance.

I have a client that until about two months ago was fully indexed in Google but is no longer indexed by Google. This site has had no purposeful SEO work done on it. There has never been any link recruiting, buying, exchanging and the like done for it and there has been no content or code optimization done for it.

That being said, because this is a publicly traded company the site does have quite a few backlinks and ranked well for very specific keyword terms.

From what I can tell there is no reason why this site should be banned. In an effort to get the site reindexed we have added a Google Sitemap and Google Analytics to the site. The Google Webmaster tool shows that the site map is good but says that Google has not yet indexed the site, it does not report that the site is banned. This tool also shows that the robots.txt file is not blocking access. The weirdest thing is that Google is still showing statistical results like average page ranking, ranked keywords etc... for a site they claim has never been indexed.

I have waited for about a month and a half now hoping that they would eventually fix a glitch or ommission in the index but to no avail.

I would appreciate it if anyone could give me some feedback about why this site may have been excluded/banned from the index so that I could make some changes and re-submit to the Google Reinclusion tool.

URL: www.iflo.com

Thank you for your help.

[Jill]

Sounds like there are perhaps no other sites "voting" for it by linking to it. Is that the problem?

[jpmuofu]

Yahoo and MSN link: searches show 179 and 468 backlinks links to the home page respectively. A Google site: search on the URL produces no results and until 2-3 months ago the site was fully indexed and well ranked on some keywords.

So I guess in answer to that question I would say no I don't think this problem is related to having "no votes," for it.

Any other thoughts?

[qwerty]

What kind of robot activity are you seeing from Google?

[jpmuofu]

Looks like we haven't seen GoogleBot in our logs since March 2006. GoogleBot Image lasted visited in June.

[qwerty]

In other words, it hasn't succeeded in crawling any of your pages in all that time or it has attempted to -- You're not even seeing any error codes?

[Randy]

How long ago was the redirect from the www.domain.com address to simply domain.com put into place?

Not that this is bad, mind you. You want them to do one or the other, and you're doing it with a 301, which is the correct method.

However if all of the pages were indexed with the www. address previously, none of those pages now exist. Your server is telling the engines that.

If this is the problem will it correct itself? Yes, eventually. But it'll take 4-6 months for the engines to catch up.

If it won't mess up anything else, you could force the www.domain.com address, doing exactly the opposite of what's now being done. Then the site might pop back into the Google SERPs pretty quickly.

But whatever you do, you'll want to pick one address or the other and stick with it. Changing back and forth can cause some real heartbreak.

[jpmuofu]

We actually just changed that this afternoon. On another forum someone suggested that since we were accepting the domain with and without the www that Google may think we were producing duplicate content. So we decided to force the user with a 301 to the no www domain.

So this particular issue should not have caused the all out ban on Google.

Any other thoughts?

[jpmuofu]

qwerty,

We have seen Google Sitemaps spider visit the site over the last month 24 different times.

As far as I can tell from our analytics the numbers I quoted above included all visits from GoogleBot.

[incrediblehelp]

Just to be sure, you never had any errors in your robots.txt in the past that would keep Googlebot out have you?

Do you have all your pages or just some in Google sitemap?

[Randy]

That's a very good point about the robots.txt Jaan.

I'm not sure how the engines or Google specifically would react to what's currently in your robots.txt file. It says:

CODE

User-Agent: *
Allow:

I'm nowhere near and expert on the robots.txt standards (Alan!?!?) but I don't believe Allow: is a valid entry. Typically to allow all spiders you would use something like:

CODE

User-agent: *
Disallow:

I seem to recall somewhere that Google does read the non-standard Allow: entry. If this is so, would if follow that your robots.txt file is actually telling them to go away because it's the opposite of an empty Disallow: ? I'm not at all sure of the logic, but it would be worth a try. So you might want to either completely remove your robots.txt file, or upload a blank robots.txt or even upload one that uses the robots.txt standard with a blank Disallow: line.

[Shane]

It would be worth signing up for Google Sitemaps, too, even if you don't upload a sitemap, to see all the stats that they provide (e.g. crawl errors). They also have a robots.txt checker in there, too.

[Randy]

The site is signed up for Sitemaps I believe Shane. It's up there in the very first post.

I'm just wondering aloud if the Allow: may be triggering something in Googlebot and the Google robots.txt checker isn't designed to catch such an odd misconfiguration. Or in other words, some kind of fluke.

I've never heard of it happening before, but I don't run across people using an Allow: instruction much either.

[jpmuofu]

Thanks for the input.

We've updated our robots.txt file to the standard disallow:

Hopefully this will help do the trick.

Yesterday evening GoogleBot hit our site for the first time in 6 months so I think fixing the 301 redirect triggered something at Google. Unfortunately, it still didn't index anything so hopefully changing this error in the robots.txt file will help get us over the hump.

If anyone else has any other good suggestions please chime in, this thread has been extremely helpful for us.

[jpmuofu]

Just as an update...our site was reincluded in the Google index this week. Thanks, for all your suggestions.