SEO Class in Chicago, IL
Are you a Google Analytics enthusiast?
More SEO Content
Googlebot Keeps Asking For 404 Pages
Posted 30 June 2012 - 07:15 AM
I know that I can just prevent these crawling attempts via robots.txt. But if I do that, the urls of the non-existing pages will appear in the SERPS (without snippet), when doing a site: mydomain.com search.
What is the best practice to get Google to understand, that these pages don't exist? 404 responses, robots.txt, or something else? Please share your experiences on this matter...
Posted 30 June 2012 - 10:44 AM
Posted 01 July 2012 - 04:12 PM
I'll try the 410. I'm worried about all the links from the spam pages, now where Google is penalizing sites that have these bad links (the Penguin update), and I'm not sure if 410 will change that... I have definitely seen a drop in rankings after Penguin.
Posted 05 July 2012 - 01:53 PM
Probably you should just redirect all the bad URLs to a single page that uses a "noindex,follow" robots meta tag. The page should have a link pointing to your HTML sitemap (or your root URL).
I would NOT simply redirect the bad URLs to an important page.
Posted 06 July 2012 - 04:18 AM
The 410 won't help. Google disclosed earlier this year that they are treating 410 and 404 about the same.
What John Mueller (John Mu) posted in May 2012 was this
Which hasn't really changed from this: (2009 quote from Google Groups) where a 410 response was being handled more or less correctly (according to the the RFC 2616 and the W3c Status Code definitions
However I think he is a bit 'off' in the time scale difference, we have probably all seen that a URL returning a 404 response often hangs around for months. Though the crawl frequency does start to tail off after a few weeks.
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users