Are you a Google Analytics enthusiast?
More SEO Content
Cleaning Up Your Bot Garbage.
Posted 25 January 2010 - 07:54 PM
My goal was to reduce the number of indexable urls google or any other bot could find. My site might have had billions of different indexable URLs due to my use of parameters in urls to aid in the sorting of items at my website. There were all kinds of parameters that merely changed the appearance of lists of items slightly. Some had no effect at all. But to google it was a mountain of URLs.
I guess it's really just personal preference if you choose to use robots.txt, or use other methods to weed-down the number of links a bot can find. I'm not a robots.txt believer for reasons of which I'll spare you the details. So, I did this without the use of robots.txt. I did 3 major things:
1. Changed all of my indexable urls to clean urls with modrewrite.
2. Put any urls whose purpose was to help people sort items behind form posts.
3. 301 redirected any already indexed urls that contained the kind of things I was trying to keep bots way from back to a clean url without all the extra parameters.
In a nutshell, I got rid of all the bot-garbage on my site, and since bots like to hold onto urls for so long, I told google and all the other bots what they could get rid of. And google at the very least seemed to really like this. It must have been delighted to crawl along my website and find page after page of truly original content. I'm truthfully not sure what kinds of things would delight a bot, but I'd think this would be one of them.
So anyway, I'm sure I'm not telling any of the long-time patrons here anything they don't already know. But if you have a site that has hundreds of thousands, millions, billions, etc. indexable urls it's well worth your time to get rid of your bot garbage. And make sure you take away the garbage it already has, too.
Posted 25 January 2010 - 08:06 PM
So you cleaned up your internal linking in order to show fewer competing URLs for the same content, and now your crawl has improved.
Is that correct?
If so, then well done.
If not, then I have to admit to being somewhat confused about what you're saying.
Posted 26 January 2010 - 08:17 AM
I wouldn't get excited too quickly about this. You may be mixing up cause and effect as Google's index has been going cuckoo lately with the number of pages appearing to be indexed changing at any time depending on which database you're seeing.
Now, you may very well be on to something, but imo, you're gonna need a lot more time to see if it remains consistent over time. Otherwise, you're just seeing the ole mixing up cause and effect thingee in action which is all too common in SEO.
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users