2 replies to this topic

[greggb]

Hi all. I started a topic a few days ago that lead to a conversation about the number of links on a website. Specifically, how a large number of indexable links can hurt your site, as far as SEO goes. Well, I just finished a link cleanup on a site of mine about 1:00 am this morning. It's possible that it's just coincidence, but I did a site:www.mysite.com at google and found that my PR2 site, which used to have about 500 urls indexed according to what google reported (and had so for the last 6 months or longer), is now reporting 1200. Some real-time traffic tracking scripts are showing me that I'm already benefitting from the additional indexed URLs. To what degree it may be too soon to tell yet, but in my book going from a reported 500 links to 1200 is a nice little pat on the back from google.

My goal was to reduce the number of indexable urls google or any other bot could find. My site might have had billions of different indexable URLs due to my use of parameters in urls to aid in the sorting of items at my website. There were all kinds of parameters that merely changed the appearance of lists of items slightly. Some had no effect at all. But to google it was a mountain of URLs.

I guess it's really just personal preference if you choose to use robots.txt, or use other methods to weed-down the number of links a bot can find. I'm not a robots.txt believer for reasons of which I'll spare you the details. So, I did this without the use of robots.txt. I did 3 major things:

1. Changed all of my indexable urls to clean urls with modrewrite.
2. Put any urls whose purpose was to help people sort items behind form posts.
3. 301 redirected any already indexed urls that contained the kind of things I was trying to keep bots way from back to a clean url without all the extra parameters.

In a nutshell, I got rid of all the bot-garbage on my site, and since bots like to hold onto urls for so long, I told google and all the other bots what they could get rid of. And google at the very least seemed to really like this. It must have been delighted to crawl along my website and find page after page of truly original content. I'm truthfully not sure what kinds of things would delight a bot, but I'd think this would be one of them.

So anyway, I'm sure I'm not telling any of the long-time patrons here anything they don't already know. But if you have a site that has hundreds of thousands, millions, billions, etc. indexable urls it's well worth your time to get rid of your bot garbage. And make sure you take away the garbage it already has, too.

[Michael Martinez]

Your post title didn't make it clear to me what you actually did. If I understand you correctly, you had dynamic URLs that were possibly confusing the crawlers and/or were so convoluted the crawlers simply didn't follow them.

So you cleaned up your internal linking in order to show fewer competing URLs for the same content, and now your crawl has improved.

Is that correct?

If so, then well done.

If not, then I have to admit to being somewhat confused about what you're saying.

[Jill]

Just a quick word of warning...

I wouldn't get excited too quickly about this. You may be mixing up cause and effect as Google's index has been going cuckoo lately with the number of pages appearing to be indexed changing at any time depending on which database you're seeing.

Now, you may very well be on to something, but imo, you're gonna need a lot more time to see if it remains consistent over time. Otherwise, you're just seeing the ole mixing up cause and effect thingee in action which is all too common in SEO.