Jump to content

  • Log in with Facebook Log in with Twitter Log In with Google      Sign In   
  • Create Account

Subscribe to HRA Now!

 



Are you a Google Analytics enthusiast?

Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE! 

 



 

 www.CustomReportSharing.com 

From the folks who brought you High Rankings!



Photo
- - - - -

Indexing Of Large Site


  • Please log in to reply
23 replies to this topic

#16 Michael Martinez

Michael Martinez

    HR 10

  • Active Members
  • PipPipPipPipPipPipPipPipPipPip
  • 5,325 posts
  • Location:Georgia

Posted 25 May 2009 - 02:56 AM

QUOTE(BBCoach @ May 24 2009, 03:13 PM) View Post
You can't have it both ways Michael. Which is it? Stand for something! Apparently you haven't tested it and that's why you have a confilicting opinion.

I'll say it straight up. Sitemaps will NOT get you crawled deep and fast as Michael's logic would lead you to believe! It's your site design that will determine the depth of a crawl and the external links to those deep pages. ...


You're just plain wrong. Site design can certainly help with crawling (I've made that point many times through the years) but you can compensate for poor site design through the use of sitemaps and links from other sites. A lot of people who have had to do just that for clients have discussed their experiences in multiple forums and blogs -- and I am one of those people.

As I stated previously, I've gotten hundreds of large sites crawled indexed, pretty thoroughly, by doing nothing more than submitting XML sitemaps.

Many other people have done the same and shared their experiences on the Web.

There is no one right way to get a site crawled. The empirically proven most efficient method, however, is to combine all three available options: good site design, sitemaps, and external links.




#17 Gerry White

Gerry White

    HR 2

  • Active Members
  • PipPip
  • 48 posts
  • Location:UK

Posted 25 May 2009 - 05:22 AM

QUOTE(BBCoach @ May 24 2009, 11:13 PM) View Post
You can't have it both ways Michael. Which is it? Stand for something! Apparently you haven't tested it and that's why you have a confilicting opinion.

I'll say it straight up. Sitemaps will NOT get you crawled deep and fast as Michael's logic would lead you to believe! It's your site design that will determine the depth of a crawl and the external links to those deep pages. My testing and as a result, my opinion, is don't waste your time with a sitemap file. Better time would be to submit a product upload file to the varying SEs. Spitooh on sitemaps. Submit product feeds instead.



Well, despite my earlier posts anti-sitemaps (which I still stand by) if you can do it right, what you can do is influence WHICH pages get indexed quicker, there is also an option in Google Webmaster tools to speed up indexing, not sure how thats going it is something we are trying on a client who was struggling ...

Sitemaps CAN provide additional info on new pages etc.. so unless you have a relatively small flat hierachial site and recency of content isn't quite as important personally I would not create a sitemap... However use Google webmaster tools and the canonical tag to influence google. (canonical to get rid of the 3 or 4 duplicate homepages that many people have, case, subdomain and "index" file ...

#18 Bri

Bri

    HR 4

  • Active Members
  • PipPipPipPip
  • 207 posts
  • Location:Toronto, Ontario

Posted 25 May 2009 - 11:13 AM

QUOTE
there is also an option in Google Webmaster tools to speed up indexing, not sure how thats going it is something we are trying on a client who was struggling ...


This option does not deal with the frequency of which google visits your site to crawl. It is meant to keep from overwhelming your server. Not as a direction to Google what content is of value that Google should be indexing.

Can't remember who said it, maybe Bruce Clay.. (i could be wrong though) This is an arms race... The content needs to be of value for significant resources to be implemented in indexing it. Would google put a control on webmasters that would allow us to tell them if our content was of value?

#19 BBCoach

BBCoach

    HR 5

  • Moderator
  • 402 posts

Posted 26 May 2009 - 10:42 AM

QUOTE
Michael - but you can compensate for poor site design through the use of sitemaps and links from other sites
Again, you're mixing and matching. If the site is designed poorly or is a member login site, then of course a sitemap is paramount in getting a site indexed. However, saying this:
QUOTE
Michael - submit a sitemap and ignore advice against doing so. It doesn't guarantee that you'll be crawled more quickly but it doesn't hurt anything.
confuses the topic.

Plainly stated. If you design a site whereby it is well organized and easily (with a fast server response) crawlable by the bots, then the need for a sitemap is MOOT and is a waste of time and resources because it does not assist in getting pages indexed in such a magnified (quicker) way as to justify creating a sitemap. However, if a site doesn't present it's information in such a fashion, then you would be ignorant to NOT create and present a sitemap.

For those that disagree with this, no skin off my back, but I'm telling you that I have tested this many times on some very large sites and it doesn't improve the speed with which your site will get indexed, unless that site is designed poorly. If you want to jump through the hoops of creating a sitemap, then please do so, but your time would be better spent submitting product uploads to the various SEs.

#20 Jill

Jill

    Recovering SEO

  • Admin
  • 33,244 posts

Posted 26 May 2009 - 11:11 AM

Umm...I think you are both on the same page at this point:

QUOTE(bbcoach)
but I'm telling you that I have tested this many times on some very large sites and it doesn't improve the speed with which your site will get indexed, unless that site is designed poorly.


QUOTE(Michael Martinez)
but you can compensate for poor site design through the use of sitemaps and links from other sites.


Are we all in agreement that a sitemap isn't necessary for a well-designed site (won't have an advantage), but for one that's not so well-organized, a sitemap can potentially get more pages indexed, more quickly?

#21 Randy

Randy

    Convert Me!

  • Moderator
  • 17,540 posts

Posted 26 May 2009 - 12:08 PM

Yeah, I think we all agree that an xml sitemap can sometimes be used as a crutch for doing things the right way the first time. But that it's neither a requirement, nor a panacea for a well designed and well constructed site.

jester.gif

#22 BBCoach

BBCoach

    HR 5

  • Moderator
  • 402 posts

Posted 26 May 2009 - 12:14 PM

I agree with Jill, Randy and myself. whitehat.gif

#23 JakeG

JakeG

    HR 4

  • Active Members
  • PipPipPipPip
  • 212 posts
  • Location:Nottingham, UK

Posted 29 May 2009 - 10:19 AM

Thanks for the advice Michael, it's interesting what you say about using blog posts, some of the press releases we sent out had deep links in, and these pages got indexed and are bringing traffic. With so many URLs though it is hard to know where to start, I guess we need to get a priority list sorted out.

#24 adibranch

adibranch

    HR 5

  • Active Members
  • PipPipPipPipPip
  • 332 posts

Posted 29 May 2009 - 11:35 AM

sitemaps are good for a couple of reasons which havent been mentioned.

1) i use it as a reference for the number of spiderable linked pages, against the number of pages indexed. Using something like the microsys sitemap generator (which produces a sitemap based on internal linking, popularity, and other, and it obeys robots.txt). You then go to the webmaster tools and it states '500 pages indexed out of 750' etc. Now, at this point i'll know i still have a couple of hundred pages which arent yet in the index. Following this, it is assumed that until these pages ARE indexed, i'll be losing out on potential site visitors. I'll be missing indexed pages, and i'll be missing visits that would have come from them.
I also state this figure in the report i send to clients. If i get 90% or over of indexed page against total pages, its a result.

2) in the sitemap is the time the page was created, the importance relative to other pages, and other such goodies. Its not just a list of URL's.

I use them all the time, sometimes for reference as in 1, sometimes because i think hey it cant do any harm, only good. To not use them is kinda odd to me...

Edited by adibranch, 29 May 2009 - 12:00 PM.





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

We are now a read-only forum.
 
No new posts or registrations allowed.