Jump to content

  • Log in with Facebook Log in with Twitter Log In with Google      Sign In   
  • Create Account

Subscribe to HRA Now!

 



Are you a Google Analytics enthusiast?

Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE! 

 



 

 www.CustomReportSharing.com 

From the folks who brought you High Rankings!



Photo
- - - - -

Does Faster Spidering = More Indexing?


  • Please log in to reply
11 replies to this topic

#1 detzx

detzx

    HR 2

  • Active Members
  • PipPip
  • 23 posts

Posted 02 September 2011 - 04:49 AM

We have around 1.3m total pages, Google currently crawls on average 87k a day and our average page load is 1.7 seconds. Out of those 1.3m pages(1.2m being "spun up") google has only indexed around 368k and our SEO person is telling us that if we speed up the pages they will crawl the pages faster and thus will index more of them.

I personally don't believe this. At 87k pages a day Google has crawled our entire site in 2 weeks so they should have all of our pages in their DB by now(it's been months since we launched them) and I think they are not index because they are poorly auto spun pages. Am I correct? Would speeding up the pages make Google crawl them faster and thus get more pages indexed?


#2 Jill

Jill

    Recovering SEO

  • Admin
  • 33,005 posts

Posted 02 September 2011 - 06:58 AM

IMO you are correct.

#3 Michael Martinez

Michael Martinez

    HR 10

  • Active Members
  • PipPipPipPipPipPipPipPipPipPip
  • 5,157 posts
  • Location:Georgia

Posted 02 September 2011 - 12:42 PM

I think Jill is saying that you are correct in your feeling that speeding up the pages will NOT result in more being indexed BECAUSE they are (as you say) "poorly auto spun pages".

Google's crawling and indexing are driven in large part by the internal PageRank they calculate (this is NOT the number you see in the Google Toolbar). But Google engineers have occasionally disclosed they may choose not to crawl parts of a Website that they deem to be of low quality or unusual to visitors.

#4 piskie

piskie

    HR 7

  • Active Members
  • PipPipPipPipPipPipPip
  • 1,098 posts
  • Location:Cornwall

Posted 02 September 2011 - 08:01 PM

MM, could you please expand the meaning of:
"unusual to visitors"
It is probably very obvious, but I can't quite put a definitive meaning to it.

#5 Michael Martinez

Michael Martinez

    HR 10

  • Active Members
  • PipPipPipPipPipPipPipPipPipPip
  • 5,157 posts
  • Location:Georgia

Posted 04 September 2011 - 02:01 PM

"unusual to visitors" is anything that falls outside the norm of what a Web surfer might expect to find on a Website. The last time I saw a Googler comment on something like that, I believe the page was highly widgetized to the point that it confused visitors. It wasn't clear what the point of the page was. I don't remember the exact conversation.

#6 piskie

piskie

    HR 7

  • Active Members
  • PipPipPipPipPipPipPip
  • 1,098 posts
  • Location:Cornwall

Posted 05 September 2011 - 12:37 PM

Thanks MM, that's the first time I've heard the term, so now I know.
However it sounds like I would never be tempted to do nor accept instruction to produce that type of crud.

#7 chrishirst

chrishirst

    A not so moderate moderator.

  • Moderator
  • 7,103 posts
  • Location:Blackpool UK

Posted 07 September 2011 - 02:36 AM

QUOTE
and our SEO person is telling us that if we speed up the pages they will crawl the pages faster and thus will index more of them.
Fire him/her/them!

#8 portentint

portentint

    HR 2

  • Active Members
  • PipPip
  • 30 posts

Posted 09 September 2011 - 02:20 PM

OK, but what about crawl "efficiency"?

My gut tells me that a faster site may see ranking faster updates following optimization, and that new content added 2-3 layers down in the site hierarchy will see faster indexing. I know Google and Bing don't recrawl the entire site every time, but they have to find the new stuff somehow, even if external links don't yet exist.

Plus, it seems likely that site speed is a factor in Google's "quality" calculations.

#9 Jill

Jill

    Recovering SEO

  • Admin
  • 33,005 posts

Posted 09 September 2011 - 02:35 PM

I would agree, if your site was humongously slow and you made it measurably faster, but that's not usually the case with most sites. Even then, my own gut says it wouldn't get more pages indexed.

#10 Michael Martinez

Michael Martinez

    HR 10

  • Active Members
  • PipPipPipPipPipPipPipPipPipPip
  • 5,157 posts
  • Location:Georgia

Posted 09 September 2011 - 04:31 PM

Crawl efficiency can be measured and managed in more than one way. Site performance is absolutely one metric. If you install a widget that drags your page load/render time down to 30 seconds, you have a user issue but not necessarily a crawl issue. If your server is in need of an upgrade or reboot and simply cannot respond quickly to fetch requests, that's a crawl issue.

Where the content is placed in the hierarchy usually doesn't matter once the search engine has a record of the location. It can schedule new visits based on PageRank-like value, finding new links pointing to the content, or both.

I think the most important crawl efficiency is managed through the internal navigation. The pages that SHOULD be crawled the most are the pages that update their internal links the most. On many sites that includes the home page but on some sites the home page is considered sacred ground and new links are rarely posted there. A LOT of crawl inefficiency channels the crawlers toward the home page without giving them a real reason to go back there every hour or whenever.

This is one reason why so many blogs do so well in search engines. As they push older content down and ultimately off the front page, new links are discovered and crawled and content is indexed more quickly.

#11 portentint

portentint

    HR 2

  • Active Members
  • PipPip
  • 30 posts

Posted 09 September 2011 - 04:34 PM

QUOTE(Jill @ Sep 9 2011, 12:35 PM) View Post
but that's not usually the case with most sites


You're lucky smile.gif

Half the sites I work on have page load times of 10+ seconds when we start.

#12 chrishirst

chrishirst

    A not so moderate moderator.

  • Moderator
  • 7,103 posts
  • Location:Blackpool UK

Posted 10 September 2011 - 04:28 AM

QUOTE(portentint @ Sep 9 2011, 10:34 PM) View Post
You're lucky smile.gif

Half the sites I work on have page load times of 10+ seconds when we start.

load times or render times?




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

SPAM FREE FORUM!
 
If you are just registering to spam,
don't bother. You will be wasting your
time as your spam will never see the
light of day!