11 replies to this topic

[detzx]

We have around 1.3m total pages, Google currently crawls on average 87k a day and our average page load is 1.7 seconds. Out of those 1.3m pages(1.2m being "spun up") google has only indexed around 368k and our SEO person is telling us that if we speed up the pages they will crawl the pages faster and thus will index more of them.

I personally don't believe this. At 87k pages a day Google has crawled our entire site in 2 weeks so they should have all of our pages in their DB by now(it's been months since we launched them) and I think they are not index because they are poorly auto spun pages. Am I correct? Would speeding up the pages make Google crawl them faster and thus get more pages indexed?

[Jill]

IMO you are correct.

[Michael Martinez]

I think Jill is saying that you are correct in your feeling that speeding up the pages will NOT result in more being indexed BECAUSE they are (as you say) "poorly auto spun pages".

Google's crawling and indexing are driven in large part by the internal PageRank they calculate (this is NOT the number you see in the Google Toolbar). But Google engineers have occasionally disclosed they may choose not to crawl parts of a Website that they deem to be of low quality or unusual to visitors.

[piskie]

MM, could you please expand the meaning of:
"unusual to visitors"
It is probably very obvious, but I can't quite put a definitive meaning to it.

[Michael Martinez]

"unusual to visitors" is anything that falls outside the norm of what a Web surfer might expect to find on a Website. The last time I saw a Googler comment on something like that, I believe the page was highly widgetized to the point that it confused visitors. It wasn't clear what the point of the page was. I don't remember the exact conversation.

[piskie]

Thanks MM, that's the first time I've heard the term, so now I know.
However it sounds like I would never be tempted to do nor accept instruction to produce that type of crud.

[chrishirst]

and our SEO person is telling us that if we speed up the pages they will crawl the pages faster and thus will index more of them.

Fire him/her/them!

[portentint]

OK, but what about crawl "efficiency"?

My gut tells me that a faster site may see ranking faster updates following optimization, and that new content added 2-3 layers down in the site hierarchy will see faster indexing. I know Google and Bing don't recrawl the entire site every time, but they have to find the new stuff somehow, even if external links don't yet exist.

Plus, it seems likely that site speed is a factor in Google's "quality" calculations.

[Jill]

I would agree, if your site was humongously slow and you made it measurably faster, but that's not usually the case with most sites. Even then, my own gut says it wouldn't get more pages indexed.

[Michael Martinez]

Crawl efficiency can be measured and managed in more than one way. Site performance is absolutely one metric. If you install a widget that drags your page load/render time down to 30 seconds, you have a user issue but not necessarily a crawl issue. If your server is in need of an upgrade or reboot and simply cannot respond quickly to fetch requests, that's a crawl issue.

Where the content is placed in the hierarchy usually doesn't matter once the search engine has a record of the location. It can schedule new visits based on PageRank-like value, finding new links pointing to the content, or both.

I think the most important crawl efficiency is managed through the internal navigation. The pages that SHOULD be crawled the most are the pages that update their internal links the most. On many sites that includes the home page but on some sites the home page is considered sacred ground and new links are rarely posted there. A LOT of crawl inefficiency channels the crawlers toward the home page without giving them a real reason to go back there every hour or whenever.

This is one reason why so many blogs do so well in search engines. As they push older content down and ultimately off the front page, new links are discovered and crawled and content is indexed more quickly.

[portentint]

but that's not usually the case with most sites

You're lucky

Half the sites I work on have page load times of 10+ seconds when we start.

[chrishirst]

You're lucky

Half the sites I work on have page load times of 10+ seconds when we start.

load times or render times?