We have around 1.3m total pages, Google currently crawls on average 87k a day and our average page load is 1.7 seconds. Out of those 1.3m pages(1.2m being "spun up") google has only indexed around 368k and our SEO person is telling us that if we speed up the pages they will crawl the pages faster and thus will index more of them.
I personally don't believe this. At 87k pages a day Google has crawled our entire site in 2 weeks so they should have all of our pages in their DB by now(it's been months since we launched them) and I think they are not index because they are poorly auto spun pages. Am I correct? Would speeding up the pages make Google crawl them faster and thus get more pages indexed?
Subscribe to HRA Now!
Are you a Google Analytics enthusiast?
Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE!
www.CustomReportSharing.com
From the folks who brought you High Rankings!
More SEO Content
International SEM | Social Media | Search Friendly Design | SEO | Paid Search / PPC | Seminars | Forum Threads | Q&A | Copywriting | Keyword Research | Web Analytics / Conversions | Blogging | Dynamic Sites | Linking | SEO Services | Site Architecture | Search Engine Spam | Wrap-ups | Business Issues | HRA Questions | Online Courses
Does Faster Spidering = More Indexing?
Started by
detzx
, Sep 02 2011 04:49 AM
11 replies to this topic
#3
Michael Martinez
-
- Active Members
-
- 5,325 posts
HR 10
- Location:Georgia
Posted 02 September 2011 - 12:42 PM
I think Jill is saying that you are correct in your feeling that speeding up the pages will NOT result in more being indexed BECAUSE they are (as you say) "poorly auto spun pages".
Google's crawling and indexing are driven in large part by the internal PageRank they calculate (this is NOT the number you see in the Google Toolbar). But Google engineers have occasionally disclosed they may choose not to crawl parts of a Website that they deem to be of low quality or unusual to visitors.
Google's crawling and indexing are driven in large part by the internal PageRank they calculate (this is NOT the number you see in the Google Toolbar). But Google engineers have occasionally disclosed they may choose not to crawl parts of a Website that they deem to be of low quality or unusual to visitors.
#4
piskie
-
- Active Members
-
- 1,098 posts
HR 7
- Location:Cornwall
Posted 02 September 2011 - 08:01 PM
MM, could you please expand the meaning of:
"unusual to visitors"
It is probably very obvious, but I can't quite put a definitive meaning to it.
"unusual to visitors"
It is probably very obvious, but I can't quite put a definitive meaning to it.
#5
Michael Martinez
-
- Active Members
-
- 5,325 posts
HR 10
- Location:Georgia
Posted 04 September 2011 - 02:01 PM
"unusual to visitors" is anything that falls outside the norm of what a Web surfer might expect to find on a Website. The last time I saw a Googler comment on something like that, I believe the page was highly widgetized to the point that it confused visitors. It wasn't clear what the point of the page was. I don't remember the exact conversation.
#6
piskie
-
- Active Members
-
- 1,098 posts
HR 7
- Location:Cornwall
Posted 05 September 2011 - 12:37 PM
Thanks MM, that's the first time I've heard the term, so now I know.
However it sounds like I would never be tempted to do nor accept instruction to produce that type of crud.
However it sounds like I would never be tempted to do nor accept instruction to produce that type of crud.
#7
chrishirst
-
- Moderator
-
- 7,718 posts
A not so moderate moderator.
- Location:Blackpool UK
Posted 07 September 2011 - 02:36 AM
QUOTE
and our SEO person is telling us that if we speed up the pages they will crawl the pages faster and thus will index more of them.
Fire him/her/them!
#8
portentint
-
- Active Members
-
- 30 posts
HR 2
Posted 09 September 2011 - 02:20 PM
OK, but what about crawl "efficiency"?
My gut tells me that a faster site may see ranking faster updates following optimization, and that new content added 2-3 layers down in the site hierarchy will see faster indexing. I know Google and Bing don't recrawl the entire site every time, but they have to find the new stuff somehow, even if external links don't yet exist.
Plus, it seems likely that site speed is a factor in Google's "quality" calculations.
My gut tells me that a faster site may see ranking faster updates following optimization, and that new content added 2-3 layers down in the site hierarchy will see faster indexing. I know Google and Bing don't recrawl the entire site every time, but they have to find the new stuff somehow, even if external links don't yet exist.
Plus, it seems likely that site speed is a factor in Google's "quality" calculations.
#9
Jill
-
- Admin
-
- 33,244 posts
Recovering SEO
Posted 09 September 2011 - 02:35 PM
I would agree, if your site was humongously slow and you made it measurably faster, but that's not usually the case with most sites. Even then, my own gut says it wouldn't get more pages indexed.
#10
Michael Martinez
-
- Active Members
-
- 5,325 posts
HR 10
- Location:Georgia
Posted 09 September 2011 - 04:31 PM
Crawl efficiency can be measured and managed in more than one way. Site performance is absolutely one metric. If you install a widget that drags your page load/render time down to 30 seconds, you have a user issue but not necessarily a crawl issue. If your server is in need of an upgrade or reboot and simply cannot respond quickly to fetch requests, that's a crawl issue.
Where the content is placed in the hierarchy usually doesn't matter once the search engine has a record of the location. It can schedule new visits based on PageRank-like value, finding new links pointing to the content, or both.
I think the most important crawl efficiency is managed through the internal navigation. The pages that SHOULD be crawled the most are the pages that update their internal links the most. On many sites that includes the home page but on some sites the home page is considered sacred ground and new links are rarely posted there. A LOT of crawl inefficiency channels the crawlers toward the home page without giving them a real reason to go back there every hour or whenever.
This is one reason why so many blogs do so well in search engines. As they push older content down and ultimately off the front page, new links are discovered and crawled and content is indexed more quickly.
Where the content is placed in the hierarchy usually doesn't matter once the search engine has a record of the location. It can schedule new visits based on PageRank-like value, finding new links pointing to the content, or both.
I think the most important crawl efficiency is managed through the internal navigation. The pages that SHOULD be crawled the most are the pages that update their internal links the most. On many sites that includes the home page but on some sites the home page is considered sacred ground and new links are rarely posted there. A LOT of crawl inefficiency channels the crawlers toward the home page without giving them a real reason to go back there every hour or whenever.
This is one reason why so many blogs do so well in search engines. As they push older content down and ultimately off the front page, new links are discovered and crawled and content is indexed more quickly.
#11
portentint
-
- Active Members
-
- 30 posts
HR 2
Posted 09 September 2011 - 04:34 PM
QUOTE(Jill @ Sep 9 2011, 12:35 PM) 
but that's not usually the case with most sites
You're lucky
Half the sites I work on have page load times of 10+ seconds when we start.
#12
chrishirst
-
- Moderator
-
- 7,718 posts
A not so moderate moderator.
- Location:Blackpool UK
Posted 10 September 2011 - 04:28 AM
QUOTE(portentint @ Sep 9 2011, 10:34 PM)
You're lucky
Half the sites I work on have page load times of 10+ seconds when we start.
Half the sites I work on have page load times of 10+ seconds when we start.
load times or render times?
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users
