9 replies to this topic

[Mark S]

Is it true that if your site is linked from a good PR site, your site will get crawled more comprehensively and more frequently?

[qwerty]

Perhaps. If you update certain pages often, they'll certainly get crawled more.

But the question is why you want to be crawled more. You want your changes to be picked up quickly, but if you're making changes in order to get crawled, that doesn't make a lot of sense. Getting crawled more doesn't help rankings.

[Jill]

It does seem to be true that popular sites will be more apt to get more deeply crawled, more quickly.

[Scottie]

Past discussion on Frequency of crawling might be interesting to you.

[Mark S]

Thanks guys.

I'm not personally bothered about how frequently my site gets crawled. The only reason I ask about this is because I was having a little debate with a friend over SEO and he mentioned that getting a site linking to you which has a good PR will get your site crawled more frequently. I was a bit suprised as I thought it only depended on how much you updated you content.

[Scottie]

Here's the logic in that...

Search engines follow links. High PR sites have a lot of links pointing to them (since PR is simply a measure of links) so the crawlers get directed to that high PR site from links in a lot of other sites. As they crawl that high PR site, they follow the links on it, so having a link on that site may get your site crawled more often.

Not that it matters, unless you change your content frequently.

[bkernst]

If the search engine notices regular notworthy updates (not just one or two words changed), then it might possibly come more. As mentioned, links are needed having links pointing to your site does not mean, the search engine has to index your site more often since that would be a waste of processing power. If your site isn't updated much, then it doesn't need to be indexed a lot to be up to date in the search engine database.

I have a site with top 10 rankings for several phrases (for some the number 1 spot for months) since there is not much competition. That site currently only has got 51 links pointing to it according to Google. The entrance page has got a PR of 4. As Scottie mentioned, the PR is just used for measurement.

Site useability should should not be sacrificed for PR.

[Jill]

If your site isn't updated much, then it doesn't need to be indexed a lot to be up to date in the search engine database.

I find that even sites that aren't updated often still get spidered often if they are popular sites.

[Ron Carnell]

Frequency and depth, I think, are two different things, though the latter is often going to be interrelated to the former.

Yea, there is evidence that Google more frequently crawls pages that change often, obviously so their SERPs can stay current with current content. There's no evidence to suggest this affects the depth of the crawl.

I personally feel it's fairly clear that the depth of the crawl is influenced by the importance (PR) of the pages. Google clearly isn't going to ignore a PR10 page just because it's buried four or five levels deep in a large site. Okay, so what about a PR9 or PR8? At what point a page becomes too important to ignore is anyone's best guess, but my experience indicates a PR4 at the bottom of your site hierarchy pretty much guarantees Google with take everything you have to offer. Those page, I believe, are simply too important to be ignored.

The depth of a crawl can affect frequency on even moderately large sites, simply because by the time Googlebot has finished the deep crawl, it's almost time to start at the top of the site again. I know of many sites where the spider visits every single day, but that doesn't necessarily mean it looks at every single page every day.

[qwerty]

On a Google Sitemap you have the opportunity to indicate how often a given file is updated. I wonder if Google responds to that information, and if so, how. I suppose they must use it for something, or they wouldn't have made it part of the file. But do they just take your word for it? For example, let's say I have a file that was indexed a year ago and hasn't been updated once since then, and then I file a sitemap that indicates that that file is updated daily. Will Google come by once or twice on a daily basis, see that the file hasn't changed, and stop believing me? Will they give me the benefit of the doubt at all, or just ignore my claim that the page gets updated every day?