7 replies to this topic

[Jerky19]

Many people have said on this forum that you need a sitemap on your website in order to help Google index it, especially if parts of your web site are dynamically created.

Great, I've done that. Within a few days, I've noticed that my site map became part of Google's cached pages.

Unfortunately though, all of the links FROM the site map that point to other pages have NOT been cached.

So, how does Google do it? Do they crawl 'home pages' every few days and an entire web site only once every 'x' number of weeks? I'm a little lost here.

Thanks for any help...

[bkernst]

As far as I can tell from my log files for various sites, Google seems to check if a page has been updated and adds it to its list of pages to index. While going through other pages of one sites, pags from other sites are added to that list. The indexing itself takes several days because of this.

Keep in mind that several sites are indexed at the same time, and not one site getting all of the attention. What is considered a monthly (or shorter) indexing, is actually Google just looping through a list of pages to index.
It seems like the toplevel entrance pages get more priority, since those are indicators if the site has changes, is under construction and to check if the site actually does still exist.

Bernhard

[bwelford]

Typically, Jerky19, I've found that Google is working on a 4 - 8 week cycle in fully getting to all web pages on the average say 75 web page website with PageRank of say 4 or 5. Some may be picked up fast as you have seen but the final few may take quite a time.

As they say, mileage may vary ...

[bkernst]

For a couple of weeks the updated date shown for some of my sites when using site: is always two days behind the current date. If I do a change today, it might be listed in two days. This does not apply to all pages.

Bernhard

[Jerky19]

Hmmm. I wasn't expecting instant success, but I also wasn't expecting to have my site half-indexed. These search engines are a strange lot indeed! You might say they are to me...

[Jill]

This isn't a job for the impatient!

[bkernst]

To actually get every page of your site indexed can easily take months. Some pages are repeatedly indexed again, while others seem to be ignored for a long time. New sites take a while to get indexed and to get any kind of rankigns takes even longer, and then it is a very long walk up the hill to the top ten.

Bernhard

[Jerky19]

Oh, I'm prepared to wait...and to fight the good fight.
I guess I like the all or nothing approach to indexing. Not the here a page, there a page index method.