6 replies to this topic

[jonbchristiansen]

We recently revamped and launch a customer website that has been indexed by the major search engines and directories in the past (poor rankings however).

An allinsite: search in Google produces 122 cached pages from the old site. Yesterday the Google's spider made it to site again BUT....


The spider only indexed the home page and eight other pages in the domain. This has me scratching my head when I look at the indexed pages closely. There is absolutely no rhyme or reason as to what subpages where indexed and what subpages where skipped.


I am grateful for any assistance on why most subpages linked from the home page where not indexed. (It is very important that the subpages are indexed as they contain 10s of pertinent keyphrases not represented on the home page.)


Thanks in advance,

Jonathan

[Scottie]

You can drive yourself crazy watching every spider visit. The details of where the spider goes on each visit and why really aren't that important.

The more good incoming links you have to your site, the more times the spiders run across your site in a given time period. Also, when they find new content, they tend to check back more frequently in an effort to keep their index up to date.

Keep getting good links and adding new content and don't worry over an individual visit here and there.

[kology]

You give the spiders too much credit.

They came to your site that's good. They got a taste of the new site and if they like the taste, they'll be back with bigger appetites.


I'll assume all your page URLs change. So on the first visit, they got the idea of what's out there that's it. Check your logs in a day or two for more visits.

Did you delete your old pages? I find this helps clean out the SE's old cache of pages. If you're just waiting for them to dissolve and vanish forget it.

If you did delete the old pages, do you have a proper 404 page up there? Or even better a 301 redirect for major sections. These can also work wonders in getting the new pages indexed quickly.


As Scottie pointed out, get lots and lots of inbound links. This will temp the spiders even more to come and visit.

[Peter]

Jonathan,

I would have some questions first:

How many pages does the site have?
How many levels does the site have?
Are the pages dynamically generated with dynamic URLs with lots of parameters or just plain static html pages?
How is the navigation? Are all text links or do you use script menus?
Do you have a site map?

Answer these questions and it will be possible to determine if you just have to be patient or if you have a real problem.

Regards,

Peter

[BrianR]

Jonathan

Since the last major algo update, Google has been much more cautious about how it indexes larger sites. It can often take several weeks for all pages to be indexed and ranked.

In between times, you will often see that many pages are only partially indexed - ie. the listing does not show the title or the description snippet but just the url.

BrianR

[jonbchristiansen]

It can often take several weeks for all pages to be indexed and ranked.

Keep getting good links and adding new content and don't worry over an individual visit here and there

Good confirmation. I am still green at SEO and this is the info that I needed.


Are there any "proven techniques" to encourage the indexing of subpages? For example:

>> Do search engines "look for" site map pages and give greater weight to indexing links on the site map.
>> Does submitting subpages to the SEs hurt or help?

How many pages does the site have?

A: 25 static pages. Literally thousands of dynamic pages.

How many levels does the site have?

A: Ususally no more than 4 clicks to the deepest level.

Are the pages dynamically generated with dynamic URLs with lots of parameters or just plain static html pages?

A: Almost all URLs of dynamic pages into the product catalog (10000+ items) employ Mod_Rewrite. We have 100+ product categories. Each category should be indexed, with 20% as targeted for keyphrase SEO.

How is the navigation? Are all text links or do you use script menus?

A: All text links. No scripted menus.

Do you have a site map?

A: Not yet.


Thanks again.

[BrianR]

Are there any "proven techniques" to encourage the indexing of subpages? For example:

>> Do search engines "look for" site map pages and give greater weight to indexing links on the site map.
>> Does submitting subpages to the SEs hurt or help?

Ensure that you have a sound internal linking structure.

With larger sites, site maps are helpful, but especially so for users.

Submitting to SEs is unnecessary - it will neither hurt nor help.

BrianR