18 replies to this topic

[markm]

Hi,

I am dealing with a dynamic site indexing problem. This an ASP site, 1000 pages and only the home page is getting indexed.

I think a good deal of the site should get indexed by Google. There is a sitemap (static) with links to all the pages in the site. The sitemap is not getting indexed - that for sure should get indexed. Also many of the pages the site map links to have single variable URLs, which I believe Google should index as well.

But it is not happening. Before I go the route of something drastic (like a URL rewrite), I want to make sure nothing else is going on.

So what else could cause it to not get indexed? Its an old site with high PR (7).

My plan is:

1 - clean up the HTML -There are HTML errors which I will ask them to clean up.

2. I was wondering if there was a spider trap which is preventing Google from spidering the site. How to I detect this? From the logs?

3. Anything else for me to look for?

Thanks,

Mark

Edited by markm, 29 September 2003 - 11:37 PM.

[Jill]

Mark,

How's the site look in a Lynx Browser?

Are the links followable?

Jill

[SearchRank]

Hi Mark,

Is the site map accessible from the home page?

Do the links to the .asp pages have anything like session ids attached to them?

How about the robots.txt file - is it limiting crawlers?

Possibly you can post the URL so others can have a looksy?

[markm]

Mark,

How's the site look in a Lynx Browser?

Are the links followable?

Jill

Hi Jill,

I tried the Lynx browser. Several of the initial links on the home page are to https pages and that causes a problem for Lynx. Even if Google doesn't like those pages, is should still go on - right?



Mark

[markm]

Do the links to the .asp pages have anything like session ids attached to them?

How about the robots.txt file - is it limiting crawlers?

Hi David,

No session IDs in the URLs but there is an 'ID" parameter - sometimes not numeric. I've heard that can cause problems (but I would think only for those specific pages).

I am having the IT guy check on the robots.txt to make sure it is has been correctly formatted. (I don't have access to the server yet).

If they are okay with it, I'll post the URL.

Thanks for the help!



Mark

[Jill]

I tried the Lynx browser. Several of the initial links on the home page are to https pages and that causes a problem for Lynx. Even if Google doesn't like those pages, is should still go on - right?

Nope, Google won't index https pages as a rule. That's your problem. Take them out of the SSL area.

Jill

[Alan Perkins]

Nope, Google won't index https pages as a rule. That's your problem. Take them out of the SSL area.

It does...

http://www.google.co...allinurl: https

Better not to, though ...

[markm]

Nope, Google won't index https pages as a rule.  That's your problem.  Take them out of the SSL area.

Jill

What I mean is this....

The home page links to many pages maybe (100). The first link and then a few others are https pages, but shouldn't it pick up the other links. For example,

home-->https://shopping-cart
home-->https://sign-on
home-->https://your-account
home-->http://shop
home-->http//category1
.
.
.
home-->http//sitemap

Shouldn't it pick up the non-secure pages or do you think the secure pages could be stopping Google in its tracks.

Also I have heard that Google maybe only traverses the first 100 links on a page. There are more than 100 links on this page. That could be stopping Google from reaching the Sitemap - right?

My strategy will be to address anything that could be causing the problem - since the customer once to get indexed ASAP (Santa is coming)

Cheers,

Mark

[Jill]

Mark, I think this is one of those cases where we're going to need a URL. It could be anything or nothing, and we can guess all day, but without seeing the site, I don't think we'll be able to know for sure about anything.

Jill

[markm]

I think this is one of those cases where we're going to need a URL.

Jill,

They would rather not show up on newsgroup thread on their site...BUT...I placed the URL in my profile under "Interests" - since getting them indexed is my prime interest right now!


Thanks,

Mark

PS

that link to an index page at the bottom of the each page is what I am calling a sitemap. I think it is left over from some previous SEO attempt. It's been there quite a while.

Edited by markm, 30 September 2003 - 02:39 PM.

[Alan Perkins]

Mark

There is some history behind that domain. It has been used as a case study in the past.

That "index" page is a bad one. I would get rid of it straight away. Then find out the history of the domain. Then plan your strategy.

[Jill]

Ahh yes, it has been in a case study! What happened? I thought everything was great with it?

I think there's more to this story than meets the eye!

Jill

[Matt B]

Whoa! Alan's not kidding, that index page is amazing!!

You may be better off formatting it to be more pleasing to your website visitors, rather than search engines - (after you remove it.) The size of that page is almost 2250K! I'd have a better count, but my browser crashed. Too, too many links and not a good presentation for visitors.

The rest of your problem my be the dynamic structure of your site, but I'm hard pressed (HA, get it? hard-pressed - wine? ) to think that your dynamic programming is the reason you are not being indexed.

How has your search engine traffic been for the last few months? Any trends up or down?

Edited by Matt B, 30 September 2003 - 03:58 PM.

[Matt B]

Here's another issue that may hinder your success - right from your privacy page:

If a user rejects the cookie, they may still browse our site. Unfortunately, the user experience will be greatly curtailed and only the California catalog will be available. In addition, the user will be unable to log in or make a purchase.

Search engines can't accept cookies. That might be something to look into.

Your California catalog link from the homepage may have too many parameters for a search engine to follow. Still, you have many pages that shouldn't be a problem to follow from the homepage. Very, very intruiging.

[markm]

Wow.....

I found the case study on search engine watch! What did I step into here?
I just walked into this as a sub-contracter to help with the SEO! After reading that case study, I am thinking maybe some penalties are still in place.

Mark