17 replies to this topic

[BrianR]

In reading earlier posts on this topic, Jill and others have said that the SEs no longer have any problems reading dynamic urls.

I understand that Google is now listing dynamic urls - I checked out a site I'm working on that is over half dynamic pages (content management system) and Google reported 74 pages, lots of which have dynamic urls.

Then I go over to Alltheweb and I get just 8 pages, only one of which is a dynamic url. And MSN reports just 4 pages, none of which are dynamic urls.

The difference is so vast that I find it difficult to believe that MSN and Alltheweb are searching dynamic urls thoroughly, especially as Google and Alltheweb both search roughly 3 billion pages.

What am I missing??

Thanks.

BrianR

[Scottie]

To different degrees, all the engines are reading some dynamic URL's.

I have a forum program that uses 5 criteria in the URL- I've eliminated every other possibility for why it is not getting spidered and the URL is definitely the issue.

So- not all dynamic URLs are spiderable.

[BrianR]

I accept that complex dynamic urls are not spiderable by the SEs. But the urls in this particular dynamic site are quite simple, eg:

www.mydomain.com/about/press_page.asp?ID=123

So, while Google may read and list the more simple dynamic urls, my little test seems to indicate that some other major SEs do not to any significant degree, if at all.

Scottie - do you (or indeed, anyone else) have any experience of SEs other than Google doing a thorough listing job of simple dynamic urls?

What I'm trying to get a fix on is which SEs we will lose out on even if we present properly optimised dynamic pages with simple urls.

Thanks.

BrianR

[Ron Carnell]

From what I've seen, there are three factors to consider.

1. For reasons explored in other threads, every search engine is going to avoid anything resembling a session ID and the potentially infinite number of pages a session ID can present. Of the three considerations, this is the only one that makes perfect sense to me and the only one I am CERTAIN is true. What I'm not certain about is just how paranoid the spiders will be. Googleguy as been quoted as saying we shouldn't use ID as a parameter in a dynamic URL. I've talked to people at other forums who have found it impossible to get pages with long strings of numbers (which definitely looks like PHP or ASP generated session IDs) indexed. Beyond that, everything seems to be a guess.

2. Common wisdom is that Googlebot will spider a page with up to three parameters. That really makes no sense to me, since the number of parameters has very little relationship to anything Google would care about. Three parameters could produce a gazillion pages just as easily as can seven or eight parameters. Just as importantly, strictly from a programming standpoint, I see absolutely no reason the spider would ever waste time parsing the parameters from the query string.

3. Common wisdom also suggests that a dynamic URL linked to from a static page is much more likely to be indexed than the same dynamic URL linked to from another dynamic page. This makes some sense, especially if both dynamic URLs are calling the same exact script. The potential for an infinite loop becomes very real.

A few years ago, of course, it was impossible to get any dynamic pages indexed. So I guess we're lucky we what we have. But I think the current situation is still too new, and probably still in too much flux, to make many hard and fast rules. As usual, we're limited to guesses, and this list of three concerns is what I've seen for us to guess about.

Anyone think of a fourth?

[BrianR]

Thanks very much, Ron - that's a very helpful summary of the current situation.

BrianR

[Alan Perkins]

strictly from a programming standpoint, I see absolutely no reason the spider would ever waste time parsing the parameters from the query string.

Not from the query string of the page it is accessing, but from the query strings it finds in links (i.e. the pages it will, or at least might, access).

I've written a spider that crawls dynamic sites, and it does parse the query parameters. It then sorts them into alpha order and reformulates the URL. This helps classify something like these two as the same URL:

www.site.com?a=1&b=2
www.site.com?b=2&a=1

[Ron Carnell]

I stand corrected, Alan. I agree, reorganizing the URL for comparisons would be a valid reason to parse the query string in a link.

I'm curious if your program counted the parameters and refused to follow a link with too many?

If one were to believe that the number of parameters was the only problem with an URL, it would sure be easy to circumvent. The ampersand delimiter, after all, is merely a convention. If you're not using a utility to parse the string (like cgi.pm), there's no reason the delimiter can't be changed to something else entirely. Let Google try to count the parameter THEN.

[Jill]

Not sure if it's true or not, but it seems that some of the engines that charge for inclusion may purposely NOT be including dynamic pages in hopes that you'll pay for trusted feed on those.

Certainly Inktomi and FAST can index dynamic pages as they used to do it all the time.

Food for thought...

Jill

[Alan Perkins]

I'm curious if your program counted the parameters and refused to follow a link with too many?

No.

[BrianR]

Jill

The rough stats I indicated at the beginning of this thread seem to indicate there's some truth in the PFI rumour.

But, with my non-paranoid hat on, it runs counter to the search engines' promise to deliver relevant and reasonanly comprehensive results.

But perhaps I'm just being naive...

BrianR

[Jill]

But, with my non-paranoid hat on, it runs counter to the search engines' promise to deliver relevant and reasonanly comprehensive results.

Yeah, no kidding.

Which is why they'll never catch Google in a trilliion, zillion, quadrillion years.



Jill

[Searchaware]

Not sure if it's true or not, but it seems that some of the engines that charge for inclusion may purposely NOT be including dynamic pages in hopes that you'll pay for trusted feed on those. 

Certainly Inktomi and FAST can index dynamic pages as they used to do it all the time. 

Food for thought...

Jill

Jill

I have wondered that too. Of course it seems to me that in some cases if you submit via paid inclusion (to Inktomi in particular) it seems to have a negative effect on them actually spidering your site naturally, as was discussed in another thread.

I wonder if it actually a dynamic pages issue rather than just about PFI. I am in the process of getting a site converted from working on the principle of URL parameters to using a URL rewriter to server "static looking" pages. I will post again if this makes a difference to Inktomi crawling the site.

Simon

[mopacfan]

My site is nearly completely dynamic. I've had very good success in getting all of my pages spidered. After reading this forum as well as some other resources, I've added a site map to my site to ensure the spider will find all of my content. I know that the 'pages' are all accessible from my current navigational structure, but I wanted to be sure, without a doubt, that a spider can find each and every page with a direct link and not have to try to guess what criteria to place in the query string.

[BrianR]

mopacfan

Your interesting post raises a couple of questions for me:

1. Which search engines are you having success with?
2. Can you give me an example of a typical dynamic url from your site?

Thanks for your help

BrianR

[Matt B]

From building similar dynamic sites, I've seen results in every search engine. From one specific website that is completely dynamic, both Google and FAST have every page indexed (over 1,100), while Inktomi and Teoma have indexed 200-500 and are adding new pages each month.

A typical query string for this site looks like this: www.website.edu/graduate/ categorymain.php?cid=1&sid=1406