4 replies to this topic

[domokun]

Ive read from several sources that certain characters (&, =, ?) contained within a url will instruct the Spider to stop in its tracks a return whatever section of the url it has crawled over

eg.

www.bobswidgetemporium.com/default.asp?widget=green&length=squat

will be retuned as

www.bobswidgetemporium.com/default.asp

now thats all well and good and, up to a point, makes some sense. the problem is (you knew it was coming!) that according to my referrer logs googlebot has been scraping urls such as

www.chrishasgreatwidgets.com/default.asp?widget=lemon&length=tall

from my site and displaying it to users in its search results. how can this be?
even more perplexing, is that there are several occaisons when a huge url (with several occurances of &, ? and = in it) has been scraped!!

has anyone else encountered this?

[Jill]

Those characters used to be a problem because they signaled the search engines that it was dynamic URL and probably dynamic content which might be a duplicate.

Google, at least, seems to have figured out how to deal with it and they are not stop characters at this point.

Not sure what you mean about the urls being scraped, could you elaborate?

Jill

[domokun]

"scraped" - sorry, force of habit! my old job involved building scripts to retreive information from web pages; we used to call this 'scraping'!
i simply meant that it appears that googlebot can follow links with these supposed 'stop characters' in them.
so google's figured it out, thats good news, what about other engines?
is the issue of stop characters no longer an issue?

[Matt B]

Ive read from several sources that certain characters (&, =, ?) contained within a url will instruct the Spider to stop in its tracks a return whatever section of the url it has crawled over

Absolute Crap.

Google, Inktomi, Ask Jeeves, and the lot can follow dynamic links, even with characters such as =, &, %, and ?.

It all depends on how well the code is written, not on the characters used. If the code is too convoluted and involves too many parameters, it won't be spidered. But simply having a ? does not mean that a spider stops and runs away.

That excuse is what amatuer programmers use to explain their lack of programming skills.

Obviously, you are getting your dynamic site indexed, so your code must be doing the job. The URL you referred to as being returned as

www.bobswidgetemporium.com/default.asp?widget=green&length=squat

will be retuned as

www.bobswidgetemporium.com/default.asp

The URL being re-written like this only happens when a mod-rewrite has been initiated by the server, not by the search engine.

Edited by Matt B, 21 October 2003 - 02:25 PM.

[powerofeyes]

Hello,
What you are talking about must be a old document or some sites which is selling some products to expose the dynami URLs,
Now google can follow any dynamic URLs even with multiple parameters passed after it, INKTOMI and FAST too can handle any type of query strings, So no need to worry about dynamic generated URLs,
But make sure you optimize the contents in your Database(like the description of a prduct) too it will help you in rankings,
VIJAY.
(WEB PROMOTIONS).