17 replies to this topic

[BrianR]

Thanks, Matt - that encourages me greatly - I was concerned that we wouldn't be able to get much exposure on any SE other than Google.

And, although both of our query string have three parameters, your dynamic parameter is longer and more complex, so I'm very hopeful of getting some pretty thorough listings in FAST and Inktomi.

Thanks for your input.

BrianR

[alexxx]

Scottie - do you (or indeed, anyone else) have any experience of SEs other than Google doing a thorough listing job of simple dynamic urls?

What I'm trying to get a fix on is which SEs we will lose out on even if we present properly optimised dynamic pages with simple urls.

for IIS exists a great ISAPI Filter to make more SE friendly URLs

URL REWRITE

[projectphp]

What spiders hate are session IDs and same variables, different order.

The variables "id" "id", "sid" and "PHPSESSID" are all read flags to Google, as these idicate session names. As such (www.mydomain.com/about/press_page.asp?ID=123) is a terrible URL, as it includes a variable "id".

Apart from that, crawling dynamic URLs is a bit of a lottery. There are so many problems associated that many just don't bother. This isn't Conspiracy theory (Sorry Jill), just a problem of technology. Inktomi only tends to index long URLs if you PFI them. The reason is because that way they KNOW the URL exists and is Unique. With a Natural crawl, they don't.

Recently, with Yahoo's bidding, Inktomki have tried to index more URLs, with the result that many forums around have found Inktomi's spiders on their site for two straight days, indexing and reindexing pages with different session IDs.