5 replies to this topic

[000]

which method is the most appropriate for a given type of Dynamic site

XML feeds -versus- Server-side Dynamic URL changes

What are the PR0s and CONs of each...

Is there a type of Website one is more appropriate for than another

[MakeMeTop]

The answer is both!

XML feeds are primarily suitable for sites who want a lot of pages indexed quickly and where data changes very frequently. However, I would still use a URL re-write for standard indexing for sites in a feed.

XML feeds are not suitable for sites which have static data that is unlikely to change frequently and don't care about fast inclusion - the re-write function is all you need.

[daniel]

It's also worth remembering that there isn't a Trusted Feed for Google, so this isn't an option if you want to optimise the site for inclusion on Google.

For Google you will have to consider URL-rewriting, unless Google can spider your URLs as they are. Google seems to be getting better and better at dealing with URLs that include query strings, so in many cases there might not be a problem.

However, session ids are a big problem, and will stop googlebot in it's tracks - so ensure that your pages aren't serving up session ids to search engine spiders.

[and]

Amazon is using a very productive technique - it may be server-side changes - but it seems to be getting them very high rankings for almost every competative keyword in Google and Inktomi.

Inktomi will now have a new importance because of their upcoming influence in both MSN and Yahoo SERPs.

Until MSN creates it's own Search engine - Inktomi may jump briefly to becomming the #1 Search engine - from near the bottom in popularity.

[john892]

Hi All,

Google seems to be getting better and better at dealing with URLs that include query strings, so in many cases there might not be a problem.

I also think that Google is getting better and better at dealing with urls that include query strings.

The reasons for this is because most of my site is dynamic and I have not used the mod re-write yet, and by looking at Google's search results for my site, about half of my pages are listed within Google with the query string attached.

Don't get me wrong though! I would still use the mod re-write. As I mentioned above, only about half of my pages are listed with query string attahced and not all of them.

[Grumpus]

How many pages are on your site? Bear in mind that PR comes into play when depth of crawl is considered. Dynamic or static, it doesn't really matter. As a general rule:

A PR4 homepage can expect about 13,000-18,000 pages in the index.
A PR5 homepage can expect about 17,000-25,000 pages in the index.
A PR6 homepage can expect about 24,000-38,000 pages in the index.

Those numbers aren't exactly right, but the ranges are close.

On large sites, using a mod-rewrite can cause more harm than it's worth. Not only does it add overhead to the server, but spiders crawl dynamic pages more slowly because they take a lot of server processing to generate them. (It doesn't crawl less, it just spends more time at your site getting it all). The only way a spider knows a page is dynamic is by it's name and how the page is called. If you are tricking a spider into thinking a page is static, then it's going to hit your site a lot more quickly. Combine the fast hits with regular usage (and sometimes several engines have spiders on your site at the same time) and you'll suddenly find your site's speed coming down to a crawl. Pretty soon, as that request queue on the server gets bigger and bigger, the server can even crash.

There are other benefits to using dynamic naming. I won't go into them here as it's a whole different area and would take considerable time to explain. But your time is far better spent learning what makes a dynamic site crawlable (which you are all set with since you are in there) and about the really cool things you can do with dynamic sites that'll increase your ranking even before the content of the page is figured in than to use a mod-rewrite and make your site so slow that all the traffic you get can't use your site anyway.

G.