6 replies to this topic

[Pets]

Hello,
I've been talking to my site builder about changing our dynamic URL's into static URL's. I've seen a lot of information on this forum and in other places about this.
The response I've gotten from my builder follows:

First of all, I want to clarify a misconception that many database-driven sites have all pages indexed by Google. There are, in fact, very few sites (Petsmart does happen to be one of them) that did go to the expense of having their site constructed without a query string. Plus, Google doesn’t like sites that are constructed that way because it bogs down their system and can make searches for a simple item very difficult. Can you imagine the challenge of a store with 10,000 items having every page indexed by an engine? That site would literally take over an engine, leaving no room for other stores to be seen 9for example, a search for “angel” would pull every page with an angel ornament on it and obliterate the search engine pages). Google will not allow that to happen (which is why they probably started the Froogle program to give stores an opportunity to showcase some product pages as web developers begin to figure out work-arounds for active server page catalogs). The other issue is that having direct path URLs for every product page would use tremendous server resources that are beyond the scope of the typical shared web server.

We build sites with query strings as that is the accepted (standard) method of building a store. To do otherwise is not only very time consuming and expensive to the client but impractical.

It may be possible for a third-party software or plug-in to help the pages of your store be seen by Google. Should you decide to go this route, a dedicated web server would be required in order to meet the load of server calls by search engines and also due to the fact we cannot install 3rd-party software on a shared server. We offer a dedicated server for $350.00/month, so it would increase your hosting cost, a call you’ll have to make. There would also be a set-up fee for the software (Scott set up 3rd party software on a dedicated server for another client a few days ago and it took him 12 hours to get the software installed and configured), and we cannot guarantee that any 3rd-party solution will work with the hosted web application (the guarantee would have to come from the software company). If it didn’t work, you would still incur the cost of setting up the server and the software, plus at least 3 months of server leasing as we would have to take out a contract for the server and be obliged to fulfill the contract.

Are they blowing wind my way or are these valid issues? Basically, they make it look like URL manipulation is like sending a man to mars. It can be done - it is very expensive - and it might not work. Also, what 3rd party software or plug-in might they be refering to??

Our e-commerce site has products at the 4th level of query, so google doesn't see any of our products.

Your thoughts and help are GREATLY appreciated.

Thank you very much!

[Haystack]

Hi Pets,
you're definitely losing out on some business until that's straightened out. I've seen dynamic sites receive as much as 40% of their traffic entering through pages other than the front page through searches on specific products or product categories. That traffic also tends to convert to sales so you can probably imagine the potential for your sales.

It sounds like this hasn't been done by your programmers before and they don't really care to learn how. There are others on this board who will probably have more to add to this and are very capable of helping you straighten things out. For now, if you'd like to learn more about the issue check out the article Alan wrote for Jill's newsletter last year:

http://www.highranki...ue022.htm#guest

[dimok]

To do otherwise is not only very time consuming and expensive to the client but impractical.

They are wrong. Our programmers do a pseudo static generation for one time and use it on each site.

Force them to do pseudo static by some way: 404 error processing or ModRewrite adjustment. It's not so difficult as they say

Our e-commerce site has products at the 4th level of query, so google doesn't see any of our products.

They solution is simple - do and extensive site map, where links to all pages are available.

[Mel]

If this is the best advice from your programmers start looking for new ones.

[Mel]

Hi Pets;
I guess my last post should have been longer, but I am really tired of .NET Programmers who have trouble creating sites they way they could and should, and then proceed to "educate" thier customers as to why it simply cannot be done and even if it could be why its a bad idea .

You might mention to your "programmers" that spidering a site of 10,000 pages is childs play to Googlebot, and to remind them that they currently index nearly 4 billion pages.

If you are having problems getting your pages spidered a good set of sitemap pages will help; download a copy of Xenu, turn on the sitemap option and let it spider your site and create a sitemap for you.

A few warnings however,
Your site spidering will often reveal that all your page titles are the same, and you may not have any meta description tags or Keywords tags. (I know that only Inktomi is reading the keywords tag, but I think it prudent to be prepared in case Yahoo makes something of Inktomi now that they own it)

For a large site like yours you should break your sitemaps into pages of less than 100 links on a page, and you should replace the titles and descriptions that are generated by keyword links and one or two sentence descriptions (viewers remember).

I have had almost a 100% failure rate on my customers IIS sites responding properly to the if modified since query. Look at your logs and if you are seeing lots of 304 responses it is likely that these are being sent by default, which seems to limit Googles spidering of a site.

BTW for FWIW it sounds to me like your programmers are not setting up the caching of your pages correctly, which can reduce the server load a lot, as well as speeding up loading of the pages themselves.

[Pets]

Thank you all. I knew I could rely on this group to be of help!
Please feel free to add any other thoughts you may have.
In your debt, - Pets

[Crifer]

Hi!

I start having second thought about dynamic content and dynamic urls. Those programmers are in a way right. 10 000 items as alot.. far from every store have that much but we have to start imagine and calculate. There is alot of dynamic content, almost every site today has a database as a backbone. Does anyone know how many online stores there is today or how many % of all the sites on the net?

Static pages always outrank dynamic pages, even if you remove the ? it might not help. I don't know exactly how every rewrite engine works, i made a quick check and apache mod_rewrite could produce a url like this:

ww.somesite.com/clothes/jeans/brand/3/model/4/
from
ww.somesite.com/clothes/jeans.php?brand=3&model=4

If i understood it correctly (any objections?). So my tought was.. yes, it removes the ? but if every page would rewrite like this, then where would google stop spidering if it is so SE friendly? Google must know that a url that look like that is dynamic, and put limitation on the spider. It's like math, 3 + 2 = 5 and 2 + 3 = 5. The same with the url. You can change places with the terms and get the same result. So don't bet before the race is over. The advantage is that you'll have a cleaner url that is easy to type. But i see it as dynamic as having the ? in the url. What do you think?

I found two articles about "dirty" urls (they are almost identical), i took a quick peek, and it looked like good reading.
http://www.masternew...nd_preserve.htm

http://www.port80sof...tgenerationurls

Pets
I concur with the others, get new programmers. If you want my recommendation, maybe even think about changing shoppingsystem..

Edited by Crifer, 23 September 2003 - 05:24 PM.