2 replies to this topic

[DJKay]

Hi,

I just had a developer email me what he thinks should be in the robots txt. Its an ecommerce site its built in interspire, they have a license.

User-agent: *
Disallow: /cgi-bin/
Disallow: /other/
Disallow: /javascript/
Disallow: /config/
Disallow: /cache/

I don't understand why you would disallow the cache. I thought google worked from the cache and then just spidered whatever was new? Maybe I am mixing up cache's...Maybe its google working from their own cache...I know you want to disallow the cart, etc.

Is it an attempt to manage server load? I went on interspire's forum and found this configuration as a suggestion:

User-agent: *
Disallow: /account.php
Disallow: /cart.php
Disallow: /checkout.php
Disallow: /finishorder.php
Disallow: /login.php
Disallow: /orderstatus.php
Disallow: /postreview.php
Disallow: /productimage.php
Disallow: /productupdates.php
Disallow: /remote.php
Disallow: /rss.php
Disallow: /search.php
Disallow: /viewfile.php
Disallow: /wishlist.php
Disallow: /admin/
Disallow: /templates/

Thanks in advance. DJKay

[Alan Perkins]

I don't understand why you would disallow the cache. I thought google worked from the cache and then just spidered whatever was new? Maybe I am mixing up cache's...Maybe its google working from their own cache...I know you want to disallow the cart, etc.

I'm not familiar with Interspire, but it's clear that you are mixing up caches. Without looking at Interspire in detail, "Disallow: /cache/" sounds like a perfectly sensible thing to be doing to prevent duplicate content and freshness issues.

[DJKay]

I thought I was mixing up caches but was not sure...okay..thanks,