Jump to content

  • Log in with Facebook Log in with Twitter Log In with Google      Sign In   
  • Create Account

Subscribe to HRA Now!

 



Are you a Google Analytics enthusiast?

Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE! 

 



 

 www.CustomReportSharing.com 

From the folks who brought you High Rankings!



Photo

Why Would You Disallow The Cache?


  • Please log in to reply
2 replies to this topic

#1 DJKay

DJKay

    HR 5

  • Active Members
  • PipPipPipPipPip
  • 355 posts

Posted 27 August 2010 - 10:40 AM

Hi,

I just had a developer email me what he thinks should be in the robots txt. Its an ecommerce site its built in interspire, they have a license.

User-agent: *
Disallow: /cgi-bin/
Disallow: /other/
Disallow: /javascript/
Disallow: /config/
Disallow: /cache/

I don't understand why you would disallow the cache. I thought google worked from the cache and then just spidered whatever was new? Maybe I am mixing up cache's...Maybe its google working from their own cache...I know you want to disallow the cart, etc.

Is it an attempt to manage server load? I went on interspire's forum and found this configuration as a suggestion:

User-agent: *
Disallow: /account.php
Disallow: /cart.php
Disallow: /checkout.php
Disallow: /finishorder.php
Disallow: /login.php
Disallow: /orderstatus.php
Disallow: /postreview.php
Disallow: /productimage.php
Disallow: /productupdates.php
Disallow: /remote.php
Disallow: /rss.php
Disallow: /search.php
Disallow: /viewfile.php
Disallow: /wishlist.php
Disallow: /admin/
Disallow: /templates/

Thanks in advance. DJKay

#2 Alan Perkins

Alan Perkins

    Token male admin

  • Admin
  • 1,630 posts
  • Location:UK

Posted 27 August 2010 - 11:00 AM

QUOTE(DJKay @ Aug 27 2010, 04:40 PM) View Post
I don't understand why you would disallow the cache. I thought google worked from the cache and then just spidered whatever was new? Maybe I am mixing up cache's...Maybe its google working from their own cache...I know you want to disallow the cart, etc.


I'm not familiar with Interspire, but it's clear that you are mixing up caches. Without looking at Interspire in detail, "Disallow: /cache/" sounds like a perfectly sensible thing to be doing to prevent duplicate content and freshness issues.


#3 DJKay

DJKay

    HR 5

  • Active Members
  • PipPipPipPipPip
  • 355 posts

Posted 27 August 2010 - 11:51 AM

I thought I was mixing up caches but was not sure...okay..thanks,




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

SPAM FREE FORUM!
 
If you are just registering to spam,
don't bother. You will be wasting your
time as your spam will never see the
light of day!