Jump to content

  • Log in with Facebook Log in with Twitter Log In with Google      Sign In   
  • Create Account

Subscribe to HRA Now!

 



Are you a Google Analytics enthusiast?

Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE! 

 



 

 www.CustomReportSharing.com 

From the folks who brought you High Rankings!



Photo

Does Have Robots.txt Have Any Glaring Mistakes?


  • Please log in to reply
1 reply to this topic

#1 lister

lister

    HR 5

  • Active Members
  • PipPipPipPipPip
  • 415 posts

Posted 10 July 2011 - 09:15 AM

Obviously for SEO purposes I need to make sure that this has been done correctly, Id really appreciate your feedback on this thanks:

(This is for a Word Press Blog)

Main thing I guess is to see that there is no glaring mistake thanks!!!

User-agent: *
Allow: /
Disallow: /cgi-bin
Disallow: /7/wp-admin
Disallow: /7/wp-includes
Disallow: /7/wp-content/plugins
Disallow: /7/wp-content/cache
Disallow: /7/wp-content/themes
Disallow: /7/wp-register.php
Disallow: /7/wp-login.php

Sitemap: http://www.site/site_map.html

*** I get confused between the / at the beginning and / at the end....some articles I have read said that a trailing / is vital to DISALLOW robots - hence this post. Thanks

#2 Alan Perkins

Alan Perkins

    Token male admin

  • Admin
  • 1,642 posts
  • Location:UK

Posted 10 July 2011 - 09:34 AM

It looks OK. It could maybe be more efficient. Be careful with the sitemap line - it must contain the full URL of the sitemap.

QUOTE(lister)
*** I get confused between the / at the beginning and / at the end....some articles I have read said that a trailing / is vital to DISALLOW robots - hence this post. Thanks

Disallow lines in robots.txt specify a head match relative to "/", the root of the domain. Therefore, every disallow value must BEGIN with a slash.

A disallow value will only END with a slash if it is specifying a directory to disallow. So a trailing slash is certainly not vital. Since disallow lines specify a head match, it's not even vital to have a trailing slash to disallow a directoy. E.g. both of these lines would disallow a directory called test:

Disallow: /test
Disallow: /test/

The difference is that the first line above would also disallow a file called "test.php", whereas the second line would not.





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

SPAM FREE FORUM!
 
If you are just registering to spam,
don't bother. You will be wasting your
time as your spam will never see the
light of day!