Obviously for SEO purposes I need to make sure that this has been done correctly, Id really appreciate your feedback on this thanks:
(This is for a Word Press Blog)
Main thing I guess is to see that there is no glaring mistake thanks!!!
User-agent: *
Allow: /
Disallow: /cgi-bin
Disallow: /7/wp-admin
Disallow: /7/wp-includes
Disallow: /7/wp-content/plugins
Disallow: /7/wp-content/cache
Disallow: /7/wp-content/themes
Disallow: /7/wp-register.php
Disallow: /7/wp-login.php
Sitemap: http://www.site/site_map.html
*** I get confused between the / at the beginning and / at the end....some articles I have read said that a trailing / is vital to DISALLOW robots - hence this post. Thanks
Subscribe to HRA Now!
Are you a Google Analytics enthusiast?
Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE!
www.CustomReportSharing.com
From the folks who brought you High Rankings!
More SEO Content
International SEM | Social Media | Search Friendly Design | SEO | Paid Search / PPC | Seminars | Forum Threads | Q&A | Copywriting | Keyword Research | Web Analytics / Conversions | Blogging | Dynamic Sites | Linking | SEO Services | Site Architecture | Search Engine Spam | Wrap-ups | Business Issues | HRA Questions | Online Courses
Sponsored Content
Does Have Robots.txt Have Any Glaring Mistakes?
Started by
lister
, Jul 10 2011 09:15 AM
1 reply to this topic
#2
Alan Perkins
-
- Admin
-
- 1,559 posts
Token male admin
- Location:UK
Posted 10 July 2011 - 09:34 AM
It looks OK. It could maybe be more efficient. Be careful with the sitemap line - it must contain the full URL of the sitemap.
Disallow lines in robots.txt specify a head match relative to "/", the root of the domain. Therefore, every disallow value must BEGIN with a slash.
A disallow value will only END with a slash if it is specifying a directory to disallow. So a trailing slash is certainly not vital. Since disallow lines specify a head match, it's not even vital to have a trailing slash to disallow a directoy. E.g. both of these lines would disallow a directory called test:
Disallow: /test
Disallow: /test/
The difference is that the first line above would also disallow a file called "test.php", whereas the second line would not.
QUOTE(lister)
*** I get confused between the / at the beginning and / at the end....some articles I have read said that a trailing / is vital to DISALLOW robots - hence this post. Thanks
Disallow lines in robots.txt specify a head match relative to "/", the root of the domain. Therefore, every disallow value must BEGIN with a slash.
A disallow value will only END with a slash if it is specifying a directory to disallow. So a trailing slash is certainly not vital. Since disallow lines specify a head match, it's not even vital to have a trailing slash to disallow a directoy. E.g. both of these lines would disallow a directory called test:
Disallow: /test
Disallow: /test/
The difference is that the first line above would also disallow a file called "test.php", whereas the second line would not.
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users
