 Double Robots.txt?, Why should websites use Double Robots.txt?? |
| Important Announcement: ***Need an Affordable SEO Website Review?*** |
  |
 Mar 31 2009, 04:08 PM
Post
#1
|
|
 HR 6  Group: Active Members Posts: 590 Joined: 5-August 03 User's local time: Feb 9 2010, 03:20 PM From: Kentucky Member No.: 301  |
I noticed the other day that a website had a robots.txt in the root of the domain and as well as one in the root of the blog directory. I am wondering how many other people out there do this? Do you find the bots listening to both of them properly?
My feeling is that you only need the one in the website root and direct the bot to do what you want from there? Why use two of them? |
 |
 
|
|
Mar 31 2009, 04:36 PM
Post
#2
|
|
 Convert Me!  Group: Admin Posts: 17,377 Joined: 17-August 03 User's local time: Feb 9 2010, 01:20 PM Member No.: 551 |
robots.txt anywhere but the Root level will be ignored by the spiders. In fact it would surprise me if it's ever even queried. robots.txt is not like .htaccess where you can control things on a per directory level.
The only way a subdirectory robots.txt might be valid is the rare case where someone has a domain name parked on a subdirectory of another domain. Or possibly if the subdirectory is really a subdomain, though that one too is questionable in my mind and isn't something I've tested to see if spiders look for a robots.txt for each subdomain. Maybe Alan knows the answer to that one? |
|
|
|
|
Mar 31 2009, 05:07 PM
Post
#3
|
|
|
HR 6 Group: Active Members Posts: 590 Joined: 5-August 03 User's local time: Feb 9 2010, 03:20 PM From: Kentucky Member No.: 301 |
Actually that is what I figured Randy. Thanks for the feedback.
I have heard of different robots.txt for https and http sites before |
|
|
|
|
Mar 31 2009, 06:22 PM
Post
#4
|
|
|
HR 2 Group: Members Posts: 20 Joined: 8-December 08 User's local time: Feb 9 2010, 02:20 PM Member No.: 22,821 |
open: ./htdocs/robots.txt:
User-agent: Googlebot Disallow: /blog/ |
|
|
|
|
Apr 1 2009, 12:07 AM
Post
#5
|
|
 HR 6  Group: Moderator Posts: 918 Joined: 24-July 03 User's local time: Feb 9 2010, 02:20 PM From: Michigan USA Member No.: 17 |
QUOTE Or possibly if the subdirectory is really a subdomain, though that one too is questionable in my mind and isn't something I've tested to see if spiders look for a robots.txt for each subdomain. They do, Randy. They do. FWIW, I almost always back up a file before modifying it. My ex-wife always said I had trust issues? At any rate, I probably have a few copies of robots.txt laying around on more than a few sites. I don't worry about it because, as you pointed out, the only one that counts is in the root. |
|
|
|
|
Apr 1 2009, 06:26 AM
Post
#6
|
|
|
HR 2 Group: Active Members Posts: 25 Joined: 28-March 09 User's local time: Feb 9 2010, 02:20 PM Member No.: 25,440 |
QUOTE(Ron Carnell @ Apr 1 2009, 12:07 AM)  They do, Randy. They do. FWIW, I almost always back up a file before modifying it. My ex-wife always said I had trust issues? At any rate, I probably have a few copies of robots.txt laying around on more than a few sites. I don't worry about it because, as you pointed out, the only one that counts is in the root. or even better, source control everything; and i mean EVERYTHING (ok, maybe not the wife) |
|
|
|
 |
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:
This forum is sponsored by High Rankings, a Boston SEO Agency
|
Lo-Fi Version | Time is now: 9th February 2010 - 02:20 PM |