28 replies to this topic

[alnany]

[Moved to [url=http://www.highrankings.com/forum/index.php?showforum=38]Technology and Coding forum[/url].]


What exactly should the robot.txt contain? Should all sites have one? Thanks.

Edited by Jill, 18 September 2003 - 10:20 AM.

[Scottie]

Welcome Al Nany!

Check out http://www.robotstxt.org .

The short answer is yes, you should have one. If you aren't worried about excluding any robots at all, save a blank document in notepad as robots.txt and upload that to the root directory of your site.

[Mel]

Note that the correct file name is robots.txt (plural). If you want an allinclusive specimen which is set up to exclude most of the badbots out there, Brett Tabbke offers his for free downloading

[fred]

Hi

do you have a link to that file

thanks

[BrianR]

Mel

I've just checked Brett Tabke's site (www.searchengineworld.com) and I can't find the template robots.txt file - do you know where it's hidden!?

Thanks.

BrianR

[Jill]

Perhaps it's the regular robots.txt file that they have at WebmasterWorld. Did you look at www.webmasterworld.com/robots.txt?

Jill

[air-dog]

Hello All,

Mel mentioned "badbots" are some of these the spiders that capture email address', currently been discussed in Been Offered A Reciprical Link?

If so, does this mean that robots.txt files can be set up to deter these spammers?

[qwerty]

Bots don't automatically comply with the robots exclusion protocol. You can name as many spiders in your robots.txt you want, but a lot of them don't even look for it.

[BrianR]

Thanks, Jill - that's it exactly.

But Qwerty's comment seems to say that using the robots.txt file is next to useless because the badbots listed just ignore it - so why bother??

BrianR

[Scottie]

It's kind of like locking the doors when you leave the house- it's not going to stop a determined thief who wants to get in, but at least you tried.

[qwerty]

I wasn't saying that there's no point in having a robots.txt. If you have files or entire directories you don't want indexed, you can block all compliant spiders. If there are particular compliant spiders you want to block, you can.

It's just that if you happen to have the user-agent name of some email harvester, putting in an entry like

user-agent: evilharvester
disallow: /

probably won't help you.

[Scottie]

Another point too-

Sometimes your competitors or other nosy people will look in robots.txt to see what you don't want to have indexed. If it's important, use password protection, not robots.txt/

[BrianR]

Thanks Qwerty & Scottie - I understand now.

It's been a l-o-n-g week, and it's getting kinda late, and I'm just being thick!

BrianR

[guidaro]

Hi, you can check more information about robots.txt at http://www.google.com/webmasters/, basically you may to allow or not search engines submit your site.
Guidaro

[Corey Bryant]

You can also check out this one:

http://www.thinkhost.com/robots.txt