24 replies to this topic

[Jill]

Surely anyone could access the robots.txt file (of any site) and get a complete road-map into the stuff you asked the bot´s NOT to show!

That's why you should never actually have it show any link that someone could get to directly, that you wouldn't want a person to see.

I use directories for that purpose. Just put your secret file in the blocked directory, and nobody can get to it, assuming your server is set up correctly so that it doesn't allow people to browse directories.

[Alan Perkins]

Just put your secret file in the blocked directory, and nobody can get to it

That's not actually true. "Security by obscurity" is quite a low form of security. Once the URL is discovered, the security is broken. It's fine if you just want to put up minimum barriers and keep things simple without worrying about passwords, secure servers etc.

[Jill]

Right, but they can't find stuff just by using your robots.txt file, correct? Or am I wrong in thinking that?

[Alan Perkins]

Right, but they can't find stuff just by using your robots.txt file, correct?

Correct, not if you use obscure file names and hide directory listings.

[1dmf]

Surely anyone could access the robots.txt file (of any site) and get a complete road-map into the stuff you asked the bot´s NOT to show!

Sounds like hiding the front door key under a flower pot and then sticking a note on the front door with instructions where to find it

love ya thinking

[Jill]

Correct, not if you use obscure file names and hide directory listings.

Good because that's what I meant in my previous post!

[Affan Laghari]

Seems I'm a bit late but just read the [url=http://www.highrankings.com/newsletter]High Rankings Advisor[/url] Feeds.

Anyway, talking about indexing, I was quite amazed to see Brett Tabke run a blog in webmasterworld's robots.txt (www.webmasterworld.com/robots.txt). Google would surely miss unless it starts indexing these. The blog seemed to have regular posting around an year ago though it's dead now.

[JohnMu]

Hi everyone
This is always an interesting topic so I thought I'd jump on over and leave my 2 cents

- The quickest way to get it out of the index is to use the "noindex" x-robots-tag (as Alan mentioned a bit back). These HTTP headers are understood by all of the major search engines (as far as I know).

- Another way to get it removed is to block it with the robots.txt using a "disallow" directive. In a situation like that, we'll still read your robots.txt (because we have to check it anyway), but it'll generally take a bit of time for it to drop out of the index (because "disallow" blocks crawling, but not indexing). A way to speed that up is to use the urgent URL removal tool within your Google Webmaster Tools account.

- That said, it's not like your robots.txt file is going to rank for anything really interesting, so I wouldn't worry about it being indexed.

For what it's worth, the WMW robots.txt is using cloaking. You can see that by looking at the cached file:
http://www.webmaster....com/robots.txt
vs
http://www.google.co....com/robots.txt

I would certainly NOT recommend doing that. Not only is it against our Webmaster Guidelines, but if you have the slightest hickup in the cloaking code you'll suddenly be serving the wrong robots.txt file. Don't play with your robots.txt file

John

[Randy]

Welcome back John !

Is this your way of reminding me that I haven't gotten by your blog in a month or so?

[Robert W. Franson]

That WebMasterWorld blog certainly is an example of applying creative thinking to a corner of the Web that we ordinarily don't think of as anything but minimally utilitarian! ... No doubt for good reason.

As for security: It's also an example of one of the oldest tricks in the cryptology books for hiding secrets, including "secret writing": hide it in plain sight. Of course that's only a make-do substitute for real hiding or real encryption or both.