3 replies to this topic

[lister]

If I place a robots.txt file in a directory will that stop SE from indexing the content?

Or does the robots.txt need to be at the root level? Thanks

[torka]

Robots.txt needs to be at the root level.

You can find everything you need to know about robots.txt at the http://www.robotstxt.org/ site.

--Torka

[lister]

thanks very helpful - is this correct?

User-agent: *
Disallow: /contact-files/

I.e. i do NOT want any SE to index what is in /contact-files/

(I know that some bots do ignore the robots.txt though.....)

Edited by lister, 01 April 2012 - 12:01 AM.

[Michael Martinez]

If you don't want robots to crawl those pages, either put them behind a login or don't publish them. You can keep the good robots out of a directory but not the bad ones.