Jump to content

  • Log in with Facebook Log in with Twitter Log In with Google      Sign In   
  • Create Account

Subscribe to HRA Now!


Are you a Google Analytics enthusiast?

Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE! 




From the folks who brought you High Rankings!

- - - - -

How To Prevent A Directory Within Site Being Indexed?

  • Please log in to reply
3 replies to this topic

#1 lister


    HR 5

  • Active Members
  • PipPipPipPipPip
  • 426 posts

Posted 31 March 2012 - 07:45 PM

If I place a robots.txt file in a directory will that stop SE from indexing the content?

Or does the robots.txt need to be at the root level? Thanks

#2 torka


    Vintage Babe

  • Moderator
  • 4,825 posts
  • Location:Triangle area, NC, USA, Earth (usually)

Posted 31 March 2012 - 09:06 PM

Robots.txt needs to be at the root level.

You can find everything you need to know about robots.txt at the http://www.robotstxt.org/ site.

--Torka :propeller:

#3 lister


    HR 5

  • Active Members
  • PipPipPipPipPip
  • 426 posts

Posted 01 April 2012 - 12:00 AM

thanks very helpful - is this correct?

User-agent: *
Disallow: /contact-files/

I.e. i do NOT want any SE to index what is in /contact-files/

(I know that some bots do ignore the robots.txt though.....)

Edited by lister, 01 April 2012 - 12:01 AM.

#4 Michael Martinez

Michael Martinez

    HR 10

  • Active Members
  • PipPipPipPipPipPipPipPipPipPip
  • 5,325 posts
  • Location:Georgia

Posted 01 April 2012 - 01:19 AM

If you don't want robots to crawl those pages, either put them behind a login or don't publish them. You can keep the good robots out of a directory but not the bad ones.

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

We are now a read-only forum.
No new posts or registrations allowed.