Jump to content

  • Log in with Facebook Log in with Twitter Log In with Google      Sign In   
  • Create Account

Subscribe to HRA Now!

 



Are you a Google Analytics enthusiast?

Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE! 

 



 

 www.CustomReportSharing.com 

From the folks who brought you High Rankings!



Photo
- - - - -

How To Prevent A Directory Within Site Being Indexed?


  • Please log in to reply
3 replies to this topic

#1 lister

lister

    HR 5

  • Active Members
  • PipPipPipPipPip
  • 415 posts

Posted 31 March 2012 - 07:45 PM

If I place a robots.txt file in a directory will that stop SE from indexing the content?

Or does the robots.txt need to be at the root level? Thanks

#2 torka

torka

    Vintage Babe

  • Moderator
  • 4,581 posts
  • Location:Triangle area, NC, USA, Earth (usually)

Posted 31 March 2012 - 09:06 PM

Robots.txt needs to be at the root level.

You can find everything you need to know about robots.txt at the http://www.robotstxt.org/ site.

--Torka :propeller:

#3 lister

lister

    HR 5

  • Active Members
  • PipPipPipPipPip
  • 415 posts

Posted 01 April 2012 - 12:00 AM

thanks very helpful - is this correct?

User-agent: *
Disallow: /contact-files/

I.e. i do NOT want any SE to index what is in /contact-files/

(I know that some bots do ignore the robots.txt though.....)

Edited by lister, 01 April 2012 - 12:01 AM.


#4 Michael Martinez

Michael Martinez

    HR 10

  • Active Members
  • PipPipPipPipPipPipPipPipPipPip
  • 5,042 posts
  • Location:Georgia

Posted 01 April 2012 - 01:19 AM

If you don't want robots to crawl those pages, either put them behind a login or don't publish them. You can keep the good robots out of a directory but not the bad ones.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

SPAM FREE FORUM!
 
If you are just registering to spam,
don't bother. You will be wasting your
time as your spam will never see the
light of day!