If I place a robots.txt file in a directory will that stop SE from indexing the content?
Or does the robots.txt need to be at the root level? Thanks
Subscribe to HRA Now!
SEO Class in Chicago, IL
Learn How To Optimize Your Website on July 26, 2013
Looking for personalized in-depth SEO training among your peers?
High Rankings is offering a 1-day customized SEO training class in Chicago. Class size is limited so please sign-up now if you want in!
Are you a Google Analytics enthusiast?
Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE!
www.CustomReportSharing.com
From the folks who brought you High Rankings!
More SEO Content
International SEM | Social Media | Search Friendly Design | SEO | Paid Search / PPC | Seminars | Forum Threads | Q&A | Copywriting | Keyword Research | Web Analytics / Conversions | Blogging | Dynamic Sites | Linking | SEO Services | Site Architecture | Search Engine Spam | Wrap-ups | Business Issues | HRA Questions | Online Courses
How To Prevent A Directory Within Site Being Indexed?
Started by
lister
, Mar 31 2012 07:45 PM
3 replies to this topic
#2
torka
-
- Moderator
-
- 4,392 posts
Vintage Babe
- Location:Triangle area, NC, USA, Earth (usually)
Posted 31 March 2012 - 09:06 PM
Robots.txt needs to be at the root level.
You can find everything you need to know about robots.txt at the http://www.robotstxt.org/ site.
--Torka
You can find everything you need to know about robots.txt at the http://www.robotstxt.org/ site.
--Torka
#3
lister
-
- Active Members
-
- 343 posts
HR 5
Posted 01 April 2012 - 12:00 AM
thanks very helpful - is this correct?
User-agent: *
Disallow: /contact-files/
I.e. i do NOT want any SE to index what is in /contact-files/
(I know that some bots do ignore the robots.txt though.....)
User-agent: *
Disallow: /contact-files/
I.e. i do NOT want any SE to index what is in /contact-files/
(I know that some bots do ignore the robots.txt though.....)
Edited by lister, 01 April 2012 - 12:01 AM.
#4
Michael Martinez
-
- Active Members
-
- 4,805 posts
HR 9
- Location:Georgia
Posted 01 April 2012 - 01:19 AM
If you don't want robots to crawl those pages, either put them behind a login or don't publish them. You can keep the good robots out of a directory but not the bad ones.
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users
