will it hurt me to not have a robot.txt file? I see that most bots look for them before they search my site, but there is nothing on it that I want to hide from the bot. What do you think?
Subscribe to HRA Now!
Are you a Google Analytics enthusiast?
Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE!
www.CustomReportSharing.com
From the folks who brought you High Rankings!
More SEO Content
International SEM | Social Media | Search Friendly Design | SEO | Paid Search / PPC | Seminars | Forum Threads | Q&A | Copywriting | Keyword Research | Web Analytics / Conversions | Blogging | Dynamic Sites | Linking | SEO Services | Site Architecture | Search Engine Spam | Wrap-ups | Business Issues | HRA Questions | Online Courses
Sponsored Content
Need A Robot.txt File?
Started by
NEWIBE-2-YOUBE
, Jun 09 2004 11:50 PM
4 replies to this topic
#2
mcanerin
-
- Active Members
-
- 2,242 posts
HR 7
- Location:Calgary, Alberta, Canada
Posted 10 June 2004 - 12:16 AM
It won't *hurt* you to not have one, other than clogging up your logs with a bunch of useless errors.
But its so simple to add one, why not have it?
Ian
But its so simple to add one, why not have it?
Ian
#3
NEWIBE-2-YOUBE
-
- Active Members
-
- 67 posts
HR 3
Posted 10 June 2004 - 12:26 AM
so what should I have if I want to allow everything?
#4
mcanerin
-
- Active Members
-
- 2,242 posts
HR 7
- Location:Calgary, Alberta, Canada
Posted 10 June 2004 - 12:36 AM
The most complete resource is here:
http://www.robotstxt.../wc/robots.html
but you can basically create a brand new text file and put *only* the following text in:
Another good one would be:
Which prevents the spiders from going into those directories but allows everywhere else
Spider Banning:
The only change here is the slash / but it will turn an "allow everything" into an "allow nothing" - so be very careful.
Once you have created this file call it robots.txt and upload it so that you could find it here:
www.yourdomain.com/robots.txt
Make sure that you have set permissions so that it's readable, and you are done.
Ian
http://www.robotstxt.../wc/robots.html
but you can basically create a brand new text file and put *only* the following text in:
User-agent: *
Disallow:
Another good one would be:
User-agent: *
Disallow: /cgi-bin
Disallow: /logs
Which prevents the spiders from going into those directories but allows everywhere else
Spider Banning:
User-agent: *
Disallow: /
The only change here is the slash / but it will turn an "allow everything" into an "allow nothing" - so be very careful.
Once you have created this file call it robots.txt and upload it so that you could find it here:
www.yourdomain.com/robots.txt
Make sure that you have set permissions so that it's readable, and you are done.
Ian
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users
