Are you a Google Analytics enthusiast?
More SEO Content
This May Be Dumb... But What Is Robot.txt?
Posted 24 October 2004 - 10:26 PM
Posted 24 October 2004 - 10:59 PM
The purpose of the robots.txt file is to prevent robots (bots, spiders, et al) from indexing pages and/or following links from particular areas of your site.
Eg. if you have a set of articles, and in a separate directory you have a print version.
You may decide you only want the for-display version to be indexed by the search engines. You would then specify in your robots.txt file:
Disallow: /print/ # printable version of our articles
This translates as for all crawlers, please don't visit my /print directory.
its important to note that you tell robots where NOT to go, not where they may go.
There is nothing that requires robots to adhere to the Robots Exclusion Protocol, but the big ones (google, yaho, etc) all do.
More details at:
Posted 25 October 2004 - 01:49 AM
Posted 25 October 2004 - 02:31 AM
If you dont have any areas that need excluding, the major use is to get rid of those pesky entries in your error file
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users