Are you a Google Analytics enthusiast?
More SEO Content
Help Please Re: /robots.txt File
Posted 09 July 2004 - 09:24 AM
I launched a new website this week and have been in touch with a few companies re:SEO etc and one company said for a start that the site did not have a /robots.txt file which could hinder search engines ability to rank it, whilst another company advised me that it was better to have the file but it was not mandatory. So i'm asking the experts,
a. What is this file
b. Is it required
c. How do I create and add it?
Posted 09 July 2004 - 09:30 AM
What is the robots.txt file? See The Robots Exclusion Protocol
The file is not required, but it's often a good idea to have one anyway.
How do I create and add it? Assuming you want search engines to view your entire site, create the following file in Notepad:
User-agent: * Disallow:Save it as robots.txt, and upload to the root directory of your Web site.
Posted 09 July 2004 - 09:45 AM
Posted 13 July 2004 - 06:57 AM
Since in robots.txt file there are only fields for types of robots(user agents) and Disallow for specifying files that should not be indexed, is there any method by which I can also specify the time duration for the robots to crawl. You can specify it in metas tag, is there no provision here in robots.txt file?
Posted 13 July 2004 - 07:11 AM
Usually, you will have no need to do this, though, as search engine spiders are usually pretty good about not over-spidering.
Posted 13 July 2004 - 08:35 AM
User-agent: msnbot Crawl-Delay: 20
Where the number is the minimum you want the bot to wait before retrieving another page.
All of the major spiders these days measure who quickly your server responds to requests and try to factor that into their crawl rate, so normally it's not an issue.
I've never seen anywhere that Google mentions supporting this particular non-standard instruction. They simply ask you to email them (email@example.com) if you need to slow down their spider for some reason, including as many details as possible and snippets from your log file.
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users