Are you a Google Analytics enthusiast?
More SEO Content
Is It Ok Not To Have A Robots.txt ?
Posted 24 March 2004 - 09:46 AM
I want all spiders to fully spider my site, I don't need to tell them not to do anything.
So my question is:
Yes or No, is it still better to have a robots.txt even if I don't want it to say anything?
Does it look 'bad' to not have one?
Posted 24 March 2004 - 09:54 AM
However, for various reasons I suggest having an empty one.
Posted 24 March 2004 - 09:57 AM
Posted 24 March 2004 - 10:02 AM
Posted 24 March 2004 - 10:05 AM
Having a robots.txt means your Web server can handle the request without throwing an exception. It only causes one line in your logs, rather than a line in your access log and a line in your error log. I guess from the server's perspective it's more efficient to have one than to NOT have one, though I haven't measured this for each Web server.
Second reason: sooner or later you implement a 404 handler. Then requests for a missing robots.txt file cause the 404 handler to be called, and it has to deal with it. If it fails to deal with it (because you never thought of it!) then it may output something invalid, which may cause robots to avoid your site.
Posted 24 March 2004 - 10:19 AM
Posted 24 March 2004 - 10:21 AM
User-agent: * Disallow:Make sure the file is readable! If it generates a 403, that's far worse than a 404. Your site will not be indexed.
Posted 24 March 2004 - 11:56 AM
Thanks! How can I test to make sure the file is readable?
Make sure the file is readable! If it generates a 403, that's far worse than a 404. Your site will not be indexed.
Posted 24 March 2004 - 12:10 PM
Posted 25 March 2004 - 05:14 PM
Posted 25 March 2004 - 05:25 PM
Oh, if only everything else with spiders was as simple and easy.
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users