13 replies to this topic

[josh1r]

I'm not too familiar with the robots file but i've beeb reading a lot about it, yet I still can't figure out if it's better to have one or not.

I want all spiders to fully spider my site, I don't need to tell them not to do anything.

So my question is:

Yes or No, is it still better to have a robots.txt even if I don't want it to say anything?

Does it look 'bad' to not have one?

[Alan Perkins]

It's fine not to have one. No problem. It doesn't look bad.

However, for various reasons I suggest having an empty one.

[Denyse]

Alan could you expand on those various reasons - I would like to understand the rational

[Grumpus]

If for no other reason, it keeps your error log from filling up with 404 calls to the robots.txt file.

G.

[Alan Perkins]

Here's a couple of reasons.

Having a robots.txt means your Web server can handle the request without throwing an exception. It only causes one line in your logs, rather than a line in your access log and a line in your error log. I guess from the server's perspective it's more efficient to have one than to NOT have one, though I haven't measured this for each Web server.

Second reason: sooner or later you implement a 404 handler. Then requests for a missing robots.txt file cause the 404 handler to be called, and it has to deal with it. If it fails to deal with it (because you never thought of it!) then it may output something invalid, which may cause robots to avoid your site.

[josh1r]

In order to create a blank robots.txt do I literally just create a blank text file, save it as robots.txt, and upload it to my site?

Thanks.

[Alan Perkins]

That will do, although I use:

User-agent: *

Disallow:

Make sure the file is readable! If it generates a 403, that's far worse than a 404. Your site will not be indexed.

[josh1r]

Make sure the file is readable!  If it generates a 403, that's far worse than a 404.  Your site will not be indexed.

Thanks! How can I test to make sure the file is readable?

[Alan Perkins]

Try to access it using a standard Web browser and see what happens.

[josh1r]

Try to access it using a standard Web browser and see what happens.

I've added the robots.txt and I think it is ok. You wouldn't mind double-checking though would you?

http://www.italycharms.com/robots.txt



Thanks!

[Alan Perkins]

Perfect.

[josh1r]

Thanks Alan!

[josh1r]

Sorry, one more quick question. Now that I have the robots.txt uploaded to my site, do I need to put any code in my index file to tell the spiders to look for it? Or since it's basically blank does it not matter anyway?

Thanks again!

[Randy]

Nope, you don't need to do a thing Josh. The spiders will automatically look for the robots.txt file to see if you have any special instructions for them.

Oh, if only everything else with spiders was as simple and easy.