9 replies to this topic

[Webseeker]

I was doing some reading and came across the talk of the /robot.txt after an URL.

What is this and how is it used? I recall doing a search of my domain using that after my URL and nothing came up.

Is it something I need to be using or installing at my site?

Could someone explain or point me in the right direction so I can understand this more thoroughly?

I am new at this whole SEO. I'm the SEO expert in my business. LOL! My webdesigner doesn't know a thing about it. In fact, I had to tell him to install keywords, meta description, titles... everything! ... so your help is appreciated.

Thanks,

Webseeker

[robwatts]

Hi webseeker

Basically its a text file that is used to tell robots how you would like them to behave on your site.

When a spider/bot visits your domain it will usually requuest a robots.txt file. This can tell the bot what it can and cannot touch.

You can restrict files and directories, block some agents and allow others.

Its not mandatory, and not all bots are robot.txt compliant.

You'd need to create one in a text editor and upload it to your domains root directory. Be sure that it is syntactically correct or you could find yourself inadvertantly blocking robots.


This query has some handy resources

Welcome to HR

[Webseeker]

Hello robwatts,

Wow, that's one info-filled link you gave me! I did a lot of reading, but couldn't really find what I was looking for...

... and wanted to know if it is something that "should" be on my site as I have no robot.txt right now. However, your answer above "Its not mandatory..." puts me at ease a little. I thought maybe I'd get penalized in some fashion by not having it, but I guess not.

Thanks for your help. I appreciate it!

Webseeker

[Jill]

Welcome Webseeker!

We've also discussed  robots.txt[/hr] here a few times.

Jill

[Randy]

Welcome Webseeker !

Two quick hits to add to the above...

Make very sure that you use robots.txt (plural) and not robot.txt in the singular. The former is what they look for. The latter will do nothing either way.

Yeah, I know I'm being pendantic. It's just a very minor error that I see an awful lot. It's such an easy typo to make after all.

Second, robots.txt or any other type of robots Meta tag is an exclusionary thing. If you have no reason to exclude anything from the spiders view there is no need to use it. You cannot use robots.txt to tell a spider to do something that it normally wouldn't do.

Or put a better way, the only time need to use a robots.txt is when you want to put up a big, flashing Don't Go Here sign for the spiders.

[mcanerin]

I've put together a free robots.txt creator, if you are interested in using it - but if your intent is to allow all robots everywhere, then you don't need to use anything at all.

What happens is that when a crawler first lands on a page in your domain, it looks for a robots.txt file - if it finds one, it obeys it. If it doesn't, then it assumes that the site is fully public and indexes whatever it can find - which is probably exactly what you want

Two last thoughts - 1) this file is NOT intended for security! It has no effect on humans visiting or finding pages on your site - you need to use actual file level security to do that.

2) If you do have areas that you want to stop robots from visiting, the best way to do it is to use the robots.txt to deal with files inside sub-folders, and the robots meta tag to deal with individual files.

For example, if you have a large number of PDF's, or a test site, you would put them in their own folder and use the robots.txt to tell the SE to avoid that whole folder and everything in it. If it was just a few pages in a folder you ordinarily want indexed (like your root directory) then you would use the robots metatag to stop the indexing of those particular pages.

One reason many SEO's prefer to have a robots.txt file on a site that is otherwise open is simply housekeeping. Since everytime a search engine comes to your site it checks for a robots.txt file, if you don't have on it generates a 404 Not Found error in your logs.

Since looking at and understanding web logs is important to SEO, we like to have nice clean, easy to read logs without a lot of errors in them - so we recommend the existance of a robots.txt file. That's why you will see the recommendation.

In short, it's a good idea, but don't worry about it if you don't have one and your site is intended for the public.

Ian

[Webseeker]

Hello? Nice people! You are being too nice. I'm not use to getting so much good input in so short a time... I'm shocked! Your replies has really enlightened my on this subject matter. I see eventually I'll have a robots.txt at the site, but for now, I'll go without, being I see it's not entirely necessary.

Now I can breathe a sigh of relief.

Thanks for the help. I needed it. LOL!

Webseeker

[Randy]

Too nice?

hmmm... No just normal. Maybe those other folks are abnormal.

On the other hand, unless you enjoy a combination of whips driven by propellers you may want to steer clear of Torka.

[leadegroot]

Ooh! I want to be nice too!
I always have a robots.txt file, even if it is empty, or more likely has the basic 2-line 'Welcome! Come on in!' syntax.
I love a clean error file, and it can really fill up with 404s on the robots.txt file

[torka]

I saw that, Randy.

--Torka