13 replies to this topic

[shakin318]

After GoDaddy's Search Engine Visibility product warned me with flashing blinking lights that "your site does not have a robots.txt file!" I made sure to generate and add one real quick! This was 3-4 weeks ago, when I was still a relative SEO "newbie," and my only frame of reference was the info from GoDaddy's SEV tool

Seems to me after reading through this section of the forum, there's no need for me to have one if there's nothing on my site I want to hide from the search engines. But just to err on the side of caution, I'm asking the question here...

[Michael Martinez]

Well, if you create XML sitemaps you can link to them from a robots.txt file. But you probably want to disallow your /cgi-bin/ directory from being crawled in any event.

[Jill]

It also prevents a lot of 404 errors on your server as the search engines will check for a robots.txt file often.

[shakin318]

Thanks for the replies. My site doesn't have a /cgi bin/ directory, nor have I determined the need for one (or lack thereof) at this point in my ongoing learning process of taking full webmaster control of my site. I have generated and included an XML sitemap for my site, although I seem to recall seeing something here on HighRankings about those being rather unnecessary as well. Sounds to me like there's no harm in having either of them, and there could be issues without them, so I leave them be.

Now I'm off to learn what I can about /cgi bin/ directories...

[Jill]

What makes you think you need a cgi-bin directory? You only need that if you have cgi scripts that need to run within one.

I think you're looking at lots of unimportant things when you could instead be focusing on stuff that matters.

[shakin318]

I don't necessarily think I need a cgi bin directory...Michael's reply above is a good example of someone more knowledgeable than myself adding something new/foreign to the mix for me, which I then have to look into. I'm learning how to manage the technical/coding aspects of my site (without training) while at the same time sorting out the good info from the bad regarding SEO -- and then implementing the good. The results so far have been tremendous (going from "invisible" to google to having spots on the 1st, 2nd and 3rd pages for my main keyword phrases in roughly 5 weeks). It's a learning process, and it's all helpful.

[Michael Martinez]

To be honest, I cannot think of a hosting account I have had access to that did not have a cgi-bin directory. But I'm with Jill. If you don't have one and don't have a need for one, you probably don't need to investigate it.

My biggest reason for blocking robot access to a /cgi-bin/ directory is to reduce the chances of hackers finding my scripts. No need to leave my tools laying out in the front yard for anyone to easily exploit.

[Twix]

Are these necessary for the images folder now? I haven't had one up to this point because there wasn't anything I really needed to keep search engines out of, but am told Google is requiring them now for permission to crawl image folders ( because of copyright issues in using images ) so if I want to make use of uploading my product to Googlebase ( shopping ) I need to give Google permission to crawl my images folder with a robots.txt file?

[Jill]

The default is that they will crawl everything. Robots.txt is only to disallow.

[Twix]

Thanks. That's what I had thought, and why I didn't have one, but am told Google is ready to roll out a new thing and will be requiring them to crawl image folders - because of all the copyright issues encountered from high end image stock image sites like corbis and getty.

[Jill]

Told by whom?

[Twix]

A fellow website owner and web designer

Told by whom?

[Jill]

They were likely misunderstanding the information here:
http://base.google.c...mp;answer=50333

(Which is in fact, ambiguous.)

[Twix]

Thanks for the reply. It's possible that's where it started and then snowballed - the information I got was that there was talk of this new thing rolling out ( requiring robots.txt to allow crawling of the image folder ) because of lawsuits Google faces from places like Getty/Corbis

Thanks again.