Jump to content

  • Log in with Facebook Log in with Twitter Log In with Google      Sign In   
  • Create Account

Subscribe to HRA Now!

 



Are you a Google Analytics enthusiast?

Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE! 

 



 

 www.CustomReportSharing.com 

From the folks who brought you High Rankings!



Photo

So If I Want Spiders To See My Whole Site...


  • Please log in to reply
13 replies to this topic

#1 shakin318

shakin318

    HR 2

  • Active Members
  • PipPip
  • 26 posts

Posted 03 March 2011 - 09:37 PM

After GoDaddy's Search Engine Visibility product warned me with flashing blinking lights that "your site does not have a robots.txt file!" I made sure to generate and add one real quick! This was 3-4 weeks ago, when I was still a relative SEO "newbie," and my only frame of reference was the info from GoDaddy's SEV tool blink.gif

Seems to me after reading through this section of the forum, there's no need for me to have one if there's nothing on my site I want to hide from the search engines. But just to err on the side of caution, I'm asking the question here...

#2 Michael Martinez

Michael Martinez

    HR 10

  • Active Members
  • PipPipPipPipPipPipPipPipPipPip
  • 5,157 posts
  • Location:Georgia

Posted 04 March 2011 - 03:53 AM

Well, if you create XML sitemaps you can link to them from a robots.txt file. But you probably want to disallow your /cgi-bin/ directory from being crawled in any event.

#3 Jill

Jill

    Recovering SEO

  • Admin
  • 33,003 posts

Posted 04 March 2011 - 09:54 AM

It also prevents a lot of 404 errors on your server as the search engines will check for a robots.txt file often.

#4 shakin318

shakin318

    HR 2

  • Active Members
  • PipPip
  • 26 posts

Posted 04 March 2011 - 10:25 AM

Thanks for the replies. My site doesn't have a /cgi bin/ directory, nor have I determined the need for one (or lack thereof) at this point in my ongoing learning process of taking full webmaster control of my site. I have generated and included an XML sitemap for my site, although I seem to recall seeing something here on HighRankings about those being rather unnecessary as well. Sounds to me like there's no harm in having either of them, and there could be issues without them, so I leave them be.

Now I'm off to learn what I can about /cgi bin/ directories... biggrin.gif

#5 Jill

Jill

    Recovering SEO

  • Admin
  • 33,003 posts

Posted 04 March 2011 - 10:34 AM

What makes you think you need a cgi-bin directory? You only need that if you have cgi scripts that need to run within one.

I think you're looking at lots of unimportant things when you could instead be focusing on stuff that matters.

#6 shakin318

shakin318

    HR 2

  • Active Members
  • PipPip
  • 26 posts

Posted 04 March 2011 - 10:45 AM

I don't necessarily think I need a cgi bin directory...Michael's reply above is a good example of someone more knowledgeable than myself adding something new/foreign to the mix for me, which I then have to look into. I'm learning how to manage the technical/coding aspects of my site (without training) while at the same time sorting out the good info from the bad regarding SEO -- and then implementing the good. The results so far have been tremendous (going from "invisible" to google to having spots on the 1st, 2nd and 3rd pages for my main keyword phrases in roughly 5 weeks). It's a learning process, and it's all helpful.

#7 Michael Martinez

Michael Martinez

    HR 10

  • Active Members
  • PipPipPipPipPipPipPipPipPipPip
  • 5,157 posts
  • Location:Georgia

Posted 04 March 2011 - 01:54 PM

To be honest, I cannot think of a hosting account I have had access to that did not have a cgi-bin directory. But I'm with Jill. If you don't have one and don't have a need for one, you probably don't need to investigate it.

My biggest reason for blocking robot access to a /cgi-bin/ directory is to reduce the chances of hackers finding my scripts. No need to leave my tools laying out in the front yard for anyone to easily exploit.

#8 Twix

Twix

    HR 2

  • Members
  • PipPip
  • 21 posts

Posted 12 April 2011 - 07:12 AM

Are these necessary for the images folder now? I haven't had one up to this point because there wasn't anything I really needed to keep search engines out of, but am told Google is requiring them now for permission to crawl image folders ( because of copyright issues in using images ) so if I want to make use of uploading my product to Googlebase ( shopping ) I need to give Google permission to crawl my images folder with a robots.txt file?

#9 Jill

Jill

    Recovering SEO

  • Admin
  • 33,003 posts

Posted 12 April 2011 - 07:43 AM

The default is that they will crawl everything. Robots.txt is only to disallow.

#10 Twix

Twix

    HR 2

  • Members
  • PipPip
  • 21 posts

Posted 12 April 2011 - 07:57 AM

Thanks. That's what I had thought, and why I didn't have one, but am told Google is ready to roll out a new thing and will be requiring them to crawl image folders - because of all the copyright issues encountered from high end image stock image sites like corbis and getty.

#11 Jill

Jill

    Recovering SEO

  • Admin
  • 33,003 posts

Posted 12 April 2011 - 08:44 AM

Told by whom?

#12 Twix

Twix

    HR 2

  • Members
  • PipPip
  • 21 posts

Posted 12 April 2011 - 08:49 AM

A fellow website owner and web designer


QUOTE(Jill @ Apr 12 2011, 08:44 AM) View Post
Told by whom?



#13 Jill

Jill

    Recovering SEO

  • Admin
  • 33,003 posts

Posted 12 April 2011 - 09:33 AM

They were likely misunderstanding the information here:
http://base.google.c...mp;answer=50333

(Which is in fact, ambiguous.)

#14 Twix

Twix

    HR 2

  • Members
  • PipPip
  • 21 posts

Posted 12 April 2011 - 09:56 AM

Thanks for the reply. It's possible that's where it started and then snowballed - the information I got was that there was talk of this new thing rolling out ( requiring robots.txt to allow crawling of the image folder ) because of lawsuits Google faces from places like Getty/Corbis

Thanks again.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

SPAM FREE FORUM!
 
If you are just registering to spam,
don't bother. You will be wasting your
time as your spam will never see the
light of day!