Jump to content

  • Log in with Facebook Log in with Twitter Log In with Google      Sign In   
  • Create Account

Subscribe to HRA Now!

 



SEO Class in Chicago, IL

Learn How To Optimize Your Website on July 26, 2013


Looking for personalized in-depth SEO training among your peers?



High Rankings is offering a 1-day customized SEO training class in Chicago. Class size is limited so please sign-up now if you want in!



 


Are you a Google Analytics enthusiast?

Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE! 

 



 

 www.CustomReportSharing.com 

From the folks who brought you High Rankings!



Photo

Removing Robots.txt Exclusion


  • Please log in to reply
10 replies to this topic

#1 wizardprincess

wizardprincess

    HR 2

  • Active Members
  • PipPip
  • 32 posts

Posted 29 November 2004 - 01:29 AM

need help about my site back in google.
we have 4 different domain for the same site and all have same content and everything so it was making duplication of our website so i suggested to add either robots.txt or to redirect every domain to main site and finally it was decied to add robots.txt and our system admin did mistake of adding robots.txt file to main site which was made to stop all the spiders crawling other domains, and it was there for a week .
so google removed our site from database.. now even when i do URL search nothing comes up, i lost PR which i achived so far, and everything is gone.

i need help in this , if you have any tips related to how to get site back in google soon..
please give me... crossfingers.gif

#2 mcanerin

mcanerin

    HR 7

  • Active Members
  • PipPipPipPipPipPipPip
  • 2,242 posts
  • Location:Calgary, Alberta, Canada

Posted 29 November 2004 - 01:47 AM

I'm assuming you've removed or edited the robots.txt file now - when did you do this?

Google checks for the robots.txt once each session - so it will remember the exclusions during that visit, but forget them upon leaving the site.

I don't think you have a serious long term issue - but you will have to wait until the spider visits again and grabs the new robots.txt. Then there will be a slow "respidering" of the site.

I have had a client that accidently did something similar, so I contacted Google on their behalf and explained the situation, and it was fixed the following month.

However, I'm not sure that the fix was any faster than it would have been if I had not contacted them. It did give my client some relief to know that G knew about it and that the robots were coming back, so from a stress/psychological perspective contacting G was very helpful.

The speed you are re-indexed will be related to the size of your site and the number and PR of your incoming links (as well as your navigation structure and spiderability, of course).

Ian

#3 wizardprincess

wizardprincess

    HR 2

  • Active Members
  • PipPip
  • 32 posts

Posted 29 November 2004 - 01:58 AM

hi
currently i dont have any robots.txt file on my site. do you suggest adding one giving full access of the site to spiders.

#4 Scottie

Scottie

    Psycho Mom

  • Admin
  • 6,294 posts
  • Location:Columbia, SC

Posted 29 November 2004 - 02:04 AM

robots.txt will simply prevent a lot of 404 errors being generated when the spider asks for it- the absence of one is considered to be an OK-to-index.

Just open Notepad, create an empty file named robots.txt, and upload it to the root directory.

#5 wizardprincess

wizardprincess

    HR 2

  • Active Members
  • PipPip
  • 32 posts

Posted 29 November 2004 - 02:11 AM

ok i will do this..

will this help me in getting back in google soon.... ???

#6 Scottie

Scottie

    Psycho Mom

  • Admin
  • 6,294 posts
  • Location:Columbia, SC

Posted 29 November 2004 - 02:44 AM

It won't help or hurt your search engine optimization... it will simply prevent a lot of 404 errors showing in your error logs.

#7 wizardprincess

wizardprincess

    HR 2

  • Active Members
  • PipPip
  • 32 posts

Posted 29 November 2004 - 02:48 AM

ok.. i will add it...

and any tips for how to get back again in Google. ????

#8 Randy

Randy

    Convert Me!

  • Moderator
  • 17,540 posts

Posted 29 November 2004 - 08:12 AM

About all you can do at this point is wait for Googlebot to pay you another visit. As soon as it does it will see that it you are not excluding anything and crawl the entire site again.

#9 Jill

Jill

    High Rankings Advisor

  • Admin
  • 32,376 posts

Posted 29 November 2004 - 09:12 AM

Wizard Princess, if your various domains are all just one site on the same server, there's no way to use a robots.txt (that I know of) to exclude the robots from certain domains. I believe this is where you made the original mistake.

Read my article How to Redirect Multiple Domain Names to figure out how to straighten your situation out.

As to Google, just don't think about it and your site will soon be back in the database. There's really nothing you can do but wait, so just forget about it for a few days or weeks and eventually all will be fine. In the meantime, figure out how to do the redirect so you can get the other domains doing what you want.

Good luck!

#10 wizardprincess

wizardprincess

    HR 2

  • Active Members
  • PipPip
  • 32 posts

Posted 30 November 2004 - 12:47 AM

hi Jill

Thanks for your advice... and ARTICLE is also great help for me...


Thanks...

#11 wizardprincess

wizardprincess

    HR 2

  • Active Members
  • PipPip
  • 32 posts

Posted 01 December 2004 - 08:21 AM

hi
Thanks.. to all of you.. for helping me and solving my problem.....

Thanks..Scottie........Jill.........mcanerin for helping me....




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users