Jump to content

  • Log in with Facebook Log in with Twitter Log In with Google      Sign In   
  • Create Account

Subscribe to HRA Now!

 



Are you a Google Analytics enthusiast?

Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE! 

 



 

 www.CustomReportSharing.com 

From the folks who brought you High Rankings!



Photo
- - - - -

Turning Off The Search Engine Bots From Indexing Your Site


  • Please log in to reply
7 replies to this topic

#1 ttw

ttw

    HR 5

  • Active Members
  • PipPipPipPipPip
  • 378 posts
  • Location:San Mateo, California

Posted 27 January 2010 - 09:46 AM

I was sitting in a new client meeting this week and the prospect told me that they had blocked the search engine bots from indexing their site while they were updating content. As far as I could tell this content update was taking weeks and weeks to accomplish and it sounded like SEO-optimization rather than dramatic content changes. Apparently their SEO firm suggested that they do this.

The boss of the man who told me this wild scenario was sitting in the meeting and while I wanted to shout "what a stupid thing to do", I did hold my tongue not wanting to embarrass the guy.

What am I missing here? What possible reason could an SEO consultant have for recommending this? Are there any longer term consequences once they remove the instructions from the Robots.txt file?

Rosemary

#2 Jill

Jill

    Recovering SEO

  • Admin
  • 32,961 posts

Posted 27 January 2010 - 09:53 AM

There is no possible reason why anyone would want to to that knowingly that I can think of.

You should strongly recommend that they turn the crawling back on ASAP!

#3 Mhoram

Mhoram

    HR 4

  • Active Members
  • PipPipPipPip
  • 114 posts
  • Location:Quincy, Illinois, USA

Posted 27 January 2010 - 10:33 AM

I could almost understand it if they were talking about a NEW part of the site, and they hadn't decided for certain how the URLs would work or something like that, then you might want to prevent access to just that part of the site until it was settled. But to block access to the entire site....yeah, that's just crazy talk.

#4 Michael Martinez

Michael Martinez

    HR 10

  • Active Members
  • PipPipPipPipPipPipPipPipPipPip
  • 5,120 posts
  • Location:Georgia

Posted 27 January 2010 - 05:47 PM

I'm having trouble imagining a good reason to disable crawling as well, although if they are conducting A/B tests through PPC to see which copy variations work best for obtaining conversions, that might explain the odd decision.

#5 ttw

ttw

    HR 5

  • Active Members
  • PipPipPipPipPip
  • 378 posts
  • Location:San Mateo, California

Posted 28 January 2010 - 09:32 AM

I was also wondering if there are any repercussions from turning bots on/off. Might Google trust this site less in the near future?

#6 Michael Martinez

Michael Martinez

    HR 10

  • Active Members
  • PipPipPipPipPipPipPipPipPipPip
  • 5,120 posts
  • Location:Georgia

Posted 28 January 2010 - 06:33 PM

None of the major search engines should care or penalize a site for disabling access to various portions of its content.

HOWEVER, the broken crawling pattern might hurt the internal link profile and it could strip some relevant content out of the index. I think the site may take a hit in long tail search traffic but it's hard to say if that will be significant.

Only time will tell.


#7 maleman

maleman

    HR 6

  • Active Members
  • PipPipPipPipPipPip
  • 677 posts

Posted 29 January 2010 - 07:46 PM

QUOTE
I was sitting in a new client meeting this week and the prospect told me that they had blocked the search engine bots from indexing their site while they were updating content

"blocked the search engine bots"...Why not shoot yourself in the foot while you're at it?

If it was my site, I would leave the existing site live and intact while I worked on the overhaul. I'd get all my updates completed and then upload the whole shootin-match to the server. An alternative would be to update a bit at a time. But I would never block the bots during the updating process.

Why were the bots blocked? whitehat.gif

#8 Vintor

Vintor

    HR 1

  • Members
  • Pip
  • 5 posts

Posted 20 February 2010 - 07:26 AM

Tell them they're stupid. Remove the blocks in robots.txt!




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

SPAM FREE FORUM!
 
If you are just registering to spam,
don't bother. You will be wasting your
time as your spam will never see the
light of day!