Jump to content

  • Log in with Facebook Log in with Twitter Log In with Google      Sign In   
  • Create Account

Subscribe to HRA Now!

 



Are you a Google Analytics enthusiast?

Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE! 

 



 

 www.CustomReportSharing.com 

From the folks who brought you High Rankings!



Photo
- - - - -

Robots.txt Question


Best Answer Alan Perkins , 09 October 2013 - 09:29 AM

How can I add that to the robots.txt file without stopping google from crawing the whole site.

 

You can't do it with robots.txt.

 

There are ways to attempt to NOINDEX /, even when / is a HTTP redirect, but I'm not going to tell you what they are because it's a really bad idea! Basically, you need to accept that / is going to be indexed. It's no big deal.

Go to the full post


  • Please log in to reply
3 replies to this topic

#1 dank

dank

    HR 1

  • Members
  • Pip
  • 1 posts

Posted 08 October 2013 - 02:31 PM

Hello guys,

I have a question about how to do a disallow.

 

When some one hits my site they are directed to a language based on their ip address. For example I have /us and /uk that it redirects to. Google tries to index the root /

How can I add that to the robots.txt file without stopping google from crawing the whole site.

 

Thanks in advanced for you help!

 

Dan



#2 chrishirst

chrishirst

    A not so moderate moderator.

  • Moderator
  • 6,714 posts
  • Location:Blackpool UK

Posted 08 October 2013 - 04:30 PM

Google tries to index the root /

So you are using a javascript redirect then I guess, switch to a server side redirect and Google will then index starting from the /us/ sub folder.

 

BUT if you do block Google from the root folder in robots.txt, ... then as you say, you block EVERY single URI on the hostname from being indexed by Google, and there is no way around that.



#3 qwerty

qwerty

    HR 10

  • Moderator
  • 8,585 posts
  • Location:Somerville, MA

Posted 08 October 2013 - 06:05 PM

If you're redirecting based on IP, then as Chris mentioned, Google will index starting from the US folder, because all Googlebots have US IP addresses. That means that Google can't access any content outside that folder.

 

This is a little off topic, but you may find it useful to give users (and bots) a choice. You can redirect them to the country/language based content their IP suggests, but if you leave up links to other language options, you'll allow your other content to be indexed, and you'll give French people visiting the UK something a bit easier to read.



#4 Alan Perkins

Alan Perkins

    Token male admin

  • Admin
  • 1,642 posts
  • Location:UK

Posted 09 October 2013 - 09:29 AM   Best Answer

How can I add that to the robots.txt file without stopping google from crawing the whole site.

 

You can't do it with robots.txt.

 

There are ways to attempt to NOINDEX /, even when / is a HTTP redirect, but I'm not going to tell you what they are because it's a really bad idea! Basically, you need to accept that / is going to be indexed. It's no big deal.


  • dank likes this




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

SPAM FREE FORUM!
 
If you are just registering to spam,
don't bother. You will be wasting your
time as your spam will never see the
light of day!