Jump to content

  • Log in with Facebook Log in with Twitter Log In with Google      Sign In   
  • Create Account

Subscribe to HRA Now!

 



Are you a Google Analytics enthusiast?

Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE! 

 



 

 www.CustomReportSharing.com 

From the folks who brought you High Rankings!



Photo

Google Webmaster Tools Not Reading My Robots.txt


  • Please log in to reply
3 replies to this topic

#1 WaldenPond

WaldenPond

    HR 2

  • Active Members
  • PipPip
  • 42 posts

Posted 01 July 2010 - 09:13 PM

I just verified a site and wanted to test a newly created robots.txt file to block the entire site from google.

User-agent: *
Disallow: /

However after testing using google webmaster tools it appears that google is ignoring the file. Is this because the site was recently verified.
CODE
http://mydomain.com/
Allowed
Detected as a directory; specific files may have different restrictions
    
Allowed
Detected as a directory; specific files may have different restrictions


However, if I test using a non-google robots.txt file it appears to be working.

CODE
Access blocked by line 2
Disallow: /
Bot: googlebot
URL: http://mydomain.com/


I suspect for some reason google can't see the robots.txt file. Below is what's displayed inside of webmaster tools. However, if I click on the url, the contents are correctly displayed. I used a third-party syntax checker and it passed without any errors.


CODE
robots.txt file     Downloaded     Status
http://mydomain.com/robots.txt    7 hours ago     404 (Not found)


#2 chrishirst

chrishirst

    A not so moderate moderator.

  • Moderator
  • 6,795 posts
  • Location:Blackpool UK

Posted 02 July 2010 - 04:05 AM

robots.txt is cached for 24 hours, so if you uploaded a different file after it had been requested Google will be using the earlier version for 24 hours.

#3 WaldenPond

WaldenPond

    HR 2

  • Active Members
  • PipPip
  • 42 posts

Posted 02 July 2010 - 10:05 AM

Thanks I suspected as much. Hmm kind of defeats the purpose of testing tweaks to robots.txt file. But who am I to comment on how google does things ;-0


#4 chrishirst

chrishirst

    A not so moderate moderator.

  • Moderator
  • 6,795 posts
  • Location:Blackpool UK

Posted 04 July 2010 - 03:54 AM

QUOTE
Hmm kind of defeats the purpose of testing tweaks to robots.txt file
Not really, you simply have to be aware of the delay, and of course the fact that different DCs may have different copies of your robots.txt.

Check your site logs for the times that the robots.txt gets retrieved and organize your "tweaking" around that.





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

SPAM FREE FORUM!
 
If you are just registering to spam,
don't bother. You will be wasting your
time as your spam will never see the
light of day!