3 replies to this topic

[WaldenPond]

I just verified a site and wanted to test a newly created robots.txt file to block the entire site from google.

User-agent: *
Disallow: /

However after testing using google webmaster tools it appears that google is ignoring the file. Is this because the site was recently verified.

CODE

http://mydomain.com/
Allowed
Detected as a directory; specific files may have different restrictions
    
Allowed
Detected as a directory; specific files may have different restrictions

However, if I test using a non-google robots.txt file it appears to be working.

CODE

Access blocked by line 2
Disallow: /
Bot: googlebot
URL: http://mydomain.com/

I suspect for some reason google can't see the robots.txt file. Below is what's displayed inside of webmaster tools. However, if I click on the url, the contents are correctly displayed. I used a third-party syntax checker and it passed without any errors.

CODE

robots.txt file     Downloaded     Status
http://mydomain.com/robots.txt    7 hours ago     404 (Not found)

[chrishirst]

robots.txt is cached for 24 hours, so if you uploaded a different file after it had been requested Google will be using the earlier version for 24 hours.

[WaldenPond]

Thanks I suspected as much. Hmm kind of defeats the purpose of testing tweaks to robots.txt file. But who am I to comment on how google does things ;-0

[chrishirst]

Hmm kind of defeats the purpose of testing tweaks to robots.txt file

Not really, you simply have to be aware of the delay, and of course the fact that different DCs may have different copies of your robots.txt.

Check your site logs for the times that the robots.txt gets retrieved and organize your "tweaking" around that.