2 replies to this topic

[1dmf]

I'm confised with this error in GWMT

www.example.com/network.asp robots.txt unreachable Jun 7, 2008

HUH? here's my robots.txt file....

CODE

# Stop 404 page errors! and block unwanted indexing
User-agent: *
Disallow: /cgi-bin/
Disallow: /includes/
Disallow: /scripts/
Disallow: /int-sys/
Disallow: /websponsor/
Disallow: /tabs/
Disallow: /style/
Disallow: /depositit/
Disallow: /fees/
Disallow: /flash/
Disallow: /BFG/
Disallow: /members/
Disallow: /ARWebsites/
Disallow: /fees_old.asp

Can you see where i'm blocking the URL in the error, coz I can't!

The network.asp file exits, isn't being blocked, there is nothing wrong that I can see, is this just another bug in GWMT?

[Alan Perkins]

Don't worry about it. /network.asp was not generating a 404 error. It was your robots.txt file that was unreachable ...

Google Webmaster Help : What are URL unreachable errors?

robots.txt file unreachable

Before we crawled the pages of your site, we tried to check your robots.txt file to ensure we didn't crawl any pages that you had roboted out. However, your robots.txt file was unreachable. To make sure we didn't crawl any pages listed in that file, we postponed our crawl. When this happens, we return to your site later and crawl it once we can reach your robots.txt file. Note that this is different from a 404 response when looking for a robots.txt file. If we receive a 404, we assume that a robots.txt file does not exist and we continue the crawl.

[1dmf]

Well either way you look at it none of the files in questions are 404's!

I guess I'll just have to ignore errors in GWMT from now on 9/10 they are false negatives!