Are you a Google Analytics enthusiast?
More SEO Content
Google Crawling Locations Disallowed In Robots.txt?
Posted 21 February 2010 - 07:49 AM
And low behold what do I see reported by Crawl Rate Tracker within WordPress:
/wp-admin/ 2 crawls Links
/wp-login.php?redirect_to=http%3A%2F%2Fwww.mydomain... 1 crawls Links
/wp-login.php 1 crawls Links
/wp-admin/index-extra.php?jax=dashboard_incoming_l... 1 crawls Links
/wp-admin/index-extra.php?jax=dashboard_primary 1 crawls Links
/wp-admin/index-extra.php?jax=dashboard_secondary 1 crawls Links
/wp-admin/index-extra.php?jax=dashboard_plugins 1 crawls Links
/wp-admin/options-general.php?page=robots-meta 1 crawls Links
/wp-login.php?action=logout&_wpnonce=932793d5bb 1 crawls Links
/wp-login.php?loggedout=true 1 crawls Links
What the hell Google!
Is there something wrong with my Robots.txt?
Posted 21 February 2010 - 10:29 AM
Posted 21 February 2010 - 06:29 PM
Which I usually do, but forgot to do this time, despite me forgetting...the info in the robots.txt in my root directory should prevent those locations to be crawled.
The same Robots.txt is also showing up in Google Webmaster Central, so it's really odd.
Posted 21 February 2010 - 06:46 PM
Yeah, it definitely should. I don't think I've ever seen Googlebot request a page that was disallowed in robots.txt.
What happens if you test one of the disallowed URLs in the Crawler Access section of WMT? Does it report back that it's blocked?
Posted 22 February 2010 - 04:47 AM
I guess Googlebot just doesn't always adhere to the Robots.txt
1 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users