Jump to content

  • Log in with Facebook Log in with Twitter Log In with Google      Sign In   
  • Create Account

Subscribe to HRA Now!


Are you a Google Analytics enthusiast?

Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE! 




From the folks who brought you High Rankings!


Robots.txt Was Ignored

  • Please log in to reply
2 replies to this topic

#1 seotiger


    HR 2

  • Members
  • PipPip
  • 18 posts

Posted 09 June 2009 - 02:32 PM

Recently noticed that Google indexed pages that are being blocked in my robots.txt file. I do a whole directory block. Everything is correct in terms of setup. robots.txt is in the root directory and the disallow is correct.

I successfully removed the directory manually through webmaster tools and that seems to have cleared the unwanted pages from the index, the only thing I can't figure out is why it all of a sudden ignored my robots.txt file.

I put a few more countermeasures in place to stop them like noindexing in the meta tag for the page and a rel="noindex" in the url to the page from other pages.

Wondering if anyony has experienced this and if you were able to determine the exact cause.

#2 Randy


    Convert Me!

  • Moderator
  • 17,540 posts

Posted 09 June 2009 - 03:02 PM

The only other times I've seen that happen is when there is an error in the robots.txt that kept the spiders from reading all of it. All of the major engine's bot are notoriously robots.txt compliant. Often going to the other extreme if there is a fatal error in the robots.txt.

I'd run the robots.txt through one or more of the several checkers out there to see what they say. Since this was a Google specific problem I'd surely run it through the one available in the Webmaster Tools area of Google.

#3 Michael Martinez

Michael Martinez

    HR 10

  • Active Members
  • PipPipPipPipPipPipPipPipPipPip
  • 5,325 posts
  • Location:Georgia

Posted 09 June 2009 - 07:31 PM

If you saw URL-only listings (no titles, snippets, caches, etc.) then the URL-removal tool may be your only option. Search engines will sometimes show a URL-only listing for pages that have inbound links pointing at them.

Google's safesearch mode will also show URL-only listings for some types of content.

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

We are now a read-only forum.
No new posts or registrations allowed.