5 replies to this topic

[bumble]

Hi everyone

A little while ago I created a robots.txt file and put it at the root of my website to stop some webpages being indexed. However, it hasn't stopped the pages from being indexed. Can you confirm that I have the correct format please:

User-agent: *

Disallow: /site_info/contact.htm
Disallow: /site_info/contact_confirmation.htm
Disallow: /site_info/privacy_policy.htm



Many thanks <--This is not in the robots.txt :o)

[SearchRank]

If you are waiting for those pages to be removed from the index, that may take a while. Have you looked at your logs to see if these pages have been getting fetched? Have you compared the page info in the indices to see if it is current with what you currently have? How do you know that the engines are not fetching these pages is what I am trying to get at.

[bumble]

I had these pages in the robots.txt before I even put these pages online. I know google is constantly indexing these pages. When I do site:www.mydomain.com it shows that the pages were updated only a couple of days ago??

[qwerty]

Is that blank line in there after

User-agent: *

like you have it in your post? If that's the case, you need to remove it. Quoting from http://www.robotstxt...sion-admin.html :

...you may not have blank lines in a record, as they are used to delimit multiple records.

[bumble]

Thanks qwerty

I did have a blank line in there. That explains it.

What will happen now? Will google remove those pages the next time it reads my robots.txt?

[qwerty]

They should drop out over time, but I don't know how long it will take. If it's urgent, Google has the "automatic URL removal system". I've never used it, but you might want to give it a try.

I didn't find anything similar in Yahoo, except a way to remove a site from their directory, and you obviously don't want that.