8 replies to this topic

[mitash]

Could the experts clear my doubt pls...

Is it true that spiders do not always obey robots.txt..?

If that's true has someone done any testing?

Or how can we proove that robots.txt is not always obeyed by spiders / bots ..?

Thanks

[Jill]

It depends on the spider. The major search engines obey it.

[mitash]

Ok lets say google, msn and yahoo for example...

Would spiders from above engines ignore the robots.txt file...

Do you know what other engines will igonre the robots.txt...?

[projectphp]

All those obey it, it just comes down to whether you get the syntax right.

Try signing up for sitemaps, I am pretty sure they have a sitemap checker.

[torka]

Yes, at Google Webmaster Tools (formerly Google Sitemaps), they do offer a robots.txt checker.

--Torka

[Alan Perkins]

Ok lets say google, msn and yahoo for example...

Would spiders from above engines ignore the robots.txt file...

No, they wouldn't ignore it ... but they may not treat it as you would expect.

• Google may index URLs that are protected by robots.txt, without reading or indexing the content at those URLs. This leads to the so-called "PIPs" (partially indexed pages).
• Google's AdwordsBot only obeys instructions directed specifically at it. It does not obey "User-Agent: *". This is only an issue if you you use Adwords on the domain.

[mitash]

No, they wouldn't ignore it ... but they may not treat it as you would expect.

• Google may index URLs that are protected by robots.txt, without reading or indexing the content at those URLs. This leads to the so-called "PIPs" (partially indexed pages).
• Google's AdwordsBot only obeys instructions directed specifically at it. It does not obey "User-Agent: *". This is only an issue if you you use Adwords on the domain.

Perhaps I should have said that do spiders ignore some of the statements made in the robots.txt..?

You said Google may index URL's that are protected by robots.txt without reading or indexing the content at those URLs.

How can we show that the above happens..?

Thanks

[Jill]

Do a search for the particular URL in question that you have blocked via robots.txt. If Google has done that, you'll still see it in the results, but without any title or description, just the URL.

[mitash]

Excellent,
Thanks for the quick reply Jill.

I will do that.

Thanks for your help.

Cheers