4 replies to this topic

[Marfola]

I would like to exclude all pages ending in /print.html and all pages with a ? in the url string in my robots.txt file. Is the following syntax correct for Yahoo, Google and MSN?

Disallow: /*print.html$
Disallow: /*?

[Randy]

Though *'d wildcards aren't part of the robots.txt specification, yes those examples will work for the search engines you mentioned. They do support such wildcards.

[Marfola]

Though *'d wildcards aren't part of the robots.txt specification, yes those examples will work for the search engines you mentioned. They do support such wildcards.

Thanks.

Do I need to have two seperate sets of directions in my robots.txt, one for google, yahoo and msn and a second for all others?

[Randy]

Depends upon what you want.

If you want those instructions to apply to only certain spiders, then you need to set that in the User-agent field. If not, you can use a User-agent: * to cover them all.

Those spiders who don't understand/use wildcards will simply ignore those lines.

[Marfola]

Thanks Randy. That is exactly what I wanted to know. ‘Those spiders who don't understand/use wildcards will simply ignore those lines.’