2 replies to this topic

[djingo]

Does

Disallow: /*xyz

- in a Robots.txt mean that all urls containing xyz will be ignored?

In www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449 it says

To block access to all URLs that include a question mark (?) (more specifically, any URL that begins with your domain name, followed by any string, followed by a question mark, followed by any string):
User-agent: Googlebot
Disallow: /*?

I am a little in doubt if this only goes for "?" or any string following the star.

[Randy]

Disallow: /*xyz

- in a Robots.txt mean that all urls containing xyz will be ignored?

In Google's implementation, yes. All urls containing the text string "xyz" would be excluded from spidering. They allow use of a * as a wildcard character.

I'm putting this into a Google perspective because technically I don't believe the robots.txt specs had any such wildcard character. Most if not all search engine spiders support it though.

Moving this one to the robots.txt area of the forums.

[djingo]

Ok, thanks!