6 replies to this topic

[ladybird]

Hi There

I've been reading on this forum that robots.txt files are a good way to stop SEs from spidering certain files on a web site.

If I want to stop ALL spiders from crawling say XYZ.htm and ABC.htm do I set up the code like this:

User-Agent: *
Disallow: XYZ.htm

User-Agent: *
Disallow: ABC.htm

OR like this:

User-Agent: *
Disallow: XYZ.htm
Disallow: ABC.htm

Thanks in advance

[chrishirst]

the format should be

User-Agent: *
Disallow: /XYZ.htm
Disallow: /ABC.htm


the leading "/" is important but any of them are correct provided the "/" is added.

[Tom Philo]

robots.txt lives in a DOS oriented world - the / tells it to start at the root of the web server and start matching the pattern from the root level of the site using what comes after the / to be the pattern to be excluded.

/images/*

would exclude all items in images directory being indexed

/_vti*

would exclude all files (and directories) that start with _vti from being indexed anywhere in the site

/file.htm would only exclude that single file.htm from being indexed

[Alan Perkins]

/images/*

would exclude all items in images directory being indexed

The * is Google specific and not part of the robots.txt standard.

In fact, just

Disallow: /images/

would exclude all items in the /images/ directory. Similarly,

Disallow: /_vti*

is non-standard, and

Disallow: /_vti

would exclude all files (and directories) that start with _vti from being indexed, not anywhere in the site but at the root level of the site. So /blah/_vti_xyz would still be indexed.

/file.htm would only exclude that single file.htm from being indexed

Technically, it would prevent any file or directory whose URL began "/file.htm" from being indexed. So, for example, this would also prevent "/file.html" being indexed.

[Tom Philo]

got caught in my DOS world by adding in the * . . .

[Alan Perkins]

[ladybird]

Thanks guys- you're the best!