4 replies to this topic

[JakeG]

Is there a neat way to exclude all pages other than the homepage?

[Alan Perkins]

It ain't easy! BUT, it's easier than allowing access to everything except the homepage (which is impossible using robots.txt).

With Google, you can try using a wildcard:

CODE

User-agent: Googlebot
Disallow: /*

Otherwise, you need to explicitly deny access to almost everything:

CODE

User-agent: *
Disallow: /a
Disallow: /b
Disallow: /c

That will disallow anything beginning with an a, b or c. You would need to add the rest (d to z, numbers, etc.).

WARNING: Please treat both the above with extreme care. Though theoretically accurate, you will need to test. The safest approach is to use the meta robots tag.

[JakeG]

Thanks, I'll experiment!

Maybe the back-end is set up in a way that will let me generate one meta tag for the homepage and a different one for all other pages quite easilly... will have a look.

Is the syntax of robots.txt case sensitive? Would I have to disallow /a and A/ ?

[Alan Perkins]

Ah. you're actually wanting to do this for real. In that case I'd definitely go for the meta tag.

robots.txt is case sensitive if your server is (e.g. Apache) and not if your server isn't (e.g. IIS). That's not part of the standard (it's case-sensitive there), but it's my experience of all search engines. Otherwise, it be unworkable on an IIS server.

[JakeG]

Yep I'm doing it for real. It is a duplicate of an existing site where only the homepage is different, it has a different brand name and brings in lots of traffic from it so I don't want to lose the ranking.

Thanks for the info I'll go down the META tag route.