3 replies to this topic

[MADTIES]

Our site is structured in such a way that partners/affiliates have imaginary virtual folders i.e. url/partner/page.htm where our system will automatically identify the partner and set various session variables.

What I need to know is whether the robots.txt file can be used to stop spiders recording links in these partner folders as ultimately the page will redirect to just url/page.htm and could trigger some duplicate content warnings as the partner page would have the same content as our standard page.

Would Disallow: /partner/ in the robots file work in this situation or only just for real folders?

Thanks

MadTies

[Alan Perkins]

Would Disallow: /partner/ in the robots file work in this situation or only just for real folders?

It would work in this situation, assuming that when you say "url/partner/page.htm " you mean "domain/partner/page.htm", i.e. that "partner" is a top level directory.

[bobsledbob]

I think what he means is that "partner" would be replaced by the actual partner identifier, such that it's different for each affiliate.

Disallowing /url/ would work.

If you can't restrict that far up the directory structure, then you'll need to move your partner level down some. ie.

/url/partners/<partner>/page.htm

Then disallow /url/partners/ in your robots.txt

[domokun]

an alternative would be to have

url/external/partner/page

then disallow robots access to

url/external