3 replies to this topic

[1dmf]

Hi,

I have written a script for my affiliates system, generally speaking I want to disallow the indexing of the cgi-bin, but allow indexing of one particular script, as it does a 301 reditrect to the correct page and so want link-juice to be passed on.

However,

if I have in my robots.txt file

CODE

allow: /cgi-bin/myscript.pl
disallow: /cgi-bin/

but peoples websites will have links that include a query string, does just allowing indexing / accesses to the script on it's own also allow the SE's to follow the link.

or am i missunderstanding the robots.txt file, is following links and evaluating link juice nothing to do with the robots.txt and is that purely indexing.

do I even need to allow the base perl script in the robots.txt for the link juice 301 redirect mechanism to work?

Advice understanding this is appreciated.

[Randy]

Not sure I understand the question.

The engeins are going to see the links to your perl script as going to different pages anyway since the query string value will make each a little bit different. Are you 301'ing those to another pge after saving the affiliate data?

If so, expressly allowing access to a script in your robots.txt should cover all query strings. robots.txt has a unexpressed wildcard at the end of each line, so if the engines are actually following the protocol they should see those affiliate queries, followed by the subsequent 301 redirect.

[1dmf]

unexpressed wildcard at the end of each line

You understood perfectly Randy

That just what I wanted to hear, they apply the wildcard to myscript.pl so myscript.pl?Affiliate=10 & myscript.pl?Affiliate=20 , are all allowed due to the allow for the basic myscript.pl line in the robots file.

Many thanks Randy.

[Randy]

That's my understanding of the robots.txt protocol anyway 1dmf. I believe the engines follow this basic protocol, but have never tested it personally.

I'm sure Alan or someone who has tested it will be along to tell is it's wrong if the wildcard element is not there in real world practice, but I think you're safe.