11 replies to this topic

[doughayman]

I was reviewing my weblogs tonight, and see an anomoly which I cannot explain, and want to know if anyone has a reasonable explanation for this.
Here's the scenario:

On some of my webpages, I have links, that when clicked, execute a local CGI program (let's call it CLICK.EXE). This CGI program provides a redirect (I force it to a 302 in my code), and displays a new browser window with the targeted page. The page that is displayed is totally controlled by my CGI logic - it takes an integer parameter which indexes into a local database table, and based on the look-up into this table, determines what pages should be displayed in the new browser window. My CGI logic also increases a counter for the effected lookup record, so I can keep track of how many clicks for a particular link occurred.

What I have noticed of late, is that a lot of these "clicks" are now displaying as webserver status 200 (instead of "302"), and my counters are not incrementing accordingly in my database. The logic in my CGI program is simple and clean. It always returns a "302", and always writes to the database.

I thought that perhaps pop-up blockers were potentially causing this phenomenon, but I have tested that scenario locally, and it is not the case.

Does anyone have a reasonable explanation for this behavior ?

Thanks in advance !

[Randy]

A question since I'm not sure.

After passing through your script does the url address remain your CGI script location? Or are the clicks being sent off to other static or dynamic pages?

The reason I ask is that I'm wondering if those final destination pages haven't been indexed somehow or another so that people (and/or spiders?) are hitting them directly instead of passing through your CGI script.

What do you see in your raw log files? They should show you the status code the server or your script is setting also. And it may help to see if there are hits to the final destination that don't pass through the script, thus aren't generating a 302.

[doughayman]

Hi Randy,

Thanks for your reply. The entire sequence of typical events that yield the "200" status are as follows (and the sequence of events are the same when I get the expected "302"):

1) User (or perhaps spider) comes into my site via a search engine referral.

2) A subordinate page in my site is visited. On this page, the user (or spider) clicks on a link that activates my CGI script.

3) My CGI script does the following:

a) It takes the parameter Id from my call (a sample call is /cgi-win/click.exe?1) and indexes into a local database of mine.
b] A URL (in this case, an affiliate link) is retrieved from my database based on this parameter Id.
c) I update a counter in my database for the record in my database corresponding to this parameter Id.
d) I force a "302", create a new browser window, and populate this new browser window with the URL retrieved in step B] above.


Hence, to answer your question above, the user now has a secondary window displayed that contains this new URL, and the original window that contains the executed CGI script is still on the screen as well. I thought the same thing, that perhaps these URL's that I'm displaying in a secondary window have
somehow been indexed in search engines (that is, the URL's with my hard-coded affiliate ID's embedded within them).

However, when I get the "200" response (as opposed to the desired "302" response), from my web log files it appears as if they are entering
my website "through the front door", as expected. The CGI script is executed (according to my web log), but the resultant webserver status code yields a "200" response, and my database counter (step 2c above) is not getting incremented. It's as if the CGI script is not getting executed properly. No where in my CGI script do I return a "200"; it's a simple simon script that returns a "302" and does the aforementioned redirect.

I am baffled ??????

[Randy]

I am too!

Another question.

Do you have some logic in the script that'll handle errors? For instance, if a parameter isn't recognized for some reason, what happens? Does the script still attempt to fire a 302 and popup window? Does it stop and deliver a 404 or some other status code dynamically? Or does the script try to execute, but can't?

Here's what I'm getting to with the questions that'll hopefully lead you somewhere even if the situation I describe is completely wrong. When you're setting a status code via a script you have to be a little bit careful to make sure it's the very first information getting sent back to the browser. So if something is happening that's trying to send something back to the browser for any reason --and some sort of unusual situation or unexpected input would be my first suspect-- the server is naturally going to deliver a 200 OK (because the click.exe file does exist) followed by your scripted 302, which doesn't actually register. The server will never see the 302 in this case because you only get one status code per page load.

Make sense?

[doughayman]

Randy,

You are the man !! You definitely put me on the right track. Lo and behold, it just tested my /cgi-win/click.exe?parameterId with a Parameter Id that doesn't exist in my lookup table, and sure enough, the browser yields a "No current record" error from my webserver that is displayed visually, but in my log file, it is registering a "200", since like you said, it found "click.exe". Hence, some how my script must have a bug in it.

Now, what is really strange is that I checked the web page that contains all of my redirect links (via the CGI script execution), where I'm getting some of these '200" statuses, and all of my Parameter Id's are valid......hmmmmm.

Additionally, when I eliminate passing the Parameter Id (i.e., /cgi-win/click.exe versus /cgi-win/click.exe?ParameterId), this too, visually display an error ("Missing Operator"), but also yields a "200".

Can you think of any browser-related issues that might somehow suppress the passing of a Parameter Id in the calling string ?

Again, thank you for your efforts here. You have definitely put me on the right track.

Regards,

Doug

[Randy]

Can you think of any browser-related issues that might somehow suppress the passing of a Parameter Id in the calling string ?

No. This would be a pretty silly thing for browsers to do since so many URLs these days carry some kind of parameter string, and they can all be constructed differently depending upon the backend server support.

What you've discovered should be a pretty easy one to fix. Simply add an Error routine to your scripted redirect so that it has a fallover 404 Error status if something doesn't process correctly. eg if no Parem ID is detected, they get a 404 status. Your page can continue to say the same thing as it now does, just make sure it sends a 404 Not Found status as the first thing to the browser. The same goes for Parem ID's that aren't recognized.

The 404 will keep those click.exe urls from getting indexed when they shouldn't be.

[doughayman]

No. This would be a pretty silly thing for browsers to do since so many URLs these days carry some kind of parameter string, and they can all be constructed differently depending upon the backend server support.

What you've discovered should be a pretty easy one to fix. Simply add an Error routine to your scripted redirect so that it has a fallover 404 Error status if something doesn't process correctly. eg if no Parem ID is detected, they get a 404 status. Your page can continue to say the same thing as it now does, just make sure it sends a 404 Not Found status as the first thing to the browser. The same goes for Parem ID's that aren't recognized.

The 404 will keep those click.exe urls from getting indexed when they shouldn't be.

Randy,

Thanks again, and agreed. It's a very easy fix to force a 404. I currently have a webserver debugger feature turned on, which will record, among other things, the QUERY string Parameter Id, so at least I can see if it is being passed in as NULL or some bogus values. However, even with this fix, it doesn't really solve my true problem - for some reason my affiliate link redirects are not executing properly. For selfish monetary reasons, this is not a good thing. Having a more gracious error handler, provides a better webserver status code, and can provide a friendlier browser error message, but still doesn't solve my core problem as to why this happening. After all, I am hard-coding these Parameters in the URL string from my base page. In theory, if users are coming in from the front-door, they should only be passing in valid parameter values. And from weblog review, most of these "200" returns are coming from legitimate "front-door" access to my site. This is what puzzles me. I'll let you know if my debugger turns up anything useful.

Forcing a 404 should eliminate any indexing of these URL's with a NULL or bogus parameter value. This should get rid of the "200's" that I'm seeing with any spiders, but still doesn't answer the question of why the "front door" users are seeing this behavior.

Thanks again for all of your help,

Doug

Edited by doughayman, 11 May 2008 - 09:48 AM.

[doughayman]

Randy,

I just wanted to update you on this, since I know you are on pins and needles about it

Thank you very, very much for pointing me in the right direction here. I enabled a webserver debugging flag,
which allowed me to capture all CGI input that was being fed into my webserver. Through this, I was able to capture
some CGI input that was causing the "200" status error.

As you suggested above, the problem lied in inappropriate error handling. As I mentioned to you, I capture clicks,
but also capture IP address, and have a subroutine that looks up the enumerated IP address and converts it to a 3
character country code, which I also store in my DB. As it turns out, my error handling for this country code lookup
was extremely poor - there were some enumerated IP's that didn't map to anything in my lookup table, and this caused
a "no current record" VB error, which in turn invoked standard CGI error handling, which displayed this message in
the browser. Since click.exe was found, this resulted in a "200" status being returned from my webserver. CASE
SOLVED.

Thank you for extremely valuable input here......you are a great resource. I hope I can return the favor some day.

Best Regards,

Doug

[Randy]

Woo! Well sleuthed Doug.

This sort of convoluted situaiton is why coders should always put as much thought into their error handling as they do anything else. Every app really does need a fairly robust error handling process. This is something I admit to glossing over far more often than I should.

I think we all simply get caught up in the fascination of a working app and don't put enough thought into the myriad ways it might break for some users. You'd think by now I'd know better.

[doughayman]

I agree with you, no doubt. I am neglectful in that area as well. In the rapidly evolving web arena, I've learned
several things that are invaluable as well:

1) Error Handling (newly renewed from this case). Due to this neglect on my part, I estimate that I lost about 20 %
of my clickthroughs over the last 6 months or so.

2) Constant review of web logs. It's amazing the kind of things that you can discover from them (this problem, case
in point). Reviewing of web logs comes in several flavors: examination of raw log files, and use of various web
analytic tools that can review user usage, prevailing trends, etc.

3) The most important thing is to keep track of changes made to websites (i.e., an audit trail). This can be a primitive
text file outlining the dates and nature of changes made to web pages, or can be the use of a configuration mgmt
tool (e.g., SourceSafe) to help automate the delta changes.

I have found that the use of an audit history of changes coupled with examination of raw log files to be an invaluable
tool, when it comes to "ranking tanking" investigations. It has saved my butt on more than one occassion.

Again, thanks Randy !!!

Regards,

Doug

[Jill]

Nice job both of you!

[1dmf]

Well done to the 'HR Server Wizard'