98 replies to this topic

[madams]

Just uplaod the data to your gmail account

[Randy]

It depends upon how you're chunking up the referrer string 1dmf. At least for the strange 1 or 2 character search phrases. I've looked at probably 100 of these referrals and they all were pretty much the same, and all different than normal hits. An example would probably be helpful for you. Here's a real one of these referrals:

CODE

24.161.xxx.xxx - - [21/Nov/2009:12:55:33 -0500] "GET / HTTP/1.1" 200 21169 "http://www.google.com/search?sourceid=navclient&aq=0h&oq=&ie=UTF-8&rlz=1T4ADFA_enUS342US343&q=keyword+phrase" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"

Before I tweaked the code what the PHP GRE saved as the search phrase from the above hit was 0h. Meaning it was triggering on the &aq= variable instead of the &q= variable, namely because for this particular browser the &aq= came first in the string.

As soon as I got more specific in how it looks for the info (triggering on &q= instead of q=) the above would then be recorded correctly.

On a totally different subject I'm going to leave things running for a bit but my initial glance at the couple of blank referrals I have had since the code update are both hits from Google Image Search and not a normal search. So it's entirely possible the user may be loading only the framed page that comes as the default and this may be having an unintended effect.

I don't believe I've ever seen a Google Images search that contains any ranking data. So if that's the case I may simply add a little check like you have that looks for a null value in the search phrase and silently drop them when it happens. We're talking something less than 1% of all Google hits on my test site for those and besides that I don't consider Google Image traffic to be real traffic. I'm gonna let my test run a few more days though and review each of the blank phrase hits by hand to make sure they're all Google Image searches.

[1dmf]

Thanks for the sample Randy, I'll look when I get a chance and see if I suffer the same problem, but i think i'm ok as I use the standard CGI module for the query string, which is pretty robust.

Interesting about the image search, didn't even consider it! hmm, I thought it might be some glitch or an occasion when it's missing, do we want to exclude these, I understand your thought on them not being 'real' searches.

I look forward to your test results we need to be sure 100% these are image searches, before removing them from the capture. Is there anything on the QS that identifies image searches that we could parse?

on the DB note, i was also thinking a Max_Records attribute might be worth including for those sites which have humungus visitor stats.

Is 1,000,000 a good number? how many records could MS/my SQL handle before it grinds to a halt?

I have an added niggling issue I know the perl version suffers from, as I wrote my own SQL wrapper module, when records are retrieved from the DB they are placed in a memory resident data hash, do you know what sort of memory and processing requirements are needed to hold an array of hashes, with a million array indexes?

Anyways, I think the server would baulk with the CGI timeout issue if you tried to return a million rows of data back to the browser, wouldn't it?

[1dmf]

I've checked the perl version and luckily the 'q' search phrase data isn't an issue. - Though we still need to be sure about these image search records!

I've also fixed the small Ignore Zero Rank display problem which was still showing even if you were not collecting missing Ranking Position visits. However, I've now incorporated it so, if you at any point collect zero rank data and then switch it off, but you still have zero ranked data in the table, the option will still show. It will only not be available if you don't have any zero rank records. Making it very flexible for you to chop and change your mind with the $pos_req setting.

Also by popular demand I've added a new 'display as' type -> CSV , you can still use all the filters and sorting options, but instead of showing them as a tabular result or a flash graph, you can also download the results as a CSV text file.

This can be handy when used in conjunction with the tabular display option, so you can narrow down your results using tabular display, when you have the records you want to export, simply change the display type to CSV, click the query button again and you will be displayed a link to download the results as a CSV file.

On top of this new display feature I have added aged record purging and archiving options.

1. $age_delay - this option allows you to set how many days (based on visit date), records should be kept in the database, after records pass the age_delay setting they are deleted from the database.
use this option with caution. leaving it set to zero (default) will NOT delete or archive any records.

2. $arc_aged - this option when set to 1 (default) will archive all aged records into a CSV file and save it in your CGI-BIN folder before they are deleted from the database.
if you set it to zero (not recommended) , records will NOT be saved before deleted.

Before you alter the default settings, ensure you understand how they work!

Ensure perl has write permissions on the CGI-BIN folder (it should, but it's worth checking before you set an age delay).

All instructions and full information is supplied in the README.TXT file.

These features have been released under the Perl v1.6 beta version. you can download the new version and try out the new features via my signature link.

I'd like to thank Jill and others for their input and welcome all feedback and suggestions .

Edited by 1dmf, 24 November 2009 - 06:15 PM.

[1dmf]

I'm pleased to confirm Perl v1.6 has been implemented and tested on both Linux/mySQL & Windows/MSSQL without errors!

[piskie]

Question for you Randy
I have been using GRE for some time now and still don't know how I used to get by without it.
I moved Hosting a short while ago and since then, there has been a small Glitch with GRE.

The problem is with date selection.
I can sort by dates ascending or descending but cannot select between dates.
Setting from and to dates just returns a full list of all entries.

Other select and sort functions work fine as far as I can tell.
I have checked for "Consecutive id" entries and there are none missing.
The only significant difference that comes to mind on the server is I now have "Globals Off"

I have introduced a few extras into the Fields and columns, but I have reverted back to the bog standard version to make sure.

Any suggestions please Randy ??

[Jill]

Unfortunately, Randy's not been around here for awhile, so it's doubtful he'll see this and be able to help, sorry.

[piskie]

Thanks Jill.
That is a pity, he's a fantastic resource, hope he returns soon.

[1dmf]

piskie, if you ran a manual query via phpMyAdmin against the DB with a 'between' date WHERE clause, do you get the right results?