14 replies to this topic

[Dwain Dibley]

Hi

I work for a large ecommerce web site and we do get many thousands of customers hitting the site everyday.

Obviously all this is very good BUT it causes us to have huge log files and I dont know if this is the reason WebTrends Log Analyser is struggling with them.

Does this sound feasible that the logs are too big for the software?

In IIS what are the best settings to have ticked to make sure best stats are being recorded and also that we are not logging useless data also.

Average file is about 800,000KB

Apologies if posted in wrong area but I know I'll get some answers from you wonderful people!!

[websage]

Here are few things I would do:

In IIS go through the list of parameters you want logged and consider removing those you do not care about. I am not sure what you are trying to analyze so it is difficult to suggest what to store and what not.

Also, are the log files daily or weekly or monthly? You might want to consider having them saved more often so the file size is not overwhelming.

Also, if it is an ecommerce site, I take it you store login information. Then you might want to consider building your own application to store the most important information into a database and analyse it directly. I would highly recommend a book Practical Web Traffic Analysis which can teach you how to build such a system with tools as simple as MS Access.

Last but not least consider a different web analytics tool like ClickTracks which can work with large log files without problems.

Hope this helps!

Edited by Jill, 07 January 2004 - 12:49 PM.

[Jill]

ClickTracks also has a problem with huge logfiles. At least the version I have used does (might be an older one).

Since starting the forum, my logfiles are also out of hand.

Jill

[SearchRank]

You will probably want to go to a daily format on the log files so that each one is not so large.

[websage]

I had somewhere a document ClickTracks sent me on comparison logfile processing with WebTrends but I can't find it.

At any rate, here is from the ClickTracks FAQ:

Q: My site traffic is huge. Can ClickTracks scale?
A: Yes. ClickTracks handles even very large sites with millions of hits per day. There is no limit to the size of log file you can read. A 1Gb file takes about 2 minutes to fully process on a 1Ghz PIII. 100Gb of log data can realistically be processed.

[Grumpus]

If it's merely (disk)size you are worrying about and you have access to the IIS admin controls, you could set it up so that those older files are zipped up daily/weekly/whatever until you are ready/need to do something with them. Since they are text files, they'll compress up nicely (about 80% I think). Many log analysis programs will even access archived logs directly.

G.

[lidog]

You may want to baseline against the expected filesizes at [http://www.chadsteel...LFET/LFET.aspx]

Edited by Jill, 29 January 2004 - 12:15 AM.

[bkernst]

The exact same problem happened to me with some clients on a monthly basis.
Out of some reason it is always the same two clients with big log files, but other clients with much bigger log files don't give me that problem.
Currently the log files are weekly, and they compress very well.
Some of those log files where 130MB for the monh, and WebTrends stopped processing just after day one of a log file, went on to the next file, and also stopped just after the first day, sometimes it stopped after the second day.

What I did, was looking for alternative log file analyzers, and I can recommend WebLog Expert (full or lite). The full version can even use command-line parameters, which means that you could use batch file with task scheduler to automatically generate your reports. WebTrends is far too unstable to be used like that on a server IMHO.

You could however, change your log file format (I suspect, that WebTrends has got issues with that), and you can also set IIS/Apache/Whatever to make more, but smaller log files, and hope that WebTrends can handle those.

Bernhard

[Maria]

Interestingly my el-Cheapo log analysis program handles my >1G log files just fine. ;-)

It's WebLog Expert -- simple but cheap. I used the 30 day trial and paid the money (ok not after 30 days, but I did in the end!).

Maria

[medkraft]

Obviously all this is very good BUT it causes us to have huge log files and I dont know if this is the reason WebTrends Log Analyser is struggling with them.

The problem likely has to do with your RAM. The larger the log files the more RAM WT needs to run the reports quickly. For me, moving from a machine with a P3 800 and 192 MB RAM to one with a P4 2.2 and 1GB a RAM, made all the difference. The new system now spits out reports (analyzing log files as large as 250MB to 500 MB) in minutes rather than hours.

Cheers

[Scottie]

Welcome, Medkraft!

[bkernst]

That might explain some problems I have been having, but in many cases WebTrends still analyses other log fiels which are bigger without any problems.

Bernhard

[medkraft]

Hi Bernhard,

Feel free to send me one of the large (zipped) files you're having trouble with and I'll run a test for you. It's probably best to place it on your server and send me the link in a private message.

Cheers... Tom

[zestor]

I don't know webtrends that well. But it might have a problem if it's doing any type of reverse DNS lookup on IP addresses with so much information.

[bkernst]

The fact that WebTrends doesn't want to analyse one or two out of about fourty websites, isn't a major problem, and it doesn't happen every month. Occasionally there is no problem at all. I just need to automate the entire system a bit more, but there is a tight budget involved (the budget isn't my decision).
For those files that do give problems, I have an adequate replacement.
I am still suspecting that something with the columns involved doesn't go too well with WebTrends.

Bernhard