12 replies to this topic

[Ultra]

Hi,

I'm pretty new to SEO & SEM and I just build a little and simple business site - gonna take it on a test/training trip regarding SEO & SEM for the next 6-12 months.

It's hosted simple and cheap with GoDaddy.

Unfortunately, my plan does not provide a (stats) logfile - GoDaddy provides simple stats via html files, which are okay, but of course I can't really 'work' with them, meaning I can't change the look or extract the data for creating my own charts and comparisons...

When I called them they told me if I upgrade to a different plan than there is software included that generates a logfile - which I then could use with (other) software to analyze & work with the data.

This is the first time I'm doing this, so here's my nOOb question:

Is there a software that I can (buy and) install myself to generate that logfile (to avoid monthly extra costs) ?

I know there are several online services that offer that service for cash, which means I could also consider the GoDaddy upgrade option - and I'm also looking into GoAn.

Just wanna check out ALL possible options...

Thanx for your help in advance !!!

Mike
Edit/Delete Message

[nethy]

GoAn is going to be the answer for most in your position. It provides a more complete picture then most log analysers preceding it.
* GoAn actually uses javascript to capture data rather the log files, this is the more common approach these days.
Probably a good idea to start with this tool.

IMO, it is not really worthwhile moving to a paid package unless you have specific needs or you are ready to upgrade to something like clicktracks for the benefit of the higher potential level of analysis.

If your intention is to learn, you may want to install 3-4 different (preferably free) tools - javascript ones & log analyser ones. They will have 95% of the same information but looking through different tools is good for looking at things different ways. Irecomend reading the following post:
http://www.kaushik.n...lternative.html
This is written for the 'corporate' internet marketer, but (apart from the eventual upgrade to $250K tools) it holds true (and gives suggestions for free tools).

[Jill]

Yep, what nethy said.

[Ultra]

Nethy & Jill,

thanx for your answers.

Have a few questions regarding your comment(s):

1. Why is it 'common' these days to use javascript rather than analyzing log files ?

a. Besides the comfort of not having to 'get' the logfile - which I don't mind at all - is there a real 'advantage' ?
b. Do log files contain more errors ?

--> Although I really don't like putting my business data on a 'presentation plate' as it is with GoAn, I will definitely give it a shot, I'm just really curious why people prefer it over the log file analysis, besides that GoAn is free...

2. So what are my options of generating a logfile on my own ? Is there a software based solution that I can install to generate those logfiles ?

--> I checked out AWstats (free) - and they offer (if u don't have a log file generated by your host) a perl/php script to be pasted into every page, which then generates a log file...

problem: too many log files generated...


Thanx for your help in advance !!!

Mike

[rolf]

AWstats is a great program to start with, but if you're serious about digging into what visitors are doing and what you can do to improve things then very quickly you can reach the limits of it's usefulness. (I don't know how their perl/php one compares to the installed one, so I'm specifically talking about the installed one here)

AWstats will tell you if/when you're spidered by the engines and which phrases are bringing in traffic, and it will tell you with acceptable accuracy how many people are coming from where and how many people are seeing which pages, plus a load of other useful stuff. If that's all you want to know then it's fine.

The problem is that you may soon be asking questions like - how many pages are people from country X looking at on average? What do people who land on page X do next? How many people finding us on X keyword from X search engine? These, and many other questions, are all things you might want to know, but AWstyats is not detailed enough to tell you.

I can't comment on other stats programs apart from GoAn, but GoAn will answer all of these questions and plenty more. I've been using it for a while now and I don't feel I'm anywhere near stretching what it can do. If I had to criticise, I'd say that there are so many features that sometimes it is a little intimidating and it sometimes takes a bit of researching/reading up to find out how to get the exact data you want, however, I'll happily accept this as a trade off for the level of detail it gives.

Hope that helps

[piskie]

Have a look at WebLog Expert Pro as a cost effectivefirst program for log analysis. There is a Free Lite version to wet your apetite and see if it works rhe way you want before buying.

[Ultra]

rolf & piskie,

will definitely take a deeper look at GoAn - Weblog Expert unfortunately can only be installed on Windows servers...

My server runs on Linux... ;(

Are there any other programs that I can install on my linux server that will produce a detailed logfile ?

Thanx again for your help !!!

Mike

[Jill]

1. Why is it 'common' these days to use javascript rather than analyzing log files ?

It's just easier.

If you have a website of any size (or more than one) log files and processing them via web analytics tools simply becomes unwieldly. You need to have a dedicated server just for that purpose many times. It's expensive and takes a long time too.

Using the javacript method is just simpler. Not more accurate though. I have heard that log file analysis is probably more accurate.

[Ultra]

It's just easier.

If you have a website of any size (or more than one) log files and processing them via web analytics tools simply becomes unwieldly. You need to have a dedicated server just for that purpose many times. It's expensive and takes a long time too.

Using the javacript method is just simpler. Not more accurate though. I have heard that log file analysis is probably more accurate.

Jill,

thanx for the info.

Got three more questions:

1. Can you please explain specific reasons why it becomes unwieldy ?

2. Does the javascript method generate a 'javascript log file' or is the info sent to the (i.e.) GoAn server automatically every time ?

3. Do I need to consider certain things if I decide to place certain scripts in my sites to generate log files ? For example the size of the script or anything else ?

Thanx again for your help.

Mike

[Randy]

The three questions first.

1. Because if you start having to deal with multiple log files, or God forbid log files located on different servers, you'll quickly run into license restrictions and/or permission problems. Servers need to maintain a certain level of security. If you do not run your own servers so can selectively open up holes to allow data to flow back and forth you're often screwed. Hosts do not like opening up holes on shared hosting platforms in the security layers, for obvious reasons. Javascript stats solutions overcome these things quite easily and don't have much of a downside.

2. It's sent to the GoAn server directly, via a small javascript call you embed in the html code of your pages.

3. The size of the script probably isn't going to be a concern, but it shouldn't be huge. Too big and it's going to cause loading speed of the pages to slow down. However this isn't usually any concern since all it's going to be doing is writing some stuff to a file.

A bigger concern you should prepare for is what you're going to do with old log files. I'm assuming here you're not just going to let some script keep appending to the end of its log file forever, but will make it rotate its logs. If it doesn't rotate your log files you could run out of disk space pretty easily. Really quickly if your site gets much traffic. And then if they rotate you need to figure out how long you're going to keep them, what happens to them, is the data stripped out and saved in a database somewhere, etc, etc.

On my servers the built in stats packages save the data to a separate location nightly. I set up domains so that the raw log files are available for three days --even then the previous days are automatically GZipped to save space-- then they get zapped. You're probably asking why.

Well, on my little personal site that only gets around 1500 or so visits per day the log file each and every day will usually run right around 25 megs in its raw form. GZipping will take that down to 2 megs or so.

And on a busier site of mine I just looked at the log (round about 20,000 uniques per day with good stickiness) it's not at all unusual to see a single days raw log file grow up to right around 2 Gigs. Per day. And that's just the access_ log. I'm skipping the SSL log (another 100 megs per day) and relatively small error_logs.

If you didn't bother to GZip, rotate and delete the log files for a site anything like that one you'd end up filling up an 80 gig hard drive in about 4-6 weeks. So yeah, it's a consideration you need to have in the mix.

Okay, last add...

Depending upon what kind of budget you have for your stats solution there are several good ones out there. Most of the Installed versions though are going to require a log file. (FWIW, you host won't tell you this, but if you're on a typical *nix system as you indicated and happen to have access to the apache configuration file for your domain you maybe able to instruct the server to create an access log file for you.)

There is an installed version of Google Analytics. Actually Google Analytics was built off of Urchin, which is an installed version. Right now the GoAn version of Urchin is in Beta testing --it had some issues last I checked-- but if you had it and the ability to process log files it would certainly work. Price wise, I believe they said the installed version license was going to run around $2500. Which is pretty much in line with all of the rest of the best of the best.

[Jill]

1. Can you please explain specific reasons why it becomes unwieldy ?

Try it for yourself and you'll see pretty quickly.

2. Does the javascript method generate a 'javascript log file' or is the info sent to the (i.e.) GoAn server automatically every time ?

No idea, but presumably, Randy has answered that.

3. Do I need to consider certain things if I decide to place certain scripts in my sites to generate log files ? For example the size of the script or anything else ?

No, they're just a few lines of code. Shouldn't be a problem for any page.

[Ultra]

Randy,

thank you for the very detailed information - very, very helpful !

Got two more questions:

A. When a server generates a logfile does it 'record' all possible data ?

--> For example: Rolf wrote in this thread that AWstats has limits when it comes to certain comparisons/calculations. This is a 'limitation' of the software - can/does it happen that your software is up to the task but your logfile is missing certain info (maybe because the code that generates the logfile was not sufficiently) ?

B. Where would the apache configuration file be located and how would I need to instruct it to create a logfile ?

Thanx again for your help !!!

Mike

Edited by Ultra, 28 January 2008 - 03:17 AM.

[Randy]

A. It could be either or Mike. It depends upon the exact configuration being used, but as mentioned previously you can change what info apache stores in its logs and in what order.

With a typical Unix/Linux setup all of the info you'd need would be automatically included in the log files, so there's no need to tweak anything. However this doesn't mean you can't if you needed to, if you have enough permissions on the server.

In the case of AWS, I believe what Rolf is referring to is an issue with the AWS software itself, not the info that's being saved to the log files.

B. It depends upon what software is installed on the server.

On a typical *nix system the base (server wide) Apache configuration file is going to be at /etc/httpd/conf/httpd.conf. For per domain stuff it's going to depend entirely upon the control panel software that's installed. For instance on a Plesk system the virtual host config files are going to be at /var/www/vhosts/yourdomain.com/conf/vhost.conf (for non-ssl stuff) and vhost_ssl.conf for ssl connections. It's been awhile since I worked on a cPanel/WHM server, however they used to be at something like /usr/local/apache/conf/userdata.

With all of those however you may well run into permission restrictions that are in place for security reasons. If you're on a shared hosting platform I rather doubt you'd be able to edit the base Apache configuration file. I know individual users cannot edit the vhost.conf files on Plesk either, and suspect the same is true of cPanel any any others. There are some significant security implications, so unless you run your own server you're going to need to bring your hosting company into the picture in how Apache works.