9 replies to this topic

[jbelle]

This is a little random but I updated a site a few weeks ago and optimized the text. Naturally, I'm anxious to see the results, so I irrationally check my sites stats daily (sometimes more frequently than that!) to see what spiders have been crawling. There are faithfully a few hits an hour from all the major bots, but nothing more.

I've never watched these spiders so carefully; will I see a big spike when they've finally decided to do a "deep crawl"?

[robwatts]

Hi jbelle

It all depends on the size of the domain really.

if I were you, I wouldnt look at your log files every day (it'll drive you potty)

Get yourself a good log analysis tool { there are loads of free ones such as webaliser, analog and a few more sophisticated ones clicktracks webtrends there are loads more too, but they escape me at the minute} and then have a little look once a week or so and see what they say.

They should all be able to show you the freq of the bot visit and the pages they grabbed too.

I'd use the time you'd otherwise spend pouring over the logs to perhaps, build links or add some more content - far more productive IMO.

[Matt B]

I agree with Rob, I wouldn't worry too much about spiders.

If you have your initial directory links in order, and good, spiderable content, then the spiders will naturally come around. Especially after making changes - give it some time.

[jbelle]

Of course there are better things to do with my time, but I'm too busy checking my stats daily still to leave time to evaluate more efficient use of my time!

I noted some jumps out of several spiders upon deleting some old pages, so I'm still curious to see what happens with spiders in their natural habitat.

I think my work can handle the 2 minutes it takes to check my stats once a day to feed my addiction--but don't worry, I'll deny it if asked!

[robwatts]

Of course there are better things to do with my time, but I'm too busy checking my stats daily still to leave time to evaluate more efficient use of my time! 

ah! you cant beat a bit of convoluted fuzzy logic!

[Scottie]

I think my work can handle the 2 minutes it takes to check my stats once a day to feed my addiction--but don't worry, I'll deny it if asked!

Denial is a sure sign of addiction...

Try not to look at all. "Just a quick look" always turns into a more in-depth time-waster than it should be. Like Ebay or forums. One thing leads to another and suddenly the day is gone. :stunned: The only way to manage is to go cold turkey. (good advice from one addict to another- do as I say... not as I do )

Does anyone have a favorite free stat analysis program? I'm not thrilled with Webalyzer (offered by my host) and haven't really found a freebie that covered all the bases yet. I haven't researched it much because I try to stay away from my stats... I can spend hours going through them! But every now and then, I do want to check things.

[jbelle]

Try not to look at all.  "Just a quick look" always turns into a more in-depth time-waster than it should be.  Like Ebay or forums.

LOL; these forums are getting the best of me today!

Well, this was supposed to be a productive post: I just went to check on the free stats package the web firm I used to work with used, and, it's not free anymore! Oh well; just another time waster

[K.S. Katz]

Here at Webcast 1, Inc., we use a Perl script modified from AWStats. It works really good for us and is completely free (you just have to do the setup).

To check it out, go to: http://awstats.sourceforge.net/

They do a wonderful graph where they compare AWStats to Webalizer and a few other stats programs. I've seen a few, and by far (in my not so humble opinion) I believe this is the best free stats program out there.

Sincerely,

K.S. Katz

[Webmaster-Toolkit.com]

Yeah I'm an avid awstats user too

[Scottie]

Thanks!

I have used AWStats before and like it- I didn't realize it was open source.