6 replies to this topic

[GeordieSEO]

Hi guys,

Not sure if I've put this in the correct place...

I've been evaluating Robot Manager as I'm interested in some software to easily track spider / crawler visits to the websites we host. However, it seems quite buggy and it seems to crash often when FTPing to our virtual host. Also it doesn't seem to get all the spider / crawler visits. I downloaded logs from one site and looking at the raw log a googlebot had visited the site but Robot Manager didn't report it.

Can anybody recommend an alternative or easy ways to track spider visits to our sites?

Thanks

:learn:

[Jill]

Hi GeordieSEO, I've moved this to the Statistics and ROI tracking area. I think if you look at some of the previous posts in this category, you'll find lots of programs to experiment with.

Good luck!

Jill

[Vertster]

Hi GordieSEO,

I started using one called Robots.txt It is really good, and you can get a free trial at their site. All it reports is robot visits, and tells your every page, each robot visited. You can filter it by date, robot, or URL, which is pretty handy.

[qwerty]

Can you post a link to that one? I did a search on robots.txt software, but all I got was software for generating a robots.txt file.

[Vertster]

It actually is an editor, but has a built in spider log analyzer as well

http://www.robots-txt.net/

[GeordieSEO]

Jill thanks for moving the post - I will search the section further

Vertster - Thanks for the link.

PS. I must find a suitable image for myself - I have a tendency to break camera lenses when photos are taken :tooth:

[Matt B]

I've been using NetTracker pretty frequently these days in watching spider activity for both new and existing sites. NetTracker is pretty detailed and allows you to drill down to what pages were requested; by session, by spider, etc. So far, I've been very impressed with the level of detail.