Is this common today to have this much spider activity crawling web sites during a week period? (This is not searches, this is spider crawling pages as seen in the web site log files.)
1 Googlebot 227
2 AltaVista Robot 141
3 Hotbot-Lycos Robot 129
4 Wisenut.com Robot 54
5 www.baidu.com 34
6 Alexa Robot 26
7 Fast WebCrawler 26
8 Direct Hit/Ask/Askjeeves Robot 23
9 MSIE Crawler 11
10 www.ilab.sztaki.hu 10
11 NameProtect Inc. Crawler 9
12 LookSmart Directory Bot 6
13 whois.sc 5
14 www.merl.com 4
15 psbot 3
16 LinkWalker 2
17 Netcraft Web Server 2
18 dloader 2
19 Xenu Link Sleuth 1
20 seed.net.tw Search 1
21 Link Checker 1
22 libwww-perl 1
23 Vagabondo 1
24 galaxy.com 1
Webstream
Subscribe to HRA Now!
SEO Class in Chicago, IL
Learn How To Optimize Your Website on July 26, 2013
Looking for personalized in-depth SEO training among your peers?
High Rankings is offering a 1-day customized SEO training class in Chicago. Class size is limited so please sign-up now if you want in!
Are you a Google Analytics enthusiast?
Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE!
www.CustomReportSharing.com
From the folks who brought you High Rankings!
More SEO Content
International SEM | Social Media | Search Friendly Design | SEO | Paid Search / PPC | Seminars | Forum Threads | Q&A | Copywriting | Keyword Research | Web Analytics / Conversions | Blogging | Dynamic Sites | Linking | SEO Services | Site Architecture | Search Engine Spam | Wrap-ups | Business Issues | HRA Questions | Online Courses
Spider Activity
Started by
webstream
, Oct 20 2003 11:35 AM
3 replies to this topic
#2
Cygnus
-
- Active Members
-
- 167 posts
<Title>Insert Title Here</Title>
- Location:Cave Creek, AZ
Posted 20 October 2003 - 12:49 PM
Some get a lot more than that, some get a lot less. So long as you're being visited, you're cool [just hope those itsy bitsy little spiders aren't eating your bandwidth]. 
Cygnus
Cygnus
#3
Vertster
-
- Active Members
-
- 327 posts
Google wristbar installed
- Location:Salt Lake City, Utah
Posted 20 October 2003 - 02:04 PM
Thats not unusual at all. I know some sites that get more than that per day!
#4
webstream
-
- Active Members
-
- 195 posts
HR 4
Posted 20 October 2003 - 03:34 PM
Thanks! Some weeks the sites I work on see huge spider activity and sometimes it appears as above. But it definitely is not slowing down. BTW, bandwidth is not a concern on any of the sites I normally work on.
Webstream
Webstream
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users
