SEO Class in Chicago, IL
Are you a Google Analytics enthusiast?
More SEO Content
Posted 25 January 2004 - 12:22 AM
Like I said, I thought it was for Overture adveritised sites ony. However over the last few days (Wed and Thurs.) I've seen MMCrawler hit several non-Overture sites, and spider them pretty deeply too.
Has it changed what it used to be doing? Or am I remembering this spider incorrectly?
Posted 28 January 2004 - 02:08 AM
Posted 28 January 2004 - 05:30 AM
That kind of fits with the IP number it came in on too. Looking back at old historical info, the IP range was the same as I used to see from the old FAST-WebCrawler/3.x Multimedia spider. The trd dot overture dot com in the logs may be a red herring. The FAST multimedia crawler came through with that same info if memory serves, but was strictly looking for image and multimedia files and had nothing to do with Overture. Maybe it's just a User Agent change from the old FAST multimedia crawler? Dunno.
I'll keep an eye out for it to hit again, or snap up another site to see what files it grabs next time. Every domain of mine that it's found so far it's hit pretty hard for a single day. It doesn't take much to notice it in the logs when a spider hits your site for 500+ files and over 10 megs of transfer over just a couple of hours.
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users