Last I heard, I thought the Yahoo spider that comes through as MMCrawler was only to crawl Overture listed sites. Specifically, this one came through Yahoo-MMCrawler/3.x.
Like I said, I thought it was for Overture adveritised sites ony. However over the last few days (Wed and Thurs.) I've seen MMCrawler hit several non-Overture sites, and spider them pretty deeply too.
Has it changed what it used to be doing? Or am I remembering this spider incorrectly?
Subscribe to HRA Now!
SEO Class in Chicago, IL
Learn How To Optimize Your Website on July 26, 2013
Looking for personalized in-depth SEO training among your peers?
High Rankings is offering a 1-day customized SEO training class in Chicago. Class size is limited so please sign-up now if you want in!
Are you a Google Analytics enthusiast?
Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE!
www.CustomReportSharing.com
From the folks who brought you High Rankings!
More SEO Content
International SEM | Social Media | Search Friendly Design | SEO | Paid Search / PPC | Seminars | Forum Threads | Q&A | Copywriting | Keyword Research | Web Analytics / Conversions | Blogging | Dynamic Sites | Linking | SEO Services | Site Architecture | Search Engine Spam | Wrap-ups | Business Issues | HRA Questions | Online Courses
Yahoo Mmcrawler
Started by
Randy
, Jan 25 2004 12:22 AM
2 replies to this topic
#2
Phoenix
-
- Active Members
-
- 24 posts
HR 2
- Location:Lone Star State!
Posted 28 January 2004 - 02:08 AM
Randy, I have seen this spider too on several sites, that are not listed in overture in anyway, and yes it has gone deep on one site. I can't seem to find much info about the bot however. Anyone know more about it? I think I remember seeing that it was a image bot, but can't confirm that.
#3
Randy
-
- Moderator
-
- 17,540 posts
Convert Me!
Posted 28 January 2004 - 05:30 AM
Being an image (or actually Multimedia) crawler would make sense from what I'm seeing Phoenix. I did some more watching and checking, and on my sites it appears to be pulling a ton of non-text files. jpg's, gif's, png's, pdf's...even avi's and mpg's!
That kind of fits with the IP number it came in on too. Looking back at old historical info, the IP range was the same as I used to see from the old FAST-WebCrawler/3.x Multimedia spider. The trd dot overture dot com in the logs may be a red herring. The FAST multimedia crawler came through with that same info if memory serves, but was strictly looking for image and multimedia files and had nothing to do with Overture. Maybe it's just a User Agent change from the old FAST multimedia crawler? Dunno.
I'll keep an eye out for it to hit again, or snap up another site to see what files it grabs next time. Every domain of mine that it's found so far it's hit pretty hard for a single day. It doesn't take much to notice it in the logs when a spider hits your site for 500+ files and over 10 megs of transfer over just a couple of hours.
That kind of fits with the IP number it came in on too. Looking back at old historical info, the IP range was the same as I used to see from the old FAST-WebCrawler/3.x Multimedia spider. The trd dot overture dot com in the logs may be a red herring. The FAST multimedia crawler came through with that same info if memory serves, but was strictly looking for image and multimedia files and had nothing to do with Overture. Maybe it's just a User Agent change from the old FAST multimedia crawler? Dunno.
I'll keep an eye out for it to hit again, or snap up another site to see what files it grabs next time. Every domain of mine that it's found so far it's hit pretty hard for a single day. It doesn't take much to notice it in the logs when a spider hits your site for 500+ files and over 10 megs of transfer over just a couple of hours.
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users
