20 replies to this topic

[Michael Martinez]

Okay, the server name is not necessarily the crawler name. It's not really MSNBot any more even though it comes from a server using that name.

What does your robots.txt look like?

[hegu]

# All robots will spider the domain
User-agent: *
Disallow: /*?print$
Disallow: /cgi-bin/
Disallow: /blog/admin/
Disallow: /update/
Disallow: /images/

I have some other folders too. But I removed them to post it here.

[hegu]

I did some search and find out this article on Bing.com coomunity.

http://www.bing.com/community/site_blogs/b/search/archive/2006/11/29/search-robots-in-disguise.aspx

It tells us that how to identify the real msnbot. It seems that ip is for real msnbot.

But my question is: why am I not seeing bingbot?

[Michael Martinez]

It IS BingBot. Microsoft just didn't follow through on their announcement to replace MSNBot with BingBot (or else they have quietly returned to the earlier user-agent). Don't worry about it.

Back to your robots.txt -- it looks okay. If your XML Sitemap is relatively small, please send me a PM with a sample of the code and I'll take a look for you. If it's a huge file, just PM me the domain name.

[hegu]

Michael,

My xml sitemap is correctly created. No problem with it. Thanks for the offer though.

[hegu]

AN UPDATE:

Msnbot going deep into the site. And 3 clicks came from bing.com with three new keywords.

I made sure site doesn't have html erros or page not found etc etc.

yayyyyy!