Jump to content

  • Log in with Facebook Log in with Twitter Log In with Google      Sign In   
  • Create Account

Subscribe to HRA Now!

 



Are you a Google Analytics enthusiast?

Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE! 

 



 

 www.CustomReportSharing.com 

From the folks who brought you High Rankings!



Photo
- - - - -

Msnbot Crawling Only Homepage


  • Please log in to reply
20 replies to this topic

#16 Michael Martinez

Michael Martinez

    HR 10

  • Active Members
  • PipPipPipPipPipPipPipPipPipPip
  • 5,028 posts
  • Location:Georgia

Posted 19 March 2012 - 01:00 PM

Okay, the server name is not necessarily the crawler name. It's not really MSNBot any more even though it comes from a server using that name.

What does your robots.txt look like?

#17 hegu

hegu

    HR 4

  • Active Members
  • PipPipPipPip
  • 118 posts

Posted 19 March 2012 - 09:45 PM

# All robots will spider the domain
User-agent: *
Disallow: /*?print$
Disallow: /cgi-bin/
Disallow: /blog/admin/
Disallow: /update/
Disallow: /images/

I have some other folders too. But I removed them to post it here.

#18 hegu

hegu

    HR 4

  • Active Members
  • PipPipPipPip
  • 118 posts

Posted 20 March 2012 - 12:29 PM

I did some search and find out this article on Bing.com coomunity.

http://www.bing.com/community/site_blogs/b/search/archive/2006/11/29/search-robots-in-disguise.aspx

It tells us that how to identify the real msnbot. It seems that ip is for real msnbot.

But my question is: why am I not seeing bingbot?

#19 Michael Martinez

Michael Martinez

    HR 10

  • Active Members
  • PipPipPipPipPipPipPipPipPipPip
  • 5,028 posts
  • Location:Georgia

Posted 20 March 2012 - 03:20 PM

It IS BingBot. Microsoft just didn't follow through on their announcement to replace MSNBot with BingBot (or else they have quietly returned to the earlier user-agent). Don't worry about it.

Back to your robots.txt -- it looks okay. If your XML Sitemap is relatively small, please send me a PM with a sample of the code and I'll take a look for you. If it's a huge file, just PM me the domain name.

#20 hegu

hegu

    HR 4

  • Active Members
  • PipPipPipPip
  • 118 posts

Posted 23 March 2012 - 10:21 AM

Michael,

My xml sitemap is correctly created. No problem with it. Thanks for the offer though.

#21 hegu

hegu

    HR 4

  • Active Members
  • PipPipPipPip
  • 118 posts

Posted 28 March 2012 - 01:16 PM

AN UPDATE:

Msnbot going deep into the site. And 3 clicks came from bing.com with three new keywords. :D

I made sure site doesn't have html erros or page not found etc etc.

:applause: yayyyyy!




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

SPAM FREE FORUM!
 
If you are just registering to spam,
don't bother. You will be wasting your
time as your spam will never see the
light of day!