Jump to content

  • Log in with Facebook Log in with Twitter Log In with Google      Sign In   
  • Create Account

Subscribe to HRA Now!

 



Are you a Google Analytics enthusiast?

Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE! 

 



 

 www.CustomReportSharing.com 

From the folks who brought you High Rankings!



Photo

Sitemap Placement In Robots.txt


  • Please log in to reply
2 replies to this topic

#1 JeremyH

JeremyH

    HR 5

  • Active Members
  • PipPipPipPipPip
  • 367 posts
  • Location:San Diego

Posted 27 March 2011 - 11:09 PM

I've been trying to find out this information on http://www.robotstxt.org/ and http://sitemaps.org/ but I can't find it online.

In my robots.txt file do I place a link to my sitemap at the very top, or below a statement addressing all user agents? And can the place a link to a gz compressed sitemap?

Right now my robots.txt looks like this:

CODE
User-agent: *
Sitemap:  http://www.widget.com/sitemaps/sitemap-568511_widget-en_-NS_0-0.xml.gz
Disallow: /*redlink=1


#2 qwerty

qwerty

    HR 10

  • Moderator
  • 8,625 posts
  • Location:Somerville, MA

Posted 28 March 2011 - 06:28 AM

As I understand it, the sitemap declaration is a message for all bots. As such, it does not go in the middle of a standard user-agent/disallow couplet, as you've done it. Those two lines belong together and shouldn't be broken up, because they represent a single message: for this bot (or these bots), disallow these files/folders.

I don't think it matters whether you put the sitemap at the beginning or the end of the file, but I think it ought to be one of those two positions, and yes, you can certainly point to a zipped file.

#3 JeremyH

JeremyH

    HR 5

  • Active Members
  • PipPipPipPipPip
  • 367 posts
  • Location:San Diego

Posted 28 March 2011 - 06:57 AM

Thank you,

I have changed it back to:

CODE
Sitemap: http://www.widget.com/sitemaps/sitemap-568511_widget-en_-NS_0-0.xml.gz

User-agent: *
Disallow: /*redlink=1


My cocern was when I had it that way before and pointed to a gz file, Google was still requesting I submit a sitemap to my site in Google Webmaster Tools. It made me think it wasn't seeing the link to the sitemaps in the robots.txt file.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

SPAM FREE FORUM!
 
If you are just registering to spam,
don't bother. You will be wasting your
time as your spam will never see the
light of day!