2 replies to this topic

[JeremyH]

I've been trying to find out this information on http://www.robotstxt.org/ and http://sitemaps.org/ but I can't find it online.

In my robots.txt file do I place a link to my sitemap at the very top, or below a statement addressing all user agents? And can the place a link to a gz compressed sitemap?

Right now my robots.txt looks like this:

CODE

User-agent: *
Sitemap:  http://www.widget.com/sitemaps/sitemap-568511_widget-en_-NS_0-0.xml.gz
Disallow: /*redlink=1

[qwerty]

As I understand it, the sitemap declaration is a message for all bots. As such, it does not go in the middle of a standard user-agent/disallow couplet, as you've done it. Those two lines belong together and shouldn't be broken up, because they represent a single message: for this bot (or these bots), disallow these files/folders.

I don't think it matters whether you put the sitemap at the beginning or the end of the file, but I think it ought to be one of those two positions, and yes, you can certainly point to a zipped file.

[JeremyH]

Thank you,

I have changed it back to:

CODE

Sitemap: http://www.widget.com/sitemaps/sitemap-568511_widget-en_-NS_0-0.xml.gz

User-agent: *
Disallow: /*redlink=1

My cocern was when I had it that way before and pointed to a gz file, Google was still requesting I submit a sitemap to my site in Google Webmaster Tools. It made me think it wasn't seeing the link to the sitemaps in the robots.txt file.