31 replies to this topic

[qwerty]

You can now point spiders to your sitemap from your robots.txt. Just add

CODE

Sitemap: <http://www.domain.com/sitemap.xml.gz>

(or whatever the URL of your sitemap is) anywhere in the file.

See http://www.sitemaps....l#submit_robots

[Jill]

Cool! Thanks for the info, Bob!

I was going to attend the sitemap session here at SES, but the room was packed tight by the time I got there.

[harpsound]

Okay
Up until now I have used a site map generator to create my .xml
One with Google specs as a Google-sitemap.xml
One to Yahoo specs as a urllist.txt

I have now generated a a non-Google specific generic .xml
Put it in the robot.txt
And resubmitted it to Google webmaster Tools also.
The webmaster tools rejected the generic .xml file outright.

Am I being too early an adopter here?
The two different submission approaches appear to be incompatible at this time for the same generic.xml file.

The generator
www.auditmypc.com/free-sitemap-generator.asp

Thoughts Ideas?

Stephen V

[qwerty]

The version that Google has accepted in the past should work for Yahoo, MSN and (now) Ask as well. I don't know the generator you're using, but if it's ok with G, it should be ok.

[harpsound]

That appears to be the case.
My Yahoo account has "processed" the Google version.

S

[Shlomo]

The possibility to define the location of sitemap.xml in robots.txt
is an important advancement.
I think that the Search-engines should go one step further
and allow the webmasters to include in the robots.txt
all the functions of the sitemap.
So that we will able to provide these 2 sets of info
in one file.

[qwerty]

That would be nice, but I doubt it would be possible. The sitemap protocol calls for an XML file, and robots.txt has a very different structure from that. If the robots exclusion protocol were updated to allow for XML, then maybe the two could be combined.

[Shlomo]

Of course you are right.
These 2 files have various formats.
But the Search-Engines' teams are very strong teams, and they can perform this change.
I think that the robots.txt should also be transformed to the XML format.
It will save much work for the bots. Now they must identify and read 2 files.

I'm not an American. But you can have more influence over this improvement.

[qwerty]

Here's a tiny complication. When the announcement about adding the URL of your sitemap to the robots.txt file, I set one up for a client of mine whose robots.txt had been empty, so now its only content is

CODE

Sitemap: <http://www.domain.com/sitemap.xml.gz>

So what happens when I check the robots.txt file in Google Webmaster Tools?

robots.txt file does not appear to be valid

I'm guessing it's just that this particular tool hasn't been updated to recognize the update to the protocol, but I'm going to see if I can get an answer from Google.

[torka]

Wonder what it would say if you added

CODE

User-agent: *
Disallow:

to the robots.txt file? Maybe it's looking for a User-agent: bit to start things off before you hit the Sitemap: part?

Will be interesting to hear what Google has to say by way of explanation/workaround...

--Torka

[qwerty]

I got a response, but not from a Googler. The person who responded is very active in the forum there, and s/he says that I was right in thinking it's just a matter of the webmaster tool needing to be updated.

<added>Here's some official info from Vanessa Fox:

We're adding support for the new instruction in our robots.txt analysis tool. The tool should be updated soon!

[Nueromancer]

Of course you are right.
These 2 files have various formats.
But the Search-Engines' teams are very strong teams, and they can perform this change.
I think that the robots.txt should also be transformed to the XML format.
It will save much work for the bots. Now they must identify and read 2 files.

I'm not an American. But you can have more influence over this improvement.

"Search-Engines' teams are very strong teams"

Yeh right why isnt the ssl bug fixed on the uk version of base then :-) look at the discuson on matts blog about how G handles robots.txt - i'me with the camp that says they parse incorectly in certain degenerate cases.

The robots.txt standard has been around for a number of years and trying to fit a totaly diferent protocol sitemaps into it is not a good idea - quite why it couldn't have been the sitemaps.xml files goes at the root of a site and leave it at that.

oh just noticed 100th post W00t!

[Randy]

WOOt indeed ! Congrats on 100.

I agree with your general sentiments. I too don't get why the engines would want to use robots.txt to do this. It only muddies the water further IMO. But on the other hand I can see why they might not want to force everybody to use the same name for a sitemap.xml file. Not all software packages that are out there can be easily tweaked to use the same xml filename.

Easy solution. Instead of querying the site only for a robots.txt file, as they all already do every visit, also query for another plain text file called sitemap.txt file. Then anyone who wanted to provide a sitemap could put the location info in this separate file. Maybe even slap a second line in there to allow people to help them out by identifying the filetype.

CODE

Filetye: xml-gzip
Sitemap: <http://www.domain.com/sitemap.xml.gz>

[qwerty]

Gah. You want them to change the protocol so that everyone creates a new file that will always be in the same place so the bot can find it easily, and that file will serve the purpose of indicating the location of another file that will always be in the same place? Why not skip part one, and even skip pointing to it from robots.txt, and just have us all call our sitemap file either domain.com/sitemap.xml or domain.com/sitemap.xml.gz, and then set the bots to always check for it at those two URLs, the same way they always look for robots.txt at domain.com/robots.txt?

Autodiscovery without having to tell the spiders to autodiscover something seems a lot more like autodiscovery to me.

[Randy]

ROFL A new Sitemaps protocol rather than having to fudge around with robots.txt Bob, that's all I'm saying. robots.txt already has enough issues with different engines supporting different non-standard features.

The sitemap protocol could automatically tell the spider to look for sitemap.xml, sure. But if whatever tool that made it doesn't name it that exact name and it's a constantly updating page as so many are where the webmaster couldn't rename the page by hand, that particular site would be left high and dry.

So in my view with sitemap.txt they give full control to the webmaster, just like with robots.txt. They specify the sitemap filename, and can even include filetype info so that the spider knows what to do with it when it gets it.