6 replies to this topic

[ForumEngine]

I believe my sites are crawlable, but I'm curious...

Is an XML sitemap enough to let the engines know where your content is?

For the sake of discussion let's assume that we've verified in Google Webmaster Tools that Google has found our sitemap.

[qwerty]

It gives them a list of your URLs. It certainly doesn't guarantee that they'll crawl and index all the pages at those URLs, but nothing guarantees that.

One thing I'm not sure of is whether the priority, lastmod or change frequency data accomplishes anything. If, for example, you give a page a priority of 1 (the highest setting), but you don't link to it from any of your other pages, will a search engine necessarily treat is as important and crawl it more than others? I have my doubts.

[ForumEngine]

It gives them a list of your URLs.

Right, it seems to solve the problem of "can they find my pages". I like solutions like that, fix the whole problem in one fell swoop.

It certainly doesn't guarantee that they'll crawl and index all the pages at those URLs, but nothing guarantees that.

Agreed, but why wouldn't they?

One thing I'm not sure of is whether the priority, lastmod or change frequency data accomplishes anything.

I don't know either, but have my doubts as well. As a coder myself, I know that sometimes we put features in just because we can, and it seemed like a cool idea on the day we did it. The features you mention smell like those kind of features to me, but I really have no idea.

[qwerty]

Agreed, but why wouldn't they?

That's easy. Having content doesn't necessarily equate to having worthwhile content. That's pretty much what PageRank (real PageRank, that is) is for. Are the pages of your site valuable enough that they're all worth keeping track of, or would Google and its users be better off grabbing what appears to be the most important stuff and skipping the rest? Especially if you've got a big site, it's unlikely that all of your pages are going to be indexed. You can increase the odds of more pages being indexed by making sure that when Googlebot requests a URL, everything goes according to plan: it finds the page and finds unique content there. If not, you may have just missed an opportunity, and that means one less of your pages is going to get indexed.

[Catz]

How about a site map created for human visitors, linked to from your home page with html links to your pages?

[qwerty]

One thing I'm not sure of is whether the priority, lastmod or change frequency data accomplishes anything.

Thanks to John Mueller of Google (and Search Engine Roundtable, where I found the link to this), we've got a fairly official answer on priority settings in sitemaps:

I wouldn't worry too much about the "priority" values. If you're able to generate them based on useful information from your website, then it's a good idea to do that. If you can't automatically generate them and have too many pages to set the values manually, then it's fine to submit the Sitemap file without "priority" information. The "priority" value gives us a bit more information about your site, but changing values here generally won't lead to visible changes in crawling, indexing, or ranking overall.

[ForumEngine]

Thanks Bob, that's the kind of answer I like to hear. Don't worry about it. Appreciate your research!