7 replies to this topic

[eteare]

So, your site has 50 MM pages (really, no kidding). But in Google's webmaster tools you can only submit Sitemaps with 50k URLs or fewer. You can make a Sitemap index that link to Sitemap indexes. But this quickly gets insane. Especially when you have millions of URLs. Any suggestions?

[Michael Martinez]

So, your site has 50 MM pages (really, no kidding). But in Google's webmaster tools you can only submit Sitemaps with 50k URLs or fewer. You can make a Sitemap index that link to Sitemap indexes. But this quickly gets insane. Especially when you have millions of URLs. Any suggestions?

You seem to be suggesting that you have the capability of adding 50,000,000 URLs to a single file but that you cannot generate 1,000 files with 50,000 URLs in each. Is that correct?

ON EDIT: I'm not trying to be snide. I'm trying to determine if the tools/resources you have available have forced you into an all-or-nothing scenario with respect to dumping 50,000,000 URLs into a text file.

Edited by Michael Martinez, 02 September 2010 - 01:23 AM.

[Mooro]

I'd either follow the guidelines or not bother, I actually think not bothering is the best method, I came to that conclusion when making up over a gig of map content for a site with over five million urls, it doesn't make a squat of difference in my opinion and you'd be better off spending more time working on the internal navigation and discoverablity of your content.

I'm convinced on site discovery means more to the engines than you feeding them a site map, one is what they can find the other being what you want them to find....

The biggest benefit a selection of maps will bring has been mentioned here a number of times after Vanessa Fox mentioned it, categorise your site maps then you can see which sections of your site have indexation issues.

That's the best use for a map I know as I'm convinced they don't contribute to visitors or $£$£ one little bit.

[eteare]

You seem to be suggesting that you have the capability of adding 50,000,000 URLs to a single file but that you cannot generate 1,000 files with 50,000 URLs in each. Is that correct?

ON EDIT: I'm not trying to be snide. I'm trying to determine if the tools/resources you have available have forced you into an all-or-nothing scenario with respect to dumping 50,000,000 URLs into a text file.

Ok, I see your point and you have every right to be snide. I am the SEO person, not in the engineering dept where they build this ginormous directory, so give me a little bit of a break. Trying to do with without dev help, you know?

I do, however, believe VERY strongly in having an xml Sitemap that can be submitted to various webmaster tools, so I am just looking for tools that have helped people out. That's all.

[Michael Martinez]

Ok, I see your point and you have every right to be snide. I am the SEO person, not in the engineering dept where they build this ginormous directory, so give me a little bit of a break. Trying to do with without dev help, you know?

I do, however, believe VERY strongly in having an xml Sitemap that can be submitted to various webmaster tools, so I am just looking for tools that have helped people out. That's all.

Really, I was NOT trying to be snide or rude. I was only asking what the scope/limit of your capabilities are.

Am I inferring correctly that you don't have the means to build a text file, and therefore you just want to know if something out there will crawl the site and build the sitemaps?

With 50,000,000 URLs I would strongly advise AGAINST using an outside crawler. It might be more productive to work with the engineering team to develop a redundant site navigation to assist with crawling and link-based prioritizing of important hub pages.

[Mooro]

It might be more productive to work with the engineering team to develop a redundant site navigation to assist with crawling and link-based prioritizing of important hub pages.

Wow, there's a strange distrotion to the echo in here!

you'd be better off spending more time working on the internal navigation and discoverablity of your content.

I feel that if you have to rely on site maps for crawlers to discover your content your navigation and architecture are letting you down.

@eteare - how come you feel so strongly that they bring value? What value and how is it quantifiable?

[Michael Martinez]

Wow, there's a strange distrotion to the echo in here!

Not when people look at the original, unadulterated comment that talks about NOT using external crawlers.

[eteare]

In my experience, giving WMTs the URLs does get them indexed more readily. But even better, Google will tell you how many of the pages in your Sitemap have been indexed, which is a nice stat to watch over time.