14 replies to this topic

[rshilkrot]

Hi,
I have a question regarding sitemaps and their importance.
I'v done some tests on some of my sites, and i found out that when i put a sitemap that covers the whole of my site - Google indexed it faster. Faster than sites i havn't put sitemaps in, that is.
Which is pretty obviouse since he gets all the pages in the site grouped in one page...

My question is, doesn't this affect how Google sees my site?
I mean, does it really think that all these links in the sitemap page are "children" to that page? or, assuming it realy scanned the whole site, understands that this is only a site-map and doesn't reflect the true "depth" of these pages?

This brings up another important question: making a site "tree-like" (categorized and sub-categorized) is better than making the site "flat"? (no categorizing).

Thanks in advance,
Roy.

[dbmasters]

Personally, I use sitemaps on many sites just for the spiders...I really don't even care if visitors use them or not, Ihope they don't, because a well layed out site shouldn't really need one. I have no statistics or anything to indicate how useful it is, but I do know a lot of spiders visit it, and it is a one-stop shop to the whole site...which can't be bad.

[Scottie]

Hey Roy!

My question is, doesn't this affect how Google sees my site?
I mean, does it really think that all these links in the sitemap page are "children" to that page? or, assuming it realy scanned the whole site, understands that this is only a site-map and doesn't reflect the true "depth" of these pages?

Google sees every page as an independant page, not parent and child. It does look at the links pointing to that page to help determine PR and that includes internal links.

This brings up another important question: making a site "tree-like" (categorized and sub-categorized) is better than making the site "flat"? (no categorizing).

That depends on the size of your site. Obviously, it doesn't make sense for a small site. On a large site, it is definitely better to group the information and use breadcrumb navigation to help users and spiders find things more easily. The consistent linking (esp at lower levels) using breadcrumbs reinforces to the spider what that page is about.

Don't discount sitemaps as navigational tools! I think you'd be surprised how many people use them. I know I do- I can tell at a glance if the information I want is on the site and where it is.

[bwelford]

On a large site, it is definitely better to group the information and use breadcrumb navigation to help users and spiders find things more easily.  The consistent linking (esp at lower levels) using breadcrumbs reinforces to the spider what that page is about.

I can see where sub-directories help the website designer find particular web pages so as to make the design process easier. However I think your earlier comment was the more important, Scottie.

Google sees every page as an independant page, not parent and child.

I still am concerned by depth and would prefer all web pages in the root directory. I believe for the search engines this may avoid a depth penalty for "deeper" pages. I believe this is therefore better for spiders.

For users, you can always give the illusion of sub-directories and breadcrumbs for web pages that are really all in the root directory.

[SearchRank]

For large sites (with lots of pages) I believe site maps are good for spider crawling because you can place links to all of your pages whereas on the home page it may be a navigational nightmare to place links to every page. Google does say in its Webmaster guidelines that it is best not to have more than a 100 links on any given page so one shouldn't go crazy with a site map. If your site map has more than a 100 links, better to break it into categories instead of trying to link to every page.

If your site is small, say 10 pages and you have a link to every page on the home page, then a site map will not make any difference to crawling. It is just my recommendation that if you have graphic links to also have a set of text links along with it, say in the footer of the site. I think Google seems to favor text links over graphic links.

[Grumpus]

Google sees every page as an independant page, not parent and child.

I disagree.

Parent/Child probably isn't the right term, but Google does take into consideration how one page relates to another page. Depth in URL's isn't as critical as the logical formation of the URL.

By looking at how Froogle works (in assessing the contents of a page and identifying products within the page) we can even see that Google sometimes even relates pages to each other even if they don't directly link to each other - using the common page that links to both as the tie between the two.

G.

[JohnHocking]

I have a dynamic site and I generate a sitemap to help the robots
understand and follow the structure.

I find it helps the process by linking to the sitemap in the footer of the
page.

Now every page links to every other page through 1 to 2 hops.

Google seems to like my site, I have a pr5 with only 13 inbound links
and so far it has indexed over 1400 pages.

John Hocking

[Grumpus]

I think a "structured" site map is a good idea. I think providing a single page that links to every other page (which is only possible on small or mid sized sites) can be a detriment as it indicates (structurally) that each page is equally important and of equal focus. A flat map will work and it'll get your pages crawled, but it'll contribute greatly to the elemination of any benefits that come from a well structured site navigation.

If you site map pages match the overall structure of your site, then it reinforces the structural pattern and you'll see some benefits to that.

---

John - how old is your site? 1400 pages isn't a lot (unless that's all the pages you have) for a site with a base PR5. 24,000 is more in the range you are looking for. (Though that number will dip from time to time as your site goes through the update process - be it in a regular update or the more recent "rolling update").

G.

[Scottie]

Let me explain my point of view a little more...

I don't believe Google uses a parent/child type of categorization where the "parent" or higher directory carries more weight. Think about it like this... where is the real information... the article, the product description, the bio? Usually on the detail page. That's what the SE ought to WANT to place more importance on in order to return more relevant results.

Like Grumpus, I do believe the URL structure makes a difference. Not that it's too long and gets discounted, but that the directories in the URL tell you more about the info on that page.

www.domain.com/ezine/authors/bios/person.html

Do you have any idea what you might find on that page? Probably so! It makes sense to analyze the URL structure to determine what the page is about. If you've got a 10 pg site, it doesn't make sense to organize your materials this way, but as the site grows there are multiple benefits to logically creating folders and organizing your content.

When it comes to finding the page and assigning PR, the URL doesn't matter as much as how it is linked to. So if the link path is home page>sitemap>page, you've got a pretty easy way to get those pages found in addition to the path for the user which is more like home page> ezines> authors > Person.

[powerofeyes]

John - how old is your site? 1400 pages isn't a lot (unless that's all the pages you have) for a site with a base PR5. 24,000 is more in the range you are looking for.

We have a site with PR5 and had 25,000 pages listed in google now it reduced to 17,000 though,
One thing I am not sure how comfortable search engines are with dynamic content,
For example 2 of my client sites has big inventories of products, one around 20,000 products and other somewhere around 14000 products,
Just as a normal query string google listed upto 4000 pages(20,000 products site) for about 3 months and didnt extend the No of pages,
But once we used XQASP tool those No of pages in google's database went to 25,000+ plus(extra pages same URL in both ways indexed), same thing happened to the other site too,
Why this happens, what is happening here, Can someone shed some light on this,

VIJAY,

[Grumpus]

I don't believe Google uses a parent/child type of categorization where the "parent" or higher directory carries more weight.

Now that I can agree with 100%. The page relationships are derived from linking structure, not URL structure. (Which is why I suggest that a flat site map page can be a detriment - it doesn't give life to the relationship structure).

Let's look at this a little more, too because there are things where directory/page naming methods do influence relationships:

www.domain.com/ezine/authors/bios/person.html

Let's say I changed it to:

www.domain.com/ezine/authors/xyz/person.html

Even though I didn't use the word "bios" anymore, every page in the XYZ directory is likely a biography. Once the spider gets to enough pages in that XYZ directory to see that "Biography" is the common "general theme" of each page and that "person.html" controls the specifics of what that biography contains, then when you add a new page, the spider knows some of what to expect from it.

Compounding this, your "person.html" pages most likely work from a common layout. The photo of the person is always "here", the person's name is always "there", and their birthday is always "down here", and the details of the biography are "right there".

So, now that Google knows that all the pages in this directory are bios, it also knows where on the page the "who's bio is it?" information is going to be. If that section of your site lists "Inventor's Biographies", then there is a decent change that your "hub" page (the page that lists all the people and links to the bios) will rank fairly well for "inventor birthdays" even if the word "birthday" isn't 'optimized' on the hub page (though it probably will need to appear once or twice). If you search for "Thomas Edison's Birthday" the specific page for Edison may rank pretty well - even though the word birthday only appears once on the page - why? Because you've shown a pattern of consistently providing birthdays of inventors on your pages. (I know that this type of thing has been happening for some time as my movie site - that came down late last year - used to have celebrity birthdays listed and I tended to rank very well for this type of search. There are similar type things going on in various other sites I oversee, too).

Consistent naming conventions and page layout all play a part in this and it's all tied together by the linking structure that echoes/mirrors your naming convention structure.

G.

[lepp]

John - how old is your site? 1400 pages isn't a lot (unless that's all the pages you have) for a site with a base PR5. 24,000 is more in the range you are looking for.

Still new to this and hadn't come across this before. Does a large number of pages on your website improve your PR? Our website has about 2500 pages and had a PR of 5 until January when suddenly it dropped to a 4.

[Jill]

The dropping of your PR has nothing to do with anything you may have done, etc. It's simply that as more and more pages get indexed by Google, they change the way they calculate it and it gets harder to get a higher number. It's nothing bad though, you still have the same links you always had, etc.

And no, a larger number of pages doesn't increase your PR. 2500 pages is a nice large number anyway!

Jill

[Grumpus]

Does a large number of pages on your website improve your PR?

No - the opposite: Your "mean PR" (the average PR across your site) improves the number of pages in the index. The numbers I give are an educated "guess" in the range of number of pages that a person can expect to have in the index but the number varies from month to month. This month, overall, seems lower than most for everyone, so it's hard to tell for sure. Based upon some quick observations this morning, PR5 sites seem to have 12K-22K pages (or thereabouts) rather than the normal 20K-28K pages (probably a short deep crawl period for some/most sites? I dunno.)

If you site isn't set up so that PR can "flow through it" properly, then your mean average is going to be down, so by looking at the PR on the front page, it's much guess how deep the crawl may be.

Again, the numbers I give have a lot of variables and, if you'd like to discuss some of them, feel free to start a thread (since this really isn't a sitemap issue - though that is a part of it). I'll be happy to discuss my observations from over the years - though keep in mind, that there are no steadfast rules just guidelines that can be followed that tend to provide good results.

G.

[lepp]

Thank you Jill & Grumpus for clarifying that. Even if # of pages was an issue, our site has the pages it needs and there's no end user reason to add extra unnecessary pages even it helped with PR. My boss is big on Less is More. But I am thinking hard about adding a sitemap to ensure a proper flow through.