45 replies to this topic

[Grumpus]

It's been going on for a while. Back in the day, you'd optimize a page that you wanted to appear in the search engines all by itself. The only external key to SEO was that it needed some links to it to be found, and on some search engines (like Google) it had to have lots of links in order to rank well. So, we'd take our home page (the one that had all the links coming to it) and optimize the hell out of it. Maybe then, we'd create another page (sometimes on a different domain because it's easier to get links to a top level page than a deep page) optimize it, and set on about getting links to it.

As recently as a year and a half ago, it was the rule that search engines looked at each page on its own and therefore you had to make sure that each page had all the elements the engines were looking for (links, keywords, content and clean code) and the page would do well.

Then, suddenly, people started noticing that news sites, forums, some web directories, blogs, and the like started doing a lot better in searches. The resoundingly agreed upon explanation of this was that the engines (namely Google) prefered new stuff and these types of sites tended to be updated more frequently. That may have been the case to some extent, but there's far more to it than that. There were, and are, plenty of other sites that manage to rank pretty well - even for pages that hadn't been updated for years.

The commonality between all the sites was their basic structure. The way they start with a broad concept and focus in on the specifics of the various elements of that concept the deeper you go into the site.

For those of you who like to play along at home, go ahead and open up a new window with the front page of these forums in it. We're gong to follow the trail to this thread to see how it all works.

By looking at that page we can see that this place is about Search Engine Optimization, even if the title didn't tell us so. SEO is a very broad topic and as we move down through the page, we see a link and a description to all of the various key elements of SEO.

When you click the "Friendly Design" link, you are brought to a page that is still about SEO, but it's more specifically geared toward design and usability as it relates to the search engines. The line next to the visually confirms this for us, but more importantly is shows the way "back" from this more specific idea to the more general notion of Search Engine Optimization.

Below that we are greeted with another list of even more specific notions - each of them dealing with design and or usability issues as they relate to SEO. If you find the link that says, "Learn from a Forum's Structural Design" you'll end up in this thread. Without even looking at what I've written here in the body of this message, you already know that this thread is about Learning from Structural Design from a design and usuability perspective on the topic of search engine optimization. We know this because of the path we took on our way to finding this post.

Today's search engine is perfectly capable of making those same observations that we just made. And, as time goes on, it's getting better and better at putting those observations into practical use. Therefore, even if this was a completely blank post, both users and search engines alike could develop certain expectations about what should be here on this page.

Obviously, this can work both for and against you.

I see a lot of posts here (and on other SEO forums) where people say things akin to "I've got all the keywords on my page and lots of links to it, but it won't rank!" One thing to consider (though it's definitely not the only possible answer) is that your page may be contradicting the expectations the search engine(s) had for what should written there.

In the same way that you need to meet your user's expectations in order to convert their traffic into sales (or merely just to keep them hanging around your site) search engines are developing their own expectations for you to meet.

Even if you have a small site, I suggest that you don't necessarily link directly from your front page to every page in there - or at least develop some sort of "notion path" to suppliment the front page linking in an effort to help search engines follow the idea-flow.

So, if your site has three services that you offer (say web design, SEO, and graphics design) along with an About Us page, your portfolio page, etc. Then even if you link to all three of your services, you should also have a "Services Overview" page that clearly says, "These are my services" and then links to each specific service.

That way rather than having:

Me -> About Me
Me -> Portfolio
Me -> Graphics Design
Me -> Web Design
Me -> Seo

You have:
Me -> About Me
Me -> Portfolio
Me -> Services
-Me -> Services -> Graphics Design
-Me -> Services -> Web Design
-Me -> Services -> Seo

Help the engines to refine their expectations one step at a time rather than asking them to make a huge leap. They aren't quite as smart as us people yet.

Hope this is helpful!

G.

[bobsledbob]

I'm wondering how a sitemap plays into this. I have found that sitemaps are both useful to my users and have been told (mostly through Jill's readings) that sitemaps are useful for search engines. It sounds like you're saying that sitemaps may sidestep this heirachical way of organizing your pages. Are sitemaps a bad thing then?

[Grumpus]

Not necessarily bad. As I said in my example with the web design site, you can still have the links that go directly to each of your services, but adding that "tweener" page to show how it all fits in will be a big help. So, a sitemap, especially in cases for a large site, where your "train of thought" navigation will take someone 5 clicks to get where they want, can be very useful.

The problem with a sitemap is when it's used because it's the only way a spider can get into your pages (like if you have Javascript Links or you need to submit a form to get into the site or whatever). You lose all of this structural benefit I talk about, you lose a lot of your link power because there are so many links on the page, you lose Local Rank benefits (that's where a separate value similar to pagerank is given to a page at the time the search is run. It counts the inbound links to that page from the top 20 or so other pages that appear in that search - in essence, it's context sensitive PageRank).

Another problem with a sitemap is that it's often used as a crutch or an alternative to good navigational structure. So often a site is started, and people add something and haphazardly link to it. Eventually, there are so many new pages and links all over the place that the only way a user can possibly find what they want is to get at it via the site map. Spiders get confused too, so if your site structure is confusing, you'll get none of these benefits.

These forums here have a sort of "site map" thing going that is very useful to the search engines. That's the "Today's Active Posts" link. Though, by the way it's set up, the spiders can only get to today's and yesterday's posts, it's a great way to get fresh results into the search engines as it puts all the new stuff there in one handy spot. If I were the SEO person for these forums though, I'd change that link text and the title of the results page to say "Today's New Posts" probably - or some other term that might be commonly searched for? (As I say, I'm a mechanics guy, not a content/keywords themselves, kinda guy).

(BTW - Google does learn what page(s) on your site are best for popping in and seeing what's new and what's updated.)

For me, though, I wouldn't use a sitemap. True, if you have an application (like forum software) that isn't crawlable, it's easier to make one page with crawlable links on it than to rewrite the whole application (which I why I think sitemaps for dynamic sites is a popular suggestion), but if you have the opportunity, your time is better spent working on your nav structure than making a page full of links.

G.

[Grumpus]

I should also mention here that there's another potential danger to site maps. As I'm sure has been discussed here over the past several months, link pages (i.e. Pages with nothing on them but links to other sites and places - sometime reciprocal sometimes not) don't really pack any punch anymore. The link will probably count for PR purposes, but it really does absolutely no good for any other purpose.

Part of the reason of this is because of link exchange programs and such, but these types of pages are showing a lack of power across the board nowadays. Primarily, I guess, because the links page just doesn't have anything on it to enable it to rank on its own. It's just a page of links. The same is true of a site map - it's just a page of links with no power of its own.

---

And, I guess - even though I rarely talk about the value of PR anymore - I should mention how a sitemap is murder on your PR. PR works in a way where a page has a certain "number" assigned to it. 80% of that number is divided up equally amongst all the outgoing links on that page and passed along to the next page. So if you had regular navigation that passed through a couple of pages, your site would start with a high PR out front and get a little lower at the next level and a little lower at the next level and so on. Your "broader" less focused pages (the ones with the more general and more competitive search terms) will have a higher PR and your more specific pages (the ones with focused terms that aren't as competitive and don't need as much PR to rank well) get less PR.

When you link through the sitemap, though, all of your pages get an equal share of a highly dilluted amount of pagerank.

---

Oh - and Google's only going to crawl a certain number of links on a page. I think their site says 100, but I suspect that it may be higher in some cases. PR of the page with all the links on it seems to have a bearing upon how many of those links on it Google will bother to crawl.

So, if your site has 125 pages, then 25 pages of your site will never get crawled.

--

I guess there are more reasons than I originally thought not to use a sitemap, huh?

G.

[bobsledbob]

Very interesting, dare I say even revolutionary, thinking. Thanks a ton.

One more question then, or clarification, on the sitemap thing.

If you use a sitemap to link to pages that can also be found through a more hierarchical click path via generic vanilla hyperlinks, you don't see as much or any problem with this? Or, does some of what you've just written still weigh in?

[Grumpus]

I think it would depend, really, on the size of the site. Once you reach a certain size, (for example, this forum is huge in respects to number of pages) the site map is both impractical and self defeating. If, on the other hand, we're talking about a 20 or 30 page site, the map page isn't going to muddy things up too much I don't think.

It's a tough call, really. I think that if you are really good at watching and keeping on top of a good number of other good SEO techniques, then one little thing - like a site map - isn't going to make a huge change in the grand scheme of things.

The thing that does make a difference is consistency. Whether it's navigation structure, URL structure (different than nav structure, but it works along the same lines), or even how the page is laid out. If you have a bunch of products, the price should always appear in the same place with the same qualifying words around it on every page. This helps Google (and Froogle, really) know that it's a price for the product that's displayed there.

Back to map pages - I guess at the end of the day, I'd never use one (to the level of putting all the links to all the pages on my site onto one page). I'd never suggest to one of my clients to use one. I have never and probably wouldn't (in today's atmosphere anyway) advise someone to remove a map page that's already there, though.

G.

[SEO-Richard]

Oh - and Google's only going to crawl a certain number of links on a page. I think their site says 100, but I suspect that it may be higher in some cases. PR of the page with all the links on it seems to have a bearing upon how many of those links on it Google will bother to crawl.

So, if your site has 125 pages, then 25 pages of your site will never get crawled.

I think I know what you're trying to say, but as written it imples that a site that is accessed only through a sitemap with 125 links on it won't have 25 of its pages indexed.

Very interesting stuff, though. It sounds like hubs, buckets and breadcrumbs ....

[powerofeyes]

Hello Grumpus,
I dont know how much I agree with you on your definition on sitemaps, I absolutely feel there is a lot of value for sitemaps in this update, Infact most of the sites I am working on the PR value of the Sitemap has Jumped from 2,3 to 4,5 6.
When it comes to google I will definetely say there is a lot of weight added to pages which line to lots of quality relevant pages, Just like how a directory page works, I feel G considers sitemap pages s Authority pages and is giving lot of benefit to them,
And regarding your defining of PR distribution, As far as I see sitemaps are not and will not affect any PR distribution on a site, We know PR is passed from pages to pages, Mainly pages connected to the index page is given more weightage than pages connected to the sublevel pages,

Basically PR sharing in static pages this is what I am seeing,

Homepage (PR 6) <--> page connected reciprocally will get the best PR (possible PR6 or PR5),
Homepage (PR6) <--> sublevel page (PR5) <---> Second level (page connected and reciprocated with the sublevel page and connected to the Homepage but not reciprocated with the Homepage)(PR4 or PR5)

Ok when it comes to sitemaps, How you say it will affect PR, Afterall Sitemap page is one more sublevel page of the site, All it does is gets PR from Homepage and passes on to pages in the sitemap,
So why does a sitemap affect the PR distribution of the site, The only PR connection is between the sitemap and the page it is connecting, Why does it affect the overall PR distribution of the site, Infact the sites I worked on and the ones I have researched I have not seen anything like you say,
We all know PageRank is mostly Page to Page talk so why is it affecting the site,

link pages (i.e. Pages with nothing on them but links to other sites and places - sometime reciprocal sometimes not) don't really pack any punch anymore.

Then how you define all the directories, There is a big difference between a link farm and a sitemap, i think you are relating both of them, We can kind of relate sitemap pages to directories, And as we all see directories are performing exceedingly well in this update, I feel sitemaps will not affect a site in anyway and most of the time only helps,

VIJAY,

[Grumpus]

I think I know what you're trying to say, but as written it imples that a site that is accessed only through a sitemap with 125 links on it won't have 25 of its pages indexed.

Yup. That's exactly what I'm saying. Interestingly, Google is still suggesting using a sitemap in the same sentence where they say keep it below 100 links.

From Google's Guidelines

Offer a site map to your users with links that point to the important parts of your site. If the site map is larger than 100 or so links, you may want to break the site map into separate pages.

And several lines below that:

Keep the links on a given page to a reasonable number (fewer than 100).

And from Vijay:

And regarding your defining of PR distribution, As far as I see sitemaps are not and will not affect any PR distribution on a site, We know PR is passed from pages to pages, Mainly pages connected to the index page is given more weightage than pages connected to the sublevel pages,

Nope. PR passes through any link on any page to any page. Here's how PR works.

----

Also, my first post doesn't say don't use a site map, it merely states that structuring your site with a logical flow helps. In essence, each page on your site maps to the next page in the logical sequence. Note that in Google's own words you should have a site map that goes to important pages on your site (i.e. The key gateways into the deeper information it contains). It also doesn't say that this site map need be on a separate page.

G.

[powerofeyes]

Nope. PR passes through any link on any page to any page. Here's how PR works.

That document is good one and that is what I meant too, I have read many PR documents speculating the calculation of PR on the web, Most of them are good speculations, I am not coming to PR question, only question is the sitemap, You say sitemap affects a site and I am trying to explain no way it is affecting a site,

each page on your site - maps to the next page in the logical sequence.

I agree on this, Brett of WMW has formulated a nice theme structure for a site long back, it is a really valuble document and will still help people designing site,
Theme Pyramids and structural sequence of a site formulated by Brett,

It also doesn't say that this site map need be on a separate page.

From Google Webmaster Guide lines:

Offer a site map to your users with links that point to the important parts of your site. If the site map is larger than 100 or so links, you may want to break the site map into separate pages.

I feel they are pointing to seperate page for site map here,

[Grumpus]

I don't see that "Offer a site map to your users with links that point to the important parts of your site" says anything about putting it on a separate page. It only says make more pages if there are more than 100 links.

Yes, Brett's article is good - and was way way way before its time.

I do have to say that, as we were discussing, the site map is the only way a spider has of getting into your site, it will most surely have a different effect (on PR and other areas) than if your site had a sequential navigation structure. PR is passed along by taking the "power of the linking page" (a number that most say is 80% of the page's PR) and dividing it out equally amongst all the links on a page. Any page that links to all the other pages on the site therefore can only pass an equal amount of PR to each of the pages. Structured navigation passes a proportionate amount of PR down the through "tree" (or pyramid, in Brett's stuff).

G.

[Dyan]

Please bear with me while I try to put something into very basic terms to see if I grasp this......

If I run a site with a high number of pages, I am better off organizing them so that my main page only links to my main section pages (which there are maybe six of). Then from each section page, I can link to the pages that relate to that section. Thus I avoid running into Google's link limit per page, right?

But if I use a site map link on my main page (because my exisiting navigation is javascript), then I'm defeating the purpose of my organizational structure because I've put all my links on one page and have now exceeded Google's link limit per page?

ohhhhhhh *expletive deleted* I just spent the weekend designing a site map because I learned that spiders couldn't follow my javascript menus. Okay......... back to the drawing board.

[t49]

Why not build a multi-page sitemap?
Map Page 1: contains links to section maps,and descriptions of sections of site.
Map Page 2: links to pages in section 1
Map Page 3: links to pages in section 2
Map Page 4: links to pages in section 3
etc.
etc.

added benefit: descrriptions add 'content' to site map pages.
Tom

[Grumpus]

It's not just a matter of link limits - but yes - Google's only going to hit 100 links or so per page.

There's also the definite possibility (I can't prove it, but my observations over the past year or so seem to confirm it) that a relationship between various pages in established in greater detail than "which page links to which page" - but in "how each page links to each page".

If all your site's links are on one page, then according to you, all of your pages have the same amount of importance, focus, and scope as each other. If you link in as a pyramid, then your general terms come up for the general terms and as the search term gets more focused, Google knows which of your pages are the "focused" ones and will opt to display that page for the focused search rather than the broadly ranged one - even if keyword counts are equal between them.

None of this is really a science, yet, but it is something to start considering and thinking about as Google is dumping some big money into further research within this area.

G.

[Jill]

If I run a site with a high number of pages, I am better off organizing them so that my main page only links to my main section pages (which there are maybe six of). Then from each section page, I can link to the pages that relate to that section.

You should definitely do this.

And you should also then do what Tom suggests, as far as the sitemap goes.

Then you'll have a perfect site for your visitors AND the search engine spiders!

Jill