12 replies to this topic

[Pete L]

My site-map link is currently at the bottom of my main page. It is a short page, so it is easy to scroll down and find it; however, the search engines seem to find the article link, FAQ link, etc. in the upper part of the page but I have not seen the site-map listed yet as a reference.

I have these options I could do to move the site-map closer to the top.

1) Place the site-map hyperlink in my logo at the top as an SRC hyperlink. My logo is currently used as a home link. It also appears on other pages, but I could change it to a site-map link, from the main page. (I would have to enclose it in anchors to make it picked up, right?)

2) Place a site-map text-style hyperlink in the top banner (It is an 80% text banner, only a couple of small graphic images.)

3) Move the site-map up into the middle of the page.

Also, is there any disadvantage to having more than one site-map reference on the same page? (Especially, if I use the logo for the SRC link and keep the bottom text-style hyperlink to the site-map.)

Thanks,

[Scottie]

Your logo should link to your home page.

The site map link should be with the rest of your housekeeping links (contact us, about us, site map) in either the header or the footer of your page.

There's no need to link to it more than once.

Why would you want to put a site map link in the middle of the page? That doesn't really make any sense for people reading the page, or does it?

[Michael Martinez]

Before you do anything, search on the URL of the site map to see if it is indexed at all. Ideally, you don't want a site map to rank better for anything than any page on your site. Sometimes that happens, but it should not.

You can design a page layout so that a text link to your site map underneath your logo (which I agree should link to the home page) looks okay. One link to the site map per page should be sufficient.

Remember that a site map should be designed for your users, not for your search engines. It's a great bonus that search engines will crawl site maps, and so many of us recommend that people create site maps. But the primary function of the site map should be to help your visitors.

Make the link plainly visible. The rest usually follows.

And maybe this goes without saying, but the site map itself needs to be in straight HTML -- nothing fancy (no Flash, no graphics, no Javascript). You can embellish the page itself with some pictures if you want, but the site map itself needs to be a simple text link list with minimal formatting (style sheet formatting is perfectly fine).

[Pete L]

Hi Scottie and Michael,

I did an updated index search. The site-map is not indexed. One of the three links near the top of the site is not indexed, either. None of the three links in the footer are indexed.

The page is only around 35K, so, I cannot see why the SE's are not picking these links up. Also, that page probably only has 20-30 links on, presently, No flash or animation.

To Scottie, the choice of a middle placement is possible and would not look odd, at all; however, after doing the URL check and noting a link near the top was not indexed, I get the feeling that placement is not the whole problem.

To Scottie and Michael, I agree, it would be better to keep the logo as a home page link, even though it is on the homepage. It makes for good consistency throughout the site, as well.

Also, this is why that site-map link is so important:

1) The site is around 6,000 pages

2) Since it is a US directory, it is organized into 50 states. It uses a CGI redirect form to point to a selected state page within the site. These links are not listed on the main page, but they are listed on the site-map. The only way those state pages will get indexed is to have the site-map they are listed on indexed.

So, is this just a hit or miss situation on the part of the SE that indexed my site, or should I be looking for some other reason the site-map and a couple of other links did not get indexed? (Incidently, I did run a check with XENU and it did show the site-map as part of the link structure.

[Scottie]

Hey Pete-

If they aren't indexing the entire home page, they aren't going to get very far through your site map either.

It sounds like you need to work on your links more than worry about the placement of the site map link. Incoming links are what make your site "important" enough for a deep crawl.

[Pete L]

Hi Scottie,

Incoming links affect the depth of a crawl? Wow, there is something I didn't know. I've put together about 70 links over the past two weeks. It will take some time before they all show up. I have to make a couple of changes on my main page, so I might try a better placement for the site index along with the other changes.

It certainly is great to have a better understanding of SE's through people like yourself.

Thanks,

Pete

[Michael Martinez]

Hi Scottie,

Incoming links affect the depth of a crawl? Wow, there is something I didn't know. I've put together about 70 links over the past two weeks. It will take some time before they all show up. I have to make a couple of changes on my main page, so I might try a better placement for the site index along with the other changes.

Google engineers have told me in the past that they limit the number of pages their spiders will retrieve from a site in any typical crawl path, especially where dynamic URLs are concerned, to avoid dragging down server performance (but they occasionally achieve that anyway).

It does sound like you need more deep links (static, not dynamic). A static deep link to a dynamic page is just a hard-coded link that includes all the variables and stuff that would normally be produced by your dynamic script.

Your site map, of course, provides such links, but since the site map isn't being indexed, it's not being crawled.

You need to get a deep link to the site map, too.

I would have to see the site to make any really useful suggestions, but keep in mind that I'm traveling this week and not likely to have consistent access to the Internet.


ON EDIT: Well, I assume you're talking about the site in your signature. I looked at it. The link to the site map is all the way down near the bottom of the page in the source code. Your formatting displays fine in my browser, so I don't think you have an offset table element problem (which could prevent a crawler from indexing the link), but you might want to see what your page looks like in a few spider simulators. If they cannot identify the site map link, you have a technical problem.

[Pete L]

Thanks Michael. Have a great Thanksgiving.

I did try a couple of online spider simulators and all of them picked up the site-map hyperlink from the main page, so it is not a technical problem. However, the site map does get spidered last, in all cases.

As far as static vs dynamic links, Chris linked me to a site that I might be able to benefit from. It had a demo of a scrollable drop-down menu that includes the hyperlinks in anchors, so each of the state directories would get spidered. This will be great if I can code it to work on my site, as I would not have to rely on the site-map for these links to get indexed or I would not have to add an additional set of static hyperlinks to those states towards the bottom of my main page.

You need to get a deep link to the site map, too.

I am not sure what Michael meant by that. Does this mean to have links from my other pages point back to the site-map or to go out and get backlinks from other sites and point a few to my site map?

Anyway, lots to do, that's for sure. I think I will also start another discussion thread on the use of multiple site maps, as with over 6000 individual city hyperlinks, I will need something more than a single site page map when I'm ready to add in those links!

Thanks,

Pete

[Michael Martinez]

Any page that is being crawled, if it links directly to your site map so that the site map is crawled, is deep linking to the site map. It doesn't matter where the page is located.

I normally have a link to my site maps on all the pages of my sites (except for my forums -- I don't believe we link to the site map). That pretty much ensures that my site maps get found. I also put my site map links high on my pages, though. They are among the first content.

But you shouldn't have to put the site map link first. Plenty of sites get crawled, where their site map links are in the footers.

Still, you at least have good reason to rule out a technical glitch. The table code formats properly and the spider simulators extracted the site map link properly.
In looking at your site map, I see you embed all the links in table code. The code looks clean, but it's generally not advisable to format a site map page that much. The whole purpose of the site map is to provide a simple alternative to your regular navigation method.

But, again, you can easily validate the page by submitting it to some spider simulators and see if they extract the links correctly.

You can also check your .htaccess (if you have one) and make sure that it doesn't prevent spiders from deep-crawling your site.

Have you checked your server logs to see if the spiders actually grab the site map?

[Pete L]

Hi Michael,

I do have an .htaccess file, however, it is just an empty file:

File Type: ASCII text (0 KB) Located in the Public HTML (root) directory. Permission 644

I am unfamiliar with the purpose of this file. Possibly something should be added in it?

I just downloaded my server logs and wrote a quick program to check them. The logs show that the site-map was crawled by Google, MSN, Becom.bot, and PS0. BTW - the site has only been up a couple of weeks.

I will place some site-map links on some of the internal pages as well as move the sit-map on the main page up towards the top. I'll let you know how it all works out in the coming weeks.

Thanks,

Pete

Edited by Pete L, 23 November 2005 - 02:11 AM.

[Alan Perkins]

Pete

If you're talking about your findachiropractor site, the site map is indexed. You have a grand total of 11 pages indexed from findachiropractor.com, including the site map page.

[Pete L]

"Token Male Admin" - lol.

Hi Alan,

That's the first time I've seen it. I don't think I have checked Google since last week. BTW - I could not find it on Yahoo, but I did find the site-map indexed as well as a new state page, which showed up this morning on MSN.

Just growing pains, I guess.

Thanks Alan,

Pete

[Jill]

If you're interested in SEO, the first thing you need to know is that patience is the key.

Before you get all worked up about stuff regarding your site, you should probably figure that you need to wait 4 - 6 weeks. If you still have the same problem, that's when it's time to ask about it here.