Are you a Google Analytics enthusiast?
More SEO Content
Does Yahoo Find Your Site Maps?
Posted 08 August 2006 - 10:58 AM
Google and MSN find it no problem. Any ideas?
Has anyone had the same problem? What do you do?
Posted 08 August 2006 - 01:24 PM
A question, since you have some real examples to look at.
Do these site maps deliver a 200 OK server response as one would expect when you go directly to the page? And if it does, are these sites perhaps configured to send people to the site map file if they get a 404 Not Found response from some other non-existent page?
Yes, I'm shooting in the dark, but I can see how both might cause a bit of confusion for a spider.
Posted 08 August 2006 - 01:52 PM
To see what I am talking about, you can look at the official website for the "Cherokee North Carolina" indians. (The agency that developed the site is a firm believer in large flash.)
But even so, why would Yahoo slurp find the pages of the site, but not follow the text "site map" link at the bottom of the page?
They see it in the cached page. But they do not come to spider it. I have a few other sites doing the same thing.
Posted 08 August 2006 - 05:39 PM
Posted 08 August 2006 - 06:15 PM
How do you know this?
Are there actually pages of your site that are only being linked to from the sitemap? And if so, why?
Your navigation should be such that the engines can get to all the pages through it.
Posted 09 August 2006 - 08:40 AM
First, the Cherokee Indian site. This site is the official site for the eastern band of cherokee indians in North Carolina. If you search for it, you'll know when you see it. It's mostly red.
There is a text link at the bottom of every page of the site to a site map. If you do a site: search in Yahoo, you will find most all of the pages in Yahoo's index. But the site map is not in there. It's very frustrating. As most of the pages are in the index, it's not too big of a deal. Just strange.
(BTW, I did not make this site. It's not up for review. I'm working with the agency to change many of the other things that are obviously hindering our seo)
The other site is "Better Home Connections". It comes up #1 (as of now) for that search phrase in Google. They sell homes in Asheville, NC. For this page, there is a text link to a site map at the bottom of the page as well. Google found it and spidered all the pages no problem. MSN did too. Yahoo has only seen the home page.
This one is a little more frustrating because the site has been live for a month or two and Yahoo will not go past the home page.
I know there are work arounds for this and I am moving forward with them.
But the question remains: Why won't Yahoo crawl my site maps?
(These sites were made by two different developers.)
Posted 09 August 2006 - 09:38 AM
Many months ago I did a test on a few sites that had a lot of pages with a low index ratio (under 50% of pages were indexed) in which I placed a link to the XML sitemap - the same one I had submitted to what was then called Google Sitemaps. After one month, and many visits by both MSN and Yahoo, neither search engine indexed any more pages than they had in the first place. The link to our xml sitemap was prominent on the page and easy to crawl. I figured they would treat it like a blog or any other XML feed and crawl the links, but I guess not.
Posted 09 August 2006 - 10:07 AM
You'd be better off sticking them in your Signature temporarily (the engines don't see those) or putting them in your profile and pointing people to that.
I don't have any free time right now to look at either site to see if I can spot some technical problem, but as a general rule Yahoo's Slurp is much slower on the uptake. They're also considerably more picky about including fluff content in their index, even though they may know about it. One thing you would want to be able to find out, and can't if you don't have access to the logs, is if Slurp has visited these site map pages.
That's what you need to know first. Has Slurp actually visited those pages. If they have but the pages don't appear in the index it means that Yahoo! has simply chosen not to list those pages for one reason or another. Nothing at all strange in that.
Posted 09 August 2006 - 10:26 AM
Your comment is very interesting. That's a good question. I will try to look into if Slurp has visited and just chooses not to index. It seems strange to me that they would not index a site map page.
I thought search engines wanted us to make site maps.
Posted 09 August 2006 - 07:12 PM
Is it my imagination that Yahoo is not spidering as well?
Posted 10 August 2006 - 06:25 PM
Posted 10 August 2006 - 06:58 PM
The thing is i only have 4 of my pages index by yahoo and don't show up in search result.
So my qustion is how do i tell when and if yahoo is crawling my site and chooseing not to index pages.
Posted 11 August 2006 - 03:36 PM
If Yahoo has visited your page, but the page doesn't appear in their index after a decent interval of time (I'd give it at least a few weeks, maybe even a month or more just to be safe), I would think you could make an assumption that they had decided to not index it for some reason.
Of course, you can never know for sure short of somebody from Yahoo telling you. And, no, I don't have any inside contacts to find out, sorry. I guess you could try emailing them, but I wouldn't hold my breath waiting for a useful reply.
Posted 25 September 2006 - 08:32 AM
I don't know if this will help, but you can setup a free account at the Yahoo Site Explorer and submit your sites and a feed for each site.
Interesting thing is that according to the text on their submission page, if you give them the url, they will crawl the web site. However, I found out this morning that they only indexed 4 pages from my site. So I created a text file in the correct format and submitted it as a feed a few minutes ago. I'll let you know what happens.
QUESTION slightly off this topic, but related: I made a bunch of modifications to my home page specifically for Yahoo, and then resubmitted my site. They indexed that page the very next day - yesterday.
My ranking did improve quite a bit, but it still has a long way to go to get into the top 10. Do I assume that after reindexing, the page is ranked approximately as high as it's going to go until I make more changes and re-submit? Or is it possible for the ranking to change drastically (up or down within in a week or so) without doing any more optimizing?
Posted 25 September 2006 - 09:35 AM
The SEs are working on their spidering and ranking algos all the time, too, and changes in those can change how your pages show up.
On top of that, the SEs run multiple datacenters in order to serve the huge volume of search requests they get, and those datacenters are rarely (as in, never ) totally in synch with each other. You can potentially connect to different datacenters at any time; there's no guarantee that just because your last query ran at DC#1 your next one won't run at DC#25 (or whatever).
So your rankings can appear to fluctuate wildly up and down from one day to the next (heck, sometimes one minute to the next) even if you haven't done anything at all. And they're subject to change at any time, often for reasons that are totally unrelated to your pages or anything you've done to them.
Which means, basically, your question is unanswerable. It may be that your page has already moved up as much as it can in its present state. It may be that it's already ranking even higher (or has already dropped back down) in some datacenters and you just haven't been lucky enough to hit one of them in your queries. It's impossible to tell.
If you make a change and your rankings go up, you can probably safely assume the change had something to do with it, but you can't know. (I guess you could try removing the change and seeing if your rankings go back to approximately where they were before, then put it back in and see if they go back up... but that seems like a lot of work for not much benefit. )
If you can still think of things that would make your page better for your visitors, then by all means do them. If, on the other hand, you're reduced to tweaking minor things in an effort to eke the last molecule of SE benefit out of your pages, I'd say it's time to move on to something else instead. The point is, I wouldn't worry much about what the SE rankings say, because they may well be different the next time you look. Do what's best for ALL your visitors (human and spider), make sure you haven't put any technical blockades in anybody's way, then move on.
The cool thing about the web is that nothing is really permanent. If the changes you make don't seem to be working the way you'd like, you can always change things back.
(BTW, you probably don't need to manually submit at all. If your pages have good links pointing at them, the SEs will find them that way, and likely long before they work through the backlog of submission requests they've got. Any time they appear to index a page rapidly following a manual submission, it's likely that page was scheduled to be spidered anyway. The best, most reliable way to get your changes noticed more quickly is to have high quality links pointing at your site pages.)
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users