2 replies to this topic

[firedog]

Can't get a corporate site listed (spidered) in Yahoo index from free submissions.

The site exists in the index due to its listing in the Yahoo Web Directory; no page other than the home page is listed

The site was previously listed with multiple pages - no spam techniques of SEO have been added to site, but all references to site other than the listing from directory are there.

A few month ago the site pages disappeared and have not come back.

It occured to me that perhaps Yahoo purposely dropped the site B/c they are filtering out company sites that have a stock exchange listed company from the free submissions. The assumption being that such corporate sites will be able to pay for paid inclusion.

What do you think?

[OldWelshGuy]

I think you need to make a tin foil hat

There are many reasons why yahoo and other spiders do this, most of which we will never know. The people with small sites cry conspiracy to bow to the large corporations, while the large corporations cry conspiracy against them to force them to buy advertising.

There are new site, old sites, large sites, small sites, uk sites, US sites, and a host more, all suffering the same fate. Losing pages, and struggling to get spidered. It is just how things are IMO, no conspiracy.

[Tom Philo]

Any SE must exist to fit a need of the population using it. Unless a labor of love, then it must also earn money in order to stay in existence. If it does not fulfill that need then people will go elsewhere to find that need. Soon that SE will cease to exist since they are not earning enough income to sustain the SE.

With 6 billion pages (and the "hidden web") out there, indexing everything is now impossible. Each SE has to decide how often and IF it should include pages that it finds. Each page lives and dies on the value the SE thinks that page has to the users if included - and if it will be missed if not.

If 50,000 pages exist on a subject, then including a new one may not be of value. How a SE determines if a new page should be included and then what page should be dropped is the arcane art that a SE program must come up with. They may play with science but it still is more of an art to program the science to do SE ranking.

How much resouces a SE puts to bots, to storage, etc is really an economic one. To each page owner every page needs to be included, to the SE collective as a whole missing a few billion pages, while still allowing people to find information being searched for it , is not important since the shear number in the collective allows most every question posed to be found.

I think that, over time of 4 to 7 years, each SE will program in limits on the number of pages of similar subjects that it includes in the index. Then, if a new page with a similar overall subject comes in, it will be weighed against the existing and if better it comes in and the worse one goes out.

Thus, one would hope, only quality pages will eventually "float" to the SERP that people see.

I have a site out there that only the first page (out of 7) has been indexed by Google or Yahoo. Google has at least come back and updated, but Yahoo has not come back for two months so it still sees the intital 5 pages I put out onto it in August.

I think the days are gone where a site is put up on Monday, and by the next week it is top of the heap. Too many pages out there. It is now a very long term 3 to 6 months wait to see how things go in SEs.

Instant gradification is gone.