95% Drop In Number Of Pages Indexed, Recurring issue and spikes with google Options

[mluggy]

We run a community-generated-content website around textual summaries and short articles written by users.

For the 4th time, Google had dropped 95% of our pages although it constantly spider our site through normal crawling sessions and sitemaps.

The following is repeated monthly:
1) Number of pages start raising in 200-500 per day
2) Number of pages reaches around 20,000
3) All of a sudden, a sharp drop to 900 pages
4) Daily drop of 40-50 pages per day up to around 600 pages

More info:
1) New unique articles are written in a rate of 600 per day.
2) Site is alive for 18 months
3) Site is indexed very well by other search engines
4) Site is ranked well for the remaining pages in Google, even for highly-competitive phrases

Anyone had experienced this?
Thank you!

Michael

[Jill]

600 new pages a day? Doesn't sound like that's a natural phenomenon to me, and I imagine it doesn't to Google either.

[mluggy]

600 new pages a day? Doesn't sound like that's a natural phenomenon to me, and I imagine it doesn't to Google either.

Am I allowed to post links?


[No. Please see [url=http://www.highrankings.com/forum/index.php?act=boardrules]Forum Rules[/url]. - Jill]

This post has been edited by Jill: Jul 23 2006, 02:06 PM

[roxyyo]

1,000 monkeys at 1,000 typewriters will build the greatest website ever known. Apparently Google doesn't agree?

[mluggy]

Jill - I wish I could post the link so this discusion can become more relevent. I really don't need the href, just to get more solid ideas.

Maybe the following will be approved and will clue you in the site in question. Absolutly weird behaviour.

shvoong.com

This post has been edited by Jill: Jul 25 2006, 06:22 PM

[Comtrad]

It may be because there is a lot of crap articles. I think you should monitor each article that is posted, and not approve each one. Then maybe google won't find it as duplicate content and delist it.

[mluggy]

Thanks comtrad,

Don't you think it's odd that Google drops down consistently back to 800 pages and don't leave out at least some of our 1000s summaries?

Michael.

[Comtrad]

No, because Google is crazy.

[mluggy]

I think otherwise. Crazyness would've bring us to completely different page counts.

Something is definitly wrong (and consistent) with what we're doing, otherwise I cannot explain why Google drops down everytime to around 800. It was never higher or lower for a period of more then a week.

[Comtrad]

Has it been doing this for long? Did you change anything you did when you started seeing the drops? It just seems like a article farm, and maybe Google is finally doing something about these types of sites.

[mluggy]

It has done so about 6-7 times already. going up all the way to 19,000 indexed webpages, then dropping to 800.

I wouldn't call that an article farm. User content is not shared with other websites. They are receiving royalties to keep things genuine.

Any idea how I can raise this to Matt Cutts? I know he wants to hear about unusual issues. Maybe he can help us understand what's wrong here.

[Michael Martinez]

Looks to me like your server response time is very slow. Google may not be retrieving whole pages frequently enough to keep you in the index.

And there are quite a few inbound links that promote your site as an article farm (they don't call it an article farm, but that is how they are promoting it). You have some good quality links and a lot of low-quality links. I don't think linkage is your problem, but if you have an issue with article quality, it's probably because of the way people are talking about your site across the Internet. They are sending wouldbe writers by the droves to submit articles.

[mluggy]

Thanks Michael,

Your clue about slow server response is being checked, although our chinese subdomain, for example, which is hosted on the same server was indexed quite well with over 20K count.

As for external linking, we're in the process of removing these crappy mentionings but webmasters are slow to response. I don't think these can really harm us or were you speaking in general of the way users preceive us?

Michael.

[mluggy]

Just reminded myself I took some days earlier last month to have a serious look at the IIS logs.

I've seen hunderds of thousands of good (status=200) connections to all of our subdomains (www included). All were served without a hitch - so connectivity and speed issues doesn't sound like a reason.

[Michael Martinez]

As for external linking, we're in the process of removing these crappy mentionings but webmasters are slow to response. I don't think these can really harm us or were you speaking in general of the way users preceive us?

I just mean that people who write articles are running around the Web telling other people who write articles that your site is a resource they can use for their own purposes. I have no idea of how much impact that has had on your traffic or the quality of your site.

Generally speaking, when pages drop out of the index and then get back in and then drop out again in a cyclic fashion, the most usual culprits are (in no particular order):

1) Technical issues such as: server availability, misconfigured or corrupted DNS tables that influence the crawlers, improperly configured or unauthorized redirection, spider blocking .htaccess or robots.txt or meta tag content, etc.

2) Poor inbound linkage (especially in the wake of the Big Daddy overhaul, which resulted in changes in linking priorities)

3) Poor internal navigational (persistent, static text links placed prominently on every page)


Now, there may be other issues, but these are the three most common types of causes I see discussed in various forums. My initial thought, when I first looked at this thread, was that you have poor inbound linkage. But some of your links are from some pretty good sources. I would be doubtful of any harm coming from inbound links.

It may still be that you need more inbound links. Good links cannot hurt you. But it could also be your own internal linkage is an issue. I didn't have time to really look at it (especially since I couldn't get to any pages but your home page).