7 replies to this topic

[markm]

Hi,

How is everyone doing on my favorite SEO forum?

This is a follow-up on a posting I originally posted in the dynamic site forum. I now realize it is not a dynamic site issue so I have moved the discussion here. I am looking for some feedback on my current theory.

To summarize: the company in question, due to acquisitions and mergers, had 4 domains and had a duplicate content situation. If you read the case study on searchengine watch, you can see that Marketleap was hired to clean up the situation. Apparently they did.

What isn't in the case study and wasn't presented at the SES conference, is that shortly after SES (12/2002)- Google started indexing the site and some time around March 2003 - the site was extremely well indexed with some 8K pages in the index. Great - right? :tooth:

Wrong - the pages started dropping out of the index again and within 3 months all the pages were out of the index. So here is my theory to what is going on.

Originally, after conferring with Google, they put up the 301 redirects on the old domains. The indexing did not seem to improve and Google still seemed to be indexing the old domain pages.

What they did next was take take down the 301 redirects and put up single pages with a "click here to go to our new domain" link - they did this some time around late January, 2003. A month later their pages started getting indexed so they concluded these these new pages linking to the single domain solved the problem. But as I mention, soon after getting indexed the pages all dropped out of the index again. Weird - huh?

Here is my theory and it might be a bit confusing.

My theory is that the 301 actually solved the problem. But since Google is so slow to react and drop the duplicate pages from the index - the cause and affect was misinterpreted.

Thus, when they put these new "click here" pages up and the pages started getting indexed, they thought it was because of those pages but really it was because the duplicate pages finally got dropped from the index.

Then, when the 'click here' pages got indexed, it triggered the old Google penalty from those old domains and the pages got de-indexed again.

So, what do you all think of this theory, does it hold water?

Thanks for your feedback for this perplexing situation!




Mark

[Jill]

Hey Mark,

I'm not sure about your theory, but what I'm wondering about is how you didn't about the whole history thing? What happened to MarketLeap? How did you get involved, and why didn't they tell you any of the background?

All of this weirdness makes me wonder what else the company may be hiding from you and/or that they hid from MarketLeap. Have you discussed this stuff with them?

I think that's what we need to know before even attempting to figure anything else out.

Jill

[markm]

Hi Jill,

Marketleap's contract simply expired. Probably because of reorgs and new people at the client.

There are a totally new set of people are at the company in question who were in fact ignorant of the history. The Christmas season coming up with a new marketing campaign, so there is a renewed focus on SE traffic.

The person at Marketleap has been quite helpful in providing me the history.

There is something else that happened just in the last month that has convinced me that the problem has nothing to do with the site itself.

Through a partnership with major retailer, they put up another domain. Dynamically generated from the same source code, this is essentially duplicate (with the exception of the partner's logo which is added in, along with their logo - nothing deceptive going on).

So I quickly set up robots.txt for exclusion (as soon as I found out about it). But I was too late and Google started indexing the new domain and some of the interior pages.

This just happened so it cannot be the cause of the pages getting dropped from their primary domain.

If the same source files are getting indexed under a different domain, then it has to be something specific to their domain. This leads me back to the links from their old domains. And the fact that they removed the 301s 3 months before they started getting de-indexed. The timeline looks like something this:

301's from old duplicate domains around 8/2002
12/2002 - still not getting indexed and duplicate pages still in Google index.
2/2003 - 301s taken down and 'click here' pages put up
3/2003 - 4/2003 Google indexes all the interior pages
6/2003 - Pages drop out of the index, again!

I am working very closely with the new people at the customer to uncover other missing information. The person in charge of the Website is new as well as the person in charge of marketing.

My initial thoughts were of course to put the 301 redirect up ASAP. I have been a bit hesitant since the conclusion from the original SEO work was that taking the 301's down solved the problem (and you can see from the timeline how they would reach that conclusion).

This is all quite confusing and a bit difficult to describe.

Mark

Edited by markm, 13 October 2003 - 10:14 AM.

[Alan Perkins]

The target has shifted considerably in the past - it's possible you will never get a focus on what happened now.

I notice you have added a robots.txt file in recent weeks - previously it redirected to "http://www.wine.com/...Population_ID=" which is a HTML page, which probably did not help.

However the robots.txt file you have published is invalid. I suggest you change it to:

# Allow all spider agents to crawl the site.
# Do not crawl the images or labels directory.
User-agent: *
Disallow: /images/
Disallow: /labels/

Don't use blank lines in robots.txt files except to separate User-agent records.

[markm]

The target has shifted considerably in the past - it's possible you will never get a focus on what happened now.

Yeah - I am trying filter through all the fog.

Thanks noticing the robots file. The webmaster had told me he put up a blank robots.txt file and I had not yet verified it.

Mark

[lots0]

My theory is that the 301 actually solved the problem. But since Google is so slow to react and drop the duplicate pages from the index - the cause and affect was misinterpreted.

Don't think it's due to google being slow, more like someone needed to update the ALL the links pointing to those pages...

If we continue to list an 'old' version of your site (i.e. we continue to list www.my123site.org despite the fact that your site now lives at www.my456site.org ) you need to update the links that are pointing to the sites. Since our robots jump from page to page via hyperlinks, someone must still be linking to the defunct page. Once others correct their links, we can too. Once your new site is live, you may wish to place a permanent redirect (using a "301" code in HTTP headers) on your old site to inform visitors and search engines that your site has moved.

http://www.google.co...sters/3.html#A2
(Bolding added by me)

[Jill]

Thanks, Lots0, that's an important point.

I guess I have some emailing to do in that case to update some old links to my old site I just recently redirectred.

Jill

[markm]

There were no links to the interior pages that were indexed. The links were to the old domain home pages which had a 301 redirect so the spiders could not continue to find those pages. In fact, they wouldn't find them anyway because they no longer existed in this case.

Nevertheless, they hung around in the Google index for quite a while.

Despite what Google says...defunct pages will stay in the index for some period of time (that is true from the day you take them down, they will still be in Google).

Now of course, I don't really know what happened, just what people tell me. Maybe they forgot to put up one of the 301s (5 total domains) or take down one of the duplicate sites.

It just does not make sense that taking down the 301s and putting up the 'click here' pages would solve the problem (and temporarily at that).

Thanks for your thoughts!

Mark