8 replies to this topic

[Sarah]

Hope you're all well?!

I work on a site with ~30,000 products which we want to de-index as a great deal of the content on those products is duplicated on other sites.

The products are uploaded from feeds, and a number of other sites also receive the feeds, creating not only duplicated content but also causing us to be flagged as an affiliate site (we know this is a.bad.thing. and are now fixing this up!). Our plan is to significantly reduce the products we offer on the site, write up some great unique content for these and allow engines back in to crawl them.

• Problem one: The URLs for the products do not contain a file that I can block engines from crawling eg: we have /the-big-blue-widget.aspx instead of /products/the-big-blue-widget.aspx
• Problem two: The products need to be inaccessible to engines, but our customers still need to be able to access the pages if they want to see more information. They can add to their basket from the listing in the category page

The sites developer/ former SEO has suggested that we use a 410 response to stop the engines from accessing and then de-index all the products until we are ready to allow the spiders back in. My immediate reaction is that this will not work, at.all. It's a trickery and when discovered will just monumentally screw the site over. Perhaps for eternity.

My own solution that I am still working through is to 301 all the "/the-big-blue-widget.aspx" pages to "/products/the-big-blue-widget.aspx", get those pages indexed and then block the /products/ file in the robots.txt and submit a de-index request.

I'd really appreciate your thoughts on using a 410 response like this!

Many thanks,
Sarah

[Sarah]

And as many of you are just waking up to your cornflakes, a quick update. I just META NOINDEX all products until they are properly written and then remove the tag each time I update something?! Win?

[chrishirst]

The robots meta element would work I would have thought, however why is being flagged as "affiliate site" a "bad thing"?

Personally I wouldn't even worry about it and let the search engines index the URLs and allow them to decide on what is "duplicate", that way as you do rewrite the content you are not going to have any problem getting the URLs indexed again.

[Jill]

I liked your first idea of redirecting to /products/

But also chris maybe right that it's unnecessary.

[Michael Martinez]

Definitely use the NoIndex solution, especially if you have a site search function that can still search those NoIndexed pages.

[Sarah]

Thanks Jill, Chris and Michael for your feedback - there are so many things that we're working on right now, and our glorious Bank Holiday interrupting that I've only just come back to this issue!

So we decided; the best solution we can get to which suits us all is that we will 410 all URLs which exist now, and reformat all the product URLs, and use a NOINDEX. The products will then have the NOINDEX removed as the content is written for our customers needs.

why is being flagged as "affiliate site" a "bad thing"?

It's not so much the affiliate tag, it's that we're getting our content out of a supplier feed, along with tens of other competitors, and we all have the same content, on the same products, and that's just not what we want to be doing. Getting the latest prices and latest products is great, but the same exact content as a number of other sites is not what we want to have. We may never let the engines index our product information again!

As much as I want to 301 old products to the category level pages straight away, there will be no 301's - I can hear faces being palmed! The 410 page will be custom built to include a site search - and we will be using analytics to understand what page was requested. If there is a strong pattern of a broken link being used multiple times, I can 301 that BUT I really want to try to disassociate as much as possible from duplicated product information.

I have my flak-jacket on...

[chrishirst]

Ok, then it's the "boilerplate" text that concerns you, which I can understand.

Many moons ago I worked on a drop shipping site I ended up writing code that used the "feed" description data unless there was a rewritten version in the local database for that product ID

[Michael Martinez]

Thanks Jill, Chris and Michael for your feedback - there are so many things that we're working on right now, and our glorious Bank Holiday interrupting that I've only just come back to this issue!

So we decided; the best solution we can get to which suits us all is that we will 410 all URLs which exist now, and reformat all the product URLs, and use a NOINDEX. The products will then have the NOINDEX removed as the content is written for our customers needs.

The 410 made no sense to me in your first post (it turns out it's not being treated any differently than a 404 in most cases). I would 301 all the old URLs to an intermediary or search page and then build out the new product URLs as you project (NoIndexing them until they have content).

I understand that a concern over duplicate content prompted this by why is there concern over duplicate content? Product listings are found on millions of Websites. You cannot trip across a brand Website that isn't reusing manufacturer/distributor text for at least *some* listings. Unless the site lost traffic, this kind of work only constitutes faux pre-emptive SEO.

Unless there is a problem to fix, fixing a problem on this scale can be extremely and unnecessarily expensive. There is no SEO case in "well, everyone is up in arms about duplicate content". If that's the only reason why you're doing this, I think you're burdening yourself for a very weak cause.

[Sarah]

It's definitely a loss of position and traffic since April's updates - I joined after Easter so I am on a little bit of a mop-up project, and we have now hit product level!

We're using this manufacturer listings on ALL the products, so we're essentially just a collection of duplicate information with our own category content. Looking at all of the data, this is one step we can take to improve things - especially for our end users - minimal landing page traffic to these pages with an almost untraceable low conversion rate seems to be the order of the day. The keyword research shows that due to the evolution of products within the industry, products are only searched for if they happen to be very long standing and unchangeable items such as a wrench. We can maintain this change, it's just how to handle it.

Another issue (of course) is the linking; most of the links built pre-April were (politely) not great - it's more now a case of slowly catchy monkey, with an approach which is to give people the tools and content to encourage a link for now. Many of the links that were in place were swiftly dropped by link sellers and spammers, presumably to try and hide their 2nd or 3rd income from engines. So I am glad I haven't got to deal with these any more, but also, wishing I still had those votes!

I love the intermediary search idea Michael - a very valuable tip I plan to put into use!