New York Times And "approved Cloaking" Options

[qwerty]

We had a brief discussion a few months ago in which I brought up the somewhat problematic fact that National Public Radio provides the text of their programs to Googlebot but charges people for access to those pages. Basically, this is what many have called "approved cloaking," in spite of Google's public statement,

Don't deceive your users or present different content to search engines than you display to users, which is commonly referred to as "cloaking."

Today, Search Engine Watch ran an article on efforts at the New York Times to be more friendly to the search engines, and guess what that involves:

To access this ["Times Select"] content, you must be a subscriber to the print edition of the Times, or pay a $49.95 annual subscription fee.

Despite this restriction for human users, Marshall says that both Google and Yahoo have been allowed to fully index the premium content, and will display results for matching queries (for example, a search for popular Times Op-Ed columnist Thomas Friedman in Google and Yahoo returns hundreds of results).

Click through on many of these links from both engines, however, and you won't see the content indexed by the search engines. Rather, the Times web site detects that the user agent is a browser, and serves up a shorter abstract page with a login form to access the premium content.

Isn't this cloaking—serving different pages to a search engine and an individual web browser? Yes, it is.

I still have issues with this sort of thing, but I imagine a lot of people see it as convenient.

[Scottie]

Nice of G to make exceptions for the big boys. Wonder if they get a cut of new subscriptions from the Times?

[qwerty]

I don't recall if this was mentioned in the article or not, but I've noticed that if you search on the title of one of their Op-Ed columns, the result doesn't include a link to a cached copy.

[Jill]

I still have issues with this sort of thing, but I imagine a lot of people see it as convenient.

Why do you have issues with it?

It's their search engine, they can index or not index what they want, no?

[projectphp]

I think this is fine, but they really should label the content, so that users aren't annoyed.

[Jill]

I should add...

"Go Marshall!"

It's really cool what Marshall Simmonds is doing there. He's a long time SEO, speaker on the SES circuit, etc. Very neat that he's making SEO mainstream at the NY Times!

[arteworks]

I'm not seeing the cloaking issue here. NYT is not presenting different content to the search engines than to humans. Humans can have the content if they pay for it? There is no deception involved here.

I know, I know, I was the one ranting about intent in the other thread, but I think this is different. As long as the content G is able to crawl is the same as the content premium members see, seems fine to me.

[qwerty]

Cloaking is showing one thing to search engines and something else to people. If I go to the page that's in the top position of the SERP I linked to, I'm going to see the one sentence in the snippet ("As long as I see Iraqis ready to take a stand for their country, I think we have to stand with them.') followed by "To continue reading this article, you must be a subscriber to TimesSelect. Log in now."

In that particular case, I can't complain. The SERP is showing me a line from the page, and that line is on the page. But if the information in the SEW article is correct, then any line from that column could have shown up in the SERP, and it would still take me to a page with just those lines.

That would be wrong, in my opinion. If I see something in the SERP, I expect to see it when I click through to the page, rather than a message that I can read the page if I'm willing to pay for it.

I applied for a full-time SEO job years ago with a company that published "premium content". The job of the SEO was in part to create abstracts of the articles that would be available to anyone -- readers or spiders. These would need to be optimized to gain traffic from organic searches. Not an easy trick for just a paragraph or so. Using this method, this company wouldn't really need an SEO. They'd just let googlebot read the article, get searchers interested, and send them to the "buy now" page.

That really seems dishonest to me. At the very least they could do what Northern Light used to: label it as premium content requiring a subscription, so we know not to click through to it if we have no intention of paying.

[projectphp]

I ahgree with that absolutely Bob. I want to know that I aint getting what I searched for without paying.

I am all for more content being indexed, but I hate teh thought i won;t know going in. I think such a move may actually make the web less usable, as we wills ee all manner of people doing the same, and SERPs filled with subscription content.

IMHO, this is an issue that needs careful consideration. Indexing the invisible web is a great idea, but not if that creates a totally invisible web.

[train99]

If you search Google with the NEWS tab, Google tells you when it requires a subscription:

See note by South African star on this search result:
http://news.google.com/news?ie=UTF-8&oe=ut...tnG=Search+News


Google should be telling you in all cases because it lowers the usabilty of the results to hit a subscription wall. Maybe they don't because there is public content and subscription content and the robots aren't smart enought to look for the line between them?

Terry

PS When I click on the link to the Star I get the full story. Maybe it's regional?

[Sarah]

So we're designing, developing and writing content for websites that will be accessible and wonderful all singing all dancing for our end user. Because surely that's what it is all about? Along comes a Big Boy and their subscription only content is indexed, shown to be relevant to the search term yet you have to subscribe to view it? And you don't know until you get there. Not end user friendly then..... (IMG:http://www.highrankings.com/forum/style_emoticons/default/censored.gif)

I have a site which has not had it's content pages indexed because of a login/register script on the first page. So I know unless someone logs in they don't get the content, so it's not indexed and we're sorting it...

I'm with Bob & php on this.

[Jill]

That would be wrong, in my opinion. If I see something in the SERP, I expect to see it when I click through to the page, rather than a message that I can read the page if I'm willing to pay for it.

I agree. And there may be some deception to the end user.

But just like trusted feed, since the engine has approved it, it's obviously not search engine spam or anything like that.

[mal4mac]

I recently read an excellent article in the NY Times called "Scan This Book!" by Kevin Kelly. It's about the possibility of placing every book in the world within the reach of everyone, for free. It was long, so I book-marked it. When I opened it again the NY times tried to charge me for it - very ironic Mr Editor!

Also, I was not warned that this would happen. This is very devious.

I found a back way in by searching for the article. I hope "The Observer" doesn't start using these tricks - I'll stop buying it if it does.

[qwerty]

But just like trusted feed, since the engine has approved it, it's obviously not search engine spam or anything like that.

Whether or not this is spam isn't really the issue, IMO. If spam is an attempt to deceive a search engine, this isn't spam. If spam is a successful attempt to deceive a search engine, this isn't spam. If spam is a successful attempt to deceive a search engine that gets caught this isn't spam. It basically is trusted feed, but of course Google doesn't have a trusted feed program, and I believe they've stated on a number of occasions that they never will.

It's been argued that "approved cloaking" is an oxymoron -- if it's approved, it can't be cloaking. But it certainly fits every sensible definition of cloaking I've ever seen... and it's apparently approved.

At the very least, I wish Google would change their guidelines about cloaking to stress that you shouldn't do it without their approval, and it would be nice if they'd come out and list the sites for which they've decided it's ok. What if somebody figures out what the Times is doing, sees that it's apparently ok, and replicates it? They're going to get their site banned.

If a kid stays out after their curfew and gets grounded, but he says that he thought it would be ok because nobody ever complains when his older sister ignores her curfew, I think Big Daddy has some explaining to do.

[mcanerin]

This is also an issue with sites intended for adults - porn, gambling, etc.

Responsible site owners would want to put up some sort of age verification method, but since a search engine can deliver somone to any page in a site (not just the home page) the age verification has to be available from anywhere.

I don't see this as presenting different data - the page is still there, and as long as you prove you are entitled to it, you can have it.

I do agree that the fact that it's subscription content should not only be made clear, it should be made a LOT clearer - it's really annoying to be challenged when I'm not expecting it.

This happens to me all the time with that goddam webmasterworld site. I hate it, hate it, hate it. Let me know CLEARLY I need a login, dammit! (IMG:http://www.highrankings.com/forum/style_emoticons/default/ranting.gif)

Ian