Did You Know That Google Supposedly Doesn't Crawl Nofollow'd L, That's what Matt C said at TW Options

[Jill]

Over at Threadwatch, there was a post about how Google is nofollowing links to some videos, but not to their partners' videos. That's interesting and all, but the part I found even more interesting was where Matt Cutts came in to explain things and wrote:

I had a half-hour meeting with a Google Video person last week (before this situation happened), just to see what they wanted to do with nofollow. It turns out that they understand the semantics very well, and the nofollow's on some of their internal links are deliberate power-user choices on their part. *For example, imagine if you had a site with articles in both printer-friendly and web-friendly format. If you wanted to sculpt where Googlebot was going to prevent Googlebot from crawling the printer-friendly pages, you might use nofollow on internal links then. So I went into the meeting wanting to indicate "Hey, a few people are curious about the nofollow's on internal links; is that what you intended?" and I walked out of the meeting very comfortable that the Video folks understood the nofollow semantics and were using it exactly how they wanted to.

*emphasis mine

Really? Nofollow on your links is supposed to stop Google from crawling the links? Well, that sure was news to me. So while everyone else was thanking Matt profusely for explaining why they were doing what they were doing, I was left scratching my head and said:

You would? Isn't that what other tags are for?

Matt, when did the nofollow attribute on links stop Google from crawling those links?

And he responded:

Jill, the meta-tag level nofollow works at a page level. You can do other ways to prevent Googlebot from crawling a specific link (e.g. link through a redirect blocked by robots.txt), but nofollow is one of the easiest ways to control crawling at a link granularity level.

Umm...ok.

So, has anyone tested whether Google actually does crawl those nofollow links? I know it was never intended to stop crawling and I was shocked he is now saying that's what it does. It was (I thought) created so that people could mark links that they didn't control, so that Google knew that they were not a "vote" for the site being linked to. There was no mention of Google not actually crawling the link.

So have thinks changed? Does Google really not crawl them? Did he just make that up to get out of a sticky situation?

And if they're not crawling them, when did that start? Anyone notice?

[piskie]

From Jills Maat quote, note the:
"and were using it exactly how they wanted to"

How they wanted to implies to me something other than designated meaning.

[mcanerin]

It makes sense that they would interprete it that way - nofollow would seem to mean "don't follow/crawl" to the average webmaster. I also agree that it's more useful that way - robots.txt for directories, robots meta for pages, and nofollow for links.

But, like Jill, I thought I remembered VERY distinctly that this wasn't how Google was going to treat them! I went looking, but all I found was the original announcment, which implies exactly this:

http://googleblog.blogspot.com/2005/01/pre...mment-spam.html

And Danny's article on the subject said the same:

http://blog.searchenginewatch.com/blog/050118-204728

"If Google sees nofollow as part of a link, it will:

NOT follow through to that page.
NOT count the link in calculating PageRank link popularity scores.
NOT count the anchor text in determining what terms the page being linked to is relevant for. "

Words are Danny's, Bold is mine.

I'm not sure where I got the idea that Google would follow them but not count them, but it's not supported by anything I just dug up. Wierd.

Ian

[Jill]

So does anyone know from real tests if that's indeed how G handles nofollow?

[qwerty]

Just anecdotal. I see Yahoo reporting pages as backlinks that Google doesn't (such as comments on Matt's blog), but of course Google isn't reporting all of the links they know about anyway.

[jehochman]

This seems to be a sloppy way to control Googlebot. If somebody else creates his own link to the target, Googlebot may still find and index the target.

[Scottie]

When the attribute first came out, Matt said they currently would not follow a nofollow link but reserved the right to change how they treated them.

So, he did say at that time they wouldn't follow the link but didn't commit to always not following a nofollow link and he emphasized that it was meant to tell the search engines that you don't vouch for that link.

Since I don't use the attribute, I haven't tested it.

[Michael Martinez]

All Google said originally about 'nofollow' was that "From now on, when Google sees the attribute (rel="nofollow") on hyperlinks, those links won't get any credit when we rank websites in our search results." It was Danny Sullivan who said they actually would not crawl the links.

Last December, at the Q&A on Links, it was reported that both Yahoo! and Google will crawl 'nofollowed' links "for discovery" but not for link popularity (the reference is inspecific about who actually made the claim). On December 7, Rand Fishkin wrote at SEOmoz

...Adam Lasnik & Tim Converse both said on a panel today at SES Chicago that they don't "obey" the no following part of nofollow. Both agreed that a more accurate name woudl actually be "nolinkjuice" - it doesn't give anchor text benefit or link pop benefit. They still get crawled as a discovery method.

It is possible that Adam Lasnik may be responsible for the general confusion on how Google handles 'nofollow'. For most of his posts about 'nofollow' on his own blog, Matt has not said whether Google would crawl it or not. A few people have claimed they've seen Google crawl through 'nofollowed' links.

However, in July 2006 Matt did say "At a link level, you can add a nofollow tag on the granularity of individual links to prevent Googlebot from crawling individual links (you could also make the link redirect through a page that is forbidden by robots.txt). Bear in mind that if other pages link to a url, Googlebot may find the url through those other paths."

So he has made this statement before, but apparently most of us have missed it. I wonder what Adam actually said at SES Chicago.

This post has been edited by Michael Martinez: Jan 31 2007, 12:43 PM

[MaKa]

I've setup a quick test today. Remind me in a couple of weeks and I'll check the results.

[Erik]

Jill,

Regardless of the outcome, thanks for reigniting this issue. It's been a real pet peeve of mine.

[Alan Perkins]

Hmm, I'm confused.

First, why did Matt say this:

Jill, the meta-tag level nofollow works at a page level.

Nobody was talking about the meta-tag level nofollow.

The meta-tag-level nofollow and link-level nofollow are both really unsound mechanisms for stopping robots crawling a page. To say "nofollow is one of the easiest ways to control crawling at a link granularity level" is poor advice, especially coming from Matt. If you want to reliably control crawling, you must use robots.txt. If you want to control indexing, you can use robots.txt or the robots meta tag.

"Nofollow" in the meta tag is almost useless. I never use it.

"Nofollow" in the link simply means "Please assign no ranking weight to this link". It does not control crawling, certainly not consistently across every engine.

[OldWelshGuy]

To me that is exactly how it should work If you are telling the spider 'don't follow' then they should obey it. That said, Spiders don't actually follow links anyhow do they, but Maybe we can run a test on this. I will set up a page, and a few of us can link from our blogs to it using the no follow atribute. That should sort it shouldn't it? If we have another page with a normal link to it on the same posts, then technically if IT gets indexed and the no followed one does not, we have our answer.

[qwerty]

OK, count me in.

[Jill]

"Nofollow" in the link simply means "Please assign no ranking weight to this link". It does not control crawling, certainly not consistently across every engine.

Exactly. But apparently Matt is saying otherwise.

[Randy]

Count me in too OWG. Just let me know where to link to with a nofollow.