Matt Cutts On Big Daddy: Major Message To Seos Options

[Michael Martinez]

Matt Cutts' Indexing Timeline blog post has finally let the cat out of the bag, I think.

Here are some excerpts:

The sites that fit “no pages in Bigdaddy” criteria were sites where our algorithms had very low trust in the inlinks or the outlinks of that site. Examples that might cause that include excessive reciprocal links, linking to spammy neighborhoods on the web, or link buying/selling. The Bigdaddy update is independent of our supplemental results, so when Bigdaddy didn’t select pages from a site, that would expose more supplemental results for a site.

...

Linking to a free ringtones site, an SEO contest, and an Omega 3 fish oil site? I think I’ve found your problem. I’d think about the quality of your links if you’d prefer to have more pages crawled. As these indexing changes have rolled out, we’ve improving how we handle reciprocal link exchanges and link buying/selling.

...

This time, I’m seeing links to mortgages sites, credit card sites, and exercise equipment. I think this is covered by the same guidance as above; if you were getting crawled more before and you’re trading a bunch of reciprocal links, don’t be surprised if the new crawler has different crawl priorities and doesn’t crawl as much.

...

You didn’t remove your entire domain, but you removed all the important subdirectories. That self-removal just lapsed a few weeks ago. That said, your site also has very few links pointing to you. A few more relevant links would help us know to crawl more pages from your site.

...

it’s by design in Bigdaddy that we crawl somewhat more than we index in Bigdaddy

...

With Bigdaddy, it’s expected behavior that we’ll crawl some more pages than we index. That’s done so that we can improve our crawling and indexing over time, and it doesn’t mean that we don’t like your site.

...

typically the depth of the directory doesn’t make any difference for us; PageRank is a much larger factor. So without knowing your site, I’d look at trying to make sure that your site is using your PageRank well. A tree structure with a certain fanout at each level is usually a good way of doing it.

...

the supplemental results are typically refreshed less often than the main results. If your page is showing up as supplemental one day and then as a regular result the next, the most likely explanation is that your page is near the crawl fringe. When it’s in the main results, we’ll show that url. If we didn’t crawl the url to show in the main results, then you’ll often see an earlier version that we crawled in the supplemental results. Hope that helps explain things...Google is less likely to give those links as much weight now. That’s the simple explanation for why we don’t crawl you as deeply, in my opinion.

...

you’ve got some people who were relying on reciprocal linking or link buying complaining specifically that they’re not crawled as much.

...

we’ve been checking spam reports more closely lately. You ask “I’m wondering how you gain relevant links, in some sectors, without reciprocating, or paying? Do you believe that rivals would give you a free one way link, lol?” My answer is that trying to force your way up to the top of search engines is in many ways not working in the most efficient way. To the degree that search engines reflect reputation on the web, the best way to gather links is to offer services or information that attract visitors and links on your own. Things like blogs are a great way to attract links because you’re offering a look behind the curtain of whatever your subject is, for example.

...

Having a sitemap for your site should *never* hurt your domain. On the other hand, don’t expect that just listing a sitemap is enough to get a domain crawled. If no one ever links to your site, that makes Googlebot less likely to crawl your pages.

...

it’s true that if you had N backlinks and some fraction of those are considered lower quality, we’d crawl your site less than if all N were fantastic.

...

it’s not that reciprocal links are automatically bad. It’s more that many reciprocal links exist for the wrong reasons.

And I specifically asked Matt to explain the Supplemental Index, as much as he is allowed to. He replied with:

...personally I’d think of it as a fallback way that we can return results for specific queries where we might not have as many results in the main index.

Sorry we didn't get a more precise explanation, but I am pretty confident there is a lot of really interesting trade secretish stuff wrapped up in the Supplemental Index.

[qwerty]

Thanks for boiling it down. I really didn't want to read through all of the original post (IMG:http://www.highrankings.com/forum/style_emoticons/default/smile.gif)

Regarding the supplemental index, I noticed that a client's page wasn't ranking as well for a particular term than I'd expected it to, so I ran a search for the phrase just within the site earlier today. Of the top four results, one was the home page, and the other three were supplemental results for pages we deleted at least two months ago. The page I wanted was way down, but that's not the point. I'm just curious as to why these pages came up when 1) they don't exist and 2) they're not IMO as relevant to the phrase as the page I was looking for.

[projectphp]

Perhaps the solution to that Bob is to use http://services.google.com:8882/urlconsole...d&lastcmd=login and put those pages in the robots.txt to exclude them. That should get 'em punted pretty quickly!

[qwerty]

The server responds with a 404. That ought to be enough, but I guess I'm just being silly thinking that.

[Scottie]

OK...

When I read that, here's what I get:

Now who links to you CAN hurt you.

If there are a bunch of scrapers sites linking to me, it CAN hurt my "relative importance" because if those nasty, low-quality (and covered with AdSense Ads) sites link to me, it lowers the ratio of "good" to "bad" links.

So, how do I go about forcing scraper sites to remove their links to my site in order to keep my "good to bad" link ratio up where it ought to be?

This also gives credence to the "Google Bowling" theory that you can damage someone's site (particularly a newer site) by buying a bunch of crap links to their site.

While I applaud their efforts and I think they are honestly trying to deal with low quality links... a link identified as "low quality" simply should not count, it should not damage the importance of your site or be figured into any calculations about the importance of your site.

This gives a tool to competitors to damage sites and is a VERY VERY bad idea if it really has been implemented as Matt says.

As many people believe that Matt often tells things as Google wishes them to be rather than as they really are, this could simply be a scare tactic to keep people from flocking to the text link buyers. Let's hope it is, or else they give us a way to remove our links from sites that we never authorized in the first place.

Better yet, how about removing some of those low-quality Adsense scraper sites from the index so they can't hurt the innocent real sites who were unfortunate enough to be listed in the Google search result that was scraped and placed on the low-quality site? (IMG:http://www.highrankings.com/forum/style_emoticons/default/nah.gif)

[projectphp]

Or revoking their rights to use AdSense? Seems fair and reasonable to me!

[Scottie]

Let's face it, it's not like they don't know who they are paying...

[qwerty]

This also gives credence to the "Google Bowling" theory that you can damage someone's site (particularly a newer site) by buying a bunch of crap links to their site.

Ah, but if they buy those links, they're all going to have nofollow put on them, and then they won't be able to hurt you (IMG:http://www.highrankings.com/forum/style_emoticons/default/tongue.gif)

[Scottie]

Not if they "give a donation" in return for the links. Cause, you know, they aren't really buying links, they are donating to a good cause and the company appreciates that and places a few links out there to recommend such good, helpful people...

Define "buy" and I can give you 50 ways around it... (IMG:http://www.highrankings.com/forum/style_emoticons/default/shades.gif)

[incrediblehelp]

Great post Michael (and Matt). Answers a lot of questions for me and many newbies I am sure.

[Andrew Gates]

anyone now can hurt my index, this is answer to my question - where are my pages?

"it’s by design in Bigdaddy that we crawl somewhat more than we index in Bigdaddy"

I have only 0.05% of all pages indexed in google. Though google crawler steal about 2-3Gb every week and do not index those pages.

I actually have about 200-300 backlinks , but when I search "mywebsite.com" in google, it displays 50 000 pages and 98% of those links just adsense sites who steal content from my site.

[making-it-big]

As I posted on Matt's blog I just don't think junk links will harm a site. People are reading into and jumping a bit too much on this.

What I do see happening is that sites with natural (think scrapers here) or unnatural (paid/recip) links that make up the BULK of their links will more than likely see adverse effects. Not directly because of those links rather those links no longer play a huge role in PR and ranking. So what happens is that those sites who experienced good rankings and indexing because of such links are now seeing a "correction". It is not a penalty although it may seem like it. Just a correction. Of course there are some sites that are complete spammers who still enjoy good rankings. Eventually their time will come.

Pretty much seems like reinforces what Jill and others have been preaching about for years, QUALITY; Quality in content, quality in business, quality in promotion and links, etc. This is where the game should be played. Not who can out SEO each other but who can out quality each other. Who can be the best in their business.

[Andrew Gates]

  It is not a penalty although it may seem like it.  Just a correction. 

Just a correction? remove most of the pages from index? It is penalty.

[making-it-big]

Explain how it is a penalty.

[Andrew Gates]

Explain how it is a penalty.

You buy links, you ranking is increased, then google devaluates links and your ranking is going down - it is correction.
When you do nothing and you lose almost all indexed pages - it is penalty.