7 replies to this topic

[Andrew Gates]

I'm receiving this message in Google webmaster tool every 2-3 months
"Googlebot found an extremely high number of URLs on your site"
We have many product pages, some pages are blocked by robots.txt file.

The problem is that Google webmaster tool shows URLs in example ("Here's a list of sample URLs with potential problems") which are blocked by robots.txt

Do you know why they do it? Thanks.

[Jill]

Are you positive they're blocked via robots.txt?

If so, then I wouldn't worry about it. But I would double and triple check that you've done that correctly.

[Michael Martinez]

robots.txt only tells Googlebot not to fetch those pages. It FINDS them through links on other pages, or through an XML sitemap. Make sure your XML sitemap doesn't include pages you're telling Googlebot and/or Bingbot not to crawl. It's okay to have navigational links that point to blocked pages.

[OldWelshGuy]

I was about to post the same as MM. make sure you are not generating a list of pages someplace on your site. Many shopping carts will do this. Seems like google is gearing up for a deep crawl every couple of months, and then generating this error.

Are you by any chance submitting these urls to google products? that will do it for you.

[Andrew Gates]

Are you positive they're blocked via robots.txt?

If so, then I wouldn't worry about it. But I would double and triple check that you've done that correctly.

I checked every page in webmaster tool (Crawler access) which shows is it blocked or not. It says that pages are blocked.
This is why I'm surprised.

[Andrew Gates]

robots.txt only tells Googlebot not to fetch those pages. It FINDS them through links on other pages, or through an XML sitemap. Make sure your XML sitemap doesn't include pages you're telling Googlebot and/or Bingbot not to crawl. It's okay to have navigational links that point to blocked pages.

We don't have such links in xml map, but there are many links from other pages on our web site. Though all links to those pages are "nofollow" links.
Why they still keep such links in database?

[Michael Martinez]

I have seen Google keep links for years, but maybe that's only because it still finds them in rare, hard-to-find, obscure documents.

You should be careful about using "Nofollow" so extensively across your Website, however. It is essentially spreading your PageRank across the Web.

[chrishirst]

but there are many links from other pages on our web site. Though all links to those pages are "nofollow" links.

Why???