Jump to content

  • Log in with Facebook Log in with Twitter Log In with Google      Sign In   
  • Create Account

Subscribe to HRA Now!

 



Are you a Google Analytics enthusiast?

Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE! 

 



 

 www.CustomReportSharing.com 

From the folks who brought you High Rankings!



Photo
- - - - -

Googlebot Found An Extremely High Number Of Urls On Your Site


  • Please log in to reply
7 replies to this topic

#1 Andrew Gates

Andrew Gates

    HR 3

  • Active Members
  • PipPipPip
  • 69 posts

Posted 30 October 2011 - 11:33 PM

I'm receiving this message in Google webmaster tool every 2-3 months
"Googlebot found an extremely high number of URLs on your site"
We have many product pages, some pages are blocked by robots.txt file.

The problem is that Google webmaster tool shows URLs in example ("Here's a list of sample URLs with potential problems") which are blocked by robots.txt

Do you know why they do it? Thanks.


#2 Jill

Jill

    Recovering SEO

  • Admin
  • 33,005 posts

Posted 31 October 2011 - 08:45 AM

Are you positive they're blocked via robots.txt?

If so, then I wouldn't worry about it. But I would double and triple check that you've done that correctly.

#3 Michael Martinez

Michael Martinez

    HR 10

  • Active Members
  • PipPipPipPipPipPipPipPipPipPip
  • 5,157 posts
  • Location:Georgia

Posted 31 October 2011 - 02:17 PM

robots.txt only tells Googlebot not to fetch those pages. It FINDS them through links on other pages, or through an XML sitemap. Make sure your XML sitemap doesn't include pages you're telling Googlebot and/or Bingbot not to crawl. It's okay to have navigational links that point to blocked pages.

#4 OldWelshGuy

OldWelshGuy

    Work is Fun

  • Moderator
  • 4,713 posts
  • Location:Neath, South Wales, UK

Posted 31 October 2011 - 05:22 PM

I was about to post the same as MM. make sure you are not generating a list of pages someplace on your site. Many shopping carts will do this. Seems like google is gearing up for a deep crawl every couple of months, and then generating this error.

Are you by any chance submitting these urls to google products? that will do it for you.

#5 Andrew Gates

Andrew Gates

    HR 3

  • Active Members
  • PipPipPip
  • 69 posts

Posted 31 October 2011 - 09:30 PM

QUOTE(Jill @ Nov 1 2011, 12:45 AM) View Post
Are you positive they're blocked via robots.txt?

If so, then I wouldn't worry about it. But I would double and triple check that you've done that correctly.


I checked every page in webmaster tool (Crawler access) which shows is it blocked or not. It says that pages are blocked.
This is why I'm surprised.


#6 Andrew Gates

Andrew Gates

    HR 3

  • Active Members
  • PipPipPip
  • 69 posts

Posted 31 October 2011 - 09:33 PM

QUOTE(Michael Martinez @ Nov 1 2011, 06:17 AM) View Post
robots.txt only tells Googlebot not to fetch those pages. It FINDS them through links on other pages, or through an XML sitemap. Make sure your XML sitemap doesn't include pages you're telling Googlebot and/or Bingbot not to crawl. It's okay to have navigational links that point to blocked pages.


We don't have such links in xml map, but there are many links from other pages on our web site. Though all links to those pages are "nofollow" links.
Why they still keep such links in database?


#7 Michael Martinez

Michael Martinez

    HR 10

  • Active Members
  • PipPipPipPipPipPipPipPipPipPip
  • 5,157 posts
  • Location:Georgia

Posted 01 November 2011 - 12:30 PM

I have seen Google keep links for years, but maybe that's only because it still finds them in rare, hard-to-find, obscure documents.

You should be careful about using "Nofollow" so extensively across your Website, however. It is essentially spreading your PageRank across the Web.

#8 chrishirst

chrishirst

    A not so moderate moderator.

  • Moderator
  • 7,103 posts
  • Location:Blackpool UK

Posted 02 November 2011 - 07:59 AM

QUOTE
but there are many links from other pages on our web site. Though all links to those pages are "nofollow" links.
Why???




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

SPAM FREE FORUM!
 
If you are just registering to spam,
don't bother. You will be wasting your
time as your spam will never see the
light of day!