Jump to content

  • Log in with Facebook Log in with Twitter Log In with Google      Sign In   
  • Create Account

Subscribe to HRA Now!

 



Are you a Google Analytics enthusiast?

Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE! 

 



 

 www.CustomReportSharing.com 

From the folks who brought you High Rankings!



Photo
- - - - -

Googlebot Habits Influencing Indexation?


  • Please log in to reply
10 replies to this topic

#1 BzzKristen

BzzKristen

    HR 1

  • Members
  • Pip
  • 5 posts

Posted 14 December 2009 - 11:16 PM

Question about Googlebot habits and indexation...

My site has a very long tail due lots of user generated content across many topics. The trend has been the more pages that are indexed, the more visits and unique keywords we get.

Recently (6 weeks) there's been a slow trend downward on key indicators:
-great fluctuation in number of pages visited by google bot on a daily basis (e.g. 20K page spread day over day now vs 5K before)
-fewer pages in index (XML sitemap in place) (-20%)
-fewer unique keywords (-30%)
-fewer visits (-30%)

Does anyone have experience with bot visit fluctuation like this? Is this a crawl problem or something else? Other factors (unfortunately negative) I should mention are very few backlinks to these content pages, high bounce rate for these pages (we're working on it).

I appreciate your thoughts!
Thx,
Kristen

#2 Jill

Jill

    Recovering SEO

  • Admin
  • 33,009 posts

Posted 15 December 2009 - 11:58 AM

Guessing it's a pretty new site?

If so, that's normal.

#3 Michael Martinez

Michael Martinez

    HR 10

  • Active Members
  • PipPipPipPipPipPipPipPipPipPip
  • 5,157 posts
  • Location:Georgia

Posted 15 December 2009 - 02:43 PM

You really haven't provided much information. Like Jill says, if this is a relatively new site (less than a year old) that sounds like normal behavior.

But have you redesigned the site?

Are you trying to "sculpt" PageRank with "rel='nofollow'"?

Did you change the navigation?

Have you added new sections?

Have you been acquiring "easy" links using popular tricks?

Have you changed hosting services?

A lot of things could explain what you're seeing. Google itself recently announced some major changes in its service and maybe it's just recrawling the entire Web to rebuild its index.

#4 BzzKristen

BzzKristen

    HR 1

  • Members
  • Pip
  • 5 posts

Posted 15 December 2009 - 11:00 PM

Thx very much for your thoughts! To answer your questions:
- The overall site is 8 years old. But the section of the site that I'm refering to only launched in May. Immediately following the launch in May we had incredible SEO results for 2 months. Then we got burned by using canonical link code (because pagination was the primary way for the bot to get to content). Once that was removed, results started to go up again for 6-8 week. Then a slow decline which I was writing about above. (I didn't want to overwhelm with all this info in my original post so I tried to be brief).
- Yes we have lots of nofollow. Not really for page rank sculpting purposes but rather to direct the spider to the pages of importance for the index and to ignore pages not worthy (unfortunately many unworthy pages)
- navigation has been stable. Recently added a taxinomy-style category page(s) to help bot/users find content easily (otherwise bot had to follow deep pagination to get to all the content)
- We have lots of backlinks to the overall site, but very few to this site section and virtually none to all the user generated content pages (tens to hundreds of thousands of pages of it). Have project in place to try to grow this. But in terms of easy links/popular tricks, would always love to learn more if you have a posting about this anywhere.
- I don't think we've changed hosting services, but I will check.

If you have any additional thoughts based on this info, would love to hear! Also, happy to provide name/URL of our site (I'm a newbie here, not sure if that's common practice).

Thank you!
Kristen

#5 Michael Martinez

Michael Martinez

    HR 10

  • Active Members
  • PipPipPipPipPipPipPipPipPipPip
  • 5,157 posts
  • Location:Georgia

Posted 16 December 2009 - 02:26 PM

QUOTE(BzzKristen @ Dec 15 2009, 08:00 PM) View Post
- Yes we have lots of nofollow. Not really for page rank sculpting purposes but rather to direct the spider to the pages of importance for the index and to ignore pages not worthy (unfortunately many unworthy pages)


Get rid of the nofollows. You're screwing your Website. Google announced in June that it deducts an equivalent amount of PageRank from a page's outbound flow for all nofollowed links.

I argued against this nonsense approach for two years, as did other people in the community. Nofollow flag wavers proudly insisted their "tests" showed that the technique worked.

Google showed everyone their "tests" weren't revealing anything at all, as they changed how PageRank is affected by nofollow about two years ago and no one in the SEO community noticed.

Links are not water. You cannot increase the pressure in the hose by squeezing it -- that is, nofollowing some of your internal links will not in any way make crawlers visit your followed links any more often -- and it certainly won't cause those destinations to accrue more PageRank.

QUOTE
- navigation has been stable. Recently added a taxinomy-style category page(s) to help bot/users find content easily (otherwise bot had to follow deep pagination to get to all the content)


You're compensating for bad site architecture. Look at what the work you're doing tells about how you've screwed up your site's crawlability.

QUOTE
- We have lots of backlinks to the overall site, but very few to this site section and virtually none to all the user generated content pages (tens to hundreds of thousands of pages of it). Have project in place to try to grow this. But in terms of easy links/popular tricks, would always love to learn more if you have a posting about this anywhere.


Good links are always nice to have, but you're basically trying to compensate for a very serious on-site detriment. That is asking a great deal of your link building program.

Generally speaking, any time a Website is "improved" or expanded, that is an opportunity for search engines to reassess which portions of the site are most important and valuable. The fact of reorganization or expansion itself is neither good nor bad.

But when you screw up your site's architecture and crawlability, the outcome is usually disastrous.

Google changed the way it handled PageRank for nofollowed links because their internal analyses revealed that SEOs were screwing up their clients' search visibility. I am not exaggerating. That is the precise reason why Google intervened.

The sooner you stop nofollowing, the sooner you can start to put things right.

There may be other issues that your punctured architecture is hiding, but first you need to resolve THAT problem.



#6 BzzKristen

BzzKristen

    HR 1

  • Members
  • Pip
  • 5 posts

Posted 16 December 2009 - 09:17 PM

Iíd first like to thank you for the time/energy youíve taken responding to my questions and situation. Itís greatly appreciated. I hope others are benefiting from this as well.

Iíve thought about your comments. I want to be clear that we are not using nofollow for pagerank purposes. Iím aware of the announcement about how link juice disappears, etc. I did attend the SMX East conference and in a session on nofollow and page rank sculpting it was suggested repeatedly that while nofollow doesnít work for page rank purposes, it has benefits for crawling and indexing. That is why we did itóso the spiderís time is used efficiently to crawl only the pages valuable to the index. We have lots of pages that are dead end and have no search value (I promise) and we didnít want to waste spider time on them. So a question I have is have I interpreted this use of nofollow incorrectly? implemented it incorrectly? or is it just plain wrong? Our results are going downhill so clearly something is wrong.

Could using nofollow really result in great fluctuation in bot visitation patterns and loss of pages from the index? I wouldnít have thought so, but youíve clearly said itís screwing up our website. So the inverse... giving the bot access to tens of thousands of non-search-worth pages will ultimately increase the number of important pages in the index (even if it also increases the number of unimportant pages in the index)? Iíd really like to hear your (and anyone elseís!) thoughts on that before I go and say goodbye to nofollow for good. Hmm, and all we wanted to do was create a path for the bot to the good stuff.

Obviously youíre right about good site architecture. But frankly thatís a work in progress that doesnít go quickly and is bigger than me. Iím just trying to get as many valuable pages in the index, because my results tell me that the more we have in the index, the more keywords we rank for and the longer my tail gets.

Thank you again,
Kristen


#7 Jill

Jill

    Recovering SEO

  • Admin
  • 33,009 posts

Posted 17 December 2009 - 02:43 PM

Sounds like you should be using your robots.txt file for that, not nofollow. They are different things for different purposes.

#8 Michael Martinez

Michael Martinez

    HR 10

  • Active Members
  • PipPipPipPipPipPipPipPipPipPip
  • 5,157 posts
  • Location:Georgia

Posted 17 December 2009 - 05:39 PM

QUOTE(BzzKristen @ Dec 16 2009, 06:17 PM) View Post
... I did attend the SMX East conference and in a session on nofollow and page rank sculpting it was suggested repeatedly that while nofollow doesnít work for page rank purposes, it has benefits for crawling and indexing. ..


And whomever said that was wrong, provably wrong, and has been proven wrong time and time again.

To date not one person in the industry has ever been able to publish any credible report showing that use of "nofollow" to manage crawl works.

But you don't have put yourself between two arguing SEO opinions.

Just remove the nofollows.

If your problem clears up, you'll be fine.

If it gets worse or doesn't change you can put the nofollows back into place.

There is absolutely no way that dropping internal links AND prematurely dissipating your site's PageRank is going to INCREASE crawl for the rest of your site. It doesn't work that way.

ON EDIT: And let me reiterate that this is precisely why Google made its announcement in the first place. All those SEOs preaching the benefits of nofollowing internal links FAILED TO NOTICE when Google changed how PageRank is handled more than a year prior to the announcement. Their tests and their analyses SUCKED and the advice they dispense on the basis of those sucky tests and analyses also sucks.

Edited by Michael Martinez, 17 December 2009 - 05:59 PM.


#9 BzzKristen

BzzKristen

    HR 1

  • Members
  • Pip
  • 5 posts

Posted 21 December 2009 - 01:11 PM

Thank you both for the advice. I'm taking a leap of faith here....the nofollows are coming off tomorrow. I will be in touch after the new year to report back on the early results. I genuinely believe that a few of them at least were very good to have, but I'm willing to go full-in with the solution for now.

I'm curious and have a follow up question: do you all feel the same way about noindex as you do nofollow? Is robots.txt the only good choice for keeping pages out of the index?

Thank you,
Kristen

#10 qwerty

qwerty

    HR 10

  • Moderator
  • 8,628 posts
  • Location:Somerville, MA

Posted 21 December 2009 - 03:30 PM

I agree with Michael. Nofollow was created as a way of saying that the person publishing a link didn't necessarily trust or recommend the page to which the link pointed. My immediate reaction to that (and it hasn't changed) was to question why anyone would link to a page they didn't trust or recommend in the first place.

Google then started telling people they should put nofollow on paid links as a way of labeling them as advertisements. I don't run any sites containing paid links, so it didn't affect me, but if I did publish paid links, I think they'd still only be to pages I felt OK about recommending.

To date, I don't think I've put nofollow on any links, either on my own sites or any clients', with one exception: I think my blog nofollows links in comments made by people who've posted less than three times. I'm OK with that, but I usually delete comments containing irrelevant links and reset relevant links to be followed, and considering the scarcity of comments on my blog, I'm not sure there are actually any nofollowed links in there.

Using nofollow as a way to try to control the flow of internal PR was bound to happen, but even if it worked (and Michael's right that nobody ever proved conclusively that it did), it always seemed to me like the kind of scheme that relied on the misuse of code and hiding information from search engines. Because of that, I figured that whether it worked in the short term or not, it was the sort of thing the search engines would eventually frown upon. If I implemented nofollow on client sites, at some point in the future that would put me in a position of either charging clients to undo something I'd already charged them to do, or fixing it for free (and if I felt really guilty, refunding them whatever they'd already paid me to implement it). That's the kind of thing that puts a given practice on my "I don't do that" list.

If you've got junky pages you want to keep out of the index, that's what the robots exclusion protocol was designed to handle. You can also limit the number of pages that link to the ones you don't like. Assuming you don't want to simply delete these pages, I'm sure they're only relevant to people who are interested in the information on certain pages, so just link to them from those pages. When you link to a page from your home page, it sends the message that the content you're linking to is of interest to the people reading your home page. If you link to it from every page, that means it's potentially of interest to everyone reading any of your pages. As such, it's important. Search engines pick up on that message and they're more likely to index the page you're linking to.

If you've got some deep page that's of interest to a limited subset of your audience, it doesn't need a link from every page. Only the people who are looking for lederhosen are going to be interested in your red lederhosen with brass bells on the suspenders. And if you've got a special page about replacement brass bells just for the red lederhosen (they tend to get damaged in the laundry), then that's probably only going to be of interest to the people on the page about the red lederhosen with brass bells on the suspenders. If you only link to it from there, I think you can feel pretty safe in assuming the About Us page that's linked to from every page is more likely to get indexed than the replacement brass bells for red lederhosen page.

#11 Michael Martinez

Michael Martinez

    HR 10

  • Active Members
  • PipPipPipPipPipPipPipPipPipPip
  • 5,157 posts
  • Location:Georgia

Posted 21 December 2009 - 06:21 PM

I don't think it can be put better than that, Bob. Well said.





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

SPAM FREE FORUM!
 
If you are just registering to spam,
don't bother. You will be wasting your
time as your spam will never see the
light of day!