Are you a Google Analytics enthusiast?
More SEO Content
Googlebot Habits Influencing Indexation?
Posted 14 December 2009 - 11:16 PM
My site has a very long tail due lots of user generated content across many topics. The trend has been the more pages that are indexed, the more visits and unique keywords we get.
Recently (6 weeks) there's been a slow trend downward on key indicators:
-great fluctuation in number of pages visited by google bot on a daily basis (e.g. 20K page spread day over day now vs 5K before)
-fewer pages in index (XML sitemap in place) (-20%)
-fewer unique keywords (-30%)
-fewer visits (-30%)
Does anyone have experience with bot visit fluctuation like this? Is this a crawl problem or something else? Other factors (unfortunately negative) I should mention are very few backlinks to these content pages, high bounce rate for these pages (we're working on it).
I appreciate your thoughts!
Posted 15 December 2009 - 11:58 AM
If so, that's normal.
Posted 15 December 2009 - 02:43 PM
But have you redesigned the site?
Are you trying to "sculpt" PageRank with "rel='nofollow'"?
Did you change the navigation?
Have you added new sections?
Have you been acquiring "easy" links using popular tricks?
Have you changed hosting services?
A lot of things could explain what you're seeing. Google itself recently announced some major changes in its service and maybe it's just recrawling the entire Web to rebuild its index.
Posted 15 December 2009 - 11:00 PM
- The overall site is 8 years old. But the section of the site that I'm refering to only launched in May. Immediately following the launch in May we had incredible SEO results for 2 months. Then we got burned by using canonical link code (because pagination was the primary way for the bot to get to content). Once that was removed, results started to go up again for 6-8 week. Then a slow decline which I was writing about above. (I didn't want to overwhelm with all this info in my original post so I tried to be brief).
- Yes we have lots of nofollow. Not really for page rank sculpting purposes but rather to direct the spider to the pages of importance for the index and to ignore pages not worthy (unfortunately many unworthy pages)
- navigation has been stable. Recently added a taxinomy-style category page(s) to help bot/users find content easily (otherwise bot had to follow deep pagination to get to all the content)
- We have lots of backlinks to the overall site, but very few to this site section and virtually none to all the user generated content pages (tens to hundreds of thousands of pages of it). Have project in place to try to grow this. But in terms of easy links/popular tricks, would always love to learn more if you have a posting about this anywhere.
- I don't think we've changed hosting services, but I will check.
If you have any additional thoughts based on this info, would love to hear! Also, happy to provide name/URL of our site (I'm a newbie here, not sure if that's common practice).
Posted 16 December 2009 - 02:26 PM
Get rid of the nofollows. You're screwing your Website. Google announced in June that it deducts an equivalent amount of PageRank from a page's outbound flow for all nofollowed links.
I argued against this nonsense approach for two years, as did other people in the community. Nofollow flag wavers proudly insisted their "tests" showed that the technique worked.
Google showed everyone their "tests" weren't revealing anything at all, as they changed how PageRank is affected by nofollow about two years ago and no one in the SEO community noticed.
Links are not water. You cannot increase the pressure in the hose by squeezing it -- that is, nofollowing some of your internal links will not in any way make crawlers visit your followed links any more often -- and it certainly won't cause those destinations to accrue more PageRank.
You're compensating for bad site architecture. Look at what the work you're doing tells about how you've screwed up your site's crawlability.
Good links are always nice to have, but you're basically trying to compensate for a very serious on-site detriment. That is asking a great deal of your link building program.
Generally speaking, any time a Website is "improved" or expanded, that is an opportunity for search engines to reassess which portions of the site are most important and valuable. The fact of reorganization or expansion itself is neither good nor bad.
But when you screw up your site's architecture and crawlability, the outcome is usually disastrous.
Google changed the way it handled PageRank for nofollowed links because their internal analyses revealed that SEOs were screwing up their clients' search visibility. I am not exaggerating. That is the precise reason why Google intervened.
The sooner you stop nofollowing, the sooner you can start to put things right.
There may be other issues that your punctured architecture is hiding, but first you need to resolve THAT problem.
Posted 16 December 2009 - 09:17 PM
Iíve thought about your comments. I want to be clear that we are not using nofollow for pagerank purposes. Iím aware of the announcement about how link juice disappears, etc. I did attend the SMX East conference and in a session on nofollow and page rank sculpting it was suggested repeatedly that while nofollow doesnít work for page rank purposes, it has benefits for crawling and indexing. That is why we did itóso the spiderís time is used efficiently to crawl only the pages valuable to the index. We have lots of pages that are dead end and have no search value (I promise) and we didnít want to waste spider time on them. So a question I have is have I interpreted this use of nofollow incorrectly? implemented it incorrectly? or is it just plain wrong? Our results are going downhill so clearly something is wrong.
Could using nofollow really result in great fluctuation in bot visitation patterns and loss of pages from the index? I wouldnít have thought so, but youíve clearly said itís screwing up our website. So the inverse... giving the bot access to tens of thousands of non-search-worth pages will ultimately increase the number of important pages in the index (even if it also increases the number of unimportant pages in the index)? Iíd really like to hear your (and anyone elseís!) thoughts on that before I go and say goodbye to nofollow for good. Hmm, and all we wanted to do was create a path for the bot to the good stuff.
Obviously youíre right about good site architecture. But frankly thatís a work in progress that doesnít go quickly and is bigger than me. Iím just trying to get as many valuable pages in the index, because my results tell me that the more we have in the index, the more keywords we rank for and the longer my tail gets.
Thank you again,
Posted 17 December 2009 - 02:43 PM
Posted 17 December 2009 - 05:39 PM
And whomever said that was wrong, provably wrong, and has been proven wrong time and time again.
To date not one person in the industry has ever been able to publish any credible report showing that use of "nofollow" to manage crawl works.
But you don't have put yourself between two arguing SEO opinions.
Just remove the nofollows.
If your problem clears up, you'll be fine.
If it gets worse or doesn't change you can put the nofollows back into place.
There is absolutely no way that dropping internal links AND prematurely dissipating your site's PageRank is going to INCREASE crawl for the rest of your site. It doesn't work that way.
ON EDIT: And let me reiterate that this is precisely why Google made its announcement in the first place. All those SEOs preaching the benefits of nofollowing internal links FAILED TO NOTICE when Google changed how PageRank is handled more than a year prior to the announcement. Their tests and their analyses SUCKED and the advice they dispense on the basis of those sucky tests and analyses also sucks.
Edited by Michael Martinez, 17 December 2009 - 05:59 PM.
Posted 21 December 2009 - 01:11 PM
I'm curious and have a follow up question: do you all feel the same way about noindex as you do nofollow? Is robots.txt the only good choice for keeping pages out of the index?
Posted 21 December 2009 - 03:30 PM
Google then started telling people they should put nofollow on paid links as a way of labeling them as advertisements. I don't run any sites containing paid links, so it didn't affect me, but if I did publish paid links, I think they'd still only be to pages I felt OK about recommending.
To date, I don't think I've put nofollow on any links, either on my own sites or any clients', with one exception: I think my blog nofollows links in comments made by people who've posted less than three times. I'm OK with that, but I usually delete comments containing irrelevant links and reset relevant links to be followed, and considering the scarcity of comments on my blog, I'm not sure there are actually any nofollowed links in there.
Using nofollow as a way to try to control the flow of internal PR was bound to happen, but even if it worked (and Michael's right that nobody ever proved conclusively that it did), it always seemed to me like the kind of scheme that relied on the misuse of code and hiding information from search engines. Because of that, I figured that whether it worked in the short term or not, it was the sort of thing the search engines would eventually frown upon. If I implemented nofollow on client sites, at some point in the future that would put me in a position of either charging clients to undo something I'd already charged them to do, or fixing it for free (and if I felt really guilty, refunding them whatever they'd already paid me to implement it). That's the kind of thing that puts a given practice on my "I don't do that" list.
If you've got junky pages you want to keep out of the index, that's what the robots exclusion protocol was designed to handle. You can also limit the number of pages that link to the ones you don't like. Assuming you don't want to simply delete these pages, I'm sure they're only relevant to people who are interested in the information on certain pages, so just link to them from those pages. When you link to a page from your home page, it sends the message that the content you're linking to is of interest to the people reading your home page. If you link to it from every page, that means it's potentially of interest to everyone reading any of your pages. As such, it's important. Search engines pick up on that message and they're more likely to index the page you're linking to.
If you've got some deep page that's of interest to a limited subset of your audience, it doesn't need a link from every page. Only the people who are looking for lederhosen are going to be interested in your red lederhosen with brass bells on the suspenders. And if you've got a special page about replacement brass bells just for the red lederhosen (they tend to get damaged in the laundry), then that's probably only going to be of interest to the people on the page about the red lederhosen with brass bells on the suspenders. If you only link to it from there, I think you can feel pretty safe in assuming the About Us page that's linked to from every page is more likely to get indexed than the replacement brass bells for red lederhosen page.
Posted 21 December 2009 - 06:21 PM
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users