Jump to content

  • Log in with Facebook Log in with Twitter Log In with Google      Sign In   
  • Create Account

Subscribe to HRA Now!

 



Are you a Google Analytics enthusiast?

Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE! 

 



 

 www.CustomReportSharing.com 

From the folks who brought you High Rankings!



Photo
- - - - -

Huge Drop In Links To My Website In Gwt - Due To Robots.txt?


  • Please log in to reply
7 replies to this topic

#1 Louistextrep

Louistextrep

    HR 1

  • Members
  • Pip
  • 4 posts
  • Location:Dublin, Ireland

Posted 14 June 2013 - 04:12 AM

Hi Guys,

I'm new to this forum and this is my first post. I'm not usually posting question on forums as I usually managed to get answers by myself, but I can't find the answer for the following.

I have a business website with a PR3, with around 100 pages. Last week, the links to our website in Google Webmaster Tool was approx. 800. We had a good amount of searches for long-tail keywords.

On Friday evening, our developpers uploaded by accident a demo site as long as a robots.txt with NO INDEX/NO FOLLOW. The demo site stayed live for 3 days.

On Monday, GWT alerted me that there was an issue with the robots.txt. We replaced the demo site with the current one and we obviously replaced the no index robots.txt by the previous one.

The next day, GWT displayed only 240 links to our website. The day after, 140. As a result we do not get any traffic from long-tail keywords anymore.

GWT stopped displaying the error related to the robots.txt the same day.

However, we still did not have recover from this huge drop yet (it's been a week since it happened).

So, my question is: how long does it take for Google to re-displayed the missing links and for our website to recover?

Thanks in advance for your help,

Louis



#2 qwerty

qwerty

    HR 10

  • Moderator
  • 8,695 posts
  • Location:Somerville, MA

Posted 14 June 2013 - 08:58 AM

Welcome to HR.

 

First off, please let me clarify something, just to make sure we're on the same page (so to speak): you can't have noindex, nofollow on your robots.txt file. You can only have disallow, which isn't really the same thing. So are you talking about disallowing everything via robots.txt, or adding robots meta tags of noindex, nofollow on your pages?

 

Secondly, I'm not sure that blocking robots from your pages would cause your reported backlinks to drop. I suppose it's possible, but I don't recall ever seeing that happen. Yes, if Google is aware of a link to a given page, but doesn't have crawler access to that page, it wouldn't surprise me to find that whatever benefit the link was sending the page was lessened, but the page could still show up on a SERP -- all you'd see would be its URL and a note indicating that there was no snippet because of robots.txt. But even under those circumstances, I wouldn't expect Google to stop reporting that the link exists. I mean, I've got Google continuing to report backlinks to pages that have not existed for a long time. They're telling me the pages 404 because there are links to those URLs.

 

Also, I'm not sure that the correlation between Google reporting less backlinks and long-tail traffic dropping (I assume for the keyword phrases anchoring some of those links) necessarily means causation, although it's certainly possible.

 

I think the important thing to watch is whether Google has indexed and cached all your pages since the bot was allowed back in. What are you seeing under Health > Index Status in Webmaster Tools? When you use the site: operator to get a list of pages, are they all displaying titles and snippets?

 

And I'm sure you've done this, but I better ask, just to be on the safe side: are the links that Google's no longer reporting actually still present? Could some of them have been removed, or switched to nofollow?


  • Jill likes this

#3 Louistextrep

Louistextrep

    HR 1

  • Members
  • Pip
  • 4 posts
  • Location:Dublin, Ireland

Posted 15 June 2013 - 03:22 AM

Thanks qwerty!

 

"First off, please let me clarify something, just to make sure we're on the same page (so to speak): you can't have noindex, nofollow on your robots.txt file. You can only have disallow, which isn't really the same thing. So are you talking about disallowing everything via robots.txt, or adding robots meta tags of noindex, nofollow on your pages?"

 

Yes, you are right. The robots.txt disawowed the entire directory of our website. There were also noindex & nofollow in the meta tags of the pages for 72 hours.

 

"Secondly, I'm not sure that blocking robots from your pages would cause your reported backlinks to drop. I suppose it's possible, but I don't recall ever seeing that happen. Yes, if Google is aware of a link to a given page, but doesn't have crawler access to that page, it wouldn't surprise me to find that whatever benefit the link was sending the page was lessened, but the page could still show up on a SERP -- all you'd see would be its URL and a note indicating that there was no snippet because of robots.txt. But even under those circumstances, I wouldn't expect Google to stop reporting that the link exists. I mean, I've got Google continuing to report backlinks to pages that have not existed for a long time. They're telling me the pages 404 because there are links to those URLs."

 

The backlinks drop occured the same day as GWT began to display a health issue with my website (i.e. issue with the robots.txt).

 

"Also, I'm not sure that the correlation between Google reporting less backlinks and long-tail traffic dropping (I assume for the keyword phrases anchoring some of those links) necessarily means causation, although it's certainly possible."

 

Would I be right to say that if Google stopped to index my pages, he wouldn't show my pages in the search results anymore?

 

"I think the important thing to watch is whether Google has indexed and cached all your pages since the bot was allowed back in. What are you seeing under Health > Index Status in Webmaster Tools? When you use the site: operator to get a list of pages, are they all displaying titles and snippets?"

 

There is no error message in GWT anymore. When i make a search using the "site:" operator, not all the pages appear. Only 2 blog posts still appear but we have more than 40 blog posts in total. Also, the G+ Authorship is removed (the author is not shown next to each post).

 

"And I'm sure you've done this, but I better ask, just to be on the safe side: are the links that Google's no longer reporting actually still present? Could some of them have been removed, or switched to nofollow?"

 

The vast majority of the links that have been removed still exist.

 

Hope my explanations are clearer?

 

Louis



#4 qwerty

qwerty

    HR 10

  • Moderator
  • 8,695 posts
  • Location:Somerville, MA

Posted 15 June 2013 - 08:43 AM

There is no error message in GWT anymore. When i make a search using the "site:" operator, not all the pages appear. Only 2 blog posts still appear but we have more than 40 blog posts in total.

I think that's the important point. You're not going to get rankings and organic traffic to those pages until they're back in the index. And unless there are serious issues with your information architecture, that's going to happen, but I can't give you any clue as to how long that's going to take, as the crawl rate for every site is different. You may be able to increase the crawl rate by adding content on a regular basis or getting (and/or creating) new links to the pages that aren't back in the index yet.

 

If you don't already have an XML sitemap, submitting one might help a little. If you do have one, make certain that Webmaster Tools isn't reporting any errors with the file. And by "errors," I don't mean a difference between the number of pages submitted and the number indexed -- based on what you've written, that should currently be the case. I mean warnings about the content of the sitemap, like Google telling you that there are URLs listed there that are returning error codes.



#5 Louistextrep

Louistextrep

    HR 1

  • Members
  • Pip
  • 4 posts
  • Location:Dublin, Ireland

Posted 15 June 2013 - 09:49 AM

Hi qwerty,

 

Thanks again for your reply.

 

My robots.txt file contains this:

 

User-agent: *

 
We do have a sitemap.
 
Under the "Blocked URL" section, it's written that 52 are blocked, even though the correct robots.txt has been dowloaded 3 days ago.
 
When I use the section "URLs Specify the URLs and user-agents to test against." in GWT, and I test several URLs such as Blog Posts or Pages, all of them are "allowed".
 
About the sitemap: "Valid Sitemap reference detected".
 
Everything seems back to normal, except the blocked URLs.
 
Do I just need to wait until Google index them again?
 
Louis


#6 qwerty

qwerty

    HR 10

  • Moderator
  • 8,695 posts
  • Location:Somerville, MA

Posted 15 June 2013 - 07:03 PM

Yeah, I think it's just a matter of waiting. It's probably not a problem (certainly not what's keeping your pages from being reindexed) but there is a small syntax error in your robots.txt file. A user-agent line should be followed by one or more disallow lines, even if you're indicating that nothing is disallowed. Without a disallow line, the user-agent lines means nothing. It can, in fact, be removed. Assuming you want to disallow nothing and you want to point to the sitemap, the text of the file should be:

 

User-agent: *
Disallow:
Sitemap: http://www.example.com/sitemap.xml


#7 qwerty

qwerty

    HR 10

  • Moderator
  • 8,695 posts
  • Location:Somerville, MA

Posted 17 June 2013 - 05:53 PM

Well, this is interesting: Link Count Continues To Drop On Google Webmaster Tools

 

Last Wednesday we reported complaints in the forums of webmasters noticing their links are dropping within Google Webmaster Tools. In short, the links shown in Google Webmaster Tools today versus two weeks ago was a fraction of the number.

 

 

Most likely, this is a bug and will be fixed. But it may be a change to how Google calculates links, but that seems less likely.



#8 Louistextrep

Louistextrep

    HR 1

  • Members
  • Pip
  • 4 posts
  • Location:Dublin, Ireland

Posted 18 June 2013 - 03:23 AM

Hi qwerty,

 

Thanks for this link.

 

It seems like we've recovered well from the situation we were in: we now have an even greater number of links pointing towards our website.

 

It doesn't seem however that we have fully recovered in term of our pages being reindexed by Google, but this is just a matter of time.

 

Anyway, lesson learned from our side.

 

Thanks for your help, I will come back to this forum often.

 

Louis






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

We are now a read-only forum.
 
No new posts or registrations allowed.