Jump to content

  • Log in with Facebook Log in with Twitter Log In with Google      Sign In   
  • Create Account

Subscribe to HRA Now!


Are you a Google Analytics enthusiast?

Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE! 




From the folks who brought you High Rankings!


Custom 404 Page Disallowed In The Robots.txt?

  • Please log in to reply
4 replies to this topic

#1 DJKay


    HR 5

  • Active Members
  • PipPipPipPipPip
  • 355 posts

Posted 25 July 2011 - 01:37 PM


Hope you all are well. I tried researching this first through the forum, but did not find anything that applies, but had fun reading some good old pinned posts:) Been out searching on the web too...did not come up for what I was looking for.

I have a question. For some reason, I am remembering that I somewhere I saw in a forum, went to a conference and heard..read somewhere etc. that it was a 'best practice' for the custom 404 page on a web site to be disallowed in the robots.txt so it would not get crawled. I think I am remembering the reason is that it could open you up to hacker attacks, but I could be wrong. Is this a best practice? Does it make no difference whatsoever?

The reason I am asking is that I am doing some clean up on all our sites, reviewing them, looking at a bunch of things, like how their robots.txt are constructed and is there anything that needs to be fixed or added. And when I was thinking about this over the weekend, for some reason, this popped in my brain.

Any help or input on this is appreciated. Candidly, I am at the point in my career that if I learned something a while ago, I don't always remember it. So I may have known the answer to this question at one time. Since I am learning new things all the time, its like the new information keeps being put in and stuff is falling out the other side because there is only so much room...smile.gifsmile.gif

Thanks in advance. DJKay

#2 Alan Perkins

Alan Perkins

    Token male admin

  • Admin
  • 1,648 posts
  • Location:UK

Posted 25 July 2011 - 02:00 PM

Hi DJKay

Best practice is for the custom 404 page to return a 404. This stops your site appearing infinitely large and full of dupe content. I believe this is what you vaguely remember.

As for disallowing the custom 404 page itself, again, best practice would not be to redirect to a custom 404 page, but to serve a 404 at the URL that was not found. Thus the custom 404 page would be a template, not a given URL.

If you do serve a custom 404 at a given URL (i.e. not following best practice), then I see nothing wrong with disallowing that in robots.txt.

#3 DJKay


    HR 5

  • Active Members
  • PipPipPipPipPip
  • 355 posts

Posted 25 July 2011 - 03:02 PM

Hi Alan,

Yes, we are in fact serving a custom 404 page to return a 404. When I say disallowing the error page, I mean the 'custom 404 page template'.

So do we disallow the 'custom 404 page template'? Or does it matter?

Thanks, DJKay

#4 Jill


    Recovering SEO

  • Admin
  • 33,244 posts

Posted 25 July 2011 - 03:11 PM

You can't disallow a template, only a URL.

#5 DJKay


    HR 5

  • Active Members
  • PipPipPipPipPip
  • 355 posts

Posted 26 July 2011 - 08:43 AM

Okay, thanks.

QUOTE(Jill @ Jul 25 2011, 04:11 PM) View Post
You can't disallow a template, only a URL.

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

We are now a read-only forum.
No new posts or registrations allowed.