4 replies to this topic

[DJKay]

Hello,

Hope you all are well. I tried researching this first through the forum, but did not find anything that applies, but had fun reading some good old pinned posts:) Been out searching on the web too...did not come up for what I was looking for.

I have a question. For some reason, I am remembering that I somewhere I saw in a forum, went to a conference and heard..read somewhere etc. that it was a 'best practice' for the custom 404 page on a web site to be disallowed in the robots.txt so it would not get crawled. I think I am remembering the reason is that it could open you up to hacker attacks, but I could be wrong. Is this a best practice? Does it make no difference whatsoever?

The reason I am asking is that I am doing some clean up on all our sites, reviewing them, looking at a bunch of things, like how their robots.txt are constructed and is there anything that needs to be fixed or added. And when I was thinking about this over the weekend, for some reason, this popped in my brain.

Any help or input on this is appreciated. Candidly, I am at the point in my career that if I learned something a while ago, I don't always remember it. So I may have known the answer to this question at one time. Since I am learning new things all the time, its like the new information keeps being put in and stuff is falling out the other side because there is only so much room...

Thanks in advance. DJKay

[Alan Perkins]

Hi DJKay

Best practice is for the custom 404 page to return a 404. This stops your site appearing infinitely large and full of dupe content. I believe this is what you vaguely remember.

As for disallowing the custom 404 page itself, again, best practice would not be to redirect to a custom 404 page, but to serve a 404 at the URL that was not found. Thus the custom 404 page would be a template, not a given URL.

If you do serve a custom 404 at a given URL (i.e. not following best practice), then I see nothing wrong with disallowing that in robots.txt.

[DJKay]

Hi Alan,

Yes, we are in fact serving a custom 404 page to return a 404. When I say disallowing the error page, I mean the 'custom 404 page template'.

So do we disallow the 'custom 404 page template'? Or does it matter?

Thanks, DJKay

[Jill]

You can't disallow a template, only a URL.

[DJKay]

Okay, thanks.

You can't disallow a template, only a URL.