Custom 404 Pages, Will search engine see them? Options

[braveheartdesign]

I was thinking about adding a custom 404 page to my site for those pages that have moved or the visitor misspelled the URL. My question is that if I make the page keyword dense with links to the rest of my site would a search engine spider/bot find it and would they consider it in any way?

David

[Bernard]

Most likely not unless you link to it from your site. I would imagine that most custom 404 pages are orphan pages and Google will not index them (although it might keep them in the supplemental index). I would be suprised if Google gave them consideration when computing rankings, link reputation or PR.

[braveheartdesign]

That is my initial thought Bernard. So while the pages can be keyword rich and perfect for the user, from the search engine perspective they would not exist.

[Bernard]

Google might index the 404 page if there are links to non-existant pages resolving to the 404 page. I don't have that situation, so I'm not 100% sure about that.

[qwerty]

No, I think you'd need to actually link to the 404 page for the search engine to index it. It's only under those circumstances that a user agent would receive a server response of 200 (OK).

If you link to domain.com/non-existent-page.htm and the 404 comes up, the spider will get the message that that document is unavailable, not that the contents it finds on the 404 page are the contents of the document it sought.

[Scottie]

There are a ton of 404 pages in the SERP's- they get indexed when pages are renamed or moved and a link somewhere still points to the non-existant URL. A 404 page is a great way to let users know a page has moved or they've linked to or typed in a wrong URL. I highly recommend having one and at least having a limited site map on it.

It's not an SEO technique though... you don't want people landing on your 404 page; it's not very professional looking or useful. You want them landing on your real pages where the real info is.

[qwerty]

Can you show me an example of an indexed 404 page that's coming up because of a bad link? I don't think I've ever seen that, and it really isn't supposed to happen.

[Scottie]

http://www.google.com/search?as_q=404.shtm...ch=&safe=images

There used to be more- it looks like Google is doing a better job of filtering them out. I was helping a friend a while back who was very upset that her 404 page was the top ranking page for her business name...

[qwerty]

But it looks like all of those examples are the actual 404 pages, as opposed to the 404 coming up for some bad URL. At least on my sites, if I try to go to a nonexistent URL, the 404 will come up, but not at 404.shtml -- the URL in the address bar will be the nonexistent one.

But if I put up a link to my 404 page, then sure, it's going to get indexed as itself.

Sorry to be so picky :tooth: , but I'd need to see a SERP pointing to an indexed URL that looks like it's pointing to some content page, but the snippet of which is something like, "Sorry, file not found".

[Scottie]

Search for "Page not Found"

The first page gives you a lot of jokes and descriptions of error pages, but starting on page 2 you can see custom error pages with their original linked URL still showing as the result.

For example: encarta.msn.com/schoolhouse/safety.asp is result #16.


Is that what you mean?

[arlen]

Poking my nose in again, and slightly off subject as usual.

If I build a custom 404 page, somthing like the cajun chef did, can't I just tell the bots not to index it?

[Scottie]

Sure. Put in robots.txt and use the robots meta tags. But it's really not anything to worry about, IMO.

[qwerty]

Yup, that's what I mean. You win.

If I'm reading the results in WebBug correctly, that URL actually brings up a 302 response rather than a 404:

CODE

HTTP/1.1 302 Found
Connection: close
Date: Fri, 14 May 2004 22:39:57 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
P3P:CP="BUS CUR CONo FIN IVDo ONL OUR PHY SAMo TELo"
X-AspNet-Version: 1.1.4322
Location: /encnet/error/Error.aspx?mesgid=404
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Length: 152

I guess that means they didn't set this up correctly.

[Scottie]

Aw... I didn't even know it was a game! I win! I win!

Do I get a medal or a sash or something...?

For the most part, I think the engines do a good job of filtering out the error pages- you ought to have them for misdirected links and users and they are nothing to worry about from an SEO standpoint, IMO.

[Jill]

That's the only time that you'll see 404-s get indexed (theoretically). If they aren't actually serving up the right header.

If you do it right (which you should), it won't get indexed.

If you do it wrong and it does get indexed, it can be a real pain in the butt because you'll often have a string of old URLs that might still get indexed, but are simply the 404-error page.

That makes it harder for them to get the good pages, if they're busy with tons of error pages. One of my client's had this problem for awhile. I think they finally got things set up properly and the error pages are now gone.

But you definitely don't want to optimize and get a 404-page indexed. Definitely not!

Jill