11 replies to this topic

[thx1138]

Hi,

I don't use a robots.txt file because there's nothing on my site that I want to exclude from the search engines, I presume this is still good practice.

However one odd thing I did notice in Google webmaster tools is in the box where your robots.txt is displayed if you have one, my entire page code is displayed, is this just the default behaviour if you don't have robots.txt?

Charlie

[Randy]

I'm not sure how the tool works, but that doesn't sound right to me Charlie.

Is the server possibly redirecting to an error page or something since there is no robots.txt file? And if so, is it giving a 404 Not Found response or is it maybe redirecting those over to the home page or an error page using a 302 or something similar?

If that's the case it might be easier to simply upload a blank robots.txt file, or one that doesn't exclude anything.

[thx1138]

I'm not sure what's going on Randy but for some reason google was using the source code of my index.htm page for what it considered to be our robots.txt file?

I've followed your advice and uploaded a blank robots.txt file, thanks for that suggestion.

Charlie

[Jill]

Sounds like you must have a 404 error page that redirects to your home page. It may also not be showing a 404 in the http header. This could in fact be what's troubling your website as G may be indexing numerous copies of your home page and filtering them all out.

Check on your 404 and get it fixed asap and you may find yourself out of the doghouse.

[thx1138]

Sounds like you must have a 404 error page that redirects to your home page. It may also not be showing a 404 in the http header. This could in fact be what's troubling your website as G may be indexing numerous copies of your home page and filtering them all out.

Check on your 404 and get it fixed asap and you may find yourself out of the doghouse.

Thanks Jill,

You're spot on, the server was redirecting failed requests to the home page, I've now created a customised 404 page and uploaded a blank robots.txt file, I hope it helps.

Charlie

[Jill]

Does the custom 404 give an actual 404 response? Make sure it's not giving a 200 OK one.

[thx1138]

Does the custom 404 give an actual 404 response? Make sure it's not giving a 200 OK one.

What a nightmare, my server control panel has an option to upload a custom error page which is of course preferable to the default one, however when I do this it returns a 200 OK response.

I've switched it back to the default error page which returns the correct 404 response however it's not exactly user friendly. No wonder Google's so fed up with my site, it probably gets caught in some sort of feedback loop when trying to spider it, I hope if this is the problem the changes I've made will have an impact in the short term.

Is there anyway to upload a customer error page and have it return the correct response?

Charlie

[Jill]

If you have .htaccess you can do it with that.

[Randy]

Following up on Jill's comment, the .htaccess instruction is simply something like:

CODE

ErrorDocument 404 /404.php

The above assumes your custom error page has been named "404.php" and is located at the root level of the site. You can change the file name and location to suit your needs. I simply always name my custom error pages after the error number because I typically have custom pages for 400, 401, 403, 404, 500, and 503 errors. Easier for me to keep straight what's what.

Offtopic

For those wondering what those are:
400 = Bad Request
401 = Unauthorized
403 = Forbidden
404 = Not Found
500 = Internal Server Error
503 = Service Unavailable

[thx1138]

Wow, thanks guys, I know I've said it before but this forum is just so very special, Randy and Jill, you're officially too good to me.

Charlie

[Jill]

Charlie, had you recently enabled that 404 thing to point to your home?

If so, that could very well have been the cause of your lost rankings.

[thx1138]

Charlie, had you recently enabled that 404 thing to point to your home?

If so, that could very well have been the cause of your lost rankings.

No, it's been like it for a few years, I think I set it like that in my naievity 3 or 4 years ago, it would have been more reassuring had I changed it recently, it would have explained a lot, though it may be that Google has tolerated it for so long and then had enough.

I have noticed now that many of my competitors have moved around as well (though not as dramatically as us) whereas I thought only I was affected so it seems there is some change propagating its way through.

Charlie